I'm about 17 and my Vanguard Roth IRA is now ready for an investment. I was wondering which of the above two funds would be suited as a retirement fund, or if there is any significant difference at all over the long term.
I don't know the specifics of those two funds, but this is a general answer for comparing different investment strategies in your situation.
The first thing to look at is the expense ratio of each fund. If a fund were to appreciate at a long term average of 7%, but has an expense ratio of 0.5%, then your return on investment will be about 6.5%. Especially for long-term investment, favor funds with low expense ratios.
Higher risk -> higher reward
If your retirement is ~50 years away and it is likely you will not want to withdraw those funds earlier, you should favor the riskier fund. In general, the higher the risk, the higher the reward. If you can ride out the ups and downs this risk tends to pay off. A long-term investment is generally less risky than a short-term investment.
Lower reward -> lower risk
If you think you might need some of that money before retirement (e.g. to pay for college or grad school, or as a down payment on a home) you may want to choose a more conservative fund with lower volatility. That way if the market crashes between now and when you need that money, you won't be out as much as if you had invested in a higher-risk fund.
Past performance is not future performance
What I recommend you don't consider is the past performance of the fund. In general, past perfomance does not predict future performance . Fund managers with many different funds will of course get lucky with some of their funds and unlucky with others, and might tout their highest-performing funds as evidence of their investment prowess, even though these funds tend to regress back towards the mean .
The main difference between these two funds is the composition.
Vanguard Total Stock Market is a 100% equity fund that is intended to track the performance of the "overall stock market", including large-cap stocks that are represented by the S&P 500 and small-cal stocks that are riskier but generally have larger returns.
Vanguard STAR is a "fund of funds", meaning its strategy is to buy other vanguard funds, maintaining a mixture of about 60-70% stock funds and 30-40% bond funds.
Since bonds are lower risk (and lower reward) that stocks, STAR will have lower average return but will have lower risk, meaning it will not swing up and down as much as the stock fund.
The STAR fund is a managed fund, so it has a slightly higher expense ratio, but both are low (0.32% vs 0.14%). So the expense ratio is not a significant decision point since they are close and the funds are not equivalent.
So it comes down to your risk tolerance. If you can handle larger swings in value in return for the potential of better performance over the long run, then the stock fund might be better for you.
That said, either is appropriate for a retirement fund - it just matters what risk/return profile you are looking for.