What is the point of comparing a security portfolio's returns with a benchmark? You can find out whether you incurred an opportunity cost for not investing in the benchmark, but isn't the benchmark as arbitrary a collection of securities as any other portfolio? You might as well compare your returns to any other randomly selected portfolio's. Some years your portfolio may perform better than the benchmark, and some years it may be the other way around. It seems that what really matters is whether you're making or losing money, not whether you're beating the S&P 500.
Markets tend to go up over time, so most things you could buy would make money. A benchmark is meant to represent the market as a whole (or a subset that is relevant to what you are trading), so you can tell if your specific choices helped or hurt your return.
As an example, say you pick two financial stocks, Citi and Goldman. They get you a return of 10% for the year, so you think you made good choices. But if the financial sector as a whole had a return of 20%, your choices weren't actually that great.
Yes an index is by definition any arbitrary selection.
In general, to measure performance there are 2 ways:
By absolute return - meaning you want a positive return at all times ie. 10% is good. -1% is bad.
By relative return - this means beating the benchmark. For example, if the benchmark returns -20% and your portfolio returns -10%, then it has delivered +10% relative returns as compared to the benchmark.
Some years your portfolio may perform better than the benchmark, and some years it may be the other way around.
Without a benchmark you will never know. And by the way if you choose poorly, you will never beat the benchmark. If the benchmark goes up 20% but your fund/investment only went up 3% you did make money, but you might want to reevaluate your strategy.
One reason it matters whether or not you're beating the S&P 500 (or the Wilshire 5000, or whatever benchmark you choose to use) is to determine whether or not you'd be better off investing in an index fund (or some other investment vehicle) instead of pursuing whatever your current investment strategy happens to be.
Even if your investment strategy makes money, earning what the S&P 500 has averaged over multiple decades (around 10%) with an index fund means a lot more money than a 5% return with an actively managed portfolio (especially when you consider factors like compound interest and inflation).
I use the S&P 500 as one of my criteria for judging how well (or poorly) my financial adviser is doing for me. If his recommendations (or trading activity on my behalf, if authorized) are inferior to the S&P 500, for too long, then I have a basis to discontinue the relationship.
Check out this Wikipedia entry on stock market indices. There are legitimate criticisms, but on the whole I think they are useful.
As an aside, the reason I point to index funds specifically is that they are the one of the lowest-cost, fire-and-forget investment strategies around. If you compare the return of the S&P 500 index over multiple decades with most actively managed mutual funds, the S&P 500 index comes out ahead.