What evidence exists for claiming that you cannot beat the market?

Question

There seems to be a common sentiment that no investor can consistently beat the market on returns. What evidence exists for or against this?

I have never seen any study or credible book that claims beating the market is possible, and justifies this claim. I have however seen many claim the opposite. In my own experience, I have also very rarely seen any strategies or mutual funds that exceed index returns in a way that can't be explained by luck and coincidence.

To make the question less complicated, suppose I am asking only about an investor who purchases stocks, bonds, ETFs and options in the North American market. The investor starts every year with $10,000, makes trades with a typical fee/commission scheme, harvests return at the end of the year, and then starts again with $10,000 (any gains are thus taken by the investor as profit, and any losses are covered by her out of pocket). She pays tax as if she is a single US citizen living in a state with no income tax, and investment of this $10,000 is her sole income. The benchmark is $10,000 invested in the S&P 500 through a zero-commission ETF.

Is there really no investment strategy that would make it likely for this investor to consistently outperform her benchmark?

Answer 1

The reason for this is arbitrage. In an free and open market, investments that are certain to generate above-average profits would do so by being sold cheaply, while having a high return on investment after that. But in a free market, prices are set by supply and demand. There is a high demand and little supply for investments that would certainly outperform the market. The demand is in fact so high, that the purchase price rises to the point of eliminating that excess return. And with high-frequency automated trading, that price hike is instant. But who would even want to sell such guaranteed outperformers in the first place?

Of course, there are uncertainties associated with stocks, and individual stocks therefore move independently. As "the market" is an average, some stocks will therefore beat the market over certain time periods. That's random statistical variation.

The only realistic path to above-average returns is to accept higher risks. As discussed above, nobody wants to sell you safe bets. But risky bets are another matter. Different actors will price risk differently. If you aren't worried much about risk, you can pick up stocks that are cheap by your standards. That is possible only because such stocks aren't cheap by risk-averse standards.

Looking a bit deeper, we see that arbitrage works in a free market because there's essentially perfect information. But risk is precisely the absence of such information, and that can lead to price variations. Yet, as the lack of information means a lack of certainty, you can't use this to reliably beat the market.

Answer 2

common sentiment that no investor can consistently beat the market on returns.

I guess it's more like very few investors can beat the market, the vast majority cannot / do not.

What evidence exists for or against this?

Obviously we can have a comparison of all investors. If we start taking a look at some of the Actively Managed Funds. Given that Fund Managers are experts compared to common individual investors, if we compare this, we can potentially extend it more generically to others. Most funds beat the markets for few years, as you keep increasing the timeline, i.e. try seeing 10 year, 15 year, 20 year returns (this is easy as the data is available), you would realize that no fund consistently beat the index. A few years are quite good, a few years are quite bad. On average, most funds were below market returns especially if one compares on longer terms or 10 - 20 years. Hence the perception.

Of course we all know Warren Buffett has beat the market by leaps and bounds. After the initial success, people like Warren Buffett develop the power of "Self Fulfilling Prophecy". There would be many other individuals.

Answer 3

No such evidence exists, because many people do beat the market. And many people fail to earn market rate of return. The way you achieve the former is generally to take risks that also increase the likelihood of the latter. The amount of time and effort you invest may bias that result, but generally risk and potential reward tend to track pretty closely since everyone else is making the same evaluations.

You can't prove a negative. We can't prove unicorns don't exist either. We can advise you that hunting for one is probably not productive; many others have been trying, and if there was one we'd probably have seen at least something that encourages us to continue looking. Not impossible, but the evidence is far from encouraging.

Market-rate-of-return can be achieved fairy reliably with minimal risk and minimal effort, and at mostly long-term tax rates. I consider that sufficient for my needs. Others will feel otherwise.

Answer 4

"Will the investor beat the benchmark for a given period" will follow a Bernoulli distribution -- each period is a coin toss, and heads mean the investor beat the market for that period. I can't prove the negative that there is no investor ever whose probability function p = 1, but you can statistically expect a number of individual investors with p ~ 0.5 to have a sequence of many heads in a row, as a function of the total population.

By example, my father explained investment scams and hot-hand theory to me this way when I was younger: Imagine an investor newsletter which mails out to a mailing list of 1024 prospects (or alternately, a field of 1024 amateur investor bloggers in a challenge). Half the letters or bloggers state AAPL will go up this week, half that AAPL will go down this week. In the newsletter case, next week ignore the people we got wrong. In the blogger case, they're losers, so we don't pay attention to them. Next week, similar split: half newsletters or bloggers claim GOOG go up, half GOOG go down. This continues for a 10 week cycle.

Now, in week 10: the newsletter has a prospect they have hit correct 10x in a row: how much will he pay for a subscription? Or, one amateur investor blogger has been on a 10 week winning streak and wins the challenge, so of course let's give her a CNBC show after Jim Cramer. No matter what, next week, this newsletter or investor is shooting 50-50.

How do you know this person is not the statistically expected instance backed up by a pyramid of 1023 Bernoulli distribution losers? Alternately, if you think you're going to be the winner, you've got a 1/1024 shot.

Answer 5

There seems to be a common sentiment that no investor can consistently beat the market on returns. What evidence exists for or against this?

First off, even if the markets were entirely random there would be individual investors that would consistently beat the market throughout their lifetime entirely by luck. There are just so many people this is a statistical certainty. So let's talk about evidence of beating the market due to persistent skill.

I should hedge by saying there isn't a lot of good data here as most understandably most individual investors don't give out their investment information but there are some ok datasets. There is weak evidence, for instance, that the best individual investors keep outperforming ¹ and interestingly that the trading of individual investors can predict future market movements ². Though the evidence is more clear that individual investors make a lot of mistakes ³ and that these winning portfolios are not from commonly available strategies and involve portfolios that are much riskier than most would recommend ¹.

Is there really no investment strategy that would make it likely for this investor to consistently outperform her benchmark?

There are so many papers ^{4 5 6} (many reasonable even) out there about how to outperform benchmarks (especially risk-adjusted basis). Not too mention some advisers with great track records and a sea of questionable websites. You can even copy most of what Buffett does if you want.

Remember though that the average investor by definition makes the average "market" return and then pays fees on top of that. If there is a strategy out there that is obviously better than the market and a bunch of people start doing it, it quickly becomes expensive to do and becomes part the market. If there was a proven, easy to implement way to beat the market everyone would do it and it would be the market.

So why is it that on this site or elsewhere, whenever an active trading strategy is discussed that potentially beats the market, there is always a claim that it probably won't work?

To start with there are a large number of clearly bad ideas posed here and elsewhere. Sometimes though the ideas might be good and may even have a good chance to beat the market. Like so many of the portfolios that beat the market though and they add a lot of uncertainty and in particular, for this personal finance site, risk that the person will not be able to live comfortably in retirement (see: Enron).

There is so much uncertainty in the market and that is why there will always be people that consistently outperform the market but at the same time why there will be few, if any, strategies that will outperform consistently with any certainty.