I've seen it said a number of places that investors can never beat the market. Why use investors, then? Why not just put your money in a market index fund?
Let me start by giving you a snippet of a report that will floor you. Beat the market? Investors lag the market by so much that many call the industry a scam.
This is the 2015 year end data from a report titled Quantitive Analysis of Investor Behavior by a firm, Dalbar. It boggles the mind that the disparity could be this bad. A mix of stocks and bonds over 30 years should average 8.5% or so. Take out fees, and even 7.5% would be the result I expect. The average investor return was less than half of this.
Jack Bogle, founder of Vanguard, and considered the father of the index fund, was ridiculed. A pamphlet I got from Vanguard decades ago quoted fund managers as saying that "indexing is a path to mediocrity." Fortunately, I was a numbers guy, read all I could that Jack wrote and got most of that 10.35%, less .05, down to .02% over the years.
To answer the question: psychology. People are easily scammed as they want to believe they can beat the market. Or that they'll somehow find a fund that does it for them. I'm tempted to say ignorance or some other hint at lack of intelligence, but that would be unfair to the professionals, all of which were scammed by Madoff. Individual funds may not be scams, but investors are partly to blame, buy high, sell low, and you get the results above, I dare say, an investor claiming to use index funds might not fare much better than the 3.66% 30 year return above, if they follow that path, buying high, selling low.
Edit - I am adding this line to be clear - My conclusion, if any, is that the huge disparity cannot be attributed to management, a 6.7% lag from the S&P return to what the average investor sees likely comes from bad trading. To the comments by Dave, we have a manager that consistently beats the market over any 2-3 year period. You have been with him 30 years and are clearly smiling about your relationship and investing decision. Yet, he still has flows in and out. People buy at the top when reading how good he is, and selling right after a 30% drop even when he actually beat by dropping just 22%. By getting in and out, he has a set of clients with a 30 year record of 6% returns, while you have just over 11%. This paragraph speaks to the behavior of the investor, not managed vs indexed.
The market is efficient, but it is not perfectly efficient. There are entities out there that consistently, legitimately, and significantly outperform the market because of asymmetric information (not necessarily insider trading) and their competitive advantage (access to data and proprietary, highly sophisticated models)*. I say this despite most hedge funds performing worse than their respective benchmarks. For most people (even very smart people) it makes a lot of sense to invest in index funds with a reasonable asset allocation (based on desired volatility, tax situation, rebalancing methods etc.).
* The usual example that is cited is RT's Medallion Fund because it has enjoyed quite dramatic returns. Other groups that have been successful include Citadel and Soros Fund Management.
Perhaps someone has an investment objective different than following the market. If one is investing in stocks with an intent on getting dividend income then there may be other options that make more sense than owning the whole market.
Secondly, there is Slice and Dice where one may try to find a more optimal investment idea by using a combination of indices and so one may choose to invest 25% into each of large-cap value, large-cap growth, small-cap value and small-cap growth with an intent to pick up benefits that have been seen since 1927 looking at Fama and French's work.
Most of it is probably due to ignorance and disbelief.
A few years ago, I started doing week-long trades with my IRA. For a while I would make money each time, and over the first year I had about a 20% rate of return. If you asked me if I thought I was smarter than other people in the market, I would've told you no - I just spent more time, and most people accepted a small financial penalty for not having to spend the time directly managing their portfolio.
Then I made a few poor choices, and all my previous earnings disappeared quickly. In the short term, yeah, things were great, but that didn't extrapolate out.
So now that I'm a few years into investing, I'm almost entirely in index funds.
Index funds do leech a "free ride" on the coattails of active traders.
Consider what would happen if literally everyone bought index funds. For a company there would be no motivation to excel. Get listed; all the index funds are forced to buy your stock; now sit on your derriere playing Freecell, or otherwise scam/loot the company. Go bankrupt. Rinse wash repeat. This "who cares who John Galt is" philosophy would kill the economy dead.
Somebody has to actually buy stocks based on research, analysis and value. Company managers need to actively fear, respect and court those people. They don't need to be mutual-fund managers, but they do need to be somebody. Maybe activist investors like Warren Buffett will suffice. Maybe retirement fund or endowment managers like CalPERS or Harvard can do this. Better be somebody!
I'm all for index funds... Just saying only a fraction of the market's capital can be in index funds before it starts into a tragedy of the commons.