Why is the stock market so unpredictable?

Question

Why is the stock market so unpredictable? If you think about it, most of the things we do in a day are pretty predictable. Waking up, deciding what sites to browse, etc. So if we have a good model about how people behave on a day to day basis, why can't this be transferred to stocks?

Answer 1

The stock market measures how things change from what we predicted. If only things we predicted happened, the prices of stocks wouldn't really change.

You might think, "But Apple went from a tiny company to a giant company."

Sure, but if that was predictable, we all would have bought Apple stock back when it was a tiny company (since we knew it would become a giant company) and the price back then would have been much higher. The price of Apple stock was low in the past because we didn't know how big Apple would become. As that information became apparent, the price went up.

Simply put, anything people expect to happen will already be built into the price. The changes we see come from things unexpected.

A sudden trade war with China. Trump gets elected. A hurricane hits North Carolina. A particular drug is discovered to cure a disease, or maybe one is taken off the market because it causes birth defects.

So if we have a good model about how. People behave on a day to day basis, why can't this be transferred to stocks?

It is. The price of a stock today does include everything people expect to happen. The possibility that Apple would become what it is today has been factored into Apple's price all along. But that was mixed in with other then possible outcomes that didn't come to pass.

Answer 2

If you think about, most of the things we do in a day are pretty predictable.

Really?

Waking up,

You never oversleep.

deciding what sites to browse,

Not sure how many people plan that in the morning. I do not.

So if we have a good model about how. People behave on a day to day basis, why can't his be transferred to stocks?

Because:

• You do not even come close to have a good model. You assume because you have or can name a couple of data points that is a good model. It is not even close to be an idea for a possibly model.
• Because the stock market is not only people and it is a LOT of actors all actions quite possibly quite rationally but together forming a complex chaotic system.

You can not go easily from "I know how Joe would react" to "I have a model that can model millions of different actors reliably".

Answer 3

Short answer: arbitrage.

The way I see it, you're not really betting on anything physical, nor even on how well companies perform; you're betting on what all the other investors/speculators think about all that.  And what everyone else thinks they think.  And so on, and so on…  It's a massive game of second-guessing, third-guessing, and umpteenth-guesing.

As Ben Kingley's character said in Sneakers, it's all about the information.  The markets are designed to use information as efficiently as possible.  (That's why insider trading is one of the most heinous financial crimes; it prevents the markets working efficiently.)

The result is that anything predictable cancels out.  As soon as something is predictable, traders react by buying or selling, which moves the price to counteract it.  And the more efficient the markets become, the faster and more accurately this happens — to the point where the only price movements you see are completely unpredictable.

As soon as there's a difference between what the price is, and what people think it should be or is going to be, they will take advantage of that difference (arbitrage), and the difference will disappear.  In effect, it's a very tight negative-feedback loop, filtering out everything but noise.

Few things in life are based around such a rapid and vast feedback loop, which is why it seems counterintuitive.

Answer 4

Whatever is predictable about a financial product is already factored into its price. For instance, if the company is predicted to do well, its stock becomes overvalued, and the opposite happens to companies those future looks grim.

The remaining uncertainty in a financial product after such corrections is truly random from the point of view of anyone who hasn't got a better prediction model. And if someone does, they will have the incentive to trade in a way that again factors that better knowledge into the price: they will buy products that they expect to perform well (thus increasing their price) and sell the products that they expect to perform poorly.

Answer 5

The elephant in the room missing in the other answers is the irrationality inherent in stock (and other) markets.

Sure enough: There are many measurable, rational, "real" things influencing stock value. The monetary value of a company, how we expect the market for their products to develop, how expensive the stock is compared to the company's earnings, capital cost etc.

But one of the most influential reasons to buy or sell stock — driving its value up and down — is simply that we expect that stock to rise or fall in value. Speculation is as old as the markets.

The feedback loop which is thus established is pretty obvious: Part of the demand for stock has no discernible root in physical or financial reality; instead, it is driven by what other people think, which in turn is strongly influenced by that very stock value development.

This is how bubbles expand and burst, with oscillations that are orders of magnitude larger than the effects of the actual tangible operations of a company. Such feedback loops are non-linear and correspondingly hard to predict, and predicting them involves psychology more than economics. Some people do that better than others; you may be interested in Michael Lewis' The Big Short, specifically the story of Michael Burry², or examine the point in time when Warren Buffett started buying again in 2008 while everybody else thought the world was going to end.¹

My impression is that it is better for one's nerves to buy an index paper and stick with it ;-).

---

_{¹ Finding the right point in time is like [playing chicken](https://www.urbandictionary.com/define.php?term=Play%20Chicken) which probably already the Neanderthal teenagers did with mammoths: If you are early, you are uncool; if you are late, you are dead; only if you hit the perfect spot you are cool and get the girl. And past successes are at best a weak indicator for the future.}

_{² Although truth be told, Michael Burry's shorting of CDOs was based on hard economics. In fact, his Asperger's condition not only enabled him to actually read the prospectuses, something few others could bring themselves to do; it probably shielded him against the frenzy holding other investors in its grip. That is, his relative inability to read social and psychological cues allowed him to see the underlying reality.}

Answer 6

There are two related reasons. The first is that if today's price is equal to yesterday's prices times a reward plus an appraisal error then the resulting difference equation can be proven to be intrinsically unstable, p(t+1)=Rp(t)+e(t+1). As long as the appraisal error has finite variance centered on zero, then any Frequentist statistical estimator will have no predictive power at all.

The second reason is that if an equilibrium price exists at each moment in time and the equilibrium reward plus a shock is the true reward then the distribution of returns around the equilibrium return will be the truncated Cauchy distribution if you have factored out liquidity risks, dividend risks, merger risks, and bankruptcy risks.

The Cauchy distribution is unusual in that it cannot have an average. You can always calculate the sample average but it won't ever settle down and converge to the true center of the distribution, which for stocks is the mode.

John Cook provides a graphical example of the difference of the behavior of the sample mean of a Cauchy distribution and the mean of the normal distribution. You can find it at Cauchy versus Normal

The mathematician Benoit Mandelbrot classified the behavior of random variables as having seven classifications. The type you generally encounter in day-to-day life is proper mild randomness. Equity security returns are the seventh class of randomness, which is extreme randomness.

In fact, you can predict things in equity markets. You can only use Bayesian methods. For a variety of reasons, it can be shown that it would be profoundly unwise to use a Frequentist methodology, but most do because it is what they know.

The difficulty is that most people try to predict things using Frequentist methods of proper mild randomness rather than Bayesian methods for extreme randomness. That includes professionals.

As there is no way to use MathML in this forum but I am about to produce a resource you could use though it is intended for those with doctorates in statistics, mathematics, finance, and economics. I will try and remember to post a link to it when it is in its final form. Elements of it could be used by anyone with a basic understanding of probability and basic calculus methods.

The physicist and statistical polemicist E.T. Jaynes wrote that had the first problems in statistics involved the Cauchy distribution the trajectory of the field of statistics would have been radically different.

EDIT I made an 18-minute video that partially answers your question.

Answer 7

I'm not sure why you think it is unpredictable. Good companies do well bad companies do poorly. Virtually all companies will fall when the market is overpriced and wall Street finds a reason to offload. Virtually all companies with solid earnings will rise when the market is undervalued. You see the market is very predictable when it reaches obvious valuations, both over and under. The reason you say it is unpredictable is because you are entering and exiting too frequently. If all you did was jump in at the bottom of a crash and jump out when it reached lofty valuations you would say it was very predictable

Answer 8

A first stab at an essential answer would be to suggest that people are simply not very predictable and are irrational in a great number of cases. This strikes at the fundamental assumption underlying classical economics.

Perhaps worse are the second-order effects: To be completely predictable, every actor in the market would have to be rational and completely predictable. As long as there were even a small number of irrational actors, everyone else would be working to respond and react to their surprise moves, and the others' reactions, etc., etc., and the entire market continues in an ongoing flux. E.g., How many people are prone to peer pressure (i.e., herd behavior causing inertia in the market)? Or, what is the rate of mental illness? I would argue these are fairly large numbers.

Related: Hyman Minsky was a professor in the 20th century who developed a theory whose central tenet was: "Stability breeds instability". That is: If the market is safely predictable, then people will take on more and more debt to invest in that market, and the debt burden will be pushed right to the edge of systemic collapse from any small surprise in the market. Many analysts started attending the theory quite a bit more after the 2008 crash which looked quite a bit like Minsky's theory.

A third theory: The Fractal Market Hypothesis (FMH), which suggests that a wide variety in investor goal horizons causes different investors to respond to news and events with very different reactions (with more or less sensitivity), and these differences cause volatility in the markets.

There are numerous other non-traditional economic theories in this vein, counter to the predominant Efficient Market Hypothesis (EMH), that pricing incorporates full information in a rational manner. This seems reasonable in light of the fact that traditional economic assumptions like EMH fail to show explanatory power for the regular and well-known market bubbles, panics, crashes, etc. that do in fact take place. In my view they all orbit around the rather obvious fact that people are not essentially rational or predictable in the first place.

Answer 9

A term that has been used to describe the stock market is anti-inductive. See Markets are Anti-Inductive and The Phatic And The Anti-Inductive. Our knowledge of a planet's orbit doesn't affect it. So we can learn more about an orbit, and then the orbit will be easier to predict. But with the market, learning about it changes its behavior. Whenever we learn more about the market, people act on that information, changing the market. So the more people work on trying to understand the market, the more people are affecting the market, and the harder it is to understand the market. Unlike with predicting planet's orbits, when you predict the market, you have to include the behavior of people, and people will be reacting to your analysis of them market. So your predictions have to predict the actions of people reacting to analysis that predict the behavior of people reacting to analysis that predicts ... and so on. If one person happens to be better than everyone else at analyzing the market, then they might be successful in predicting it. But people in general can't predict the market, because the average person, by definition, isn't any better than any other average person, so if one average person can figure out that a stock will go up, then every other average person can figure that out too and bid the price up. Stock market fluctuations represent the unpredicted parts of the economy. If they were predicted, then they would have already been included in the price. We can't predict market fluctuations because the fluctuations are by definition what isn't predicted.

Answer 10

There are two types of prediction problems typically.

The level 1 prediction models are for things like weather - one could predict something, wait and observe, and then correct the error for next prediction. The key aspect is the fact whether it would or would not rain on a certain day is not dependent, i.e., it does not change, due to the prediction (that it is going to rain, or it is not going to rain).

However level 2 predictions are harder - which is precisely the stock market. Even if one develops an excellent model that predicts how the stock market is going to behave, the moment somebody actually starts using the model to buy and sell stocks - the market "sees" the increased buying/selling of a certain stock, and corrects itself appropriately. Soon this excellent model becomes obsolete.

And the duration for which this excellent model predicted the stock behavior correctly could be really small, that its short term success is pretty much indistinguishable from noise.

Answer 11

Why is the stock market so unpredictable? ...most of the things we do in a day are pretty predictable.

If you mean overall and you mean in the long run then the US stock market is predictable.

It is like how people are predictable in large numbers - what a random person will do is not predictable.

At a US theme park (like Disney) people, when given a choice, will tend to turn right instead of left. That can be established as fact, because people study this stuff and that is what they've found.
That does not mean that you can predict whether a specific person will choose to go to the left or right at a specific place.

The stock market is the same:
A certified financial planner will likely tell you that, "If you will need the money in the next five years, you probably should not put it in the stock market."

Why would they say that?
Because if you look at mutual funds with 10+ year track records, you'll find that they have a positive return in almost all periods of five consecutive years of their history.
(four point something times out of five if I remember correctly)

TL;DR;
The stock market is very likely to go up if you are looking at period of five or more consecutive years.

What a particular stock will do is anybody's guess...
and they're all guessing differently...
and people change their guess from one day to the next...
and that is what sets the price...
which why a particular stock moves so erratically...
But... over time (measured in years)... the market goes up :-)

Answer 12

Stock price changes are second order predictions, which makes them less predictable than first order predictions.

Stock prices are crowd sourced first order predictions. They reflect what people predict other people will be willing to pay for the stock in the future. If "everyone" predicts a certain stock will be worth $200 in the future, "everyone" is now willing to buy that stock at close to $200, "everyone" is now willing to sell that stock at close to $200, so the predicted future price becomes the current price. (There are exceptions, related to liquidity, interest, and inflation, which I ignore for simplicity)

Answer 13

When you find a way to predict the stock market, you will probably want to use that to make a ton of money. Even if you don't try to act on it, announcing the discovery will entice someone else. Even if you keep it secret, the very fact that you discovered means it exists, therefore someone else will probably discover it eventually on their own. The market is very popular, so thousands of very smart people are constantly looking for patterns in it.

When you try to make money from the market based on some pattern, your trading creates buying and selling pressure. As per supply and demand, buying drives price up and selling drives it down. Generally, your attempt to trade on the pattern you found will result in reducing the amount of profit to be made from that pattern. This can be thought of as an arbitrage. Of course, a rational actor will keep trading the pattern until there is no profit left to be made. Eventually the particular inefficiency that gave rise to the pattern will be eliminated, courtesy of you, the speculator. This is the efficient market hypothesis in very broad strokes.

If you think of it in another way, the market is a system that pays people to make it more complex. Every time you find a predictable behavior, you trade on it, which generates a profit for you, but at the same time makes that behavior less predictable. Because people love money, they keep playing this game over and over. As a result, the market has become very complex: Every simple pattern has been arbitraged away.

Leaving aside the fallacy of your premise (we do not have good models predicting day-to-day behavior), the complexity of the market stems from two main factors:

• There are a lot of participants doing a lot of trades every day
• Everyone's behavior is influenced by what they think everyone else will do

If you consider that the second point implies infinitely recursive "mind games", it is easy to see why the market is unpredictable.

Answer 14

So if we have a good model about how. People behave on a day to day basis, why can't this be transferred to stocks?

Stocks are, in essence, just a commodity to be bought and sold. And that means you have a price.

A price is simply a value we assign something (and we tend to go based on someone being willing to pay said price, not the pure opinion of the previous owner). Prices can be volatile. Milk might cost $3 per gallon, but what if some disease wipes out half the cows that make the milk? The price of milk goes up. Is there any way to predict that? And that's assuming we only have one factor driving the price to change.

Prices contain too much information to process meaningfully:

Why prices are a mystery isn’t a mystery. The why of prices — or at least all the whys of prices — are simply unknowable. Sure, you might be able to know why person X wants $28.99 for his vintage Batman talking alarm clock while another person wants $80.99 for the same masterpiece, but you can’t know all the reasons why it originally sold for $9.99. Once you make your peace with that fact, the mystery ceases being mysterious.

What I think is fascinating is that prices aren’t unique. We think they are a category unto themselves. A price isn’t like anything else, right? Well, wrong — maybe. If prices are simply the agglomeration of disparate and complex information boiled down to a number, can’t other things be agglomerations of disparate and complex information boiled down to something else?

Anyway, the point I’m getting at is that there’s nothing — nothing — that we say, do, own, make, or believe that isn’t brimming with all of the sorts of information that go into a price.

The stock market is driven by a myriad of forces and interests. I have money in there for my retirement. Warren Buffett has money in there to make money for other people. John Doe has money in there because he likes owning a certain brand of stock. Different goals. Maybe Warren Buffett dumps his stock in Company X, which drives the price down, while John hangs on to his no matter the price. Or maybe a bug in stock trading software causes havoc:

When the market opened at 9:30 AM people quickly knew something was wrong. By 9:31 AM it was evident to many people on Wall Street that something serious was happening. The market was being flooded with orders out of the ordinary for regular trading volumes on certain stocks. By 9:32 AM many people on Wall Street were wondering why it hadn’t stopped. This was an eternity in high-speed trading terms. Why hadn’t someone hit the kill-switch on whatever system was doing this? As it turns out there was no kill switch. During the first 45-minutes of trading Knight’s executions constituted more than 50% of the trading volume, driving certain stocks up over 10% of their value. As a result other stocks decreased in value in response to the erroneous trades.

TL;DR

Stocks are based on prices (what people are willing to pay for them). Prices are based on too much information to know why they are priced the way they are at any given moment. Stocks are based on even more information than most commodities.

Answer 15

Three reasons:

1. thin margins

   Companies sell a product for price P, and buy the stuff/services needed to build the product for price 0.9P. The profit is 0.1P, rather thin. If for some reason customers' willingness to pay decreases by 10%, the whole profit is lost. Thus, in times when economic conditions are not optimal, the company cannot pay dividend. The only reason people invest in stocks is in fact it, namely the dividend. (Some stocks don't pay dividend but their value is based on the expectation that someday they could start paying dividend. Note that dividends are equivalent to share buybacks, so if a company is buying back its shares, it's effectively paying dividend.)

2. supply and demand

   Prices do vary based on supply and demand of savings instruments. At times when saving money into stocks is trendy, stocks are expensive. At times when spending all of your money is trendy, stocks are cheap. For example, during a recession a huge number of people want to access their savings, driving down prices. If you are countercyclical, you will find that during a recession, you have a huge amount of good investment opportunities at ridiculously low prices.

3. liquidity

   Stock market is actually a safer investment than e.g. the housing market. See for example this: A Long Run House Price Index: The Herengracht Index, 1628–1973 by Piet M.A. Eichholtz

   In this file, it is explained how house prices have gone both up and down. The maximum decrease was 80%. It's about the same than stocks during a very large depression. However, with stocks it's easy to diversify whereas most people have the money to purchase only 1 or perhaps 2 houses. So, due to lack of diversification it generally doesn't make sense to invest money into housing market but rather invest it into stock market.

   Why do people think stocks are unpredictable and houses are predictable, then? Because stock market is liquid. The 80% crash in real house prices happened over a period of over 35 years between 1778-1779 and 1814-1815. Stocks could crash in few days or less.