In a perfect market, the extra losses of a high risk portfolio should exactly match the difference in returns between that and a low risk portfolio. In other words, a sufficiently diversified portfolio will return the same, on average, independent of the risk level.
So why is so much money made by providers choosing a suitable risk level? Yes, I know the market is not perfect. But we do not know in which way, so the argument still applies.
Because we don't have the rest of time to recover our losses. Since we'll all want to actually spend our money in a few decades at most, an investment that may have suffered a sharp fall at the time when we want to spend it is not as suitable as a lower-risk one.
Before getting into a practical example, I think it would be valuable to define what "risk" actually is.
Risk is by definition the magnitude of variance in possible results. When you put $1k in the bank, there is zero risk. You know that the money will be there tomorrow. When you put $1k on the stock market, there is some risk. Tomorrow, you might have .5% more, or 1% less, or 2% more, or the same amount. When you bet $1k on a horse race, there is an insane amount of risk - you might get $5k back, or you might lose it all. What we inherently consider risky, is fundamentally this: the chance that an outcome will vary substantially between many possible results.
The more different a 'good' outcome is from a 'bad' outcome, the riskier it is.
Consider risk through the lens of gambling: I'll offer you 3 bets costing you $1,000 on a fair coinflip, all with the same average payout, and you tell me whether they are all truly identical:
If you win you get $1,100; if you lose you get $900 back.
If you win you get $2,000; if you lose you get nothing.
If you win you get $4,000; if you lose you owe me an extra $2,000.
Remember - with our definition of Risk above, we can see that option 1 is the lowest risk, because both success and failure have results that are quite close together. Similarly, option 3 is incredibly high risk, because the positive and negative results are 6x your original bet apart from each other.
These are extreme examples, but they illustrate the differences of options available in the market. All investments in an efficient market theoretically have 'fair' payouts that appropriately compensate for risk. If you want to quibble over details, then bet 2 should actually pay you something like $2,050 on a win to compensate you for the extra risk, and bet 3 should actually pay you something like $4,200, to compensate you for extreme risk.
Still, it is not fair to say that the most extreme risk is always 'best'. What is best depends on your short and long term financial goals, and your ability to withstand a downturn. For example, if you are going to buy a house next year, then high risk is bad, because the market could fall before you need to make a down payment. Similarly, if you have a lot of cash to invest, it might be prudent to consider high-risk investments for a portion of your money, to capitalize on the fact that you can earn higher returns on average, while still being able to afford rent if the market collapses.
Now you might say 'I can afford to lose $1,000; I should take the most extremely risk bet, because it pays more on average'. And that's the point: in this case, you can afford to lose that money. If I told you instead, that the cost of the bet was $100,000, it wouldn't be so easy to choose bet number 2 or 3, would it?
a sufficiently diversified portfolio will return the same, on average, independent of the risk level
Not true. Theoretically, if two portfolios have equal risk but one has higher expected returns, then no one would invest in the lower-return portfolio. Similarly, if two portfolios have the same expected return, but one has lower risk, then no one would invest in the riskier portfolio.
This is evidenced int he fact that bond funds have both lower risk and lower average returns than, say, equity funds.
True, this is a very broad theory and does not always play out in reality due to numerous other variables, but on average it is reasonable to positively correlate risk and return in diversified portfolios.
It is true that uncorrelated risk can be mitigated through diversification, and anti-correlated risk can be eliminated completely.
But financial risk is rarely uncorrelated, and the degree of uncorrelation is unknown.
Beta is a measurement of the correlation of the investment with the market as a whole. An unlimited number of investments with a large Beta won't eliminate market risk, because on top of their individual random price swings, they'll have price swings correlated to the market.
In effect, diversification cannot mitigate risk if the risk is correlated. Which leads us to why we care about risk.
There is the naive answer -- that we don't have infinity to wait for our investments to recover, and we actually need to resources at some point. While this answer is correct, that is not all it is.
The Kelly Criterion mathematically places a price on risk.
If you have an investment that returns 1% per year reliably, you should invest all of your money into it. If you have an investment that returns 102% half of the time, and loses all of your money the other half, you should not invest all of your money in it. Because if you do, in 20 years you'll be completely broke.
Kelly says you should invest about 1% of your bankroll for longterm best results; and that means this risky investment is 100x worse than the safe one. To mitigate the risk, you have to keep a huge pile of money not growing.
The same logic describes why individual investors should mix volatile with less-volatile assets with a weight determined by how soon they'll need it; in effect this reflects your personal income as part of your risk mitigation.
Now, not all risk is Beta -- not all risk correlates with the market. And there are investment strategies based on this; by looking for risk that is not Beta, they can find investments that can only be safely made when you diversify with negatively correlated investments, with the hope that they are underpriced.
But such management requires effort, and that effort brings costs (and profits for those who manage it for you).
Even in perfect markets, market pricing works in the aggregate and over time, so unless you can build a portfolio that mirrors the entire market and hold it for an arbitrarily long period, your personal portfolio is not guaranteed to reflect a perfect market. Famously "Markets can remain irrational a lot longer than you and I can remain solvent." Plus, as you already conceded, markets are not perfect. Your remark that we don't know which way it's imperfect, so it doesn't matter, flies in the face of human psychology. In human terms, a hedge against our families being put out on the streets with no money for food, housing, or medicine is far more valuable then a shot at the riches of Croesus.
Consider: in 2007 the market priced S&P 500 at 1561. Two years later the market decided the S&P should be priced at 683. Now the market has decided that the 2007 price was a bargain, and prices it at 2690. Those of us who didn't need our portfolio for current living expenses were free to hang on to our stocks and saw no loss. However there were a lot of folks who retired in 2007 who absolutely relied on the value of those stocks for their living expenses. They were forced to sell at bargain basement prices and ran through their retirement savings in short order. Try chatting with those folks about rational pricing of risk.
The fallacy here is thinking that average returns is the proper metric to begin with. The average is just that: the average. It's simply a formula for taking a bunch of different numbers and summarizing them with another number. There are some situation where the average contains all the needed information about the entire set of numbers, but there are many, many situations where it doesn't. For instance, if you take two numbers from a random distribution and want to know how much, on average, the product of the two numbers will be, it's not enough to know what the average of the random distribution is.
If all you care about is what the expected value of your portfolio is, that's called "risk-neutral". Maybe you are risk-neutral, but most people aren't. Most people would prefer a 100% chance of $1M to a 99% chance of 0 and 1% of $100M. This phenomenon can be modeled as saying that each dollar amount provides a person with a certain amount of utility, and utility isn't necessarily proportional to net worth. When you spend money, you presumably start off buying whatever has the highest utility-to-money ratio for you. Once you've bought whatever has the highest such ratio, you move on to things with smaller and smaller ratios. So getting more money means that you're spending even further down the list of utility-to-money ratio, which means that as you get more money, each additional dollar is worth less utility.
I think a separate way to look at things may help explain this - the economic concept of utility. Utility describes the idea that the 'real value' of $1 depends on how it is used. And for an individual, your first $1 means more to you than your 1 Billionth dollar. Your first $1 gets you a burger at McDonald's. Your 1 Billionth dollar does nothing.
Consider a retiree with annual living expenses of $40k. Assume that person has a pension of $35k. This person will need to earn $5k / year from their investments. Now assume they have $300k in savings to invest.
They could earn guaranteed 2% in interest each year ($6k), which means they would be able to cover their living expenses.
Or, they could invest in the stock market earning an average of 7% each year (21k). However, some years they will earn nothing, or even lose money, when the stock market dips.
In scenario 1, investing low risk means they will be able to cover their cost of living. In scenario 2, investing high risk means that they either have a bit of extra cash, or, to put it bluntly, they starve.
The point is that for this person, the first $40k / year means far more to them than any additional amounts. To earn less means they starve, to earn more means... a better cable package, perhaps.
So, the market must compensate investors for taking on risk, because to those investors, the cost of losing money could mean extreme reduction in quality of life, while earning extra money would not increase quality of life so much.
Others have good answers but a simple illustrative example might help to build intuition.
Once a week you have a chance to bet on a coinflip, and you get 3 times your bet back if you win (so you either lose x or win 2x). How big portion of your bankroll should you invest on each bet?
(Note that real life investing is a lot different than this scenario. My point is just to demonstrate why betting too much can be considered actually losing in the long term, even if it one high risk bet has big monetary expected value.)
Because the expected value of that bet is huge, you should bet as much as you can, right? It's quite straightforward to show that betting as much as you can on every bet gives the best expected value for 100 bets. But let's try different values.
(Note that I'm not assuming any utility function here, I'm just showing what probably happens.)
Here are three cases with 20 simulations over 100 weeks. In the first, we bet 25 % of the bankroll each time, in the second 50 %, and in the third 75 %. (These are logarithmic plots, and the unit is your initial bankroll.)
If you bet 25 %, you are very likely to win. If you bet 50 %, you might win a lot more (note that it's a logarithmic plot), but there is also a very significant chance of losing over 100 bets and in the long run. And if you bet 75 %, there is a very slight probability (not visible for just 20 trial runs) that you win a lot, but you will very probably lose in the long term! What's happening?
You could say that this is because you need money to make money by investing: If you lose once, you not only lose your bet but also you lose all your future winnings that money might give you. Therefore, in the extreme case, the possibility of losing 0.75×bankroll outweighs the possibility of winning 1.5×bankroll in the long run.
Let's compare the two extreme cases:
If you bet 75 % each time, you'll either drop your bankroll to 0.25 times the original value or increase it to 2.5 times the original value. So you'll need on average log_{2.5}(1/0.25) = 1.51 coinflips to recover from a single loss. In the long run, you will have a lot less than 1.51 wins for each loss, so it's quite clear that with this strategy there is a very small probability that you will earn money in the long term, and the probability quickly gets smaller the higher the number of bets is. Actually, if you keep betting forever, your your bankroll will get below any arbitrary positive value with probability 1.
Betting only 25 %: With one bet you'll either drop your bankroll to 0.75 times your original bankroll or increase it to 1.5 times your original bankroll. Now you need on average only log_{1.5}(1/0.75) = 0.71 wins to recover from a single loss. In the long run, you will have a lot more than 0.71 wins for each loss, so in the long run, you will with some money.
If you bet too much on high risk, recovering from a loss is hard compared to what you gain by winning.
Higher risk means higher likelihood of a loss. Even in a perfectly diverse portfolio, a completely high risk portfolio will be more likely to lose ground than a portfolio diversified in risk also.
Diversified risk is a trade off between having a longer time to recover your money and getting higher overall returns. Generally speaking, a good high risk investment is also high growth. The company being invested in is in a weak position so they will give more for the investment, but many of the investments will go belly up.
Since many investors may need to access their finances in the short term, they can not reliably recoup their investment within the time frame they need. Because of this, there are fewer funds available for high risk investment which also helps with the negotiating power for those wanting to receive investment.
So, since high risk investment has fewer investors, the deal, on average, is a better return in exchange for the longer term investment that may be required before the diversified investment pays off and yields a return.
If investing £10,000 over ten years, most people would prefer one that guarantees to return £15,000 to one which has a 75% chance of returning £20,000 but a 25% chance of returning £0.
The point is that we feel losses (or an absence of moderate gains) more strongly than we feel large gains.
As I have commented above, I believe the answer is as follows- Because we fear loss more than we want return, the market undervalues risky assets. Thus the return is higher (on average) because of this undervaluation.
This alone can cause the effect. The many comments on other causes, such as length of investment, are not required to explain.
Given that the higher the risk the greater the above fear, the undervaluation will increase with risk. A possible investment strategy could be to put 90% (for example) in gov. bonds and the rest in as risky a set of investments as possible. This is actually the method espoused by Nassim Taleb (he of the Black Swan et al). Most will fail (max loss 100% (out of 10%)), only one or two need to succeed (max gain unlimited) to 'win'. This is also similar to venture capital methodology.
This is such a simple question and yet the answer is not simple. To some, it just seems obvious that risk imposes cost. But it's not really obvious exactly how. But it is true -- risk imposes cost.
Consider the following two options:
I definitely give you $100,000.
You flip a coin. Heads I give you $301,000, tails you give me $100,000.
You are arguing that no rational person would prefer option 1. But say that losing $100,000 would be a significant harm to your lifestyle.
Of course, you could take option 2 but acquire some insurance to cover part of your loss. That insurance will have some cost because nobody's going to give you free insurance. That cost will eat some of the extra value choice two has, potentially enough to make option 1 the better choice.
Taking risks imposes costs because you have to prepare for a wider variety of future outcomes and some of those outcomes will require you to prepare for them by doing things that impose costs on you. This makes risky investments more complex to manage and requires more reserve capital.
For example, consider an investment that has a 50% chance of a -10% day and a 50% chance of a +11% day. That sounds like a pretty good investment. But is it?
Say we invest $1,000 and it has a -10% day followed by a +11% day. Sounds awesome right! Did we just make 1% in two days?
Nope, we didn't. $1,000 minus 10% is $900. And $900 plus 11% is $999. Wait, what happened. We lost $1?!
Of course we can fix this. When we lose 10%, we can put in another $100, to get back to our $1,000. Then if we have a +11% day, we're at $1,110 on our $1,100 investment. And it's awesome, as it should be. No investment is that good.
But we had to have the $100 standing by to do it or we had to pull that $100 out of something else. The cost of doing that negates some of the benefit of the risky investments. And if we have two -10% days in a row, we either have to put more capital at risk than we wanted to or put less capital in initially just in case we had to increase our position.
In other words, risky investments require active management and reserve capital or you can take losses even when the investment performs well.
I assume you're talking about comparing two investments who have the same expected return (i.e. 10%) and different risk levels, (i.e. st dev of 7% vs. 14%).
And you're arguing that in the short term, there will be more volatility for the 14% investment compared to the 7% investment, but over the long term, they'll both return 10%... Is that right?
In this case, you're right... risk doesn't really matter.
But, the most obvious counterargument would be someone who wants to retire in 5 years and start taking income, and doesn't want to risk a sharper downturn in the market. So they don't have as long of a horizon as someone else. So risk does matter in that situation, as you'd be more likely to deplete your savings in a down market with the higher volatility investment.
I'll make my comment into an answer since it appears nobody has addressed it.
As a simple example, consider the following game (a "Markov chain"):
You start with $1.00
You toss a coin, and if it is heads, you gain p = $1.00. If it is tails, you lose p $1.00.
If you ever reach $0.00, your investment is dead.
You repeat the above for a total of totaling n tosses.
Why? Intuitively, it's because you always have a chance to go to zero no matter how much you have, but you never have a chance to come back up after reaching zero. (Mathematically, it's because the state of having zero money is an "absorbing" state of the Markov chain.)
The more you stand to gain at each toss, the faster you may also lose the same amount, and hence the faster you might reach a state of losing everything.
So this means you have to balance the probability of losing everything against the probability of making a decent-sized profit.
Which means the statement that the average return is the same regardless of p is only correct because it is always zero. (!!)
Maybe p would be more like $0.02 (or more correctly, it may be a multiplicative factor of 2%), but to a large extent, the same idea applies: if your variance is too high, then it is too easy to lose everything*, after which you cannot come back up anymore.
*Note: Yes, with a multiplicative factor you can't literally lose everything in finite time, but a $100,000 investment that turns into $100 is equivalent to one that turns into $0 for all intents and purposes.
