The question is: how do you quantify investment risk?
As Michael S says, one approach is to treat investment returns as a random variable. Bill Goetzmann (Yale finance professor) told me that if you accept that markets are efficient or that the price of an asset reflects it's underlying value, then changes in price represent changes in value, so standard deviation naturally becomes the appropriate measure for riskiness of an asset. Essentially, the more volatile an asset, the riskier it is.
There is another school of thought that comes from Ben Graham and Warren Buffett, which says that volatility is not inherently risky. Rather, risk should be defined as the permanent loss of capital, so the riskiness of an asset is the probability of a permanent loss of capital invested.
This is easy to do in casino games, based on basic probability such as
roulette or slots. But what has been done with the various kinds of
investment risks? My point is saying that certain bonds are "low risk"
isn't good enough; I'd like some numbers--or at least a range of
numbers--and therefore one could calculate expected payoff (in the
statistics sense).
Or can it not be done--and if not, why not?
Investing is more art than science. In theory, a Triple-A bond rating means the asset is riskless or nearly riskless, but we saw that this was obviously wrong since several of the AAA mortgage backed securities (MBS) went under prior to the recent US recession. More recently, the current threat of default suggests that bond ratings are not entirely accurate, since US Treasuries are considered riskless assets.
Investors often use bond ratings to evaluate investments - a bond is considered investment grade if it's BBB- or higher. To adequately price bonds and evaluate risk, there are too many factors to simply refer to a chart because things like the issuer, credit quality, liquidity risk, systematic risk, and unsystematic risk all play a factor.
Another factor you have to consider is the overall portfolio. Markowitz showed that adding a riskier asset can actually lower the overall risk of a portfolio because of diversification. This is all under the assumption that risk = variance, which I think is bunk.
I'm aware that Wall Street is nothing like roulette, but then again
there must be some math and heavy economics behind calculating risk
for individual investors. This is, after all, what "quants" are paid
to do, in part. Is it all voodoo? I suspect some of it is, but not all
of it.
Quants are often involved in high frequency trading as well, but that's another note. There are complicated risk management products, such as the Aladdin system by BlackRock, which incorporate modern portfolio theory (Markowitz, Fama, Sharpe, Samuelson, etc) and financial formulas to manage risk.
Crouhy's Risk Management covers some of the concepts applied.
I also tend to think that when people point to the last x number of
years of stock market performance, that is of less value than they
expect. Even going back to 1900 provides "only" 110 years of data, and
in my view, complex systems need more data than those 40,500 data
points. 10,000 years' worth of data, ok, but not 110.
Any books or articles that address these issues, or your own informed
views, would be helfpul.
I fully agree with you here. A lot of work is done in the Santa Fe Institute to study "complex adaptive systems," and we don't have any big, clear theory as of yet. Conventional risk management is based on the ideas of modern portfolio theory, but a lot of that is seen to be wrong. Behavioral finance is introducing new ideas on how investors behave and why the old models are wrong, which is why I cannot suggest you study risk management and risk models because I and many skilled investors consider them to be largely wrong.
There are many good books on investing, the best of which is Benjamin Graham's The Intelligent Investor. Although not a book on risk solely, it provides a different viewpoint on how to invest and covers how to protect investments via a "Margin of Safety."
Lastly, I'd recommend Against the Gods by Peter Bernstein, which covers the history of risk and risk analysis. It's not solely a finance book but rather a fascinating historical view of risk, and it helps but many things in context.
Hope it helps!