Im trying to understand the VIX Index, it gives the implied volatility of the S&P500.
Now if I was to calculate the volatility of the S&P500 using standard deviations of the returns over a period of time & compared this with the VIX index, should I expect them to be similar?
Would the VIX index be higher or lower? (Presumably you can arbitrage so it is lower?)
The VIX is basically the option market's forecast of the volatility that will be realized in the S&P500 in the next 30 days. The index is calculated (simply said) as average implied volatility of a wide range of options with average time to expiration of 30 days (see VIX White Paper at CBOE for details).
When you compare the VIX with volatility realized in the recent history (standard deviation of returns over some recent period), it will often be very close, but rarely exactly the same. Also note the length of period you choose for the historical volatility calculation can have significant effect on the result (HV in the last 20 days may be quite different from HV in the last 60 days, for example).
More often than not, the VIX is a bit higher than recent realized volatility, especially when volatility in general is very low, like in most of 2017. Sometimes it can be lower though, typically when realized volatility has just spiked (like in a flash crash) - the VIX spikes too, but less, as the market expects volatility to calm down again in the near future.
You can trade the relationship between VIX and (the future - not the past) realized volatility, using S&P500 or SPY options, or VIX futures, options or ETNs. That said, it is not that simple (VIX derivatives can move quite differently from the spot VIX index, which is not directly tradable itself). It can also be very risky and I wouldn't call it arbitrage (no guaranteed profits, as the future is unknown).
Historically, implied volatility tends to be a bit higher than subsequent realized volatility. This is known as volatility risk premium and explained as a compensation for the risk of volatility spikes / market crashes that option sellers take (and option buyers insure themselves against).
Short volatility strategies which try to capture the volatility risk premium have been very popular in the recent years, and also very profitable, thanks to the steady low-volatility bull market we've had. But these strategies often lead to heavy losses when the market turns and volatility spikes, sometimes wiping out all their previous gains. There is no guaranteed profit without risk.
The implied volatility that that VIX is based on is a measure of what the market participants expect the future volatility of the S&P 500 to be. Many people buy and sell near-term options which most experts agree depend heavily on future volatility. The VIX index uses a weighted blend of these options to back-out a reasonable estimate of what the market believes the future volatility will be.
Expected volatility in the future, just like realized volatility in the future, can certainly be higher or lower than past volatility (what you measure with standard deviation). Often you expect future volatility to be roughly similar to past volatility and there is plenty of data and models that support this. However, this is only a rough statistical relationship so no arbitrage is possible.
VIX will almost always be higher than realized (historical) volatility (RV) of the underlying for at least two reasons.
1 ) Empirically, IVOL tends to overestimate RV, commonly referred to as Volatility Risk Premium. To make the numbers comparable, you also need to shift a series because VIX is 1m forward looking as of today, and it will take the full month to have the data to compute RV.
2 ) The VIX will in almost all cases be higher than traditional ATM IVOL as can be seen in the picture below.
TL;DR
VIX incorporates the information of the shape of the vol surface into its computation (technically, it is computed like the fair variance swap strike). IVOL is the only free parameter in the Black-Scholes-Merton (BSM) model. Higher IV can be a result of compensation for tail risk. The vol surface exists mainly because asset returns have fat tails, skewness, heteroscedasticity, jumps (crashes), and so forth. None of these real-world phenomena are featured in the Black Scholes formula. The market just developed ways to account for many of the shortcomings of Black Scholes. Adding skew (the line tilts to one side) or adding kurtosis (the curve bends upward), not only adjusts for the shortcoming in Black Scholes, but also adjusts the VIX. Usually, VIX is higher, relative to ATM, the more pronounced skew and kurtosis are.
The GIF below is from this answer. For anyone interested in the details, the formula and most of the Julia code used here can be found in this answer from Quantitative Finance SE.
Since imgur does not allow larger files, I am also adding a screenshot of the GIF, so that it is easier to see the details.
