Index funds are well-known to give the best long-term investment.
Not exactly. Indexes give the best long term performance when compared to actively managing investments directly in the underlying stocks. That is, if you compare an S&P500 index to trying to pick stocks that are part of it, you're more likely to succeed with blindly following the index than trying to actively beat it.
That said, no-one promises that investing in S&P500 is better than investing in DJIA, for example. These are two different indexes tracking different stocks and areas.
So when advisers say "diversify" they don't mean it that you should diversify between different stocks that build up the S&P500 index. They mean that you should diversify your investments in different areas. Some in S&P500, some in DJIA, some in international indexes, some in bond indexes, etc.
Still, investing in various indexes will likely yield better results than actively managing the investments trying to beat those indexes, but you should not invest in only one, and that is the meaning of diversification.
In the comments you asked "why diversify at all?", and that is entirely a different question from your original "what diversification is?".
You diversify to reduce the risk of loss from one side, and widen the net for gains from another.
The thing is that any single investment can eventually fail, regardless of how it performed before. You can see that the S&P500 index lost 50% of its value twice within ten years, whereas before it was doubling itself every several years. Many people who were only invested in that index (or what's underlying to it) lost a lot of money.
But consider you've diversified, and in the last 20 years you've invested in a blend of indexes that include the S&P500, but also other investments like S&P BSE SENSEX mentioned by Victor below. You would reduce your risk of loss on the American market by increasing your gains on the Indian market. Add to the mix soaring Chinese Real Estate market during the time of the collapse of the US real-estate, gains on the dollar losing its value by investing in other currencies (Canadian dollar, for example), etc.
There are many risks, and by diversifying you mitigate them, and also have a chance to create other potential gains.
Now, another question is why invest in indexes. That has been answered before on this site. It is my opinion that some methods of investing are just gambling by trying to catch the wave and they will almost always fail, and rarely will individual stock picking beat the market. Of course, after the fact its easy to be smart and pick the winning stocks. But the problem is to be able to predict those charts ahead of time.