First of all, the annual returns are an average, there are probably some years where their return was several thousand percent, this can make a decade of 2% a year become an average of 20% .
Second of all, accredited investors are allowed to do many things that the majority of the population cannot do. Although this is mostly tied to net worth, less than 3% of the US population is registered as accredited investors. Accredited Investors are allowed to participate in private offerings of securities that do not have to be registered with the SEC, although theoretically riskier, these can have greater returns. Indeed a lot of companies that go public these days only do so after the majority of the growth potential is done. For example, a company like Facebook in the 90s would have gone public when it was a million dollar company, instead Facebook went public when it was already a 100 billion dollar company. The people that were privileged enough to be ALLOWED to invest in Facebook while it was private, experienced 10000% returns, public stock market investors from Facebook's IPO have experienced a nearly 100% return, in comparison.
Third, there are even more rules that are simply different between the "underclass" and the "upperclass". Especially when it comes to leverage, the rules on margin in the stock market and options markets are simply different between classes of investors. The more capital you have, the less you actually have to use to open a trade. Imagine a situation where a retail investor can invest in a stock by only putting down 25% of the value of the stock's shares. Someone with the net worth of an accredited investor could put down 5% of the value of the shares. So if the stock goes up, the person that already has money would earn a greater percentage than the peon thats actually investing to earn money at all.
Fourth, Warren Buffett's fund and George Soros' funds aren't just in stocks. George Soros' claim to fame was taking big bets in the foreign exchange market. The leverage in that market is much greater than one can experience in the stock market.
Fifth, Options. Anyone can open an options contract, but getting someone else to be on the other side of it is harder. Someone with clout can negotiate a 10 year options contract for pretty cheap and gain greatly if their stock or other asset appreciates in value much greater. There are cultural limitations that prompt some people to make a distinction between investing and gambling, but others are not bound by those limitations and can take any kind of bet they like.