However as I'm now in my 20s I've noticed that my broker has been getting returns of about 5% a year which I'm not really happy with.
Have you found out why are the returns about 5% a year?
I can see four reasons for such poor returns:
Too high fees, i.e. return could be 7-8% but fees are 2-3%
Too little diversification combined with poor luck