However as I'm now in my 20s I've noticed that my broker has been getting returns of about 5% a year which I'm not really happy with.
Have you found out why are the returns about 5% a year?
I can see four reasons for such poor returns:
- Too high fees, i.e. return could be 7-8% but fees are 2-3%
- Too little diversification combined with poor luck
- Too cautious portfolio, i.e. 50% bonds and 50% stocks instead of 100% stocks
- Active management gone wrong. Sometimes actively managed funds yield more than index funds, sometimes less.
If it's (1) don't walk to the next broker; run!
If it's (2) your broker isn't particularly competent. Run to the next broker! Regarding (2), you don't want your portfolio to return by pure luck.
If it's (3) changing a broker won't solve a thing, if the next broker is going to have 50% in bonds too. You need to change the risk of your portfolio. Note that by increasing the risk of your portfolio in the hope of getting better returns, you get larger risk of poor returns, even negative returns, as well (although that should be obvious).
If it's (4) you should note that sometimes a non-index portfolio yields more than the index, sometimes less. So just by waiting the returns could improve. However, in most cases, if you have problem (4) you have problem (1) as well. An exception to this would be naturally Berkshire Hathaway which is the world's cheapest actively managed fund.
I hope that by transferring it would be private, but I want to find out first before I proceed.
If you have find out why the returns are about 5% a year, and why the next broker would have better returns, I'm sure he'll understand. If you haven't found out the reason for poor returns, in my opinion you should do your research first.
I would here like to underline that it's you whose responsibility is to ensure your money is invested with low fees (1) and good diversification (2). So, for example, don't invest in just US stocks. Invest in European, Asian, emerging markets, etc. stocks too. Prefer index funds.
And then, what returns can you expect? Since when I started investing in stocks, my money-weighted annualized rate of return is 11.1% with index return being 10.8%. I never expected this good return; my expectation was 8%. This has been during an extremely good period in global stock markets. Include one poor period, and you'll see the return go low, even negative, temporarily.