Ok so I am adding another answer because I think it is appropriate,
So I have thought about what you put here and consulted my financial adviser, and I believe you have a few flawed assumptions. Firstly, At least with my Whole Life, it gains a guaranted amount of interest (there are non-guarantee ones) in my case this is 3%. Remember you could always take a loan for your cash balance and invest that.
You should not use a Mutual fund here, because it is an entirely different investment with a lot higher risk profile. The proper comparison (at least for the guaranteed ones) is a risk free (or semi risk free) vehicle. For this I would use either T-Bills or a portfolio trying to replicate exactly the risk profile of the company (since your only risk is the default of the life insurance company). You could even borrow against your policy and invest that in a mutual fund (not advised) or in my example T-Bills so that $750 now becomes more like $350. That $350 is more than made up for on year 31 (assuming a 30 year term policy) when you lose all death benefit. Even all this doesn't give a good profile because there are tax implications but we will ignore these. Another thing most people miss is the "gain" from the death benefit. As we talked before if you have a 1/2000 chance you will die, you can multiply that by the benefit (300k in my case) in order to get a better "Overall Asset Balance".
After doing the math this way, you can easily see, in a lot of cases, Whole Life is a more prudent investment than Term Life, especially with T-Bills at <3%. If you have stuck with the Whole life for longer than say 3 years and are going through a good company, keep it current and augment it, but I would be hesitant to say you will be saving (or making) heaps of money by switching.
The most important thing to any portfolio is balance and a lack of emotional speculation.
UPDATE:
In order to prevent another -1 because they didn't read the question, 100% whole life may be cost prohibitive. This means you still may need to use term life but that is outside of the scope of this question. Again please refer to my comment about balance.