Just ask Kelly
Yes, it can sometimes make sense to not insure as a high-net-worth individual (or corporation!), and the math that tells you when is called the Kelly Criterion.
The Kelly Criterion is on its face about how much you should bet on a positive-sum game.
Imagine you have a game where you flip a coin, and if heads you are given 3 times your bet, and if tails you lose your bet. Naively you'd think "great, I should play, and bet every dollar I have!" -- after all, it has a 50% average return on investment. You get back on average 1.5$ for every dollar you bet, so every dollar you don't bet is a 0.5$ loss.
But if you do this and you play every day for 10 years, you'll almost always end up bankrupt. Funny that.
On the other hand, if you bet nothing, you are losing out on a great investment. So under certain assumptions, you neither want to bet everything, nor do you want to bet nothing (assuming you can repeat the bet almost indefinitely).
How does using Kelly help?
The question that Kelly can answer is what percentage of your bankroll should you bet?
The typical Kelly Criterion case is where we are making a bet with positive returns, not an insurance against loss; but with a tiny bit of mathematical trickery, we can use it to determine how much you should spend on insuring against loss.
An "easy" way to understand the Kelly Criterion is that you want to maximize the logarithm of your worth after a given choice. Such a maximization results in the largest long-term value in some situations.
Let us invent an insurance case and try out the math.
Suppose you have a 1 million dollar asset. It has a 1% chance per year of being destroyed by some random event (flood, fire, taxes, pitchforks).
You can buy insurance against this for 2% of its value per year (so, 20,000$). It even covers pitchforks.
On its face this looks like a bad deal. Your expected loss is only 1%, but the cost to hide the loss is 2%?
If this is your only asset, then the loss makes your net worth 0. The log of zero is negative infinity. Under Kelly, any insurance (no matter how inefficient) is worth it. This is a bit of an extreme case, and we'll cover why it doesn't apply even when it seems like it does elsewhere.
Now suppose you have 1 million dollars in other assets. In the insured case, we always end the year with 1.98 million dollars, regardless of if the disaster happens. In the non-insured case, 99% of the time we have 2 million dollars, and 1% of the time we have 1 million dollars.
We want to maximize the expected log value of our worth. We have log(2 million - 20,000) (the insured case) vs 1% * log(1 million) + 99% * log(2 million).
Or 14.49 (insured) vs 13.80 (uninsured). The Kelly Criterion says insurance is worth it; while you could "afford" to replace your home, however because it makes up so much of your net worth, Kelly Criterion effectively says the "the financial hit - it is too painful" and you should just pay for insurance.
Now suppose you are worth 1 billion. We have log(1 billion - 20k) on the insured side, and 1%*log(999 million) + 99% * log(1 billion) on the uninsured side.
The logs of each side are 20.72 (insured) vs 21.42 (uninsured). (Note that the base of the logarithm doesn't matter, so long as you use the same base on each side; here I used the natural logarithm).
According to Kelly, we have just found a case where insurance isn't worth it. When you are worth 1 billion dollars, insuring a 1 million dollar asset with a 1% loss chance at 2% of its value isn't worth the overhead.
(Aside: You can view this as the counter-party solution. By pooling risks and capital, the insurance company acts like a "large net worth" owner who is engaged in the exact opposite deal as you are: in the limit as net worth goes to infinity, Kelly turns into a linear expected value test!)
The Kelly Criterion roughly tells you "if I took this bet every (period of time), would I be on average richer after (many repeats of this bet) than if I didn't take this bet?" When the answer is "no", it implies self-insurance is more efficient than using external insurance. The answer is going to be sensitive to the profit margin of the insurance product you are buying, and the size of the asset relative to your total wealth.
Wait a second...
Now, the Kelly Criterion can easily be misapplied. Being worth financially zero in current assets can easily ignore non-financial assets (like your ability to work, or friends, or whatever). And it presumes repeat to infinity, and people tend not to live that long.
You can patch this somewhat by incorporating "expected future earnings" at "net present value" into your net worth; to do this correctly, you'd have to include the uncertainty of those future earnings and their correlation with the insured loss. (Ie, if your house burns down, the chance you suffer burns making you unemployed (cutting your future income) is higher than if your house didn't burn down: if you are relying on future income to cushion the loss of your home, this is bad news.)
But Kelly is a good starting spot.
Note that the option of bankruptcy can easily make insurance not "worth it" for people far poorer; this is one of the reasons why banks insist you have insurance on your property.
You can use Kelly to calculate how much insurance you should purchase at a given profit margin for the insurance company given your net worth and the risk involved. This can be used in Finance to work out how much you should hedge your bets in an investment as well; in effect, it quantifies how having money makes it easier to make money.
Why does it work?
The Kelly criteria implicitly assumes a logarithmic utility function; going from 100$ net worth to 10,000$ net worth has the same "distance" as going from 10,000$ to 1,000,000$ net worth, as both are a factor of 100x.
Replacing a linear utility function (1$ is 1$ no matter your net worth) seems to fit how money actually interacts with our well being better. You can use any utility function you want; but note that the logarithmic base won't change the conclusion. You have to use a function that is significantly different (and not just a constant factor different).
You could redo Kelly math using the square root of your net worth, for example, or even net worth squared. You'd get different answers, especially if your curve is convex up. (A situation where a convex up utility curve makes sense is one where everyone is going to die except people who can spend 1 million dollars on a ticket off the island: here, having 100000$ vs 1$ doesn't matter, but having 999,900$ vs 1,000,000$ matters a lot. You could imagine a less contrived example, I am sure.)