We all know you can arrange a higher deductible (USA) or excess (UK) to pay a lower insurance premium.
My question is, is there an established method to identify what is the optimal size of deductible?
You have to take two things into account when considering how much you should spend on insurance: frequency of claims (how often they happen) and severity (how bad any given claim is likely to be); this is how your insurance company looks at it as well: Ideally, you want to purchase insurance for the unexpected (low frequency, high severity) because that is the least predictable risk, and take on the low severity and higher frequency losses because these are the most predictable and the insurance company is just going to charge you for it anyway.
As an (extreme and simple) example, let's say that you have had $1,000 worth of losses every year for the last 5 years and you have policy with a zero deductible - your premium is going to be at least $1,000 plus the cost to insure losses above $1,000 to the limit of insurance; therefore, if you know that you'll have at least $1,000 worth of losses you should have a deductible of at least $1,000 because then you're reducing the dollar swapping between you and the insurance company (and you actually pay more than $1,000 due to transaction costs). This can be calculated by taking all of your losses you have, indexing them for inflation to present value, and then dividing by the number of years they cover - Ins. Cos. usually use 5-7 years.
In your example, if you are paying for the $100 deductible and have $1,000 worth of losses and just not reporting them, you are overpaying because deductibles are supposed to reduce the FREQUENCY of losses, not SEVERITY. You are just reducing the severity of a very infrequent event (a large loss) when you could just put that money in your pocket and pay for the loss when it happens - because, again, the insurance company is just charging you for that + transaction costs.
With respect to the idea that
10 claims will put the company out of business, so will 30, 40, or 1,000,000 - but how likely is it and how much premium will you save if you increase your the deductible? If it's not likely and you save a lot of premium, the decision is simple.