Caution: somewhat opinionated answer (although I don't have/need car insurance, so do not have a particular "axe to grind").
Why might this happen?
You weren't supposed to notice. You were supposed to just let it renew quietly.
The insurance industry (somewhat like Utility supply industries) is particularly cut-throat: margins are tight, so companies need to attract as much new business as they can. They want to compensate for low profit-per-customer by having as many customers as possible.
So, particularly in the age of the comparison website, they will offer extremely attractive deals to new customers switching from other companies. A cynic might believe they even offer loss-leaders. Once a new customer has joined them, despite the prevalence of comparison sites, there is an increased chance that they will stay with that company, at least for a couple of years. This might be due to a feeling of loyalty on the part of the customer, but – I suspect – many people will not think about insurance until next year's reminder arrives, at which point sufficient numbers will not be bothered wading through the comparison sites to try and find a better offer and will simply allow the renewal to go ahead1. The price for these years will tend to revert from the "attract new customers price" to one that more accurately reflects the risks involved (and, probably, beyond that figure, to help recover revenue-reductions due to the initially-low prices).
When either someone does notice the price-hikes on renewal (or they rise by too much that "can't be bothered looking around" ceases to be an option) then customers will return to the price-comparison websites, and the whole process starts again. (As the OP noted, the current insurer may offer a modest discount on their original renewal quote, but the fact that the customer is questioning the figure means they've probably lost them already).
In case anyone thinks I am having an undue go at insurance companies, I'm not. In a lot of ways, this is how they have to do business in the current climate.
In the "good old days", before the internet, hunting around for "the best deal" was much more time consuming, so a lot of people would tend to stick with the same insurer for longer. Some would go through an insurance broker to do the searching for them, but they – quite reasonably – would need to be paid for their work (either directly or indirectly). I strongly suspect that – while "new customer deals" probably did happen – they were not as dramatic as they are now, and the price settled to a more uniform level representing the actual risks involved.
Since then there have been two major factors that I think have affected the industry:
Price Comparison Sites
In some ways, a double-edged sword for the consumer. Yes, they allow customers to search for "the best deal" (although not always: some insurers don't work with any/some comparison sites, and some sites may "favour" those companies they have deals with).
On the flip-side, it means insurers that do participate are under constant pressure to offer as cheap a deal as they can, leading to the current situation of artificially-low prices for new customers that have to be offset by larger jumps for renewing ones.
No Win, No Fee Lawyers
Over the last couple of decades there has been a dramatic increase in the number of companies offering no win, no fee services. While this will probably have enabled a lot of people to get justice that previously they would not have had access to, it has also led (IMHO) to a vast increase in essentially frivolous claims. These are often at levels where – taken case by case – it is cheaper for insurance companies to settle than to fight. Collectively, though, they must impose a great burden on insurance companies, which – necessarily – has to be reflected in the premiums.
1 Interestingly, an article on This is Money, carrying similar stories to the OP's, notes that when you try to renew can affect the prices on offer:
Timing affects the quotes you get
Insurers will offer cheaper quotes to customers they view as less of a risk. So if you buy your car insurance before it needs to be renewed you will be judged as a more organised, and probably safer, driver.
An investigation by MoneySavingExpert suggested 21 days before renewal is the cheapest time to get your car insurance. You could expect to pay £587, on average.
If you buy it on the day it was due for renewal, you can expect to pay up to an extra £567, taking the average annual premium to £1,156 a year.
Part of this will be simply that the later someone leaves looking to renew, the more they are in a "take it or leave it" situation – they need insurance by the end of the day, so will have to pay whatever is demanded. Another reason could be that someone looking ahead of renewal is more likely to be on top of the situation, and therefore less likely to passively take the auto-renewal figure when it arrives in a couple of weeks.
