The calculation you are trying to make is laid out in some detail in the German context, that is mostly applicable to your situation, in the various books by Gerd Kommer (Kaufen oder Mieten etc.). I encourage you to read it.
His end result is that in most cases, renting is financially more prudent than buying given the current market, assuming you take the difference between your rent and the mortage payments and stick it in an ETF or similar investment.

This has many reasons from the very renter friendly setting in Austria to the global market for investments. To answer your question about how this comes to pass: Houses are considered a safe investment by many investors and thus roughly correlate with the returns on bonds (simplified). That means a gross return of 2.5% (or a factor of 40 as real estate investors tend to phrase it) is considered OKish and roughly corresponds to slightly negative returns on long running bonds and also has expectations of raised rents in the future price into it. If you want to pay off your house in 20 years, you are essentially assuming something in the 5% range of return, which is just way higher than the market currently bears.
The end result is that it's probably not really feasible for the median income household to actually own a place in that setting. They are sadly simply priced out of the market.

Whether the market will cool off, as you imply, is a matter of speculation at least beyond the demographic point of view. I would tend to say that it will fall off once the boomer generation mostly passed away, which however will be too far in the future for you to sensibly wait for.