You have correctly identified a couple of the main risks associated with investing through debt. Namely, that you would need to be able to bear interest costs for the period of time that your investments might be 'underwater', or you risk needing to liquidate your assets which might not cover your debt, leaving you with less than nothing. You've also correctly identified that housing losses in particular have borne the brunt of recent major economic downturns.
However, the risk of using home equity for investing is not just in the leverage itself, it is also in what that mortgage really means to you from a flexibility standpoint. Consider the order of events of the Global Financial Crisis that started in 2007:
(1) The early 2000's had a significant upturn in the US housing market, which dominated attention and captivated the national interest (see: DIY / house flipping show boom around the same time). People began buying larger and larger houses, using the oft-repeated proviso of 'they aren't building more land!!!!', taking on debt that banks were practically begging them to accept, on terms that would now be considered literally criminal [No Income, No Job or Asset 'NINJA' loans being perhaps the most infamous]. The financial pressure riding on the continued success of the housing market became massive. The point here is that markets are tied in a very intricate way, and it is often difficult to predict the outcome of a collapse in one sector, on another. Even the Global Financial Crisis may have been a non-worst case scenario of how this might happen in the future.
(2) By 2007, many homeowners were overleveraged, having borrowed more than they could reasonably afford, temporarily buoyed by an expectation that ever-rising house prices would always allow refinancing to occur, with loans to be ultimately paid off by the value of the underlying housing assets themselves [sounds like a bubble, right?]. In 2007, when housing supply finally outpaced speculation-laden demand, prices plateaued, and then started to drop. Suddenly, people couldn't refinance their homes which they couldn't afford on their own (especially if they had infamous balloon payment mortgages which deferred some of the larger principal repayments of the loan for 5+ years, inherently necessitating refinancing to be afforded). So make sure you are truly able to afford your total debt, not just that the bank will approve you for it. They don't have your best interests at heart.
(3) As some people became unable to refinance their homes, they were forced to sell. Mass sellings occurring at the same time dumped housing supply on the market which had simultaneously reduced demand. Housing prices of course dropped. How unique is your house if you needed to sell at the same time as your neighbors? Toronto's sub-550 sq ft condo market continues to spiral, as literally thousands of identical units came on the market for rent and sale simultaneously due to new builds coming online, and air-bnb's being taken off that market due to Covid.
(4) As the housing market crashed, financial markets which were directly tied to it followed suit. Financial market collapse precipitated other economic crashes, as secondary and tertiary industries also faced reduced cashflows. No matter how far removed your investments are from housing, the way our markets work now means that it is very difficult for a major collapse in one sector, to not hit all sectors.
(5) Job losses created big problems - if you can't sell your house, you can't move for work, so temporary unemployment became permanent unemployment for many. The point here is that tying your financial success to your house physically places you in one space, and reduces your flexibility to (a) cut housing costs if needed, or (b) move to better employment elsewhere. If you lose your job while a housing crisis is in process, and you have overleveraged yourself, and financial markets get hit at the same time as part of a 2008-style GFC snowball effect, you may not have any desirable financial options outside of bankruptcy.
(6) Recovery took a long time, and it was not universal across the country, or across the world. Some areas have housing markets which have still not fully recovered on an inflation-adjusted basis, and some areas are many times higher than they have every been. The point here is that it is not truly fair to expect that weathering a few years of poor financial results would be enough in a future downturn. As an example of how this might happen now - consider 'what if' the world truly moves towards more remote work, after the practical test-case of COVID: will San Francisco's housing market collapse, as tech works leave for literally greener, lower cost pastures? The US is not a singular housing market, it is many thousands of housing markets, and your house is in exactly one of them. Sure hope you don't double-down on house debt right before they put a hog rendering plant next door!
So before taking on these risks, consider not just whether you could whether a 3 year storm of dropped equity prices, but also dropped housing prices, and what job loss might mean for you particularly if it happened while your house was underwater. Interest rates are very attractive right now, but debt still has costs. And leverage itself inherently increases the risk of a portfolio, which you admit is already high-risk.
Be careful out there.
