There is a general advice that borrowing against 401k is to be avoided. When I was looking at related questions here in SE I found a few discussions where some experts challenged the generality. For ex, source vs collateral question. To me that somewhat makes sense.
In the recent past I have been thinking about this option more and more as I have been looking to buy my first home in the San Francisco Bay Area, California. Needless to say most houses are unreasonably highly priced (and that discussion could be out of scope for now).
I am currently faced with a sort of a dilemma/quandary and I am looking to evaluate my thought process and remove misconceptions.
Back story: In the last two years we had many expenses (unplanned medical, wedding, deferred medical, expenses from putting purchases on credit cards for a period close to a year when my now wife wasn't being paid by her professor for postdoc research).
So, currently, even though together we bring a decent pay home, we're aggressively paying down debt and we've been doing so by juggling balance transfer cards and scoring 0% APR cards. My own question - balance transfer vs loan - total interest paid for reference.
Until even a year ago I was not looking to buy a house in the Bay Area and was hoping to find a job in another state but that is seeming uncertain now. It also looks like the market is somewhat saturating with more houses staying longer on the market. So as we started talking about borrowing options it soon became evident that the biggest blocking factor for us is the down payment.
Constraints/Challenge: Because of our paying down the debt we aren't building/saving for the down payment. A few options where they need just 3% down seem a little unconvincing/shady, but even if we wanted to we can't do 3% currently.
With the stock market having bad and good days, my 401k balance is not showing consistent growth. I understand that over the long term with the assumption of a growth in stocks the 401k will grow and compound. But...
So the questions: Can I use this time of ups and downs and trade-war uncertainties to withdraw from 401k? Also I keep hearing fed rates are low. Will this help my case in securing lower APR mortgages, and thus help me in the long run?
I am also hazy on the 'you pay yourself the interest' because then what is the flip side? And what do the banks get for this?