You're correct that in theory it should't matter; "sell low + buy low" is effectively a "wash sale" in terms of your equity.
But you bet you care. If you are trading a house when the market is soft...
You may not be able to get financing
The reason prices went up in the first place is that you had many people bidding on the available housing stock, and they had a bigger bag of money to bid with than they did before.
Where did that come from? The bank loaned it to them (when they didn't before). Almost everyone buys on mortgage, and the size of their mortgage (thus their "bag of money") is limited by the size of the mortgage the bank will give them. In housing run-ups, banks are typically more liberal with the loans they are writing. They are a) offering marginal loan products such as teaser ARMs, IO loans, neg-am loans etc. that allow a bigger loan to be made on the same monthly payment, or b) offering bigger loans to people with worse credit than they normally lend. In other words, housing prices go up because credit is loose and flowing freely.
Flash forward to your down market. Why is it down? Nobody's bidding the prices up. Why aren't they? They can't get loans.
Well, neither can you.
You can sell the old house (can you?) and pay off that loan easily enough, but loans don't just "transfer" (even in the UK, they don't transfer without the bank's consent, because it must be willing to accept the new home as collateral). And that's the problem: in a down market, the bank isn't so happy with the house as collateral.
For cash buyers, it's the reverse
On the other hand, an up/bubble housing market is bad news for cash buyers, because borrowers' "bag of bidding money" are getting larger and larger at the whimsy of the bank. They are the vast majority of house buyers, so they set the market. Cash buyers can get priced out, and told "get a mortgage like everyone else".
But just wait. When the market is down, and nobody's writing mortgages, it tends to crash housing prices. And then, the cash buyer is king. In 2008, San Francisco flipped from 90% of sales being extreme mortgages (IO or neg-am), to 90% of sales being cash.
Also, you could be upside-down on your mortgage
In a down market, you lose equity in your home. That means you could find yourself either entirely upside-down on your mortgage, or simply with too high an loan-to-value ratio to get a comparable mortgage.
Say you buy a $250,000 house with a $150,000 note (normal 60% LTV). You're ready to move, but the house has fallen to $150,000. Good news is you paid off $15,000, so you have $20k equity (you owe $135k, 90% LTV). In this bad market, nobody's willing to write you a 90% LTV loan, so buying a comparable $150,000 house is right out. At a normal 80% LTV loan, your $15,000 down will only get you a $75,000 house. Ouch.
The fact that they're already in a mortgage relationship with you doesn't mean they want another one today based on their current, more conservative underwriting doctrine.
Your credit may not be as good
Down markets tend to have knock-on effects that impact a lot of people. Job loss, revolving credit lenders canceling credit lines, lenders being more itchy about calling notes... nobody's credit gets better in a down market, and you have a fair chance of it getting worse.
For instance even if you never get laid off, if your bank sees your employer is doing rounds of layoffs, it gives them pause to write a loan.