Does it ever make sense? Yes, but almost never.
If you're in a situation where you're invested in something with low rates (think government securities) or cash equivalents, then you do need to think about rate spread as you mention. Does the savings over the life over the mortgage beat out the 35% hit now, plus all the interest you would earn over those 20 years? Have you factored in other considerations such as mortgage interest deduction on taxes? Don't forget you need to think about how rates will change down the line (they can't go much lower, so potentially you'll get better rates in the 401(k) down the line). Don't forget there's also the impact of inflation; again the rates on your savings may go up, but your mortgage is a fixed payment, so with even a low rate of inflation, your payments effectively become "less" over time.
If your investments are in something like stocks and bonds, then I would say undoubtedly you would want to keep the money in the 401(k). Time in market and compounding are your best friends over a long time horizon.
Also, as mentioned by @JohnFX, the hit of your 35% now is something you will absolutely feel now. Hopefully not, but your life situation could change where you have an emergency and need to drain your savings or you may not see the end of that 20 years.