If I can drop from 4.375% APR to 3.7% (or slightly lower) by going from a 30- to a 20-year note, I can save over $100,000 in interest over the life of my loan. However, like most folks, I have no idea if I will still be in this house 20 years from now. Is this a bad idea? My payments will go up slightly ($150/month), and allegedly I can deduct the points from my taxes (in the US), but how long would I need to stay in this house before selling would represent a financial mistake, on, say, $4000 closing costs?
For a refinance, you deduct the points over the terms of the loan, here 1/20 per year.
Since you don't mention the mortgage balance, I'll offer a back of envelope answer. You will save .675%/yr on the declining balance. If your loan is $200K, it will take about 3 years to break even, saving the $4000 cost. After that, it's all gravy. You can use your own balance and get a precise calculation.