It is true that all else being equal, you will pay a lower amount of total interest by paying down your highest interest rate debts first. However, all else is not always equal. I'm going to try to come up with some reasons why it might be better in some circumstances to pay your debts in a different order. And I'll try to use as much math as possible. :)
Let's say that your goal is to eliminate all of your debt as fast as possible. The faster you do this, the lower the total interest that you will pay. Now, let's consider the different methods that you could take to get there: You could pay the highest interest first, you could pay the lowest interest first, or you could pay something in the middle first. No matter which path you choose, the quicker you pay everything off, the lower total interest you will pay. In addition to that, the quicker you pay everything off, the difference in total interest paid between the most optimal method and the least optimal method will be less. To put this in mathematical notation:
limt→0 Δ Interest(t) = 0
Given that, anything we can do to speed up the time it takes to get to "debt free" is to our advantage.
When paying large amounts of debt as fast as possible, sacrifice is needed. And this means that psychology comes into play. I don't know about you, but for me, gamifying the system makes everything easier. (After all, gamification is what gets us to write answers here on SE.)
One way to do this is to eliminate individual debts as quickly as possible. For example, let's say that I've got 10 debts. 5 of them are for $1k each. 3 of them are for $5k each, 1 is a $20k car loan, and 1 is a $100k mortgage. Each one has a monthly payment.
Let's say that I've got $3k sitting in the bank that I want to use to kickstart my debt reduction. I could pay all $3k toward one of my larger loans, or I could immediately pay off 3 of my 10 loans. Ignore interest for the moment, and let's say that we are going to pay off the smallest loans first. When I eliminate these three loans, three of my monthly payments are also gone. Now let's say that with the money I was paying toward these eliminated debts, and some other money I was able to scrape together $500 a month that I want to use toward debt reduction. In four months, I've eliminated the last two $1k debts, and I'm down to 5 debts instead of 10. Achievement Unlocked!
Instead of this strategy, I could have paid toward my largest interest rate. Let's say that was one of the $5k loans. I paid the $3k toward the bank to it, and because I still had all the monthly payments after that, I was only able to scrape together $400 a month extra toward debt reduction. In four months, I still have 10 debts.
Now let's say that after these four months, I have a bad month, and some unexpected expenses come up. If I've eliminated 5 of my debts, my monthly payments are less, and I'll have an easier month then I would have had if I still had 10 monthly payments to deal with.
Each time I eliminate a debt, the amount extra I have each month to tackle the remaining debts gets bigger. And if your goal is eliminating debt quickly, these early wins can really help motivate you on. It really feels like you are getting somewhere when your monthly bills go down. It also helps you with the debt free mindset. You start to see a future where you aren't sending payments to the banks each month. This method of paying your smaller debts first has been popularized in recent years by Dave Ramsey, and he calls it the debt snowball method.
There might be other reasons why you would pick one debt over another to pay first. For example, let's say that one of your loans is with a bank that has terrible customer service. They don't send you bills on time, they process your payment late, their website stinks, they are a constant source of stress, and you are getting sick of them. That would be a great reason to pay that debt first, and never set foot in that bank again.
In conclusion:
If you have a constant amount of extra cash each month that you are going to use to reduce your debt, and this will never change, then, yes, you will save money over the long run by paying the highest interest debt first. However, if you are trying to eliminate your debt as fast as possible, and you are sacrificing in your budget, sending every extra penny you can scrape together toward debt reduction, the "snowball" method of knocking out the small debts first can help motivate you to continue to sacrifice toward your goal, and can also ease the cash flow situation in difficult months when you find yourself with less extra to send in.