It looks like the tax investment penalty is about 2.6%, which makes me wonder why anyone pays quarterly estimated taxes. Wouldn't you be better off putting that money into stocks and using the proceeds to pay the penalty? Obviously some people have cash flow issues and in some years stocks lose value, but assuming you're planning long term, doesn't paying all your taxes in the next year make more sense?
Same argument and answer for investing instead of paying off debt, or borrowing to invest. Risk. What happens if the stocks drop by 10%? Sure, you might come out ahead on average, but a drop in the market could be catastrophic from a cash flow point of view.
In addition, federal tax debt is arguably the worst kind. The IRS has the authority to garnish wages and has virtually unlimited resources they can use to collect.
Your logic is not wrong. But the risk is more significant than you seem to assume. Essentially you are proposing taking a 2.6% loan to buy stocks.
Is that a good strategy? On average, probably. But if your stocks crash you might have significant liabilities.
In 1929, the Dow Jones dropped 89%. In 1989, >30%. In 2008-9, 54%. This is a huge risk if this is money that you owe in taxes. If you operate the same system year after year the chance of it going horribly wrong increases.
In addition to the other answers, which cover the risks of what is essentially leveraged investing, I'd like to point out that the 2.6% penalty is a flat rate. If you are responsible for withholding your own taxes then you are paying tax four times a year. So any underpayment on your first quarterly tax payment will have much more time to accrue in the stock market than your last payment, although each underpayment will be penalized by the 2.6%. It may make sense for someone to make full payments on later payments but underpay on earlier ones.
"While the US tax code does not directly impose an obligation to pay estimated taxes, it does impose a penalty on individuals for failure to pay enough taxes either through withholding or estimated tax." USMTG
Anyone can choose how s/he wants to pay their taxes but they better deal with any consequences of not paying them instead of just complaining about it like most people do.
Most people get the hatred towards the IRS but most complaints are misdirected and should be directed towards Congress who creates and messes around with the US Tax Code.
Some people actually do not make estimated payments and pay any possible taxes with their returns knowing that there may be underpayment penalty. For those people, the penalty is relatively small compared to what they can do with the cash over a year's time (i.e. investing or paying down debt). It's their choice!