One reason is that you can trade in the IRA without incurring incremental taxes along the way. This may be especially important if you intend to shift your portfolio allocation as you approach retirement. For instance, gradually selling stocks and buying bonds can incur taxes if you do it in a taxable account (if you do it while you have other income and thus may face capital gains taxes). Also, if you have mutual funds in a taxable account, they may distribute capital gains to you that you'll owe taxes on, but holding the funds in an IRA will shield you from that.
There are also some other side benefits to IRAs because they are considered to "not count" for certain purposes when determining what you're worth. For instance, if you go bankrupt, you could be forced to sell assets in taxable accounts to pay your creditors, whereas IRAs are protected in many cases. Likewise, if you try to get financial aid to pay for college for your kids, money in an IRA won't be counted among your assets in determining your aid eligibility, potentially giving your kids access to more aid money.
Finally, an especially prominent benefit is, paradoxically, the early withdrawal penalty. For many people, part of the purpose of an IRA is to "lock away" their money and prevent themselves from accessing it until retirement. Early withdrawal penalties provide a concrete consequence that psychologically deters them from raiding their retirement savings willy-nilly.