Ditto @GradeEhBacon, but let me add a couple of comments:

1. Ignorance of the law. There might be all sorts of tax savings that you or I could take but that we don't because we don't know about them. And that doesn't mean that we're stupid. US tax code is now over 73,000 pages long. Who knows what's in there?

But more relevantly:

2. GradeEhBacon mentioned transaction costs. Yes. Many tax shelters require setting up accounts, doing paperwork, etc. Often you have to get a lawyer or accountant to do this right. If the tax shelter could save you $1 million a year in taxes, it makes sense to pay a lawyer $10,000 to set it up right. If it could save you $100 a year in taxes, paying $10,000 to set it up would be foolish. In some cases the tax savings would be so small that it wouldn't be worth the investment of spending $20 on a FedEx package to ship the paperwork.

3. Inconvenience. Arguably this is a special case of transaction costs: the cost of your time. Suppose I knew that a certain tax shelter would save me $100 a year in taxes, but it would take me 20 hours a year to do the paperwork or whatever to manage it. I probably wouldn't bother, because my free time is worth more than $5 an hour to me. If the payoff was bigger or if I was poorer, I might be willing.

4. Complexity. Perhaps a special case of 3. If the rules to manage the tax shelter are complicated, it may not be worth the trouble. You have to spend a bunch of time, and if you do it wrong, you may get audited and slapped with fines and penalties. Even if you do it right, a shelter might increase your chance of being audited, and thus create uncertainty and anxiety. I've never intentionally cheated on my taxes, but every year when I do my taxes I worry, What if I make an honest mistake but the government decides that it's attempted fraud and nails me to the wall?

5. Qualification. Again, as others have noted, tax shelters aren't generally, "if you fill out this form and check box (d) you get 50% off on your taxes". The shelters exist because the government decided that it would be unfair to impose taxes in this particular situation, or that giving a tax break encourages investment, or some other worthy goal. (Sometimes that worthy goal is "pay off my campaign contributors", but that's another subject.) The rules may have unintended loopholes, but any truly gaping ones tend to get plugged. So if, say, they say that you get a special tax break for investing in medical research, you can't just declare that your cigarette and whiskey purchases are medical research and claim the tax break. Or you talked about off-shore tax havens. The idea here is that the US government cannot tax income earned in another country and that has never even entered the US. If you make $10 in France and deposit it in a French bank account and spend it in France, the US can't tax that. So American companies sometimes set up bank accounts outside the US to hold income earned outside the US, so they don't have to bring it into the US and pay the high US tax rate. (US corporate taxes are now the highest of any industrialized country.) You could, I suppose, open an account in the Caymans and deposit the income you earned from your US job there. But if the money was earned in the US, working at a factory or office in the US, by a person living in the US, the IRS is not going to accept that this is foreign income.

Ditto GradeEhBacon, but let me add a couple of comments:

1. Ignorance of the law. There might be all sorts of tax savings that you or I could take but that we don't because we don't know about them. And that doesn't mean that we're stupid. US tax code is now over 73,000 pages long. Who knows what's in there?

But more relevantly:

2. GradeEhBacon mentioned transaction costs. Yes. Many tax shelters require setting up accounts, doing paperwork, etc. Often you have to get a lawyer or accountant to do this right. If the tax shelter could save you $1 million a year in taxes, it makes sense to pay a lawyer $10,000 to set it up right. If it could save you $100 a year in taxes, paying $10,000 to set it up would be foolish. In some cases the tax savings would be so small that it wouldn't be worth the investment of spending $20 on a FedEx package to ship the paperwork.

3. Inconvenience. Arguably this is a special case of transaction costs: the cost of your time. Suppose I knew that a certain tax shelter would save me $100 a year in taxes, but it would take me 20 hours a year to do the paperwork or whatever to manage it. I probably wouldn't bother, because my free time is worth more than $5 an hour to me. If the payoff was bigger or if I was poorer, I might be willing.

4. Complexity. Perhaps a special case of 3. If the rules to manage the tax shelter are complicated, it may not be worth the trouble. You have to spend a bunch of time, and if you do it wrong, you may get audited and slapped with fines and penalties. Even if you do it right, a shelter might increase your chance of being audited, and thus create uncertainty and anxiety. I've never intentionally cheated on my taxes, but every year when I do my taxes I worry, What if I make an honest mistake but the government decides that it's attempted fraud and nails me to the wall?

5. Qualification. Again, as others have noted, tax shelters aren't generally, "if you fill out this form and check box (d) you get 50% off on your taxes". The shelters exist because the government decided that it would be unfair to impose taxes in this particular situation, or that giving a tax break encourages investment, or some other worthy goal. (Sometimes that worthy goal is "pay off my campaign contributors", but that's another subject.) The rules may have unintended loopholes, but any truly gaping ones tend to get plugged. So if, say, they say that you get a special tax break for investing in medical research, you can't just declare that your cigarette and whiskey purchases are medical research and claim the tax break. Or you talked about off-shore tax havens. The idea here is that the US government cannot tax income earned in another country and that has never even entered the US. If you make $10 in France and deposit it in a French bank account and spend it in France, the US can't tax that. So American companies sometimes set up bank accounts outside the US to hold income earned outside the US, so they don't have to bring it into the US and pay the high US tax rate. (US corporate taxes are now the highest of any industrialized country.) You could, I suppose, open an account in the Caymans and deposit the income you earned from your US job there. But if the money was earned in the US, working at a factory or office in the US, by a person living in the US, the IRS is not going to accept that this is foreign income.