The scenario you mention regarding capital gains is pretty much the core of the issue. Here's a run-down from PolitiFact.com that explains it a bit. It's important to focus on it being the tax rate, not the tax amount (which I think you get, but I want to reinforce that for other readers).
Basically, most of Buffett's income comes from capital gains and
dividends, income from investments he makes with the money he already
has. Income earned by buying and selling stocks or from stock
dividends is generally taxed at 15 percent, the rate for long-term
capital gains and qualified dividends.
Buffett also mentioned that some of the "mega-rich" are hedge fund
managers "who earn billions from our daily labors but are allowed to
classify our income as 'carried interest,' thereby getting a bargain
15 percent tax rate."
We don't know the taxes paid by Buffett's secretary, who was mentioned
by Obama but not by Buffett. Buffet's secretary would have to make a
high salary, or else typical deductions (such as the child tax credit)
would offset taxes owed. Let's say the secretary is a particularly
well-compensated executive assistant, making adjusted income more than
$83,600 in income. (Yes, that sounds like a lot to us, too, but
remember: We're talking about the secretary to one of the richest
people in the world.) In that case, marginal tax rates of 28 percent
would apply. Then, there would be payroll taxes of 6.25 percent on the
first $106,800, money that goes to Social Security, and another 1.45
percent on all income, which goes to Medicare. The secretary’s overall
tax rate would be lower than 28 percent, since not all the income
would be taxed at that rate, only the income above $83,600.
Buffett, meanwhile, would pay very little, if anything, in payroll
taxes. In the New York Times op-ed, Buffett said he paid 17.4 percent
in taxes. Thinking of the secretary, it gets a little complicated,
given how the tax brackets work, but basically, people who make
between $100,000 and $200,000 are paying around 20 percent in federal
taxes, including payroll and income taxes, according to an analysis
from the nonpartisan Tax Policy Center.
So in this case, the secretary's rate is higher because so much of
Buffett's income comes from investments and is taxed at the lower
capital gains rate.
Here's Buffet's original Op-Ed in the NYT for those of you that aren't familiar.
