There are circumstances under which you don't have to pay the penalty:
- You die and the account is paid to your beneficiary
- You become disabled
- You withdraw an amount less than is allowable as a medical expense deduction
- You begin substantially equal periodic payments
- Your withdrawal is related to a qualified domestic relations order (QDRO)
- Your withdrawal is used to pay qualified higher education expenses
- Your withdrawal is used for a qualified “first-home” purchase (up to $10,000)
Unless it is one of those circumstances, your 10% penalty is on the pre-tax amount of the withdrawal that is subject to income tax (so if some of it is not, then there is no penalty on that amount). Some states may also impose additional penalties. More here.
Note that the other penalty you pay is a hidden one - the lost opportunity cost of letting that money stay in place and grow until age 59 1/2 or later.