It is a very bad idea in almost all circumstances to borrow/withdraw from an IRA (or other retirement account) for a down payment on a house and I would only suggest it as a last resort. Even as a last resort it is a bad idea, and I'd suggest you aren't ready to buy a house until you have enough non-retirement money to do the deal.
In your case you have money in savings, you should absolutely do that. If you don't feel comfortable using your emergency fund, you need to keep saving until you can afford to buy a house. Your retirement fund isn't a piggy bank for when you need money now, it is a payment you make to keep a decent lifestyle later in life. When you are retired your options for alternate income streams will be much more limited.
The biggest hidden cost is the opportunity cost of not having that money working for you. Your final retirement balance is very likely (when done correctly) going to be mostly comprised of the return on your investment more so than the money you put in. While you are young your greatest asset is time for that money to grow, don't throw that away when you have viable options!
The other problem, if you are talking about an employer plan (401K/Roth) is that if you lose your job you will have 60-90 days to pay back the loan or it will be considered a distribution and subject to taxes plus a 10% early withdrawal penalty. This happened to a bunch of my friends from work when our company got bought by another company forcing them to close the old 401K plan and transfer to the new one.
Short version: Don't touch retirement money for non-retirement things unless it is a life/death emergency and you have no other options.