Given that you have your emergency fund, and no other high interest rate debts (credit cards, etc.) you will want to put down at least enough to not have to pay Private Mortgage Insurance (PMI). PMI is solely to protect the lender if you default. It has no benefit to you. It generally means that you will need at least 20% down. After that, its a personal decision, depending on what else you are going to do with the money. If you are the type to spend money frivolously if you have it, it might make sense to put as much down as you can. If you think that you can invest the money and over the long-term make more than the historically low mortgage interest rates, it might make sense to invest.
One thing to keep in mind is that money that you put into the down-payment is relatively illiquid, meaning that it is hard to turn back into cash. If you have large expenses in the future, like health problems or college for the kids, it might be better to have the money in something easier to turn into cash.