"One for me and one for my wife"
In the old days (the tax code prior to the one implemented for 2018), the tax process included "exemptions," a $4050 per person deduction against income. In other words, a family of 4 had $16K that would not be taxed, along with the rest of the math of their return, whatever their other deductions were, standard vs itemized, etc.
The W4 used the term "allowances". One "allowance" meant "please don't tax $4050 worth of my income." It could have been from a person or from $4K worth of mortgage interest, etc. Again, to be clear, a schedule A offering me $40K worth of itemized deductions might result in 10 allowances.
Now, there are no exemptions. So, the question you asked about the baby, has an answer but not based on an exemption, rather the child tax credit.
The W4 does not appear to have been updated yet. But Publication 15 is ready for 2018. In my opinion, the W4 calculators always left a lot to be desired, and going right to Pub 15 made the most sense. You can do a dry run of your taxes, estimate your total tax bill, and look up on Pub 15 exactly how many allowances to claim to get close to your total payment goal.