I think the answer depends very much on where you are.
I believe the other answer covers north america.
On contrast, in (continental) Europe, giving the account and bank number (IBAN and BIC) is a (the most) common way to enable someone to send money to you.
E.g. in Germany, you need much more than account number and bank number to withdraw money:
To "push" money to another account (wire transfer from your account to someone who gave you the other account + bank numbers), you either have to hand-sign a certain form, or (online) certain credentials (e.g. login & password / PIN + TAN) are needed. I.e. for defrauding you, the other would need to get your online credentials (for mTAN also your mobile phone, for chipTAN a TAN generator of your bank [easy] and your bank card, for (i)TAN your TAN list) or fake your signature.
There are also ways to allow someone to pull money from your account, see e.g. direct debit
For that you sign that the other side is allowed to withdraw specified amounts of money (at specified dates). This is either
between you and the other (i.e. your bank cannot check and doesn't reject withdrawals that are not authorized). However, the other side needs to have signed a contract with their bank that they'll only try to withdraw money they're entitled to.
or you sign such a thing with your bank (then they do know whether the other side is allowed to withdraw money, and you can tell the bank that you won't accept any further withdrawals from XYZ).
In the first case, the withdrawal technically still needs your approval. In order not to create a huge risk of fraud, the rejecting here is really easy:
If you tell your bank that you reject the payment,
- the bank will immediately roll back the transaction.
- You need not give a reason for rejecting such a payment,
- the bank is not allowed to charge you anything for the rejection, and
- has to give you the interest you'd have gotten if the money had stayed on your account.
- However, the bank can charge fees to the other side who tried to withdraw money from your account without approval.
- And of course, it's criminal to try defrauding someone.
- As the banks are much interested that this procedure is accepted as a safe way of payments, they're much after persons who try to defraud by this procedure.
- But even if the one who tried to withdraw money was entitled to the money (and you just say "I don't pay"), the bank rolls back the transaction. In that case the other has to go to a civil court to get his money.
The practical rule is that the payment is approved if you didn't reject within the first 6 weeks after the bank sent the account statement. In other words, until 4 1/2 months after the withdrawal (in case you have a bank that does only quarterly account statements), the one to get the money cannot be really sure that he actually has the money.
I think (but I'm not completely sure, maybe someone else can comment/edit) that these two possibilities are also what is used with debit card payments (EC/Maestro card - these are much more common here than real credit card payments).
-- end of Germany specific example --