Generally just giving a Bank Account Number does not cause damage. It depends on what other information the user has and the country you are in.
Generally Bank take telephone instruction for certain [non-transactional] activities , and they would authenticate you by asking account number, address, date of birth and some additional info. In today's world this info can be pretty easily accessible, for example facebook or a details posted on Jobsites etc.
It is best avoided to give the bank account details, unless you are sure of the person.
Typical other misuse is using your bank account to Launder black money. The typical modus is transfer funds to you and then ask you to transfer it elsewhere. At times its also a scam and you loose money as they trick you in sending money before you receive it.