Here’s what happens:
When you thought the scammer deposited money in your account, it didn’t really happen. It was essentially fake money. This could happen in a few different ways. It could be that the scammer gave you a bad/fake check, which isn’t discovered until several days after you deposited it. Banks don’t know for sure if checks are good until they send it on to the issuing bank. As a courtesy, your bank will assume it is good as soon as you deposit it and credit your account immediately, but if the check turns out to be bad, they will withdraw that amount back from your account, and if you have already spent that money, the bank will want you to pay it back. There is also an electronic version of this scenario: If you are electronically sent money from an account by mistake or an account that was hacked, this can be reversed.
On the other side of the scam, when you send money out of your account to the scammer, you are doing so willingly, even if you were tricked into doing so. You went to the bank and withdrew the cash, or you wrote the check, or you initiated the electronic transfer out. Your account was not hijacked; you told the bank what you wanted to do with your account. You can’t normally just change your mind after the fact.
If the scammer took control of your account electronically and stole the money, or the scammer forged a check without your knowledge, you would generally be able to get your money back from the bank. But if you willingly hand money over to the scammer, you will need to go after the scammer to get your money back.