You make a charge, your bank receives the request and texts your phone to say, "did you authorize this?" And you respond yes or no. If you say no, you can't walk out of the store with the stuff, and the store has nothing on file but a one-time request number.
There are payment systems that work exactly this way. I believe the largest is Alipay (primarily in China), but there's probably others. With this system, your phone generates a one-time barcode or QR code, which the merchant scans. Your account gets charged, and that code is never used again.
I believe it also has an alternate mode where the customer scans the merchant's QR code, which then causes them to send money to the merchant, but I'm not certain of that. The biggest problem here, and really the biggest problem in payments in general is "What if communication fails?"
If the merchant reads your card data (or QR code) and can't send it to your bank (or Alipay) to approve it, they have two choices: They can let you leave with the goods anyway, or they can say "Sorry, we can't sell you that right now. Come back tomorrow." In the vast majority of cases, it's better for them to let you leave with the goods, on the assumption that they will be able to successfully charge your card later. After all, most people's cards are good, and if done seamlessly enough, fraudsters won't know that the store is offline to take advantage of it. And if you can't buy what you wanted right then, you may just go to a competitor or give up altogether, and they lose out.
The flip side is that if they have to rely on your phone to send money, instead of them requesting it, they have to deal with bad cell service, customers whose phones are dead, or any other reason that you might be unable to send it right then. And while you sit there waiting for the right app to load on your phone with a single bar, other customers are piling up behind you.
Additionally, there needs to be a way for them to generate a unique enough number that your payment app knows exactly how much to send to which merchant, for which order, or they won't have any way to know that you did, in fact, successfully pay. And if they're offline, they'd have no way to receive that notification, so there's now two points of communication failure that will cause the sale to fail. (This is also why most sales don't trigger text messages for pre-approval confirmation - what if you don't get the text?)
Finally, it's in the card brand's interests to only accept payment requests from verified sources (i.e. merchant processors), and not have publicly exposed endpoints that anyone with an internet connection can DDOS or otherwise disrupt. And it's more profitable to them to eat the costs of fraud (or - better - make the merchant pay) in the name of making payments flow as quickly as possible.
Note: This answer is focused on the retail and quick-serve/fast-food industries. Hotels, sit-down restaurants and bars, gas stations, high-value merchants (like jewelers) etc. all have different requirements that affect what risks they're willing to take.