I see three possible reasons:
- He doesn't have a secure way (or any way) to manually enter card details. Most payment terminals have a keypad and can support manual entry (or PINs), but everything else in the ecosystem has to as well. If he doesn't have a way (or doesn't know how) to get the terminal to prompt for manual entry, then that's not an option.
- PCI-DSS. It's possible that his PCI scope (how much liability he has) is based on never actually having the card number himself. Giving it to him over the phone would violate that. PCI violations could lead to the major card brands saying "You're not allowed to take credit cards any more", which would be fatal to most businesses these days. Violations would also leave him liable for any fraud that can be traced back to his store.
- Interchange rates. He almost certainly pays more for a manually entered card than a swiped one, because the latter is more secure. If he has a way for you to insert your chip, that's even better, as well as making him not liable for fraud if your card was stolen (since the card brands would eat it). So by making you travel out there to physically present your card, he's saving himself money.
To put #3 another way: By making you show up in person, he makes it less likely you're using a stolen card number (because you'll have a physical card) and easier for him to prove that you did actually authorize the payment (because you'll sign a receipt and/or be caught on a security camera). That makes it less likely that it's a fraudulent transaction, which is why it gets a lower interchange rate.