I think you have a misapprehension as to what the CVV is.
First off, every time you enter it into a website, you're "giving" it to someone, same as if you give it over the phone. Maybe you trust that they won't store it, but why do you trust a website more than a person? I don't.
The purpose of CVV is to have a second form of check for when you are performing a card-absent transaction (meaning the card is not physically in the hand of the merchant and not swiped or chip-read), to make it more likely that the person is actually holding the card. Using your card over the phone is one form of card-absent transaction, just like over the web.
It does not guarantee safety. What it does, is has a second thing that the potential thief would have to steal. So when a thief downloads a database of cards used at some merchant, for example, assuming that merchant followed correct practice they didn't also get CVV numbers - so they're more limited in where they can use the stolen card numbers. The person asking it for the over-the-phone transaction is helping you out here: they're making it less likely that someone who has stolen your card number will be able to make a fraudulent transaction with them.
Beyond that, CVV isn't really offering very much protection. It's certainly possible for someone, whether a person on the phone or a person behind a website, to take that CVV, write it down, and then misuse your credit card number and CVV. That's not your fault, and it's nothing your credit card issuer is going to hold against you.
You can see some additional information in some articles, like this one from AVG:
You may also be asked for your credit card security code when processing a payment over the telephone. As with online transactions, it’s usually safe to do this – you just need to be sure that no one overhears the details you give out (so avoid public places when doing this).
Here's an article by nerdwallet that discusses things from a business owner's perspective:
How can I mitigate the risk of fraud?
Get all relevant information when taking down a customer’s credit card information — including card number, expiration date, CVV code and billing ZIP code. Also, make sure you get the customer’s name exactly as it appears on the card, including middle initial if applicable.
Consumerist discusses a case where you should not have to enter it - in person. But it does include a comment from Mastercard confirming over-the-phone is considered normal:
“It’s difficult to think of a reason why an employee would ask a customer to recite their CVC2 code for an in-store transaction,” the spokesperson said. “As you note, the codes were created to help authenticate cardholders for online and over the phone purchases.”
Finally, from the Visa merchant agreement, some notes:
Mail order/telephone order (MO/TO) and electronic commerce merchants must verify—to the greatest
extent possible—the cardholder’s identity and the validity of the transaction.
Mail/telephone order are treated identically to e-commerce, in this section.
If available, use fraud prevention tools such as Card Verification Value 2 (CVV2)*, Address Verification
Service (AVS)**, Verified by Visa, and Visa Checkout. For more information visit www.visa.com.
Note the instruction for phone order (and even mail order!) to use CVV2.
Ask for CVV2
The Card Verification Value 2 (CVV2)* is a three-digit security number printed on the back of Visa cards
on the signature panel and helps validate that a customer is in possession of the card at the time of an
order. (See Visa Card Features and Security Elements on page 26 in Section 2: Card-Present Transactions of this
Of course, it's strictly prohibited to retain this information regardless of what type of merchant it is:
A cardholder’s CVV2 may never be stored as a part of order information or customer data. The storage of CVV2 is strictly prohibited subsequent to authorization.