Several events will always result in a reduction of your score, including:
- Reducing the average age of your credit cards (would result from cancelling this card)
- Reducing your available credit as measured by debt-to-credit ratio (as you mentioned)
- Opening a new credit account
These will show up in the short term, but I don't think it's worth $40 per year in perpetuity to avoid this. These aren't serious "black marks" in the same category as missing payments, carrying too much debt, or foreclosures/evictions, etc. These effects are designed to signal issuers when someone acquires a large amount of credit in a very short period of time, which may indicate a greater risk.
If your credit is good and you are using your other cards responsibly, closing the card (given the annual fee) would not cause me great concern if it were me.
Since you are so much better of a risk than you likely were in college, you can also call Capital One, ask to speak with a supervisor, and ask them to drop the fee and increase your credit limit. They should be able to easily verify that you meet the requirements for other types of preferred cards they offer, and they should be willing to offer you improved terms rather than losing your business. It is very possible they simply haven't re-evaluated your risk since you initially applied.
Also, remember that these types of effects determine only a portion of your overall score. Activity is also a major component. Rather than leaving an unused card open for history and debt-to-limit purposes only, I would also recommend having some minimum level of activity, such as an automatic bill payment, on each card you carry. The effect of using your cards over time will have a significant positive effect on your score.
Best of luck!