When you become an Authorized User on someone else's credit card, you inherit the card's history, good or bad.
If the card has a good payment history, is well-seasoned (4 years or more), and has a consistent low credit utilization (balance is 10% of the limit or less), removing yourself may result in a lower credit score.
If your credit is "thin" (you have few accounts reported), losing one account may cost you some points. If you have accounts in your own name that are relatively new (less than two years old), removing an Authorized User account may cause the Average Age of Accounts (15 per cent of your credit score) to be younger, causing you to lose some points.
The best advice I can give is to keep the account until you have four or five credit cards of your own if it is always in good standing with a low balance and the owner is willing. Dropping the Authorized User account would then have very little effect on the score.