There are two methods of accounting for buybacks, and only one of them affects retained earnings. It makes sense if you think of "retained earnings" as all earnings that have not been spent or distributed - If a company earns $1000 and gives out $100 in dividends, it "retains" $900 in earnings. That's a very simple example but if you think about it that way, you can see how stock buybacks can affect retained earnings.
If a company issues stock at $10 and buys some back at $50, that $40 increase in share price is not accounted for anywhere on the balance sheet before the buyback, so the difference between what was received for the stock and what it paid to buy it back comes from whatever earnings were "retained" before the buyback. Conceptually it is an "expense" that must be accounted for.
In reality the charge to retained earnings is just used to balance out the debits and credits, but conceptually it makes sense to think of it as an "expense", even though it is not recorded as such on the Expense Report.
In the "cost" method (which is more common in my experience), the difference is accounted for in a contra-equity account called "treasury stock" that reduces the shareholder portion of net equity.
In either case, the net effect is a reduction of the equity available to shareholders (similar to a dividend, it's "cash out the door"), and it's important to understand the reason behind unusual numbers, whether it's negative retained earnings or large amounts of treasury stock and how they affect future profitability.