Yes, you have to pay it. The person you borrowed the shares from is entitled to the dividend, whether small or large.
You are likely to have enough cash in your account to pay the dividend, because you are required to maintain margin for the short position.
Of course, the stock price (all else equal) will go down by a similar amount on ex-dividend day, so you will make an offsetting gain on the short position.
The dividend you pay may be tax-deductible:
So long as you keep your short position open for longer than 45 days, then you're allowed to deduct payments in lieu of dividends on short sales as investment interest... [Otherwise] you're not allowed to deduct those payments as itemized deductions. Instead, you have to treat them as part of the total price you paid to buy the shares to close the position, so the payment becomes part of the capital gain or loss you realize on your short sale.
Note that there's a special IRS rule to deal with special dividends. If a dividend is greater than 10% of the price of a share of common stock or 5% of a preferred stock, then the required holding period becomes a year, rather than 45 days.