Calculate your original total cost basis. Divide the original total cost basis by the number of shares you own after the corporate event to determine the new cost basis per share. Multiply this number by the number of fractional shares sold - that is the cost basis of the fractional share sold. Subtract this from the the cash received to determine your capital gain/loss.
Here's an example for a 50% stock dividend (see below). It's not the same circumstances as your situation but the process is the same. If you want a match, google for one.
Many people simply report the payment as sales proceeds with zero cost and pay capital gains tax on the entire amount. The proper method, which also achieves tax savings, is to allocate your adjusted cost basis to the fractional shares and pay capital gains tax only on the gain or loss.
You can use the calculators on this website to compute the cost basis of your fractional share based on the type of transaction that produced it. An "all stock" merger, a "stock & cash to boot" merger, a stock split, or a spinoff can all result in fractional shares.
Here is an example of the actual calculations:
You own 75 shares of Company XYZ, and a 50% stock dividend has just been declared. You are entitled to receive 37.5 shares, but the company will only issue whole shares. You therefore receive 37 shares, plus a cash in lieu payment of $10.00 for the 1/2 share. Your adjusted cost basis for the initial 75 shares was $1000.00.
After the stock split, you own 75 plus 37.5 shares, for a total of 112.5 shares. Your adjusted cost basis per share is $1000.00 divided by 112.5 shares, or $8.89 per share.
The cost basis allocable to the fractional .5 shares is $8.89 x .5 = $4.44. The net gain to be reported on your Schedule D for the cash in lieu payment is therefore $10.00 less $4.44 or a net gain of only $5.56.
Instead of being taxed on $10.00, with a little effort on your part to allocate your cost basis to the fractional share, you will be taxed on only $5.56 of gain.
Disclaimer: Make sure that you verify such advice with your accountant :->)