You did not provide the timing so you will need to make adjustments to what I am writing. If each of these transactions were one year apart, for example, then the sum of the present values should equal zero at the proper internal rate of return. It should solve:
10000-20000/(1+r)+20000/(1+r)^2-40000/(1+r)^3=0.
There is only one solution to this problem, which is 100%, as r=1. Now, if your example were different, such as if the first two transactions were firm XYZ and the second two were firm ABC and you sold XYZ concurrently with buying ABC, then you would solve:
10000-20000/(1+r)+20000/(1+r)-40000/(1+r)^2=0
Although r=1 is still the valid solution, but for a different reason. You cannot solve this problem without a clear definition of timing.
Timing matters. Let us imagine you had 53 days between the first and second transactions and you waited 30 days between the second and third and 800 days between the third and the fourth. The correct solution for DAILY return would be:
10000-20000/(1+r)^53+20000/(1+r)^83-40000/(1+r)^883=0
In this case, the DAILY return is approximately .1678%. To find the annual rate of return you would solve this as (1+.001678)^365, which is 84.4%.
You will have to be careful though, because there could, in theory, be 883 solutions to the problem as there are 883 roots. This is not actually a problem because there will be one root for every time it swaps back and forth between positive cash flows and negative cash flows. There will be 883 roots, but it will work out in the above problem that all 883 are equal to the same number, .1678%. Still, it is possible, in your problem format, to have up to three roots. The simplest solution to plug each root into Microsoft Excel's NPV function and only one of them will create a NPV of 0 dollars.
Doing it this way also allows you to include the impact of dividends and any custodial fees if this were something like an IRA. Every time there is a cash flow, you create a term, no matter why the cash flow happened of A/(1+r)^t, with out flows and in flows having opposite signs and netting out all transactions on the same day.
Response to edit
There is only one way to solve it, it is through the internal rate of return, though there is a more complicated modified internal rate of return method, you did not provide additional information so I am assuming it is inappropriate.
You have to include the amount of time that passes to solve this question. If you doubled your money over 100 years or doubled your money in one day, your periodic rate of return would be different depending on how long you decided a period was.
Also note that the number of shares does not matter, only the cash flow matters.
You would have to solve the equation:
10000-10000/(1+r)+15000/(1+r)^2-40000/(1+r)^3=0.
At one year apart, that is roughly a 61% annual rate of return. It simplifies to 2r^3+4r^2+5r=5. You would solve from there. There is no simpler way to do it. Neither formula one nor formula two that you provided are valid. Once you have anything more than a buy and a sell, you cannot use formulas like one or two.