Or is there a theory that proposes inaction is worse?
No, inaction is not always the rational choice in your scenario. To give an extreme example as proof:
Let's say that you have $1,000. The expected value of inaction is $0 - you will end up with the same amount (minus inflation, but we can ignore that for this exercise).
If your only other option is an investment which has a 75% chance of losing all of your money, and a 25% chance of doubling your money, then your expected value is ((75% * -$1,000)+ (25% * $1,000)) = -$500. In that case your best option is obviously inaction.
However, let's say that you have a 75% chance of losing all of your money, but a 25% chance of making $1,000,000 on your investment. In that case, the expected value of investing is ((75% * -$1,000)+ (25% * $1,000,000)) = $249,250. This means that the rational thing is to make the investment. Basically, the 25% chance of making $1M is worth the 75% chance of losing everything.
Other answers here explain very well the intent of your question. However, a word should be said about why people make different choices at all. So to directly answer your question, yes there is a theory but it does not say inaction is worse/better and instead explains why different people make different choices and how this information can be used to construct optimal portfolios for a specific individual. And that theory is called the Risk Aversion Theory. You can visit the following URLs to discover more about this theory:
- Wikipedia article on the Indifference Curve
- Wikipedia article on Risk Aversion: The very first example here attempts to directly answer the question in your post's subject.
- A university's lecture slides on the theory itself
What you're looking for is expected value.
Let's say you invest $1000 in option A, and later discover that you are going to lose 20%. So the expected value of option A is $800.
If option B has a 50% chance of losing 50%, and a 50% of gaining 20%, then your expected value is $650 ($650 = $1000 * (50% * 50% + 50% * 20%)).
$800 > $650, so you should stay with option A.
In the real world, you usually don't know the exact odds and percentages, so substitute in your best guesses. This may lead to suboptimal results, but only if your estimations are wrong; the formula itself is ideal.