If you're looking for some formula, I don't think one exists. People talk about this all the time and give conflicting advice. If there was a proven-accurate formula, they wouldn't be debating it.
There are basically 3 reasons to do a home improvement project:
(a) Correct a problem so that you prevent on-going damage to your home. For example, have a leaking roof patched or replaced, or exterminate termites. Such a job is worthwhile if the cost of fixing the problem is less than the cost of future damage. In the case of my termite and leaking roof examples, this is almost always worth doing. Lesser maintenance problems might be more debatable.
Similarly, some improvements may reduce expenses. Like replacing an old furnace with a newer model may cut your heating bills. Here the question is: how long does it take to repay the investment, compared to other things you might invest your money in. Just to make up numbers: Suppose you find that a new furnace will save you $500 per year. If the new furnace costs $2000, then it will take 4 years to pay for itself. I'd consider that a good investment. If that same $2000 furnace will only cut your heating bills by $100 per year, then it will take 20 years to pay for itself. You'd probably be better off putting the $2000 into the stock market and using the gains to help pay your heating bill.
(b) Increase the resale value of your home. If you are paying someone else to do the work, the harsh reality here is: Almost no job will increase the resale value by more than the cost of getting the job done. I've seen many articles over the years citing studies on this. I think most conclude that kitchen remodeling comes closet to paying for itself, and bathrooms come next. New windows are also up there.
I don't have studies to prove this, but my guesses would be: Replacing something that is basically nice with a different style will rarely pay for itself. Like, replacing oak cabinets with cherry cabinets.
Replacing something that is in terrible shape with something decent is more likely to pay back than replacing something decent with something beautiful. Like if you have an old iron bathtub that's rusting and falling apart, replacing it may pay off. If you have a 5-year-old bathtub that's in good shape but is not premium, top of the line, replacing it with a premium bathtub will probably do very little for resale value.
If you can do a lot of the work yourself, the story changes. Many home improvement jobs don't require a lot of materials, but do require a lot of work. If you do the labor, you can often get the job done very cheaply, and it's likely that the increase in resale value will be more than what you spend. For example, most of my house has hardwood floors. Lots of people like pretty hardwood floors. I just restained the floors in two rooms. It cost me, I don't know, maybe $20 or $30 for stain and some brushes. I'm sure if I tried to sell the house tomorrow I'd get my twenty bucks back in higher sale value. Realtors often advise sellers to paint. Again, if you do it yourself, the cost of paint may be a hundred dollars, and it can increase the sale price of the house by thousands. Of course if you do the work yourself, you have to consider the value of your time.
(c) To make your home more pleasant to live in. This is totally subjective. You have to make the decision on the same basis that you decide whether anything that is not essential to survival is worth buying. To some people, a bottle of fancy imported wine is worth thousands, even millions, of dollars. Others can't tell the difference between a $10,000 wine and a $15 wine.
The thing to ask yourself is, How important is this home improvement to me, compared to other things I could do with the money? Like, suppose you're considering spending $20,000 remodeling your kitchen. What else could you do with $20,000? You could buy a car, go on an elaborate vacation, eat out several times a week for years, retire a little earlier, etc. No one can tell you how much something is worth to you.
Any given home improvement may involve a combination of these factors. Like say you're considering that $20,000 kitchen remodeling. Say you somehow find out that this will increase the resale value by $15,000. If the only reason you were considering it was to increase resale value, then it's not worth it -- you'd lose $5,000. But if you also want the nicer kitchen, then it is fair to say, Okay, it will cost me $20,000, but ultimately I'll get $15,000 of that back. So in the long run it will only cost me $5,000. Is having a nicer kitchen worth $5,000 to me?
Note, by the way, that resale value only matters if and when you sell the house. If you expect to stay in this house for 20 years, any improvements done are VERY long-term investments. If you live in it until you die, the resale value may matter to your heirs.