There are many different cash flows involved here— both positive (ie. capital gains) and negative (ie. rent due from each person that lived in the house, and paid to all those with an ownership interest)— with different people having different proportional shares of each. There are multiple start dates on which those various cash flows became relevant to the issue at hand. The good news is that it sounds like you are pretty certain about those dates, and the actual dollar amounts are well established. That turns a lot of this into straightforward cash flow analysis that any CPA should be able to do in their sleep.
The part where it will probably get sticky is the fact that they (your family) were living in the house for part of that time and you lived elsewhere. In that situation, they would owe you rent paid at a fair market rate for that property and in the same proportion as your ownership stake in the property. This is because they received the benefit of your investment (the down payment), while you gave up the potential income you could have had by investing that money elsewhere. Calling it "rent" is troublesome and tends to provoke negative emotions, so you will make this conversation easier if you talk about it as if it was a cash loan.
Note that they do not get to use the capital gains you have made on the property to offset the amount they owe you. There are several ways to explain this:
- First, and most directly: capital gains is not the same as investment income (ie.interest). The windfall from selling the house does not offset the opportunity cost of the lost investment income you could have made had you used that money for some other purpose.
- They have also received capital gains on their own investments. They have no more right to devalue your gains for their own benefit than you do to simply take a portion of their gains just because you feel like it.
- The fact that you lived elsewhere is irrelevant: they are not retroactively paying your rent. Rather, you paid your own rent out of pocket in addition to the opportunity cost borne by holding a non-liquid asset that pays no interest.
- The time they spent living in the house was made possible by your investment, but they must mentally disconnect you from that money. It is no different than a bank loan.
You need to separate the $60,000 loan from your brother, versus the value of the house and the improvements your father made. The amount owed on the house is still $349,000, not $400,000. Even if your dad directly used that money to increase the value of the property to $750,000, your brother is not entitled to a greater portion of that gain because of the loan. What he is entitled to is the outstanding balance plus any interest owed, period. That is how loans work.
Further, your brother is now owed that money from your mom, as she most likely inherited your dads share of the property as well as the debt. How she pays it has no impact on the proportional ownership interest that each of you now has in the house. When she dies, and if a portion of the loan is still outstanding, your brother would be able to collect the remaining balance from the estate before you divide up the remaining assets, and in that case it would reduce the total value that you would stand to receive from your moms estate. In no case does it alter the proportional interest each of you has in the property, because. it. is. separate.
These situations are always difficult, and it's good to remember that you're dealing with family. Try to empathize, but also set the expectation that everyone must do the same. You all deserve to be treated fairly and you all deserve your fair share. If you truly believe that the others are not mature enough to approach this objectively, and if you are unsure about the financial math involved, you can hire a CPA. A good one will be able to give you an exact breakdown of the entire situation using well established methods and explain all the details in a way that should leave everyone with the sense that they were treated fairly (not necessarily happy). I'd skip the lawyer unless things get really contentious; you (or your accountant) will probably know if things have reached a point where lawyers are necessary.