It's not helpful for me to tell you that you should have considered this 10 years ago. But I say this as a warning to any reader who is planning to buy a house jointly. You need to have an agreement in advance to avoid this kind of issue in the future.
For you - you should get your 20% back, off the top, of course. The real question is - what return? (a) The same rate as the mortgage (b) the same percent the property grew in value over the ten years (c) other.
The improvements are similar but another issue to be resolved. Hopefully, you have contemporaneous records.
In response to OP's comment regarding Joint Tenancy with Rights of Survivorship (JTWROS) - this provision really is important on the death of one owner. The other portion (say half) of the property passes to you without going through probate. It doesn't specifically address your two issues, the downpayment and the upgrades.
I offered 3 choices of how to determine a growth number. JTWROS doesn't address which method to use. In fact, it deems that you own the property equally. So the partner can claim that your $10K 'extra' deposit was a gift, under the limit at the time, and they still own the full 50%. I am back to "without an agreement you need to sit down calmly and discuss the dissolution of this partnership."