First off, your commitment to paying down debt and apparent strong relationship with your brother is admirable. However, I think you are overcomplicating your situation and potentially endangering your relationship by attempting to combine debts in this way.
You could consider a simple example where you have interest bearing at 5% and your brother has interest bearing debt at 10%. If you both pay down his higher interest debt first, and then both pay down your debt after, then clearly you will have paid less interest combined. But, by waiting to pay off your debt until later, you have accrued more interest yourself. So who has saved money by doing this? Your brother. You will have paid (let's say, without getting into balances) $50 extra interest to save your brother $70 in interest.
So why would you want to give your brother $50? Total interest savings between both of you in this simplified example are $20. So, in theory your brother could pay you $60 after the fact, effectively meaning you end up $10 ahead, and your brother ends up $10 ahead. Here, you end up in a position where you could still say, in theory 'we both came out ahead'. But what if your brother loses his job while you're both paying off your debt, and he can't help any more? Does he accrue some type of calculated interest until he pays you back? What if he's off work for 2 years and still owes you 30k? What if he just never makes his payments to you on time? At what point do you resent your brother for failing to uphold his end of the deal?
Money and friends don't mix. Money and family mixes even worse.
In rare circumstances where you absolutely must mix family and money, get everything in writing. Get it signed, make it legal. Outline all details of the transaction, including interest rates, and examples of how the balances calculate. In 5 years when things go haywire, following the letter of the law is what will keep you from becoming enemies. But with family, often people have an expectation that "while we agreed I would pay x, he's my brother, so he should take pity on me and allow me to pay only y, if I need to".
Finally, to your question about how to calculate amounts to pay: it will be very complicated. You will need to track minimum balance payments, interest rates, and even potentially the lost income which one of you gives up to pay down the other's debt. You could do these things in a simplified way close to what I've set out above, but then ultimately one of you will lose out. If you pay down your debts first, how can you calculate the lost living potential for your brother, who might want to buy a house but can't save for a down payment for an extra year? What if he has to move, and without sufficient down payment, he needs to pay extra Mortgage Insurance on his loan from the bank? Will you compensate him for that?
My recommendation, if you haven't caught it yet, is Do not do this. Your potential savings are not going to be worth the potential heartache of breaking your relationship with your brother.
Instead, look at joining your minds, not your money. Set goals for yourselves individually, and hold each other accountable. Make this an open conversation between yourselves, as it can be difficult to talk about finances with other people. Your support will help the other person, and hopefully help keep you on track as well.
To provide numerical context for potential savings, which you appear to still want, consider the numbers you've provided [you have 40k debt at 10%, your brother has 20k of debt at 5%]. Let's assume you each can pay up to 20k against the principal of your loans each year. Finally assume for simplicity that you also have enough to pay off interest as it gets charged [so no compounding], and you pay in even instalments each year. Mathematically that means your interest each year is equal to your interest rate * your average annual balance.
If you each go alone, then you will accrue 10% on an average balance of [(40k+20k)/2] = 30k per year, which equals 3,000 in interest in year 1, then [(20k+0)/2] = 10k * .10 = 1,000 interest in year 2. Total interest for you = 4,000. Your brother will accrue [(20k+0k)/2] = 10k * .05 = 500 in interest in total. Total interest for both of you combined would be 4,500.
If you pool your debt snowball, then you will clear your debt first. So the interest on your debt would be [(40k+0k)/2] = 20k * .1 = 2,000. Your brother's debt would fully accrue 5% of interest on the full balance in year 1, so interest in year 1 would be 20k * .05 = 1,000. In year 2, your brother's debt would be cleared half way through the year; interest charged would be [(20k+0k)/2] = 10k * .05 * 50% = 250. You would then owe your brother 10k, which you would pay him over the remainder of year 2. His total interest paid to the bank would be 1,000 + 250 = 1,250. Total interest for both of you combined would be 3,250.
In a simplified payment example using your numbers, maximum interest savings would be about $1,250 combined. How you allocate those savings would be pretty subjective; assuming a 50:50 split, this yields $625 in savings to each of you. If you aren't able to each save 20k per year, then savings would be greater for snowballing, because otherwise it will take you even longer to pay off your high interest debt. This is similar to your brother loaning you 20k today that you can use to pay off your debts, after which you pay him back so he can pay off his. Because you will owe him 20k for 2 years, but an average of ~10k at any one time [because he slowly advances it to you today, and you slowly pay him back until the end of year 2], at $650 in benefit passed to your brother, this is roughly equivalent to him loaning you money at 6.5% interest.
