I'll assume you live in the US for the start of my answer -
Do you maximize your retirement savings at work, at least getting your employer's match in full, if they do this.
Do you have any other debt that's at a higher rate?
Is your emergency account funded to your satisfaction? If you lost your job and tenant on the same day, how long before you were in trouble?
The "pay early" question seems to hit an emotional nerve with most people. While I start with the above and then segue to "would you be happy with a long term 5% return?" there's one major point not to miss - money paid to either mortgage isn't liquid. The idea of owing out no money at all is great, but paying anything less than "paid in full" leaves you still owing that monthly payment. You can send $400K against your $500K mortgage, and still owe $3K per month until paid. And if you lose your job, you may not so easily refinance the remaining $100K to a lower payment so easily.
If your goal is to continue with real estate, you don't prepay, you save cash for the next deal. Don't know if that was your intent at some point.
Disclosure - my situation - Maxing out retirement accounts was my priority, then saving for college. Over the years, I had multiple refinances, each of which was a no-cost deal. The first refi saved with a lower rate. The second, was in early 2000s when back interest was so low I took a chunk of cash, paid principal down and went to a 20yr from the original 30. The kid starts college, and we target retirement in 6 years. I am paying the mortgage (now 2 years into a 10yr) to be done the month before the kid flies out. If I were younger, I'd be at the start of a new 30 yr at the recent 4.5% bottom. I think that a cost of near 3% after tax, and inflation soon to near/exceed 3% makes borrowing free, and I can invest conservatively in stocks that will have a dividend yield above this. Jane and I discussed the plan, and agree to retire mortgage free.