Lenders are generally cautious about consumers that own income property which is mortgaged. This is based on the perception that such arrangements are very high risk - what happens if there's an issue with the rental property or it's mortgage that changes your finances to the point that you can no longer afford your newly-purchased primary residence? In part, this fear was driven by the fallout from the mortgage crisis in the US in 2008 - people had tried to build mini-empires by stacking income properties - borrowing against one property to fund downpayments on another - and sometimes including their primary residence in such arrangements. It was easy for these mini-empires to crumble, because an issue in one income property would mean that you no longer had the income to support it.
So, you've got a bit of an uphill battle compared to someone with more "traditional" income from a salary.
And of course, the rules will vary depending on what type of mortgage you're taking out on your new primary residence (i.e. is it fannie/freddie backed?) and also somewhat from bank to bank. For a typical conforming loan for your new primary residence, most banks will count rental income as income in DTI only once you've had at least two annual tax returns which show that income. And the rental income is often discounted by 25%, so if you've got $1,000 of monthly income it'll only count as $750 when figuring DTI.
In terms of which of your two example DTI calculations is used, it's typically neither of those. If you've got two years of tax returns showing rental income, the bank will simply use your net rental income from your tax return (times .75) as income in the denominator of the calculation, along with salary or any other income you have. This is nice and clean because it keeps the whole mess of figuring out rental income as a separate problem from DTI.
If you don't have two years of tax returns showing rental income, you may have to search for a bank that's willing to work with you (and likely it will be on a nonconforming loan for your new primary residence), and you will likely find that they have their own formula for deciding how to include rental income and expense in DTI. Typically, this is done by derating rent by a standard factor - usually 75% - then subtracting PITI (on the rental property, not your new loan), and including that as an estimate of rental income.