It is not so much that you are miscalculating, but that what you are calculating is not meaningful in the context you're trying to use it ("the annual rate of the loan"). You are essentially trying to compare apples with oranges.
You have taken the total interest paid (over one or 20 years), expressed this as a fraction of the original loan amount, and then "un-compunded" that rate to get a "yearly percentage".
(Your figures are 4.84% for 1 year; 2.71% for 20 years. As noted below, these are "slightly wrong" because (a) you took the interest after an extra month, and (b) you should have used the number of months, not years, in the un-compounding process. My corresponding figures are 4.38% and 2.67%.)
The question then is: what do these rates represent?
They are the rates of return necessary on a single, lump-sum deposit (where interest is compounded monthly) in order to earn the equivalent of the interest paid on the mortgage over the same period.
So, $165,000 deposited for one year at an annual rate of 4.38%, where interest is compounded monthly, would grow by $7,373 after one year.
Similarly, $165,000 deposited for 20 years at an annual rate of 2.67%, where interest is compounded monthly, would grow by $116,280 after twenty years.
(These figures would also apply to an extreme form of a "balloon mortgage" where no repayments were made over the term of the loan, and the entire amount owing (accumulated interest + original principal) was repaid as a lump-sum at the end of the period.)
Yes, the 2.67% is a lot lower than 4.38%, but this just shows the effect of compound-interest over a longer period.
Using the same repayment calculator as you1 and a fixed-deposit calculator I found2, I prepared the following table (which also shows the figures for the full 30-year term):
4.5% Loan repaid after 1 Year 20 Years 30 Years
------- -------- --------
Months: 12 240 360
Loan amount: 165,000 165,000 165,000
Interest paid (a): 7,370 116,315 135,971
Total paid: 172,370 281,315 300,971
Interest as % of loan (b): 4.47% 70.49% 82.41%
Effective annual rate (c): 4.38% 2.67% 2.01%
Return on investment (d): 172,373 281,280 301,400
(a) Interest paid is taken from the loan calculator page, using the October 20xx line in each case. Note: I originally mis-transcribed the 1-year figure as 7,360 (giving very slightly different % rates).
(b) This is the interest divided by the loan amount and expressed as a percentage.
(c) This is the figure from (b) converted to an effective annual rate, using
( ( 1 + interest/loan ) ^ ( 1 / months ) - 1 ) * 12
This is similar to the calculation you made to get 2.71%, except (i) I use the number of months when taking the nth-root, and then multiply by 12; and (ii) I used the October figures where you used the December figures.
(d) These figures are from the fixed-deposit calculator, using the rate as shown in the line above (e.g. "2.67"). If the exact, calculated EAR is used, the calculator's figures agree to the dollar with the interest paid figures from (a).
And the final word: it really does not matter what rate you calculate: the bottom line is that by paying the mortgage off early, you're only paying a little over $7k instead of over $116k in interest!
2 https://www.calculator.com.my/fd-savings. The fixed-deposit calculator happens to show amounts in Malaysian Ringgits (RM), but that does not matter for our purposes: we're only interested in the numbers.