It appears that you have been looking for funds through a broker's interface or web site. The transaction fee is one that is being charged by that broker, not by the mutual fund you are buying. (The broker is probably receiving a payment from the high-expense fund and so it eliminates its charge in that case in order to promote that fund. The fund can afford the payment via the higher expenses.)
If you deal directly with the mutual fund you usually will not be charged transaction fees. The Vanguard VTTSX fund you mentioned for instance, shows on its web page that there is no purchase or redemption fee. It's good to read the Prospectus, though. It shows that there is a $20 account service fee each year if the balance is less than $10,000.
And yes, it is important to consider all charges when deciding on an investment. If you invest $1000, the broker's $76 fee is 7.6%, on $10,000 is .76% both of which are substantial parts of your return. Also, the broker probably will charge a fee for getting your money back.
Since some costs are one time but some are continuing (like the expense ratio), you should compare by choosing some likely values and working out the results. E.G., you might think of putting $10000 in, have say a 5% per year investment return less the expenses, and see what you would have after redeeming your investment in 45 years, 10 years, and 2 years (in order to get a feel for the effect of different scenarios).