Personally, my solution for this is to use a Transferwise borderless account and debit card (available since 2018). Although not free, rather than adding a load/spread on top of the mid-market rate, they quote the live mid-market rate and then add an explicit fee, which is usually much smaller than the banks. For example, if you were to add 1000 AUD to a Transferwise account and then convert them to JPY, you would pay a fee of 10.32 AUD and get 75,247 JPY at an exchange rate of 76.0319 JPY/AUD, which is the current live mid-market rate. You can then use the debit card associated with your account to take out cash (without extra fees within relatively small limits) or pay within in Japan (without extra fees).
I have personally used such a card for USD, GBP, EUR, SEK, NOK, CHF, and RUB, and it works well for me. You might be cheaper off only with a bank card that truly charges no foreign exchange fee and does not add a load/spread on the mid-market rate, but those may be hard to find (see Is there such a thing as a credit card with no (currency exchange) fees?).