I am astonished that financial institutions charge such exorbitant prices for currency conversions
Just to clarify the mark-up is on the exchange rate and not on the total amount. Say the rate is 1 unit of Currency A = 50 units of Currency B. The mark of 4% means the rate is will 50*1.04 = 52 units of Currency B.
Why do these fees exist?
From a Banks point of view, they are operating in Currency A; Currency B is a commodity [similar to Oil, Grains, Goods, etc]. So they will only buy if they can sell it at a margin.
Currency Conversion have inherent risks, on small amount, the Bank generally does not hedge these risks as it is expensive; but balances the position end of day or if the exposure becomes large. The rate they may get then may be different and the margin covers it. Hence on highly traded currency pairs; the spread is less.
Are there back-end processes and requirements that require financial institutions to pass off the loss to consumers as a fee?
The processes are to ensure bank does not make loss.
is it just to make money on the convenience of international transactions?
Banks do make money on such transactions; however they also take some risks. The Forex market is not single market, but is a collective hybrid market place. There are costs a bank incurs to carry and square off positions and some of it is reflected in fees.
If you see some of the remittance corridors, banks have optimized a remittance service; say USD to INR, there is a huge flow often in small amounts. The remittance service aggregates such amounts to make it a large amount to get a better deal for themselves and passes on the benefits to individuals. Such volume of scale is not available for other pairs / corridors.