Part 1: Government Loans
The first thing to know is who offers student loans in the US. The first source is the US federal government itself, primarily as a form of financial aid through the FAFSA program. The interest rates are not 9%, but rather more like 3.75% to 6.5%. These rates are basically set by government fiat - specifically, they are set by legislative acts by the US congress. In about 2013, congress set the rate of student loans to be tagged to the 10-year US treasury bond + a premium of about 2%.
So, why doesn't Congress just set them lower? In the last election cycle I believe both Clinton and Trump both suggested they would do just that, and generally every election cycle for the last 10 or more years has various candidates promise to cut the price of loans and/or college somehow. A quick look at the cost of college in the US - and the cost of student loans to borrowers - makes it pretty clear this hasn't actually been achieved.
But still - why not just set it lower? Of course the answer is at least partly money. After all, does the US government not make a profit on student loans?
Well, that's the funny thing. It is not entirely clear if the government turns a profit, breaks even, or actually loses money on undergraduate student loans. This CNN Money article gives a good overview of the topic, but the short answer is: we don't actually know, because you don't really know what the cost/profit of a student loan is until its paid back.
The problem with estimating student loans is that they are decidedly unlike 10-year treasury bills, in that student loans regularly take 10, 20, 30, 40+ years to pay back in full - and sometimes they never get paid back at all (death, leaving the country, permanent disability, etc.).
If you want a full and delightfully in depth review of this, please see the General Accounting Office report entitled, "Borrower Interest Rates Cannot Be Set in Advance to Precisely and Consistently Balance Federal Revenues and Costs".
So in short, some ways of estimating the costs of providing loans suggests the government runs around a $1.6 billion profit per year on lending out $100 billion in student loans a year. Other estimates put the price as -$20 billion a year in losses (cost greater than direct return) on the loans, with continued estimated losses for the next decade. Though this is for undergraduate loans - estimates for graduate and graduate PLUS loans (where parents co-sign graduate school loans) seem to agree that this category decidedly turns a profit, perhaps up to 14% a year. This is at minimum partly due to the fact that the rates are higher than undergraduate loans.
No matter how you slice it, reducing the price of student loans would impact the federal government budget. It is therefore a political issue as to what the best way to allocate the budget is. Should they cut spending to fund lower priced loans? Raise taxes, and on who, to cover the cost? Something else? It is not a trivial decision, and requires political will and capital to make it happen. The desire to reduce the cost of education has continually been set against concerns of quality/standards, availability to a broader range of people, education as a long-term investment that is worth more than the loans themselves provide (through increased salaries, international competitiveness, less social costs), all piled on top of a lot less savory reasons (ask who benefits, follow the money, and all the usual vague warnings). Thus the system we have today.
The government loans cost what they cost, have the restrictions and limits they have, because congress has set them that way. Congress has set them that way "because Congress" and because that's the way the US political system has worked so far, trading off the varying competing wills and incentives in this way.
Part 2: Private Student Loan Lenders
The other big source of funds is private lenders, which includes Sallie Mae (which used to be a government organization, but was privatized and acts as an independent corporation...with lots of special political advantages that are out of scope here, but this is the modern nature of business for large corporations for the US - the public-private line is often blurred).
These loans are generally at a higher interest rate - 9% and up is in no way unusual for them, and they go higher.
Why would students take them when they can get cheaper loans? Well, sometimes it is just that they didn't know better, but I'd say the vast majority of the time is because they did not in fact have a cheaper loan option. Federal loans are considered a form of aid and have various requirements, and are subject to lifetime lending limits as well, generally capping out at $31k-$58k. If you need more loans than that, or don't qualify for other reasons (it does require parents who are willing to cooperate and give their financial information, file taxes, etc.), there are a few other types of financial aid loans (like Perkins loans), but these have limits of their own and only are available in smaller amounts.
So when the government tap out, private lending raises its hand and extends its checkbook - for a fee. This interest rate is not so much set by government fiat, but by "what the market will bear". Competition ideally works to decrease rates, but again as with the GAO report it is hard to estimate either the cost of loans and their collect-ability over decades of time (especially as default rates go up and down due to economic changes and fluctuations, and with technological change so fast it is hard to know who will be able to afford to pay back any given loan 10+ years from now). Thus the costs go up as companies seek more profits and a cover for the risk of having no idea when or whether they will get the money back and in what amounts, driving the costs of further - and of course the bigger the profit the better.
Private loan companies have also "lobbied" quite successfully in making it easier to collect on student loans, including making it so they are almost never discharged in bankruptcy or otherwise (other than death, at least) - yet the cost of providing loans has not plummeted. I hope you won't find me excessively cynical if I suggest there is likely a lot of governmentally protected rent-seeking behavior and regulatory-capture going on, at least, none of which are making loans cheaper.
Part 3: The Cost of Education and Reduced Government Support
Over the last few decades the federal government has tried to reduce its spending on college education, and to provide less non-research support to colleges and Universities to help defray the cost of education. At the same time, State governments have largely done the same, seeking to free up their budgets from the expense of supporting their own state colleges and Universities as well. Colleges and Universities have responded by trying to seek increased income other ways, including rapidly rising tuition - and of course spending has not gone down either.
Another major income source has been seeking to recruit international students and wealthier domestic students, as they receive less aid from colleges and pay higher rates, with international students often paying more tuition than would be legally allowed to be charged to domestic students. These students don't get student loans, generally, because they don't qualify or don't need them - but the demands of trying to recruit high-dollar students may actually be driving up the cost of education as well, but that's going a bit out of scope for our purposes here.
So if the government wants to spend less, and lets politely acknowledge that educational institutions are more than a little bit interested in growing and getting more money for their own panoply of reasons, what's the result?
Well, exactly the system we have today in the US. Costs of education increases have vastly outpaced inflation for decades, student loans are fundamentally hard to account for and predict financially at a governmental level, rapid changes in the economy of left traditional non-extended-education routes to middle incomes harder to secure than prior decades, incomes for the majority of the population have not grown faster than the increased costs of life in the society as it is (considering healthcare and education + general inflation - middle incomes are almost flat against inflation by itself),
A) the US Congress sets the government loan rate provide to the majority of US students, which is less than 9% but now by law more than the treasury rate by a set premium, yet it is unclear how profitable or costly the federal loan program even is, B) private lenders charge as much as the market will bear (with assistance from the government in laws to make it easier to collect and likely limiting competition from reducing rates), and C) in normal financial terms for most potential lenders, student loans are a crazy bad option, with hard to estimate repayment lengths (but which are generally long and unsteady), no collateral, and a highly difficult to predict economy which can heavily influence future repayment. Oh, and the cost and demand of college to students and parents has exploded over the last few decades, so that ain't helping matters either, increasing demands for loans all the while.
Many countries have decided that education is a good that is worth direct government spending and guarantees which are not measured as a direct return-on-investment for capital outlay, and that the society is not well served by a young generation entering adulthood with large debts from such an education - but in general, the US is not one of them when it comes to education beyond high school.
Opinions vary on the folly and wisdom of such a state of things, but after all:
The best laid schemes o’ Mice an’ Men / Gang aft agley, / An’ lea’e us nought but grief an’ pain, / For promis’d joy! - To a Mouse, by Robert Burns