The government issues higher yield bonds because it has to.
Government bonds have a nominal value and yearly interest payment. For example, in an environment where a 10-year interest rate is 4.5%, the government most likely chooses a yearly interest payment of 4.5%. This means that when the government borrows $1 million, it has to pay back $1 million at the end of the 10-year period. (In theory, it might be possible for a government to for example choose 0% yearly interest payment, in which case a bond paying back $1 million at end of 10-year period will be sold at a price of 1000000/1.045^10 = $675564.17 so the government can't get million dollars for a million-dollar bond if there are no yearly interest payments)
As for old bonds, the discounted value of the bond changes if interest rates change. So for example if a bond was recently issued at 3.5% interest rate, at this rate the discounted present value is (in GNU Octave notation):
sum(0.035*1000000./1.035.^[1:10]) + 1000000/1.035^10 = 1000000
However, if the interest rate suddenly jumps to 4.5% overnight, then the new value of the bond is:
sum(0.035*1000000./1.045.^[1:10]) + 1000000/1.045^10 = 920872.82
So although you get 3.5% yearly interest payments, the 1% difference in yield is from the fact that you can purchase the bond for $920872.82 and get $1000000 back at the end of the 10-year period.
Note that if you purchase a 10-year bond at interest rate of 3.5% and then the rate jumps to 4.5%, your investment is worth only 92% of what it used to be before that interest rate jump. You made an immediate 8% loss, therefore.
This is why I never purchased (and still haven't purchased) bonds when the interest rates were low. Maybe if the 10-year rate in Europe rises to 4.5% I will reconsider. I also made sure to protect all my loans against interest rate rise whenever such protection was available from the bank.
Recently, bond values have dropped as much as stock values! That's due to interest rate rise reducing their value (and equivalently, increasing their future yield).