Primary or secondary is a distinction without a difference. You both "co-signed." You both signed the loan, you both owe the money. You both agreed to the terms of the contract.
Paying off the loan would remove your name from the loan. Perhaps your boyfriend can go get a different loan to refinance the car.
If you are buying the next house together, all your loans will be considered, so whether they are joint or sole will not matter.
What will matter is that you may be considered able to service a larger loan if she co-signs, because your car payments will be lower. That won't be the case if your limiting factor is the Loan-to-Value ratio, which will not be ...