I am not a U.S. citizen and I am leaving the country in a few months. Can I get the money back that I paid for Social Security taxes, since I am not going to be retiring in the United States?
No, you cannot. The FICA taxes paid are not refunded if you're not reaching the benefits threshold. They're gone. That is why foreigners who are not tax residents (mainly students) are not required to pay them.
If your home country has a social security reciprocity agreement with the US - you can have a credit in your home country.