Since you did not treat the house as a QBU, you have to use USD as your functional currency. To calculate capital gains, you need to calculate the USD value at the time of purchase using the exchange rate at the time of purchase and the USD value at the time of sale using the exchange rate at the time of sale. The capital gain / loss is then the difference between the two. This link describes it in more detail and provides some references:
http://www.maximadvisors.com/2013/06/foreign-residence/
That link also discusses additional potential complications if you have a mortgage on the house.
This link gives more detail on the court case referenced in the above link:
http://www.uniset.ca/other/cs5/93F3d26.html
The court cases references Rev. Rul 54-105. This link from the IRS has some details from that (https://www.irs.gov/pub/irs-wd/0303021.pdf):
Rev. Rul. 54-105, 1954-1 C.B. 12, states that for purposes of determining
gain, the basis and selling price of property acquired by a U.S. citizen living in a foreign country should be expressed in United States dollars at the rates of exchange prevailing as of the dates of purchase and sale of the property,
respectively.
The text of this implies it is for U.S. citizen is living in a foreign country, but the court case makes it clear that it also applies in your scenario (house purchased while living abroad but now residing in the US):
Appellants agree that the 453,374 pounds received for their residence should be translated into U.S. dollars at the $1.82 exchange rate prevailing at the date of sale. They argue, however, that the 343,147 pound adjusted cost basis of the residence, consisting of the 297,500 pound purchase price and the 45,647 pounds paid for capital improvements, likewise should be expressed in U.S. dollar terms as of the date of the sale. Appellants correctly state that, viewed “in the foreign currency in which it was transacted,” the purchase generated a 110,227 pound gain as of the date of the sale, which translates to approximately $200,000 at the $1.82 per pound exchange rate.
...
However fair and reasonable their argument may be, it amounts to an untenable attempt to convert their “functional currency” from the U.S. dollar to the pound sterling.
...
Under I.R.C. § 985(b)(1), use of a functional currency other than the U.S. dollar is restricted to qualified business units ("QBU"s).
...
appellants correctly assert that their residence was purchased “for a pound-denominated value” while they were “living and working in a pound-denominated economy,”
...
And since appellants concede that the purchase and sale of their residence was not carried out by a QBU, the district court properly rejected their plea to treat the pound as their functional currency.
