TL;DR State keeps everything.
What exactly happens largely depends on the particular state's probate laws. These laws are notoriously complicated and vary greatly between the states.
In very rough lines, here's what happens using California as an example.
If a family member left a valid will, the assets are going to be distributed according to the will.
If there's no will (see, intestacy), the state will determine heirs according to intestate succession laws. At this stage the assets may go to distant relatives, e.g. California Probate Code (PROB) §6402:
(g) If there is no surviving next of kin of the decedent and no surviving issue of a predeceased spouse of the decedent, but the decedent is survived by the parents of a predeceased spouse or the issue of those parents, to the parent or parents equally, or to the issue of those parents if both are deceased, the issue taking equally if they are all of the same degree of kinship to the predeceased spouse, but if of unequal degree those of more remote degree take in the manner provided in Section 240.
And, if there're no inheritors, the assets escheat to the state, e.g. PROB §6800:
(a) If a decedent, whether or not the decedent was domiciled in this state, leaves no one to take the decedent’s estate or any portion thereof by testate succession, and no one other than a government or governmental subdivision or agency to take the estate or a portion thereof by intestate succession, under the laws of this state or of any other jurisdiction, the same escheats at the time of the decedent’s death in accordance with this part.
The state will liquidate the assets and distribute the money according to law, e.g. PROB Ch. 6. Distribution to State:
11900. (a) The court shall order property that is not ordered distributed to known beneficiaries to be distributed to the state.
(b) Insofar as practicable, any real property or tangible personal property shall be converted to money before distribution to the state.
11903. (a) Property distributed to the state shall be held by the Treasurer for a period of five years from the date of the order for distribution, within which time any person may claim the property in the manner provided by Title 10 (commencing with Section 1300) of Part 3 of the Code of Civil Procedure.
(b) A person who does not claim the property within the time prescribed in this section is forever barred, and the property vests absolutely in the state, subject to the provisions of Title 10 (commencing with Section 1300) of Part 3 of the Code of Civil Procedure.
Each of these steps has the potential to become Jarndyce v Jarndyce-level nightmare with outstanding debts, multiple wills, and competing heirs.