No. There are two parts to the stimulus payment: the advance refund (the "check") and the tax credit (in the case of the third stimulus payment, it's a tax credit for the 2021 tax year). For the advance refund, you received the correct amount; and for the tax credit, the lowest it can be is 0 (it cannot be negative).
Let's look at the text of the law. For the 3rd stimulus payment, this is in the American Rescue Plan Act of 2021, section 9601, which inserts 26 USC 6428B. Subsection (g)(1-2) specifies the eligibility and amount for the advance refund:
(1) IN GENERAL.—Subject to paragraphs (5) and (6), each individual who
was an eligible individual for such individual’s first taxable year
beginning in 2019 shall be treated as having made a payment against
the tax imposed by chapter 1 for such taxable year in an amount equal
to the advance refund amount for such taxable year.
(2) ADVANCE REFUND AMOUNT.—
(A) IN GENERAL.—For purposes of paragraph (1), the advance refund
amount is the amount that would have been allowed as a credit under
this section for such taxable year if this section (other than
subsection (f) and this subsection) had applied to such taxable year.
So the eligibility and amount for the advance refund are the eligibility and amount of the tax credit, if the tax credit were applied to the 2019 tax year instead of 2021. There is a provision in subsection (g)(5)(A) where it will use the 2020 tax year instead of 2019 if you had filed your 2020 tax return prior to the initial determination date:
(5) APPLICATION TO INDIVIDUALS WHO HAVE FILED A RETURN OF TAX FOR 2020.—
(A) APPLICATION TO 2020 RETURNS FILED AT TIME OF INITIAL
DETERMINATION.—If, at the time of any determination made pursuant to
paragraph (3), the individual referred to in paragraph (1) has filed a
return of tax for the individual’s first taxable year beginning in
2020, paragraph (1) shall be applied with respect to such individual
by substituting ‘2020’ for ‘2019’.
But this did not apply to you because you had not filed a 2020 tax return prior to the initial determination date. So based on the facts of your situation and the provisions of the law, the amount based on your 2019 tax year, is the correct, and the only correct, amount you should get for the advance refund (the "check").
There is a provision in subsection (f)(1) that coordinates between the advance refund and the tax credit, whereby your stimulus tax credit in the 2021 tax year will be reduced by the amount of the advance refund that you received, but (and this is the key part) it cannot be reduced below 0:
(f) Coordination with advance refunds of credit.—
(1) REDUCTION OF REFUNDABLE CREDIT.—The amount of the credit which
would (but for this paragraph) be allowable under subsection (a) shall
be reduced (but not below zero) by the aggregate refunds and
credits made or allowed to the taxpayer (or, except as otherwise
provided by the Secretary, any dependent of the taxpayer) under
subsection (g). Any failure to so reduce the credit shall be treated
as arising out of a mathematical or clerical error and assessed
according to section 6213(b)(1).
So, in the case where the amount of advance refund you received is greater than or equal to your stimulus tax credit (before subtracting the advance refund), at the worst, you would get 0 tax credit in the 2021 tax year (i.e. you get no additional credit). You would never have "pay the difference back" because that is only possible with a negative stimulus tax credit, and the law does not allow that.