This is really more of a legal question, but I am almost certain the answer is no. I do know that some real estate brokers have a custom of splitting the commission between the seller's broker and the buyer's, but that would be between them and not involve you.
I'd ask this broker to provide you documentation that you owe them the commission. Unless you signed a contract with them it seems unlikely that they will be able to produce anything.
Even if they tried to sue you (unlikely) they would need to provide some evidence that a debt was owed to prevail. No one can unilaterally make you participate in a contract without your consent.
I found this on a Real Estate Attorney's website. It is New York Based, but I'm betting the same principle applies.
A broker is not entitled to compensation on the principle of
ratification merely because a person accepts the result of unsolicited
service. In order to create an implied contract entitling a broker to
a commission, the broker's services must have been performed under
such circumstances as to notify the recipient that the services were
being performed for him and not for another person.
Further, the recipient must have reason to understand that the
services were rendered in expectation of payment of a commission by
the recipient. The recipient must have acted in some manner to
indicate acceptance of the services. Merely selling or leasing a
property to a party voluntarily brought forward by a real estate
broker will not render the seller liable to the broker for a
commission. Consent to a real estate broker's rendition of unsolicited
services will not entitle the broker to a commission without further
indications that the recipient intended to enter into a contract for
the broker's services.
Source