(I am neither a lawyer nor financial adviser; this neither legal nor financial advise.)
TL;DR: Refusing to execute an order because of a broker's competing product would seem to be at least against the spirit of both SEC/FINRA regulations. If there are no clauses that specifically prohibit such action, it certainly seems to be against the overarching principle of "play fair" that permeates those regulations.
Excerpting from Section [V.A.1] of the SEC's Guide to Broker-Dealer Registration (and adding some emphasis of my own):
V. CONDUCT REGULATION OF BROKER-DEALERS
A. Antifraud Provisions (Sections 9(a), 10(b), and 15(c)(1) and (2))
The "antifraud" provisions prohibit misstatements or misleading omissions of material facts, and fraudulent or manipulative acts and practices, in connection with the purchase or sale of securities.3
1. Duty of Fair Dealing
Broker-dealers owe their customers a duty of fair dealing. This fundamental duty derives from the Act's antifraud provisions mentioned above. [...] a broker-dealer represents to its customers that it will deal fairly with them, consistent with the standards of the profession. Based on this important representation, the SEC, through interpretive statements and enforcement actions, and the courts, through case law, have set forth over time certain duties for broker-dealers. These include the duties to execute orders promptly, disclose certain material information (i.e., information the customer would consider important as an investor), charge prices reasonably related to the prevailing market, and fully disclose any conflict of interest.
SRO rules also reflect the importance of fair dealing. For example, FINRA members must comply with NASD's Rules of Fair Practice. These rules generally require broker-dealers to observe high standards of commercial honor and just and equitable principles of trade in conducting their business. The exchanges and the MSRB have similar rules.
In general terms, these indicate to me that at all times a broker should act "fair and above board". In particular, if there is a conflict of interest, this must be fully disclosed. In the context of the original question, I would take this to mean that – if they are allowed to refuse to trade some securities at all – then at a minimum any "banned" securities should be clearly listed upfront, before an attempt to trade in them.
(As noted in comments, a broker can refuse to allow trading in certain classes of securities such as options or futures (indeed, in many cases they must refuse to allow someone to trade unless or until they are satisfied that the person is aware of the risks). But refusing to trade specific stocks or ETFs would seem to go against the spirit of the above guidance).
The above guidance says "FINRA members must comply with NASD's Rules of Fair Practice". These rules are many and varied, but Rule 5420 from the FINRA manual includes (emphasis mine):
5240. Anti-Intimidation/Coordination
(a) No member or person associated with a member shall:
[some specifics omitted]
This includes, but is not limited to, any attempt to influence another member or person associated with a member to adjust or maintain a price or quotation, whether displayed on any facility operated by FINRA or otherwise, or refusals to trade or other conduct that retaliates against or discourages the competitive activities of another market maker or market participant.
I believe refusing an order because the broker has their own, competing product, could well be argued as falling foul of the highlighted section.
From the same source, Rule 5310. Best Execution and Interpositioning covers a broker's duty and obligations to execute an order at the best available price (and the required diligence in finding the best price). Some of the supplementary material to that rule would seem applicable to the original question:
.01 Execution of Marketable Customer Orders. A member must make every effort to execute a marketable customer order that it receives fully and promptly.
No mention of "unless the member has a competing product".
.02 Definition of "Market." For the purposes of Rule 5310 and the accompanying Supplementary Material, the term “market” or “markets” is to be construed broadly, and it encompasses a variety of different venues, including, but not limited to, market centers that are trading a particular security. This expansive interpretation is meant to both inform broker-dealers as to the breadth of the scope of venues that must be considered in the furtherance of their best execution obligations and to promote fair competition among broker-dealers, exchange markets, and markets other than exchange markets, as well as any other venue that may emerge, by not mandating that certain trading venues have less relevance than others in the course of determining a firm's best execution obligations.
While the thrust of the rule is about the lengths the broker has to go to in ensuring they get the best price, the highlighted passages indicate that FINRA's expectations of "fair play" are high. Refusing to execute an order because the broker has their own competing fund would not, I suggest, "promote fair competition".
.08 Customer Instructions Regarding Order Handling. If a member receives an unsolicited instruction from a customer to route that customer's order to a particular market for execution, the member is not required to make a best execution determination beyond the customer's specific instruction. Members are, however, still required to process that customer's order promptly and in accordance with the terms of the order.
Again, the main thrust of this supplement is about "best price" (specifically when the member is not obligated to fine one). However, once again the highlighted passage seem to convey FINRA's intention that all such orders must be executed, and, for example, cannot be refused just because the broker has a competing fund.
