Please review the royal mint's guidelines on legal tender. The only silver coins that are legal tender in the UK are commemorative or limited edition coins minted by the royal mint.
If your coins are from the US, they are no different from any other good/product made from silver e.g. cutlery or jewelry and are taxed just the same.
For a full understanding of what qualifies as legal tender in the UK, you should read up on the 1971 Coinage Act. In summary, gold coins generally pass as legal tender while silver passes as legal tender within a strict set of parameters. This is a vestige of a time when currency was gold-backed, hence the lower bar for gold coins relative to other metals (e.g. silver or bronze)
This isn't about fairness of law, it is a question of extant law. Anyone is free to push for a change to what qualifies as legal tender via parliament, but absent such a change, these are the rules.