If you made an excess contribution to Roth IRAs which you did not withdraw by the tax filing deadline, then you are required to fill out Form 5329 Part 4, computing the penalty, when you file your tax return. (And continue to fill it out in future years until the excess is withdrawn or absorbed into a future year's contribution limit.) Form 8606 is not involved unless there is a withdrawal. So it's not the IRS's responsibility to "know" -- it's your responsibility to proactively report and pay the penalty on your tax return.
If you fail to file Form 5329 and fail to add the penalty to your taxes, then your tax return is incorrect. As other answers have mentioned, the IRS receives Form 5498 from the IRA custodian so they can easily find out that your tax return is incorrect. But in general, this belongs to the category of questions of "if I file my tax return incorrectly, how will the IRS know?" For example, if you are married, but you file as Single filing status, since you do not have to provide evidence that you are single, how will the IRS enforce it? Well, they might or might not find out quickly. But if and when they do find out, even if years down the line, you will be in big trouble, and be looking at potential back taxes and penalties with interest.