Assuming that the conversion was completely non-taxable (i.e. your Traditional IRA was 100% basis), then the converted money can be taken out at any time whatsoever (no 5 year or age stuff), without tax or penalty, similar to directly contributed money. For withdrawing conversions and rollovers within 5 years of the conversion or rollover, the penalty only applies to the part of the conversion or rollover that was taxable. Since in this case the conversion was completely non-taxable, there is no penalty on the withdrawal.
However, note that the ordering of the conversion money is not the same as for contribution money, and this may be significant in some cases. When you take money out of Roth IRA, it goes 1) contributions, 2) rollovers and conversions, and 3) earnings. However, money within (2) is then further divided by year, with rollovers and contributions for earlier years ordered before rollovers and contributions for later years, and then within each year, the taxable rollover and conversion money are ordered first, before the non-taxable money.
So what does that mean? Well, suppose you made a Roth IRA conversion that was taxable one year, and then the next year you make a contribution. If you withdraw a little bit, it comes from the contribution which is ordered first, which means no penalty. Now suppose in that second year you had a backdoor Roth IRA contribution instead of a regular contribution. If you withdraw, the first year's conversion is ordered first, and since it's within 5 years, there's a penalty. It's still true that withdrawing the backdoor Roth IRA has no penalty; but, you don't get to that money until you finish the other one. If you've never made a taxable conversion before, then this issue doesn't exist.