FHA insured loans must 'go hand in hand' with PMI, because the FHA element is the insurance itself. The FHA isn't actually giving you a loan, that's coming from a lender; instead, the FHA is insuring the loan, at some cost to you - but allowing a loan to folks who may not be able to afford it normally (lower down payment requirements and a somewhat cheaper PMI).
FHA-insured loans may be lower rates in some cases than non-FHA insured loans because of this backing; that's because they make it easier for people of poorer credit histories with smaller down payments to get a house in the first place. Those people would tend to have a harder time getting a loan, and be charged sometimes usurious rates to get it. Low down payment and mediocre credit history (think 580-620) mean higher risk, even beyond the risk directly coming from the poor loan to value ratio.
Comparing this table of Freddie Mac rates to this table of FHA-backed loan rates, the loan rates seem comparable (though somewhat lagging in changes in some cases). FHA loans are not nearly the size or complexity of loan population as Freddie Mac, so be wary of making direct comparisons.
Looking into this in more detail, pre-collapse (before 12/07), FHA rates were a bit lower - average rate was about .5 points lower - but starting with 12/07, FHA average rates were usually higher than Freddie Mac rates for 30 year fixed loans: in 1/2009 for example they were almost a point higher. As of the last data I see (5/13) the rates were within 0.1 points most months.
This may be in part because Freddie Mac had looser requirements to get a loan pre-collapse, then tightened significantly, then started to loosen some (also around June 2013, rates climbed significantly due to some signals from the Fed, although they're almost back to their lows thanks to the Fed again). These are averages across all loans, so you get some noise as a result.
Loan interest rates are very personal, in general: they depend on your credit, your house and down payment, and your bank (which varies by your location). The best thing to do is to shop around yourself and just see what you get, and ask your lender any questions you have: if you pick a local lender with a good service history and who is willing to talk to you in person (ie, has a direct phone number), you'll have no trouble getting answers.