I looked at a stock's ROE for Q3 2011 and it was more than 100%, but the operating margin wasn't very high, around 5%. How is this possible?
An operating margin will not compare with ROE. If a company has even a small margin on a large turnover and has a comparative lower shareholder equity, it ROE will be much higher.
One ratio alone can not analyse a company. You need a full set of ratios and figures.
A company's Return on Equity (ROE) is its net income divided by its shareholder's equity. The shareholder's equity is the difference between total assets and total liabilities, and is not dependent on the stock price.
What it takes to have a ROE over 100% is to have the income be greater than the equity. This might happen for a variety of reasons, but one way a high ROE happens is if the shareholder's equity (the divisor) is small, which can occur if past losses have eroded the company's capital (the original invested cash and retained earnings). If the equity has become a small value, the income for some period might exceed it, and so the ROE would be over 100%.
Operating margin is not closely related to ROE. Although operating income is related to net income, to calculate the margin you divide by sales, which is completely unrelated to shareholder's equity. So there is no relationship with ROE to be expected. Operating margin is primarily dependent on market conditions, and can be substantially different in different industries.