I looked at a stock's ROE for Q3 2011 and it was more than 100%, but the operating margin wasn't very high, around 5%. How is this possible?
A company's Return on Equity (ROE) is its net income divided by its shareholder's equity. The shareholder's equity is the difference between total assets and total liabilities, and is not dependent on the stock price.
What it takes to have a ROE over 100% is to have the income be greater than the equity. This might happen for a variety of reasons, but one way a high ROE happens is if the shareholder's equity (the divisor) is small, which can occur if past losses have eroded the company's capital (the original invested cash and retained earnings). If the equity has become a small value, the income for some period might exceed it, and so the ROE would be over 100%.
Operating margin is not closely related to ROE. Although operating income is related to net income, to calculate the margin you divide by sales, which is completely unrelated to shareholder's equity. So there is no relationship with ROE to be expected. Operating margin is primarily dependent on market conditions, and can be substantially different in different industries.