If you are a permanent resident (and it wasn't taken away or abandoned), then you are a resident alien for U.S. tax purposes. (One of the two tests for being a resident alien is the "green card test".)
Being a resident alien means all your worldwide income is subject to U.S. taxes, regardless of where you live or work. That doesn't necessarily mean you need to actually pay taxes on your income again if you've already paid it -- you may be able to use the Foreign Tax Credit to reduce your taxes by the amount already paid to a foreign government -- but you need to report it on U.S. tax forms just like income from the U.S., and you can then apply any tax credits that you may qualify for.
As a resident alien, you file taxes using Form 1040. You are required to file taxes if your income for a particular year is above a certain threshold. This threshold is described in the first few pages of the 1040 instructions for each year. For 2013, for Single filing status under 65, it is $10000. The only way you can legally not file is if your income the whole year was below this amount.
You should go back and file taxes if you were required to but failed to. Having filed taxes when required is very important if you want to naturalize later on. It is also one component of demonstrating you're maintaining residency in the U.S., which you're required to do as a permanent resident being outside the U.S. for a long time, or else you'll lose your permanent residency. (Even filing taxes might not be enough, as your description of your presence in the U.S. shows you only go there for brief periods each year, not really living there. You're lucky you haven't lost your green card already; any time you go there you run a great risk of them noticing and taking it away.)