From my understanding, income based repayment for federal student loans is based off of my yearly income. How does it work if I am self-employed and don't have a steady and predictable paycheck every month?
If you're self-employed they'll typically base repayment amounts on your income as shown on prior year tax-return. You re-certify your income annually with your loan servicer, but you can ask your loan servicer to recalculate your payment at any time. You do this by submitting a new application for IBR, and under 'reason for application' you'll indicate that you want a payment recalculation.
To stray from prior year income when self-employed, they'll need documentation of current income, this could be a copy of your ledger or other, you'll have to discuss supporting documentation with your loan servicer.
Here's a good article on Income-driven repayment plans.