I want to build a small empire of my own over time in real estate. I want to pay 20% for a house and rent it out for a small profit (hopefully make around 3 - 5k a year in profit) and then pay it off as soon as possible. Once I pay the house off, I want to repeat the process over and over again. I have no intentions of selling off my house, rather just keep buying homes and renting the homes out. I don't expect to make a lot of money when I first start, but in the long run I am hoping to make a lot of money. I know this won't be a passive income and will be like a part time job when I start out, but this seems like the best opportunity for me. Both in making money and learning a valuable skill in property management. Also, I will not be spending any money I make in real estate on myself. It will go back into my real estate business, unless something comes up where I have to spend on myself for emergency reasons.
Is there a huge flaw in this plan? Is this something realistic or will this not work out? I know it won't be 100% smooth sailing (nothing in life is), but I feel like this could be a good goal to have.
Edit: Sorry, but I forgot to include this. I am using my own money that I make from my main job as well. So it won't be just the 5k in profit I get from renting. I will only dip into my rental profits if I absolutely have to.