Any car manufacturer that undercuts their own dealer network would have that network fall apart quickly. Tesla is using a dealer-free distribution model from the start, so they don't have that problem. Toyota doesn't work that way, though.
GM imposed a uniform no-haggling policy with their Saturn brand, but that policy was coupled with local monopolies for dealers to make it work. Lexus has also experimented with no-haggling and online ordering (with delivery still taking place at a dealership). The rest of Toyota doesn't work that way, though.
Some car manufacturers, such as BMW and Audi, allow you to take delivery of your new car at the factory for a discount. But even then, the transaction still takes place through a dealer. Toyota doesn't work that way, though. For one thing, they work at a different scale. If you buy a Camry in the US, it might be produced in Kentucky, Indiana, or Aichi, depending on business conditions.
You say that you want to cut out the middleman, but the fact is that you do require someone to deliver a Toyota to you, like it or not.
If you're interested in saving money, consider trying various well documented tips, such as negotiating by e-mail before showing up, pitting dealerships against each other. If you don't want to negotiate, you might be able to take advantage of pre-negotiated dealer prices through Costco.
You mentioned that the dealership offered you a 7.99% interest rate for your 710 FICO score. That sounds insanely high — I'd expect deals more like 2% advertised by buyatoyota.com. (Remember, Toyota Motor Credit Corporation exists to help Toyota Motor Corporation sell more cars cheaply.) You can also seek alternate financing online (example) or through your own bank.