You don't have to use an agent (broker, as you call it), but it is strongly advised. In some counties lawyers are required, in some not. Check your local requirements. Similarly the escrow companies that usually deal with recording and disbursing of money.
You will probably not be able to get a title insurance without using an escrow service (I'm guessing here, but it makes sense to me). You will not be able to secure financing through a bank or a mortgage broker without an escrow company, and it might be hard without an agent.
Agents required by law to know all the details of the process, and they can guide you through what to do and what to look into. They have experience reading and understanding the inspection reports, they know what to demand from the seller (disclosures, information, etc), they know how and from where to get the HOA docs and disclosures, and can help you negotiate the price knowing the market information (comparable sales, comparable listings, list vs sales statistics, etc). It is hard to do all that alone, but if you do - you should definitely get a discount over the market price of the property of about 5% (the agents' fees are up to 5% mostly).
I bought several properties in California and in other states, and I wouldn't do it without an agent on my side. But if you trust the other side entirely and willing to take the risk of missing a step and having problems later with title, mortgage, insurance or resale, then you can definitely save some money and do it without an agent, and there are people doing that.