Growth and efficiency can occur independently of each other. For instance, if an economy consists of one inefficient business and then a second more efficient business opens to compete agains the first the overall efficiency increases while the economy grows.
New industries tend to be inefficient at the beginning (since initiation is more important than optimisation) and then become more efficient over time. Agriculture is an amazingly efficient business if you consider how many people now produce the amount of food we consume in comparison to only 100 years ago.
Plus, efficiency is not only about producing extra widgets. You could produce the same number of widgets for lower cost. Outsourcing to China (taking advantage of their lower cost of production) increases the efficiency of the US economy, but also increases the efficiency of the Chinese economy (since extra work is created producing more things).
Lower costs in the US lead to increased investment in other production. Increased production in China leads to the rising wages there. Growth can be achieved in both places for very different reasons.
So, no, growth doesn't have to come about through less efficiency.