To compare apples-to-apples, let's pretend that you are considering saving $5,500 each year ($458 per month), which is the current IRA contribution limit. And we'll compare your 401(k) to a traditional IRA, which is pre-tax just like your 401(k). Let's assume the rate of growth on your investment is 8%.
In the 401(k), when you contribute $458 per month, your employer kicks in another $46 with your 10% match. After 5 years of investment at a rate of growth of 8%, your account is worth $37,032.
In the traditional IRA, with no match, after 5 years of investment at $458 per month and a rate of growth of 8%, your account is worth $33,652.
If the rate of growth and expenses are equal between the 401(k) and the IRA, the match makes a big difference. However, as you noted, 401(k) accounts typically have a limited investment options and higher expenses.
To beat the 401(k) with the IRA in 5 years, you would need to earn about an 11.6% growth rate in the IRA. This is a difference of 3.6%, which can cover a lot of 401(k) expenses.
The longer you are investing, the easier it is for the IRA to beat the 401(k). Looking 10 years into the future, the IRA would only need to earn 9.7% (or 1.7% higher) to meet the 401(k), and a 20 year outlook only requires that the IRA earn 8.75% (only 0.75% higher).
It is important to remember that once you leave your job, you can roll your 401(k) funds into an IRA. So the timeframe you need to look at for this comparison is how long you will be at your current job.
Ultimately, the answer to this problem depends on how long you plan on staying with this employer (which determines how long you will be getting this match) and how bad the 401(k)'s investment options are.
You also mentioned the vesting period. Obviously, if you are planning on leaving the employer before the vesting period is complete, this entire discussion on the match is moot, as you won't be receiving any of the match.
Note: I used Bankrate's investment calculator to determine these returns. It is pretty handy for comparing different investment strategies.