Welcome to Money.SE. Please forgive what might sound like a cliche, "How well do you sleep at night?" I mean, specific to the mortgage. There are those who are in a group who consider debt, at any rate, to be inherently bad, and would not take on a 2% mortgage even if a different bank were offering 4% CDs. You just need to understand the risk.
Your mortgage cost after taxes may be 2.625% (if you are in the 25% bracket) therefore, your break even is 3.09% for long term investments. The recent "lost decade" had a return of -9.5% for the full 10 year period. This is just about the worst decade in modern history. The average 10 year return is a cumulative 183% gain, with a standard deviation of 138%. If a perfect bell curve, this means that 1 10 year in 6 will give you a return under 45%. In fact, of the last 100 10 year periods, 15 had returns less than 45%, and just 8 were less than 30%, right in line with the bell curve stats.
We always need to say "past performance is no guarantee of future results," yet, when it comes to the market (I use the S&P for my numbers, by the way) we do have history to give us an idea of the kind of volatility we might see over the years. In my opinion, your approach is sound, and your returns very skewed to the positive, the median 10 year return being 138%, vs your cost of money of 40% or so for a decade.
It's pretty easy to pull S&P data into a spreadsheet and analyze as you wish.