You're talking about modern portfolio theory. The wiki article goes into the math. Here's the gist:
Modern portfolio theory (MPT) is a theory of finance that attempts to maximize portfolio expected return for a given amount of portfolio risk, or equivalently minimize risk for a given level of expected return, by carefully choosing the proportions of various assets.
At the most basic level, you either a) pick a level of risk (standard deviation of your whole portfolio) that you're ok with and find the maximum return you can achieve while not exceeding your risk level, or b) pick a level of expected return that you want and minimize risk (again, the standard deviation of your portfolio). You don't maximize both moments at once.
The techniques behind actually solving them in all but the most trivial cases (portfolios of two or three assets are trivial cases) are basically quadratic programming because to be realistic, you might have a portfolio that a) doesn't allow short sales for all instruments, and/or b) has some securities that can't be held in fractional amounts (like ETF's or bonds). Then there isn't a closed form solution and you need computational techniques like mixed integer quadratic programming
Plenty of firms and people use these techniques, even in their most basic form.
Also your terms are a bit strange:
It has correlation table p11, p12, ... pij, pnn for i and j running from 1 to n
This is usually called the covariance matrix.
I want to maximize 2 variables. Namely the expected return and the additive inverse of the standard deviation of the mixed investments.
Like I said above you don't maximize two moments (return and inverse of risk). I realize that you're trying to minimize risk by maximizing "negative risk" so to speak but since risk and return are inherently a tradeoff you can't achieve the best of both worlds.
Maybe I should point out that although the above sounds nice, and, theoretically, it's sound, as one of the comments points out, it's harder to apply in practice. For example it's easy to calculate a covariance matrix between the returns of two or more assets, but in the simplest case of modern portfolio theory, the assumption is that those covariances don't change over your time horizon.
Also coming up with a realistic measure of your level of risk can be tricky. For example you may be ok with a standard deviation of 20% in the positive direction but only be ok with a standard deviation of 5% in the negative direction. Basically in your head, the distribution of returns you want probably has negative skewness:
because on the whole you want more positive returns than negative returns. Like I said this can get complicated because then you start minimizing other forms of risk like value at risk, for example, and then modern portfolio theory doesn't necessarily give you closed form solutions anymore.
Any actively managed fund that applies this in practice (since obviously a completely passive fund will just replicate the index and not try to minimize risk or anything like that) will probably be using something like the above, or at least something that's more complicated than the basic undergrad portfolio optimization that I talked about above.
We'll quickly get beyond what I know at this rate, so maybe I should stop there.
