Statistical models for predicting stock prices and models for optimum stock trading do not tell you how many shares to buy for your portfolio. This is the role of Portfolio Optimisation.

You have to optimise for a specific objective and this determines the type of optimisation that is required. Some common types are discussed below.

All the different optimisations have the common element of maximising some measure of return and minimising some measure of risk. This is the basis of "modern portfolio theory."

Mean-Variance optimisation is more a return/risk concept than a practical method. "Mean" refers to an asset's expected return and "variance" to its risk of not achieving that return (usually standard deviation rather than variance is used). The objective is to choose assets to maximise the return for any given level of risk. Or to minimise the risk for any given level of return.

This produces a curve called the efficient frontier and a spot on the frontier is chosen according to an extra parameter outside the optimisation itself such as a utility function or a constraint that the risk must not exceed a certain value.

In real life this optimisation does not work very well. Noone knows what the expected return or risk of a stock really is. Or if you are using a model to estimate them then you still only have quantities that are subject to error and the mean-variance framework does not take into account this estimating error. For example, no matter how you fill in the values and do the optimisation you will always get an efficient frontier that says for some levels of return you can reduce your risk by using leveraging. This goes against intuitive thinking which says that leverage increases risk. The paradox is due to the fact that your efficient frontier is only an estimate of the true frontier and this extra risk is not taken into account by the optimisation.

Another problem with Mean-Variance optimisation is that the variance is not necessarily the best measure of risk. Other more practical measure might be the downside variance, downside skewness, or kurtosis of the asset. A further complication is that the optimisation does not consider transaction costs.

For these reasons there are hundreds of books and thousands of paper on the extension of the Mean-Variance approach. We discuss some extensions below.

Long-Short portfolio construction is not as simple as combining a long and a short portfolio (usually a long portfolio and a hedge). For example, if you do this you may find the long side and the short side having two stocks in common which should never happen in an optimal portfolio. There are correlations between the long and the short portfolios that need to be taken into account. So if you are hedging you need to do portfolio optimisation to calculate the optimum hedge - you just can't apply a hedge as a percentage of the long.

But with Long-Short portfolios there are other constraints that come into play. Usually some constraints will be that the portfolio is dollar neutral or market neutral or both. Dollar neutral means equal dollars are spent on the long and the short sides. Market neutral means that the sum of the betas of the stocks in the portfolio is zero (so that the overall portfolio has no bias if the market moves up or down). These constraints will sacrifice some return - perhaps it will be better to sacrifice some neutrality. So in this sense an "optimal" portfolio may not be what the investor desires.

Standard Mean-Variance optimised portfolio weights are remarkably volatile and unstable over time due to volatility in the estimates of returns and risk. At the very least this will add to transaction costs. In the worst case this will create portfolios that don't even average out to be optimal over time.

A Robust Portfolio optimisation is one that stays largely optimal in face of errors in estimation of returns and risks. A robust optimisation incorporates estimation error directly into the optimisation. There are numerous approaches to this. A simple one is to assume the worst case for all estimates and to optimise accordingly. This produces results that are too pessimistic but helps explain the concepts involved in better methods.

Other methods may use ranks of stock returns instead of the returns themselves. Presumably it is easier to rank stocks than to calculate their returns. Another methods is to use intervals rather than point estimates.

This is a hot area for new research and offers the potential for you to gain an edge over your competitors.

Another approach to robust portfolios is the Black-Litterman method which combines results from models with subjective human views. It sounds like a good way of investing but the drawback is that the humans have to quantify their views by specifying numbers for their expected returns. This adds more uncertainty into the overall optimisation.

The method can also be used to combine estimates from more than one prediction model. It does this by using a Bayesian approach (see Statistical Learning for more details) to combine estimates in what may be regarded as a weighted average. The actual methodology is complicated and uses techniques such as reverse optimisation and implied returns.

The Black-Litterman method is our preferred method for portfolio optimisation.

Tactical asset allocation implement short-term tactical tilts around a long-term policy target. This means two models are operating at the same time - the short term and long term. Black-Litterman methodology is a good way to implement the tilt.

Tilts are often applied qualitatively since it is often a human decision how much of the short term model to include. And then the tilt is applied as a percentage weighting to the whole portfolio rather than looking at individual assets. Black-Litterman gives you a quantitative value for the tilt which means that you get the optimum amount. It also can look into correlations between the two models and adjust the tilt appropriately. So the tilt is applied optimally to individual holdings.

All the optimisation methods discussed above a single time frame methods. They assume that the same conditions will hold for a specified duration such as a month or a year. Stochastic Optimisation allows for variations in the static quantities and can optimise over multiple time frames. Another example is when portfolio requirements change over time such as when planning for retirement.

Stochastic methods can get complicated and are usually simplified by breaking the time frame into discrete time periods. Then the final period is started with first and the optimisation steps backwards in time optimising each step.

There are two two interpretations of index tracking: 1. A passive strategy that seeks to reproduce as closely as possible an index or benchmark portfolio by minimizing the tracking error of the replicating portfolio; or 2. An active strategy that seeks to outperform an index or benchmark portfolio while staying within certain risk boundaries defined by the benchmark.

Either case requires an optimisation that is similar to portfolio optimisation (they give rise to tracking error frontiers). Doing such an optimisation increases the chances of your portfolio tracking the target.

Transaction Costs mess up most portfolio optimisations because they make the optimisation more complicated mathematically. In fact, the optimisation can get so difficult that you only have enough computer time to do it approximately. So the skill in this optimisation is to formulate the problem and use heuristics to get the best accuracy given the resources available.

But it is essential to include transaction costs into a portfolio optimisation because such costs can eat up a significant portion of returns. In fact, many trading schemes and models that appear to make high returns "on paper" make zero returns when transaction costs are included. One of the reasons that stock markets are not perfectly efficient is that transaction costs are too high to eke out every last possible bit of efficiency.

Once you have made some returns in the stock market you would like to know how much of that return was due to your model and how much due to luck. Also you will want to know how much return came from style allocation (choosing, say, large companies over small companies), sector allocation (choosing, say, tech stocks over finance stocks), stock selection (picking better stocks than others), and stock timing (buying stocks at the right time).

Performance Attribution looks back on your portfolio and tells you where your returns came from. It is quite a mathematical process because it has to decompose your returns into all the different categories and the decomposition is done using matrix algebra and optimisation.

Risk Attribution does for volatility what Performance Attribution does for returns - it tells you where your volatility came from.

It may not be as useful as Performance Attribution because it is harder to measure volatility (takes more data) and because it is backwards looking. It just tells you where your past risk was whereas risk management is almost exclusively forward looking and has to consider all risks that may arise even if they have not occurred in the past.

But it is still essential to look at how much volatility there was compared to how much you expected. So even if you don't do Risk Attribution you still need to look at past volatility to calibrate your models.