Why indexing outperforms stock-picking

Traders gather on the floor of the New York Stock Exchange. Photo / AP

Professional equity managers - of mutual funds, pension funds or hedge funds - use a number of methods to try to beat their benchmark index.

These broadly fall into one of two camps, "active" and "passive," and a debate over which gets better results has raged since at least the late 1960s.

Passive managers have been able to claim the upper hand by pointing to the well-known tendency of active equity managers to underperform their benchmark index.

Although this conclusion is widely accepted, there have been few attempts to find out why the index crowd outperforms.

After all, it is one thing for active equity managers to fail to beat the benchmark index -- that could imply only a lack of skill to do better than random selection.

But it is much harder to explain why most active equity managers fail to keep up with the benchmark index, a shortcoming that implies these investment professionals are doing something that systematically leads to underperformance.

The answer, we believe, lies in the fact that the best performing stocks in a broad index perform much better than the other stocks in the index, most of which perform relatively badly. That means average index returns depend heavily on the relatively small set of winners.

We developed a simple model that incorporates this idea.

By randomly selecting a small subset of securities from the index, we maximized the chance of outperforming the index - which validates the allure of active equity management - but it also maximized the chance of underperforming the index, with the chance of underperformance being larger than the chance of overperformance.

Here is a simple illustration of our main idea:

Consider an index of five securities. Four (though we don't know which) will return 10 percent and one will return 50 percent.

Suppose active managers choose portfolios of one or two securities and each investment is weighted equally. There are 15 possible one or two security "portfolios." Of these, 10 will earn returns of 10 percent, because they will include only the 10 percent securities.

Just five of the 15 portfolios will include the 50 percent winner, earning 30 percent if part of a two-security portfolio and 50 percent if it is the sole asset in a one-security portfolio.

The mean average return for all possible actively managed portfolios will be 18 percent; the median actively managed portfolio will earn 10 percent. The equally weighted index of all five securities will earn 18 percent.

In other words, the average active-management return will be the same as the index, but two-thirds of the actively managed portfolios will underperform the index because they will omit the 50 percent winner.

That is, if the typical active manager's picks are no better than a random selection from the benchmark - a view many active managers might endorse, at least when judging their peers - then we can expect those active managers to underperform the index.

Researchers have focused on the costs of active management as primarily the fees paid to the managers.

Our results suggest a much much higher cost: the inherently high probability of underperformance that comes with attempts at stock selection, which disproportionately increases the chance of underperformance relative to the chance of overperformance.

Those allocating assets (such as pension funds and endowments) should revisit the assumption that the only cost of active investing compared with indexing is the active managers' fees.

The benefits of identifying the best active managers may be greater than previously thought.

J.B. Heaton is an attorney with Bartlit Beck Herman Palenchar and Scott in Chicago. Nicholas G. Polson is professor of econometrics and statistics at the University of Chicago Booth School of Business. Jan Hendrik Witte is affiliated with the Mathematical Institute of the University of Oxford.