How markets become efficient (answer: they don't)

A staggering amount of effort has been spent -- and wasted -- exploring the idea of market efficiency. The notoriously malleable efficient markets hypothesis (EMH) claims (in its weakest form) that markets are "information efficient" -- market movements are unpredictable because smart investors keep them that way. They should quickly -- even, "instantaneously" in some statements -- pounce on any predictable pattern in the market, and by profiting will act to wipe out that pattern.

I've written too many times (here, here, here, for example) about the masses of evidence against this idea. It's not that predictable patterns don't attract investors who often act in ways that tend to wipe out those patterns through arbitrage. Part of the problem is that investors often act in ways that amplify the pattern (following trends, for example). Moreover, there are fundamental limits to arbitrage -- "the markets can stay irrational longer than you can stay solvent." Still, the EMH stumbles onward like a zombie -- dead, proven incorrect and misleading, yet still taking center place in the way many people think of markets.

I found an illuminating new perspective on the matter in this recent paper by Doyne Farmer and Spyros Skouras, which explore analogies between finance and ecology. This analogy is itself deeply suggestive. They note, for example, how the interactions between hedge funds can be useful viewed in ecological terms -- funds sometimes act as direct competitors (the profits of one reducing opportunities for another), and in other cases as predator and prey or as symbiotic partners. But I want to look specifically at an effort they make to give a rough estimate of the timescale over which the actions of sophisticated arbitragers might reasonably be expected to wipe out a new predictable pattern in the market. That is, if the market for whatever reason is temporarily inefficient -- showing a predictable pattern -- how quickly should it be returned to efficiency? How long is the time to relaxation that the EMH claims is "instantaneous" or close to it?

The gist of their idea is very simple. Before you can exploit a predictable pattern, you first have to identify it. If you're going to invest money trading against it, you need to be fairly sure you've identified a real pattern, not just a statistical fluke. If you're going to invest somebody else's money, you have to convince them. This takes some time. The stronger the pattern, the more it stands out and the less time it should take to be sure. Weaker signals will be hidden by more noise, and reliable identification will take longer. Looking at how much time it should take to get good statistics should give an order of magnitude of how long a pattern should persist before any smart investor can begin reliably trading against it and (perhaps) erasing it.

Here's the specific argument, expressed using the Sharpe ratio (ratio of expected return to standard deviation of a strategy exploiting the pattern):

This makes obvious intuitive sense. If S is very large, making the pattern obvious, more deterministic and easier to exploit, then the time over which it might be expected to vanish is smaller. Truly obvious patterns can be expected to vanish quickly. But if S is small, the timescale for identification and exploitation grows.

As Farmer and Skouras note, successful investment strategies often have Sharpe ratios of about S = 1, so this gives a result of about 10 years. [This is the result if one makes the analysis on an annual timescale, with the Sharpe ratio calculated on a yearly basis. If we're talking about fast algorithmic trading, then the analysis takes place on a shorter timescale.]

So, 10 years is the order of magnitude estimate -- which is a rather peculiar interpretation of the word "instantaneous." Perhaps that word should be replaced in the EMH with "very slowly," although that somewhat dampens the appeal of the idea: "The EMH asserts that sophisticated investors will very slowly identify and exploit any inefficiencies in the market, tending the erase those inefficiencies over a few decades or so." Given that new inefficiencies can be expected the arise in the mean time, you might as well call this more plausible hypothesis the PIMH: the perpetually inefficient markets hypothesis.

And their estimate, Farmer and Skouras point out, is actually optimistic:

We should stress that this estimate is based on idealized assumptions, such as log-normal returns – heavy tails, autocorrelations, and other effects will tend to make the timescale even longer.... As a given inefficiency is exploited, it will become weaker and the Sharpe ratio of investment strategies associated with it drops. As the Sharpe ratio becomes smaller the fluctuations in its returns become bigger, which can generate uncertainty about whether or not the strategy is still viable. This slows down the approach to inefficiency even more.

Of course, as I mentioned above, this analysis depends on timescale. Take t in years and we're thinking about predictable patterns emerging on the usual investment horizon of a year or longer, patterns exploited by hedge funds and mutual funds of the more traditional (not high frequency) kind.  Here we see that the time to expect predictable patterns to be wiped out is very long indeed. If 10 years is the order of magnitude, then it's likely some of these patterns persist for several decades -- getting up to the time of a typical investing career. Hardcore supporters of the EMH should learn to speak more honestly: "We have every reason to expect that predictable market inefficiencies should be wiped out fairly quickly, at least on the timescale of a human investment career."

All in all, this way of estimating the time for relaxation back to the "efficient equilibrium" suggests that the relaxation is anything but fast, and often very slow. The EMH may be right that there likely aren't any obvious patterns, but more subtle predictable patterns will likely persist for long periods of time, even while they present real profit opportunities. The market is not in equilibrium. And with no mechanism to prevent them, new predictable patterns and "inefficiencies" should be emerging all the time.

a little more on power laws

I wanted to respond to several insightful comments on my recent post on power laws in finance. And, after that, pose a question on the economics/finance history of financial time series that I hope someone out there might be able to help me with.

First, comments:

ivansml said...

Why exactly is power-law distribution for asset returns inconsistent with EMH? It is trivial to write "standard" economic model where returns have fat tails, e.g. if we assume that stochastic process for dividends / firm profits has fat tails. That of course may not be very satisfactory explanation, but it still shows that EMH != normal distribution. In fact, Fama wrote about non-gaussian returns back in 1960's (and Mandelbrot before him), so the idea is not exactly new. The work you describe here is certainly useful and interesting, but pure patterns in data (or "stylized facts", as economists would call them) by themselves are not enough - we need some theory to make sense of them, and it would be interesting to hear more about contributions from econophysics in that area.

James Picerno said...

It's also worth pointing out that EMH, as I understand it, doesn't assume or dismiss that returns follow some specific distribution. Rather, EMH simply posits that prices reflect known information. For many years, analysts presumed that EMH implies a random distribution, but the empirical record says otherwise. But the random walk isn't a condition of EMH. Andrew Lo of MIT has discussed this point at length. The market may or may not be efficient, but it's not conditional on random price fluctuations. Separately, ivansmi makes a good point about models. You need a model to reject EMH. But that only brings you so far. Let's say we have a model of asset pricing that rejects EMH. Then the question is whether EMH or the model is wrong? That requires another model. In short, it's ultimately impossible to reject or accept EMH, unless of course you completely trust a given model. But that brings us back to square one. Welcome to economics.

I actually agree with these statements. Let me try to clarify. In my post I said, referring to the fat tails in returns and 1/t decay of volatility correlations, that  "None of these patterns can be explained by anything in the standard economic theories of markets (the EMH etc)." The key word is of course "explained."

The EMH has so much flexibility and is so loosely linked to real data that it is indeed consistent with these observations, as Ivansml (Mark) and James rightly point out. I think it is probably consistent with any conceivable time series of prices. But "being consistent with" isn't a very strong claim, especially if the consistency comes from making further subsidiary assumptions about how these fat tails might come from fluctuations in fundamental values. This seems like a "just so" story (even if the idea that fluctuations in fundamental values could have fat tails is not at all preposterous).

The point I wanted to make is that nothing (that I know of) in traditional economics/finance (i.e. coming out of the EMH paradigm) gives a natural and convincing explanation of these statistical regularities. Such an explanation would start from simple well accepted facts about the behaviour of individuals, firms, etc., market structures and so on, and then demonstrate how -- because of certain logical consequences following from these facts and their interactions -- we should actually expect to find just these kinds of power laws, with the same exponents, etc., and in many different markets. Reading such an explanation, you would say "Oh, now I see where it comes from and how it works!"

To illustrate some possibilities, one class of proposed explanations sees large market movements as having inherently collective origins, i.e. as reflecting large avalanches of trading behaviour coming out of the interactions of market participants. Early models in this class include the famous Santa Fe Institute Stock Market model developed in the mid 1990s. This nice historical summary by Blake LeBaron explores the motivations of this early agent-based model, the first of which was to include a focus on the interactions among market participants, and so go beyond the usual simplifying assumptions of standard theories which assume interactions can be ignored. As LeBaron notes, this work began in part...

... from a desire to understand the impact of agent interactions and group learning dynamics in a financial setting. While agent-based markets have many goals, I see their first scientific use as a tool for understanding the dynamics in relatively traditional economic models. It is these models for which economists often invoke the heroic assumption of convergence to rational expectations equilibrium where agents’ beliefs and behavior have converged to a self-consistent world view. Obviously, this would be a nice place to get to, but the dynamics of this journey are rarely spelled out. Given that financial markets appear to thrive on diverse opinions and behavior, a first level test of rational expectations from a heterogeneous learning perspective was always needed.   

I'm going to write posts on this kind of work soon looking in much more detail. This early model has been greatly extended and had many diverse offspring; a more recent review by LeBaron gives an updated view. In many such models one finds the natural emergence of power law distributions for returns, and also long-term correlations in volatility. These appear to be linked to various kinds of interactions between participants. Essentially, the market is an ecology of interacting trading strategies, and it has naturally rich dynamics as new strategies invade and old strategies, which had been successful, fall into disuse. The market never settles into an equilibrium, but has continuous ongoing fluctuations.

Now, these various models haven't yet explained anything, but they do pose potentially explanatory mechanisms, which need to be tested in detail. Just because these mechanisms CAN produce the right numbers doesn't mean this is really how it works in markets. Indeed, some physicists and economists working together have proposed a very different kind of explanation for the power law with exponent 3 for the (cumulative) distribution of returns which links it to the known power law distribution of the wealth of investors (and hence the size of the trades they can make). This model sees large movements as arising in the large actions of very wealthy market participants. However, this is more than merely attributing the effect to unknown fat tails in fundamentals, as would be the case with EMH based explanations. It starts with empirical observations of tail behaviour in several market quantities and argues that these together imply what we see for market returns.

There are more models and proposed explanations, and I hope to get into all this in some detail soon. But I hope this explains a little why I don't find the EMH based ideas very interesting. Being consistent with these statistical regularities is not as interesting as suggesting clear paths by which they arise.

Of course, I might make one other point too, and maybe this is, deep down, what I find most empty about the EMH paradigm. It essentially assumes away any dynamics in the market. Fundamentals get changed by external forces and the theory supposes that this great complex mass of heterogenous humanity which is the market responds instantaneously to find the new equilibrium which incorporates all information correctly. So, it treats the non-market part of the world -- the weather, politics, business, technology and so on -- as a rich thing with potentially complicated dynamics. Then it treats the market as a really simply dynamical thing which just gets driven in slave fashion by the outside. This to me seems perversely unnatural and impossible to take seriously. But it is indeed very difficult to rule out with hard data. The idea can always be contorted to remain consistent with observations.

Finally, another valuable comment:

David K. Waltz said...

In one of Taleeb's books, didn't he make mention that something cannot be proven true, only disproven? I think it was the whole swan thing - if you have an appropriate sample and count 100% white swans does not prove there are ONLY white swans, while a sample that has a black one proves that there are not ONLY white swans.

Again, I agree completely. This is a basic point about science. We don't ever prove a theory, only disprove it. And the best science works by trying to find data to disprove a hypothesis, not by trying to prove it.

I assume David is referring to my discussion of the empirical cubic power law for market returns. This is indeed a tentative stylized fact which seems to hold with appreciable accuracy in many markets, but there may well be markets in which it doesn't hold (or periods in which the exponent changes). Finding such deviations  would be very interesting as it might offer further clues as to the mechanism behind this phenomenon.

NOW, for the question I wanted to pose. I've been doing some research on the history of finance, and there's something I can't quite understand. Here's the problem:

1. Mandelbrot in the early 1960s showed that market returns had fat tails; he conjectured that they fit the so-called Stable Paretian (now called Stable Levy) distributions which have power law tails. These have the nice property (like the Gaussian) that the composition of the returns for longer intervals, built up from component Stable Paretian distributions, also has the same form. The market looks the same at different time scales.
2. However, Mandelbrot noted in that same paper a shortcoming of his proposal. You can't think of returns as being independent and identically distributed (i.i.d.) over different time intervals because the volatility clusters -- high volatility predicts more to follow, and vice versa. We don't just have an i.i.d. process.
3. Lots of people documented volatility clustering over the next few decades, and in the 1980s Robert Engle and others introduced ARCH/GARCH and all that -- simple time series models able to reproduce the realistic properties of financial times, including volatility clustering.
4. But today I found several papers from the 1990s (and later) still discussing the Stable Paretian distribution as a plausible model for financial time series.

My question is simply -- why was anyone even 20 years ago still writing about the Stable Paretian distribution when the reality of volatility clustering was so well known? My understanding is that this distribution was proposed as a way to save the i.i.d. property (by showing that such a process can still create market fluctuations having similar character on all time scales). But volatility clustering is enough on its own to rule out any i.i.d. process.

Of course, the Stable Paretian business has by now been completely ruled out by empirical work establishing the value of the exponent for returns, which is too large to be consistent with such distributions. I just can't see why it wasn't relegated to the history books long before.

The only possibility, it just dawns on me, is that people may have thought that some minor variation of the original Mandelbrot view might work best. That is, let the distribution over any interval be Stable Paretian, but let the parameters vary a little from one moment to the next. You give up the i.i.d. but might still get some kind of nice stability properties as short intervals get put together into longer ones. You could put Mandelbrot's distribution into ARCH/GARCH rather than the Gaussian. But this is only a guess. Does anyone know?

What moves the markets? Part I

It's a key assertion of the Efficient Markets Hypothesis that markets move because of news and information. When new information becomes available, investors quickly respond by buying and selling to register their views on the implications of that information. This is obviously partially true -- new information, or what seems like information, does impact markets.

Yesterday, for example, US Treasury Secretary Timothy Geithner said publicly that, despite the ominous economic and financial climate, there is “absolutely” no chance that another United States financial institution will fail. (At least that's what the New York Times says he said; they don't give a link to the speech.) That was around 10 a.m. Pretty much immediately (see the figure below) the value of Morgan Stanley stock jumped upwards by about 4% as, presumably, investors piled into this stock, now believing that the government would step in the prevent any possible Morgan Stanley collapse in the near future. A clear case of information driving the market:

Of course, this just one example and one can find further examples, hundreds every day. Information moves markets. Academics in finance have made careers by documenting this fact in so-called "event studies" -- looking at the consequences for stock prices of mergers, for example.

But I'm not sure how widely it is appreciated that the Efficient Markets Hypothesis doesn't only say that information moves markets. It also requires that markets should ONLY move when new information becomes available. If rational investors have already taken all available information into account and settled on their portfolios, then there's no reason to change in the absence of new information. Is this true? The evidence -- and there is quite a lot of it -- suggests very strongly that it is not. Markets move all the time, and sometimes quite violently, even in the total absence of any new information.

This is important because it suggests that markets have rich internal dynamics -- they move on their own without any need for external shocks. Theories which have been developed to model such dynamics give markets with realistic statistical fluctuations, including abrupt rallies or crashes. I'm going to explore some of these models in detail at some point, but I wanted first to explore a little of evidence which really does nail the case against the EMH as an adequate picture of markets "in efficient equilibrium."

Anyone watching markets might guess that they fluctuate rather more strongly than any news or information could possibly explain. But research has made this case in quantitative terms as well, beginning with a famous paper by Robert Shiller back in 1981. If you believe the efficient markets idea, then the value of a stock ought to remain roughly equal to the present value of all the future dividends a stock owner can anticipate getting from it. Or, a little more technically, real stock prices should, in Shiller's words, "equal the present value of rationally expected or optimally forecasted future real dividends discounted by a constant real discount rate." Data he studied suggest this isn't close to being true.

For example, the two figures below from his paper plot the real price P of the S&P Index (with the upward growth trend removed) and of the Dow Jones Index versus the actual discounted value P* of dividends those stocks later paid out. The solid lines for the real prices bounce up and down quite wildly while the "rational" prices based on dividends stay fairly smooth (dividends don't fluctuate so strongly, and calculating P* involves taking a moving average over many years, smoothing fluctuations even further).

These figures show what has come to be known as "excess volatility" -- excess movement in markets over and above what you should expect on the basis of markets moving on information alone.

Further evidence that it's more than information driving markets comes from studies specifically looking for correlations between new events and market movements. On Monday, October 19, 1987, the Dow Jones Industrial Average fell by more than 22% in one day. Given a conspicuous lack of any major news on that day, economists David Cutler, James Poterba and Larry Summers (yes, that Larry Summers) were moved soon after to wonder if this was a one-off weird event or if violent movements in the absence of any plausible news might have been common in history. They found that they are. Their study from 1989 looked at news and price movements in a variety of ways, but the most interesting results concern news on the days of the 50 largest singe day movements since the Second World War. A section of their table below shows the date of the event, how much the market moved, and the principle reasons given in the press for why it moved so much:

Within this list, you find some events that seem to fit the EMH idea of information as the driving force. The market fell 6.62 percent on the day in 1955 on which Eisenhower had a heart attack. The outbreak of the Korean War knocked 5.38% off the market. But for many of the events the press struggled mightily to find any plausible causal news. When markets fell 6.73% on September 3, 1946, the press even admitted that there was "No basic reason for the assault on prices."

[Curiously, I seem to have found what looks like a tiny error in this table. It lists the outbreak of the Korean War (25 June, 1950) as explaining the big movement one day later on June 26, 1950. But then it lists "Korean war continues" as an explanation for a movement on June 19, 1950, five days before the war even started!]

Cutler and colleagues ultimately concluded that the arrival of news or information could only explain about one half of the actual observed variation in stock prices. In other words, the EMH leaves out of the picture something which is roughly of equal importance as investors' response to new information.

More recently in 2000, economist Ray Fair of Yale University undertook a similar study which found quite similar conclusions. His abstract explains what he found quite succinctly:

Tick data on the S&P 500 futures contract and newswire searches are used to match events to large five minute stock price changes. 58 events that led to large stock price changes are identified between 1982 and 1999, 41 of which are directly or indirectly related to monetary policy. Many large five minute stock price changes have no events associated with them.

 All in all, not a lot of evidence supporting the EMH view on the exclusive role of information in driving markets. Admittedly, these studies all have a semi-qualitative character based on history, linear regressions and other fairly crude techniques. Still, they make a fairly convincing case.

In the past few years, some physicists have taken this all a bit further using modern news feeds. More on that in the second part of this post. The conclusion doesn't change, however -- the markets appear to have a rich world of internal dynamics even in the absence of any new information arriving from outside.