Wednesday, February 15, 2012

How markets become efficient (answer: they don't)

A staggering amount of effort has been spent -- and wasted -- exploring the idea of market efficiency. The notoriously malleable efficient markets hypothesis (EMH) claims (in its weakest form) that markets are "information efficient" -- market movements are unpredictable because smart investors keep them that way. They should quickly -- even, "instantaneously" in some statements -- pounce on any predictable pattern in the market, and by profiting will act to wipe out that pattern.

I've written too many times (here, here, here, for example) about the masses of evidence against this idea. It's not that predictable patterns don't attract investors who often act in ways that tend to wipe out those patterns through arbitrage. Part of the problem is that investors often act in ways that amplify the pattern (following trends, for example). Moreover, there are fundamental limits to arbitrage -- "the markets can stay irrational longer than you can stay solvent." Still, the EMH stumbles onward like a zombie -- dead, proven incorrect and misleading, yet still taking center place in the way many people think of markets.

I found an illuminating new perspective on the matter in this recent paper by Doyne Farmer and Spyros Skouras, which explore analogies between finance and ecology. This analogy is itself deeply suggestive. They note, for example, how the interactions between hedge funds can be useful viewed in ecological terms -- funds sometimes act as direct competitors (the profits of one reducing opportunities for another), and in other cases as predator and prey or as symbiotic partners. But I want to look specifically at an effort they make to give a rough estimate of the timescale over which the actions of sophisticated arbitragers might reasonably be expected to wipe out a new predictable pattern in the market. That is, if the market for whatever reason is temporarily inefficient -- showing a predictable pattern -- how quickly should it be returned to efficiency? How long is the time to relaxation that the EMH claims is "instantaneous" or close to it?

The gist of their idea is very simple. Before you can exploit a predictable pattern, you first have to identify it. If you're going to invest money trading against it, you need to be fairly sure you've identified a real pattern, not just a statistical fluke. If you're going to invest somebody else's money, you have to convince them. This takes some time. The stronger the pattern, the more it stands out and the less time it should take to be sure. Weaker signals will be hidden by more noise, and reliable identification will take longer. Looking at how much time it should take to get good statistics should give an order of magnitude of how long a pattern should persist before any smart investor can begin reliably trading against it and (perhaps) erasing it.

Here's the specific argument, expressed using the Sharpe ratio (ratio of expected return to standard deviation of a strategy exploiting the pattern):

This makes obvious intuitive sense. If S is very large, making the pattern obvious, more deterministic and easier to exploit, then the time over which it might be expected to vanish is smaller. Truly obvious patterns can be expected to vanish quickly. But if S is small, the timescale for identification and exploitation grows.

As Farmer and Skouras note, successful investment strategies often have Sharpe ratios of about S = 1, so this gives a result of about 10 years. [This is the result if one makes the analysis on an annual timescale, with the Sharpe ratio calculated on a yearly basis. If we're talking about fast algorithmic trading, then the analysis takes place on a shorter timescale.]

So, 10 years is the order of magnitude estimate -- which is a rather peculiar interpretation of the word "instantaneous." Perhaps that word should be replaced in the EMH with "very slowly," although that somewhat dampens the appeal of the idea: "The EMH asserts that sophisticated investors will very slowly identify and exploit any inefficiencies in the market, tending the erase those inefficiencies over a few decades or so." Given that new inefficiencies can be expected the arise in the mean time, you might as well call this more plausible hypothesis the PIMH: the perpetually inefficient markets hypothesis.

And their estimate, Farmer and Skouras point out, is actually optimistic:
We should stress that this estimate is based on idealized assumptions, such as log-normal returns – heavy tails, autocorrelations, and other effects will tend to make the timescale even longer.... As a given inefficiency is exploited, it will become weaker and the Sharpe ratio of investment strategies associated with it drops. As the Sharpe ratio becomes smaller the fluctuations in its returns become bigger, which can generate uncertainty about whether or not the strategy is still viable. This slows down the approach to inefficiency even more.
Of course, as I mentioned above, this analysis depends on timescale. Take t in years and we're thinking about predictable patterns emerging on the usual investment horizon of a year or longer, patterns exploited by hedge funds and mutual funds of the more traditional (not high frequency) kind.  Here we see that the time to expect predictable patterns to be wiped out is very long indeed. If 10 years is the order of magnitude, then it's likely some of these patterns persist for several decades -- getting up to the time of a typical investing career. Hardcore supporters of the EMH should learn to speak more honestly: "We have every reason to expect that predictable market inefficiencies should be wiped out fairly quickly, at least on the timescale of a human investment career."

All in all, this way of estimating the time for relaxation back to the "efficient equilibrium" suggests that the relaxation is anything but fast, and often very slow. The EMH may be right that there likely aren't any obvious patterns, but more subtle predictable patterns will likely persist for long periods of time, even while they present real profit opportunities. The market is not in equilibrium. And with no mechanism to prevent them, new predictable patterns and "inefficiencies" should be emerging all the time.


  1. Interesting. Could you give me a specific example of a predictable pattern that allows the me to generate systematic abnormal returns?

  2. Every bubble :
    Stock markets, Housing, Subprime, Gold . Now farmland? farmland ?

    Great site btw!

  3. Congrats on finding Doyne Farmer. He's about the best economist there is, these days. Of course, the mainstream label him as an "econo-physicist" much the way writers of "genre fiction" are sneered at by high brow lit types (who also are bitter about why the public prefers the 'genre' stuff). But if you make the naive, idealistic mistake of thinking "economics" is about "scientific understanding of the economy", then by that definition Farmer has few equals.

    Sure the market is never in equilibrium. Neither is the ocean. But on the other hand, you wouldn't want to claim "gravity has no effect on large bodies of water" simply because it can't bring the Pacific ocean to a complete stop. Instead you have to move toward adding models of wave propagation and currents, etc. Fluid dynamics, in other words. The EMH is just the point that "there is a force pushing markets towards price equilibrium". Then an elementary engagement with the real world will quickly demonstrate that "we never really get to equilibrium" and (for extra credit) "because equilibrium itself is a moving target that moves faster than we can catch up to it". (There's a lot more to be said about timescales, of course, since breaking news or ex-dividend dates can produce new consensus prices *very* quickly ... but the point is that the new price is a medium term 'consensus' not an equilibrium.)

    Rob above also illustrates why economics itself as a discipline has trouble converging on the right problems:

    "Interesting. Could you give me a specific example of a predictable pattern that allows the me to generate systematic abnormal returns?"

    It's just way too profitable to make money from people who want specific trading advice instead of actually DOING SCIENCE. You can't reliably predict markets for the same reason you can't build a perpetual motion machine. It's a matter of thermodynamics. (Yes, people can and do exploit trading signals over various time frames, but they do thermodynamic WORK to tune their algorithms to the evolving signals and earn the returns.) But the distraction is ENORMOUS. People smart enough to answer Rob's question correctly are also smart enough to estimate the dollar value of answering it -- and where to go to collect those dollars.

    Farmer has been there (with Prediction Company) yet for whatever reason doesn't conform to contemporary culture by defining himself in dollar terms. (Or at least his marginal utility for dollars has flattened out severely.) The world needs a few people with deviant values like that in order to eke out a little bit of knowledge.

    1. Seth, completely agree with you there.

      Would like to ask you (and/or to Mark), do you believe that the reason the target (price equilibrium) being the moving target is mainly due to lot of participants being fooled by randomness ?

      If you ask right questions you will evidently find right answers. You will be surprised how much less of prediction you need to do to make money.

  4. Aaditya,

    "participants being fooled by randomness" is probably part of the story. It's really an empirical question -- albeit one fraught with difficulties.

    I don't think it is the major explanatory factor, however. A lot of the noise is just dissipation of ambient energy.

    In the ocean analogy, it's good to remember that the ocean is actually *very* 'flat' (or 'spherical' really) on a larger scale. From orbit it probably looks glassy smooth. The problem is that we all operate at smaller scales -- we live *in* the economy, not above it.

    The ambient energy in the ocean is from the rotation of the earth, tides, solar radiation, etc. and it works itself out through circulating currents, surface waves, etc.

    In the financial markets, the differing subjective valuations of securities comes from the different risk exposures/profiles/preferences of the participants. The market was unstable even when it was a highly elite activity (no amateur day traders jumping in with 'unsophisticated' strategies, etc.)

    Find the article "Noise" by Fischer Black for a smart take on this question. It's an earlier mediation on some of the themes in the Farmer & Skouras which Mark cited above.

  5. Mark,

    The article quoted uses the lognormal distribution of returns assumption, which is often cited these days as an inappropriate characterization of markets due to their "fat-tailed" nature. Is it correct that for a more "complicated" return assumption the time delay to discovery would be even longer?

    I also wonder if this is applicable in slower moving situations. I remember my microeconomics class where "in the long run all costs are variable", with the result being a lot of supply and demand graphs are evaluated with the supply curve representing a flat, price-taker firm. Might your comments be applicable to some extent in that situation as well?

  6. Hi,

    Great site! I'm trying to find an email address to contact you on to ask if you would please consider adding a link to my website. I'd really appreciate if you could email me back.

    Thanks and have a great day!

  7. Good website! I truly love how it is easy on my eyes it is.

  8. In this blog, market updates are superb. If anyone want to get latest precious metal updates then join Epic Research, which gives amazing market news.

  9. Did you know that you can create short links with Shortest and make dollars for every click on your shortened links.


  10. شركة نقل عفش
    اهم شركات مكافحة حشرات بالخبر كذلك معرض اهم شركة مكافحة حشرات بالدمام والخبر والجبيل والخبر والاحساء والقطيف كذلك شركة رش حشرات بالدمام ومكافحة الحشرات بالخبر
    شركة مكافحة حشرات بالدمام
    شركة تنظيف خزانات بجدة الجوهرة من افضل شركات تنظيف الخزانات بجدة حيث ان تنظيف خزانات بجدة يحتاج الى مهارة فى كيفية غسيل وتنظيف الخزانات الكبيرة والصغيرة بجدة على ايدى متخصصين فى تنظيف الخزانات بجدة
    شركة تنظيف خزانات بجدة
    شركة كشف تسربات المياه بالدمام
    شركة نقل عفش واثاث

  11. شركة نقل عفش بالرياض وجدة والدمام والخبر والجبيل اولقطيف والاحساء والرياض وجدة ومكة المدينة المنورة والخرج والطائف وخميس مشيط وبجدة افضل شركة نقل عفش بجدة نعرضها مجموعة الفا لنقل العفش بمكة والخرج والقصيم والطائف وتبوك وخميس مشيط ونجران وجيزان وبريدة والمدينة المنورة وينبع افضل شركات نقل الاثاث بالجبيل والطائف وخميس مشيط وبريدة وعنيزو وابها ونجران المدينة وينبع تبوك والقصيم الخرج حفر الباطن والظهران
    شركة نقل عفش بجدة
    شركة نقل عفش بالمدينة المنورة
    شركة نقل اثاث بالرياض
    شركة نقل عفش بالدمام