Markets are rational even if they're irrational

I promise very soon to stop beating on the dead carcass of the efficient markets hypothesis (EMH). It's a generally discredited and ill-defined idea which has done a great deal, in my opinion, to prevent clear thinking in finance. But I happened recently on a defense of the EMH by a prominent finance theorist that is simply a wonder to behold -- its logic a true empirical testament to the powers of human rationalization. It also illustrates the borderline Orwellian techniques to which diehard EMH-ers will resort to cling to their favourite idea.

The paper was written in 2000 by Mark Rubinstein, a finance professor at University of California, Berkeley, and is entitled "Rational Markets: Yes or No. The Affirmative Case." It is Rubinstein's attempt to explain away all the evidence against the EMH, from excess volatility to anomalous predictable patterns in price movements and the existence of massive crashes such as the crash of 1987. I'm not going to get into too much detail, but will limit myself to three rather remarkable arguments put forth in the paper. They reveal, it seems to me, the mind of the true believer at work:

1. Rubinstein asserts that his thinking follows from what he calls The Prime Directive. This commitment is itself interesting:

When I went to financial economist training school, I was taught The Prime Directive. That is, as a trained financial economist, with the special knowledge about financial markets and statistics that I had learned, enhanced with the new high-tech computers, databases and software, I would have to be careful how I used this power. Whatever else I would do, I should follow The Prime Directive:

Explain asset prices by rational models. Only if all attempts fail, resort to irrational investor behavior.

One has the feeling from the burgeoning behavioralist literature that it has lost all the constraints of this directive – that whatever anomalies are discovered, illusory or not, behavioralists will come up with an explanation grounded in systematic irrational investor behavior.

Rubinstein here is at least being very honest. He's going to jump through intellectual hoops to preserve his prior belief that people are rational, even though (as he readily admits elsewhere in the text) we know that people are not rational. Hence, he's going to approach reality by assuming something that is definitely not true and seeing what its consequences are. Only if all his effort and imagination fails to come up with a suitable scheme will he actually consider paying attention to the messy details of real human behaviour.

What's amazing is that, having made this admission, he then goes on to criticize behavioural economists for having found out that human behaviour is indeed messy and complicated:

The behavioral cure may be worse than the disease. Here is a litany of cures drawn from the burgeoning and clearly undisciplined and unparsimonious behavioral literature:

Reference points and loss aversion (not necessarily inconsistent with rationality):
Endowment effect: what you start with matters
Status quo bias: more to lose than to gain by departing from current situation
House money effect: nouveau riche are not very risk averse

Overconfidence:
Overconfidence about the precision of private information
Biased self-attribution (perhaps leading to overconfidence)
Illusion of knowledge: overconfidence arising from being given partial information
Disposition effect: want to hold losers but sell winners
Illusion of control: unfounded belief of being able to influence events

Statistical errors:
Gambler’s fallacy: need to see patterns when in fact there are none
Very rare events assigned probabilities much too high or too low
Ellsberg Paradox: perceiving differences between risk and uncertainty
Extrapolation bias: failure to correct for regression to the mean and sample size
Excessive weight given to personal or antidotal experiences over large sample statistics
Overreaction: excessive weight placed on recent over historical evidence
Failure to adjust probabilities for hindsight and selection bias

Miscellaneous errors in reasoning:Violations of basic Savage axioms: sure-thing principle, dominance, transitivity
Sunk costs influence decisions
Preferences not independent of elicitation methods
Compartmentalization and mental accounting
“Magical” thinking: believing you can influence the outcome when you can’t
Dynamic inconsistency: negative discount rates, “debt aversion”
Tendency to gamble and take on unnecessary risks
Overpricing long-shots
Selective attention and herding (as evidenced by fads and fashions)
Poor self-control
Selective recall
Anchoring and framing biases
Cognitive dissonance and minimizing regret (“confirmation trap”)
Disjunction effect: wait for information even if not important to decision
Time-diversification
Tendency of experts to overweight the results of models and theories
Conjunction fallacy: probability of two co-occurring more probable than a single one

Many of these errors in human reasoning are no doubt systematic across individuals and time, just as behavioralists argue. But, for many reasons, as I shall argue, they are unlikely to aggregate up to affect market prices. It is too soon to fall back to what should be the last line of defense, market irrationality, to explain asset prices. With patience, the anomalies that appear puzzling today will either be shown to be empirical illusions or explained by further model generalization in the context of rationality.

Now, there's sense in the idea that, for various reasons, individual behavioural patterns might not be reflected at the aggregate level. Rubinstein's further arguments on this point aren't very convincing, but at least it's a fair argument. What I find more remarkable is the a priori decision that an explanation based on rational behaviour is taken to be inherently superior to any other kind of explanation, even though we know that people are not empirically rational. Surely an explanation based on a realistic view of human behaviour is more convincing and more likely to be correct than one based on unrealistic assumptions (Milton Friedman's fantasies notwithstanding). Even if you could somehow show that market outcomes are what you would expect if people acted as if they were rational (a dubious proposition), I fail to see why that would be superior to an explanation which assumes that people act as if they were real human beings with realistic behavioural quirks, which they are.

But that's not how Rubinstein sees it. Explanations based on a commitment to taking real human behaviour into account, in his view, have "too much of a flavor of being concocted to explain ex-post observations – much like the medievalists used to suppose there were a different angel providing the motive power for each planet." The people making a commitment to realism in their theories, in other words, are like the medievalists adding epicycles to epicycles. The comparison would seem more plausibly applied to Rubinstein's own rational approach.

2. Rubinstein also relies on the wisdom of crowds idea, but doesn't at all consider the many paths by which a crowd's average assessment of something can go very much awry because individuals are often strongly influenced in their decisions and views by what they see others doing. We've known this going all the way back to the famous 1950s experiments of Solomon Asch on group conformity. Rubinstein pays no attention to that, and simply asserts that we can trust that the market will aggregate information effectively and get at the truth, because this is what group behaviour does in lots of cases:

The securities market is not the only example for which the aggregation of information across different individuals leads to the truth. At 3:15 p.m. on May 27, 1968, the submarine USS Scorpion was officially declared missing with all 99 men aboard. She was somewhere within a 20-mile-wide circle in the Atlantic, far below implosion depth. Five months later, after extensive search efforts, her location within that circle was still undetermined. John Craven, the Navy’s top deep-water scientist, had all but given up. As a last gasp, he asked a group of submarine and salvage experts to bet on the probabilities of different scenarios that could have occurred. Averaging their responses, he pinpointed the exact location (within 220 yards) where the missing sub was found. 

Now I don't doubt the veracity of this account or that crowds, when people make decisions independently and have no biases in their decisions, can be a source of wisdom. But it's hardly fair to cite one example where the wisdom of the crowd worked out, without acknowledging the at least equally numerous examples where crowd behaviour leads to very poor outcomes. It's highly ironic that Rubinstein wrote this paper just as the dot.com bubble was collapsing. How could the rational markets have made such mistaken valuations of Internet companies? It's clear that many people judge values at least in part by looking to see how others were valuing them, and when that happens you can forget the wisdom of the crowds.

Obviously I can't fault Rubinstein for not citing these experiments  from earlier this year which illustrate just how fragile the conditions are under which crowds make collectively wise decisions, but such experiments only document more carefully what has been obvious for decades. You can't appeal to the wisdom of crowds to proclaim the wisdom of markets without also acknowledging the frequent stupidity of crowds and hence the associated stupidity of markets.

3. Just one further point. I've pointed out before that defenders of the EMH in their arguments often switch between two meanings of the idea. One is that the markets are unpredictable and hard to beat, the other is that markets do a good job of valuing assets and therefore lead to efficient resource allocations. The trick often employed is to present evidence for the first meaning -- markets are hard to predict -- and then take this in support of the second meaning, that markets do a great job valuing assets. Rubinstein follows this pattern as well, although in a slightly modified way. At the outset, he begins making various definitions of the "rational market":

I will say markets are maximally rational if all investors are rational.

This, he readily admits, isn't true:

Although most academic models in finance are based on this assumption, I don’t think financial economists really take it seriously. Indeed, they need only talk to their spouses or to their brokers.

But he then offers a weaker version:

... what is in contention is whether or not markets are simply rational, that is, asset prices are set as if all investors are rational.

In such a market, investors may not be rational, they may trade too much or fail to diversify properly, but still the market overall may reflect fairly rational behaviour:

In these cases, I would like to say that although markets are not perfectly rational, they are at least minimally rational: although prices are not set as if all investors are rational, there are still no abnormal profit opportunities for the investors that are rational.

This is the version of "rational markets" he then tries to defend throughout the paper. Note what has happened: the definition of the rational market has now been weakened to only say that markets move unpredictably and give no easy way to make a profit. This really has nothing whatsoever to do with the market being rational, and the definition would be improved if the word "rational" were removed entirely. But I suppose readers would wonder why he was bothering if he said "I'm going to defend the hypothesis that markets are very hard to predict and hard to beat" -- does anyone not believe that? Indeed, this idea of a "minimally rational"  market is equally consistent with a "maximally irrational" market. If investors simply flipped coins to make their decisions, then there would also be no easy profit opportunities, as you'd have a truly random market.

Why not just say "the markets are hard to predict" hypothesis? The reason, I suspect, is that this idea isn't very surprising and, more importantly, doesn't imply anything about markets being good or accurate or efficient. And that's really what EMH people want to conclude -- leave the markets alone because they are wonderful information processors and allocate resources efficiently. Trouble is, you can't conclude that just from the fact that markets are hard to beat. Trying to do so with various redefinitions of the hypothesis is like trying to prove that 2 = 1. Watching the effort, to quote physicist John Bell in another context, "...is like watching a snake trying to eat itself from the tail. It becomes embarrassing for the spectator long before it becomes painful for the snake."

What moves the markets? Part II

High frequency trading makes for markets that produce enormous volumes of data. Such data make it possible to test some of the old chestnuts of market theory -- the efficient markets hypothesis, in particular -- more carefully than ever before. Studies in the past few years show quite clearly, it seems to me, that the EMH is very seriously misleading and isn't really even a good first approximation.

Let me give a little more detail. In a recent post I began a somewhat leisurely exploration of considerable evidence which contradicts the efficient markets idea. As the efficient markets hypothesis (the "weak" version, at least) claims, market prices fully reflect all publicly available information. When new information becomes available, prices respond. In the absence of new information, prices should remain more or less fixed.

Striking evidence against this view comes from studies (now almost ten or twenty years old) showing that markets often make quite dramatic movements even in the absence of any news. I looked at some older studies along these lines in the last post, but stronger evidence comes from studies using electronic news feeds and high-frequency stock data. Are sudden jumps in prices in high frequency markets linked to the arrival of new information, as the EMH says? In a word -- no!

The idea in these studies is to look for big price movements which, in a sense, "stand out" from what is typical, and then see if such movements might have been caused by some "news". A good example is this study by Armand Joulin and colleagues from 2008. Here's how they proceeded. Suppose R(t) is the minute by minute return for some stock. You might take the absolute value of these returns, average them over a couple hours and use this as a crude measure -- call it σ -- of the "typical size" of one-minute stock movements over this interval. An unusually big jump over any minute-long interval will be one for which the magnitude of R is much bigger than σ. 

To make this more specific, Joulin and colleagues defined "s jumps" as jumps for which the ratio |R/σ| > s. The value of s can be 2 or 10 or anything you like. You can look at the data for different values of s, and the first thing the data shows -- and this isn't surprising -- is a distinctive pattern for the probability of observing jumps of size s. It falls off with increasing s, meaning that larger jumps are less likely, and the mathematical form is very simple -- a power law with P(s) being proportional to s^-4, especially as s becomes large (from 2 up to 10 and beyond). This is shown in the figure below (the upper curve):

This pattern reflects the well known "fat tailed" distribution of market returns, with large returns being much more likely than they would be if the statistics followed a Gaussian curve. Translating the numbers into daily events, s jumps of size s = 4 turn out to happen about 8 times each day, while larger jumps of s = 8 occur about once every day and one-half (this is true for each stock).

Now the question is -- are these jumps linked to the announcement of some new information? To test this idea, Joulin and colleagues looked at various news feeds including feeds from Dow Jones and Reuters covering about 900 stocks. These can be automatically scanned for mention of any specific company, and then compared to price movements for that company. The first thing they found is that, on average, a new piece of news arrives for a company about once every 3 days. Given that a stock on average experiences one jump every day and one-half, this immediately implies an imbalance between the number of stock movements and the number of news items. There's not enough news to cause the jumps observed. Stocks move -- indeed, jump -- too frequently.

Conclusion: News sometimes but not always causes market movements, and significant market movements are sometimes but not always caused by news. The EMH is wrong, unless you want to make further excuses that there could have been news that caused the movement, and we just don't recognize it or haven't yet figured out what it is. But that seems like simply positing the existence of further epicycles.

But another part of the Joulin et al. study is even more interesting. Having found a way to divide price jumps into two categories: A) those caused by news (clearly linked to some item in a news feed) and B) those unrelated to any news, it is then possible to look for any systematic differences in the way the market settled down after such a jump. The data show that the volatility of prices, just after a jump, becomes quite high; it then relaxes over time back to the average volatility before the jump. But the relaxation works differently depending on whether the jump was of type A or B: caused by news or not caused by news. The figure below shows how the volatility relaxes back to the norm first for jumps linked to news, and second to jumps not linked to news. The later shows a much slower relaxation:

As the authors comment on this figure,

In both cases, we find (Figure 5) that the relaxation of the excess-volatility follows a power-law in time σ(t) − σ(∞) ∝ t− β (see also [22, 23]). The exponent of the decay is, however, markedly different in the two cases: for news jumps, we find β ≈ 1, whereas for endogenous jumps one has β ≈ 1/2. Our results are compatible with those of [22], who find β ≈ 0.35.

Of course, β ≈ 1/2 implies a much slower relaxation back to the norm (whatever that is!) than does β ≈ 1. Hence, it seems that the market takes a longer time to get back to normal after a no-news jump, whereas it goes back to normal quite quickly after a news-related jump.

No one knows why this should be, but Joulin and colleagues made the quite sensible speculation that a jump clearly related to news is not really surprising, and certainly not unnerving. It's understandable, and traders and investors can decide what they think it means and get on with their usual business. In contrast, a no-news event -- think of the Flash Crash, for example -- is very different. It is a real shock and presents a lingering unexplained mystery. It is unnerving and makes investors uneasy. The resulting uncertainty registers in high volatility.

What I've written here only scratches the surface of this study. For example, one might object that lots of news isn't just linked to the fate of one company, but pertains to larger macroeconomic factors. It may not even mention a specific company but point to a likely rise in the prices of oil or semiconductors, changes influencing whole sectors of the economy and many stocks all at once. Joulin and colleagues tried to take this into account by looking for correlated jumps in the prices of multiple stocks, and indeed changes driven by this kind of news do show up quite frequently. But even accounting for this more broad-based kind of news, they still found that a large fraction of the price movements of individual stocks do not appear to be linked to anything coming in through news feeds. As they concluded in the paper:

Our main result is indeed that most large jumps... are not related to any broadcasted news, even if we extend the notion of ‘news’ to a (possibly endogenous) collective market or sector jump. We find that the volatility pattern around jumps and around news is quite different, confirming that these are distinct market phenomena [17]. We also provide direct evidence that large transaction volumes are not responsible for large price jumps, as also shown in [30]. We conjecture that most price jumps are in fact due to endogenous liquidity micro-crises [19], induced by order flow fluctuations in a situation close to vanishing outstanding liquidity.

Their suggestion in the final sentence is intriguing and may suggest the roots of a theory going far beyond the EMH. I've touched before on early work developing this theory, but there is much more to be said. In any event, however, data emerging from high-frequency markets backs up everything found before -- markets often make violent movements which have no link to news. Markets do not just respond to new information. Like the weather, they have a rich -- and as yet mostly unstudied -- internal dynamics.

Learning in macroeconomics...

I've posted before on macroeconomic models that try to go beyond the "rational expectations" framework by assuming that the agents in an economy are different (they have heterogeneous expectations) and are also not necessarily rational. This approach seems wholly more realistic and believable to me.

In a recent comment, however, ivansml pointed me to this very interesting paper from 2009, which I've enjoyed reading. What the paper does is explore what happens in some of the common rational expectations models if you suppose that agents' expectations aren't formed rationally but rather on the basis of some learning algorithm. The paper shows that learning algorithms of a certain kind lead to the same equilibrium outcome as the rational expectations viewpoint. This IS interesting and seems very impressive. However, I'm not sure it's as interesting as it seems at first.

The reason is that the learning algorithm is indeed of a rather special kind. Most of the models studied in the paper, if I understand correctly, suppose that agents in the market already know the right mathematical form they should use to form expectations about prices in the future. All they lack is knowledge of the values of some parameters in the equation. This is a little like assuming that people who start out trying to learn the equations for, say, electricity and magnetism, already know the right form of Maxwell's equations, with all the right space and time derivatives, though they are ignorant of the correct coefficients. The paper shows that, given this assumption in which the form of the expectations equation is already known, agents soon evolve to the correct rational expectations solution. In this sense, rational expectations emerges from adaptive behaviour.

I don't find this very convincing as it makes the problem far too easy. More plausible, it seems to me, would be to assume that people start out with not much knowledge at all of how future prices will most likely be linked by inflation to current prices, make guesses with all kinds of crazy ideas, and learn by trial and error. Given the difficulty of this problem, and the lack even among economists themselves of great predictive success, this would seem more reasonable. However, it is also likely to lead to far more complexity in the economy itself, because a broader class of expectations will lead to a broader class of dynamics for future prices. In this sense, the models in this paper assume away any kind of complexity from a diversity of views.

To be fair to the authors of the paper, they do spell out their assumptions clearly. They state in fact that they assume that people in their economy form views on likely future prices in the same way modern econometricians do (i.e. using the very same mathematical models). So the gist seems to be that in a world in which all people think like economists and use the equations of modern econometrics to form their expectations, then, even if they start out with some of the coefficients "mis-specified," their ability to learn to use the right coefficients can drive the economy to a rational expectations equilibrium. Does this tell us much?

I'd be very interested in others' reactions to this. I do not claim to know much of anything about macroeconomics. Indeed, one of the nice things about this paper is its clear introduction to some of the standard models. This in itself is quite illuminating. I hadn't realized that the standard models are not any more complex than linear first-order time difference equations (if I have this right) with some terms including expectations. I had seen these equations before and always thought they must be toy models just meant to illustrate the far more complex and detailed models used in real calculations and located in some deep economic book I haven't yet seen, but now I'm not so sure.

Difficulties with learning...

I just finished reading this wonderful short review of game theory (many thanks to ivansml for pointing this out to me) and its applications and limitations by Martin Shubik. It's a little old -- it appeared in the journal Complexity in 1998 -- but offers a very broad perspective which I think still holds today. Game theory in the pure sense generally views agents as coming to their strategies through rational calculation; this perspective has had huge influence in economics, especially in the context of relatively simple games with few players and not too many possible strategies. This part of game theory is well developed, although Shubik suggests there are probably many surprises left to learn.

Where the article really comes alive, however, is in considering the limitations to this strictly rational approach in games of greater complexity. In physics, the problem of two rigid bodies in gravitational interaction can be solved exactly (ignoring radiation, of course), but you get generic chaos as soon as you have three bodies or more. The same is true, Shubik argues, in game theory. Extend the number of players above three and as the number of possible permutations of strategies proliferates it is no longer plausible to assume that agents act rationally. The decision problems become too complex. One might still try to search for optimal N player solutions as a guide to what might be possible, but the rational agent approach isn't likely to be profitable as a guide to the likely behaviour and dynamics in such complex games. I highly recommend Shubik's short article to anyone interested in game theory, and especially its application to real world problems where people (or other agents) really can't hope to act on the basis of rational calculation, but instead have to use heuristics, follow hunches, and learn adaptively as they go.

Some of the points Shubik raises find perfect illustration in a recent study (I posted on it here) of typical dynamics in two-player games when the number of possible strategies gets large. Choose the structure of the games at random and the most likely outcome is a rich ongoing evolution of strategic behaviour which never settles down into any equilibrium. But these games do seem to show characteristic dynamical behaviour such as "punctuated equilibrium" -- long periods of relative quiescence which get broken apart sporadically by episodes of tumultuous change -- and clustered volatility -- the natural clustering together of periods of high variability. These qualitative aspects appear to be generic features of the non-equilibrium dynamics of complex games. Interesting that they show up generically in markets as well.

When problems are too complex -- which is typically the case -- we try to learn and adapt rather than "solving" the problem in any sense. Our learning itself may also never settle down into any stable form, but continually change as we find something that works well for a time, and then suddenly find it fails and we need to learn again.

Crazy economic models

**UPDATED AT END OF POST**

In a recent post I commented on the "fetish of rationality" present in a great deal of mathematical economic theory. Agents in the theories are often assumed to have super-human reasoning abilities and to determine their behaviour and expectations solely through completely rational calculation. In comments, Relja suggested that maybe I'd gone too far and that economists version of rationality isn't all that extreme:

I think critiques like this about rationality in economics miss the point. The rationality assumed in economics is concerned with general trends; generally people pursue pleasure, not pain (according to their own utility functions), they prefer more money to less (an expanded budget constraint leaves them on a higher indifference curve, thus better off), they have consistent preferences (when they're in the mood for chocolate, they're not going to choose vanilla). Correspondingly, firms have the goal of profit maximization - they produce products that somebody will want to buy or they go out of business. Taking the rationality assumption to its "umpteenth" iteration is really quite irrational in itself. A consumer knows that spending 6 years to calculate the mathematically optimal choice of ice-cream is irrational. An economist accordingly knows the same thing. And although assumptions are required for modelling economic scenarios (micro or macro), I seriously doubt that any serious economist would make assumptions that infer such irrationality. :).

I think Relja expressed a well-balanced perspective, has learned some economics in detail, and has taken away from it some conclusions that are, all in all, pretty sound. Indeed, people are goal oriented, don't (usually) prefer pain, and businesses do try to make profits (although whether they try to 'maximize' is an open question). If economists were really just following these reasonable ideas, I would have no problem.

But I also think the problem is worse than Relja may realize. The use of rationality assumptions is more extreme than this, and also decisive in some of the most important areas of economic theory, especially in macroeconomics. A few days ago, John Kay offered this very long and critical essay on the form of modern economic theory. It's worth a read all the way through, but in essence, Kay argues that economics is excessively based on logical deduction of theories from a set of axioms, one of which (usually) is the complete rationality of economic agents:

Rigour and consistency are the two most powerful words in economics today.... They have undeniable virtues, but for economists they have particular interpretations.  Consistency means that any statement about the world must be made in the light of a comprehensive descriptive theory of the world.  Rigour means that the only valid claims are logical deductions from specified assumptions.  Consistency is therefore an invitation to ideology, rigour an invitation to mathematics.  This curious combination of ideology and mathematics is the hallmark of what is often called ‘freshwater economics’ – the name reflecting the proximity of Chicago, and other centres such as Minneapolis and Rochester, to the Great Lakes.

Consistency and rigour are features of a deductive approach, which draws conclusions from a group of axioms – and whose empirical relevance depends entirely on the universal validity of the axioms.

Kay isn't quite as explicit as he might have been, but economist Michael Woodford, in a comment on Kay's argument, goes further in spelling out what Key finds most objectionable -- the so-called rational expectations framework, originally proposed by Robert Lucas, which forms the foundations of today's DGSE (dynamic stochastic equilibrium models). A core assumption of such models is that all individuals in the economy have rational expectations about the future, and that such expectations affect their current behaviour.

Now, if this meant something like Relja's comment suggests it might -- that people are simply forward looking, as we know they are -- this would be fine. But it's not. The form this assumption ultimately takes in these models is to assume that everyone in the economy has fully rational expectations, in that they form their expectations in accordance with the conceivably best and most accurate economic models, even if solving those models might require considerable mathematics and computation (and knowledge of everyones' expectations). As Woodford puts it in his comment,

It has been standard for at least the past three decades to use models in which not only does the model give a complete description of a hypothetical world, and not only is this description one in which outcomes follow from rational behavior on the part of the decision makers in the model, but the decision makers in the model are assumed to understand the world in exactly the way it is represented in the model. More precisely, in making predictions about the consequences of their actions (a necessary component of an accounting for their behavior in terms of rational choice), they are assumed to make exactly the predictions that the model implies are correct (conditional on the information available to them in their personal situation).
                 
This postulate of “rational expectations,” as it is commonly though rather misleadingly known, is the crucial theoretical assumption behind such doctrines as “efficient markets” in asset pricing theory and “Ricardian equivalence” in macroeconomics.  

It is precisely here that modern economics takes the assumption of rationality much too far merely for the sake of mathematical and theoretical rigour. Do economists really believe people form their expectations in this way? It's hard to imagine they could as the live the rest of their lives with people who do not do this. But the important question isn't what they really believe but on what do they base their theories which then get used by governments in policy making? Sadly, these unrealistic assumptions remain in the key models. But these assumptions really have zero plausibility. Woodford again,

[The rational expectations assumption] is often  presented as if it were a simple consequence of an aspiration to internal consistency in one’s model and/or explanation of people’s choices in terms of individual rationality, but in fact it is not a  necessary implication of these methodological commitments. It does not follow from the fact that one believes in the validity of one’s own  model and that one believes that people can be assumed to make rational  choices that they must be assumed to make the choices that would be seen  to be correct by someone who (like the economist) believes in the validity of the predictions of that model. Still less would it follow, if the economist herself accepts the necessity of entertaining the  possibility of a variety of possible models, that the only models that  she should consider are ones -- in each of which everyone in the economy is assumed to understand the correctness of that particular model, -- rather than entertaining beliefs that might (for example) be consistent with  one of the other models in the set that she herself regards as possibly  correct.

This is the sense in which hyper-rationality really does enter into economic theories. It's still pervasive, and still indefensible. It would be infinitely preferable if macro-economists such as Lucas and his followers (one of whom, Thomas Sargent, was perversely and outrageously just awarded the Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel).

**UPDATE**

Blogger sometimes doesn't seem to register comments. Email alerted me to a sharp criticism by ivansml of some of the points I made, but the comment isn't, at least for my browser, yet showing up. Just so it doesn't get lost, ivansml said:

Every assumption is false when understood literally, including rational expectations. The important thing is whether people behave as if they had rational expectations - and answer to that will fortunately depend on particular model and data, not on emotional arguments and expressive vocabulary.

By the way, if you reject RE but accept that expectations matter and should be forward-looking, how do you actually propose to model them? One possible alternative is to have agents who estimate laws of motion from past data and continously update their estimates, which is something that macroeconomists have actually investigated before. And guess what - this process will often converge to rational expectations equilibrium.

Finally, the comment about Nobel Prize (yeah, it's not real Nobel, whatever) for Sargent is a sign of ignorance. Sargent has published a lot on generalizations or relaxations of RE, including the learning literature mentioned above, literature on robustness (where agents distrust their model and choose actions which are robust to model misspecifications) and even agent-based models. In addition to that, the prize citation focuses on his empirical contributions (i.e. testing theories against data). This does not seem like someone who is religiously devoted to "hyper-rationality" and ideology.

To points in response:

1. Yes, the point is precisely to include expectations but to model their formation in some more behaviourally realistic way, through learning algorithms as suggested. I am aware of such work and think it is very important. Indeed, the latter portion of this post from earlier this year looked precisely at this and considered a recent review of work in this area by Cars Hommes and others. The idea is not to assume that everyone forms their expectations identically, that learning is important, that their may be systematic biases and so on. As ivansml notes, there are circumstances in which the model may settle into a rational expectations equilibrium. But there are also many in which it does not. My hunch -- not backed by any evidence that I can point to readily -- is that the rational expectations equilibrium will be increasingly unlikely as the decisions faced by agents in the model become increasingly complex. Very possibly the system won't settle into any equilibrium at all.

But I think ivansml for pointing this out. It is certainly the case that expectations matter, and these should be brought into theory in some plausible and convincing way. Just to finish on this point, this is a quote from the Hommes review article, suggesting that the RE equilibrium doesn't come up very often:

Learning to forecast experiments are tailor-made to test the expectations hypothesis, with all other model assumptions computerized and under control of the experimenter. Different types of aggregate behavior have been observed in different market settings. To our best knowledge, no homogeneous expectations model [rational or irrational] fits the experimental data across different market settings. Quick convergence to the RE-benchmark only occurs in stable (i.e. stable under naive expectations) cobweb markets with negative expectations feedback, as in Muth's (1961) seminal rational expectations paper. In all other market settings persistent deviations from the RE fundamental benchmark seem to be the rule rather than the exception.

2. On his second point about Thomas Sargent, I plead guilty. ivansml is right -- his work is not as one dimensional as my comments made it seem. Indeed, I had been looking into his work over the past weekend for different reasons and had noticed that his work has been fairly wide ranging, and he does deserve credit for trying to relax RE assumptions. (Although he did seem a little snide in one interview I read, suggesting that mainstream macro-economists were not at all surprised by the recent financial crisis.)

So thanks also ivansml for setting me straight. I've changed the offending text above.

Creating desires with advertising...

Vance Packard, American journalist of some 50 years ago, quoted in Satyajit Das' book Extreme Money:

A toothbrush does little but clean teeth. Alcohol is important mostly for making people more or less drunk. An automobile can take one reliably to a destination and back...  There being so little to be said, much must be invented. Social distinction must be associated with a house... sexual fulfillment with a particular... automobile, social acceptance with... a mouthwash, etc. We live surrounded by a systematic appeal to a dream world which all mature, scientific reality would reject. We, quite literally, advertise our commitment to immaturity, mendacity and profound gullibility. It is the hallmark of our culture.

 And this was before color television.

What moves the markets? Part I

It's a key assertion of the Efficient Markets Hypothesis that markets move because of news and information. When new information becomes available, investors quickly respond by buying and selling to register their views on the implications of that information. This is obviously partially true -- new information, or what seems like information, does impact markets.

Yesterday, for example, US Treasury Secretary Timothy Geithner said publicly that, despite the ominous economic and financial climate, there is “absolutely” no chance that another United States financial institution will fail. (At least that's what the New York Times says he said; they don't give a link to the speech.) That was around 10 a.m. Pretty much immediately (see the figure below) the value of Morgan Stanley stock jumped upwards by about 4% as, presumably, investors piled into this stock, now believing that the government would step in the prevent any possible Morgan Stanley collapse in the near future. A clear case of information driving the market:

Of course, this just one example and one can find further examples, hundreds every day. Information moves markets. Academics in finance have made careers by documenting this fact in so-called "event studies" -- looking at the consequences for stock prices of mergers, for example.

But I'm not sure how widely it is appreciated that the Efficient Markets Hypothesis doesn't only say that information moves markets. It also requires that markets should ONLY move when new information becomes available. If rational investors have already taken all available information into account and settled on their portfolios, then there's no reason to change in the absence of new information. Is this true? The evidence -- and there is quite a lot of it -- suggests very strongly that it is not. Markets move all the time, and sometimes quite violently, even in the total absence of any new information.

This is important because it suggests that markets have rich internal dynamics -- they move on their own without any need for external shocks. Theories which have been developed to model such dynamics give markets with realistic statistical fluctuations, including abrupt rallies or crashes. I'm going to explore some of these models in detail at some point, but I wanted first to explore a little of evidence which really does nail the case against the EMH as an adequate picture of markets "in efficient equilibrium."

Anyone watching markets might guess that they fluctuate rather more strongly than any news or information could possibly explain. But research has made this case in quantitative terms as well, beginning with a famous paper by Robert Shiller back in 1981. If you believe the efficient markets idea, then the value of a stock ought to remain roughly equal to the present value of all the future dividends a stock owner can anticipate getting from it. Or, a little more technically, real stock prices should, in Shiller's words, "equal the present value of rationally expected or optimally forecasted future real dividends discounted by a constant real discount rate." Data he studied suggest this isn't close to being true.

For example, the two figures below from his paper plot the real price P of the S&P Index (with the upward growth trend removed) and of the Dow Jones Index versus the actual discounted value P* of dividends those stocks later paid out. The solid lines for the real prices bounce up and down quite wildly while the "rational" prices based on dividends stay fairly smooth (dividends don't fluctuate so strongly, and calculating P* involves taking a moving average over many years, smoothing fluctuations even further).

These figures show what has come to be known as "excess volatility" -- excess movement in markets over and above what you should expect on the basis of markets moving on information alone.

Further evidence that it's more than information driving markets comes from studies specifically looking for correlations between new events and market movements. On Monday, October 19, 1987, the Dow Jones Industrial Average fell by more than 22% in one day. Given a conspicuous lack of any major news on that day, economists David Cutler, James Poterba and Larry Summers (yes, that Larry Summers) were moved soon after to wonder if this was a one-off weird event or if violent movements in the absence of any plausible news might have been common in history. They found that they are. Their study from 1989 looked at news and price movements in a variety of ways, but the most interesting results concern news on the days of the 50 largest singe day movements since the Second World War. A section of their table below shows the date of the event, how much the market moved, and the principle reasons given in the press for why it moved so much:

Within this list, you find some events that seem to fit the EMH idea of information as the driving force. The market fell 6.62 percent on the day in 1955 on which Eisenhower had a heart attack. The outbreak of the Korean War knocked 5.38% off the market. But for many of the events the press struggled mightily to find any plausible causal news. When markets fell 6.73% on September 3, 1946, the press even admitted that there was "No basic reason for the assault on prices."

[Curiously, I seem to have found what looks like a tiny error in this table. It lists the outbreak of the Korean War (25 June, 1950) as explaining the big movement one day later on June 26, 1950. But then it lists "Korean war continues" as an explanation for a movement on June 19, 1950, five days before the war even started!]

Cutler and colleagues ultimately concluded that the arrival of news or information could only explain about one half of the actual observed variation in stock prices. In other words, the EMH leaves out of the picture something which is roughly of equal importance as investors' response to new information.

More recently in 2000, economist Ray Fair of Yale University undertook a similar study which found quite similar conclusions. His abstract explains what he found quite succinctly:

Tick data on the S&P 500 futures contract and newswire searches are used to match events to large five minute stock price changes. 58 events that led to large stock price changes are identified between 1982 and 1999, 41 of which are directly or indirectly related to monetary policy. Many large five minute stock price changes have no events associated with them.

 All in all, not a lot of evidence supporting the EMH view on the exclusive role of information in driving markets. Admittedly, these studies all have a semi-qualitative character based on history, linear regressions and other fairly crude techniques. Still, they make a fairly convincing case.

In the past few years, some physicists have taken this all a bit further using modern news feeds. More on that in the second part of this post. The conclusion doesn't change, however -- the markets appear to have a rich world of internal dynamics even in the absence of any new information arriving from outside.