Blind on purpose: equilibrium as a conceptual filter in economics

A couple of years ago, I came across this article written in The Huffington Post by economist and game theorist David Levine. It carried the provocative title "Why Economists Are Right," and argued back against all those who were then criticizing economics -- especially the rational expectations assumption -- in the aftermath of the financial crisis. Levine's article is delicately crafted and sounds superficially convincing. Indeed, it seems to make the rational expectations idea almost obvious. His argument is a masterpiece of showmanship in the manner of Milton Friedman -- its conclusion seems unavoidable, yet the logic seems somehow fishy, though in a way that is hard to pin down.

The most notable passage in this sense is the following:

In simple language what rational expectations means is "if people believe this forecast it will be true." By contrast if a theory is not one of rational expectations it means "if people believe this forecast it will not be true." Obviously such a theory has limited usefulness. Or put differently: if there is a correct theory, eventually most people will believe it, so it must necessarily be rational expectations. Any other theory has the property that people must forever disbelieve the theory regardless of overwhelming evidence -- for as soon as the theory is believed it is wrong.

Seems convincing, doesn't it? Or at least almost convincing. Is this the only claim made by the rational expectations assumption? If so, maybe it is reasonable. But there's a lot lurking in this paragraph.

When I first read this I thought -- well, he's just assuming that people will learn over time to hold rational beliefs. In other words, he simply asserts (maybe because he believes this) that the only possible outcome in our world has to be an equilibrium. If people have certain beliefs, and their actions based on these lead to a collective outcome that does not confirm those beliefs, then they'll have to adjust those beliefs. There's no equilibrium but ongoing change. From this, Levine assumes that if this goes on for a while that peoples' beliefs will adjust until they lead to actions and collective outcomes that confirm these beliefs and bring about an equilibrium. But this is simply his personal assumption, presumably because he likes game theory and has expertise in game theory and so likes to think about equilibria.

The world is much more flexible. The more general possibility is that people adjust their beliefs, act differently, and their collective behaviour leads to another outcome that against does not confirm their beliefs (at least not perfectly), so they adjust again. And there's an ongoing dance and co-evolution between beliefs and outcomes that never settles into any equilibrium.

But I've kept this essay in the back of my mind, never quite sure if my interpretation made sense, or if the hole in Levine's logic could really be this blazingly obvious. I'm now more strongly convinced that it is, in part because of a beautiful paper I came across yesterday by economist Brian Arthur. Arthur's paper is a wonderful review of the motivation behind complexity science and its application to economics. Two passages resonate in particular with Levine's argument about rational expectations:

One of the earliest insights of economics—it certainly goes back to Smith—is that aggregate patterns [in the economy] form from individual behavior, and individual behavior in turn responds to these aggregate patterns: there is a recursive loop. It is this recursive loop that connects with complexity. Complexity is not a theory but a movement in the sciences that studies how the interacting elements in a system create overall patterns, and how these overall patterns in turn cause the interacting elements to change or adapt. It might study how individual cars together act to form patterns in traffic, and how these patterns in turn cause the cars to alter their position. Complexity is about formation—the formation of structures—and how this formation affects the objects causing it.

To look at the economy, or areas within the economy, from a complexity viewpoint then would mean asking how it evolves, and this means examining in detail how individual agents’ behaviors together form some outcome and how this might in turn alter their behavior as a result. Complexity in other words asks how individual behaviors might react to the pattern they together create, and how that pattern would alter itself as a result. This is often a difficult question; we are asking how a process is created from the purposed actions of multiple agents. And so economics early in its history took a simpler approach, one more amenable to mathematical analysis. It asked not how agents’ behaviors would react to the aggregate patterns these created, but what behaviors (actions, strategies, expectations) would be upheld by—would be consistent with—the aggregate patterns these caused. It asked in other words what patterns would call for no changes in micro-behavior, and would therefore be in stasis, or equilibrium. (General equilibrium theory thus asked what prices and quantities of goods produced and consumed would be consistent with—would pose no incentives for change to—the overall pattern of prices and quantities in the economy’s markets. Classical game theory asked what strategies, moves, or allocations would be consistent with—would be the best course of action for an agent (under some criterion)—given the strategies, moves, allocations his rivals might choose. And rational expectations economics asked what expectations would be consistent with—would on average be validated by—the outcomes these expectations together created.)

This equilibrium shortcut was a natural way to examine patterns in the economy and render them open to mathematical analysis. It was an understandable—even proper—way to push economics forward. And it achieved a great deal. ...  But there has been a price for this equilibrium finesse. Economists have objected to it—to the neoclassical construction it has brought about—on the grounds that it posits an idealized, rationalized world that distorts reality, one whose underlying assumptions are often chosen for analytical convenience. I share these objections. Like many economists I admire the beauty of the neoclassical economy; but for me the construct is too pure, too brittle—too bled of reality. It lives in a Platonic world of order, stasis, knowableness, and perfection. Absent from it is the ambiguous, the messy, the real.

Here I think Arthur has perfectly described the limitation of Levine's position. Levine is happy with rational expectations because he is willing to restrict his field of interest only to those very few special cases in which peoples' expectations do correspond to collective outcomes. Anything else he thinks is uninteresting. I'm not even sure that Levine realizes he has so restricted his field of interest only to equilibrium, thereby neglecting the much larger and richer field of phenomena outside of it.

One other final comment from Arthur, with which I totally agree:

If we assume equilibrium we place a very strong filter on what we can see in the economy. Under equilibrium by definition there is no scope for improvement or further adjustment, no scope for exploration, no scope for creation, no scope for transitory phenomena, so anything in the economy that takes adjustment—adaptation, innovation, structural change, history itself—must be bypassed or dropped from theory. The result may be a beautiful structure, but it is one that lacks authenticity, aliveness, and creation.

Markets are rational even if they're irrational

I promise very soon to stop beating on the dead carcass of the efficient markets hypothesis (EMH). It's a generally discredited and ill-defined idea which has done a great deal, in my opinion, to prevent clear thinking in finance. But I happened recently on a defense of the EMH by a prominent finance theorist that is simply a wonder to behold -- its logic a true empirical testament to the powers of human rationalization. It also illustrates the borderline Orwellian techniques to which diehard EMH-ers will resort to cling to their favourite idea.

The paper was written in 2000 by Mark Rubinstein, a finance professor at University of California, Berkeley, and is entitled "Rational Markets: Yes or No. The Affirmative Case." It is Rubinstein's attempt to explain away all the evidence against the EMH, from excess volatility to anomalous predictable patterns in price movements and the existence of massive crashes such as the crash of 1987. I'm not going to get into too much detail, but will limit myself to three rather remarkable arguments put forth in the paper. They reveal, it seems to me, the mind of the true believer at work:

1. Rubinstein asserts that his thinking follows from what he calls The Prime Directive. This commitment is itself interesting:

When I went to financial economist training school, I was taught The Prime Directive. That is, as a trained financial economist, with the special knowledge about financial markets and statistics that I had learned, enhanced with the new high-tech computers, databases and software, I would have to be careful how I used this power. Whatever else I would do, I should follow The Prime Directive:

Explain asset prices by rational models. Only if all attempts fail, resort to irrational investor behavior.

One has the feeling from the burgeoning behavioralist literature that it has lost all the constraints of this directive – that whatever anomalies are discovered, illusory or not, behavioralists will come up with an explanation grounded in systematic irrational investor behavior.

Rubinstein here is at least being very honest. He's going to jump through intellectual hoops to preserve his prior belief that people are rational, even though (as he readily admits elsewhere in the text) we know that people are not rational. Hence, he's going to approach reality by assuming something that is definitely not true and seeing what its consequences are. Only if all his effort and imagination fails to come up with a suitable scheme will he actually consider paying attention to the messy details of real human behaviour.

What's amazing is that, having made this admission, he then goes on to criticize behavioural economists for having found out that human behaviour is indeed messy and complicated:

The behavioral cure may be worse than the disease. Here is a litany of cures drawn from the burgeoning and clearly undisciplined and unparsimonious behavioral literature:

Reference points and loss aversion (not necessarily inconsistent with rationality):
Endowment effect: what you start with matters
Status quo bias: more to lose than to gain by departing from current situation
House money effect: nouveau riche are not very risk averse

Overconfidence:
Overconfidence about the precision of private information
Biased self-attribution (perhaps leading to overconfidence)
Illusion of knowledge: overconfidence arising from being given partial information
Disposition effect: want to hold losers but sell winners
Illusion of control: unfounded belief of being able to influence events

Statistical errors:
Gambler’s fallacy: need to see patterns when in fact there are none
Very rare events assigned probabilities much too high or too low
Ellsberg Paradox: perceiving differences between risk and uncertainty
Extrapolation bias: failure to correct for regression to the mean and sample size
Excessive weight given to personal or antidotal experiences over large sample statistics
Overreaction: excessive weight placed on recent over historical evidence
Failure to adjust probabilities for hindsight and selection bias

Miscellaneous errors in reasoning:Violations of basic Savage axioms: sure-thing principle, dominance, transitivity
Sunk costs influence decisions
Preferences not independent of elicitation methods
Compartmentalization and mental accounting
“Magical” thinking: believing you can influence the outcome when you can’t
Dynamic inconsistency: negative discount rates, “debt aversion”
Tendency to gamble and take on unnecessary risks
Overpricing long-shots
Selective attention and herding (as evidenced by fads and fashions)
Poor self-control
Selective recall
Anchoring and framing biases
Cognitive dissonance and minimizing regret (“confirmation trap”)
Disjunction effect: wait for information even if not important to decision
Time-diversification
Tendency of experts to overweight the results of models and theories
Conjunction fallacy: probability of two co-occurring more probable than a single one

Many of these errors in human reasoning are no doubt systematic across individuals and time, just as behavioralists argue. But, for many reasons, as I shall argue, they are unlikely to aggregate up to affect market prices. It is too soon to fall back to what should be the last line of defense, market irrationality, to explain asset prices. With patience, the anomalies that appear puzzling today will either be shown to be empirical illusions or explained by further model generalization in the context of rationality.

Now, there's sense in the idea that, for various reasons, individual behavioural patterns might not be reflected at the aggregate level. Rubinstein's further arguments on this point aren't very convincing, but at least it's a fair argument. What I find more remarkable is the a priori decision that an explanation based on rational behaviour is taken to be inherently superior to any other kind of explanation, even though we know that people are not empirically rational. Surely an explanation based on a realistic view of human behaviour is more convincing and more likely to be correct than one based on unrealistic assumptions (Milton Friedman's fantasies notwithstanding). Even if you could somehow show that market outcomes are what you would expect if people acted as if they were rational (a dubious proposition), I fail to see why that would be superior to an explanation which assumes that people act as if they were real human beings with realistic behavioural quirks, which they are.

But that's not how Rubinstein sees it. Explanations based on a commitment to taking real human behaviour into account, in his view, have "too much of a flavor of being concocted to explain ex-post observations – much like the medievalists used to suppose there were a different angel providing the motive power for each planet." The people making a commitment to realism in their theories, in other words, are like the medievalists adding epicycles to epicycles. The comparison would seem more plausibly applied to Rubinstein's own rational approach.

2. Rubinstein also relies on the wisdom of crowds idea, but doesn't at all consider the many paths by which a crowd's average assessment of something can go very much awry because individuals are often strongly influenced in their decisions and views by what they see others doing. We've known this going all the way back to the famous 1950s experiments of Solomon Asch on group conformity. Rubinstein pays no attention to that, and simply asserts that we can trust that the market will aggregate information effectively and get at the truth, because this is what group behaviour does in lots of cases:

The securities market is not the only example for which the aggregation of information across different individuals leads to the truth. At 3:15 p.m. on May 27, 1968, the submarine USS Scorpion was officially declared missing with all 99 men aboard. She was somewhere within a 20-mile-wide circle in the Atlantic, far below implosion depth. Five months later, after extensive search efforts, her location within that circle was still undetermined. John Craven, the Navy’s top deep-water scientist, had all but given up. As a last gasp, he asked a group of submarine and salvage experts to bet on the probabilities of different scenarios that could have occurred. Averaging their responses, he pinpointed the exact location (within 220 yards) where the missing sub was found. 

Now I don't doubt the veracity of this account or that crowds, when people make decisions independently and have no biases in their decisions, can be a source of wisdom. But it's hardly fair to cite one example where the wisdom of the crowd worked out, without acknowledging the at least equally numerous examples where crowd behaviour leads to very poor outcomes. It's highly ironic that Rubinstein wrote this paper just as the dot.com bubble was collapsing. How could the rational markets have made such mistaken valuations of Internet companies? It's clear that many people judge values at least in part by looking to see how others were valuing them, and when that happens you can forget the wisdom of the crowds.

Obviously I can't fault Rubinstein for not citing these experiments  from earlier this year which illustrate just how fragile the conditions are under which crowds make collectively wise decisions, but such experiments only document more carefully what has been obvious for decades. You can't appeal to the wisdom of crowds to proclaim the wisdom of markets without also acknowledging the frequent stupidity of crowds and hence the associated stupidity of markets.

3. Just one further point. I've pointed out before that defenders of the EMH in their arguments often switch between two meanings of the idea. One is that the markets are unpredictable and hard to beat, the other is that markets do a good job of valuing assets and therefore lead to efficient resource allocations. The trick often employed is to present evidence for the first meaning -- markets are hard to predict -- and then take this in support of the second meaning, that markets do a great job valuing assets. Rubinstein follows this pattern as well, although in a slightly modified way. At the outset, he begins making various definitions of the "rational market":

I will say markets are maximally rational if all investors are rational.

This, he readily admits, isn't true:

Although most academic models in finance are based on this assumption, I don’t think financial economists really take it seriously. Indeed, they need only talk to their spouses or to their brokers.

But he then offers a weaker version:

... what is in contention is whether or not markets are simply rational, that is, asset prices are set as if all investors are rational.

In such a market, investors may not be rational, they may trade too much or fail to diversify properly, but still the market overall may reflect fairly rational behaviour:

In these cases, I would like to say that although markets are not perfectly rational, they are at least minimally rational: although prices are not set as if all investors are rational, there are still no abnormal profit opportunities for the investors that are rational.

This is the version of "rational markets" he then tries to defend throughout the paper. Note what has happened: the definition of the rational market has now been weakened to only say that markets move unpredictably and give no easy way to make a profit. This really has nothing whatsoever to do with the market being rational, and the definition would be improved if the word "rational" were removed entirely. But I suppose readers would wonder why he was bothering if he said "I'm going to defend the hypothesis that markets are very hard to predict and hard to beat" -- does anyone not believe that? Indeed, this idea of a "minimally rational"  market is equally consistent with a "maximally irrational" market. If investors simply flipped coins to make their decisions, then there would also be no easy profit opportunities, as you'd have a truly random market.

Why not just say "the markets are hard to predict" hypothesis? The reason, I suspect, is that this idea isn't very surprising and, more importantly, doesn't imply anything about markets being good or accurate or efficient. And that's really what EMH people want to conclude -- leave the markets alone because they are wonderful information processors and allocate resources efficiently. Trouble is, you can't conclude that just from the fact that markets are hard to beat. Trying to do so with various redefinitions of the hypothesis is like trying to prove that 2 = 1. Watching the effort, to quote physicist John Bell in another context, "...is like watching a snake trying to eat itself from the tail. It becomes embarrassing for the spectator long before it becomes painful for the snake."

Quantum thinking

I just had a feature article for New Scientist magazine covering research showing some rather peculiar connections between the mathematics of quantum theory and patterns of human decision making. I don't want to say too much more here, but would like to clarify one very important point and give some links.

I was inspired to write this article a couple years ago at a brain storming session held by the European Commission. Participants were supposed to be bold and propose radical visions about where the most promising avenues for research lay in the near future (this was in the context of information and computing technology). One Belgian researcher gave a fascinating talk on the application of quantum mathematics to human decision making, claiming that quantum logic fits actual human behaviour more closely than does classical logic. There are many famous "anomalies" -- such as the Ellsberg Paradox -- where people systematically violate the laws of classical logic and probability when making decisions of economic importance. The Belgian researcher explained that the quantum formalism is able to accommodate such behaviour, and was therefore surprisingly useful in understanding how people organize and use concepts.

What struck me then was the derision with which several other scientists (physicists) greeted this suggestion, while completely mis-understanding what the man had said. One physicist came close to screaming that this was "embarrassing mumbo jumbo" somehow linked to the idea that quantum physics underlies brain function (the idea proposed over a decade ago by Roger Penrose in his profound book Shadows of the Mind). He had dismissed the idea so quickly that he hadn't listened. The Belgian physicist had actually pointed out that he wasn't at all suggesting that quantum physics plays a role in the brain, only that the mathematics of quantum physics is useful in describing human behaviour.

This is a very important point -- the mathematics of quantum theory (the mathematics of Hilbert spaces) isn't identical with the theory and somehow owned by it, but stands quite independent of that theory and existed for at least a century before quantum theory was invented. The Belgian was saying -- this mathematics which turned out to be so useful for quantum physics is now turning out to be profoundly useful in quite another setting.

The New Scientist article is just a very brief introduction to some of the work. A few other things I found utterly fascinating while researching the article are:

1. This research paper called A Quantum Logic of Down Below which falls somewhere in between philosophy, psychology and computer science. The second author Dominic Widdows is a computer scientist at Google working on information retrieval. The paper essentially argues that philosophers historically devised classical logic and then took it as a model for what human logic must be or at least should be. They suggest this was the wrong way around. Pure logic isn't our best example of reasoning. The best example of reasoning systems is people, and so a logic of what reasoning is and can be ought to start with people rather than mathematics. This is a powerful idea. As the authors put it:

... what reasoning is (or should be) can only be read off from what reasoners are (and can be). Such a view one finds, for example in [Gabbay and Woods, 2001] and [Gabbay and Woods, 2003b], among logicians, and, also in the social scientific literature [Simon, 1957, Stanovich, 1999, Gigerenzer and Selten, 2001b]. Here the leading idea of the “new logic” is twofold. First, that logic’s original mission as a theory of human reasoning should be re-affirmed. Second, that a theory of human reasoning must take empirical account of what human reasoners are like – what they are interested in and what they are capable of.

 They then go on to argue that whatever the accurate logic of human reasoning is, it is more similar to quantum logic than to classical.

2. A second fascinating paper is more technical and describes some applications of this in computer science and information retrieval. Here the idea is that if people create concepts and texts and organize them using a quantum-style logic, then search methods based on classical logic aren't likely to search such conceptual spaces very effectively. This paper describes applications in which literature search can be improved by using quantum logic operations. Most interesting (and I did mention this in the New Scientist piece) is the use of quantum operations to generate what might be closely akin to "hunches" or "guesses" about where in a mass of textual data interesting ideas might be found -- guesses not based on logical deduction, but on something less tightly constrained and ultimately more powerful.