Thursday, October 2, 2008

Complexity theory and the financial crisis

The Washington Post has a very good Op-Ed piece about how complexity theory explains the financial crisis. Finally. I've been thinking about how complexity theory could help us all makes sense of this mess. This is a good first step.

They key takeaway is that companies evaluated risks sui generis, as individual phenomenon, without analyzing connections to the system as a whole as thoroughly as necessary. Given normal conditions, the computer models predict that financial instrument X has Y percentage of going wrong. However, if conditions change very dramatically, beyond the parameters of the model, the chance that something will do seriously wrong also change dramatically.

What's key to understand is that the introduction of these computer models were themselves a change in the conditions of the markets.


Anonymous said...

I agree that making such heavy reliance on computer models itself represents a change of the market environment.

But I think the impact of ever-more-sophisticated computer models may have a more fundamental effect. I myself use datamining in my own work, which happens to be Genomics where I basically try to infer biological relationships by throwing pattern-recognition tools at large databases. Similarly, the financial types use datamining tools to seek correlations.

BUT when I discover some apparent relationship between these genes and those genes, my discovery of such a relationship does not thereby effect human physiology. When some analyst discovers a correlation and uses that correlation to guide market trades, THERE IS A FEEDBACK EFFECT. I strongly suspect these computer models are generating complex feedback loops that no human being understands.

Fundamentally, datamining on the stock market represents statistical arbitrage: if I can find patterns before you do, then I can make money. But if everybody is doing it, then it will become harder and harder to spot patterns because any correlation gets traded away.

JohnTEQP said...

Exactly. One thing datamining like that does on the market is correct imbalances very quickly. But it also introduces volatility, because massive amounts of money can be moved very quickly.

The big problem with the complex feedback loops is WHO doesn't understand them: not only traders, but also regulators. And, most importantly, the public. This is part of the frustration of the public. Stocks and bonds are easy to understand, at least in basic principle. But complex derivatives are very difficult to understand, so the public feels like smart people are making money just by manipulating symbols, rather than generating value.

Milton said...

I am not a financial analyst but something else that complexity theory offers us is an increased ability to recognize when a system is reaching a critical state. It can't tell us what might trigger the dramatic shift but we can increase our ability to see when a signifiant number of factors are all building up.

Complexity theory isn't about future prediction because the underlying theory requires that we accept future states as unknowable in principle. The further in the future we push our predictions, the less likely they are to be correct. I can tell you with reasonable certainty what the weather will be like here five minutes from now but if I extend that to five days or five weeks, detailed prediction degrades.

Patterns and cycles, however, do guide us. We can know with reasonable certainty that it will drop below freezing this winter in Canada, even if I can't predict a given temperature on a given day five weeks from now. Good post. I'll take time to read the news article you suggest.

Josh said...

Nice post and good reference article. My understanding is the author is referencing Per Bak's power law which is why the use of the coin flip analogy is right on. What concerns me is how few people understand the ideas inherent in complexity theory. Unfortunately it will require a change in thinking on the part of both the traditional left as well as the traditional right. You can't know EXACTLY how to change a system and you can't assume by letting a system run wild that it will self-regulate. Intellect and humility are two characteristics that are hard to find in the same political leader. Sigh.