Predictions, Experts, and the Lucas Critique

An explanation not of how to be right, but of how to not be wrong.

I

Experts often have a life-cycle. Someone such as Michael Burry (Christian Bale from the Big Short) comes along. He makes several grand predictions about a housing bubble, even bets money on it, ends up being right (sort of), and receives amazing press.

Over time, they keep making predictions. But this time, they keep being wrong. To return to the Michael Burry example, he has been betting on another market crash for years now. He’s been wrong each time.

In politics, people will often look to someone like Samuel Huntington or Francis Fukuyama. They were correct in diagnosing that something had changed in 1991, but then made fundamental mistakes in analyzing the post-91 order. In economics, Larry Summers is a prime example of someone who correctly predicted an inflationary surge, but then made ludicrous predictions regarding unemployment.

All of the above individuals have two major issues that lead to this pattern of initial success and subsequent failure. The first is overfitting to the environment.

II

If you missed your introductory statistics classes, here is a quick explanation of overfitting.

You observe a bunch of data, which after plotting looks like this:

Now, by eyeballing, you might be able to tell this was generated by a parabola. Unfortunately, you usually won’t be able to guess distributions that easily. So you end up doing what every good economist does. You run an OLS regression.

That looks stupid, right? Clearly the line does a very poor job of explaining where the data comes from. So now you go up one power and fit a parabola.

Now that’s a lot better! Your parabola explains the data well…but there’s still a lot of points that seem to drift from it. Perhaps you could do even better?

Ah, there we have it. An 80th degree polynomial. This does an even better job of fitting the data.

You show how well your model predicts the world to your colleagues. They are quite impressed. Then, one of them discovers some new data that your model was not initially exposed to. You decide to test how well it does at predicting this new data.

Oh wow…That looks really bad. It thinks y hits a million when x is equal to 3? Yeah, that’s pretty bad. But it’s fine, you’ve already been offered a job as a New York Times columnist, so not much to worry about.

III

The problem described in the previous section is not unique to statistics, economics, or any other field. It concerns anyone who makes theories. Observe a pattern that historians often fall into “well, Rome was a republic, it was the most powerful state at the time, and then had a charismatic person take power who eventually caused a collapse into Imperial Rome”.

Every time a historian says this, an American thinks of the United States and Donald Trump. Parallels run wild and everyone becomes convinced that the demise of American democracy is upon us. But the logic described is…hilariously bad? You clearly can’t argue that every powerful republic will find its end due to a charismatic leader. That model is extremely overfit to a few data points.

Or consider sports fans. I remember many people arguing that Germany was doomed in 2014 because it had to face Brazil, which it had traditionally struggled against. A few days later, Germany won 7-1. Another few days after that, they became world champions.

Theories and models that are fit based on an existing environment work fine (perhaps even well) while that environment persists, but the minute things change, your predictions become terrible and your favorite model loses 7-1.

IV

This is the first problem. But there is another one which is closely related.

When Larry Summers made his prediction about the necessity of high unemployment to fight inflation, he was inspired by what economists now call the naïve Phillips Curve. In general, the Phillips Curve refers to an observed negative correlation between unemployment and inflation. When one is high, the other is low, and vice versa.

This observed correlation from the data was taken by many to not just be a causal relationship (which it likely is), but essentially a menu of options for policymakers. The idea was that policymakers could essentially choose different combinations of inflation and unemployment based on their preferences.

This turned out to be false. The Phillips Curve relationship likely held because of how inflation affects wages. When inflation is higher, wages are falling in real terms and firms can hire more workers. If you have deflation, real wages are rising, so hiring stalls.

And that makes perfect sense…until you try to exploit it. You see, the minute you try to choose a combination of inflation and unemployment as if you were choosing from a menu of options, firms and workers change their inflation expectations. They now negotiate salaries knowing that inflation will be high. As a result, they make sure to price in yearly wage increases. As a result, firms lose the incentive to hire more workers, and you’re left with high inflation without low unemployment.

And this is precisely what occurred in the 1970s United States. After Nixon pressured the Fed to ease policy, inflation went up, and eventually unemployment did as well. The Phillips Curve model was right, but it was incomplete. It applied only up to the moment when you tried to use it.

V

In economics, the Phillips Curve’s collapse is often related to the Lucas Critique. The idea that any good model (or theory) should not leave high level parameters invariant to policy changes.

Let’s translate what that means. In the standard Phillips Curve story, the relationship between inflation and unemployment was invariant to policy. Regardless of what policy was, that relationship was supposed to maintain. That is clearly wrong. Instead, economists now try to build models based on micro-foundations. All decisions are made by individuals with different preferences. The main invariant part of the model, or the “model primitives” are said preferences. People’s level of patience or their choice between two apples and three bananas are what the model takes as given. Now perhaps that is also imperfect, but it’s a much more reasonable standard to take as given than the slope of the relationship between inflation and unemployment.

But the fundamental idea of the Lucas Critique has massive implications for much of social science inquiry. Indeed, most empirical evidence since the Credibility Revolution rests on finding natural experiments in the real world. We use instrumental variables, difference-in-differences, and regression discontinuities to find the relationship between minimum wage hikes and employment. But what we end up finding is the relationship at that moment in that specific location. Our estimates usually have little to no external validity. It is difficult to argue that the causal effect of a minimum wage hike in New Jersey 1992 is informative as to what the effect would be for the broader United States in 2026. Moreover, it is very difficult to argue that the relationship would not change if the government set out to exploit it.

The implications of this are huge when one considers it. It implicates practically all of our research that focuses on finding point estimates. These small reduced-form models build the intuitions of experts, who use it to make predictions that work for a few years, but then fall flat as the environment changes. We overfit and suffer.

And this is especially relevant in modernity when the world is clearly changing at a rapid pace due to both geopolitical shifts and AI. As Gramsci (sort of) said, “The old world is dying, and the new world struggles to be born: now is the time of monsters.” But what he neglected to mention is that in addition to monsters, the new world will invalidate our old models due to their dependence on invariant high-level parameters.

VI

So how does one solve this dilemma? How can we build models that do not fail the test of external validity? Well, one option is to use micro-foundations and big structural models everywhere. In other words, instead of using natural experiments to find how a minimum wage hike affects unemployment, build a model that analyzes firm and consumer behavior based on preferences. Then fit the model to the data and see what preferences best explain what we observe. Then, based on the now-solved-for preferences, simulate what would happen in case of a minimum wage hike.

This is how industrial organization and macroeconomics tackle this issue. It does not depend on invariant high-level parameters and claims to have external validity. So why doesn’t everyone use this?

Well, for starters, it does not work. DSGE models in macroeconomics are complete failures when it comes to forecasting what will happen. The ugly secret is that building good structural models is hard. You have to know what the correct model of the world is and build something accordingly. But even small tweaks to models can have massive implications! One economist uses Calvo pricing and finds that inflation targeting is optimal, while another uses menu costs and finds that NGDP targeting is optimal. Small changes make a large difference and it’s unrealistic to expect ourselves to get everything right. As a result, our models are just…bad.

VII

At this point, it is worth mentioning the concept of Kuhn-Loss. You will notice a pattern in your life: (1) your grandma gives you advice (2) you question the advice and ask your grandma for evidence or logic that substantiates her advice (3) she cannot provide either (4) you dismiss the advice (5) years later you discover evidence and logic that substantiates her advice. This pattern applies not just to your grandma, but surprisingly, to all of science.

Observe for example that people used to forecast the weather using cloud shapes. As science developed, we decided that the evidence and causal mechanisms connecting cloud shapes to weather outcomes were weak. As a result, we discarded the cloud-shape theory of weather. But as time passed, meteorology showed us that cloud formations do in fact predict cyclonic storms.

That is Kuhn-Loss. We lose knowledge when we raise our standards. That is precisely what happens when we insist on the use of larger structural models in lieu of simple reduced-form estimates of what the minimum wage does. Our standards increase, and, as a result, our predictive power often goes down.

VIII

So, what do we do?

Unfortunately, the correct answer is to be a slightly more thoughtful Larry Summers (sans all the Jeffrey Epstein stuff). You will find in life that it is completely impossible to build intuitions about reality without massive simplifications. Most of your reasoning will come from reduced-form models, a few key observations, and theories that are overfit to the environment. That is not ideal, but like democracy, we don’t really have a better choice.

But unlike many who find themselves stuck in the same model forever, a good forecaster will try to figure out when the world around them is changing. This is not straightforward. Unlike in video games, real life does not give us a large red warning sign when something is going wrong. The music does not turn into an ominous orchestra indicating a boss fight and we do not get dreams telling us to adopt new theories.

The only way forward is to qualify each statement in your head and remember that all of the massively simplified causal relationships you believe in are likely contingent on something else. If you do this, you will still be shocked and upset when your internal model of the world collapses. But hopefully, you’ll survive, adapt, and overcome.