A colossal counterfactual screw up

A few weeks ago I wrote about the importance of choosing the right counterfactual in policy analysis. I noted that quite often people choose ‘no change’ as their comparison for whatever they are considering. This is not a very interesting comparison. More relevant is a comparison to other plausibly good options, or at least what would be likely to happen if no conscious path were chosen.

This may seem a bit academic but in fact it can radically change a debate or how you approach a piece of research.

A major case in point has been identified by Scott Sumner in the debate over the effect of fiscal stimulus in the USA after the 2008 financial crisis.

Fifty elite economists were asked what I thought was a very simple question:

Because of the American Recovery and Reinvestment Act of 2009, the U.S. unemployment rate was lower at the end of 2010 than it would have been without the stimulus bill.

And only Pete Klenow got it right:

Agree. Caveat: how much was it offset by less aggressive (than otherwise) unconventional monetary policy?

…

Pete Klenow was the only one of 50 who seemed to understand the question. They were asking if fiscal policy lowered unemployment, i.e. boosted RGDP. But the standard model says that only occurs if it boosts AD. And that only occurs if NGDP rises. And the standard new Keynesian and monetarist and new classical models all agree that monetary policy drives NGDP. So it’s really asking if the 2009 fiscal stimulus in some way caused NGDP to evolve differently than otherwise, which is inescapably a question about how monetary policy would have evolved in the absence of the ARRA. And only one guy seemed to understand that.

The correct answer was; “What kind of question is that! How the hell can I answer that if you don’t tell me the monetary policy counterfactual.”

A simple explanation of this for non-economists woud be this.

1. In theory there are two ways of stimulating the economy in a downturn – fiscal policy (higher government spending) and monetary policy (lowering interest rates, which is to say printing money). What matters to the economy is primarily the combined effect of both fiscal and monetary policy.
2. In 2009 the United States Congress passed a huge fiscal stimulus.
3. Had they not done so, the US Federal Reserve would have noticed that the economy was doing worse and chosen to do more monetary stimulus. This might have included unusual practices like ‘asset purchases’ to get money into the economy even if interest rates had been stuck at zero.
4. Almost all analysis of ‘the impact of fiscal stimulus’ assumes that without that stimulus monetary policy would otherwise have stayed the same.
5. That is an irrelevant issue.

If you use the correct counterfactual you are left with a different research question. Rather than just estimating the effect of fiscal you also need some idea of what the reserve bank would have done, and how effective it would have been. Considering this will probably make fiscal policy look less effective.

Some methods for estimating the effect of fiscal stimulus will compare to the actual counterfactual, while others will not. No surprise then when they disagree!

Even though this is something hundreds of the smartest people in the world research and it is of huge consequence almost everyone in newspapers and general policy circles is asking the wrong question. A sobering thought.