It's Not Just the Data; It's Also the Model

I was recently hired to teach a short course on quantitative methods to a firm that contracts with the Federal government. It was a good experience for me on a number of levels. First, it required me to go back through all my notes and "re-learn" a lot of methodological approaches that I was taught but don't frequently employ. Second, despite minoring in methodology at grad school I'd never taught a methods course. Third, I'd never taught a course in this format -- roughly 32 hours of instruction over 4 days -- at all. Plus I got paid. So a good experience.

It also means that a lot of issues related to data and statistical modeling are in the front of my mind, so when I saw posts from Matt Yglesias and Noah Smith on possibilities and problems for economists looking to do empirical work using quantitative methods it sparked a few responses.

First, both Yglesias and Smith are correct that it isn't enough to just mine data. Yglesias mentions that, under the arbitrary but common decision-rule for inference that social scientists employ, generally without thinking because of a convention that has nothing to do with social inquiry and which is neither scientific nor especially rigorous, we're willing to be wrong 5% of the time.

But I do worry sometimes that social sciences are becoming an arena in which number crunching sometimes trumps sound analysis. Given a nice big dataset and a good computer, you can come up with any number of correlations that hold up at a 95 percent confidence interval, about 1 in 20 of which will be completely spurious. But those spurious ones might be the most interesting findings in the batch, so you end up publishing them! 

This doesn't concern me that much in general, but Yglesias' next point concerns me quite a lot:

[T]here's also the problem of Milton Friedman's thermostat. Take a room with a furnace that's regulated by a really good thermostat. Your data is going to show that the amount of fuel burned by the furnace is uncorrelated with the temperature in the room. Thus you'll discover that burning fossil fuels doesn't cause heat. Oooops!

What this problem illustrates has nothing to do with data or with statistical analysis per se. After all, Friedman could come to the same wrong conclusion simply by sitting in his room and looking at each month's power bill. Some months the bill goes up, some months the bill goes down, but the room always feels the same. Friedman could then, without any statistical analysis at all, come to the same erroneous conclusion about the relationship between fossil fuel consumption and temperature inside his room.

The problem is in the model. Friedman's example suggests that a univariate model -- burning fossil fuels causes temperature changes -- is insufficient when there are other variables -- the thermostat, the temperature outside the room -- involved in the causal process. Similarly, an additive model will give false results if the relationships between variables are conditioned by the presence or absence of other variables. The difficulty is that, contrary to the assumption underlying the most commonly-used statistical models, we do not have perfect information regarding the data-generating process. Therefore we will never know if we have included the correct variables in our model. There is no statistical test for this.

There are many approaches to overcoming this problem. The inductive approach encourages gathering as much data as it can and mining it to look for correlations. Yes, some of them will be spurious but most won't be. And, contrary to the impression Yglesias gives, there are methods short of randomized controlled experiments that can help us determine which correlations are spurious and which are robust. None of them are perfect but all of them are better than chucking out statistical analysis entirely.

The data mining approach is fundamentally atheoretical. As Yglesias notes, sometimes we'll get very robust correlations that have nothing to do with causality. The canonical (fake) example of this is the finding that ice cream consumption causes crime, but there are many real-world examples of it, particularly in epidemiology research. Social scientists, however, try to get around this problem by developing theory about the phenomena of interest, deriving hypotheses from this theory, and then using statistical (or other) methods to evaluate whether the hypotheses can be eliminated. Thus the model is determined by theory: if we were trying to explain fluctuations in crime rates and had no theoretical reason to include ice cream consumption in our model, then we wouldn't include it in our model. Or, to bring this back to Yglesias, if we think that the temperature of the room is caused by fossil fuel consumption as regulated by a thermostat, then we would include both the fuel consumption and the thermostat in our model. Typically the way this is done is by estimating the average size of the effect of variable x on outcome y, while allowing for "random" -- i.e. unmodeled and therefore unexplained -- variance in y, which we call the "residual" and denote e.

This brings us to Smith's post. Much development in econometrics has concerned e. Without getting too wonky, if there are systematic processes operating within e then it is not random and our model has major problems. So methodologists have spent a lot of time trying to "fix" e, by which I mean making it resemble a "random" process even when it isn't. Fixing e is fine is what we're primarily interested in is predicting the central tendency of y. But simply fixing e is not fine at all if what we're interested in understanding what causes y.

This gets into Smith's criticism about economics. Often, economists find some factor that they either can't measure or can't explain but that nevertheless seems to have an important causal impact on outcomes like economic growth or the wealth of nations. Frequently they'll give a name to this -- such as "total factor productivity" -- despite the fact that they don't really know what it is:

This residual represents how productive capital and labor are, which is why we call it "total factr productivity" (TFP). What determines TFP? It could be "human capital". It could be technology. It could be institutions like property rights, corporate governance, etc. It could be government inputs like roads, bridges, and schools. It could be taxes and regulations. It could be land and natural resources. It could be some complicated function of a country's position in global supply chains. It could be a country's terms of trade. It could be transport costs and urban agglomeration. It could be culture. It could be inborn racial superpowers. It could be God, Buddha, Cthulhu, or the Flying Spaghetti Monster. It could be an ironic joke by the vast artificial intelligences that govern the computer simulation that generates our "reality", putting their metaphorical thumb on the scales because they are bored underpaid research assistants with nothing better to do.

A similar thing happens with "institutions" in economics. Again, the issue is not with the data. It's with the model. More specifically, it's with the theory from which the model is derived. Put more precisely, any approach that focus on fixing e rather than modeling the processes that are happening within it has no theory. When we "label the residual", as Smith puts it, we might think we are doing theory because we now have things to say about "institutions" or "total factor productivity", but really we are just defining the problem away using new terminology. In fact what we are doing is omitting relevant variables from our analysis and then concluding that we've solved the problem!

There are many problems with this, some of which Smith notes. For one thing this can lead to "semantic bias" (Smith's term), which in turn can lead us to all sorts of wrong intuitions about the way the world works. Political science is far from immune from these same problems*, but I think the problems are even bigger in economics, particularly when economists shift from academic exercise to policy prescription. Quite often economists neglect politics entirely -- i.e. they leave them in e -- when modeling economic outcomes. This is, as I put it once, like re-arranging the furniture for a party with no invited guests... perhaps interesting academically, but not  helpful for much else. It's also very likely to lead to a biased model.

The solution is not, as Friedman seemingly suggests, to just stop doing statistical analysis. It's to do better statistical analysis, in conjunction with better theory and better operationalizations of key variables. It's to try to improve the model rather than fix the residual, and to specify exactly what we do and don't know.

*The most common instance of this phenomenon is to use "regime type" as an explanatory variable in comparative studies. Sometimes this might make sense, but most commonly it's just dumped into a model with little theoretical justification and almost no interpretive value.

Good Sense and Critical Intelligence

Following up on my post below, here's how our elected leaders view social science:

“We’re spending $70 per person to fill out [The American Community Survey]. That’s just not cost effective,” [Rep. Daniel Webster] continued, “especially since in the end this is not a scientific survey. It’s a random survey.”

I hope you can spot the egregious error at the end.

We've been doing the American Community Survey since 1850, and it is used to learn about the demography and needs of the citizenry as a way to guide spending programs in a more sensible way:

It is the largest (and only) data set of its kind and is used across the federal government in formulas that determine how much funding states and communities get for things like education and public health.

The House of Representatives has already voted to abolish it.

Via all the poli sci grad students in my Facebook feed.

More on IR, Professionalization, Etc.

Dan Nexon linked to my response to his post on the horrors of the professionalization of international relations students and called it a cage match, but then was very gracious thereafter (despite misspelling my name twice). Still, I agree with some parts more than others. For example, I love this part:

On the one hand, I hope that one day Kindred will sit on a hiring committee (because I'd like to see him land a job).

Me too! Let's hope he's on a hiring committee next year. But Nexon seemed me to want me to come back strong, so here we go. To his quibble:

But while some of my comments are applicable to all journals, regardless of orientation, others are pretty clearly geared toward the "prestige" journals that occupy a central place in academic certification in the United States.

Sure. The prestige journals don't thrill me with everything they publish either. In fact, I don't think I've ever heard any political scientist ever say "You know what? I really really loved the most recent issue of APSR. I had no problems with it at all. None. It was perfect. Every article was great." I have heard complaints from nearly every possible corner on almost every conceivable level about almost every article published (at least those that are even noticed). The old saying "If you're making everyone unhappy then you must be doing something right" is a sophistry, but in this situation it's pretty hard for the "prestige" journals to escape criticism. They mostly publish the sort of research that is mostly being done. Does that leave less space for less common approaches? Yes, by definition, but what else can they do? There are niche journals for niche paradigms and mainstream journals for mainstream paradigms.

Does that lead to path dependence? I'm sure it does. Again, what else can be done? Path dependence is a favorite causal explanation for social scientists when analyzing other social systems... why should we think we're any different?

Is this healthy for the discipline? I don't know. That's not hand-waving... I actually don't know. Anyway, there's an element of have-my-cake-and-eat-it-too-ism that always bothers me in these discussions. E.g., in comments to Nexon's post, PTJ -- for whom I have a lot of respect -- responded to something I wrote thus:

It may not take a philosophical commitment to understand what a Gaussian distribution is, but it takes a philosophical commitment to consider it relevant. Ditto discourse analysis. Methods are portable; methodologies are not.

This reminds me of the tired old Keohane/Tickner You-Just-Don't-Understand "debate".* It seems somewhat perverse to me to take a position that sums to "I reject everything you do, I reject your epistemology and your ontology, I consider your entire intellectual approach to be invalid (if not oppressive), and oh by the way it's really screwed up that you aren't publishing me in your journals" or maybe "I don't consider what you do to be 'relevant' but it's so not fair that you feel the same way about me".

I am caricaturing but only just. Put another way: If critical theorists were 95% of the APSR I doubt they'd be demanding more neopositivist articles for pluralism's sake. Alternatively, neopositivists aren't demanding equal time in Alternatives. I'm all for letting a thousand flowers bloom, but that doesn't mean I expect them all to flourish equally in all climates. Nor do I know how I could make that happen even if I wanted to.**

I understand that there are things at stake. Jobs, promotions, etc. I don't take that lightly. On the other hand... this is the profession. No profession equally weights every preference or caters to every idiosyncrasy and academia does better than most.*** There are outside demands, there is an externally-imposed incentive structure, etc. The way that those shape grad student behavior was the subject of my previous post. Does that mean that I've not engaged in some types of inquiry that might be fun or interesting for me to pursue? Yes. But what I'm doing now is fun and interesting too, and it's not clear to me that I'd be better off -- intellectually or otherwise -- with fewer institutional/disciplinary constraints.

Back to Nexon:

Of course, most of what we do in graduate school should be about learning methods of inquiry, albeit understood in the broadest terms. The idea that one does this only in designated methods classes, though, is a major part of the problem that I've complained about. As is the apparent bifurcation of "substantive" and "methods of inquiry."And if you didn't get anything useful out of your "substantive" classes because you hadn't yet had your coursework in stochastic modeling... well, something just isn't right there.

I don't think there's much space between us here. I will say that one difference between "methods" classes and "substantive" classes (in my experience) is that methods classes contain students from all subfields of political science as well as other disciplines so a strong applied substantive focus is not feasible. Nor is it necessarily desirable. Similarly, "substantive" classes are taught by -- and attended by -- folks with high varied methodological backgrounds. I think methods classes should be fairly theoretical. UNC is, I think, well above the mean in this regard -- we don't just teach folks how to type 'reg y x' into Stata and then let them loose into the world -- but the point is that a strong methods training isn't bounded by substantive interest.

As some other students have said, it's not that I didn't get anything useful out of my substantive classes in my first year or so... it's just that I got very little. Mostly, it was learning what was out there. Kind of. For that I spent hundreds of hours, and had to go back and re-read everything (related to my research at least) once I had a better sense of what was actually going on in these articles. Not all of that was methods-related but a lot of it was.

And I agree... something just isn't right there. That was my whole point. From what I can tell from other grad students who contacted me after my last post, this isn't an uncommon experience. I'm not sure what can be done about it except for demanding more from undergraduates. Quite a lot of grad students' time is wasted -- in the sense that it is a prerequisite for later productive work but there is a "burn-in" period -- and this might just be one of those things.

*FWIW, I think both sides came off awfully in that exchange, and there's something half-shameful in professional educators not being able to communicate their ideas in such a way as to at least understand each other.

**A commenter on Nexon's post picked up on something I'd hinted at in my previous post: that incentives within the academia are -- to at least some extent -- dictated by the non-academic job market. Call me a dirty neopositivist, but opportunity costs are real.

***All of this is well beyond even "First World Problems". I generally find this kind of navel-gazing to be embarrassingly self-important. Yet here I am.

(Terrible, No Good) GOTD

This, apparently, was Bloomberg's "Chart of the Day" (via, and HT to Leigh Caldwell on Twitter). In case you can't tell, the orange line is the U.S.'s legal debt ceiling, the white line is the spot price of gold.

It is a very bad, no good, terrible, very bad graph. Misleading at best. Can you tell why?

Here's a clue, but it won't work until tomorrow.

Insta-Classic

Andrew Gelman:

If some dude scratched my car, I wouldn’t be so quick to jump to the conclusion that he’s a rapist.

But would you think he was an atheist?

Context.

Apologies for the light posting. Hopefully we'll be back to normal soon.

Few Links

-- Cool interactive map of migration patterns.

-- Jeffry Frieden on Europe's "Lehman moment".

-- When hierarchical models are appropriate for time series cross-sectional data.

-- Excellent report on the economic situation, mostly in graphical form. Gist: it's about the banks.

-- Twitter debate between me and Dan Trombly over Hitchens, game theory, Marxism, GWOT, hegemonic stability theory, and other topics.

Why Doesn't Anybody Use Slopegraphs?

I'm going to start. This is a slopegraph, from Tufte (p. 158):



(click for bigger, less blurry, image)

Lot of information, conveyed elegantly and simply. Discussion here. R code here. Via Kottke.

Misc IPE Developments and Research...

... That I won't properly blog.

-- US and Mexico finally reach agreement on cross-border trucking. This issue has been festering since NAFTA's beginning, but came to a head a few years ago when Mexico got fed up and enacted retaliatory tariffs. To be clear: the US has been in the wrong here all along. Via Greg Weeks on Twitter, who also noticed Chomsky criticizing Chavez.

-- US, EU, and Mexico win case against China in the WTO over Chinese restrictions of exports of rare earth metals. China said the restrictions were for environmental reasons, but they were pretty clearly intended to benefit domestic manufacturers by subsidizing the price of an important input into production.

-- Brazil's real continues to rise, and they don't like it. Is this what adjustment looks like? And is it just me, or do all of Latin America's major economies look fairly fragile right now?

-- John Quiggin has an excellent series of posts [1, 2, 3] on what remains of Marxism if we ditch the assumption that revolution is both necessary and sufficient for positive social change. The answer, it seems to me, is very little that we can't get elsewhere. I've been thinking about this a lot lately, prompted by this awful book by Terry Eagleton (and even worse promo essay). I will probably be returning to the topic eventually.

-- Chinn, Eichengreen, and Ito do a "forensic analysis" on global macro imbalances in the run-up to the crisis. They aren't very optimistic about corrections in the near future. And Chinn describes an important-looking paper by Rose and Eichengreen on the effects of abandoning currency pegs for gaining policy flexibility elsewhere.

-- Tim Harford explains what all the fuss about bank capital is about. Excellent intro for those unfamiliar with the topic.

-- An aggravated take on the current US political economy of banking and bailouts.

-- The BIS compares the internationalization of the 2008 banking crisis, as compared to the 1931 banking crisis. The takeaway is that we only got a Great Recession (Little Depression? What are we calling this?) in 2008 because the lack of a gold standard allowed large liquidity injections.

-- Ha-Joon Chang and Jagdish Bhagwati debate the merits of a strong manufacturing base in The Economist. I wonder if the vote tally is a sign of the times? Ryan Avent adds some levity.

-- Cosma Shalizi reviews the new Easley-Kleinberg book on networks in The American Scientist. A free pre-print of the book is still available here (very large pdf).

Policy Research and Stats

In the past I bashed on Andrew Exum for being too dismissive of stats work in IR, so it's only fair that I praise him for running this guest-post on his to do policy-relevant conflict stats analysis. I haven't read the paper Charlie Simpson is criticizing, but as a general lesson the post is good. Some of them may be less relevant for academics than policy folks -- "Moneyball that shit and find the COIN version of on-base percentage or WHIP" is about exploiting arbitrage opportunities and snapping up under-valued assets at basement prices... not really what academic work is about -- but that's a good lesson too. Methodological approaches that are valid in some contexts are not appropriate in all contexts. Knowing the difference is half the battle. Maybe the most important half.

A few caveats to Simpson's post:

- #3 applies to regression as much as cross-tabs or whatever. I think her point is that multivariate analysis is what is needed, but that's not what she actually says so I can't be too sure.

- #4 is a very good point. Which is why...

- ...It sucks that #5 is wrong. Well, wrong at the end. Not all variables need to be measured at the same level of analysis. That's what HLMs are for. I'm becoming an HLM advocate more and more all the time, so I'd like to see this get out there more.

Stats Nerd Post

Fortunately, I learned time series from the estimable Jim Stimson, so I would never make such a mistake:

Apparently, the frequency with which the words "simulation" and "Monte Carlo" appear in physics journals has a statistically significant correlation with average incomes in Argentina, the number of doctoral degrees granted in the USA and the number of collaborations at CERN. What they don't publish is that, via the spurious regression phenomenon, their dataset will also show a statistically significant correlation with anything else that has gone up over the period in a sufficiently smooth fashion.

(NB: physicists should only feel a little bit ashamed of this; most econometrics papers before the early 70s also suffered from the problem of regressing two nonstationary series on each other and then patting yourself on the back for getting a high r^2).

These are physicists, not rubes. I will say that every political science conference I go to I see time series results presented without any discussion of trends in the series, or without any corrections to (probably) correlated errors. It's... disheartening. It's gotten to the point where I distrust almost every time series I see unless the researcher is very specific and up-front about his modeling decisions.

Making Quant IR Credible

Phil Arena, IR assistant professor at SUNY-Buffalo, has a new blog. He focuses on the security side of IR, but some of it crosses over. For example, this post on the difficulty of isolating causal processes from associative stats analysis. Basically the idea is this: Suppose there are two possible states of the world, one in which A causes B and one in which C causes both A and B but is not directly observable. Standard stats methods would not be able to distinguish between the two.

I think he does a pretty good job of describing the problem, so I'm not going to rehash his post. Just go read it. And I think we're seeing an increased use of Bayesian stats, instrumental variables, experimental approaches, and network analysis to try to mitigate the problem. In other words, I think things might be improving as the discipline matures. I completely disagree with this, however:

If you ask Jas Sekhon, one of the most talented methodologists we have in political science, he'll tell you that the answer for IR scholars is to give up on quantitative work altogether, learn how to do credible qualitative, and start asking more policy relevant questions.

An argument for better stats (or better theory) is not an argument for qualitative methods, which must be made on its own merit. I like qualitative methods and value their inclusion in the discipline, but it's not as if qualitative analysis is definitionally error-free, and "policy relevant" is very much in the eye of the beholder.

I also think that his suggestion that we focus more on theory -- which is unsurprising, since he does formal theoretical work -- is pablum. True pablum, but pablum nonetheless. Of course creating credible, rigorous theory is important and perhaps under-valued in IR, but the whole point of using stats is to evaluate theory. We don't know if a theory is credible or rigorous until we put it to some evidence-based test. Internal logic is important, but is not the end of the story.

Nevertheless it's a good post, and worth thinking about.

Note: Updated slightly for clarity shortly after posting.