Thursday, March 4, 2010

That's a Nice Straw Man You've Got There

. Thursday, March 4, 2010

So this Andrew Exum post bashing the use of quantitative methodologies in IR research has gotten a lot of play in the IR blogosphere. (Aside: it was sure nice of him to write a manifesto for those of us who use stats in our analysis... would he mind if I wrote one for him?) The Foreign Policy bloggers are split: Stephen Walt got a thrill from it while Drezner thinks it's "unadulterated horsesh*t". Parts of it, at least.

Drezner also posts to this nice rebuttal from Drew Conway, which I solicited on Twitter, and received. My goal was to tag-team with Conway and unite the American grad school IR blogosphere in solidarity, but I got busy. Now that's it's all in the news, tho, I'll sacrifice a half-hour of sleep and pipe up.

Look, it's obvious that, like all straw men, there's a small shred of truth in what Exum is saying. This, for example, could have come verbatim from all of our Scope and Methods instructors' mouths on the first day of grad school:

I will use quantitative analysis in conjunction with theory and qualitative analysis to describe what I see as phenomena in war and peace. I will be honest about the limits of both my theory and my analysis.

It's true, but it's banal. In fact, I would argue that people with solid training in quantitative methods have a much, much stronger skepticism towards statistical results than people without it. Moreover, we actually have reason to be skeptical, since (unlike Exum) we know what we're talking about. Do some do bad work? Of course they do. Does Exum think that every policy analyst using qualitative methods (or no systematic methods at all) perfectly avoid mistakes? I hope not.

But more importantly, if the above statement is true and meaningful than its corollary must also be:

I will use qualitative analysis in conjunction with theory and quantitative analysis to describe what I see as phenomena in war and peace. I will be honest about the limits of both my theory and my analysis.

And yet Exum purposefully rejects this statement when he rejects the validity of quantitative analysis.

But wait... it gets weirder:

In war and peace, the variables are infinite, and not everything can be measured or assigned a numerical value.

Again, to the extent that this is true it's trivial: that's not the claim that quantitative researchers make. At. All. Second, Exum's statement as written is generalizable to any field of inquiry and any systematic methodology. Somehow I doubt he'd sign a manifesto that included the statement "In physics, the variables are infinite, and not everything can be measured or assigned a numerical value". Why? Because he recognizes that despite the fact that the models that physicists build are simplifications of the universe, based on assumptions which are almost certainly false, with tons of missing information, they nevertheless have significant value in explaining causal processes. It's as if Exum wishes to make the perfect the enemy of the good.

So Exum must be claiming one of two things: 1. social systems cannot be studied systematically; 2. the scientific method is wrong. I don't want to put words in his mouth so I'll let him pick which one represents his true belief, but either one is absurd. Especially since he criticizes the exact same attitude among sports journalists. (After I typed that I noticed that Justin Logan nailed this point even harder: Exum doesn't believe in science.) But there's more:

I will not use numbers to signify what are fundamentally qualitative assessments without acknowledging to my reader that I have done so in order to satisfy a departmental requirement, gain tenure, or get published in the APSR. Or because I have been in graduate school for so long that I have forgotten how to effectively write in prose.

This is... I can't... ... ... sigh. For one thing it's slander. (At least I think it is; it makes so little sense that I'm having trouble understanding what he means. What's a "fundamentally qualitative assessment", for example?) For another it's wrong. The first part of the first sentence is provably wrong in the context of Bayesian analysis, but I wouldn't expect Exum to know that. The second sentence is just rude. More importantly, it just doesn't represent the motivation of anyone I know, and it doesn't accurately characterize the discipline. Are we motivated by getting and keeping our jobs? Yes. Who isn't? We're also motivated by trying to learn more about the world, which is why we're doing this instead of getting MBAs and making a bunch more money. If Exum is saying that quantitative social scientists as a class of people, like every other class of people in every other profession including qualitative researchers and policy professionals, market their work for their audience, then he's right. (Although it's common practice in the quantitative IR literature to list one's problematic assumptions, discuss the limitations in one's results, mention alternative interpretations, and propose ways in which to extend and falsify the claims made. But Exum wouldn't know that because he refuses to read it.) Again... what's the point?

I recognize that very few squad leaders in the 10th Mountain Division have ever taken a course in statistics yet probably know more about the conduct and realities of war than I do.

Fine. I recognize that they know how to fight and kill people better than I do, and I've never read or heard a quantitative researcher claim othewise. But I know a lot of service-people -- including some officers -- who know far less than I do about the causes of war and how to avoid it, and conflict isn't even my subfield! I know officers, serving today, who are very good at their operational jobs and yet believe that Saddam Hussein was allied with Iran to help al Qaeda attack the United States or somesuch nonsense. I know some servicepeople who believe every statistic they hear, and some that don't believe any of them. We're good at different things.

Considering the immense recent mistakes made by the American administrations, in which not a single quant had a high-ranking position but plenty of experienced policy professionals did, and the ongoing abject failure that is American foreign policy, I'm just not all that amused by Exum's cheekiness on this point. Those mistakes could not have been justified using quantitative methodologies, but came perfectly naturally to people relying on their experience to make qualitative judgments. I really have a hard time buying the notion that the quant researchers are the ones with too much hubris in the foreign policy community.

Walt responds to all of this with this bizarre statement:

Wise words indeed. I’d just add that Nobel prize-winning economist and strategic guru Thomas Schelling offered a similar warning in The Strategy of Conflict, cautioning against any tendency “to treat the subject of strategy as though it were, or should be, solely a branch of mathematics.”

But... we don't. I dare Exum or Walt to show me an IR scholar who treats strategy solely as a branch of mathematics. Mathematics is a tool. It's a language that allows us to organize facts in a systematic manner in order to make judgments. Like any tool it can be used well or poorly. But when used properly it can do things that other tools cannot, which is why Thomas Schelling used mathematics.

All I'm saying is... there's a lot of data in the world, and we can learn a lot from it. We have learned a lot from it. Eschewing that in favor of one's own experience or intuition or ideological biases seems like a really irresponsible choice. Doing that and then accusing others of willfully obfuscating for personal gain is slanderous. Disregarding quantitative work out of hand is logically fallacious.

That's a helluva trifecta. I wouldn't want it under my name.


That's a Nice Straw Man You've Got There




Add to Technorati Favorites