Thursday, March 4, 2010

I'm Losing My Religion: Methods as Fetish

. Thursday, March 4, 2010

(Disclaimer: The majority of my published research employs statistical analysis. Everything I am currently working on employs statistical analysis. )

I fear for the future of my profession. Part I

The IR blogging community's response to Andrew Exum's (rather insipid) manifesto, which Will ably collects here, illustrates something I have been contemplating for a while: the talismanic status of quantitative analysis in contemporary study of international politics. Consider Drew Conway's response (I don't intend this to be a dig at Drew Conway, who by all appearances is an intelligent, thoughtful, and highly-skilled individual. I focus on his post because I think his response is representative of median IR-man. I could equally pick on Will, but I do that enough already).

(edited for content, 5:23)

Drew advances a positive case for quantitative IR based entirely on first principles. Yet, the issue isn't whether statistical analysis can in theory produce good knowledge (of course it can). A scientific defense of quantitative IR is empirical: what original knowledge has this approach to the study of conflict (and IPE, for that matter) produced that is both substantively important and generally accepted as "true?" And is this knowledge better than the knowledge which would have been produced had we dedicated those same resources to studying conflict using an alternative method? And not only does Q-IR not have an answer to this question, but its adherence to first-principle defenses suggests that quantitative IR doesn't recognize that this is the question. This suggests that quantitative IR has ceased to be a method (a tool scholars employ when it is best suited to answer the question at hand). Quantitative IR has become dogma: Q-IR believes it offers the best possible answer to every question.

And thus I fear for the future of my profession because it is becoming (or has become already) a primitive religion based upon a methods fetish. It believes that statistical methods have a magical power to deliver truth. It thinks it need not demonstrate that quantitative analysis is better than alternatives because first principles (faith) tell practitioners that statistical analysis is always better than the alternatives. It clings to methods because it believes that therein lies the discipline's salvation. Now, as a Vermonter and a skeptic, I never paid all that much attention to religion. But I do seem to recall some sort of admonition against the worship of false idols.


Kindred Winecoff said...

I had a feeling this was coming...

There are two things: what Exum is saying (and what Drew and I are responding to), and what your broader thoughts are. As you say, Exum's manifesto is insipid. So that's settled. I'll move on to the other.

I don't think you're being fair to Drew. Drew is rightfully pointing out that Exum is trying to have it both ways: he tries to keep his qual hands clean by denying that his views have been tainted by dirty statistics, but then tries to say that he knows what he's talking about. He tries to say that his pure policy mind hasn't been distorted by the corrupting influence of the APSR, but then says he knows exactly what's in it and why it's wrong. (Some of that comes from a different recent post, which I linked.)

Drew isn't doing that at all. Drew acknowledges Exum's expertise in COIN, doesn't question it or Exum's right to ask questions of quant researchers, and is courteous throughout his response.

Then you claim that the issue isn't whether statistical analysis can produce good knowledge. But yes it is! Exum is denying that possibility out of hand. And you seemingly ask the same question a few sentences down.

To the other question: *has* statistical work improved our knowledge of the world? Yes it has. Absolutely. Are there good stats studies of COIN that reach different conclusions than qual studies? I'm not sure b/c it isn't my area of expertise (perhaps Drew can enlighten), but the new study linked by Monkey Cage looks interesting and possibly important. Have we discovered any universal laws? No, but that's not really what stats are for.

Finally, I'd like to take a step back and think about why stats are used at all in contemporary IR. For decades (centuries?) they weren't. And for decades we spent most of our time arguing about paradigms and grand theories. One case study contradicted another. Two analysts examining the same case contradicted each other. Was that state of IR really so much more productive than the present state?

Stats are no panacea. Theory is still the most important aspect of social science. But as economists are (hopefully) learning, theory without empirical support is dangerous. And as Drew mentions, stats provide mechanisms for examining the external validity of theory in ways that other methods cannot. That's important, and there is value added there.

That's doesn't mean that all statistical studies are flawless, and it doesn't mean that quant analysis is best for all questions. But it does mean that quant analysis is best for *some* questions, and it does mean that a more productive avenue for criticism is at poorly-constructed statistical studies, not stats in general.

Thomas Oatley said...

First, I apologize to Drew if I have been unfair. As I write in the first paragraph, this isn't about him. He just provided a vehicle. I deleted the first paragraph from the post because it seemed unkind.

Second, Exum doesn't deny the utility of statistics generally. Let me quote him: "Strategic studies scholars I admire like Steve Biddle show the utility of quantitative analysis in their own work, and Steve in particular makes a strong case for why policy papers and academic research backed up by quantitative analysis have more of an impact than do papers based on strictly qualitative or theoretical work."

So who is being unfair to whom?

Third, please read my post again. Your comments suggest that you think the point of the post is anti-statistics. I am not anti-statistics. I am anti-dogma. There's a big difference.

Fourth, you neglect the opportunity cost argument. Q-IR assumes rather than demonstrates that a statistical test is superior to an alternative method. Seems reasonable to ask whether that's a smart approach to the problem in an empirical discipline such as ours. Also seems reasonable in a world of refereed journals to demand Q-IR adherents to demonstrate that it's actually empirically accurate.

Fifth, this probably isn't my last post on this set of questions for reasons you know but which I would rather you kept to yourself for the time being.

Kindred Winecoff said...

I read that post where Exum praises Biddle -- I linked to it, after all -- but it reminded me of someone prefacing a bigoted comment with "Now don't get me wrong; not all ____ people are skeevy. Take my token friend Steve Biddle, for example: he's one of the good ones. But the rest of them..." like the whole APSR and everyone except for Steve Biddle, they're awful.

That little caveat makes it worse not better, so I don't think I've been unfair to Exum at all.

(Notice, also, that his defense of Biddle is focused on the *impact* that quant studies have and not their actual worth as Exum sees it. Taken in context of the rest of his arguments Exum may very well believe that Biddle is accurately describing a tragic reality. Exum has yet to mention an example of a statistical study that was useful in a different way than qual studies, or even acknowledge that the possibility might exist.)

I've read your post probably a dozen times. I see what you're saying. But you're reaching. Exum's post wasn't anti-dogma, it was anti-stats. Drew's response (and mine) wasn't pro-dogma, unless the dogma is "stats are good at some things but not everything".

I didn't neglect the opportunity cost argument at all. That's why I tried to compare the way IR is now to the way it was before stats took hold. You didn't take the bait, but I was trying to directly address the opportunity cost argument. Your argument seems to be that we're spinning our wheels doing stats; my response is that we were spinning our wheels before we did stats too.

Finally, I have no intention to steal your thunder.

Drew Conway said...

Now I wish I knew what was in that first paragraph...

Rather than repeat much of what Will has written, I would only note that if you are wondering what lasting contribution quant-IR has made I refer you to my response to Exum's point 5, which is of the same sentiment.

Thomas Oatley said...

I meant to say also, not thunder, Will, anonymity.

Thomas Oatley said...

Hi Drew--I saw the works you cite before I wrote the post. I omitted to mention them because with the (partial) exception of BdM, none are based on statistical analysis of data generated by real world processes. So, regardless of lasting impact, they don't satisfy the minimum requirement of my request. I am going to cross post this comment to be sure you see it.


Drew Conway said...

That's not actually true. In all of the cases I cited the authors present a model, make predictions, and then test those predictions against data.

I think the trouble is that data is hard to come by in IR.

I suppose more contemporary authors such as Will Reed, Walter Enders, Jacob Shaprio and Jason Lyall would be better examples of the type of research you are looking for.

I'm Losing My Religion: Methods as Fetish




Add to Technorati Favorites