NSF Recipient Has "No Idea" if He Should Be an NSF Recipient

I'm not sure why I'm jumping in on this again. Oh well.

I suggested in a previous post that if social scientists want to continue to get federal funding we should be prepared to explain why we should continue to get federal funding. I further explained that a simple recitation of "interesting" or "important" findings was not sufficient. Neither was it enough to say that research is worth of subsidy because it is a public good (which it actually isn't). We needed to explain why our work was more in the public interest than some other spending program. After all, if poor folks now have to take drug tests before they can get food aid, the least we should be expected to do is explain why our work deserves federal money instead of, I don't know, condition-less food aid. Or biomedical research, or deficit reduction, or tax cuts, or early childhood education, or universal post-secondary education or or or.

Either we need to explain why our work is in the public interest or we need to admit we're rent-seekers and start trying to be better at it. Presumably we should know how to do that.

So we were given a chance! Krugman went after Cantor, Cantor fired back by naming a particular NSF-funded political science research program, and the recipient political scientist -- Walter Stone of UC-Davis -- took to the Monkey Cage to state his case. Here's his (Stone's! Not Cantor's!) answer to the question:

Are these and other results we are reporting worth the $267,000 support NSF granted the project? I have no idea. Could the money have created more value for the nation if it had been devoted to medical or biological research? Possibly.

Nice. Credit to Walter Stone for honesty and humility, but I doubt Cantor feels chastised.

The actual findings of the study are interesting to me. They also support things we generally already knew: proximity models of elections work pretty well in most cases. They worked pretty well in 2010, just as they had previously worked pretty well. 2010 is actually an interesting case for this, since many had assumed that the rise of the Tea Party are thrown a wrench in proximity models. This wasn't so. The findings reinforce my priors, so I like that too.

But in the end it doesn't matter what the findings were. The question is whether it was worth $267,000. I don't know any more than Stone, but it'd be easier for me to argue that it isn't than that it is. The problem with social science is not that we haven't done enough voter surveys or tested the median voter theorem enough, at least in my view. Even if it was... is there any public interest in it? Other than the "knowledge for its sake" sense I can't see it. There's an academic interest in it, but that is not the same as a public interest. Just because findings are interesting or important doesn't mean that they are worth public subsidy. Lots of things are interesting or important that do not.

John Sides previously objected to my suggestion that social scientists were acting like rent-seekers when they defend their, erm, rents. I didn't mean it pejoratively, but the first commenter at the Monkey Cage (an anonymous grad student, apparently) made my point:

I thought we political scientists were supposed to know something about the practice of politics. Quotes likes this are why we are at serious risk of losing our funding. No professional lobbyist would EVER make these kinds of concessions. If I were in the APSA executive office, I would be furious after having read this response.

Indeed. But if we acted like lobbyists it'd be even harder to argue that our work was in the public interest, wouldn't it?

Stone ends his response to Cantor with this:

I am confident, however, that some small investment in understanding citizen behavior in the world’s oldest democracy is worthwhile. If we find that voters act reasonably in selecting candidates for seats in the “people’s House”—that they are not dominated by money and other distorting influences—perhaps we will learn to trust that deliberations in Congress, including over how best to spend federal research dollars, will ultimately reflect the public interest.

Maybe. But the APSA will disagree if the House votes to cease funding political science through the NSF.

Does Social Science Deserve Public Funding?

John Sides notes that the Congressional Republicans have resumed their attack on federal funding of social science. Here's Eric Cantor, as quoted by Sides:

There is an appropriate and necessary role for the federal government to ensure funding for basic medical research. Doing all we can to facilitate medical breakthroughs for people … should be a priority. We can and must do better. 

This includes cutting unnecessary red tape in order to speed up the availability of life saving drugs and treatments and reprioritizing existing federal research spending. Funds currently spent by the government on social science – including on politics of all things – would be better spent helping find cures to diseases.

Cantor's argument is not that social science has no merit; it is that other policy goals should have priority. This is possibly wrong, but it is not unreasonable on its face. When development organizations start programs in less developed countries they do not fund social science. They fund health care, infrastructure, and basic education. This suggests that publicly-funded social science is, to some extent at least, a luxury good. The US government obviously does not face the same budget constraint as say Liberia, but at some margin there is a tradeoff between funding program A and funding program B. If 'A' is medical research and 'B' is social science research, it might make sense to prioritize the medical research.

Many people believe that the US is not spending nearly enough on infrastructure, health care for all, education, alternative energy programs, public transportation systems, and biomedical research. Or foreign aid, for that matter. Indeed, social scientists frequently make these claims. Cantor has laid down a challenge: can the social sciences demonstrate that their work is a better investment than research into new medical procedures, alternative energy sources, infrastructure upgrades, etc.? Can the social sciences demonstrate that money spent on their programs is worth more to society than whatever the next-best option is? More technically, Cantor is asking us to think about the relative opportunity costs given actual budget constraints.

This is an opportunity for the social sciences to demonstrate their value by making a clear, coherent argument. Simply pointing to research on topics of possible public interest (as Sides does) is not enough... it must be accompanied by an argument that that research is more deserving of public funding than something else. So far I have not seen such an argument made. I have seen social scientists act like any other interest group: they want public spending on programs that benefit them because those programs benefit them. There's nothing wrong with that, but it's a bit distasteful to equate common rent-seeking behavior with a broad public interest. If the social sciences deserve public funding they ought to be able to make the case on its merits. In a way, Cantor is challenging us to think like civically-minded social scientists.

Sides concludes his post:

The broader point is that Cantor’s goal, curing disease and saving lives, can be better accomplished by including social and political science alongside the “hard” sciences and medicine.

Maybe that's true (I'm not being sarcastic here), but it is indisputable that we cannot cure diseases without medicine. However, we can administer medicine without studies showing how we have previously administered medicine, however useful those might be. (If that wasn't the case we social scientists would have no cases to study!) If the efficiency gains and complementarity effects from combining research in the social and physical sciences are sufficiently high that they out-weigh the costs, then we ought to be able to demonstrate that fact using the tools of social science. In other words, it is incumbent upon social scientists -- not congressional representatives -- to demonstrate their value to society. The question is whether we can do it.

UPDATE: John Sides has responded with a good post. I don't disagree with much of; maybe not any of it. But also see my comment.

Defending Social Science Against Those Who Would Prefer to Know Nothing

Earlier this week PM included this line in defense of NSF funding for political science at DoM:

Indeed, the alternative to good social science is not no social science but bad social science.

The implication from that is not that all social science is good but that it strives to be, is useful when it is, and in aggregate has improved the stock of human knowledge. Enough has been written about the battle over NSF funding that I feel no need to weigh in, but I must object to this ridiculous post by Gary Gutting at the NYT (via The Monkey Cage). It's titled "How Reliable Are the Social Sciences?" and it argues, I guess, "not enough for it to influence policy". Or in the authors words:

How much authority should we give to such work in our policy decisions? The question is important because media reports often seem to assume that any result presented as “scientific” has a claim to our serious attention. But this is hardly a reasonable view. There is considerable distance between, say, the confidence we should place in astronomers’ calculations of eclipses and a small marketing study suggesting that consumers prefer laundry soap in blue boxes.

He then goes on to make a nearly uncountable number of false assertions, misplaced blames, understatements of the usefulness of the social sciences, and overstatements of the development of the natural sciences. It's hard to know where to begin. Take the above quote for starters: The first sentence asks about policy decisions. The concluding sentences reference two potential avenues for inquiry that, so far as I can tell, have nothing whatsoever to do with policy.

Or take this:

But often, as I have pointed out for the case of biomedical research, popular reports often do not make clear the limited value of a journalistically exciting result. Good headlines can make for bad reporting.

Biomedical research may have implications for policy -- e.g. what treatments should be covered in insurance plans or something -- but it is not a social science. Additionally, if journalists misread research findings that is an indictment of journalists, not researchers.

In the next paragraph:

Second, and even more important, there is our overall assessment of work in a given science in comparison with other sciences. The core natural sciences (e.g., physics, chemistry, biology) are so well established that we readily accept their best-supported conclusions as definitive. (No one, for example, was concerned about the validity of the fundamental physics on which our space program was based.) Even the best-developed social sciences like economics have nothing like this status.

Are the core sciences so settled? I'm no expert in any of them, but my impression is that they are filled with hotly-contested debates concerning almost all of their "definitive" conclusions. Do we exist in a multiverse? Does evolution operate at the level of the genotype or phenotype? Can a universe (or multiverse) arise from nothing, and if so what is the definition of "nothing"? Is consciousness a biological function, and if so how did it arise and to what species does it extend?

Should we accept the validity of bodies of work based only on their perceived "status"? As determined by who, exactly? 40% of Americans don't believe that evolution occurred. Despite clear evidence, only 60% of Americans believe that climate change is happening at all, and a majority believe that humans do not play a major role in altering the climate. Granted, those polls are examples of social science so Gutting would likely give them no credence but even then the question remains: where does this status come from, and why should it matter?

Perhaps more pointedly, everyone should well be concerned about the "validity of the fundamental physics on which our space program was based" considering the impressive number of boondoggles and outright tragedies that resulted from it.

There are many questions of policy relevance that social scientists have not yet answered sufficiently well to base policy on them. There are also many questions with policy relevance that are more or less "settled" by social scientists, in that they have coherent theoretical explanations that are supported by multiple empirical studies. There's no point in running down a list, which could only be illustrative in any case, but one example might be that if you pay people to put silly arguments into print under their own name then they will be more likely to do it than if they were taxed for it.

A bit further down:

Is there any work on the effectiveness of teaching that is solidly enough established to support major policy decisions?

The case for a negative answer lies in the predictive power of the core natural sciences compared with even the most highly developed social sciences. Social sciences may be surrounded by the “paraphernalia” of the natural sciences, such as technical terminology, mathematical equations, empirical data and even carefully designed experiments. But when it comes to generating reliable scientific knowledge, there is nothing more important than frequent and detailed predictions of future events. We may have a theory that explains all the known data, but that may be just the result of our having fitted the theory to that data. The strongest support for a theory comes from its ability to correctly predict data that it was not designed to explain.

This is both true and one of the weaknesses of the social sciences. But refer to the quote from PM with which I began this post, and consider the relative youth of social scientific inquiry that has employed the "paraphernalia" of the natural sciences. In some cases we've only been doing it for a decade or several, mostly with pretty poor data and not much sophistication. But these problems are being corrected with time and experience -- as science should do -- and even the flawed efforts of the past are better, in aggregate, than their alternative: setting policy according to how Thomas Friedman's (or Gary Gutting's) gut feels about it.

Gutting then goes on to suggest that social science's problem is that it cannot do randomized controlled experiments on its subjects and therefore cannot make "detailed and precise predictions". Setting aside the fact that, in many cases, neither can astrophysicists or evolutionary biologists, this is a) changing in the social sciences; b) not a panacea in any science, social or otherwise; c) RCTs are about testing hypotheses (or, sometimes, just seeing what happens) not generating them; d) the analysis of observational data has always been considered a valid way to examine the rightness of theories in the sciences. Should Galileo not have been trusted because he couldn't perform a randomized controlled experiment on the earth's orbit around the sun? Had Darwin no insight on natural selection because he could not perform a trial to determine why the beaks of finches in different places varied in thickness?

Even more absurdly Gutting then writes:

Without a strong track record of experiments leading to successful predictions, there is seldom a basis for taking social scientific results as definitive.

True. True not because social science findings are so unreliable as to make them useless. True because all findings in all sciences are provisional. That is the hallmark of science. To criticize science for not being definitive is like criticizing philosophers for asking questions.

Having painted himself into such a corner Gutting has no choice but to conclude:

Given the limited predictive success and the lack of consensus in social sciences, their conclusions can seldom be primary guides to setting policy. At best, they can supplement the general knowledge, practical experience, good sense and critical intelligence that we can only hope our political leaders will have. 

Well then God help us all.

More seriously, perhaps we should use research to influence policy decisions when the research is relevant and when there is strong empirical support from multiple research programs, while recognizing that changed circumstances or new information may cause us to modify these programs later. I'm not suggesting that ever will quite happen -- if they notice it all opportunistic leaders will cherrypick findings that support their preferences and disregard the rest -- but surely that's a better goal than relying on the "good sense and critical intelligence" our overlords do not possess to solve our problems.

And when our leaders stray, or when they set policy cynically, it is the role of intellectuals to use every faculty at their disposal to point out the emperor's nakedness. That is easier done with the arsenal of social science at one's disposal than with intuition or impression alone.

Gutting is a professor of philosophy at Notre Dame. It's worrying to see him so dismissive of his colleagues in the academy. Perhaps he should give them some more of his time.

On IR Professionalization, Grad School, and What I Think I've Been Doing for the Past Four Years

Dan Nexon has written a lengthy diatribe opposing what he calls the "over-professionalization" of IR, particularly as it involves graduate student instruction. Over-professionalization seems to reduce to, basically, extensive training in neopositivist approaches (mostly but not only quantitative) to the study of international relations, with less space for idiosyncratic approaches. Nexon believes this has led to, and is leading to, an impoverishment of IR theory. It's well worth reading, as is Erik Voeten's critique, and comments on both posts.

No one has responded to Nexon from my perspective -- a grad student about to go onto the job market -- and I've been thinking a decent amount about these issues lately, so I'm going to chime in. Before I get into more specific discussion, let me note that it's been very interesting to watch the ways in which my approach to grad school has evolved over the past four years, and the ways in which my fellow grad students have taken different approaches to their time in grad school. For me, having an academic career is only valuable if I can do it more or less my way. I'm not a "careerist" in that sense... if what it takes for me to get a job at a good school is to do a bunch of work that I don't enjoy or find interesting, then I don't want a job at a good school. I'd much rather have a place at a lesser school or in a non-academic setting. Therefore, I'm somewhat risk-acceptant regarding both my professionalization (evidence: my willingness to pick blog fights with tenured faculty) and my research program (which carries a nonzero chance of being an abject failure). Some of my fellow graduate students don't seem to feel the same way, and they make their choices accordingly. There's nothing at all wrong with that, it's just not the approach I'm comfortable taking.

At this point I have almost no idea what the values of "the discipline" at large are, and I'm somewhat skeptical that there is any such thing.* It seems to depend very much on which departments we're talking about, and even then it's somewhat contingent upon the balance of recent hires/turnover within departments. Same with the journals. While different journals (naturally) tend to publish different types of work, it's not clear whether that is because authors are submitting strategically, editors are dedicated to advancing their preferred research paradigms, both, or neither. There are so many journals that any discussion of them as doing any one thing -- or privileging any one type of work -- seems like painting with much too wide a brush. Nevertheless, on to Nexon's claims.

Perhaps ironically, I think the phenomenon that Nexon is trying to describe (to the extent that it exists at all) is best understood using language taken from formal bargaining theory. That is, in an environment where competition for any decent academic job is fierce -- much less a so-called "top tier" job -- there is no possible way that grad students will not be "over-professionalized". Grad students will always be incentivized to game the job market as best as they can, and their advisors and departments are incentivized to help them do it. The job market rewards graduate students that can strongly signal an ability to maintain a productive research agenda into the indefinite future. Even if the efficient outcome is for a less professionalized discipline (if e.g. it leads to more interesting theorizing), and Nexon does little to establish the case that it is (more below), those who defect and become more professionalized are more likely to get jobs because they will be the ones who can better signal their quality to hiring departments. In politics it is often said that you can't enact the policy without being elected, and you can't have a long career doing interesting research in IR without first getting hired somewhere. Getting a job is therefore the first goal of every student, advisor, and department. There is no other measure of success, nor can there be.

From where I sit, as a grad student there appear to be only so many ways you can signal your value to hiring committees that receive tons of applications for each job -- particularly if you're coming from a department outside of the top 10. One way is to demonstrate mastery over difficult material and/or techniques for inquiry that will be useful in conducting research over a long career. Another is by having work published -- indicating that you actually can do research -- and in progress -- indicating that you don't have only one decent idea -- and by getting well-regarded scholars to write strong letters on your behalf. For the first, taking an extra methods course adds to the "toolkit" in ways that taking an extra class out of your major field does not. Or hell, even within your major field, if it's on a topic that isn't directly relevant to your dissertation. Sure, the substantive class will expose you to new ideas, but nothing's stopping you from reading those books on your own. The value added of discussing the substantive work in seminars with other grad students who only had time to skim the reading is almost surely less than the time spent learning a new method. Having more training in more methods gives you a greater ability to ask a wider range of questions and a greater ability to answer them. We're always being told to not let method determine research question, but if you only know one method you have no other option. Moreover, without some methodological chops it is difficult to conduct research that is publishable (the second signal) or that is interesting enough to excite jaded faculty that are writing letters for you (the third). In short, it's important for graduate students in political science to be able to do political science. Methods training is a prerequisite for that.

Unlike Nexon, I don't see this as a very bad thing.** I've taken more methods classes in my graduate education than substantive classes. I don't regret that. I've come to believe that the majority of coursework in a graduate education in most disciplines should be learning methods of inquiry. Theory-development should be a smaller percentage of classes and (most importantly) come from time spent working with your advisor and dissertation committee. While there are strategic reasons for this -- signaling to hiring committees, etc. -- there are also good practical reasons for it. The time I spent on my first few substantive classes was little more than wasted; I had no way to evaluate the quality of the work. I had no ability to question whether the theoretical and empirical assumptions the authors were making were valid. I did not even have the ability to locate what assumptions were being made, and why it was important to know what those are. Those questions, which are central to any study in political science (or should be, anyway), can only be answered once one has some sense of how theory is constructed, how models perform, how data is collected and analyzed, and why every choice made by the analyst is important. Coming into grad school I didn't have that ability despite having being fairly comfortable with basic statistics (what Nexon lumps into GLR -- "general linear reality") and models as an undergraduate economics major. (In my opinion it's still a weakness for me, and for nearly everyone I come across in academia.)

The fact remains: students at architecture schools are not asked to design a skyscraper on the first day. Someone taking their first saxophone lesson is not encouraged to abandon traditional chord structures in favor of free improvisation. Nor would it be productive for most graduate students to be "thinking outside the box" during their early years in graduate school. At that point I didn't even know where the box was. I couldn't possible have been any less professional, and so I was incapable of producing anything of theoretical or substantive value. I don't think a single one of my grad student colleagues was any different.

More importantly, I don't think that learning a bunch of methods closed off avenues of research for me; on the contrary, it opened them up! Learning that not every real-world variable has a Gaussian distribution, that assumptions regarding the data-generating process are very important, that (yes) not every process in the world is linear, that bias can come in all shapes and sizes (regardless of what method one uses, and I don't just mean quant here)... this is important knowledge to possess! I could not acquire it without "professionalizing" myself.

So with the caveat that he has much more experience and knowledge than me, I don't think I can agree with Nexon's core argument: that professionalization of graduate students has led to a lack of interesting theorizing. I believe a decent amount of professionalization is a prerequisite for understanding how good theories are constructed and how they may be evaluated. I do think that there is not all that much interesting theory being produced in IR currently, but I think that's because it's really hard to come up with original theory. Not just in IR... theoretical development moves slowly in all sciences. Most of the work is in pushing at the edges of existing theory in more-or-less straightforward empirical ways.*** It takes something approaching genius to develop original theories and let's face it... most graduate students (and professors, practitioners, and lay-people) just don't possess the necessary quality whether they've been professionalized or not.

If there is one thing that I wish I had spent more time on in my graduate classes it would be taking the philosophy of science more seriously. I suspect, from other things he's written, that Nexon would agree. Other than the first-semester scope and methods class -- which came too soon to be of much use, and doesn't seem to be conducted super-rigorously almost anywhere -- attention paid to the philosophy of science has been spotty, sporadic, and mostly ad hoc. This is, I believe, a problem with social science more generally and not just graduate instruction at the top 20 (or so) IR grad programs. But such a focus would almost necessarily have to lead to more professionalization rather than less.

*When we discuss what "IR" wants in terms of hiring decisions, it seems relevant to note that hiring decisions are generally made by departments as a whole and not just the IR people. I'm sure that in many cases faculty from other disciplines will defer to IR people when making IR hires, but I imagine that there's quite a lot of variation there.

**I'm not sure he does either. He seems to be saying different things at different times. I think what he'd like to say is that a strong methods training would be a very good thing if it didn't lead to the sort of professionalization that he doesn't like. But he doesn't quite say that, and I can't tell if it's really true.

***On a somewhat related point, tangential to my post as well as Nexon's: if I was more familiar with the literature I think I'd like to write a post arguing that political "science" must mean something specific. Political practice is different from political science, and I think there are a lot of uses in keeping the distinction between normative and positive theory intact. The falsifiability criterion seems to me to be as good as any. I think there are good reasons for arguing that certain types of theory is not political "science" without diminishing its utility for people who are interested in politics, theories about the world, etc. The word "science", if it is to have any meaning, must distinguish between styles of inquiry based on their approach, their methodology. I'm not sure where that line falls, but it must exist.

Is This Really True? QOTD

From an article on the multidisciplinary influence of Kahneman and Tversky:

Political scientists use prospect theory to model foreign-policy decision making. Some international-relations scholars argue that cognitive biases favor hawkish policies, making wars more likely to begin and more difficult to end. (Kahneman shares that view.)

That's the only part of the article discussion IR (no hypen needed) or other political science. I'll admit that I don't follow the FP decision-making lit very closely, but this doesn't ring true to me. Yes, the classic Kahneman/Tversky prospect theory article was on my Intro to IR Theory first-year syllabus, but it was tucked into the "Other Approaches" week at the end of the semester. I have to say that I've come across few major IR articles that have explicitly adopted a behavioralist frame.

Perhaps others can fill in the gaps in my education. I welcome comment.

So Let Me Get This Straight

Political scientists can't predict political behaviors, sociologists can't predict social movements, economists can't predict economic crises, biologists have no idea how many species there are, and let's not even mention meteorologists.

We're a really dumb species.

Using Social Science in Russia

In Russia, social scientists use competing metrics to try to influence policy:

Social science was part of that from early on; one factor in Mr. Putin’s selection as president was a survey that showed that Russians’ most-admired figures were fictional tough guys — the undercover spy Max Otto von Stirlitz and the homicide detective Gleb Zheglov, said Igor V. Zadorin, who headed the Kremlin’s in-house sociology department at the time. ...

The opposite argument is coming from a liberal set of social scientists, who say the data shows the public is demanding a more open and competitive political model.

The economist Mikhail E. Dmitriyev — whose research group was originally founded to shape Mr. Putin’s economic platform— began warning of a “pretty abnormal” spike in dissatisfaction he observed in political focus groups, first among middle-class Muscovites and then appearing in other large cities.

This brings to mind Fabio Rojas' post from the other day. In this case social science is being used to undermine democracy, but only by making Russia's "competitive autocracy" more responsive to citizens. Which is supposedly a key element of democracy. Interesting stuff.

Social Science, Democracy, and Being Contrary

Fabio Rojas:

It’s often thought that social science is a tool for progress and democracy. Overall, I agree. However, there’s a sense in which social science is anti-democratic:

- Social scientists may discover that popular behaviors have bad outcomes.
- Social scientists may discover that popular government policies have bad outcomes.
- Social scientists privilege experts over the “person in the street.”
- Social scientists may find that policies favoring certain political, social or corporate actors may be bad.

You might think of this as the Ibsen view of politics and social science. And you see this already. It’s now a ritual among some politicians to trash social scientists.

I believe my thoughts on social science are pretty idiosyncratic, but it has always been part of my thinking that social scientists should have a pretty strong streak of contrarianism in them. This can manifest itself in different ways. There's the lack of one-handed economists, eg, but what I really mean was once articulated by Christopher Hitchens like so: "Sit me down across a table with an ashtray and a bottle on it, and cue the other person to make an argument, and I am programmed by the practice of a lifetime to take a contrary position." This is not contrarianism for its own sake, but instead a constant probing and refinement of argument and evidence, as if it were a battle. Which it is. In any case it's an impulse for me, as fellow students and professors who have been unfortunate enough to have me in seminars would no doubt attest. More than anything else that impulse is what attracted me to academia in the first place.

Other social scientists seem not to share this impulse, instead enjoying the pursuit of consensus. And there's nothing wrong with that. I certainly don't want to disparage hard-won insights from social science that enjoy broad acceptance among experts. There's usually very good reasons why they are broadly accepted. But the enduring works in almost any social science field -- some of my favorites in political economy are Hume, Mill, Smith, and Marx -- are those which seek to overthrow the received wisdom rather than reinforce it. Even intellectual failures can be much more interesting than successes.

Rojas' post reminded of Joshua Tucker's post on social science and torture from awhile back:

My original thought was that good social science research that shows that torture does not extract useful intelligence information would be the final nail in the coffin in any public argument in support of torture. But what happens if one of us gets access to the relevant data, does the empirical analysis, and then discovers the opposite: that torture does lead to useful intelligence information. What do you do then? Sit on the results? Would any political science journal publish such a paper? How would that look in a tenure review? (“Right, she’s the one who said torture was valuable…”).

I couldn't imagine not publishing that paper if I'd written it. Not because I think such a finding would suggest that we should torture more, but because I think that moral and intellectual progress comes from tackling difficult questions head-on rather than shrinking from them. And because I strongly believe that all beliefs, especially core beliefs, must be carefully weighed against the best arguments in opposition to them. Without that there's no point in any intellectual enterprise. I'm very much a Mill-ian in that respect.

I agree with Rojas that social scientists often avow the merits of democracy despite the fact that much of our own research, behavior, and intuition suggests that democracy has deep flaws along many dimensions. Many of our assumptions about democracy do not actually make much sense in theory, and the empirical record is equally spotty. I think that many social scientists are too slow to ask who benefits from political institutions, and to quick to assume that in the case of democracy the answer is more or less "everyone". But we know that isn't true. I was recently reading debates from state legislatures over constitutional reforms (at the state level) in America in the 1820s, and it was amazing how up-front some statesmen were in arguing that democracy threatened their interests, while their opponents admitted that that was indeed the point. Of course the whole point of constitutions as legal documents is to restrict democratic tendencies. So why should we start from the assumption that democracy is somehow normatively "good"? Good for who?

I don't really have a takeaway point for this post, other than I'd like to see more social scientists really question core assumptions, admit (and then defend) the normative biases in their work, and give more value to interesting intellectual experiments, even interesting intellectual failures.

Deep Thoughts

Tyler Cowen has two posts up about bad tendencies of economists, one for right-leaning and one for left-leaning. But really both are the same post, and all the items in them sum up to two things:

1. Economists don't understand politics.

Therefore (or also),

2. Economists don't understand economics.

The problem is that economists study how to maximize aggregate utility in a social environment in which that is literally no one's priority. There is some value in understanding the realm of the possible, but it needs to be tempered by context.

Hard Problems in Social Science, Redux

Last April, Thomas blogged about the Harvard "Hard Problems in Social Science" initiative. Here were his contenders:

A. Aggregation
Across levels of analysis--micro to macro
Across space--society is not defined by national borders.
Across policy domains--Very little happens in isolation.
In short, constructing general equilibrium models based on the knowledge generated from the last thirty years of partial equilibrium research.

B. Endogenizing Change
Partial equilibrium models kind of rule this out. Thus, endogenizing change requires prior development of theoretical models that are not wedded to an equilibrium conception of politics.

Here is what he wasn't so much interested in:

Defining what is meant by "a problem in social science" is a necessary first step that the conveners appear unwilling to take. As a result, the participants in the Saturday convention held very different conceptions. Many seemed to drop "science" to focus on social problems, e.g., the gender gap in pay; the racial gap in educational achievement; inability to engage in effective state-building operations. These aren't problems of social science, these are social problems. Some went the other way and focused on the science to the neglect of the "social" (like the lone political scientist presenter who focused on post-treatment bias. Perhaps important, but a problem of method not of social science per se. Taleb focused on inability to estimate tiny probabilities. Again, clearly important, not obviously a problem of social science in particular.

Here is the result of the symposium. In some ways Thomas should be pleased. In other ways, not so much.

(Edited to better characterize Thomas' previous post.)

The Double-Standard Ethics of RCT

On the one hand I definitely understand this:

“I don’t think homeless people in our time, or in any time, should be treated like lab rats,” Ms. Palma said.

On the other hand, this is also true:

“It’s a very effective way to find out what works and what doesn’t,” said Esther Duflo, an economist at the Massachusetts Institute of Technology who has advanced the testing of social programs in the third world. “Everybody, every country, has a limited budget and wants to find out what programs are effective.” ...

“This is about putting emotions aside,” [Seth Diamond] said. “When you’re making decisions about millions of dollars and thousands of people’s lives, you have to do this on data, and that is what this is about.”

Esther Duflo recently won the John Bates Clark award for her use of RCTs in the developing world.

A complication arises when applying discount weights to the calculus. While we do not want to hurt people by excluding them from a potentially-helpful treatment, resources are scarce. If the treatment is not in fact helpful, we can divert those resources to better uses. Thus, while some people risk being harmed in the present by excluding them from the treatment, many more people might be helped in the future if we learn that there is a better use for the funds. In other words, there is some probability that the program is useful, and an inverse probability that it is not. We are essentially weighing the cost to the homeless from being removed from the program if it is useful against the hypothetical gain to future people of having access to those funds if it is not useful.

I'd also note that nobody complains when this happens in Africa or India. If it's controversial when it happens in NYC, shouldn't it be controversial when it happens in Monrovia?

On balance, put me on the side of RCTs. Not without some reservations, but on balance.

Who Says We've Got Economics Envy?

Cowen Tabarrok says it should be the other way around:

(Economics, and perhaps social science in general, seems behind its time compared say with political science.)

That is mostly (I think) in reference to the incorporation of psychology and experimental methods into political science. The post is about what ideas are behind their time.

(Updated to note correct MR writer. Thanks Anon.)

Against the Kristof/Friedman Comparative "Method"

Texas in Africa takes out the trash:

Even though Kristof has more anecdotes than your average observer, it's still not evidence. As @WrongingRights tweeted while quoting her dad, "The plural of anecdote isn't data." Data that isn't gathered systematically isn't data at all. ...

Because Kristof's only research method is his personal observation, we can't be sure that he's not simply making general claims on outliers. He's not using data; he's using anecdotes. And anecdotes are a slippery slope on which to base claims about the kind of aid work that will best aid the world's poor.

At least Kristof talks to professional aid workers and NGOs. It's possible that their opinions are based on underlying data, or at least a model, and that he's getting good information from them. Friedman's method -- "the boy pulling my rickshaw in Mumbai is more optimistic than the farmer at the Dairy Queen in Topeka, therefore India must be in better shape than America" -- is even worse. How much worse I can't say. But it has to be worse.

Good Research = ...

Fabio Rojas describes his attitude towards social science research:

Here’s how I evaluate research importance in social science. Averages, of course:

1. Great problem, great solution.
2. Great problem, partial/boring solution.
3. Decent problem, great solution.
4. Great problem, empirical/descriptive work.
5. Decent problem, partial/boring solution.
6. Decent problem, empirical/descriptive work.
7. Small problem, any solution.
8. Bizarre/highly technical approaches to any problem.
9. Definitional/taxonomic/philosophical writings on any problem.
10. History of social thought.

Sounds about right, but all of those categories are more-or-less in the eye of the beholder, yes?

Fighting Science with Snobbery

(Like everyone else I grow tired of rehashing this stuff, but a friend asked what I thought. As long as I was typing it up I thought I'd post it here.)

Ryan Sager asks a question...

Why do so few people in politics seem to know or care a whit about political science? People in sports care about sports science. People in business care about the science of how to do whatever it is they do better. But folks involved in politics — campaign consultants, journalists, and politicians themselves — could hardly be any more ignorant or disengaged when it comes to the science of politics.

... then answers it:

[M]y theory of why no one in politics likes to think about political science: because it renders them powerless. How do you do your job as a political consultant when the truth is that 90% of the success or failure of what you do will be determined by the unemployment rate? If you’re a political journalist, how do you write a story every day for a year (or three years, given our current presidential election system) saying, essentially, “Well, the fundamentals still make it exceedingly likely the president will be reelected.”

This echoes previous thinking on this blog and elsewhere, and I'm glad Sager came to that conclusion. I do think he's too generous to the wrong people -- I don't think that pundits and commentators realize how little they know about politics -- but at least he's barking up the right tree. Unlike Andrew Sullivan, who responded to Sager's article thusly:

And then there's the fact that so many political scientists are quant-wonks you'd run from if you met them in a Starbucks. And, yes, I have a PhD in political "science".

Well, maybe his PhD deserves the scare-quotes (he did it in Political Theory), but why generalize? I know (think?) he's only being half-serious, but Sullivan has an opportunity to say something constructive and spurns it in favor of name-calling. That's annoying.

I'll ignore the dig on "quant-wonks" (what?) for now and focus on what little resembles a substantive point: Is political science science or "science". In other words, is it a serious discipline or just a group of people playing with regressions and being creepy in Starbucks? I think I know exactly what Sullivan is claiming; I also think I know exactly why his insinuation is wrong.

Sullivan has made the common error of mistaking science for results. What do I mean by that? I have a (non-scientific) belief that when people hear the word "science" the first association is Laws: of Thermodynamics, Motion, Gravitation, Relativity, etc. In other words, the association of science is with old theories that have never been refuted. This is a mistake, in my view, and confuses results with method. I'm not a philosopher of science and I'm generally bored by those kinds of arguments, but I think there is near-universal agreement that what defines science is not findings but method: scientists examine the world in systematic ways, deriving hypotheses to explain observable phenomena, and testing those hypotheses against empirical evidence. If that definition is operable then modern political science, however flawed, qualifies easily. No scare quotes necessary, and no need to run for the hills at the first sight of someone with Stata installed on their laptop.

Judged by the standard of science as method rather than science as results the criticism that political scientists (or social scientists more generally) have discovered no Laws is a non sequitur. Laws are hard to come by in any discipline, which is why there are so few of them, but they are even harder to come by when the objects under study are not controllable. This does not condemn a systematic examination of the social world, however, especially when 90% of the variation of at least some political outcomes is pretty well pinned down (to use Sager's example).

The irony of course is that by citing Sager's piece Sullivan is implicating himself and nearly everyone he links to every day. But rather than acknowledging it and trying to incorporate more systematic approaches into his work he resorts to making fun of the nerds. (To be fair, Sullivan does sometimes link to research in a variety of fields. But not enough, and not nearly as much as other sources.) Well fine. But by denigrating method in favor of God knows what -- let's be generous and say Laws -- Sullivan is actually attacking science itself. I'm sure it's not his intention, but then... it's really hard to discern what his intention is. Other than snobbery, of course.

Connecting Government and Academia

Yesterday, I spent the day in Washington, DC attending an all-day panel on political violence that brought together an array of academics from various fields including political science, criminology, sociology, public policy, statistics, mathematics and psychology, private sector analysts and researchers, and government practitioners from various departments and agencies. The panel was put together by the Institute for Homeland Security Solutions, a collaborative effort between Research Triangle Institute International, the University of North Carolina at Chapel Hill and Duke University, and the Human Factors and Behavioral Sciences Division (HFD) of the Department of Homeland Security’s Science and Technology Directorate, which is the primary research and development arm of DHS.

I'm not allowed to divulge any details from the all-day event nor can I talk about who was present or what they said since government officials and agencies were involved and are funding the project (which kind of sucks but makes sense and is understandable), but this kind of academia-government interaction is exactly the type of collaborative effort that is needed to foster cooperation between academics and practitioners, and the kind of interaction that Joe Nye, Dan Drezner and others have called for in recent months.

One of the primary goals of the discussion was to bring social science scholarship and methodology to the table to help government agencies find answers to problems and questions that they face on a daily basis. There is a lot that government practitioners can learn from social science research and the ways we approach complex questions. Furthermore, these types of meetings let academics see the types of questions that government officials are dealing with, the types of problems and puzzles that need more attention, and can potentially help us frame our work in a way that can help these agencies solve everyday problems. I'm not saying that we should only be asking questions that are relevant to policy-makers, but rather that engaging the policy community can help keep us relevant and may make our work better. Yesterday's meetings were definitely a step in the right direction towards making social science research and approaches more relevant to the policy-making community.

The Eye of the Beholder

From a (surprisingly) good David Broder column:

From too many years of covering politics, I have come to believe as Axiom One that the absolute worst advice politicians ever receive comes from journalists who fancy themselves great campaign strategists.

Really? And all this time I thought the worst advice came from social scientists. I guess it all depends on where you're sitting.

Quote of the Day

See this is why you have to put rational expectations in your model. Otherwise you get these silly "fool people systematically" results that defy common sense. As bad as people think macro is with RE, trust me it was way worse before RE.

Kevin Grier, aka Angus, on some short-sighted Saudi Arabian policymakers.

It does bring up a larger point that has always bothered me about those who criticize rational choice modeling: what else can we do? If we model actors as irrational then... we have nothing to model. It's definitionally impossible to model stochastic processes. Yeah, yeah, "bounded rationality". I'm all for that, but all it does it blur the edges. The operative assumption is still rationality, just "bounded" by informational or other constraints.

Anyway, social scientists who insist that the rationality assumption is problematic confuse me. What are they studying?