A quasi-summary of the International Conference on Qualitative Research for Policy-Making 2010 by Tim Nelson

The only defense I have for what you are about to read is that Jasper asked for a copy of my rather rambling summary at the end of Day 2 of the conference. Seizing upon this meager excuse, I decided not only to write up some of those points but to plague you with some further thoughts that have occurred to me since that time.

First, let me say that this is not an attempt at an objective summary, but a take on the conference from the perspective of one who has only recently come into the world of policy-relevant research (as I think I told any of you who came within three feet of me at the conference). As a relative newbie then, the conference raised a lot of questions for me - many of which I also picked up from other participants during presentations and discussions.

• One factor might be the strength of lobbyists and pressure groups within that sector—organizations which will undoubtedly get the ear of elected officials before academic researchers. This may vary by policy level or issue (national health insurance policy in the US for example seems to be strongly influenced by insurance companies, the American Medical Association and the AARP), but local policy or policies around such things as H1N1 may be relatively free from such political pressures.
• Another source of influence that elected officials will have to contend with is the average constituent. This is where educational researchers have a disadvantage (beyond the influence of strong teachers’ unions) as every “man on the street” can pull an education policy out of his back pocket and thinks that he knows how to fix the schools. In addition, as Gerald Galway pointed out, local schools are bound up in issues of community pride and identity, which make them less amenable to the tinkering of “rational” policy initiatives borne out of data. I think the average citizen is more circumspect when it comes to health care (beyond the issue of insurance), as they recognize this as a field for the “experts.”
• Speaking of “nonrational” factors which compete with research for the policymakers’ attention, we also have to take account of the moral role of government as well as its “problem-solving” and pragmatic function. As an example, I don’t think there is a credible piece of research which has demonstrated that capital punishment is a deterrent to homicide, but for many groups that is beside the point. For them it is a matter of moral justice to execute murderers, despite its lack of a deterrent effect. Thus, it seems that the more that a policy sector is seen in moral terms, the less it is amenable to a purely pragmatic efficiencies and outcomes approach that can be informed by research. This “morality effect” operates not only in criminal justice, but also education and poverty policy where the key issues are deservedness and fairness and especially in family policy where issues of morality abound at every turn. Exacerbating this is the fact that, at least in the US, those most likely to emphasize the moral dimensions of policy view academics with deep suspicion and sometimes open hostility as the “liberal elite” who want to impose their own “anything goes” culture on them. There is a huge cultural class divide between educated elites and Middle America which policymakers ignore at their peril.

2. Policy Players: The policy world is very complex with many layers (national, state or provincial, county, municipal) with a parallel layered structure of NGOs, service providers, advocacy groups, and various forms of media. Several conference presenters pointed out this complexity, and it is very helpful to recognize this, as not all policy is done by elected officials or their staffs (and thank the Lord for that). A few thoughts on this.

• First, this complexity should not scare us. We are, after all, qualitative researchers by God, used to looking complexity and contextuality in the eye and wrestling it to the mat. OK, maybe too vigorous of a metaphor, but my point is that if we turned our analytical and observational powers honed in research and trained it in the direction of policy actors, their constraints, resources and fields of influence, we could develop a much better, more finely targeted approach for having our voices heard in the policy process. Eric Lai’s suggestion about opportunities inherent in new national legislation is a very good case in point here.
• However, a note of caution. Even when policy-involved groups are not beholden to the constituencies or pressure groups I pointed out earlier, they may still not be open to research input. Last year my wife, Kathryn Edin, who is a national poverty expert in the US, was summoned to the Gates Foundation along with a small group of other academics. Melinda Gates and her team were to be there at this meeting which was to set the Foundation’s agenda for the 21st century. I was quite excited by this (I had these delusional fantasies about Kathy and Melinda hitting it off at this meeting and subsequently being invited to out to Redmond for casual soirees with Bill and Mel—as we would call them—and their crowd of billionaire friends.) When she returned and I asked how it went, she said that the foundation had already decided on a course of action and simply wanted the academics there to rubber-stamp it as a good idea. Unfortunately for my dreams of hoisting a few cold ones with Bill, she thought what they proposed was really a bad idea and told them so. They haven’t called back.
• It is folly to ignore the role of the media. The media is incredibly powerful in agenda-setting for policy (a point raised by many in the conference) and, I would argue, even less subject to the influence of research than are policymakers.
  • I think that our natural inclination is to see journalists as comrades in arms and compatriots who “speak truth to power” and are critical in the same way that much research is critical of the status quo. Like academics, journalists are knowledge workers who prize autonomy, are generally left-leaning, and love skewering those in charge.
  • Yet we should not be naïve to the fact that media organizations are primarily businesses which act according to their own logic and purposes. And those are not to solve social problems but to exploit them. I think it is the experience of many who want to “use the media” for their own ends (even noble ones like communicating important research findings) to come away from the experience realizing that they were the ones who were used.
  • The media tends to trade in stereotypes, particularly of stigmatized groups, and many of these might go directly against our research findings. The reason for this is simple: although journalists do interviews, they do not understand or care about sampling or representativeness (not their fault—it is not rewarded in their field). Therefore they will find people to interview who match their preconceived idea of that population and make them a mouthpiece for the group, or social issue, in their articles. For this reason, the media is often more of a hindrance than a help in effecting policy change in line with our research findings.

3. When Worlds Collide: One recurring theme, expressed verbally and non-verbally during the conference, seemed to be a deep ambivalence about getting our hands dirty in the world of policy and politics. Here we have two institutional cultures in collision. On the one hand there is politics, which involves the arts of compromise and strategic maneuvering. It is relentlessly time-bound and opportunistic, willing to be “not perfect, but good enough for now.” On the other there is academic research, striving for the pure and ideal (hence our obsession with methods), and willing to expend limitless resources and time in the pursuit of Truth. Needless to say, these two worlds don’t fit together well.

What makes us even more uneasy is that we can’t get them to play on our turf. If we want our research to have a policy impact, we have to go to them—only in rare instances will they come to us (though the “evidence-based” policy mandate in the UK seems like it may have altered this dynamic somewhat in favor of the researchers). As Willard Waller observed almost a century ago, whoever has the least interest in a relationship holds the most power in that relationship. As academics and researchers who prize our autonomy above all else, the idea of being beholden to some bureaucrat or politician who thinks snowball sampling is something that Eskimos do at dinner parties is downright galling, to say the least.

4. Uniqueness of Qualitative Research: While much of what I have written above applies to research in general, there are several issues which pertain specifically to qualitative research which should be mentioned.

• First, a question: Is qualitative research in an advantageous or disadvantageous position relative to quantitative research when it moves into the policy world? Conference participants expressed points of view on both sides of this issue. Some mentioned numbers as having more rhetorical weight with policymakers, while others countered that the stories qualitative researchers tell can be more effective. My sense is that we don’t really know, and it probably varies by policy sector and policy player. It may even depend on the personalities and biographies of gatekeepers and authorities within policy organizations. Logically, it could go either way. It is, as we say, an empirical question to which I don’t think we have enough hard data (in either stories or numbers) to make any sound conclusions.
• While it is fun to point out the foibles of quantitative researchers and the often fetishistic use of numbers by organizations (as in Giuseppe Veltri’s amusing presentation), my sense is that there is an increasing dissatisfaction with purely quantitative work on the part of policymakers—particularly those in housing, poverty, education and even criminal justice. There is a good reason for this: numbers often don’t illuminate the mechanisms or processes through which causes have their effects, and to make good policy these are things you need to know. Not always of course. Some pharmaceuticals work even when doctors have no idea why they work, and that seems to be good enough for them. But applying this to social policy is risky. First, those pharmaceuticals have been tested to make sure that they have minimal side effects. But can you test for unintended effects in social policy? We often don’t have the means to test the effectiveness of the intended outcome of the treatment, let alone any unanticipated consequences. It seems to me that this is qualitative research’s great point of advantage over quantitative methods when it comes to policy research: we can say how something happens. The example I gave was from the Moving to Opportunity experimental study of families given housing vouchers to move out of public housing. The quantitative research showed that among “experimentals” (those who moved to lower poverty neighborhoods), adolescent girls had better outcomes, while adolescent boys had worse outcomes relative to their control peers. The numbers could not say why this gender difference existed. But the qualitative interviews, built into the larger study, could. (It had to do with the gendered patterns of leisure activities and friendships made upon the move to the new neighborhood—if you are interested I’ll send you the paper). If you are making policy, it seems you would absolutely need to know the pathways and mechanisms through which negative outcomes occur so that you could sabotage them, while giving funds and other resources to encourage mechanisms producing good outcomes. Mixed-method studies often involve researchers of both quantitative and qualitative persuasions, and that’s where David Nicholas’ helpful insights into effective team management could greatly help the process.
• While quantitative researchers juggle numbers, qualitative researchers have a special relationship with, and therefore, I think, a special responsibility to our own research subjects. For myself, I feel that this responsibility constrains my ability to jump with both feet into the expediency-driven world of policy and politics. Perhaps this violates good research practice, but I see myself as more of an advocate for the low-income men I interviewed than a dispassionate researcher. This is not always the case of course—some sociologists have studied white supremacists and other groups for which they had a deep antipathy. But I suspect that most qualitative researchers tend to adopt a more sympathetic stance toward their subjects—in some sense it is necessary to be a decent researcher in the first place. What do we do with that when we enter the policy arena? Is it a betrayal to place our findings (which they have entrusted to us in a sense) like so many poker chips on the political table, when the outcome can so greatly affect their lives? But what if we don’t? Is that even more of a betrayal—to stay in the ivory tower with “clean hands,” looking the other way while bad policies or complete indifference continue unabated? Tough questions which we should all be thinking about, but to which there are no easy answers.

5. What about Theory? The role of theory appeared several times during presentations and in the Q & A. This is something I have struggled with myself. When I first came to the Kennedy school I approached my section head with the idea of teaching a course to our Masters of Public Policy students on basic concepts of social science—things like “social capital” and the like, phrases that people bandied about without any kind of precise idea of what they were talking about. My idea was that it would be kind a “Conceptual Toolkit for Policy Analysis.” I thought it was a great idea, but Tony—a usually mild-mannered guy with glasses and given to wearing bow ties—shot me a look like I had just suggested teaching “Basics of Internet Pornography.” “Our students,” he gently reminded me, “look more for the practical nature of the courses—something they feel they can use in their eventual careers.” I bit my tongue before I could ask him then why the hell Microeconomics was part of the core curriculum. But I digress. I think that if we are communicating directly to policymakers, it would be a mistake to even mention the “T word.” If they don’t want to know methodology, they certainly don’t want to know theory. But I am torn here. Isn’t all policy based on some theory of human behavior? Doesn’t it all boil down to some variation on the “carrot, the stick or the sermon” as Jenny Owen said in her closing presentation on Day 1? Perhaps the role of theory is in the formulation of policy-relevant research in the first place. As far as the policymakers, it would remain an invisible but an essential element of the final product. Another topic for further discussion.

6. The Role of Linking Institutions and Networks. The presentations by Sarah Morton, Jennifer Fluekiger, and the team from the QCDA (Tom, Peter and Kate) brought home the potential role of institutions and networks which bridge the worlds of academic research and policymaking.

• First, what a brilliant idea—hiring a smart and media-savvy person to package research for the potential users, including policymakers. We should clone Jennifer and each have a copy of her working for our organizations.
• Second, the dense social ties between academics and policymakers in Scotland seem to have contributed to the successful integration of research and policy in that country. Yet this seems to be an environmental feature—how does one replicate that in other kinds of contexts with very loose and indirect connections between these two worlds?
• The idea of having intermediary organizations which sit between these worlds and serve as a clearinghouse for research in the service of policymaking, as QCDA does for the use of curriculum in England, seems to be a very optimal solution. It would be fascinating to explore what other types of bridging institutions could be formed, in which policy sectors, perhaps funded by foundations or government grants. Again, another topic for future discussion

by Tim Nelson, Harvard Kennedy School - Harvard University

Want to know more about this conference? Click here to see highlights and pictures of this past event!