When policymakers and academics collide OR evidence is in the eye of the beholder

The demand from our universities, our funders and our government that our research impact on society by changing policy and practice, has brought researchers into ever-greater contact with an increasing-variety of ‘non-academic users’. This raises tensions. In another post, Heather discussed how our research got lost in translation between us, as researchers, and a journalist. In this post, she looks at how we’ve found that we have a different conception of evidence than that which predominates among policymakers.

Evidence has been a key term in policy circles for sometime now as part of a push to so-called Evidence-Based Practice. I’ve always been a cynic about the relationship between policy and research. As Estelle Morris said, years ago, speaking at a British Educational Research Association Conference, the research evidence clearly supports mixed-ability teaching but no Labour government will ever speak out about the damaging impacts of setting and streaming because of what they feel would be the influential media and public reactions against this (not least from the Daily Mail newspaper).

Turing to this government, Kim and Laura have drawn attention to flimsiness and selectivity in Education Minister Michael Gove’s choice of evidence on which to base his policies. Recently debates around evidence in education have been reignited by Ben Goldacre’s report which, as Harry Torrance points out, ignores the debates around evidence that happen within educational research (and social research generally) and instead positions education researchers as resistant to progress or, as Gove puts it, ‘enemies of promise’.

However, there are undoubtedly many people engaged in policymaking who, knowing the imperfections of the process, as far as possible try to enact what the evidence shows to be most effective. My concern in this post is to unpick what – at least some – such people construct as evidence and to show how this means that some research counts and other research doesn’t.

We have talked with policymakers from the start of this study – through our advisory group, our interim workshop, our twitter account, and so on. Through these conversations, we’ve encountered the view that a project entitled ‘the role of celebrity in young people’s classed and gendered aspirations’ should be able to isolate the role that celebrity plays, independent of all other factors, within young people’s imagined futures. We don’t think this is possible, as I discuss in the video below, and instead are exploring how celebrity operates as part of our sociocultural milieu within which young people come to hold particular aspirations.

Even if we can successfully make the argument for the impossibility of establishing a cause-and-effect relationship between celebrity representations and youth identities, we have then been told that this renders our research unusable as evidence for policy. Evidence, in this view, must be transparent and indisputable; it must speak in the same way to all who encounter it, whether now or an unidentified number of years into the future.

But evidence doesn’t work like this. Mathematical proof is often seen as the ultimate in asocial, ahistorical evidence: a standard for all cultures and all times. But Imre Lakatos‘ book Proofs and Refutations, among other work, shows that what counts as proof is open to contestation and has changed dramatically across time. Even proof turns out to be a product of society when you look at it closely.

Instead evidence comes in many forms. Howard Becker in his book Telling About Society, talks about mathematical models and statistical tables alongside novels, films and the many other ways we use to communicate what we know about the social world. All these he sees as representations, with both possibilities and constraints. This makes sense to me: I referenced Jonathan Franzen’s novel The Corrections in developing my approach to identity in my thesis and reckon I’ve learnt more about gender and sexuality from watching Buffy the Vampire Slayer than from most of the feminist books I’ve read. Given that Buffy has generated academic conferences and an online journal Slayage,  many scholars must feel similarly.

As Howard Becker eloquently explains, much as we might wish it were otherwise:

Every statement of fact presupposes a theory that explains what entities are out there to describe, what characteristics can be observed and which can only be inferred from characteristics that are observable and so on. … Instead of facts supported by evidence that makes them acceptable as fact, then, we have facts based on a theory, accepted by some people because they have been gathered in a way acceptable to some community of makers and users.

Knowledge is inseparable from power.  So, when policymakers, as one community, encode a particular form of evidence as the only one that counts, not only does much research, including ours fail to count, but we get a perverted form of policymaking, that presents its facts as theory-free.  While we agree with our funders, the ESRC, that ‘it is important that researchers and policymakers share a mutual understanding of the relevance of each other’s interests and activities’, we feel that this needs policymakers to be as open to changing how they do things as we’re expected to be.