Philip Sayer
This section of Discover Society is provided in collaboration with the journal, Policy and Politics. It is curated by Sarah Brown.
‘Evidence-based policy’ and ‘what works’ are phrases that have become increasingly embedded in debate surrounding good policy-making over the last 20 years. This period has seen no shortage of critiques of these terms and the ways in which they have been employed, but relatively few attempts to articulate the precise foundations of knowledge on which they rest. Yet there are many interesting and important questions that might be asked. How exactly are stronger forms of evidence to be separated from weaker forms? What foundational assumptions lie behind the frequent endorsement of experimental methods? Or, most fundamentally, what precisely is the nature of the proposed link between good evidence and good policy?
This last question has, I think, sometimes been overlooked because we take it for granted that better evidence will lead to better decision-making. How, after all, could seeking better understanding of ‘what works’ possibly have a detrimental effect on policy-making? But perhaps there is merit in looking more closely at what this claim entails. If good evidence is cashed out in terms of the epistemic (truth-directed) properties of particular research methods, as is typically the case, then does excellence in this respect necessarily result in improved policy-making? Should it be expected to always do so, or just under certain circumstances? To what degree can this knowledge-based component of decision-making be separated from questions of value that are required to evaluate policy outcomes? These all strike me as legitimate and important questions that are too often overlooked in the evidence-based policy discourse.
Framed in this way, the relationship between good evidence and good policy begins to look much more contingent and indirect. In the absence of existing answers to the above questions from those endorsing the ‘what works’ approach, how might such an approach be justified? In other words, what are the epistemological foundations on which the edifice of evidence-based policy discourse rests? In my recent Policy & Politics article entitled ‘A new epistemology of evidence-based policy’, I suggest that an epistemological theory known as ‘process reliabilism’ shares significant affinities with the claims made by those endorsing ‘what works’ versions of evidence-based policy and therefore provides a plausible framework. I then present a number of problems that such an account faces, arguing that it fails to provide a warrant for the epistemic emphasis of ‘what works’ discourse.
The growth of ‘what works’ terminology
The phrases ‘evidence-based policy’ and ‘what works’ rose to prominence in the late 1990s in the aftermath of existing criticisms of research in social sciences disciplines. Educational research, in particular, received criticism for being non-cumulative, small-scale, biased and having non-empirical tendencies. The model offered by evidence-based medicine was frequently held up as a positive contrast, where a systematic evidence base was being generated through robustly reliable methods such as randomised controlled trials. Calls for social policy research to emulate the more reliably truth-yielding qualities of such methods proliferated, with RCTs held up as a gold-standard research method for their ability to determine ‘what works’. This terminology was picked up as part of the modernising agenda of New Labour in its 1997 manifesto and, at least rhetorically, as a core feature of governance after the 1997 election. Twenty years later, the language of ‘what works’ continues to play a central role in the conceptualisation of the research and policy-making nexus and the ‘What Works Network’ was set up by the UK government in 2013 to ‘ensure that robust evidence shapes decision-making at every level’ (What Works Blog, Cabinet Office). The network includes nine ‘What Works Centres’, each focusing on a different policy area.
Two questions for the ‘what works’ framework
Now that we have some idea of the character of the discourse, we can return to the key question posed above: does excellence in this respect [determining what works] necessarily result in improved policy-making? The first question we may wish to ask is whether the methods privileged by the evidence-based policy and ‘what works’ discourses are indeed the best way to generate predictions of policy effectiveness. That is, to temporarily leave aside broader issues regarding the value of policies to different interest groups and to focus solely on epistemic concerns. A great deal has been written on this, with a growing consensus that the initial enthusiasm for evidence hierarchies and the randomised controlled trials sitting at their apex needs to be tempered somewhat, with greater acknowledgement that evidence from a range of different methods will often be the best way to establish the policy effectiveness.
A second question, that has received rather less attention, is whether the heavy emphasis on epistemic aspects of decision-making – on the production of effectiveness predictions – is warranted. This brings us back to the central concern; why will better evidence of this kind lead to better policy-making? What are the epistemological assumptions that allow us to make the intended link between researching ‘what works’ and effective policy? The following line of reasoning seems to best capture the intuition that better evidence will inevitably lead to better decision-making:
Premise 1: Implementing the most reliable methods available will produce the greatest possible proportion of true beliefs
Premise 2: True beliefs are required to successfully enact policies
Premise 3: Consequently, if we have any ends at all that we wish to realise then we ought to value true beliefs
Conclusion: We ought to prioritise the processing of information gathered by the most reliable methods in our decision-making processes. What works is what counts.
Parallels with epistemological debates in analytical philosophy
Similar arguments have been put forward by epistemologists in the tradition of analytic philosophy and I believe this captures the intuition that discovering ‘what works’ can only have a positive impact on policy-making. However, an additional premise is required that is rarely stated or discussed. To be convincing, we need to assume that the search for epistemic evidence regarding policy outcomes has no negative consequences upon other aspects of decision-making processes. Given the finite resources available for the policy-making process, the decision to prioritise forms of research due to their reliability in producing truth inevitably produces trade-offs.
What is the role, for example, of research which seeks to foster and understand public discussion of policy issues? These forms of research are difficult to present in terms of contribution to understanding ‘what works’ and therefore tend to be marginalised by discourse which focuses on the epistemic qualities of the evidence base. A policy-making infrastructure that allocates funding and prestige on the basis of determining ‘what works’ may be effective in the goal of producing reliable true beliefs whilst having deleterious consequences for non-epistemic aspects of sound decision-making.
One way of thinking about this is to invoke William James’ distinction between the aim of seeking truth, on the one hand, and avoiding error on the other. The ‘what works’ discourse typically presents an ideal of policy-making that is aligned with the former aim but a more circumspect, and perhaps more realistic, role for research evidence might be achieved if it was recast around the idea of avoiding error. The current formulation is at risk of both overstating the kind of certainty that is possible in the social sciences and of creating a policy discourse that is dominated by questions of truth rather than those of public value. While epistemic concerns are clearly of great importance to effective decision-making, their current dominance of policy discourse may not be warranted.
My article therefore adds to calls for a policy discourse and decision-making infrastructure that more accurately reflects the sociological and political dimensions of public decision-making today. It is important to make clear, though, that this is not an argument against the use of evidence in policy per se. Rather, it is a call for renewed discussion about the relationship between knowledge-based and value-based dimensions of decision-making.
At a time when public mistrust of experts is widespread and concerns over ‘fake news’ abound, it is crucial that we find ways to connect our best empirical research with discussions and explorations of value claims. I argue that this requires a more circumspect and less fundamental role for knowledge of ‘what works’ than supporters of evidence-based policy are typically willing to allow. The epistemic emphasis of ‘what works’ discourse needs to be complemented and counterbalanced with greater appreciation of the value of conceptual clarity, public deliberation, local and experiential forms of knowledge.
Philip Sayer is Lecturer in the School of sociology, Politics and International Studies at the University of Bristol.