Foreword

Not all evidence is born equal. Some evidence is stronger, more trustworthy, and more relevant, than other evidence. But figuring out what is good can be tricky. For the specialist, there are things like Magenta and Green Books, the 100-page bibles on appraisals and evaluations produced by Her Majesty's Treasury. They talk you through such things as cost-benefit analysis, randomised controlled trials, quasi-experimental impact evaluations, and the like.

Such detailed guides are fine and needed. But what is also needed are easy-to-grasp frameworks, for both specialists and lay audiences, that judge the evidence backing your policy, programme or practice. This report sets out these frameworks - currently 18 in the UK, and rising - and helps you navigate between their different approaches.

Usually the frameworks have some sort of one to five scale, and usually the focus is on evidence of impact. But what our mapping reveals is that there are important variations around this stereotype. A good example of this is that some standards, like those produced by Nesta or Project Oracle, look at the strength of the designs of single evaluations or interventions, while others look at whole bodies of evidence, such as the meta-analysis and systematic reviews used by the Education Endowment Foundation. BOND's international development evidence principles go even further. It has added new criteria such as 'voice and inclusion'; taking into account the perspectives of the marginalised or people living in poverty. There is no carbon copy for these standards. All have been tweaked or rebuilt to meet the needs of their sector, and that is a good thing.

However, this diversity can create confusion. It is alarming to hear that some interventions have been rated a decent two on one scale, but a poor zero by another. That sends a garbled message to our sector. We need to standardise the standards, set up an independent accreditation system, like those curated by the Kitemarking body the British Standards Institute, or the ISO. Another option is to create something like the US Results First Clearing House Database, a one-stop online shop that compares the ratings of interventions across eight national what works-type centres. We will certainly endeavour to encourage knowledge sharing amongst providers of standards, including working internationally with OECD and others, and create a toolkit or good practice guide. We also hope to do more to understand the 'demand side' of users such as grantees and funders, frontline professionals, commissioners of services, and others. But for now, we should reflect on how 18 standards represent our still nascent sector. We should be proud of how they have grown, learnt from each other, adapted, and flourished.

Jonathan Breckon, Director, Alliance for Useful Evidence, Nesta.

1. Introduction

We all know that evidence is vital if good decisions are to be made. Yet evidence can be lacking, there can be confusion about what evidence looks like, or there can be different types of evidence making it unclear how confident we can be in what the data is saying.

This is where standards of evidence come in.

Over the past decade, there has been a growing interest in standards of evidence, and other frameworks that help us understand what is working, and what isn't, by grading effectiveness or impact against a scale or level. Typically, the lower levels indicate that there is some evidence, and as the scale or level is ascended, more evidence is available to increase confidence in deciding whether the intervention or practice is working.

This paper maps the 18 standards of evidence currently used in UK social policy. It is intended to help all of us - innovators, commissioners, providers, policymakers, services users, and practitioners – navigate the field and understand where standards of evidence exist, what they do, similarities between them, and where they differ.

In summary we have found:

• There has been a rapid proliferation of standards of evidence and other evidence frameworks since 2000. This is a very positive development and reflects the increasing sophistication of how evidence is generated and used in social policy.
• There are common principles underpinning them, particularly the shared goal of improving decision-making, but they often ask different questions, are engaging different audiences, generate different content, and have varying uses. This variance reflects the host organisation's goals, which can be to inform its funding decisions, to make recommendations to the wider field, or to provide a resource for providers to help them evaluate.
• It may be expected that all evidence frameworks assess whether an intervention is working, but this is not always the case, with some frameworks assessing the quality of evidence, not the success of the intervention itself.
• The differences between the standards of evidence are often for practical reasons and reflect the host organisation's goals. However, there is a need to consider more philosophical and theoretical tensions about what constitutes good evidence. We identified examples of different organisations reaching different conclusions about the same intervention; one thought it worked well, and the other was less confident. This is a problem: Who is right? Does the intervention work, or not? As the field develops, it is crucial that confusion and disagreement is minimised.
• One suggested response to minimise confusion is to develop a single set of standards of evidence. Although this sounds inherently sensible, our research has identified several major challenges which would need to be overcome to achieve this.
• We propose that the creation of a single set of standards of evidence is considered in greater depth through engagement with both those using standards of evidence, and those being assessed against them. This engagement would also help share learning and insights to ensure that standards of evidence are effectively achieving their goals.

2. Current landscape of standards of evidence in social policy

We have analysed the 18 standards of evidence currently used in UK social policy. [^3] We have only looked at impact standards and those relevant to the social sector, we have also only included those currently used in the UK.

Our analysis focuses on 18 frameworks used by 16 UK organisations for judging evidence used in UK domestic social policy which are relevant to government, charities, and public service providers. These are:

The standards of evidence we have analysed are in a range of charitable and government organisations. It is worth noting that three of the UK What Works network and its affiliates – the Centre for Ageing Better, the Wales Centre for Public Policy, and What Works Scotland – are omitted from the list. This is because they currently do not use a standards of evidence framework. We have also excluded the standards of evidence used by Pearson Plc [^5] as these are the same as Nesta's Standards of Evidence.

Standards of Evidence have a long history, and there has been a rapid proliferation of standards of evidence since 2000. The oldest that we have examined is the GRADE system used by NICE and the What Works Centre for Wellbeing, followed by the Standards of Evidence for London, commissioned by the Greater London Authority for Project Oracle and delivered by the Dartington Social Research Unit in collaboration with international experts such as Delbert Elliot at the University of Colorado, and Steve Aos at the Washington State Institute for Public Policy (WSIPP). [^6] , [^7] The most recent to emerge was the What Works Centre for Wellbeing's CERQual framework in 2017, and in 2016, HACT's Standards of Evidence, Realising Ambition's Confidence Review, and the What Works Centre for Crime Reduction's EMMIE framework.

The standards of evidence span numerous social policy areas. Some, such as those used by Nesta and Realising Ambition, are designed to accommodate all policy areas. Three of the frameworks – developed by Project Oracle, the Early Intervention Foundation, and Dartington Service Design Lab - focus upon children and young people. Bond focuses on international development, with the other frameworks focusing on health, local economic growth, housing, education, wellbeing, social care, and crime. It is worth noting, that although most have a specific policy focus, all of them could be used in other policy areas.

The evolution of standards of evidence is a very positive development and reflects the increasing sophistication of how evidence is generated and used in social policy. And it also reflects the determination of those organisations involved. Often the creation of evidence and the increased scrutiny which accompanies it, is met with controversy and resistance. [^8]

Although there has been a rapid uptake of standards of evidence, they are not without their critics. We do not have space to go into these here, but it is worth noting the ongoing debates about whether standards of evidence focus too heavily on quantitative methods, and in particular, randomised controlled trials (RCTs), or whether lists of interventions to tackle complex social challenges is too simplistic a response. These discussions are valuable in helping sharpen the use of evaluation across social policy, but they should not distract from the value that standards of evidence are providing in the quest of finding what is working.

Standards of Evidence timeline

Office for Fair Access Standards of Evidence

In 2017, the Office for Fair Access (OFFA), the independent regulator of fair access to higher education in England developed standards of evidence. To create these, OFFA adapted Nesta's Standards of Evidence and those developed by The Social Innovation Partnership (TSIP). The OFFA Standards of Evidence are on a 1 to 3 scale as follows: [^9]

Level 1

The Higher Education Institution (HEI) can provide a narrative to motivate its selection of outreach activities in the context of a coherent outreach strategy.

Level 2

In addition to a narrative account, the HEI has collected data on impact and can report evidence that those receiving an intervention treatment have better outcomes, though this does not establish any direct causal effect.

Level 3

The HEI has implemented an evaluation methodology which provides evidence of a causal effect of an intervention.

The Crime Reduction Toolkit

The What Works Centre for Crime Reduction has developed the Crime Reduction Toolkit. The EMMIE Framework - discussed in this paper - is part of this Toolkit. The Toolkit ranks interventions that have undergone at least one systematic review on the impact on crime, how it works, whether it works, how to do it, and how much it costs. A snapshot of the Toolkit is shown below.

Source: whatworks.college.police.uk/toolkit/Pages/Toolkit.aspx

Realist and Meta-narrative Evidence Syntheses (RAMESES)

In 2011, the National Institute of Health Research Health Services & Delivery Research Programme funded the RAMESES project to develop quality and reporting standards and training materials for realist reviews. The aims were to increase the quality of realist reviews, its reporting and to help build realist review research capacity. In 2015 it funded the RAMESES II project to do the same for realist evaluations. [^10]

RAMESES was the first attempt to provide standards for reporting realist synthesis. Realist synthesis is seen as an alternative to the systematic review method [^11] and draws on theory-driven, qualitative, and mixed methods approaches. These approaches can help expand the knowledge base by explaining the success, failure, or mixed fortunes of complex interventions. [^12]

What they do

The standards of evidence vary widely in their aims, scope, and objectives. The table below summarises what they do. It is ordered chronologically so that the evolution and progression of the landscape of standards of evidence can be observed.