Introduction

There has been much discussion about whether we need a 'NICE for social policy'. By this, people mean an equivalent to the National Institute for Health and Clinical Excellence (NICE), which provides evidence to the NHS on which drugs and treatments are cost-effective. The suggestion is that a NICE for social policy could be created to improve the ways evidence is used and made useful in other areas of policy. This is a timely debate, although NICE can't be copied wholesale, we do believe that the strong interconnections between evidence and practice across much of the UK healthcare system should be our aspiration in other areas of social and public policy.

This paper outlines why we need to explore a centre or a network of evidence centres - which help to institutionalise evidence in the decision making process. When exploring new centres or infrastructural changes there are a number of key features which need to be considered; in summary these include:

• That any new developments are demand led, that is to say they are useful to those who will be the target audiences and for whom the evidence will be relevant. As well as being useful, evidence will need to be used, which means new developments will need to institutionalise the demand for evidence, amongst commissioners, policymakers, and other decision makers.
• Any new evidence centres should be close enough to government to have impact, but with enough independence to be critically objective of it.
• It could perform the roles of advancing methodological development, evidence synthesis and translation, ensuring high quality and rigour, and allocating kite-marking and accreditation to exemplar performance, policy and programmes.
• To innovate and advance new methodologies, particularly those that enable agile, rapid, low-cost impact evaluations to be undertaken, but which don't compromise on quality and rigour.
• To advance the use of administrative data and other sources, as well as developing new ways of visualising and communicating such information.
• To have a clear definition of the types of evidence and what quality looks like at different stages and levels, but balancing the need for evidence without trampling innovation.

The role of the National Institute for Health and Clinical Excellence

The National Institute for Health and Clinical Excellence (NICE) is a special health authority that serves the National Health Service (NHS) in England and Wales. NICE provides guidance to support healthcare professionals, and others, to help make sure that care provided is of the best possible quality and offers the best value for money. NICE provides independent, authoritative and evidence-based guidance on the most effective ways to prevent, diagnose and treat disease and ill health, for the NHS, local authorities, charities, and anyone with a responsibility for commissioning or providing healthcare, public health or social care services.¹

Why we need a NICE for social policy

Before we outline what a NICE for social policy could look like, we will start by saying why it is needed. Despite the idea that policy and practice should be underpinned by rigorous evidence being widely accepted, billions of pounds are spent every year on social policies and programmes with little rigorous evidence on the impacts that these have.² Although the drive for evidence-based policy and practice is not new, the context of reduced budgets, devolved commissioning, and the development of new procurement mechanisms such as payment by results and social impact bonds, brings a renewed focus on finding what is effective.³

In some fields there is a need for more evidence, but just as often the priority is to ensure that the evidence being collected and analysed is made relevant to the needs of decision makers. Equally, there are challenges surrounding evidence use, such as commissioning models, budget planning, and organisational cultures. Overall, one of the most striking factors is the absence of organisations tasked with linking the supply and demand. There is the Social Care Institute for Excellence (SCIE) and the National Institute for Health and Clinical Excellence (NICE) in health, yet there are no equivalent centres or institutions in other areas such as education, criminal justice, or children's services.

It should be emphasised that there are a myriad of reasons why NICE is a good ambition, but not a good blueprint. For instance, the R&D system in health is completely different from other areas of social policy so that NICE doesn't undertake much, if indeed any, primary data collection. Instead there are a number of other institutions that produce high-quality research, from academia to the Wellcome Foundation and Nuffield, alongside the ways in which the pharmaceutical industry funds a great deal of clinical trials. In addition, the organisational cultures across the medical field, such how the line between practice and research is indistinguishable within teaching hospitals, or the ways in which doctors are trained, all have a contributing role to play, and may be vastly different across different policy areas. This means that although we can learn useful lessons from NICE, we need to think much more laterally if new evidence centres are to be embedded and function effectively across the different fields of social and public policy.

What an equivalent to NICE in social policy could look like

The creation of any new institution or set of institutions is complex, and it becomes even more so when it comes to establishing an organisation that could potentially perform the role of NICE across social policy - from education, criminal justice, youth services, and beyond, whilst being relevant to the spectrum of decision makers that need to be engaged and influenced. This section outlines what is needed to enable this to happen.

Ensuring it engages with the widest possible audience

Although engagement and consultation will vary at different stages when exploring and developing new centres, it is imperative that any NICE for Social Policy involves a wide spectrum of evidence producers, users, and others, to ensure it does not replicate what is happening elsewhere, and to help ensure that it is fit for purpose.

There is a huge amount of excellent work underway. We have a world-class university sector, and a variety of other producers of evidence - ranging from independent researchers, consultants, providers themselves, and beyond. Rather than trying to compete or duplicate these, they need to be engaged and supported ensuring evidence is visible, accessible, usable and used.

As we increasingly move towards a world of decentralised decision making, the number of decision makers continually expands. This means that any NICE equivalents will need to engage and influence a range of both producers and users of evidence, encompassing policymakers and commissioners across central and local government, academia, third sector, independent research organisations, service providers, and the end users of services themselves. We have argued before that service users have been left out of much of the debates surrounding evidence for social policy, when they are in fact a vital ally in helping to identify problems and issues in service delivery, whilst also helping to hold decision makers to account for the ways in which services are delivered.⁴

Deciding upon size and organisational form

Alonside agreeing the overall objectives for a NICE for social policy, there needs to be discussion on what resource is needed to achieve this.

The National Institute for Health and Clinical Excellence has around 250 staff and a budget of approximately £60 million per year. In the current spending context, it is probably unlikely that any equivalent developments in social policy would be of a similar size. Though there is an argument that investing to identify the most effective approaches would deliver more efficient public services, helping to recoup any investment.

In terms of organisational design there are a number of options to be considered, with three listed in the table below.

Defining the areas of focus for a NICE for social policy

Interlinked with the discussion about size and form will be an overlapping examination and consideration of where to focus.

Social policy is vast, spanning education, health, social care, criminal justice, and beyond. This could require any new centres to focus on certain areas initially, or to develop individual centres which focus on specific topics. We have previously proposed a model which involved a central hub that developed expertise in a number of areas and then applied these skills to help a network of centres make developments in particular fields, such as education, early years, reducing crime, or sustainability. This would enable cross-pollination of ideas and innovative approaches across different fields.

If areas of subject focus are to be explored these could be selected based upon a variety of criteria, including in the areas of greatest government expenditure, where there is a strong availability of evidence, but there are barriers affecting its update, or alternatively, where the field is particularly underdeveloped and little is known about what is or isn't effective. These challenge areas could then be reviewed, altered, expanded or changed over time.

Arguably we shouldn't limit the scope of any new infrastructure to simply social policy alone. Regardless of whether social policy, innovation policy, or business policy, are being discussed, there are very similar challenges and barriers that affect them. Yet there is a counter argument about needing to balance breadth and depth when deciding where to focus and prioritise initially.

There is also a decision to be made around whether any NICE for social policy focuses upon policy, programmes and practice, or a mix of these. Of course, in reality the split is rarely neat, but the different spheres will require different resources and encompass different audiences. Although the distinctions are rarely mutually exclusive, the table below begins to outline some of the differences between these three components.

NICE arguably excludes the first two options and instead focuses upon evidence and publishes guidelines for medical treatment and procedures, clinical practice and guidance on health promotion. A 'NICE for social policy' will need to be explicit about whether it wants to encompass all the elements, or whether it is excluding certain spheres.⁶

Defining what evidence means

Any new centre will need to be very clear as to what 'evidence' means and what 'quality' looks like. Many institutions, such as Washington State Institute for Public Policy and the Cochrane Collaboration, draw upon academically published literature. Although to an extent this can assure certain quality criteria have been met, it can mean that publication lag can delay new approaches being identified, or arguably more commonly, those approaches which aren't on the radar of academia could be missed out. This means a NICE for social policy should consider wider sources of evidence, such as from independent research organisations or evaluations commissioned from the providers themselves.⁷ Many argue that evaluations undertaken by the provider can lead to overstated claims and unreliable findings, however, not including them at all could miss out on a potentially rich and insightful evidence stream. These two views, however, aren't irreconcilable, if there are appropriate levels and standards of evidence which clearly denote the strength - and by extension - reliability of the claims, then all evidence could be duly considered.⁸

By defining what is working, we also need to be clear on when something is failing. At what point does developing and testing stop, when it's recognised that a programme or policy is not going to be effective?⁹

Deciding upon a location

NICE is quasi-autonomous from government. Any social policy equivalent should ideally strive to be the same. Too far from government and it could lose impact, yet too close and it could lose powers of objectivity or critique. Another possibility is being based in an academic setting. Although advantageous as it could mean it would draw upon cutting-edge expertise and research, it may also mean that it doesn't engender 'buy in' from other audiences (see Annex 1 for the locations of other evidence centres).

What a NICE for social policy could do

In some fields new research will need to be generated, whilst in other areas the challenge is one of awareness-raising, dissemination and translation. The crucial role that a NICE for social policy should do is effectively marry the supply of evidence with demand, plugging gaps where needed, and ensuring evidence is available, useable, and crucially, it is used. This section briefly outlines what a NICE for social policy could do to help meet these goals.

Increasing and improving the prevalence of useful evaluations

Social programmes and policies have changed, evolved and been amended over the past few decades, yet the ways in which we evaluate these have remained relatively static. With the opening up of public services there is a plethora of new approaches and interventions available that may be more efficient, so the need for experimentation and testing becomes ever more important. Therefore any new evidence centres must help facilitate the generation of low cost and agile impact evaluation tools and methods.

This does not necessarily need commissioning costly research, instead we can find new ways of utilising the information already available and empowering society to make use of it. This involves using administrative data more efficiently, or opening up the wealth of evidence held by the funders of social programmes, such as the Big Lottery Fund.¹⁰

There may be times when any new initiative realises that there is little available evidence in certain fields. If generating this is beyond its scope or capabilities, then there could be systems in place to trigger academia, research councils, government departments, or others, to begin to fill these gaps.

Another key area to explore is the ways in which we use experimentation in policy development. The Obama Administration has recently developed a way in which more policies are evaluated and tested.¹¹ It is proposed that federal agencies should demonstrate how they have used evidence and how they will commit to effective testing and experiment of policies within their Budget 2014 submissions. This includes a decisive call for greater testing and experimenting, including details on how departments have sponsored low-cost rigorous evaluations using administrative data, how agencies have 'waived' legal or regulatory provisions in order to enable rigorous testing of programmes to be carried out, as well as potentially incentivising evidence-based practices in agency grant-making programmes.¹²

Balancing evidence with innovation

Standards and scales of evidence can be exceptionally useful in helping to judge the evidence behind claims. Many organisations use terms like 'top tier' or 'proven' to classify programmes and help decision makers to select those interventions that are deemed to be working. These commonly draw upon studies where the intervention has been evaluated using random assignment. This can be useful in fields where there is a strong evidence base of well conducted RCTs; however, there are many fields where these types of evaluations may be lacking. Equally, there are instances when this method isn't appropriate, such as when the intervention is at an early stage of development, it is very localised or involves a small sample size. This can make it hard to judge and compare innovations which have alternative types of evidence. RCTs could then be seen as what to aim for, but recognising that other types of evaluation need to be considered, especially when an intervention is at an earlier stage of development. Any new evidence centre should then ensure that the bar isn't set so high that providers can start to work their through the levels. Project Oracle's Standards of Evidence are a very useful starting point.¹³

Powers of accreditation

To help aide decision making, a suggested role for a NICE for social policy has been as an accreditation body.¹⁴ The Cabinet Secretary Jeremy Heywood has suggested that kite-marks could be awarded to effective social policy schemes such as those rehabilitating prisoners or drug addicts.¹⁵ There could also be accreditation to acknowledge high performing organisations, such as the Early Intervention Foundation or the Education Endowment Fund. The signalling as to what is effective could be useful in guiding commissioners and others in decision making, however it does assume that there will be rationality in decision making, a key point which we will return to.

Advice and guidance for both evidence users and producers

To help advance the understanding of evidence and building the case for what is working, a key role for a NICE for social policy will be to draw all this research, insight and expertise together to provide a rich source of advice and guidance, in a timely useable and accessible way. This can be advice on what is or is not effective, guidance on what counts as evidence at different stages, or the ways in which this can be measured. This could involve engaging policymakers, practitioners, providers, researchers, users, and beyond, demanding greater consideration to be paid to ensuring evidence is timely, appropriate and accessible to different audiences.

Stimulating the demand for evidence and enabling it to be acted upon

Even when rigorous evidence does exist, a range of barriers can prevent its uptake, ranging from a lack of awareness, perceived unreliability, or facets of the organisational culture that can resist or block. NICE has power to ensure its guidelines are implemented, and although any new centre may not have such authority, it will need to ensure the available levers and incentives to ensure that the demand for evidence becomes institutionalised in decision making. If accrediting powers are granted to a NICE for social policy, then there will need to be incentives for funding to flow to these policies or programmes.

Evidence must be seen as iterative and evolving

The expression 'what works' is commonly used when talking about effective programmes. However, we would not want this to be taken that the adoption of an 'evidence-based programme' is the end result.¹⁶ Issues will need revisiting to ensure what is working now continues to be effective. This means any accreditation and kite marking scheme will need very careful thinking through.

This also has implications for how any ‘NICE for social policy' offers other forms of advice and guidance. It should be made clear that the identification of 'proven practice' is not the end result. There are countless examples of perceived wisdom subsequently being overturned,¹⁷ therefore there needs to be continual monitoring and evaluating to ensure that the 'what works' is 'still working'.¹⁸

What else needs to happen

There are a number of wider systemic changes that may need to occur if a NICE for social policy is to become embedded successfully:

• Decommissioning regimes: for instance, once alternative and more effective approaches have been identified, we need to ensure decommissioning of the incumbent service can take place.¹⁹
• Dealing with negative findings: to be truly experimental, we need to be frank and honest about what is not working, as much as what is. This involves accepting that for some programme developers and service delivery organisations this could mean the termination of funding.²⁰
• Training and skills development to ensure new ways of working can take place, as well as incentives to encourage changing practice.

Learning from other organisations and other countries

As the table in Annex 1 shows there are a wealth of research institutions that produce, generate, synthesise and communicate evidence in a variety of ways. Although we can easily see their size, scale, budget and operational model, it should be noted that there are very few evaluations of their actual performance and impact.²¹ To get a better understanding of what mechanisms are and are not - effective at generating evidence and then getting this used there is a need to be 'evaluating the evaluators'.²²

The Alliance for Useful Evidence will be a useful ally when developing a NICE for Social Policy. The Alliance takes a campaigning stance to ensure evidence is not ignored. This resource could be drawn upon to feed into exploration and development of evidence centres, as well as helping to embed the centres once they have been established.²³

Concluding remarks

We strongly endorse plans for a centre akin to NICE to be established in other areas of social policy, yet we equally recognise that evidence is only one piece of the puzzle. Any creation of a 'NICE for social policy' should be set within the context that decision making is influenced by a plethora of other factors, including, ideology, politics, cost, timescales, opinion, values and personal judgement. In addition, evidence is rarely definitive or has absolute objectivity. However, the challenges should not excuse action. Although these issues and concerns need careful consideration, the overall ambition of finding safe, efficient ways of delivering our public services to improve the outcomes for all service users, should not be compromised.

Annex 1: What do we know about institutions that already operate in this field and what they do?

We have looked at a number of institutions that already operate and summarised the range of approaches taken. Further details are provided in Table 1.

Working groups or research boards to establish standards and/or direct research

In 2003, in response to recommendations made by the Coalition for Evidence-Based Policy, the US Office of Justice Programs and other Federal agencies developed a common scale based on evaluation methods to rank the effectiveness of programmes and to decide whether or not they are ready for dissemination.

A broader proposal has been made for a UK national public services research board in which services could share and learn evidence skills, expertise and organisational arrangements from each other and then apply these.²⁴ For example police and probation services would learn about medical schools and profession-led practice guidelines from the NHS, and the NHS would learn about management training from the police service.

Similar approaches have been taken for specific sectors. Crime prevention panels in countries such as Canada, Finland and Australia also set a national research agenda, support systematic reviews, provide technical assistance to local agencies and fund pilot programmes.

Expert institutes and/or databases of effective approaches

The most well-known expert institute, the National Institute for Health and Clinical Excellence (NICE), produces science-based practice guidance and manages a national database of evaluations of new and existing medicines, treatments and procedures. NHS Evidence acts as a 'Google for healthcare', ranking search results from credible medical sources; the NHS Evidence 'Accreditation Mark' also denotes those organisations that have met criteria for producing high-quality information. Similarly, the Cochrane Collaboration is an international network of researchers and practitioners who prepare systematic reviews of healthcare evidence.²⁵

Somewhat similar initiatives exist in other fields, but they do not as yet ensure universal coverage across social policy. The Social Care Institute for Excellence (SCIE) is an independent charity, funded by the Department of Health and the devolved administrations in Wales and Northern Ireland, which produces and disseminates evidence about what works in social care. The Evidence for Policy and Practice Information Co-ordinating Centre (EPPI Centre), based at the Institute of Education and part-funded by the ESRC, reviews evidence in education and health promotion, while the Campbell Collaboration produces systematic reviews in education, crime and justice, and social welfare.

In the US context, the University of Colorado's Center for the Study and Prevention of Violence 'Blueprints' framework identifies effective violence and drug prevention programmes. Similarly, the Promising Practices Network operated by the RAND Corporation offers research-based information on children and families. Amongst others, two academic research centres - J-PAL (MIT) and EdLabs (Harvard University) - conduct experimental evaluations in poverty reduction and education respectively.

The Blueprints for Violence Prevention programme at the University of Colorado has a data base of 'proven' and 'promising' approaches. (Dartington Social Research Unit is currently seeking support to move Blueprints to the UK.) Similarly, RAND's Promising Practices Network is a searchable database of 'promising' and 'proven' interventions from across children services.

University-based 'public service schools'

Following the healthcare model, it has been suggested that service-specific 'public service schools' should be established in research-intensive universities to integrate training, practice and research.²⁶ These schools would train practitioner-academics, distil and disseminate international evidence to inform professional practice (so helping to develop an evidence-informed profession), generate and implement new evidence, and lead national research efforts.

Practitioner-academics would also be well-placed to develop prototype services based on their research. They could work with providers, including small providers, to help them establish the effectiveness of their approaches and receive increased investment, so enabling these approaches to scale. Where some of these schools already exist in areas of social policy they should adhere to a scientific approach to ensure that the most effective approaches are identified and replicated.

A somewhat analogous model which blurs boundaries between practitioner and academics is the Center for Court Innovation, a New York-based think tank that acts as the court system's independent research and development arm, testing demonstration projects and assisting practitioners to launch their own programmes. The Center is currently working with The Young Foundation to launch an independent institution in the UK (the Centre for Justice Innovation).

Applied policy research institutes

The Washington State Institute for Public Policy carries out practical non-partisan research at legislative direction on issues such as education, welfare, children and adult services, and criminal justice. The Institute uses its own analysts and economists, academic specialists and consultants to undertake systematic reviews and compile 'consumer reports' to detail the potential benefits of service users, taxpayers and the wider state that could be accrued against the cost of certain interventions.²⁷

Some ESRC research centres also fall into this category. For example, the Third Sector Research Centre is currently reviewing research on civil society organisations. Its 'Knowledge Portal' will allow practitioners to search for evidence and other resources relating to the third sector.

Assistance for smaller providers to establish rigorous evidence for their approaches

The GLA's Project Oracle combines of number of these models, with a particular emphasis on working with providers to establish evidence for their approaches. Project Oracle's aim is to establish a coordinated approach to evaluation and establish mechanisms for identifying and sharing effective practice for youth services, starting with youth violence. The project includes an evaluation toolkit and is developing an easy-to-use web resource of effective practice. It has now been expanded to draw upon academic research capabilities to enable providers to move up through the standards.²⁸

The Early Intervention Foundation, as recommended by the Allen Review, represents another hybrid approach, in its proposed role as advocate for early intervention, a hub for evidence, and a broker between investors and programmes.

In summary, there are some common features if the examples we have reviewed in Annex 1 are:

• Most, but not all, tend towards supply-side interventions: generating new research.
• Comprised predominantly by supply-side professions (academic researchers, policy-researchers).
• Almost all are publicly funded; in the US private foundations also play a considerable role.
• Most are core-funded, but some are funded on a project-by-project basis (e.g. WSIPP).
• Many are structured as networks, but many also maintain a large core of staff.
• It is possible to leverage a large amount of voluntary contributions from the academic community, though this may be dependent on the main activities being the production of journal-published papers.
• Most institutions appear to focus on a particular policy domain, rather than more across the market.
• We know little about the comparative impacts on practices and policy between these institutions.

Endnotes