Foreword
From criminal justice, children's services to poverty reduction, this report contains essays from organisations using different methodologies and approaches to generate evidence and influence policy and practice in a number of service areas.
The idea that policy and practice should be underpinned by rigorous evidence is internationally accepted, yet there is recognition that the level of rigour in evaluating ‘what works’ in social policy remains limited. In a time of public service reform and more decentralised decision making, the need for timely, accessible and reliable evidence is becoming ever more important.
At NESTA we both use and produce evidence through a combination of practical programmes and research projects. We are keen to learn from the organisations featured here – as well as others working in the field – to help strengthen the evidence base and improve the sharing of this knowledge.
We recognise that with local autonomy comes an enormous opportunity for innovative approaches to thrive, but in order for this to happen practitioners, commissioners, users and other decision makers must know what works – and what doesn't. Sharing evidence will help improve outcomes whilst helping to prevent duplicated efforts and wasted resources.
Different policy areas will undoubtedly need different support and development in the production, dissemination and use of research and evidence, with many organisations having a key role to play. At NESTA we are working to forge connections with a range of organisations and explore possible ways forward. As you will see from the essays there is much activity happening and, beyond the examples included here, there is a great deal of interest and enthusiasm in progressing this agenda.
We would welcome your thoughts.
Ruth Puttick
Policy Advisor, Public and Social Innovation, NESTA
April 2011
NESTA is the UK's foremost independent expert on how innovation can solve some of the country's major economic and social challenges. Its work is enabled by an endowment, funded by the National Lottery, and it operates at no cost to the government or taxpayer.
NESTA is a world leader in its field and carries out its work through a blend of experimental programmes, analytical research and investment in early-stage companies. www.nesta.org.uk
Part 1: Project Oracle
Mat Ilic, Greater London Authority and Stephen Bediako, The Social Innovation Partnership
1. Introduction
This paper has been written against a backdrop of change in public services, taking place at an unprecedented speed and scale. The Government has committed to a series of wide-ranging reforms, with a renewed focus on achieving better social outcomes at a reduced cost. Some of the more challenging debate now is about what constitutes reliable evidence of social change, and how outcomes or results of specific interventions can be verified.
In the United Kingdom there is rare cross-party consensus on the benefits of early intervention. For example Iain Duncan Smith MP and Graham Allen MP have progressed the 'what works' agenda.
- The Allen Review recommended a portfolio of evidence-based, early intervention programmes aimed at improving outcomes for children, young people and their families;
- The Centre for Social Justice identified in a recent report "...a fatal failure at the heart of government spending decisions.” namely the absence of adequate evidence in the construction of social policies and programmes, in turn leading to poorly designed provision with little or no proof of positive change.
In addition, with the onset of an era of localism (or 'mass localism') where individuals and communities need to come together to resolve their problems with local solutions, one is entitled to ask how communities and individuals will obtain the knowledge and evidence they need to make sound decisions.
As a response, the Greater London Authority (GLA), with support from The Social Innovation Partnership, contributes this paper on Project Oracle. The project, having emerged from the Mayor's proposals in Time for Action (his programme to tackle serious youth violence), seeks to: “Understand and share what really works” in supporting young people and preventing youth violence. It does this by combining, for use by service providers, funders and policymakers:
- A tried and tested programme evaluation methodology.
- A technical platform - the Project Oracle website.
- A supportive centralised officer resource.
2. The origins of Project Oracle
Project Oracle seeks to further the aims of evidence-based policymaking by stimulating collaboration between government, academia and the wider social intervention community. It was established in recognition of four key factors:
1 There is currently no clear understanding of what programmes work, in what conditions they work, and whether they therefore represent 'value for money'.
2 There needs to be a sustainable body of evidence so that the knowledge base evolves for future policymaking.
3 Evidence needs to be grown from somewhere using a consistent method, requiring a stimulus and a mechanism for providers to develop continuously.
4 The root causes of social problems require a deep understanding that should be captured and documented, particularly given the pace of social change.
Project Oracle emerged as one of the six work streams from Time for Action. The project seeks to establish a standard for evidence-informed decision making on children and young peoples' policy in London. The Mayor believes that he is: "Strategically best placed to initiate joint work to... evaluate programmes that have greatest benefits and to identify those that don't..." Recognising the vital contribution that partners in policy, funders, service providers and academics can make, the project's Delivery Board represents cross-sector leadership.
The challenge for Project Oracle is how to build the evidence from London-based activities, avoiding the temptation of importing 'proven' interventions from other countries and contexts without further validation.
3. Project Oracle: an overview
How Oracle works (a summary):
- Standards of Evidence for London were commissioned by the GLA and delivered by Dartington Social Research Unit, in collaboration with international experts such as Delbert Elliot (University of Colorado) and Steve Aos (Washington State Institute for Public Policy).
- The Standards document has been translated into a set of practical activities that can be conducted by providers of social programmes. These activities are part of a continuous improvement process that involves providers going through an online self-assessment, which tests the robustness of the evidence in support of their programme(s).
- The self-assessment is carried out by the provider alongside a practitioner guidebook. Depending on how their project scores (Levels 1-5, with 1 being entry level), practitioners are expected to undertake an improvement plan aimed at increasing the level of evidence that supports their claim to positive outcomes.
- This self-assessment activity requires a degree of human interaction and peer challenge, which in the current pilot stages is facilitated by a dedicated project officer from the GLA, with support from Masters students from social research disciplines. The GLA also runs monthly workshops and will engage in online collaboration once the Oracle website has benefited from further development (April 2011).
4. Standards of evidence
In the context of youth violence prevention studies, the current state-of-the-art standards of evidence methodology can be traced back to initiatives from the Center for the Study and Prevention of Violence, University of Colorado, USA, where the Blueprints research and evaluation programme was begun. In 2009, the GLA commissioned Dartington Social Research Unit to take forward the experience of evidence-based programmes from the United States, working with experts in the field to put together the standards needed for London.
In the absence of any standards it is likely that many social interventions for children and families are being used in ignorance; worse, some are being implemented in spite of indications that they can be harmful. Better information, communicated and used well, should mean that resources are confidently directed toward more effective projects to address the right problem. Widespread use of the London Standards should bring greater consistency to judgments about how to support young Londoners and prevent violence.
This is a long-term ambition. The GLA recognises that the London Standards will need their own body of knowledge to reflect the capital's complex social and political environment and culture.
The London Standards of Evidence can be summarised in the following way:
|
Table 1: The Oracle levels of evidence |
| Level 1 |
Sound theory of change or logic model, with clear plans for evaluation |
| Level 2 |
Demonstrating emerging evidence of impact |
| Level 3 |
Effective – comparison group (ideally random controlled trials), statistical analyses, 'effect sizes' |
| Level 4 |
Model – analysis of 'dosage', of 'fidelity of implementation' |
| Level 5 |
System-ready – 'multiple independent replication evaluations', and cost-benefit analysis |
NB: These standards have been summarised for the purpose of this report.
5. Making Oracle practical
The academically rigorous framework proposed by Dartington is valuable, but its ambitious goals of random controlled trials, longitudinal studies, and so on, are a long way away from the realities of projects aimed at children and young people in London.
Translating the Standards document for the use of mostly frontline providers not trained in evaluation had to be a creative process in itself. It required consultation with a very wide range of groups across London, as well as input from external advisers, notably Professor Betsy Stanko and other members of the Oracle Challenge Panel, Community Links, and The Social Innovation Partnership.
The Standards, with further development, have introduced the concept of a 'theory of change', an approach to planning or articulating social change developed in the US over the last 25 years. It is an interactive method for obtaining group and community consensus on the long-term goal, preconditions, and assumptions behind proposed social programmes. The GLA tested this approach in sessions with Oracle pilot projects (see below), and in ten workshops with 110 participants from 78 organisations from across London.
Ongoing support is necessary because achieving improved evidence of effectiveness is a developmental process, which requires those involved in commissioning and delivering social
6. The pilot approach
change projects to work together. To date this support has been provided by a single project officer and Masters students on work placements, but it is anticipated that as the volume and quality of evidence from providers improves, the quality of these resources must also be enhanced.
In September 2010, the Oracle Challenge Panel agreed to initial work with ten Oracle pilot projects. These were current, mostly voluntary sector projects, varying in both size and prominence, but all working with the aims of combating youth crime and violence.
The pilot has involved working intensively with these organisations, and has allowed the GLA to consider the benefits of Project Oracle through the eyes of funders, practitioners and academics. It has pointed to common issues that evidence and evaluation-building exercises face in the real world. This includes factors such as a recognised lack of proactive funding for research and evaluation, as well as poorly designed (and therefore unpersuasive) research.
7. Oracle's applications
Three scenarios have been identified in which Project Oracle and its methodology can make a difference:
1 When commissioners are buying new services or developing new products 'in-house' (thus the GLA is working with London Funders to move towards commissioning along Oracle lines).
2 Helping statutory organisations to reconfigure their operations or programmes (this has already started with 'GLA family' organisations, including the Metropolitan Police Service).
3 Improving projects or programmes directly and championing a single standard of evaluation (where Oracle works directly with providers to improve their evidence base).
Throughout the development of the project, the GLA has focussed on simplicity and clarity, to ensure that all concerned can successfully make the journey to improved outcomes for young Londoners.
8. Activities proposed for 2011-2012
The GLA expects to embed the principles of this project across organisations which are a part of the GLA group, thus leading by example. In 2008-2009, youth engagement and programme expenditure in the GLA group was estimated to be in the region of £40 million, which makes any effort to 'make it better' a worthwhile exercise in its own right.
Through its London-wide Delivery Board, the GLA aims to build a sustainability strategy for Project Oracle with the intention of advancing the initiative through the support of its key partners across London. This work will detail the lessons learned from the project and explore whether and how the initiative can be sustained in the context of a changing political and social environment.
9. Conclusions
NESTA has put forward for consideration Ten Steps to Transformation, one of which is 'innovation culture only comes from practice'. This is not about creating an innovative culture for its own
sake, but a culture where people feel empowered and supported to affect change and adapt their practice. Empowering involves providing advice and practical support. In this way, Project Oracle offers a pathway from practice to evidence, enabling projects to reflect on what made them (or can make them) succeed and why.
An important question to conclude with is: 'what is the purpose of knowledge in seeking social change?' When demonstration projects are funded as a test for national solutions (as NESTA and others do), how does the learning get retained for future decision-making activity? If a promising initiative is identified, what are the chances that it will be sustained?
The GLA aims for Project Oracle to be a source of information, evidence and inspiration for city-wide policymaking and decision making. The model may become one that other cities could adopt. There is no reason why, with time, the wider public cannot be empowered to scrutinise investment in social programmes by being given access to information about 'what works'. If future resources are directed in this way, the outcomes for young Londoners should improve more predictably over time.
Part 2: Combining research and practice: The Center for Court Innovation's approach to justice reform
Greg Berman and Aubrey Fox, Center for Court Innovation
1. Summary
The Center for Court Innovation is a public/private partnership dedicated to reforming the justice system through demonstration projects, original research, training and technical assistance. In New York, the Center for Court Innovation functions as the State court system's independent research and development arm – studying chronic problems, devising new solutions and testing their feasibility. After the Center has tested a model and documented its effectiveness, it then looks to disseminate the lessons more widely, helping reformers around the world adopt new ways of doing business.
2. Background
Figure 1: The Center for Court Innovation's business model
Demonstration
Field-testing new ideas
↓
Documentation
Tracking the impacts of experiments
↓
Replication
Helping government officials adopt lessons from demonstration projects
↓
Institutionalisation
New practices become embedded within the justice system
The Center's first demonstration project, the Midtown Community Court, was created in 1993 to address low-level offending in the Manhattan neighborhoods around Times Square. The Midtown Court combines punishment and help, sentencing misdemeanor offenders to perform community service and receive social services. The goal is to reduce both crime and the use of incarceration. Independent evaluators from the National Center for State Courts confirmed that the community court helped to curb street crime: prostitution arrests dropped by 56 per cent and illegal vending by 24 per cent after the court opened. Other results included improved compliance with court orders and reductions in case processing time. In addition, more than two out of three local residents surveyed in a telephone poll said that they would be willing to pay additional taxes to support the community court.
3. Current activities
The Center's next experiment was the Brooklyn Treatment Court. The first such court in New York City, Brooklyn Treatment Court worked with more serious cases: felony offenders with long histories of addiction. Following a model originally established in Florida, participants were linked to long-term drug treatment in lieu of incarceration. Progress in treatment was regularly monitored by a judge using a system of sanctions and rewards. Researchers from the Center for Court Innovation tracked the performance of drug court participants for three years after they left the program and found statistically significant reductions in both substance abuse and recidivism. Based in no small part in these findings, the New York State Court System made an institutional commitment to spread the drug court model statewide.
The drug court model also attracted the attention of the executive and legislative branches of government in New York. In April 2009, the Governor of New York signed into law a significant revision of the infamous Rockefeller Drug Laws, long regarded as the toughest in the United States for mandating prison for non-violent drug offenders. One of the explicit goals of this reform, which was celebrated by the Governor with an event at the Brooklyn Treatment Court, was to increase the number of defendants who would participate in drug court.
Figure 2: Problem-solving courts in New York
- 192 drug courts
- 8 community courts
- 41 domestic violence courts
- 28 mental health courts
- 55 integrated domestic violence courts
- 8 sex offender management courts
All 62 counties have at least one problem-solving court
Numbers include projects in operation and planning as of January 2010
Source: Center for Court Innovation.
4. Structure of the Center for Court Innovation
While community courts and drug courts are the most prominent of the programs that the Center for Court Innovation has implemented, they are far from the only ones. The Center has also been responsible for establishing New York's first mental health court, domestic violence court, reentry court and others. While each of these projects is unique, they have come to be known collectively as 'problem-solving courts' for their efforts to address the underlying issues that bring defendants into the justice system. Over the past 15 years, problem-solving courts have been widely replicated throughout New York State (see Figure 2). Most of these programs have either been based on models created by the Center for Court Innovation or created with training and technical assistance from the Center's team of expert consultants.
The Center for Court Innovation is a non-governmental organization that is operated as a project of a larger non-profit agency, the Fund for the City of New York. In fiscal year 2010, the Center had a budget of $17.6 million which was underwritten by a range of funders at the city, state and federal level (87 per cent of the Center's revenues come from government grants, 13 per cent from private foundations and fee-for-service contracts). The Center for Court Innovation seeks to combine action and reflection, doing and thinking. It brings together professionals from an array of disciplines (attorneys, social workers, mediators, victim advocates, social scientists, journalists, technologists and others) to promote justice reform. Broadly speaking, the Center's 175 full-time employees work in four principal areas:
1 Demonstration projects. As described above, the Center's primary business is the planning, implementation and operation of demonstration projects, and has been responsible for creating 20 different model projects. These vary in size (for example, the Red Hook Community Justice Center hears several thousand cases each year while the Brooklyn Mental Health Court less than 100) and focus (for example, the Brooklyn Domestic Violence Court handles serious felony cases involving intimate abuse while the Harlem Youth Court deals with minor offenses committed by juveniles).
2 Research. The Center is committed to the idea of 'action research' and employs a team of in-house researchers that regularly monitors the work of the Center's demonstration projects, tracking impacts on street crime, substance abuse, sentencing practice, levels of neighborhood fear and public trust in justice. Researchers from the Center also conduct independent evaluations with national policy implications, including a multi-year evaluation of court-mandated drug treatment, a randomized trial of the effectiveness of batterer intervention programs in domestic violence cases and an examination of the extent to which a judge's practice is changed after assignment to a problem-solving court.
3 Strategic communication. The Center uses a number of strategies to spread information, including a website that attracts 82,000 unique visitors per month, and clearly written how-to manuals and best-practice guides for criminal justice officials (more than 50,000 publications are downloaded each month). In addition, the Center produces a monthly podcast, New Thinking, which features interviews with leading criminal justice thinkers. Staff from the Center have published numerous books, articles in the popular press, trade publications and academic journals; and regularly make presentations at national conferences and workshops.
4 Technical assistance/training. The Center provides hands-on, expert assistance to reformers - judges, attorneys, probation officials, community organizers and others – from around the world, offering guidance on assessing public safety problems and crafting workable, practical solutions. Structured site visits to the Center's demonstration projects are a key part of the organization's approach to technical assistance; over the past five years, they have hosted an average of 625 visitors per year. The Center is currently working with innovators both in the United States and abroad to help create new responses to problems like drugs, domestic violence, delinquency, and neighborhood disorder.
5. Lessons from the Center for Court Innovation
The Center for Court Innovation's approach to promoting evidence-based justice sector reform is multi-faceted. Key elements include:
- Balancing independence and access. The Center for Court Innovation has sought to work closely with government while being formally independent from it. Working with government decision makers helps to ensure the relevance of the organization's work. Yet the Center's independence from government grants it the freedom it needs to think beyond the electoral cycle and to pursue a long-term vision of justice reform. The Center's independence from government also means that it does not operate under some of the institutional constraints (e.g. civil service regulations, union rules) that hamper efforts to create an entrepreneurial culture within government. Finally, the Center's independence gives it the room it needs to issue findings that are less than positive. For example, the Center's randomized trial that examined the use of batterers' intervention programs in the Bronx found no evidence of impact on the behavior of offenders. Although this finding directly questioned common practice of local judges, the Center's study was not suppressed. Rather, it was featured in a front-page story in the New York Law Journal and in multiple journals.
- Combining research and practice. Having researchers work alongside criminal justice practitioners has multiple benefits for the Center. Firstly, it forces practitioners to think more rigorously. In particular, it encourages those who plan and implement demonstration projects to be more disciplined about articulating measurable goals and objectives for their work. On the other side of the coin, researchers benefit because they become grounded in the messy realities of day-to-day implementation, which makes their work more nuanced and their writing easier to read. Unlike some academic evaluators, researchers from the Center also know enough not to hold new programs to unrealistic standards of performance.
- Bridging the local and the national. The Center for Court Innovation has always had one foot in the world of local practice and one foot in the world of national policy. The Center's sustained engagement on the ground in New York has given it credibility and enabled it to build trust with local practitioners and policymakers. But the Center has a broader worldview than the typical local organization and its national reach (and connections) means that it can bring ideas from across the country to New York. For example, the Center recently adapted the Ceasefire anti-violence program that has shown success in reducing gun crime in Chicago to the Brooklyn neighborhood of Crown Heights.
- Using multiple methods of analysis. While many researchers extol the importance of randomized trials, these trials rarely happen in the world of criminal justice. Instead the Center believes that randomized trials are not the only kinds of studies that have value and that there is a lot to learn from other types of research, including quasi-experimental designs, qualitative studies and process evaluations. The Center is also committed to moving beyond a pass-fail approach to evaluating social programs, involving not just tracking the impact on crime rates but examining a much wider set of program outcomes, such as compliance with court orders, impacts on system efficacy, public confidence in justice, whether court litigants feel the justice system is treating them fairly.
6. Wider influence and impact
The idea of problem-solving justice has spread far beyond the borders of New York. Mainstream American legal organizations, including the American Bar Association and the Conference of Chief Justices, have endorsed the concept. On the campaign trail, Barack Obama explicitly endorsed drug courts, making reference to the Center's research, saying: “Drug courts have proven successful in dealing with non-violent offenders. These courts offer a mix of treatment and sanctions, in lieu of traditional incarceration... The success of these programs has been dramatic: One New York study found that drug court graduates had a re-arrest rate that was on average 29 per cent lower than comparable offenders who had not participated in the drug court program. These programs are also far cheaper than incarceration.” In the FY2012 executive budget recently submitted to Congress, President Obama allocated $57 million to support problem-solving courts.
International interest in the Center for Court Innovation and its demonstration projects is growing. Staffers from the Center have worked with criminal justice reformers in 50 countries, including aiding the development of new community courts in England, South Africa, New Zealand, Australia and Canada. Australia has recently launched its own Center for Court Innovation, operating out of Monash University. Several organizations (including the Young Foundation and Policy Exchange) have recently issued calls for a Center for Court Innovation in the United Kingdom.
Part 3: Washington State Institute for Public Policy
This summary was prepared by NESTA
1. Summary
This summary has been written by NESTA and unless otherwise indicated is based upon documents provided by the Washington State Institute for Public Policy.
This summary covers the development of the Washington State Institute for Public Policy and the role it performs currently, as set out in the 2009 Legislature. The Institute's ‘4-Step’ approach to research is outlined to show how different interventions are evaluated, and how the findings generated are presented to Washington State to enable research to influence policy decisions. This note also covers the current work to translate the Institute's model to the UK.
2. Roles and responsibilities
The Washington State Legislature created the Washington State Institute for Public Policy (WSIPP) in 1983 to carry out practical, non-partisan research – at legislative direction – on issues of State importance. A Board of Directors representing the Legislature, the Governor, and public universities govern the Institute and guide the development of all activities. Alongside an in-house research team of 13, the Institute collaborates with universities and other experts.
The role of the institute has evolved since it was established. In the mid-1990s, the Legislature directed the Institute to identify evidence-based juvenile justice programs that could lower crime. The Institute built its first benefits-cost analytical tool in 1997 to help the Legislature select sound investments, identifying several programs that could reduce crime and save Washington taxpayers’ money. In subsequent sessions the Legislature used the results to begin a series of policy reforms. Then in the early 2000s, the Legislature began to direct the Institute to apply the same benefit-cost approach to other public policy areas, including education, child welfare, adult mental health and substance abuse.
The current areas of staff expertise include education, criminal justice, welfare, children and adult services, health, utilities and general government. WSIPP also collaborates with universities to extend its capacity into other subject areas.
3. Current policy areas
The 2009 Legislature directed WSIPP: "To calculate the return on investment to taxpayers from evidence-based prevention and intervention programs and policies." The Institute was instructed in the Legislature to produce a: “Comprehensive list of programs and policies that improve [...] outcomes for children and adults in Washington and result in more cost-efficient use of public resources.” The following public policy areas are focussed upon:
- Crime
- Employment
- Public assistance
- Child maltreatment
- Housing
- Public health
- Education
- Mental health
- Substance abuse
The Legislative cycle is two years long, with the next Legislature due in April 2011. A full report covering the 2009 period will be published in June 2011, and provide a ranking of programs currently implemented in Washington, alongside a broader scope of policy options available.
4. Approach to research and evaluation: The 4-Step Model
WSIPP conducts research using its own policy analysts and economists, specialists from universities and consultants. Over the last decade, and continuing the current legislature, WSIPP has continued to develop and improve its 4-step research approach. This includes the use of a cost-benefit model which is used to calculate policy costs against likely economic returns, alongside predicting the impact of competing investment options. WSIPP's cost-benefit model is used as part of a four-step program to identify 'what works':
Step 1: What works
A systematic review of the available literature, from the US and elsewhere, is assessed against high standards of scientific evidence to identify those interventions that have best achieved the outcomes (and which ones have not). The empirical approach follows a meta-analytic framework to assess systematically the entire research literature on a given topic. The research studies with strong, credible evaluation designs are prioritised, whilst those with weak design are discarded.
Step 2: What makes economic sense?
WSIPP then calculate the costs and benefits for Washington State. A cost-benefit analysis is undertaken by answering two questions:
1 How much does it cost to produce the effect found in Step 1?
2 How much is it worth to people in Washington State to achieve the outcome?
To answer these questions, WSIPP developed an economic model that provides estimates presented from three distinct perspectives: the benefits that accrue solely to programme participants, those received by the taxpayers, and any other measurable (non-participant and non-taxpayer) benefits. The sum of these perspectives provides a 'Total Washington State view' on whether a programme benefits exceed its costs.
Step 3: Impacts on State-wide outcomes
The findings from Step 1 and Step 2 are then used to prepare 'Consumer Reports' of what works – and what does not – ranked by benefit-cost estimates. In Step 3, the degree to which a 'portfolio' of policies is likely to affect big-picture state-wide outcomes such as crime or high school graduation rates produces a ranking of public options to reveal the pros and cons of different options.
Step 4: Assessing risk
The final analytical step involves testing the robustness of the findings. Considerable uncertainty can exist in any estimate of benefits and costs; therefore it is important to understand how conclusions might change when assumptions alter. To test risk, WSIPP perform a 'Monte Carlo simulation' in which the key factors used in the calculation are varied to re-estimate the results of the analysis. The purpose is to test the riskiness of decision making to determine the probability that costs would outweigh benefits if a particular policy were adopted.
5. Dissemination of findings
The cost-benefit model has been used to inform policy decisions for over 13 years and can be adapted for a variety of policy areas.
The Institute produces evaluation reports covering the 4-step model outlined above to show what does and doesn't work, and the rate of return on taxpayer investments. An overview of two evaluations is provided below:
1 Benefits and costs of prevention and early intervention programs for youth
This evaluation set out to investigate whether prevention pays; specifically if for each dollar the State spends that there will be greater than one dollar worth of benefits generated. The programmes included in the evaluation span education, youth development, child welfare, mentoring and youth substance abuse prevention programmes, and were analysed with monetary values assigned to any observed changes in education, crime, substance abuse, child abuse, teen pregnancy and public assistance outcomes. The report concluded that there is credible evidence that well-implemented programs can achieve significantly more benefit than costs. The programmes identified included 'Nurse Family Partnership for low income women' which was found to generate $2.88 per dollar spent; and 'Project Towards No Tobacco Use' which generates $55.84 per dollar spent.
2 To reduce future prison construction, criminal justice costs and crime rates
In 2006 it was forecast that Washington State would need two new prisons by 2020, and possibly a third by 2030. Since a new prisons costs around $250 million to build and a further $45 million a year to operate, the Washington State Legislature expressed an interest in indentifying alternative 'evidence-based' options that could reduce the future need for prison beds, save money for state and local taxpayers and contribute to lower crime rates. To address this, the Institute conducted a systematic review of 571 rigorous comparison group evaluations of adult corrections, juvenile corrections and prevention programmes, most of which were operating in the US, to estimate the benefits and costs of evidence-based options. Programmes evaluated include vocational education in prisons, Teen Courts, and Adult Drug Courts. The evaluation concluded that if Washington successfully implemented a moderate-to-aggressive portfolio of evidence-based options, involving expansion of programmes to reach between 20 to 40 per cent of the remaining eligible population, then a significant level of future prison costs can be avoided, with taxpayers saving about $2 billion and crime rates reduced.
6. International influence: Transferring WSIPP's Model to the UK
The Institute is currently working with the UK-based Dartington Social Research Unit (SRU) to transfer Washington State Institute's Economic Model to the UK. SRU will translate the model into the UK context by replacing the US data in the model with UK inputs. The first results will be presented as a cost-benefit analysis of interventions selected in policy areas relevant to child well-being. The programme, supported in the UK by Birmingham and Manchester City Councils, and by Greater London Authority, will result in software available to all local and health authorities that predict the costs and benefits over a child's lifetime of competing investments in services.
Following the translation of the model into child well-being policy and interventions, the full model will then be translated into ten further policy areas affecting both adults and children. The final software package will be freely available and enable both central and local government commissioners to generate custom reports based on the resources and needs in their area, and calculate the costs and benefits of competing investing options. As well as showing 'what works', the software is also planned to reveal harmful, and therefore expensive, policies and programmes. The software is due to be ready from January 2012 onwards. Whilst the software is being developed, the Social Research Unit is using existing data in the policy areas of youth justice and child welfare to respond to immediate requests for investment advice.
The Social Research Unit cited three reasons for selecting and developing the WSIPP model:
1 It is cautious in its estimates of potential savings.
2 It can be consistently applied across a range of policy areas.
3 It has been proven to influence major policy decisions, including the shifting of resources from prisons to prevention.
Part 4: Using impact evaluations to inform policy
Iqbal Dhaliwal, Global Director of Policy and Caitlin Tulloch, Policy Analyst, J-PAL, Department of Economics, Massachusetts Institute of Technology (MIT)
1. Introduction
A. Mission
Billions of dollars are spent every year on development policies and programs, but there is relatively little rigorous evidence on the true impact these programs have on the lives of the poor. Rigorous scientific evidence on what programs or policies work is hard to come by, in part because it is so difficult to attribute changes in people's lives to a program, rather than other external factors. This scarcity of rigorous evidence on program impact, and the technical language in which the little evidence that does exist is presented, makes it inaccessible to policymakers, many of whom then rely on intuition and anecdotal evidence in deciding which programs to fund and implement.
The Abdul Latif Jameel Poverty Action Lab (J-PAL) is a network of 54 affiliated professors who use randomized evaluations to answer questions critical to poverty alleviation. With offices on five continents, J-PAL works to reduce poverty by ensuring that policy is based on scientific evidence. Besides undertaking research, J-PAL also helps build the capacity of policymakers to evaluate the impacts of programs and to widely disseminate the results of field research to policymakers, to ensure that policy is based on scientific evidence and effective programs are replicated and scaled up.
B. J-PAL's history
The Poverty Action Lab was founded in 2003 by MIT professors Abhijit Banerjee, Esther Duflo, and Sendhil Mullainathan. The concept behind J-PAL was not only to facilitate a larger number of evaluations by affiliated professors and partners, but also to exponentially increase their impact by promoting the sharing of methods and results. In 2003, NGO Development Innovations, run by affiliate Dean Karlan, changed its name to Innovations for Poverty Action and began an ongoing partnership with J-PAL. J-PAL offices are located at universities and provide access to academic resources, while IPA country offices are not based at universities. In 2005, J-PAL had affiliated professors at only four universities running 20 evaluations. Since then the number of affiliates has grown to 54, and the number of projects has increased to more than 250.
C. Structure of J-PAL
There are two main components of J-PAL: the network of affiliated professors, and the staff who support their work at J-PAL offices. Affiliated professors are based at universities around the world, where they pursue their own research agendas in diverse areas of development including education, health, finance, environment, governance, and agriculture. In their field projects, researchers evaluate programs and policies that are being implemented by governments, NGOs, international development organizations, and foundations. Contrary to commonly held perceptions that evaluations are forensic fact-finding exercises, J-PAL's affiliates often work closely with the implementing organizations to design new and innovative approaches informed by economic theory and past research findings, to help solve intractable problems in development. Researchers bring a high degree of rigor, objectivity, and independence to the measurement of program impact, and many of these evaluations go on to be published in top peer-reviewed academic journals like the Quarterly Journal of Economics and The American Economic Review.
D. Policy outreach
In 2009, J-PAL created a dedicated Policy Group that works to create new materials for policy outreach, disseminate knowledge about 'what works' to foundations, NGOs, international development organizations and governments, and works with these organizations to scale up programs that have been found to be effective by J-PAL research. There are many channels to inform policy effectively, and J-PAL's Policy Group focuses on a few of these methods, as described below. This does not mean that these are the only ways to influence policy, or necessarily the most effective, but they are the ones that J-PAL believes are best suited to its mission of ensuring that policy is informed by research, its organizational structure, and the countries in which it works.
2. Generating evidence to influence policy
A. Using randomized evaluations to generate rigorous evidence
Evaluations that accurately measure the impact of a program on target participants can provide policymakers with very useful evidence in deciding which programs to fund or discontinue. This is done by comparing the outcome of interest (e.g. test scores in schools) of beneficiaries who received a program (the treatment group) to another group (the comparison group) that is similar in all respects except that it did not receive the program (e.g. free textbooks for students). Measuring outcomes in this comparison group is as close as one can get to measuring how program participants would have fared without the program. Therefore, we can only trust an estimate of impact when we believe that the treatment and comparison groups were equivalent at the start of the program.
There are many methods of creating a comparable comparison group, but some methods do a better job than others. Randomized evaluations are often considered the gold standard of impact evaluations because they generate a comparison group that is statistically identical to the treatment group, by randomly assigning the targeted population into the treatment and comparison groups. Random assignment prevents possible differences between the treatment and comparison groups, which could arise if, for example, a certain type of person chose to opt into a program, or if either the treatment or comparison groups had very specific unobservable characteristics that systematically differentiated them from each other. This random assignment is the same methodology used in clinical trials to determine who gets a drug versus a placebo when testing the effectiveness of new medicines. Because members of the treatment and comparison groups do not differ systematically at the outset of the experiment in both observable and unobservable characteristics, any difference that subsequently arises between them can be attributed to the program rather than to other factors.
This randomized method of evaluation helps researchers and policymakers identify programs that work and those that do not, so that effective programs can be promoted and ineffective ones can be discontinued. In this way, evidence can help policymakers get a bigger effect for their development spending, and this in turn may lead to greater support and funding commitments for programs that have been evaluated and found to be effective.
The randomized evaluation methodology has long been used in medical trials as well as social programs in the United States, but an important feature of work by J-PAL affiliates is that it addresses key questions that are relevant for the selection and implementation of effective anti-poverty policies in the field. In many cases this means testing a program or a variation of a program which could be scaled up to large numbers of users, but it could just as well mean examining some psychological or behavioral mechanism that influences the lives of the poor. J-PAL affiliates set their own research agendas, which are often driven by issues of interest to policymakers.
In addition, J-PAL tries to promote active dialogue between researchers and policymakers to identify the issues that are considered most pressing for both. J-PAL hosts matchmaking conferences to provide opportunities for relationships to be built between researchers and implementing organizations with programs (or questions) that they need evaluated. J-PAL staff identify organizations engaged in an area of work from around the world, and talk to them about their research priorities and programs that they need evaluated. Staff spend extensive time screening out organizations that do not have specific research questions or would not be able to adopt the randomized methodology, and then invite the remaining group to a conference. Simultaneously, researchers whose work focuses on related questions are invited to attend and give presentations on their research interests and past partnerships. At the end of the conference, researchers and policymakers are able to share their priorities for new research and find common questions from which new evaluations can proceed.
Another way that J-PAL helps bring the policy perspective into research is through the creation of special initiatives in focus areas, like the adoption of agricultural technologies or ways to reduce corruption in public programs. These Initiatives are driven by funding organizations, policymakers, and J-PAL affiliates identifying key areas for future research, and inviting proposals from researchers working with field partners to propose evaluations that attempt to answer these questions. To encourage dialogue about the most pressing policy questions facing a Latin American country, J-PAL helped set up a commission of academics and policymakers. This commission met several times to present and discuss various policy questions, and collaboratively designed an evaluation to examine solutions to these problems, which the government committed to fund (both the program and the evaluation). Similarly, J-PAL affiliates have been involved with the creation of evaluation funds and units within state and national governments, to build the capacity (both intellectual and financial) of countries to evaluate their own pressing questions.
C. Partnering in the field to improve program design
The close interaction between researchers and implementing partners that takes place during the evaluation process is another key means of influencing policy. The structure of such a partnership can affect the way in which evidence can subsequently be used, because it significantly helps determine the design of the program being examined and the scale at which it is evaluated. Often people assume that program evaluations are a forensic, ex post evaluation of whether a program worked or not, but a typical J-PAL evaluation begins long before a program is implemented. Researchers and implementing organizations will have an extensive discussion of the underlying problem (e.g. unauthorized absenteeism among government healthcare workers) and various possible solutions (e.g. higher salaries or better monitoring), along with the associated theory of change (e.g. role of incentives versus penalties in changing behavior and intrinsic motivation).
As a part of this process, researchers may share the results of previous evaluations, and work with implementers to try to identify promising interventions to test. Early small-scale pilots of such interventions can further improve program design by gathering input from the field, even before the program is launched. Once a program starts, multiple variations of the treatment can help policymakers understand which components of the program work best, while qualitative surveys and diaries provide continuous feedback on why some components of the program worked better than others. In this way, evaluations build partnerships in the field that can have a profound impact on program design, help organizations question the rationale behind every step of their processes, and become better at rigorously measuring the impact of their programs.
While partnering on a randomized evaluation always involves significant ex ante interaction with the implementer, these partnerships can take many forms. It can be advantageous to partner with governments because they offer the chance to implement programs on a large scale in a sustainable way, but it may be harder to convince politicians and civil servants to implement new programs that may not be well aligned with their interests, or rigorously evaluate others that may have large potential for benefitting their favored constituencies.
Work with governments can also involve long and cumbersome approval processes and the risk of programs being 'orphaned' and discontinued when key civil servants are transferred or politicians are voted out. On the other hand, partnerships with NGOs may limit the program scale, but they can be faster and more flexible in implementing new approaches and offer very dedicated staff. Large foundations tend to focus on one particular issue, such as health or education, but they offer highly professional staff and possible funding for scale ups of successful programs. And international development agencies, though they often have a narrower geographic focus corresponding to their home country's strategic interests, can bring significant funding for the program and offer the chance to implement at a large scale.
3. Using evidence to influence policy
J-PAL's Policy Group spends a significant amount of time communicating with funders and implementers about the need for evidence-based policy, stimulating demand by explaining how evidence can be incorporated in the policymaking process, as well as the pros and cons of various kinds of evidence from different types of evaluations. In many cases this can serve as an introduction to the presentation of specific evidence from field projects, but in others it can mean providing policymakers the tools to evaluate their own programs and to become more informed consumers of evidence.
B. Making research accessible
One of the most effective ways to promote evidence-based policy is by communicating the results from field evaluations around the world. However, the results of impact evaluations are most often presented in academic working papers or journals, and these papers tend to be written in a very technical way which can limit their potential audience among policymakers. One of the primary responsibilities of the J-PAL Policy Group is to make research more accessible by extracting the most compelling results from longer papers and reports and presenting them in non-technical language. There are more than 250 ongoing and completed evaluations by J-PAL affiliates, and another 100 or more are in the process of being launched.
For all of these evaluations conducted by J-PAL researchers, the Policy Group creates an 'evaluation summary': a two-page synopsis of the relevant policy questions, the program being evaluated, and the results of the evaluation. These summaries are targeted at a non-academic audience, while still presenting all relevant information. They clearly outline testable research questions, explain the context and design of an evaluation, and present the results as clearly as possible, including the implications these research results have for identifying which policies and programs are effective. These evaluation summaries are made available on the J-PAL website, and are particularly useful for putting together reports of the current research on particular topics: for example, all J-PAL evaluations examining business training for micro-entrepreneurs or all projects looking at improving immunization rates.
Sometimes an evaluation is particularly compelling and policy-relevant, so J-PAL creates an expanded publication about the project. These 'briefcases,' around six pages in length, provide a longer summary of the project and allow outreach to a larger audience due to their print run. They expand upon the policy questions that the evaluation addresses, provide more detail on the program being evaluated and how the evaluation was designed. These longer publications also provide additional information on the context in which the program was implemented and discussion of how the results can be extrapolated to other contexts.
In addition to communicating the results of academic evaluations to a non-technical audience, the Policy Group also synthesizes the results from multiple evaluations to draw out general lessons about 'what works' in a given area of anti-poverty policy. This kind of analysis takes two main forms: cost-effectiveness analyses and longer printed publications called 'bulletins.
One way to analyze results from multiple evaluations examining the same policy question or program is to combine them in a cost-effectiveness analysis. In simple terms, a cost-effectiveness analysis calculates the ratio of the amount of 'effect' a program achieves for a given amount of cost incurred, or, conversely, the amount of cost required to achieve a given impact. This ratio, when calculated for a number of alternative programs addressing the same policy goal, conveys the relative impacts and costs of these alternative programs in an easy and intuitive way. However, relatively few studies published in academic journals include cost data on the programs they are evaluating, and what data is available is presented in a wide variety of formats that do not allow for easy comparison between programs.
J-PAL's Policy Group collects results and cost data from programs which aim to achieve one policy goal (for example, reducing diarrheal disease in children), and provides cost-effectiveness analyses as a tool to inform policymakers about these different programs. One challenge in this analysis is to strike the right balance in the trade-off between a format which is easy and intuitive to understand, and a representation of the programs which reflects all of the nuances of their implementation and results.
The results of these cost-effectiveness analyses, as well as syntheses of evaluations which cannot necessarily be quantified in cost-effectiveness analyses, are presented in longer print publications called 'bulletins.' For example, the Policy Group is currently working on a bulletin synthesizing the results from ten evaluations which examine the effects of charging a price for preventive health products and other goods that have social benefits. From the results of these evaluations, J-PAL finds that distributing products for a nominal fee significantly reduces take-up relative to free distribution, and does not appear to generate significant revenue for implementing organizations. Bulletins include information on each of the evaluations they cover, and also present general lessons that can be drawn from the collective body of evidence.
These bulletins, while longer than an evaluation summary or briefcase, are still targeted for a policy audience and written in accessible language. Because they tend to contain evaluations of programs from multiple contexts and where multiple variations have been tested, they are also able to provide more information on the sensitivity of this type of program to different contexts and assumptions, such as population density or the cost of program inputs.
D. Disseminating evidence
In addition to creating individual evaluations summaries and synthesizing multiple results for a policy audience, the Policy Group also tries to reach out to organizations and policymakers that have the potential for best affecting development programs and policies based on rigorous evidence. One of the main tools for such evidence dissemination is the J-PAL website, where all evaluation summaries, printed publications, and cost-effectiveness analyses can be found. This website, currently available in English and Spanish and shortly to be available in French, is intended to be a clearing house for the evidence generated by J-PAL researchers, and to provide the policy lessons that have been derived from that evidence base.
J-PAL also hosts outreach conferences to communicate the results of its affiliates' evaluations. These conferences, generally organized around a particular geographic region or research theme, target funders and decision makers who are in a position to use that evidence to effect policy changes. A recent example of such a conference was held in Bihar state in India, to communicate research results that could inform policy in that state. The conference consisted of presentations by researchers and the organizations with whom they had partnered to evaluate programs, and focused on issues as diverse as informational campaigns to improve voter knowledge, and de-worming schoolchildren to improve absenteeism. In this way it served two purposes: to explain pertinent results from field evaluations, and to provide examples of how researchers and policymakers can collaborate to generate useful lessons to improve future decisions.
E. Scale ups
J-PAL also works with implementing agencies like governments, NGOs, foundations, and international development organizations who may be interested in replicating or scaling up a particular program found to be effective by a randomized evaluation. In such cases, J-PAL works in close partnership with the implementing organization to understand the underlying problem and the context-specific factors like infrastructure, the severity of the problem, and the capacity of implementing agencies. If this study reveals that the program has potential to be successfully replicated in the new context, J-PAL works with the implementing organization to design a 'policy pilot' that introduces the program in a small area. The aim of the pilot is not only to measure the size of the impact of the program in this new context, but also to test the delivery mechanism for the program using the new implementing agency's resources. Such a pilot provides useful information to help policymakers decide whether or not to scale up the program, and if so then what challenges and issues they may need to consider to ensure a successful launch.
4. Conclusion
There are many ways to effectively influence policy and J-PAL does not believe that there is just one right way of doing so. J-PAL's strategy is to inform development and social policy in countries around the world by first using randomized evaluations to generate evidence on the impact of programs and policies that are most critical for development practitioners, and then effectively disseminate this evidence among policymakers. This paper provides an overview of J-PAL's policy strategy, and in a more detailed paper to be issued later we hope to illustrate our approach in greater detail with actual examples from our work.
Part 5: RAND's Promising Practices Network
Rebecca Kilburn, RAND Corporation
1. Introduction
This document summarizes RAND's Promising Practices Network on Children, Families and Communities. The RAND Corporation operates the Promising Practices Network (PPN), a project that presents evidence-based information on what works to improve outcomes for children and families in the United States. The main goal of this paper is to outline the lessons drawn from PPN in order to help inform the exploration of how to best strengthen the evidence base for UK social policy.
2. Project overview and genesis
PPN provides information via the Internet on programs and practices that credible research indicates are effective in improving outcomes for children, youth, and families. The information on the PPN website pertains to children from the prenatal period to age 18, as well as the families and communities in which they live. This site provides useful information to decision makers, practitioners, and program funders who must choose among many possibilities for improving results for children, youth, and families.
PPN was founded in 1998 by four state-level intermediary organizations: California Foundation Consortium, Colorado Foundation for Families and Children, Georgia Family Connection Partnership (formerly Georgia Academy), and Missouri's Family and Community Trust. These organizations started PPN because they recognized the value of providing better access to evidence-based information on improving outcomes for children and families, and they saw an opportunity to realize efficiencies by pooling resources to undertake this common activity. The RAND Corporation joined PPN as the operating partner in October of 2000. RAND is now responsible for all operations, fundraising and marketing. The original Governing Board for the project included RAND plus the four 'founding partners'. This has been replaced by a Board of Advisors consisting of a representative from each of the Network members.
3. Mission and distinguishing features
With the input of the PPN Board of Advisors, RAND developed the following PPN mission:
To be the premier source of evidence-based information used by those working to improve outcomes for children and families.
These are some features of PPN that distinguish it from other websites and best practices sites:
- Targets policymakers, practitioners, and other decision makers rather than only parents or researchers.
- Includes only evidence-based information screened for objectivity and scientific credibility.
- Provides information from the full range of pertinent sources, not just from the organizations sponsoring the site.
- Offers information in a form that the intended users find useful and easy to understand.
- Covers the breadth of topics related to children, not just one issue such as violence or health.
- Provides free use to anyone with access to the Internet rather than only being available to Members.
4. Audience and products
PPN's primary product is the website: www.promisingpractices.net. The website is user-friendly, focusing on providing brief, easy-to-understand information for busy non-researchers. The primary audience for the website includes policymakers, practitioners, the media, and other decision makers who may not have research training. Data also indicate that secondary audiences, such as scholars, also use the site, as well as professors in teaching courses.
The site content is organized into five main areas:
- Proven and Promising Programs. Provides brief descriptions of programs that have evidence showing they improve outcomes for children and families. RAND teams write these descriptions expressly for PPN. Hundreds of programs have been reviewed in detail for inclusion on the site, and we continue to review programs on an ongoing basis.
- Research in Brief. Provides links to hundreds of easy-to-understand summaries of reliable research information from a broad range of sources such as the National Center for Education Statistics, Child Trends, Office of Juvenile Justice and Delinquency Prevention, Urban Institute, the journal Pediatrics, and dozens of other sources. Information in this section is also organized by outcome area and is continually updated.
- Resources and Tools. This section includes links to databases, fact sheets, screening tools, seminal reports, and a variety of other resources that are among the best research-based materials available on children and families. The PPN team reviews material for this new section on a continuous basis, and we welcome recommendations for additional resources.
- Expert Perspectives. Leading scientists provide insights into pressing policy issues through via multimedia, such as videos or archived Webinars, videos, answers to users' questions from our Ask the Expert feature, or other products.
- Partner Pages. Links to information on PPN and other resources that are related to 'hot topics' in child and family policy in that state or organization. This feature is only available to organizations that are members of the Network. The member organizations select the hot topics, which are updated on a monthly basis.
The content in the major sections is organized in several ways to facilitate users finding what they need. Much of the information is organized by broad Outcome Areas (Healthy and Safe Children, Children Ready for School, Children Succeeding in School, and Strong Families), 13 Indicators (such as Children not using tobacco, alcohol or illegal drugs, and Students graduating from high school), and four types of Topic categories (Age of Child, Type of Outcome Improved, Type of Setting, and Type of Service).
5. Review criteria
Another product associated with the website is the monthly e-mail newsletter, which is sent to individuals who sign up. The newsletter announces content that has been added to the site since the last newsletter.
All content posted on the site must meet pre-established evidence criteria, and two PPN team members and at least one outside reviewer (can be a RAND or non-RAND researcher) must agree that the research meets these criteria. All information listed on the site must meet the 'Promising' designation at a minimum, and programs listed in the Programs that Work section can be designated either 'Promising' or 'Proven', where the latter designation is reserved for programs with evaluations meeting the highest standards of rigor. The primary review criteria are displayed in Table 2. Note that the complete review criteria include about 20 additional issues that reviewers consider, such as whether the evaluator was also the program developer and the rate of attrition in longitudinal studies.
6. Use and marketing
PPN engages in the following ongoing marketing activities:
- Issuing a monthly e-mail newsletter, which reminds subscribers to visit the site, and usually results in new users as the newsletter gets forwarded to others.
- Distributing hard-copy materials at conferences around the US and ad hoc direct mailings.
- Face-to-face meetings.
- Optimizing the site so as to maximize hits from search engines (e.g. Google, Yahoo).
| Type of Information | Table 2: Primary PPN Evaluation Criteria | | | Proven Program | Promising Program | Not Listed on Site |
| :---------- | :--------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
This is the second of two publications.
Introduction to this summary
This summary has been compiled by the National Audit Office (NAO) drawing on content from three publications which provide overviews of evidence-based policymaking in different contexts:
- Evidence for Social Policy and Practice: Perspectives on how research and evidence can influence decision making in public services (2011) – NESTA.
- A Simple Guide to Supported Self-Management: For Commissioners and Service Providers (2010) – The Health Foundation.
- The Obama Administration's evidence-based social policy initiatives: An overview (2011) – Ron Haskins, Advisory Board and Jon Baron, President, Coalition for Evidence-Based Policy.
These publications contain numerous footnotes which have been retained in full for this summary. Unless otherwise stated, any content or analysis presented here is from the original publications.
Executive summary
Policymakers face many difficult decisions about how to make the best use of increasingly limited resources to solve major social challenges. The idea that policy and practice should be underpinned by rigorous evidence is internationally accepted. However, there is recognition that the level of rigour in evaluating 'what works' in social policy remains limited.
Evidence-based policymaking involves using high quality research and evaluation to ensure that public policies and programmes are effective in achieving their intended outcomes. This can mean:
- Identifying what works: Systematically reviewing available evidence to determine which interventions are most effective in achieving desired outcomes. This includes using statistically significant data and, ideally, randomised controlled trials for robust evidence.
- Promoting policy-relevant evaluations: Designing evaluations that address key questions relevant to policy selection and implementation, ensuring research is practical and useful for decision-makers.
- Partnering in the field to improve programme design: Collaborating with implementing organisations to integrate research findings into programme design and refine interventions based on continuous feedback.
- Disseminating evidence: Making research findings accessible to policymakers, practitioners, and the public in non-technical language to inform decision-making.
- Scaling up effective programmes: Working with implementing agencies to replicate and expand successful programmes, carefully testing delivery mechanisms and impacts in new contexts.
Various organisations have developed models to promote evidence-based policymaking. For example:
- Project Oracle (NESTA): Seeks to establish a standard for evidence-informed decision-making on children and young people's policy in London. It combines a programme evaluation methodology, a technical platform, and a centralised officer resource. Key factors include understanding what programmes work, building a sustainable body of evidence, using consistent methods for evidence growth, and understanding the root causes of social problems.
- The Center for Court Innovation: A public/private partnership in New York dedicated to reforming the justice system through demonstration projects, original research, training, and technical assistance. Its business model involves demonstration (field-testing new ideas), documentation (tracking impacts), replication (helping officials adopt lessons), and institutionalisation (embedding new practices).
- Washington State Institute for Public Policy (WSIPP): Created by the Legislature to carry out practical, non-partisan research on issues of State importance. Its ‘4-Step’ approach involves identifying what works, assessing economic sense through cost-benefit analysis, evaluating impacts on State-wide outcomes, and assessing risk through Monte Carlo simulations. The WSIPP model is being transferred to the UK to inform investment decisions in child well-being.
- Abdul Latif Jameel Poverty Action Lab (J-PAL): A network of affiliated professors using randomized evaluations to answer questions critical to poverty alleviation. J-PAL works to reduce poverty by ensuring policy is based on scientific evidence, building capacity for policymakers, and disseminating research results.
- RAND's Promising Practices Network (PPN): Provides evidence-based information on what works to improve outcomes for children and families in the United States via its website www.promisingpractices.net. It targets policymakers and practitioners, screens information for objectivity, and covers a breadth of topics.
- The Obama Administration’s initiatives: Sought to strengthen the evidence base for US social policy by bringing evidence inside the decision-making process. Key components included selecting social problems, identifying evidence-based model programs, obtaining funds to scale up such programs, making funds available for replication and development of new models, and continuously evaluating projects. These initiatives cover areas like home visiting, teen pregnancy prevention, and the Social Innovation Fund.
Across these models, common themes emerge: the importance of independent evaluation, consistent methodologies, stakeholder engagement, and making evidence accessible and actionable for practitioners and policymakers. The ultimate goal is to improve social outcomes by ensuring that public resources are directed towards programmes that are proven to be effective.
Endnotes
