| Header 1 | Header 2 # Financial Reporting Council

This is not a heading but a bold paragraph, per the new instruction.

3.17 Rapid cycle testing

USP: Iterative experiments useful for local problem-solving, that create a rapid feedback loop between testing, re-design and re-testing.

Rapid cycle tests are a family of experiments being pioneered by early adopters in the US, Sweden, Canada and the UK. They draw on diverse sources of inspiration – like improvement studies and implementation science in health, and 'lean startup' and 'lean impact' approaches in the emerging social research and development (social R&D) sector.¹⁶² Rapid cycle tests take a more flexible approach to evaluation, building on basic pre-post designs to create an iterative approach. They take multiple measurements over a longer time period, as innovations are re-designed and re-tested, and focus on solving the problems that face decision-makers locally.

Rapid cycle tests are conducted in lots of ways, but most are underpinned by 'plan-do-study-act' (PDSA), a template for involving practitioners, leaders and researchers in developing and testing innovations that was developed in healthcare improvement. Rapid cycle tests build on this practitioner-led model to create series of small experiments. Figure 22 shows this iterative approach, in which multiple small trials are conducted, and results are built upon over time. Other ways of doing rapid testing are usually a twist on this design: Dartington Service Design Lab, which is leading the way on doing rapid testing in new policy areas in the UK, uses a five-step process that aims to combine new evidence with user-centred design.¹⁶³ Rapid tests emphasise continuous learning and often require the whole organisation to plan, collect and analyse data. Some approaches rely on frontline staff testing things out and have little or no input from expert evaluators. Others draw on the expertise of both practitioners and researchers, and use more advanced measurement techniques.

Rapid cycle testing obviously lacks the rigour that more formal experimentation methods gain by using a comparison or control group, but this may not matter very much depending on the context. If you are running an online platform for example, you control the entire user journey and, if it is not particularly susceptible to outside events, then any change you observe post-treatment can reasonably be attributed to the change that you made. This is particularly true given the rapid nature of the experiments: there has been little time for anything in the outside world to change.

Some of the most developed examples of rapid testing can be found in the US. At Harvard University, the Centre on the Developing Child is testing out ideas highlighted by America's National Scientific Council on the Developing Child, a multi-disciplinary collaboration that works to promote the science of early childhood development.¹⁶⁴ Through a project called Frontiers of Innovation, researchers and practitioners work together to turn ideas from research into interventions for children and families. One example of this is Learning Through Play, a programme developed with researchers at the University of California that draws on science about how children develop 'executive function' - the cognitive skills that help them pursue tasks and goals – in their early years.¹⁶⁵ The researchers created a programme that supports adults and children to play games together to develop these skills. Rapid cycle testing will often start with a small feasibility test with about ten families, then be iterated with 20 or 30, and then, if an innovation shows promise, tested with a bigger sample and a control or comparison group. In this context, 'promise' does not usually mean a statistically significant improvement (the experiment is too small for that): any improvement after the intervention would be considered evidence of promise, and grounds for further testing.

Other organisations are using rapid tests to innovate in public services. Nesta's People Powered Results team runs 100 Day Challenges that search for new ways improve health and care services by bringing together multi-disciplinary teams.¹⁶⁶ Like others, they look at how changes to the practice of service delivery can create a more effective system. The US Centre for Employment Opportunities, which supports people who have been in prison, is using rapid tests to explore having staff with experience of the criminal justice system deliver their support programme. Based on evidence that suggests that people with lived experience might be better at building relationships, they are providing special training to these staff mentors, and running rapid cycle tests to see if the innovation can improve how engaged participants are, or reduce drop-outs from the programme.¹⁶⁷

Many applications of rapid testing (including the one above) focus on adapting projects to make them more appropriate for a specific group of people or in a particular context.¹⁶⁸ The Family Nurse Partnership (Box 17), a home-visiting programme providing support for first-time young mothers, is exploring how this might be done at a national scale in the UK, adapting a big evidence-based programme originally created in the US.¹⁶⁹

Rapid cycle tests require good quality data which is easily and quickly accessible to monitor the effect that changes have. Being able to test innovations this way requires having good 'baseline' measurements – that is, a good knowledge of the service, system and participants before changes are made, as well as during and after. They also require training for and commitment from staff across the organisation, to monitor and test innovations.¹⁷⁵ There may be certain kinds of social policy intervention that are more amenable to rapid testing; NSPCC, for example, is using rapid tests to evaluate place-based approaches, like its Together for Childhood initiative.¹⁷⁶ Rapid tests may be a particularly good fit for projects like these, which foreground local, community-centred knowledge creation.

At the moment, we don't have strong evidence to show that rapid cycle testing is consistently effective at improving the impact of innovations in the long-term. One challenge is that rapid testing can look very different across projects and sectors, so it's hard to work out what might make them effective in general.¹⁷⁷ A second limitation is that, in some cases, we don't know what the unintended consequences of making changes to an existing programme might be. This is a question that researchers and practitioners in public health have been grappling with, and they are creating new frameworks, guidelines and tools to help decision-makers make safe and successful changes to existing projects and policies.¹⁷⁸

There is potential to combine rapid cycle tests with 'nimble RCTs' or 'hybrid designs' - randomised experiments that we looked at in Sections 3.3 and 3.10 respectively. The use of these designs would give organisations more certainty about the effects of rapid innovations and the impacts they have for the people they work for, while still maintaining the pace of innovation. The result is similar in spirit to the A/B test (see Section 3.4) which is standard practice in industries where innovation and optimisation is routine, such as web design.

3.18 Prototyping

USP: An approach to trying out ideas in practice before an innovation is offered to recipients, to get feedback and input from stakeholders, improve the idea and increase its chance of success.

Prototyping is focused on front-loading the risk of failure: by trying out ideas as early as possible, it aims to make small failures happen at the beginning stages of a project, so that the idea which is more formally trialled has a better chance of success. It can also be 'human-centred', focused on user and stakeholder engagement, a way to gather feedback, engage partners and communities, and create buy-in for a new idea. If an idea proves to be unworkable, this is found out before significant resources have been committed; if it needs to be improved then this is discovered at a point where a meaningful decision can be made as to whether to invest in improving it or whether it is better to try something else.

In social science terms, prototypes are not experiments because they don't involve structured testing and evaluation. Nevertheless, prototyping has an important role in helping organisations to think and act more experimentally. They are part of the puzzle of how to embed experimental mindsets, as well as methods. Nesta's former CEO Geoff Mulgan has argued that there is a craft of developing ideas experimentally: too much too soon can kill a good idea, while too little too late risks social irresponsibility.¹⁷⁹ Prototypes encourage us to reframe potential solutions as hypotheses and approach ideas with a testing mindset. Nesta's publications Designing for Public Services and Prototyping Framework aim to make prototyping practical and accessible for government practitioners.¹⁸⁰

A prototype follows the 'double diamond': an iterative process that moves from a version with little detail or functionality (like a rough draft that illustrates the idea) to a version with much more detail and functionality (giving test users a better sense of how it is intended).¹⁸¹ What's learnt through early iterations helps create a more refined solution that's eventually tried out in practice.¹⁸²

Footnotes

Figure 23: The Design Double Diamond

'Exploratory prototypes' investigate how a solution might look and feel through low-fidelity, low-resource models. A product can be created in cardboard form, a website through a hand-drawn wireframe, a service through role play. ¹⁸³ These mock-ups can be shared with colleagues, stakeholders and users to elicit feedback. ¹⁸⁴ The ideas for exploratory prototypes like these are usually based on pared-down ethnographic methods, where a designer or team observes or interviews service users, or alternatively a workshop or focus group.

'Live prototyping' covers later rounds of iteration, when a product or service is trialled at a small scale with users. Live prototypes aim to make operational errors and design flaws obvious and easier to avoid. ¹⁸⁵ Unlike a full pilot, a prototype is often a simulation: the new service or website will not be fully up and running and may not be implemented in its normal setting (though it is preferable if it is). Online prototypes are often fairly straightforward to implement and evaluate; the UK Government Digital Service, for example, has a downloadable Prototype Kit which builds new versions of online government services. ¹⁸⁶ Online prototypes can be evaluated with A/B tests, an online randomised trial that we covered in Section 3.4. Other types of service innovations can be trickier to develop and evaluate. In Chile, Laboratorio de Gobierno's 'We are Community' project provides a good example of how an innovation lab used a quasi-experimental design (see Sections 3.12-3.15) to robustly evaluate a series of prototyped innovations that aimed to help citizens in Santiago feel safer in their neighbourhoods.

Box 19: iSomos Comunidad! Making Santiago safer for citizens

Somos Comunidad (We are Community) aimed to find new solutions to citizens' perception of insecurity and risk in Santiago. Chile's Laboratorio de Gobierno (LabGob) took a robust approach to evaluating prototypes that were developed collaboratively with the public.

Research showed that Santiago's citizens had a much higher perception of the likelihood of crime happening than was apparent in the official statistics. To explore the problem, LabGob ran a consultation involving more than 300 people in workshops including police and local government, academics, community leaders and citizens. Several different solutions were proposed and went through iterative prototyping by teams supported by the government labs' practitioners. One solution was online: an app to report crime or fear of crime, developed with the police. Another was social: the creation of community 'Task Forces' who worked with local mayors. These Task Forces explored municipal solutions to anti-social behaviour and involved community police, security experts, and local community members, and tried different ideas out in six local areas. Each Task Force also trained citizens in how to use the online reporting app safely.

While each community was given autonomy to develop its own solutions, the programme was evaluated with the help of the Inter-American Development Bank, using a quasi-experimental method that compares communities to an untreated comparison group. At the time of writing, results were not yet in but Beatriz Hasbún, LabGob's Learning Experience Designer, is open about the decision to evaluate: "This is a large-scale project and we wanted to work with an independent partner. There are so many different factors that affect people's perception of security – we felt it was crucial to use a comparison group." If LabGob sees an impact on outcomes in the evaluation, it will support more neighbourhoods to test the programme. If not, solutions will be improved or discontinued.

Some ways of doing prototyping emphasise the 'quick and dirty' over a need for good research design. Prototypes can and should draw on existing research to understand the problem area they are trying to address, like exploring what's been done before, to see which ideas are most promising. Ad hoc or one-off field visits can only tell us so much because what's seen in one place, on one day, is unlikely to be representative. And while doing prototyping in workshops or user feedback sessions might provide a useful sounding board, it's often unclear how groups are managed or who gets invited. It's likely that different kinds of bias might be at work, like 'groupthink', which can lead to poor group decision-making. ¹⁸⁷

When it comes to live prototyping, there is still relatively little published literature on how these tests work in practice, and the process is likely to vary significantly from project to project. One question is whether prototypes are being evaluated in a way that would really allow them to fail, or whether they could fall into the trap of 'single loop learning', where preferred options are given the best chance of success while other ideas are not given a similar chance to show promise. ¹⁸⁸ A possible solution to this could be to do more 'parallel prototyping' where multiple solutions are developed and tested by multiple designers, and to share learning among organisations about what great prototyping looks like. ¹⁸⁹ US designer Bruce Hanington, co-author of the book Universal Methods of Design, points out that design should aspire to research excellence in order to bring 'relevance' together with 'rigour'. ¹⁹⁰

Conclusion

Not every experiment is right for your needs and strengths. For instance, running quasi-experimental designs (Section 3.12-3.15) requires complex statistical know-how, whereas there are simple online experimental packages that anybody can use to run A/B tests on the digital delivery of services. Crucially, what's right for you will depend on what you want to learn about. The important principle is 'horses for courses': choosing the kind of experiment that is suitable for your challenge and environment. To help you here, Table 1, summarises all 18 experiments in our inventory.

When making your choice, it is worth thinking about how far you are on your journey of developing an idea. Some early ideas may just not be ready for full-blown RCTs or QEDs. Prototyping may be more what you are looking for (Section 3.18) and emphasises learning with stakeholders about how an idea can be made a good fit for a community or group of users. If you want useful insight quickly, nimble RCTs offer pragmatic, operational learning and formative (improvement-focused) evaluation. Nesta runs 100 Day Challenges in healthcare to focus minds, and focus results. When problem-solving requires pooling the expertise of teams more effectively, rapid cycle testing can combine a collaborative model with a degree of evaluative rigour (see Section 3.17). These approaches can still be grounded in the best available science - and we found great examples of this work from organisations like the Dartington Service Design Lab in the UK and the Harvard Centre on the Developing Child in the US. Prototyping can also be combined with impact evaluation designs, as we saw in the example from Chile's Laboratorio de Gobierno (see Box 19).

Figure 24: Nesta's Innovation Spiral

However, if you have significant funding, a developed policy intervention, and are about to embark on a national roll-out, a more robust experiment like a randomised trial will give you the evidence you need to make crucial investment decisions. There are many designs out there, suited to a wide range of contexts, sectors and innovations (covered in Sections 3.1-3.11). If a policy or programme is already underway, then you may have to evaluate things retrospectively (not prospectively, as with RCTs or pre-experiments). If this is the case, non-randomised and quasi-experimental designs (QEDs) will help – although it's worth keeping in mind that they rest on assumptions that may or may not be testable.

The issue may not, however, be time, or how far developed your innovation is. You might want to ask different sorts of questions of your innovation: instead of 'did this work', the question could be 'how and where will this work in practice?'. If that is the case, then hybrid trials (Section 3.10) or realist trials (Section 3.9) – drawing on qualitative research and openness to theory - are your best bet. Multi-site trials too (Section 3.8) point to how we might grow the evidence base by testing portable theory. It's also worth keeping in mind different approaches to experimental design, particularly the stepped wedge and wait-list designs (Section 3.6), which allow you the option of ensuring that all possible recipients can benefit in the end. When randomisation isn't possible, there are valuable alternatives: differences-in-differences, propensity score matching, and regression discontinuity (Sections 3.12-3.15).

Whatever experiment you choose, we hope that you avoid the common myths and misunderstandings - like those about ethics, costs and speed - and do not use these as reasons not to experiment and improve. We have shown that there is a diversity of trials, and that some design innovations have helped to answer critics (see our history of experimentation in Section 1 along with Annex B, which covers both criticisms and responses).

No approach is perfect, and no design offers easy answers. Many innovations fail to make a difference; making better policy takes hard work and investment. According to Jon Baron, Vice President of Evidence Based Policy at the Arnold Foundation, and a long-standing advocate of randomised experiments, most projects evaluated by an RCT show little impact. In education, 90 per cent of interventions evaluated in RCTs by the US Institute for Education Sciences had weak or no positive effects. ¹⁹¹

This lack of progress should not, however, depress us. It is a reminder of how difficult it is to make a difference (and how unwise it is to assume that just because an innovation is new and sounds attractive, that it will improve outcomes in practice). It has been dubbed Rossi's Iron Law of evaluation, after the American sociologist Peter Rossi, who declared that:

"The better designed the impact assessment of a social program, the more likely is the resulting estimate of net impact to be zero. "¹⁹²

This law means that the more technically rigorous we are in how we evaluate, the more likely are its results to show no effect. As former Australian Government minister and advocate for RCTs, Andrew Leigh has argued:

"Rossi's Law doesn't mean we should give up hope in changing the world for the better. But we ought to be sceptical of anyone peddling panaceas. The belief that some social programs are flawed should lead to more rigorous evaluation and patient sifting through the evidence until we find a program that works." ¹⁹³

To do this, we need to draw on many different tools and drive innovation in how we learn about making change. 'Sifting through the evidence' is not solitary work. Organisations and teams must take an open approach, that allows us to strengthen collective expertise, and identify important lessons more effectively. We must also work across sectors, to link up what we learn above and beyond our own areas of specialism – including with the wealth of knowledge housed in universities and research organisations. ¹⁹⁴ There are many areas of policy where experiments are likely to be critical to progress. Fields like renewable energy and Green Deals may be ideal spaces for learning experimentally about how to improve efficiency in the energy sector, or what will work to change behaviour and help businesses take up low carbon practices. Similarly, devolution and regional policy changes, as well as future international collaborations, may provide more opportunities to learn about what works, when, and for whom.

As innovators seeking social good, we know how hard it can be to move the dial towards sustained success. Yet with humility and hard work, we have a duty put our ideas to the test to stop doing what doesn't make a difference, find out what does, and understand how we can improve people's lives.

Section 4. Useful resources

There are now many resources available to support experimenters. If you are reading this in the UK, a dedicated Trials Advice Panel sits at the Cabinet Office to provide support and advice to government, and for others such as in NGOs, the Test+Build platform, run by the Behavioural Insights Team, provides support on online experimenting.

To encourage the adoption of technology and modern management practices in small and medium-sized enterprises (SMEs), the UK Department for Business, Energy and Industrial Strategy is trialling a new approach to funding experiments through its Business Basics Fund, supported by Innovate UK and Nesta's Innovation Growth Lab. We now have a network of 12 What Works Centres, many of whom curate and communicate the results of experiments for frontline professionals and decision-makers. ¹⁹⁵ This trend is not unique to Britain.

Below is a list of some of our favourite, freely accessible resources:

• Bit by Bit: Social Research in the Digital Age (2017) by Matthew Salganik. See chapter 4, Running Experiments. A digital copy is available free online at bitbybitbook.com

• Goldilocks Deep Dive: Introduction to rapid-fire operational testing for social programs (2016) from Innovations for Poverty Action (IPA). A short guide on running Nimble trials.

• Designing for Public Services (2017) from Nesta and IDEO covers prototyping and other design-based approaches to innovating in services.

• Impact Evaluation in Practice (2011) from the World Bank. Clear and accessible textbook on doing randomised and quasi-experimental evaluation, with examples.

• Developing and Evaluating Complex Interventions (2006) from Medical Research Council. An excellent source of guidance and advice on evaluating complex projects, with an update due for publication in 2019/2020.

• Impact evaluation methods: What are they and what assumptions must hold for each to be valid? (2015). A summary factsheet from J-PAL North America.

• Evaluation: What to consider (2015) from The Health Foundation covers rapid evaluation, along with approaches to evaluating improvement in health. More insight and blogs can be found on the webpages of its Improvement Analytics Unit.

• Mastering Metrics: The paths from cause to effect (2014) by Joshua Angrist & Jörn-Steffen Pischke. Excellent guidance on quasi-experimental designs.

• Experimentation Toolkit (2018) from Innovation Growth Lab (IGL). This practical toolkit is designed to build an understanding of how adopting an experimental approach can be used to make policies more effective.

• Measuring Impact by Design: A guide to methods for impact measurement (2019) from Impact and Innovation Unit, Government of Canada. A new reference guide for those involved in the design, delivery, procurement or appraisal of impact measurement strategies in Canada and elsewhere.

• Getting to Moonshot: Inspiring R&D practices in Canada's social impact sector (2016) from SiG Canada covers promising examples of social R&D practice.

• Prototyping Framework (2013). The prototyping process outlined in this toolkit was developed by Nesta and thinkpublic.

• GOV.UK Prototype Kit. This kit rapidly creates HTML prototypes of GOV.UK services.

• Test+Build and Predictiv platforms from the Behavioural Insights Team (BIT) allow you to build experiments online, supported by behavioural science.

• Rapid Cycle Design and Testing: The what and the why (forthcoming, 2020) from Lowther, K., Simpson, D., Green, F., Morpeth, L,, Axford, N., and Hobbs, T. at the Dartington Service Design Lab. Some blogs and resources on rapid cycle testing are available on Dartington Service Design Lab's website.

• Testing Social Policy Innovation (2014) from the European Commission. This guide covers both randomised and quasi-experimental designs and can be found on the Alliance for Useful Evidence website.

• Running Randomised Controlled Trials in Innovation, Entrepreneurship and Growth: An introductory guide (2016) from Innovation Growth Lab. IGL's step-by-step guide on running randomised trials.

• Test, Learn, Adapt: Developing public policy with randomised controlled trials (2014) from Haynes, Service, Goldacre & Torgerson. A classic guide on RCTs from the Behavioural Insights Team in collaboration with Ben Goldacre, author of Bad Science, and David Torgerson, Director of the University of York Trials Unit.

• Randomised controlled trials: Gold standard or fool's gold? The role of experimental methods in voluntary sector impact assessment (2016) from NCVO/Charities Evaluation Service.

• The Magenta Book: Guidance for Evaluation (2011) from HM Treasury. UK Government guidance on what to consider when designing an evaluation.

• Quality & Improvement: Theory & Practice in Healthcare (2008) from NHS Institute for Innovation and Improvement & Manchester Business School summarises some of the improvement approaches discussed. More resources can be found at NHS Improvement online.

• The London School of Hygiene and Tropical Medicine's Centre for Evaluation provides resources on methodologies for evaluation in public health.

• Quasi-Experimental Designs and Methods (2014) from UNICEF. A brief for the charity sector on QEDs.

• Using Randomised Controlled Trials in Education (2017) from SAGE. A quality resource from education specialists at Queen's University Belfast.

Annex A: Experimental jargon buster

Bias: In simple terms, bias is when ideas or evidence about something are skewed or unfairly weighted in one direction or another. In experiments, bias is a general term for any phenomenon which causes us to mis-estimate the effect of an intervention in a systematic way (for an example, see selection bias below). Bias can be present in experiments if they are not well designed and will affect the strength of findings; it may cause researchers to over- or underestimate the effects of an intervention, for example. Bias is usually something to be avoided, but in some situations, it may be better to accept some bias if the alternative is to have a very imprecise estimate (known as a bias-variance trade-off).

Causal inference: When we say 'intervention X causes effect Y' we are making a causal statement (literally, a statement referring to what causes what). With a sufficiently robust experiment (such as an RCT or regression discontinuity), we can make such statements with confidence if the results of the experiment support them. With weaker or more exploratory designs, such as the pre-post design, we cannot say that any change is caused by the intervention, so those designs are not considered to be causal. Sometimes, if a trial would be causal but some unexpected event introduces a significant bias or other threat to the validity of the trial, the trial is said to have 'lost causality', meaning that you can no longer make a causal statement about the effect of the intervention.

Confounder: Trials generally try to 'isolate' the effect of the treatment as much as possible. A confounder is anything that could have an effect on the outcome which we would mistakenly attribute to the treatment. This usually depends on the design as well as the phenomenon. For example, if there were a positive time trend in the outcome and we were using a pre-post design, then we could mistakenly attribute the increase to the intervention and the time trend is a confounder. If we were using an RCT or a DiD design, then the time trend would not be a confounder, since it also applies to the comparison group (it 'cancels out' and we do not attribute it to the intervention).

Construct validity: This is the extent to which the theoretical concepts that form the basis of an experiment are accurate - and so whether or not an experiment is really testing the hypothesis it aims to. This is tricky because experiments sometimes test abstract constructs, with contested meaning. In a quasi-experimental evaluation a few years ago on the UK social programme Troubled Families, evaluators faced a problem when they realised that although the programme claimed to be for 'troubled families', it had actually been piloted with poorer families. These are not the same thing, so it became very hard to test the programme's theory – and also likely that the pilot targeted the wrong people. ¹⁹⁶ (It's worth noting though that a new evaluation from the Ministry of Housing, Communities and Local Government has addressed some of these limitations. ¹⁹⁷)

Contamination: Sometimes called spillover or spillover effect, contamination is when the treatment has an effect on 'units' outside the intervention group. These effects can be either positive or negative. If the intervention is some form of advice, then individuals who receive it may tell their friends outside the intervention group, affecting their outcomes. Alternatively, if the intervention uses messaging along the lines of 'you have been specially selected for...' then participants outside the intervention group may think they haven't been selected and aren't eligible, which reduces their chances of doing whatever you wanted them to do. Contamination is a source of bias and is to be avoided if possible, unless doing so would make the experiment prohibitively large.

Controlled experiment: These are experiments that allocate some people to receive an intervention and others to a control group. In a randomised experiment, people are allocated to different groups by chance. In quasi-experimental designs, a comparison group will be created using statistical methods.

Counterfactual: The literal definition of counterfactual is 'something that didn't happen'. In the context of an experiment, the word is usually used to refer to 'what would have happened to the treatment group if we hadn't treated them'. This is never directly observed (that's why it's called a counterfactual), and the quality of an experiment can be judged by how well the control or comparison group approximates the counterfactual.

Experimentation vs innovation: Experimentation and innovation are sometimes used interchangeably, but they are not the same thing. Innovation is defined in the Cambridge English Dictionary as a new idea or method, or the use of new ideas and methods in the development of products, designs or ideas. ¹⁹⁸ Social innovation applies this process to social problems. The European Commission describes social innovations as 'new ideas that meet social needs, create social relationships and form new collaborations. These innovations can be products, services or models addressing unmet needs more effectively.' ¹⁹⁹ Experimentation, on the other hand, is an approach or set of tools that innovators might use.

External validity: Sometimes called 'transportability' or 'generalisability', this means the extent to which the results of an experiment can be applied more generally, in other places and times than those in the experiment. ²⁰⁰ This depends on several things, like the kind of question the experiment addressed, the sample size and composition (in particular how representative the sample was of the wider population), and the generalisability of the context in which the experiment took place, i.e. if the experiment took place in a very unusual setting, it's less likely the results will be relevant in other settings. ²⁰¹ It also depends on how much we know about the innovation in question, and how it aims to create change: do we understand the mechanisms? Did we only learn what works, or also why? It's worth keeping in mind that theory is portable.

Fidelity: This word is commonly used to mean faithfulness. In an experimental context, it describes the extent to which an intervention is implemented as its designers intended.

Hypothesis: A statement that is to be tested. If posed as a question, it is a research question (for example, 'what is the effect of taking an aspirin each day on a patient's stroke risk?'). Research questions are used in all kinds of research and will guide the research design, methodology, data collection method, and analysis approach used. In statistical testing, the null hypothesis states that an intervention has no effect (for example, 'taking an aspirin each day has no effect on a patient's chance of having a stroke') and the alternative hypothesis states that the treatment does have an effect. The alternative hypothesis may or may not specify more about the nature of the effect. For example, 'an aspirin each day reduces the stroke risk'; 'an aspirin each day changes the stroke risk' or 'an aspirin each day reduces the stroke risk by 20 per cent' are all valid alternative hypotheses. An experiment will either provide enough evidence to reject the null hypothesis in favour of the alternative, or it will provide insufficient evidence (the null hypothesis can never be 'proven', similar to how a court of law will not find a defendant 'innocent', only 'not guilty').

Heterogeneity (of treatment effects): This is about how an intervention might affect people differently: treatment effect heterogeneity is the extent to which a treatment has differential causal effects on different people or units of the experiment. These are important because different units might respond differently to different programmes or solutions. For example, households with high levels of energy consumption might respond more to an experiment testing the effect of an electricity-saving intervention than households whose energy consumption is already low. ²⁰²

Internal validity: Internal validity is about whether the experimental process was performed correctly, in a way that makes the test administered valid within the context of the experiment. Good internal validity allows us to say with confidence that X caused Y, because we have eliminated A, B and C – and any other factors that might be messing with our test within the structure of our experiment. This is one of the strengths of lab experiments: they enable precise and clear tests by controlling the environment of the experiment. Internal validity is why it's important to get the implementation of experiments on the ground right.

Natural experiments: These are events which replicate experimental conditions, but which happened naturally (or if they are man-made, weren't intended as experiments when they were implemented). For example, when a new policy was rolled out across the country it may have been implemented on a rolling region-by-region basis, with the next region being chosen randomly each time, because it was logistically impossible to roll out to everywhere at the same time. The roll-out happened in this manner for purely logistical reasons, but because it happens to be identical to the stepped wedge experimental design, we can treat it as though it were an experiment and use the resulting data to evaluate the policy. Natural experiments generally arise when some external factor causes the innovation to be implemented in an apparently random way.

Selection bias: A phenomenon where trials can produce misleading results because of how the experimental groups were sampled or observed. For example, if we are measuring the effect of a voluntary programme that citizens can enrol themselves in, we might consider comparing the later outcomes of those who enrol themselves versus those who don't. This introduces selection bias, because people who proactively sign up for programmes tend to be more motivated and may be different in other ways, and this has knock-on effects on their final outcomes. Even if the intervention didn't do anything, you would probably conclude that it did because the intervention and comparison groups were different.

Unit (also experimental unit): This usually refers to whoever or whatever we measure outcomes for. Not to be confused with a cluster. If, for example, we have a randomised trial where each school receives the same intervention, but the outcome of interest is pupil performance, then the pupils are the units and the schools are the clusters. If the outcome of interest was a school's Ofsted rating then the school would be the unit, since the outcome relates to the whole school rather than an individual pupil. This usage of the word unit is not universal, and some sources will use the word unit to mean what would normally be called a cluster.

(Un)observable: As the name suggests, a quantity or action is observable if it can be observed in a quantitative way (such as exam results, an individual's age, whether someone changes their energy provider and so on). Something is unobservable in quantitative research if it is an intrinsic trait, such as an individual's motivation or laziness. You may be able to make an imperfect measure of these traits with a survey, but there will always be aspects which are not captured with a numeric measurement. Unobservable quantities are a common source of bias in experimentation and the more robust experimental designs, such as the RCT, are able to reduce the negative consequences of unobserved variables.

Limited generalisability: It can be hard to generalise from a single experiment to the wider world. The unique context of one time and place means that it can be difficult to learn policy lessons to roll out nationally. The best way to respond to this is to replicate any trial in multiple locations. Replication is an important part of experimental science and social science. It's also possible to test interventions in multiple places to grow the evidence base (see multi-site trials, Section 3.8), and to use more mixed-methods research (see realist trials, Section 3.9). Local context will always be a critical factor in learning from experiments. But it's also an issue for any form of evaluation or any roll-out of a policy.

In a recent piece online, Eva Vivalt has pointed out, "It's not that RCTs have uniquely bad external validity – it's just that generalizing from any study to guess implications for different programs elsewhere on the planet is hard work." ²⁰³

We cannot practice blinding in research trials involving social policy: In a drug trial, patients, doctors and researchers can be made unaware of what treatment patients are receiving (a double-blind placebo trial). In research in other areas such as education, welfare and crime reduction - this is rarely possible since the intervention is clearly visible to all concerned. We are, however, often able to blind those involved in the measurement of outcomes. The lack of blinding is a problem for social policy research trials due to the propensity for intervention group participants to 'try harder' (known as the Hawthorne Effect). However, other evaluation methods will rarely improve on this since they will also not operate blinding.

An RCT doesn't tell us how something worked - and for whom, and under what circumstances: In general, a trial does not tell us how the intervention works. True. It tells us whether an intervention has worked. This is a good argument for qualitative work happening in parallel with any trial that also seeks to clarify how the intervention is causing any measured effect. Some types of trials can also help us 'look under the hood' to see why something has worked, such as realist trials (see Section 3.9), or 'how' something works using hybrid trials (Section 3.10), as well as multi-site trials, that tell us more about 'who' (Section 3.8). In some areas, like economics, trials may be more deeply rooted in theory, and explore the mechanisms driving results.

The complex interplay of systems and structures within social policy means that large-scale policy changes are difficult to trial with an RCT: The introduction of a new social policy or programme may involve a whole raft of different interventions, laws, regulations, funding, and changes to practice. Trialling the whole policy may be unfeasible. However, it could be possible to test some specific aspects of the policy, such as a change to welfare programme, or a new tax incentive. If that is not possible, it may just be that an RCT is unrealistic, and other types of evaluation are more practical, such as quasi-experimental designs (see Sections 3.12-3.15), or pre-experiments (see Sections 3.16-3.17) on rapid iterations in the roll-out of the policy. The Medical Research Council is leading the way on providing guidance on evaluating and experimenting in complex systems (see our list of resources in Section 4).

It is also worth noting that experiments are possible in challenging multifaceted areas, such as on 'complex contagion' and how innovations spread like social movements or new social norms (see Box 8, which covers the work of Damon Centola). ²⁰⁴

Endnotes

Roman numerals

1. Definitions of what constitutes an experiment tend not be consistent. We follow William Shadish, Thomas Cook and Donald Campbell (2002); Martin Ravallion (2018) and Matthew Salganik (2017) who define an experiment as a kind of trial, with different designs and forms of control. Try William Shadish, Thomas Cook, and Donald Campbell, *Experimental and quasi-experimental designs for generalized causal inference* (Wadsworth Cengage Learning, Boston, MA, 2002).
2. We discuss several of these debates in the section on 'The Basic RCT' in this report. For reference, see critiques from Lant Pritchett (2018) 'The Debate about RCTs in Development is over. We won. They lost.' Development Research Institute Blog, NYU. Available at: [http://www.nyudri.org/events-index/2018/2/22/lant-pritchett-talk-the-debate-about-rcts-in-development-is-over-we-won-they-lost](http://www.nyudri.org/events-index/2018/2/22/lant-pritchett-talk-the-debate-about-rcts-in-development-is-over-we-won-they-lost). Also, this article summarising some debates resulting from the Nobel Prize in Economics: Piper (2019) 'The Nobel went to economists who changed how we help the poor. But some critics oppose their big idea.' Vox. Available at: [https://www.vox.com/future-perfect/2019/12/11/20938915/nobel-prize-economics-banerjee-duflo-kremer-rcts](https://www.vox.com/future-perfect/2019/12/11/20938915/nobel-prize-economics-banerjee-duflo-kremer-rcts)
3. Advocates of RCTs are often termed 'randomistas'. In some cases, this is considered an insult - but many also use it positively, such as Andrew Leigh in his 2018 book, *Randomistas*.
4. Many early attempts at experimenting in the UK failed because decision-makers jumped the gun on making policy before results were in. Gough and Breckon provide a review of early UK policy experiments in Breckon & Gough (2019) 'Using Evidence in the UK' in What Works Now, edited by Boaz, Davies, Fraser & Nutley. The Magenta Book is included in our list of further resources in Section 4.
5. Find out more about how NSPCC are learning about their services and explore their resources at NSPCC Learning: [learning.nspcc.org.uk](https://learning.nspcc.org.uk)
6. Our Using Research Evidence Practice Guide describes a theory of change. 'In the early stages of any intervention, it's important to logically describe what you do and why it matters coherently, clearly and convincingly. This is often referred to as a theory of change, and aims to give a: "...clear, concise and convincing explanation of what you do, what impact you aim to have, and how you believe you will have it. It is a vital foundation of any programme, and a prerequisite for effective evaluation." A theory of change is a useful way to be more explicit about what evidence you are using from others - and to be clearer about how you are going to get results.' Jonathan Breckon, *Using Research Evidence: A Practice Guide* (Alliance for Useful Evidence, Nesta, London, 2016) p16.
7. One example of a natural experiment is a study of the 2008 Beijing Olympics. It looked at the impact of air pollution in the city, and how this affected health. In preparation for the Olympics, the Chinese Government introduced measures to improve the city's poor air quality. These measures were reversed after the Olympic period, creating a small window of better quality air in the city. A researcher used this window to compare the weight of babies born in this period to those born in 2007 and 2009, and found they were 23 grams heavier on average, suggesting that air pollution can interfere with how babies develop in the womb. Seizing the opportunity to run this natural experiment made it possible to study something that it would be unethical to study in a randomised experiment. This natural experiment generated important evidence about the harmful effects of pollution. Rich et al (2015) 'Differences in Birth Weight Associated with the 2008 Beijing Olympics Air Pollution Reduction: Results from a Natural Experiment' in *Environ Health Perspect*. 2015 Sep;123(9):880-7. doi: [10.1289/ehp.1408795](https://doi.org/10.1289/ehp.1408795).
8. Taking the same example, if we decide to compare people over 65 who received a flu jab to individuals over 65 who didn't, this is unlikely to resolve all our problems. On the one hand, people who received a flu jab might be different to those who didn't in some way we can observe - maybe they were the targets of an advertising campaign or received a letter from their doctor. On the other, there could be differences that we can't necessarily observe or measure - perhaps those who didn't get a jab are less engaged with health services and avoid going to check-ups. This is likely to impact on their health long-term.
9. The Campbell Collaboration is a central part of the evidence landscape worldwide. It has produced systematic reviews on social interventions since 2000 and has centres in Denmark, the UK and India. Find out more: [https://campbellcollaboration.org](https://campbellcollaboration.org)
10. Traditional matching can mean running into problems creating matches and is problematic when using large amounts of data. For example: what if few good comparison individuals exist? Or if no matches are available? In these situations it's difficult to determine what constitutes a good-enough match.
11. The UK's Health Foundation has for example pointed out programmes can't be seen as packages that can just be 'copied and pasted' to new locations; they need to be implemented in a way that's sensitive to context. See Dixon (2016) 'Spreading improvement: how to accelerate and the importance of archetypes' in Khan, *The Future of People Powered Health*, Nesta, London.
12. One systematic review of the 'plan-do-study-act' method, for example, conducted in 2014, reported that many interventions using PDSA didn't actually comply with many of the method's core features, and were often poorly evaluated. Only 15 per cent of the studies in the review reported using quantitative data at monthly or more frequent data intervals to inform cycles of change and learning. Another, published in 2019, pointed out that, 'widespread challenges with low adherence to key methodological features in the individual projects pose a challenge for the legitimacy of PDSA-based QI'. Other ways of doing rapid cycle testing may face similar challenges. See the following reviews: Knudsen, S.V., Laursen, H.V.B., Johnsen, S.P. et al. Can quality improvement improve the quality of care? A systematic review of reported effects and methodological rigor in plan-do-study-act projects. *BMC Health Serv Res* 19, 683 (2019) doi: [10.1186/s12913-019-4482-6](https://doi.org/10.1186/s12913-019-4482-6) (Open Access); Taylor MJ, McNicholas C, Nicolay C, et al. (2014) Systematic review of the application of the plan-do-study-act method to improve quality in healthcare, *BMJ Quality & Safety* 23:290-298.
13. Many public health programmes are adapted by local teams. One review found that 62 per cent of US evidence-based programmes focusing on HIV/AIDS, mental health, substance abuse, and chronic illnesses had been adapted locally; see Cam Escoffery et al, 'A systematic review of adaptations of evidence-based public health interventions globally', *Implementation Science* 13, no. 125 (September 2018). How these adaptations are made is important: there's a risk that making changes to programmes could contradict their original evidence-base, making them less effective, or even causing harmful consequences. But, it's often not clear what the 'core elements' or 'active ingredients' of a solution are. In public health there are now frameworks to help organisations make safe adaptations, such as ADAPT-ITT, which provides guidance on adapting and testing evidence-based HIV Interventions. Some projects, like US sex education programme Get Real, have specific guidance on 'green light', 'yellow light', and 'red light' adaptations. See Gina Wingood and Ralph DiClemente, 'The ADAPT-ITT model: A novel method of adapting evidence-based HIV Interventions', *Journal of Acquired Immune Deficiency Syndromes* 47 (March 2008) Suppl 1:S40-6, and the Get Real website: [https://www.etr.org/ebi/programs/get-real](https://www.etr.org/ebi/programs/get-real)
14. See Coalition for Evidence Based Policy (2013) *Randomized Controlled Trials Commissioned by the Institute of Education Sciences Since 2002: How Many Found Positive Versus Weak or No Effects*, Coalition for Evidence Based Policy. For employment and training programmes 75 per cent of RCTs commissioned by the US Department of Labour show weak or no positive effects. The same is true in business. According to the author Jim Manzi, 80 to 90 per cent of RCTs on new products and strategies run by Google and Microsoft have found no significant effects (Jim Manzi, *Uncontrolled: The Surprising Payoff of Trial-and-Error for Business, Politics, and Society*, Perseus Books Group, New York, 2012, pp. 128 and 142).

Numbers

1. Find out about our EdTech Innovation Testbed at: https://www.nesta.org.uk/project/edtech-innovation-testbed; and read about the Business Basics Programme: https://www.nesta.org.uk/blog/why-you-should-know-about-business-basics-programme
2. Jonathan Breckon, Better Public Services Through Experimental Government (Alliance for Useful Evidence, Nesta, London, 2015).
3. The Franklin Institute, Science and Learning Resources, 'Edison's Lightbulb' https://www.fi.edu/history-resources/edisons-lightbulb
4. Nicholas Maxwell, Karl Popper, Science and Enlightenment (London: UCL Press, 2017).
5. Attributed to Kurt Lewin in Charles W. Tolman (1996) Problems of Theoretical Psychology - ISTP p.31 and Johnson & Christensen (2016) Educational Research: Quantitative, Qualitative, and Mixed Approaches, SAGE Publications.
6. Jim Manzi, head of Applied Predictive Technologies claims in his book Uncontrolled that his firm is running RCTs for '30 to 40 per cent of the largest retailers, hotel chains, restaurant chains, and retail banks in America'. Jim Manzi, Uncontrolled: The Surprising Payoff of Trial-and-Error for Business, Politics, and Society (New York: Basic Books, 2012) p147.
7. Brian Christian, 'The A/B Test: Inside the Technology That's Changing the Rules of Business', Wired, 25 April 2012 https://www.wired.com/2012/04/ff-abtesting
8. Andrew Leigh, Randomistas: How Radical Researchers Changed Our World (Yale University Press, 2018).
9. Gerry Stoker, 'Exploring the Promise of Experimentation in Political Science: Micro-Foundational Insights and Policy Relevance', Political Studies 58, no. 2 (March 2010): 300-319. https://doi.org/10.1111/j.1467-9248.2009.00812.x p316
10. Alvin Roth, 'Introduction to Experimental Economics' in The Handbook of Experimental Economics, edited by John Kagel and Alvin E. Roth, 3-109. Princeton University Press (1995).
11. Amy Edmondson, 'Strategies for Learning from Failure', Harvard Business Review, 2011 https://hbr.org/2011/04/strategies-for-learning-from-failure
12. Kincaid, H (2012) Introduction: Doing Social Science, The Oxford Handbook of Philosophy of Social Science Oxford University Press.
13. Gerry Stoker, 'Exploring the Promise of Experimentation in Political Science: Micro-Foundational Insights and Policy Relevance', Political Studies 58, no.2 (March 2010): 300-319. https://doi.org/10.1111/j.1467-9248.2009.00812.x p316
14. Jonathan Breckon, Better Public Services Through Experimental Government (Nesta, London, 2015) p7.
15. Jen Gold, 'How Experiments Craft Better Policy: Inside the world's What Works teams', Apolitical, 2019 https://apolitical.co/solution_article/inside-worlds-what-works-teams
16. Conversations with Nicholas Chesterley and Myra Latendresse Drapeau, Government of Canada.
17. Health Canada's Experimentation Works (EW) Team, 'Health Canada's PRODigy experiment: An experience in learning-by-doing', Medium, 2018 https://medium.com/@exp_works/health-canadas-prodigy-experiment-an-experience-in-learning-by-doing-87598ab222c3
18. Experimentation Works (EW) is on Medium: https://medium.com/@exp_works
19. See Department for Work and Pensions press release on Universal Credit, 2013: https://www.gov.uk/government/news/universal-credit-progress
20. Whistleblowers have reported that the expensive IT system is riddled with design flaws and a January 2017 parliamentary inquiry into the system is ongoing. See Commons Select Committee, Work & Pensions, Benefits Sanctions, The role of Universal Support: https://publications.parliament.uk/pa/cm201719/cmselect/cmworpen/1667/166704.htm#_idTextAnchor002 Also see Patrick Butler, 'Universal credit IT system 'broken', whistleblowers say', The Guardian, 2018 https://www.theguardian.com/society/2018/jul/22/universal-credit-it-system-broken-service-centre-whistleblowers-say
21. Amy Edmondson, 'Strategies for Learning from Failure', Harvard Business Review, 2011. https://hbr.org/2011/04/strategies-for-learning-from-failure
22. EEF Blog, 'Magic Breakfast: A case study in scaling evidence for impact', Education Endowment Foundation, 2018 https://educationendowmentfoundation.org.uk/news/eef-blog-magic-breakfast-a-case-study-in-scaling-evidence-for-impact
23. Jonathan Breckon, Using Research Evidence: A Practice Guide (Alliance for Useful Evidence, Nesta, London, 2016).
24. Jonathan Breckon, Better Public Services Through Experimental Government (Alliance for Useful Evidence, Nesta, London, 2015).
25. Anthony King and Ivor Crewe, The Blunders of Our Governments (London: Oneworld Publications, 2013).
26. A. J. Macdonald, Guide to the Classical and other Long-term experiments, Datasets and Sample Archive (Rothamsted Research, Herts, 2018) https://www.rothamsted.ac.uk/sites/default/files/Web_LTE%20Guidebook_2019%20Final2.pdf
27. Ronald Fisher, The Design of Experiments (New York: Hafner Publishing Company, 1971 - originally 1935).
28. Sebastian Heilmann, 'From Local Experiments to National Policy: The Origins of China's Distinctive Policy Process', The China Journal No. 59 (January 2008): 1-30 Retrieved from: www.jstor.org/stable/20066378
29. Jonathan Breckon, Better Public Services Through Experimental Government (Alliance for Useful Evidence, Nesta, London, 2015).
30. Judith Gueron and Harold Rolston, Fighting for Reliable Evidence (New York: Russell Sage Foundation, 2013).
31. Robert Brook et al, The Health Insurance Experiment: A Classic RAND Study Speaks to the Current Health Care Reform Debate (RAND Research Brief, RAND Corporation, 2006) https://www.rand.org/pubs/research_briefs/RB9174.html
32. Jonathan Breckon, Better Public Services Through Experimental Government (Alliance for Useful Evidence, Nesta, London, 2015).
33. The report was titled When Will We Ever Learn?, the quote is provided in a history of evidence in Howard White (2019) The twenty-first century experimenting society: the four waves of the evidence revolution Palgrave Communications p3. https://www.nature.com/articles/s41599-019-0253-6.pdf
34. Jonathan Breckon, Better Public Services Through Experimental Government (Alliance for Useful Evidence, Nesta, London, 2015) p19.
35. Stephen Kidd, 'The demise of Mexico's Prospera programme: a tragedy foretold', Development Pathways Blog, 2019 https://www.developmentpathways.co.uk/blog/the-demise-of-mexicos-prospera-programme-a-tragedy-foretold/
36. What Works Network, The Rise of Experimental Government: Cross-Government Trial Advice Panel Update Report (What Works Network and Economic and Social Research Council, London, 2018).
37. For a 2019 update on the work of these teams: Jen Gold, 'How Experiments Craft Better Policy: Inside the world's What Works teams', Apolitical, 2019 https://apolitical.co/solution_article/inside-worlds-what-works-teams
38. Find out more about Experimental Finland at: https://kokeilevasuomi.fi/en/frontpage
39. The French Government website on the 'Garantie Jeune' can be found here: https://www.service-public.fr/particuliers/vosdroits/F32700

Footnotes

1. Caroline Allard and Ben Rickey, British What Works Centres: What lessons for evidence-based policy in France? (Agence nouvelle des solidarités actives, 2017) p 86. https://www.alliance4usefulevidence.org/assets/Ansa_A4UE_whatworks_final_Full-report-standard.pdf
2. Shatha Alhashmi and Giulio Quaggiotto, 'Promoting experimentation in government - learning from Canada's experience', Nesta blog, June 2017 https://www.nesta.org.uk/blog/promoting-experimentation-in-government-learning-from-canadas-experience
3. Experimental Finland website: https://kokeilevasuomi.fi/en/frontpage
4. Albert Bravo-Biosca and Lou-Davina Stouffs, 'What are we learning from policy experiments to increase innovation and entrepreneurship?', Nesta blog, June 2018 https://www.nesta.org.uk/blog/what-are-we-learning-policy-experiments-increase-innovation-and-entrepreneurship
5. James Phipps and Lou-Davina Stouffs, 'Coming soon... the Vouchers Story', Innovation Growth Lab blog, April 2018 https://www.innovationgrowthlab.org/blog/coming-soon-vouchers-story
6. Dan Hodges, 'How experiments are improving the way we work', Innovation Growth Lab blog, August 2018 https://www.innovationgrowthlab.org/blog/how-experiments-are-improving-way-we-work
7. Find out more about Giving Evidence at: https://giving-evidence.com
8. Christian Bason, Design for Policy (Gower Publishing, 2014).
9. Geoff Mulgan, 'Mindsets and Methods: The 21st century curriculum for public servants', Nesta blog, April 2019 https://www.nesta.org.uk/blog/mindsets-and-methods
10. Find out more about States of Change at: https://states-of-change.org
11. William Trochim, 'Donald T. Campbell and Research Design', American Journal of Evaluation 19, no. 3 (1998): 407-409.
12. For an overview, see this report from the World Bank Group: Zeina Afif, William Wade Islan, Oscar Calvo-Gonzalez, Abigail Goodnow Dalton, Behavioral Science Around the World: Profiles of 10 Countries (English) (eMBeD brief, World Bank Group, Washington, D.C, 2019).
13. Available from the James Lind Library. MRC, 'Streptomycin treatment of pulmonary tuberculosis: A Medical Research Council investigation', BMJ 2 (1948): 769-78. https://www.jameslindlibrary.org/medical-research-council-1948b
14. See discussion in Harry Rutter et al, 'The need for a complex systems model of evidence for public health', The Lancet 390, no. 10112 (2017) 2602-2604.
15. Donald Berwick, 'The Science of Improvement', JAMA. 299 no.10 (March 2008): 1182-1184. https://doi.org/10.1001/jama.299.10.1182
16. Find out more about Edna McConnell Clark Foundation at: www.emcf.org
17. Vinod Rajasekaran, 'Getting to Moonshot: Inspiring R&D Practices in Canada's Social Impact Sector', SiG, 2016, p9.
18. Collins Dictionary Online: https://www.collinsdictionary.com/dictionary/english/experiment
19. James Druckman et al, 'The Growth and Development of Experimental Research in Political Science', American Political Science Review 100, no. 4 (2006).
20. Definitions of what constitutes an experiment tend not be consistent. We follow William Shadish, Thomas Cook and Donald Campbell (2002); Martin Ravallion (2018) and Matthew Salganik (2017) who define an experiment as a kind of trial, with different designs and forms of control. Try William Shadish, Thomas Cook, and Donald Campbell, Experimental and quasi-experimental designs for generalized causal inference (Wadsworth Cengage Learning, Boston, MA, 2002).
21. Jonathan Breckon, Better Public Services Through Experimental Government (Alliance for Useful Evidence, Nesta, London, 2015).
22. EEF, Classification of the security of findings from EEF evaluations (Education Endowment Foundation, July 2019) https://educationendowmentfoundation.org.uk/public/files/Evaluation/Carrying_out_a_Peer_Review/Classifying_the_security_of_EEF_findings_2019.pdf
23. Matthew Salganik, Bit by Bit: Social Research in the Digital Age (Princeton, NJ: Princeton University Press, 2017) p148.
24. Neil Salkind, 'Pre-Experimental Methods' defined in SAGE Encyclopaedia of Research Design, SAGE Publishing, 2010 https://methods.sagepub.com/Reference//encyc-of-research-design/n330.xml
25. Figure 3 is informed by: Neil Salkind, 'Pre-Experimental Methods' defined in SAGE Encyclopaedia of Research Design, SAGE Publishing, 2010 https://methods.sagepub.com/Reference//encyc-of-research-design/n330.xml; Sandesh Adhikari, '20 differences between Randomized Controlled Trial (RCT) and Quasi-experimental study design', Public Health Notes, 2018 https://www.publichealthnotes.com/20-differences-between-randomized-controlled-trial-rct-and-quasi-experimental-study-design; and Matthew Salganik, Bit by Bit: Social Research in the Digital Age (Princeton, NJ: Princeton University Press, 2017).
26. Elizabeth Lillie, Bradley Patay, Joel Diamant, Brian Issell, Eric Topol, Nicholas Schork, 'The n-of-1 clinical trial: The ultimate strategy for individualizing medicine?' Personalized medicine 8 no. 2 (2011): 161-173. https://doi.org/10.2217/pme.11.7
27. Taken from: Siobhan Campbell, Gemma Harper, 'Quality in policy impact evaluation: Understanding the effects of policy from other influences' (Magenta Guide: HM Treasury, DECC and DEFRA, 2012).
28. Paul Connolly, Ciara Keenan and Karolina Urbanska, 'The trials of evidence-based practice in education: A systematic review of randomised controlled trials in education research 1980-2016', Educational Research, 60 no.3 (2018): 276-291, DOI: 10.1080/00131881.2018.1493353.
29. Bruce Thyer, 'A Bibliography of Randomized Controlled Experiments in Social Work (1949-2013)', Research on Social Work Practice 25 no.7 (2015): 753-793.
30. Find out more about the Global Policing Database: https://global-policing-database.dhssp.cloud.edu.au/s/gpd/page/about
31. Robert G. St. Pierre and Jean I. Layzer, 'Using Home Visits for Multiple Purposes: The Comprehensive Child Development Program,' The Future of Children 9, no. 1, spring/summer (1999): 134-151; Robert G. St Pierre et al, National Impact Evaluation of the Comprehensive Child Development Program (Administration for Children and Families, U.S. Department of Health and Human Services, 1997) https://www.acf.hhs.gov/sites/default/files/opre/ccdp_fullreport.pdf
32. Find out more about Creative Credit at: https://www.nesta.org.uk/feature/measuring-our-impact/creative-credits
33. Cancer Research UK discusses the phases on clinical trials at: https://www.cancerresearchuk.org/about-cancer/find-a-clinical-trial/what-clinical-trials-are/phases-of-clinical-trials
34. Graham Moore et al, Process evaluation of complex interventions: UK Medical Research Council (MRC) guidance (MRC Population Health Science Research Network, 2015) https://mrc.ukri.org/documents/pdf/mrc-phsrn-process-evaluation-guidance-final
35. Angus Deaton and Nancy Cartwright, 'Understanding and misunderstanding randomized controlled trials', Social Science & Medicine 210 (2018): 2-21. https://www.sciencedirect.com/science/article/pii/S0277953617307359
36. For more details, see Nesta's Innovation Growth Lab Experimentation Toolkit, Section 1.7 on Limitations of RCTs, included in our list of further resources in Section 4. https://www.nesta.org.uk/toolkit/innovation-growth-lab-experimentation-toolkit
37. Dougal Hutchison and Ben Styles, A Guide to Running Randomised Controlled Trials for Educational Researchers. (NFER, Slough, 2010) https://www.nfer.ac.uk/media/2114/rct01.pdf
38. Angus Deaton and Nancy Cartwright, 'Understanding and misunderstanding randomized controlled trials' Social Science & Medicine 210 (2018): 2-21. https://www.sciencedirect.com/science/article/pii/S0277953617307359
39. See Mary Ann Bates and Rachel Glennerster, 'The Generalizability Puzzle', Stanford Social Innovation Review (Summer 2017). Available through Open Access online at: https://ssir.org/articles/entry/the_generalizability_puzzle#
40. Gerry Stoker, 'Exploring the Promise of Experimentation in Political Science: Micro-Foundational Insights and Policy Relevance', Political Studies 58 no. 2 (2010).: 300-319. https://doi.org/10.1111/j.1467-9248.2009.00812.x p303.
41. Jonathan Breckon, Better Public Services Through Experimental Government (Alliance for Useful Evidence, Nesta, London) p32.
42. Jonathan Breckon, Better Public Services Through Experimental Government (Alliance for Useful Evidence, Nesta, London) p43.
43. Phil Alderson, Ian Roberts, 'Corticosteroids for acute traumatic brain injury', Cochrane Database of Systematic Reviews 1 Art. No. CD000196 (2005). Available from Cochrane Library: https://www.cochrane.org/CD000196/INJ_corticosteroids-to-treat-brain-injury
44. For example, if an intervention is shown to be effective in one setting but uncertainty remains about how it will work in others, how much evidence - and how many experiments - do we need to make further trials unnecessary, or unethical?
45. See Demos Helsinki's website, the organisation helped author the Code of Conduct: https://www.demoshelsinki.fi/en/referenssit/code-conduct-national-ethical-code-conduct-societal-experiments
46. Adrian O'Dowd, 'Scientists call for more multi-arm clinical trials to speed up approval of new drugs', BMJ 349:g4812 (2014) https://www.bmj.com/content/349/bmj.g4812
47. MRC Blog, 'Clinical trials: why multi-arms are better than two', Medical Research Council Blog, 25 July 2014 https://mrc.ukri.org/news/blog/clinical-trials-why-multi-arms-are-better-than-two
48. Ibid.
49. See Optimizely's Glossary: https://www.optimizely.com/uk/optimization-glossary/multi-armed-bandit
50. Sofia Villar, Jack Bowden, James Wason, 'Multi-armed Bandit Models for the Optimal Design of Clinical Trials: Benefits and Challenges', Statistical Science 30 no. 2 (2015):199-215.4. Statistical Science 30(2):199-215.4.
51. Chris Kaibel and Torsten Biemann, 'Rethinking the Gold Standard With Multi-armed Bandits: Machine Learning Allocation Algorithms for Experiments', Organizational Research Methods (2019) Organizational Research Methods.
52. Aaron Dibner-Dunlap and Yumna Rathore, 'Beyond RCTs: How Rapid-Fire Testing Can Build Better Financial Products', IPA Blog, 2016 https://www.poverty-action.org/blog/beyond-rcts-how-rapid-fire-testing-can-build-better-financial-products
53. Find out more about the EdTech Innovation Testbed: https://www.nesta.org.uk/project/edtech-innovation-testbed
54. Alaka Holla, 'Seeking Nimble Plumbers', World Bank Blogs, April 4 2018 https://blogs.worldbank.org/impactevaluations/seeking-nimble-plumbers
55. Read a summary of the World Bank's Nimble Evaluations: https://www.worldbank.org/en/programs/sief-trust-fund/brief/nimble-summaries
56. Dean Karlan and Mary Kay Gugert, The Goldilocks Challenge: Right-Fit Evidence for the Social Sector (OUP USA, 2018).
57. Adapted from Matthew Salganik, Bit by Bit: Social Research in the Digital Age. (Princeton, NJ: Princeton University Press, 2017) Chapter 4: Running Experiments.
58. Behavioural Insights Team et al, Applying Behavioural Insights to Organ Donation: Preliminary results from a randomised controlled trial Cabinet Office, London, 2013).
59. Joseph Henrich, Steve J. Heine, Ara Norenzayan, 'The Weirdest People in the World?,' Working Paper Series of the German Council for Social and Economic Data 139, German Council for Social and Economic Data (RatSWD) (2010) https://ideas.repec.org/p/rsw/rswwps/rswwps139.html
60. Amazon Mechanical Turk (MTurk): https://www.mturk.com
61. LabintheWild: https://www.labinthewild.org
62. The Web Experiment List: http://www.wexlist.net
63. Damon Centola, How Behaviour Spreads (Princeton, NJ: Princeton University Press, 2018).
64. Adam Kramer, Jamie Guillory, Jeffrey Hancock, 'Experimental evidence of massive-scale emotional contagion through social networks', Proc Natl Acad Sci US A 111 no. 24 (2014):8788-8790. doi: 10.1073/pnas.1320040111.
65. Michael H. Birnbaum, 'Human Research and Data Collection via the Internet', Annual Review of Psychology 55, no. 1 (2004): 803-832. https://doi.org/10.1146/annurev.psych.55.090902.141601
66. Jesse Chandler et al, 'Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers', Behavior Research Methods 46 (2013): 112. https://doi.org/10.3758/s13428-013-0365-7 doi: 10.3758/s13428-013-0365-7
67. Andrew Leigh, Randomistas: How Radical Researchers Changed Our World (Yale University Press, 2018).
68. Andrew Leigh, Randomistas: How Radical Researchers Changed Our World (Yale University Press, 2018) p131.
69. See Obama for America's Digital Fundraising Machine: https://www.optimizely.com/uk/customers/obama2012
70. Behavioural Insights Team et al, Applying Behavioural Insights to Organ Donation: preliminary results from a randomised controlled trial (Behavioural Insights Team, Cabinet Office, London, 2013) https://www.bi.team/publications/applying-behavioural-insights-to-organ-donation
71. Ibid.
72. Matthew Salganik, Bit by Bit: Social Research in the Digital Age (Princeton, NJ: Princeton University Press, 2017).
73. Test & Build platform: https://www.testandbuild.com; Predictiv platform: https://www.bi.team/bi-ventures/predictiv
74. POST, Understanding Research Methods (Parliamentary Office for Science and Technology, forthcoming in 2020).
75. Thomas Dee and Benjamin Keys, 'Does merit pay reward good teachers? Evidence from a randomized experiment', Journal of Policy Analysis and Management 23 no. 3 (Summer 2004): 471-488.
76. For example, in a subsidised meal programme experiment in Kenya, upset parents in over half of the control schools organised to raise funds for student meals to match what was being received in the treatment group. See David Greenber and Mark Shroder, The Digest of Social Experiments (3rd ed.) (Washington, DC: Urban Institute Press, 2004) p.399.
77. Gary King, Emmanuela Gakidou, Nirmala Ravishankar, Ryan Moore, Jason Lakin, Manett Vargas et al, 'A 'politically robust' experimental design for public policy evaluation, with application to the Mexican universal health insurance program', Journal of Policy Analysis and Management 26 (2007): 479-506.
78. Ibid.
79. Laura Haynes, Owain Service, Ben Goldacre, David Torgerson, Test, Learn, Adapt: Developing Public Policy with Randomised Controlled Trials (Cabinet Office, London, 2010) p.22. https://www.gov.uk/government/publications/test-learn-adapt-developing-public-policy-with-randomised-controlled-trials
80. Emma Beard et al, 'Stepped wedge randomised controlled trials: systematic review of studies published between 2010 and 2014', Trials 16, no. 353 (2015).
81. Noreen Mdege et al, 'Systematic review of stepped wedge cluster randomized trials shows that design is particularly used to evaluate interventions during routine implementation' Journal of Clinical Epidemiology Sep;64(9) (2011): 936-48. doi: 10.1016/j.jclinepi.2010.12.003
82. Dominic Pearson, David Torgerson, Cynthia McDougall, Roger Bowles 'A parable of two agencies, one of which randomises', Annals of the American Academy of Political & Social Sciences 628 (2010): 11-29.
83. Ministry of Housing, Communities and Local Government, Measuring the impact of Community-Based English Language Provision Findings from a Randomised Controlled Trial (National Learning and Work Institute, Ministry of Housing, Communities and Local Government, HM Government, 2018).
84. Interview with Dr Jen Gold, Head of the What Works Team.
85. Ministry for Housing, Communities and Local Government, Integrated Communities Strategy green paper (Ministry for Housing, Communities and Local Government, HM Government, 2018).
86. What Works Network, The Rise of Experimental Government: Cross Government Trials Advice Panel Update Report, (What Works Network and Economic and Social Research Council, London) p17.
87. Neil Salkind, 'Crossover Design', defined in SAGE Encyclopaedia of Research Design, SAGE Publishing, 2010 https://methods.sagepub.com/reference/encyc-of-research-design/n95.xml
88. Kerry Dwan, Tianjing Li, Douglas G Altman, Diana Elbourne, 'CONSORT 2010 statement: extension to randomised crossover trials', BMJ 366 (2019) :14378. https://www.bmj.com/content/366/bmj.14378
89. Hui-Ling Lai and Yin-Ming Li, 'The effect of music on biochemical markers and self-perceived stress among first-line nurses: a randomized controlled crossover trial,' Journal of Advanced Nursing 67(11) (2011): 2414-24. https://doi.org/10.1111/j.1365-2648.2011.05670.x
90. Halima Khan, 'Nesta's work on Personalised Care: How we got here', Nesta Blog, 4 February 2019 https://www.nesta.org.uk/blog/nestas-work-personalised-care-how-we-got-here
91. See Health Education England's page on Person-centred Care: https://www.hee.nhs.uk/our-work/person-centred-care
92. Eva van der Ploeg, Barbara Eppingstall, Cameron Camp, Susannah Runci, John Taffe, Daniel O'Connor, 'A randomized crossover trial to study the effect of personalized, one-to-one interaction using Montessori-based activities on agitation, affect, and engagement in nursing home residents with Dementia', International Psychogeriatrics 25(4), (2013): 565-575. https://doi.org/10.1017/S1041610212002128
93. Kerry Dwan, Tianjing Li, Douglas G Altman, Diana Elbourne, 'CONSORT 2010 statement: extension to randomised crossover trials', BMJ 366 (2019) :14378. https://www.bmj.com/content/366/bmj.14378
94. Dean Karlan and Jacob Appe, Failing in the Field (Princeton, NJ: Princeton University Press, 2016) p8.
95. University of Cambridge Research News: https://www.cam.ac.uk/research/news/first-scientific-report-shows-police-body-worn-cameras-can-prevent-unacceptable-use-of-force
96. Alex Sutherland et al, 'Post-experimental follow-ups - Fade-out versus persistence effects: The Rialto police body-worn camera experiment four years on', Journal of Criminal Justice 53 (November 2017): 110-116.
97. Barak Ariel, et al, 'The Effect of Police Body-Worn Cameras on Use of Force and Citizens' Complaints Against the Police: A Randomized Controlled Trial', Journal of Quantitative Criminology 31 no.3 (November 2014): 509-535.
98. Barak Ariel et al, 'Report: increases in police use of force in the presence of body-worn cameras are driven by officer discretion: a protocol-based subgroup analysis of ten randomized experiments', Journal of Experimental Criminology 12 (2016): 453-463; Barak Ariel et al, 'Contagious Accountability': A Global Multisite Randomized Controlled Trial on the Effect of Police Body-Worn Cameras on Citizens' Complaints Against the Police', Criminal Justice and Behavior, 44(2) (2017): 293-316. https://doi.org/10.1177/0093854816668218
99. Alex Sutherland et al, 'Post-experimental follow-ups - Fade-out versus persistence effects: The Rialto police body-worn camera experiment four years on', Journal of Criminal Justice 53 (November 2017): 110-116.
100. This has been discussed by researchers over the years. See this perspective from the US in 2004: Lawrence Sherman and Heather Strang, 'Experimental Ethnography: The Marriage of Qualitative and Quantitative Research', The ANNALS of the American Academy of Political and Social Science 595 no. 1 (2004): 204-222. https://doi.org/10.1177/0002716204267481
101. Ray Pawson and Nicholas Tilley, Realistic Evaluation (London: SAGE Publications Ltd, 1997, 2014); Chris Bonnell et al, 'Realist randomised controlled trials: A new approach to evaluating complex public health interventions', Social Science & Medicine 75(12) (December 2012): 2299-306. doi: 10.1016/j.socscimed.2012.08.032
102. Peter Hedström and Petri Ylikoski, 'Causal Mechanisms in the Social Sciences', Annual Review of Sociology Vol 36 (2010): 49-67. Doi: 10.1146/annurev.soc.012809.102632
103. Ray Pawson and Nicholas Tilley, Realistic Evaluation (London: SAGE Publications Ltd, 1997, 2014)
104. Farah Jamal et al, 'The three stages of building and testing mid-level theories in a realist RCT: A theoretical and methodological case-example', Trials 16, no. 466 (2015) Open Access: https://trialsjournal.biomedcentral.com/articles/10.1186/s13063-015-0980-y
105. Chris Bonnell et al, 'Realist randomised controlled trials: A new approach to evaluating complex public health interventions', Social Science & Medicine 75(12) (December 2012): 2299-306. doi: 10.1016/j.socscimed.2012.08.032
106. Lyndal Bond et al, 'The Gatehouse Project: Can a multilevel school intervention affect emotional wellbeing and health risk behaviours?', Journal of Epidemiology and Community Health 58, no.12 (2004): 997-1003. doi: 10.1136/jech.2003.009449
107. Wolfgang Markham and Paul Aveyard, 'A new theory of health promoting schools based on human functioning, school organisation and pedagogic practice', Social Science & Medicine 56, no.6 (2003): 1209-20. doi: 10.1016/s0277-9536(02)00120-x
108. Chris Bonell, et al, 'Initiating change locally in bullying and aggression through the school environment (INCLUSIVE): a pilot randomised controlled trial', Health Technology Assessment 19(53) (2015).
109. Chris Bonell et al, 'Examining intervention mechanisms of action using mediation analysis within a randomised trial of a whole-school health intervention', Journal of Epidemiology & Community Health 73 no. 5 (2019): 455-464.
110. LSHTM, 'Making UK schools more inclusive places could help reduce bullying and promote well-being', London School of Hygiene & Tropical Medicine Blog, 2018 https://www.lshtm.ac.uk/newsevents/news/2018/making-uk-schools-more-inclusive-places-could-help-reduce-bullying-and-promote
111. Chris Bonell et al, 'Examining intervention mechanisms of action using mediation analysis within a randomised trial of a whole-school health intervention', Journal of Epidemiology & Community Health 73 no. 5 (2019): 455-464.
112. C. Hendricks Brown et al, An Overview of Research and Evaluation Designs for Dissemination and Implementation', Annual Review of Public Health 38 (March 2017): 1-22.
113. Ibid.
114. Geoffrey Curran et al, 'Effectiveness-Implementation Hybrid Designs', Medical Care 50, no.3 (March 2012): 217-26.
115. C. Hendricks Brown et al, 'An Overview of Research and Evaluation Designs for Dissemination and Implementation', Annual Review of Public Health 38 (March 2017): 1-22.
116. Geoffrey Curran et al, 'Implementation of the CALM intervention for anxiety disorders: A qualitative study', Implementation Science 7, no. 14 (2012) doi:10.1186/1748-5908-7-14. Open Access Review: https://implementationscience.biomedcentral.com/articles/10.1186/1748-5908-7-14
117. Bryan Garner et al, 'Testing the effectiveness of a motivational interviewing-based brief intervention for substance use as an adjunct to usual care in community-based AIDS service organizations: Study protocol for a multisite randomized controlled trial', Addiction Science & Clinical Practice, 12, no.31 (2017) doi: 10.1186/s13722-017-0095-8
118. Kari Gloppen et al, 'Sustaining adoption of science-based prevention through communities that care', Journal of Community Psychology 44:1 (2016) https://onlinelibrary.wiley.com/doi/abs/10.1002/jcop.21743
119. Registration of The Community Youth Development Study on Clinical Trials.gov: https://clinicaltrials.gov/ct2/show/study/NCT01088542?show_desc=Y#desc
120. Weichung Joe Shih, 'Plan to be Flexible: A Commentary on Adaptive Designs', Biometrical Journal / Biometrische Zeitschrift 48:4 (2006) 656-9; discussion 660. doi: 10.1002/bimj.200610241.
121. Philip Pallmann et al, 'Adaptive designs in clinical trials: Why use them, and how to run and report them', BMC medicine 16(1), no. 29 (2018) doi:10.1186/s12916-018-1017-7.
122. Ibid.
123. Margaret Handley et al, 'Selecting and Improving Quasi-Experimental Designs in Effectiveness and Implementation Research', Annual Review of Public Health 39:1 (2018): 5-25. doi: 10.1146/annurev-publhealth-040617-014128.
124. Sanjay Basu, Ankita Meghani, Arjumand Siddiqi, 'Evaluating the Health Impact of Large-Scale Public Policy Changes: Classical and Novel Approaches', Annual Review of Public Health 38 (2017): 351-370. doi: 10.1146/annurev-publhealth-031816-044208.
125. Steven Glazerman, Dan Levy, David Myers, 'Nonexperimental versus Experimental Estimates of Earnings Impacts', The Annals of the American Academy of Political and Social Science 589 (2003): 63-93. Retrieved from www.jstor.org/stable/3658561
126. Sanjay Basu, Ankita Meghani, Arjumand Siddiqi, 'Evaluating the Health Impact of Large-Scale Public Policy Changes: Classical and Novel Approaches', Annual Review of Public Health 38 (2017): 351-370. doi: 10.1146/annurev-publhealth-031816-044208.
127. Martin Ravallion, Should the Randomistas (Continue to) rule? (Working Paper 492, Centre for Global Development, August 2018) https://www.cgdev.org/publication/should-randomistas-continue-rule
128. Margaret Handley et al, 'Selecting and Improving Quasi-Experimental Designs in Effectiveness and Implementation Research', Annual Review of Public Health 39:1 (2018): 5-25. doi: 10.1146/annurev-publhealth-040617-014128; Sanjay Basu, Ankita Meghani, Arjumand Siddiqi, 'Evaluating the Health Impact of Large-Scale Public Policy Changes: Classical and Novel Approaches', Annual Review of Public Health 38 (2017): 351-370. doi: 10.1146/annurev-publhealth-031816-044208.
129. Thomas Cook, "Waiting for Life to Arrive': A history of the regression-discontinuity design in Psychology, Statistics and Economics', Journal of Econometrics 142, no. 2, (February 2008): 636-654.
130. Ibid.
131. Paul Gertler et al, Impact Evaluation in Practice (Washington DC: The International Bank for Reconstruction & Development/World Bank, 2011).
132. Ibid.
133. Ibid.
134. Adapted from Paul Gertler et al, Impact Evaluation in Practice (Washington DC: The International Bank for Reconstruction & Development / World Bank, 2011) p103.
135. Ojmarrh Mitchell, Joshua Cochran, Daniel Mears, William Bales, 'Examining Prison Effects on Recidivism: A Regression Discontinuity Approach', Justice Quarterly 34:4 (2017): 571-596. doi: 10.1080/07418825.2016.1219762.
136. European Commission, Testing Social Policy Innovation (Employment, Social Affairs & Inclusion, European Commission, 2014).
137. Ibid.
138. William Shadish, Thomas Cook, and Donald Campbell, Experimental and quasi-experimental designs for generalized causal inference (Wadsworth Cengage Learning, Boston, MA, 2002).
139. David Lee and Thomas Lemieux (2010) Regression Discontinuity Designs in Economics Journal of Economic Literature 48 281-355. Available online: [https://www.princeton.edu/davidlee/wp/RDDEconomics.pdf](https://www.princeton.edu/davidlee/wp/RDDEconomics.pdf)
140. Paul Gertler et al, Impact Evaluation in Practice (Washington DC: The International Bank for Reconstruction & Development/World Bank, 2011).
141. Sanjay Basu, Ankita Meghani, Arjumand Siddiqi, 'Evaluating the Health Impact of Large-Scale Public Policy Changes: Classical and Novel Approaches', Annual Review of Public Health 38 (2017): 351-370. doi: 10.1146/annurev-publhealth-031816-044208.
142. Paul Gertler et al, Impact Evaluation in Practice (Washington DC: The International Bank for Reconstruction & Development/World Bank, 2011).
143. Martin Ravallion, Should the Randomistas (Continue to) rule? (Working Paper 492, Centre for Global Development, August 2018) https://www.cgdev.org/publication/should-randomistas-continue-rule
144. All evaluation documents are available at the evaluation homepage at Birkbeck University's site: http://www.bbk.ac.uk
145. Sarah Cattan et al, The health effects of Sure Start (Institute for Fiscal Studies, London, June 2019) https://www.ifs.org.uk/publications/14139
146. Find out more about the Millennium Cohort Study: cls.ucl.ac.uk/cls-studies/millennium-cohort-study
147. Professor Edward Melhuish et al, The Impact of Sure Start Local Programmes on Seven Year Olds and Their Families, (The National Evaluation of Sure Start (NESS) Team, Institute for the Study of Children, Families and Social Issues; Birkbeck, University of London; and the Department for Education, 2012).
148. Ibid.
149. Ibid.
150. Sarah Cattan et al, The health effects of Sure Start (Institute for Fiscal Studies, London, June 2019) https://www.ifs.org.uk/publications/14139
151. Paul Gertler et al, Impact Evaluation in Practice (Washington DC: The International Bank for Reconstruction & Development/World Bank, 2011)
152. Ibid.
153. Margaret Handley et al, 'Selecting and Improving Quasi-Experimental Designs in Effectiveness and Implementation Research', Annual Review of Public Health 39:1 (2018): 5-25. doi: 10.1146/annurev-publhealth-040617-014128.
154. Coady Wing et al, 'Designing Difference in Difference Studies: Best Practices for Public Health Policy Research', Annual Review of Public Health 39 (2018): 453-469; John Snow, On the Mode of Communication of Cholera London (John Churchill, London, 1855).
155. Simon Burgess, Deborah Wilson, Jack Worth, A natural experiment in school accountability: The impact of school performance information on pupil progress and sorting (The Centre For Market And Public Organisation, Bristol University, Brisol, October 2010) http://www.bristol.ac.uk/media-library/sites/cmpo/migrated/documents/wp246.pdf p1.
156. Ibid
157. Coady Wing et al, 'Designing Difference in Difference Studies: Best Practices for Public Health Policy Research', Annual Review of Public Health 39 (2018): 453-469.
158. Ibid.
159. Sanjay Basu, Ankita Meghani, Arjumand Siddiqi, 'Evaluating the Health Impact of Large-Scale Public Policy Changes: Classical and Novel Approaches', Annual Review of Public Health 38 (2017): 351-370. doi: 10.1146/annurev-publhealth-031816-044208.
160. Margaret Handley et al, 'Selecting and Improving Quasi-Experimental Designs in Effectiveness and Implementation Research', Annual Review of Public Health 39:1 (2018): 5-25. doi: 10.1146/annurev-publhealth-040617-014128.
161. Ibid.
162. Coady Wing et al, 'Designing Difference in Difference Studies: Best Practices for Public Health Policy Research', Annual Review of Public Health 39 (2018): 453-469.
163. Note that averages are 'weighted', which affects how they are combined. See Robert McClelland and Sarah Gault, The Synthetic Control Method as a Tool to Understand State Policy' (Urban Institute, Washington DC, March 2017).
164. Alberto Abadie and Javier Gardeazabal, 'The Economic Costs of Conflict: A Case Study of the Basque Country', American Economic Review 93, no. 1 (March 2003): 113-132. doi: 10.1257/000282803321455188.
165. Robert McClelland and Sarah Gault, The Synthetic Control Method as a Tool to Understand State Policy' (Urban Institute, Washington DC, March 2017).
166. Ibid.
167. Alberto Abadie, Alexis Diamond, Jens Hainmueller, 'Synthetic Control Methods for Comparative Case Studies', Journal of the American Statistical Association 105, No. 490, Applications and Case Studies (2010).
168. Sanjay Basu, Ankita Meghani, Arjumand Siddiqi, 'Evaluating the Health Impact of Large-Scale Public Policy Changes: Classical and Novel Approaches', Annual Review of Public Health 38 (2017): 351-370. doi: 10.1146/annurev-publhealth-031816-044208.
169. Ibid.
170. Michihito Ando, 'Dreams of Urbanization: Quantitative Case Studies on the Local Impacts of Nuclear Power Facilities Using the Synthetic Control Method', Journal of Urban Economics 85 (January 2015): 68-85.
171. Robert McClelland and Sarah Gault, The Synthetic Control Method as a Tool to Understand State Policy' (Urban Institute, Washington DC, March 2017).
172. Ibid.
173. Adapted from Hank Jenkins-Smith et al, Quantitative Research Methods for Political Science, Public Policy and Public Administration: 4th Edition with Applications in R (2017), a free resource available online from multiple sources. See: https://bookdown.org/ripberjt/qrmbook
174. Julie Reed, Alan Card, 'The problem with Plan-Do-Study-Act cycles', BMJ Quality & Safety 25 (2016):147-152; Ann Mei Chang, Lean Impact: How to Innovate for Radically Greater Social Good (John Wiley & Sons, 2018).
175. You can find out more on Dartington's approach and its work with Chance UK, a mentoring charity, on its website in this blog article: Deon Simpson, 'To help London's children, one mentoring service is improving itself first', Dartingon Service Design Lab blog, 26 July 2019 https://www.dartington.org.uk/ourblog/tohelplondonchildren
176. Find out more at: https://developingchild.harvard.edu/science/national-scientific-council-on-the-developing-child
177. Maia Barrow et al, 'Improved cognitive flexibility after a structured play intervention with a high-risk sample of preschoolers', (2015) doi: 10.13140/RG.2.1.2093.2966.
178. Find out more online about People Powered Results: https://www.nesta.org.uk/project/people-powered-results
179. Interview with Brad Dudding, formerly Chief Impact Officer at the Centre for Employment Opportunities.
180. Dartington Service Design Lab and Family Nurse Partnership National Unit, FNP Adapt Interim Report (Interim report of the FNP ADAPT project, London, 2018). http://fnp.nhs.uk/media/1246/fnp-adapt-imterim-report.pdf
181. Tom McBride, 'What Happens When the Evidence Is Mixed?' Early Intervention Foundation blog, 29 March 2018 https://www.eif.org.uk/blog/what-happens-when-the-evidence-is-mixed
182. Michael Robling et al, 'Effectiveness of a Nurse-Led Intensive Home-Visitation Programme for First-Time Teenage Mothers (Building Blocks): A Pragmatic Randomised Controlled Trial,' The Lancet 387, no. 10014 (2016): 146-55. See also the process evaluation: J. Sanders et al, 'Implementation of the Family Nurse Partnership programme in England: Experiences of key health professionals explored through trial parallel process evaluation', BMC Nursing 18, no. 13 (2019) doi:10.1186/s12912-019-0338-y Available Open Access: https://bmcnurs.biomedcentral.com/articles/10.1186/s12912-019-0338-y
183. See 'FNP Next Steps: Developing and Improving the Programme', Family Nurse Partnership: https://fnp.nhs.uk/fnp-next-steps
184. Dr Nick Axford, 'Applying "Dark Logic" in Practice', Alliance for Useful Evidence blog, 29 November 2017 https://www.alliance4usefulevidence.org/applying-dark-logic-in-practice
185. Dartington Service Design Lab and Family Nurse Partnership National Unit, FNP Adapt Interim Report (Interim report of the FNP ADAPT project, London, 2018). http://fnp.nhs.uk/media/1246/fnp-adapt-imterim-report.pdf
186. Interview with Tim Hobbs, Director of Dartington Service Design Lab.
187. Interview with Jon Brown, Head of Impact at NSPCC.
188. See Geoff Mulgan, 'Teach a Society to Fish: The twin revolutions that could transform social science', Nesta blog, 22 September 2017 https://www.nesta.org.uk/blog/teach-a-society-to-fish-the-twin-revolutions-that-could-transform-social-science; and Geoff Mulgan, 'Social Science and Intelligence Design', Nesta blog, 25 January 2019 https://www.nesta.org.uk/blog/social-science-and-intelligence-design
189. See our list of useful resources in Section 4.
190. Nesta, 'Prototyping' in 20 Tools for Innovation in Government (Nesta, London, September 2019) https://www.nesta.org.uk/report/20-tools-innovating-government
191. Nesta, Prototyping in Public Services (Nesta, London, November 2011) https://www.nesta.org.uk/report/prototyping-in-public-services
192. Bella Martin and Bruce Hanington, Universal Methods of Design: 100 Ways to Research Complex Problems, Develop Innovative Ideas, and Design Effective Solutions (Gloucester, MA: Rockport Publishers Inc, February 2012); Camilla Buchanan, 'Prototyping for Policy', Open Policy blog, 27 November 2018 https://openpolicy.blog.gov.uk/2018/11/27/prototyping-for-policy
193. For guides on roleplaying, creating simulations and other ideas, see our Designing for Public Services Guide in Section 4: Useful resources.
194. Camilla Buchanan, 'Prototyping for Policy', Open Policy blog, 27 November 2018 https://openpolicy.blog.gov.uk/2018/11/27/prototyping-for-policy
195. GOV.UK Prototype Kit: https://govuk-prototype-kit.herokuapp.com/docs
196. James K Esser, 'Alive and Well after 25 Years: A Review of Groupthink Research, Organizational Behavior and Human Decision Processes', 73, no. 2-3 (1998): 116-141. ISSN 0749-5978 https://doi.org/10.1006/obhd.1998.2758
197. Ian Sanderson, 'Evaluation, Policy Learning and Evidence-Based Policy Making', Public Administration 80, no. 1 (2002) 1-22. doi: 10.1111/1467-9299.00292.
198. Bella Martin and Bruce Hanington, Universal Methods of Design: 100 Ways to Research Complex Problems, Develop Innovative Ideas, and Design Effective Solutions (Gloucester, MA: Rockport Publishers Inc, February 2012).
199. Bruce Hanington, 'Relevant and Rigorous: Human-Centered Research and Design Education', Design Issues 26, no. 3 (Summer 2010): 18-26. Retrieved from: http://www.jstor.org/stable/20749956
200. Peter H. Rossi, 'The Iron Law of Evaluation and Other Metallic Rules'. Research in Social Problems and Public Policy 4 (1987): 3-20.
201. Andrew Leigh, Randomistas: How Radical Researchers Changed Our World (Yale University Press, 2018).
202. Kathryn Oliver and Annette Boaz, 'Transforming evidence for policy and practice: Creating space for new conversations', Palgrave Commununications 5, no. 60 (2019) doi: 10.1057/s41599-019-0266-1 Open Access at: https://www.nature.com/articles/s41599-019-0266-1
203. Nesta, 'The UK What Works Network' in 20 Tools for Innovation in Government (Nesta, London, September 2019) https://www.nesta.org.uk/report/20-tools-innovating-government
204. See Jill Rutter, 'Troubled Familiies Takeaways', Institute for Government blog, 18 October 2016 https://www.instituteforgovernment.org.uk/blog/troubled-families-takeaways
205. See the 2019 evaluation by Ministry of Housing, Communities and Local Government, National evaluation of the Troubled Families Programme 2015-2020: Findings (Ministry of Housing, Communities and Local Government, 2019).
206. Cambridge Online Dictionary definition of innovation at time of writing.
207. See European Commission website on Social Innovation: https://ec.europa.eu/growth/industry/innovation/policy/social_en
208. Judea Pearl and Elias Bareinboim, Transportability of Causal and Statistical Relations: A Formal Approach, (Proceedings of the 25th AAAI Conference on Artificial Intelligence, August 7-11, 2011, San Francisco, CA).
209. Michelle Jackson and D.R. Cox, 'The Principles of Experimental Design and Their Application in Sociology', Annual Review of Sociology 39 (2013): 27-49. Retrieved from http://www.jstor.org/stable/43049624
210. Matthew Salganik, Bit by Bit: Social Research in the Digital Age (Princeton, NJ: Princeton University Press, 2017) p176.
211. Ben Styles and Dougal Hutchinson, A Guide to Running Randomised Controlled Trials for Educational Researchers (Slough: NFER, 2010).
212. Kelsey Piper, 'The Nobel went to economists who changed how we help the poor. But some critics oppose their big idea', Vox, 11 December 2019 https://www.vox.com/future-perfect/2019/12/11/20938915/nobel-prize-economics-banerjee-duflo-kremer-rcts

About Nesta

Nesta is an innovation foundation. For us, innovation means turning bold ideas into reality and changing lives for the better.

We use our expertise, skills and funding in areas where there are big challenges facing society.

Nesta is based in the UK and supported by a financial endowment. We work with partners around the globe to bring bold ideas to life to change the world for good.

www.nesta.org.uk

58 Victoria Embankment London EC4Y 0DS

+44 (0)20 7438 2500 [email protected] @nesta_uk www.facebook.com/nesta.uk www.nesta.org.uk

Nesta is a registered charity in England and Wales with company number 7706036 and charity number 1144091. Registered as a charity in Scotland number SC042833. Registered office: 58 Victoria Embankment, London, EC4Y ODS.