Policy Innovation Blog

Understanding what is working in the real world: evaluating Project Oracle

Ruth Puttick - 04.10.2012

At Nesta we are interested in exploring how rigorous evidence can be better used to drive innovation. Part of this is being able to identify the most effective policies and programmes. Yet the thing we find striking is how little is known about the actual mechanisms that can effectively ensure evidence is useful and useable for decision makers. In other words, what works in helping identify what is working?

To begin exploring these issues we are evaluating Project Oracle, an innovative city-wide evidence generation campaign. 

Over the past couple of years Nesta has published a number of reports and hosted a number of events to explore the infrastructural changes needed to improve the generation and use of evidence. Our work has helped to inform the Open Public Services White Paper and the more recent Civil Service Reform White Paper, both of which contained commitments to explore new evidence centres, in short hand, the equivalent of a 'NICE for social policy'. We have previously written what such as NICE for Social Policy could look like.  

As part of this we have looked at what evidence centres and infrastructures already exist. Although it is relatively easy to see the areas covered or the size and structure of particular institutions, it is much harder to gauge how successfully they actually are at influencing decision making. A striking thing often encountered is the lack of evaluations or evidence about these 'evaluators' or 'evidence centres'. This raises the question, how do we know what is an effective way linking rigorous evidence into the decision making process?

To begin answering these questions and help inform the current work to develop new evidence centres we are evaluating Project Oracle, the only city-wide evidence generation campaign of its kind anywhere in the world. Already Project Oracle has demonstrated that it is possible to bring providers of youth services - many of which are small and have struggled to evaluate their work - in line with academically rigorous standards of evidence.  It does this by helping make it easy for providers, supporting charities and social enterprise to build the evidence behind their approach at a pace an in a way which is appropriate to them.

Project Oracle has now moved into a new phase.  Over the next 12 months an experienced consortium will lead on an ambitious programme to improve the evidence behind more providers of youth services in London, and strive to ensure there is demand for this information. To do this Project Oracle will work directly with decision makers, such as local authorities, to help make evidence a central consideration of the commissioning and decision making process.

We are interested in the impact that Project Oracle has overall to improve youth service outcomes, and through testing the different mechanisms that are being trialed to help achieve this. These mechanisms consist of training and involving commissioners in the research process, matchmaking the availability of academic resource in London with organizations needing support to evaluate their work, and through running evidence competitions with prizes available to those who move up through the Oracle Standards of Evidence. Details of these and other workstreams are available on the Project Oracle website.

We recognize that the generation of evidence doesn't simply translate into decision making in a linear fashion. We are also aware that decision makers are influenced by a plethora of factors besides evidence. Yet we hope that this evaluation will help to develop our understanding of how to make evidence available, useable and critically, used.

We are publishing our initial findings from the evaluation of Project Oracle in Spring 2013, with final results due later in the year. To stay up to date with developments sign up to the Nesta Newsletter.

If you have any questions or comments about Nesta's evidence work programme please contact Ruth Puttick, Ruth.Puttick@Nesta.org.uk

Further details about Project Oracle are available on the new website, www.project-oracle.com