Alliance for Useful Evidence

Using evidence effectively: a ‘top ten’ list of barriers

01.11.2012

The following is a guest blog by Rosie Chadwick, Director of Public Affairs, Policy and Innovation, Catch22

What gets in the way of organisations using evidence effectively? Based on fifteen years' involvement in neighbourhood crime reduction, youth crime prevention and youth development programmes, here's a suggested 'top ten' list of impediments that I think we need to overcome.

1. Short-termism: too many initiatives that are here today and gone tomorrow, leaving insufficient time, money and motivation to put data collection systems in place, or - where they are in place - to capture trend data. In this context it's worth reminding ourselves that the evidence base for Family Nurse Partnerships was more than two decades in the making.

2. The fact that, even with longer-term programmes, outcomes may only be apparent long after an organisation has ceased working with a person or family, with whom they may no longer be in touch.

3. The lack of interest funders sometimes show in 'intermediate' indicators - the small steps along the way that signal progress being made. The measurement system developed by Substance for the Home Office's Positive Futures programme was an honourable exception to this, tracking young people's movements along a scale from disengagement to autonomy. (In keeping with real life, there was also recognition that this movement could be 'down' as well as 'up.')

4. The vexed question of attribution, an issue that's especially problematic for work in community settings which may be awash with interventions.

5. Widespread confusion about what standards of evidence are good enough for what purpose.

6. the lack of agreed 'industry standard' counting rules, hindering robust benchmarking of different organisations and approaches.

7. In some cases, protectionism - hedging interventions about with licence fees and requirements that hinder their wider adoption. As somebody put it on the radio recently, where would we be today if Tim Berners-Lee had been proprietorial about his invention of the internet?

Getting down to the rub', I think that in the youth arena certainly we also need to acknowledge:

8. a historically weak culture of performance management;

9. a related lack of skills in data analysis; and

10. a tendency to see performance reporting and review as divorced from service delivery - rather than fundamental to it. Asked to capture performance data, too often the reply that comes back is 'I'm too busy delivering the service.'  Yet as a monitoring and evaluation specialist once put it to me very powerfully: 'how can you possibly be providing a good service when you don't know how effective you are being?'

Let's hope that items 8-10 will feature in the recently announced review of skills in the voluntary sector, being led by Dame Mary Marsh, though I don't think these deficits are the sole preserve of the voluntary sector. 

Dare I say it, a silver lining of austerity may also be the stronger use of evidence, with a blunt message that 'if you can't prove it, we won't pay for it' allied increasingly to payment by results. I don't think it's any coincidence that one of, if not the best, example of Catch22's  use of evidence is our Payments By Results-based contract at Doncaster Prison.  If we can frame Payments By Results targets in ways that benefit the most marginalised in our society then we will be well on the way to the effective use of evidence. 



About Rosie Chadwick: Formerly a civil servant, Rosie has spent the last 15 years in policy, development, operational and strategic management roles at Crime Concern then Catch22.

(The views in this blog are the author's own and not necessarily those of the Alliance for Useful Evidence)