Kids Company: what are the lessons?
Now, more than ever, charities of every size are expected to provide evidence of impact. Tony Munton suggests that a system of peer review, and better use of existing research, would help charities generate cost-effective, credible evidence.
Commentators have attributed the high profile demise of Kids Company to several factors. Some, (see TheTelegraph and The Spectator), have suggested that donors withdrew financial support because they lost confidence in the charity’s capacity to deliver effective support for children. That loss of confidence was in part due to the perceived failure of Kids Company to provide robust evidence of impact.
Caroline Fiennes, Director of Giving Evidence, has opined that this failure to collect and disseminate evidence of impact is a characteristic shared by too many charities.
Donors are upping their demands for data. They want charities to provide evidence that supports transparency, ensures accountability, and promotes effective practice. Organisations unable to comply are going to find it increasing difficult to source funding.
That is not to say that charities never evaluate their services. Most donors require some evidence of how their money is spent. Unfortunately, that does not always translate into robust impact evaluation. A report from the Paul Hamlyn Foundation assessed the quality of 120 reports from their grant holders. It labelled just one third as ‘good’, over half as ‘average’, and the rest as ‘poor’.
Donors are upping their demands for data
What is the solution? Some have suggested that poor quality evidence is a consequence of charities funding their own evaluation research. Conflict of interest militates against validity. They suggest that funding should come from elsewhere. Alternatively, they suggest that charities should simply not do impact evaluations at all because they cannot afford to fund the kind of robust research required to provide real evidence of impact and good practice.
Both proposals are short on credibility. Of course organisations should be encouraged to monitor their performance; in terms of transparency and accountability, to do less would be positively irresponsible. And who, other than themselves, would have the financial incentive to fund performance monitoring?
For a solution to improving the independence, and by implication, the quality of evaluations, we need look no further than the process of peer review widely used in academic research. To convince donors that evidence of impact is valid and robust, charities should routinely submit research reports they have either produced or commissioned to independent peer review. The cost would be marginal, the effect disproportionate.
On the issue of affordability, rather than not spending on research, charities might think more strategically about the nature of the research they need. Robust impact evaluation can be expensive. Kids Company reportedly paid the London School of Economics nearly £40,000 for work dismissed by some as lacking rigor.
The solution, at least as charities take the first steps towards compiling a credible evidence base, lies in commissioning robust evidence reviews rather than primary research. The principle, captured eloquently by Caroline Fiennes, is 'cite research, don’t necessarily produce it’. The advantages are reduced cost (good quality evidence reviews are typically cheaper than impact evaluations), and more reliable conclusions (based on multiple studies conducted across different contexts and over several years). Kids Company was itself committed to using interventions that were based on research. One of their principles was for:
“Active use of cutting-edge research combining areas that are usually apart and separated by rigid academic boundaries: interventions are informed by research on neurological and immunological changes linked to developmental adversity, psychotherapeutic theory and the social world of children and young people.” Kids Company: A diagnosis of the organisation and its interventions Final Report, LSE September 2013. p.19
The key lesson from the Kids Company collapse is that in the search for transparency and accountability, philanthropic organisations and individuals are becoming increasingly research literate. Charities can no longer ignore demands for robust evidence of effectiveness. Given the plethora of small organisations that make up the sector (over 60,000 providing support for children and young people in the UK) meeting those demands effectively is likely to require co-ordinated action.