About The Alliance for Useful Evidence

The Alliance for Useful Evidence is a network, hosted by Nesta, that champions the smarter use of evidence in social policy and practice. We do this through advocacy, convening events, sharing ideas and resources, and supporting individuals and organisations through advice and training. We promote our work through our network of more than 3,700 individuals from across government, universities, charities, businesses, and local authorities in the UK and internationally. Anyone can join the Alliance network at no cost.

To sign up please visit: www.alliance4usefulevidence.org/join

We are funded by the Big Lottery Fund, the Economic and Social Research Council and Nesta.

About the Wales Centre for Public Policy

The Wales Centre for Public Policy was established in October 2017. Its mission is to improve policymaking and public services by supporting ministers and public services to access rigorous independent evidence about what works. The Centre collaborates with leading researchers and other policy experts to synthesise and mobilise existing evidence and identify gaps where there is a need to generate new knowledge. The Centre is independent of government but works closely with policymakers and practitioners to develop fresh thinking about how to address strategic challenges in health and social care, education, housing, the economy and other devolved responsibilities.

For further information please visit our website at www.wcpp.org.uk

We are funded by the Economic and Social Research Council, Welsh Government, and Cardiff University.

About Nesta

Nesta is a global innovation foundation. We back new ideas to tackle the big challenges of our time.

We use our knowledge, networks, funding and skills – working in partnership with others, including governments, businesses and charities. We are a UK charity but work all over the world, supported by a financial endowment.

To find out more visit www.nesta.org.uk

If you'd like this publication in an alternative format such as Braille, large print or audio, please contact us at: [email protected]

Acknowledgements

I am hugely grateful to Tony Munton, Helen Mthiyane and Jane O'Brien for their hard work on earlier drafts of this report. Thank you also to each of the charities featured in this report for their willingness to give honest and insightful advice to other charities, and their help in writing the case studies: Melissa Henry at Common Cause; Alex Smith at the Cares Family; Harriet Laurie at TheHorseCourse; Sam Austin at Llamau; Jamila Boughelaf at Mentor UK; Rob Cope at Remember a Charity; and Linda Hutton and Rob Gowans at Citizens Advice Scotland. Thanks are also due to colleagues Steve Martin, Jonathan Breckon, Pippa Coutts, and Steve Wise for their support in writing and publishing this report.

All errors and omissions remain the author's own.

About the author

Emma Taylor-Collins is Senior Research Officer at the Alliance for Useful Evidence and the Wales Centre for Public Policy, leading research and engagement projects that champion the use of evidence in social policy and practice, with a focus on Wales. Emma's background is in the charity sector, and she is doing a PhD part-time on youth volunteering at the University of Birmingham's Third Sector Research Centre.

Introduction

The Alliance for Useful Evidence and the Wales Centre for Public Policy champion and promote the smarter use of evidence in social policy and practice. We do this because, although billions of pounds are spent on social policy programmes, all too often this is done without any rigorous evidence that policies actually work. And it isn't simply because the evidence isn't available – often it is, but it isn't always used or automatically integrated into decision-making.

To date, the Alliance and Wales Centre for Public Policy have done a lot of work with decision-makers in central and local government and alongside the UK's What Works network to make sure that public services commissioners and practitioners have access to independent, high quality, accessible evidence to inform their work. The Alliance approach includes making the case for evidence-informed policy and decision-making through training, seminars, events, publications and developing our network of individuals and organisations who share our desire for the better use of evidence in social policy and practice. The Wales Centre for Public Policy collaborates with leading policy experts to provide ministers, the Civil Service, and public services with high quality evidence and independent advice that helps them to improve policy decisions and outcomes. The Centre also undertakes research to advance understanding of the role which evidence can play in supporting better policy making and public service delivery.

Many of the Alliance's 3,700 network members come from the charity sector, as do several of our Evidence Champions. And we're not alone in thinking that it's vital for charities to use evidence effectively. As the Select Committee on Charities' Stronger Charities for a Stronger Society report stated:

We are living through a time of profound economic, social and technological change and the environment in which charities are working is altering dramatically. These changes have posed new challenges for charities, resulted in some high-profile failures, and led to greater scrutiny of the sector than ever before.¹

Partly as a result of this, charities are also under increasing pressure to measure the impact of what they do – it's part of the deal when they're delivering public services, as charities increasingly are, and often written into funding requirements.² As such, in recent years there has been a proliferation of reports, toolkits and guidance designed to help charities measure their impact, from NPC's Inspiring Impact programme, to Evaluation Support Scotland, to the Centre for Youth Impact's resource hub.

While there's plenty of advice on impact evaluation, there's a lot less to be found on how charities might use evidence in other ways, such as when they're first getting started, when they're scaling up, or when they're influencing policy. In this report we aim to fill that gap, exploring ways that charities can use evidence to improve the way they work.

Who is this report for?

The report is aimed at anyone working in the charity sector – whether a frontline practitioner dealing with clients, a CEO, or a trustee. Previous research tells us that the messenger is important in encouraging evidence take-up, and that people of influence can help persuade peers of the value of evidence.³ As such, we've spoken to charities working on a range of causes from across the UK, showcasing seven charities which have used different types of evidence to make their organisations more effective, from the early days of setting up a charity, to improving practice, to changing the environment in which they operate.

What do we mean by evidence?

We take a deliberately inclusive view of evidence, from research and evaluation studies to expert knowledge and stakeholder consultations. For us, good – and useful - evidence is robust and appropriate for the issue at hand.

Importantly, evidence is more than just data. Charities often collect a great deal of monitoring data - the number of people who use a service, where they live, what they thought of the service they received, and so on. But data like these give no information about what actually works, what has an impact, or insights into cost effectiveness. The key is turning such monitoring data into evidence.

Quantitative data is something that has been counted or measured – it expresses quantity, amount or range. Qualitative data is gathered from opinions, views, notes or observations. Often it's words, expressions, drawings or symbols. Neither form of data is better than the other - they have different uses - and in their raw form, neither is particularly useful.

For example, say a charity is collecting data on how many people use their services, how often, who they are, and what they think of it. Data alone won't tell us much. The starting point for interpreting data is to ask questions of it, and to code it and analyse it accordingly. This produces information, and once we have information we have something useful; information can tell us something or shed light on a question. It's at this point, when information is giving insight and answering a question, that we have evidence.

For example, imagine a foodbank wanted to know if it was reaching local, young, unemployed women.

It collects information on users as they come in and use the service. Volunteers gather standard demographic information including names, addresses, age, ethnicity, employment status, if they have an illness or disability, if they have children or live alone, and so on. It's only when we organise and analyse the data that we can start to answer that initial question.

For charities, good evidence can provide insight into what Geoff Mulgan describes as the 'six Ws': what works, for whom, when, where, and with whom.⁴ And as our Using Research Evidence: A Practice Guide states:

Evidence can make organisations more effective. From more persuasive campaigning to winning grant funding, and from developing a board's decision-making abilities to making sure programmes deliver results – evidence can bolster your work. It doesn't matter if you are a small voluntary organisation or a large government department.⁵

In this report, we focus on research evidence - 'evidence that is underpinned by research rather than expert opinion or stakeholder views'⁶ – because, while 'research is only one sort of evidence, [it] has the advantages of greater rigour, relevance and independence when compared to other types of evidence'.⁷

Case studies

Each case study in this report offers advice in the form of short takeaways from the charities themselves about how you might do something similar.

GETTING OFF THE GROUND

Setting up a charity

Identifying need

Getting commissioned

BETTER PRACTICE

Improving staff training

Designing new services

CHANGING THE LANDSCAPE

Campaigning

Informing policy

Setting up a charity: Common Cause

Most charities are set up to address a particular need; evidence use is what comes next, rather than what comes first. What's less common is a charity that grows out of an evidence base - like Common Cause Foundation, based in Wales but working across the UK, which aims to place values that prioritise community, environment and equality at the heart of cultural, political, and civic institutions.

For years it was assumed that people could be persuaded to be pro-environment if they were told what was in it for them, but Common Cause argued that this wasn't necessarily the case.⁸ Social psychology research by academics at Cardiff University and elsewhere has shown that people are often motivated to do good things when campaigns appeal to their 'intrinsic' values, or values that connect with concern for other people, community, and the wider world. What's more, appealing to selfish motives can actually undermine people's motivation to engage in environmental behaviours.⁹

The research also identified a values 'perception gap': people typically underestimate the importance that fellow citizens place on 'intrinsic' values, and typically overestimate the importance that other people place on values of financial success or public image.¹⁰ This 'perception gap' is associated with people feeling less positive about civic engagement, and experiencing greater social alienation.¹¹

Probably the most significant outcome of this evidence was the establishment of Common Cause Foundation. Originally a programme within WWF, in 2015 it became a not-for-profit in its own right, working with institutions across a range of sectors to put into practice what the evidence on values shows. As Melissa Henry, Director at Common Cause Foundation, puts it:

"Evidence gives us permission to challenge business as usual. It confirms it is possible to do things differently, drawing on a robust evidence base".

Common Cause Foundation has developed a series of practical resources and workshops for campaigners, communicators, fundraisers and advocates to develop a values-based approach to engagement. More recently, a partnership with Manchester Museum saw Common Cause Foundation introduce a raft of changes in the museum to invoke in visitors, volunteers, and staff the values that promote social justice, equality and care for our natural environment.

As Tom Crompton, Director at Common Cause Foundation, says,

"This has enabled the development of ways to engage and model 'intrinsic' values that are transferable to a range of settings. We're now interested in coalescing networks of organisations like Manchester Museum that are able to help normalise 'intrinsic' values as part of people's everyday identity, celebrating the values that bring people together and underpin citizens' demand for action on social and environmental issues."

Next, Common Cause plans to develop evidence-informed training to help other organisations put into practice what the evidence shows on values and perceptions.

Takeaways:

• Assumptions made without evidence to back them up can not only lead to ineffective work, but can actually undermine the very issue you're trying to address.
• Without robust evidence, it can be hard to challenge 'business as usual'. And yet as NPC has recently argued, business as usual will no longer cut it in a tough funding environment - charities have to think about how they can achieve the greatest impact, making the most of available resources, innovating and transforming their work.¹² As Common Cause shows, evidence can help charities do just that.

Identifying need: Cares Family

A common criticism made about new charities is that they are set up with no evidence that there is demonstrable need.¹³ Not so The Cares Family, a group of charities (currently North London Cares, South London Cares and Manchester Cares) aiming to reduce loneliness and social isolation among older and younger people alike in rapidly changing cities.

It's taken a while for politicians in the UK to take loneliness seriously, with 2018 marking the year the UK got its first 'Minister for Loneliness' and the Scottish Government set out a national strategy for tackling loneliness and isolation - said to be the first of its kind anywhere in the world¹⁴ - but loneliness has been recognised as an important issue for a lot longer than that. Not least by The Cares Family, founded by Alex Smith in 2011 after he made friends with an older man in his neighbourhood who had become isolated from the people and places around him. While personal experience provided the inspiration, before setting up the charity Alex wanted to check whether the need he'd identified in his own community existed on a wider scale. Alex made use of the growing Campaign to End Loneliness' research hub, which curates evidence on loneliness from the UK and around the world through accepted and trusted measures, and found that loneliness is bad for people's mental and physical health. As Alex says,

"The Cares Family have always sought good evidence. It can teach us so much about how we operate, the people we are reaching and the impact we are having on their lives – and of course it's vital for building a coalition of supporters to understand our vision and to fund our work."

As The Cares Family expanded it drew on solid existing evidence demonstrating that loneliness is as bad for people's health as smoking.¹⁵ It had powerful messages about the number of older people (two in five)¹⁶ who said the TV was their main form of company, and demonstrated the effects of loneliness on public services.¹⁷

The Cares Family's commitment to this approach saw it contribute to the growing evidence base around loneliness by conducting its own evaluations of its work, too – one of which was funded as part of the Nesta and Cabinet Office's Centre for Social Action Innovation Fund¹⁸ which highlighted some surprising findings that revolutionised its funding model.

Previously, the group had generated almost all its income from grants from trusts and foundations. But its evaluations found that there was a real appetite among young people and other local partners to help fundraise. Now, money raised through those community networks has grown from 0 per cent of overall income in 2013-2014 to 46 per cent of income in 2016-2017. And these activities don't just raise funds - they bring people together to address the very issue of social isolation and loneliness that the Cares Family aims to tackle. Importantly, Alex is clear that:

"We don't prioritise data over people. We are always conscious of building an evidence base, through curation and creation, that directly measures our progress on our unique objectives, rather than just on abstract numbers which can sometimes mislead. That's the number one concern - good evidence that puts people first - and as we experiment with new evaluation models in the future, it's something we're looking forward to learning more about".¹⁹

There are now 5,000 young professionals and around 4,000 older people who are part of North London Cares, South London Cares and Manchester Cares. The group is planning to expand further in the coming years, continuing to make good use of evidence as they scale up.

Takeaways:

• There is a wealth of robust, often freely-available evidence in the public domain, on a whole host of social issues. Before diving into your own research, and certainly before setting up your own charity, think about how you can make use of what's already out there. If you need help wading through it, research hubs like the Campaign to End Loneliness can help synthesise.
• Impact evaluations can tell you more than you bargained for. Be open minded about what you might find, think creatively about what you're measuring, and be prepared to think a bit differently about how evidence could change the way you do business, not just the way you deliver your programmes.

Getting commissioned: TheHorseCourse

When money's tight, evidence can feel like a nice to have rather than a means to survival. But for TheHorseCourse, it was robust evidence - combined with the positive relationships it built with commissioners – that ultimately prevented it from folding.

TheHorseCourse aims to reduce reoffending and improve attendance and engagement at school through intensive programmes which "fill the gap where talking isn't working", as Harriet Laurie, CEO and Founder, describes it. Working in prisons and with children and families, this small, Dorset-based charity runs activities involving horses by referral for people facing multiple challenges - from poor mental health to drug and alcohol misuse and homelessness.

Harriet has made evidence a seamless part of day-to-day practice at TheHorseCourse - from having Bournemouth University undergraduates, PhD students, and professors in to work on various projects, to a partnership with NPC on a theory of change and contribution analysis, and work with the NCVO Charities Evaluation Service (NCVO CES). Harriet argues that this:

"Adds richness to our work, always. Evidence is disruptive, and that's a good thing: it provokes discussions, raises questions, and brings a fresh perspective to what we do".

The support that Harriet's been able to attract from academics and other research institutions - to the extent that Bournemouth University is now calling for funding for a randomised controlled trial (RCT) of TheHorseCourse - means that most of this research has been done for free or at low cost. One piece of research in particular, conducted with help from NCVO CES, helped TheHorseCourse out of a difficult funding situation. TheHorseCourse needed to show that they could work in community settings as well as prison, so they offered 20 free places on a course to young people known to the local children's services, and evaluated it, producing the Youth Outcomes Summary (2015).

"We got to a point where we had about three months' funding left and we knew we had to reach commissioners if we were to survive. What we got from NCVO CES was about a day's worth of time, spread across a few weeks, of bespoke guidance and advice from an expert CES consultant".

The CES consultant ensured that the work was robustly designed and that the findings were correctly interpreted. Harriet says that:

"What I could bring was closeness to the data and knowledge of what was possible and realistic for us to do; what the consultant brought was the kind of knowledge we just didn't have in-house at the time, it being just me and a few freelancers at that point."

The aim of the work was to get commissioned - and that's what they got. Harriet took the Youth Outcomes report to Children in Need and to the Troubled Families team at the local authority, and used it to get a mixture of charitable funding and commissioning for TheHorseCourse to work with young people and families in Dorset. Key to Harriet's success was combining hard evidence of outcomes with stories, and building a good relationship with commissioners. She invited commissioners to see TheHorseCourse in action, first hand. As Harriet says:

"Giving people a bunch of research reports and leaving it at that doesn't automatically make doors or cheque books open. It doesn't matter how good the evidence is – if you don't invest in the people side, telling stories, engaging people on an emotional level, they aren't going to take much notice of it. That's what gets commissioners involved in what you do; the evidence makes them feel comfortable about ultimately making that decision to commission your service."

Now with a team of five full-time staff and 20 volunteers, TheHorseCourse has broadened its reach to work with 500 beneficiaries a year. It's now hoping to get an RCT funded so that it can further demonstrate the difference it's making to young people's lives.

Takeaways

• Good evidence needs to be communicated in the right way to the right audiences if it is to make a difference. Any investment in evidence should also be accompanied by investment in relationship building to help maximise its impact.
• With academics under increasing pressure to demonstrate the impact of their work on society as part of the Research Excellence Framework (REF), and charities facing significant budget restraints and a more competitive funding environment, research collaborations between universities and charities can be mutually beneficial and can sometimes provide pro bono research support. Charities benefit from robust, independent research expertise and universities benefit from access to groups and individuals that might otherwise be difficult to engage in research.
• Even when it isn't free, research expertise doesn't have to be prohibitively expensive – a relatively small investment in advice can help to develop robust evidence, helping to secure new sources of funding and provide the credibility and trustworthiness needed to impress commissioners.

Improving staff training: Llamau

Cardiff-based Llamau aims to eradicate homelessness for young people and women in Wales. It provides safe accommodation, preventative services, employment opportunities, skills training, and mental health support across Wales. Llamau also leads the UK-wide End Youth Homelessness initiative (a 2016 New Radical), and part of Nesta and Welsh Government's Innovate to Save project to test a new, sustainable social investment funding model.

Sam Austin, Deputy CEO at Llamau, talks about how:

"we've always been interested in research – one of our core values is about learning – both to help us improve as an organisation, but also to help us stand out in what is an increasingly competitive tendering environment for charities like ours".

This commitment to research has led to a longstanding partnership between Llamau and Cardiff University. Back in 2012, reading about Cardiff University's research on homelessness in the USA, Llamau CEO Frances Beecher got in touch with Cardiff University to see if they could shift their focus a little closer to home. It was the beginning of a partnership that has seen 12 undergraduate Psychology student placements with Llamau, two PhDs on Llamau through to completion, and a recent deal to bring in a Consultant Clinical Psychologist and Research Assistant from Cardiff University to work on a new project with them up to 2021.

Completed in 2014 as a Knowledge Transfer Partnership,²⁰ the first of these PhDs,²¹ on mental health and youth homelessness, won the Social Innovation Award at Cardiff University's 2015 Innovation and Impact Awards. This qualitative longitudinal study involved interviews with 121 young people aged 16-24 who were originally housed in Llamau's temporary accommodation. Ninety of these young people were interviewed again either 8-12 months later or 16-24 months later. The study found that these young people were significantly more likely to have a mental health problem than the age-matched general population, and that access to mental health services for young people experiencing homelessness was worryingly low. As a result of this research, in 2013 Llamau introduced a screening tool to identify mental health problems at an early stage.

The research influenced Llamau's work in three main ways. First, it changed the way it trains its mental health workers, who are now better equipped to support young people to access the healthcare they need. Second, Llamau used the findings to demonstrate to the then Minister for Health and Social Service in Wales²² the link between mental health and homelessness, leading to collaboration between Llamau and Public Health Wales. Finally, young people using Llamau's services started taking copies of the screening tool to their GPs, and GPs have since approached Llamau about using the tool in their practices.

The second PhD – a mixed methods study involving experiments and a questionnaire related to cognitive functioning with 69 young people experiencing homelessness – was completed in 2018 and the full findings are due to be shared with Llamau soon. Already Llamau has used the interim findings to inform the way that staff work with service users – providing information in small chunks, encouraging slower decision-making, and minimising distractions, for instance - and Sam anticipates that this study too will make a big difference to the way Llamau works.

The key to embedding the learning from this research across the whole organisation, Sam suggests, is a result not only of buy-in from Llamau's board and senior leadership team, who believe in the importance of an evidence-informed approach to their work, but also of bringing the rest of the organisation with them as research projects develop. For each piece of research, presentations are made at all-team meetings (and sent round afterwards) at the start, during and at the end of the project, and integrated into staff inductions and training.

Takeaways

• Research can inform service design, as the Llamau research did with improving staff's mental health awareness and skills to work appropriately with the young people and women they help. It can also have unintended wider effects on systems, as demonstrated by the take-up by GPs.
• Involving staff of all levels in a piece of research from start to finish is key to embedding learning across the whole organisation.
• It's worth exploring university funding and funding council opportunities - many have schemes to encourage applied and collaborative research, which can provide charities with access to evidence at low or no cost.
• The collaborative research conducted with Cardiff University also helped to increase fundraising from new sources - in 2017-18, for example, Llamau was named Cardiff Business School's Charity of the Year.

Designing new services: Mentor UK

While there's a lot that charities can learn from across the UK about what works on a whole host of issues (see the learning coming out of the UK's ten What Works Centres²³), looking further afield, as Mentor UK has done, can also reap rewards. Mentor UK is the UK's leading charity in promoting evidence-informed practice in preventing alcohol and drug misuse among children and young people.

The team from Mentor UK first heard about The Good Behaviour Game (GBG) at a European conference a few years back. Far from being one of those programmes which gets a lot of airtime but isn't actually effective (or is even harmful - you'll no doubt have heard of the controversies surrounding Scared Straight²⁴), GBG, developed in the US in 1969, has over 40 years' worth of robust evidence to back it up. Through longitudinal research and RCTs, GBG claimed powerful and dramatic results from better classroom behaviour to improved academic success, as well as improved mental health and lower substance use outcomes later in life.²⁵ In the US the programme is listed on the National Registry of Evidence Based Programs and Practices, and it's also been tested in a culturally diverse range of settings, from the Netherlands to Chile to Canada.²⁶ Given the strong case for it, Mentor UK could have simply bought this programme off the shelf and rolled it out at scale - but there are examples of evidence-informed programmes that fail to work in different countries or settings.²⁷ How could they be confident the programme would work in the UK?

Although the American Institute for Research (AIR), which has the license to deliver GBG, has tight controls over how the programme should be run, Mentor UK was able to work with AIR to adapt the programme for a UK context. Mentor UK have recently completed RCTs of GBG in northern England and the East Midlands, with support from the University of Manchester and the Educational Endowment Foundation - the What Works Centre for education - and are currently awaiting the results.²⁸ As Mentor UK's Head of Programmes, and Evidence Champion for the Alliance for Useful Evidence, Jamila Boughelaf, points out,

"in current times, with budget cuts and increasing public health priorities, it's important that we invest in activities that have an impact on society".

While testing a programme like GBG can't deliver immediate results, Mentor UK recognises that in the long run, it's more effective - and more cost-effective - to be sure about whether something works before it's rolled out en masse.

Takeaways

• While it might be tempting to assume that a programme which works elsewhere will work in the UK, it's impossible to know that. Although not every charity can win an £800k grant to test their programme before rolling it out, systematic reviews such as those published by the Campbell Collaboration are a good place to start for understanding what works (and what doesn't) in different contexts.
• We're much more likely to hear about programmes that do work than programmes that don't. It's the same in the academic world, where studies showing null effects are less likely to be published.²⁹ If you try out a programme and it doesn't work, it might feel like you've just wasted a whole load of time and money - but if you share that knowledge with other organisations, you can help them make more informed decisions in the future, in turn supporting the charity sector to be that bit better.

Getting politicians to back your cause: Remember a Charity

Even when a way of working can seem intuitively right, it can be difficult to persuade others without an evidence base to back it up. Remember a Charity, based at the Institute of Fundraising, is a network of over 200 charities working together to encourage more people to leave charitable gifts in their wills. Until recently, while Remember a Charity had long believed that solicitors have an important role to play in increasing legacy giving, they didn't have the evidence base to show that. Nor did they know which messages worked best and for whom. Not only did this mean that Remember a Charity had a fairly small pool of solicitors' firms championing their legacy giving cause, but it also meant that it was hard to persuade government, and other stakeholders, to take seriously the importance of solicitors asking clients about legacy giving. As Remember a Charity's Director, Rob Cope, puts it,

"without an independent voice helping us make that case, no one was going to listen to us".

So in 2013 Rob commissioned the Behavioural Insights Team (BIT) to conduct an experiment at a legal call centre,³⁰ testing the effectiveness of different prompts for people to leave money in their will.³¹ They found that certain prompts did influence people's decision to leave money: it was more effective for a solicitor to say 'many of our customers like to leave a gift to charity in their will. Are there any charitable causes that you're passionate about?' than simply 'would you like to leave a charitable gift in your will?'³²

Then, in 2016, Remember a Charity worked with BIT and the University of Bristol on another piece of research - this time, a series of RCTs with eight solicitors' firms – to take the initial study further. They looked at a wider set of prompts, whether these worked on the telephone as well as face-to-face, and what will-writers and the public thought about them.³³ Rob talks about how this approach aligns with BIT's mantra of test, learn adapt:

"It's important that we don't assume to know all the answers, and to recognise that one trial can't tell us everything. Often, RCTs throw up things we don't expect, so a series of trials has helped us to test things out as we go".

Put together, these studies have contributed a significant body of evidence to the field of legacy giving, and have made a huge difference to the way that Remember a Charity works, as well as the influence that they have had over government, solicitors, and charities. Helped by the fact that BIT was then based in the Cabinet Office, Remember a Charity was able to use its findings to persuade the Minister for Civil Society to support their campaign. For four years now, the Minister for Civil Society - and there have been four Ministers in that time - has backed the campaign, writing to thousands of solicitors asking them to raise awareness of legacy giving with their clients.³⁴ Rob says that without the evidence to make their case, that simply wouldn't have happened. What's more, Rob also credits this evidence with helping to increase the number of solicitors' firms who back the Remember a Charity campaign from around 100 to over 1,300 between 2011 and 2016, and in that same time period increase the number of wills that include a charitable gift by 12 per cent.

Following the success of this research, Remember a Charity has recently signed up to use BIT's new Test + Build platform, where they plan to look at how employers can increase legacy giving among staff. For Rob, it's crucial that they continue to build the evidence base around their work, helping make their campaigning as effective as it can be.

Takeaways

• While it might seem intuitively right to work in a certain way, without robust evidence to back it up, it can be hard to get buy-in from other stakeholders. Evidence that affirms our existing assumptions can be just as important as that which tells us something unexpected.
• For a charity like Remember a Charity, which has relied upon external organisations such as law firms to carry out research, it's crucial that these organisations are committed to championing the work internally among staff, and that they believe in the value of the research.
• Selecting the right research partner can bring additional benefits - an organisation which is well-respected by the audiences you're aiming to influence and has good networks can not only help to attract partner organisations, if relevant, but can also open doors for you when it comes to using your evidence to achieve impact.

Informing policy: Scottish Citizens Advice Bureaux

Scottish Citizens Advice Bureaux (CAB) deliver advice services at 300 places across the country, from city centres to island communities. In 2016-2017 the CAB network helped 275,000 people - around one in 14 Scottish adults - deal with over 930,000 advice issues. Over two-fifths of these advice issues were in relation to social security benefits.

The umbrella organisation for bureaux – Citizens Advice Scotland (CAS) – addresses one of the twin aims of the service, namely to use the data gathered by bureaux to exert a positive influence on policy development. The sheer volume of CAB clients, along with direct access to those who are directly affected by legislative and policy changes, provides a strong base upon which arguments can be built.

Scotland has been replacing Disability Living Allowance (DLA) with Personal Independence Payment (PIP) since June 2013. The Scottish Government wants to develop a distinct approach to social security in Scotland, including disability and carers' benefits. In 2016, it consulted on this, and CAS were able to respond based on their evidence gathered from the frontline and discrete research.

The Scottish Government partnered with CAS, and provided a small grant, to strengthen their response by funding a number of activities. First, CAS convened an 'adviser focus group' where Scottish Government officials could directly interact with those on the frontline of support. Following this, 14 bureaux (from Skye to Glasgow) undertook focus groups (n=10) or individual interviews (n=18) with clients, supported by Research Toolkits. In addition, there was an online Adviser survey across Scotland.

The toolkits were developed jointly by the Research and Community Action Teams within CAS to provide bureaux with a guide on how to organise and run an effective focus group, along with standard 'reporting frameworks' in which to feedback information. Additionally, focus group outlines (including questions and prompts) and interview schedules were provided.

While this approach of using untrained researchers can be challenging³⁵ in terms of quality and consistency, CAS are confident that because of the support provided by the CAS researchers, and the Community Action Team, who have regular contact with bureaux, the challenges were effectively overcome. CAS actively communicated with bureaux when developing the approach to understand their needs and constraints, such as the large number of people they see every day facing complex issues.

The research highlighted the experience and ideas of skilled CAB advisers, and their clients - people who use the social security system - and it fed directly into the Scottish Government's decision-making process. It is a considerable achievement to reach so many people who are seldom heard, and has provided a model that CAS will look to use again in the future.

As Rob Gowans, Policy Officer at Citizens Advice Scotland explains,

"Having a strong evidence base is the lifeblood of our policy work. Decision-makers respect and value our input because of the strength and authenticity of our evidence – we wouldn't be able to do what we do without research."

The Scottish Government has found both the direct interaction with advisers and information gathered by citizens advice bureaux to be highly valuable, and have made considerable changes to their approach on how devolved powers in relation to disability benefits will operate. For example, the Scottish Government has included a set of guiding principles in the Social Security Scotland Bill, particularly that respect for individuals has to be at the heart of the social security system, which was a recommendation from CAS's research, and the Government is leading an awareness campaign in light of CAS's recommendation to increase uptake by people eligible for disability and carers benefits.

Takeaways:

• When seeking the views of those occasionally labelled as 'seldom heard', sometimes the best approach is to have those already in touch facilitate access.
• Working in partnership with organisations looking to achieve similar aims can often achieve more than a solo approach.
• With the right support and information, it is possible for non-researchers to make a valuable contribution to the evidence-gathering process.

What have we learnt?

Across the seven case studies, we've identified five key learning points. An overview of each and our reflections on these are below.

Using evidence doesn't have to be costly

We're keen advocates for the use of existing research evidence that's rigorous, relevant, and independent.³⁶ It's what we teach at the Alliance in our Evidence Masterclasses and talk about at our events and in our publications, and as the case studies in this report show, charities of all shapes and sizes are making use of research that's already out there. If you'd like to learn more, take a look at Nesta's Data for Good report for further examples.³⁷

The Wales Centre for Public Policy is one of a number of organisations that synthesise existing research to make it accessible and useful to policymakers and practitioners. For a comprehensive list of organisations helping to synthesise evidence in a range of areas, check out the Alliance's UK Evidence Ecosystem tool - the research summaries for policymakers and practitioners and the What Works Centres may be especially helpful.

For areas where there currently isn't sufficient evidence, free or affordable research support is available from organisations like NCVO's Charities Evaluation Service and the OR Society's pro bono scheme, or from university academics with a mutual interest in your cause. Knowledge Transfer Partnerships can be a good way to collaborate, or take a look at the National Coordinating Centre for Public Engagement's Community University Partnership Initiative.

Transparency improves the sector

We heard from Mentor UK about the importance of publishing research findings, about what works as well as what doesn't. This also applies to funders: 360 Giving has encouraged many to publish data on what they fund and where, and funders are also well placed to share some of their aggregated learning about outcomes with the rest of the sector, as Nesta does in sharing its lessons learned as a funder.³⁸

Being transparent also means being open about the evidence behind a particular programme. We've been working hard to get government to be better at this, collaborating with Sense about Science and Institute for Government to produce a transparency framework,³⁹ and conducting an assessment⁴⁰ and a spot check⁴¹ of how transparent various government departments are about the evidence they use to inform their policies. A similar approach by the charity sector could highlight research that goes on behind the scenes, and contribute to a more open culture in the sector. This can help charities to improve their effectiveness and enhance their external credibility. Giving Evidence has written about non-publication of research by charities if you're interested in reading more on this.⁴²

Proving what we think we already know

It's tempting to focus on evidence which tells us things we're unsure about, that's more likely to surprise us, than evidence which confirms what we already thought. But it's important we test our assumptions, too, as the case studies in this report have highlighted – not only because it helps strengthen our case if the assumptions are right, but also (and perhaps more importantly) because it can show us where those assumptions are mistaken, and 'uncover dangerous implicit assumptions that would otherwise slip unnoticed and thus unchallenged' into our work, as a Harvard Business Review article on 'Discovery-driven planning' warns.⁴³ It's risky, of course, and not always palatable to funders, as evidence consultant Elliot Trevithick has blogged about.⁴⁴ Being comfortable with learning hard truths and ensuring from the start that your funders are comfortable with that too is a crucial part of evidence gathering.

What's more, as we've written about in our Practice Guide, there's a heightened risk that when we test our assumptions we're more likely to fall prey to confirmation bias, where we see the evidence that fits our prior beliefs, and discount what doesn't.⁴⁵ In a 2016 speech, Nesta's Geoff Mulgan urged us to "seek disconfirmation as well as confirmation. To learn and not just repeat ideas because they are plausible, or fashionable".⁴⁶ Reflecting on our biases and seeking independent perspectives can help avoid confirmation bias.

Evidence can give you more than you bargained for

From having unintended effects on the ways GPs treat mental health, to identifying new fundraising streams, to your research partner opening doors for you in policy terms, the case studies in this report show that using good evidence can have far-reaching and unexpected consequences. While these might not always be positive, of course, it's worth thinking about any ways you can make the most of the relationships you're building.

The #iwill Fund, for example, which funds youth social action projects, aims to 'create a learning environment' to 'share what works, as well as initiatives that haven't been as successful, ensuring that funding helps to grow organisational knowledge across delivery partners and funders'. The Institute for Voluntary Action Research work on 'funding plus' gives some interesting examples of how funders provide more than money, such as introductions to new partners, training, and lobbying support.⁴⁷

What's more, as some of the charities featured in this report show, evidence can (more than) pay for itself by generating awareness and increased revenue.

Don't forget the people

Embedding learning across an organisation is important not only for making the most of what evidence tells you, but it also makes it easier to draw upon support from frontline staff and volunteers to gather evidence themselves. This recent blog from Citizens Advice Bureau explains how they make evidence relevant to their volunteers and staff - unsurprisingly, the way evidence is communicated is a key part of this. This is reflected in our study of what works in enabling evidence use in decision-making – framing and tailoring evidence to the right audiences in targeted, user-friendly ways, telling stories, and paying attention to branding to increase trustworthiness are all important in communicating evidence.⁴⁸

The last word

From all the case studies featured here, what comes across clearly is evidence can have a range of practical benefits. Whether that's challenging the way charities promote their cause, changing how staff interact with those they're helping, or getting government to think differently about the welfare system, each of these charities shows what can happen when good evidence is used effectively. We hope this report might inspire you to think about how you can use evidence to enhance your influence, effectiveness, and impact.

Endnotes