Substance is an experienced social research co-operative established in 2005 by four academics who wanted to make a real world impact beyond the university sector. It undertakes both large-scale national research and smaller, focused projects using an action research approach. Substance has grown rapidly and now embraces a range of research and technical skills that are used to provide organisations delivering personal, community and social development with user-centred tools and approaches to help them improve and demonstrate impact and value.

NESTA: Innovation in public services

Our public services face unprecedented challenges, made more urgent by the impact of the current economic crisis. Traditional approaches to public service reform are unlikely to provide the answers we need.

NESTA's Public Service Lab is identifying, testing and developing new ways of responding to social challenges and delivering better public services at lower cost.

Introduction

The crisis in public finances is deep and urgent. With the announcement in the Comprehensive Spending Review of a reduction in public expenditure of £81 billion¹ over the next four years, public services now face the stark reality of needing to shift the focus from how much to cut, to what to cut and, perhaps more importantly, how to make the right decisions. Even before the recession, services were struggling to respond to changing expectations, demographic shifts and new demands from problems associated with obesity, substance abuse and chronic disease. The current financial climate has increased the pressure on services and made the demand for new solutions even more urgent.

We have argued before² that the only way public agencies may be able to make sizeable savings while at the same time maintaining the same level of or improving outcomes is by adopting radically different approaches and finding more efficient ways to prevent and solve problems for citizens. We believe that the challenges we are facing call for a rapid expansion of the role of innovative civil society organisations, micro providers, social enterprises and other non-traditional providers in the market for public services. Such a radical change will be prompted by the opening up of state contracts and the emergence of new forms of finance, including social finance instruments, philanthropic and commercial investment.

At the heart of the effort to open up the market for public services and attract new and alternative sources of funding – be they philanthropic or commercial – is the need to find simpler, cheaper and better ways to capture and evidence the impact of services. This goes along with shifting the focus of commissioning and performance management away from mere accounting of resources (inputs) and activities (outputs) towards engineering and demonstrating results (outcomes).

Our experience shows that focusing on outcomes can also significantly increase the effectiveness of public services in meeting needs and reducing demand, by:

• creating a consensus and sense of ownership between funders, frontline staff, leaders, service users and communities around the problem that a specific service is designed to solve;
• consistently involving frontline staff, service users and communities in the design of services that are better placed to respond to local need;
• building frontline staff and communities' motivation, engagement and sense of ownership of services;
• promoting innovation by allowing people to design the most appropriate solutions rather than prescribing ready-made ones;
• reorienting performance management towards outcome and evidence-based rewards and a more efficient use of public resources;
• generating data that can support continuous service improvement;
• and, in time, producing a base of evidence on the effectiveness of different public services, which can inform decisions about the allocation of scarce resources.

The shift towards commissioning and monitoring for outcomes demands culture change amongst both purchasing and service delivery agencies. This needs to be supported by the development and spread of processes and infrastructure that can enable new forms of dialogue between commissioners, service providers and other stakeholders. The current trends towards outcomes-based commissioning and evidence-based funding inevitably point to the need for social enterprises, civil society organisations and the like to adopt processes that enable them to better evidence their outcomes and, importantly, use these data to improve the quality of their services.

Monitoring and evaluation systems are part of this crucial infrastructure. However, traditional monitoring and evaluation systems are often ill equipped to support and prompt the change that is required. It has also been argued that they struggle to adequately measure innovative approaches or models that focus on prevention, such as co-production. Too often evaluation criteria are imposed from the top down and result in the collection of information that is tailored to meet external requirements rather than to capture the essence of a service. This can lead to a disconnect between funders and the frontline where workers and service users are unsure as to why they have been asked to collect certain types of data or are required to work across a number of different tools linked to multiple funding streams. Aware of how monitoring and evaluation is often perceived by service providers as a bureaucratic burden, commissioners and funders have often tried to simplify the process, with a resultant over-reliance on monitoring (or auditing) of resources and activity, but not much in the way of evaluation of impact or value.³

A lot of work has been done to date by organisations such as Charities Evaluation Services, the National Council for Voluntary Organisations, Triangle Consulting, nef (new economics foundation), the Social Return on Investment Network and others on refocusing attention on outcomes and impact and on designing tools to capture the achievement of outcomes through service delivery.

The social research co-operative Substance has also been active in this field with the development of cutting edge technology for monitoring and evaluation. Between 2008 and 2010 NESTA supported a Substance project, which set out to help service provider organisations to:

• produce qualitative reports that evidence impact without having to rely on expensive expert researchers;
• simplify reporting by aligning communications about their activities to different funders' outcomes;
• streamline data collection through a platform that supports interaction with different media (audio, video and social media) and with third party outcome measurement tools (such as the Outcomes STAR, SROI, etc.).

The result of this work is Views, a suite of online tools designed to capture and document the impact of a service through the use of a wide range of media, techniques and reporting formats. Substance's experience of developing Views has highlighted how systems that evidence impact in innovative ways can support culture change in both service providers and commissioning bodies. It also demonstrated that enabling practitioner-led learning and evaluation has the potential to radically change the scope and role of performance monitoring.

In this document we present the learning. The sections that follow set out the top tips to transform monitoring and evaluation from a box-ticking exercise into a major organisational asset, and share the story of how the cutting edge monitoring system Views was developed.

Monitoring outcomes: ingredients of success

Shifting performance management in public services away from a mechanical recording of inputs and outputs towards a more strategic focus on outcomes requires a radical culture change. This involves funders, organisational leaders, managers, frontline staff and service users, and needs to be supported by appropriate infrastructure. Collaboration between these groups can turn evaluation from a laborious 'add-on task' to a dynamic process that can transform and improve services.

Substance's experience of exploring practical and scalable monitoring and evaluation approaches that reconcile the needs of funders with the priorities and processes of service providers has surfaced some key guiding principles:

1. Start with the quality of your users' lives.
2. Empower frontline staff.
3. Use your imagination and be creative.
4. Embed evaluation in everything you do.
5. Release the data.
6. Use evidence to make resourcing decisions.

This paper will go on to explore these guiding principles in more depth.

1. Start with the quality of your users' lives

It seems obvious to say that the starting point of a service should be the need it is trying to address. People's lives are complex and their needs often do not fit neatly within the narrow silos of existing services. Understanding the lived experiences of users and the impact of services upon their lives should be at the heart of any effort to assess, design or improve a service. As NESTA has argued⁴ through its work on co-production and radical efficiency, equal and reciprocal partnerships between professionals and service users are the crucial ingredients in services that are radically more effective and efficient. Co-producing services does not just mean designing services with users – it goes far beyond that, to include considering how users can play a role in delivery and co-measuring of outcomes.

Traditional monitoring systems can make it difficult for organisations to capture the richness of users' experience of a service and the complexities of their needs. Substance tried to address this issue through the development of the Substance Project Reporting System (SPRS) – an online monitoring tool developed by Substance in 2006 so that organisations could get a better understanding of their users' journeys and be able to relate them to higher level outcomes, such as a reduction in overall crime figures.

“It's not just about the crime reduction figures at the end of the year – it's about recording the journey of each youngster on an ongoing basis.”
Respect Programme,⁵ Portsmouth

2. Empower frontline staff

Frontline staff are the first interface with service users and as such they are in the best position to capture the journeys and stories of their clients and to piece together a rich picture of the impact a service has on the lives of its users. They are also best placed to facilitate user involvement in the way the services are delivered and monitored. As we have found in our work, employees play a vital role in an organisation's capacity to innovate. Our experience shows that empowering frontline staff to lead on monitoring and evaluation efforts will unfold hidden potential for innovation in service design and delivery. It can also help to reduce costs by devolving responsibility for data capture down to the frontline. Ultimately, the greatest level of effectiveness is achieved when there is alignment and communication flows between frontline staff, the strategy level and funders' requirements.

To successfully devolve monitoring and evaluation to frontline staff, data collection needs to be made as simple and relevant as possible. Service practitioners are committed to devoting time and attention to their customers and often resent the bureaucratic burden placed by monitoring and evaluation requirements, seeing it as a distraction from 'real work'. It is critical to ensure that monitoring arrangements are not layered on top of existing work profiles, but built into them.

In building Views, Substance was led by organisations' demand for a streamlined system that minimised duplication and made it easy for frontline staff to collect evidence of the impact of services while carrying out client facing activities. By providing: smart work scheduling and bookings tools; attendance registers; resource allocation and income/expenditure profiles; media storage and case study creation functions, Views enables practitioners to build a repository of searchable data and generate evidence to meet their organisation's monitoring requirements while carrying out their daily tasks.

“The Substance Database has made the monitoring side of Positive Futures easier and more accessible for all staff. It has also allowed the project coordinator to have a simplified overview of the project, and allowed easier information sharing with partners. This accessibility of information has enabled us to review the success of sessions, and make adjustments accordingly and review the progress of individual young people and the achievement that they have made.”
North West Leicestershire Positive Futures¹¹

3. Use your imagination and be creative

A growing appetite for instant stories and technology that enables real-time communication and supports user-generated content are pushing the boundaries of work practices. The combination of technology and creativity can revolutionise monitoring and evaluation by using new media (i.e. video, audio, social media) that ‘make impact real' and offer a 'live' portrayal of user journeys. This may mean capitalising on people's willingness to use a range of new media by enabling them to link to content uploaded to popular social media platforms rather than having to duplicate the process via bespoke monitoring tools. In order to acquire greater meaning and validity this data will then need to be knitted together with other evidence and linked to the achievement of specific outcomes.

The revolutionary speed at which people have engaged with social media and turned it into a trend is driven by the immediacy with which content can be shared. A ‘streaming' process for the collection and broadcasting of evidence is increasingly likely to complement and significantly reduce the prominence of conventional point-in-time evaluation and annual reporting mechanisms.

For example, through tagging of files and keywords to appropriate outcomes frameworks, Views enables seamless links so that a YouTube hosted video can be used to provide an additional layer of evidence of how a specific intervention has contributed to the desired outcome.

"We have focused on capturing the stories of those individuals, like Francois, who we work with. It's not just about the re-offending rates, it's recording little things like youngsters turning up on time – which can be a major achievement but might get forgotten. We record case studies and interviews with young people to capture every element of their personal journey.”
Leyton Orient Community Sports Programme¹⁴

4. Embed evaluation in everything you do

Throughout this document we have been making the case for the value of collecting meaningful data and information on services. We believe that data collection should not happen only once a year with the purpose of fulfilling tedious reporting requirements, but rather that it should be embedded in job descriptions and carried out regularly. Similarly, evaluation should not be based on information that is disjointed from daily operations but rather that it should build on data that the organisation collects anyway as part of its functions. This will ensure that evaluation captures salient aspects of a service rather than marginal and artificial information and, crucially, it avoids duplication of effort. Moreover, gathering information about services in real time, gives staff a valuable insight on what is and isn't working.

Funders, on the other hand, should move away from imposing top down reporting criteria and treating evaluation as just a formal annual duty, in favour of shaping relevant reporting frameworks in collaboration with provider agencies and engaging in more regular conversations to monitor and reflect on outcome achievement.

Users of the Substance Project Reporting System (SPRS) have consistently reported that one of the main benefits of the system is the practical support it provides to staff in organising and managing their work. For example, collecting data on weekly session attendance, background information on a young person and key personal details allows staff to capture soft outcomes (i.e. distance travelled by a young person) while at the same time keeping the consent and referral details up to date.

“We take pride in recording it all and when we train new staff we explain how important it is to put everything down."
Respect Programme, Portsmouth

5. Release the data

Given the amount of effort invested in collecting evidence for reporting purposes, it is surprising how little this data is generally used to inform a genuine and shared reflection on the service, both on the part of service providers and funders. Just as commercial providers go to great lengths to access information that will reveal consumer preferences and inform their marketing and operational strategies, so public service providers and commissioners need to make use of information they hold to regularly review the effectiveness of their services. Quite often they are sitting on a mine of precious information, which could help not only to improve a service, but also to deliver it more cheaply.

A data query carried out through SPRS by a project delivering the Kickz programme in the West Midlands revealed a collapse in attendance for participants from a particular postcode area. Informed by this data, project staff enquired further and discovered there had been a conflict between these participants and another group living in the vicinity of the project venue. This prompted a review, which led to a change of venue and the successful re-engagement of the original participants.

6. Use evidence to make resourcing decisions

From the point of view of organisations that are competing for limited sources of funding, evidence is the raw material for a strong business case. From the perspective of commissioners it is the basis for making informed decisions about the allocation of limited resources. Ultimately, building a strong base of knowledge about different services' effectiveness in meeting designated outcomes allows both service providers and commissioners to draw out important lessons for future delivery. These lessons can and should be used not only to revise and improve service specifications but also, where appropriate, to inform commissioning plans and policy.

Positive Futures is a Home Office-funded programme targeting vulnerable 10-19 year olds living in the most disadvantaged neighbourhoods. The programme uses a locally based approach to evaluation that assesses and benchmarks the impact of each project on the lives of young people and maps results against national policy priorities. Building on the confidence in the evidence generated by projects through the SPRS, over the last 18 months the programme has moved to a payment-by-results funding model that rewards those projects delivering sustained positive results and impact. The Respect Programme in Portsmouth, which since 2007 has contributed to a reduction in youth crime in their target areas of 20 per cent and a fall in anti-social behaviour of 60 per cent, has recently received increased funding as a result.

"Results-based funding provides a financial incentive for organisations to exceed their baseline targets and move on from a culture of just doing what they need to do.”
Leyton Orient Community Sports Programme

Making it happen: the Views story

Views was born out of a desire to develop a scalable approach to process monitoring and outcome measurement that combines the richness of qualitative research methods with the ease and functional simplicity of more technical data capture and reporting systems.

Substance's involvement in and evaluation of a series of government-funded social policy programmes had brought it into close contact with the frontline realities of existing monitoring and evaluation arrangements, often characterised by sterile, box ticking exercises that few people really believed in. The challenge was to develop a meaningful alternative that was woven into the very fabric of delivery processes.

As we considered how to capture the mix of mundane, euphoric, familial, chaotic and distressing real-world stories we had observed in the context of our more engaged qualitative research, we were confronted with the question of how to collect and represent this evidence in a coherent and accessible form. We also wondered how an intrinsically qualitative and resource-intensive approach could be streamlined and delivered at scale, so that richer forms of evaluation and impact assessment could be made available to the widest possible array of delivery organisations.

Despite our own professional biographies being imbued with social science perspectives, we felt that the need to engage frontline staff in the process and to deliver a scalable and cost-effective solution meant the answer must lie in technology.

In September 2006, a year's consultation and development culminated in the first release of the Substance Project Reporting System (SPRS). This prototype software provided a comprehensive online monitoring, evaluation and reporting framework that neatly combined quantitative and qualitative data entry to enable organisations to capture and represent a fuller range of their work.

Whilst working with the Positive Futures programme, for which Substance determined the parameters of the monitoring requirements, we were largely successful in ensuring a user-friendly system that minimised the burden presented to users. But as new projects from beyond this programme came on board, they each brought fresh ideas and new requirements with them that encouraged us to take account of:

• Growing enthusiasm for storing ‘qualitative' forms of evidence across social media platforms.
• The continuing demands placed on projects to deliver distinct information sets in different formats to multiple funders.
• The benefits of engaging system developers in a more open process of mutually beneficial sharing of data and functionality.
• The disjuncture between projects' focus on what they are doing or planning and the continuing demands of annual reporting cycles which focus on what they have done.
• The critical need to clearly and robustly demonstrate social impact and value in the context of revised public spending priorities and reforms to public service provision.

With NESTA's backing we sought to address these issues by creating the state of the art Views outcome-monitoring platform. The cornerstone of the platform lies in its data structures, which deliver an infinite degree of flexibility by enabling the storage of different types of data and the ability to associate it with a limitless range of individuals, work profiles and outcomes. System users are able to map data seamlessly to organisational objectives as well as local, regional and national priorities and outcome frameworks through a single process.

This search for simplicity and need to avoid duplication is also reflected in the recognition that Views could only ever be part of the answer. Practitioners already routinely engage with a range of powerful data collection tools that should be complementary to the system rather than standing in opposition. Embracing this reality, Views has the ability to interface with third party software, whether it is media files hosted on social media sites such as YouTube, Flickr and Ipadio or bespoke outcome measurement tools such as the Outcomes Star and others.

Ultimately Views will enable:

• Delivery organisations to focus on developing the most effective ways to deliver results rather than being the slave to alienating reporting regimes.
• Data capture mechanisms to be embedded in local, 'real time' project management, delivery and appraisal processes rather than reporting cycles.
• Easy collation and storage of multiple forms of evidence.
• Interface with third party software providing an umbrella for a range of outcome measurement tools.
• Simplified and flexible reporting formats.
• Commissioners to track and benchmark the effectiveness of services and access rich, real time data with which to review resource allocations.

If adopted on a broad basis the guiding principles of effective outcome monitoring outlined in this pamphlet could allow for a revolution in public service efficiency, as money is encouraged to follow superior value. In order to help realise this goal, over the next 12 months and beyond, Substance will be working to maximise take-up of Views by working with a wide range of pioneer partners who currently include: ACEVO, Catch-22, Football League Trust, London Borough of Camden, Mentoring and Befriending Foundation, NESTA, Novas Scarman Group, Social Investment Business, Sported Foundation, Sunderland AFC.

Endnotes

www.nesta.org.uk www.substance.coop Published: November 2010 ISBN 978-1-84875-111-8