PERFORMANCE MANAGEMENT AND REPORTING

This paper summarises Nesta's approach to performance management

1. Introduction

1.1 Nesta has a broad mission to support innovation. That involves us in two main types of activity:

• first, supporting particular ideas or ventures and helping them to develop, and then grow to scale and achieve impact;
• second, influencing the environment so that it's easier for innovations to flourish, for example through changing government policy or the policies of big firms such as banks.

1.2 So how can we know if we're succeeding? How can we judge if our resources could be better spent elsewhere? How can we, our board, or our stakeholders, judge if we're doing a good job?

1.3 Over the last few years we have developed a rigorous approach to evidence of impact (summarised later in this report), and have worked with hundreds of organisations to help them achieve greater impact. But the management of performance and impact in innovation is not easy for many reasons – because of the often long gap between actions and results; because radical innovation involves many failures as well as successes; and because some of the best innovations end up achieving outcomes different from those they originally intended.

1.4 Nevertheless we think it's important to do our best in managing effectiveness, and it's also an important aspect of our charitable status that we should demonstrate the public benefit we achieve. In this paper we set out some of our thinking, with the aim of prompting responses and ideas on how we could do better.

The challenge of measuring innovation

1.5 The challenges of measurement and management are shared by many other agencies dedicated to innovation – including public agencies and foundations. They overlap with the measurement challenges faced by many others – including research teams within large firms; departments in universities; and laboratories of all kinds. In every case overly simple metrics and management methods (for example only managing what can be easily measured, or setting rigid or quick output targets) can easily lead to distorted decisions and poor results. All non-profits struggle to measure the value they create; and although in theory for-profit firms are helped by the unambiguous metric of shareholder value, a succession of scandals and crises have confirmed that they can struggle almost as much because of the absence of any objective ways of allocating costs and linking them to future revenues.

1.6 There are many ways to embed a commitment to impact in everyday work. Nesta's executive team and board receive regular dashboards which provide a mix of quantitative data and qualitative assessments of progress in every area of activity – from investments to programmes, research projects to communications. Each board meeting also includes a detailed dive into a particular area of work. Each team addresses how to achieve and assess impact.

1.7 Some of Nesta's activities are relatively easy to measure and manage: investment activity can be assessed in terms of capital valuations and exits; communications activity can be measured in terms of reach and depth, and satisfaction. The management of Nesta's Trust assets can be measured in terms of long-term value benchmarked against comparable foundations.

1.8 Other areas of activity are inherently harder to measure, including research. And some, including programmes, involve a mix of relatively easily measurable aspects (such as financial and project management, or numbers of projects funded) and much harder ones (in particular outcomes). Overly rigid metrics can destroy a creative culture – by its nature anything involving discovery can't be measured in advance. Yet there are good reasons for wanting to understand what is and isn't working.

2. Nesta's approach and what success looks like

2.1 The first question to ask of any activity is: "what does success look like"? We have tried to answer this through strategy exercises which sharpen up thinking on programme outcomes, investment goals, research and policy influence etc. Within each activity it's useful to return to this question repeatedly. The answers then need to be translated into tangible goals for investments, programmes, research, networks and skills. Wherever possible these should be articulated in terms of outcomes – the change we want to see in the world, with supplementary measures that articulate our contribution in a way that means we can hold ourselves to account. These will change and evolve over time – it's in the nature of innovation that it will take us in unexpected directions. But it's valuable to have a framework that provides confidence and accountability, with clear expectations of organisational performance, including the management of finance, people, assets, delivery and risks.

2.2 Each field of Nesta activity has a distinct view of success:

Investments

At first glance the metrics and goals are reasonably straightforward: a growing portfolio of ventures achieving both commercial and in some cases social impacts, with successful exits and impact (Nesta has achieved several very profitable exits over the last two years). However, the reporting system entails a series of judgements about ventures, the state of the management team, product and technology development, market conditions etc. There are also some new complexities. In Impact Investing, we've launched with external investors, a new £17.6m FCA regulated Impact Investment Fund, whose goal is to grow a portfolio of ventures that drive social impact in our key chosen fields on a sustainable basis - that involves new tools for assessing success. In addition we help in building the market for impact investing in the UK, through our work with intermediaries, accelerators and our approach to impact measurement. Our venture portfolio now includes 30 early stage companies and funding has been provided to 35 start ups through accelerator programmes that we support. We've also helped create a 4 year, £2m incubator programme with Bethnal Green Ventures that will support 80 early stage social innovations. Investments are regularly assessed for their health or sickness and key decisions on new and further funding involving intensive discussion at the Investment Committees, drawing on a combination of data and qualitative assessment.

Programmes

In relation to programmes the goals include tangible impacts on important fields of challenge and opportunity – eg substantially expanding the amount of digital making (coding, programming etc) in UK schools with, in the longer run, an impact on the economy; accelerating the growth of a hyper local media sector, with visibility across the UK; making co-production mainstream in primary care in the NHS, with widely used tools for commissioning, assessment and provision. Where we run open calls we aim to successfully identify and support the most ground breaking innovations, and help them to grow and be adopted by others. For example, the Innovation in Giving programme has supported 75 projects, leveraged £9.3m in additional funding, and reached several million citizens, with the longer term goal of reaching a majority of the population. Our 'People powered health' projects reached many patients in six areas of the country – but some of its biggest impact has come through demonstrating that if these were scaled up they could save the NHS over £4bn.

Our main problem in assessing performance in the past was an oscillation between over-reaching ambitions at design phase and then overly modest objectives. Often programmes were run for too short a period, or without adequate lesson learning. Moving to longer time horizons should make it easier to articulate the bigger strategic goals which are being targeted; the quality of pipelines of early stage ideas; how many of these are moving into implementation and scale; and the richness of insights being generated. Each programme needs to clarify its intended model of causation or 'theory of change' which should then shape metrics. But we should expect a wide range of types of impact and goals.

Policy and Research

Here there are some crude metrics available such as media coverage, citations, and policy take-up by political parties or government, though each risks skewing the programme towards shorter-term issues. Impact is bound to be harder to judge – and will be most meaningfully assessed only after the project is completed, but we can attempt to assess at least four routes to impact for our work: impact on policy, impact on thinking ('changing the debate'), impact on practice and impact on Nesta programmes. Sometimes our main goals may be new ways of seeing or thinking – such as new statistical measures like the Innovation Index. Sometimes they may be new policies (for example the commitment to introduce computer science into the school curriculum), or changes in the way things are done (such as the expansion of the small business research initiative). Some projects directly benefit a new field – like the crowdfunding site that supports and links to 31 other platforms. The Alliance for Useful Evidence which we host has over 1000 members – but just as important has been its success in persuading government to a back network of half a dozen ‘What Works' centres. These examples confirm that we also need to track the performance of overall programmes of work rather than only disparate elements, so as to demonstrate impact over longer periods of time.

Skills

In relation to skills, which is a newer area of activity for Nesta, the goals include the reach and depth of use of our skills offerings – the broader goal is to raise capacities to innovate, whether in start-up firms or local councils. In the longer run we will be able to measure the value of skills work in terms of reach, depth, NPS, benchmarking etc. Willingness to pay is another obvious measure – though our primary goal is to provide tools free or at low cost for public benefit.

Networks and Communications

For our communications team there are relatively straightforward metrics in terms of reach and depth, and in time measures such as net promoter scores. For example, our unique website visitors now run at around 1m a year; we have some 40,000 Twitter followers, and some 12000 visitors to events at Nesta each year. The team now collect a wide range of data, including media hits across trades, nationals, regional, broadcast (which now runs at a far higher level than ever in the past); website useage and engagement; and engagement with particular elements, such as staff blogs. Another tool we can use is klout which analyses influence online and although not totally accurate is a good gauge of the reach and impact of new content.

2.2 As we develop measures we recognise that there are plenty of traps to avoid. These are a few:

• backing incremental rather than radical innovations so that there is a higher chance of being able to claim success in having them implemented;
• backing specific innovations which are already on track to success and claiming them as successes;
• recommending policies that we know are anyway likely to appear in government policy

Quantitative methods on their own are very vulnerable to these traps, and can easily distort behaviour. That's why we emphasis dialogue and scrutiny to dig beneath the surface of the numbers.

3. Elements of performance

3.1 Some of the elements of performance management are relatively straightforward to define and assess. These include measures such as overall levels of investment and grants; numbers of organisations supported and so on. We also seek to manage our own internal processes:

Project management

– for each programme or area of research there is normally a clear agreed forecast timeline and set of activities to be performed. Project slippage and risk can be caused by poor Nesta project management but also by the failings of partners, by external factors or the difficulty of the project. We therefore usually include very clear milestones when working with partners, commissioning work, or providing grant funding. This then provides a common language for internal reporting and reporting to the board, showing which projects and programmes are on track.

Project / Programme spend

– each quarter the executive receive management accounts which provide data down to the individual programme level. All internal programme staff are fully costed to programmes and projects. All programme spend is compared to budget each quarter. For core costs we increasingly seek to benchmark spending and results against other comparable organisations.

Financial Performance management

- as described above quarterly management accounts are produced for the Executive team and compared to budget for each business unit. This highlights variances against budget and helps to identify projects and programmes that are not performing to plan. A reforecast is prepared three times a year by the Executive team to reflect any changes in the timing of major areas of work, any changes to partner funding relationships and any business development spend.

Staff performance management

- This process is conducted by Directors and Executive Directors in the organisation, and overseen by HR. All staff agree annual objectives with their line manager, receive regular performance feedback and an annual appraisal. Individual objectives should provide a line of sight to broader programme, team and organisational objectives.

Some of these aspects of management are subjected to rigorous scrutiny. We use internal audits by third parties to scrutinise how well we manage our own processes. The most recent ones have been completed by Grant Thornton.

4. Goals: achieving impact

4.1 The more demanding aspects of performance management concern impact. We can require each area of the organisation to be rigorous in establishing project and programme goals and to establish how we will measure and define success. However, this does not then provide a simple framework for assessment. As section 2 showed, impact means different things in different fields of activity. There are also some general issues.

4.2 It is in the nature of early stage innovation projects that they evolve; generate surprises; and change their goals as well as their means. The impact and success for a programme can be evidenced by such things as reach, scale, replication, changes in behaviours and recognition of innovation, but these may take a long time to materialise. In addition very different measures are likely to be useful for different programmes, which makes it difficult to draw direct comparisons of impact across programmes, for example, ones supporting new tools for charitable giving, a new method of running GP consultations or a new business model in the media.

4.3 Our own experience points to many examples of the challenge of performance management in relation to innovation: the current success of some investments which had previously been written off; and the success of some of Nestaʼs earlier awardees – many of whom took many years to reach the top of their fields, and were wrongly deemed not to have succeeded in earlier evaluations.

4.4 Timescales are crucial here. Relatively incremental innovations and ideas can expect to move into implementation or take-up quite fast. More radical ideas are bound to take much longer. Research is bound to have more impact after projects have been completed than during their progress. So we need within each programme or project a realistic sense of milestones and rhythms of progress. Equally we will need to strike a balance between relying on the judgements of experts, and hunches which may be at odds with conventional wisdom. The key lesson is that it's vital to grow a strong capacity to make judgements. Metrics can help. But these are only inputs to judgement which develops in the light of experience.

4.6 Each team also has revenue generation targets – at present we aim to roughly match the income from our Trust with external income. The need to bring in partners is a good pressure to ensure that projects and programmes are seen as valuable by others. It also enables us to achieve more with our limited resources. However here too there are risks. Too much chasing of external income can easily work against other goals (for example it encourages cookie cutter consultancy rather than originality; it can encourage commodity grant making rather than the more innovation-oriented, hands on approaches we favour). The only way to ensure the right balance is being struck is through intensive discussion amongst staff, executives and board.

Applying the evidence framework

4.8 Over the last year we have developed a common language for evidence, which draws on the work many others have done. This sets out a series of steps that define how certain we can be that a particular activity or programme is working.

Most of our work involves projects at the lower rungs of this ladder – it's often helpful to work with projects and programmes to clarify exactly how they think they may achieve change, sharpening up their understanding of causation (this is sometimes misleadingly called a 'theory of change' though it is strictly speaking not a theory). This approach is also important for our own work in designing and implementing programmes, and involves repeatedly returning to fundamental questions: – what are we trying to achieve, with what tools, and drawing on what knowledge and experience?

We are also increasingly working with organisations to help them gain more evidence of impact. This is particularly the case where our funding is helping an organisation to scale. Over time this should ensure that we have access to much more project-level and programme-level data on impact, and greater confidence in making claims about impact.

5. Information and Reporting

5.1 Executives, trustees and external stakeholders, need different types of information, albeit with a substantial overlap. Trustees need more strategic oversight of effectiveness of spending, resource allocation and integrity, while executives will be more concerned with management issues, remedial actions, programme adaptation (or exit). Both need data as well as qualitative judgements. And both need to be able to calibrate both performance and the difficulty of tasks and their levels of ambition (a green isn't always as good as a red on a programme dashboard). External stakeholders and regulators – such as the Charity Commission – need to be confident that we are achieving public benefit and that we are making good use of resources.

5.2 For our annual report we aim to steadily increase the proportion of reporting related to impacts. As a NDPB, Nesta tended to use its annual report to describe current initiatives, rather than achievements and impacts. As a charity, with a more explicit requirement to achieve public benefit, we will shift to reporting on:

• Activities
• Impacts achieved
• The longer term impacts that are sought

5.3 A key issue we are grappling with in all fields of reporting is to get the right granularity. There has been a lot of interest in recent years in new aggregate measures – such as SROIS, or single metrics to summarise the activities of an organisation. In our view these have not succeeded because the act of aggregation risks losing the most important aspects of information and feedback. The numbers that result can be rather meaningless and hard to act on, whether for managements or boards. This is very different from performance management of investment funds, service delivery or consumer product organisations where aggregation and standardisation of metrics are not only possible but essential. By contrast in the types of activity Nesta does over-aggregation is misleading. It's not hard to define some organisation-wide KPIs, but these risk being meaningless or at worst distorting behaviour. The most useful management involves some engagement with detailed findings, achievements and surprises. In practice like many organisations we use what's sometimes called a 'balanced scorecard' – but it's the processes that surround the scorecard rather than the scorecard itself which matter most.

5.4 That's why, alongside this reporting system, it's vital to grow a culture which is able to acknowledge problems and failures; able to reflect on lessons; able to distinguish between organisational performance and personal performance; and to institutionalise post programme and project learning in an open and rigorous way. Performance management is an art and craft rather than a science. This may be particularly true for organisations involved in finance, whose actions are necessarily one or more steps removed from direct provision of goods and services. Harsh lessons were learnt in the last decade when organisations with apparently sophisticated internal metrics turned out to be measuring the wrong things, measuring the right things but in the wrong ways, or relying on algorithms that systematically distorted decision-making. The key lesson is not to give up on measurement and evaluation – but rather to combine any metrics with intensive, sceptical discussion to support more intelligent judgements. These judgements are bound to be made in conditions of uncertainty, particularly where innovation is involved. But systematic description of desired outcomes, combined with systematic learning about what actually happens, makes it likely that judgements will improve over time.

Nesta 1 Plough Place London EC4A 1DE [email protected] www.twitter.com/nesta_uk www.facebook.com/nesta.uk www.nesta.org.uk © Nesta. June 2013 Nesta operating company is a registered charity in England and Wales with company number 7706036 and charity number 1144091. Registered as a charity in Scotland number SC042833. Registered office: 1 Plough Place, London, EC4A 1DE.