MI/25 / Published: July 2008 Policy Briefing

Measuring Innovation

Evidence-based policy demands reliable data. Existing innovation metrics constrain policymakers by reinforcing a linear model of innovation that bears little relation to the innovation that is most relevant to the modern UK economy. Traditional metrics fail to adequately capture 'hidden innovation': innovation in services, the public sector and the creative industries, or new trends in open and user-led innovation.

Around the world, policymakers are demanding new ways to measure innovation. In Innovation Nation, NESTA was called upon to develop a new Innovation Index for the UK. Similar efforts are underway in other countries and in international bodies such as the OECD and European Commission. Correctly developed, these will ensure that the UK is better positioned to understand and support innovation in the future.

Over the next two years, the Innovation Index Project will mobilise the best of the UK's innovation practitioners, researchers and policymakers to develop the most relevant, rigorous and accessible measures of innovation.

Metrics matter

Strong data are the feedstock of social research

Data are the raw material of quantitative social science research. The best data shed light on the evolution of societies and economies. They reveal their driving forces, future challenges and potential policy responses.

Policy is affected by the targets used to measure 'success'

Policymakers now rely on a number of outcome targets. From inflation to child poverty, successive governments have used metrics to identify areas of concern, and targets to drive public service improvements.

However, just as poor data can lead to unreliable research, the wrong targets can produce weak policy and poor outcomes. Convenient measurements may not capture what matters most and can lead to harmful unintended consequences. For instance, a switch to biofuels to meet carbon reduction targets has resulted in deforestation to plant crops, thereby reducing carbon sequestration.¹

Traditional innovation measurements have produced traditional innovation policy

Modern understandings of innovation have their genesis in science policy and the manufacturing economy

During the 1940s and 1950s, military concerns led governments to undertake large-scale, resource-intensive, 'big science' projects. These had enormous budgets, large numbers of staff and investment in hi-tech machinery.² Over time, science and research and development (R&D) became synonymous with innovation.

As a result, the innovation process was believed to conform to a 'linear model'. Here, innovation started with scientific discovery and basic research, passed through applied research, engineering and manufacturing activities, and ended with a new, commercialised product.³

Innovation metrics were designed for a manufacturing-based linear model of innovation

'Innovation' metrics followed this innovation policy, focusing on the creation of new scientific knowledge of relevance to manufacturing industries.⁴ Current internationally-agreed indicators include R&D expenditure, patent production and numbers of science and technology graduates.⁵

Innovation policy 'targets' followed the available metrics

Our most readily-available measures are the product of traditional innovation policy. After several decades of R&D statistics output, boosting the levels of R&D is now a major focus of international innovation policy. The UK government introduced R&D tax credits, while the European Council set a 3 per cent GDP target for R&D expenditure for EU members, as part of the Lisbon agenda.⁶

Innovation, like the economy, is changing

Services now dominate the UK economy

Since the 1970s, the UK has shifted from a manufacturing economy to one where service industries dominate. The service sector now accounts for over 75 per cent of value-added, compared to just 16 per cent for manufacturing.

But the UK's service-based economy is not well-served by existing innovation metrics. On aggregate figures, the UK spends less on R&D and creates fewer patents than other developed economies. However, once its economic structure is taken into account, the gap between the UK and Finnish R&D intensity (for example) halves from 1.9 per cent to 0.8 per cent of GDP. The gap with Germany, on triadic patents per million of population, falls from 38 to 10.⁷

Innovation occurs in new forms and places

The 'linear' model of innovation is no longer universally applicable. Innovation is now understood as a multidirectional process involving multiple actors, which is strengthened by repetition.⁸ It encompasses not only the development of new components and products but new services, technical standards, business models and processes. It is as much a feature of developments in the public and non-profit sectors as in the private sector.⁹

Furthermore, much of the economic benefit from innovation comes from incremental innovations arising from the wide diffusion of knowledge and technology rather than from the creation of that knowledge in the first place. And just as service sectors are now significant innovators, manufacturing industries are learning to innovate in new ways. In the scientific equipment, chemical production and semiconductor manufacture sectors, most commercially-relevant innovations arise from user modifications or demand.¹⁰

Internationally, policy and research are catching up with these new kinds of innovation

The UK, Denmark, Sweden, Finland and Singapore are among the countries to put themselves at the forefront of research into new modes of innovation, as part of the development of a more comprehensive innovation policy.

Innovation Nation, the recent Department for Innovation, Universities and Skills' (DIUS) White Paper, explicitly recognised the 'changing face of innovation'.¹¹ Innovation in services and the public sector were given specific policy recommendations for the first time, while demand was identified as a main driver of innovation.

New innovation is driving the production of new metrics

But if this new innovation policy is to be embedded in economic policymaking, it needs new metrics to measure performance. This is now a global priority.

In the UK, DIUS has asked NESTA to develop a new 'Innovation Index', to measure the UK's innovation performance.¹² A recent report to the US Secretary of Commerce stated that "the need to advance innovation [measurement] cannot be understated", and suggested several improvements to current metrics.¹³

Within the US, states like Massachusetts have independently developed new approaches to measure relevant innovation.¹⁴ Canada and Australia have also begun to develop improved metrics.¹⁵

The OECD and European Commission are working on new innovation metrics

The Organisation for Economic Co-operation and Development (OECD) was the driving force behind the adoption of the standard science and technology indicators, as contained in the Frascati Manual. More recently, the OECD has been working with Eurostat to develop and update these definitions, so that they capture wider forms of innovation.¹⁶

The development of better indicators to capture non-technological innovation is a major component of the OECD's new 'Innovation Strategy'. The strategy also seeks to improve our understanding of how the climate for innovation is linked to innovation performance.

Eurostat, the statistical agency of the European Commission, in coordination with national statistical offices, implements the Community Innovation Survey. It is a rare example of an attempt to develop a more comprehensive picture of innovation, providing comparable cross-country data on products, processes, and organisational and marketing innovations, among others.

The European Innovation Scoreboard benchmarks the performance of EU countries and regions in various aspects of the innovation process. Although previous versions had a focus on technological innovation, several alternatives to capture new forms of innovation in future versions are also currently being considered.

Conceptualising innovation poses a problem for researchers and policymakers

Innovation is difficult to define

The former Department for Trade and Industry (DTI) defined innovation as "the successful exploitation of new ideas".¹⁷ While this definition is short and simple, identifying its components is far harder.

NESTA has completed research on hidden and non-traditional innovation that has begun to identify and quantify new ways in which innovation occurs.¹⁸ These findings, however, have highlighted further methodological difficulties.

Different sectors innovate differently

Innovation activities are often industry-specific; few measures effectively capture innovation in all industries. R&D expenditure and patent creation, for example, capture innovation well in the pharmaceutical industry, but fail to do so for financial services. Any accounting methodology needs to allow for this diversity, developing specific indicators where necessary for each industry.

'Productive failures' are hard to quantify

Many innovative, path-breaking projects end up in failure; risk is inherent to the innovation process. But apparent failures still contribute to a wider innovation process, providing a source of future ideas, and indicating an inherent dynamism within the economy.

Diffusion is often more important than creation

Society benefits most from an innovation that is widespread. The adoption of innovation by others is therefore often as important as its original development. However, developing an innovation and adopting innovations diffused by others are very different activities requiring different metrics.

Innovation is now an international activity

Innovation is about more than turning local ideas into successful products; it also draws on ideas and innovations developed elsewhere. Consequently, an economy's 'absorptive capacity' determines its ability to take on and utilise international ideas to drive innovation.

But this raises problems of definition. Many UK firms innovate in their overseas branches, while the UK benefits from R&D by foreign multinationals. According to the R&D Scoreboard, nearly a quarter of R&D expenditure by the top 850 R&D-investing companies in the UK is actually invested by foreign-owned firms.¹⁹

Innovation is influenced by 'wider conditions' in the UK's economy and society

Innovation does not happen in a vacuum. Instead, successful innovation ecosystems rely on a range of factors, some influenced by public policy. The innovation performance of firms partly depends on the quantity and quality of skills available, the effectiveness of universities, the nature of the intellectual property regime and the regulatory and competitive environment. Other less tangible factors, such as attitudes to risk or consumers' willingness to try new products, are also important.

Measuring innovation in public services presents a specific set of challenges

The delivery of public services has a significant impact on the welfare of citizens, regardless of whether they are supplied by the public, third or private sector. The UK Office of National Statistics (ONS) has made significant progress in measuring public sector productivity, but quantifying innovation and its effectiveness remains a challenge.

The NHS Institute for Innovation and Improvement has an ongoing project intended to improve innovation metrics within the National Health Service. And Innovation Nation called for DIUS to review innovation-related aspects of government activity in an 'Annual Innovation Report'.²⁰ However, devising a credible and widely-accepted metric across public services will pose substantial methodological challenges without the benefit of the decades of research available on private sector innovation.

Creating new metrics poses further difficulties

Developing new metrics means balancing accuracy, longevity, comparability and ease of collection

A perfect innovation metric would balance accuracy, longevity, comparability and ease of collection. It would precisely measure the innovation that matters to a particular industry. It would do so consistently over a long period of time, allowing comparisons across sectors and countries, and be easy to collect (something innovation indicators have traditionally placed at a premium).

But these factors are often in conflict with each other

Developing accurate innovation measures relevant to particular sectors is likely to involve the collection of new data, which may not be easily available. Sector-specific indicators, although more accurate, are obviously less comparable across sectors. Similarly, innovation processes evolve over time, so a metric that represents today's innovation very accurately may miss tomorrow's innovations completely, and result in meaningless comparisons between both periods.

Measuring innovation may require new data

A far greater understanding of innovation may be achieved by interrogating and aggregating existing data sources – for instance, distinguishing by sector, and focusing on sector specific metrics that measure relevant innovation.

But fully accounting for hidden innovation will require new data. This presents a major challenge for statistical agencies, which often rely on what companies already track, since many firms do not yet monitor their own innovation activities in a consistent manner.

Effective presentation is essential

For data to be effectively used by policymakers and the media, it must be presented in an accessible format – sufficiently aggregated to be meaningful, but detailed enough to identify any underperforming areas. To be most effective, innovation performance data should be presented by industrial sector, geographic region and innovation system function.

The Innovation Index project is an opportunity to develop and deliver better measures of innovation

Innovation Nation stated that NESTA will "...develop an Innovation Index to measure UK innovation in the round drawing on input and expertise from partners such as the ONS, DIUS, BERR, AIM, the CBI and others. A pilot index will be published in 2009 with a fuller system in place by 2010".²¹

The Innovation Index project will address the main challenges of measuring innovation

The new Index will seek to track the most relevant innovation to the UK and seek to embed its findings in future policymaking. It will look to capture hidden innovation, while recognising the structure of UK's economy. And it will provide clear insights for the main innovation actors in the UK, establishing a firm evidence base for future policy.

It will also set an agenda for further research and provide a framework for a systematic body of work on innovation measurement with recommendations for the improvement of underlying statistics.

Collaboration is essential to success

Understanding innovation requires specialised knowledge distributed around the UK – in firms, universities and trade bodies. Moreover, the UK 'innovation system' is, in fact, the sum of multiple, smaller sectoral and local innovation systems. Moreover, innovation systems that benefit the UK reach well beyond its geographical boundaries.

As a consequence, NESTA has built a collaborative project plan, involving partner organisations in many parts of the Index design and delivery process.

NESTA is launching a Call for Ideas

NESTA is seeking contributions from individuals and organisations that can help develop the best measures of innovation for the UK. This will inform the development and implementation phase of the Index, starting in October 2008. The consultation closes on September 12th. More information is available on the Innovation Index website.

innovationindex.org.uk

The Innovation Index website aims to be the focal point of the Index project. It provides a collaborative space where individuals can access materials, share ideas and suggest and critique potential solutions. Our final goal is to build a community of interested individuals who can collectively guide the development of the Innovation Index.

NESTA Policy & Research Unit 1 Plough Place London EC4A 1 DE [email protected] www.nesta.org.uk

Footnotes