---

Innovation policy mix and instrument interaction: a review

Paul Cunningham Jakob Edler Kieron Flanagan Philippe Laredo

Nesta Working Paper No. 13/20 November 2013

www.nesta.org.uk/wp13-20

Abstract

This paper is part of the Compendium of Evidence on the Effectiveness of Innovation Policy Intervention.

This review briefly discusses and develops a concept of policy mixes. This concept acknowledges that policy actions inevitably interact in a stream of existing events and activities, including other policy interventions. Policy mixes are characterised by the policy domains they cover (as mixes may not be confined to one domain), the underlying policy agendas and rationales, the targets (actors, processes) and the instruments employed.

A systematic search in both literature databases and specialised evaluation and innovation policy databases reveals a rather thin base of evidence regarding deliberate policy mix design and intended or unintended interactions between instruments.

The empirical findings first concentrate on the interplay of individual instruments. For example, the review shows that the benefits of interventions to support R&D in firms can be improved if accompanied by complementary support in terms of HR and skills, managerial support or market introduction of new products and services. It also shows examples of instruments portfolios that are targeted towards specific technologies, highlighting the importance of a clear definition of the underlying catalogue of goals and their relative importance, a clear understanding of the effects of the instrument mix over time and the sequencing of interventions.

The review, second, finds a limited take-up of policy mix thinking, often reduced to one lead agency or ministry in one country. In addition, policy mixes most often develop when individual measures that are designed to address different systemic gaps over time. However, existing reviews at country level do not make any systematic attempt to analyse overall effects of the mixes.

In a final empirical section, our review presents evidence on four cases of deliberate policy mix designs in different countries. This shows the diversity of designs and emergence of policy mixes and the different ways in which ministries and agencies approach the issue of mixes.

The study concludes with the shortcomings of evaluation practice, which has far to go regarding the systematic capture of effects of mixes and interplay, partly due to a lack of awareness, partly due to methodological limits for the assessment of complex portfolios and interplays.

JEL Classification: 038 Keywords: Innovation policy mix, interplay of instruments

The Compendium of Evidence on the Effectiveness of Innovation Policy Intervention Project is led by the Manchester Institute of Innovation Research (MIoIR), University of Manchester, and funded by Nesta, an independent charity with the mission to make the UK more innovative. The compendium is organised around 20 innovation policy topics categorised primarily according to their policy objectives. Currently, some of these reports are available. All reports are available at http://www.innovation-policy.org.uk. Also at this location is an online strategic intelligence tool with an extensive list of references that present evidence for the effectiveness of each particular innovation policy objective. Summaries and download links are provided for key references. These can also be reached by clicking in the references in this document. Corresponding Author: Paul Cunningham, [email protected].

The Nesta Working Paper Series is intended to make available early results of research undertaken or supported by Nesta and its partners in order to elicit comments and suggestions for revisions and to encourage discussion and further debate prior to publication (ISSN 2050-9820). © Year 2013 by the author(s). Short sections of text, tables and figures may be reproduced without explicit permission provided that full credit is given to the source. The views expressed in this working paper are those of the author(s) and do not necessarily represent those of Nesta.

This report is part of the Compendium of Evidence on the Effectiveness of Innovation Policy Intervention Project led by the Manchester Institute of Innovation Research (MIoIR), University of Manchester. The project is funded by the National Endowment for Science, Technology and the Arts (NESTA) – an independent body with the mission to make the UK more innovative.

The compendium is organised around 20 innovation policy topics categorised primarily according to their policy objectives. Currently, some of these reports are available:

All reports are available at http://www.innovation-policy.org.uk. Also at this location is an online strategic intelligence tool with an extensive list of references that present evidence for the effectiveness of each particular innovation policy objective. Summaries and download links are provided for key references. These can also be reached by clicking in the references in this document.

List of Figures

1. Figure 1: Market transformation - policy measures and market maturity level
2. Figure 2: Comparison of effects of policy tools: isolated and combined

List of Tables

1. Table 1: Five forms of influence or confluence in policy instrument blends or mixes
2. Table 2: Important elements underpinning composition and interaction in the policy mix from the literature
3. Table 3: Policy mix interaction types and dimensions
4. Table 4: Comparison of direct and indirect measures to support R&D
5. Table 5: Differences in target groups and effects between indirect and direct measures
6. Table 6: The instrument mix within the German High Tech Strategy
7. Table 7: LMI measures in the six markets

Executive Summary

The reports within the review of evidence on innovation policy measures compiled in the NESTA Compendium have focused on the impact of specific types of innovation policy measures. This report complements those 18 reports of the Compendium by looking at the extent to which what we will call "policy mix" issues have been considered in the evidence base for innovation policy. The "policy mix" concept acknowledges that policy actions inevitably interact in a stream of existing events and activities, including other policy interventions. Policy interventions (or instruments) may interact with the portfolio of already existing interventions for a given target group, technology, sector or societal issue. When designing or assessing policies and policy instruments, existing interventions are inevitably part of the context or may necessitate complementary intervention to work best. The goals and modalities of different policies or instruments may conflict and the effects of instruments may reinforce or contradict and weaken each other. However, the interplay of policies and instruments, as well as the deliberate design of policy mixes and portfolios of interventions, has received surprisingly little practical and theoretical attention so far.

To understand the existing evidence on mixes, the review briefly discusses and develops a concept of policy mix. It reviews the recent attempts at conceptualization that have been offered, from both prescriptive and analytical perspectives. All such attempts highlight the importance of understanding the composition of policies as well as their possible and actual interaction. On the basis of these recent conceptual developments, the report develops a simple concept of policy mixes characterised by the policy domains they cover (as mixes may not be confined to one domain), the underlying policy agendas and rationales, the targets (actors, processes) and the instruments employed. The instruments within mixes inevitably interact as they target (1) the same actor groups, (2) different actors involved in the same social or economic processes or (3) different processes across policy domains. Furthermore, instruments interact over time, both within and across policy domains (issues), policy spaces (actors) and geographical areas (levels). Finally, the concept distinguishes between the interactions of policies and mixes on the one hand and the deliberate design and implementation of instrument mixes, on the other. It should also be noted that path dependency, i.e. past policy developments and national policy styles, can shape the development of policy mixes by influencing policy choices.

A systematic search in both literature databases and specialised evaluation and innovation policy databases reveals a rather thin base of evidence regarding deliberate policy mix design and intended or unintended interactions between instruments.

The discussion of the interaction of instruments in the first empirical part of the report is organised around five dimensions. It first suggests that the benefits of interventions to support R&D in firms can be improved if accompanied by complementary support in terms of HR and skills, managerial support or market introduction of new products and services. Secondly, and in the same vein, for business and innovation measures more generally the existing evidence highlights the importance of interaction for the design and implementation of policy to improve its effectiveness. It also shows one recurrent gap, i.e. the lack of follow up support after direct R&D support or collaboration schemes: valorisation and commercialisation are left to the market, while in many cases this sequential stage may need additional support. It should be noted that these findings are based on evaluations that identify a lack of complementary measures, rather than through evaluations of complementary measures. Thirdly, the review presents the findings of evaluations of portfolios of instruments that are targeted towards specific technologies. All these examples are situated in the energy sector which has a tradition of portfolio approaches and where policy mix thinking is more advanced. These analyses show that the successful deployment of multiple instruments starts with a clear definition of the underlying catalogue of goals and their relative importance (e.g., energy efficiency vs. innovation effects within a given country). They also stress the importance of a clear understanding of the effects of the instrument mix over time and the sequencing of interventions. This also points to the need to design specific interventions for specific stages of technological and market development. Fourth, reflecting the rise of interest in indirect measures and their effects, the review summarises the effect of the interplay between direct and indirect measures. Again, a number of studies recommend the design of intelligent combinations, whilst only a very few have actually looked at such effects empirically. The findings of the analyses are not very surprising: firms that are supported with a direct grant are more likely to take advantage of a tax incentive scheme as well, and firms benefitting from both types of intervention show a higher level of innovation input and output and a higher propensity for radical innovations. Fifth, an analysis of the interplay of public procurement and supply side measures indicates that firms which benefit from both interventions have a considerably higher input additionality, but there is no significant effect on the share of innovative output. However, the search for evidence on the interplay of traditional supply side measures with demand side interventions is still in its infancy.

In a second part, the review looks at evidence about policy mix thinking at the national level, Our review shows a limited take-up of policy mix thinking in the literature, often reduced to one lead agency or ministry in one country. A systematic review of EU country R&D policy mixes in 2008 found no explicit policy-mix designs and implementations in those countries. However, the study confirms that path dependency is the critical factor underlying the formation of policy mixes: individual measures are designed to address systemic gaps over time and the policy mix develops as a consequence: the mix itself is not designed as an entity but is shaped by the interplay of the design features of its component measures. We find a limited number of broader country reviews, intended to be learning tools for policy makers that apply the policy mix approach as an analytical lens. On the basis of a qualitative assessment, they recommend a reduction in complexity by simplifying the mixes and the adoption of a more "balanced mix" approach (e.g. between supply and demand side measures). Due to the lack of underlying data, these reviews are unable to comment on the effects of the specific mix they find, but our review shows that they can be a useful tool to encourage reflection about potential complementarities and conflicts between policies.

In a final empirical section, our review presents evidence on four cases of deliberate policy mix designs. The German High Tech Strategy illustrates the design of a cross-government mix based on a unifying strategic vision which is institutionalised over time; not through creating new agencies or centralising its implementation, but through the bundling of existing activities, the subsequent design of new and complementary instruments implemented by diverse agencies and ministries with their own budgets, and monitored and supported by an accompanying evaluation. The European Lead Market Initiative is a multi-measure framework to support market building based on public procurement arrangements. The review shows how the communication and concertation of four different measures mobilised a small - transnational community and changed awareness and behaviour as a first step to the creation of markets. However, the framework of instruments had a wide variety of effects in different sectors, showing the need for a tailored implementation of mixes. It further shows how the discourse on an explicit framework served as a reference for policy makers at different levels. Two further examples highlight the effects of a portfolio of instruments implemented by two agencies. In a first example, a comparison of two portfolio reviews concluded that an agency with a broad remit in research and innovation funding, the Research Council of Norway, changed its portfolio over time in reaction to an earlier evaluation. The French Energy Agency ADEME is an example of an agency with a broad set of interventions. These interventions are managed through a broad interactive process with stakeholders, and a tailored application based on specific needs and gaps. Importantly, this mix differs between technologies, and changes over time. The evaluations of the portfolio and its elements over time show an effect on the overall change in energy systems. However, these effects or the specific contribution of the ADEME instruments are not quantified, rather they are established using individual instrument evaluations and the monitoring of energy consumption change and, on that basis, a broad and institutionalised discourse with other stakeholders involved in the energy systems and in the definition of programmes.

The study finally considers the shortcomings of evaluation practice regarding the assessment of policy mixes and interplay of instruments. The evaluation community has far to go regarding the systematic capture of effects. Assessments of the evolution of mixes and their changing effects over time are extremely rare, but are crucial to developing an understanding of the interplay of their component measures over time. We do not have evaluations that examine the interplay of instruments between levels and, given the organisation of evaluation practice, we lack any meaningful evaluation of instruments implemented by different ministries and agencies, especially across different policy domains. Furthermore, while 'traditional' methods may be able to cope with the interplay of two or three instruments, they are overwhelmed by any attempt to delineate the contributions of individual instruments within broad mixes. Finally, evaluations often fail to understand the specific implementation challenges associated with implementing a policy mix. Until new approaches are developed, any evaluation of complex mixes will need to continue to rely on assessments of the combined effects of instruments based on parallel evaluations.

1 Introduction

The innovation policy Compendium reports have reviewed a large body of evidence on the impact of specific innovation policy instruments. Reflecting the general practice in evaluation, this approach has led to a perspective that isolates individual instruments from the overall policy context they are in. It also tends to neglect policy designs that employ more than one instrument in order to target a specific actor group or reach a specific policy goal. However, looking at the multitude – and overlap - of policy goals, at the variety of instruments that target the same actor group and at the overlap of intervention rationales between instruments, one has to look at how policies or instruments interact. This is important to understand the relative contribution of an instrument, but also in order to understand synergetic or counter-productive effects through this interaction. For policy makers it raises the question as to whether instruments can achieve synergies and positive complementarities, and what principles and processes to follow in order to minimise any negative interaction between instruments. More generally, it begs the question whether (and how) policy makers can actively design a mix of policies and instruments.

Against this background, this report reviews evidence on policy mixes and the interplay of instruments in innovation policy. It introduces a conceptualisation of mix and interplay (section 2) and summarises the methodologies employed for the review (section 3). It then reviews the evidence we have on policy mixes and instrument interplay. This is done along three major lines: first, we look at evaluations which have explicitly looked at how instruments interact (section 4), we review studies or peer reviews that look at the policy mix at the country or system level (section 5) and we look at instances where individual policies or instruments have been deliberately used together, as designed mixes across policy institutions or as portfolios of specific agencies (section 6). We finish by commenting on the evaluation of mixes (section 7) and by summarising the main lessons derived from the review (section 8).

2 The policy mix for innovation: the emergence and development of the concept

2.1 Emergence of the concept

An interest in possible complementarities or tensions between innovation policy “instruments" is nothing new (see e.g. Smith, 1994; Branscomb & Florida, 1998), and the shift in emphasis towards more “systemic” views of innovation in the late 1980s/early 1990s implies - though has rarely delivered a greater focus on the range of policies that might affect the relationships and processes that underpin innovation. However, explicit discussion of the "policy mix" affecting innovation processes is a fairly recent development in the innovation policy discourse.

The term has a longer history in economic policy debates, being coined by Nobel Economics prize-winner Robert Mundell (1962). During the 1990s and early 2000s the phrase came to prominence in the policy studies literature (see e.g. Howlett, 2005) by way of the literature on environmental policy and regulation (e.g. ETAN Expert Group, 1998; Sorrel and Sijm, 2003). The policy mix concept seems to have found its way into the European innovation policy discourse via these two routes.

Analysts and policy-makers alike have grasped at the concept not only to help deal with the growing complexity of the innovation policy agenda in a systemic world, but also to help rationalize the relative failure of two decades of active R&D and innovation policy efforts at the European level to transform the innovation performance of the European economy. Most recently this thinking can be seen in the OECD Science, Technology and Industry Outlook (2010a), which devotes an entire chapter to "The Innovation Policy Mix” (OECD, 2010a), and in the OECD Innovation Strategy document of the same year (OECD, 2010).

2.2 Issues and challenges of policy mix analysis

2.2.1 Composition and interaction

At its most basic, policy mix thinking must be concerned with the composition of policies "in the mix" and with how they might interact to mutually shape each other's effectiveness. In particular, the idea of interaction between policies is fundamental to the policy mix concept. At a basic level, it is possible to hypothesise that policy mix interactions might take the form of complementarities between instruments (so that the presence of one instrument in the mix increases the effectiveness of another) or trade-offs (where one instrument attenuates the effectiveness of another). It might also be that one instrument has no effect on another when both are present in a mix. Gunningham and Sinclair (1999), discussing environmental policy mixes, go further, hypothesizing four classes of interaction in an instrument mix: instruments that are inherently incompatible; instruments that are inherently complementary; instruments that are complementary if sequenced in a particular way; and instruments whose complementarity or otherwise is essentially context-specific. Writing in the general political science literature, Bressers and O'Toole (2005) identify five classes of interaction between instruments in a policy mix, depending on the actors or groups, processes targeted by policy, on the one hand, and the interdependence between different domains of policy action, on the other (see Table 1, below).

In a similar vein, the factors which might be important in thinking about composition and interaction in innovation policy mixes have been explored by OECD (2010a), by Flanagan et al. (2011) and most recently by Borrás and Edquist (2013). Broadly, OECD and Borrás and Edquist are preoccupied with providing a prescriptive model of how innovation policy mixes can or should be designed whilst Flanagan et al. (2010, 2011) are concerned with establishing a basis for better understanding the evolution of actual policy mixes as a precondition towards any evaluative or prescriptive efforts. Below we summarise and synthesise the key elements of these three approaches to suggest a working conceptualisation.

Source: Flanagan et al, 2011, based on Bressers and O'Toole, 2005.

OECD (2010a) offers a framework for thinking about composition and interaction which emphasises four inter-related considerations, namely: the policy domains in question; the rationales offered in support of policy intervention; the strategic tasks pursued; and the policy instruments deployed (OECD, 2010a, p26). Similarly Flanagan et al. (2010, 2011) emphasise policy subsystems (the network of state and non-state actors and institutions that shape policies focused on a particular problem area in a particular jurisdiction at a particular time), rationales, policy goals, policy targets (actors or processes in society or economy targeted for behaviour change by policy action) and interacting instruments. Most recently Borrás and Edquist (2013) offer a framework that emphasizes policy objectives (derived from identified systemic problems), activities in the innovation system (to be targeted by policy intervention) and instrument selection (done in the context of national policy styles). Clearly, despite the different objectives of these three sets of authors, there is some commonality in the basic conceptualization (summarized in Table 2, below).

2.2.2 Instrument selection and the policy mix

Borrás and Edquist (2013) emphasise the influence of national policy styles in shaping policy instrument choice. Flanagan et al. (2011) stress that decisions about implementation are key to the 'interpretive flexibility' of policy instruments in the real world, and have a key role in influencing outcomes and interactions suggesting that policy instruments are not the substitutable "tools" they are often claimed to be. Both OECD (2010a) and Flanagan et al. (2010, 2011) emphasise that mixes cannot be reduced simply to portfolios of policy instruments - OECD identify "mixes" for each of their four key elements (domains, rationales, tasks and instruments) whilst Flanagan et al. argue that policy mix interactions between instruments will stem from tensions, trade-offs or complementarities between rationales, goals and approaches to policy implementation. Borrás and Edquist (2013) are centrally concerned with designing instrument mixes, rather than understanding policy mixes more broadly, although they do emphasise the importance of problems, objectives, activities and national policy styles in shaping instrument selection and thus they do at least imply that these factors can interact.

2.2.3 Designed versus emergent mixes, and the time dimension

The early EU and OECD policy literature which pioneered the application of the policy mix concept to innovation policy analysis implicitly tended to treat mixes as designed portfolios, the result of purposive action and co-ordination (Flanagan et al., 2011). As already noted, Borrás and Edquist (2013) also emphasise designed mixes, though they acknowledge that real policy mixes are shaped over time.

In contrast, Flanagan et al. emphasise the emergent nature of policy mixes affecting innovation processes and their evolution over time. Much policy analysis is undertaken on a "snapshot" basis (Kay, 2006) yet public policy unfolds over time, exhibiting non-linear, sometimes path-dependent dynamics¹. These policy dynamics co-evolve with other dynamics - such as budgetary, political and economic cycles. Goals, rationales and instruments emerge, evolve, are institutionalised or fade away. Systemic roles played by actors may change and actors will learn over time - meaning that the response of actors and processes targeted by policy action may change over time, even where the intervention remains the same. New policy instruments do not emerge into a vacuum but represent an intervention in a continuous stream of unfolding developments. Thus, even with a designed mix, it is unclear to what extent theorised complementarities between policy interventions will be borne out in practice: at some point, instruments which may seem in principle to be complementary may begin to interact in unpredictable or negative ways. Thus, the interactional characteristics of policy instruments are always potentially transient.

2.2.4 Dimensions of the policy mix

Flanagan et al. (2011) propose a conceptual model of policy mix interactions that emphasises several dimensions. This is summarised below in Table 3. Thus they argue that policy mix interactions can occur across the policy space represented by different policy sub-systems, between different levels of governance (governance space), across geographical space and over time. Modifying the ideas of Bressers and O'Toole (2005) they propose three types of interaction between different instruments based on the target of the intervention. Further, because policy instruments are flexible and change over time, they also propose a fourth kind of interaction, between nominally similar instruments across the policy, governance, geographical and time dimensions.

Source: adapted from Flanagan et al, 2011

2.2.5 Coherence and co-ordination of the policy mix

Improving the coherence and coordination of the broader policy mix affecting innovation processes has been a central preoccupation of the policy mix literature, especially in its EU and OECD forms. OECD (2010a) suggest coherence can best be thought of as a goal whilst coordination should be seen as the means of achieving that goal. In turn, they argue that coordination must be informed by evaluation evidence. Mechanisms such as high-level policy coordination councils, shared cross-governmental visions and foresight exercises are identified as means by which coordination can be improved (the merger of policy ministries into "super-ministries" is also mentioned). However introducing additional structures and processes may actually make coordination challenges more severe (Flanagan et al., 2011).

Both OECD and Flanagan et al. emphasise the significant co-ordination challenges presented by the complexity of the policy mix likely to affect innovation processes and outcomes. In contrast, Borrás and Edquist (2013) do not address these co-ordination challenges. In their idealised approach, coherence comes from careful problem identification and policy mix design. In this view, if the right decision framework is rationally applied to the question of instrument selection and layering, then a coherent mix should always result.

2.2.6 Summary

Based on the above review, we can derive a simple framework for thinking about policy mix composition and interaction. This emphasises complementarities or tensions and contradictions between policy domains or sub-systems; the policy agendas which identify problems to be resolved and the policy rationales offered in support of policy intervention; the targets of policy action and the selection and implementation of instruments. Tensions or complementarities can arise, and therefore interactions occur, between instruments addressing the same or different targets across the dimensions of policy space (domains or sub-systems), governance space (levels of governance), geography and over time. Policy makers may seek to address these potential sources of tension, or take advantage of potential complementarities, by efforts at improved coordination, or by attempting the rational, stepwise design of a coherent policy mix. There are enormous challenges for evaluation in seeking to understand whether interactions arise from these sources of potential tension or complementarity (in both designed and emergent/evolving mixes) but the effort should be made. Evaluation should also critically explore the extent to which coordination and design efforts actually do avoid tensions or take advantage of complementarities.

3 Scope and method of this report

Having briefly reviewed the emergence and conceptualisation of the "policy mix" concept in innovation policy analysis, we now turn to a review of the evidence as regards interaction between instruments, designed portfolios or other policy mix thinking. We began by defining a range of key words that are related to mix, interplay and complementarity of instruments. With those key words, we first searched the ISI web of knowledge and Google Scholar for academic literature in the area of STI policy. Second, we scanned an existing database of innovation policy evaluations (InnoAppraisal) to find reports that explicitly discussed and analysed the interplay between policies/instruments. Both searches unearthed remarkably few examples. Third, we reviewed all the existing 18 NESTA Compendium reports for systematic discussion and analysis of dedicated policy mixes and portfolios as well as policy interplay. Fourth, we systematically reviewed existing country reports provided by the OECD and EU reviews (CREST/ERAC) which are not evaluations but rather are based on expert assessment and secondary data, but which sometimes reach conclusions about the "appropriateness" of mixes. Fifth, we re-visited a research project that conceptualised and described the R&D (rather than innovation) policy mixes of different countries². Again, this project is more a description and an expert assessment about the mix than a rigorous empirical evaluation of impact. Sixth, we have analysed recent system evaluations (Czech Republic, Norway, Austria, Finland) to understand whether (and if so, how) the mix of policies and instruments at system level has been analysed.

As already noted, evidence for the impact of policy mixes and interplay is remarkably thin. As discussed in section 2, policy "mix" and "interplay” are concepts most often used in a normative sense, as part of a demand for improved policy making and improved evaluation. As a matter of fact, however, we find only very few instances both of deliberate designs of mixes, and of systematic evaluations of the interplay of policies and instruments. For example, our key word search in the InnoAppraisal database (Edler et al. 2012) showed 14 instances of "policy mix", all of them referring normatively to a need to see the bigger picture, the "mix", and none of them related to any actual systematic investigation of a policy mix.

4 Evidence on interplay of individual instruments

4.1 Introduction

Policy makers and analysts understand that the way an instrument impacts on behaviour, innovation input and innovation output is connected to the ways in which other framework conditions and instruments influence the same target groups and technologies or pursue the same policy goals. Nevertheless, one important finding of the NESTA Compendium exercise as well as the Inno Appraisal study (Edler et al. 2012) is that evaluations of policy instruments are largely done in isolation. Very rarely do we see a conscious, explicit attempt to evaluate that interplay. In this section we present a few notable exceptions in order to illustrate the benefits and limits of existing practice to understand impact more holistically. We summarise some key examples of interplay derived from the analysis across the board of our Compendium reports as well as our additional searches. It appears that the area in which interplay is most commonly analysed is the demand side, where measures influence the purchasing decisions of consumers and firms and thus the uptake and diffusion of innovation.

4.2 R&D support measures

General R&D support measures: need for complementarity

Some attention has been addressed to the lack of complementary measures with regard to the supply of direct subsidies for R&D in firms. Remarkably, however, the evaluations that address the issue all do so by identifying "gaps” in the portfolio and by recommending the addition of complementary measures, rather than by evaluating the complementarity of existing instruments. For example, a review of the French agency ANVAR's “Procédure d'aide au projet d'innovation" (de Laat et al., 2001) proposed that young companies utilising the scheme could "benefit from complementary services other than financial aid, especially concerning their connection to external partners” which could include specific services for young companies and coaching to starting entrepreneurs. Likewise, a review of the UK Small Firms Loan Guarantee (SFLG) (Cowling 2010) found that "as a significant minority of SFLG supported businesses are seeking to innovate and/or expand into new geographical, particularly international, markets, there may be a case for SFLG supported businesses to be offered advisory support programmes in parallel with their financial support". Lastly, an evaluation of a large Japanese medical technologies programme (PREST, 2003) noted that "the programme should offer complementary measures to assist small firms with preliminary (international) market studies before they commit to a full project".

These and similar findings led to the following conclusions in the Direct Measures report in this series (Cunningham et al. 2012)

• Support complementary to direct subsidies for firms is important to allow the likelihood of successful outcomes by developing the capacity of the recipient firm to capitalise on all aspects of the supported project and build up capabilities to be used in other and in future projects. This needs to be balanced against the increase in overall costs, the reduction in the number of firms supported and increasing the coordination costs.
• The combination of direct measures (which may be employed in a strategic manner by governments, for example by targeting specific sectors, regions or types of firm) and fiscal incentives (which engage firms in a broader, reactive fashion) seems to represent a effective mix of measures (below we provide empirical evidence for this claim).

R&D collaboration and complementarity

The need for complementary measures to accompany schemes that foster R&D collaboration is also expressed in the evaluation reports reviewed. However, evidence as to interplay is rare. In the late 1980s, when collaborative R&D programmes were on the rise, the landmark evaluation of the UK ALVEY scheme (Guy et al., 1991) recommended that "R&D support should at the very least be accompanied by complementary action to address skills shortages". This point was reinforced by many evaluations and reviews analysed by Cunningham and Gök (2012) in this NESTA Compendium, leading the authors (in line with, e.g. Lambert, 2003) to conclude that collaboration in R&D is one stage in the overall innovation process and that the tangible outputs and intangible outcomes of that stage may lead to further demands for policy support and to "align collaborative support programmes within a broader supporting and complementary set of policies which can capitalise on their outcomes (both tangible and intangible)" (Cunningham and Gök, 2012, p. 5).

A recent review of the UK Collaborative R&D Programme (PACEC, 2011) examined its relation to the alternative sources of funding available in the event of unsuccessful applications to the scheme. In this regard, the authors sought to determine which alternative sources of (public and private) finance participants would have applied for if they had not been successful. This helps to understand the degree of overlap, and define which gaps the programme fills. In terms of the alternative sources of public support, 12% said that they would have applied for funding under the (then) Grant for Research and Development (now SMART) and 5% would have sought funding from the (former) Regional Development Agencies. What is particularly interesting is that the alternative sources of funding were all direct support for R&D; alternative collaboration schemes (Knowledge Transfer Networks and Knowledge Transfer Partnerships) were not considered as potential sources. This suggests that the Collaborative R&D Programme has been complementary to other existing schemes, its mode and target groups sufficiently distinctive from other schemes, avoiding overlaps and crowding out effects.

4.3 Interplay of business support measures

A few studies have looked at the interplay of various kinds of direct support measures. A synthesis study of a variety of UK business support schemes – based on existing evaluations conducted in 2009 (SWQ 2009), did include the remit, inter alia, to:

• to assess the relative impact of different types of interventions and
• to "examine the degree to which the interventions were competing or complementary in terms of this contribution".

In itself, this is one of the very rare examples of attempting to understand a policy portfolio across a policy area. However, as it was based on reviewing existing evaluations, the available data and analysis as to the interplay of measures was somewhat patchy.

The report mainly looked at SME support policies, regional policy schemes, business support measures and energy policies. It found evidence that there were issues both of complementarity and competition in the same policy domain. The authors define two levels of complementarity. Operational complementarity describes the "degree to which programmes and projects reinforce each other in their delivery on the ground", whereas strategic complementarity refers to the extent to which "policies and programmes are designed and introduced in a coherent and harmonised way to contribute to the same higher level objectives". The study notes that evaluations that take complementarity into account tend to focus on operational rather than strategic complementarity.

The broad findings of the review were that (SQW 2009, executive summary):

• For business support interventions the main issue was operational duplication (rather than complementarity) between services and the interaction with other sources of advice and support (especially from the private sector).
• The interaction most evident in the small firm policy interventions was the strategic and operational one between Business Link, other public sector interventions such as the Manufacturing Advisory Service and private sector provision of advisory services.
• The energy policy interventions present a strategic issue of complementarity or competition which was not addressed in the evaluations, namely the potential tension of supporting different energy forms at roughly the same time."

The authors recommended that “assessment of the degree and nature of interactions between interventions and their effects should be a matter of course in the design of interventions and their evaluation” – in other words, that the potential inter-relationships between existing and planned policy interventions should be considered in the formulation and design stage of any new policy instrument (ibid, p. vi).

4.4 Portfolios for specific technologies - Examples from the energy efficiency area

In the area of energy policy and energy efficiency measures we found a couple of analyses that looked how different measures impact upon the absorption and diffusion of energy efficient technologies over time. A first example is an analysis of the Danish and Norwegian wind industry (Buen, 2006). Buen's starting claim is that the "cumulative impact" on "permanent technical change at company and sector level" increases if

• individual instruments are designed for specific needs
• the "totality" of instruments covers all needs in the target arena and stimulates both technology push and pull (supply and demand side)
• the employment of different policy instruments is coordinated.

Interestingly, this definition of important criteria does not include the interplay of instruments over time, even if the analysis is about the effects over time. When analysing the impact, Buen shows how the succession of different instruments has influenced the development of total installation and efficiency gains in Denmark and Norway over the last 30 years. The Danish case is one of long-term success mobilising dynamic technical change. A first wave of installations was pushed by investment subsidies (to local residents), rigorous approval processes and awareness campaigns. A second wave was influenced by sending long-term signals (long-term planning), guaranteed grid connections and feed-in guarantees from large suppliers as well as an increase in energy surcharges. An abrupt change of policy in the late 1980s (reduction of incentives on the supply and demand side), in conjunction with a slowdown in foreign demand, subsequently led to a crisis in the industry. After 1994, installed capacity grew due to the clarification of framework conditions and long-term expectations (government contracts, long-term goals) as well as an increase in foreign demand. In anticipation of a stricter price regime planned for 2001, installation peaked towards the year 2000. After that, the policy turned towards an upgrade of efficiency, driven by a new feed-in tariff to replace old turbines with newer ones. Those factors that were argued to be conducive to this overall policy success of the Danish innovation story in wind turbines included the build up of a general consensus on the overall importance of this area, the long-term focus, predictability and commitment, the linking of growth and sustainability goals, the combination of supply and demand measures, and the specific tradition of Danish cooperation, ensuring long-term support through subsidies targeted at co-operatives.

In contrast, Buen's analysis of Norway concludes that a conscious sequence of instruments was missing. Enormous R&D subsidies in the late 1970s and early 1980s were not followed up by further tailored supply or demand measures. When demand measures were finally set up, they favoured foreign designs. Overall, Buen concludes that the Norwegian mix was designed to secure energy supply rather than to support domestic innovation in the sector, lacked demand side measures altogether and thus framework conditions and instruments were too unpredictable to provide domestic industry with consistent incentives.

An important first lesson from this for policy making is the need to re-visit the instrument mix that is offered over time, to remove instruments that have served their purpose and which may start to become detrimental and to maintain long-term expectations in the market. Secondly, the overall objective of the mix of instruments and the relationship between partly competing objectives needs to be clear rather than oscillating between them. Finally, local framework conditions and traditions make a difference, and policy instruments need to be tailored towards these specificities.

A second example is provided by Johnstone et al. (2010) who analyse the effects of different policy instruments on patenting in specific energy efficient technologies in a range of countries, using an econometric approach. This allows an analysis of the impact of individual instruments on innovation across different technologies. The instrument mix is characterised by a sequencing of policies: the introduction of R&D support in the 1970s was followed by the introduction of investment incentives, tax measures and feed-in tariffs (starting in the late 1970s and peaking in the 1990s), voluntary programmes (1990s), obligations (1990s) and tradable permits (2000s), (ibid, p. 144). This demonstrates the growth of the instrument mix over time, as most of the interventions introduced, in one form or another, remained in place. The authors show how the same policies affect different technologies differently. For policy making this means that in order to achieve a desired mix of energy technologies one would need an appropriate policy instrument mix. For example, feed-in tariffs establish markets even for high cost technologies (solar), while energy certificates drive innovation in wind power. Significantly, the authors then also look at the co-linearity of instruments. They analyse how the impact of two policies (feed-in tariffs and renewable energy certificates) changes with the presence of another policy. Except for one example (ocean technologies and renewable energy certificates) there are no significant effects. In sum, the authors do not find strong evidence of significant change of impact of one instrument (feed-in tariffs or energy certificates) on patent activities through the interplay with another instrument. From this econometric analysis it appears that for a given technology, it may be the sequencing of instruments over time that is of key importance, rather than the actual interplay of instruments at a given phase.

A third example in the area of energy technology is the mix of instruments deliberately created to transform markets for a defined energy-efficient innovative technology. The object was to shift diffusion patterns towards innovative solutions that are more sustainable by combining different instruments on the demand side. For any given technology to be supported, a specific mix of interventions was deployed, again with adjustments over time as the technology matured and its acceptance rose. The Compendium report on demand side measures has discussed Swedish examples from the 1980s in some detail (Edler 2013, p. 26-28). Each mix was tailored to take account of the specificities of each technology and consumer behaviours (Geller et al., 2001).

The figure below shows the combination of a large range of demand measures for different phases of an innovation and diffusion cycle (see Figure 1), complemented in some instances by R&D support of various kinds. The instruments deployed were mainly within the responsibility of the energy agency STEM/NUTEK.

Source: Neij (2001, p. 69) ¹, taken from Edler 2013, p. 28

As can be seen, the mix of policies changed between different phases of the market development. Further, the mix is targeted at different kinds of actors, end users, producers and intermediaries of various sorts (trade, installation). At the beginning of each activity, a broad analysis of the technology (and its societal benefits when deployed), and the behaviour patterns of the various actors in the related innovation system was conducted. The analysis defined opportunities and bottlenecks in terms of capabilities, awareness, and connectivity, and thus delivered the basis for the definition of the mix of interventions. Each specific mix was designed with specific quantitative targets in mind and a strategy for the phasing out of interventions once certain thresholds had been reached. The well documented Swedish approach to support energy efficient technologies is centred around the idea that public procurement can be the catalyst for diffusion. Thus, bundling demand and defining forefront specifications was at the core of most approaches, harnessing public purchasing power to leverage broader markets. However, a variety of awareness raising (marketing campaigns of various sorts) and education measures plus financial incentives were deployed that sought to trigger a broader, private market:

• Media campaigns, press releases, interviews
• Targeted information for specific groups, brochures
• Labels and performance standards
• Further education of intermediaries (maintenance, installation).
• Mobilising of producers to support the demand base measures
• Demand subsidies for early adopters
• Demonstration projects (in specific circumstances)

Evaluations have found a generally positive impact both on market introduction and on diffusion, albeit with differences between areas, especially with regard to diffusion (Edler and Hafner, 2007, Neji, 1998, NUTEK, 1994, Suvilehto and Överholm, 1998, p.102-104). While market introduction was achieved in most of the approaches, diffusion patterns differed, as some programmes had difficulties in reaching consumers that were far from the initial adoption frontier. The factors contributing to the overall success of transformation that can be identified from this exercise are as follows (Edler 2013):

• a thorough prior analysis of the market (underlying technologies, supply and demand chains);
• user groups involvement²;
• bundling of demand (public-public and public-private);
• accompanying monitoring of market developments;
• sustained efforts, long term benchmarks.

Regarding the mix and interplay dimension, there was a single agency was responsible for the various approaches, from market analysis through to media campaigns, further education activities and supporting public procurers. This agency had a high credibility in the market, knowing both consumption patterns and the producing sectors and supply chains, and being able to mobilise public procurers.

4.5 Interplay of direct and indirect measures to improve R&D spending

The academic literature has uncovered substantive evidence in comparing direct and indirect forms of Government support: for example, a study of government support instruments in Shanghai by Zhu et al. (2006) found that "(stable) direct funding by government has a positive effect on industrial R&D investment whereas tax incentives led enterprises in the observed industrial sectors to switch to more general and less costly science and technology (i.e. low-tech) activities, which was seen as a less desirable outcome". Likewise, Carboni (2011) used a comprehensive firm-level data set in the manufacturing sector to compare the performance of direct measures and tax credit schemes in Italy. His results suggested that public assistance enabled recipient firms to achieve more private R&D than they would have in the absence of public support and that tax incentives appeared to be more effective than direct grants, although grants encourage the use of funding sources internal to the firm. Conversely, Grilli and Murtinu (2012), concluded from their study of Italian new technology based firms (NTBFs) that "selective R&D subsidies outperform other types of scheme in fostering NTBF performance"..

While a range of studies claim that an intelligent mix of direct and indirect measures offers beneficial complementarities to firms (see Cunningham et al. 2012, p. 26 and p. 41), only a few studies have actually looked at the interplay of direct and indirect measures. In a general study of Canadian industry, Bérubé and Mohnen (2009) examined the effectiveness of R&D grants for Canadian firms that were already benefiting from R&D tax credits. They found that firms that benefited from both types of policy measures introduced more new products, made more world-first product innovations and were more successful in commercialising their innovations when compared to their counterparts who only benefited from R&D tax incentives³. Corchuelo and Martínez-Ros 2009, analysing the Spanish tax credit, found that firms that receive a direct subsidy are more likely to take advantage of the tax credit, as their internal awareness and procedures in R&D make them more capable of doing so. Similarly, in the study of the Smart/Grant for R&D scheme in the UK in 2009 it was found that 30% of those firms that had a project supported subsequently claimed R&D tax credit for this project, since following the end of the grant period, lack of finance persisted as a major obstacle for R&D (PACEC, 2011; see also Cunningham et al.. 2012, p. 29).

The Austrian government has been concerned with the systemic effects and the overall efficiency and effectiveness of their framework conditions and policies in the area of science, innovation and technology. As part of a broad, very comprehensive "system evaluation", an analysis of the interplay of direct and indirect measures was performed (Falk 2009). The study was based on a broad company survey and used econometric techniques. This to our knowledge - is the first such thorough quantitative and qualitative analysis of the interplay of direct and indirect measures. The main questions asked were (Falk 2009p. 2).

• What is the relationship between effects of direct and indirect measures (contradictions, substitutive or complementary)?
• What are the overall net incentives out of that mix?
• Are the incentives compatible with each other?

The basis for the analysis is the conceptual distinction between the two types of instruments as shown in Table 4 below (Falk 2009)

Source: Falk. 2009

In Austria, there has been a relative shift to more indirect forms of support, which in 2005 made up around 50% of all public support for private R&D. Approximately 50% of direct support is directed towards the promotion of collaboration between firms, i.e. in contrast to many other countries, half the share for direct measures is spent on R&D activities by individual firms. Some 13.2% of industrial R&D was financed by public sector, 6.3% of this through direct support measures and 6.9% through tax schemes (ibid., p. 14).

The evaluation found a very broad overlap of direct and indirect measures. Around half of all R&D active firms in the country benefit from both direct and indirect support, roughly 80% of firms that benefit from tax incentives get private support in addition, and two thirds of those firms getting direct support also benefit from tax incentives (ibid., p. 23).

A range of interesting findings emerge:

• Firms who only receive direct support have a much higher research intensity and a higher growth rate of employment (which has to do with the fact that younger firms tend to get direct rather than indirect support)
• Input additionality: In general, firms that are supported do not show growth in R&D expenses, not even when direct and indirect measures are combined. Firms with high growth rates, however, do tend to benefit from direct support or a combination of support (but not from indirect support only) (ibid. p. 41)
• Output additionality: Support does not make a significant difference for incremental innovation or adaptation, in fact firms that are supported by direct measures only are even less likely to adapt, modify or introduce an innovation that those not funded at all
• Compared to firms that are not supported at all, firms supported by indirect measures have a higher likelihood to innovate radically (15%), as are those supported by direct measures (14%) and by both (21%) (ibid. p. 43), but there is no effect on adaptation or incremental innovation. This analysis used a rigorous econometric approach that controlled for other intervening variables.
• Thematic programmes in themselves tend not to lead to radical innovations; for this, a combination with indirect measures or other open support measures is needed (ibid. p. 43)
• There is no significant effect of either direct or indirect support on growth or employment: however, firms that benefit from direct and indirect support show 4.7% more growth than firms not supported (p. 49).

Overall, the econometric study concludes that indirect and direct measures are complementary. First, firms that benefit from both interventions in combination tend to show higher R&D input, better growth performance and are more likely to introduce radical innovation (Table 5). Second, the application of the two instruments should not be based on the question which is "better”, but rather, which firms are to be targeted and for what purpose. Early stage firms, R&D and firms in areas that do not fall squarely into the Frascati definition of research (which is the basis for indirect support) seem to benefit more from targeted direct support, while manufacturing firms with medium technology intensity seem to get greater benefit from indirect measures.

Source: Falk. 2009; p. 51-52

4.6 Demand and supply interaction

As demand based measures are an increasingly important part of innovation policies (OECD 2011), the question of the interplay between them and measures on the supply side is becoming more important. However, given that demand side measures have only recently gained prominence as explicit innovation policy tools in the policy and analytical communities, we have found only three studies that explicitly analyse this interplay. One of them is an interim analysis of the Lead Market Initiative (LMI), which applies a portfolio of instruments with public procurement support at its core. The LMI is described in more detail below (section 6.3). Two other studies look at public procurement and R&D subsidies: Guerzoni and Raiteri (2012) provide a rare analysis to test the interplay of public procurement and R&D subsidies on innovation output and innovation input. Their analysis starts with the assumption made by David et al. (2000), who suggest an interplay of reduction of R&D risk (through R&D subsidies) and market risk (public procurement). By looking at this interplay, they add to the literature on intervening factors for the impact of R&D subsidies on R&D investment. Using data from the Innobarometer survey 2006-2008, they differentiate between firms that had a public procurement leading to innovation and/or an R&D subsidy. In their model, both innovation output and input are dichotomous: i.e. increase in R&D subsidies (yes/no) and majority of turnover through innovation (yes/no). They then apply a non-parametric matching approach for each of the outcome variables.

Figure 2 shows their overall results. It indicates that both policy interventions individually, without any other public treatment, lead to input and output additionality. Comparing firms that only receive R&D subsidies to those that only benefitted from a procurement of an innovation, procurement appears to have higher effects, both on input additionality and on output additionality. This is roughly in line with a similar study by Aschhoff and Sofka (2009) who find a strong effect of public procurement on input additionality, but no significant effect of R&D subsidies.

Source: Guerzoni and Raiteri 2013, p. 22, figure 4. Note: The first and second pair of columns in each box depict the effect of innovative procurement and R&D subsidy respectively together with any other public treatment.

Concerning the interaction between instruments, two observations are crucial. First, the effect of both instruments is considerably weaker if the instrument is the only intervention from which the firm benefits. In particular, subsidies seem to impact much more in conjunction with other policy interventions. Second, if we look at the interplay of subsidies and procurement in particular, we see that firms that benefit from both instruments have a considerably higher input additionality, but there is no significant effect on the share of innovative output.

5 Reviews of mix at country or system level

In recent years, increased attempts have been made to understand the overall performance of policies supporting research and innovation at the level of the nation state. In this section we review a number of country (or system) level reports and analyses of various kinds: EU Peer reviews, which are based on qualitative assessments made by policy makers and analysts, policy mix reviews, which are based on dedicated studies of mixes, and system evaluations, which are research or innovation system focused assessments commissioned by governments and typically using a range of methods.

5.1 Policy mix reviews

In 2007-2008 a European consortium conducted a conceptual and empirical study on policy mixes in R&D (not innovation) policy. It considered both mixes within the R&D policy domain and those between R&D and other domains and framework conditions as they affected the level of R&D performed in a national system. 34 case studies were compiled, covering various aspects of policy mixes in 14 country settings, 10 regional settings, and 10 sectoral settings. This study did not analyse the impact of mixes, but rather reviewed existing literature and evaluation on the cases and on that basis attempted to draw general lessons on patterns and trends in the context of a simple conceptualisation of policy mixes. It found a variety of de facto mixes, contextualised in their specific settings.⁴

The important message for the purpose of this study is that no analyses were found concerning the overall effects of policy mixes and interplay of instruments. The synthesis report, focusing on the interaction of R&D (rather than innovation) policy instruments (Hofer/Dinges, 2008) concludes that policy rationales in the EU countries investigated are not concerned with policy mix in the design of policies, with very few noticeable exceptions: in specific countries (German High Tech Strategy), for specific technologies (energy efficiency technologies) or in countries that are required to adhere to certain mix considerations by outside forces, e.g. when receiving structural fund support. The review did not find any common or converging patterns for typical combinations of policies, as the "emergence of policy mixes appears to be highly path dependent and results in quite diverging trends". However, it finds one governance and coordination trend, i.e. in order to minimise tensions and to maximise synergies and complementarities, countries have tended to centralise innovation policy within large intermediary agencies. Further, the observed trend towards more indirect support versus direct support is seen as driven partly by the need to reduce coordination costs and to let firms design their own activity portfolios and behavioural patterns.

On the limited evidence available, the study drew the following lessons:

• While the composition of the direct measures may evolve differently according to context, (producing shifts between thematic and generic orientations), positive complementary effects prevail below a certain threshold of complexity and number of measures.
• Interacting policies are more likely to be complementary if the context in which they operate is taken carefully into consideration and when the incentive structures do not point in different directions.
• As regards effects between instruments at different policy levels, the review found that the overall complementarity of instruments prevails. Similarly, regional and national policies are more often complementary than conflicting, as similar instruments are often implemented differently at both levels, allowing for different contributions at the different levels.

Overall, then, since policy mixes are generally a product of attempts made over time to plug system gaps, based on a system failure rationale, they evolve as further measures are introduced: the interplay of measures is not designed a priori but is a feature of the system itself. However, the more limited the number of measures within the mix, the easier it is to design complementarity.

Finally, the report states the obvious – that if the intention is to design and evaluate mixes, then contextualisation is even more important than for individual instruments. To sum up, the key message of the study is that generalisations can only be made to a very limited extent and then with caution, and that thorough analysis of contextual conditions will always be required.

5.2 EU reviews

ERAC, the European Research Area Committee, is a strategic policy advisory body set up to assist the European Commission and the Council of the European Union in performing tasks relevant to the area of research and technological development. Formerly known as CREST, the committee dates from the early 1970s, ERAC has become increasingly important due to the growth in significance of the role of research and technological development in Europe, which is now widely seen as a crucial element of Europe's competitiveness and economic growth. In 2009, the European Council redefined the mission of the then CREST in the context of an enhanced governance of the European Research Area and in 2010, it was renamed the European Research Area Committee (ERAC) in order to better align its role with the new emphasis given to the ERA.

ERAC's mandate is to facilitate the formation and function of the European Research Area by, amongst other things, providing strategic guidance for RDI policy, monitoring the European Research Area (ERA) and promoting evaluation of the policy mix in the member states. ERAC reviews take the form of reviews performed by peers (policy-makers from other Member States) of a national research, development and innovation system, supported by academic experts. However, it should be noted that it was not within the scope of the CREST or ERAC reviews to scrutinise individual policies in detail or to comment on their detailed functioning. Thus, any evidence concerning the interaction between policies is typically presented at a more generic level. Nevertheless, a number of the reviews do reflect on the nature of the relationship between the functions of different policy types. Two, Denmark and the UK, illustrate some key findings of those reports.

One of the most recent ERAC reviews, that of Denmark, was conducted in 2012 (Cunningham et al. 2012). Regarding the streamlining and marketing of innovation funding schemes, the report notes that any mix of policy must fit the system properties, criticising the large number of sub-critical measures in innovation policy compared to countries of similar size. It also found that the Danish policy mix is characterised by a strong bias towards the supply side, whereby direct financial R&D incentives are geared mainly towards public R&D, with insufficient attention paid to issues of valorisation and the commercialisation of research results. This can be explained by the composition of the Danish economy, which lacks a sufficient number of large companies with large R&D activities that would push for more support for private R&D, e.g. through tax incentives.

The report most relevant to the issue of policy mix interactions is that of the UK, which includes specific observations relating to synergies within the policy mix, conflicts within the innovation policy mix, synergies between innovation policy and other policy domains and conflicts between innovation policy and other policy domains. Again, while this review was not an evaluation in the technical sense, it was informed by a range of high level interviews, documentary analysis and a cross-country comparative lens (Cunningham et al. 2007).

With regard to policy mix synergies, the report notes that in a well-designed policy mix, ideally "all policies should demonstrate some degree of synergy or complementarity. At worst, policies should be neutral in regard to their interaction with other policies". It goes on:

"In the UK policy mix, it is clear that a number of measures and instruments are complementary. For example, measures to stimulate the creation of start-ups are reinforced with a set of incentives to promote related types of entrepreneurial activity, either at the general level or in a more focused context (such as through the promotion of incubators and similar activities in universities); such measures are also complemented by a range of measures intended to promote the availability of start-up capital and know-how, seed funding and other forms of early stage venture capital. In turn, there are a range of incentives and support measures for later stage funding and support tailored to the needs of SMEs".

The authors find that the UK's policy mix is "a product of an extended period of innovation support which has been shaped by a series of reviews of both the national innovation system or parts of it and by evaluation and monitoring of the performance and effects of instruments comprising the policy mix" and that this “seems to offer a logical approach for the design of complementary policy interventions". More specifically, the authors propose that the modification and rationalisation of elements of the policy mix appear to have been implemented in response to the following outcomes:

• "success of the instrument in adjusting the behaviour of the target actors (which might lead to its continuation, extension of scope or refinement)
• overlap with other instruments, i.e. similarity to other mechanisms in terms of objectives, targets, etc. (which might lead to the merging of complementary schemes or the simplification of bureaucratic or administrative requirements)".

The report identifies two possible examples of conflicting measures within the (admittedly broader) policy mix affecting innovation outcomes. The first concerns the apparent mismatch of objectives between the mechanism by which the allocation of block grant funding has been made to HEIs by the higher education funding bodies in the UK the (then) Research Assessment Exercise (RAE) and those of a range of measures intended to stimulate improved linkages and the transfer of knowledge between the higher education sector and businesses. The report notes that, aside from the enormous resources in terms of time invested by HEIs and academic staffs in compiling data for the RAE, there are tensions between the focus on excellent academic publications on the one hand and the policy goal of stimulating greater interaction with business and, more recently, local communities. It could be argued that the final character of the RAE's successor (the Research Evaluation Framework), wherein there are indications of the inclusion of "impact", may moderate this tension.

A second potential example of tension between instruments is seen in the numerous interventions that target "the provision of venture capital and also those aiming at the promotion of interaction between the research base and industry". The report authors argue that, at first sight, there is an apparent degree of duplication within these two sets of measures, which might lead to substitution or crowding out effects between their target groups. The review also notes that the (then) Department for Trade and Industry had rationalised a number of its linkage measures following a detailed review and suggests that scope for further rationalisation could exist, particularly with regard to the various finance instruments then available. This in fact turned out to be the case, although the simplification exercise was conducted under the rationale of easing the administrative burden on (small) firms.

5.3 Systems evaluations

A number of system level evaluations have been conducted in recent years: in Norway, the Czech Republic, Finland and Austria. These evaluations have used a variety of different approaches, but their common interest has been to understand the current structure and performance of the research and innovation system and the role that framework conditions and policies play.

Only one of these reviews takes the interplay of policies or instruments seriously in the sense of attempting an empirical analysis rather than mentioning it only as a general, conceptual consideration or recommendation (Falk. 2009). This focus in the Austrian case was discussed above (Section 4.5) as it focuses on the interplay of two instruments that are both targeted at the increase of R&D activities and subsequent innovation activities.

In the case of the review of the Czech Republic (Arnold 2011), the interplay of policies or instruments was not a target of the evaluation. However, the report highlights the lack of integrative governance structures and coordinated policies. Poor joined up thinking on RTI policies across the Czech government is seen as a major hindrance to realising the potential and increasing the capabilities of the Czech system. The limited remit of the Czech Technology Agency, at the time acting as innovation policy integrator, was seen as a major obstacle to the effectiveness of the system and its supporting structures. Backed by findings from other countries provided as a background, the review also suggested the implementation of an innovation council or functional equivalent which would conduct regular reviews of policy measures and act as arbiter and adviser for decisions on the implementation of instruments (ibid, p. 24-25).

The Finnish system evaluation stresses the need for "broad based” innovation policy and the overall importance of coordination. Broad based innovation involves service and organisational innovation and includes "all determinants of the development and diffusion of innovations [to be taken] into account when designing and implementing innovation policies", which essentially means to add demand side policies. Although this approach is applauded in the evaluation (Edquist et al. 2009 p. 48), the evaluation finds a series of gaps in the provision of framework conditions and appropriate incentive schemes in the Finnish system, such as closer user-producer interaction, and the utilisation of innovations in the production and delivery of public services, whereby research is directed towards specified needs. The main lesson from this largely qualitative review is the need to take a holistic view, to start off with a clear understanding of the breadth of innovation policy and the related instruments. However, the recommendations are made on the basis of a large scale interview programme with context experts rather than on an analysis of the impacts of an existing portfolio of instruments.

6 Evidence on deliberate mixes and agency portfolios

6.1 Introduction

The increased prominence of policy mix concerns in the innovation policy literature (e.g. Braun 2008) and discourse has not thus far translated into a multiplication of deliberate portfolios or mixes of innovation policy measures towards a broader goal. We have scanned the existing evaluation literature and found very few examples of deliberate design of a policy mix. This does not mean that other examples of the conscious creation of instrument portfolios or mix do not exist, but the initiatives we discuss below appear to be the only ones that have had some form of assessment from which we can potentially draw lessons as to the design and coordination of such initiatives.

The four examples we have drawn on are: the German High Tech Strategy (HTS) and the Lead Market Initiative (LMI) at macro level, and the cases of two funding agencies, the Research Council of Norway and ADEME, at the meso level. Surprisingly they provide interesting material and issues, indicating the very high potential of such systemic approaches. We present the cases in this section while keeping the conclusions we derive for the overall conclusion.

6.2 German High Tech Strategy

In 2006, the German government bundled a range of existing policies and instruments together under the umbrella of one overarching High Tech Strategy (HTS). The main initial goal was to contribute to higher R&D spending through increasing public R&D and incentivising private R&D in a more coordinated manner. To that end, existing initiatives across government were bundled with innovation and research and industry was involved as a stakeholder in the design and further development of measures. Initially, the exercise was a response to the 3% Barcelona goal and the underlying requests to coordinate activities within EU member countries. Over the years, however, a more strategic definition of goals was developed, with buy-in from a range of departments and a more fine-tuned, pro-active design of instrument mixes. In 2012, an action plan (until 2020) was agreed at Cabinet level, giving a clear signal that the approach would be continued and broadened for the longer term. It included 10 "future projects", derived from clear societal needs, but defined broadly enough to allow for flexibility and inclusion. The policy mix was not, therefore, designed from scratch, but evolved from existing bundled activities that were, over time, connected together more systematically against defined goals.

Two reviews of the High Tech Initiatives are available (Rammer 2008; Heimer et al. 2011). Neither of these reviews performed a systematic impact assessment based on quantitative analyses, rather they are based on monitoring data, documentary analysis and interviews.

In the early phase, the HTS was seen as a major step towards a more comprehensive approach to R&D and innovation policy. Two thirds of all the non-institutional funding of the Federal Ministry for Research and Education (BMBF) was channelled through the HTS (Rammer 2009, p. 11) in the years 2006 to 2009. This involved an absolute increase in public spending and a harmonising effect on R&D. It brought together and coordinated major existing activities and four new initiatives (see Table 6), focused on five need areas and, originally, 17 distinct technological fields (now reduced to ten future projects).

Source: Rammer 2009

Rammer (2009) sees the HTS as an example showing that "effective and substantial coordination among R&D policy activities of different agencies (i.e. Federal Ministries) as well as joint development of a policy strategy and measures that integrate existing policy actions can take place" (ibid, p. 30).

The main success factors for this persistency and the buy-in have been, according to Rammer (2009):

• High-level political backing and additional funding: the initiative had backing from the Chancellor and was decided in the Cabinet. As well as bundling existing activities, a further €6bn was earmarked to induce "strategic capacity" to catalyse the joined up approach and to reduce inter-departmental budget battles
• Partnerships with other relevant actors: in each Federal Ministry one unit was made responsible for the linkage to the BMBF and in each of those ministries the ministers (undersecretaries of State) gave official backing; non-R&D agencies are involved, as the High Tech Strategy is intended not to be science driven but rather holistic
• Precise measures, clearly defined actions: the strategy defined very clear actions that spoke to all actors involved, first providing some continuity with existing activities, which then folded into a more holistic definition of joined up future initiatives over time; joined up action included thematic funding of R&D, regulatory framework condition, standardisation and demand side measures
• Clear links to other policy initiatives: research and innovation policy in Germany has a different "corporate identity" and thus the link to other domain initiatives (energy, climate), is more direct and straightforward through the structures that are created within the HTS.
• Clear targets and milestones, regular review: the initiative has been accompanied and supported by a range of strategic intelligence initiatives such as accompanying evaluation, a reporting system for the various focus areas, industry and stakeholder dialogues, and a high level expert group reporting to the Chancellor. Importantly, the nature of the initiative has led to a more holistic discussion of framework conditions⁵ across each of the focus areas that then feeds into an action plan for specific instruments to improve bottlenecks horizontally (Heimer et al. 2011).
• Direct industry involvement: for the discourse at the level of technologies or challenge areas, a new body including the highest level industry representatives (Research Union) has been established which in turn has raised further awareness in all the ministries involved to engage at highest the level. It also signalled a commitment by industry to the overall strategy.

As said above, these findings have not as yet been backed up through a systematic analysis of the impact of improved coordination.⁶

6.3 The EU Lead Market Initiative

In the Compendium report on public procurement (Uyarra 2013), the Lead Market Initiative has been discussed in some detail. It focused on the role of public procurement and summarised key findings of a recent evaluation. Here we concentrate on the instrument mix and its effectiveness. The Lead Market Initiative is an attempt by the EU to create the development of six markets whereby the supply and demand conditions co-develop and lead to a powerful market development that can spill over in non-European export markets.

The build up of lead markets necessitates a mix of measures, because it needs to tackle multiple dimensions simultaneously, avoiding a missing link. Consequently, the LMI is implemented through a combination of four broad types of measures (see Table 7): Legislation proposals (new legislation or modifications) and regulatory measures, promotion of the use of public procurement, development of more consistent standardisation, labelling and certification and other complementary actions to support the impact of the above instruments (including supply side instruments as needed).

The interim evaluation finds very mixed results across the six areas. The example worth looking at in more detail is sustainable construction, which “has been almost completely implemented and presents the clearest example of the benefits of addressing a well-structured set of interrelated demand-side issues" (CSES, 2011, executive summary).

The main characteristics of the sector, which are felt to render a mix of instruments necessary, are:

• A traditional focus on initial costs
• A high proportion of small contractors working in a traditional way
• The fragmentation of the supply chain and thus the challenges of learning across chains.
• Public procurement represents 40% of the market.
• The crucial role of standardisation and norms.

The Action Plan for sustainable construction envisaged eleven actions in total, nine of which were implemented:

• Screening of regulation and legislation, regulatory cost analysis
• New Directive on Energy Performance of Buildings Directive resulted in Directive 2010/31/
• Establishment of a network of procurers (SCI), coordinated by ICLEI (a European organisation with a large number of local organisations as members), with high visibility and multiplier effects
• Mobilisation of the Low Carbon Building (LCB) Healthcare network providing a platform for public procurement
• Development of guidance for procurement
• Standardisation: broadening the scope of European codes for construction design (CEN working group), increasing cross-referencing in standards
• Research funded as input for sustainable assessment
• Initiative for up-grading of skills for SME and guidance including procurement award criterion that takes account of Life Cycle Costing
• Additional studies to support businesses (e.g. on voluntary schemes)

While this catalogue of instruments has been implemented, the evaluation stressed the importance of national and EU follow up and the sustainability of those measures. Maintaining the momentum in a coordinated instrument mix is seen as a major challenge, especially as the impacts on the market take time.

The evaluation is largely based on qualitative analyses and a range of trend data along key indicators. It concludes that, in the sustainable construction sector, the LMI has "devised a relatively basic programme of inter-related actions that have been able to act as a focus for achieving important changes" (ibid p. 147). While it is contested how much of the activities have actually been initiated by the LMI (e.g. regulations were already in place), the explicit bundling, the set up and the communication about the coordination is felt to have made a real difference, as a range of actors, hitherto not involved, is now mobilised for concerted action. For example, the SCI network has mobilised more than 100 organisations in the course of the LMI initiative (ibid, p. 149). Furthermore, interviews showed a change in attitudes and understanding as to the potential of public procurement and accompanying measures. Challenges are seen in terms of follow-on activities at the national level, the interplay with policies that are outside the remit of this initiative and the need to engage with an ever increasing number of stakeholders when it comes to standardisation and end-user involvement (ibid). In addition, the necessary capability building and strategic intelligence to underpin long-term measures has been perceived by some industry representatives as an interference causing delay (ibid, p. 147). As the architecture of the initiative, the mix, was new, a range of stakeholders had to be coordinated in novel ways, which is potentially beneficial for the governance of innovation in the future, but which appeared to have created an initial imbalance as “incumbent" innovation policy stakeholders have different expectations and capabilities to stakeholders newly pulled into the mix.

6.4 Instrument portfolios of holistic agencies

There has been a trend in more and more countries to develop and/or reshape the ways in Government departments delegate responsibilities to funding agencies, whether covering the field broadly (RCN in Norway) or focused on specific domains (ADEME in France for energy issues). In both cases, these agencies are given a broad remit and it is the responsibility of their board and management to design the specific approach to intervention, the portfolio of instruments used, and to subsequently evaluate and adjust these as necessary. Thus, the design and operation of the policy mix becomes an agency-level responsibility. Broader, institutional evaluations by the government occur on an exceptional rather than periodic basis (e.g. once in a decade for RCN). We draw on the evaluations made for presenting the two cases.

6.4.1 The case of the Research Council of Norway (RCN)

RCN is the result of a full reshaping of public funding of research in Norway. It was created in 1993 by the merger of 6 disciplinary research councils. It underwent an extensive evaluation in 2001-2002, which is still considered as an exemplar of organisational evaluation. This evaluation underlined the need for a new step of integration and advised a full structural transformation. Today the council is organised along 5 divisions: science, energy resources and the environment, society and health, innovation and administration. With a budget of around €1bn, RCN serves as a national strategic body for research, and supports basic and applied research as well as innovation. It operates the national funds dedicated for its central mission, but is also the distributer of targeted funding for government administrations and regional bodies (operating seven regional funding schemes for innovation).

It underwent a second evaluation in 2011-12 which focused on stakeholder views about the role and performance of the council. The analysis asked researchers, head of research institutes and industry about the importance and attractiveness of schemes, and their satisfaction as users. An interesting point is that all other existing schemes (especially those supporting innovation) were included in the questioning, allowing for comparisons between schemes and actor groups (Langfeldt et al. 2012, p. 14; Fridholm andMelin 2012, p. 4-9). This is a helpful means to understand how actor groups rate the relative importance of a scheme and thus how the portfolio of schemes is assessed. However, the analysis did not examine how the various schemes interact with each other. There is also no analysis of the effects for the different measures and funding sources beyond the assessment questions asked. As this evaluation is a follow up from one done 10 years previously, it is able to conclude that since the first evaluation, companies have become more satisfied with the way the Council is now able to cover early and late stage research, while 10 years ago the focus was largely on early stage research. In this sense, the Council can be an example of a funding body that offers support for the full range of research, with a set of different instruments. This has been highlighted as very positive in the evaluation (ibid., p. 21). This simple portfolio and matching analysis has been helpful in the case of the rather small Norwegian system, as a tool to understand how funding bodies meet the needs of different kinds of stakeholders and thus identify gaps or the oversupply of opportunities across the portfolio.

6.4.2 The case of the French Environment and Energy Agency ADEME

ADEME (Agence de l'Environnement et de la Maîtrise de l'Energie) is the French agency in charge of supporting research and innovation for waste management, energy conservation and new alternative energies. With a budget of €800m and 1000 staff, ADEME is present in all French regions. It has a dual organisation crossing domains of intervention and types of interventions. ADEME's portfolio of instruments covers four main 'activities': 'knowing' (12% of funds distributed in 2011 and 24% of staff), 'convince and mobilise' (6% of funds but 18% of staff), 'advise' (4% of funds and 22% of staff) and 'support realisations' (78% of funds and 36% of staff). How does ADEME mix these different instruments? We take here only energy conservation and the case of three 'domains' to illustrate it.

In any domain, ADEME starts by establishing a 'roadmap' with stakeholders. For urban consumption, the roadmap focuses on the energy efficiency of buildings and on consumption patterns of shops and private housing. For the latter, the mix is focused on the downstream dimensions, not on innovation but on diffusion, mixing communication, regulation and support for change. Communication (based on what you can save, for instance, in powering off your TV set rather than leaving it in standby mode) and advice are used for changing behaviours. For instance, the agency's 250 'espaces information' have been visited by more than 3 million citizens in the last 8 years to get estimates of costs and return on consumption and investment. One important aspect deals with regulatory initiatives: for instance, the evaluation of ADEME's experimentation on shop lighting at night has been turned into a new regulation about night lighting. Its 'energy certificates' deliver more in-depth information than standard classifications about energy aspects of products. Moreover, the agency has a specific focus on public procurement by local authorities mixing training of their staff with evaluations of present situations and advice on potential transformations.

For the energy efficiency of buildings, the approach is radically different. Based on a previous approach, it targets professionals who are able to obtain a new label created in 2011 and is supported by the development of guides and training sessions (regionalised in 30 places). In order to further develop techniques and make them readily visible, there is an important experimentation and demonstration programme (PREBAT). A recent evaluation demonstrated its value but showed that it was mostly focused on new construction which led to a reorientation towards rehabilitation (while support for new building was limited to positive energy buildings).

Transport and mobility is often considered as a difficult area due to the predominance of road transport both for private and freight mobility. The 2030 transport roadmap is interesting to look at for the ways it mixes general trends with the specific French energy situation. ADEME on its own initiative has chosen to focus on improving the present structural situation addressing goods (truck energy saving driving, alternative options for distribution of goods to end consumers) and individuals (with a focus on car classification and mobilising government regulation of 'bonus/malus' taxes on new vehicles). ADEME's regional offices also work with city authorities on short distance mobility and the organisation of the city (for pedestrians, for bicycles, and for information on common transportation availability). For longer-term options ADEME mobilises the funds of the 'programme d'investissements d'avenir' that have been delegated to it (over €1.6bn over 10 years) focusing on new urban "non-carbon vehicles" and upon the infrastructure needed to refuel them. As a whole, this leads to a completely different policy mix compared to the two previous programmes.

In term of evaluations, ADEME has two complementary policies. One is to evaluate all its programmes individually to measure their relevance and efficiency (three evaluations finished in 2011, and five have been planned for 2012-13). This provides input for implementation and strategy focus, rather than in terms of overall impact. Like other on-going developments (e.g. evaluating the impact of agricultural research, Collinet at al., 2012), ADEME considers that impacts are generated by the whole network of actors involved in innovation processes, and that attribution to one actor (in this case ADEME itself) is both very difficult and always contestable. ADEME thus develops overall measures of change that are articulated in annual reports. For instance, overall 'energy efficiency' in the country has been increased by 20% since 1990, enabling an overall stabilisation of consumption; this in turn can be translated in different domains, e.g. housing energy efficiency has increased by 25% (focused on existing buildings) and in transport the figure has been 14% (mostly linked to car replacement). These figures help to better 'position' the action of the agency, but it is not stricto sensu an evaluation of its impact.

7 Observations on evaluation practice and challenges

As stated in section 3, our review shows a lack of attention on the part of evaluators and those who commission evaluations to the interplay of instruments and other policy mix issues. Our review suggests that two levels of evaluation should be distinguished. One group of evaluations looks at how different instruments in combination affect a target group, a technology or a sector (Cunningham et al. 2012), the other focuses on understanding the interplay of different policies for different or overlapping target groups in a more systemic sense. For the former, we have identified a small number of quantitative and qualitative evaluations. For the latter, there are a limited number of peer and expert reviews and "system” evaluations, which examine in detail the components of the system and subsequently develop an overall picture, but do not explicitly look at the effects of the various mixes they find. An initial, yet simple conclusion from this observation is that evaluation practice does not undertake sufficient systematic efforts to tackle the challenge of the interplay of instruments and polices, at both levels.

Equally, innovation policy makers and evaluators pay too little attention to considering the impacts of the joint or sequential use of innovation policy instruments and their interplay with other instruments (such as fiscal, human capital, and other policies). This suggests a need for greater consideration of portfolio evaluations which can consider systems of policy intervention more broadly, or the establishment of evaluation approaches which consider the impacts of multiple programmes on particular target groups over time.

Evaluating the combined effect of interventions is important for two reasons: first, as Guerzonie and Ratieri (2012) point out, "...evaluating the impact of a policy tool in a quasi-experimental setting without controlling for simultaneous public programmes aiming at the same objective, can lead to procedural confounding due to hidden treatments”. In other words, looking only at one particular treatment without controlling for other treatments distorts the results of the target treatment. Second, the variety of innovation policy tools that is applied in any given system can be employed more effectively if one knows if the combined net effects of a treatment are higher than the combined costs of the treatment.

We have seen that evaluations can indeed empirically and quantitatively analyse the effects of combinations of instruments. They may be used to compare changes of input or output as a consequence of one or a few isolated instruments against the effects of a combination of instruments. However, sophisticated econometric analyses which do this, even taking other intervening variables into account, are very rare. The examples we found for this report centred around the combination of direct and indirect R&D support and could take advantage of a dedicated large scale survey that allowed the collection of appropriate indicators for the use and effect of both direct and indirect measures (Falk, 2009). The really problematic issues for quantitative analysis of interplay concern interplay over time and over-complexity. The data required to establish causality and sequentiality over time, in the face of random external shocks, can be excessively challenging. The example of the French agency, ADEME, demonstrated a more holistic analysis of overall instrument portfolio development and was able to establish, based on expert knowledge and monitoring data, the relative contribution made by the set of policies in stimulating research and innovation activities and contributing to their diffusion. A sound, credible and demonstrable narrative emerged over time and demonstrated the overall role that the agency played in supporting systemic change as regards energy technologies, rather than claiming hard figure contribution to the change.

This last point raises the issue of the complexity of evaluating mixes particularly in terms of their strategic complementarity. The two agencies covered in this report (RCN and ADEME) both address a range of different stakeholders with a range of different measures to achieve their goals, a very broad one (R&D performance of the Norwegian system) and a very targeted one (the transformation of energy systems towards greater sustainability). In both cases, the strategic complementarities over time and across diverse actor groups cannot be meaningfully determined in quantitative terms. Thus the capacity for assessing the overall effects of mixes must rely on qualitative judgements which will be more robust if the construction of the mix and its individual instruments is based on broad stakeholder involvement (see the ADEME case).

Finally, the design and practice of evaluations into policy interplay and policy mix is limited due to the fragmented responsibilities that agencies have for the range of programmes being implemented. No agency is likely to have an interest in the effects of interplay and the interrelationship of policies beyond its own remit. For this reason, evaluations also suffer from the same coordination issues as faced by the governance of the mix itself. This is compounded by the fact that evaluations are increasingly commoditised (i.e. undertaken on a commercial basis by external professional contractors) and are applied to clearly specified instruments and situations (because those commissioning such evaluations have budgetary boundaries which relate to their ‘own' measures): however, the evaluation of the interplay of measures must be designed very specifically according to a highly specific set of rationales, purpose, scope and methodologies.

8 Conclusion

This report has presented a brief overview of the conceptualisation attempts as regards policy mix and interplay of instruments and on that basis draws out a set of key dimensions for understanding policy mixes and interplay of instruments. It then reviewed existing knowledge on the interplay of policies and instruments and addressed pro-active attempts to developing policy mixes. In the following, we summarise the lessons derived from this report first for interplay of measures, and then for the attempts of deliberate mixes.

Conceptualisation: the multi-dimensional nature of policy mixes

The recent work done on conceptualisation of policy mixes (reviewed in section 2) tells us that a policy mix is more than a de facto or even deliberate deployment of a set of instruments, whether simultaneously or over time. It tells us that policy mixes have to consider the pre-existing delineation of a domain and previous developments and national, regional and local (and perhaps even agency by agency) policies and policy styles, and that rationales justifying public action, objectives pursued and audiences targeted differ between different elements of the mix. It also tells us that a policy mix is seldom the outcome of one policy actor (implicitly the department in charge within the central government), but of a myriad of policy actors positioned in different governance spaces (from the city to the world) acting upon one given geographical space.

Lessons from evidence on the interplay of instruments

There are remarkably few evaluations that have addressed the issue of interplays other than in normative terms (what should be and not what is). This holds true even including the range of evaluations and meta-evaluations covering country reviews or material supporting strategic visions. Our report focuses on the existing knowledge on (1) interplays considering direct support only, (2) the interplays between direct and indirect support, and (3) the interplays between supply-oriented and demand-oriented supports. Five main lessons can be derived:

First, direct financial support, which is mostly targeted to small and/or young firms, is more fruitful if it is complemented by non-financial support both for internal management (often around HR and production) and for market access (from classical participation in trade fairs, to more sophisticated networking with 'key' suppliers, often provided through geographical or sectoral structures – clusters or technical centres). This point has been explicitly developed in the CREST report on the UK. What evaluations have not tackled in this respect is the way in which direct public support measures interplay with private support, in other words, if the financial support provided leverages private investment.

Second, the same conclusion applies when the analysis extends beyond R&D. However, the only explicit evaluation shows that the issue of ‘non-financial' support is not a trivial one, since there is a multiplication of (and competition between) public sources (between departments/agencies, and between levels of governance) and, on selected issues, the competition extends to private services.

Third, one central - and debated issue of interplays lies in the combination of direct and indirect support to business R&D. One has to be clear here: most financial support targets R&D only (because there is a fiscal requirement for an evidenced-based definition of the basis for support, and the main established one is associated with the Frascati manual. This is an important issue since OECD countries (with the important exception of Germany) have tended to adopt or enlarge fiscal policies during the last 10 years. This interplay has, however, been seldom addressed by econometric evaluations. Those that have done so tend to conclude that there is a significant level of complementarity, especially for small firms. Some consider that it has no long-term benefit (even in term of input additionality) for large firms, being at best a counter-cyclic instrument. In some cases (e.g. Austria), one can no longer speak of complementarities since nearly all firms undertaking R&D combine both types of public support. Finally, some studies consider that tax credits are more useful in helping firms develop their product range (mostly incremental innovations), while targeted support tends to focus on new technologies and might be more supportive of radical innovations (with of course a far greater rate of failure, meaning that new products are less often generated- even indirectly). The balance between direct and indirect support to R&D would thus depend on the targeted population and its characteristics in terms of sectors, roles within sectors and size.

Fourth, as far as the interplay between supply and demand driven instruments is concerned, we can only be cautious at this early stage of their deployment in most countries. We have only one recent study that tries (in an approximate way) to analyse complementarities between direct R&D supports and innovation procurement. The results are very positive for public intervention in general (which is a view not shared by other studies). There is a positive effect on innovation input and output for both the supply side and the innovation procurement. However, the figures are not so clear-cut for the combination of both interventions, which seems to generate more R&D activity but not more innovations. Thus at the present stage of accumulated knowledge, this interplay remains mostly a theoretical possibility. However, the picture changes if we look of the interplay of demand and supply for innovation processes as a whole and across time, leading us to the fifth conclusion.

When looking at innovation processes as a whole, and the attempts to push certain technologies in the market place rather than at individual aspects within an innovation process (such as R&D), one clear lesson emerges from the different evaluations of energy policies (probably the sector most studied). There is an interest in building a portfolio of instruments that covers the spectrum of stages of an innovation process (whether one adopts a linear or a network model of innovation). It is a portfolio because instruments are more efficient if they are specialised to focus each on one stage. However policies must consider their deployment over time: the overall efficiency, if one follows the Danish case, depends upon a shifting balance over time linked to market emergence and growth and the dynamics between technological developments and maturity in the market place. One crucial aspect is the development of the interplay between demand and supply measures in a technological space over time. The focus should shift over time between specific demand side measures and supply side measures, as a new generation of technologies necessitates R&D investment before demand for it can kick in and be supported.

Lessons from evidence on country or system level mixes

The report reviewed existing evidence on policy mixes at national policy level assuming that the rhetoric about policy mixes and attempts to "optimise” innovation policy within innovation systems have led to systematic approaches to design mixes. The multiple dimensions that characterise a mix theoretically offer a vast number of potential combinations for the deployment of such policy mixes. However, in practice, policy mix definitions are simplistic, hardly encompassing more than the standard definition of policymaking (that is policies developed by one Government department in isolation). A review in the context of an EC supported project that we summarise in section 5.1 nicely demonstrates this. At the time (2008), the authors were driven to conclude that "EU countries were not concerned with policy mixes in the design of their policies". In order to learn about potential benefits of policy mixes in absence of any evaluation of those mixes, the analysis compared a set of 'de facto' cases'. The central result highlighted the strength of path dependency and of 'national preferences' in terms of the types of instruments selected. The only convergence the review could identify was the search for simplification in the handling of policy at central government through the process of centralisation, an answer to which the long term limits is now well established in political science. The review could not, however, establish any impact on target groups or the system as such.

We further looked at some recent country reviews that explicitly took a systems approach. Those reviews are intended to be learning tools for the policy makers in the countries reviewed and in other countries. They are not evaluative in a technical sense. Interestingly, the idea of policy mix is borne by the reviewers that use it as a canvas to benchmark policies and make recommendations, following the established practice of the OECD innovation policy reviews. These reviews thus lead to rather traditional and conventional recommendations, corresponding to classical OECD answers (such as the setting up of transversal advisory bodies) or to the dominant policy ideas of the time and the reviewing group. The only review that goes a step further in its assessment of mixes (the UK CREST review) mainly focuses on 'operational complementarities' and tensions within the present portfolio of instruments, finding once more the lessons we have already highlighted.

Lessons from evidence on proactive, deliberate policy mixes for specific missions

There are a few cases of pro-active policy mixes that go beyond the 'alignment' in a given space or over time of the existing set of instruments. The report identified four cases that allow us to draw lessons, even if at a rather generic level.

The review of the German High Tech Strategy highlights two sets of issues. First, at the level of alignments, there needs to be a top level initiative to drive the sector-based administrations to communicate and coordinate their views, visions, strategies and activities. This cannot be a one-off and has to be periodically reiterated, especially since in democratic countries the time span of such initiatives goes far beyond one electoral term. To bear fruit, it should gather a quasi-consensus, which points towards the importance of political stakeholder involvement. Second, comprehensive mixes do not pose traditional public coordination challenges, since the broader the strategic complementarity, the more diverse the actor groups engaged in the governance of policies. The experience of the German High Tech Strategy suggests that additional budgets and shared visions can help to mobilise stakeholders around 'joined-up policies', whilst the bundling of existing activities with existing budgets may trigger defensive reactions This not only simplifies compliance with target groups, it also embeds the strategy 'in practice' and over time and drives the implementing actors to progressively streamline their instruments. Thus, it is possible to progressively learn about what works and what does not, which allows the adaptation of the portfolio. Over time, instruments can therefore be made more sophisticated and responsive to the needs of the system.

The German High Tech Strategy is a clear example of a high level policy mix playing on all the layers of policymaking in one space (both institutional and geographic) and over time, having been institutionalised out of an umbrella into a more coordinated strategy deployed on the ground.

The European Lead Market Initiative is an attempt to tackle a lasting problem (that of poor (public) demand for innovations) with a comprehensive approach. The LMI offers an approach that is generating the evolution of similar policies across borders, supporting transnational "integration” not at the policy side but at the implementation and impact side. It is a meta-framework consisting of four sets of instruments offered to national policymakers at various levels. This framework is not only a concrete set of options for national actors to implement, it is also a way to benchmark their existing national portfolios. It thus operates as a heuristic device, allowing countries to do their self-assessment and define the relevant developments in their instruments and in their portfolio. Whilst not all policy audiences are equally receptive of the LMI approach to a policy mix, it can nevertheless be regarded as important since it de facto triggers (a) attention, (b) ways of addressing problems that mobilise the framework and address the four dimensions, thereby offering ways in which policies may become more integrated. It further highlights, again, the need for adjustments not only between technologies or sectors, but also over time. Finally it highlights the need to understand the context of immediate target groups (demand side) holistically, as for demand measures to trigger innovation and growth one needs to understand the potential of the supply side.

The final two examples centred around agencies and their portfolios and illustrated the organisational competencies and strategies to deploy and re-adjust portfolios over time. The Research Council of Norway is an interesting example of an agency being able to adjust its portfolio over time. The RCN is the central Norwegian funding agency covering all domains and all stages from fundamental research to innovation. It was evaluated 10 years ago (an evaluation that remains a landmark in the evaluation of funding bodies). This evaluation underlined a quite general phenomenon, i.e. the very unbalanced portfolio of instruments and the absence of instruments for a number of innovation issues. Ten years after, a new evaluation was conducted, mainly focused on the views of recipients. It highlights a very important phenomenon: while no high level policy action had been taken on the weaknesses highlighted by the 2002 evaluation, the evaluation shows that, ten years later, these problems have been addressed, highlighting the ability of the organisation to internally address, in a progressive manner, the issues raised. This informs us about the importance of professionalisation for solving implementation issues (rather than always looking at their 'policy' angle). However, it also requires that a "relevant” delegation is made, and that controls are not placed on the use of inputs (as is too often the case) but on the impacts generated. This, of course, implies that delegation periods are sufficiently long to enable instruments to start generating effects and for evaluations to assess them.

ADEME is another case of such delegation, which has further embedded a broad range of instruments (similar to the LMI in having four main types of instruments, along which the agency reports its activities). The case highlights three important issues. First, the mix of the four types of instruments differs depending upon sectors, and, within a given sector, it also evolves over time, demonstrating the organisational strategic capability to understand the different needs in different areas over time. Second the notion of a target sector is not given ex-ante, but is redefined over time to address failures and opportunities as they evolve. The ways in which urban consumption is for instance redefined is a good example of the need to rethink existing delineations, as highlighted in the theoretical policy mix approach. These adjustments over time cannot be done as an administrative task but have to be shared with all parties concerned, operationalised in long-term domain roadmaps. Finally, the evaluation work done by ADEME shows that it is possible to measure transformations (often through physical rather than monetary indicators) overall: it is completely artificial to look for a 'share of impacts' that derive from specific ADEME investments. Broad, largely quantitative assessments of large programmes (for example, see the famous case of the Advanced Technology Programme, Ruegg and Feller 2004) have never resisted critics and have subsequently never been mobilised in policy debates. The ADEME example suggests that it is better to assess an overall transformation of the domain addressed and to characterise and broadly discuss the roles played by the policies as a change agent.

Across most cases of deliberate mixes we have seen the importance of one aspect that is neglected in assessments and evaluations, i.e. the role of implementation structures and processes in policy mixes. The way implementing agencies interpret a policy and are able to put it into practice is a key determinant of impact for any instrument in any policy field. However, for a deliberate mix this challenge is exacerbated since the mix manifests itself according to the way the diverse organisations and actors implementing the programmes and instruments of the mix interpret the overarching goal and the role their instrument plays. Further, the overall impact is determined by the way the implementing actors are able to align their particular implementation action with other elements of the mix, understanding the interactions between instruments. Therefore, if policy makers and analysts are to understand how interactions and similar policy mix issues materialise in specific cases, there is a need to focus on the ways in which policy ideas are put into practice. Of course, good evaluation should always focus on implementation. However there is an ever-present danger that, particularly as evaluation and monitoring become routine and commoditised, evaluations that attempt to take an overview of a portfolio, agency, mix or system will take for granted that a policy is implemented as initially intended. Addressing the reality of implementation, and how that plays out in the mix, is perhaps the most important challenge for policy mix evaluation.

This report has shown that we can learn about the nature of interplay and the potential of policy mixes only by looking at specific cases in detail. It has also shown that, currently, there are not many evaluations available that do this. Consequently, we need to continue to study the existing cases where mixes of policy instruments emerge or are deliberately designed. In turn, policy making processes must accept that any intervention, whatever the rationale and whatever the previous track record of that policy idea in other jurisdictions or policy areas, is a new intervention in an unique, complex and evolving system. Even with improved evaluations, thinking about systemic effects will have to be done with limited evidence and in conditions of uncertainty - but it should be attempted nonetheless. Being open about this, and inviting the widest possible debate about what the evidence tells us would seem to us to be a good starting point.

References

• Arnold, E., 2011. International Audit of Research, Development & Innovation in the Czech Republic : Synthesis Report . Technopolis
• Bérubé, C., Mohnen, P., 2009. Are firms that receive R&D subsidies more innovative? . Canadian Journal of Economics, 42: 206-225.
• Borrás, S., Edquist, C., 2013. The Choice of Innovation Policy Instruments. CIRCLE Working Paper No 2013/04, Lund: http://www.circle.lu.se/?wpfb_dl=5.
• Branscomb, L.M., Florida, R., 1998. Challenges to Technology Policy, in Branscomb, L.M., Keller, J.H. (Ed.), Investing in Innovation: creating a research and innovation policy that works, MIT Press: Cambridge, pp.3-39.
• Bressers, H.A., O'Toole, L.J., 2005 . Instrument selection and implementation in a networked context, in Eliadis, P., Hill, M., Howlett, M. (Ed.), Designing Government: from instruments to Governance, McGill-Queens University Press: Montreal, pp.132-184.
• BUEN, J., 2006. Danish and Norwegian wind industry: The relationship between policy instruments, innovation and diffusion. Energy Policy, 34 (18): 3887-3897.
• Carboni, O.A., 2011. R&D subsidies and private R&D expenditures: evidence from Italian manufacturing data. International Review of Applied Economics, 25: 419-439.
• Corchuelo, B. Martínez-Ros, E., 2009. The Effects of Fiscal Incentives for R & D in Spain.. IDEAS, : http://ideas.repec.org/p/cte/wbrepe/wb092302.html.
• Cowling, M., 2010. Institute for Employment Studies, Economic evaluation of the Small Firms Loan Guarantee (SFLG) scheme. BIS: London.
• CSES, 2011. Final Evaluation of The Lead Market Initiative (Final Report). Framework Service Contract for the Procurement of Studies and other Supporting Services on Commission Impact Assessments and Evaluations Interim, final and ex-post evaluations of policies. centre for strategy and evaluation services: Oxford.
• Cunningham, P.N. CREST 3% OMC Third Cycle Policy Mix Peer Review: Country Report United Kingdom, European Parliament, September 2007.
• Cunningham, P.N., Crasemann, W., Lehto, P., Starzer, O., van der Zwaan, A., 2012. Peer-Review of the Danish Research and Innovation System: Strengthening innovation performance. Expert Group Report prepared for the European Research Area Committee: Brussels.
• Cunningham, P.N., Gök, A., 2012. Impact of Innovation Policy Schemes for Collaboration, Compendium of Evidence on the Effectiveness of Innovation Policy Intervention. NESTA/MIOIR: London/Manchester.
• David, P. A., Hall, B. H., Toole, A. A, 2000. Is public R&D a complement or substitute for private R&D? A review of the econometric evidence. Research Policy, 29 (4-5): 497–529.
• de Laat, B., Warta, K., Williams, K., 2001. Evaluation of ANVAR’s “Procédure d'aide au projet d'innovation”. Technopolis: Paris.
• Edler, J., Berger, M., Dinges, M., Gök, A, 2012. The practice of evaluation in innovation policy in Europe. Research Evaluation, 21 (3): 167-182.
• Edler, J., HAFNER, S., 2007. Nachfrageorientierte Innovationspolitik in Schweden, in EDLER, J. (Ed.), Bedürfnisse als Innovationsmotor. Konzepte und Instrumente nachfrageorientierter Innovationspolitik, edition sigma: Berlin, pp..
• ETAN Expert Working Group, 1998. Climate Change and the Challenge for Research and Technological Development (RTD) Policy. ETAN Working Paper (Directorate General XII Directorate D - Environment and Climate RTD Programme, Directorate AS - RTD Actions: Strategy and Coordination; Directorate General Joint Research Center- Institute for Prospective Technological Studies).
• Falk, R., 2009. The Coherence of the Instrument Mix. Report Nr. 8 in the context of the Study: Evaluation of Government Funding in RTDI from a Systems Perspective in Austria. : Vienna.
• Farre-Barbosa L., Lhoste E., Gibelli F., 2013. 'ADEME et l'accompagnement de l'innovation énergétique. report for the master course on research and innovation policy, Ecole des Ponts: Paris.
• Flanagan, K., Uyarra, E., Laranja, M., 2010. The ‘policy mix' for innovation: Re-thinking innovation policy in a multi-level, multi-actor context. Manchester Institute of Innovation Research Working Paper Series, Manchester: https://www.escholar.manchester.ac.uk/uk-ac-man-scw:82698.
• Flanagan, K., Uyarra, E., Laranja, M., 2011. Reconceptualising the ‘policy mix' for innovation. Research Policy, 40 (5): 702-713.
• Fridholm, T., Melin, G., 2012. Evaluation of the Research Council of Norway: Background Report No 8. Company Survey . Technopolis: Oslo.
• Geller, H., Nadel, S., 1994. Market Transformation Strategies to Promote End-Use Efficiency. Annual Review of Energy and the Environment, 19: 301-346.
• Grilli, L., Murtinu, S., 2012. Do public subsidies affect the performance of new technologybased firms? The importance of evaluation schemes and agency goals. Prometheus, 30: 97-111.
• Guerzoni, M., Raiteri, E., 2013. Innovative procurement and R&D subsidies: compounding effects and new empirical evidence on technological policies in a quasi-experimental setting. Bureau of Research in Innovation, Torino: http://s3.amazonaws.com/academia.edu.documents/30874969/18_wp_2012.pdf?AWSAccesKeyId=AKIAIR6FSIMDFXPEERSA&Expires=1381782095&Signature=Mf%2FTW3Dqq5GY1R8tHYWk3AAICz8%3D&response-content-disposition=inline.
• Gunningham, N., Sinclair, D., 2002. 'Regulatory pluralism: designing policy mixes for environmental protection. Law and Policy, 21 (1): 49-76.
• Guy, K., Georghiou, L., Quintas, P., Cameron, H., Hobday, M., Ray, T., 1991. Evaluation of the Alvey Programme for Advanced Information Technology. HMSO: .
• Hofer, R., Dinges, M, 2008. R&D – R&D Policy Interaction: Thematic report. Reinhold Hofer and Michael Dinges Joanneum Research: Brussels.
• Howlett, M., 2005. What is a policy instrument? Policy tools, policy mixes, and policy-implementation styles, in (Ed.), Designing Government: from instruments to Governance, McGill-Queens University Press: Montreal, pp..
• JOHNSTONE, N., HASCIC, I., POPP, D., 2010. Renewable Energy Policies and Technological Innovation: Evidence Based on Patent Counts. Environmental and Resource Economics, 45 (1): 133-155.
• Kay, A., 2010. The Dynamics of Public Policy: Theory and Evidence. Edward Elgar Publishing: Cheltenham.
• Lambert, R., 2003. Lambert Review of Business-University Collaboration. HM Treasury: Norwich.
• Langfeldt, L., Piro, F., Ramberg, I., Gunn, H., 2012. Evaluation of the Research Council of Norway. Background Report No 7 - Users' experiences of and interaction with the Research Council of Norway . Technololis-NIFU: Oslo.
• Mundell, R., 1962 . The Appropriate Use of Monetary and Fiscal Policy for Internal and External Stability' IMF Staff Papers. International Monetary Fund: .
• Neij, L., 2001. Methods of evaluating market transformation programmes: experience in Sweden. Energy Policy, 29 (1): 67–79.
• Neji, L., 1998. Evaluation of Swedish Market Transformation Programmes. Department of Energy and Environmental Energy Systems Studies: Lund.
• NUTEK, 1994. Effective Market Influence - an effect chain analysis of NUTEK's high frequency lightning campaign. NUTEK: Stockholm.
• OECD, 2010a. The Innovation Policy Mix, in OECD (Ed.), OECD Science, Technology and Industry Outlook 2010, OECD: Paris, pp.251–279.
• OECD, 2010. The OECD Innovation Strategy: getting a head start on tomorrow. OECD: Paris.
• OECD, 2011. Demand Side Innovatio Policies. OECD: Paris.
• PACEC, 2011. Evaluation of the Collaborative Research and Development Programmes: Final Report.
• PREST, 2003. Evaluation of National Research and Development Programme for Medical and Welfare Apparatus. Japanese Ministry for Economy, Trade and Industry: Manchester.
• Smith, K., 1994. New directions in research and technology policy: identifying the key issues. STEP rapport / report: Oslo.
• Sorrel, S., Sijm, J., 2003. Carbon trading in the policy mix. Oxford Review of Economic Policy, 19 (3): 420-437.
• SQW Consulting, 2009. Synthesis of Evaluation Evidence: Final report for the Department for Business, Enterprise and Regulatory Reform. BIS: London.
• Suvilehto, H.-M., Överholm, E., 1998. Swedish Procurment and Marcet Activities - Different Design Solutions on Different Markets. ACEEE, : http://aceee.org/proceedings-paper/ss98/panel07/paper27.
• Zhu, P., Xu, W., Lundin, N., 2006. The impact of government's fundings and tax incentives on industrial R&D investments - Empirical evidences from industrial sectors in Shanghai. China Economic Review, 17 (1): 51–69.