This paper is part of the Compendium of Evidence on the Effectiveness of Innovation Policy Intervention. The objective of this report is to review the upsurge of initiatives to support public procurement of innovation and offer insights into the effectiveness of these policies. To do so, it firstly elaborates a delineation of these policies and summarises the main academic rationales for intervention. It further considers conceptual and methodological issues influencing the assessment of such policies. It then moves on to examine specific policy measures that have been introduced in support of the public procurement of innovation and existing evidence of their impact. Finally, conclusions and lessons learnt are presented in terms of policy interventions and methodological issues.

JEL Classification: 038 Keywords: Public procurement, demand side innovation policy, evaluation.

The Compendium of Evidence on the Effectiveness of Innovation Policy Intervention Project is led by the Manchester Institute of Innovation Research (MIoIR), University of Manchester, and funded by Nesta, an independent charity with the mission to make the UK more innovative. The compendium is organised around 20 innovation policy topics categorised primarily according to their policy objectives. Currently, some of these reports are available. All reports are available at http://www.innovation-policy.org.uk. Also at this location is an online strategic intelligence tool with an extensive list of references that present evidence for the effectiveness of each particular innovation policy objective. Summaries and download links are provided for key references. These can also be reached by clicking in the references in this document. Corresponding Author: Dr Elvira Uyarra | Manchester Institute of Innovation Research, Manchester Business School | The University of Manchester, Oxford Road, Manchester M13 9PL, UK | T:+44(0)161-275 7055 | F:+44(0)161-275 0923 | E: [email protected]

The Nesta Working Paper Series is intended to make available early results of research undertaken or supported by Nesta and its partners in order to elicit comments and suggestions for revisions and to encourage discussion and further debate prior to publication (ISSN 2050-9820). Year 2013 by the author(s). Short sections of text, tables and figures may be reproduced without explicit permission provided that full credit is given to the source. The views expressed in this working paper are those of the author(s) and do not necessarily represent those of Nesta.

This report is part of the Compendium of Evidence on the Effectiveness of Innovation Policy Intervention Project led by the Manchester Institute of Innovation Research (MIoIR), University of Manchester. The project is funded by the National Endowment for Science, Technology and the Arts (NESTA) – an independent body with the mission to make the UK more innovative.

The compendium is organised around 20 innovation policy topics categorised primarily according to their policy objectives. Currently, some of these reports are available:

All reports are available at http://www.innovation-policy.org.uk. Also at this location is an online strategic intelligence tool with an extensive list of references that present evidence for the effectiveness of each particular innovation policy objective. Summaries and download links are provided for key references. These can also be reached by clicking in the references in this document.

1 Introduction

This report is a literature-based review of government initiatives supporting public procurement of innovation. Policy makers at regional, national and supranational levels have in recent years shown an increased interest in the use of demand side measures (see Edler, 2013) and in particular public procurement to harness innovation. This has coincided with a renewed interest in the academic literature on the importance of demand for innovation, and the relevance of the public sector as an innovator in its own right.

Despite this academic and policy interest, issues remain in relation to the precise definition and articulation of such policies. Problems in delineating these interventions make it difficult to precisely allocate specific instruments into the category of public procurement of innovation. Boundaries between public procurement of innovation and normal procurement, and between procurement and other demand side measures and even supply side interventions are blurred, rendering its delineation somewhat artificial (Edler, 2013).

Further, the aspirations and political rhetoric on demand side policies are often not translated into concrete implementation, let alone provisions for their assessment and evaluation. Existing evidence of the implementation of these policy instruments is therefore limited, and when it exists implementation is rarely accompanied by monitoring and evaluation activities to evaluate impacts.

The objective of this report is to review the upsurge of initiatives to support public procurement of innovation and offer insights into the effectiveness of these policies. To do so, it firstly elaborates a delineation of these policies and summarises the main academic rationales for intervention. It further considers conceptual and methodological issues influencing the assessment of such policies. It then moves on to examine specific policy measures that have been introduced in support of the public procurement of innovation and existing evidence of their impact. Finally, conclusions and lessons learnt are presented in terms of policy interventions and methodological issues.

This report complements other reports of the NESTA Compendium of Innovation dealing with related interventions on the demand side. In particular Edler (2013) focuses on support measures for private demand, covering direct financial support and, to a much lesser extent due to poor existing evidence, awareness measures and labels. Rigby (2013) covers pre-commercial public procurement schemes such as the US SBIR and the UK SBRI and Gok (2013) deals with innovation inducement prices. Four more of our Compendium reports have some demand side aspects: regulation and innovation (Blind 2012), standardisation and innovation (Blind, 2013) and foresight and innovation policy (Harper, 2013). Edler (2013) presents a broad picture of the importance of demand on innovation and a typology and main rationales of demand side policies. This report only makes a brief mention to such debates.

2 Context and Definition

Interest in procurement as an innovation policy tool has seen a renewed interest in recent years. Reports such as the Fraunhofer Institute report (Edler et al., 2005) and the Wilkinson Report (Wilkinson et al., 2005) helped inform the procurement of innovation agenda at the EU level. The Kok Report (Kok, 2004), reviewing progress on the Lisbon strategy, also emphasized a need to promote policies driving demand for innovation, including public procurement. The Aho Group Report Creating an Innovative Europe (Aho et al., 2006) highlighted the lack of demand as one of the key barriers inhibiting innovation in Europe. It recommended a set of policies on the demand side involving the creation of a favourable regulatory environment, the use of standards, a better deployment of public procurement to drive demand for innovation, and a culture that celebrates innovation.

Responding to the recommendations of the Aho report, the Lead Market Initiative (European Commission 2007) sees public procurement as one of the policy pillars enabling “lead markets" in Europe. More recently, the ‘Europe 2020 strategy' specifically includes public procurement as one of the market-based instruments that should be used to achieve the objectives of smart, sustainable and inclusive growth (European Commission 2011a). The 2011 Commission Green Paper on 'the modernisation of procurement policy' (European Commission 2011b) suggests ways to improve existing procurement tools in order to make them better suited to pursue common societal goals and the provision of high quality public services.

Following a similar trend, OECD launched a project on 'Demand-Side Innovation Policies' in 2008 that provided input the OECD Innovation Strategy (see OECD, 2011). The OECD (2011) report provides an overview of the experiences in many countries in incorporating public procurement as one element in the enlarged instrument mix fostering the demand for innovation.

This interest in demand-side measures can be observed not only in large OECD economies, but also in very small countries (e.g. Georghiou et al., 2010), in Central and Eastern European countries (Edler, 2011) and emerging economies such as China (Edler et al., 2008; Li, 2011). The degree of adoption of demand side policies is very diverse however. The OECD Science, Technology and Industry Outlook 2010 policy questionnaire evidenced that demand-side policies are not top policy priority for most governments, and also that differences exist across governments (OECD, 2011). For instance Finland reports such policies as high priority, while Denmark, France, Israel, New Zealand and the US is of relatively low priority. In Finland, Demand- and user-driven innovation policy is one of the key priorities in the 2008 national innovation and its 2010 Action Plan for implementation includes several public procurement mechanisms. In the UK, the debate on the use of procurement has been accompanied with the launch of a host of initiatives and reports to mobilise the use of procurement to support competitiveness and innovation (DIUS, 2008; HM Treasury, 2007).

Despite a generalized optimism on the potential of public procurement to spur innovation, the implementation of strategies and initiatives to exploit this potential has been limited. This is because of considerable challenges associated with the implementation of innovation procurement practices (Edler and Uyarra, 2013). For instance in the UK the OGC (2004) listed key barriers preventing the government from fully 'capturing innovation', including inadequate early warning, risk aversion, and client capability shortfalls in the public sector. The 2008 Innovation Nation White Paper expressed concern that as a result of a risk- averse culture, difficulties in defining what constitutes innovation in procurement terms and insufficient capability in procurement, “procuring innovative solutions has tended to be a low priority" (DIUS, 2008: p.23). The House of Lords (2011) report considers the barriers that inhibit the promotion of innovation through public procurement, and groups them in four broad categories, namely: lack of capability, expertise and incentives; risk aversion; need for more effective engagement between procurers, suppliers and academia; and overly prescriptive and burdensome procurement processes. According to NESTA (2012) the main reasons why progress in relation to the public procurement of innovation has been slow are inertia and cost reasons (tried-and-tested solutions are generally cheaper) that act as a disincentive to procurers, particularly when considering the public sector's inherent risk aversion. From a supplier perspective, the UNDERPINN survey (Georghiou et al., 20123) conducted among 800 public sector suppliers in the UK, revealed a number of perceived barriers to innovation in the way procurement is conducted. In particular, almost 60% of suppliers complain about a dominant emphasis on price rather than quality. Other barriers that are perceived to be a significant hurdle to innovation include too prescriptive specifications (36%), lack of interaction with procuring organisations (43%), risk aversion of procurers (37%), and lack of competence of procuring organizations (32%).

3 The Rationales for Public Procurement of Innovation

3.1 Policy delineation and definitions

Upsurge of interest in the topic of public procurement and innovation has been accompanied by attempts to further delineate the policy, including a definition of the rationales for its use and classifications of the types of intervention. Edler (2013) situates the use of public demand within the broader range of demand based measures.

Proponents of the use of public procurement to stimulate innovation increasingly refer to "innovative procurement” or “procurement of innovation", rather than “public technology procurement', in an attempt to reflect a broader view of innovation beyond R&D (Edler and Georghiou, 2007). Further, a key part of such delineation includes the distinction between regular or 'off-the-self' procurement and procurement of innovation, consisting of public bodies seeking to purchase a product or service which does not exist at the time (Edler et al., 2005; Edquist and Hommen, 2000). Such strict definition has been considered elsewhere (see e.g. Uyarra and Flanagan, 2010) as too restrictive, in that it tends to overlook innovation effects beyond the initial purchase and is biased towards radical (new to the world) innovations, paying less attention to other categories of innovation.

A more nuanced distinction is made between procurement of solutions that do not exist (triggering demand) and procurement to acquire and adopt a service that exists in the marketplace, but is new for the organization (responsive demand) (Edler and Uyarra, 2013). Edquist and Hommen (2000, p.21) also differentiate between 'developmental' and 'adaptive' public technology procurement. 'Developmental' or 'triggering' modalities have been associated with the public sector intervening early in the procurement process to directly procure R&D to support the activities and decisions of government and public authorities (Edler and Uyarra, 2013). This form of intervention has been labeled 'pre-commercial procurement'. Pre-commercial procurement, such as the US Small Business Innovation Research (SBIR) programme, is not procurement in the strictest sense since it does not involve the purchase of a product or service. It is therefore not included within the scope of the present report (see Rigby, 2013 for a review of this policy instrument).

Further, (Edler, 2013) differentiates between general and strategic procurement of innovation. Whereas the former includes innovation as additional criterion in tenders, in strategic procurement innovation is an explicit goal. It is worth noting however that many of the most frequently cited examples of procurement-induced innovation are likely to have been the result not of an intentional or conscious drive to encourage innovation but rather a by-product of “normal” or general procurement. Linked to this, the delineation of innovation procurement could thus be placed along a continuum, which distinguishes between pre-commercial procurement on one end, active attempts to stimulate innovation (via the use of new measures such as competitive dialogue and forward commitment procurement) and the use of 'innovation friendly' procurement at the other end, namely the use of practices and competences in general procurement that make sure that innovative solutions are not excluded or disadvantaged (Uyarra and Flanagan, 2010). The latter may require 'innovation in procurement', that is change or innovation in the procurement process itself.

A further distinction relates to whether the procuring organization is also the end user of the good and service (direct) or whether it seeks to address other needs and therefore is done in conjunction with or on behalf of private users (catalytic procurement) (Edler and Georghiou, 2007; Hommen and Rolfstam, 2009).

Finally, and reflecting on the nature of the actual goods and services procured (in particular the extend to which they are specialized or standardized, and whether they are generic or dedicated) Uyarra and Flanagan (2010) suggest four stylized types of procurement. Their starting point is that procurement influences innovation through shaping the size, specificity, technical sophistication or standardisation of demand and, adapting Kraljic (1983) purchasing portfolio models to incorporate innovation, they argue that different categories of products would necessitate a differentiated procurement strategy.

3.2 The Justifications for Public Procurement of Innovation

Interest on the demand side policies has coincided with a resurgence of interest in innovation studies on the importance of demand side factors on innovation beyond supply push effects; the realisation of a relative failure of traditional innovation policies (mainly supply side factors) to boost competitiveness (Edler and Georghiou, 2007; Flanagan et al., 2011), and an interest in the public sector as an innovative actor in its own right (Flanagan et al., 2011) (Edler and Uyarra, 2013). In particular, the use of procurement as an innovation policy tool has been justified as a means to overcome a number of market and systemic failures hindering innovation, as well as the pursuit of additional economic or social goals (for an elaboration see Edler, 2013).

Put simply, the public sector can overcome market failures by enlarging the market for certain goods and services, thus ensuring sufficient critical mass to encourage R&D investment. The public sector can also enable standards through procurement, thus facilitating diffusion. Procurement can also help overcome systemic failures by enabling interaction between users and producers, or by articulating and signalling unmet needs to the market. Government can be highly influential when it is itself the end user of the innovation (Dalpé et al., 1992). Associated with the idea of the lead user, public procurement can contribute to the emergence of so-called “lead markets" (see e.g. Edler and Georghiou, 2007; Georghiou, 2007; Geroski, 1990), understood as “regional markets with specific attributes that increase the probability that a locally preferred innovation design becomes internationally successful as well" (Beise and Cleff, 2004, p.455).

Additional justifications to use public procurement to spur innovation rest on the assumptions that innovation procurement can contribute to the achievement of particular economic and societal goals such as sustainability, social inclusion, etc. (Edler and Georghiou, 2007; McCrudden, 2004). Finally, procurement of innovation can improve the quality and availability of public service delivery (OECD, 2011).

4 Scope

4.1 Limitations and Challenges of Existing Evaluations

The use of public procurement for innovation has increased considerably in the last years, as mentioned above. However, demand-side policies in general, and public procurement of innovation in particular, have been under-evaluated (Edler et al., 2012; OECD, 2011). This lack of evaluation "makes evidence-based policy making in this area difficult” (OECD, 2011; p.12).

Lack of evaluation of demand side innovation policies may be the result of lack of experience given their relative underutilisation. However these instruments also face additional challenges compared to traditional policy tools such as R&D subsidies. Tracing the impact of demand side measures is difficult, and it is problematic to attribute their effects to the intervention. For instance it is difficult to define a control group to assess the effects of policies such as standards and regulation, for they are non-discriminatory (OECD, 2011). It is also difficult to separate the influence of other policies from the supply-side.

Secondly, it is difficult, as we have seen above, to establish a clear boundary of the intervention, particularly in the case of public procurement of innovation. What actually constitutes the policy can be sometimes difficult to define. As Edler et al. (2012, p.35) note, “demand-based policies in this area do not easily lend themselves to 'programmatization' in the sense of having verifiable objectives, a schedule of deliverables, and a budget". Furthermore, procurement is a multi-objective policy that rarely has one single policy objective, and it is therefore difficult to disentangle what aspect of it has which effect. Evaluations of procurement related intervention have generally value for money and other efficiency related objectives and rarely is innovation considered, let alone constitutes the key focus of the evaluation.

Thirdly, demand-side policies often come in a 'mix', and their use in combination makes their evaluation difficult. Different measures may reinforce or conflict with each other. Their effect may be cumulative or successive so for instance one measure may be dependent on another being in place. Evaluations of discrete instruments may thus not be appropriate, requiring instead system-level evaluations that take into account such synergies between instruments and over time (Flanagan et al., 2011; OECD, 2011).

Fourthly, another difficulty lies in defining the market or the target group that the policy is intended to influence. The objectives of policies aimed at transforming markets or creating lead markets are by definition new and difficult to identify by conventional measures. In the case of public procurement, understanding its influence on innovation requires an identification of the target group, namely the population of firms that provide goods or services to or on behalf of the public sector. Lack of reliable data on public procurement poses an important challenge. Understanding the influence of procurement on innovation is further complicated by the fact that public sector suppliers have both government and private sector clients, so it is difficult to disentangle the effect of public expenditure from overall demand. In the case of organizations such as third sector providers, it is necessary to differentiate the effect of government contracts from other forms of government support. Key challenges associated with procurement data thus involve differentiating what is procurement expenditure from other sources such as subsidies or transfers, as well as adequately categorizing public sector spend by sector or functional areas, and capturing the full extent of procurement activities in a traditionally very fragmented public sector.

Systematic evaluations of public procurement of innovation are unsurprisingly rare. Evidence of impact is rather fragmented and restricted to individual ‘success' cases that examine alleged factors contributing to their success, rather than actual impact (Edler et al., 2005; Edquist and Hommen, 2000; Lember et al., 2011).

Quantitative studies have also sought to assess the impact of public procurement, generally vis a vis or in combination with other innovation policy measures. However, the effects of specific measures designed to support public sector of innovation (see next section) are rarely assessed. For instance, research by Rothwell and Zegveld (1981) compared R&D subsidies and state procurement contracts and more recently Aschhoff and Sofka (2009) contrasted the effects of procurement vis-à-vis other instruments (regulation, R&D subsidies and university research) on the innovation activities of German enterprises. Using a survey of 1100 German firms, they found that both public procurement and the provision of knowledge infrastructure in universities had positive effects on innovation success, procurement being particularly effective for smaller firms in regional areas under economic stress and in distributive and technological services.

Following Aschhoff and Sofka (2009), Guerzoni and Raiteri (2012) consider not just the effect of public procurement and R&D subsidies, but also how they may potentially interact. Using data of 5238 firms from the Innobarometer on Strategic trends in innovation between 2006-2008 and in the 27 EU Member States, Norway and Switzerland; they design a quasi-experimental framework to assess the causal effect of both, potentially coexisting policy tools. They found that public procurement not only tends to be more effective than R&D subsidies but also that strongest effects occur when the two policies interact. While these quantitative studies are valuable in demonstrating that procurement has an impact, they are less useful in teasing out the nature of that impact and the precise design of the intervention that is likely to have an impact.

The Underpinn survey (Georghiou et al., 2013) of 800 public sector suppliers in the UK reported that that, of those had introduced innovations, 67% admitted that bidding for or delivering contracts to public sector clients has had some impact on their innovation activity: 25% of the innovating organisations claim that all of their innovations have been the result of public procurement. Furthermore, influence of procurement was reported to be stronger for service than for product innovations, partly reflecting the public sector purchasing portfolio.

From a policy perspective, what matters is not so much whether procurement influences innovation or not, but attempting to better understand the mechanisms by which those impacts occur (Geroski, 1990). As with any other policy instrument, and particularly those that are design sensitive such as public procurement, there is uncertainty about which aspect of its articulation and implementation is actually responsible for any observed effect. The more flexible the measure, the larger number of variables will influence the outcomes and therefore the more difficult it will be to judge its effectiveness (Flanagan et al., 2011).

4.2 Scope

The following section reviews those specific measures that have been introduced as deliberate support to public procurement of innovation. They consist of academic papers but mainly grey literature including policy reports and reviews by the EU and the OECD as well as national policy reports. A number of initiatives where more evidence of impact exists are selected for closer scrutiny although none of them have been rigorously evaluated.

The report does not purport to cover all possible interventions, but only a selection of measures where information is available on their implementation and/or impact.

The basis for the search has been a key word search involving: public procurement, demand, innovation. In addition, it has involved a search through EU and OECD policy databases and reports, as well as recent country reports and thematic cross-country reviews.

5 Findings

5.1 Classification of Interventions

Interventions directed at stimulating innovation via public procurement vary in nature, rationale, and implementation modes. In general, approaches differ from the introduction of legislative measures, to the issue of guidelines, to organizational changes within departments to accommodate for innovation procurement, to competence and skills upgrading, and specific instruments to incentivize the uptake of innovations by procurers. In certain cases, countries may develop integrated strategies or umbrella programmes incorporating one or several of those dimensions.

Georghiou et al. (2013) elaborate a policy framework and policy taxonomy to understand such interventions (see table 1). The framework revolves around a functional approach to procurement, namely the functions that are sought in order to support the procurement of innovation and the deficiencies they seek to remedy. Such a framework thus distinguishes between interventions seeking to address framework conditions for procurement, including the legislative background and broader governance underpinning procurement processes, measures addressed at improving organizational arrangements and capabilities for innovation procurement, mechanisms intended to improve the identification, specification and signalling of needs, and incentives for suppliers to take up innovative solutions.

Table 1. Policy measures in support of innovation public procurement ¹

5.2 Framework Conditions

Framework conditions include modifications in the legal framework, such as those made to the European directives in 2005, which, for the first time, incorporated the promotion of innovation as an explicit objective. It also includes measures to improve the access of SMEs to procurement contracts. Dedicated SME procurement policies are common in certain countries, most notably in the US. They range from the introduction of set asides and targets for public contracting with SMEs to indirect mechanisms to enable their participation such as breaking contracts into lots, information provision and electronic portals.

5.2.1 SME procurement policies

Measures to support SME procurement in the US date back to the modification of the Small Business Act in 1978 to make government procurement more accessible to small businesses. Currently, a statutory minimum goal is set for all government agencies for 23 per cent of all prime contracts by value to be awarded to small businesses. Under EU law however, similar mandatory SME quotas or set asides are not permitted.

There are a number of arguments that could be held in support to the idea of helping SMEs in procurement. For instance it could be argued that small firms face higher costs in terms of regulatory compliance and unequal conditions in credit markets and that they therefore compete in unequal situations. As such they cannot be subject to equal treatment as they are not equals (Yukins and Piga, 2012). Establishing subsidies and set asides for SMES can be justified to ensure sufficient competition in those cases where the number of bidders is insufficient or generally more competition needs to be stimulated, and small firms are identified as a relatively weaker class of bidders (Yukins and Piga, 2012).

Helping SME participation would thus favour greater competition and would make the formation of cartels more unlikely. From a dynamic competition perspective, increased participation would have a variety increasing effect, raising the number of competing solutions and improving the chance of an innovative solution being selected (Uyarra and Flanagan, 2010). In the longer term greater participation would ensure survival of small firms that may become large/successful firms, and contribute to growth and innovation, as opposed to privileging an increasingly narrow number of incumbents. There are also potential downsides associated with such forms of assistance, such as the likelihood of fraudulent behaviour and the high bureaucratic cost associated with establishing those programmes (HM Treasury/OGT, 2008; Yukins and Piga, 2012).

Evidence is inconclusive about the impact of such SME measures. In places where such assistance is well established, such as the US, the rationales are not always clear and small business programmes tend to be dominated by a political agenda (Yukins and Piga, 2012). Information gathered by NESTA on international SME policy in the context of the Glover review on the barriers to SME procurement in the UK; p.38) led them to conclude: "there is strong evidence to suggest that targets alone, even where mandatory on procurers, are not sufficient to encourage SME procurement."

The Glover review concluded that introducing quotas would not be effective in improving SME participation in procurement. It suggested instead that "the main priority must be for the Government to tackle the main issues of transparency, simplicity and strategic procurement and the position of SMEs in the supply chain." (HM Treasury/OGT, 2008: p.5).

This notwithstanding, the UK government has set an aspirational commitment to award 25% of contracts to SMEs. A range of measures have been introduced to enable this objective, including the launch of a new contracts finder website, which would advertise all opportunities over £10,000, the appointment of a Crown Commercial Representative (CCR) for SMEs, and a mandate for a single, simplified PQQ for all main commodities (and the elimination of PQQs for central procurements under £100,000). The ‘one year on' progress (Cabinet Office, 2012) reported that the share of central government direct spending with SMEs was expected to double from 6.5% in 2009/10 to 13.7% in 2011/12.

Indeed, rather than quotas, policy attention has been placed in general on dealing with the barriers influencing the ability of SMEs to access public sector contracts, including too complex tender procedures, inconsistent specifications, size of contracts, length and cost of bidding, and lack of competition. An "Evaluation of SMEs' access to public procurement" in Europe (GHK, 2010) used statistical analysis of contract award notices and contracts awarded between 2006 and 2008 as well as two surveys to companies and contracting authorities respectively to assess the factors influencing SMEs access to public tenders. In that period SMEs won 58% to 61% of public procurement contracts above the EU thresholds, with an associated total contract value between 31% and 38%, which is lower than their overall weight in the economy (52% of total turnover). More specifically, the evaluation addressed the extent to which specific procedures such as dividing tenders into lots, the inclusion of innovation requirements in tenders and eProcurement practices increased SMEs' access to public procurement. They found that certain practices have indeed an effect on SMEs' success rate. In particular, breaking down tenders into lots. After a certain value threshold (€300,000), SMEs participation tends to diminish. Breaking down tenders, either of smaller contracts or geographically, increases SME participation. SMEs tend to secure more contracts from local government than from central government, and gain more contracts in business services and certain manufacturing goods and rarely in contracts for the supply or pharmaceuticals, commodities and food.

Other practices that are useful in improving SME access include the provision of information, through e.g. web portals and other forms of notification, improving dialogue with SMEs and simplifying tendering procedures. The review found a number of barriers preventing contracting authorities from introducing such measures, including (i) a lack of a concrete policy focus on SMEs, (ii) a lack of time and human resources, and (iii) risk averseness. Simple eProcurement tools (publication of notices on websites and electronic access tender documentation) were identified as the most frequently used solutions in the EU, while other, more ambitious, systems such as supplier profiles, online Q&A are less widely implemented.

5.3 Specific procurement policies and strategies

Moving on to the next dimension identified by Georghiou et al. (2013), a number of initiatives have been introduced directed at improving the organizational setting and competence set associated with the use of procurement to promote innovation. Organisational strategies within departments to promote innovation through procurement have been encouraged for instance in the UK.

The Innovation Nation White Paper (DIUS, 2008) included a commitment that “each Government Department will include an Innovation Procurement Plan as part of its commercial strategy, setting out how they will drive innovation through procurement and use innovative procurement practices... These Plans will include details of how Departments will seek to increase their procurement of innovative products and services, fulfil their commitments under existing initiatives such as the Small Business Research Initiative and how they will make use of innovative procurement mechanisms." Such innovation procurement plans (IPPs) were to be regularly reviewed.

Most central government departments subsequently published an innovation procurement plan. The plans provide an indication of current activities and aspirations of how they seek to capture innovative solutions through procurement. However, they have been very different in quality and influence. Giving evidence to the House of Lords, Fergus Harradence of BIS commented that "it would be fair to say that the quality of the plans was variable; some from those departments that had more experience of procuring innovative products and services, such as the Ministry of Defence, were relatively strong. I think others were relatively weak and were perhaps more focused on some of these distinct procurement mechanisms and activities that were under way, rather than being more forward-looking documents of the sort that we were trying to encourage departments to produce" (House of Lords, 2011).

While valuable to identify the degree of commitment to innovative procurement practices and the extent to which these are embedded in commercial activities of Government departments, there has been no evaluation of such plans, and the commitment to produce IPPs has been discontinued by the Coalition Government. The IPPs lacked any key performance indicators or measurable objectives, which made it difficult to assess the performance of departments against their goals. The House of Lords (2011) committee thus described them as "little more than a statement of good intentions".

Related to capacity and competence improvement of the procurement process, a number of initiatives have been introduced directed at improving the skills of procuring organisations. Lack of skills has been identified as a key barrier to innovation procurement. This is because procuring off-the-shelf goods for the lowest possible price requires relatively little in-house competence, whilst greater competence is needed to procure innovative solutions (Rothwell and Zegveld, 1981).

A number of initiatives have been introduced across the EU directed at improving expertise and spreading best practice, such as the Dutch PIANOo initiative and the EC Lead Market Initiative networks of contracting authorities (Georghiou et al., 2013). In Austria, a PPI service point has been established as part of the Federal Procurement Agency (BBG) in the context of the Public Procurement of Innovation Action Plan, the role of which is to provide systematic information exchange among stakeholders, the provision of PPPI tools (e. g. guidelines) and PPPI training.

In the UK concerns about capability shortfalls in central government departments led the OGC to start in 2007 a series of Procurement Capability Reviews, intended to look in detail at key elements of procurement capability in central government departments. Capacity constraints at the local level have been also identified (see e.g. (Byatt 2001) (Uyarra and Flanagan, 2010). As a response, Centres of Procurement Excellent and later on Regional Improvement and Efficiency Partnerships (RIEPs) were set up to overcome a perceived lack of procurement expertise in local government.

A relative lack of experience and capabilities can lead to higher cost in procurement, an issue that has been addressed by initiatives such as TEKES financial support scheme of up to 75% of project expenses in the planning stage (Georghiou et al., 2013).

5.3.1 The PIANOo initiative (The Netherlands)

PIANOo is a Public Procurement Expertise Centre for the Netherlands. Set up in 2005, it is part of the Dutch Ministry of Economic Affairs, Agriculture and Industry. The objectives of PIANOo are to promote high professional standards in purchasing by Dutch public authorities and improvements in their compliance with public procurement rules. It is intended for all those in the public sector who are engaged in the procurement of works, supplies and services.

It started as a knowledge network for government procurement officers and contracting authorities and currently comprises a network of more than 3,000 Dutch procurement professionals (CESS/Oxford Research 2011).

PIANOo supports procurement via a suite of services, including: training, expert guidelines and publications, regular meetings and conferences, support for partnerships and a national portal for information and advice. The portal includes a virtual marketplace for tenders (tendernet) and an on-line discussion platform for public sector procurement professionals and contracting authorities (PIANOo-desk). PIANOo-desk is an online knowledge sharing tool set up in 2002 and incorporated into PIANOo in 2005. These services are provided by a team of around 20 employees with a range of backgrounds and experience in procurement.

PIANOo was evaluated in 2008, using desk research, an internet survey of users of services (652 responses) and some 40 interviews to users and stakeholders. The evaluation mainly focused on the use of the services and their impact against the objectives set, in particular the extent to which it contributed to compliance with procurement rules, professionalism, innovation and value for money.

The evaluation found that PIANOo services are used extensively. Of all the services, PIANOo desk was the most used service, with 81% of the respondents using it, and also the most important according to respondents, followed by the portal PIANOo.nl. All services were highly rated, and the users responded that they will probably (29% of respondents) or definitely (67%) continue to use the services of PIANOo. An even larger proportion of users responded that they were likely to (22%) or definitely would (75%) recommend the use the services of PIANOo to others. The most valued aspects of the services were their customer friendliness and accessibility, rated good or excellent by more than 70% of the respondents. For 70% of respondents to the online survey, the existence of PIANOo is important or very important for the procurement activities of their organization. Respondents from local governments were the most positive of the services of PIANOo and more likely to continue using them in the future.

In terms of the impact of PIANOo services on the user organizations, respondents were asked to rate their influence on compliance with procurement rules, professionalism, innovation and value (on a five point scale from 'completely applicable' to 'not applicable '). Responses showed that services have the greatest effect on the professionalism of the tender conditions, followed by compliance with the procurement rules while the smallest impact is on innovation activity.

The evaluation finds the lag in the effect on innovation unsurprising. It considers that it can probably be explained by the need to first improve the professional levels of contracting authorities before attention can be placed on innovation. The influence on innovation is therefore indirect, through its impact on procurers' competences.

The study concludes that PIANOo has contributed to the objectives set, particularly better compliance with procurement rules, better professionalism and, to a lower extent, innovation and better procurement outcomes (price-quality ratio); and that it has over the years developed into an important source of knowledge and experience for buyers in public organizations.

According to Tazelaar (2008), the success of PIANOo and particularly PIANOo-desk can be explained by the fact that it responds to a real need for professionalism and cost savings in procurement, an increasing demand for these services, as the procurement function becomes more relevant, with a consequent growth of interest in the management and the development of the profession. In particular, he argues that PIANOo-desk has managed to create an environment where actors feel they can share knowledge freely, and where requests are given due attention.

5.4 Identification, Specification & Signalling of Needs

A third set of initiatives contribute to aiding in the phase of planning and articulating demand for innovation. Such initiatives aim to support the identification, articulation and communication of needs from buyers to the market. They strive to prevent too narrowly defined specifications that prescribe specific solutions and reduce the scope for innovative solutions.

One such mechanism is the use of Pre-commercial procurement (PCP) aimed at conducting R&D to develop and demonstrate innovative solutions before commercialization. These policies are explored in more detail in Rigby (2013). Another mechanism is the Competitive Dialogue procurement procedure, which aims to allow pre-qualified tenderers to engage in a discussion to fine-tune the solutions proposed (see 5.4.1 below). Also intended to bridge the gap between suppliers and users are a number of platform initiatives such as the lead market initiative (see 5.4.2). Other mechanisms for signalling demand include the development of catalogues and standards to increase visibility of innovations. For instance the Chinese government introduced a system of 'catalogues' in 2006 (Li, 2011), consisting of innovation catalogues on the one side, namely of products that were innovative and indigenous and catalogues of equipment on the other. The latter represent a 'what we want' list, namely the needs of ministries in terms of new technologies. Such technological areas would also have priority in terms of supporting measures such as R&D subsidies, tax reduction and pre-commercial procurement (Li, 2011).

5.4.1 The Competitive Dialogue Procedure

Incorporated by the European Commission Directive 2004/18/EC and intended to replace the negotiated procedure, the Competitive Dialogue procedure is aimed at "particularly complex contracts" such as integrated transport systems, Private Finance Initiative (PFI) and other forms of Public Private Partnerships (PPPs). The use of the procedure is intended for those instances where procurers are not able to fully define the technical means or the legal and/or financial make up of the project, and where the use of the open or restricted procedure "will not allow the award of the contract" (HM Treasury/OGC, 2008; p.9).

The use of competitive dialogue has not been evaluated but its implementation in different countries has been the subject of several academic studies and policy reviews (see e.g. Arrowsmith and Treumer, 2012; Telles, 2010, 2012). The take up of the procedure has been very uneven in different countries, with 80% of the total procurements using competitive dialogue in Europe taking place in the UK and France. The extensive use of competitive dialogue in the UK can be partly explained by the history of PFI and PPI arrangements for the delivery of public services.

The (HM Treasury, 2010) conducted in 2010 a review of the use of competitive dialogue in the UK, based on desk research, a survey to targeted individuals in the public sector on the use of competitive dialogue, a general web based survey and roundtable discussions with stakeholders. The review identified a few areas of concern, in relation to the extent and mode of application, its performance and its effect on innovation.

Firstly, the review identified an extended use and potentially a misuse of the procedure. By 2010 there were thousands of competitive dialogue cases in the UK, almost half of which were conducted by local authorities. In general, the review found that the procedure is used not just in complex procurements, but almost as a default process for all but the most straightforward procurements. Local authorities are more likely to use competitive dialogue as default, while central government departments tend to use a broader range of procurement options.

A second and related concern relates to the competences required to effectively conduct competitive dialogue. Different parts of the public sector present differential capacities to deliver the procedure; while for Local Authorities the review found few cases of repeat experience of Competitive Dialogue procurements and little evidence of knowledge sharing between delivery teams, central government contracting authorities appear to have more capacities to assume the increased resources needed to deliver the procedure.

A third concern has to do with whether the procedure was able to secure improved solutions in procurement. According to the survey conducted, 78% of respondents considered that the use of competitive dialogue has contributed to improved solutions compared with the negotiated procedure.

However, the use of competitive dialogue has been associated with higher costs of bidding. Increased cost was a concern for 86% of respondents to the HM Treasury survey and 55% of contracting authorities. From the public sector, increased costs are the result of additional administration, evaluation and support costs associated with the increased number of bidders entered into dialogue. For suppliers, greater costs result from delayed procurement timetables and the need for greater expertise. Costs for the private sector are estimated to have risen from 2-3% of contract size under negotiated procedure to 5-6% under competitive dialogue. This is particularly a problem for smaller bidders who may lack the resources and skills to undertake such procurement. The procedure may thus act as a barrier to entry into public sector contracts.

A final consideration, more directly related to innovation, relates to the danger of 'cherry picking' that can take place in the process of competitive dialogue, resulting in participants seeing their innovative ideas or solutions revealed to their competitors. The directives establish that the contracting authorities may not reveal to the other participants confidential information presented by candidates during the dialogue without their agreement. 'Cherry picking' is therefore not allowed unless the participant agrees. This ban was introduced in order to balance the interests of the contracting authority and the participants in the procedure (Treumer and Uyarra, 2012). Even though information about proposed solutions must be kept confidential, it may be difficult to avoid the disclosure of some information. Contracting authorities may be tempted to set aside the ban to secure value for money, or try to get the agreement of the tenderers to cherry picking. Despite this danger, and the contentious nature of the issue of cherry picking, there have been very few disputes resulting from the use of competitive dialogue and there is generally a lack of case law in relation to its application (Treumer and Uyarra, 2012).

The HM Treasury review notes that this is an issue of significant concern in the UK (highlighted by 55% of public sector respondents to their survey); however it appears that contracting authorities are giving due care to respecting confidentiality, the level of objection therefore being low. However, the review also notes that bidders often wait until relatively late in the process to introduce their most innovative ideas.

5.4.2 The Lead Market Initiative

The Lead Market Initiative (LMI) was adopted in 2007 as a response to the 2006 Aho report 'Creating an Innovative Europe'. The Lead Market Initiative aims to foster the emergence of specific lead markets that are important in both economic and social terms and that are likely to become very significant on a global scale. It has identified six broad markets (eHealth, protective textiles, sustainable construction, recycling, bio-based products and renewable energies) as having the potential to become 'lead markets', based on their strategic societal and economic interests, the importance of demand and the added value of targeted policy instruments.

Support was articulated through action plans for each lead market lasting for 3-5 years and comprising a mix of demand-side policy measures in the fields of legislation, standardisation and labelling, public procurement and complementary activities.

In 2011 an evaluation was conducted focusing on the development of each of the markets measured by market growth, employment rate, turnover, growth in procurement, number of patents/trademarks etc (CESS/Oxford Research 2011) and progress in implementing the action plans of each of the 6 lead markets in the chosen policies. In terms of assessing progress of the action plans, it considered, following Edler et al (2009) the relevance and coherence of the intervention; 2) the effectiveness of the intervention (particularly in terms of the specification of objectives and implementation process); 3) efficiency; 4) sustainability of the intervention; 5) value-added; and 6) utility.

Assessing the development of each of the markets is problematic given the difficulties associated with the definition of the relevant sectors, in particular a lack of correspondence with established NACE codes. This in turn renders data used for the evaluation of lead markets very provisional and subject to constant revision. For instance, development within the renewable energy sector is especially difficult to predict due to discrepancies in definition of the sector and internal variability among subsectors. Growth in procurement according to the TED database has been very uneven across sectors: while public procurement for e-health and protective textile has been low, in recycling, bio products and specially renewable energy the overall level of public procurement has risen. Finally, patent applications have also seen an uneven development, with low growth and a decrease in protective textiles and recycling respectively and a large increase in other sectors such as e-health and renewable energy.

In relation to the attribution of impacts (in employment, turnover, patents and procurement activity) to the LMI, the evaluation tentatively considers that 20% of the anticipated growth in the markets could be attributed to the Initiative, except for the renewable energy sector (more dependent on the establishment of targets for the share of renewable energy). However it recognizes the difficulty of attributing causality, especially to measures designed to improve the market environment. The evaluation also acknowledges a lack of evidence of impact of the LMI on demand-side instruments such as public procurement. Furthermore, the activities considered may not be attributable to the lead market initiative but to other policy developments.

In terms of progress, most of the selected markets were considered to be relevant and representative and in each of them demand side policies were seen as key for their growth. In protective textiles, relevance was highlighted in relation to its potential to spill over into other markets such as consumer goods. However it was felt that, for renewable energy, the broader range of policy efforts addressing energy issues was not fully addressed. In particular, the evaluation argued that greater attention should be paid to supporting the development of technological capacity in the sector. This is complicated by the fact that the sector includes a number of sub-sectors (such as wind, solar, bio-fuels, sustainable nuclear fusion, grid technology) with very different levels of technological and market development and different dynamics. Similar issues were raised in relation to the wider range of policy developments that affect the construction industry. In recycling, the review highlighted the disparities in relation to policy and actions across member states in the field of recycling.

Some action plans (such as bio-products, textile, construction) were perceived to be appropriately focused and effective in relation to the challenges facing the sector. Some of the actions have been completed and some haven't and thus effectiveness has been uneven. For instance, effectiveness in relation to public procurement was found to be low in e-health and recycling but high in textiles. In Energy, the evaluation found that the LMI has made very little contribution, and all activities appear to have taken place outside of the LMI framework. Construction, albeit effective and relevant, has faced difficulties in its implementation given the complexities associated with stakeholder engagement.

In terms of efficiency, some questions have been raised in terms of the small volume of investment (e-health), fragmentation and lack of accessibility of funding (bio-based products). In textiles, particularly because of the industry support, the Initiative has been able to proceed with a relatively restricted use on public resources. Funding issues do not appear to have posed any problems so far and this market has been able to make good use of funds available from other sources. Given the modest development of the LMI in renewable energy, efficiency is difficult to assess.

Utility, sustainability and value added have been found to be high in the LMI of biobased products and e-health. In e-health, policy initiatives to enable EU cross-border co-operation are seen as particularly positive to enable beneficial outcomes such as economies of scale. Value added in textiles was favoured by its more targeted and focused nature relative to other LMIs, which as helped engage suppliers. In Recycling, greater coordination and standardisation in recycling activity is necessary before its full potential can be realized. The LMI was found to have little value added or benefit for renewable energy. In construction, value added was perceived to be important, although still limited given the needs of the sector.

Overall, the evaluation concluded that the LMI has fallen short of the ambitions and high expectations raised by the Aho report. Limited results have been attributed to the experimental nature of the initiative, limited funding, and different degrees of implementation of the action plans. Nevertheless the evaluation considers that the six markets continue to show market potential and that a major strength of the initiative was targeting interrelated policy areas that would not otherwise be picked up.

5.5 Concrete Measures to Incentivize Solutions.

A final set of interventions are directed at changing practices, overcoming the inherent risk of the public sector, associated with the uncertainty associated with the adoption of innovative solutions. It seeks to mitigate the risks of procurers not purchasing the innovative solutions developed by suppliers.

From the supplier perspective the biggest risk is that the purchaser will not respond to their offering of an innovative solution. The UK's Forward Commitment Procurement (FCP) for instance incorporates a guarantee of purchase of a developed solution if it fulfils the performance requirements set at a price commensurate with its benefits (see 5.5.1).

Through legislation, Germany has included innovation demands from contracting authorities upon suppliers. As part of the PPI action plan, the Austrian government also explicitly includes innovation as a secondary procurement objective (on a par with social, ecological and SME issues already included as secondary objectives.)

The development of catalogues, standards and certifications of products, such as the Chinese 'innovation catalogues' or the Korean excellent technology product certification (see 5.5.2) also enable the take up of innovative solutions (Li, 2011). In England, the NHS has introduced quality certificates and demonstration procurement procedures (via the National Institute for Health and Clinical Excellence, NICE, and the National Technology Adoption Centre, NTAC, respectively) (Georghiou et al., 2013). In some cases additional incentives (or targets) are added to encourage the take up of those products. This is the case of the South Korean government's New Technology Purchasing Assurance Scheme (see 5.5.2), which includes a financial insurance for procurers to mitigate the risks involved in purchasing high tech products.

5.5.1 Forward Commitment Procurement

Initially developed in procurement related to environmental innovations (Defra 2006), forward commitment procurement (FCP), seeks to address the need for a product or service that does not exist through providing market incentives for its development. In other words, it helps to lower the level of perceived risk associated with investing in innovation by increasing the confidence that there will be a market for the product or service once the solution is proven. FCP consists of three stages (identification of need, market engagement and procurement). In a first stage, the purchasing authority signals the market the need for innovative solutions to a particular problem in a Prior Information Notice. The notice defines the requirements in terms of particular performance outcomes. A second stage consists of engagement with potential suppliers, followed by a formal procurement stage. Such procurement may incorporate a forward commitment, namely an agreement to purchase the developed solution at a price that is commensurate with its benefits.

One of the most celebrated cases of FCP is the procurement of zero waste mattresses by HM Prison Service (HMPS), which used the model to procure a solution that prevented disposal of mattresses and pillows into landfill. The result were innovative new covers that reduced turnover and a fully recyclable mattress with cost savings estimated to be in the region of £5 million over the life of the contract. Other projects developed following this methodology include the procurement of Ultra Efficient Lighting of the Rotherham NHS Foundation Trust, with a solution that involved biodynamic lighting enabling energy consumption and maintenance savings of 30% and 88% respectively, and a recent innovation for sustainability pilot project competition launched by Defra and DIUS.

The FCP initiative has not yet been evaluated. Evidence of impact stems from a number of good practice cases and there is no evidence of the extent to which such practices have become embedded in public sector procurement. According to Gaynor Whyles², “there seems to be a fair amount of resistance" associated with the use of the FCP framework, since aspects such as outcome based specifications based on a problem is "opposite to what procurers normally do".

5.5.2 New technology purchasing assurance programme (Korea)

New technology purchasing assurance programme for public procurement of SME technology products started in 1996 aimed at stimulating technology development by SMEs. At its inception it consisted of a series of recommendations by the Korean Small and Medium Business Administration (SMBA) to public institutions to purchase SMEs technological products that have been approved for performance by the government.

The programme initially had a poor uptake of high quality SME technology products, due to various factors, including (OECD, 2011): Lack of recognition and insufficient interests of public organizations in procuring SMEs products; lack of product quality verification and thus lack of confidence of the quality and performance of SME goods; legal and institutional barriers, and in particular weak legal grounds for enforcing public procurement. The programme was not legally binding and therefore not enforceable by law.

In the light of these findings the programme was revised during 2005-6. Among the changes adopted, a Performance Certification System and a Performance Insurance system were incorporated to increase the take up of the scheme by public authorities. It included preferential conditions for the procurement of performance-certified products, such as an insurance scheme where procurers are compensated for their potential losses. The objective was to provide buyers with immunity from losses incurred due to the procurement of such products.

The effectiveness of the programme is secured by the adoption of a mandatory purchasing of 20% of new products, whereas previously the programme was based on recommendations. Moreover, the implementation is reinforced by annual reporting by the SMBA and by the requirement of notification by public organizations or their preferred procurement performance and reasons of non-procurement. Finally, a Technology Product Procurement Promotion Committee (composed of around 20 specialist) was introduced to select and recommend technology products for preferred procurement.

Even though the programme has not been evaluated, some results are available about the take up of the programme. Before the modification of the programme, the procurement of new SME products was merely 3% of total public procurement of SMEs. After 2005, however, the share has risen, reaching 9.3% in 2009. Equally, public procurement of SME technology products amounted USD 1.9 billion in early stages of the programme, and more than tripled after 2005.

6 Evaluation and Public Procurement of Innovation

The previous section has examined policy practice of a range of measures introduced to facilitate the promotion of innovation through public procurement. They consisted of a variety of interventions, ranging from legislative measures, to financial incentives, targets, information provision and mechanisms to secure dialogue between users and producers. They range from more formal interventions, to umbrella programmes and strategies, to concrete instruments.

The report does not intend to be exhaustive, but to provide an overview of the diverse measures developed in different countries and in different stages of the procurement life cycle. As such, this portfolio of measures presents a number of challenges with regards to evaluation. Firstly, given their diversity, it is difficult to draw general lessons about impact. Secondly, it can be sometimes difficult to delineate the specific intervention and differentiate it from other demand side and even supply side measures. The objectives and rationales of such measures are not always clear-cut. Thirdly, some of these measures involve a mix of demand side interventions (such as the lead market initiative). In practice, most of the interventions described in section 5 come in a mix and it is difficult to disentangle the effects of different measures.

Some interventions may thus not have a clear rationale by which to judge its effectiveness (e.g. set asides for SMEs). Many procurement related interventions have a mix of objectives, innovation being just one of them. Desired outcomes in procurement (such as value for money, sustainability and innovation) can sometimes be contradictory or can complement each other. Innovation outcomes may not be an explicit objective per se, but emerge as a by-product of increased professionalism (e.g. competence building programmes such as PIANOo).

As a result innovation outcomes may only be evident some time after the intervention. Most of the evaluations encountered have assessed the programmes soon after their completion, thus not leaving sufficient time for certain impacts to materialise.

In other instances, the programme has not been in place for long enough or has been discontinued even before their impact was evaluated (e.g. the IPPs in the UK, product catalogues in China). In other cases, the measure is still almost at a pilot stage (e.g. FCP in the UK), with only a limited number of examples and thus not sufficient evidence of impact. In a number of countries, and pre-commercial procurement activities aside, initiatives oriented to stimulate innovation through procurement often do not go beyond statements of intent and guidelines containing aspirations to support innovation procurement, without concrete implementation, let alone monitoring and evaluation mechanisms.

Data availability constitutes another hurdle in assessing impact. Procurement data is fragmented and unreliable, allowing limited comparability across countries, and consequently the share of innovative procurement is difficult to estimate. This constitutes a non trivial issue: how can we assess the impact of a particular intervention on the procurement of innovation if data on innovation procurement does not exist? Impact on the target organizations (suppliers to the public sector) requires the identification of such organisations and the relative importance of public sector procurement in their innovation activity. Dedicated surveys present limitations such as self-reporting, and sometimes underrepresentation of small organizations, potential selection bias (firms do not participate randomly in public procurement) and the difficulty of establishing a control group.

Finally, procurement measures are mediated by their implementation. In particular, most interventions are conditioned by the way they are conducted, and particularly by the competences and skills of procurers. The HM Treasury (2010) review on competitive dialogue noted that 'the outcome of a procurement will be influenced as much by the capacity and capability of those party to the process as by the nature of the contract to be delivered'. Equally, the delivery of initiatives such as the FCP in the UK is subject to procurement professionals possessing the commercial skills needed to engage with suppliers for the development of innovative solutions. How the measures are designed and implemented heavily condition their outcome and make comparison across measures even more difficult.

7 Conclusions and general lessons learned

Despite the general interest in the public procurement of innovation, evidence of actual policy practice in this area is still scarce. And when it exists, it is either too early to properly assess impact or no evaluation exists of such measures. Evaluations are limited in their scope and the methodologies adopted, and rarely has innovation been the main focus of the evaluations.

This notwithstanding, the present review shows a portfolio of initiatives seeking to encourage innovation outcomes in the procurement process. Following Georghiou et al. (2013) we have adopted a broad framework that considers all possible interventions seeking to address all the dimensions enabling innovation across the procurement life cycle, namely the framework conditions, competences, articulation of demand and incentives.

The review suggests first, that more efforts are needed to understand the nature of procurement related interventions, namely the characteristics, instrument design and implementation of such measures. Given their distinct nature, they require a different framework of analysis compared to supply side measures in terms of programmatic design, budget and objectives. More information is needed on the precise rationales, logic and objectives of some measures. These are rarely straightforward in the case of demand side policies. In addition, many initiatives are still in pilot stage, with relatively low funding attached. The EU lead market initiative is an example of a policy that generated high expectations and yet under delivered in terms of implementation and impact. Secondly, better metrics are required to assess impact, particularly on emerging markets. Equally, procurement data needs improvement to enable comparability and rigorous evaluation. Finally, improved methodologies are needed to understand and trace the impact on innovation of demand side interventions in general, and procurement in particular. All evaluations reviewed in this report fall short of providing a rigorous and transparent assessment of policy impacts.

References

Aho, E., Cornu, J., Georghiou, L., Subira, A., 2006. Creating an Innovative Europe: Report of the Independent Expert Group on R&D and Innovation appointed following the Hampton Court Summit. European Communities: Luxembourg.

Arrowsmith, S., Treumer, S., 2012. Competitive Dialogue in EU Procurement. Cambridge University Press: Cambridge.

Aschhoff, B., Sofka, W., 2009. Innovation on demand—Can public procurement drive market success of innovations. Research Policy, 38 (8): 1235–1247.

Beise, M., Cleff, T., 2004. Assessing the lead market potential of countries for innovation projects. Journal of International Management, 10 (4): 453-477.

Blind, K., 2013. The Impact of Standardization and Standards on Innovation. Report within the MIOIR-NESTA Compendium of Evidence on Innovation Policy. MIOIR-NESTA: Manchester/London.

Blind, K., 2012. The Impact of Regulation on Innovation. MIoIR-NESTA: Manchester/London.

Byatt, I., 2001. Delivering Better Services for Citizens-a Review of Local Government Procurement in England. Department for Transport, Local Government and the Regions: London.

Cabinet Office, 2012. Making Government business more accessible to SMEs - One Year On. CabinetOffice: London.

CESS/Oxford Research, 2011. Final Evaluation of the Lead Market Initiative: Final Report. Publications Office of the European Union : Luxembourg.

Dalpé, R., DeBresson, C., Ciaoping, H., 1992. The public sector as first user of innovations. Research Policy, 21 (3): 251-263.

DEFRA, 2006. Procuring the Future. Department for Environment, Food and Rural Affairs: London.

DIUS, 2008. Innovation Nation. Department for Innovation, Universities & Skills: London.

Edler, J., 2011. Innovation in EU CEE: the role of demand-based policy, in Radosevic, S., Kaderabkova, A. (Ed.), Challenges for European Innovation Policy: Cohesion and Excellence from a Schumpeterian Perspective, Edward Elgar: Cheltenham, pp.177-208.

Edler, J., 2013. Review of Policy Measures to Stimulate Private Demand for Innovation. MIoIR-NESTA: Manchester/London.

Edler, J., Corvers, S., Xielin, L., 2008. Public Procurement and Innovation: OECD Experience and Reflections on China, in OECD (Ed.), OECD Reviews of Innovation Policy, OECD: Paris, pp.563-591.

Edler, J., Georghiou, L., 2007. Public procurement and innovation - Resurrecting the demand side. Research Policy, 36 (7): 949–963.

Edler, J., Georghiou, L., Blind, K., Uyarra, E., 2012. Evaluating the demand side: New challenges for evaluation. Research Evaluation, 21 (1): 33-47.

Edler, J., L. Georghiou, E. Uyarra, D. Cox, J. Rigby, Nugroho, Y., Blind, K., 2009. Monitoring and Evaluation Methodology for the EU Lead Market Initiative A Concept Development. Manchester Institute of Innovation Research: Manchester.

Edler, J., Ruhland, S., Hafner, S., Rigby, J., Georghiou, L., Hommen, L., Rolfstam, M., Edquist, C., Tsipouri, L., Papadakou, M., 2005. Innovation and Public Procurement. Review of Issues at Stake. European Commission: Brussels.

Edler, J., Uyarra, E., 2013. Public Procurement of Innovation, in Brown, L., Osborne, S. (Ed.), The Handbook of Innovation and Change in Public Sector Services, Edward Elgar: Cheltenham.

Edquist, C., Hommen, L., 2000. Public Technology Procurement and Innovation Theory, in Edquist, C., Hommen, L., Tsipouri, L. (Ed.), Public Technology Procurement and Innovation, Springer: New York, pp.5-70.

European Commission, 2007. A Lead Market Initiative for Europe. European Commission: Brussels.

European Commission, 2011a. Communication from the Commission to the European Parliament, the Council and the European Economic and Social Committee. European Commission: Brussels.

European Commission, 2011b. Green Paper on the modernisation of EU public procurement policy Towards a more efficient European Procurement Market. European Commission: Brussels.

Flanagan, K., Uyarra, E., Laranja, M., 2011. Reconceptualising the 'policy mix' for innovation. Research Policy, 40 (5): 702-713.

Georghiou, L., 2007. Demanding Innovation Lead markets, public procurement and innovation. NESTA: London.

Georghiou, L., Edler, J., Uyarra, E., Yeow, J., 2013. Public procurement as an innovation policy tool: choice, design and assessment.. Technological Forecasting and Social Change, Available online 24 October 2013

Georghiou, L., Li, Y., Uyarra, E., Edler, J., 2010. Public Procurement for Innovation in Small European Countries. Report to the European Commission in the context of the ERAPRISM OPMC-Net project (Policies for Research and Innovation in Small Member States to Advance the European Research Area): Manchester/Brussels.

Geroski, P.A., 1990. Procurement policy as a tool of industrial policy. International Review of Applied Economics, 4 (2): 182-198.

GHK, 2010. Evaluation of SMEs' access to Public procurement markets in the EU. European Commission: Brussels.

Gok, A., 2013. The Impact of Innovation Inducement Prizes. MIoIR-NESTA: Manchester/London.

Guerzoni, M., Raiteri, E., 2012. Innovative procurement and R&D subsidies: compounding effects and new empirical evidence on technological policies in a quasi-experimental setting. Bureau of Research in Innovation, Torino: http://brick.carloalberto.org/images/publications/18_wp_2012.pdf.

Harper, J., 2013. Impact of Technology Foresight. MIoIR-NESTA: Manchester/London.

HM Treasury, 2007. The Race to the Top A Review of Government's Science and Innovation Policies. HM Treasury: London.

HM Treasury, 2010. Review of Competitive Dialogue. HM Treasury: London.

HM Treasury/OGC, 2008. Competitive Dialogue in 2008. HM Treasury: London.

Hommen, L., Rolfstam, M., 2009. Public procurement and innovation: Towards a taxonomy. Journal of public procurement, 9: 17-56.

House of Lords, 2011. Public procurement as a tool to stimulate innovation. House of Lords: London.

Kraljic, P., 1983. Purchasing Must Become Supply Management. Harvard Business Review, 61 (5): 109-117.

Lember, V., Kalvet, T., Kattel, R., 2011. Urban Competitiveness and Public Procurement for Innovation. Urban Studies, 48 (7): 1373-1395.

Li, Y., 2011. Public procurement as a demand-side innovation policy tool in China a national level case study. DRUID, Copenhagen, Jun 15 2011 12:00AM.

McCrudden, C., 2004. Using public procurement to achieve societal outcomes. Natural Resources Forum, 28 (4): 257–267.

NESTA, 2012. Plan I. The case for Innovation-led growth. NESTA: London.

OECD, 2011. Demand Side Innovation Policy. OECD: Paris.

OGC, 2004. Capturing Innovation: Nurturing. Suppliers' Ideas in the Public Sector. Office of Governement Commerce: London.

Rigby, J., 2013. Review of Pre-commercial Procurement Approaches. MIoIR-NESTA: Manchester/London.

Rothwell, R., Zegveld, W., 1981. Government regulations and innovation, in National Academy of Engineering (Ed.), Industrial innovation and public policy options: report of a colloquium, National Academies Press: Gaithersburg, pp.116-147.

Tazelaar, K., 2008. PIANOo, A KNOWLEDGE NETWORK FOR PUBLIC PROCUREMENT. Dutch Public Procurement Expertise Centre, The Hague: http://www.ippa.org/IPPC4/Proceedings/14ProcurementProfession/Paper14-6.pdf.

Telles, P., 2010. The competitive dialogue in Portugal. Public Procurement Law Review, 1.

Telles, P., 2012. Competitive dialogue in Spain and innovation: the case of the health sector, in Piga, G., Treumer, S. (Ed.), The Applied Law and Economics of Public Procurement, Routledge: London.

Treumer, S., Uyarra, E., 2012. Competitive dialogue and contractual design fostering innovation and need analysis, in Piga, G., Treumer, S. (Ed.), The Applied Law and Economics of Public Procurement, Routledge: London.

Uyarra, E., Flanagan, K., 2010. Understanding the Innovation Impacts of Public Procurement. European Planning Studies, 18 (1): 123-143.

Wilkinson, R., Georghiou, L., Cave, J., Bosch, C., Caloghirou, Y., Corvers, S., Dalpé, R., Edler, J., Hornbanger, K., Mabile, M., Montejo, M.J., Nilsson, H., O'Leary, R., Piga, G., Tronslin, P., Ward, E., 2005. Public Procurement for Research and Innovation: Developing procurement practices favourable to R&D and innovation. European Commission: Brussels.

Yukins, P., 2012. Dialogue: Set-asides for small and medium firms, as in the USA system with the Small Business Act that reserves shares of tenders to SMEs only, in Piga, G., Treumer, S. (Ed.), The Applied Law and Economics of Public Procurement, Routledge: London.