Nesta Working Paper 13/19

July 2014

www.nesta.org.uk/wp13-19

Abstract

This report identifies and reviews literature that evaluates the impacts of technology and innovation advisory services. These services provide information, technical assistance, consulting, mentoring, and other services to support enterprises in adopting and deploying new technologies and in commercialising innovations. Examples include the: Manufacturing Advisory Service (England), the Manufacturing Extension Partnership (USA), and the Industrial Research Assistance Program (Canada). Technology and innovation advisory services are also provided by technology centres and other business assistance programmes. Such services are typically targeted at small and medium-sized enterprises. A defining capability of technology and innovation advisory services is the offer of expert one-on-one guidance to individual companies through extension staff, field offices, or dispersed technology centres. The available studies generally find that technology and innovation advisory services provide positive benefits for participating firms. The types of benefits achieved include reductions in costs, improved quality, reduced waste and improved environmental performance, higher productivity, and new product development and innovation. Net benefits achieved are typically relatively modest, although commensurate with the relatively small amounts of public funding usually invested in technology and innovation advisory services. More intense and customised services can improve results. Applying best practices in the operation and delivery of technology and innovation services, including attention to outreach, staff capability, long-term engagement, flexibility, organisational stability, and linkage with other services, increases the likelihood of programme effectiveness and impact.

JEL Classification: 038

Keywords: Technology advisory services, extension services, evaluation, innovation policy

The Compendium of Evidence on the Effectiveness of Innovation Policy Intervention Project is led by the Manchester Institute of Innovation Research (MIoIR), University of Manchester, and sponsored by Nesta, an independent charity with the mission to make the UK more innovative. The compendium is organised around innovation policy topics categorised primarily according to their policy objectives. All reports are available at http://www.innovation-policy.org.uk. Also at this location is an online strategic intelligence tool with an extensive list of references that present evidence for the effectiveness of each particular innovation policy objective. Summaries and download links are provided for key references. Corresponding Author: Philip Shapira, Manchester Institute of Innovation Research, H Hankins Building, Manchester Business School, University of Manchester, Manchester M13 9PL. Email: [email protected].

The Nesta Working Paper Series is intended to make available early results of research undertaken or supported by Nesta and its partners in order to elicit comments and suggestions for revisions and to encourage discussion and further debate prior to publication (ISSN 2050-9820). © Year 2014 by the author(s). Short sections of text, tables and figures may be reproduced without explicit permission provided that full credit is given to the source. The views expressed in this working paper are those of the author(s) and do not necessarily represent those of Nesta.

This report is part of the Compendium of Evidence on the Effectiveness of Innovation Policy Intervention Project led by the Manchester Institute of Innovation Research (MIoIR), University of Manchester. The project is funded by NESTA – an independent body with the mission to make the UK more innovative.

The compendium is organised around 20 innovation policy topics categorised primarily according to their policy objectives.

All reports are available at http://www.innovation-policy.org.uk. Also at this location is an online strategic intelligence tool with an extensive list of references that present evidence for the effectiveness of each particular innovation policy objective. Summaries and download links are provided for key references. These can also be reached by clicking in the references in this document.

Authors and Acknowledgements

This report was authored by:

Philip Shapira¹,² and Jan Youtie³

1. Manchester Institute of Innovation Research, Manchester Business School, University of Manchester, Manchester M13 9PL, UK; [email protected]
2. School of Public Policy, Georgia Institute of Technology, Atlanta, GA 30332-0235, USA
3. Enterprise Innovation Institute, Georgia Institute of Technology, Atlanta, GA 30332-0390, USA; [email protected]

The authors gratefully acknowledge research assistance provided by Jaime Humberto Sierra Gonzalez and Omid Omidvar.

Executive Summary

Technology and innovation advisory services provide information, technical assistance, consulting, mentoring, and other services to support enterprises in adopting and deploying new technologies and in commercialising innovations. Such services are typically but not exclusively targeted at small and medium-sized enterprises (SMEs) in manufacturing. A defining capability of technology and innovation advisory services is the offer of expert one-on-one guidance to individual companies through extension staff, field offices, or dispersed technology centres, although there are complementary efforts to provide services to groups of companies and through online means.

The organisation and operations of technology and innovation advisory services differ by country and region. Services may be organised nationally or locally and may be provided through universities, non-profit organisations, technology centres, and private companies. Sponsorship may involve a mix of public and private revenue sources, with considerable variations in the range and intensity of services offered. Technical assistance is not usually coupled with direct financial support, although there are exceptions and technology and innovation guidance can lead to subsequent private and public funding. Leading examples of technology and innovation advisory services include the:

• Manufacturing Advisory Service (MAS), offering manufacturing business support to companies in England through a network of expert advisors. MAS is sponsored by the UK Department for Business, Innovation and Skills.
• Manufacturing Extension Partnership (MEP), providing field services, including process improvement, green manufacturing, and innovation strategies, to SMEs in the United States. MEP centres and offices are located in all US states, with funding from the US National Institute of Standards and Technology, state governments and private sources.
• Industrial Research Assistance Program (IRAP), which combines advisory services, funding, networking, and training to promote technology, innovation and growth for SMEs in Canada. IRAP is sponsored by Canada's National Research Council.

The major rationale for public support in providing technology and innovation advisory services is market failures. If unaddressed, these will result in economically and socially sub-optimal investment in new technology and innovation by enterprises. Market failures may be generated by a combination of reasons, including lack of information awareness and expertise among firms, difficulties in choosing between technologies, lack of access to finance or inability to justify investment in new technology, expensive or weak private consulting availability, and short-term contracting and lack of support from major customers. Such issues typically affect mature SMEs. Additionally, there are system and institutional failures, where the needs of existing SMEs receive low priority in practice, for example by universities focused on basic research, partnerships with larger companies, or high technology start-ups, or by innovation programmes which target advanced science and technologies rather than more routine technology assistance needs. Public policies for technology and innovation advisory services seek to address these market and system failures by providing dedicated services to SMEs to address their technological needs and opportunities. Immediate intended effects include encouraging investment in new technologies, training, lean production, collaboration, strategy and market development, with longer-term effects on innovation and productivity. Broader effects are also sought, such as retaining and creating employment, supporting business growth and competitiveness, stimulating economic development and promoting exports.

This report collates and examines literature that evaluates the impacts of technology and innovation advisory services. The methods used in evaluations include surveys, case studies, centre reviews, and econometric studies. In several cases, quasi-experimental control group evaluations have been conducted, comparing assisted and non-assisted firms, controlling for the performance of firms prior to programme entry. As with evaluations in other areas of business assistance, there can be issues related to self-reported impacts, selection bias, and attribution (since multiple factors, including service assistance, may lead to reported impacts). These problems are more in evidence in simple evaluations, but are addressed in relatively robust ways in more complex studies, especially those with control groups.

The available studies generally find that technology and innovation advisory services provide positive benefits for participating firms. The types of benefits achieved include reductions in costs, improved quality, reduced waste and improved environmental performance, higher productivity, and new product development and innovation. Levels of investment involved (by both the public sector and private participating firms) are typically not high. Similarly, the net benefits achieved are often relatively modest for individual projects, although such incremental improvements add up and can make the difference to SME survival or decline. There are examples of significant and fundamental improvements for participating firms, but this is not the mode. Controlled studies generally tend to show lower net effects. Broader benefits to the economy are typically estimated through leveraging and multiplier assumptions about gross value added, although studies recognise that it is difficult to precisely estimate broader effects and spillovers. Studies do not commonly find unintended effects. For example, while concerns have been raised about negative impacts of publicly supported technology advisory services on private consultants, the studies that have examined this do not find such effects. Indeed, private consultants are often engaged by public technology advisory services allowing them to serve and market to SMEs that would not otherwise have engaged their services.

Technology and innovation advisory services are typically funded at relatively modest levels, but there have been tensions in sustaining funding support and particularly in balancing the mix of public and fee income. Efforts to increase the amount of service costs paid by participating firms appear to allow leveraging and reduce pressure on public funds; however, as targets for private fee income are increased, services tend to go "up market" to serve larger firms that can afford to pay higher levels of fees. There is also a tension in technology advisory services in selecting who to serve and how. Some evaluations suggest that more intense and customised services provide improved results, although this will mean serving fewer firms; at the same time, there is often policy pressure to serve as many firms as possible, which means that service assistance is less intense than may be desirable. Programmes often address this challenge by providing initial assistance and then referring companies to qualified private consultants and other public services. Many studies discuss the preconditions for effective programme performance, including professional and industrially experienced staff, good outreach and branding, effective systems to diagnose enterprise needs, flexibility, and the ability to maintain a long-term perspective. However, challenges are often raised between providing short-term assistance – for instance, by helping a firm better operate its existing technology to save costs – which has immediate and measurable effects and providing support for strategic development, mentoring and innovation which takes much more time and whose effects are harder to measure.

The key findings and insights for policy makers to emerge from the analysis are:

• Technology and innovation advisory services provide positive benefits for participating firms: reductions in costs, improved quality, reduced waste and improved environmental performance, higher productivity, and new product development and innovation.
• Net benefits achieved are typically relatively modest, although commensurate with the relatively small amounts of public funding usually invested in technology and innovation advisory services. Nonetheless, such incremental improvements add up and can make the difference to SME survival or decline. Controlled studies generally tend to show lower net effects.
• More intense and customised services provide improved results; but there is often policy pressure to serve as many firms as possible, which means that service assistance is less intense than may be desirable. Programmes typically seek to address this challenge by providing initial assistance and then referring companies to qualified private consultants.
• Efforts to increase the amount of service costs paid by participating firms allow leveraging; however, technology and innovation services then tend to go "up market" to serve larger firms that can afford to pay higher levels of fees.
• Applying best practices in the operation and delivery of technology and innovation services, including attention to outreach, staff capability, long-term engagement, flexibility, organisational stability, and linkage with other services, increases the likelihood of programme effectiveness and impact.

1 Introduction

Technology and innovation advisory services are services provided directly by specialists particularly to small and medium-sized enterprises (SMEs)¹ to support and stimulate improvements in business operations including productivity, efficiency, production, quality, waste reduction, information technology and logistics. Increasingly, such services also focus on innovation in design, products and services, and business models. These are "real services," using the terminology of Bellini (2003), in that they are provided through direct expert engagement with SME personnel (usually managers) as contrasted with indirect offerings such as tax credits or measures that only provide financial subsidies. Technology and innovation advisory services may be associated with other business support services, but their focus on technological and innovation aspects distinguishes them from general business assistance services that focus on business planning, finance, accounting, and marketing. Much has been written on small business growth and assistance through general business support services and entrepreneurship development programmes (for example, see Storey, 1994; Boter and Lundström, 2005; Cumming and Fischer, 2012). However, the literature on technology and innovation advisory services is smaller and more focused.

This review focuses on developed economies where well-established technology and innovation advisory services are found. Technology and innovation advisory services are usually centred on manufacturing, not only because of the prominent use of technology in this sector, but also because manufacturing is a critical "traded sector" and once lost, is not easily replaced. As developed economies, including the United Kingdom (UK) and the United States (US), seek to rebalance their economies following crises in financial services sectors in recent years and ever growing global competitive pressure, many initiatives are being considered or implemented to rebuild manufacturing and to stimulate manufacturing firms to be more innovative (Dyson, 2010; Heseltine, 2012). This re-emphasis places ever more attention on technology and innovation advisory services to manufacturers as part of broader manufacturing, industry and innovation strategies. It also reinforces the leveraging role of technology and innovation advisory services in supporting manufacturing SMEs to access and take advantage of advanced manufacturing and innovation centres, such as the Catapult Centres now established in the UK or the National Network of Manufacturing Institutes which is proposed in the US. At the same time, opportunities are presented to expand the reach of technology and innovation advisory services beyond their conventional client groups with the increasing importance of technology in services sectors, including services linked with manufacturing.

Across all industries and sectors, SMEs usually comprise the majority of all enterprises irrespective of country. Technology and innovation advisory services typically focus on SMEs because these enterprises, on the whole, face greater barriers than larger firms in adopting improved or new technologies and methods. This is evidenced in the lower productivity of smaller firms relative to larger enterprises. For example, data from the 2007 US Census of Manufacturers indicates that SMEs (with fewer than 500 employees) have 58% of the labour productivity of their larger manufacturing counterparts, where labour productivity is measured as value-added per employee.² However, the importance of supply chains as well as the role that branch facilities of multinationals and original equipment manufacturers play in regional development often leads programmes to provide services to some larger companies as well. In addition, while these types of programmes are typically structured to assist existing companies, some services – Canada's Industrial Research Assistance Program (IRAP) being a case in point – explicitly work with start-ups in manufacturing and related technology-intensive sectors.

This report is one of a series produced as part of the Compendium of Evidence on the Effectiveness of Innovation Policy Intervention, with project sponsorship from NESTA.³ The report begins by providing an overview of technology and innovation advisory services including the major types of approaches and services adopted and the rationale for public support. This is followed by a discussion of leading examples of technology advisory and innovation services in the UK and in other developed economies. The report then collates and examines available literature, including formal evaluation studies, selected academic papers, and accessible grey literature, to assess the evidence as to impact. We review the methods and metrics used and organise the evidence according to the nature of the impacts reported. This assessment highlights studies that examine dedicated technology and innovation advisory services, although it also considers literature on other programmes where such services are provided as part of larger technology centres or innovation support initiatives. Finally, the report highlights lessons and insights for policymakers. These conclusions will likely also be useful for others, including programme managers, business organisations, and development agencies. The diversity of national innovation systems, institutional and industrial factors, related programmes and policies, and other specific contextual influences should be kept in mind when assessing the transferability of the lessons and insights highlighted in the study.

2 Technology and Innovation Advisory Services: Overview and Justification

This section provides an overview of technology and innovation advisory services. There is a discussion of the services that are generally offered, what kinds of companies these services are targeted to, and how they are delivered. This is followed by a discussion of the challenges and market failures typically faced by SMEs in upgrading their technological and innovation capabilities and the rationales for public support of technology and innovation advisory services to address these challenges and failures.

2.1 What are Technology and Innovation Advisory Services?

The core component of a technology and innovation advisory service is guidance provided directly to firms by experts to foster improved business performance through technological modernisation and innovation in products, services and methods. One of the key characteristics inherent in this definition is that the service is provided directly to the enterprise, often at the company's facility, rather than the firm having to travel to a central location. Classic technology and innovation advisory services, such as state industrial extension services in the US, are often based around a network of offices distributed throughout a geographic region rather than a single central technology centre. Of course, technology centres can have expert advisors who will visit companies in the field, as in Japan's prefectural public technology centres. Additionally, to comprehensively address the many facets associated with technology upgrading and innovation, there are vital roles for group services such as supply chain improvement initiatives, quality circles, and workshops, for demonstration facilities and online information, and for access to training, finance, export assistance and other complementary services.

A related characteristic is the use of highly capable professionals to provide technology and innovation services to firms. A typical technology and innovation service professional has significant industry experience, often (but not always) with engineering or other technological training, and broad knowledge of business and financial processes. While general advice or referrals can be provided indirectly, on site clients visits allow the professional advisor to see first hand the operations of the company, engage directly with key managerial and operational employees, diagnose issues, and customise any recommendations. Direct interaction between the advisory professional and company is also essential because of the importance of tacit and experiential knowledge in modernization and upgrading – knowledge which cannot be acquired or transferred solely from documented or online sources. Such on-on-one relationships (when developed professionally and over time) build up trust between the company and the advisor and can increase the confidence of business managers, particularly in smaller companies, in making and implementing new technology or innovation decisions. In some cases, technology and innovation advisory services use a "brokering" approach where, after diagnosis, the advisory specialist refers the company to a third-party external provider who delivers the necessary services. Nonetheless, the advisory specialist will still use experiential knowledge to identify the appropriate provider and it is a good practice to "stay in the loop" to oversee and manage the quality of the engagement between the company and the external provider.

The services provided through technology and innovation advisory services vary by programme, the industrial context, and the needs and capabilities of clients. Generally, however, the range of services offered combines strategic and practical forms of assistance. A common set of technology and innovation advisory services includes:

• Initial assessment at plant and company levels.
• Lean operations, including lean production and lean office, use of methods such as “5 S”, value stream mapping, pull systems, quick changeover, cellular manufacturing, waste minimization, preventive maintenance, and just-in-time systems.
• Quality improvement, including certification to standards such as ISO 9000, TS 16949, ISO 13485, ISO/IEC 17025) and the use of statistical process control and methods such as root cause and six sigma
• Product development and marketing, e.g., lean product development, ideation, market research, business development assistance, lead generation, technology validation
• General business analysis, strategic planning successive planning executive coaching
• Financial analysis, activity-based costing
• Energy management
• Environmental, health, and safety (e.g., food safety, ergonomics, waste management, compliance, pollution prevention, carbon footprinting)
• Information technology, including computer software/packages for accounting, planning, order entry, scheduling, inventory, such as MRP II, ERP
• Technology, supplier and vendor sourcing and matching
• Referral to qualified private consultants, technology centres, or other relevant assistance programmes

These services typically draw on, or make referrals, to related services to assist with design, testing, or prototyping, intellectual property support, basic or technical skills training, accounting, information technology, networking, exporting, and marketing.

In considering the scope of technology and innovation advisory services, it is important to note that the emphasis is not on technology transfer from labs to firms (although some projects may contain such elements), but rather about systemic measures to improve firms' technological and innovation capabilities and ultimately their business performance. Technology and innovation advisory services do not necessarily impart advanced technology in isolation, but are frequently involved in diagnosing and facilitating pragmatic improvements in operations and practices, usually with commercially-proven technologies. Although employment impact of service is a commonly asked question, it should be recognized that technology and innovation advisory service is not a short-term jobs programme. Results will take time to materialize and require sustained efforts and, indeed, some direct jobs may be lost as productivity increased. Technology and innovation services are frequently embedded in processes that are driven by industry needs and market opportunities and which leverage existing resources. Such services can have significant impacts on companies served, but from an economy-wide perspective their effects are typically incremental.

Technology and innovation advisory services link with, but can also be usefully distinguished from other business support services including those which provide general non-technological business assistance as well as technology-oriented initiatives such as those which seek to transfer novel technologies from universities to industries. Broad distinctions can also be drawn about the business targets of various business and technology support services. A schematic positioning of technology and innovation advisory services compared with selected other forms of business and technological assistance is presented in Figure 1. The Y-axis denotes the technological focus of the firm: none (or very limited), mature off-the-shelf, or advanced. The X-axis differentiates where a firm (SME) is positioned in terms of its life cycle: start-up or spinout enterprise, an established firm with existing products, services, and markets, or a firm that is facing immediate business crisis. Within these matrices, services are situated based on the extent to which they primarily serve certain types of clients (from a life cycle standpoint) and provide assistance at different levels of technological sophistication. Using US examples, the top left hand area shows R&D oriented services that are often associated with advanced technology start-ups: incubators, technology licensing offices, and the Small Business Innovation Research (SBIR) programme (which does serve mature companies as well as start-ups but provides funding for new-to-the-world offerings). Services represented at the bottom of the curve typically have little technological orientation, rather they are focused on basic business planning and marketing. Technology and innovation advisory services fit between these poles in that they are targeted to existing companies and they provides assistance with technologies that are new-to-the-firm rather than those that are new-to-the-world.

2.2 Rationales for Technology and Innovation Advisory Services

SMEs typically face a series of challenges that can hinder their efforts to upgrade and adopt modern technologies and implement innovations in products, processes and services (NAPA, 2003; National Academies, 2013). One of the most prevalent is lack of capital. SMEs often find it difficult to access financial resources to invest in technological upgrading and innovation. In some cases, difficulties in accessing financial resources are compounded by weaknesses in the ability to cost-justify new technologies or the benefit-risk relationships involved in innovation. Human capital challenges are also important. SMEs typically have limited internal know-how to manage the innovation process. SMEs typically under-invest in training and skills, and they tend to have a smaller pool of qualified personnel for selecting, operating, and integrating new technology into the firm's flow of production. In addition, few employees have the level of absorptive capacity to learn from, manage, transfer knowledge associated with the technology throughout the firm, and leverage the technology to produce innovative outcomes.

SMEs can also be limited in their ability to gain knowledge for upgrading from external sources. In part, this reflects the day-to-day pressures of managing immediate tasks within the business (including marketing, production, inventory, delivery, and payroll), restricting time and resources available for developing innovation strategies and engaging with external sources of expertise. These issues can be exacerbated by weaknesses in business relationships, value chains, and networks. The external environment also challenges SME's ability to identify trends and develop an approach to capitalize on them. Universities are too complicated to deal with, while private consultants are often viewed as expensive.

Such challenges in acquiring expertise, information, skill, and financial resources often lead SMEs to under-invest in technology and innovation, below what might be an economically optimal level. This underlies the rationale for publicly-sponsored intervention. Technology and innovation advisory services are usually justified through their role in addressing some of these market failures (National Academies, 2013). Market failures can exist on both the demand and supply sides. Market failures on the demand-side include the lack of information, expertise and skills, training, resources, strategy, and confidence among SMEs to adopt new technologies and techniques. Market failures on the supply side involve the costs for vendors, customers, consultants, and other business assistance sources to reach and service SMEs. System-level failures concern the limited quantity and sometimes poor quality of education and training services available to SMEs, lack of access for SMEs to universities and national laboratories and technology centres focused on research missions and high-end technologies, and existing government programs focused on economic development or generic non-technological services. The lack of coordination across different levels of local, regional, and national governments, and among different agencies and branches from the perspective of SME manufacturers is a further system-level failure.

3 Types of Programmes

While many publicly sponsored programmes and organizations offer variations on technology and innovation advisory services, three major institutional categories of publicly sponsored programmes can be identified (Table 2).

In the first group are programmes that concentrate almost exclusively on delivering technology innovation and advisory services through a dedicated network of field agents. Examples include the Manufacturing Extension Partnership (MEP) in the United States and the Manufacturing Assistance Service (MAS) in England. These programs tend to serve a diverse range of companies, are flexible and decentralized, and providing services with more of an application orientation than a research orientation. The US Manufacturing Extension Partnership serves a broad base of manufacturing SMEs through a decentralized service delivery system that employs industrially experienced extension agents and qualified third-party consultants. A range of organizations (some university-based, others private non-profit or government) run state and local MEP centres and offices, with significant involvement of state government and other partners. Federal funds for the MEP are matched by state government contributions and fee income. The Manufacturing Assistance Service (MAS) provides assistance to SMEs in England through a network of industrially experienced expert advisors. Funding is provided by the national government, with the programme operated under contract to a consortium of private and non-profit organizations.

The second type of programme offers a range of technology-oriented business services, which may include technology transfer, venture start-up assistance, seed funding, and technology and innovation advice. An example is Canada's Industrial Research Assistance Program (IRAP), which serves a range of SMEs in manufacturing and other sectors, including start-up and established firms. Organized by the National Research Council of Canada, IRAP operates a decentralized network of offices, industrial technology advisors, and third party partners. IRAP offers funding directly to companies for investments in technologies to facilitate innovation.

In the third group are applied technology centres that undertake research and technology projects with industry and which may also provide associated technology and innovation advice and consultancy. For example, Germany's Fraunhofer Society is comprised of self-managed contract research institutes each with particular specialisations. Services of Fraunhofer Institutes include applied contract research, collaborative projects with companies, and expert consultancy. Japan's Public Industrial Technology Research Institutes (Kohsetsushi centres) are administered by prefectural governments under the Ministry of Economy, Trade and Industry, and undertake applied R&D, testing, and technical assistance services for SMEs.

In addition to these three major organizational types, companies can also be served by other types of organizations who provide technology and advisory services as an adjunct to their primary missions. Universities may have innovation or business support services which can provide advice to small firms. For example, in the UK, Sheffield Hallam University offers research and consultancy services provided by academics and other researchers to businesses in materials and engineering, digital technologies, bioscience, management and other areas, with sponsorship in part from European Regional Development Funds.⁵ Similarly, in the US, a group of technical specialists in the Pennsylvania Technical Assistance Program (PennTAP) at Pennsylvania State University serve SMEs by providing information, technical assistance, or making referrals. PennTAP's expertise includes advanced information technology, energy, environmental health and safety, and innovation services.⁶ Community, technical, and further education colleges offer technical training or apprenticeship programs that can assist SMEs, and can make equipment or facilities available including for training, prototyping, or machining. Some community colleges in the US offer advice and assistance on technology transfer, intellectual property, and commercialisation in their region, as in the case of the Office of Community Technology Transfer at Lorain County Community College, Northern Ohio.⁷ Trade associations typically focus on advancing their sector's interests with policymakers and other external organizations, but may also offer technical assistance, training, and networking to member companies.

The provisions of technology advisory and innovation services may also be facilitated by a firm's participation in other programmes or initiatives. In Japan, for instance, a broad array of services are offered to SMEs, including access to information, business and machinery credit, insurance and loans, assistance with tax credits and R&D subsidies, management training, support for new business creation and innovation, technology transfer and modernisation, internationalisation, mutual insurance, assistance with succession, mergers, and the avoidance of bankruptcy. In aggregate, there are thousands of (publicly subsidized) public, private, and non-profit organizations involved in the provision of these services, including regional and prefectural business support centres, public technology centres, chambers of commerce, and small business associations and societies (Shapira et al., 2011). Different organisations tend to focus on particular types of firms, services, and local areas, and no organisation provides all services at once. In accessing one programme or service, a company may also receive, or be referred to, technology advisory and innovation services (or, indeed, be requested to undertake a review performed by such services so as to receive a loan or another service). The small business support systems found in the UK, US, and other developed countries are similarly multi-organisational and complex, where individual firms may receive, or be referred to, technology advisory and innovation services in conjunction with a mix of other services. At the same time, notwithstanding periodic policy efforts to rationalise or join-up business support services, there are often issues of poor coordination, administrative pre-requisites, lack of business awareness of the service mix, and small business disinclination to engage with external service providers. In the UK and other countries, vouchers for business support and innovation have been offered by governments (DG ENTR-Unit D2, 2009). In such schemes, a small business can receive a publicly subsidised credit that can be applied to procure services, including technology and innovation services, from universities, technology centres, universities, and business support organisations. Vouchers seek to enhance small business demand and inclination to engage with external service providers, as well as encouraging those service providers to interact with small firms. Vouchers are generally small in value (a few thousand pounds in the UK, usually with a business matching requirement), and hence play a role in "introducing" SMEs to service providers (Golding, 2012). The voucher is then used to cover the costs of an initial service such as a diagnosis, assessment, or a few days of consulting. These introductions may then lead to a more significant, separately funded, technology or innovation advisory project.

4 Anticipated Impacts and Measurement Challenges

In innovation policy, as well as in other areas of public policy, evaluation is facilitated not only by the exposition of goals and anticipated impacts but also by a clear understanding of the programme logic of how inputs are expected to lead to intermediate and end results (Kellogg, 2001). This section considers the logic of intervention and anticipated impacts typically associated with technology and innovation advisory services. This is followed by a discussion of several of the major measurement challenges associated with efforts to evaluate these services.

4.1 Programme Logic Model and Pathway to Impacts

The typical intervention logic for technology advisory and innovation services is illustrated in Table 2. In this model, advice and guidance inputs from the programme and associated actions by the company and third party services, including the implementation of projects, lead to intermediate outputs. These can range from new innovation strategies and investments in process improvements to new product development and enhanced supplier relationships.

In the programme logic model presented in Table 2, intermediate outputs are, in turn, anticipated to lead to outcomes for the business, such as improved workforce productivity, new sales, or cost savings, and to broader outcomes including enhanced contribution to regional and national gross value added and enhanced industrial competitiveness. The programme logic model is simplified since not all possible inputs, actions, outputs or outcomes are illustrated. Additionally, in practice, relationships between the service and the business, and the progression pathway of the business itself as it pursues technological upgrading and innovation, may well be more iterative. Realised impacts will also depend not only on other management, firm and economic influences, but also on the specific service received. For example, if the advisory service provides quality systems services, and the firm pursued implementation, it could be expected that ISO certification might be achieved and that there would be improvements in the quality of the firm's products and services. On the other hand, if the service provided advice on technological upgrading, the outputs achieved could include investment in new equipment and associated training, leading to improved business productivity outcomes. In some cases, advice contributes to management know-how and might lead to the avoidance of investment in a sub-optimal technology and the pursuit of other non-technological strategies to support business growth.

4.2 Measurement Challenges

While the programme logic model for technology and innovation advisory services is relatively straightforward, there are still significant challenges in measurement and monitoring. Challenges are present at the firm, programme and system levels. Small firms have limited capabilities to answer detailed questions about quantitative effects of the programme and there are wide variations in how company management systems define, collect, and report data. For example, while it is standard to collect (although not always to disclose) sales data, small firms may not track the time of employees involved in technology and innovation projects or their spending on research and development.

While outcome effects on business and the broader economy are expected from technology and innovation advisory services, care needs to be taken in how outcome changes are attributed. Multiple other influences, including changes made separately by the company, interactions with other business support or technology programmes, or economic factors, could affect impacts on sales and jobs. Programmes effects can be difficult to separate from general firm performance or from other in-house actions taken. The attribution of programme effects is particularly difficult for outputs that require actions from others such as bank lenders and supply chains. In certain cases, an initial effect could be to reduce the number of direct jobs in the short-run through implementation of efficiency techniques, although this might improve the underlying stability of the company and its longer-term growth prospects. Broader economic and market conditions also effect the extent of outcomes that can be realized from technology and innovation advisory services.

Improvements in productivity are often targeted by technology and innovation advisory services because of the contribution enhanced productivity can make to business competitiveness. Value-added per employee is often used as a proxy for productivity, while net contribution to regional or national gross value added is used in some countries as a measure of programme effectiveness (ONS, 2010). However, small companies often do not track the data needed to accurately calculate value-added per employee, nor that part which is attributable to programme support, leading to approximation or even non-response (Luria, 2011). From a regional or national perspective, one assisted company's gain might be another non-assisted company's loss, particularly if the overall market segment is stagnant and exports have not increased (or imports decreased).

To address these types of confounding effects, efforts to measure the improvements in the performance of assisted firms should ideally incorporate a control element. This addresses how a service has improved the performance of its customers compared with changes in the performance of similar non-assisted firms (overall, by industry, by size group, or by region). In some qualitative evaluation situations, control can be addressed by introducing counter-factual probes, for example, what would have happened if programme services had not been received (see, for example, Cosmos Corporation 1999; SRI and Georgia Tech, 2008). Introducing formal control groups of non-assisted companies does, however, increases complexity and cost in terms of evaluation design, data collection, and measurement. Using already collected secondary data on firms (via other surveys or national statistics) and matching with data on assisted firms can be a way to retrospectively construct control groups (Jarmin, 1999; Ordowich et al., 2012). However, secondary data sources can be outdated, have relatively few relevant fields to match performance, and lack information on what other forms of support may have been provided to assisted and control firms.

Impacts on system-level performance may result from technology and innovation advisory services. These might be in regional or state policies or in ease of access to other specialised providers of technology and business support services. These types of impacts are more difficult to measure from a quantitative viewpoint. Such changes can be picked up in questionnaires, dialogues with firms, and case studies.

There are a series of further issues related to the impact measurement of technology and innovation advisory services. In most, although not all cases, programme managers and sponsoring agencies seek regular, detailed information on programme activities and impacts. This can raise the burden of data collection and reporting on the programmes themselves and also on the companies who are asked to provide information. Some types of firm-level effects lend themselves more easily to quantitative measurement than do others; effects that are short-term and sufficiently distinguishable- such as capital investment—can be easier to assess that those that are intangible and take longer to realize-such as skills upgrading or inter-firm collaboration. Cost savings may be realized immediately whereas sales from new product innovations can require several years beyond a reasonable period of surveying to be achieved. Broader outcomes, such as effects on industrial or national competitiveness are characteristically difficult to measure. Effects on gross regional or national value added appear to be more readily quantifiable, yet notwithstanding the potential problems of reporting accuracy and attribution, most technology and innovation advisory services programmes are relatively small in scale and are likely to have incremental rather than fundamental impacts on the broader economy (National Academies, 2013). Technology and innovation advisory services may have valuable and, in some cases, rather significant effects for particular companies, and generate positive returns on public investment. However, in aggregate these services tend to assist a relatively small portion of the potential business population in any one year, effects may take time to emerge and be hard to measure, and some important impacts may not directly influence short-run economic results (for example, where assistance is related to capabilities or know-how).

5 Evaluations of Technology and Innovation Advisory Services

This section identifies and reviews analyses and evaluations of technology and innovation advisory services, drawing from the available body of published and accessible work. There is a discussion of the broad scope of the literature. This is followed by in-depth review of evaluations of three leading examples of technology and innovation advisory services in Canada, the United Kingdom, and the United States.

5.1 Scope of Literature

There is a broad array of literature that analyses or evaluates the operation and impacts of various types of technology and innovation advisory services and centres. The literature includes studies and reviews of the major institutional categories of publicly sponsored programmes (as discussed in Section 3, Types of Programmes) as well as other programmes which offer aspects of technology and innovation advice.

Much of the available literature focuses on operational aspects and the identification of best practices of programmes (see, for example, Bellini, 2003; Shapira et al., 2011). These studies are usually based on assessments of existing secondary literature and may include findings from case study visits or interviews with programme managers. Often these studies are undertaken in an international comparative context. For example, Ezell and Atkinson's (2011) review of programmes to support innovation in manufacturing SMEs raises examples from more than a dozen countries, including Canada, Germany, Japan, South Korea, Spain, the United Kingdom and the United States. They highlight as global best practices the importance of targeting services in relationship to the current capabilities of the majority of a nation's SMEs, identifying gaps in processes, technology and innovation capabilities, and offering services that will take firms to the next level of capability. The National Academies (2013) review examined best practices from several applied research and technology programmes which work with SMEs to support innovation. The publicly sponsored programmes examined included Fraunhofer Institutes in Germany, Carnot Centres in France, the Industrial Technology Research Institute in Taiwan, Catapult centres in the United Kingdom, and the Industrial Research Assistance Program in Canada. The first four of these programs are examples of applied technology centres, while the Canadian programme is technology-oriented business service (see Table 2). All programs offer an element of technology and innovation advice to SMEs. The review highlighted 12 best practice characteristics. These practices included substantial and sustained funding over a long-term horizon, a high degree of autonomy in operations and in working with the private sector, links to local clusters, support for training, business expertise, fostering collaboration and organizational networking, and regular centre reviews by independent reviewers.

There is also literature which considers the broad orientation of policies and programmes to upgrade SMEs and stimulate innovation among these firms and in the sectors and regions where SMEs are clustered. An ongoing theme is the orientation to nonmanufacturing as well as manufacturing sectors. While manufacturing remains a core target for technology and innovation services, there is growing attention to supporting innovation in services (Mas-Verdu 2007; Aboal and Garda 2012). There is also debate in the literature about whether services should have a technology, sectoral, or regional focus. In technology centre alliances in Spain such as FEDIT (Fedit es la Federación Española de Centros Tecnológico⁸) or Tecnalia (in the Basque Country⁹), individual technology centres tend to pursue a sectoral focus because they depend on company memberships or close relationships with firms in sectorally-oriented local clusters. In Japan, there is typically at least one Kohsetushi technology centre in each prefecture which has a general orientation, while other centres are focused on particular sectors or technologies (Shapira, 2008). Fraunhofer Institutes follow a technological orientation, reflecting the highly-specialised foundation of firms in Germany (National Academies, 2013). These approaches are not necessarily mutually exclusive, and other programmes within a country may adopt contrasting approaches. A further theme in the literature considers the institutional and policy mix within which technology and advisory services can be effective (see, for example, Molina-Morales and Mas-Verdu, 2008). Efforts to support innovation in SMEs also have to address financial, regulatory, management, labour market, training, and infrastructural issues. There are variations among countries in the allocation of responsibilities for different elements of these policies, including between national and regional jurisdictions and among various agencies. For example, the federal policy framework in the United States supports labour market flexibility but has less emphasis on with vocational training in SMEs (reflecting structural weaknesses in non-college training and because education is a state and local role), while cost and regulatory issues are often highlighted. In Germany, there is a well-established national system for vocational training which involves SMEs and is viewed as beneficial for the take up of technology particularly by Germany's leading middle-sized companies (Holz, 2013). In Japan, innovation policymaking tends to be centralized, with structural constraints on the operation of labour markets and the flexibility of local initiatives (Shapira et al., 2011).

The literature on the review of the operation of technology and innovation advisory services includes studies of programs in Europe, North America, Japan, and multiple other countries. In some cases, formal evaluation studies have been undertaken which focus on the effects of these programs on business, innovation, and economic outcomes. However, there is a much smaller of programmes which has been subjected not only to formal impact evaluation but also to repeated evaluation measurement by a range of evaluation performers, including independent evaluators. The next part of this study focuses on three technology and innovation advisory services which have been subject to formal, repeated and multi-performer impact evaluation.

5.2 In-depth Review of Evaluations of MEP, MAS, IRAP and related programmes

This section examines the results of a multiple evaluations from three focused technology and innovation advisory services: the Manufacturing Extension Partnership (MEP) in the United States, the Industrial Research and Assistance Program (IRAP) in Canada, and the Manufacturing Advisory Service (MAS) in England, and similar or related advisory support services in the United Kingdom.¹⁰ The identified evaluations in this review occur from early assessments of the programme as it has evolved, starting from the early 1990s, through to current evaluations. Some of these evaluations are oriented to firm-level effects, some to program-level effects, and a smaller set to system-level effects.

Organisational, service and funding profiles of the MEP, IRAP, and MAS are summarised in Section 3 (Types of Programmes) and in Table 1. The MEP and MAS are examples of technology and innovation advisory services that use dedicated field staff (also known as field agents) to provide a range of standard and customised services to SMEs. Both focus on manufacturing SMEs. IRAP is an example of a technology oriented business service that offers funding for research and innovation projects and start-up assistance but which also provides field services to SMEs through a field agent network. Each of these programmes has been the focus of multiple and recurrent formal evaluations by a range of authors and organizations.¹¹ While the US MEP has been the target of the greatest number of studies, all three programs have undergone a series of formal and repeated evaluations.¹²

This review focuses on evaluations related to the MEP, IRAP, and MAS that have implications for understanding pathways to impact and programme outcomes. We do not include annual programme reports or analyses of customer and financial activity levels, customer satisfaction ratings, or programme-reported economic impacts, although all these data types are frequently incorporated into programme evaluations. The emphasis in this section is on studies with a credible evaluation methodology and a learning orientation, rather than monitoring reports or reports that primarily are focused on self-justifying a programme.

The set of evaluations upon which the discussion in this section draws is summarised in Appendix 1. These studies use a variety of methods. Not only is this range apparent when looking across the studies, but also when looking within these studies: many of these studies rely on mixed methods approaches to capture qualitative as well as quantitative results. These evaluative methods use data and metrics derived from administrative databases of field specialist activities and customer interactions, surveys sent to clients after service delivery, comparative studies (i.e., customer progress and longitudinal benchmarking which use progress reports over multiple periods and non-customer controls often obtained from surveys or administrative datasets), simulations, in-depth case studies to examine linkages between the service provide and firm operations and performance, and institutional assessments of programme and system level operations and impacts. These evaluations appear in a diversity of media, including government oversight reports, expert or advisory panel reviews, scholarly journal articles, books, and working papers.

The evaluations reflect conditions and developments in the national innovation systems within which they are located. The US MEP has been the subject of an extensive set of diverse evaluations, some sponsored by the programme or its operating agency (the National Institute of Standards and Technology), with other studies undertaken by government oversight agencies or academics. In addition to evaluations at the federal level, there are a number of evaluations of state manufacturing and industrial extension programmes. Canada's IRAP is evaluated through formal external evaluations on a five year cycle although academic studies of the programme have also been conducted. The MAS programme has evolved over time, being re-launched in 2002 with a focus on productivity improvement and technical and strategic advice to SMEs. From 2002-2011, MAS was organised on a regional basis, but was reorganized in 2012 as a national service for England operated by a private consortium with government sponsorship. In recent years, MAS has adopted a greater orientation towards supporting innovation among client SMEs. Available MAS evaluations cover the period before the latest (2012) reorganization. MAS core services, including its five-level service model, have continued through to the present.¹³

When reviewing the evaluations to ascertain insights, two caveats should be considered. One caveat for drawing insights across these evaluations is that programs differ by country and by region, province, or state within a country in terms of who the customer is, what types of assistance are given, what investments and resources are used, and what kinds of outcomes are produced. Another caveat is that despite efforts in a number of studies to control for unobserved effects with comparison groups of similar unassisted firms, larger scale economic and market forces can override the ability to distinguish impacts from these relatively small scale programs on, for example, a nation's manufacturing sector.

Within these limitations, these evaluations suggest a series of findings related to the impacts of technology and innovation advisory services. Findings have been produced which relate to the categories of intermediate business outputs, business outcomes, and broader outcomes delineated in the programme intervention model presented in Section 4.1 (Programme Logic Model and Pathway to Impacts). Additionally, evaluations have developed findings which relate to programme and system levels. Findings from these five categories are considered in the following sections.

5.2.1 Intermediate Business Outputs

A critical element of the logic model concerns intermediate outputs. Intermediate outputs connect assistance from technology and innovation advisory services to business performance and broader outcomes. Assistance by an expert advisor or field agent usually does not directly result in increased sales or business performance, rather it usually leads to intermediate capacities to learn, make changes (such as adoption of new practices), and these practices can, in turn, produce business and broader outcomes. Intermediate outputs can be divided into three parts: (1) the capacity for change in practice, (2) changes in practice, and (3) changed firm capability (Cosmos Corporation 1997).

The capacity for change typically involves new skills or learning abilities that stem from technology and innovation advisory services. These capacities can include increased knowledge, skills, readiness to make changes, and enterprise wide capabilities to do something differently. Changes in practice involve distinctive use of materials, alternative plant layouts, changes in manufacturing processes, increased or new capital expenditures, adoption of new-to-firm technologies, new management approaches, training related to new activities and tasks, new collaborations (e.g., with other firms, universities), and/or substantial reorganization of personnel. As a result of these changes, the firm may have new capabilities, for example, new qualifications, testing milestones passed by a product, or new certifications in quality, energy, or environmental areas (Cosmos Corporation 1997).

Intermediate outputs typically are of a qualitative nature or have quantitative dimensions not readily captured in administrative reporting and thus are typically measured through client surveys and case studies. In addition to productivity enhancements, MAS participants reported improvements in manufacturing practices, waste reduction, inventory management and logistics, and investment in capital equipment, as well as organisational benefits including better networking capabilities and supply chain linkages. The DTZ Consultancy evaluation of MAS reported from their survey of MAS clients from 2002 to 2005 that 47% of clients had improved productive use of equipment, 37% had increased investment in skills, 35% had improved just-in-time manufacturing practices, 33% had better stock turns/stock holding/delivery, 30% had increased space utilization, and 20% had increased investment in capital equipment (DTZ 2007). Four-fifths of MAS assisted firms indicated that benefits from services would be felt for at least five years.

The MEP client survey asks questions about whether services lead to an increase in investment in workforce practices and employee skills as well as in plant and equipment and information systems and software; these investments totalled $2.5 billion in 2012. The most recent IRAP review by Goss Gilroy (2012) included survey questions asking clients about intermediate technical and business capacity. Results reported that the percentage of IRAP clients agreeing that the program helped increase the firm's business skills and knowledge was 70%, helped increase its scientific and technical knowledge was 82%, and that it led to enhanced technical knowledge or capabilities (90%), enhanced ability to perform R&D (62%), and enhanced business knowledge/capabilities (68%).

5.2.2 Business Outcomes

Firm outcomes have been found to vary by service mix and intensity. Moderate but intense customised services in the product development and marketing area lead to bigger benefits, while routine services oriented around quality and process improvement leads to modest firm effects (Oldsman and Heye 1998, Thompson 1998, Youtie and Shapira 1997). This finding was reinforced in a set of case studies of a pilot product development/marketing service (SRI and Georgia Tech 2008), which found notable levels of firm growth tied to the service. UK results present more of a mixed view. DTZ (2007) reported that benefits from the MAS were higher for clients receiving intensive services. Mole et al. (2008) also found that intensive Business Links assistance was associated with employment growth. At the office level, in contrast, Mole et al. (2011) found that assistance from Business Links Offices using intensive "pipeline forcing" models was negatively associated with productivity, as measured by sales per employee, and was not significantly associated with growth. The authors did acknowledge that this intervention model was particular to certain UK regions and that confounding factors could have influenced the results (although some factors were controlled in the model).

Results do not affect all client firms in the same way, as seen in several studies. Youtie and Shapira (1998) reported that a few firms have very high impacts as a result of assistance from the MEP. Luria (1997), who set up a performance benchmarking database of manufacturers across the US, found that some manufacturers compete on product development, some on lean manufacturing, and some on new technology. His data suggested that working with firms using low price strategies produced growth but not necessarily productivity gains, while working with firms that use product oriented strategies produced gains in wages, profitability, and productivity. Mole et al. (2008) suggested that Business Links specialists' selection of certain young firms to work with was a strength. The 2012 IRAP review (Goss Gilroy 2012) also recommended that attention be given to client selection and flexibility in being able to work with clients most likely to have a successful outcome.

Various studies report differences in the ways in which firms adopt new technologies and techniques. Shapira and Rephann (1996) found that firms assisted by the industrial extension programme in West Virginia were more likely to adopt a single technology, but not an aggregation of technologies. A comparison of Community Innovation Survey (CIS) question results by Roper et al. (2010) found that manufacturers in a US region were more likely to adopt an innovative product or process than those in two UK regions, but the UK firms were more likely to have bundled (i.e., complementary) innovations. Helper and Wial (2010) have argued that the MEP should expand its product and market development services and better coordinate with other programmes so as to provide more strategic services to firms.

Positive results have been reported from the use of financial incentives and linked project funding in association with technology and innovation advisory services. BiGGAR Economics (2010) observed that the Scottish Innovation Voucher Scheme led to new R&D partnerships, as well as quantitative outcomes. Funding services have attracted particular attention. PACEC (2009) found that R&D grants to UK SMEs filled an important funding gap, reduced risks associated with the project, led to greater technology use, and for a smaller set of participants, increased productivity. Niosi (2006) found that Canadian university spinoffs receiving IRAP funding had higher growth than those with venture backing. The IRAP 2002 evaluation (NRC 2002) underscored the importance of funding in service mix in that services tied to the receipt of funds were more highly valued than those involving advice alone. A case study of the Partnership for Regional Innovation Services to Manufacturers in Ohio (reported in National Academies, 2013, pp. 126-127) reports positive results from integrating multiple services, including centre support, finance, training, and networking, in intensively assisting smaller companies to develop innovative new products, markets and services.

Evaluations of technology and innovation advisory services have sought to discern whether programme participation has an impact on firm performance and business outcomes relative to non-participants. In the case of the MEP, the effect of programme participation on labour productivity has been found to be positive yet incremental. Compared with non-clients, MEP clients had 3.4%-16% greater growth in labour productivity over a 5-year period in the late 1980s and early 1990s, according to a well-regarded longitudinal comparison group study by Jarmin (1999) of nine centres during the formative years of the MEP. An update to this study, using a comparable control group method, found mixed results for MEP's overall net productivity impacts on assisted firms for the period 1997 to 2002, but did find that MEP services were associated with significant productivity improvements for smaller firms and certain types of services (Ordowich et al,. 2012). An econometric analysis investigating whether MAS assisted firms performed differently than non-assisted firms proved to be inconclusive (DTZ, 2007). This result was attributed not only to measurement and statistical issues but also to the likelihood that the analysis occurred before the full range of results from MAS services had materialised.

5.2.3 Broader Economic Outcomes

The DTZ (2007) evaluation of MAS services over the 2002-2005 period found that positive economic returns were generated. For Level 4 consultancy services, £1.40 – £1.80 of economic benefit was received by firms, on average, for each £1 of public funding (with an estimated internal rate of return of 15% – 17% over a five year period). MAS was reported to have contributed £155m of additional Gross Value Added through Level 4 and quantified Level 2 services between 2002-2005. The legislatively-mandated reviews of the Canadian IRAP programme (NRC and Goss Gilroy, 2007; Goss Gilroy 2012) suggested positive economic benefits of more than 10 to 1 relative to public sector costs – albeit based on methods that are expansive and based on multipliers derived from input-output models. An analysis applying this (uncontrolled) method to MEP reporting data finds $20 of sales increases for every dollar invested in the MEP (MEP, 2013).

From an aggregate perspective, economic impact is also a function of programme reach and market penetration. A review of MEP reporting data by Stone & Associates (2010) found that only 10% of manufacturers were "touched” by the programme in 2006-2008 and only 2% were served by substantive services. With only a small share of firms served, and with impacts from those firms at modest levels in aggregate, the MEP's broader impact on the national economy was limited. Stone & Associates recommended that the MEP programme needed to increase its annual clientele base by a factor of four to have a meaningful effect on the US manufacturing sector.

When considering overall aggregate economic outcomes, it should be kept in mind that the level of effort and resources involved in most MAS and MEP advisory interactions and projects is generally small. The available data suggests an average annual public spend of £2.3K ($3.7K) and £6.0K ($9.6K) per client for the MAS (all service levels) and MEP respectively.¹⁴ IRAP allocates more per client, largely because of its ability to directly provide funding in association with technology and innovation advice. Importantly, company resources committed usually match, or more than match, these public resources. DTZ (2007) estimated the average total cost to business of participating in the MAS ranged from £11K - £13K, with business costs for more intensive Level 4 services ranging from £15K - £23K per firm. In a study of industrial extension in the US state of Georgia, it was found that companies invested $3 - $13.3 for every dollar of public expenditure on the programme, with a payback period ranging from six to 22 months (Shapira and Youtie, 1998). The leveraging of business actions over and above what might otherwise is a key factor in driving the "additionality" associated with technology and innovation advisory services. At the same time, a vital ingredient imparted from such services is enhanced knowledge, capability, and networking (Schrank and Whitford, 2009). Where programme intervention provides guidance that leads to a complex set of subsequent strategic changes, new relationships, and follow-on actions, it is difficult to track and attribute overall outcomes in simple economic terms. Central measures of business outcomes can also be misleading, in that while most clients of technology and innovation advisory services report modest results, for some companies there are much larger impacts associated with programme participation which can be under-estimated by short-term measurement as it may take time before full results materialise (Shapira and Youtie, 1998).

5.2.4 Programme-level findings

The relationships between client needs and the missions of technology and innovation advisory services is an important area of evaluative attention. GAO (1991) indicated that the US extension programme was initially set up to transfer laboratory technologies to SMEs, whereas SMEs really need more pragmatic assistance. The MEP programme model was the subject of subsequent evaluations by NAPA (2004) and Stone & Associates (2010), both of which encouraged the MEP to place greater emphasis on product development and technology diffusion. MEP has placed a greater emphasis on innovation and growth services in recent years, although the National Academies (2013) observes that the performance of such services and their providers has not yet received significant attention in terms of targeted evaluation studies.

The conventional model for delivering technology and innovation advisory services relies on one-on-one assistance from expert agents to client companies. An evaluation of the Leading Enterprise and Development (LEAD) programme in the Northwest region of the UK, which used peer-to-peer interaction along with training and coaching, found that participating micro enterprises became less isolated and anticipated positive quantitative outcomes. At least one study, by Kelly (1997), found that one-on-one assistance was not necessarily effective in encouraging advanced technology adoption. Studies have also focused on the performance of other approaches to serving firms, including through networking and group projects. In 1996, the MEP funded USNet as a two-year pilot programme to explore the use of networks for service delivery; this pilot included an extensive evaluation component. Networks were found to yield positive net benefits to company participants, particularly those led by companies, but differences in the capacity of states to deliver services through networks limited their effectiveness (Shapira 1998, Kingsley and Klein 1998). Regeneris (2010) found significant qualitative benefits from UK's Knowledge Transfer Partnerships, including increased innovation capacity and positive spillovers with suppliers and other firms, as well as quantitative benefits in terms of sales, gross value added, and employment growth; the evaluation recommended more and higher impact partnerships and less bureaucratic application processes.

The effects of government funding on the programme have been studied through interviews and simulation. GAO (2011) found that MEP centres were less likely to serve SMEs in rural areas because of the difficulty of getting private sector firm match. MEP and Nexus Associates (1998) used a simulation model to show that two-thirds of the states would withdraw their state contribution if federal funding was removed.

Not every region and centre performs the same in these decentralized programs. Data envelope analysis by Chapman (1998) showed that different centres were at the frontier of different service areas, with no one centre consistently in the lead. NAPA (2004) also found considerable divergence across centres and recommended better sharing of approaches across MEP centres. DTZ (2007) observed that the adoption of different delivery models by MAS regional centres was not a factor in these centres' ability to meet programme targets. SQW (2009) found that although Business Links Offices in the South West met their targets, the "consistency of access" standard was a programme weakness. On the other hand, Schrank and Whitford (2009) viewed centre diversity as an MEP strength, giving it flexibility and experimentation.

Several of these assessments have given guidance on improvements to programme evaluation and performance management systems. The IRAP 2012 programme evaluation, for example, called for greater inclusiveness of all programme elements in the evaluation system (Goss Gilroy 2012). The NAPA review of the MEP recommended that evaluation systems be incorporated at the earliest stages of client engagements to obtain baselines, that the programme incorporate more gradations in performance at the centre level, and that customer satisfaction measures be retained in centre scoring systems (NAPA 2004). These reviews place greater emphasis on rigorous and consistently applied evaluation metrics and systems, yet other studies in this compendium have given value to flexibility and experimentation. IRAP (2012) recommended more "voice of the client" and coordination in performance measurement. How to balance the need for flexibility and experimentation with the need for consistent performance management is an ongoing tension in these studies.

5.2.5 System-level findings

Service coordination is a major system-level topic. Shapira and Youtie (1997) found that the MEP had a positive effect on integrating disparate organizations that serve or could serve SMEs, but that coordination costs were underestimated. Several of the broader programme reviews present findings with more wide-ranging policy implications, including relationships with broader industrial and SME policies. These reviews tend to emphasize the need for technology and innovation advisory services to be better integrated into the larger SME and/or manufacturing policy system rather than focusing on improving programme operation in isolation. NAPA (2004) emphasized the need for greater coordination within the MEP and between the MEP and other national and regional manufacturing programs. Lipsey and Carlaw (1998) used institutional and historical analysis to highlight the positive role of IRAP in the manufacturing service system. The IRAP (2002) evaluation reported gaps in coordination in manufacturing service delivery and recommended greater linkages with collaborators, suppliers, customers, universities and government laboratories.

A significant policy debate about whether these services conflict with or crowd out private sector offerings has occurred particularly in the US. Surveys by Oldsman (1997) indicated that there was little overlap between MEP and private sector services, and that the MEP expanded private sector business opportunities with SMEs. In reviewing the core premise of the MEP, NAPA (2003) found that US SMEs continued to be underserved by private consultancies.

6 Lessons and Conclusions

Taken together, these evaluations highlight the outputs and outcomes associated with technology and innovation advisory services, and their impacts on firms, regions, and national economies.

The reviewed studies generally find that technology and innovation advisory services provide positive benefits for participating firms. The types of benefits achieved include reductions in costs, improved quality, reduced waste and improved environmental performance, higher productivity, and new product development and innovation. Levels of investment involved (by both the public sector and private participating firms) are typically not high. Similarly, the net benefits achieved are often relatively modest for individual projects, although such incremental improvements add up and can make the difference to SME survival or decline. There are examples of significant and fundamental improvements for participating firms, but this is not the mode. Controlled studies generally tend to show lower net effects. Broader benefits to the economy are typically estimated through leveraging and multiplier assumptions about gross value added, although studies recognise that it is difficult to precisely estimate broader effects and spillovers. Studies do not commonly find unintended effects. For example, while concerns have been raised about negative impacts of publicly supported technology advisory services on private consultants, the studies that have examined this do not find such effects. Indeed, private consultants are often engaged by public technology advisory services allowing them to serve and market to SMEs that would not otherwise have engaged their services.

Technology and innovation advisory services are typically funded at relatively modest levels, but there have been tensions in sustaining funding support and particularly in balancing the mix of public and fee income. Efforts to increase the amount of service costs paid by participating firms appear to allow leveraging and reduce pressure on public funds; however, as targets for private fee income is increased, services tend to go "up market" to serve larger firms that can afford to pay higher levels of fees. There is also a tension in technology advisory services in selecting who to serve and how. Some evaluations suggest that more intense and customised services provide improved results, although this will mean serving fewer firms; at the same time, there is often policy pressure to serve as many firms as possible, which means that service assistance is less intense than may be desirable. Programmes often address this challenge by providing initial assistance and then referring companies to qualified private consultants and other public services.

Many studies discuss the preconditions for effective programme performance, including professional and industrially experienced staff, good outreach and branding, effective systems to diagnose enterprise needs, flexibility, and the ability to maintain a long-term perspective. However, challenges are often raised between providing short-term assistance – for instance, by helping a firm better operate its existing technology to save costs – which has immediate and measurable effects and providing support for strategic development, mentoring and innovation which takes much more time and whose effects are harder to measure.

A series of insights for policymakers emerge from this review of technology and innovation advisory services. These include:

• Technology and innovation advisory services provide positive benefits for participating firms: reductions in costs, improved quality, reduced waste and improved environmental performance, higher productivity, and new product development and innovation.
• Net benefits achieved are typically relatively modest, although commensurate with the relatively small amounts of public funding usually invested in technology and innovation advisory services. Nonetheless, such incremental improvements add up and can make the difference to SME survival or decline. Controlled studies generally tend to show lower net effects.
• More intense and customised services provide improved results; but there is often policy pressure to serve as many firms as possible, which means that service assistance is less intense than may be desirable. Programmes typically seek to address this challenge by providing initial assistance and then referring companies to qualified private consultants.
• Efforts to increase the amount of service costs paid by participating firms allow leveraging; however, technology and innovation services then tend to go "up market" to serve larger firms that can afford to pay higher levels of fees.
• Applying best practices in the operation and delivery of technology and innovation services, including attention to outreach, staff capability, long-term engagement, flexibility, organisational stability, and linkage with other services, increases the likelihood of programme effectiveness and impact.

From a programme operations and performance improvement perspective, these findings suggest that attention should be paid to allocation and service strategies, given the limited set of public resources that are available to provide technology and innovation advisory services. In the debate about whether it is better to have broad penetration or fewer intensive services, the weight of evaluation results is on the latter strategy, as most find that intensity of service is associated with positive firm benefits. Targeting these services to firms most apt to be able to benefit from them is suggested in several evaluations. Group processes involving other SMEs were found to be successful and are a useful counterpart to intensive one-on-one engagements. Programmes should also be encouraged to pay attention to complementarities across technologies and services to enable SMEs to take advantage of positive spillovers. Finally, although evaluations tend to emphasize quantitative business performance metrics, the provision of expert technology and innovation advice also leads to significant impacts that are intangible, hard to measure or attribute, or take time to come to fruition.

References

Aboal, D., and Garda, P. (2012). Technological and Non-technological Innovation and Productivity in Services vis a vis Manufacturing in Uruguay. Inter-American Development Bank Discussion paper IDB-DP-264.

Bellini, N. (2003). Business Support Services. Marketing and the Practice of Regional Innovation Policy. Oak Tree Press, Cork, Ireland.

Boter, H., and Lundström, A. (2005) SME perspectives on business support services: The role of company size, industry and location. Journal of Small Business and Enterprise Development, 12, 2, 244-258.

Chapman, R. (1998). Using Data Envelopment Analysis to Assess Performance of Manufacturing Extension Centers. NISTIR 6198, Gaithersburg, MD: National Institute of Standards and Technology.

Cosmos Corporation (1997). MEP Successes: A Case Study Approach. NIST Special Publication 916. Government Printing Office, Washington, DC.

Cosmos Corporation (1999). MEP Successes: Case Study Series –Exemplary Projects Case Studies. US Government Printing Office, Washington, DC.

Cumming, D.J., and Fischer, E. (2012). Publicly funded business advisory services and entrepreneurial outcomes, Research Policy, 41, 2, 467–481

DG ENTR-Unit D2 (2009). Availability and Focus on Innovation Voucher Schemes in European Regions. Enterprise and Industry Directorate General, European Commission, Brussels.

DTZ (2007), Evaluation of the Manufacturing Advisory Service. DTZ Consulting and Research. London.

Dyson (2010). Ingenious Britain: Making the UK the leading high tech exporter in Europe. A report by James Dyson. Commissioned by the Conservative Party, London.

Ezell, S., and Atkinson, R. (2011). International Benchmarking of Countries' Policies and Programs Supporting SME Manufacturers. Information Technology and Innovation Foundation, Washington, DC.

General Accountability Office (2011). NIST Manufacturing Extension Partnership Program Cost Share, GAO-11-437R. Government Printing Office, Washington DC.

General Accounting Office (1991). Technology Transfer: Federal Efforts to Enhance the Competitiveness of Small Manufacturers, GAO/RCED-92-30. Washington, D.C.

Golding, D. (2012). Development of new national innovation voucher scheme. Technology Strategy Board, Swindon, UK. (Presentation, March 29).

Goss Gilroy, Inc. (2012). Evaluation of the NRC Industrial Research Assistance Program (NRC-IRAP). Ottawa, Canada.

Helper, S., and Wial, H. (2010). Strengthening American Manufacturing: A New Federal Approach. Brookings, Washington, DC.

Heseltine (2012). No Stone Unturned in Pursuit of Growth. A report by Lord Heseltine. Department of Business, Innovation and Skills, London.

Holz, M. (2013). Strategies and policies to support the competitiveness of German Mittelstand companies, pp. 147-168. In: eds. Coltorti, F., Resciniti, R., Tunisini, A., and Varaldo. Mid-sized Manufacturing Companies: The New Driver of Italian Competitiveness. Springer, Milan.

Jarmin, R. S. (1999) Evaluating the Impact of Manufacturing Extension on Productivity Growth, Journal of Policy Analysis and Management, 18, 1, 99-119.

Kellogg (2001). Logic Model Development Guide. Using Logic Models to Bring Together Planning, Evaluation & Action. W.K. Kellogg Foundation, Battle Creek, MI.

Kelly, M. (1997). From Mission to Commercial Orientation: Perils and Possibilities for Federal Industrial Technology Policy, Economic Development Quarterly, 11, 4, 313-328.

Kingsley, G., Klein, H. (1998). Interfirm Collaboration as a Modernization Strategy: A Survey of Case Studies, Journal of Technology Transfer, 23, 1, 65-74.

LEAD (2007). Leading Enterprise and Development. Lancaster University Management School. Lancaster.

Lipsey, R.G., Carlaw, K. (1998). A Structuralist Assessment of Technology Policies: Taking Schumpeter Seriously on Policy. Working Paper 25, Simon Fraser University.

Luria, D. (1997). Toward Lean or Rich? What Performance Benchmarking Tells Us About SME Performance, and Some Implications for Extension Center Services and Mission. In P. Shapira and J. Youtie, editors. Manufacturing Modernization: Learning from Evaluation Practices and Results. Evaluation of Regionally-Based S&T Programs Proceedings of Third Workshop on the Evaluation of Industrial Modernization Programs, Atlanta, Georgia: Georgia Institute of Technology, 6-29.

Luria, D. (2011). Evaluating the MEP Evaluation. Presentation at the workshop on Strengthening American Manufacturing: The Role of the Manufacturing Extension Partnership, National Academies, Washington, DC. (November 14).

Manufacturing Extension Partnership (1998). Review of Mission and Operations of Regional Centers of the Manufacturing Extension Partnership, Gaithersburg, MD: National Institute of Standards and Technology.

Manufacturing Extension Partnership (2013). The Manufacturing Extension Partnership: Partnering for Manufacturing Innovation and Growth.

Mas-Verdu, F. (2007). Services and innovation systems: European models of Technology Centres. Service Business, 1, 7-23.

Mole, K.F., Hart, M., Roper, S., and Saal, D.S. (2008). Assessing the effectiveness of business support services in England: evidence from a theory based evaluation. Working Paper 93. Warwick Business School. Centre for Small and Medium Sized Enterprises, Coventry.

Mole, K.F., Hart, M., Roper, S., and Saal, D.S. (2011). Broader or deeper? Exploring the most effective intervention profile for public small business support. Environment and Planning A, 43, 1, 87-105.

Molina-Morales, F., and Mas-Verdu, F. (2008). Intended Ties with Local Institutions as Factors in Innovation: An Application to Spanish Manufacturing Firms. European Planning Studies, 16, 6, 811-827.

National Academies (2013). 21st Century Manufacturing: The Role of the Manufacturing Extension Partnership Program. National Academies Press, Washington DC.

National Academy of Public Administration (2003). The National Institute of Standards and Technology's Manufacturing Extension Partnership Report 1: Re-examining the Core Premise of the MEP Program. Washington DC.

National Academy of Public Administration (2004). The National Institute of Standards and Technology's Manufacturing Extension Partnership Report 2: Alternative Business Models. Washington DC.

National Research Council (2002). Evaluation of NRC's Industrial Research Assistance Program (IRAP). Ottawa, Canada.

National Research Council and Goss Gilroy, Inc. (2007). Impact Evaluation of the NRC Industrial Research Assistance Program (NRC-IRAP). Ottawa, Canada.

Niosi, J (2006). Success Factors in Canadian Academic Spin-Offs. Journal of Technology Transfer 31, 451-457.

Oldsman, E. 1997. Manufacturing extension centers and private consultants: collaboration or competition? Technovation. 17, 5, 237-244.

Oldsman, E., Heye, C. (1998). Waste not, want not—A look at the impact of manufacturing extension centers. Journal of Technology Transfer, 23, 1, 37-41.

ONS (2010). Measuring the economic impact of an intervention or investment. Paper One: Context and rationale. Office of National Statistics, Cardiff, Wales.

Ordowich, C., Cheney, D., Youtie, J., Fernández-Ribas, A., and Shapira, P. (2012). Evaluating the Impact of MEP Services on Establishment Performance: A Preliminary Empirical Investigation. Working Paper, 12-15, Center for Economic Studies, U.S. Census Bureau, Washington, DC.

PACEC (2009). Evaluation of Grant for Research and Development & Smart. Final Report. URN 09/1059. Report prepared for the London Development Agency and the Department for Innovation, Universities and Skills. Public and Corporate Economic Consultants, Cambridge and London.

Regeneris (2010). Knowledge Transfer Partnership: Strategic Review. Report for the Technology Strategy Board. Regeneris Consulting Ltd, Altrincham.

Roper, S., Youtie, J., Fernandez-Ribas, A., and Shapira, P. (2010). Knowledge, Capabilities and Manufacturing Innovation: A USA-Europe Comparison. Regional Studies, 44, 3, 253–279.

Schrank, A., and Whitford, J. (2009). Industrial Policy in the United States: A Neo-Polanyian Interpretation. Politics and Policy, 37, 4, 521-553.

Shapira, P. (1998). The Evaluation of USNet: Overview of Methods, Results, and Implications. Georgia Institute of Technology, Atlanta, GA.

Shapira, P. (2003). US Manufacturing Extension Partnership (MEP): Operation, Evaluation & Lessons. Presentation at the UK Department of Trade and Industry, London, August 4.

Shapira, P. (2008). Putting Innovation in Place: Evolution of Japanese Strategies for Industrial Services, Regional Clusters, and Manufacturing SMEs. Prometheus, 26, 1, 69-87.

Shapira, P., and Rephann, T. (1996). The adoption of new technology in West Virginia: Implications for manufacturing modernization policies, Environment and Planning C: Government and Policy, 14, 431-450.

Shapira, P., and Youtie, J. (1997). Coordinating Manufacturing Extension Services, Journal of Technology Transfer, 22, 1, 5-10.

Shapira, P., and Youtie, J. (1998). Evaluating Industrial Modernization: Methods, Results and Insights from the Georgia Manufacturing Extension Alliance, Journal of Technology Transfer, 23, 1, 17-27.

Shapira, P., Youtie, J., and Kay, L. (2011). Building capabilities for innovation in SMEs: a cross-country comparison of technology extension policies and programs. International Journal of Innovation and Regional Development, 3, 3/4, 254-272.

Shapira, P., Youtie, J., and Roessner, J.D., (1996). Current Practices in the Evaluation of U.S. Industrial Modernization Programs, Research Policy, 25, 2, 1996, 185-214.

SQW (2009). Interim evaluation of Business Link services in the South West. Final Report to SWRDA.

SRI and Georgia Tech (2008). Eureka! Winning Ways: Analysis of Early Client Experiences. SRI, Arlington, VA.

Stone & Associates and the Center for Regional. Economic Competitiveness (2010). Re-examining the Manufacturing Extension Partnership Business Model: Alternatives for Increasing the Program's Impact on US Manufacturing Sector Performance. Gaithersburg, MD.

Storey, D.J. (1994). Understanding the Small Business Sector. Thompson, London.

Thompson, C. (1998). Local Politics, National Policy, and the Taxpayer-Payback of Manufacturing Extension, Journal of Technology Transfer, 23, 1, 37-42.

Youtie J., Shapira, P. (1997). Tracking Customer Progress: A Follow-up Study of Customers of the Georgia Manufacturing Extension Alliance, Journal of Technology Transfer, 22, 2, 43-52.

Youtie J., Shapira, P. (1997). GMEA 1997: Review of Results. Evaluation Working Paper E9701. Georgia Institute of Technology, Atlanta, GA.

Appendix 1. Technology and Innovation Advisory Services – Selected Analytical and Evaluation Studies for Canada, the United Kingdom, and the United States

Footnotes: