ACKNOWLEDGEMENTS
Andrew Whitby's contributions were integral to the design of the business survey, and David Axford and Andrew Croll at Ipsos MORI supported its implementation.
Hetan Shah, Olivia Varley-Winter and Roeland Beerten at the Royal Statistical Society, and Kion Ahadi at Creative Skillset provided valuable feedback throughout the project.
Our ideas about data analytics skills and policy have benefited greatly from discussions with a large number of individuals including Sir Ian Diamond, Samuel Roseveare, Rosalind Lowe, Professor Patrick Wolfe, Professor Sofia Olhede, Paul Driver, Jane Tory, Sandy Grom, Steven Bond, Keith Dugmore, Anjali Samani and Kim Nilsson.
Nesta is an innovation charity with a mission to help people and organisations bring great ideas to life.
We are dedicated to supporting ideas that can help improve all our lives, with activities ranging from early-stage investment to in-depth research and practical programmes.
Nesta is a registered charity in England and Wales with company number 7706036 and charity number 1144091. Registered as a charity in Scotland number SCO42833. Registered office: 1 Plough Place, London, EC4A 1DE.
www.nesta.org.uk
©Nesta 2015
EXECUTIVE SUMMARY
WHAT ARE THE SKILLS IMPLICATIONS OF THE DATA REVOLUTION?
Digitisation has brought with it bigger volumes and varieties of data, which are being generated at faster velocities. The opportunities to create value from this data are quite simply bewildering. We see this manifested in innovative data-driven products and services, improvements in processes, and more informed decision-making across the economy and society.
However, realising this value requires access to the right skills: data engineering skills to develop a robust data infrastructure, data analysis skills to extract valuable insights from data, and business skills to apply them.
What are those skills exactly? How easy are they to find in the labour market? What are the implications of the data explosion for education and skills policy in the UK?
We have partnered with Creative Skillset to address these questions, focusing on the data analysts charged with transforming data into business insight that drives action. This work builds on Nesta's earlier research on the behaviour of Datavores – our name for those businesses which rely on data for commercial decision-making. The work is also part of Seizing The Data Opportunity, the Data Capability Strategy that was published by the Government in 2013, which, together with previously published research by the Tech Partnership and a new report from Universities UK on analytics skills supply, adds up to a substantive body of research on data skills in the UK.
Together this research paints a picture of severe data skills shortages and gaps which should be of deep concern to industry, educators and policymakers. Timely, corrective action is needed across the education and skills system to ensure that UK businesses can fully benefit from the opportunities that data offers. This report is accompanied by a policy briefing jointly developed by Nesta and Universities UK setting out actions to accomplish this.
3. FINDINGS
The first output of Nesta's data skills research, 2014's Model Workers, drew on 45 in-depth interviews to specify the profile of the 'perfect data analyst'. It showed that businesses exploring bigger and messier datasets are looking for analysts - data scientists - with a mix of analytical and computing expertise, domain knowledge, business know-how and communication skills. Our respondents reported that these professionals are very hard to find. The analytical talent coming out of university lack important technical skills; experienced talent is expensive; there is a dearth of individuals with the right mix of skills, and some businesses do not in any case have the internal capability to recruit effectively.
An outcome of this is a widespread perception of a data talent 'crunch' preventing UK businesses from exploiting their data, and in some cases it is even resulting in an offshoring of analytical capability to outside the UK.
In this report, we explore these issues through the experiences of a sample of 404 medium- and large-sized UK businesses in six sectors (Creative Media, Financial Services, Information and Communications Technology (ICT), Manufacturing, Pharmaceuticals and Retail).
Although our sample is not representative of the wider population of UK medium- and large-sized firms, it provides a timely snapshot of the analytical skills situation among companies where data plays an important role in business.
What does it show?
COMPANIES IN OUR SAMPLE ARE USING A VARIETY OF STRATEGIES TO CREATE VALUE FROM THEIR DATA
We have found four distinct 'Data Groups' in our sample, depending on what type of data they work with, and how. Three of them are 'Data-active' in the sense that they are either data-driven (Datavores), work with large volumes of data (Data Builders), or combine data from different streams (Data Mixers). A fourth group, comprising 30 per cent of our sample, work with few, small datasets, and rarely use analysis to make decisions. We refer to them as the 'Dataphobes'.
Analysts in Data-active companies create an impact not just by finding ways to save on costs, but also by seeking out new business opportunities, generating revenues through data products and services, and by improving customer loyalty. By contrast, the Dataphobes appear to have passed up the commercial opportunities emerging form the data revolution.
We find all Data Groups present in all industries, consistent with the idea that data analytics has features of a General Purpose Technology, in that it drives innovation and growth in many different sectors.
DATA-ACTIVE COMPANIES PERFORM BETTER THAN DATAPHOBES
Data-active companies are significantly more likely to self-identify as product/service innovators and process innovators than Dataphobes. For example, Data Builders are over 50 per cent more likely than Dataphobes to say they launch products and services ahead of their competitors.
Our econometric analysis uncovers a positive and significant association between data activity and productivity. This is particularly the case for Datavores and Data Builders, which we show to be, on average, over 10 per cent more productive than Dataphobes, after controlling for other factors.
DATA-ACTIVE COMPANIES ARE RECRUITING LARGE NUMBERS OF ANALYSTS, AND BUILDING A DATA SCIENCE CAPABILITY
They are between two and three times more likely than Dataphobes to have sought to recruit in the previous 12 months. In addition to recruiting experienced talent (from inside and outside their own industries), they are also strongly reliant on talent straight from universities. Around two-thirds of Datavores and Data Builders recruit undergraduates, and more than one-third recruit analysts straight from doctoral programmes.
Data-active companies are also building up their data science capabilities by hiring analysts from numerical disciplines like Mathematics and Statistics at the same time as programmers from Computer Science. They are between two and three times as likely to do this as Dataphobes.
DATA-ACTIVE COMPANIES ARE SUFFERING WORSE SKILLS SHORTAGES
Our findings confirm that there is a crunch in the labour market for analytical talent that is particularly afflicting highly innovative, high-performance Data-active businesses.
Data-active companies are far more likely to report difficulties filling analytical vacancies -for example, as many as two-thirds of Datavores who sought to recruit in the previous 12 months mention problems with at least one vacancy, compared with 40 per cent of Dataphobes. Finding talent with the right domain knowledge, the right mix of skills (e.g. data scientists), experience, and business know-how to apply data in a commercial context is much harder than finding people with the right technical skills (including data manipulation and analysis).
COMPANIES IN OUR SAMPLE ARE UP-SKILLING THEIR WORKFORCE INTENSIVELY, AND INNOVATIVELY
Few companies in our sample think that their analytical workforce is fully equipped with the skills needed to create value inside the business. Consistent with this, they engage in high levels of training activity. Almost 80 per cent of our respondents report providing in-house training to their analysts, and 70 per cent make use of external training.
Substantial numbers of companies (between one- and two-thirds) are involved in online and peer-based learning, reflecting the vibrant ecosystem of hands-on learning, communities of practice and meet-ups in the Big Data/Data Science/tech space. By contrast, universities are the least popular source of training, being used by just one in five companies.
WE SEE SOME SECTORAL DIFFERENCES IN DATA WORK, BUT ALSO CROSSOVER IN DATA APPLICATIONS AND TALENT FLOWS
A company's data sources, analytical methods and areas of application are shaped by the sector in which it operates, something that is reflected in our findings. At the same time, we see many instances of innovative use of data across sectors (e.g. Manufacturers routinely using social media data, and Creative Media companies tapping into Open and Government Data), and of unexpected talent flows across industries. This supports the view that there is great scope for cross-sectoral knowledge sharing and collaboration in this area. Data innovations in one sector are often applicable elsewhere in the economy, and the same is true for data analysis skills.
WHAT'S AT STAKE?
The stakes for the UK economy cannot be understated. If, for example, in our sample of firms all the Dataphobes were to behave like Datavores, our results suggest this would be associated with an overall 3 per cent uplift in productivity. To put this into context, at the macroeconomic level a 3 per cent uplift could, according to OECD statistics, represent roughly one-fifth of the UK's productivity gap with the rest of the G7.
POLICY IMPLICATIONS
One would be hard pressed to find an industry today where knowledge does not play a central role, or where the ability to learn rapidly does not confer upon its owner a significant competitive advantage.
The data explosion brought about by digitisation is multiplying the opportunities to create knowledge and accelerate learning, and consequently, the opportunities to innovate and grow. Nesta's three-year programme of research in this area has quantified the breadth and scale of data opportunities in the UK economy. It has also demonstrated the importance of developing the human resource and organisational capabilities needed to realise these opportunities a challenge that is made significantly more difficult by the presence of the skills shortages reported by the Data-active companies we have surveyed, and the apparent ambivalent attitude to data shown by a significant minority of companies - the Dataphobes we have identified too.
Undoubtedly, data analysis will only gain in importance in UK industry over time and, unless corrective action is taken, we should expect the skills shortages to worsen too.
How can we avoid this gloomy situation, and make sure that UK businesses have access to the analytical talent that they need to thrive in a data-rich world?
To answer this question, we have developed a policy briefing jointly with Universities UK, Analytic Britain, where we set out a major programme of policy recommendations to remove blockages in the pipeline for analytics talent in the UK, spanning schools, universities and the labour market.
1. INTRODUCTION
THE (BIG) DATA EXPLOSION
The digitisation of social life, industry and commerce has resulted in a data explosion. The numbers involved are extraordinary: Facebook processes 930 million photo uploads, six billion 'likes' and 12 billion messages each day. Google receives 40,000 searches every second, and YouTube users upload 100 hours of video to YouTube every minute. According to IBM, 90 per cent of the data that exists today was generated in the previous two years. The arrival of connected devices – the Internet of Things – and wearable technology is only intensifying this process, with important implications for sectors like Manufacturing and Health.
One consequence is the emergence of 'big data'. This term was reportedly first used by technology consultants Gartner in 2001 to refer to datasets that are hard to manage using traditional database and analytical technologies. The reasons for this are that these datasets have:
- High volume: They are too large to be stored in a single computer, or even on a single server. Instead, they need to be spread across an organisation's data infrastructure, using (sometimes multiple) computing clusters.
- High variety: Big datasets often draw on diverse sources and formats, including 'unstructured' ones like video, audio and text, which are very different from highly structured data, like financial information. Unstructured data has to be processed before it can be worked on, and is hard to store using traditional database architectures.
- High velocity: Big data is often generated at high frequency, and is most valuable when analysed and acted upon in 'real time'.
An ever-expanding array of technologies has emerged to help organisations deal with big data. They include Hadoop (a framework for distributed data processing), Cassandra (a big database system) and Hive (a big data warehouse), among many others. These innovations, together with new developments in statistical and analytical methods such as machine learning, have made it easier to create value from data through better-informed decision-making and the creation of data-driven products and services. This is impacting on business performance in a way that we are only just starting to quantify. In Inside the Datavores we showed how the 18 per cent of UK internet economy businesses who use their data more intensively are also, other things being equal, significantly more productive.
THE DEMAND FOR DATA TALENT
Realising these new data opportunities requires the right skills: engineering skills to develop and maintain a reliable infrastructure to collect and process data, analytical skills to extract valuable knowledge from this data, and the business know-how to identify what questions to ask, and what to do about the answers.
As more data becomes available and its value more apparent, these skills have become increasingly sought after. But is there enough data talent to go around?
Available evidence suggests the answer is no. According to McKinsey Global Institute, by 2018 the US will experience a shortfall of 140,000-190,000 ‘deep analytical experts' and 1.5 million data-savvy managers. And recent research in the UK points to a 41 per cent increase in demand for 'big data professionals' in 2013 alone. The same study suggests that 75 per cent of vacancies in big data occupations are hard to fill. The average salary for experienced big data professionals is almost double the UK average (and almost a third higher than the average for IT professionals).
NESTA'S RESEARCH PROGRAMME ON DATA ANALYSIS SKILLS
In this project we have partnered with Creative Skillset to study skills needs in one particular role, namely the data analysts charged with transforming data into business insight that drives action. The disruption this group is experiencing illustrates many of the wider workforce opportunities and challenges created by the data explosion.
Analysis of bigger and more varied data is driving innovation in all industries - be it customised search results and recommendation engines in the case of e-commerce, fraud detection in banking, robotics in manufacturing and transport, or genomics in health, to name but a few. However, harnessing many of these opportunities requires data analysts with a new combination of mathematics, statistics, computer programming skills and industry domain knowledge - they are increasingly referred to as 'data scientists'.
Anecdotal evidence suggests that data scientists are exceptionally hard to find, and there are doubts about whether the UK's education system can rise to the challenge of producing what's needed.
Our research seeks to answer three questions in order to inform policy and practice in this fast-moving field:
- What are the skills of productive data analysts?
- How easy is it to find those skills in the labour market?
- How can organisations manage their analytical talent to create the most value?
In July 2014, we published the first output of this research, Model Workers. This report explored the questions above in 45 qualitative interviews with leading industry experts in the UK. It supported the idea of severe skills shortages in data analysis.
Although some of our interviewees were sceptical about terms like 'big data' (which some perceived as a fad) or 'data scientist' (which some saw as an opportunistic rebadging of previous professions), almost all were exploring the use of novel and 'messy' datasets in their businesses, and seeking - with only mixed success talent with 'data scientist' profiles to do this.
Why were they struggling?
- First, because available analysts lacked the right technical skills and industry experience. On the one hand, analysts straight out of education were not used to working with big data volumes and varieties: training them was costly and risky (not least because analysts would often leave to pursue other opportunities afterwards). On the other hand, experienced talent was expensive.
- Second, data analysts lacked the right multi-disciplinary mix of skills: individuals who combined expertise in computing and analysis with knowledge of business were compared with unicorns, for all their rarity and mythical qualities.
- The third reason for recruitment problems reflected considerations of internal business capacity: companies lacked the knowledge about how best to recruit analysts with advanced data analysis skills and, in some cases, senior management was not fully appreciative of the commercial value of data analysis skills.
ABOUT THIS REPORT
In the present report we set out to determine if the qualitative findings in Model Workers hold for a sample of 404 UK medium and large businesses in six sectors, and to outline what the results imply for government policy.
In addition to surveying these businesses, we extracted longitudinal financial data about them from Bureau Van Dijk's FAME database, and matched it with the survey responses to explore the link between data activities and productivity.
STRUCTURE
The structure of the report is as follows:
Section 2 describes the data we collected, and how.
Section 3 presents our findings. In sub-section 3.1, we describe the data activities and performance profiles of companies in our sample by segmenting them into four 'Data Groups'. In sub-section 3.2, we consider the labour market activities and outcomes for these four groups, and in particular the strategies they deploy to tackle workforce skills shortages and gaps. In sub-section 3.3, we analyse sector-specific differences in further detail.
Section 4 presents the conclusions of our analysis and their policy implications.
2. OUR DATA
SAMPLE DESIGN
Our sample was randomly drawn from the population of UK businesses with more than 50 employees in Bureau Van Dijk's FAME database. We focused on medium- and large-sized business for three reasons:
- Previous research suggests that the levels of analytics adoption among UK SMEs is very low (less than 1 per cent do any big data work).
- We were interested in learning about the organisation and management of data analysts. Some of these issues are less relevant for small organisations with simpler structures.
- We wanted to match survey responses with financial data, but the availability of this data is patchy for companies with less than 50 employees (as they are exempt from reporting that information to Companies House).
We sampled companies from six sectors: Creative Media, Finance, ICT, Manufacturing, Pharmaceuticals and Retail, encompassing a range of domains where the literature indicates that data analysis is having - or could have a substantial impact.
QUESTIONNAIRE CONTENT
Our questionnaire is based on our previous Datavores research looking at skills and data practices, as well as the UK Commission for Employment and Skills Employer Skills survey. It covers the following areas:
- The data value chain: This includes questions on:
- Data collection: what types of data are collected by companies, and how comprehensively.
- Data processing and analysis: The methods that companies use to manage and analyse that data.
- Data application: The areas where data analysis and insight create value inside the business.
- Data talent sources: What are the talent pools and disciplines that companies tap on for their analytical talent.
- Labour market experiences: Recent recruitment experience and the nature of hard-to-fill vacancies and skills shortages.
- Skills strategies, including investments in tools and training to upgrade workforce skills.
- Data talent management practices: The practices that respondents use to create value from their analytical talent, including team make-up, organisation, communication and compensation structures.
FIELDWORK
The survey was carried out over the telephone by two research agencies - IFF Research and Ipsos MORI, between 5 March and 16 June 2014.
Data analysts may be found in any number of different departments within organisations so our interviewers made initial contact with Human Resource (HR) managers. This was based on the reasoning that HR managers were most likely to have information about the situation of data analysts inside the business. Where HR managers felt unable to answer the survey questions, the interviewers asked to be referred to someone else in the business.
Individuals were encouraged to participate in the research with a lottery draw for an iPad Mini.
OUTCOMES
We collected data from 404 businesses. The response rate was 19 per cent and the eligibility rate (companies that, having accepted to participate, passed the screener), 73 per cent.
Around two-thirds of respondents work in HR functions (other functions represented in the sample include Finance, IT and operations). Almost two-thirds of respondents work in senior roles (such as Manager, Director, Managing Director and Chief Executive Officer).
Figure 1 shows the distribution of responses by industrial sector.
FIGURE 1: SECTOR DISTRIBUTION OF SURVEY RESPONDENTS
A bar chart showing the frequency of survey respondents by sector.
Manufacturing has the highest frequency (over 100), followed by Retail, ICT, Creative Media, Financial Services, and Pharmaceuticals (lowest, under 50).
Manufacturing is over-represented, which is as expected given this sector's disproportionate presence in the population of medium- and large-sized firms. We obtain the lowest number of responses (49) in Pharmaceuticals, which is also expected, given the relatively small number of large companies in that sector.
FINANCIALS
As mentioned previously, we matched the survey responses with longitudinal financial data from FAME. This included a variety of indicators such as turnover, employment, costs of goods sold, average remuneration, and different measures of profitability. There were some inconsistencies in data availability caused by flexibility in the financial reporting timeframes required of UK businesses. For 49 per cent of companies, the last available accounts had been filed in 2013, and for 48 per cent, in 2012 (generally on the 31 December).
3. FINDINGS
WHY SEGMENT?
Instead of looking at the skills requirements of a 'typical company', we distinguish between the behaviours and performance of companies making very different uses of data. That there is no ‘one size fits all' approach to data analysis and use was a strong lesson of our Model Workers report, and it is strongly echoed by our survey exercise too.
We segmented the companies in our sample based on their responses to questions about the 'data inputs' they worked with, and their attitude towards data (see Appendix 1 for details of the clustering analysis). We used three metrics for the segmentation:
- Data variety: How many data sources does a company use, and with what intensity?
- Data volume: How big (in volume) are the datasets that the company tends to work with?
- Data drive: To what extent does the company use data and analysis over experience and intuition to make business decisions?
Our analysis revealed four distinct 'Data Groups' in the sample. They are as follows:
- Datavores (16 per cent of the sample): These are companies that make strong use of data and analysis for decision-making.
- Data Builders (22 per cent of the sample): Companies using 'big' datasets requiring dedicated servers or (possibly multiple) clusters for parallel processing.
- Data Mixers (31 per cent of the sample): Companies that collect and combine data from a variety of sources.
- Dataphobes (30 per cent of the sample): Companies that work with small datasets and few data sources, and do not use data or analysis to make decisions.
Figure 2 presents for each Data Group the normalised mean score for each of the three different data activity metrics.
FIGURE 2 DATA GROUP SCORES IN DATA VARIETY, VOLUME AND DRIVE
A bar chart showing the average score for Data Group across Data Variety, Data Volume, and Data-Drive. Each category has four bars representing Datavore, Data Builder, Data Mixer, and Dataphobe. Datavores and Data Builders show high scores across all three metrics, with Datavores highest in Data-Drive. Dataphobes show the lowest scores across all categories.
It illustrates clearly the defining features of the different Data Groups. But it also shows where there are overlaps in data practices: Datavores and Data Builders also combine a variety of data sources, for example; Datavores sometimes work with big data volumes, and Data Builders and Data Mixers also use data to make decisions.
Companies working with few data sources, small data sets and who do not use data or analysis for decision-making - 'Dataphobes' - account for a considerable number of companies in our sample. In the remainder of the report, we use the term 'Data-active' to characterise all other firms excepting these Dataphobes.
HOW DO THE DATA VALUE CHAINS OF OUR FOUR DATA GROUPS COMPARE?
In order to gain a better understanding of the differences between companies in these Data Groups, we explore their behaviour at different stages of the data value chain: specifically, can we detect differences in the data sources they are using, in their approaches to analysis, and in the nature of the impacts of data analysis on their business?
Data collection: Figure 3 shows for each Data Group, and the sample as a whole the extent to which companies regularly use different sources of data. Customer data is by far most regularly used by all companies, followed by other internal business data (e.g. financial and operations data), social media data, third party data (e.g. from credit scoring agencies, audience research etc.) and last, but not least, open and public government data (which is used routinely by 9 per cent of respondents).
Naturally, Data Mixers make more regular use of a variety of data sources compared with other firms. Dataphobes, on the other hand, rarely make use of data sources other than on their customers.
FIGURE 3 REGULAR USE OF DIFFERENT DATA SOURCES
This bar chart shows the percentage of different Data Groups (All Sample, Datavore, Data Builder, Data Mixer, Dataphobe) using various data sources regularly. The data sources are: Customer, Other internal business, Social media, Third party, and Open and Public Government. The X-axis represents 'Percentage using data source regularly' from 0 to 80, and the Y-axis lists the 'Data Source' categories. Each Data Source has five horizontal bars, one for each Data Group, showing their respective percentages.
Data analysis: Figure 4 shows what tools different companies use to manage their data, and what analytical techniques they employ to extract insights from it (see the Glossary for their description).
Unsurprisingly, older technologies and methods are more prevalent than newer tools: Over half of all respondents have SQL, basic statistics and general programming capabilities. Half report that they can do data visualisation.
A substantial minority of companies (around one-third) have skills in more technically complex areas like advanced statistics and machine learning, however. In general, Data Builders are more likely to make use of data processing and analysis tools. They are 30 per cent more likely than the average to have the skills to work with big data (such as skills with the Hadoop parallel data processing framework), and machine learning skills, and 45 per cent more likely to have NoSQL skills to store and manage unstructured data. All of this makes sense, given that they work with high volume datasets requiring a strong IT infrastructure and automated data processing.
We also find a considerable number of Datavores and Data Mixers with advanced statistics and unstructured data analysis skills.
As expected, Dataphobes lag behind in the adoption of all data management and analysis techniques.
FIGURE 4 SKILLS IN DATA MANAGEMENT AND ANALYSIS TECHNIQUES
This bar chart displays the 'Percentage equipped with skill in this area' (0-100%) for different 'Techniques' across various 'Data Group' categories (All Sample, Datavore, Data Builder, Data Mixer, Dataphobe). The techniques listed on the Y-axis are: SQL, Basic statistics, Programming, Data visualisation, Unstructured data analysis, Big data, Advanced statistics, NOSQL, Machine learning, and Experiment design. Each technique has five horizontal bars representing the different Data Groups, illustrating their proficiency levels.
Data application: Figure 5 shows the business areas where, according to our respondents, the work of data analysts is having a major impact. Although process-related improvements (cost saving and automation) are mentioned most often, firms report impacts in many other parts of their business too.
It also shows an interesting split between Dataphobes who are primarily using analytical insights to save costs, automate and improve margins, and Data-active companies, where analysis contributes to the discovery of new opportunities (this is particularly the case for Datavores), creates revenues from data products (especially for Data Builders) and improves customer loyalty (as reported by Data Mixers). This pattern is consistent with the idea that firms that are less advanced with data use it incrementally, with a focus on efficiency, whereas companies further along their data journey make it part of their innovation and strategic processes.
FIGURE 5 AREAS OF IMPACT FOR DATA ANALYSIS
This horizontal bar chart illustrates the 'Percentage where analysis is having major impacts in the area' (0-60%) across different 'Impact' categories for various 'Data Group' types (All Sample, Datavore, Data Builder, Data Mixer, Dataphobe). The impact areas listed on the Y-axis are: Save costs, Automate, Find new opportunities, Improve margins, Build data products, and Improve customer loyalty. Each impact area has five horizontal bars, corresponding to the Data Groups, showing the reported impact percentages.
DATA-ACTIVE COMPANIES ARE FOUND IN ALL SECTORS
It is generally acknowledged that the benefits of data are not confined to a single industry. On the contrary, data is transforming all sectors, reflecting the ‘general purpose’ nature of digital technologies.
Figure 6, where we look at the distribution of our Data Groups across industrial sectors confirms this view, with Data-active companies in all sectors.
That being said, some of the Data Groups are more prevalent in some sectors. So, it turns out that there are more Datavores in quant-heavy Financial Services and science-intensive Pharmaceutical companies, for example. Data Builders, working with big datasets, are more commonly found in ICT. Intuitively, there are more Data Mixers in the Creative Media sector, where a variety of web sources (social media, audience data, web analytics) are being combined to create value.
We find Dataphobes in all sectors too, although interestingly they are somewhat more common in sectors that on the whole have – at least up until now – been less disrupted by the internet (Manufacturing, Retail and Pharmaceuticals). We might speculate that Dataphobe businesses in these sectors will see particular challenges in coming years, with the growth of technologies that are connected and data-enabled, like the Internet of Things, e-Commerce and wearables.
FIGURE 6 DATA GROUPS BY SECTOR
This horizontal bar chart shows the 'Percentage of sector in Data Group' (0-100%) for different 'Data Group' types (Datavore, Data Builder, Data Mixer, Dataphobe) within various 'Sector' categories. The sectors listed on the Y-axis are: All Sample, Creative Media, Financial Services, ICT, Manufacturing, Pharmaceuticals, and Retail. For each sector, there are five stacked horizontal bars, representing the proportion of each Data Group within that sector.
DATA-ACTIVE COMPANIES ARE MORE INNOVATIVE IN THEIR PRODUCTS AND EFFICIENT IN THEIR PROCESSES
What is the quantitative link between a company’s data activities and its performance? Answering this question can help us gauge the contribution of data analysis skills to business value, and the obstacles to growth created by bottlenecks in the supply of analytical talent in the UK.
Figure 5 presented firms’ perceived impacts of analysis on their business. In Figure 7, we consider whether a company sees itself as a product innovator ‘in general’ (i.e. if it tends to launch products and services ahead of its competitors), and whether it considers itself efficient in its business processes.
The survey responses suggest that Data-active companies perform much better than Dataphobes on both counts.
In particular, Data-active companies are more likely to self-identify as being product/service innovators. Data Builders – the companies most likely to describe themselves as product/service innovators – are over 50 per cent more likely than Dataphobes to say they launch products and services ahead of their competitors.
FIGURE 7 REPORTED LEVELS OF PRODUCT/SERVICE INNOVATION BY DATA GROUP
This stacked bar chart shows the 'Percentage reporting levels of product/service innovation' (0-100%) across four 'Data Group' categories: Datavore, Data Builder, Data Mixer, and Dataphobe. Each bar is segmented by agreement level to the statement "Percentage who see themselves as product/service innovators": Don't Know, Strongly disagree, Tend to disagree, Indifferent, Tend to agree, and Strongly agree.
Figure 8 considers whether respondents think they are more efficient than their competitors. Once again, Data-active companies report higher levels of process efficiency than Dataphobes, though the differences are less marked. Datavores in particular strongly agree that they are more efficient in their processes (almost 44 per cent do, compared with only a quarter of Dataphobes).
FIGURE 8 REPORTED LEVELS OF PROCESS EFFICIENCY BY DATA GROUP
This stacked bar chart shows the 'Percentage reporting levels of process efficiency' (0-100%) across four 'Data Group' categories: Datavore, Data Builder, Data Mixer, and Dataphobe. Each bar is segmented by agreement level to the statement "Percentage who see themselves as more efficient than their competitors": Don't Know, Strongly disagree, Tend to disagree, Indifferent, Tend to agree, and Strongly agree.
DATA-ACTIVE COMPANIES ARE ALSO MORE PRODUCTIVE
Self-reported measures of performance like those we have just described obviously suffer from subjectivity. Furthermore, simple bivariate correlations (in the case above, between Data Group status and company performance) fail to consider the possibility that a third variable might be driving the relationship that has been observed.
To address these issues, we carried out an econometric analysis of the link between a company’s Data Group and its economic performance using financial data from FAME, following the same modelling strategy in Inside the Datavores (see Appendix 2 for further information). Our goal was to determine if Data-active companies were, other things being equal, more productive than Dataphobes.
Figure 9 shows the results of this analysis for three models where we progressively introduce more variables to control for other determinants of productivity (measured by value added).
1The Factor Controls model only controls for a company’s levels of production input, namely employment and capital.
2The Sector controls model adds controls for the industry a company operates in.
3The All controls model adds controls for company age and innovativeness, based on the survey responses to the aforementioned questions about product innovation and process efficiency. The idea is to account for some ‘unobserved’ differences between companies that might correlate with both productivity and data activity (like investment in Research and Development, or managerial bent towards risk-taking), thereby inducing a false impression of causality.
The height of the bars show the average differences between companies in a given Data Group and the Dataphobes ‘benchmark’, while the stars show the statistical significance of the estimate (three stars are significant at the 1 per cent level, two stars are significant at the 5 per cent level and one star is significant at the 10 per cent level). (see Appendix 2 for the full regression results).
Our estimates show that companies in all three Data-active groups – and especially Datavores and Data Builders – are significantly more productive than the Dataphobes, even after we control for potentially confounding variables. As might be expected, progressively adding more controls diminishes the marginal contribution of data to productivity, however, and in the case of Data Mixers makes them statistically insignificant (albeit still positive).
FIGURE 9: PRODUCTIVITY ACROSS DATA GROUPS
This bar chart illustrates the 'Average productive uplift (%) compared to Dataphobe benchmark' (0-25%) across three 'Model' categories: Factor controls, Sector controls, and All controls. Within each model, there are three vertical bars representing different 'Data Group' types: Data Mixer, Data Builder, and Datavore. Stars above the bars indicate statistical significance: * Significant at the 10% level, ** Significant at the 5% level, *** Significant at the 1% level.
The headline result is that, other things being equal, companies in the Datavores and Data Builders group are, on average, over 10 per cent more productive than Dataphobes. One plausible interpretation of this finding, consistent with the previous discussion, is that Datavores and Data Builders use analysis to identify new commercial opportunities, improve their efficiency and generate revenues from innovative data products and services, and that this drives up their value added.
3.2 ACCESS TO ANALYTICAL TALENT AND SKILLS
'he preceding analysis shows that higher levels of data activity are in broad terms associated with stronger company performance in terms of higher rates of product innovation, levels of process efficiency, and productivity. The implication is that analytical skills shortages will create bottlenecks for these activities, for superior company performance, and growth.
In this sub-section, we provide evidence on the labour market for data talent. In particular we use our survey findings to look at the talent pools that companies in different Data Groups draw on, the disciplines they bring together to build an analytical capability, and their recent recruitment activity and difficulties filling vacancies. We also identify the specific skills areas for which shortages and gaps exist, and the types of training which UK businesses are investing in to keep the skills of their data analysts up to date.
ALTHOUGH MOST COMPANIES PREFER TO RECRUIT EXPERIENCED TALENT, THEY HIRE FROM UNIVERSITIES TOO.
When asked about the types of analytical recruits they take, most companies (8 in 10) mention ‘experienced talent from inside their own industry’, followed by ‘experienced talent from other industries’ (mentioned by almost 7 in 10) (Figure 10). A considerable number of businesses (36–50 per cent) are also hiring analysts straight of university, including undergraduates, postgraduates and PhDs (over a third of respondents mention doctoral programmes as a source of talent).
Datavores and Data Builders recruit from universities more often. Around 75 per cent in the first group and 70 per cent in the second identify at least one academic source of talent, compared with less than one half of Dataphobes. This is consistent with the importance of advanced research skills in much analytical work, as well as the challenges of finding more experienced talent (which we return to later on).
FIGURE 10 SOURCES OF ANALYTICAL TALENT (PER CENT RECRUITING FROM SOURCE)
This horizontal bar chart shows the 'Percentage recruiting from group' (0-100%) for different 'Talent source' categories across various 'Data Group' types (All Sample, Datavore, Data Builder, Data Mixer, Dataphobe). The talent sources listed on the Y-axis are: Experts in the same industry, Experts from other industries, Undergraduates, MSc/MA, and PhD. Each talent source has five horizontal bars, representing the different Data Groups, illustrating their recruitment percentages.
DATA-ACTIVE COMPANIES ARE BUILDING UP A 'DATA SCIENCE' CAPABILITY
Figure 11 shows the academic disciplines that our respondents recruit from. Business disciplines (including Management, Marketing and Finance) are the main sources of analytical talent (almost two-thirds of companies hire analysts with this background).
Datavores and Data Builders are much more likely to recruit candidates with a Computer Science, Engineering or Mathematics background – around 6 in 10 do, which is double the figure for Dataphobes. Between 40 per cent and 50 per cent of the companies in the Data-active Group more generally (which also includes Data Mixers) hire statisticians.
This mix of computer programming and quantitative knowledge makes a good deal of sense for these companies: they work with more complex and harder to manage datasets, use more sophisticated analytical techniques, and deploy data to build new products and services. This requires developing a ‘data science capability’ that brings together coding and analysis. Around one-half of Datavores and Data Builders, and a third of Data Mixers, hire analysts from core numerical disciplines (Mathematics or Statistics) and Computer Science at the same time, compared with just 17 per cent of Dataphobes.
In Model Workers we quoted industry leaders stressing the importance of multidisciplinary analytical teams for innovation and learning. Our survey findings strongly echo this, with almost everyone in the sample (86 per cent) agreeing that ‘team diversity is important for performance’. We find that Data-active companies stand by this rhetoric in recruiting from a wide range of disciplines. The median Datavore and the median Data Builder hire from four disciplines simultaneously – twice as many as the median Dataphobe.
FIGURE 11 THE DISCIPLINES OF DATA ANALYSTS
This horizontal bar chart shows the 'Percentage recruiting from discipline' (0-80%) for different 'Discipline' categories across various 'Data Group' types (All Sample, Datavore, Data Builder, Data Mixer, Dataphobe). The disciplines listed on the Y-axis are: Business, Engineering, Computer Science, Mathematics, Economics, Statistics, Physics, and Life Sciences. Each discipline has five horizontal bars, representing the different Data Groups, illustrating their recruitment percentages from each field.
DATA-ACTIVE COMPANIES ARE HIGHLY ENGAGED IN THE LABOUR MARKET FOR ANALYTICAL TALENT...AND EXPERIENCE GREAT DIFFICULTIES RECRUITING
Figure 12 shows recent levels of analyst recruitment in our sample, and the difficulties they face in filling vacancies. The top panel displays the percentage of companies that sought to fill at least one analyst vacancy in the 12 months prior to being surveyed (we use different shades of red to illustrate the number of vacancies they sought to fill), while the bottom panel displays the percentage of companies that suffered recruitment difficulties (again, the shades of red show the number of vacancies that were hard to fill).
The headlines are:
1Data-active companies are much more engaged in the market for analytical talent. While just over 40 per cent of the companies in the sample have sought to hire a data analyst in the 12 months before the survey, the equivalent percentage for Datavores is 59 per cent, and for Data Builders 52 per cent. Almost one-half of Data Mixers have sought to recruit data analysts in the previous 12 months. These Data-active companies are also recruiting greater numbers – almost 20 per cent of Datavores have sought to recruit ten data analysts or more in the previous year, compared with just 2.3 per cent of Dataphobes.
2Around one-half of the companies that have sought to recruit data analysts report difficulties filling at least one vacancy. Here, Datavores are particularly likely to highlight difficulties (almost two-thirds of those who sought to recruit have experienced problems in filling at least one vacancy.) Fewer Dataphobes claim to suffer in this area: only 4 in 10. These differences across groups are also visible in the volumes of hard-to-fill vacancies involved: around 15 per cent of Datavores and 10 per cent of Data Builders struggle to fill five vacancies or more, compared with 3.4 per cent of Dataphobes. These findings are worrying because they suggest that many high-performing Data-active UK businesses are struggling to access the analytical talent they need to exploit their data. Finding analytical talent seems to be less of an issue for Dataphobes which, as we showed in previous sections, are less sophisticated in the analysis of their data, and innovative in its applications.
FIGURE 12 LEVELS OF RECRUITMENT AND HARD-TO-FILL VACANCIES
This chart contains two stacked bar graphs showing recruitment levels and hard-to-fill vacancies by Data Group.
The top graph, 'Percentage hiring X analysts', displays the percentage of companies across Datavore, Data Builder, Data Mixer, and Dataphobe groups that hired varying numbers of analysts. The segments represent 'Vacancies': DK (Don't Know), 0, 1 to 4, 5 to 9, 10 to 24, 25 to 49, and 50+.
The bottom graph, 'Percentage of those recruiting reporting difficulties hiring X analysts', shows the percentage of companies within the same Data Groups that experienced difficulties filling vacancies, segmented by the same 'Vacancies' categories.
ANALYSTS WITH ADEQUATE DOMAIN KNOWLEDGE, THE RIGHT SKILLS MIX AND EXPERIENCE - THE 'PERFECT ANALYST'- ARE HARDEST TO FIND
Figure 13 shows the proportions of respondents reporting difficulties in finding analysts with the right level of skills and knowledge in different areas.
It shows that the main areas of concern are insufficient domain, or industry, knowledge, talent lacking the right mix of skills, and professional experience – between 30 per cent and 40 per cent of all companies flagged up issues in these areas. A smaller (although still substantial) percentage of companies – between 15 per cent and 20 per cent – mentioned difficulties finding people with the right technical skills (that is, data analysis and data manipulation).
FIGURE 13 DIFFICULTIES FINDING DATA TALENT WITH THE RIGHT SKILLS IN DIFFERENT AREAS
This horizontal bar chart illustrates the 'Percentage reporting difficulties finding talent with adequate skills in the area' (0-60%) for different 'Skills area' categories across various 'Data Group' types (All Sample, Datavore, Data Builder, Data Mixer, Dataphobe). The skills areas listed on the Y-axis are: Domain knowledge, Right skills mix, Experience, Data manipulation, Business know-how, Analysis, and Soft skills. Each skills area has five horizontal bars, representing the different Data Groups, showing their reported difficulty percentages.
When comparing the responses of different Data Groups, it is striking that Datavores and Data Builders are more likely to report difficulties finding data talent with the right skills in all skills areas. As Figure 14 shows, 80 per cent of Datavores and around 70 per cent of Data Builders report difficulties in at least one skills area (compared with 56 per cent of Dataphobes.) One-third of Datavores and one-quarter of Dataphobes identify four or more skills shortage areas.
Datavores and Data Builders are particularly concerned with finding data talent with relevant domain knowledge (over half report problems in this area), and with the right mix of skills (flagged up by 50 per cent of Datavores, and over 40 per cent of Data Builders). Insufficient business know-how is a proportionately bigger issue for Datavores and Data Builders than other companies too.
These findings strongly reinforce a conclusion in Model Workers: namely that data analysis is becoming increasingly embedded in a wide range of industries and business processes, but in order to create value from it, it needs to be combined with domain knowledge and business know-how – a ‘data science’ mix that, our findings confirm, is in short supply. These constraints are particularly biting for Datavores and Data Builders who are doing more complex analysis and working with larger volumes of data and who, the results of this report suggest, are likely to be forgoing value added because of the recruitment difficulties they face.
FIGURE 14 NUMBER OF SKILLS SHORTAGE AREAS BY DATA GROUP
This vertical bar chart shows the 'Percentage reporting shortages in X areas' (0-50%) for the 'Number of reported skills shortage areas' (0-7) across different 'Data Group' types (All Sample, Datavore, Data Builder, Data Mixer, Dataphobe). For each number of shortage areas, there are five vertical bars, representing the different Data Groups, illustrating the percentage of companies reporting that many shortages.
MOST COMPANIES SEE OPPORTUNITIES TO UPGRADE THE SKILLS OF THEIR WORKFORCE
Figure 15 shows our respondents’ views about the current skills levels of their workforce in different areas, and the perceived opportunities and needs for further development.
In general terms, the companies we surveyed see extensive opportunities to develop the skills of their analysts. Few think their workforce is adequately equipped to address the needs of their business today. The areas where our respondents are least satisfied are data manipulation skills, business know-how and soft skills (roughly in that order).
Intuitively, unlike the case of recruitment, respondents are relatively less concerned about the domain knowledge of their workforce – expected, given that current employees pick up some of this knowledge on the job.
Interestingly, we do not find major differences between Data-active companies and Dataphobes in their reported satisfaction with workforce skills, suggesting that the low levels of Data Activity among the Dataphobes do not reflect insufficient skills in their analytical workforce; rather, they are the result of a lack of analytical ambition.
FIGURE 15 DEVELOPMENT NEEDS IN DIFFERENT SKILLS AREAS
This chart comprises five horizontal stacked bar graphs, each for a different 'Skills area': Domain knowledge, Business know-how, Data manipulation, Data analysis, and Soft skills. Within each skills area, there are five bars, representing 'Data Group' types (All sample, Datavore, Builder, Mixer, Dataphobe). Each bar is segmented to show the 'Situation in skills area (percentage)': Fully equipped, Some need for development, Considerable need for development, and Poorly equipped. The X-axis represents Percentage (0-100%).
Leaving aside specific skills areas where there are opportunities for development, Data-active companies which are more ambitious in their uses of data are more aware of skills gaps in their current workforce. They are between 40 per cent and 50 per cent more likely than Dataphobes to report such gaps (35 per cent–37 per cent say this is the case, compared with 26 per cent of Dataphobes, see left-hand panel in Figure 16).
The right-hand panel in Figure 16 shows the perceived impacts of these gaps in the companies suffering them: over 70 per cent of all companies and almost 90 per cent of Data Builders say that skills gaps in their workforces are having a negative impact on company performance.
FIGURE 16 SKILLS GAPS AND THEIR IMPACTS
This chart features two vertical bar graphs side-by-side, both showing 'Percentage' (0-100%).
The left graph, titled 'Suffered Skills Gaps', shows the percentage of each 'Data Group' (All Sample, Datavore, Data Builder, Data Mixer, Dataphobe) that reported suffering from skills gaps.
The right graph, titled 'Gaps had an impact (as a percentage of those with gaps)', shows, for each 'Data Group', the percentage of those who reported skills gaps that also stated these gaps had a negative impact.
COMPANIES IN OUR SAMPLE ARE USING MULTIPLE STRATEGIES TO ADDRESS SKILLS GAPS
The companies we interviewed in Model Workers described a number of strategies they were adopting to deal with analytical skills gaps. These included investing in software tools that substituted for data analysts (or made them more productive), outsourcing analytical work to specialist providers, employing talent from overseas, and working with universities to spot talent and train it.
Figure 17 shows the frequency with which different strategies are used by survey respondents (note that the base for these percentages are only those companies who reported skills gaps, not everyone). The main action being taken is to increase training budgets, and expand trainee programmes, underscoring the importance that firms attach to developing their in-house analytical capabilities - 73 per cent are doing this. In comparison, fewer companies are resorting to external analytical skills via outsourcing (the exception here are Datavores - 56 per cent of those with skills gaps are looking for specialist skills outside the company).[^33]
It is also worth noting that a substantial minority - just over 20 per cent are working with universities to address their skills gaps and around one-quarter are seeking to address them by tapping into overseas talent.
FIGURE 17 STRATEGIES TO ADDRESS SKILLS GAPS IN THE WORKFORCE
(Bar chart showing strategies for addressing skills gaps in the workforce, with categories for Training, Invest in tools, Strategy, Outsourcing, Recruit outside the UK, and Collaborate with universities, broken down by Data Group: All Sample, Datavore, Data Builder, Data Mixer, Dataphobe. X-axis shows percentage using strategy from 0 to 100.)
The data landscape is changing rapidly, and this requires employers to make substantial investments in workforce skills if they are to stay abreast of the most recent developments in the field. As Figure 15 showed, most companies see opportunities to build up the skills of their analytical workforce in a number of areas.
Almost 80 per cent of our respondents reported providing in-house training and 70 per cent making use of external training (Figure 18).[^34]
Going beyond this, more innovative, web- and peer-based training sites are proving remarkably popular (almost as popular as external training). Two-thirds of companies say their analysts participate in events and meet-ups, more than half use online training services and Massive Open Online Courses (MOOCs), and one-third contribute to online communities and open source software projects, including competition sites such as Kaggle. Notably, universities are the least popular source of training – with only just over one in five companies mentioning them.
These findings are consistent with the fact that the data science field is one where rapid change in experimental tools and data sources often freely available online - have resulted in a vibrant culture of informal, peer-to-peer learning informed by ‘hacker' traditions.[^35]
FIGURE 18 TRAINING ACTIVITIES AND SOURCES
(Bar chart showing percentage using source of training, with categories for Train in-house, Train externally, Attend meetups and events, Train online, Participate in online communities, and Train at universities, broken down by Data Group: All Sample, Datavore, Data Builder, Data Mixer, Dataphobe. X-axis shows percentage from 0 to 100.)
3.3 SECTORAL PROFILES
To conclude our findings section, we have compared data activities and analytical skills issues in different sectors. We summarise the outcomes of this exercise in Table 1 (focusing on the Data Value Chain) and Table 2 (focusing on access to analytical talent and skills). (See Appendix 3 for the figures underpinning this comparison.)
SECTORAL VARIATIONS IN THE DATA VALUE CHAIN AND SOURCES OF TALENT REFLECT DIFFERENCES IN MARKET STRUCTURES, BUSINESS MODELS AND TYPES OF DATA USED
There are good reasons for expecting variations in data analysis practices across sectors, depending on market structures and business models. For example, social media data should be expected to be more important for consumer-facing Creative Media companies using platforms like Twitter or Facebook for promotion and to measure audience behaviour, while Pharmaceutical companies are more likely to be reliant on open and public data (including data published by public research organisations). And this indeed is what we find (Table 1).
A sector's choice of data tools also depends on the types and volumes of data it works with. For example, ICT companies processing large data sets will have a need for non-relational database technologies like NoSQL and big data technologies like Hadoop. In terms of impacts, consumers in Creative Media markets value novelty, so it should not be surprising that analysts in that sector generate impacts by using data to find new opportunities. By comparison, B2B manufacturers prioritise analysis to identify cost efficiencies.
The pattern of differences is similar when we look at the analytical talent piece: for example, the survey findings suggest that science-based Pharmaceutical companies are more likely to recruit from PhD programmes. And in terms of disciplines, more ICT companies recruit computer scientists, while Financial Services firms value economists, and Manufacturers seek out data-expert engineers.
Regarding perceived skills shortages, we see that Creative Media companies are particularly concerned with insufficient domain expertise in the talent pool. This might plausibly be explained by the fact that, until recently, the sector has not been considered as a natural career destination for quantitative analysts, which means that knowledge of the sector in the analytical labour market could be expected to be low. Heightened concerns in the ICT sector about a lack of experienced analysts, and analysts with the right skills mix probably reflects the fast rates of change in the big data technology landscape (which make it difficult to find workers proficient with the latest data platforms and tools), as well as a dearth of multi-skilled data scientists.
We also see intuitive differences across sectors in attitudes to training, with ICT companies that operate in a fast moving technology landscape investing more in data training, including peer-to-peer and web-based solutions and communities that may be the only source of information about cutting-edge technologies and tools. Pharmaceutical companies that have stronger links with academic research departments, it turns out, use universities for training more often.
In terms of their labour market activity, the Creative Media, ICT and Financial Services sectors have been most active in their recruitment of data analysts. Almost two-thirds of Creative Media companies, and more than one-half of ICT and Financial Services companies had tried to hire at least one analyst in the previous 12 months. Manufacturing and Pharmaceuticals, arguably the least 'web-active' sectors in our study, recruited less often, with just over one-quarter of all companies trying to hire at least one analyst recently. Pharmaceutical companies were more likely than other sectors to report difficulties filling at least one vacancy: Around seven in ten did, compared with just under half in other sectors.
DESPITE THESE SECTORAL DIFFERENCES, WE SEE REMARKABLE LEVELS OF DATA INNOVATION AND TALENT 'CROSSOVER' IN THE SAMPLE
The literature abounds with examples of surprising and innovative transfers of data sources and analytical methods across domains -like the application of models developed by physicists to finance, epidemiology to interbank settlements, or machine learning to translation.[^36]
We see hints of this sectoral and interdisciplinary crossover in the data: for example, 10 per cent of Manufacturing companies report using social media data regularly, and almost 30 per cent of retailers claim to have skills in experimental design (i.e. setting up randomised control trials to test the effects of different strategies - e.g. in marketing on website users). Four in ten retailers say that analysis creates an impact through the development of data-based products and services, and a third of Creative Media companies use it for automation (as we are seeing in markets like advertising, via real-time bidding for ad inventory).
The strength of talent and disciplinary flows across industries are also striking: seven in ten companies in our sample are recruiting experienced analysts from outside the 'natural talent pool' in their industry. Over one-third of Creative Media companies hire Physicists, for example, and one-third of Pharmaceutical companies recruit Economists. A quarter of Retailers employ analysts from Life Science backgrounds. These behaviours underscore the transferability of quantitative skills across domains, and the opportunities for serendipitous 'combinatorial' innovations created by unexpected talent flows across sectors.
TABLE 1 SECTORAL DIFFERENCES IN THE DATA VALUE CHAIN
| Area | Observations The first paragraph contains a reference to the table. I need to make sure the table has a title and is wrapped in the div.
Figure 17 shows the frequency with which different strategies are used by survey respondents (note that the base for these percentages are only those companies who reported skills gaps, not everyone). The main action being taken is to increase training budgets, and expand trainee programmes, underscoring the importance that firms attach to developing their in-house analytical capabilities - 73 per cent are doing this. In comparison, fewer companies are resorting to external analytical skills via outsourcing (the exception here are Datavores - 56 per cent of those with skills gaps are looking for specialist skills outside the company).[^33]
It is also worth noting that a substantial minority - just over 20 per cent are working with universities to address their skills gaps and around one-quarter are seeking to address them by tapping into overseas talent.
<div class="chart-description" markdown="1">
FIGURE 17 STRATEGIES TO ADDRESS SKILLS GAPS IN THE WORKFORCE
(Bar chart showing strategies for addressing skills gaps in the workforce. The x-axis represents 'Percentage using strategy (percentage of those reporting skills gaps)' from 0 to 100. The y-axis lists strategies: Training, Invest in tools, Outsourcing, Recruit outside the UK, and Collaborate with universities. Data is grouped by 'All Sample', 'Datavore', 'Data Builder', 'Data Mixer', and 'Dataphobe'.)
</div>
### COMPANIES REPORT HIGH LEVELS OF TRAINING, OFTEN INVOLVING INNOVATIVE PEER-TO-PEER AND ONLINE FORMATS. UNIVERSITIES ARE THE LEAST POPULAR SUPPLIER OF TRAINING
The data landscape is changing rapidly, and this requires employers to make substantial investments in workforce skills if they are to stay abreast of the most recent developments in the field. As Figure 15 showed, most companies see opportunities to build up the skills of their analytical workforce in a number of areas.
Almost 80 per cent of our respondents reported providing in-house training and 70 per cent making use of external training (Figure 18).[^34]
Going beyond this, more innovative, web- and peer-based training sites are proving remarkably popular (almost as popular as external training). Two-thirds of companies say their analysts participate in events and meet-ups, more than half use online training services and Massive Open Online Courses (MOOCs), and one-third contribute to online communities and open source software projects, including competition sites such as Kaggle. Notably, universities are the least popular source of training – with only just over one in five companies mentioning them.
These findings are consistent with the fact that the data science field is one where rapid change in experimental tools and data sources often freely available online - have resulted in a vibrant culture of informal, peer-to-peer learning informed by ‘hacker' traditions.[^35]
<div class="chart-description" markdown="1">
FIGURE 18 TRAINING ACTIVITIES AND SOURCES
(Bar chart showing training activities and sources. The x-axis represents 'Percentage using source of training' from 0 to 100. The y-axis lists sources of training: Train in-house, Train externally, Attend meetups and events, Train online, Participate in online communities, and Train at universities. Data is grouped by 'All Sample', 'Datavore', 'Data Builder', 'Data Mixer', and 'Dataphobe'.)
</div>
## 3.3 SECTORAL PROFILES {: #section-3-3-sectoral-profiles }
To conclude our findings section, we have compared data activities and analytical skills issues in different sectors. We summarise the outcomes of this exercise in Table 1 (focusing on the Data Value Chain) and Table 2 (focusing on access to analytical talent and skills). (See Appendix 3 for the figures underpinning this comparison.)
### SECTORAL VARIATIONS IN THE DATA VALUE CHAIN AND SOURCES OF TALENT REFLECT DIFFERENCES IN MARKET STRUCTURES, BUSINESS MODELS AND TYPES OF DATA USED
There are good reasons for expecting variations in data analysis practices across sectors, depending on market structures and business models. For example, social media data should be expected to be more important for consumer-facing Creative Media companies using platforms like Twitter or Facebook for promotion and to measure audience behaviour, while Pharmaceutical companies are more likely to be reliant on open and public data (including data published by public research organisations). And this indeed is what we find (Table 1).
A sector's choice of data tools also depends on the types and volumes of data it works with. For example, ICT companies processing large data sets will have a need for non-relational database technologies like NoSQL and big data technologies like Hadoop. In terms of impacts, consumers in Creative Media markets value novelty, so it should not be surprising that analysts in that sector generate impacts by using data to find new opportunities. By comparison, B2B manufacturers prioritise analysis to identify cost efficiencies.
The pattern of differences is similar when we look at the analytical talent piece: for example, the survey findings suggest that science-based Pharmaceutical companies are more likely to recruit from PhD programmes. And in terms of disciplines, more ICT companies recruit computer scientists, while Financial Services firms value economists, and Manufacturers seek out data-expert engineers.
Regarding perceived skills shortages, we see that Creative Media companies are particularly concerned with insufficient domain expertise in the talent pool. This might plausibly be explained by the fact that, until recently, the sector has not been considered as a natural career destination for quantitative analysts, which means that knowledge of the sector in the analytical labour market could be expected to be low. Heightened concerns in the ICT sector about a lack of experienced analysts, and analysts with the right skills mix probably reflects the fast rates of change in the big data technology landscape (which make it difficult to find workers proficient with the latest data platforms and tools), as well as a dearth of multi-skilled data scientists.
We also see intuitive differences across sectors in attitudes to training, with ICT companies that operate in a fast moving technology landscape investing more in data training, including peer-to-peer and web-based solutions and communities that may be the only source of information about cutting-edge technologies and tools. Pharmaceutical companies that have stronger links with academic research departments, it turns out, use universities for training more often.
### CREATIVE MEDIA, ICT AND FINANCIAL SERVICES HAVE BEEN MORE ACTIVELY RECRUITING DATA ANALYSTS
In terms of their labour market activity, the Creative Media, ICT and Financial Services sectors have been most active in their recruitment of data analysts. Almost two-thirds of Creative Media companies, and more than one-half of ICT and Financial Services companies had tried to hire at least one analyst in the previous 12 months. Manufacturing and Pharmaceuticals, arguably the least 'web-active' sectors in our study, recruited less often, with just over one-quarter of all companies trying to hire at least one analyst recently. Pharmaceutical companies were more likely than other sectors to report difficulties filling at least one vacancy: Around seven in ten did, compared with just under half in other sectors.
### DESPITE THESE SECTORAL DIFFERENCES, WE SEE REMARKABLE LEVELS OF DATA INNOVATION AND TALENT 'CROSSOVER' IN THE SAMPLE
The literature abounds with examples of surprising and innovative transfers of data sources and analytical methods across domains -like the application of models developed by physicists to finance, epidemiology to interbank settlements, or machine learning to translation.[^36]
We see hints of this sectoral and interdisciplinary crossover in the data: for example, 10 per cent of Manufacturing companies report using social media data regularly, and almost 30 per cent of retailers claim to have skills in experimental design (i.e. setting up randomised control trials to test the effects of different strategies - e.g. in marketing on website users). Four in ten retailers say that analysis creates an impact through the development of data-based products and services, and a third of Creative Media companies use it for automation (as we are seeing in markets like advertising, via real-time bidding for inventory).
The strength of talent and disciplinary flows across industries are also striking: seven in ten companies in our sample are recruiting experienced analysts from outside the 'natural talent pool' in their industry. Over one-third of Creative Media companies hire Physicists, for example, and one-third of Pharmaceutical companies recruit Economists. A quarter of Retailers employ analysts from Life Science backgrounds. These behaviours underscore the transferability of quantitative skills across domains, and the opportunities for serendipitous 'combinatorial' innovations created by unexpected talent flows across sectors.
TABLE 1 SECTORAL DIFFERENCES IN THE DATA VALUE CHAIN
<div class="table-container" markdown="1">
| Area | Observations ```
SECTOR FIGURE 7 RECRUITMENT ACTIVITIES AND HARD-TO-FILL VACANCIES
This chart presents two stacked bar charts.
The top chart, titled "Percentage who recruited X analysts," shows the distribution of "Vacancies" (DK, 0, 1 to 4, 5 to 9, 10 to 24, 25 to 49, 50+) as a percentage for different sectors: Creative Media, Financial services, ICT, Manufacturing, Pharma, Retail, and All sample.
The bottom chart, titled "Percentage of those who recruited with X hard to fill vacancies," shows a similar distribution of "Vacancies" (DK, 0, 1 to 4, 5 to 9, 10 to 24, 25 to 49, 50+) as a percentage for the same sectors.
SECTOR FIGURE 8 DIFFICULTIES FINDING TALENT WITH THE RIGHT SKILLS IN DIFFERENT AREAS
This chart displays a clustered bar chart illustrating the "Percentage reporting hard to find skills in area" across various skill areas (Domain knowledge, Right skills mix, Experience, Data manipulation, Business know-how, Analysis, Soft skills). Each cluster represents a different sector: Creative Media, Financial Services, ICT, Manufacturing, Pharmaceuticals, Retail, and All sample.
SECTOR FIGURE 9 SKILLS DEVELOPMENT NEEDS
This chart presents a stacked horizontal bar chart showing the "Situation in skills area (percentage)" for different levels of skill development: Fully equipped, Some need for development, Considerable need for development, and Poorly equipped. The data is broken down by major skill areas (Domain, Business, Manipulation, Analysis, Soft skills), and within each area, by specific sectors (Creative, Financial, ICT, Manufacturing, Pharma, Retail, All).
SECTOR FIGURE 10 SKILLS GAPS AND IMPACTS
This figure contains two clustered bar charts.
The left chart, titled "Suffered Skills Gaps," shows the percentage of sectors that have suffered from skills gaps, categorized by Creative Media, Financial services, Manufacturing, ICT, Pharma, Retail, and All sample.
The right chart, titled "Gaps had an impact (as a percentage of those with gaps)," illustrates the percentage of entities within each sector (Creative Media, Financial services, Manufacturing, ICT, Pharma, Retail, and All sample) where skills gaps had an impact, out of those that reported having gaps.
SECTOR FIGURE 11 TRAINING ACTIVITIES AND SOURCES
This chart displays a clustered bar chart showing the "Percentage using training source" across various training activities (Train in-house, Train externally, Attend meetups and events, Train online, Participate in online communities, Train at universities). Each cluster represents a different sector: Creative Media, Financial Services, ICT, Manufacturing, Pharmaceuticals, Retail, and All sample.
ENDNOTES
- Bakhshi, H. and Mateos-Garcia, J. (2012) 'Rise of the Datavores: how UK businesses can benefit from their data.' London: Nesta; Bakhshi, Bravo-Biosca, A. and Mateos-Garcia, J. (2014) 'Inside the Datavores: estimating the effect of data and online analytics on firm performance.' London: Nesta.
- http://www.cio.com/article/2854720/data-center/how-and-why-facebook-excels-at-data-center-efficiency.html
- http://www.internetlivestats.com/google-search-statistics/; https://www.youtube.com/yt/press/en-GB/statistics.html
- http://www-01.ibm.com/software/data/bigdata/what-is-big-data.html
- Mayer-Schönberger, V. and Cukier, K. (2013) 'Big Data: A Revolution That Will Transform How We Live, Work, and Think.' Boston MA: Houghton Mifflin Harcourt.
- http://blogs.gartner.com/doug-laney/files/2012/01/ad949-3D-Data-Management-Controlling-Data-Volume-Velocity-and-Variety.pdf
- For more information on these technologies visit http://hadoop.apache.org/ (Hadoop), http://cassandra.apache.org/ (Cassandra) and https://hive.apache.org/ (Hive).
- Brynjolfsson, E., Hitt, L.M. and Kim, H.H. (2011) 'Strength in Numbers: How Does Data-Driven Decisionmaking Affect Firm Performance?' SSRN Scholarly Paper. Rochester NY: Social Science Research Network. See: http://papers.ssrn.com/abstract=1819486; Bakhshi, H., Bravo-Biosca, A. and Mateos-Garcia, J. (2014) 'Inside the Datavores: Estimating The Effect Of Data And Online Analytics On Firm Performance.' London: Nesta. See also???
- Bakhshi, H., Bravo-Biosca, A. and Mateos-Garcia, J. (2014) 'Inside the Datavores: Estimating The Effect Of Data And Online Analytics On Firm Performance.' London: Nesta.
- SAS and Tech Partnership (2014) 'Big Data Analytics Assessment of Demand for Labour and Skills 2013-2020.' London: Tech Partnership. See: http://www.thetechpartnership.com/globalassets/pdfs/bigdata_report_nov14.pdf; Manyika, J. et al., (2011) 'Big Data: The next Frontier for Innovation, Competition, and Productivity.' McKinsey Global Institute; Booz Allen Hamilton (2013) 'The Field Guide to Data Science.' Booz Allen Hamilton.
- Manyika et al., (2011) 'Big Data: The next Frontier for Innovation, Competition, and Productivity.' McKinsey Global Institute.
- SAS and Tech Partnership 'Big Data Analytics Assessment of Demand for Labour and Skills 2013-2020.' London: Tech Partnership.
- For detailed discussions about the skills and competencies of data scientists see Patil, D.J. (2011) Building Data Science Teams. 'O'Reilly Radar.' 16 September 2011. See: http://radar.oreilly.com/2011/09/building-data-science-teams.html. Conway, D. (2010) 'The Data Science Venn Diagram.' See: http://drewconway.com/zia/2013/3/26/the-data-science-venn-diagram., and Schutt, R. and O'Neil, C. (2013) 'Doing Data Science: Straight Talk from the Frontline.' O'Reilly Media, Inc.
- The aforementioned Tech Partnership research suggests that Data Scientists were the hardest to recruit big data occupation.
- Bakhshi, H., Mateos-Garcia, J. and Whitby, A. (2014) 'Model Workers.' London: Nesta.
- Bureau Van Dijk's FAME (Financial Analysis Made Easy) is a database of companies in the UK and Ireland. It draws on business registry data (Companies House in the case of the UK) and other sources to provide financial information about nine million companies. For more information, visit: http://www.bvdinfo.com/Products/Company-Information/National/FAME.aspx
- e-Skills UK (2013) 'Big Data Analytics Adoption and Employment Trends, 2012-2017.' London: e-Skills.
- Manyika et al., (2014) 'Big Data: The next Frontier for Innovation, Competition, and Productivity.' McKinsey and Company. Stone, M. (2014) 'Big Data for Media.' Oxford: Reuters Institute for the Study of Journalism. See: https://reutersinstitute.politics.ox.ac.uk/sites/default/files/Big%20Data%20For%20Media.pdf; 'How Big Data Can Revolutionize Pharmaceutical R & D | McKinsey & Company,' accessed May 6, 2014, http://www.mckinsey.com/insights/health_systems_and_services/how_big_data_can_revolutionize_pharmaceutical_r_and_d.
- Livingstone, I. and Hope, A. (2011) 'Next Gen. Transforming the UK into the World's Leading Talent Hub for the Video Games and Visual Effects Industries. London: NESTA; Bakhshi, H. and Mateos-Garcia, J. (2012) 'Rise of the Datavores.' London: Nesta; Winterbotham, M. 'UK Commission's Employer Skills Survey 2013: UK Results, January 2014.' See: http://dera.ioe.ac.uk/19271/1/evidence-report-81-ukces-employer-skills-survey-13-full-report.pdf.
- OECD (2014) 'Data-Driven Innovation for Growth and Well-Being. Interim Synthesis Report.' Paris: OECD. See: http://www.oecd.org/sti/inno/data-driven-innovation-interim-synthesis.pdf; Bakhshi, H. and Mateos-Garcia, J. (2014) 'Rise of the Datavores.' London: Nesta.
- This is almost identical to the proportion of Datavores in an independent firm survey of UK internet economy businesses in 'Rise of the Datavores.'
- Each Data Group has a statistically significant higher score than the sample overall in the variable that 'defines' it. Dataphobes have significantly lower scores than the average in all of the data variables we use in our clustering.
- http://sloanreview.mit.edu/reports/analytics-innovation/analytical-innovators/
- Rincon, A., Vecchi, M. and Venturini, F. (2013) 'ICTs as a General Purpose Technology: Spillovers, Absorptive Capacity and Productvity Performance.' NIESR Discussion Paper 416.
- Nelson, R.R. (2003) On the Uneven Evolution of Human Know-How. 'Research Policy.' 32, no. 6 (2003): 909-22.
- And with the findings of 'Rise of the Datavores.'
- All the differences are statistically significant at the 5 per cent level.
- They are still statistically significant for Datavores and Data Buillders (at the 10 per cent level), but they are not statistically significant for the Data Mixers.
- Value added is defined as sales minus costs. Our analysis considers the link between being in a Data Group and a company's output conditional on its use of inputs like capital and labour. It shows the extent to which companies in different Data Groups are more or less productive for a given level of resource. Higher levels of productivity could be expected among companies with stronger economic efficiency and superior product quality allowing them to set prices higher for a given level of input.
- Of course, our analysis cannot categorically demonstrate a causal effect of being in a Data Group on company performance. In order to do that we would need a comprehensive set of control variables or longitudinal data about levels of data activity (whereas our survey dataset is cross-sectional). This means that we cannot rule out that our findings are explained by unobserved heterogeneity (innovative, forward thinking and well-managed companies are more likely to work with data and to be more productive, for example), or reverse causality (commercially successful companies invest more in their data).
- Several companies we interviewed in 'Model Workers' said they were looking for talent in universities in response to what they perceive to be a 'data talent crunch' for more experienced talent.
- We have also compared perceived skills shortages between companies that sought to recruit in the previous 12 months, and companies that did not. In general, companies that tried to hire talent had a dimmer view of the situation in the labour market, highlighting skills shortages more often and in more areas than those that did not try to recruit. This is consistent with the idea that our findings reflect the reality of the analytical labour market, rather than just the perceptions of our respondents.
- Recall this was the Data Group that faces greatest difficulties when filling vacancies.
- By comparison, the most recent UK CES Employer Skills surveys shows that two-thirds of employers (66 per cent) had arranged or funded off or on-the-job training or development for any of their staff in the previous 12 months. Winterbotham, M. et al., (2014) 'The UK Commission's Employer Skills Survey 2013: UK Results.' London: UK commission for Employment and Skills.
- Levy, S. (1984) 'Hackers: Heroes of the Computer Revolution.' New York NY: Doubleday.
- Genzel, D. (2010) 'Automatically Learning Source-Side Reordering Rules for Large Scale Machine Translation.' In 'Proceedings of the 23rd International Conference on Computational Linguistics.' Association for Computational Linguistics. 376-84 See: http://dl.acm.org/citation.cfm?id=1873824; Gai, P. Haldane, A. and Kapadia, S. (2011) Complexity, Concentration and Contagion. 'Journal of Monetary Economics.' 58, no. 5 (2011): 453-70.
- Nelson, R.R. 'On the Uneven Evolution of Human Know-How. Columbia University.
- Kalapesi, C., Willersdorf, S. and Zwillenberg, P. (2010) 'The Connected Kingdom: How the Internet Is Transforming the UK Economy.' The Boston Consulting Group. See: https://www.bcgperspectives.com/content/articles/media_entertainment_technology_software_the_connected_kingdom/.
- See: Bakhshi, H. and Mateos-Garcia, J. (2012) 'Rise of the Datavores.' London: Nesta; Bakhshi, H., Bravo-Biosca, A. and Mateos-Garcia, J. (2014) 'Inside The Datavores.' London: Nesta; Bakhshi, H., Mateos-Garcia, J. and Whitby, A. (2014) 'Model Workers.' London: Nesta; Tech Partnership big data analytics reports.
- The Tech Partnership employer skills survey, 2015.
- See: http://www.theguardian.com/media-network/2015/feb/12/data-scientists-as-rare-as-unicorns
- The British Academy (2015) 'Count Us In: Quantitative skills for a new generation.' Available at: http://www.britac.ac.uk/policy/count_us_in_report.cfm?frmAlias=/countusin/
- The idea is to find the number of clusters beyond which adding more clusters doesn't substantially improve the precision of the cluster allocation.
Nesta
1 Plough Place
London EC4A 1DE
Email: [email protected]
Twitter: @nesta_uk
Facebook: www.facebook.com/nesta.uk
Website: www.nesta.org.uk
Nesta is a registered charity in England and Wales with company number 7706036 and charity number 1144091. Registered as a charity in Scotland number SCO42833. Registered office: 1 Plough Place, London, EC4A 1DE. This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
