Acknowledgements

This report was written by Nesta. The research team was led by Jonathan Bone with support from Codrina Cretu, Matt Stokes, Peter Baeck and Tom Symons. The authors would like to thank the many interviewees and roundtable participants who provided their input about what is most important to starting, sustaining and growing DSI initiatives and who helped shape the selection of indicators. We would also like to thank all those who responded to our surveys, and to Speedtest by Ookla for providing data on broadband and mobile internet speeds. Thanks also to our consortium partners, Nesta colleagues and others who provided valuable feedback and support throughout the framework development process, particularly Joel Klinger and Kostas Stathoulopoulos at Nesta, Massimo Menichinelli at Fab Lab Barcelona and Nicola Morelli at Aalborg University. Finally, thanks to the European Union's Horizon 2020 programme for its generous funding of the project.

Consortium partners

• nesta
• betterplace lab
• waag technology & society
• MAKE
• FAB LAB BARCELONA
• Barcelona Activa
• Fundacja ePaństwo

Funded by

Data partner

• SPEEDTEST by OOKLA

About DSI4EU

Building on five years of work led by Nesta in collaboration with DG CONNECT, the Horizon 2020-funded DSI4EU project is supporting digital social innovation in Europe to grow and scale through a combination of research, policy and practical support.

digitalsocial.eu @DSI4EU [email protected]

© European Union, 2019. This work is licensed under a Creative Commons AttributionNonCommercial-ShareAlike 4.0 International License.

DSISCALE, operating under the DSI4EU brand, is funded by the European Commission Directorate General for Communications Networks, Content & Technology, Net Futures, Administration and Finance, under Grant Agreement No. 780473.

The information, documentation and figures in this deliverable are written by the DSISCALE project consortium under EC grant agreement 780473 and do not necessarily reflect the views of the European Commission. The European Commission is not liable for any use that may be made of the information contained herein.

1. Executive summary

This executive summary provides a brief overview of the steps we took to compile the European Digital Social Innovation Index (EDSII).

1.1 Introduction

The European Digital Social Innovation Index (EDSII) contains composite indicators measuring how well the ecosystems of 60 cities in Europe support digital social innovation (DSI) — "a type of social and collaborative innovation in which innovators, users and communities collaborate using digital technologies to co-create knowledge and solutions for a wide range of social needs and at a scale and speed that was unimaginable before the rise of the Internet."²¹

The aims of the index are five-fold:

1) Identify success factors for the creation, growth and sustainability of DSI. 2) Help policymakers understand how they can better support DSI, drawing upon successful examples from other places. 3) Incentivise the development and implementation of supportive policies. 4) Inform practitioners about where in Europe has the best conditions for supporting DSI, which may be influence where practitioners decide to set up or grow their initiatives. 5) Raise awareness about, and interest in, DSI among people, communities and organisations not currently involved in the field.

The index includes 60 cities in total: 26 of the capital cities in the EU28 (Valletta and Luxembourg City were excluded due to unavailability of data) and an additional 34 non-capital cities chosen based on population size, known DSI activity, score on other indexes related to innovation, availability of data, and to ensure good representation across all of Europe.

1.2 Methodology

Below the steps used in the construction of the index. We have adhered to the 'ideal sequence' of steps detailed in the JRC/OECD Handbook on constructing composite indicators to guide our process, and these are illustrated (Figure 1) and described briefly below.

Adapted from https://digitalcityindex.eu/methodology

1.2.1 Theoretical Framework

The theoretical framework for the EDSII was built based on 11 semi-structured interviews and a roundtable event with experts from across Europe, combined with a review of existing academic literature, policy reports, and other related indexes. This research was distilled to give us a clear definition and understanding of the phenomenon we are measuring, 'capacity to support DSI', and used to identify 32 indicators for inclusion in the index and to structure these indicators into six dimensions or 'themes' (Figure 2).

From our research it was that clear that some indicators and themes are more important in measuring capacity to support DSI than others. To reflect this, indicators and themes were given different weights which were used in the later aggregation stage. Weightings were chosen based on a survey of 114 DSI experts, alongside internal workshops.

1.2.2 Source selection & data gathering

A wide variety of data sources were used in the index, including publicly available data (e.g. from the EU and other public organisations) and commercial data, as well as primary data collected through web scraping, API queries and a survey of 143 researchers, practitioners, policymakers and other people interested in DSI. Data was denominated accordingly (e.g. by city population, working age population or purchasing power) in order to make comparisons across different sized cities meaningful.

1.2.3 Data checking

• Treatment of outliers: Outliers were identified as values which are more than 1.5 times the interquartile range (IQR) above the upper quartile or below the lower quartile. Upper-end outliers were transformed to have the same value as the largest non-outlier in that variable and lower-end outliers were transformed to have the same value as the smallest non-outlier.
• Normalisation: A Min-Max normalisation was used to scale variables to be within an identical [0.1, 0.9] range.
• Imputation of missing values: After the initial data gathering exercise, 4% of the data that was needed was missing. These gaps were scattered across multiple indicators and cities. In order to work with a complete dataset, missing data were replaced with estimates calculated using multiple regression. In order to take into account uncertainty about the missing data, five predictions were made and a mean of these five values replaced the missing value.

1.2.4 Data Processing

• Multivariate Analysis: Principal Components Analysis was used to explore the underlying structure of the data, particularly how different variables change in relation to each other and how they are associated. We also used a cluster analysis to give us some insight into which cities scores similarly across themes. In addition, we analysed the correlation between themes, and between themes and the overall index score by calculating Pearson correlation coefficients.
• Weighting and Aggregation: Indicators were aggregated using a weighted arithmetic mean to create theme scores. Theme scores were then aggregated using a weighted geometric mean to produce the overall index score. Using geometric aggregation meant that compensability is lower for theme scores with low value, so a city with a low score for one theme will need a much higher score on the others to improve its score. We believe that this reflects reality as our research suggested that all themes included in the Index are necessary to supporting DSI. Indicators and themes were weighted using weightings chosen in the earlier theoretical framework stage.
• Robustness and Sensitivity Analysis: The robustness and sensitivity analysis checked the effect of several methodological decisions made throughout the index development process on the ranking of cities (e.g. indicator selection and weighting, treatment of outliers, imputation of missing data and aggregation method).
• Index validation: As a means of validating the index we compared cities' scores with the number of DSI organisations located in each city that have entered their information onto the DSI4EU platform. We find a weak to moderate positive correlation.

1.2.5 Data visualisation

A visualisation was developed using Tableau data visualisation software and embedded into the digitalsocial.eu website. The data visualisation allows viewers to compare city scores and rankings for the overall index as well as for the individual themes. It also presents examples of strategies, policies and initiatives from across Europe which may help support DSI.

1.3 Challenges and limitations

We explore some of the reasons why EDSII scores do not correlate with DSI activity more strongly, including how favourable conditions for DSI do not necessarily lead to high levels of DSI if there is not a particular need for it (e.g. in places where there are social challenges are less major and/or urgent), potential issues around the choice of indicators, data quality and how indicators are weighed and aggregated. We also explore some of the challenges in producing an index measuring the capacity to support a project across a very broad range of technologies, organisation types, social challenges and stages of development.

1. Introduction

DSI4EU is an EU-funded project which aims to support policy makers, funders and practitioners to grow and scale digital social innovation (DSI) in Europe and to harness the power of people and technology to tackle some of Europe's biggest social and environmental challenges.

A key part of DSI4EU is to better understand the systemic and macro-level conditions which support the creation, growth and sustainability of DSI initiatives, and to analyse geographically how different parts of the European Union are positioned to support DSI initiatives. Much of this builds upon research carried out by the preceding DSI4EU project, which explored the challenges to growth for DSI at both the macro (system) and micro (project) levels.¹ Building upon this, DSI4EU aims to use this analysis and understanding to influence and help policymakers (and other stakeholders) to proactively build ecosystems which are conducive to the growth of DSI.

The central activity within this area of work is the development of the experimental European Digital Social Innovation Index (EDSII), which aims to measure and compare the capacity of cities to support DSI. The aims of the index are explored below (Section 1.3).

Ultimately, our aim is to situate the Index within the wider framework of DS14EU's activities, including direct policy engagement with cities, the European Commission and national governments, practical support and peer learning for DSI practitioners and other stakeholders, creation of accessible research on the current DSI landscape and the future of DSI, and building a stronger network of DSI in Europe.

1.1 Background

Previous studies of DSI have tended to focus on measuring activity - the volume, geographical spread and characteristics of the DSI initiatives taking place around Europe. This has provided valuable insight and helped researchers, policymakers and other stakeholders to better understand and engage with the field. Nevertheless, there has been less research looking at the enabling and hindering factors behind this activity, exploring what helps or hinders DSI initiatives' creation and growth.

There are currently no comprehensive assessments, tools or methods for assessing the capacity of local, regional or national innovation ecosystems to support DSI initiatives.

A note on terminology

DSI is not the only term used to refer to the use of digital technology to address social challenges. 'Civic tech', 'tech for good' and 'social tech' are also used widely across Europe, and among many communities are much more widely recognised. For this reason, we have used these terms somewhat interchangeably in the development of the index and when speaking to people in the field. However, in the context of this report, we only use the term DSI.

1.3 Aims of the index

The purpose of the EDSII is to provide an open-source tool to measure and compare the capacity of cities to support DSI. This will support DSI projects and other stakeholders in the field in five ways:

1. Identify success factors for the creation, growth and sustainability of DSI. The Index has been produced by combining indicators for several key factors that are important to starting, sustaining and growing DSI projects. By identifying what these key factors are and disseminating our insights through the Index, we will enable a better understanding of the success factors for developing DSI ecosystems and raise awareness on how to spread and promote these conditions between funders, policymakers and other stakeholders. 2. Help policymakers understand how they can better support DSI, drawing upon successful examples from other places. The EDSII provides an overall score, allowing ecosystems in Europe to be ranked based on their capacity to support DSI at the city level. Alongside the overall score, a second output of the index is scores for how well ecosystems perform on different dimensions of the index. These dimensions, or themes, group several indicators, which together measure a particular aspect of an ecosystem's capacity to support DSI, for example, funding or skills. Using these themes, policy makers can use the index as a diagnostic tool to understand where their city is falling behind and thus what areas future policies should target for improvement. The Index is accompanied by the "Ideas Bank", a collection of case studies of supportive strategies, policies and initiatives from across Europe to enable sharing of best practice and replication/adaptation of successful experiences. 3. Incentivise the development and implementation of supportive policies. The scores allotted to ecosystems act as a competitive incentive for policymakers to implement supportive policies which aim to better support DSI within politicians and policy makers cities, regions or countries. 4. Inform practitioners about where in Europe has the best conditions for supporting DSI, which may influence where practitioners decide to set up or grow their initiatives. DSI initiatives are often set up in reaction to a particular local need. However, where there is some flexibility on where a DSI initiative could be located, or, more likely, where an existing initiative is looking to expand. The Index will provide practitioners with useful insight into where might be amenable for projects to locate or grow with maximum chances of success. Furthermore, the scores on individual themes will support practitioners to make informed decisions based on what is most relevant to their project's characteristics and needs. 5. Raise awareness about, and interest in, DSI among people, communities and organisations not currently involved in the field. By providing an accessible benchmark for how cities rank in their ability to support DSI, we hope the Index will spark wider conversations and interest - for example within the startup sector, governments and funding institutions.

1.4 What are composite indexes and why are they useful?

Composite indexes attempt to measure complex social or economic phenomena by combining several individual indicators which, individually, would not adequately describe the phenomenon in question. The individual indicators that make up the composite index are selected, combined and weighted based on an underlying model of the structure of the phenomenon that is being measured.

The EDSII adds to a growing list of composite indexes which aim to explore how different geographic regions compare against a wide range of complex issues with more or less relevance to the field of DSI, including:

• Entrepreneurship and the digital economy - such as the European Digital City Index², CITIE³, the Global Entrepreneurship Index⁴, the Digital Transformation Monitor⁵ and the Global Innovation Index⁶;
• Digital skills and infrastructure - such as the Digital Economy and Society Index⁷;
• (Open) data - such as the Open Data Barometer⁸, the Global Open Data Index⁹ and the European Data Portal¹⁰;
• Open licensing - such as Creative Commons' State of the Commons¹¹;
• Sustainability - such as the Environmental Performance Index¹² and the Sustainable Cities Index¹³,
• Wellbeing - such as the OECD Better Life Index¹⁴ and the Fab City Dashboard¹⁵.
• Social innovation and entrepreneurship - such as the Social Entrepreneurship Index¹⁶ and the Social Innovation Index¹⁷.

None yet exists which is looking at DSI specifically, and our index aims to fill this gap.

1.4.1 The strengths of composite indexes

Composite indexes are useful in two particular cases: either when a single indicator could not conceivably measure the phenomenon due to its complexity and abstract nature; or when a single indicator is conceivable but would be too difficult or costly to measure.

While in theory we could scrutinise each indicator within a composite index individually (and indeed this is sometimes useful), combining them makes it much easier to interpret information without losing sight of, or access to, the underlying information.

Indexes are particularly powerful tools for policymaking when they are captured over multiple years, allowing trends to be identified. These trends can help draw policymakers' attention to particular issues or help them evaluate the impact of specific interventions. Country or city rankings based on index scores are especially good at attracting media attention, facilitating public discussion, promoting accountability and harnessing competitive spirits to motivate policymakers into action.¹⁸,¹⁹

1.4.2 The weaknesses of composite indexes

Composite indexes, like all models, are by necessary simplified versions of reality. And while it is this simplification that makes them useful, it means that in some sense they will always have their errors and misrepresentations. Like all models, composite indexes are created through a combination of science and art, and their construction necessarily entails numerous decisions (such as the choice and weighting of indicators) which are more or less subjective. Such decisions are not always clear cut and are almost always subject to debate. Getting these decisions wrong, or having to ignore indicators that are too difficult to measure, can result in simplistic or inappropriate policy messages.

Even if constructed as accurately as possible, problems may arise if policy conclusions drawn from composite indexes are (consciously or unconsciously) erroneous. The complexity of composite indexes means misrepresentation is an ever-present risk. For this reason, it is important that those producing indexes are transparent about the processes, methodologies, decisions and experts used in constructing them, and about what their limitations are. Composite indexes should not become "black boxes" whose users are blind to the decisions and processes behind the final numbers. And policymakers should not make any policy-decisions based on a single index (or model more generally).

1.5 Geographical coverage

65 European cities were carefully selected through consultations with our partners and a process of internal deliberation. These cities were selected based on population size, known DSI activity, score on other indexes related to innovation,²⁰ availability of data and to ensure good representation across all of Europe and especially in traditionally underrepresented areas such as Eastern

Footnotes

and Central Europe. Five cities were later removed because of lack of good data: Valletta (Malta), Essen (Germany), Luxembourg City (Luxembourg), Naples (Italy) and Brno (Czech Republic). The remaining 60 cities included in the index are listed in table 1.

The EDSII focuses on ranking cities, rather than countries, for several reasons:

• This is in keeping with the principle of subsidiarity in policymaking, “the principle that social and political issues should be dealt with at the most immediate (or local) level that is consistent with their resolution”²²;
• DSI is particularly active at city level, due both to the challenges faced by cities (ranging from transport and air pollution to provision of healthcare, housing and education), and the density of people, assets, infrastructure, knowledge and skills which allows for collaborative technologies to thrive;
• Cities are increasingly becoming a hotbed for innovative policymaking and strategies (both in digital and non-digital policy);
• In many countries, although not all, cities have significant power over the policy decisions which affect people's day-to-day lives the most and effective ways of engaging with citizens.

Table 1. Cities chosen to be featured in the EDSII

2. Methodology

The methodology followed when building the EDSII is based on that used for the European Digital City Index, which itself is adapted from the "ideal sequence" of steps detailed in the JRC/OECD Handbook on constructing composite indicators (Figure 1)²³. In following this methodology and carefully documenting each stage of the process we have ensured the index is as robust and transparent as possible.

It is important to note that while Figure 1, and this document in general, presents this as a linear process in reality it was an iterative process where we were frequently required to revisit earlier steps.

2.1 Developing the theoretical framework

Throughout the development of the framework, we aimed to be as inclusive, transparent and open as possible, in keeping with the values of DSI and to create an Index which is not just useful but also accepted and welcomed as legitimate by different stakeholders in the DSI community.

2.1.1 Defining the concept

As discussed above, the index compares the capacity of cities to support DSI initiatives. Although DSI continues to evolve as a field, for this index we continue to use the definition first outlined in our first study on digital social innovation, published in 2015:²¹

"A type of social and collaborative innovation in which innovators, users and communities collaborate using digital technologies to co-create knowledge and solutions for a wide range of social needs and at a scale and speed that was unimaginable before the rise of the Internet."

When we refer to the 'capacity to support', we are looking beyond top-down government level policy levers,²¹ considering ecosystem capacity in the broadest sense of the word. For this, we will analyse the DSI ecosystem as a whole including political, economic, social, cultural and technological factors affecting a city.

2.1.2 Themes and indicators

Indicators were selected through a literature review (see Appendix 1 for a selected literature review) and consultation with experts and stakeholders in DSI through 11 semi-structured interviews²⁴ and a roundtable event.

An initial set of 69 possible indicators was narrowed down in an internal workshop at Nesta to 32 indicators to be included in the Index. Longlisted indicators were excluded from the shortlist because of either duplication (they were a subcategory of another indicator or were closely related to another indicator) or irrelevance (there was no strong link to DSI). These 32 indicators were then grouped by the research team into six themes (initially indicators were grouped into seven themes but one theme 'Support Systems' was later merged with the infrastructure and skills theme²⁵), grouped alongside other related indicators. The six themes are:

• Skills
• Infrastructure
• Funding
• Diversity and Inclusion
• Collaboration
• Civil Society

Below we describe and discuss the justification for the inclusion of the 32 indicators we have chosen. We discuss the indicators under the subheadings of the seven themes.

2.1.2.1 Skills

Anecdotal evidence suggests a wide range of both technical and non-technical skills are needed for DSI initiatives to thrive. Drawing upon literature exploring related fields like entrepreneurship, social impact and social innovation, we can distinguish a number of soft skills (e.g. communications, administration) and hard skills (e.g. programming, data skills) which are commonly used as a measurement of human capital, powering ecosystems and helping them to grow.²⁶ Other literature highlights the importance of knowledge and skills as a requirement for grassroots innovation as well as a measurement of successful outcomes.²⁷

2.1.2.2 Civil society

Most DSI originates within civil society in its broadest sense, ranging from large charities to informal community groups. Through our review of literature and interviews, we found that an active, trusted and developed civil society is important in encouraging the creation of DSI and in encouraging citizen engagement in DSI. A culture of donating money and volunteering is important to supporting DSI initiatives and is indicative of public engagement in social causes more generally.

2.1.2.3 Collaboration

Given the open and multidisciplinary nature of DSI, collaboration (both online and offline) is a key success factor. DSI works best when a diverse group of people with different expertise (such as technology, social challenges and provision of public services) work together.

We know that explicitly outward-looking technologies which are powered by, and drive, collaboration are at the heart of DSI. As the DSI ecosystem remains relatively fragmented, it can be difficult for practitioners to identify, learn from and collaborate with similar projects. Collaboration and sharing of knowledge and best practice between policymakers, practitioners, investors and other stakeholders is key to enabling peer learning and supporting the sustainable growth of DSI (and something which we aim to facilitate through DSI4EU).

2.1.2.4 Funding

Unsurprisingly, experts consistently said that the availability of funding is important for DSI initiatives at all stages of development. Grant funding in particular was highlighted as being crucial to starting and sustaining initiatives. Beyond grant funding, DSI initiatives which have potential to create financial as well as social returns may also look to raise larger amounts of money by selling equity in their business. With this in mind several interviewees suggested the importance of an active impact investment scene.

Further to this, entrepreneurship literature has highlighted the huge benefits associated with procurement from large organisations, helping startups scale up their operations and conveying validation to potential future customers²⁸. A large number of DSI initiatives work in fields where the public and social sector holds a monopoly, such as in healthcare, education and employment support. It is for this reason why we have included an indicator measuring the 'Willingness of public and social sector procure from startups'.

2.1.2.5 Infrastructure

DSI initiatives' creation, growth and sustainability is enabled by good infrastructure: digital (such as broadband and mobile internet and provision of open data), physical (such as workspaces, accelerators, makerspaces, fablabs) and process-related (such as those involved in setting up a business). While data is not often treated as infrastructure, as the Open Data Institute points out, 'Data is as important as our road, railway and energy networks and should be treated as such'. And in order to maximise data use and value, we need not only data assets but also the organisations that operate and maintain them, and guides describing how to use and manage the data.²⁹

2.1.2.6 Diversity and inclusion

Diverse and inclusive communities, environments and sectors (of different genders, sexes, ethnicities, sexual orientations, abilities, ages etc.) lead to better, more sustainable DSI initiatives. A myriad of studies has shown that more diverse and inclusive companies and sectors are more innovative.[^39] This is likely because more diverse groups encompass and bring with them a wider range of interests, experiences, backgrounds and ideas. Diversity is particularly important in the field of DSI where minority groups are more likely to have lived experience of the social challenges which DSI tries to tackle.

Furthermore, digital inclusion and digital skills among the population are important for DSI to grow because for DSI to deliver impact equitably it must be open and accessible to everyone, regardless of age, ethnicity, ability, gender, income or location. Indeed, emerging research from the civic technology community suggests that without digital inclusion, digital technologies can replicate or even exacerbate existing patterns of inequality and discrimination.[^40]

2.1.3 Weighting indicators and themes

The index takes a "nested" structure. Within this structure, each indicator has a score; these scores were combined to give a score for each theme; these theme scores were then combined to give a score for the index as a whole.

While some indexes weight individual indicators as equally important for the overall index score, it was clear from the literature review and interviews that some themes and indicators are more important than others. We therefore decided to assign indicators different weights when combined into the overall index score.

In order to determine how indicators and themes should be weighted, we conducted a survey in which 114 DSI practitioners³⁰, public sector employees, researchers, intermediaries, funders and policymakers were asked to rank the themes and indicators in order of their importance to supporting DSI³¹. We used the median ranks given to themes and indicators to inform their relative weightings.

When ranking indicators, survey respondents were only told what the indicators would be measuring (e.g. “Public advocacy for DSI"), not how exactly they would be measured (e.g. "Number of articles which mentioned a DSI related keyword over the last 12 months at the country level")³². This means that the weights given by respondents did not take into account the quality of the data sources to be used in the index.

In response to this, weightings were revised after source selection and data collection. We weighted down indicators for which we had concerns about the data's relevance to the indicator, analytical soundness, geographical coverage or age. Downweighting an indicator involved giving that indicator the same weight as that of the next highest weighted indicator in the same theme. Where the indicator already had the lowest weighting in that theme, we decreased its weight by the average difference between indicator's weightings in the theme.

The final weightings used to aggregate indicators into themes and themes into the final ranking can be found in tables 2 to 7.

2.2 Source selection and data gathering

2.2.1 Data sources

Data sources for each indicator were chosen based on four criteria:

• their relevance to the indicator;
• analytical soundness;
• geographical coverage;
• their relationship to other indicators being considered.

A wide variety of data sources were considered during this stage including publicly available data (e.g. from the EU and other public organisations) and commercial data, as well as primary data collected through web scraping and API queries. In some cases, data sources were other indexes identified through the literature review. Also, for some of the 32 indicators, we combined multiple variables. This means that the index is made up of a total of 49 variables.

Occasionally we were unable to find either a direct measure or a proxy measure for an indicator that was identified as being important during the theoretical framework development. In order to collect data for these indicators we conducted a survey of 143 researchers, practitioners, policymakers, support organisations, funders or people otherwise interested in DSI from around Europe. In Tables 2-7 below, indicators using data from this survey are marked 'primary research' in the data source column.

It is important to note that the relevance of data sources to indicators is, to some extent, subjective and so the pros and cons of different data sources was discussed among the research team before final choices were made. Where indicators are dependent on, or affected by, size-related factors or the cost of goods or services in a particular country, they were scaled by a factor (e.g. working population or purchasing power parity) to compensate for this.³³

Below we present information on the variables used for each indicator:

Table 2. Civil society (weighing = 20%)

medium-sized city might have one or two fablabs, but we would not necessarily expect a city with five times the population to have five times as many fablabs. For this reason, we did not standardise these variables by city population, as we did for some other variables. It is hard to distinguish for which variables this would be the case but decided that it was more likely to be true for those for which absolute numbers are small. We therefore decided not to standardise count variables for which the maximum count number is less than 25 - still somewhat arbitrary, but in our opinion the most sensible option.

Footnotes

3. Collaboration

The 3 variables were combined to create one indicator by first normalizing all variables to be within the same 0.1 to 0.9 scale and then calculating the mean of the 3 variables.

4. Skills

5. Infrastructure

6. Funding

7. Diversity and Inclusion

These two measures were combined to create one indicator by first normalizing both variables to be within the same 0.1 to 0.9 scale and then calculating the mean of the two variables.

These two measures were combined to create one indicator by first normalizing both variables to be within the same 0.1 to 0.9 scale and then calculating the mean of the two variables.

2.3 Data checking

2.3.1 Treatment of outliers

Index building is based on a benchmarking principle where baseline values considerably influence a city or country's index score as well as its rank. The presence of outliers may result in an inappropriate benchmark and must therefore be dealt with before the index can be constructed.

Outliers were identified as values which are more than 1.5 times the interquartile range (IQR) above the upper quartile or below the lower quartile. Large outliers were transformed to have the same value as the highest existing value in that variable which falls within the thresholds outlined above. Small outliers were transformed to have the same value as the lowest existing value in that variable which falls within the thresholds outlined above. Twenty-nine out of the 49 variables used in the index contained at least one outlier.

2.3.2 Normalisation

Indicators will have different measurement units and scales of magnitude. Data must therefore be normalised so that different indicators are on the same scale before they can be aggregated into the composite index. We used a simple min-max normalisation method with which we which we transformed variables to within an identical [0.1, 0.9] range using the equation below. ¹

Equation 1. Min-max Normalisation [0.1, 0.9]

(0.9-0.1)(xi,j-min(xj))
Zi,j = --------------------
      ((max(xj)-min(xj))+0.1

where z_i,j is the normalised value for city i and variable j? x_i,j is the original value for city i and variable j max(x_j) is the maximum value for variable j min(x_j) is the minimum value for variable j

2.3.3 Imputation of Missing Data

In order for the Index to fairly compare cities on a like for like basis, we require a complete data set with the same data points for every region. However, the geographical coverage of economic and demographic datasets is often incomplete. Although the number of overall missing values was generally low (4%), of the 32 indicators included, 9 indicators (28%) had missing data for at least one of the 60 cities.

Where possible, our first approach was to try and find another data source which is more complete. When this was not possible, we used multiple regression to assign (or 'impute') estimates to missing values. In order to perform imputation, predicted values were calculated using the iterative imputer function in the fancyimpute Python library. Where data was missing for a particular variable, predicted values were computed using the other indicators in the same dataset as explanatory terms. In order to take into account uncertainty about the missing data, five predictions were made and a mean of these five values replaced the missing value.

In the later sensitivity analysis stage, we tested the effect of regression imputation by recreating the index, replacing missing data with the mean of the other variables in that theme obtained for that city, as was used in the European Digital City Index ² (i.e. 'theme mean imputation').

2.4 Data processing

2.4.1 Factor analysis

We assessed the overall structure of the data collected and processed using Principal Components Analysis (PCA) ³. PCA helps reveal how different indicators change in relation to each other and how they are associated. This can help us decide whether the way we have grouped indicators into 'Themes' makes sense from a statistical point of view. We chose to describe the data using 7 principle components, because after 7 there is a large decrease in the amount of explained variance that each extra component adds (figure 3).

Figure 3. Variance explained by each principal component

These 7 components together explain 72% of the variance in the data. Only component loadings of more than 0.15 (or less than 0.15) were considered high enough to be taken into consideration when interpreting the results of this analysis. When indicators have high loadings within the same component, this indicates that those indicators are correlated.

We found some, but relatively low, similarity between the themes laid out in the theoretical framework stage and the statistical components calculated by the PCA (see Appendix 3 for PCA loadings). Component one of the PCA contains four out of six of the indicators in the 'Civil Society' theme suggesting that these indicators vary together and thus, make sense to be grouped under the same theme. However, the indicators 'Public advocacy for DSI (e.g from political / leading figures)' and ‘Presence of supportive government policy for social purpose initiatives', which are also in the Civil Society theme carry a low loading weight in component one. Public advocacy for DSI appears to instead be correlated with indicators related to funding. This is perhaps unsurprising as more funding is likely to be available in cities in which there is public advocacy for DSI and civil society more generally (and vice versa). While it may have made sense statistically to move the Public advocacy for DSI indicator into the Funding theme, conceptually it was thought to fit better under the civil society theme.

2.4.2 Aggregation method

The next stage involved combining individual indicators into a composite index score. We decided to first combine indicators within the same theme to give theme scores and then to aggregate these theme scores to produce an overall index score by which cities could be ranked. The benefits of this are two-fold. Firstly, it means that themes that contain more variables were not automatically more influential to the final index score than those with fewer variables. Secondly, alongside the overall index scores we can present theme scores for each city. This makes it easier for users to understand a given city's score, in turn increasing its utility as a diagnostic tool.

During the framework development stage, we initially decided that themes and indicators should be aggregated in a non-compensatable fashion, meaning that a low score for one indicator or theme cannot be completely offset by a high score in another. In later discussions we decided that due to uncertainty around the reliability of individual indicators, it would be more sensible to allow compensability between individual indicators but not between themes.

In order to have compensability between indicators but not between themes, we aggregated indicators into themes using an arithmetic mean (commonly referred to as just a 'mean'; equation 2) and then aggregated themes into the final index score using a geometric mean (equation 3). Table 8 shows the overall city scores and rankings after weighting and aggregation.

Equation 2. Weighted arithmetic mean aggregation of indicators into theme scores

     J
    ∑  w_j z_i,j
    j=1
TS_i,k = ---------
     J
    ∑  w_j
    j=1

where TS_i,k is the aggregated theme score for city i and theme k w_j is the weight given to variable j=1,...,J z_i,j is the normalised value for city i and variable j=1,...,J

Equation 3. Weighted geometric mean aggregation of theme scores into overall index score

      K
     ( Π TS_i,k^w_k ) ^ (1 / ∑ w_k)
      k=1
IS_i =

where IS_i is the aggregated Index score for city or country i w_k is the weight given to theme k=1,..., K TS_i,k is the aggregated theme score for city or country i and theme k =1,...,K

In the sensitivity analysis stage, we tested the effect of this aggregation method by recreating the index using all four combinations of arithmetic and geometric means to aggregate themes and indicators.

2.4.3 Cluster analysis

Cluster analysis is a descriptive tool which can be used to group data based on similarities/dissimilarities between cases. We used cluster analysis to give us some insight into which cities scores similarly across themes.

We used a hierarchical classification because we did not know the number of clusters there should be a priori. There are several clustering algorithms that can be used to categorise data based on the distance between data points. For all methods, a small distance is equivalent to a strong similarity. We use 'ward' as the method since it minimises the variants of distances between the clusters.

Deciding on the optimum number of clusters is largely subjective, although looking at the plot of linkage distance can be a useful guide. In this case, there is a sudden jump in the level of similarity among the 60 cities at a linkage distance of 1.5, indicating that the cities are best represented by four clusters (figure 4).

We noted the following four clusters:

1. Ljubljana, Vilnius, Budapest, Prague, Vienna, Riga, Dublin, Helsinki and Tallinn. Except for Dublin and Helsinki all these cities are situated in the Baltics or Central Europe. These cities take middle to bottom ranks in the index and generally excel in the skills theme but fall short in collaboration.
2. Cluj-Napoca, Bucharest, Zagreb, Athens, Sofia, Turin, Rome and Milan. These cities are all situated in South and Eastern Europe. They all rank towards the bottom of the index. They typically score particularly low for diversity and inclusion but do relatively well for skills.
3. Toulouse, Bilbao, Belfast, Karlsruhe, Leipzig, Dortmund, Dresden, Munich, Nice, Ghent, Düsseldorf, Frankfurt, Aarhus, Lisbon, Porto, Birmingham, Manchester, Berlin, Marseille, Lyon, Bristol, Edinburgh, Gothenburg, Warsaw, Krakow, Nicosia, Bratislava, Malmo, Hamburg, Cologne and Stuttgart. These cities are spread throughout the middle ranks of the index. They are spread throughout Europe geographically. They generally score relatively high on civil society, collaboration and diversity and inclusion, but fall behind on infrastructure and skills.
4. Amsterdam, London, Utrecht, Eindhoven, Rotterdam, The Hague, Barcelona, Madrid, Paris, Brussels, Copenhagen and Stockholm. These cities are situated in Western Europe. They all sit within top 15 ranks in the index and generally score highly across all six themes.

Figure 4. Dendrogram showing the relationship between cities determined through cluster analysis

2.4.4 Theme correlations

In order to test the correlation between themes, we calculated theme by theme Pearson correlation coefficients. We find that there is a moderate positive correlation between several themes. The strongest positive correlations exist between the Collaboration and Civil Society themes and between the Civil Society and Funding themes (table 9). These correlations can be explained conceptually. For example, the collaboration theme contains indicators which relate to the collaboration between civil society and the tech sector and government; without a developed civil society such collaborations would not be possible. Further to this, civil society is more likely to become developed where there is funding available to support it and vice versa.

Although the themes all have a distinct focus, they all contain indicators that are clearly relevant to socially-focussed projects e.g. funding contains indicators related to grants and impact investing, and infrastructure contains indicators related to socially-focussed accelerators and incubators. It is perhaps unsurprising then that the Civil Society theme has a moderately strong positive correlation with four out of five of the other themes. Interestingly, the Skills theme has very little correlation with the other themes. This may be explained by the fact that this theme has little focus on skills normally associated with socially-focussed projects or civil society more generally. Diversity and Inclusion was also less heavily correlated with other themes.

We also investigated the correlation between themes scores and the overall index scores cities received. These correlations will of course largely be the result of the weighting themes were given in the index, so we were looking for correlations that stand out even when taking weightings into account (Table 10).

Unsurprisingly, all theme scores have a moderately strong correlation with overall index score. However, for Skills, and Diversity and Inclusion this correlation is weak relative to the other themes. This is a result of these themes being less strongly correlated with the other themes. Civil Society and Infrastructure on the other hand are highly correlated with the overall index score and with the other themes.

2.4.5 Index validation

As a means of validating the index we compared cities' scores with the number of DSI organisations located in each city that have entered their information onto the DSI4EU platform (figure 5). Although we would expect some positive correlation between these two variables, we would not expect them to correlate exactly as they are not measuring the same thing. The EDSII is trying to measure the conditions which support DSI whereas the DSI4EU map is mapping DSI activity itself. Having the right conditions in place for DSI does not necessarily translate into high DSI activity. This is case for many possible reasons: for example, DSI (like social innovation more broadly) is often reactive to social challenges (such as inequality or corruption), so cities with supportive conditions for DSI but fewer or less serious social challenges may have less DSI activity than cities where the conditions are not as supportive but where there are many social challenges which demand solutions. Another reason is that the Index measures systemic conditions but does not take into account specific DSI-focused policies and initiatives taken by government.

Figure 5. Comparison of EDSII scores and number of organisations on the DSI4EU platform for each city.

We found there to be a weak to moderate positive correlation (Pearson correlation coefficient, r = 0.45). This is perhaps less of a correlation than we would have expected, even bearing in mind the differences outlined above. We explore some of the possible reasons for this below.

Where it was thought that indicators would be affected by size-related factors, such as the population or purchasing power of the city or country, indicators were scaled to account for differences in these factors. As a final check whether the final score was affected by such factors, we tested whether there was a correlation between city population and the overall index score. We found that there was only a very small positive correlation (Pearson correlation coefficient, r = 0.19), suggesting that our scaling of indicators was effective.

3. Sensitivity analysis

To understand the impact of the methodology decisions made, we tested the effect of the following:

• Variable Selection - The effect of discarding a variable
• Outlier treatment - The effect of transforming outliers to the same value as the closest non-outlying value
• Imputation method - The effect of using multiple imputation rather than theme mean imputation to account for missing data
• Aggregation method - The effect of using geometric (variable) / geometric (theme) rather than arithmetic (variable) / geometric (theme) aggregation.
• Weight Selection - The effect of varying weightings of variables.

3.1 Indicator Selection

The aim of this analysis was to determine whether a single indicator had an excessively large impact on the overall ranking. To test this, we ran a Monte Carlo simulation that tested the effect of sequentially and randomly excluding indicators from the index. The results of this simulation are shown in figure 6. The wider the box and whisker, the more variable that city's rank is when indicators are removed.

In general, the top ranking and bottom ranking cities were the least sensitive to changes in the index composition with middle ranking cities being more sensitive (figure 6). London was particularly robust to changes in the indicator composition, ranking first in nearly all of the combinations tested. Helsinki, Tallinn and Dublin were particularly sensitive to which variables were included in the Index. Nevertheless, this analysis shows that the ranking is relatively stable, even to major changes in indicator composition.

Footnotes

3. Sensitivity Analysis

3.2 Treatment of outliers

It is often not possible to determine whether outliers are the result of a mistake during data collection or just an indication of variance in the data. It is, therefore, debatable whether outliers should be treated. We tested the effect of transforming outliers to the same value as the closest non-outlying value by recreating the index without treating outliers (see ‘Rank with outliers' of table 11).

London and Paris remained in first and second position when outliers were untreated. Zagreb remained in last position and most of the bottom 10 positions remained relatively unchanged. A few cities, however, ranked considerably worse when outliers were not removed, such as Madrid, Barcelona, Brussels and Helsinki. This indicated that these cities had some low outliers which when treated improved their scores. Others did considerably better when outliers were not treated e.g. Birmingham, Berlin and Vienna, suggesting that these cities had high outliers that when treated decreased their scores. In general, however, while there are several small changes throughout the ranking, most cities did not move considerably.

Table 11. City; rankings when index re-made without treating outliers, without imputing missing values, without weighting indicators, and with different methods for aggregating data. Green cells represent higher rankings compared to the final index, red cells represent lower rankings and yellow cells represent equal rankings.

3.3 Imputation method

We tested the effect of using multiple imputation on missing values rather than using the simpler method of replacing missing data with the mean value of the other indicators in the same theme for that city ('theme mean imputation'). The imputation method used obviously has the largest effect on those cities with a lot of missing data.

London and Paris remained in first and second position when theme mean imputation was used and Zagreb remained in last position (see ‘Rank with theme mean imputation' column of Table 11). Copenhagen, which was in third position, moves to 15th position when theme mean imputation is used. Aarhus and Helsinki also rank considerably worse when theme imputation is used (23rd to 40th position and 11th to 26th position, respectively). For most cities, however, rankings only change slightly.

3.4 Aggregation method

We used an arithmetic mean to aggregate indicators and geometric mean to aggregate themes. The use of geometric or arithmetic means to aggregate indicators and themes each have their own benefits, so we tested the effect that this decision had on rankings (Table 11). To do this we re-created the index using: arithmetic means to aggregate both indicators and themes (see 'Rank with arith- arith aggregation' column), geometric means to aggregate both indicators and themes ('Rank with geo-geo aggregation'), and a geometric mean to aggregate indicators and an arithmetic mean to aggregate themes (‘Rank with geo-arith aggregation').

London remained in first position and most cities only experienced small changes in ranking with different aggregation methods. With geometric aggregation, compensability is lower for the composite indicator or themes with low values. This means that when using a geometric aggregation, a city with a low score for one indicator or theme will need a much higher score on the others to improve its score. Therefore, cities with one or a few very low scores experienced the largest effect of changing from arithmetic to geometric aggregation of indicators Brussels, for example, moved from 8th to 18th position when geometric aggregation was used.

Similarly, using arithmetic rather than geometric aggregation of themes will improve the ranking of cities which fall behind on a particular theme. Generally, this appears to have less of an effect than changing the indicator aggregation method. However, some cities did experience considerably better rankings, Karlsruhe for example moved from 38th to 24th position when arithmetic aggregation of means was used.

3.5 Weight Selection

To measure how sensitive the rank is to the weighting selection we ran a Monte Carlo simulation. In this simulation, single indicator and theme weights were randomly selected from our possible values (specified in Table 2). The results of this simulation are shown in figure 7. The wider the box & whisker, the more variable that city's rank is when the weighting is changed.

In general, weighting changes have very little effect on the ranking. This is because weightings in the index are generally not very extreme; all themes are weighted either 15%, 17.5% or 20%.

London maintains its first position in all versions of the index tested. As with the indicator selection, weight selection appears to have the largest impact on middle-ranking cities. However, even with these cities' variation is relatively small.

We also tested the effect of equally weighing all indicators and themes (see 'Rank with equal weightings' column in table 11.) In general ranking changes were minor, reflecting the relative subtlety of the weightings applied. London and Amsterdam remained in first and second position when equal weighting was applied, and Zagreb remained in last position.

4. Data visualisation

A data visualisation was developed using Public Tableau software and embedded into the digitalsocial.eu website. The interactive data visualisation allows viewers to compare city scores and rankings for the overall index as well as for the individual themes. It also presents examples of strategies, policies and initiatives from across Europe which may help support DSI.

5. Challenges and limitations

"All models are wrong, but some are useful" George Box, Statistician

As is the case with all indexes, and models more generally, the EDSII does not perfectly describe cities' abilities to support DSI. We do, however, believe it will be useful tool to help policymakers understand how they can better support DSI and to incentivise the development and implementation of supportive policies. We also think it can inform practitioners about where has the best conditions to support DSI, helping them to decide where to set up or grow their initiatives.

Throughout the development of the index, several largely subjective decisions were made with regards to indicator choice, data processing and aggregation methods. While some indexes cloak such decisions in a veil of objectivity, we wanted to be transparent about where these decisions were made and how they might affect rankings. The sensitivity analysis we conducted shows that while these decisions can have major effects on rankings, most of the general structure of the ranking is relatively robust. The top ranking positions of London and Amsterdam were also very robust to changes in the methodology.

While several cities do rank better or worse than we would have expected, in general, the ranking of cities in the index is relatively unsurprising. However, we did expect to see a stronger correlation between the index and levels of DSI activity (as mapped through the DSI4EU project), as discussed above.

There are several possible reasons for why this might be the case. Indeed, it is most likely a combination of (some of) these reasons.

• Favourable conditions for DSI don't necessarily lead to high levels of DSI. As noted above, many DSI initiatives emerge as a response to unfavourable social, political and economic contexts, in an attempt to address social issues that have been overlooked by traditional institutions. For example, a large number of digital democracy initiatives have emerged in Eastern and Southern Europe addressing localised issues such as corruption and lack of government transparency. Some of the cities which are high-ranking in the index might simply be better places to live, meaning that bottom-up DSI is less necessary; and some of the cities which are low-ranking in the index might have conditions which lead people to develop DSI despite a less favourable ecosystem.

Taking this hypothesis to its logical end, places with more (or more serious) social challenges would be better "ecosystems" for DSI to grow. But, of course, we would not want to include this as an indicator in the composite index. Furthermore, country-specific contexts affect the nature of DSI in different places. For example, DSI initiatives tackling corruption and improving transparency and accountability are thriving in countries with weaker state institutions - but we would not argue that policymakers should weaken institutions to promote DSI. While this may at first seem facetious, these geographical and contextual differences have posed a real challenge in developing the framework, and one which we have not managed to completely solve thus far. * Local and national policy initiatives focused on supporting DSI may play a greater role in supporting DSI initiatives than the wider ecosystemic factors measured by the index. While there are some indicators related to policy initiatives, such as the ease of doing business and socially-focused support, the index does not take into account governmental attempts to specifically support DSI and related fields. We are cataloguing some such attempts through the Ideas Bank. The reason for the discrepancy between the index and known activity may be because those specific policy initiatives are far more important for the growth of DSI than ecosystemic factors. * Data quality, availability and accuracy. As we had predicted from the beginning, we have faced several challenges regarding the quality, availability and accuracy of data and have had to make difficult - and ultimately subjective - decisions on data sources, including using proxy indicators, outdated data or geographically broad data, or even omitting indicators altogether. This is an unavoidable consequence of developing an experimental index for a field which receives limited attention from government and academia. It is possible that some of the trade-offs we have made have influenced the scores. * Decisions in the indicators and themes included in the theoretical framework. We based our decisions for the indicators and themes on a broad range of evidence: literature review, interviews and a survey. These involved experts and the wider DSI community and were developed in consultation with all the DSI4EU partners. It is nevertheless possible that some of these decisions regarding indicator choices and weighting are not as accurate or representative as we would like. While we are confident in our research methods, we think it is important to note that such decisions were far from being clear-cut. * Incomplete mapping of DSI activity. We know that the DSI4EU database is neither comprehensive nor truly representative: it is largely crowdsourced; it is biased towards regions where DSI4EU partners have knowledge, networks and linguistic access and to the social areas they work in. Indeed, DSI is such a broad, active and fast-developing field that we could not possibly map all activity accurately. It is possible, therefore, that there is significant activity going on in some of the higher-ranking cities which we are not aware of, and we intend to explore those places in more depth over the coming months. * The diversity of DSI as a field means a traditional index is simply not sufficient to measure ecosystemic factors which support it. The breadth of activity encompassed under the term DSI is vast: different actors (researchers, user-innovators, entrepreneurs, charities, start-ups, corporates); different social areas; reactive (e.g. an individual addressing a specific problem for them or their community) or proactive (e.g. a startup tackling an international development issue); localised or global; and much more. Most indexes focus on much more clearly-defined fields. It may be that the usual method of creating indexes cannot be applied to DSI as a whole.

A number of additional challenges arose when discussing the index and potential indicators with interviewees and roundtable participants. While we had anticipated some of these, as we continued to develop the framework we continued to come up against tricky questions. While this made it all the more important for our index methodology to be as inclusive as possible, we are aware that it is by no means definitive, and far from perfect. Largely, these challenges have resulted from the huge breadth of the field of DSI, and we outline a few specific challenges below.

• Stages of development. Several experts pointed out that the specific needs of DSI initiatives differ considerably depending on their stage of development.³⁵ For example, availability of advice and mentoring on running a social purpose initiative is likely to be much more important to early stage DSI initiatives than scaling DSI initiatives, while public procurement might be very important to an established initiative but of little interest to new initiatives.
• Breadth of organisational types. For-profit organisations will have different requirements to not-for-profit organisations, an example being that they will be more likely to be interested in selling equity and less reliant on grants.
• Breadth of social challenges. DSI is active in a vast range of social challenges - healthcare, education, transport, housing, justice, environment, democracy and migration to name only a few. Of course, each of these challenges is itself vast. The enabling and hindering factors for DSI initiatives varies enormously and it would be impossible to cater to all of these within a composite index.
• Breadth of technologies. DSI encompasses a huge range of technologies, so we have faced challenges with some infrastructural indicators. Open data and digital fabrication tools, for example, are central to the success of some DSI initiatives, but completely irrelevant for others. However, we have decided to include them as indicators and allow the survey findings to suggest how indicators like these should be weighted.

One possible way of addressing the four challenges above would be to effectively produce multiple indexes, one for each different type of DSI initiative (i.e. based on their stage of development, organisational type, social challenge they are addressing and technology they are using), by using the same indicators for each index but weighting them differently depending on characteristics of the DSI initiative. This would be presented using an online user interface allowing policymakers to pick and choose the different characteristics (e.g. sector, organisation type or development stage) they were interested in seeing an index and ranking for. While this approach was impossible within the scope of this project, it should be considered if another iteration of the index is created in the future. Furthermore, as the index is open-source and its methodology available to all, specific sectors or fields could tailor it to their needs if they desired.

We welcome questions or feedback on any aspect of the Index at [email protected]. Also please visit the DSI4EU website for more research and data on digital social innovation in Europe.

6. Appendices

Appendix 1. Selected literature review

Table 12. Literature review

Appendix 2. Keywords used in analyses

'technology for good', 'tech for good', 'techforgood', 'tech4good', 'civic tech', 'civictech', 'civic technology', 'digital social innovation', 'non-profit tech', 'non-profit technology', 'nonprofit tech', 'nonprofit technology', 'Digital democracy', 'Démocratie Numérique', 'democrazia 'digitale', 'digitale Demokratie', 'democracia digital', 'digitalt demokrati', 'ψηφιακή δημοκρατία', 'digitale democratie', 'digitálna demokracia', 'democrația digitală', 'Democracia virtual', 'digitalna demokracija', 'interneta demokrātija', 'skaitmenine demokratija', 'digitális demokrácia', 'digitální demokracie', 'цифрова демокрация', 'digital demokrati', 'digitaalseks demokraatiaks', 'demokrazija diġitali', 'cyfrowa demokracja', 'elektronička demokracija', 'daonlathas digiteach', 'e-democracy', 'e-démocratie', 'democrazia elettronica', 'e-Demokratie', 'democracia electrónica', 'Elektroninen demokratia', 'e-demokrati', 'Ηλεκτρονική δημοκρατία', 'e-democratie', 'e-demokracia', 'e- democrația', 'e-democracia', 'e-demokracija', 'E-demokrātija', 'e-demokratija', 'e- demokrácia', 'e-demokracie', 'е-демокрация', 'E-demokrati', 'e-demokraatia', 'e- demokrazija', 'e-demokracja', 'E-demokracija', 'e-daonlathas', 'open data', 'Données ouvertes', 'dati aperti', 'Offene Daten', 'Datos abiertos ', 'Avoin data', 'Åbn data', 'ανοιχτά δεδομένα', 'otvorené dáta', 'date deschise', 'dados abertos', 'odprti podatki', 'atveriet datus', 'atviri duomenys', 'nyitott adatok', 'Otevřená data', 'отворени данни', 'Öppna data', 'Avaandmed', 'data miftuħa', 'otwarte dane', 'otvoreni podaci', 'sonraí oscailte', 'open hardware', 'Matériel ouvert', 'hardware libero', 'offene Hardware', 'hardware libre', 'Avoin laitteisto', 'åben hardware', 'ανοιχτό υλικό', 'otvorený hardvér', 'Open Hardware', 'sursa deschisa', 'hardware livre', 'odprta strojna oprema', 'atvērt aparatūru', 'atvira technine iranga', 'nyitott hardver', 'otevřený hardware', 'отворен хардуер', 'Öppen hårdvara', 'avatud riistvara', 'hardwer miftuħ', 'open hardware', 'otvoreni hardver', 'crua-earraí oscailte', 'code source ouvert', 'fonte aperta', 'offene Quelle', 'código abierto ', 'Avoin lähdekoodi', 'åben kildekode', 'Ανοικτός Κώδικας', 'sursă deschisă', 'código aberto', 'Odprta koda', 'Atvērtais pirmkods', 'atviras resursas', 'nyílt forráskód', 'отворен код', 'Öppen källkod', 'Avatud lähtekood', 'sors miftuħ', 'otwarte źródło', 'Otvoreni kod', 'Foinse oscailte', 'open government', 'gouvernement ouvert ', 'governoaperto', 'offene Regierung', 'gobierno abierto', 'avoin hallitus', 'åben regering', 'ανοικτή κυβέρνηση', 'open overheid', 'otvorená vláda', 'guvern deschis', 'governo aberto', 'odprta vlada', 'atvērta valdība', 'atvira vyriausybė', 'nyitott kormány', 'otevřená vláda', 'отворено', 'правителство', 'öppen regering', 'avatud valitsus', 'gvern miftuħ', 'otwarty rząd', 'otvorena vlada', 'rialtas oscailte'.

Additional words used in Twitter analysis: tech for good, technology for good,

techforgood, #tech4good, civic tech, #civictech, Civic technology, digital social

innovation, #NGOtech, #ngotech, #nptech, #nonprofittech, Non-profit tech/ nonprofit tech, Non-profit technology/ nonprofit technology, #socialtech,

goodtech

Table 13. Keywords used in analyses with translations into English

Appendix 3. PCA loadings

Table 14. Indicator loadings for each of the 7 principle components. Loadings of more than 0.15 are highlighted green and less than -0.15 are highlighted red.