How data can help councils provide more personalised, effective and efficient services.
How data can help councils provide more personalised, effective and efficient services.
Key findings
- After years of hype, councils are now using big and small data in a range of innovative ways to improve decision making and inform public service transformation.
- Emerging trends in the use of data include predictive algorithms, mass data integration across the public sector, open data and smart cities.
- We identify seven ways in which councils can get more value from the data they hold.
What does the data revolution mean for councils and local public services? Local government collects huge amounts of data, about everything from waste collection to procurement processes to care services for some of the most vulnerable people in society. By using council data better, is there potential to make these services more personalised, more effective and more efficient?
Nesta’s Local Datavores research programme aims to answer this question. Where, how and to what extent can better data use can help councils to achieve their strategic objectives? This report is the first in a series, aimed primarily at helping local public sector staff, from senior commissioners through to frontline professionals, get more value from the data they hold.
Author
Tom Symons
ACKNOWLEDGEMENTS
We would like to thank SAP for their support for this project, in particular Chris Francis. We would also like to thank all those who gave up their time for interviews and to help with our research: Sarah Henry (Manchester City Council), Katherine Rooney (Bristol City Council), Jamie Whyte (Trafford Borough Council), Emma Prest (Datakind), Lora Armstrong (Greater London Authority), Hendrick Grothius (Cambridgeshire County Council), Niamh Walsh (London Borough of Islington) Steve Nicholas (London Borough of Newham), Gesche Schmid (Local Government Association), Wajid Shafiq (Xantura Consulting), Dan Miodovnik (Social Finance), Lauren Haynes (Data Science for Social Good), Mark Kleinman (GLA), Jen Hawes-Hewitt (Accenture), Duncan Ross (Datakind), Oliver Buckley (Cabinet Office), Lisa Clark (North Tyneside Council), Eddie Copeland (Nesta), Julie Simon (Nesta). Any errors and omissions remain the responsibility of the author.
SAP Run Simple to Improve People’s Lives
Nesta is an innovation charity with a mission to help people and organisations bring great ideas to life.
We are dedicated to supporting ideas that can help improve all our lives, with activities ranging from early-stage investment to in-depth research and practical programmes.
Nesta is a registered charity in England and Wales with company number 7706036 and charity number 1144091. Registered as a charity in Scotland number SC042833. Registered office: 1 Plough Place, London, EC4A 1DE.
www.nesta.org.uk © Nesta 2016
EXECUTIVE SUMMARY
The opportunities to improve local government through data
Data science is increasingly influential in shaping the world we live in, the choices we make and the quality of goods and services we access. Data analytics now influences the personalised ads and product recommendations we see on websites, the satellite navigations in our cars and on our smart phones, and even the creation of new TV programmes, such as Netflix's House of Cards.
But what does the data revolution mean for councils and local public services? Local government collects huge amounts of data, about everything from waste collection to procurement processes to care services for some of the most vulnerable people in society. By using council data better, is there potential to make these services more personalised, more effective and more efficient?
Nesta's Local Datavores research programme aims to answer this question. Where, how and to what extent can better data use help councils to achieve their strategic objectives? This report is the first in a series, aimed primarily at helping local public sector staff, from senior commissioners through to frontline professionals, get more value from the data they hold. In this report we set out the findings from a period of preliminary research conducted in the Spring of 2016 about how the use of data is changing how local government works, the most impactful use cases emerging in the UK and around the world, and the critical factors required for these projects to be successful.
All Data Big and Small
Powerful big data analytical tools, Internet of Things (IOT) technologies, sensors and new methods of collecting data tend to attract much of the attention about the ways data can transform councils. But in our research we saw that councils are starting to find significant impact simply through better data analysis, often of data they have held for years. Councils are finding that for now, there is as much to gain from small data as there is from big data.
We saw five emerging trends in data use in local government:
- Predictive government - governments are using data analytics to predict events from potential child abuse, to the likeliest locations for house fires and the school children most at risk of not completing their education. These insights equip local governments with more ability to take a preventative approach, putting in place interventions to try and stop problems rather than providing costly services in response.
- Integrated data - through data warehousing, councils are combining data sets from across local government and the wider local public sector to enable deeper population level analysis, and to provide frontline professionals with a much more comprehensive picture of people receiving services. Such datasets can be enablers of the historically challenging objective of partnership working across public services.
- Smart places - in some councils, the combination of sensors, Internet of Things technologies and data are improving traffic management, tracking air pollution and making more efficient use of infrastructure such as street lights. These councils are also starting to take a citizen-centric approach to smart cities, collecting data from citizens to better understand how a council can use their resources in a way which reflects the ways in which people navigate and experience places.
- Geo-spatial analytics - Councils have made considerable use of geo-spatial data to improve services, such as optimising waste collection routes and reducing inefficiency and duplication in transactional services. This is one of the most established areas of data analytics in local government, with studies finding a cost-benefit ratio of a £4 return for every £1 spent[^1] on the use of geospatial data.
- Open data - through open data portals and analytics hubs, councils are becoming more transparent and better engaged with their residents and communities. Such communities include developers, entrepreneurs and innovators who are able to use open data to create businesses, products and services, such as apps like Citymapper or the startup firm Spend Network. Alongside finding solutions for public or social problems, this is an important source of local economic growth.
Following the significant amount of hype that accompanied the emergence of big data nearly a decade ago, which failed to make real impact in government, these emerging trends represent a possible turning point for the sector. But while some councils are breaking new ground, many councils are struggling to understand how they can use data to help them with their most immediate challenges. Our research aims to use learning from the most impactful or innovative programmes to help councils get more from their data.
Seven things councils can do to make more of their data
In looking at these emerging use cases for council data, we saw some of the factors which enable data use to lead to tangible improvements. Based on this, there are seven things councils can do to get more out of the data they hold:
- Take a problem oriented mindset to working with data - data is not in and of itself useful, but using data analysis to test hypotheses or solve problems can ensure that value is created.
- Integrate data into a data warehouse to enable deeper analysis and use - linking together data creates a fuller view of issues or individuals, making problem solving or pattern spotting easier.
- Enable data sharing through use of case oriented information governance protocols - being specific about the circumstances and purposes for which data can be shared makes it easier to unlock data and integrate it.
- Support the use of data from the top - senior managers and politicians can create a data-oriented culture through asking for data and analysis as part of decision and policymaking processes.
- Invest in the data science capacity needed to perform analysis and integrate large data sets - data work increasingly requires data scientists and programmers who are currently rare in the local government workforce. Successful projects require investment in these skill-sets, either from inside or outside the organisation.
- Take an agile approach to working with data - rapid prototyping, testing and iteration improves the quality of analysis and tools, and helps build momentum.
- Ensure that hard and soft infrastructure enables integration of data and analysis - without high-speed broadband, data storage options and the right software, data approaches can be held back.
Next steps
We will be undertaking detailed case study research of councils using data in innovative and impactful ways to provide a richer understanding of the use and value cases of data, and the strategies councils can employ to get the most from the data they hold. This will culminate in a report in September 2016. Through the research programme we will be developing tools that can help councils to do more with their data, such as a data maturity framework, and a compendium of use cases. The insight gained from this work is also supporting the development of Nesta's programme of Offices of Data Analytics.
GLOSSARY
Big Data
Both large volumes of data with high levels of complexity and the analytical methods applied to them, which require more advanced techniques and technologies in order to derive meaningful information and insights in real time.[^2]
Small Data
Data which is small enough to be processed inside a single computer, using simple tools such as spreadsheet applications.
Structured data
Structured data is data which is in a traditional row-column tabular format.
Unstructured data
Unstructured data is data that needs to be cleaned and processed before analysis, or where the structure of the data is not tabular. An example of the first type would be text, and of the second, a social network.
Data Standards/Standardisation
Data Standards/Standardisation are the rules by which data are described and recorded. Sharing, exchanging, and understanding data is simplified if the format and meaning are standardised.
Middleware
Middleware is software that acts as a bridge between an operating system or database and applications, especially on a network.
APIs
APIs are a set of functions and procedures that allow the creation of applications which access the features or data of an operating system, application, or other service.
Algorithm
A self-contained step-by-step set of operations to be performed, usually by a computer.
INTRODUCTION
DEFINITION OF A LOCAL DATAVORE
Nesta developed the concept of a 'datavore' in the 2012 report Rise of the Datavores. This defined datavores as companies which "gather online customer data intensively, subject this data to sophisticated analyses (such as controlled trials and data and text mining), and use what they learn to improve their business. They also report that they are more innovative than their competitors, in products as well as processes".[^3]
In this report, we apply these principles to local authorities. While there are some important distinctions between businesses and councils, there are also many common features. For the purpose of this research programme, we define local datavores as councils which "intensively gather data about people, communities, places, businesses, council processes and services, infrastructure and the environment and subject this to sophisticated analyses. They also seek to share, integrate and use data where possible for the improvement of services and operations. They use what they learn to inform decisions about improvements to council operations, processes, services and infrastructure, and to ensure they meet the needs of their residents."
WHY LOOK AT THE WAYS COUNCILS CAN USE DATA?
Local authorities sit in the middle of a web of information. Everything from social care for vulnerable children, waste collection, procurement, council tax collection, to planning applications produces huge quantities of data. This data is sometimes garbled, hard to analyse, or personal and sensitive. But it is potentially hugely helpful in enabling councils to make services more targeted and effective, to allocate resources to where they will have the biggest impact, to save officer time in front and back office processes, and to provide insight into the causes and solutions to costly social problems.
Running a city or a local authority is to a great extent about managing and responding to information. Increasing digitisation of services, the use of sensors and other forms of data collection mean that there are emerging data sets which capture the wide variety of activities performed by councils. And while big data presents opportunities for local councils, there are equally important opportunities presented by smaller data sets already available to councils. Whether the data sets are big or small, there are major benefits to be had from using them more intelligently, sharing them more widely and making them more open.
Yet despite the hype, to date the use of data and analytics has not kept pace with this ambition. In 2013, research by Localis argued that councils were not taking full advantage of big data. In the same year, the government's Strategy for Digital Capability argued that the public sector lacked the technical skills to take advantage of the opportunities presented by sophisticated analysis of public data sets. So far, implementation of data-driven work has not supported the argument that data and analytics can unlock significant value for councils.
THE LOCAL GOVERNMENT CONTEXT
In local government there are both huge opportunities for the use of data and analytics, and also a pressing need for innovation which help councils deliver better outcomes with decreasing resources. With the Comprehensive Spending Review and the Local Government Settlement, councils are facing the twin prospects of devolution and budgetary pressures. Financial pressures are particularly acute; councils have already made significant cuts over the past five years with many council leaders now saying that there are no more efficiency savings to be made.[^4] Future cuts will therefore present councils with the choice of either radical transformation, or having to scale back or exit from certain fields. There are also broader shifts happening, with councils moving from being large organisations that provide lots of services in-house, to smaller organisations that coordinate, commission, and sometimes provide services for citizens. This shift from service provision to information and commissioning requires a different approach to the use of data.
Data and analytics may not provide the solution to all the challenges faced by local councils, but they should be part of any important decisions being made about where to save money, or the reconfiguring of services and operations. Data and analytics present local authorities with a huge opportunity; the potential to transform local government service delivery, making it more efficient, more effective and more responsive to the needs of local residents, businesses and communities. This is especially the case if data can be shared and linked across administrative boundaries, such as within the combined authorities being formed as the basis for devolved powers. Realising this potential will require councils to change the way they approach many aspects of data - from the way it is generated, stored and analysed to how it is used.
PROJECT OVERVIEW
The aim of the Local Datavores research programme is to help councils get more from the data they have. We aim to identify:
- Use- and value- cases for council data
- Strategies for overcoming common data challenges
- The critical success factors of better data use
- Tools and resources which could help councils to do more with their data, such as a data maturity framework
At the outset of the research programme, we undertook a literature review and searched for innovative approaches to local data use in the UK and overseas. We spoke to people in local authorities, people working with local authorities and experts from the UK and overseas about the major issues associated with working with data. In particular, we spoke to them about areas of opportunity and emerging use cases, challenges faced when implementing data innovation and critical success factors identified from their experience.
This report summarises these initial findings as a discussion paper. We plan to publish two further reports under this programme; a summary of our case study research and a final report.
SECTION 1
A TAXONOMY OF DATA SOURCES, METHODS AND USES IN LOCAL PUBLIC SERVICES
There are few, if any, frameworks which offer an overview of the many different types of data available to local authorities, the tools of analysis and the use-cases for data. The draft typology below is a first attempt to pull together this information for a local government context. We will be refining this taxonomy over the course of the research programme and welcome any comments or suggestions.
Figure 1: Draft typology of data available to local authorities
The diagram illustrates a feedback loop for monitoring and evaluation, connecting three main areas:
SOURCES OF DATA * Administrative/operational * Council processes * Personal * Business * Service delivery information * Web data * Sensors * Citizen-generated/crowdsourced * Partners (e.g. police, housing, charities etc.) * Commercial sources * Official statistics * Council assets * Survey (in-house or external) * Official survey * Ad hoc survey
MAKING DATA USABLE * Cleaning * Standardising * Integrating * Linking data * Using technology to access in real-time
TOOLS OF ANALYSIS * Descriptive statistics * Algorithms * Classification * Cluster analysis * Regression * Machine learning * Data visualisation
USE OF DATA * Monitoring and measuring * Understanding of events * Evaluation and testing 'what works' * Transparency and citizen engagement * Prediction (individuals, services) * Case management e.g. social care * Town planning * Optimisation of resources e.g. of traffic * Detecting fraud and error * Better targeting of resources * Automate decisions * Modelling impact of changes to services * Risk-management
SOURCES OF DATA
While big data draws much of the attention, there is also value held in smaller data sets, which currently makes up the majority of council data. And increasingly there is a view that unstructured data can also offer value, and can be generated by people as well as by public sector organisations.
The data councils have traditionally been able to use for analysis has been structured data, collected by local authorities as part of deliberate monitoring, surveys or processes. There have been far greater quantities of unstructured data, in case files or in free text responses to consultations for examples, but until recently this data could only be used if read by humans. At the same time, structured data required considerable effort to link together datasets in different formats or extracted from different IT operating systems. Now, there are resources available which can both link structured data together more easily, and which can create unstructured data from things like word documents and the web, increasing the opportunities for data analytics which can inform decision-making and improvements.
TOOLS OF ANALYSIS
Some new transactional systems come with inbuilt analytical tools, some of which are automated, making the process of analysis more straightforward. Below are some of the tools of analysis which may be built into IT systems, or which are used manually via software packages.
Descriptive Statistics
At present most local authority data analysis comprises the use of descriptive statistics. This includes:
- Basic maths and statistics, such as percentages and ratios.
- Correlations, which show the relationship between two variables.
- Cross-tabulated statistics, which enable the comparison of two variables.
Predictive Analytics
The massive increase in data available to governments, and advancing power of analytical tools, means we are starting to be able to predict events with greater accuracy, enabling better targeting of resources and prevention of social and public policy problems. These methods typically involve the use of algorithms.
Below are some examples of functions that can be achieved with algorithms. The examples are drawn from children's services where the use of algorithms is particularly promising because much of the work of commissioners or frontline professionals involves complex decision-making with lots of information. Algorithms can help in these situations by using historic data to establish patterns, and offering predictive insight for new decisions based on the presence and weight of certain variables.
Predicting the level of future risk for a child which becomes known to children's services, based on a number of observed factors. This would typically involve the use of regression, whereby data is given a real value rather than a label. The algorithm must predict values for new data, based on observations of relationships with previous data.
Identifying common groupings of needs and characteristics within the population of families known to children's social care. This would typically involve clustering, whereby data is unlabelled but can be divided into groups based on similarity or other measures of structure within the data. The algorithm tries to find the hidden structure of the data, representing patterns or groupings.
Predicting which families are most likely to respond positively to a particular intervention or support service. This would typically involve classification, whereby labelled data is used by the algorithm to guess the label to attach to new unlabelled data. The algorithm is effectively modelling the differences and similarities between groups or classes.
Machine Learning - the use of algorithms to predict outputs based on previous examples of relationships between input data and outputs (called training data). There are four different types of machine learning:[^5]
- Supervised - requires a training data set with labelled data, or data with a known output value. Classification and regression problems are solved through supervised learning.
- Unsupervised learning techniques don't use a training set and find patterns or structure in the data by themselves. Clustering problems can be solved with an unsupervised approach.
- Semi-supervised learning uses mainly unlabelled and a small amount of labelled input data. Using a small amount of labelled data can greatly increase the efficiency of unsupervised learning tasks. The model must learn the structure to organise the data as well as make predictions.
- Reinforcement learning uses input data from the environment as a stimulus for how the model should react. Feedback is not generated through a training process like supervised learning but as rewards or penalties in the environment. This type of process is used in robot control.
Data Visualisation
Data visualisation is the art of communicating and making sense of data using images. Data visualisation can be both a form of presenting data and a means of analysis, as many visualisations are interactive, enabling the viewer to interrogate data sets in novel ways and identify new insights. Data visualisation offers a way of making sense of data through visual means of coding and labelling, and with colours, shapes and movements. Data visualisation can take many forms, such as interactive dashboards, interfaces, graphs, maps and video.
Figure 2: Data visualisation of food standards in London
This interactive visualisation from Visually shows London Food Hygiene data from the Food Standards Agency on businesses based in London.
Filters: * Ratings: 5 Rating, 4 Rating, 3 Rating, 2 Rating, 1 Rating, 0 Rating (with radio button selection). * Type: Hospitals / Childcare / Care, School/college/university, Pub/bar/nightclub, Restaurant/Cafe/Canteen, Supermarkets, Takeaway / sandwich shop, Hotel / bed and breakfast/guest house (with checkbox selection).
Local Authority Break Downs: The map displays data points across London boroughs. A list of local authorities is provided with the number of data points for each (e.g., Barking and Dagenham (540), Barnet (1778), Camden (2052), Westminster (3463)).
A hygiene ratings distribution (e.g., 2 ratings shown) is also visible.
Spatial Analysis
Spatial analysis is a technique to understand the relationship between variables linked to a location and patterns in a space. Spatial analysis underpins Geographic Information Systems (GIS), such as those used for satellite navigation in cars or maps on smart phones. Some examples of spatial analysis include geo-locating of public assets or infrastructure on a map; finding the quickest routes between places, such as for waste collection services; and detecting and quantifying patterns such as the areas which have the highest prevalence of disease or poverty.[^7]
USE OF DATA
Throughout the research we identified a range of 'use cases' for data. These are specific tasks for which data and analysis can be used to make that task more straightforward, efficient or effective.
| Use Case | Examples of Use | Where? (selected examples) | |---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
