When Pulse Lab Jakarta started out in late 2012, we were pretty much the only big data shop in town. But five years later we are surrounded by a network of policy-makers, data scientists and entrepreneurs, keen to use the affordances of the data revolution for public good. This is our journey.

From supply to demand

The first couple of years felt lonely at times. Actually, quite a lot of the time. We were a big data lab looking to access new digital data sources to create proof-of-concepts, and making pitches to countless stakeholders in and out of the public sector - but not getting much back in terms of interest or uptake. In partnership with the UN, the Government of Indonesia was very progressive in hosting a data innovation lab and - alongside the Government of Australia - seriously investing in it.

However, many of our broader stakeholders didn’t realise the lengthy life cycle of a typical big data project or the demands for data access that it entails.

What the Government of Indonesia understood well, however, was the need to answer the common challenge that we all had: how do you optimise analyses and glean insights from data to make development initiatives work better for the people of Indonesia?

Over the course of a few short years, the landscape has evolved and changed dramatically, and we are seeing a more widespread realisation of how real-time data can complement existing government datasets to yield new insights needed to respond to the fast-paced, complex and dynamic development landscape that is Indonesia.

One of the main things that our team learned to do over the course of these years was to listen better

In addition to working on proof-of-concepts, we tried to develop a deeper understanding of the drivers of demand for advanced data analytics, and what the information needs were for our counterparts. We took some of our early proof-of-concepts a step further and created interactive data tools, such as the National Dashboard and Haze Gazer.

The Executive Office of the President of Indonesia and the National Ministry for Development Planning eventually adopted and developed these tools, adding additional data and functionality. What we have found most valuable from this process though, is not only how to expand proof-of-concepts into working platforms, but how to build mutual trust and understanding.

At the same time, we were starting to discover that far from being alone, there was an array of exciting development mutants in this field, such as Peta Bencana, the Urban Poor Consortium, Kota Kita and Akvo. These teams generate and use data in sophisticated ways to inform community action and policy.

The work of initiatives such as Open Government Indonesia, Open Data Lab Jakarta and Perkumpulan IDEA in advocating for better use of public data for transparency and social accountability also helped raise general awareness on the potential of combining different sources of data for new insights. Our own initial attempt to map civic innovators uncovered new, interesting forms of social activism in Indonesia, and pushed us to think of what our role would be in this landscape. Exchanging ideas and learning about the work of other actors in this nascent ecosystem helped us hone our own work and chart our path.

Blending big data with thick data

From our early forays in trying to understand the information needs and priority issues of our government counterparts, we realised that we needed to complement the Lab’s core data analytics expertise with a more in-depth understanding of how people make sense of data. In 2015, we began to bulk up our user research and service design skills in-house. We felt that we needed to provide more context and local knowledge alongside our big data tools.

Our social systems team cut its teeth on pure service design assignments in the fields of maternal health, education and entrepreneurship

But the intention was always to augment our findings from big data with qualitative insights from user research, or, in the words of Tricia Wang, to develop thick data.

In parallel, as we have integrated tools into our partners’ information systems, our assumptions have been challenged in terms of the link between the supply of evidence and action. The service design work is instrumental in helping us to understand data resistance, driven by the inertia of organisational practice. In response, we are working on methods and tools to facilitate organisational change alongside data innovation.

Building the data innovation ecosystem

Our own efforts to innovate with data are a very small part of the bigger picture. The creation of networks like Data Science Indonesia and the rich ethnographic work done by teams like the Reality Check Approach are testament to the development of this ecosystem.

Working alongside these actors, we have been playing another role as a lab in connecting the dots. Our data innovation grants with UNDP are designed to find and develop data innovators within the state and among NGOs. Alongside these, our data dives with academics and public agencies build links and help to nudge forward the research and development agenda in fields as diverse as disaster management and computational linguistics.

Identifying data privacy experts in Indonesia, and relying on their expertise in this area, has helped balance out the demand for ethical standards for the use of data, with the need to utilise new digital data sources for more agile development. We have also learned that when we combine forces with development initiatives in other sectors, our chances of tapping into the amazing potential for data innovation within Indonesia’s government units becomes much, much higher.

The rich tapestry of actors in this ecosystem came together for our recent conference on the Data Revolution for Policy Making

It was truly humbling to witness the diversity of players and see the level of engagement in the data revolution within the government.

There has been a dramatic change in the development of the data ecosystem in Indonesia, with quiet adoption by the government of Indonesia in recent years and national ownership of the data revolution well and truly established. What is also encouraging is the level at which the data revolution has permeated down through the echelons of bureaucracy. Success stories are springing up all across the Indonesian archipelago, where local governments are using technology and new sources of data to strengthen their relationship with their constituents.

Where to from here?

The dynamics of this landscape makes exploring the nexus between big data and thick data pretty exciting work, which we will continue to pursue as a research area.

Using the lessons learned (and the hard knocks received) over the past few years, we will also continue to refine methods to overcome data resistance within different organisational cultures. Immediate demand exists to share our data tools and methodological approaches with partners in other parts of the Asia-Pacific region.

We’re also working on better ways of measuring the impact of our work - and how we can actively learn from both our successes and our failures to make better decisions and produce better products.

It will be very interesting to turn our user research and data analytics lenses inward, for once.

Part of this work towards measuring impact has triggered some existential soul-searching on the future form of Pulse Lab Jakarta. We know that we need to plan the next stage of our operational model, moving from a donor-funded lab to a more sustainable model.

More on this soon - but, thanks to the vibrant and growing data for good community in Indonesia, whatever we become, it’s good to know that on this journey we are never alone.

Pulse Lab Jakarta is grateful for the generous support of the Government of Australia.