Big data

Barriers to Data Innovation

Louise Marston - 17.05.2012

Big Data is a great buzzword - but how many are really innovating with data - and what's stopping those who aren't?

We held a great event at Nesta last week on the self-hacking movement: measuring and recording personal data, and using it for self-improvement. This area includes fitness apps and devices like RunKeeper, Nike+ and Fitbit, as well as happiness and wellbeing apps like Mappiness and Moodscope.

You can read more about the event here, and you can also replay a video of the event.

This represents yet another field of personal data or big data, and another community that is bumping into some of the same issues and barriers when it comes to dealing with data. Whether you're an individual, a company, a government organisation, a startup, a charity or a think-tank, some version of all of these issues comes up when you deal with data and innovation.

I'm starting to characterise these challenges in four ways:

• Privacy: what can and should we do? How do we know where the line is? How secure is it? What are the legal duties and what will people accept? How do you balance usefulness with privacy? What should we make public and what should stay private?
• Value: how can we demonstrate the potential of data to a sceptical audience? how can we measure the return from using it? How much time or money should be invested in collecting and analysing data?
• Action: what is effective in moving from data to useful information to taking action? Is it easier to create change with a single number, or with a panel of metrics? What do we know from management metrics, behavioural economics and gaming that could be brought into other types of data design? How can data-driven decision making create disruption (e.g. Moneyball)
• Fusion: how can we bring data from different sources together in one place? Which tools should we use to get a better view across multiple data sources? How can public/open data be combined with private/corporate data to enhance the information? How can data users submit corrections to improve central data sources?

I think that these four basic areas can be mapped onto a huge range of contexts to help outline the possible challenges. They are faced by businesses, public bodies, charities - anyone trying to do more with data and convince others of the value of data innovation.
And in many cases, the solutions will need to be shared back across sectors and disciplines, to propagate best practice.

Do you recognise these barriers and challenges? Do you have any good ideas for overcoming them? I'll be looking at some of them in the next few months, and I'd like to talk to as many people as possible about them.