Collecting evidence on the impact of a digital intervention is vital to developing radical, innovative solutions to the problems facing public services today. The benefits of grounding the development of new service delivery tools in rigorous evidence should be obvious: being able to demonstrate an innovation works points to its potential longer-term impact and will increase the likelihood it will be taken to scale.

Demonstrating impact can be difficult during the prototyping phase though, partly because ideas are new and evolving, and partly because the nature of the work is often relationship-based and therefore harder to measure than a transactional service. However, developing measures and methods of data collection at this point, which can be qualitative or quantitative, will help project teams learn about their users, understand whether an approach is really having an impact, and inform the future development of a tool. 

The team from Future You, for example, worked with the University of Sussex and New Philanthropy Capital to develop evaluation tools for their mentor training programme. This gives the team a helpful and robust longitudinal perspective of the effectiveness of the support that is provided, which then feeds into the ongoing development of the platform.

Alongside traditional impact measures, collaborative technologies also capture new types of information, which offer new insights that previously would have been unavailable to a service that can be used to improve the effectiveness of an intervention. 

The behavioural data collected by Buddi and Buddy help users identify patterns of behaviour that are either having a detrimental impact on their mental health or might increase the likelihood someone will re-offend; while by moving online Person to Person has improved the flexibility of the Hertfordshire's volunteer service because it is easier for users and volunteers to identify slots that are convenient for them.

To increase the likelihood digital innovations achieve impact, it is important to target service areas that are likely to be receptive to integrating new digital approaches. The right points in the system are those that either already use open technologies, or at least would be conducive to the introduction of open technologies. It is also important these tools can be easily accessed by practitioners and users.

For example, both Future You and FLiP provide support to vulnerable young people who are not in employment, education or training, and, following the government's decision to end funding for Connexions, have sought to fill the gap that will be left by the service. Buddy meanwhile initially focused on Community Mental Health Teams, but the IT systems used here meant it was hard to integrate new technologies. As a result, they shifted their focus to Improved Access to Psychological Therapies teams where systems are open and use interoperable technologies. 

Taking steps to understand and increase the impact of a digital innovation will inevitably raise questions about the economic implications of using these tools. Service providers could potentially achieve important economies of scale, because moving support online removes some of the restrictions faced by traditional delivery models and means services can be provided for a fraction of the cost, or can achieve savings because processes are more efficient.

Taking steps at an early stage to understand this picture could service the innovation well in the longer term. Specifically, service providers need to identify what inputs go into providing a service, how much these cost and what outputs services expect to achieve as a result. These can then be compared against estimates of the same measures for the new service model to understand whether the innovation will save money and deliver improved outcomes in the longer term.

Beat Bullying did this and estimate that it costs a local authority around £2 million to run a Connexions service, whereas the cost of licensing Future You starts at £40,000 a year. Similarly, Hertfordshire Police estimate that the time and resources that Buddi saves for officers is up to £50k per investigation. When these financial savings are matched by improved social outcomes - 20 per cent of 14-25 year olds who have used Future You have gone on to access an education, employment or training opportunity, while 64 per cent of offenders who have worn the Buddi tag have not re-offended - the argument for adopting these new tools suddenly becomes much more persuasive. 

By collecting a variety of evidence at this early stage, the goal is not to gain definitive answers but instead to test the fundamental business hypothesis and clarify at a basic level that a sustainable approach can be built around a new product or service. Or let's think of it another way: applying collaborative technologies to public services should not be about the kudos of being the first to develop a new digital tool, but instead should be about the kudos of being best to use these technologies. So validating learning to develop a business case that stakes up is crucial, even at a very early stage.