For the last 18 months, Nesta’s Innovation Skills team has been working with leading innovation practitioners to develop a new approach to impact assessment for cultural change and capacity building in government.
Innovation teams and labs around the world are increasingly being tasked with building capacity and contributing to cultural change in government. There’s also an increasing recognition that we need to go beyond projects or single structures and make innovation become a part of the way governments operate more broadly.
However, there is a significant gap in our understanding of what “cultural change” or better “capacity” actually means.
At the same time, most innovation labs and teams are still being held to account in ways that don’t productively support this work. There is a lack of useful ways to measure outcomes, as opposed to outputs (for example, being asked to account for the number of workshops, rather than the increased capacity or impact that these workshops led to).
Consequently, we need a more developed awareness and understanding of what the signs of success look like, and what the intermediary outcomes (and measures) are in order to create a shift in accountability and better support ongoing capacity building.
One of the goals of States of Change, the collective we initiated last year to build this capability and culture, is to proactively address the common challenges that innovation practitioners face again and again. The field of public innovation is still emerging and evolving, and so our aim is to inspire action through practice-oriented, collaborative R&D activities and to develop the field based on practice rather than theory.
With this as our core premise, over the past 18 months we’ve been collaborating with innovation practitioners from across the world to develop and test an impact framework for cultural change in government that tackles some of the issues above. Multiple leading innovation organisations and practitioners have been involved in this process – including Laboratorio de Gobierno, MaRS Solutions Lab, Lab para la Ciudad, La 27e Region, MOM Design Lab (Singapore), Snowcone & Haystack, FutureGov, Public Policy Lab, UK Policy Lab, InWithForward, the MBRCGI (UAE government), UsCreates, Pides, OPM Lab, Y-Lab and the Colombian Department of National Planning’s innovation team EiP.
In the rest of this blog, we share the draft framework we’ve created, the process for how we did so, and our thoughts on continuing challenges and next steps.
The development process began at a practitioner gathering in London in September 2016. This was the first of multiple off- and online sessions that have focused on exploring and testing new ways of enabling strategic impact planning and assessment for cultural change in government.
At this first stage, we wanted to focus on 1) framing and developing the right focus, 2) exploring the potential use-value and 3) and testing possible approaches and tools relevant to this context.
Consequently, we asked the questions:
We focused on prototyping various versions of what an impact framework might look like and explored its use-value. A selected prototype (see below here) was then chosen as the practical starting point for the learning collective.
This led to a next phase of testing the framework in different innovation teams and labs around the world to qualify the content, format and design principles. Following this, a new version was developed and used to analyse, develop and communicate impact stories of selected projects and their overall theories of change connected to building the capacity of government organisations.
These impact stories were presented at a gathering in New York City in the Summer of 2017 to enable peer-learning about how the framework was useful to understand and support impact assessment in very different contexts.
The insights gathered through the impact stories are now informing a series of parallel experiments around the world to test the framework’s value-contribution to creating theories of change, accountability management and programme assessment. Beyond this, the Nesta Skills team is using the framework to assess and measure the development of organisational readiness and capacity development in the States of Change programme.
The main output produced so far is the impact framework illustrated below. It consists of a range of levels in which to observe cultural change and capacity developments (individual, team, organisation and ecosystem), and combines these with overall elements that should be considered when building (and assessing) innovation capacity (attitudes, abilities, behaviour, discourse, roles, relationships, environment, outputs and ripple effects).
Importantly, the framework focuses on spotting, mapping and assessing behaviour change. It is not about filling every box; rather, the framework enables a constructive dialogue about the sorts of areas we should prioritise and develop over time, and focuses on tracking intermediary outcomes. In this way, it is meant to support the ongoing customisation and design of your own contextual approach to impact assessment, and can develop over time.
At this early stage in the development process, the framework has mainly been applied as:
While the overall methodology for the impact assessment approach is still being developed, we are learning about what combination of methodologies will be part of it. For now, it is safe to say that we will build on developmental evaluation approaches.
In this light, it is worth stressing (at least) five overall principles that underpin the work of assessing the impact of innovation teams and labs:
Since innovation labs and teams are by default working to challenge existing ways of working in government, there are significant challenges and limitations to how much existing impact schemes can be influenced at this stage.
Here are three selected key attention points of focus going forward:
We see a significant lack of clarity about the purpose of impact assessment approaches. It is crucial to be clearer on whether it is about overall accountability (theory of change), documentation (tracking attributable impact and generating intelligence), justification (of resources spent and strategic directions chosen) or learning (for improving practices, strategies, processes, etc.).
A lot of capacity-building and cultural change in government implies unlearning existing ways of thinking and working, so how do we best deal with this premise? Focusing on behaviour change - the transition from traditional habits, mindsets and procedures to embracing and embedding new ones - is not a smooth process of abandoning one set and adopting an alternative set, making it a challenge to understand and assess.
Attempting to embed innovation processes and structures involves a large range of trade-offs that will need to be considered when shaping your approach to impact assessment for cultural change. Selected trade-offs include:
Our immediate next steps will be to, as part of the States of Change initiative, continue to develop a dedicated suite of practical tools based on the framework to deal with impact assessment and evaluation for cultural change.
Alongside this, we will continue to facilitate experimental user-testing with innovation teams and labs focused on redesigning their approaches to assessment and evaluation (among others Y-Lab, Health Lab and the EiP team in Colombia). We will also communicate some of the impact stories developed using the framework to enable a wider discussion about the future of impact assessment for innovation in government.
We are very interested to engage with interested fellow practitioners facing similar challenges so please get in contact to hear more and explore potential synergies ([email protected]). For more information, please also see states-of-change.org.