Metrics for government reform

How do you measure a programme of government reform? What counts as evidence that it's working or not? I've been asked this question many times, so this very brief note suggests some simple answers - mainly prompted by seeing a few writings on this question which I thought confused some basic points.

Any type of reform programme will combine elements at very different levels. These may include:

  • A new device - for example, adjusting the wording in an official letter or a call centre script to see what impact this has on such things as tax compliance.
  • A new kind of action - for example a new way of teaching maths in schools, treating patients with diabetes, handling prison leavers.
  • A new kind of policy - for example opening up planning processes to more local input; making welfare payments more conditional.
  • A new strategy - for example a scheme to cut carbon in cities, combining retrofitting of housing with promoting bicycle use; or a strategy for public health.
  • A new approach to strategy - for example making more use of foresight, scenarios or big data.
  • A new approach to governance - for example bringing hitherto excluded groups into political debate and decision-making.

This rough list hopefully shows just how different these levels are in their nature. Generally as we go down the list the following things rise:

  • The number of variables and the complexity of the processes involved
  • The timescales over which any judgements can be made
  • The difficultness involved in making judgements about causation
  • The importance of qualitative relative to quantitative assessment

In judging a programme of reform it's therefore vital not to mistake what is being assessed. So in the earlier examples - new devices or actions - evaluation can be very rigorously quantitative and often fast. There is no excuse for not measuring impacts; not using plenty of control groups; not feeding results fast into decision-making. 

For the later examples - that are more strategic - all of these tools for assessment risk becoming vices rather than virtues. Then you need complex judgement; a sense of history and context; and above all, the time and space to interrogate those judgements. The conversations prompted are likely to be as useful as any written outputs.

Author

Geoff Mulgan

Geoff Mulgan

Geoff Mulgan

Chief Executive Officer

Geoff Mulgan was Chief Executive of Nesta from 2011-2019.

View profile