In a previous blog, I invited feedback on ten principles for a Code of Standards for public sector use of AI in algorithmic decision making.
Getting the right approach to AI in a broader sense appears to be of rising importance for many individuals and organisations, most recently with Google’s CEO Sundar Picha publishing a set of principles for the company. I’ve chosen to focus on algorithmic decision making specifically as I believe it’s one of the applications of AI that is likely to pose the greatest potential benefits and risks for public sector organisations.
In this article, I want to recap why I think some form of standards or guidance is needed; explore the feedback I received; and then outline how and why I've converted them into 20 Questions that public sector organisations should be able to answer before using algorithmic decision making in a live environment.
Public sector officials have been making decisions and taking actions based on simple algorithms - following written rules or the outputs from basic software - for years.
Why the need for any new standards?
To my mind (and many others’), the added sophistication and complexity that comes with using Artificial Intelligence to create and train those algorithms changes the game.
Three broad categories of concern arise:
First, there are concerns about the inevitable limitations and biases of algorithms:
Second, there are concerns about the way algorithms might be used:
Third, there are concerns about algorithms’ opacity:
If these concerns are even slightly correct, I’d suggest some basic conclusions can be drawn about the use of algorithms in informing decisions or taking actions in a public sector context:
Given all these concerns, it’s worth pausing to ask why the use of AI for algorithm-informed decision making is desirable, and hence worth our collective effort to think through and get right. I think there are at least three good reasons. If used thoughtfully and with care, algorithms could:
And so to your feedback. I can’t hope to cover all the points that were made in just one blog, but below I discuss some of those that were made most often and which stood out to me.
What was striking about the comments was their polarised nature. Some claimed the principles did not got far enough; others claimed the code would stop any public sector organisation trying anything with AI.
Some of those offering the latter view said the code was like the requirement to wave a red flag in front of the first automobiles - shorthand for an unnecessary and overly-restrictive regulation.
Let’s consider this comparison for a moment.
In the literal case of the person waving a flag in front of a car, I’d argue that rather than being an example of excessive regulation, it supports the rationale for codes for new technologies.
When cars were first invented, people were understandably uncertain about their safety, and so measures were put in place to counter the perceived risks until drivers gained confidence, and pedestrians felt that the risk presented was acceptable.
The flag waving was never going to be a permanent feature. It has been replaced by hundreds of rules and regulations that ensure safe driving conditions; proper vehicle functioning; and much else besides. The point is that codes may initially take a cautious approach, but are necessary until we build trust and confidence in the new technology.
I feel much the same applies to public sector use of AI in algorithmic decision making. I fully expect our attitudes and codes of practice to evolve over time. But if we make a major blunder early on, we risk undermining confidence and setting back its beneficial uses by years.
Given the public sector’s track record of misjudging public opinion on previous data and technology initiatives (think of ID cards; Care.Data, etc.) I don’t think this concern is unwarranted.
Others wondered why we should be expecting higher standards of algorithms than we do of human processes and existing forms of decision making.
Tom Forth, for example, pointed out that some of the principles would be equally reasonable requests of ‘non-algorithmic decision making’. He rephrased principles 1 and 2 to illustrate his point:
It’s a good and fair challenge.
Certainly we might want to look at setting higher standards than currently exist for more traditional forms of decision making - and we could reasonably ask many of the same questions that we suggest posing of algorithms.
However, I think a case can be made that algorithm-informed decision making is relevantly different, and hence requires a new approach, albeit one that needs to evolve over time.
That’s for four main reasons.
First (to repeat a point made above), with new technologies we may need to set a higher bar initially in order to build confidence and test the real risks and benefits before we adopt a more relaxed approach. Put simply, we need time to see in what ways using AI is, in fact, the same or different to traditional decision making processes.
The second concerns accountability. For reasons that may not be entirely rational, we tend to prefer a human-made decision. The process that a person follows in their head may be flawed and biased, but we feel we have a point of accountability and recourse which does not exist (at least not automatically) with a machine.
The third is that some forms of algorithmic decision making could end up being truly game-changing in terms of the complexity of the decision making process. Just as some financial analysts eventually failed to understand the CDOs they had collectively created before 2008, it might be too hard to trace back how a given decision was reached when unlimited amounts of data contribute to its output.
The fourth is the potential scale at which decisions could be deployed. One of the chief benefits of technology is its ability to roll out solutions at massive scale. By the same trait it can also cause damage at scale.
Taking on board the point that we need to get the right balance between standards that offer protection, whilst also helping organisations to innovate, I’ve adapted, refined and rephrased the principles into a set of questions.
These questions incorporate many of the other numerous points of feedback I received, which seem better geared towards featuring in a more practical tool.
The questions are those that any public sector organisation should ask themselves before deploying an algorithm in a live environment, whether in-house or from an outsourced provider. They are intended to ensure organisations have thought through:
1) The algorithm’s purpose and appropriate circumstances of use
2) The outcomes the algorithm intended to make possible (and whether they are ethical)
3) The algorithm’s function
4) The algorithm’s limitations and biases
5) The actions that will be taken to mitigate the algorithm’s limitations and biases; and
6) The layer of accountability and transparency that will be put in place around it.
Organisations must use their own professional judgement - and the requirements of relevant legal frameworks - to decide how detailed their responses need to be, what ethical models are most appropriate, and who needs to see their answers, based on the specific context in which the algorithm will be used.
However, I believe our standard should be this: that public sector organisations should not use algorithmic decision making in a live environment if they cannot sufficiently answer one or more of the 20 questions outlined at the link below.
Do you agree?
I invite you check out the questions, and offer your feedback by commenting in the document or via Twitter.