Acknowledgements

We would like to thank the following for their insight and assistance with the report:

Geoff Huggins, Dr Barbara Barry, Dr Harpreet Sood, Dr Thomas Hope, Dr Jeremy Wyatt, Will Cavendish, Saurabh Johri, Ben Glocker, Ben Hughes, Chris Tyson, Effy Vayena, Helen Munn, Indra Joshi, Alastair Pickering, Mark Hellaby, Tim Hubbard, Tom Dewar, and Zoe Camper.

And our colleagues from Nesta, Harry Armstrong, Halima Khan, and Celia Hannon.

About Nesta

Nesta is a global innovation foundation. We back new ideas to tackle the big challenges of our time.

We use our knowledge, networks, funding and skills - working in partnership with others, including governments, businesses and charities. We are a UK charity but work all over the world, supported by a financial endowment.

To find out more visit www.nesta.org.uk

About Nesta Health Lab

Nesta Health Lab is committed to being a centre of expertise on people-powered and data-driven health: we work with partners from the health, care, voluntary, community and social enterprise sectors to test and scale new ways for people to remain healthy.

Our work focuses on new sources of support, which make it possible for people to be more involved in their health, such as peer support; new sources of data, which improve people's knowledge about their health, such as citizen-generated smartphone data; and new sources of innovation, that generate new solutions, including our 100 day People Powered Results method for transforming systems.

Over the past few years, we have backed over 100 local health systems and individual organisations with more than £20 million of funding.

www.nesta.org.uk/health-lab

Executive summary

Artificial Intelligence could become part of the front door to healthcare. It could make the health system simpler, more accessible, more responsive, more sustainable, and put patients more in control. But there's a risk that the public could experience it more as a barrier than an open door, blocking access to care, offering opaque advice and dehumanising healthcare in every sense. We're now at a crucial moment when decisions are being made which will determine whether the technology develops into People Powered AΙ.

Artificial intelligence (AI) looks like it could be one of the transformative technologies of our era. Healthcare is rich in the data that Al thrives on, and in the kinds of questions that it can tackle. While the use of Al in healthcare is at an earlier stage than the hyperbole surrounding the technology might suggest, it is developing at pace, and this raises both significant opportunities and risks.

Al has delivered some striking results. There have been research trials that successfully use machine learning on images from, for example, radiology, dermatology and ophthalmology, to a level of accuracy that matches clinicians' own abilities. This, and other Al developments, have led to the suggestion that machines are poised take the place of doctors.

However, today's Al is narrow and not capable of the holistic thinking and complex judgement required for many clinical tasks. While there are significant areas of medicine where more narrow applications of decision-making rules and expert pattern matching predominate, the path towards Al replacing humans is not solely determined by technical capability. Technology implementation will need to address trust, accountability and similar factors. And, at the same time, humans remain especially good at certain tasks, such as learning to identify rare situations from small amounts of data.

This jump to focusing on whether or not Al could replace doctors also potentially distracts from some far more immediate and likely applications of Al in health. It is far easier for Al to be adopted where there are no or few good alternatives on offer, than in areas where humans are effective and trusted. Areas where Al could be effective, and where there are few good alternatives include:

Advice and triage before seeing a doctor

When should someone first seek help from the health service?

Proactive care

When is the right time to intervene in the face of worsening symptoms?

Automated second opinion

How does the diagnosis and treatment I am getting compare to alternative options?

There are already Al products in the market in these areas, although the evidence base is not always sufficient. If Al is adopted in these areas it would be in a hugely influential position over our health and care. This could bring great benefits, but also comes with significant risks that need to be proactively managed and mitigated.

Applications of Al

1. Advice and triage before seeing a doctor

Most people find it hard to know exactly when to seek appropriate medical help. Twenty per cent of GP¹ appointments and 19 per cent of A&E attendances² are for minor medical problems that could be treated at home. This unnecessary or preventable demand creates significant pressure on health resources.

Al is already being used to offer healthcare advice and diagnoses directly to consumers. People can buy a diagnostic app or access to a chatbot to share symptoms with, and use this advice to decide whether or not to seek further medical help. In this way Al is beginning to provide a form of triage into the healthcare system and, if developed correctly, could help solve a major issue: how to support the appropriate use of limited healthcare resources including reducing the unnecessary use of health services.

However, this use of consumer-facing Al could also generate a flood of unnecessary demand (from false positives or generally risk averse advice); a new source of error in the system (from false negatives or other mistakes); and could widen health inequalities depending on the underlying business model. This area of development for Al also creates a new and urgent regulatory challenge.

These challenges need to be tackled, however, because this looks set to be a growth market and we are increasingly likely to see the use of Al as a highly influential entry point to the healthcare system.

A likely future is one where Al is a common first point of contact for health, and a front door to the health system - a highly influential position.

2. Proactive care

Al is also becoming capable of extracting signals from real-time data and giving early warning that a health problem is getting worse. For example, by listening to the breathing sounds of those with congestive heart failure to spot signs of deterioration. This could enable help to be directed to the people who need it in a more timely way, leading to a healthcare system that is more dynamic and responsive in the way it cares for people. Or it could generate a great deal of unnecessary concern, replace individualised conversations with standardised analytics, and generate an oppressive degree of monitoring.

Al would have an influence over who gets treated and when they get treated. While this could make the system much more dynamic, it could also make it more impenetrable, more unequal and less individualised.

3. Automated second opinion

Al could sit alongside doctors and offer an opinion on the same patient, for both diagnosis and treatment. This could be used to give patients a digital second opinion, which would enable to them to challenge and advocate for their care more easily. It would also allow mass comparisons of physicians' decisions centrally, perhaps with a view to understanding variation in care better. This use of Al could have significant potential influence on power dynamics within health, as well creating additional cost pressure on the system. The evidence that Al can do this reliably is not there, but products like IBM Watson claim to come close to this sort of functionality.

Both patients and managers will find it tempting to compare diagnosis and treatment to Al-generated opinions This could be helpful if applied to the narrower questions where Al is competent, but high risk if misapplied to judgements beyond the capability of the technology or quality of underlying datasets, putting pressure on clinicians to conform to it.

Consequences for power and autonomy

Like many digital technologies, Al can either democratise or centralise, empower or disempower, depending on the way it is implemented. It is easy to see how poorly designed and executed Al would be problematic:

• Poorly designed triage and prioritising systems could make the system even harder to access, making healthcare even less simple for patients.
• Increasing use of data-hungry analytics could squeeze out dialogue, which could make it harder to surface key details about the individual not captured in datasets.
• Opaque Al could reduce accountability and transparency.
• An over-monitored patient is not helped to understand what the Al is saying and why, making it harder to have a say in their care and leading to reduced control.

Further, it is likely that those worst affected by these changes will be those with hard to diagnose conditions, complex social and health needs, and who already face disadvantage.

However, it is equally possible to imagine a future where the patient is significantly more empowered:

• Al makes it simpler to know when to seek help and get to the right person.
• Ensure patient and professionals are more prepared for and have more time for their conversation, so more dialogue.
• Patients find it easier, via home diagnostics and chatbot advice to understand their condition and to ask for help when they need it, so have increased control.
• The ability to get a digital second opinion increases transparency and accountability.

Also the clinician could be freed from a lot of low priority work, fed useful insights, and more able to intervene at the right time.

Principles for People Powered Al

These are principles that apply to any form of healthcare that aims to be humane and person-centred, but are not presently being applied to the design, development and implementation of Al.

Recommendations

Al development is being driven by private companies, who are not directly incentivised to think from the system point of view. If we sleepwalk into a situation where a small number of tech companies have already monopolised access to the data to build Al, and are selling into a health service which does not fully understand the technology they are buying, the more negative scenarios for Al become much more likely.

There is currently a window of opportunity to shape the future of Al in health. Policymakers should set rules for Al and ownership of public data that ensure the public gets not only value for any data it decides to share, and privacy elsewhere, but also Al products that deliver maximum public benefit. This requires that both the providers and users of Al understand the technology, have the tools to shape the market, can understand the needs of citizens, and are able to work through the complexities of implementation. This can be achieved through the following four recommendations:

1 Public and clinical scrutiny

Involve citizens and clinical professionals in the upstream design, development and implementation of the technology. This should include the requirement of mechanisms, such as public panels made up of citizens, that ensure technology development and implementation takes account of the demands and perspectives of citizens and healthcare professionals and ensures that People Powered Al principles are applied.

2 Controlled tests in real-world conditions

Enable real-world experimentation of Al in designated test sites, with non-Al comparators, to understand how Al works in complex systems before wider take-up 'in the wild'.

3 Proactive market design

System leaders actively engage in market design to maximise public benefit and ensure a plural market with genuine choice. This should include regulation that is upstream and proactive ('anticipatory regulation'), clarity over who owns both algorithms and data, and requiring adherence to key design principles, such as People Powered Al principles. Market design should also foster a diversity of new entrants to the market including procurement processes that work for smaller companies and market structures that support a diverse range of R&D activities.

4 Decision-makers equipped to be informed users

Create a new cadre of public leaders and decision-makers with the technical skills, authority and institutional levers to scrutinise, manage and deploy Al in a responsible way. This should include incorporating artificial intelligence into medical education and health management training to enable the frontline workforce to be informed users of the technology.

Introduction

Machines are getting smarter, and doing so quickly. They can beat humans at complex games, drive cars, and, when presented with the right data, make an accurate clinical diagnosis. Machines can successfully accomplish tasks that, only a few years ago, were the exclusive domain of human beings.

This Artificial Intelligence (Al) seems likely to be a transformative technology in general, but could be particularly significant in healthcare, which is rich in the data on which Al thrives, and the types of problems that it is able to tackle.

Al-driven healthcare has made some impressive technical achievements. A growing body of research shows that Al is capable of performing at a similar level to a doctor across a range of diagnostic tasks. There are already commercially available Al products which have a conversation with a patient and put forward a tentative diagnosis, such as bablyon³, and recommend cancer treatments to patients, such as IBM Watson

This success has lead to a degree of hype, including the suggestion that Al will rapidly come to replace doctors. In fact there remain significant constraints on what the current generation of Al can do, a lack of evidence to back up some of the claims made for Al technologies, and significant issues of accountability and trust to be overcome.

But replacing doctors is not the only way Al could be significant in health. An easier route for Al is to provide solutions where few exist, such as advice before we see a doctor, predictive and real-time analytics of health data, or a digital second opinion. Al could also pick up a significant amount of 'lower value' tasks, supporting existing clinical work.

If Al can deliver these new capabilities, then despite being less headline-grabbing than robot doctors, they could really help. For example, they could transform access to care by providing 24/7 advice and triage, and relieving pressure on an overstretched health system.

However, this does require putting Al in a very influential position. For better or worse, Al could change patients' experience of care: how care is accessed, the way services are organised, and the relationship between citizen and clinician. Whether this improves or degrades the experience of care for citizens depends on how the technology is designed and implemented. If the pace of development of Al continues, and given the amount money being invested in it, and the 'burning platform' of an overstretched health service Al could become commonplace in as little five years. As the legal, regulatory and commercial rules that govern Al are being set, we have a window of opportunity to ensure that patients are placed at the centre of the process, and that Al delivers the right outcomes for them.

Consequently, the goal of this report is to:

• Explore how Al might be used in the UK, or a similar health system.
• Explore how Al-enabled healthcare might look and feel, especially from the point of view of the citizen.
• To suggest what can be done to maximise benefits and minimise harm.

1 Al primer

In this section we define what we mean by Al, highlight the varieties of Al considered in this report, and illustrate their achievements and limitations hitherto. Those familiar with the field may wish to skip ahead to the next section.

Defining Al

In this report we use the term artificial intelligence (Al) in its colloquial, informal sense to mean computers which perform cognitive tasks usually associated with human minds, particularly learning and problem-solving.

This lack of precision stems from the fact that there is no definitive understanding of what intelligence means, and what tasks require it. It is this that creates the uncertainty about what the impact of Al might be, and is consequently a question with which we will have to grapple in this report.

In doing so, we limit ourselves to capabilities of Al that do not seem to require a big technical breakthrough; as described in Box 1 above, this excludes Al that can display the same range of abilities as humans. Instead we are looking at extensions of what Al can do at the moment.

Varieties of Al

"Al is not really any single thing - [it is] a set of rich sub-disciplines and methods, uision, perception, speech and dialogue, decisions and planning, robotics"⁵

Eric Horwitz, director of Microsoft Research Labs

The term Al covers a range of distinct approaches and abilities. In order to highlight the potential of Al we will examine three of the most important approaches, which underlie much of the leading research and products at the moment. While there are other Al technologies, these were by far the most frequently mentioned in our interviews.

• Pure Machine Learning - ML is when machines learn from examples, rather than being explicitly programmed. Rather than being, in some sense, told what a cat looks like, a machine would learn by being shown many images of cats, and told "cats are things that look like this", or learn to tell cats from dogs by viewing pictures of them both.
• Chatbots - A chatbot is a programme that can conduct a conversation, and answer questions requested of it in natural language. Most of us have such a technology embedded into our phones. Beneath the natural language interface, (often built via machine learning) there is a more explicitly organised knowledge base that the chatbot can query to find its answers. This allows chatbots to be broader than pure machine learning in terms of the range of problems they can tackle.

Data challenges

Practitioners often focus on availability of data as the main limitation to the progress of ML. Machine learning requires enormous quantities of data to tease out patterns and cluster data according to common properties. Generally speaking, the more data, the better the model that will be developed, the more useful patterns might be uncovered and the better predictions can be made.

The availability of data is, at present, far from where ML developers would like it to be:

• Data protection legislation and norms makes accessing many large datasets a challenge.
• Data is often not of sufficient quality. Frequently the data is incomplete, and with a significant number of errors, fragmented between different repositories, and hard to link up accurately, or coded using incompatible frameworks.
• Data that ML developers want may simply not exist, either because it has never been digitised, or because it has never been recorded at all. A lot of potentially valuable data collected by devices, including monitoring equipment, is immediately discarded after use or display. And a lot of information that doctors use in practice never makes it to the record.

"A lot of information that doctors use for decisions isn't captured, so it isn't there for Al to learn from. Doctors could pull in information about social situation, context. If you're treating a frail, elderly person with no social support you make a different decision at 2am and 2pm."

Alastair Pickering, Clinical Lead (Urgent and Emergency Care) NHS Digital

• The data may not have been gathered in a way that makes it possible to distinguish causation from correlation – see below for an example.

All of this means preparing data for machine learning applications requires significant amounts of technology, investment, and trained human labour. Data has often been called the new oil; it should not be surprising that alongside collecting, this material also needs refining, and that this refining process requires skill and investment.

The speed of progress towards the futures sketched in this report will, to a large extent, be determined by how we solve this set of problems.

Data complications

A team at the University of Pittsburgh Medical Center wanted to better predict which pneumonia patients would develop severe complications, and so be able to manage demand and improve patient safety; sending low-risk patients to outpatient services, whilst better identifying those at high risk and admitting them to hospital. They tried both neural nets and traditional methods such as logistic regression- 'rule-based systems'. They found that neural nets outperformed other methods by a wide margin. But there was a downside.

It was hospital policy to send people with both asthma and pneumonia not only to an inpatient bed but directly to intensive care, as the combination of the two can produce severe complications. This policy worked so well that people with asthma almost never developed these complications.¹⁷

In learning from this dataset in isolation, the Al came to the conclusion that having asthma reduced the risks associated with pneumonia, the exact opposite of the truth. It recommended community management for these patients, a dangerous course of action.¹⁸ The Al was picking up a pattern that did exist in the training data, but which did not reflect reality.

The team used both an Al with explicit and visible rules, and a neural net where the logic is much harder for a human to understand. This failure was visible in the rules-based system, and so easy to catch. But the study authors conclude "If the rule-based system had learned that asthma lowers risk, certainly the neural nets had learned it, too." If the team had relied only on a neural net and trusted its conclusions it could have made deadly errors.

2 How Al might be used

In this section we consider how the present generation of Al might be adopted within the healthcare system, and hence be part of our everyday experience of healthcare. We argue that Al will see its fastest adoption in areas where it is not competing head-to-head with humans, but even so will be in a hugely influential position.

The present generation of Al is narrow compared to people. It works for tasks that are to some degree constrained. An ML algorithm is generally trained on data of a particular type (e.g. an MRI), and learns to reach certain types of conclusions (e.g. cancer present or not). And while chabots are capable of covering a broader range of situations, the conversations they can have are still relatively simple.

In contrast, much of medicine requires complex and holistic judgements. It requires a clinician to use what is effectively a general intelligence, integrating a whole range of capabilities to make a judgement - linguistic, interpersonal, physiological, clinical, and psychological understanding as well as knowledge of the local landscape. As mentioned in Box 1, it would probably take a fundamental breakthrough in Al for machines to be able to do this. Indeed, in a world where Al could do this, it could probably write this report. Consequently most people think that large sections of medicine will remain hard for the current generation of Al.

It is also fair to say that there are significant areas of medicine that involve narrow application of decision-making rules and expert pattern matching. Diagnostic specialisms such as radiology and pathology are seen as containing a significant number of tasks in this area - although there is more to these specialisms than that.

It is an open debate how much of medicine can be reduced to narrow judgements based on a consistent set of inputs, and how much requires a holistic and general intelligence.

But even when Al is successful at these more constrained tasks, there are obstacles to machines replacing people completely. To actually replace doctors, Al has to show superiority in nearly all circumstances - or equivalently, that the doctor adds almost no value in combination with the Al. In practice, machines and people have different strengths. For example people are better at learning from a smaller number of examples, and therefore are likely to do better in rare circumstances - unusual presentations, rare diseases, etc. Al needs large amounts of data to learn to perform a task, which may be hard to assemble for rare conditions and circumstances. People also have common sense, and can spot when the machine is making an obvious mistake due, for example, to a foreign object that has found its way into the medical image.

Additionally, the public prize their personal relationships with their doctors, and have a high degree of trust in the medical profession. And having a human in charge leads to clear accountability for decisions. The public are likely to set a very high bar when it comes to anything that might undermine these, and with good reason.

Generally, competing head-to-head with humans at tasks that people are already good at, is a difficult route to adoption. Easier routes are likely to be: helping people to perform their existing tasks more easily, and providing new solutions where there are presently no good alternatives.

Al can take lower value tasks that people do not want to perform. Al could, for example, help a pathologist by counting the number of abnormal cells in a sample - a task that is both dull and difficult for people to be accurate at. This is likely to be less disruptive to relationships, and so is less relevant to the focus of this report.

The other, and perhaps the easiest route for Al to come into our lives, will be offering solutions where is there is significant need but presently few good alternatives on offer. Innovations often take root in this way, as argued by the great innovation theorist Clayton Christensen.¹⁹ We term this Complementary Al. Areas that caught the interest of our interviewees included:

Advice and triage before seeing a doctor

When should someone first seek help from the health service?

Proactive care

When is the right time to intervene in the face of worsening symptoms?

Automated second opinion

How does the diagnosis and treatment I am getting compare to alternative options?

There are Al driven products in all these categories, although not always with sufficient evidence of safety or efficacy as yet. These products could be very attractive to both an overstretched health service, and to worried citizens. This would put Al in a hugely influential position for our health, our experience of care, and relationships within the system. The following paragraphs explore what this might look like.

Potential impacts of complementary Al

1. Advice and triage before seeing a doctor

One area where existing solutions are weak is before the patient chooses to first contact a doctor - what some call the 'pre-primary care'²⁰ space. Online advice is often unreliable, contradictory, and not specific enough to be helpful in individual circumstances.

With Al, members of the public could have access to high quality screening and personalised advice. Chatbots are available that answer medical queries for a range of situations, such as babylon health, Ada and your.md. And smartphone apps can make Al imaging analysis easily available e.g. for skin cancer²¹ - although app advice often stops short of a definitive diagnosis. As well as helping with self-care and diagnosis, these products will inevitably give advice on when to seek medical care - effectively triaging people into the health system.

This could be very significant. The fact that it is hard for patients to know when to seek help means A&E and GP practices are full of people who did not need to be there (but who largely could not have known that at the time). Research suggests that 20 per cent of GP²² appointments and 19 per cent of A&E attendances²³ are for more minor medical problems that could be treated at home. Equally, there are people who do not seek medical attention when they should, often causing further problems and expense at a later date. Progress on these points would be hugely significant to the economics of the health system, as well as quality of care.

There are also significant risks. The Al could give unsafe advice. It would also be easy to generate a flood of worried well, should the Al be too risk averse in its recommendations and throw off a large number of false positives. With the potential legal and reputational risks of unsafe advice, there will be significant pressure on Al developers to be cautious. Widespread consumer adoption of overly risk averse triage Al would risk overloading a health service already at breaking point.

Where there is explicit or implicit pressure to use the Al, it could also easily become a barrier to be negotiated, rather than an aid. Opaque reasoning, and a failure to cope well with complex or atypical cases, would lead to frustration and perhaps gaming of the system.

Al advice is already being rolled out here in London as part of the babylon health's GP at Hand service²⁴. This combines remote access to a GP with a triage chatbot, and is available to anyone who lives or works in central London. The service has received some criticism of is lack of an independent evidence base, and a review is underway²⁵. But in the near future data will emerge that establishes whether the advice given is safe, and whether it meets demand appropriately. If this service can safely reduce unnecessary demand, then it would seem likely that it will be rolled out at scale in coming years.

2. Proactive care

Similarly to the dilemma of care, it is hard for patients to know themselves when to seek help for a potentially deteriorating condition, and expensive for the health service to frequently assess them. Sensors and devices that can collect data from the patient, and trigger more timely care known as telehealth - have been around for many years. However, this technology has not quite broken through to the mainstream, despite considerable effort and investment from government. Al brings new analytical capabilities that could hugely expand the scope and quality of this real-time monitoring.

For example, an app from Cordio can hear signs of deterioration in patients with congestive heart failure, via an audio recording of their voice. The app detects signs of a build up of fluids around the patient's lung, before other physical symptoms are manifested.

Technologies of this kind should allow earlier intervention, and so improve outcomes and reduce costs. They would reduce unnecessary appointments, while responding more quickly and effectively to problems. It would require a significant rearrangement of staff and modes of communication between them, but the preventative savings and improved outcomes from timely intervention could be significant. Again, if this sort of technology can be established as reliable and safe, then it could be a significant part of our experience of care in coming years.

There are further risks beyond safety and reliability. Overly risk averse technologies could create overload on the system, as well as patients who ignore the warnings. Ubiquitous monitoring could easily feel excessive and even nagging. Finally the system would be data hungry, and filling in gaps in the data could easily reduce time for conversation.

3. Automated second opinion

Healthcare is one of the interactions in which we are least able to question and assess the advice we receive. Al could change this.

Al could be used to as a comparator or digital second opinion. Technology like IBM Watson (if proven to be reliable) allows a whole treatment approach to be compared to an automated alternative. The recommendations of local clinicians could, in theory, be compared to those of Al trained and programmed at the most highly respected hospitals - perhaps the Royal Marsden approach to cancer, or a Moorfields approach to eye problems.

Managers and regulators could also have this option - perhaps using it to screen for sources of variation in treatment. If Al reaches high levels of accuracy for certain situations, a doctor who gives very different diagnoses regularly might raise questions.

To be clear, the evidence certainly does not support making Al a standard to which doctors should be held, even in limited circumstances. But that will not prevent individuals and organisations with access to Al making comparisons. Patients find it hard to know if they are getting appropriate treatment; and given the importance of the issue, an easily accessible second opinion is appealing. Similarly managers and clinical leaders both seek to understand and control variation in treatment and outcome for similar patients.

Where the Al is accurate and applied within its competence, and the results are understood in context, this could be helpful. However, again, Al could easily be highly problematic in this context if misapplied. Inaccurate Al advice could undermine the trust between doctor and patient. Central monitoring could put pressure on clinicians

Overall, Al in these three categories:

• Could be very attractive to patients and managers, potentially solving problems that are serious and significant.
• Carries with it a great deal of responsibility, not only in terms of safety, accuracy and efficacy, but also in its impact on power and control for both patients and professionals.

Chapters 3 and 4 explore the impact of this sort of Al from the perspective of patients and professionals.

3 How we might experience Al

In this section we will try to be more concrete and specific about how Al might change the experience of healthcare for patients and professionals. We will do this via a series of illustrative narratives.

4 Lessons for People Powered Al

In this section we pick out some of the consequences of Al mentioned in the scenarios above. We argue that the technology allows for a broad range of options, from very empowered to very disempowered, and that choices made in implementation decide where in that range we arrive.

The impact of Al in the scenarios above varies along a number of dimensions - typical with either positive or negative consequences for the power and autonomy of individuals.

Simplicity and complexity

We have described an Al that often sits between citizens and professions - triaging, alerting, prioritising etc. This means it can make accessing care feel simpler, or more difficult. Talking to an Al chatbot that does not understand an unusual presentation, or non-standard phrasing, or take into account multiple conditions, could be immensely frustrating. It could result in people circumventing or gaming the Al; in turn, this could undermine clinical relationships

A core issue is the degree to which the capabilities of the technology match with the task that it is being asked to perform. In particular, are we asking narrow Al to tackle the complex problems that require a much more general intelligence? This is bound to result in poor advice and frustration. Al needs to be implemented in a way that recognises the scope of its own competence, and that hands over to a human when this scope is exceeded.

Control

Al could give patients more control over their health. Real-time monitoring could allow patients to have a far better understanding of their condition, how it was progressing, and if they were on track. They would be better able to ask for care when they needed it. Or it could reduce the control that patients have. Obscure and hard to question Al decision-making would make it hard to exercise control.

A key here is whether the monitoring is designed for patients as much as clinicians or managers. If its outputs are made as accessible as possible for patients, it will maximise opportunities for understanding and self-care.

Dialogue

High quality conversations should remain at the heart of healthcare. It would be easy to see the time for an individual conversation squeezed out by the time taken to get the Al the data it needs, or dominated by the Al's framing of a problem. The potential of Al to monitor clinician behaviour, if not done very carefully, could result in increasingly scripted interactions that can miss important facts about an individual, prevent flexible, creative solutions, and feel mechanical. Equally Al could create more time for conversation, enrich it with helpful analytics, and ensure that doctor and patient were as well prepared as possible.

Accountability

Certain forms of Al are so complicated that it can be impossible to understand how decisions are made and to check the logic which led to the decision. Some machine learning tools are particularly problematic. They may use hundreds of inputs, assign thousands of different weightings to those inputs and combinations of inputs, and arrive at a model that is impossible for a human to scrutinise.

A system that is obscure in its decision-making makes it hard to the patient or the professional to engage with or question it. The system also becomes unresponsive and slow to learn, as the reason for mistakes is not identified. A lack of accountability and transparency exacerbates the other issues mentioned in this section - it makes any deficiencies hard to identify and challenge.

A key here is that Al remains explicable and interrogable, at least to a similar degree to an expert, and that there is clear accountability for rectifying mistakes and learning from them.

Bias and inequality

Algorithms that give advice are vulnerable to bias just as people are. A well publicised example occured in an algorithm used for assessing the risk of reoffending within the US criminal justice system.²⁶ This algorithm influenced bail, sentencing and parole decisions, so was of great importance. When the algorithm made mistakes, it did so by overstating the reoffending risk for black people, and understating it for white people. This was despite the fact that the algorithm was not told who was black and who was white.

If we move into a world where Al is advising on who would benefit most from treatment, there is real a risk that inappropriate factors are picked up and bias is replicated and reinforced. For example, those from disadvantaged backgrounds may benefit less from a given treatment, due to lack of resources, social networks or extra stressors. A fair approach might be to give them the extra support they need to get the full benefit from treatment. But since Al can be something of a black box, it may not be clear that the drivers of poor response to treatment are factors that should not be considered, or require a different response.

In all of this, it is important to consider the dynamics of the marketplace. Al development is being driven by private companies, who are not directly incentivised to think from the system point of view and bear the points above in mind. Some companies will be incentivised, by reputational risk, to give conservative advice, even if this might increase demand. They will not necessarily consider the impact of their innovation on dialogue, or understand whether, in the context of a complex system, it makes accessing care simpler or harder. This means that the public sector has to be clear about what it wants from Al, understanding the strengths and limitations of the technology, and actively managing the wider impacts. The window of opportunity to do this is limited. If we sleepwalk into a situation where a small number of tech companies have already monopolised access to the data to build Al, and are selling into a health service which does not fully understand the technology they are buying, the capacity for public influence of Al will be much reduced.

A number of principles and metrics could guide us down the right road here.

5 Conclusion

Al in health is at an earlier stage than the debate sometimes suggests - but its implications could be even more radical.

Current-generation Al seems likely to be adopted in health where there is not much of a competing solution, rather than replacing humans at things they are not good at. Candidates include advice and triage before seeing a doctor, proactive care, and automated second opinions.

On the one hand, Al could significantly enhance health. It could be a force for the democratisation of knowledge and empowerment of citizens, as well as much better health and cheaper healthcare through better prevention, better targeting of treatment and better use of specialist clinical expertise.

But it could also be a disempowering force that reduces citizens' control of their health, creating an expensive and unreliable healthcare system.

Which path is taken depends on choices, so it is important to develop a set of principles to guide the development and implementation of Al in health. We have suggested the following principles and tests:

If the pace of development of Al continues, and given the amount money being invested in it, and the 'burning platform' of an overstretched health service Al could become commonplace in as little five years.

There is currently a window of opportunity to shape the future of Al in health. Policymakers are already working to set rules for Al and ownership of public data that ensure the public gets not only value for any data it decides to share, and privacy elsewhere. At this moment they have the leverage to also ensure that Al products that deliver maximum public benefit. This requires that both healthcare providers and users of Al understand the technology, have the tools to shape the market, can understand the needs of citizens, and are able to work through the complexities of implementation. This can be achieved through the following four recommendations:

Public and clinical scrutiny

Involve citizens and clinical professionals in the upstream design, development and implementation of the technology. This should include the requirement of mechanisms, such as public panels made up of citizens, that ensure technology development and implementation takes account of the demands and perspectives of citizens and healthcare professionals, and ensures that People Powered Al principles are applied.

The most reliable guarantors of the principles above are citizens and professionals. It is from their perspective that these issues are most obvious and urgent and the solutions most clear. To gain the benefit of their input, it is critical that they are involved throughout the design, development and implementation process, rather than being brought in late, when key decisions have already been made. Involvement of this kind takes effort to organise, and a degree of innovation in practice. If the public sector is serious about ensuring that Al works for people, this should be a rigorous and mandatory part of the process.

Controlled tests in real-world conditions

Enable real-world experimentation of Al in designated test sites, with non-Al comparators, to understand how Al works in complex systems before wider take-up 'in the wild'.

While general principles can be articulated, we still have a lot to learn about the details of implementing Al and how to design services that take advantage of it while minimising risks. For example, it will require some iteration to ensure safety of advice, while making sure the system is not overrun, or to find the correct balance between use of the Al analytics, and leaving time for human conversations.

This sort of complex technology and service redesign process is hard to do, especially in a responsible way that appropriately manages risks and unintended consequences. Space for rigorous real-world experimentation in controlled areas, before wider roll-out therefore needs to be carved out.

Proactive market design

System leaders actively engage in market design to maximise public benefit and ensure a plural market with genuine choice. This should include regulation that is upstream and proactive ('anticipatory regulation'), clarity over who owns both algorithms and data, and requiring adherence to key design principles, such as People Powered Al principles. Market design should also foster a diversity of new entrants to the market including procurement processes that work for smaller companies and market structures that support a diverse range of R&D activities.

Al development is being driven by private companies, who no matter how well-intentioned, are not directly incentivised to think from the system point of view and bear these principles in mind, or to throttle demand at the cost of a more inaccessable and inhuman system. The public sector needs to shape the market, ensuring it delivers the kind of Al that is good for people and system. This includes, minimally, giving clear requirements, reasonable procurement, and encouraging some disciplined risk-taking. The most effective time to do this is while the general rules for Al are still being set, rather than waiting until providers are well entrenched.

Decision-makers equipped to be informed users

Create a new cadre of public leaders and decision-makers with the technical skills, authority and institutional levers to scrutinise, manage and deploy Al in a responsible way. This should include incorporating artificial intelligence into medical education and health management training to enable the frontline workforce to be informed users of the technology.

Understanding the limits of Al is critical to implementing the technology well, as well as grasping its impact within a complex system. Further, taking advantage of Al will require some reorganisation of services, incentives, perhaps across institutional boundaries, as well as engaging with complex regulatory issues such as data protection. Leadership with the right skills and authority across boundaries is necessary to manage the design and implementation of Al. A health service which has sometimes struggled to get value from its technology investments needs to take special care with an unusually disruptive but high potential technology.

Endnotes

58 Victoria Embankment London EC4Y ODS +44 (0)20 7438 2500 [email protected] @nesta_uk www.facebook.com/nesta.uk www.nesta.org.uk

Nesta is a registered charity in England and Wales with company number 7706036 and charity number 1144091. Registered as a charity in Scotland number SCO42833. Registered office: 58 Victoria Embankment, London, EC4Y ODS.