The Principle of Proportionate Benefits in AIs and Ethics

AIs and the principle of proportionate benefits

Together with the numerous ethical issues associated with modern AI technologies (of which we have vastly discussed on our The House of Ethics collections), various ethical benefits may exist, too.

For instance, many supporters of the new technologies bring the example of car accidents. Say we consider UK, 2024: about 1600 people died due to car collisions. On the other hand, a major driver in developing self-driving cars is increasing safety and reducing accidents; say, they succeed in reducing accidents by 40%: more than 600 lives can thus be saved, which is a praiseworthy endeavor.

Similar reasoning can be developed in fields such as medicine: in this case, we may think of automated diagnosis. It has been established that, in the US, every year about 371,000 people die following a misdiagnosis. In AI research, much research is therefore dedicated to improve automated diagnosis in various areas of specialization. AI tools have ranging degrees of performance – in some application, they are on par or slightly better than human doctors; in others, they still struggle behind. For the sake of example, let us imagine the case where AI will eventually succeed in reducing misdiagnosis by 20%: more than 74.000 lives can thus be saved.

So, why are many people still doubtful, and what could be a reasonable improvement in AI performance that may be not only acceptable, but even desirable?

Psychological and statistical considerations

The main issue lies in the difference between global and individual rates of failures. Think of cars. Whenever we consider 1600 deadly car accidents, we implicitly refer to 1600 accidents caused by all humans within a country – for UK, say over 50 million people. That is, the accidents occur – and shouldn’t – but the single probability that a random individual may cause an accident is incredibly small. It is extremely more effective to enhance policies against alcohol, drug abuse, sleep deprivation, joyriding, or oversight of mechanical problems, rather than addressing the negligible small probabilities that single persons have to cause deadly accidents. On the other hand, given that the number of car manufacturers is relatively small, and that of autonomous car manufacturers is even smaller, the 1600 – 40% = 960 deadly accidents caused by autonomous vehicles would be shared between very few actors. In short, every single manufacturer bears a way higher probability to cause an accident than every single person.

Similarly, in the case of medicine, there are about a million licensed physicians in US, whereas the number of Medtech companies working in automated diagnosis is at least three times smaller – and only a handful (say, about a hundred to keep the numbers easy) are capable of going to market and have real impact in the healthcare sector. Surely, they may provide impressive aid when there is shortage of personnel; on the other hand, the 371.000 – 20% = 296.800 deaths resulting from misdiagnosis would be shared among a handful of companies, each of which would be associated with a large number of deadly errors.

It matters little that manufacturers sell many cars or diagnosis systems: there is only one name, and one system, under scrutiny. This is the first issue that, intuitively, is not acceptable: going from social to individual level (which, by the way, is a common tendency in societies that are more individualistic-leaning) makes autonomous driving a lesser choice than personal driving, or automated diagnosis a potentially questionable solution to replace doctors.

There is also a subtle distinction between random and systematic errors that come into play: a random error happens – by definition – at random, due to unpredictable, never-seen and hardly avoidable causes.

A truck tier exploding all of a sudden, on a straight and well-lit road, for example. Or an antibiotic-resistant bacterium evolving spontaneously, as a medical example. A systematic error happens whenever some conditions are met: maybe they are rare, but can occur. The condition of drunkenness is systematically associated with higher risk of accidents; poor hygiene is associated with concurrent pathologies and higher chance of misdiagnosis. Systematic errors are, in principle, easier to act upon, and are subject to regulation and restrictions. Studies are required in this sense, but AI-powered devices are intuitively often associated with systematic errors: they way AI is trained – through billions of examples – make it so that never-met errors are not common in the training set; and, if a never-met error occurs, the intuitive idea is that it’s anyway fault of the training set, as the error was systematically outside of it.

The idea that 1600 human-driven accidents may be associated with random errors (the systematic ones are instead blamed and punished), while 960 AI-driven accident can mostly be associated with systematic errors, therefore creates frustration and a sense of being unacceptable. Similarly, thinking that a portion of 371.0000 deadly misdiagnosis is due to unforeseeable errors, while the almost entirety of 296.800 is due to systematic faults, is hard to bear. Here is where statistics and psychology meet and may clash – and we saw several times, including during the COVID-19 pandemics, that psychology takes the upper hand on the long run.

Both ways of reasoning can be applied in domains, where the tension between population-wide statistics versus individual risks of failure exist.

Think again about medicine: from the perspective of a hospital CEO, it makes sense to reduce the risk of misdiagnosis by some percentage using automated tools. From the perspective of a patient, however, a single doctor by whom he’s visited may have an already low chance of misdiagnose; and, if he/she fails, the patient may go to another doctor and, statistically, have an even lower chance of being misdiagnosed as second time.

Finally, there is another psychological consideration to be added, which however depends on the tasks to which it is applied. Let us take driving, once more: if an autonomous car comes with a relatively systematic risk of failure, an experienced driver believes he/she has an almost null risk, due to driving history, confidence in abilities, and avoidance of situations that can drastically increase such risk (alcohol, drugs, and so on). Experienced doctors may feel similarly. Hence, someone confident in their abilities may be less prone to accept a low but ever-present risk due to automation – and again, we encounter the tension between social and individual risk management: what if the confidence is however overconfidence?

Proportionate benefits

What could be one way out this social/individual and statistical/psychological impasse? We suggest to focus on proportionate benefits, rather than relative improvement.

In technology, a certain metric may relatively improve, in terms of improvements with respect to a previous value. Having 960 deadly accident instead of 1600 is a 40% relative improvement; 296.800 misdiagnosis instead of 371.000 is a 20% improvement. However, this improvement only concerns the public sphere.

Instead, a proportionate improvement would account for the improvement given with respect to all subjects that can get a benefit.

It takes the perspective of individuals – which is often more stringent and beneficial than that of societies.

Take again the car example, and assume there are 5 autonomous car manufacturers selling in a country. Ideally, to foster acceptance also at individual level, each car manufacturer should have a risk of provoking car accidents at most equal to that of a single person, neglecting systematic errors. By the UK statistics, every person may cause 0,00003 accidents a year; this should be the number that every car manufacturer should strive to accomplish – in face of the current 960/5=192 accidents per manufacturer per year that would follow the relative improvement discussed above.

In case of medicine , we may consider the fact that each physician is associated with possibly deadly 0,371 misdiagnosis per year (of course, more refined calculations should differentiate by specialization and its intrinsic risk). Instead, assuming 100 automated diagnosis companies with a 20% improvement each, then a single company would be associated with almost 3000 fault diagnosis per year, if we only consider relative improvements. Vice-versa, to strive for a proportionate benefit in the case of automated diagnosis, AI companies should strive to get way better.

Proportionate improvements are thus an extreme challenge for technology, but bear the potential to provide massive benefits to societies and to promote trust in AI and related technologies. The thresholds may also be discussed – as mentioned, systematics shall also be taken into account, as well as many other factors, which may raise the bar acceptably and should be carefully addressed by research. Moreover, if the goal of technology is really to benefit humans, and individuals before societies, and to be really human-centric instead of statistical, the way of proportionate benefits, which focuses on who receives the benefit, before delving into numbers, would be an ethical and challenging endeavor to trigger the intellect of the most brilliant technological minds.