From law courts to cancer wards and governments handling the Covid crisis, scattergun decision-making can lead to disaster. Three internationally renowned professors explain how to remove bias and 'noise' from our thinking.
Think of two different judges passing sentence on a couple of first-time offenders, both up for cashing forged cheques for less than $100. In the first case, Judge A sentences the accused to 30 days in jail. In the second, Judge B gives the other guy 15 years.
You could say that each judge displayed his bias, one towards leniency, the other towards severity. What you couldn't say is that the joint effect of the two judgments amounted to a reasonable average. Leaving aside whether seven and a half years would be fair, the point is that neither criminal experienced that fairness.
A US judge named Marvin Frankel came across that very scenario, as well as many other shocking disparities, when he wrote a report on sentencing policy back in 1973. His findings were so revelatory that it resulted in the US adopting sentencing guidelines in an attempt to iron out such extremes.
In their new book, Noise: A flaw in human judgment, Daniel Kahneman, Olivier Sibony and Cass Sunstein cite Frankel as an early pioneer in the battle to rid institutions and systems of "unwanted variability". Or, to give it the more vernacular term the authors employ and which is the book's title: noise.
They ask us to imagine an archery target at which a number of people take aim. If all the arrows land to one side of the bullseye, the accuracy problem is one of bias. But if the arrows land scattered all around the target, it's one of noise. The same applies for all large systems, institutions and professions, from the law to medicine to the insurance business.
Hungry for justice
In recent years, we've heard a lot about unconscious bias, particularly relating to issues of identity – race, sex, class – and most organisations of any size are now seeking to address biases, with so far mixed results. However, the three authors contend that just as large a problem, and perhaps larger in terms of efficiency, is noise.
Noise isn't just a matter of competing biases. Studies show that the decision-making of people such as judges and doctors is influenced by a range of extraneous matters, such as the weather or the performance of the local sports team, or whether the decision is made before or after lunch. It's those kinds of factors that can contribute to a glaring variability in outcomes, but which are often hidden from view.
And therefore noise can show up in the kinds of places we assume are immune to external influences. In 2012, an FBI-commissioned study asked 72 fingerprint experts to look again at 25 pairs of fingerprints they had evaluated only seven months earlier. Roughly one in 10 was changed. It wasn't that clear-cut decisions were overturned, but rather that marginal cases were reclassified as unsafe. Noise can have serious repercussions.
The trio of authors represent something of a dream team in the world of social psychology. Kahneman, a Nobel laureate and grand old man of the discipline, wrote Thinking, Fast and Slow, which examined the errors that lead from two distinctive modes of thought: intuitive and logical. Sibony, a former senior partner at management consulting firm McKinsey, is a professor of strategy and business policy at HEC Paris; Sunstein is a professor at Harvard Law School and co-author of the bestselling book Nudge: Improving decisions about health, wealth and happiness.
Noise and its cousin
It's Sunstein whom I speak to on Zoom. With his high forehead that extends seamlessly into a bald pate, he looks like central casting's idea of a policy wonk, an impression that is only deepened by the careful manner in which he outlines his arguments.
The pandemic, he tells me, has been a period of great noise. "In medicine, for example, diagnosis of what people have has been quite noisy, and visibly so. In the criminal justice system, the sort of penalties people face for behaviour that contravenes restrictions is often unpredictable and highly variable. And I think it's fair to say that if different governments respond very differently to the pandemic, there is either noise itself or a close cousin of noise.
"The reason I don't want to be dogmatic and say it's noise is we understand noise to be unwanted variability. And if New Zealand responds differently from South Africa, it might be that that's responsive to different values and norms or situations."
From 2009 to 2012, Sunstein worked for the Obama administration, in charge of the Office of Information and Regulatory Affairs. But he's very much a lawyer by training and not, as he ruefully admits, a statistician. He says that he found some of the statistics in the book rather challenging. Although aimed at a general audience, there are parts of
Noise that do not make for easy reading, even if the message of the book is quite simple: too much variation in decision-making is bad, especially when the outcome is critical. Diversity of decision-making is fine when it comes to how to stage Hamlet, but not when diagnosing breast cancer.
If three doctors make three different diagnoses, "that's noise", says Sunstein, "even if we don't know what the proper diagnosis is".
Various studies have shown that algorithms and machine learning bring more consistent and often more effective results in a variety of systems. And reading the book, it's hard to escape the sense that Noise, intentionally or otherwise, is an argument against human decision-making and in favour of artificial intelligence. Sunstein, however, rejects the idea. "The title of the book isn't 'Hooray for Algorithms'," he says, with such a gentle, bone-dry delivery that you could miss the fact he's making a joke. "It isn't 'Algorithms Now!'"
Rather, he says, it's an argument for structuring decision-making in such a way that it cuts down both bias and noise. An example might be in the interviewing process, where, instead of asking an ever-changing range of questions, there is an agreed list that is tailored to the specific job each candidate has applied for.
"The most conspicuous bias of interviewing is that within a few moments of meeting someone, people basically know what they think and the rest of the interview confirms the impression, which is not the best way to decide who to hire," says Sunstein. "The structuring approach eliminates or reduces significantly that kind of bias and also reduces noise."
In practice, though, algorithms are often better at – and less bored by – following a methodical approach. One famous study in the US showed that algorithms did much better at spotting recidivism in bail applications than judges with many years' experience.
However, when the state of Kentucky legally required judges to consult algorithms to decide whether a defendant should be held in jail before trial, the court system didn't get what it bargained for. They were expecting a cheaper and fairer system. Instead, whereas previously there had been no real difference in the proportion of black and white defendants who were granted bail, after switching to an algorithm-led assessment, judges began releasing far more white defendants relative to black. In other words, it was less noisy, but much more biased.
Sunstein points out that an algorithm is only as good as the information that's fed into it. And if that information is itself biased, the outcome is unlikely to be different.
Nonetheless, algorithms are making inroads in all aspects of life, including in spheres – such as law and medicine – that have traditionally attracted some of the brightest and most independent minds. If you were to think about people who would be naturally predisposed to deferring to a more reliable authority, judges and consultant practitioners are probably not the first people who come to mind.
The stakes are high
It's perhaps no coincidence that the sentencing guidelines that were introduced following Judge Frankel's report were later watered down, following protests by judges who felt themselves to be too restricted in their options. After all, the "three-strikes" laws, which led to those found guilty of three felonies automatically being given life sentences, were rightly subject to enormous criticism. Noise reduction doesn't always lead to improvement in outcome.
So, how does a company or institution ensure that a reduction in unwanted variability doesn't lead to an increase in some deleterious effect? Sunstein says, as with most systemic change, a cost-benefit analysis is required. "The costs of eliminating unwanted variability may be higher than the cost of unwanted variability," he allows, noting that, for example, an insurance company that isn't very noisy has little reason to implement costly measures of "decision hygiene".
The way to work out whether to institute a process of decision hygiene, suggest the authors, is with a "noise audit" that establishes how error-prone a given system or organisation is. "If the noise audit uncovers a high level of noise, and the stakes are really high – it might involve money or health or safety, or, you know, who gets to go into New Zealand seeking asylum – that's a big deal," says Sunstein.
But what does decision hygiene actually entail? It sounds like just the kind of vague but resolute term behind which a whole army of consultants could mushroom. Sunstein says it involves a number of tools, including instituting guidelines, soliciting independent views from which an average is derived, and structuring judgments, which means delaying decisions rather than just hastily going on intuition. Initially, the co-authors planned to make a stronger case for algorithms and artificial intelligence but decided to focus more on a human approach to decision hygiene.
One reason for that choice, says Sunstein, "is that algorithms are not going to replace human judgment in multiple domains for numerous reasons, one of which is that human beings don't like to be displaced by algorithms. Another is that algorithms might encode bias. If you could get an algorithm that's unbiased and noise free, that's a great advance. In medicine, with respect to diagnosis of some diseases, algorithms are saving lives. We will see a lot of progress there."
What has become clear is that humans expect a great deal more from machines than they do from themselves, in much the same way that we are more prepared to take risks with nature than we are with science. In the first instance, driverless cars need to be as close to accident-free as possible to win acceptance, while we will apparently put up with thousands of deaths from accidents involving human-driven cars. In the second, the experience of the Oxford-AstraZeneca Covid vaccine shows that many nations and individuals would much rather run a far higher risk of illness or death as a result of a "natural" virus as opposed to the much lower risk of both from an "unnatural" vaccination.
All of which means that there is a significant psychological resistance to what might be called dehumanisation of the social sphere. But this isn't simply a matter of humans versus machines; it's also an issue of humans versus machine-like behaviour. As a culture, we tend to prize inspiration and ingenuity and denigrate the methodical approach as boring. As the authors argue, to make progress in efficiencies and results we need to shift towards an outlook that favours "accuracy, not individual expression".
So, is noise reduction an attack on notions of individualism and creativity? And, indeed, might it involve a reduction of status in those who are made to conform to a more regimented way of doing things? Sunstein leans his head to one side, as if he's letting the question roll around his capacious mind, and then he's off with his softly spoken but crystal-clear analysis.
"So, if the question is, 'Does someone have lung cancer or strep throat?' there's a judgement to be made where the role of imagination and creativity isn't clear," he says with another dose of undemonstrative irony.
The answer will come down to following a diagnostic formula, says Sunstein, and, as things stand, doctors are always following guidelines without any diminution in status. It's just a matter of following better guidelines more closely.
Much of what Noise recommends seems to involve ever greater numbers of observers checking that the guidelines are correct and that they are being followed, and then perhaps observers checking the observers. As mentioned earlier, the past decade has seen a huge expansion in the business of combating systemic bias. Yet, although unconscious-bias training has become widespread, its effects remain open to question, with several studies showing that it has fallen far short of its claims, failing to produce any noticeable change in underlying attitudes.
Might companies preoccupied with reducing noise similarly find that their observers are no less subject to error than those they're observing? Sunstein acknowledges that this could be the case, but says the greater danger is not knowing what kind of noise exists within the organisation in the first place.
Noisy as a disco
Of course, there are many situations in which one person's noise is another's innovation, or one person's means of noise-reduction is another's stifling of talent. For example, in book writing. A number of critics have noted the tonal changes that occur in Noise as a consequence of three different authors having their say. Sunstein talks me through the trio's method.
Before the advent of Covid, they all met in Kahneman's apartment in New York and spent two days discussing substance and structure. "For some of the chapters, Danny would write an outline, and I found that I was very bad at working from his outline. Olivier was excellent at working from his outline."
Already it sounds as noisy as a disco. "So, a number of chapters were written that way, and a number were written by one of us taking the lead after the discussion and the other two would then edit the draft very aggressively."
I'm not sure a noise observer would have supported their idiosyncratic ways of working. But Sunstein is no stranger to authorial collaboration. Nudge was written with the economist Richard Thaler. That 2008 book argued for a synthesis of two starkly different approaches to life and government – libertarianism and paternalism – and the engineering of "choice architecture".
The idea is – while maintaining a JS Mill-like respect for individual choice that doesn't harm others – that there can be subtle ways of influencing choice without direct intervention. For example, it's possible for authorities to place healthy foods at eye level in supermarkets or school cafeterias. This kind of undeclared approach is a way of "nudging" people towards more healthy and economically rational behaviour. The book was popular with many liberal governments because it recommended policies that weren't costly or draconian, but also nodded towards progressive outcomes.
Nudging a nation
In many respects, the Covid pandemic has provided a global study in how to influence populations to adopt prophylactic measures – social distancing, mask-wearing, vaccine uptake. Who does Sunstein think has performed well in striking the right balance between libertarianism and paternalism?
"The best cases for libertarian paternalism are ones in which people are making errors with respect to decisions that affect themselves, like not eating healthily or not saving for retirement."
But in situations in which people are harming others, he says, then more interventionist measures are called for. In the case of the pandemic, nudges should form part of a multifaceted government armoury.
"From afar," he says, "New Zealand and its prime minister have been masterful with respect to nudging. I don't know whether this is because of behavioural expertise within the government or a very good set of working intuitions."
It does indeed seem a long-distance view that doesn't take into account the achingly slow vaccine roll-out that led the OECD to warn of the disastrous economic impact of a late outbreak. And I'm not sure how the comprehensive lockdown that Jacinda Ardern instituted last year qualifies as a nudge, but he points to the success of mask-wearing and social distancing – neither of which New Zealanders have had to endure very much, thanks to the promptness and severity of that first lockdown – and the strict border controls that have been in operation.
Perhaps implicitly recognising these facts, he adds: "A pandemic has a feature in common with environmental harm, and not in common with, let's say, risk taking for oneself, which is if you get sick or are reckless, you'll endanger others. So, for harm to others, we want nudges and other instruments."
If anything, it could be argued that Covid has proven the limitations of laissez-faire government in general. But it has also been a breeding ground for unwanted variability. Globally, we've been awash in statistics, theories, conflicting data, variable practices, confusion, error and misrepresentation. If the pandemic has dampened enthusiasm for nudging, it should by rights increase the efforts to combat noise.
Another casualty of the virus was the trip Sunstein was planning to New Zealand, his first. "I was extremely excited about it. I hope to get to you some day."
In the meantime, he's looking forward to others taking up the baton that he, Kahneman and Sibony have laid down. He says the positive response to the book has been driven by the knowledge that there's so much more work to do on the hidden subject the authors have raised – or at least named – for arguably the first time. "People appreciate the fact that we didn't get to the promised land, and that's great," he says, allowing an inscrutable half-smile.
Whether he'll ever get there, or indeed here, remains to be seen. But there's little question that, for such an understated and quietly spoken man, he's managed to make a great deal of noise in the process of addressing noise.