In 2015, I began work as a journalist at The New Zealand Herald. During the next nine years, the value of my contribution to my employer was increasingly determined by a single data point: the number of people clicking on my articles and then clicking “subscribe to the Herald”. Bonuses for me and many of my colleagues were tied specifically to boosting that number and only that number. In one meeting, a respected senior journalist summed up the mood: “I’m sick of being treated like a number.” In May this year, I was made redundant. Data, I believed, could go and get fucked.
Having said that, it would be disingenuous not to acknowledge the genesis of this article was, at least in part, based on knowing that the line “how data can skew outcomes” would get a lot of clicks.
Life, death & data
The ultimate case of real-world, data-driven decision-making landed here in 2020, when Covid-19 turned seemingly everyone into a data scientist overnight. People who had never before thought seriously about using the phrase “regression to the mean” became instant experts in how best to apply the concepts they heard on the news. Meanwhile, real scientists were struggling to figure out what was happening and how to deal with it.
Shaun Hendy, then-director of Te Pūnaha Matatini Centre for Complex Systems and Networks, a national centre of research excellence, was at the heart of modelling the potential spread of the disease. In his first few months working on the response, he says, “We were just absolutely desperate for data”, which he describes as “situation normal” for scientists.
At the beginning of the pandemic, he was so short of usable data he was transcribing Ashley Bloomfield’s press conferences to get it.
As the data on transmission of the virus became more prevalent and available to him, the challenge shifted to one of communication. “The numbers didn’t speak for themselves,” Hendy says. “They needed interpretation for the public to make sense.”
Having data is not the same as understanding data, and neither are the same as making decisions using data. As we quickly learnt during the pandemic, data-driven decision-making is neither straightforward nor without consequences – it can be literally life and death.
“Ultimately, a great deal of the decisions we make individually and collectively come back to our ethical judgments,” says Jonathan Boston, emeritus professor of public policy in the School of Government at Victoria University of Wellington. “And those are obviously influenced by numbers – quantitative stuff – but they’re also distinct from those things.”
During Covid, Boston says, ethical judgments about the relative importance of human life had to be weighed against other values like freedom to travel or freedom to associate. How does one measure the relative importance of human life against these freedoms? Starting in 2020, different countries came to very different conclusions.
Hendy: “In a world awash with data, if you haven’t got the experts to help you interpret that data, you’re in a very dangerous place.”
Boston: “Personally, I’m very glad I was living in New Zealand and not in the UK.”
Steps to infinity
In papers that form the foundation for his forthcoming book The Score, C Thi Nguyen, associate professor of philosophy at the University of Utah, argues the use of data-based scoring systems leads us to change our values to suit the scoring systems of the institutions that impose them. He calls this “value capture”.
“This looks like: people who pursue step-counts even when it hurts their knees and exhausts their spirit; academics who pursue publications in the highest-ranked journals even when their work feels boring and meaningless; universities that pursue high rankings in [rankings publisher] the US News & World Report over richer understandings of education,” Nguyen writes. To this list, he adds “newspapers that pursue clicks and page views over their own sense of newsworthiness and social importance”.
The risk is that as we take on these impersonalised values, “we won’t even notice what we’re overlooking”. Nguyen tells an illustrative story of one of his family members who went on holiday with friends he calls John and Shelley – a couple who both wore Fitbits, the wrist bracelets that count the wearer’s steps.
“John and Shelley wouldn’t go to the opera with her: not enough steps. They’d cancel dinner dates because they hadn’t met their daily step goals yet. My guess is that John and Shelley never consciously decided that step-counts were more important than, say, art or friendship. The Fitbit just spoke more loudly in their internal deliberation, and there was no Artbit or Friendbit to compete.”
Once we start measuring things, we consciously or unconsciously begin to value the things we’re measuring and to reorient our lives to optimise for that measure. In doing so, we lose depth. We see this everywhere in our modern, tech-mediated world: follower counts, clicks, screen-time reports, sleep scores.
Then again, if a Fitbit motivates you to exercise, and exercise is good for you, what’s the problem? The problem is what you lose. Exercise comes in a wide range of forms and offers many possible benefits, but step-counting reduces all that to one: the number of steps. Not to mention that the “science” behind 10,000 steps needed daily for optimal health came from a 1960s marketing campaign in Japan for a pedometer, Manpo-kei. Manpo-kei means “10,000 steps meter” and was chosen for its catchiness. Of course, more modern studies have linked increased walking to positive health outcomes.
As Nguyen writes, “Fitbit doesn’t capture the ecstasy of complex skillful motion. It doesn’t capture the camaraderie of team sports, the meditative calm of paddling a canoe across a quiet lake, or the aesthetic loveliness of a delicate rock-climbing move.”
That might sound like a relatively trivial example, but the idea of value capture extends across many spheres. Take GDP. Although Nguyen doesn’t specifically address gross domestic product as an example, the similarities are obvious: GDP is typically presented as a scorecard for the country’s economy and more broadly an answer to the question: “Are we doing well?” But just like a Fitbit, it leaves out huge amounts of information in order to do so.
For instance, the unpaid labour that makes the paid economy possible (household chores and maintenance, caring for children, the elderly, disabled people), the effects of economic activity on people’s wellbeing, damage to the environment and income inequality.
Just as Fitbit can be used to reduce exercise to a number of steps, GDP reduces a country’s wellbeing to “total monetary value of goods and services”.
The Labour-led government of Jacinda Ardern attempted to change this, with then finance minister Grant Robertson introducing a Wellbeing Budget in 2019 that garnered worldwide interest. It focused on priority areas apart from GDP, including climate and environment, productive work, Māori and Pacific opportunities, child wellbeing, and mental and physical health. But it proved a fair-weather friend: Covid and the economic downturn that followed saw the focus on the five wellbeings fade, and the incoming coalition government officially ditched it.
Goodhart’s Law states, “When a measure becomes a target, it ceases to be a good measure.” In other words, if people are motivated to reach a target, they will find ways to do it that may prove counterproductive. Some examples are teachers teaching to the test; underreporting of crime by police to “reduce” crime rates; journalists focusing on subjects they know are likely to get attention (ADHD, gangs, sex, the Mowbrays) and steering clear of subjects that aren’t (political policy debates, the arts, poverty, climate change).
Nguyen writes, “It’s easy to justify yourself in the language of metrics, because metrics are easy to understand. They have, in fact, been engineered to be so. The cost of value capture is that we give up on the process of finely tuning our values to our own context: our personalities, our peculiar culture, our particular corner of the world.”
By the numbers
In our computerised, AI-mediated world, quantitative data has achieved pre-eminence over qualitative data. Whenever important decisions are made – from politics to sport to business – they are justified by numbers.
But why? “There is often an assumption that quantitative evidence is better than qualitative evidence, in the sense that somehow numbers provide a more accurate and reliable account of what is the truth of the matter,” says Boston. But, he says, the distinction between the two is not so clear cut. For one thing, quantitative outputs are often derived from qualitative inputs, such as surveys.
A classic example is the ubiquitous “Quality of life” reports that purport to tell us how liveable our city or country is. The decisions that go into determining that number are highly subjective. What comprises quality of life? And what weight do we give to each factor? Is “number of beaches” worth more than “low crime rate”? How many crimes are offset by a single beach? What type of crimes? What type of beach? And so on. A final number might suggest objective certainty, but its foundations often don’t.
The other issue is how our exponentially growing stream of data is being used. Boston is concerned about the misuse of data both globally and locally. He is particularly critical of the current government. “We have a government that is being very selective in its use of data and is frequently ignoring the best available data and ignoring the advice it’s receiving from those who were paid to provide the best available evidence – namely the public service.”
He’s seeing this across many areas of policy. “People talk about us living in a post-truth era or in the context of post-truth politics. And my sense is that democracy cannot thrive in such an environment. Democracy depends on a reasonably high level of trust in public institutions and in the people who inhabit those institutions. And if trust is undermined by repeated misinformation, disinformation and frequent misuse of information and analysis by those responsible for making the important decisions on matters of public policy, then this will undermine the credibility of our democratic institutions, and ultimately, potentially undermine democracy full stop. So, I don’t want to be unduly alarmist, but we are living in extraordinary times.”
Out of the park
In 2008, I read Moneyball: The Art of Winning an Unfair Game, by Michael Lewis. The book tells the true story of the Oakland Athletics baseball team, who in the early 2000s were not very good at baseball and had relatively little money. But the Oakland A’s overcame both hurdles by harnessing the power of data and maths geeks from Ivy League colleges to build a team of overlooked and undervalued sporting geniuses.
Moneyball changed the way I thought about first sport, then data, then life. I read Freakonomics, by University of Chicago economist Steven Levitt and journalist Stephen J Dubner, and Soccernomics, about statistical trends in football, and anything else with “nomics” in the title. I believed happiness could be measured on a five-point scale and that 40% of my happiness was determined by my actions and mindset. Whatever the question, I believed data had the answer.
Moneyball’s thesis was that traditional ways of valuing baseball players were flawed. Baseball had fallen prey to value capture. Specifically, batters’ value had traditionally been measured on two metrics: batting average and runs batted in (RBI), but both measures failed to value the “walk”, which occurs when a batter gets a free pass to first base by displaying sufficient judgment and patience to not swing at pitches outside the strike zone. Although the walk is of equal value to a hit that gets a player to first base, neither batting average nor RBI included it.
This meant players who were good at drawing walks were systematically undervalued. The Oakland A’s exploited this fact by putting greater focus on a measure called on-base percentage, which accounted for the value of walks. By doing so, the argument goes, the A’s were able to buy undervalued players and become the best team in baseball. All teams today track on-base percentage, along with an enormous bunch of other stats.
Hayden Croft is performance analyst for the Silver Ferns and head of sport, exercise and health at Otago Polytechnic. He wrote his PhD thesis on big data in sport and employs his expertise in the area every day. He thinks Moneyball has been incredibly damaging for sport and has led people to see data as a “secret solution”. He believes data is only a tool, and only one of many tools, for improving sporting performance.
“I know through my experience working in rugby and netball as an analyst that when you talk to a coach, a large percentage of the time they already know what the data is going to say because they have a really good tacit knowledge of the game.
“They’ve got a lot of experience. Yes, they carry biases, but we really use data just for them to challenge their own thinking and to see whether what they see stacks up against the data. You could come from the data side and say we can explain everything with data, but there’s a lot of evidence and support that that doesn’t always work.”
Croft says one of the first things he teaches his students is the danger of confirmation bias – the idea that you use data only to support your ideas. “Athletes and coaches and analysts can chase questions in the data and find something that will give some level of evidence to that, and that’s quite dangerous because you’re not looking at the whole picture – you’re just taking a snapshot of it and misrepresenting, probably, what the real information says.”
Too much information
In our increasingly digitised world, the amount of data available to us is already overwhelming, and it’s still growing. Croft cites Formula 1 motor racing, where a modern car features hundreds of sensors said to generate 1.1 million data points per second. Croft says he regularly observes young drivers entering F1 driving fast and free from constraints, then slowing down as they buckle under the weight of all the data.
“I see this in sport all the time,” he says. “Where young athletes are free, they’re confident of what they’re doing, they don’t have doubt in their mind. And then you start overloading them with information and they get really confused. They start thinking about the wrong things and then when they’re actually trying to perform in a Formula 1 car, on a netball court, rugby field, whatever it is, they’re thinking about the data or what they’ve been told to do. They’re not actually looking at what’s in front of them and reacting.”
Croft’s PhD supervisor was Professor Kirsten Spencer of Auckland University of Technology, whose research field is sports performance analysis. Spencer is a fan of the saying that in sport, the most important thing is “the top two inches”.
“The athlete is more than just the numbers,” she says. “The data is important, but it’s not everything.”
Croft puts it similarly: “Until we have a robot running around on the netball court, you’ve got to deal with humans and humans are very, very complex creatures and simply measuring them with a few numbers is not a very good representation.”