Did 4 per cent of Americans really drink bleach last year?

According to the report, 4 per cent of the respondents stated that they had drunk or gargled diluted bleach in the past month. Photo / 123RF

Early in the summer of 2020, the Centers for Disease Control and Prevention issued a report on unsafe coronavirus prevention practices in the US According to the report, 4 per cent of the 502 respondents stated that they had drunk or gargled diluted bleach in the past month, 4 per cent said the same about soapy water and 4 per cent said the same about household disinfectant. This quickly inspired a number of alarming headlines.

This media response was understandable. While 4 per cent may not seem like much, if this study sample was representative of the US population, it would imply that roughly 12 million Americans engaged in these dangerous behaviours — an alarming figure.

But there may be reason to question that conclusion. First of all, the CDC report noted that a survey of just 500 opt-in participants was not necessarily representative of the US population (though the CDC did weight responses to line up with national age, gender and race demographics). And beyond these sample limitations, a second study (currently undergoing peer review) that aimed to replicate the CDC's findings with some additional quality control suggests that the data itself could have some serious flaws.

Specifically, this new study from online research platform CloudResearch sought to address two major issues that can threaten data quality: inattentiveness (i.e., respondents who are careless or aren't paying attention) and mischievousness (i.e., respondents who intentionally lie or mislead researchers). Psychologists who study relatively rare behaviours, such as hard drug use, have long known about these challenges. For example, in a 1973 drug-use study, researchers found that when they included a fake drug in the list of drugs they asked people about, 4 per cent of respondents reported taking a drug that didn't exist, suggesting that the data was likely not totally reliable.

You might have noticed the recurring 4 per cent figure. That might not be a coincidence. The psychiatrist and blogger Scott Siskind coined it the "Lizardman's Constant" back in 2013, in reference to a widely publicised Public Policy Polling report that 4 per cent of respondents said they believed shape-shifting lizard people were controlling the world. This poll garnered a lot of media attention. But Siskind and others argue that that 4% is a lot more likely to reflect inattentive and mischievous respondents than a true belief in such an outlandish conspiracy.

As with the lizard people survey, the CDC report generated a lot of publicity. And as with the lizard people survey, these responses could be legitimate — or they could simply be bad data.

To begin to get to the bottom of this, the second study sought to identify whether inattentiveness and/or mischievousness could explain the CDC's surprising findings. After asking the same questions as the CDC survey, researchers had participants complete a short word-association exercise (i.e., circle the unrelated word from a list) to test for attentiveness. These questions were designed to be very easy for anyone with a basic English reading level to get right, as long as they were paying attention. Next, to target mischievous respondents, the researchers asked "reality check" questions: questions with only one reasonable answer, such as "Have you died of a heart attack?" or "Have you ever used the internet?" (The survey was distributed online.)

Finally, as an additional quality control measure, anyone who said "yes" to ingesting household chemicals was asked a series of follow-up questions: They had to confirm that they had intentionally selected "yes," and then provide some additional details about the context in which they ingested the chemicals.

So, what did the researchers find? They collected data from a total of 688 participants. Of these, 55 (8 per cent) stated that they had ingested at least one of three household cleaning chemicals (disinfectant, soap or bleach) — a similar result as reported by the CDC. But of those 55, only 12 passed the basic quality control questions. In other words, almost 80 per cent (43 out of 55) of respondents who claimed to have ingested a toxic chemical did not accurately identify simple related words, and/or gave completely implausible answers to the reality check questions.

That left 12 apparent chemical drinkers who passed the quality control questions. Did they really drink bleach? If so, it would only be 1.7 per cent of the sample, but that would still represent millions of Americans.

When asked to confirm whether they had in fact ingested household chemicals, 11 of the 12 stated that they had selected the "yes" option by mistake. And the one remaining participant — who verified that they had intentionally selected "yes" — responded to the question asking for more detail with "Yxgyvuguhih." They also reported being 20 years old, having 4 children, weighing 1,900 pounds (861kg) and having a height of "100." Needless to say, these responses call into question the validity of the final participant's data.

So how many Americans actually ingested bleach to ward off the coronavirus? We don't really know. We would need more research to reliably answer that question, but the fact that the percentage dropped from 4 per cent to 0 per cent after accounting for basic data quality issues suggests that the real number is most likely a lot lower than headlines suggest.

The takeaway here isn't that all survey data is garbage. But especially when that data is used to support claims with serious societal repercussions, it's essential to validate results with basic quality-control interventions, such as the attention and reality checks described above. In the case of the CDC study, a failure to do so (along with some perhaps overzealous reporting) led researchers, media and the public to believe that up to 12 million Americans were drinking bleach. This claim was likely not only false, but also potentially harmful, as it may have served to Normalise these dangerous behaviours and thus increase the number of people who might actually engage in them.

Just as chemists and physicists need to ensure that their measurement tools are well calibrated, social scientists must also ensure the quality of their data to avoid reaching misleading conclusions. While there are no surefire solutions, a bit of rudimentary quality control can go a long way in validating the accuracy and reliability self-reported data. At the same time, some of the responsibility falls on journalists as well, to fact-check and accurately report on studies — and to avoid sensationalising. And, as with any content, the final gatekeeper is you, the reader. The next time you come across something that sounds too outrageous to be true, know that your instinct may be correct. Ultimately, it's up to all of us to think critically, do our research and determine for ourselves whether a source can be trusted.

Written by: Rachel Ernstoff
© 2021 Harvard Business School Publishing Corp. Distributed by The New York Times Licensing Group