It was said you couldn't beat the test designed to detect if you were secretly racist, no matter how hard you tried.
Now it turns out the Implicit Association Test which unveiled subjects' unconscious prejudices has failed a few tests of its own.
The test has been used for years to determine. among other things, if you're racist or not.
And tackling the "implicit bias" it reveals has been the subject of myriad HR policies and training courses in private companies, schools and police forces worldwide.
But according to an article in Quartz, the test the world has relied on to identify "implicit bias" and fight racism may be flawed, news.com.au reported.
"This acclaimed and hugely influential test, though, has repeatedly fallen short of basic scientific standards," Olivia Goldhill writes.
The implicit association test (IAT), created by researchers at America's Harvard University, measures whether the brain associates good things more with one sort of person than another. It is supposed to reveal whether we subconsciously harbour beliefs about certain types of people.
It relies on measuring your reaction times, pairing concepts, like white and good, or black and good, and asking you to sort them together.
If it takes you less time to sort white with good than it does for you to sort black with good, it is supposed to reveal an implicit association of goodness with white faces.
"There's little doubt we all have some form of unconscious prejudice," Goldhill writes.
"Nearly all our thoughts and actions are influenced, at least in part, by unconscious impulses. "There's no reason prejudice should be any different.
"SHAKY" SCIENCE
"But we don't yet know how to accurately measure unconscious prejudice. We certainly don't know how to reduce implicit bias, and we don't know how to influence unconscious views to decrease racism or sexism."
There are now more than 12 versions of the IAT tests, and Goldhill says the science behind IAT tests is "shaky".
She said in recent years, a series of studies "have led to significant concerns about the IAT's reliability and validity. These findings, raising basic scientific questions about what the test actually does, can explain why trainings based on the IAT have failed to change discriminatory behaviour".
She's not alone. There are more than a few academics who think the tests are questionable.
Last year two researchers from Scandinavia argued there is "little evidence that the IAT can meaningfully predict discrimination, and we thus strongly caution against any practical applications of the IAT that rest on this assumption."
Edouard Machery, professor at the Center for Philosophy of Science at the University of Pittsburgh told Quartz the test doesn't have a strong "test-retest reliability" score, which recognises if a user can retake it and get a roughly similar result.
Perfect reliability, a 1, is defined as when a group of people repeatedly take the same test and their scores are always ranked in the exact same order.
"Current studies have found the race IAT to have a test-retest reliability score of 0.44, while the IAT overall is around 0.5 ... even the high end of that range is considered 'unacceptable' in psychology," Machery said.
"WILDLY DIFFERENT SCORES"
"It means users get wildly different scores whenever they retake the test.
"For other aspects of psychology if you have a test that's not replicated at 0.7, 0.8, you just don't use it."
The article also challenges the validity of the test — a measure of how effective it is at gauging what it aims to test.
"Validity is firmly established by showing that test results can predict related behaviours, and the creators of the IAT have long insisted their test can predict discriminatory behaviour," Goldhill writes.
"This point is absolutely crucial: after all, if a test claiming to expose unconscious prejudice does not correlate with evidence of prejudice, there's little reason to take it seriously."
She said four separate analyses between 2009 and 2015 hand show the IAT to be "a weak predictor of behaviour".
Machery argues that if the IAT cannot meaningfully predict behaviour, the results of the test are largely irrelevant.
He compared someone with a low anti-black IAT score but who behaves in a prejudiced way to someone who says they are brave, but behaves in a consistently cowardly way.
"You would not say that he's explicitly courageous and implicitly a coward," he says. "You would say he's a coward."
Calvin Lai, director of research at Harvard's Project Implicit and professor of psychological and brain sciences at Washington University-Saint Louis, told Quartz it's difficult to prove a test predicts behaviour, and future larger studies could well find stronger evidence to bolster the IAT.
He said one benefit of IAT was it helped people realise that while they may tell themselves they are not racist, they're not necessarily "bastions of equality"
"I think demonstrations like the IAT show that no, it's not just everyone else. It's you as well," he said.