The study found those grinning were more likely to giggle at the cartoons. Its results have been cited ever since.
However, a new version of the study asked 1,900 participants to hold the pens in their mouths and complete tasks including drawing lines under vowels and between numbers, before rating the cartoons.
No evidence was found to suggest inducing particular facial expressions led people to rate the cartoons differently.
Study author Professor Eric-Jan Wagenmakers, of the University of Amsterdam, said: 'This did not replicate the results [of the original authors] and failed to do so in a statistically compelling fashion.'
The findings are published in Perspectives on Psychological Science.