As politics becomes a high-stakes spectator sport, pollsters are reviewing their latest failures. But is part of the problem the public's overly high expectations of precision?
Senator Susan Collins did not lead in a single publicly released poll during the final four months of her reelection campaign in Maine. But Collins, a Republican, won the election comfortably.
Senator Thom Tillis, R-N.C., trailed in almost every poll conducted in his race. He won, too.
And most polls underestimated President Donald Trump's strength, in Iowa, Florida, Michigan, Texas, Wisconsin and elsewhere. Instead of winning a landslide, as the polls suggested, Joe Biden beat Trump by less than 2 percentage points in the states that decided the election.
For the second straight presidential election, the polling industry missed the mark. The miss was not as blatant as in 2016, when polls suggested Trump would lose, nor was the miss as large as it appeared it might be on election night. Once all the votes are counted, the polls will have correctly pointed to the winner of the presidential campaign in 48 states — all but Florida and North Carolina — and correctly signalled that Biden would win.
But this year's problems are still alarming, both to people inside the industry and to the millions of Americans who follow presidential polls. The misses are especially vexing because pollsters spent much of the last four years trying to fix the central problem of 2016 — the underestimation of the Republican vote in multiple states — and they failed.
"This was a bad year for polling," David Shor, a data scientist who advises Democratic campaigns, said. Douglas Rivers, the chief scientist of YouGov, a global polling firm, said, "We're obviously going to have a black eye on this."
The problems spanned the public polls that voters see and the private polls that campaigns use.
In response, polling firms are asking whether they need to accelerate their shift to new research methods, such as surveying people by text message. And media organizations including The New York Times, which financially support and promote polls, are reevaluating how they portray polls in future coverage.
This year's misleading polls had real-world effects, for both political parties. The Trump campaign pulled back from campaigning in Michigan and Wisconsin, reducing visits and advertising, and lost both only narrowly. In Arizona, a Republican strategist who worked on Sen. Martha McSally's reelection campaign said that public polling showing her far behind "probably cost us $4 or $5 million" in donations. McSally lost to Mark Kelly by less than 3 percentage points.
"District-level polling has rarely led us — or the parties and groups investing in House races — so astray," David Wasserman of the Cook Political Report, a nonpartisan publication that analyzes races, wrote last week.
The full explanation for the misses will not be knowable for months, until the election results are finalised, detailed poll data is released by survey firms, and public voter files are also released.
But the available facts already point to some likely conclusions:
• People's decreasing willingness to respond to polls — thanks partly to caller ID — has reduced average polling response to only 6 per cent in recent years, according to the Pew Research Center, from above 50 per cent in many polls during the 1980s. At today's level, pollsters cannot easily construct a sample of respondents who resemble the population.
• Some types of voters seem less willing to respond to polls than others, perhaps because they are less trusting of institutions, and these voters seem to lean Republican.
• The polling industry tried to fix this problem after 2016, by ensuring that polling samples included enough white working-class voters in 2020. But that is not enough if response rates also vary within groups — for instance, if the white or Hispanic working-class voters who respond to polls have a different political profile than those who do not respond.
• This year's polls may have suffered from pandemic-related problems that will not repeat in the future, including a potential turnout decline among Democratic voters who feared contracting the coronavirus at a polling place.
• A much-hyped theory that Trump supporters lie to pollsters appears to be wrong or insignificant. Polls did not underestimate his support more in liberal areas, where supporting Trump can be less socially acceptable, than in conservative areas.
• In what may be the most complex pattern, polls underestimated the support of multiple Senate Republican candidates even more than Trump. This means the polls missed a disproportionate number of Americans who voted for both Biden and a Republican Senate candidate — and that the problems do not simply involve Trump's base.
Defenders of the polling industry point out that the final national error may not be very different from the historical average — and that polls can never be perfect.
Regardless, there are reasons for concern: Polling now seems to suffer from some systemic problems, which create a misleading picture of the country's politics.
Politics has become a high-stakes spectator sport at the same time that the country's ability to understand it has weakened.
History as a preview
Pollsters have been grappling with some of the same challenges since the creation of the industry.
One of the first polls to receive widespread attention came from The Literary Digest, a magazine, and it was published days before the 1916 presidential election. The magazine asked readers in 3,000 communities to mail in sample ballots and then reported that President Woodrow Wilson was in a stronger position than his Republican opponent, Charles Evans Hughes.
Wilson won the election, and The Literary Digest poll became a national phenomenon.
Although Literary Digest's sample of 2.4 million respondents was enormous, it was not representative. In 1936, a less prominent pollster, George Gallup, surveyed many fewer people — about 50,000 — but he had been careful to ensure they matched the country's demographic mix. Gallup correctly predicted a Roosevelt win.
Gallup's methods shaped the industry. Typically, a poll surveys hundreds of or a few thousand people and then extrapolates their answers to represent the broader population. If a poll cannot reach enough people in a certain demographic group — say, white Catholics or older Black men — it counts those it does reach in the group more heavily.
Still, the Gallup methods were not perfect, for some of the same reasons that polls have struggled recently. Within some demographic groups, Gallup turned out to be interviewing more Republican voters than Democratic ones, and it overestimated the Republican vote share in 1936, 1940 and 1944.
In 1948, the pollsters' luck ran out. They continued to overestimate the Republican vote share, reporting that President Harry Truman trailed his Republican opponent, Thomas Dewey. This time, the polls pointed to the wrong winner, contributing to perhaps the most famous error in modern journalism, the Chicago Tribune's banner headline "Dewey Defeats Truman."
That error led to a new overhaul of polling. Presidential polls in the 1950s and '60s missed by about 4 percentage points on average, similar to this year's miss, according to the website FiveThirtyEight.
"Polling has always been challenging," Nate Silver, FiveThirtyEight's editor-in-chief, said.
But polls still tended to point correctly to the winner of the presidential race in those years, partly because Americans were so willing to respond to polls.
"Decades ago, most people would be happy to answer the door to a stranger or answer the phone to a stranger," Courtney Kennedy, the director of survey research at Pew, said, "and those days are long gone."
A shock in 2016
Some groups, like college graduates and politically engaged people, were more willing to respond to polls. But the differences did not break down along partisan lines. Similar numbers of Democratic and Republican voters were declining to answer questions, which allowed the polls to be accurate.
Then came the election of 2016.
Nearly every poll showed Hillary Clinton to be leading Trump in Michigan, Pennsylvania and Wisconsin. But she lost all three narrowly, giving Trump a stunning victory.
Afterward, pollsters dug into their data. In early 2017, the leading industry group, the American Association of Public Opinion Research (AAPOR), released its conclusions.
One factor was largely unsolvable: Late-deciding voters broke strongly for Trump.
A second factor was more damning for pollsters. Many had not ensured that their samples included enough people without college degrees, especially among white voters.
Perhaps most important, the polling association argued, the 2016 experience did not suggest a systematic problem.
The midterm elections of the following year, 2018, initially seemed to support this conclusion. The polls correctly suggested that Democrats would sweep to victory in the House, while Republicans would retain the Senate.
The underlying details contained some reasons for concern, though. While polls in some liberal states, like California and Massachusetts, had underestimated the Democrats' vote share, polls in several swing states and conservative states, including Florida, Georgia, Michigan and Pennsylvania, again underestimated the Republican share.
Missing Republican voters
By the final weeks of this year's campaign, the polls seemed to be telling a clear story: Biden had led Trump by a significant margin for the entire race. Some combination of the coronavirus, Trump's reaction to police brutality and his erratic behavior at the first debate had put Biden within reach of the most lopsided presidential win since Ronald Reagan's in 1984.
That, of course, did not happen.
What went wrong? One possibility is that the pandemic may have led to an unexpected falloff in Election Day voting among Democrats. Another is that Democratic voters, energized by the Trump presidency and bored during the pandemic, became newly excited to respond to polls.
But the most likely explanation remains an unwillingness among some Republican voters to answer surveys. This problem may have become more acute during Trump's presidency, because he frequently told his supporters not to trust the media.
B.J. Martino, a partner at the Tarrance Group, which works for Republicans, said, "If there is an underlying issue, it's not getting those folks on the phone to begin with."
These voters do not fit any one demographic group, which is part of why they are so difficult to reach. Instead, they appear to be a distinct group of voters within some groups.
Even if pollsters constructed samples with the right mix of groups they might not capture the electorate's mood. Those factors, Shor said, "are not enough to predict partisanship anymore."
The media's polling problem
The polling industry group, AAPOR, had announced months before the election that it would conduct a post-mortem analysis for 2020. This analysis — and others — will probably shape the specific measure that pollsters take.
One option is to create new screening questions about whether respondents trust other people and major institutions — and then weight less trustful respondents more heavily. Pew, in recent years, has asked questions about whether people spend time volunteering, as one measure of trust.
Another is to expand the use of text messages and other nonverbal communication, like Facebook messages, in surveying people. "We're going to see more diversity in polling methodologies," said Kevin Collins, the co-founder of Survey160, which collects data through text messaging.
A separate set of changes may involve how the media present polling and whether publications spend as much money on it in the future. Among other questions, editors are grappling with the best way to convey the inherent uncertainty in polls.
Virtually nobody thinks polling is going away. It is too important in a democracy, Collins, of Survey160, said. It guides campaign strategies and politicians' policy choices. And there is no alternative method of election analysis with anywhere near as good a track record as polling's imperfect record.
The only short-term solution, some people believe, is for pollsters and the media to emphasise — and for Americans to recognize — that polling can be misleading.
If nothing else, the polling of the last four years may have given Americans a better understanding that they should not take polls literally. "The narrative around polling has to change," Cornell Belcher, a Democratic pollster, said, "because it's misinforming and it's setting polling up to fail."
Written by: David Leonhardt
Photographs by: Jim Wilson, Hilary Swift, Ryan Christopher Jones, Maddie McGarvey, Damon Winter, Todd Heisler, Cameron Pollack, Ruth Fremson and Doug Mills
© 2020 THE NEW YORK TIMES