Pollsters thought they had learned from the errors of 2016. It's possible that they did, and that this election reflects new problems.
Asking for a polling post-mortem at this stage is a little bit like asking a coroner for the cause of death while the body is still at the crime scene. You're going to have to wait to conduct a full autopsy.
But make no mistake: It's not too early to say that the polls' systematic understatement of President Donald Trump's support was very similar to the polling misfire of four years ago, and might have exceeded it.
For now, there is no easy excuse. After 2016, pollsters arrived at plausible explanations for why surveys had systematically underestimated Trump in the battleground states. One was that state polls didn't properly weight respondents without a college degree. Another was that there were factors beyond the scope of polling, like the large number of undecided voters who appeared to break sharply to Trump in the final stretch.
This year, there seemed to be less cause for concern: In 2020, most state polls weighted by education, and there were far fewer undecided voters.
But in the end, the polling error in states was virtually identical to the miss from 2016, despite the steps taken to fix things. The Upshot's handy "If the polls were as wrong as they were in 2016" chart turned out to be more useful than expected, and it nailed Joe Biden's 1-point-or-less leads in Pennsylvania, Georgia and Arizona.
The national polls were even worse than they were four years ago, when the industry's most highly respected and rigorous survey houses generally found Hillary Clinton leading by 4 points or less — close to her 2.1-point popular-vote victory. This year, Biden is on track to win the national vote by around 5 percentage points; no major national live-interview telephone survey showed him leading by less than 8 percentage points over the final month of the race.
The New York Times/Siena College polls were also less accurate than they were in 2018 or four years ago. In 2016, the last two Times/Siena polls were among a very small group of polls to show Trump tied or ahead in Florida and North Carolina. This time, nearly all of the Times/Siena surveys overestimated Biden to about the same extent as other surveys.
In the months ahead, troves of data will help add context to exactly what happened in this election, like final turnout data, the results by precinct and updated records of which voters turned out or stayed home. All of this data can be appended to our polling, to nail down where the polls were off most and help point toward why. But for now, it's still too soon for a confident answer.
In the broadest sense, there are two ways to interpret the repeat of 2016's polling error. One is that pollsters were wrong about what happened in 2016. As a result, the steps they took to address it left them no better off. Another is that survey research has gotten even more challenging since 2016, and whatever steps pollsters took to improve after 2016 were canceled out by a new set of problems.
Of these two, the latter interpretation — real improvements canceled out by new challenges — may make the most sense.
"I think our polls would have been even worse this year had we employed a pre-2016 methodology," said Nick Gourevitch of Global Strategy Group, a Democratic polling firm that took steps to better represent Trump's supporters. "These things helped make our data more conservative, though clearly they were not enough on their own to solve the problem."
The explanation for 2016's polling error, while not necessarily complete or definitive, was not contrived. Many state pollsters badly underrepresented the number of voters without a college degree, who backed Trump in huge numbers. The pollsters went back to their data after 2016 and found that they would have been much closer to the election result if they had employed the standard education adjustments that national surveys have long used. An Upshot analysis of national surveys found that failing to weight by education cost Trump about 4 points in polling support — enough to cover much of the 2016 polling error. Other pollsters had similar findings.
But this time, education weighting didn't seem to help. State and national polls consistently showed Biden faring far, far better than Clinton did among white voters without a degree. Last week's results made it clear that he didn't.
Overall, the final national surveys in 2020 showed Trump leading by a margin of 58 per cent to 37 per cent among white voters without a degree. In 2016, they showed Trump ahead by far more, 59 per cent to 30 per cent. The results by county suggest that Biden made few gains at all among white voters without a degree nationwide and even did worse than Clinton's 2016 showing in many critical states.
In contrast, the 2016 polls did show the decisive and sharp shift among white voters without a degree but underestimated its effect in many states because they underestimated the size of the group. Many state polls showed college graduates representing half of the likely electorate in 2016, compared with about 35% in census estimates.
The poll results among seniors are another symptom of a deeper failure in this year's polling. Unlike in 2016, surveys consistently showed Biden winning by comfortable margins among voters 65 and older. The final NBC/WSJ poll showed Biden up 23 points among the group; the final Times/Siena poll showed him up by 10.
In the final account, there will be no reason to believe any of it was real.
This is a deeper kind of error than ones from 2016. It suggests a fundamental mismeasurement of the attitudes of a large demographic group, not just an underestimate of its share of the electorate. Put differently, the underlying raw survey data got worse over the last four years, cancelling out the changes that pollsters made to address what went wrong in 2016.
It helps explain why the national surveys were worse than in 2016; they did weight by education four years ago and have made few to no changes since. It also helps explain why the error is so tightly correlated with what happened in 2016: It focuses on the same demographic group, even if the underlying source of the error among the group is quite different.
Polling clearly has some serious challenges. The industry has always relied on statistical adjustments to ensure that each group, like white voters without a degree, represents its proper share of the sample. But this helps only if the respondents you reach are representative of those you don't. In 2016, they seemed to be representative enough for many purposes. In 2020, they were not.
So how did the polls get worse over the past four years? This is mainly speculation, but consider just a few possibilities:
The President (and the polls) hurt the polls
There was no real indication of a "hidden Trump" vote in 2016. But maybe there was one in 2020. For years, the president attacked the news media and polling, among other institutions. The polls themselves lost quite a bit of credibility in 2016.
It's hard not to wonder whether the president's supporters became less likely to respond to surveys as their scepticism of institutions mounted, leaving the polls in a worse spot than they were four years ago.
"We now have to take seriously some version of the Shy Trump hypothesis," said Patrick Ruffini, a Republican pollster for Echelon Insights. It would be a "problem of the polls simply not reaching large elements of the Trump coalition, which is causing them to underestimate Republicans across the board when he's on the ballot."
(This is different from the typical Shy Trump theory that Trump supporters don't tell pollsters the truth.)
A related possibility: During his term, Trump might have made gains among the kinds of voters who would be less likely to respond to surveys, and might have lost additional ground among voters who would be more likely to respond to surveys. College education, of course, is only a proxy for the traits that predict whether someone might back Trump or respond to a poll. There are other proxies as well, like whether you trust your neighbour; volunteer your time; are politically engaged.
Another proxy is turnout: People who vote are likelier to take political surveys. The Times/Siena surveys go to great lengths to reach non voters, which was a major reason our surveys were more favorable for the president than others in 2016. In 2020, the non voters reached by The Times were generally more favorable for Biden than those with a track record of turning out in recent elections. It's possible that, in the end, the final data will suggest that Trump did a better job of turning out non voters who backed him. But it's also possible that we reached the wrong low-turnout voters.
The resistance hurt the polls
It's well established that politically engaged voters are likelier to respond to political surveys, and it's clear that the election of Trump led to a surge of political engagement on the left. Millions attended the Women's March or took part in Black Lives Matter protests. Progressive activists donated enormous sums and turned out in record numbers for special elections that would have never earned serious national attention in a different era.
This surge of political participation might have also meant that the resistance became likelier to respond to political surveys, controlling for their demographic characteristics. Are the "MSNBC moms" now excited to take a poll while they put Rachel Maddow on mute in the background? Like most of the other theories presented here, there's no hard evidence for it — but it does fit with some well-established facts about propensity to respond to surveys.
The turnout hurt the polls
Political pollsters have often assumed that higher turnout makes polling easier, since it means that there's less uncertainty about the composition of the electorate. Maybe that's not how it worked out.
Heading into the election, many surveys showed something unusual: Democrats faring better among likely voters than among registered voters. Usually, Republicans hold the turnout edge.
Take Pennsylvania. The final CNN/SSRS poll of the state showed Biden up by 10 points among likely voters but by just 5 among registered voters. Monmouth showed Biden up by 7 among likely voters in a "high-turnout" scenario (which it ended up being) but by 5 points among registered voters. Marist? It had a lead of 6 points among likely voters and 5 points among registered voters. The ABC/Washington Post showed a 7-point lead for Biden among likely voters and a 4-point lead among registered voters.
It's still too soon to say whether Republican turnout beat Democratic turnout, but it sure seems possible. In Florida, the one state where we do have hard turnout data, registered Republicans outnumbered registered Democrats by about 2 percentage points among those who actually voted, even though Democrats outnumber Republicans among registered voters by about 1.5 points in the state. Here, there is no doubt that the turnout was better for the president than the polls suggested, whether they're private polls or the final Times/Siena poll — which showed registered Republicans with an edge of 0.7 points.
If Trump fared better among likely voters than among registered voters in Pennsylvania, a fundamental misfire on the estimate of turnout could very quickly explain some of the miss.
Unlike the other theories presented here, this one can be proved false or true. States will eventually update their voter registration files with a record of whether voters turned out in the election. We'll be able to see the exact composition of the electorate by party registration, and we'll also be able to see which of our respondents voted. Perhaps Trump's supporters were likelier to follow through. We might start to get data from North Carolina and Georgia in the next few weeks. Other states might take longer.
The pandemic hurt the polls
Remember those Times/Siena polls from October 2019 that showed Biden narrowly leading Trump? They turned out to be very close to the actual result, at least outside of Florida. They were certainly closer than the Times/Siena polls conducted since.
It wasn't just the Times/Siena polls that were closer to the mark further ahead of the election. Results from pollsters in February and March look just about dead-on in retrospect, with Biden leading by about 6 points among registered voters nationwide, with a very narrow lead in the "blue wall" states — including a tied race in Wisconsin, ultimately the state where the polls were off by the most.
One possibility is that the polls were just as poor in October 2019 as in October 2020. If so, Trump actually held a clear lead during the winter. Maybe. Another possibility is that the polls got worse over the last year. And something really big did happen in American life over that time: the coronavirus pandemic.
"The basic story is that after lockdown, Democrats just started taking surveys, because they were locked at home and didn't have anything else to do," said David Shor, a Democratic pollster who worked for the Obama campaign in 2012. "Nearly all of the national polling error can be explained by the post-Covid jump in response rates among Dems."
Circumstantial evidence is consistent with that theory. We know that the virus had an effect on the polls: Pollsters giddily reported an increase in response rates. High-powered studies showed Biden gaining in coronavirus hot spots, seeming to confirm the assumption that the pandemic was hurting the president.
But if Shor is right, the studies weren't showing a shift in the attitudes of voters in hot spots; rather, it was a shift in the tendency for supporters of Biden to respond to surveys.
Adding to the intrigue: There is no evidence that the president fared worse in coronavirus hot spots, contrary to the expectations of pundits or studies. Instead, Trump fared slightly better in places with high coronavirus cases than in places with lower coronavirus cases, controlling for demographics, based on the preliminary results by county so far. This is most obviously true in Wisconsin, one of the nation's current hot spots. The final polls in Wisconsin — including the final Times/Siena poll — showed Biden gaining in the state, even as polls elsewhere showed Trump making gains.
Don't forget the Hispanic vote
There's one state in particular where the polls were much worse in 2020 than in 2016: Florida, where Trump made huge gains among Hispanic voters.
What happened in Miami-Dade County was stunning. Biden won by just 7 points in a county where Clinton won by 29 points. No pollster saw the extent of it coming, not even those conducting polls of Miami-Dade County or its competitive congressional districts.
Most polls probably weren't even in the ballpark. The final Times/Siena poll of Florida showed Biden with a 55 per cent to 33 per cent lead among Hispanic voters. In the final account, Biden may barely win the Hispanic vote in the state.
What happened in Miami-Dade was not just about Cuban-Americans. Although Democrats flipped a Senate seat and are leading the presidential race in Arizona, Trump made huge gains in many Hispanic communities, from the agricultural Imperial Valley and the border towns along the Rio Grande to more urban Houston or Philadelphia.
Many national surveys don't release results for Hispanic voters because any given survey usually has only a small sample of the group. It will be some time until the major pollsters post their results to the Roper Center, a repository of detailed polling data. Then we'll be able to dig in and see exactly what the national polls showed among this group.
But if the Florida polls are any indication, it's at least possible that national surveys missed Trump's strength among Hispanic voters. It seems entirely possible that the polls could have missed by 10 points among the group. If true, it would account for a modest but significant part — maybe one-fourth — of the national polling error.
These are the initial guesses. Other theories will emerge. In time, to the extent they can be, all of them will be put to the test. And then we'll know more than we do now, and can revisit this question.
Written by: Nate Cohn
Photographs by: Janie Osborne, Hilary Swift and Doug Mills
© 2020 THE NEW YORK TIMES