Prime Minister Chris Hipkins and National leader Christopher Luxon. Photo / Mark Mitchell
Prime Minister Chris Hipkins and National leader Christopher Luxon. Photo / Mark Mitchell
What are political polls? Should we trust them? And what do we need to keep in mind when reading them? The Herald spoke to two statistics professors – University of Auckland’s Thomas Lumley and Victoria University’s Richard Arnold – about these questions as we inch closer to the election.
Frequentpolitical polls are a sign a general election is near. These polls are an attempt to measure New Zealanders’ voting intentions – without actually having an election – by using a small number of participants, often 1000.
But, as the experts say, such polls are not the election and – although they can be trusted when designed well and function well as indicators of the direction of the election – there are several things to consider while reading them.
Who is interviewed by election poll companies?
Richard Arnold said it’s important to consider who conducted and commissioned the political poll you’re reading – and whether the group behind it might have an agenda.
“A poll run by a political party, especially a fringe one, is likely to be less reliable, or a lobby group that’s got a particular angle.
“Just ask that question. If it’s one of the major political organisations, you’ll be fine, but if it’s not, keep your eyes open for who’s paying. Even if they’re being sincere, it’s really easy to design a bad questionnaire where you front load a whole lot of questions on the issues that you care about as a lobby group.”
A poor approach to polling is front loading – or priming – asking certain questions that get participants thinking in a particular way, potentially guiding them to answer later questions in an intended way.
What makes a good sample in an election poll?
A sample is the people who participate in the survey. Typically, national political surveys in New Zealand should use a sample of 1000 voters.
Richard Arnold said 1000 respondents was appropriate because it was small enough to conduct the survey within a week but large enough to provide an acceptable margin of error (we’ll get to this soon).
Why, then, could you not ask 1000 people on the street or 1000 callers to a radio station?
Voting in the October 14 election is a chance to choose the people and parties that represent you in Parliament.
Thomas Lumley said more important than the number of people was that the sample was representative of New Zealand and respondents were geographically dispersed over an electorate or the country.
“You can find out opinion from lots of people yourself, but you won’t get the whole country. The same would be true if [you asked] NewstalkZB callers … they’re going to tell you about a particular segment of the country but there’s also other segments of the country that they’ll miss.
“If you tried Radio New Zealand or if you tried [a] station that plays mostly reggae … you get different people listening and so you get different results”
Political polling has become more difficult with the decline in the use of landlines. Traditionally, companies would call randomly generated landline numbers, trying to get a relatively representative sample.
Today, far more people own mobile phones instead of landlines and generating a random mobile number that is active is much harder, Richard Arnold said. There is also no prior indication of where the owner of that mobile phone is located.
Margin of error: Why should we care?
Richard Arnold: “Since opinion polls aren’t censuses, we can’t be 100 per cent sure that the party support we measure in the poll is equal to the true support in the whole population. However, with a sample of 1000 people we can be 95 per cent confident what we measure is within about 3 per cent of the actual support for a party. However that 3 per cent margin of error only applies to the larger parties - those polling close to 50 per cent. For parties with less support, the margin of error is also less.”
National polling at 35 per cent has a margin of error of 3.0 percentage points, meaning we are 95 per cent sure the actual voting intention lies somewhere between 32 per cent and 38 per cent.
Labour polling at 33 per cent has a margin of error of 2.9 percentage points, or between 30.1 per cent and 35.9 per cent.
A smaller party like Te Pāti Māori polling at 3 per cent has a margin of error of 1.1 percentage points, meaning between 1.9 and 4.1 per cent.
For smaller parties, the margin of error is a smaller number, but a larger fraction of their predicted vote. This means polls provide a better picture of voting intention for the larger parties.
It is also important to remember that the margin of error refers to an ideal “perfect poll”.
Thomas Lumley: “The standard maximum margin of error calculation underestimates the real uncertainty: it would be true for the Platonic ideal poll, but it’s not really true for polls as they are conducted”.
A political poll shows people’s voting intention on the day the survey was taken and a single poll that shows a sudden change should be treated with scepticism.
People should canvas all recent polls to get an idea of what’s going in, Thomas Lumley said.
“You shouldn’t place much faith in any single poll because a three per cent [margin of error] is a lot. That’s a huge margin in a New Zealand election and so you really can’t rely on single polls.”
Percentage of ‘don’t knows’: Are they important?
Survey participants can often choose a neutral or “I don’t know” answer in a poll. What should we make of the proportion of people who respond this way?
Richard Arnold said the percentage of “don’t know” answers are important if there are two parties with close results and these answers could make a difference in swinging the result.
He said the “don’t know” respondents often fell into three groups: centrists, broadly choosing between Labour and National; leftists, choosing between Labour and the Greens; and rightists, choosing between National and ACT
“‘Don’t knows’ don’t matter if there aren’t many of them. They’ll always split up among the parties to some extent and never vote as a single bloc together.”
‘A dirty window’: Can we trust political polls?
Richard Arnold said political polls could be trusted based on how well they had done in the past. He said they were not “wildly out” from the actual results of previous elections and were indicative of what was going on.
“The politicians are very right to say any particular poll doesn’t tell you that much because there is a lot of scatter.
“Of course, there’s always a bit of road left before the election that they can have things move and they want to be optimistic. But, they are noticing these numbers because they do tell something of a story.
“I mean, it’s through a dirty window in a sense, you’re not really seeing things clearly, but you’re getting a reasonable idea.”
Julia Gabel is an Auckland-based reporter with a focus on data journalism. She joined the Herald in 2020.
