Schools’ international reading, maths results likely to be skewed - ministry

Last year's PISA test results, which show where New Zealand students' reading, maths and science skills rank compared with other countries, are expected to be skewed, the Ministry of Education says. Photo / Getty Images

By John Gerritsen of RNZ

A skewed sample is likely to have boosted New Zealand’s scores in the most recent PISA tests of reading, maths and science.

There were 4682 15-year-olds who participated in the OECD’s Programme for International Student Assessment (PISA) last year. But the group was not a representative sample and was more likely to do well in the tests than a fully representative sample, the Education of Ministry says.

The problem happened because only 72 per cent of the schools and students invited to participate did so, short of the benchmark of 85 per cent of invited schools and 80 per cent of invited students.

RNZ reported at the time that some principals refused to participate because their students were already under pressure from Covid-related disruption to their learning.

The ministry said its analysis found the results due to be published in early December would probably not be an accurate representation of New Zealand students’ performance.

“Our student sample was found to be slightly biased. The analysis showed slightly more high-achieving students participated. This means the results will likely show Aotearoa New Zealand in a more positive light than if we had an unbiased sample, potentially by about 10 PISA points.”

It said students who went to private schools and schools in richer neighbourhoods (high-decile schools) were over-represented, as were students who regularly went to their classes.

“Unsurprisingly, students who were chronically absent around the time PISA 2022 was conducted in New Zealand were less likely to take part, and students who were regular attenders were more likely to take part.”

The ministry said readers would have to keep the bias in mind when considering New Zealand’s results.

“If the data shows better outcomes than in the past, on performance or in survey data, we cannot be certain of the extent to which it is due to the full population improving or higher achievers being over-represented in the data.

“Where results show worse outcomes than in the past, we can be fairly confident that the outcomes are truly poorer.

“We cannot be certain that if there are changes, what the size of the change is. For example, country ‘rankings’ become difficult to conclude.”

New Zealand’s scores in the tests have dropped by 22 to 29 points over a period of nearly 20 years and in the previous round of testing in 2018 ranged from 508 in science to 494 in maths.

The ministry said sample bias might have affected previous years’ results and the results of other countries.

“It is important to note that the bias may not affect trend or country comparisons to its full extent, because a similar (although, likely, smaller) bias may also affect past results and the results of other countries and economies.”

University of Auckland associate professor of education Dr Aaron Wilson said PISA had become increasingly high-stakes for many countries and the ministry’s findings meant people would have to be very cautious interpreting the results. He said even without the bias in the 2022 sample, the results should not be considered in isolation from previous years.

“If we’re looking to see a pattern or a trend over time in results, even without that [bias], you need more data points anyway. You can’t conclude much from a single data point. A lot will depend on - because it’s an international comparison - knowing exactly what the level of sample bias was for other jurisdictions.”

Wilson said he was not surprised that a lot of schools opted out of the tests, particularly those in lower and mid-decile schools that had struggled most with Covid-19.

New Zealand Initiative researcher Dr Michael Johnston said a 10-point difference was “a reasonably big deal” given that reading scores had fallen 25 points in 18 years. He said it would be difficult to interpret the results with any certainty, even allowing for the difference.

“They can put a bound of 10 points on the likely extent of the error, but within that it’s very had to say where things sit.”

Johnston said schools had a lot to deal with due to Covid-19 and it was understandable that some had opted out of the tests. He said the ministry could compare the 2022 results with the results of students who sat the 2018 assessment and had similar characteristics to the 2022 sample in order to find out if achievement had changed.