Covid-19: Review confirms 'limitations' in Health Ministry's datasets

Stats NZ has recommended a raft of improvements to the Ministry of Health's own population dataset, widely criticised for under-counting Māori Covid-19 vaccination rates. Photo / Michael Craig

Stats NZ has recommended a raft of improvements to the Ministry of Health's own population dataset, which has been widely criticised for under-counting Māori Covid-19 vaccination rates.

That dataset is now to be updated, with 233,000 people being added to it next week - resulting in the two-dose coverage of the eligible population aged over 12 falling from 95 to 90 per cent.

Among Maori, coverage rates will drop from 88 per cent to 83 per cent, and for Pacific peoples, from 97 per cent to 89 per cent.

The Public Health Agency's deputy director-general Dr Andrew Old however maintains the ministry's dataset is "ideal" for supporting vaccination efforts.

This morning, Stats NZ released its long-awaited review of the ministry's Health Service User population (HSU), following debate among experts over whether it was suitable enough to be used as a denominator for measuring Covid-19 vaccine coverage.

In its reporting on Covid-19 vaccination rates to date, the ministry has been using a particular dataset called Health Service Utilisation 2020, capturing those who used the health system in 2020 with an age cut off at June 30 that year.

This meant the vaccination rate for New Zealanders aged over 12 was calculated on those who were that age prior to June 30, 2020 - and only if they had accessed health services.

The Herald earlier reported statisticians had been pushing the ministry to use a dataset that has a better capture of the entire population, but it had maintained HSU 2020 still gave an accurate picture of the population.

There has been pressure, particularly from those working in Māori health, because HSU 2020 was known to undercount Māori - a group that was likelier to use health services less, and sometimes identify as another ethnicity when doing so.

That was a criticism endorsed by the Waitangi Tribunal's finding in December that the Ministry of Health's data collection "does not collect sufficient data to accurately and equitably inform the rollout of the vaccine for Māori".

In May, work by the Herald's data team showed the HSU data had partial vaccination at 96.4 per cent of people aged 12-plus and full vaccination at 95.2 per cent of people for those aged 12-plus.

When Stats NZ's December 2021 population estimates were used, the percentages came out significantly lower with 93 per cent of those 12-plus partially vaccinated and 91.8 per cent aged 12-plus fully vaccinated.

If the whole population was taken into account — that's including children under 5 years — then 84.2 per cent were partially vaccinated and 80.4 per cent were fully vaccinated, at that point in time.

Otago University epidemiologist Professor Michael Baker has also said the lack of consistency across datasets was "a real problem".

"If you have a service you want to deliver to 'everyone', you have to agree what 'everyone' is," he told the Herald in June.

"If you're not measuring what you think you're measuring, we've got a problem. It's fundamental to running a vaccination programme to know your coverage."

Such inconsistencies between datasets were acknowledged in Stats NZ's just-released review, which also confirmed there'd been lower representation of Māori in the HSU.

Yet the review also found the HSU had "some advantages" as a denominator for health-based analyses, including for the monitoring of Covid-19 vaccine coverage.

One was a "consistency of reporting" of Māori and other ethnic groups in the numerator compared with the population denominator – including in vaccination rates.

"This use is consistent with Stats NZ's advice that customers should validate the consistency of the numerator with the denominator when undertaking calculations by ethnicity," the review found.

Another advantage of the HSU was its inclusion of visitors from overseas who may be using specific health services.

"This may be especially relevant if they are in New Zealand for several months, but not long enough to be statistically defined as a resident."

However, the HSU carried "some limitations" - notably for time series analyses because not everyone resident in New Zealand was in the HSU and its coverage varied over time.

Stats NZ said use of the HSU also didn't preclude use of other population statistics – such as its own estimated resident population (ERP) dataset - for health denominators.

"Indeed, the focus of Stats NZ's estimates on the resident population means they are more suitable for some purposes, especially if the visitor (non-resident) use of the specific health service is inconsequential."

The review also found the latest HSU dataset – capturing 2021 – better aligned with Stats NZ's own population projections than the 2020 set, indicating that vaccination data had ultimately improved the HSU's overall coverage.

Ethnicity differences between those two population measures - especially in Māori and Pacific populations – reflected the "array of collection and sourcing challenges" that existed within administrative data, it said.

"The lower representation of Māori in the HSU requires careful examination at source to identify whether it is driven by systemic issues, or genuine reporting choices by health service users."

It also flagged issues with age-based counting.

"Comparison of the age distributions between the 2020 HSU and 2021 Stats NZ projections shows that decisions around the use of a static and outdated HSU contributed to inaccurate denominators in Covid-19 vaccination coverage estimates for older ages (for example, resulting in vaccination rates greater than 100 per cent)," it found.

"Ageing the 2020 population into the correct ages for 2021 largely mitigates the discrepancies in older age groups."

Among its recommendations, Stats NZ said HSU should be derived "as at" key reference dates – so emphasising the data was captured at a certain point in time like June 30 or December 31 - before deriving mean populations from them.

"Separating these steps simplifies the demographic accounting, as well as providing alternative but consistent HSU estimates for different purposes."

The review also recommended all live births registered in New Zealand should be included in the HSU according to the date of birth of the child, regardless of their stated residence – and that all deceased whose deaths were registered here also be included.

It also suggested ethnicity data be collected and reported according to "total response" rather than "prioritised", to better reflect the ethnicities reported by people using health services.

"Ethnicity data should be collected at the most detailed level of the ethnicity classification practicable, to give ultimate flexibility in reporting."

When it came to how this data should be shared, Stats NZ said the HSU should "adhere to the principles of official statistics in its derivation and dissemination".

"This is because the HSU is an official statistic in its own right, but also used to derive other official statistics."

In particular, Stats NZ said the rules by which people were added to, and removed from, the HSU, should be openly published – and also be explained in technical and simpler terms for people.

In response to the review, Old said its recommendations had either been implemented or were in the process of being – with one exception relating to a "technicality" in ethnicity reporting.

"Importantly, we're committed to updating the data set every six months, which will help to keep our reporting as up to date as possible into the future."

The 2020 dataset would be replaced from next week – adding another 233,000 people to it.

While the update would change the reported estimate of vaccine coverage across the population, Old said this would make no difference to the total number of Kiwis vaccinated.

Old said the total number of people aged over 12 who'd had at least two vaccination doses had increased by 9.6 per cent in the past eight months – going from 3.63 million on December 1 to 3.98 million on August 1.

"However, when we switch to the new data set next week, our reported vaccination rates will show a drop, because as I said, this updated dataset has 233,000 more people in it," Old said.

"So put simply, it's the same number of people vaccinated, but we're now dividing that by a bigger number."

Using that larger dataset, the two-dose coverage of the eligible population aged over 12 would fall from 95 to 90 per cent.

"For Maori, it goes from 88 per cent to 83 per cent, and for Pacific peoples from 97 per cent to 89 per cent," Old said.

"Importantly, rates for all ethnicities for our most vulnerable people aged over 65 remain above 90 per cent," he said.

"And overall New Zealand remains one of the world's most highly vaccinated countries for Covid-19."

Old saw the HSU population count as "ideal" for this role, as it "represents real people who are here in the country interacting with the health system".

"That means that we can target and plan our campaigns both at a community level, for example, identifying areas where coverage is low, but also at an individual level."

Under another improvement to reporting, the data would reflect peoples' current age, rather than their age at the time of their last vaccination.

"This is because we need to better understand who in the population is eligible for boosters as we go through," Old said.

"So this change results in a more accurate indication of who is eligible for both boost boosters and first vaccinations as new people into the various age brackets."

The ministry was also removing the term "fully vaccinated" from a person's vaccination status and replacing it with completed primary course.

"So previously, a person was deemed fully vaccinated immediately after the last dose of a Covid-19 primary vaccination course," he said.

"So for most people, that was two doses, and that definition did not include boosters.

"Now a person is considered up to date with their vaccinations if they have received all recommended Covid-19 vaccinations.

"So in other words, they've completed both their primary course and any booster doses that they are eligible for, according to their age and other factors."