A 12-hour IT failure that created chaos in North Island hospitals was caused by a single hardware component at a commercial data centre and was not impacted by recent job cuts, a health boss says.
Speaking on Ryan Bridge TODAY, Health NZ’s acting chief IT officer for digital services Darren Douglass said the outage on Wednesday night was caused by a hardware component failure.
“A lot of our components are pretty old and that is part of our challenge,” he said.
“We have infrastructure of varying ages and varying complexity, and part of our drive is to modernise, part of our drive is to get those foundations stable because we know that clinicians rely on our systems, and it’s really not acceptable for them to not be there.”
But the Public Service Association (PSA) says it was “pure luck” the incident didn’t have more tragic consequences, with its union members reporting “constant near misses” and patients receiving insufficient care.
The outage at hospitals in Auckland and Northland left clinicians unable to communicate with each other or access patient records and forced them to use pen and paper and whiteboards.
The PSA is pointing the finger at cuts to Health NZ’s IT team, which it says left too few experts to fix outdated IT systems when they break down.
Douglass said technicians had responded “very quickly” to the intermittent technical issue, drawing on teams locally and multiple vendor partners overseas.
“In terms of restoration, that is where it’s taken time. We don’t believe that is due to us not having the expertise either internally or with those external partners.”
He said structural changes within Health NZ had not influenced the delays, despite 200 unexpected vacancies within the organisation causing “transition pain”.
“What we have now is an aligned structure, and that aligned structure allows us to respond with expertise where it’s needed,” he said, pointing to technology underinvestment as the biggest service delivery obstacle.
“We know that we need to take a national approach in order to drive maximum value out of the money we do spend. So, we need more money, and we need to spend it in the right way.”
The outage caused “a night of chaos and constant near misses”, PSA national secretary Fleur Fitzsimons told Bridge this morning.
“We’re very lucky that more significant tragic consequences did not arise. It was pure luck.”
Problems included issues accessing people’s blood type, and patients missing out on care in theatre, she said.
Patient Elinor Kennedy, who was at Auckland City Hospital emergency department when the systems crashed, called the ED “chaotic” and past overflowing, with nurses helping with admin tasks and one transcribing notes by hand.
Another patient in the same ED told the Herald his stay was increased perhaps 1.5 times by the outage, though he had “nothing but praise” for staff, who had quickly implemented their backup plan.
“The place fell into what seemed like mostly organised chaos. No doubt it could have been a lot worse if it wasn’t for their very competent handling of the situation,” said Nico, who asked his last name not be used.