Hospitals IT failure follows start of new group to fix old systems

Recent IT outages at hospitals come on the heels of controversial IT staff cuts and the beginning of a project to improve Health NZ's IT systems – but with uncertain future funding. Photo / Reece Baker / RNZ

By Phil Pennington of RNZ

An IT failure that forced some public hospitals to rely on pen and paper for 12 hours overnight Wednesday follows closely on the Government setting up a new centre to try to fix the plethora of weak old systems.

This week’s technical failure was at a commercial data centre – yet Health NZ’s plan has been that the use of exterior data centres would help stabilise its systems.

The Centre for Digital Modernisation of Health began work on December 1, with $19.5 million in funding.

An Official Information Act (OIA) response showed that funding was only until June.

“An internal funding case will be developed to identify future funding options for the centre,” Health NZ told the senior doctors’ union, the ASMS, in the OIA, in December.

On Friday it repeated that the centre had confirmed funding to June 30.

“Funding for the centre and its programmes will be progressed through Health NZ’s budget process,” said Health NZ acting chief IT officer Darren Douglass.

The centre is an addition to the agency’s digital and data ranks, after masses of cuts to it through two big restructures that put paid to hundreds of jobs and IT projects.

Staff at the time warned in internal feedback the cuts would worsen the outages.

“Without us the problems will go around and around in circles,” said one.

Unions on Friday blamed the 12-hour failure at hospitals across Auckland and Northland on the staff cuts, but Health NZ rejected that.

Health Minister Simeon Brown. Photo / Mark Mitchell

It was a technical failure in “part of our network infrastructure in one of our data centres, commercial data centre that we host a number of our systems on”, Douglass told Morning Report on Friday.

Yet Health NZ’s new 10-year fixit plan calls for more reliance on the data centres. It said that critical clinical apps would be moved out of old, at-risk servers in individual hospitals to the “cloud” in a “secure, modern national data centre”.

“This immediately reduces the risk of failure from ageing hardware or local power outages,” it said.

The outage that ended Thursday morning was the fourth hospital IT outage this month.

All four outages were technical issues, and three were due to “third-party vendor issues”, said Health NZ.

The new modernisation centre featured third-party vendors or “delivery partners”.

The centre was a “collaboration between Health New Zealand and delivery partners that brings together global innovation capabilities, artificial intelligence expertise, and world-class process engineering to co-ordinate critical investments”, said Health Minister Simeon Brown when he launched both the centre and the 10-year fixit plan at the same time last November.

Asked by RNZ about funding, Brown did not mention it.

His focus to fix the old system they inherited from the last Government was on building “reliable digital tools for staff and patients”.

The phased approach was to first put governance and capability in place, then investment cases and then move into delivery using proven international best practice, Brown said.

He did not respond to a question whether, after the four IT outages in January, he would consider boosting the centre’s funding.

Douglass said the first phase of the 10-year plan – delivered by the new centre – was to stabilise the IT system across common platforms: “The centre is addressing this through bringing together in one team digital delivery expertise and disciplines.”

The plan made stabilisation one of three focus areas: “This means less time dealing with IT outages, and more time with patients,” it said.

Senior doctors said the Auckland outage caused chaos.

University of Auckland computer scientist Dr Ulrich Speidel on Friday questioned why any hospital IT system would have a single point of failure and no back-up.

Douglass had told Brown’s office in mid-2024 that relying on the old tech would lead to “ongoing security vulnerabilities and associated breaches, more frequent service outages”, emails released previously showed.

A chief IT officer late that year told staff they could not afford to have “anything other than ... one vanilla-flavoured brown-bag common cheap solution per problem” and that continuous improvement demanded failing “early, fail often, succeed over time”.

Health NZ has been working on an IT fix since it was set up in 2022.

However, it had also cut data and digital roles and put the brakes on scores of IT upgrade projects to save $100m during 2024’s financial meltdown.

Some projects were considered crucial. Others have carried on or been newly initiated, such as Brown’s ‘accelerate’ programme to digitise patient records and end the use of paper notes for two-thirds of hospitals.

“Modernising a system this complex takes time,” Brown said at the time.

The modernisation centre had an interim director appointed last month. Recruitment for a permanent director was underway, Douglass said.

Asked what it had achieved so far and about its plans, he said: “Design of the centre has been completed and communicated, detailed processes for delivery are nearing completion and the approach to assurance has been defined.”

Business cases to develop programmes in the 10-year plan were being worked on.

The centre’s funding is from a Vote Health appropriation for “enabling health system transformation”. It is unclear if that is additional to baseline funding.

Douglass said on Friday: “We need investment, we are underinvested.”

However, he also said it had enough staff and had spread that expertise nationally.

“That isn’t removing expertise from our system, that’s making sure the experts we have can lend support where it’s needed.”

It had responded to the Auckland outage within 30 minutes, but it was intermittent so proved hard to fix, taking 12 hours.

The ASMS senior doctors’ union responded that there was “no meaningful investment ... the public deserves to know what’s going on”.

- RNZ