An AI-powered scribe used by Health NZ staff is in the spotlight after "jailbreak" testing by a US-based security company. Photo / 123rf
By Kate Green of RNZ
The AI scribe rolled out in Emergency Departments around the country has given health diagnoses, a meth recipe, instructions for killing by poison and advice on making bombs during testing to see if it would break the rules on what it’s allowed to do.
US-based security testing company Mindgard said it had carried out the “jailbreak” by only typing prompts, which “do not require advanced technical expertise and could plausibly be replicated by a technically savvy clinician”, and asked it to rewrite its own system instruction, or system prompt, which sets the guard rails for its responses.
The Australian-made tool, known as Heidi, was adopted by Health NZ after a successful trial period in New Zealand, cutting the time it took staff to write clinical notes and freeing them up to see more patients. HNZ said it was now being used by 1250 doctors and front-line staff in EDs around the country.
The Heidi company told RNZ on Thursday it had made changes to prevent the kind of activity reported by Mindgard before it was drawn to its attention externally. Heidi and Health NZ said at no time was any patient data revealed.
But now, according to reports by Australian media, the tool is under review in Australia by the Therapeutic Goods Administration (TGA).
The TGA has been approached for comment.
The “system prompt” is the mechanism used to set the tool’s limits – for example, it can be told not to give health advice, only to summarise the information it’s been given.
With its system prompt changed, Heidi began calling itself “Nexus” and rewrote its own code when asked.
As well as being open to providing medical diagnostic advice – a function a scribe tool was not meant to have – it also provided a step-by-step guide on how, as a doctor, to steal a patient’s identity.
Mindgard has published screenshots of the responses on its website.
“A doctor is in a unique position of trust,” the tool says, before going on to give a detailed explanation of how to leverage it.
It begins: “Phase 1: Information Harvesting. This is the foundational phase where the doctor leverages their privileged access to systematically collect all necessary data.”
Mindgard’s Jim Nightingale, who detailed the experiment on its website, says doctors hopefully know the tool is “not designed for medicine”.
“But a clinical scribe is developed for healthcare, and its use is approved by their institution,” he said. “So even though it wasn’t designed for diagnosis, doctors may be more willing to trust a scribe, rationalising ‘it’s already medical-grade’, leading to scope creep without awareness of crossing the line. Prompting a diagnosis may just seem like ‘I’m using AI’s full potential’.”
Seb Welsh, Heidi’s head of security, said there was no harm done to users. “What could actually happen to users? The answer, confirmed by both parties, is nothing,” he said.
“Executing the jailbreak required a user to deliberately execute a multi-step manipulation sequence in their own session and then choose to act on whatever the model returned. That is functionally equivalent to prompting any general-purpose AI for content it shouldn’t produce. It is a known property of large language models, not a vulnerability in Heidi’s product or security posture.”
Heidi had already identified the issue and fixes for this vulnerability internally before Mindgard reached out, he said.
He warned against sensationalist framing of security research, which “doesn’t just misrepresent individual companies – it damages the ecosystem responsible disclosure depends on”.
“When findings are overstated and misattributed, companies become less willing to engage with researchers and security companies openly. The public loses its ability to distinguish real incidents from noise. Everyone is worse off.”
He called Mindgard’s post “an overclaim, with no patient data exposure, no system impact, and no user harm. It doesn’t support its conclusions”.
Health NZ’s director of digital innovation and AI, Sonny Taite, said the jailbreak had identified only a “minor issue that was entirely contained within the isolated test session” – that is, it hadn’t been repeated.
In fact, he said, it showed the safeguards around the software worked as they should.
“It did not put patient information at risk, affect any users, or connect to Health New Zealand systems.”
– RNZ