Cyber-defences gone haywire are being blamed for problems with the My Vaccine Pass site yesterday - which persisted into the evening.
It appears that the system mistook the crush of visitors for a DDoS (distributed denial of service) attack, which sees an army of bots aimed at a site, with their multiple connection requests blocking out regular visitors.
"My Vaccine Pass was switched on at 9am yesterday," Ministry of Health group manager, national digital services Michael Dreyer told the Herald this morning.
"As expected, we had a lot of people across the country keen to get in early, so the My Covid Record website came under significant pressure.
"Because of the sheer number of people attempting to get a pass, some of our automated controls for restricting access kicked in. These controls are designed to prevent the site being taken down in the event of a denial-of-service attack.
"This is now all working well and by midnight we had processed 200,000 My Vaccine Passes. We will continue to work with our cloud platform partners to optimise performance today."
Dreyer also confirmed that Amazon Web Services is hosting the My Covid Record site, which "uses a mix of AWS and Microsoft Azure services."
Later, an MoH spokesperson added, "We would just like to clarify that AWS wasn't involved in the outage earlier this morning and their systems are operating as usual. There was no malfunction of AWS systems."
Yesterday, the Herald also found the ministry's toll-free number jammed.
"Our 0800 channel to support those without access to technology to get their passes is experiencing steady load and coping well."
Some tech industry experts were surprised that problems with the Vaccine Pass site continued late yesterday - some of which appeared to be the result of underlying glitches rather than the volume of visitors.
Meanwhile, a number of questions lingered for users and businesses about how the new system will work in terms of scanning, enforcement and fraud protection (and enforcement is potentially made trickier by the MoH emailing the Vaccine Pass out as an editable rather than read-only PDF - although if you change, say, the expiry date, it won't match the details that appear on the verification app when someone scans your QR code. If the verification app lives up to its spec, it will reject any automatically reject any QR code with an expired verification date).
Ben Gracewood - who held chief engineering roles at Datacom, Vend and Westpac before becoming chief technology officer at The Spinoff - defended the Ministry of Health's development team through much of the day.
Early on, as the My Covid Record Site (where people register for a My Vaccine Pass) crashed under a heavy load, Gracewood called for people to take a deep breath.
Just managed to get a Vaccine Pass fine, but a lot of others had issues, especially this morning.
— Chris Keall (@ChrisKeall) November 16, 2021
There's no where to use it yet, so no rush https://t.co/6b0M1xHJOe pic.twitter.com/k8S8mFh5kX
"Why is the site down? Because all y'all keep trying to log in. It's basically the digital equivalent of lockdown toilet paper. Just chill," he tweeted.
But around 9pm he posted, "I could understand a few minutes for them to understand the load profile and tune the scaling, but not an entire day."
After the new information emerged this morning, Gracewood said it was understandable.
"A highly sophisticated DDoS attack will look exactly like a whole ton of over-eager Kiwis. Software is dumb and cannot tell the difference, so will block both," he posted in response to a Herald update.
"Computers will always be stupid, we should just stop using them."
However, another tech industry figure, JD Trask, founder of Wellington-based, globally-operating software testing firm Raygun, responded, "I'd suggest Raygun gets more traffic at a fairly consistent rate. One customer of ours a while back, we tracked 87m users, concurrently, on their applications. You get what you plan for."
Members of the public also contacted the Herald to say they were either unable to access the site, or were getting "Something went wrong" messages that indicated something going haywire with the verification system used for My Covid Record.
"Nanogirl" Dr Michelle Dickinson called for patience yesterday evening. After one user voiced their frustration with the ongoing problems, she replied: "You could just wait a day and go when it's less busy, a bit like deciding on when to go to the supermarket."
Covid-19 Response Minister Chris Hipkins also told people while the system could issue 200 certificates a second, it would take a couple of days for the crush to subside.
Certainly, there was no immediate rush. Vaccine Passes won't come into effect until the traffic light system is introduced - at which point the likes of hospitality venues and gyms will scan your Vaccine Pass QR code before you enter.
But many businesses were unsure how the system would work from their end - either in terms of how they would scan QR codes, how they could prevent fraud or at which point they were required to scan people.
And many who were not NZ citizens or were planning to visit NZ were confused over if - or how - they could apply for a My Vaccine Pass.
The Government has an official verification app in the works, and will also make the code available for organisations to use in their own apps.
Curious to see how practical this is. Every cafe and restaurant going to have to employ bouncers?
— brewsawa (@brewsawa) November 17, 2021
International visitors will be able to display an equivalent vaccine-confirmation status document.
Retail NZ chief executive Greg Harford told Morning Report they had concerns about staff being able to police the rules, and whether all staff would need to be vaccinated also.
"It is going to be a little bit problematic for those businesses that might need to put extra staff on the door, for example, particularly if you're a cafe ... you've got less income coming in and then additional costs to have someone on the door. We really need to work that kind of thing through."
Hipkins said businesses like bars were familiar with having to check customers' age.
He also gave assurances police could be called on to help, and the infringement regime which introduces greater fines and punishments for rule-breakers would provide some extra tools.
"Police of course are available to provide support around enforcement, and they've been involved in this process and the discussions around this process so they know what's coming and they will certainly be there," Hipkins said.
Businesses would be able to conduct their own ID checks on top of scanning a My Vaccine Pass, which contains a person's name, birthdate and vaccination status.
Despite the problems, many people did manage to register and receive a My Vaccine Pass (once you complete registration on My Covid Record, you are emailed a link for adding your Vaccine Pass - which includes a QR code to your smartphone's Apple or Google Wallet. You also receive your pass as an email attachment PDF, which can be printed out. Another option is to phone the Ministry of Health on 0800 222 478 to request a hard copy of your Vaccine Pass be snail-mailed to you. The number was overloaded yesterday).
At her 1pm briefing, Prime Minister Jacinda Ardern said 60,000 had received a My Vaccine Pass (all up, some 4.3 million are eligible).
Hipkins said the Government was likely to set up walk-in facilities for those still having problems.
Tech experts told the Herald that a contract for developing the technology that underpins the My Vaccine Pass app - signed on October 13, according to a timeline supplied to the Herald by the MoH - should have been awarded much earlier.
Asked about the timing, Ministry of Health group manager national digital services Michael Dreyer said, "The use of vaccine certificates is a significant decision and we took time to assess their effectiveness and use overseas before the decision was made to use them."
The industry faces challenges but hopes to bring newcomers and veterans together.