That inconsistency is on a lot of people's minds today, but Civil Defence says the first test worked "seamlessly" and met expectations.
The alert was sent out between 6pm and 7pm on Sunday to all mobile phones in New Zealand but was only picked up by around two million cellphones.
Sarah Stuart-Black, director of the Ministry of Civil Defence and Emergency Management, said the test met expectations, going to around a third of New Zealanders.
"The initial signs are that the test was successful. People received the alert from the top of the North Island to the bottom of the South Island so the system itself worked seamlessly," she said.
Leanne Brown, councillor for the Mount Maunganui/Papamoa ward, said she posted on her Facebook page as well as a couple of community Facebook pages about the test.
"The feedback has been very mixed on those platforms, with an even spread between those who received it and those that didn't," she said.
"Of those that didn't receive it, it seems that many have not got the latest update of their model phone – [for example] iOS 11 for iPhones – so didn't get the warning."
Brown said the nationwide alerting system would be developed over time and was another form of communication "to add to the toolkit" along with texting, emails, radio, social media, TV, and, in some areas, sirens.
"Tauranga City Council are working through emergency warning sirens and a decision will be made as part of the Long Term Plan next June."
Bay of Plenty MP Todd Muller also received a lot of feedback on Sunday after he asked people on Facebook whether they had received the alert.
He said people's views were "pretty mixed".
"There were those who had received the alert and that group of people – and it's probably roughly about half – said that they were pleased, basically, that it worked for them and they found the noise quite loud. It got their attention and some of them it freaked them out a bit, until they read the message properly."
Muller said the other half generally either received nothing, or their phone was on silent and the alert did not sound out.
"I suspect it was a phone compatibility issue. I understand that not all phones yet have been integrated into the alert system which, simply, has to be sorted in my view."
The issue with phones on silent was also something that needed to be explored further, he said.
"There doesn't seem to be a mechanism for anything to overrule that."
Muller said he and his family got the alerts.
"Between myself, my wife and the two kids, I tell you, it was all on. It all happened within five seconds of each other so it certainly alerted the Muller household. In fact, we had to calm the youngest down ..."
He said he was a strong advocate for the needs of the Papamoa community to be taken seriously by Civil Defence and the council and this was a "hugely important" issue for them.
"I still remain concerned that the Papamoa community generally hasn't got enough roads out of the community and I'm very strong on the importance of the incoming Government committing to the connector of Papamoa onto the Eastern Link ..."
Muller said there also needed to be multiple notification sources.
"You can't rely on just text messages or just the siren system. It has to be a mixture of both and also people looking after each other. If something happens and you get woken at 3am in the morning, yes you quickly look after your family, but you also ring your next door neighbour or shout out to them to make sure they heard too."
Civil Defence's Sarah Stuart-Black said there was some variability in how different handsets behaved and more people would be included in the system over time as they upgrade their phones or download the latest software updates.
The technology could not be picked up by all models of phones.
She said they were looking at this through a survey people were encouraged to complete online following the test.
"One of the things that will come out of the survey is how the technology works with different phones.
"Every handset is different, the alert overrides silent mode on some of them but not others."
Stuart-Black said the survey would help Civil Defence figure out which handsets worked with the system and which didn't.
She also encouraged people to look at the setting their phone was on when the alert came through and note whether it made a sound.
Around 15,000 people had filled in the survey by late on Sunday night.
While there were still some people who complained they hadn't been warned the alert was coming, Stuart-Black said Civil Defence had done its absolute best to inform the public.
"We always understood it wouldn't have connected with everyone but we had a consolidated effort in the lead-up to make sure people knew it would happen."
Why you didn't receive the alert:
Your phone was in flight mode
*If your phone was on aeroplane mode you will not have received the alert.
*Civil Defence spokesman Anthony Firth said the setting effectively did the same as turning your phone off.
Your iPhone doesn't have the latest update
*If the software on your iPhone isn't up to date you likely will not have received the alert.
*If your phone is on Civil Defence's list of mobile alert-capable phones then you should check that your software is updated.
Your model of phone isn't supported
*Civil defence has a list online of the phones that are compatible with the alert system.
*If you have an older or unusual model of phone it might not be able to receive Emergency Mobile Alerts.
You have unselected Emergency Alerts on your phone
*The setting to opt in or out of Emergency Alerts can be found under "settings" on most smartphones.
*Make sure you have this turned on.
- Additional reporting: NZ Herald
