By RICHARD BRADDELL and MICHAEL FOREMAN
It is the kind of thing that is not meant to happen in modern telecommunications networks - but it did.
Telecom's smart, reasonably new ATM data network serving central Auckland went down for 20 hours, cutting off data traffic to several Internet service providers and 300 businesses and corporates in the central city.
In an industry where outages are measured in terms of seconds per decade, the unthinkable had happened.
And not only did it happen, it lasted for what in telecommunications terms is a very long time.
The question being asked in telecommunications circles is, how did it happen? Such systems are supposed to be fail-safe.
Had a contractor put a digger through a fibre optic cable servicing the exchange, everyone would have understood.
But not so the failure of a complex marriage of computer equipment from several international suppliers.
According to a Telecom spokesman, Glen Sowry, the ATM circuits went down at 11.25 pm on Sunday at Telecom's Mayoral Drive exchange.
ATM, or asynchronous transfer mode, is the workhorse of most modern telecommunications networks, providing the high bandwidth capacity necessary to serve the data needs of modern business.
But while Telecom had the ATM side of things running half an hour later, its problems had only just begun. Very few commercial users run ATM itself, using instead lower-capacity data products that run over the top of ATM.
Unfortunately, one of those, a frame relay, which is very commonly used by businesses, failed to perform when the ATM network came back up.
It was not until 6.50 pm on Monday, after engineers had discovered and replaced 11 faulty electronic frame relay cards, that the service was restored.
The question of why it took so long to get frame relay back up remains unanswered.
Meanwhile, Internet service providers affected by the outage say they doubt Telecom's assertion that its service provider, Xtra, had emerged unscathed.
"It's rubbish," said Nick Wood, managing director of Ihug.
"The whole of IPNet [the method by which Telecom delivers national Internet services] is served over ATM, so if the ATM network falls over then so does Xtra."
Mr Wood said he had received a call from a Telecom employee who told him that Xtra had "fallen over" for 30 to 40 minutes around 6.30 pm on Monday. Xtra users were also complaining on the nz.comp newsgroup of poor performance at other times.
He accused Telecom of "covering it up and mentioning everyone else's name" to draw attention from its own problems.
But Mr Sowry said Mr Wood was "incorrect." The ATM backbone failed at 11.25 pm Sunday and was restored at 11.51 pm.
As a result, 11 frame relay circuits failed, but Xtra was not using those circuits. However, Mr Sowry said Xtra had failed for 20 minutes on Monday between 3 pm and 4 pm, but this was unrelated to the earlier ATM failure.
Mr Wood said Ihug experienced the temporary closure of an international link provided by Telecom as well as sporadic failures at various regional connection points. He said the international failure was short-lived, as it was quickly bypassed by alternative arrangements with Clear and Telstra. However, several business customers using ATM lost their service completely for a short period.
Mr Wood said Telecom's ATM network had "fallen over quite regularly" for short periods in the past few weeks.
"This sort of outage can happen to anyone. The issue is why they didn't have a hot swappable back-up that automatically kicks in."
Ihug would not be seeking any redress from Telecom, as Telecom's contract specified an outage must last for more than two hours before a lack of service would be credited.
Voyager, which uses the IPNet network for its dial-up customers, managed to divert IPNet numbers to other equipment within its network before Telecom restored the service. This allowed some users to log on, according to Voyager's Web site, but the measure may have caused some delays due to overloading.
Mike Ward, spokesman at Voyager owner Uunet, dismissed as "media speculation" reports that Voyager was considering legal action against Telecom. He said it would be "unwise" for him to comment on yesterday's outage, as Voyager was still awaiting a report from Telecom.
AdvertisementAdvertise with NZME.
Latest from Technology
Humane AI Pin is more comic relief than iPhone-killer
The tech world's hottest gadget is, well, just too damn hot.