Spark chief executive Jolie Hodson: "It wasn't a [Spark] management issue. It was our partner and how they had configured the channels." Photo / Leon Menzies
Spark chief executive Jolie Hodson: "It wasn't a [Spark] management issue. It was our partner and how they had configured the channels." Photo / Leon Menzies
The finger has been pointed at content delivery giant Akamai for Saturday night's Spark Sport streaming fail - which saw thousands receive substandard video, and the decision made to simulcast the All Blacks-South Africa game free on TVNZ.
But a leading industry expert says that it looks more like Spark should carry the can on all levels, from technical to management.
Speaking to the Herald earlier on Monday, Spark chief executive Jolie Hodson said: "The issue that we identified was a config issue of the video stream coming in from the US, which we rectified over the 24 hours that followed.
"An international partner that we work with, the way they were configuring streams to come into New Zealand - it was getting congested in a particular channel.
"In simple form, if you think of it as lanes on the motorway. They had too much going down one particular lane," Hodson said.
A duff algorithm was apparently to blame for the misdirection of traffic. It had to be manually unpicked.
Streaming problems graphic
Hodson did not want to name the international partner concerned. But another person close to the situation bluntly identified it as Akamai Technologies, the US-based content delivery giant whose former clients include Netflix, and whose current roster includes the likes of NBC and Fox Sports.
And based on a July 8 media briefing by Spark Sport head Jeff Latch, it could only be Akamai.
Latch detailed the live Rugby World Cup feed would come down from Japan via satellite or fibre to Spark's broadcast partner TVNZ.
TVNZ would then add its production elements, including ads, then send the post-production feed up to Spark's US-based streaming partner, iStream planet, which encoded the video feed for a multitude of different devices.
iStream, in turn, would hand off the feed to Akamai, which would deliver the Spark Sport stream to a never-disclosed number of points around New Zealand.
One person close to the situation told the Herald that Akamai had failed to optimise its delivery for a typical Saturday night, where Kiwis were loading up on Netflix.
Regardless, shouldn't Spark have been monitoring everything and ensuring that Akamai was following plan for what was always going to be a huge Saturday night, and very likely Spark Sport's busiest day (given the All Blacks face soft competition in the rest of their pool games and likely quarter-final, and the semis and final will be free on TVNZ).
The Herald asked Hodson if Spark was passing the buck. Wasn't it up to Spark to manage everything.
"It wasn't a management issue. It was our partner and how they had configured the channels," she said.
However, the problem seems to be not as simple as an international partner dropping the ball.
A manager at an internet service provider told the Herald that it appeared Akamai had "broken peering protocol" and taken an unsual approach to configuring its servers.
He said there had been poor communication. "Spark need to be more upfront on the technical side with the rest of us," he said.
He forwarded the Herald the transcript of a private chat between ISP owners and managers to prove his points.
The ISP owner added, "There is also a lot of contempt towards Spark because they had the rest of the industry invest tons of money and then they did a 'classic Telecom' and screwed everyone on the wholesale deal."
In the build-up to the World Cup, Spark met with Vodafone, 2degrees, Vocus (owner of Slingshot and Orcon) and other ISPs, who agreed to accelerate network upgrades to facilitate better World Cup streaming in return for keenly-priced wholesale deal to resell Spark Sport Tournament passes.
But after Spark launched its early-bird pricing before its putative partners had a wholesale deal on the table, the spirit of cooperation evaporated. Bad blood emerged - with things descending to the point where Vocus and 2degrees bosses were openly bad-mouthing Spark.
More, industry expert, John Humphrey says it just doesn't wash to blame Akamai.
Humphrey, who currently works as an international consultant, was formerly a network and business strategy manager for Telecom and a co-founder of the Pacific Fibre bid to build a new trans-Pacific Cable.
"Blaming Akamai sounds far too easy," Humphrey told the Herald.
"Spark should have known the bandwidth-provisioning requirements based on the numbers of subscribers combined with decent forecasting for last-minute sign-ups.
"Always overprovision the expected requirement to ensure a good viewer experience is a good OTT [over-the-top or streaming] operator's and ISP's mantra."
"Spark's bandwidth-provisioning planning and network monitoring and responsiveness let them down. They just underestimated the network demand or ran into bandwidth restrictions that they could not handle at the time."
Humphrey said while there could have been an international issue, "It's Spark bandwidth to NZ not Akamai's. Akamai just runs the CDN [conente delivery network] capacity which is easily incremented on the fly. It was up to Spark to manage this and make sure there was sufficient bandwidth available. Southern Cross and Hawaii have a lot of spare bandwidth available.
"Capacity from the game to the undersea cable head-ends was possibly managed by Akamai and this is what Spark may have been referring to. If that was the case, Spark should have been checking Akamai and overseeing what they were doing."
Humphrey added, "Part of the challenge for Spark is that this was feed by unicast IP video streams from a CDN - somewhere in NZ- to each subscriber. A lot of simultaneous unicast streams occupy a lot of bandwidth. This is why it is so easy for Sky to do this and so hard for Spark. Sky can use multicast from the satellite to each home with no stress of the "network" resulting. The true power of satellite is in its one-to many usage.
"This is a common problem in many countries. The US has an OTT streaming challenge. Up to 70% of their domestic Internet usage in the evening is OTT IP Unicast video streams and if you are in an area with poor mid- and last-mile connections, you get buffering, lag, stop/start etc that we don't often experience in NZ. NZ has one of the best terrestrial networks in the world now but even it can get overloaded.
"It was not quite good enough and they are learning a hard PR lesson as a result. The next AllBlack games will show whether they have sorted this out or not."
Asked if Spark would be seeking redress, Hodson said her company had worked constructively with its international partner to implement a fix for Saturday night's problems by Sunday afternoon.
She was confident the problem would not be repeated, and Spark was returning to streaming-only coverage for last night's Wales-Georgia clash.
