Telecom NZ reveals outage's 'root cause'
Telecom New Zealand has blamed "traffic surges" in a Christchurch Radio Network Controller as the "root cause" of the major 3G network outage that affected 100,000 customers on 27 January.
Telecom has said a hardware failure caused traffic surges which overloaded the Radio Network Controller (RNC) in Christchurch and affected XT coverage south of the central North Island town of Taupo.
"The traffic surge was caused by thousands of users suddenly re-registering after a separate network routing fault took down some cell sites," Telecom CEO Paul Reynolds explained in a media statement yesterday.
Telecom spokesperson Mark Watts today confirmed a hardware failure on 27 January as causing the problems. "A piece of equipment failed. It had a flow on effect of extremely high re-registering and one thing led to another," he said.
In his statement, Reynolds also detailed capacity improvements and other enhancements to help prevent a re-occurrence of problems with the XT network. However, more outages this week on Monday and Thursday stemmed from Telecom actually attempting to make the XT network more resilient by installing a new RNC in Christchurch.
"We planned to migrate cell sites to the new RNC. Unfortunately, what happened was the migration of the 20 or so sites did not work. We struck some pitfalls and put it on pause. We will suss out the issue and resume the migration as planned," he said.
However, no full explanation has been given for Monday's problems, which mainly affected texting, other than it was "volume related" according to Watts.
The problems came as Reynolds also revealed the consultants to carry out an Independent Review into the XT network. The review by UK-based Analysys Mason Group (AMG) will commence next week and cover the design, build and operation of the XT mobile network. It is expected to be complete within eight weeks.
"AMG are bringing a multi-disciplined team of up to seven specialists who have in depth knowledge of the design, planning and operation of 3G and IP networks from a range of technology suppliers," Reynolds said in a statement. "During the review Analysys Mason will be engaging with Telecom staff, customers, Alcatel-Lucent and other key stakeholders," he said.
The XT network was supplied by Alcatel-Lucent, which said in the statement that the problem was a high priority. "Resolving the XT network problems in New Zealand is the single most important operational matter in the world for Alcatel-Lucent right now, our entire executive and senior engineering team, including global chief executive Ben Verwaayen are involved," Rajeev Singh-Molares, president Asia Pacific of Alcatel-Lucent, said.
"The full resources of our global business are available to address these problems — we will do whatever it takes."