Fixed-line telecommunications provider TPG has advised its customers of a major submarine cable outage between Sydney and Guam, which is not expected to be repaired until 8am on March 7 -- 30 days after it occurred.
The outage, which occurred shortly after 8pm AEDT on Friday night, resulted when the PPC-1 submarine cable system, which connects Australia with Papua New Guinea and Guam, lost its payload. Investigating engineers found a fibre fault on the Guam side of the cable.
"At approximately 20:00:10 AEDT on 5 February 2016, the TPG Network Operations Centre received multiple alarms for the PPC-1 cable system between Sydney and Guam. The alarms indicated that a submarine line card had lost its payload. A Network Incident ticket was created and the issue was immediately escalated to the PPC-1 transmission engineering team," TPG said in a statement on its website.
"The transmission engineering team, in collaboration with infrastructure vendors, completed preliminary investigations and have found a fibre fault approximately 4,590 kilometres from Guam. The nature for the fibre fault has resulted in an outage to the entire PPC-1 cable system. TPG has started to work with our maintenance contractor to understand when we can mobilise a repair ship to the specific location.
"Due to the nature of the fault, resolution is expected to take an extended period."
A spokesperson from TPG explained to ZDNet that it is mostly wholesale customers who have been affected by the fibre fault.
"There is a clear fibre fault about 4,652km from Guam at a depth of more than 2km. The PPC1 outage has meant that TPG Telecom has shifted all IP traffic to its redundant Southern Cross and AJC capacity.
"At this time, the affected customers are mainly wholesale customers who purchased unprotected services using the PCC1 path. Specific Asian destinations may experience an increase in latency, however TPG will further optimise traffic routing over the next few days to minimise customer impact."
The spokesperson added that TPG's maintenance contract provides for relatively fast shipments based out of New Caledonia for hardware to repair any cable faults that arise. However, the maintenance ship is already repairing a fault in the Basslink cable system, which will delay repairs on the Sydney-Guam TPG subsea cable.
"TPG Telecom has a maintenance agreement which provides the group access to the CS Ile De Re based in Noumea, New Caledonia. This ship is stocked with spare cables and other hardware for repairs to the cable system. We have been informed that the same ship is being used to repair the Basslink cable system. As a result of the Basslink repairs, extra delays can be expected and therefore TPG is currently considering other options for a faster repair using an alternative vessel. At this stage, TPG is not able to quote exact restoration times.
"Once TPG has a firm handle on the restoration times we will update our customers accordingly."
TPG customers have complained over broadband enthusiast website Whirlpool about slow international browsing.
"Since Friday night, has anyone else been noticing very slow international internet browsing? Often getting timeouts loading various web sites, or takes a very long time to load up," said one user.
Another user came up with a method to circumvent the affected cable: " I have been using a SSH tunnel into my VPS in Hong Kong to get around these issues. It goes through the Australia-Japan Cable. Singapore is also working fine, lots of cheap VPS options there, including the cheapie Digital Ocean. If you are willing to compromise on security, there are also some free vpns [sic] you can experiment with temporarily until the issue is fixed."
For the interim, TPG has rerouted customer connections to the Australian-Japan Cable system and the Southern Cross cable, which connects Australia and New Zealand to the United States.
Another Whirlpool user complained that TPG's fix was not working well, however.
"TPG's alternative routing seems to be very poor, the performance is shocking. Alas I suppose if you don't pay much for the service you can't expect they will pay for a good backup system, think i'd [sic] better find another ISP.
"Not to mention they run a major internet service and didn't even notice there were problems, then when someone reported it they said nothing was wrong - that ain't active monitoring to ensure your customers get a good service."
The Southern Cross cable was itself hit by a "catastrophic failure" in 2012, when an unauthorised software change was made to the wavelength switching platform at its landing station in Sydney.
Rival telecommunications provider Telstra had its own Australia-Singapore subsea cable outage in October last year, which caused severe delays for Telstra customers attempting to download or update their operating systems or apps across the iTunes Store and the App Store, as well as use streaming services Apple Music and Apple Radio.
Updated at 2.20pm AEDT, February 8, 2016: Added comment from TPG.