NEXTDC have confirmed this was caused by their equipment faulting.
Impact duration of 160 minutes. NXT Refs: INC0084445 / TAU454547
The root cause confirmed the OBS Network device falsely detected light signals after the polarity swap, which didn’t allow the device to automatically fail-over services to the backup link.
A case has been opened with the Supplier to investigate further.
Manual fail-over of traffic by the Network Engineer from the impacted primary link to the backup link restored connectivity.
They list the below as their next steps:
We have network upgrades planned for this quarter which should help improve routing during such conditions, including on-boarding with Edge IX Sydney to bring peering online, allowing more diverse/direct routing.
Hi again,
Following over 2 hours of impact (1.01pm through 3.12pm in QLD time, without DST) it appears that the problem has primarily been resolved - with some ancillary access still not functional, but Radio etc assets now available.
The cause appears to have been internal network configuration changes within our upstream provider’s network deployed at 1pm~ today. The incident saw half of our IPv4 space be unroutable for 2+ hours.
It does not seem that the changes were advised ahead of time. We apologise for the impact.
At this stage we are monitoring the restored services, and waiting for full restoration.
Cheers,
LEOPARD.host
Hi there,
We’re experiencing impact across corporate resources since 1.01pm today (45-50 minutes as at now).
Investigations are underway. We are working with our partners behind the scenes.
Our crew appreciates your patience during this time. Thank you.
Cheers,
LEOPARD.host
Network: AS138521