NS1 Outage (Binary Lane): DDoS impacting Superloop

Monitorando

DNS appears to be returning to service as of 6.30pm, and it seems probable that they will be adopting a new network carrier:

“We’ve now met with our primary upstream and agreed a plan to return stability to the network, and that plan is being implemented now. In parallel, over the past hour we’ve continued to work with additional providers to bring online further capacity for ongoing DDoS mitigation.”

We’ll keep providing updates as we get them.

Atualizado

The root cause is a DDoS attack against Binary Lane, large enough to be toppling Superloop’s network.

“BinaryLane is responding to a large-scale denial-of-service attack targeting our entire autonomous system.

The attack has been significant enough to affect our primary upstream provider, who has had to drop our ports several times to protect their wider network — most recently because the attack resumed each time we were re-added. We are currently announcing traffic via our secondary upstream, which is keeping the platform reachable but with reduced throughput and higher latency for traffic outside Sydney, where the secondary provider’s footprint is concentrated.

We have a meeting at 4:00 PM with our primary upstream to agree a sustainable path forward, and in parallel we are working with another Australian carrier (with whom we have direct connectivity in all our DCs) to bring additional capacity online for mitigation. Customer data is not at risk — this is a network-level attack, not an intrusion. We’ll post a further update immediately after the 4:00 PM meeting. Thank you for your patience.”

Atualizado

It seems there is slow headway towards resolving this impact:

“Servers are only accessible via IXPs at the moment, which means they are still not accessible via Telstra/Optus/Voda/TPG/Superloop and most international ISPs at the moment. The techs are still working on fixing it.”

Atualizado

Superloop has confirmed this outage to their network, stating:

“We are currently experiencing a service disruption affecting the INDIGO West link due to instability.
Our team is actively investigating & working to resolve the issue.
Updates will be provided as information becomes available.”

INDIGO West is a 4,675km-long cable system offering direct, low latency connectivity from Perth to Singapore.
Binary Lane (Mammoth Media) leverages Superloop for most traffic, so are between a rock and a hard place.

We’ll consider any post-mortem outcome and changes before evaluating whether to relocate NS1 instances. At this time most services remain entirely uncontactable. We will relay all updates received. Thank you.

Problema Identificado

G’day,

We’re sorry to report that NS1 instances are currently unavailable for usage due to a Superloop issue.
This has been gradually spreading since 12.30pm today and so far the provider hasn’t published any advisory.

Edit: Binary Lane’s advisory is available here: https://status.binarylane.com.au/incidents/bq3ypjmv6vdh

This is due to a network-level event that is above and beyond our infrastructure. We deliberately have DNS hosted externally to Merlot Digital so that in a disaster event, you do not lose resolution nor 3rd-party MX, etc.

To clarify, there is nothing currently amiss with Merlot Digital network nor infrastructure in our compute space.

So far we’re aware of this having quite some breadth to it:

  • Synergy Wholesale have reported their network impact officially yet so far there’s no reasoning
  • Binary Lane / Mammoth Media is entirely dead, even their website is offline, yet status page says all OK
  • Superloop seems to be a common point with regard to network, and if so then we’re concerned by the lack of immediate failover to other links, as this likely could have been easily avoided for all parties

We were able to get slow (100ms average) ICMP ping requests into some DNS hosts, however it’s not more constantly failing. Binary’s own website is reporting Connection Refused, so we’re hoping this isn’t security-related. The root cause around the commonality will be interesting to see. We’re concerned about breadth.

  • As of 1.10pm we are starting to see some machines return to normal service.
  • It seems certain that this is related to Superloop & improper routing.
  • When updates come to light we’ll be in contact via this advisory.

Thank you for your patience during this time.

Cheers,
Merlot Digital

Afeta 3 dos Serviços:
The Network Crew Pty Ltd (TNC)

« Merlot Digital website «

Network: AS138521