DNS (Melbourne): Intermittent Network Issues
This is now persistently stable. We’ve configured a permanent MAC Address learning restriction to avoid recurrence.
Technically-speaking:
Our provider uses a fixed method for routing on LAGs (link aggregation groups), particularly for IPv4 broadcast and multicast packets - broadcast/multicast leaving a member link on a LAG can be returned through the other member link, rather than handled within their access network. The side effect of that in a hypervisor use-case, is that the LAG “bond” (group of multiple links - ie. the LAG on our side) can falsely learn guest MAC Addresses, re-learn & repeat.
So now we have sticky static records in-place in the FDB (forwarding database) on the hypervisor itself, and a hook script enabled on all guests to add the records again at host/guest boot, to ensure they’re persistent. This way it will only learn MAC Addresses onto the bridge through the TAP (terminal access point) for the guest, never from the LAG’s bond.
We’re now restarting the Dedicated Server in Melbourne to verify persistence of the fixed state.
If this is successful and the issue doesn’t return, we’ll close this out as resolved. Thank you again.
Melbourne DNS machines are stable following mitigation of the root cause. Thanks for your patience.
We’ll keep this advisory open for a while longer, just to further verify that the issue is truly resolved.
Thanks for your patience. We’ve identified the issue as false MAC learning from the link-aggregation group.
A temporary local fix has been implemented for 50% of the Guests, which so far is proving the root cause.
Our upstream provider has been given the investigation results and is currently investigating on their side.
For now, status quo is that 50% of DNS machines are still “flapping” while the mitigated 50% are now stable.
We’ll continue to provide further updates as this progresses, and appreciate your continued patience. Thanks.
G’day,
We’re investigating intermittent networking issues on the new Dedicated Server in Melbourne (DNS).
It is potentially related to the introduction of redundant networking, though we’re still investigating it.
We’ll provide further updates, however for now please be aware of sporadic issues in Melbourne.
Cheers,
Merlot Digital
- Nameserver Clusters / DNS (Brisbane & Melbourne)
- Nameserver Clusters / DNS (Adelaide & Melbourne)
- Nameserver Clusters / DNS (Melbourne & Singapore)
The Network Crew Pty Ltd (TNC)
Network: AS138521