Network Overhaul: Major Optimisations to Core Network

Updated

We’ll be applying minor config updates to the new HPE switches after COB today/tomorrow.

Updated

Cellular Routers are now installed and configured at each end. Next up, tunnel config over-the-top.

Once that’s built, we’ll have emergency access to our data centre operations even with multiple fibre faults.

Updated

The new 10/40G HPE switches are now live with link aggregation! Many thanks for your patience.

Ended
Updated

Stacking and testing of the new 10/40G HPE switches was successful this afternoon.

We’ll finalise them, then finish the remaining items “slow burn”. See the revised advisory for details. :-)

Updated

We’ve sourced reverse-airflow fan trays for the HPE switches. This is due to their mounting positions.

Updated

The testing machine has had optical fibre capabilities added. This means extra testing options pre-prod.

Updated

Stop-work on Dell switches. Now working on HPE switches instead.

Following a wealth of work on the new Dell 10/40G optical fibre switches, we’ve decided to stop work on them and instead replace our existing HPE 1/10G copper/fibre switches with HPE 10/40G optical fibre switches.

We made the move to go more “all of Dell”, and it was a good test in showing us to not use Dell for networking! The over-engineered nightmare that is their Network Operating System is quite an abomination of a NOS. :-)

HPE hardware is en route and we’ll return over the coming weeks to implement the switching upgrade.

Updated

Major OS Upgrades on our 2x Core Routers were successful! There was no impact to service. :-)

Updated

We’re starting the major OS upgrade on Core Router #2. If it’s successful, we’ll also upgrade Router #1.

Updated

Testing of WAN Outage re-routing was successful. VRRP & iBGP re-routed traffic in less than 2 minutes!

Updated

Optical fibre pull testing is commencing now. Beware of brief impact/s up to 11am~ Sydney time.

Updated

We’ve a technician arriving on-site this morning who’ll be conducting “optical fibre pull testing”.

The goal is to verify whether or not the upgraded configurations will behave correctly in a failure scenario.

Impact should be limited to 5-10 minutes per router. If the Router 1 test fails, we will not proceed with Router 2.

Updated

The pre-requisite changes have been successfully completed on both routers!

Updated

Today we’ll be adjusting more network config to keep next week’s testing on-schedule.

It’s expected to knock connectivity offline for a short while, as ARP comes back online.

Updated

Internal routing peer configurations have been amended. Physical tests will be run Wed 30th Oct.

Updated

We’re applying some changes to internal routing filters. Our crew expect no-to-minimal impact.

Updated

Internal BGP configuration has been successfully deployed into the core network.

Updated

The major config (VRRP etc) works have been performed, though failover testing was unsuccessful.

Works will continue to improve internal routing logic. Then we’ll verify through another real-world test.

Updated

Preparation works on the routers is complete, and major works are now commencing.

Updated

We’re starting the Network Engineering around 3pm. Radio impact will be last.

Updated

Major network engineering works are scheduled for tomorrow.

Updated

Engineers are back on-site to continue major works. Thanks for your patience.

Updated

Our engineers needed to take a break after a full long weekend of work.

Updated

Final testing of planned network interface failover automations is underway.

Updated

Network Engineering works continue. Interruptions to service during this Public Holiday should be brief.

Updated

Cable Management has now been completed within our rack-space.

Updated

All hardware relocation works within the rack space are now complete.

Updated

Internet Radio clients are about to go offline briefly. We’re relocating hardware within the rack space.

Updated

The Dell Switches have been configured and stacked, and are now awaiting testing.

Our fibre supplier sent us the wrong order. They’ll re-send our transceivers & cables on Monday.

Began
Maintenance Planned

Hi there,

Following successful preparation works as well as research to determine the best ways forward with our Core Network in Sydney, we’ve now scheduled major optimisations and upgrades to be carried out.

This will involve some major works within AS138521:

  • ✅ Relocating some equipment within the rack space to optimise RU usage
  • ✅ Simplifying our IPv4 Network while improving SPOF resilience (VRRP)
  • ✅ Performing major version upgrades on our Core WAN Routers
  • ✅ Upgrading our Core Switches (HPE 1G/10G to HPE 10G/40G)
  • ✅ Repurposing 1/10G HPEs into OOB/Internal Switching
  • ✅ Remove, re-run, label & test every single cable
  • ✅ Add SFP+ fibre NIC to Testing Hypervisor

While we’ll approach the works in a staggered manner, there will be impact to services during these works due to the major nature of the network re-engineering. The benefits from completing these works are great news for the resilience of your services moving forward, so we’re looking forward to completing them!

Following the major works, we’ll remotely finalise some internal routing/security upgrades:

  • Overhauling our internal Virtual Private Network
  • Cellular emergency access to Sydney DC
  • Refined access for DR Storage

If you have any queries or concerns, please don’t hesitate to get in touch.

Cheers,
Merlot Digital

10 Affected Services:
The Network Crew Pty Ltd (TNC)

« Merlot Digital website

Network: AS138521