We’ll be applying minor config updates to the new HPE switches after COB today/tomorrow.
Cellular Routers are now installed and configured at each end. Next up, tunnel config over-the-top.
Once that’s built, we’ll have emergency access to our data centre operations even with multiple fibre faults.
The new 10/40G HPE switches are now live with link aggregation! Many thanks for your patience.
Stacking and testing of the new 10/40G HPE switches was successful this afternoon.
We’ll finalise them, then finish the remaining items “slow burn”. See the revised advisory for details. :-)
We’ve sourced reverse-airflow fan trays for the HPE switches. This is due to their mounting positions.
The testing machine has had optical fibre capabilities added. This means extra testing options pre-prod.
Stop-work on Dell switches. Now working on HPE switches instead.
Following a wealth of work on the new Dell 10/40G optical fibre switches, we’ve decided to stop work on them and instead replace our existing HPE 1/10G copper/fibre switches with HPE 10/40G optical fibre switches.
We made the move to go more “all of Dell”, and it was a good test in showing us to not use Dell for networking! The over-engineered nightmare that is their Network Operating System is quite an abomination of a NOS. :-)
HPE hardware is en route and we’ll return over the coming weeks to implement the switching upgrade.
Major OS Upgrades on our 2x Core Routers were successful! There was no impact to service. :-)
We’re starting the major OS upgrade on Core Router #2. If it’s successful, we’ll also upgrade Router #1.
Testing of WAN Outage re-routing was successful. VRRP & iBGP re-routed traffic in less than 2 minutes!
Optical fibre pull testing is commencing now. Beware of brief impact/s up to 11am~ Sydney time.
We’ve a technician arriving on-site this morning who’ll be conducting “optical fibre pull testing”.
The goal is to verify whether or not the upgraded configurations will behave correctly in a failure scenario.
Impact should be limited to 5-10 minutes per router. If the Router 1 test fails, we will not proceed with Router 2.
The pre-requisite changes have been successfully completed on both routers!
Today we’ll be adjusting more network config to keep next week’s testing on-schedule.
It’s expected to knock connectivity offline for a short while, as ARP comes back online.
Internal routing peer configurations have been amended. Physical tests will be run Wed 30th Oct.
We’re applying some changes to internal routing filters. Our crew expect no-to-minimal impact.
Internal BGP configuration has been successfully deployed into the core network.
The major config (VRRP etc) works have been performed, though failover testing was unsuccessful.
Works will continue to improve internal routing logic. Then we’ll verify through another real-world test.
Preparation works on the routers is complete, and major works are now commencing.
We’re starting the Network Engineering around 3pm. Radio impact will be last.
Major network engineering works are scheduled for tomorrow.
Engineers are back on-site to continue major works. Thanks for your patience.
Our engineers needed to take a break after a full long weekend of work.
Final testing of planned network interface failover automations is underway.
Network Engineering works continue. Interruptions to service during this Public Holiday should be brief.
Cable Management has now been completed within our rack-space.
All hardware relocation works within the rack space are now complete.
Internet Radio clients are about to go offline briefly. We’re relocating hardware within the rack space.
The Dell Switches have been configured and stacked, and are now awaiting testing.
Our fibre supplier sent us the wrong order. They’ll re-send our transceivers & cables on Monday.
Hi there,
Following successful preparation works as well as research to determine the best ways forward with our Core Network in Sydney, we’ve now scheduled major optimisations and upgrades to be carried out.
This will involve some major works within AS138521:
While we’ll approach the works in a staggered manner, there will be impact to services during these works due to the major nature of the network re-engineering. The benefits from completing these works are great news for the resilience of your services moving forward, so we’re looking forward to completing them!
Following the major works, we’ll remotely finalise some internal routing/security upgrades:
If you have any queries or concerns, please don’t hesitate to get in touch.
Cheers,
Merlot Digital
Network: AS138521