DR: Systems & Storage (SYD)
Back-ups: Storage Connectivity; System Rebuild
Backups have successfully run overnight from BackBork KISS, and off-site cloning will resume shortly.
Many thanks for your patience as we’ve worked through this. We’d been using JetBackup for 6+ years!
Sadly, they never sorted their demons - JBv5 was too unreliable (& encapsulated) to resume with.
Testing has been completed by 3x different companies (cPanel, Merlot Digital & Velocity Host), all OK.
We have now published v1.4.0-RC of BackBork KISS, and will be deploying to all servers this evening.
It will run its first backup iteration overnight and we’ll then fine-tune throughout this week coming.
Client test backups initiated earlier are on-track, and we aim to have all machines back-up overnight.
Velocity Host Pty Ltd are going to be conducting internal testing over this weekend coming.
From there, the final work is to bed down any straggler features in BackBork KISS.
Thank you again for your patience throughout this backup “project”.
BackBork KISS testing has been successfully completed, with a larger test starting at 12pm.
From there, all machines will have the new system deployed, configured & initiated ASAP.
We’ve built a WHM Plugin which wraps existing cPanel-shipped tooling to deliver what JetBackup did.
This has been done in association with Velocity Host Pty Ltd, who provide TrueNAS Storage (not to us… yet).
After intensive development, the first beta release v1.3.0 has been published and is undergoing testing now.
https://github.com/The-Network-Crew/BackBork-KISS-Plugin-for-WHM
We expect to have backups restored by Monday.
Thank you for your patience.
Update: Retention periods have been adjusted to 4x Monthly & 20x Weekdays (4x Weeks) hereon.
NAS dataset and isolation configurations have been successfully finalised, tested and deployed this morning!
We’ve decided to replace JetBackup due to its proprietary data storage & on-going years-old major bugs.
It’s being replaced with native functionality within cPanel & WHM, which lacks hourly - but includes SFTP.
- This means we’ll have to address the “restore only from local” restriction, as we only store remotely.
- Separately, there is another logic flaw in the
cpbackuptool that we’re escalating now (95861522).
As part of this outage / rebuild, we have decided to also overhaul our retention periods:
Currently, retention periods are:
- Full Backups: 2x Weekly
- Full Backups: 2x Monthly
- Incrementals: 14x Weekdays
Hereon, retention periods are:
- Incrementals: 4x Monthly (on the 1st)
- Incrementals: 20x Weekdays (ie. 4 weeks)
Many thanks for your continued patience.
This is on-track for syncs resuming tonight. More detailed info will be available then.
We’re taking this opportunity to overhaul how storage datasets are nested and configured.
For the last 36 hours, we’ve been waiting for a permissions command to cleanly exit out.
Once complete, we will then finalise the new configs, and then deploy to machines.
Due to the overhauled system, it may take us some days to finish config testing.
It’s best to approach this in-order, hence why we’re having to wait right now.
Thank you for your patience. Work is continuing on this problem.
We’re aiming to have a proper update delivered ahead of Monday.
G’day,
Following changes to internal DR replication to our off-site storage location, VPS backup jobs are failing due to problems verifying the destinations. We’ve reverted some TrueNAS changes, however need to re-activate the JBv5 destinations per-server and then successfully perform connectivity tests on each machine to verify as fixed.
Data can be accessed manually at this time should you have any urgent restoration needs. Thank you.
We’ll be working through this today and tomorrow, with many thanks for your patience.
Cheers,
Merlot Digital
The Network Crew Pty Ltd (TNC)
Network: AS138521