Replibit - Service Advisory – ZFS Offline on Most Devices

INCIDENT

August 6 2020

Early this morning, the Axcient team identified an issue with an operating system update that is causing local Replibit appliance and private cloud vault storage pools to appear offline. 

The storage pool appearing offline is causing the following issues/symptoms:

• Storage Pool degradation alerts
• New backup snapshot failures (new backups cannot occur)
• Local recovery is unavailable

Additionally, we know the following services remain available:

• Existing/previous backup snapshots on the local appliance are NOT impacted (but are currently inaccessible)
• Cloud vaults hosted in the Axcient cloud are running without impact and can be used for file recoveries or full Cloud DR

ACTION PLAN

If you have Replibit automatic updates turned on, we will be releasing a hotfix (v10.2.3) which will automatically fix this issue.  We expect this to be released prior to 11am US Mountain today (August 6th). This will automatically fix the issue, and nothing is needed on your part.

Until the issue is automatically resolved, the following resources are available to you:

• If you wish to fix the issue immediately, you can run the following command as root on each appliance/vault:
  apt-get update && apt-get purge zfs-doc -y && apt-get install zfsutils zfs-dkms -y && /sbin/initctl start asyncreboot
• Monitor the current incident - https://status.axcient.com/
• Contact our support team or your Axcient Partner Success Manager through the x360 portal – https://partner.axcient.com/login

We deeply apologize for this issue and pain it has caused. We are working to get all affected systems automatically back online as soon as possible. Please do not hesitate to reach out to us.

Manually Fix This Issue

Repairs on your affected Replibit devices can be performed with a simple command-line executed at the console or via ssh. 

1. Connect to your affected device(s) using your favorite ssh utility or access the Terminal using your Management Portal from the Managed Systems page:

2. Login as the ‘root’ user then copy and paste this command:

apt-get update && apt-get purge zfs-doc -y && apt-get install zfsutils zfs-dkms -y && /sbin/initctl start asyncreboot

3. The device will download the updated Ubuntu packages and install them, then rebuild the missing kernel driver modules required for ZFS.  This process may take 5-10 minutes on some devices.  The Replibit device will reboot automatically once the missing drivers have been repaired, and the storage pool will be online again.