Overview

While CircleCI does recommend Velero for backing up your server, the default deployment and Restic Daemon can fail, especially on larger databases. The error looks generic, but can look like the following:

error="pod volume backup failed: error running restic backup, stderr=: signal: killed"

These are some common fixes that can allow your backups to proceed.

Confirm Versions are the Same

To confirm your versions are the same on your CircleCI Server and Local CLI, you can simply run velero version which will output like the following:

$ velero version
    Client:
      Version: v1.9.1
      Git commit: -
    Server:
      Version: v1.9.1

If your Velero versions are not matching, you may need to either change the version of your CLI or your Server version.

Change CLI

To change the CLI version, you will need to download the tar file from the release you want. CircleCI recommends v1.9.1 by following the instructions on Velero's website.

Change Server

To change the server version, and you are upgrading it, you can simply change the image like so:

$ kubectl set image deployment/velero \
    velero=velero/velero:v1.9.1 \
    --namespace velero

$ kubectl set image daemonset/restic \
    restic=velero/velero:v1.9.1 \
    --namespace velero

If you need to downgrade, you will need to uninstall the Velero Server and reinstall after correcting your CLI version. Velero will match the installation to your CLI version.

Once uninstalled, you can follow our documentation to reinstall Velero.

CircleCI Server 4 Backup and Restore

Edit the CPU/RAM allocated to Velero and Restic

In some cases, simply making sure the CLI and Server versions match is not enough to ensure that the backup completes and you may need to edit the deployment and daemon to allocate more CPU/RAM.

Velero

$ kubectl edit deployment/velero -n velero 

 resources:
   limits:
     cpu: "1"
     memory: 512Mi
   requests:
     cpu: 500m
     memory: 128Mi

While the specific numbers may need to be played with for the size of the cluster you are backing up, it is best to 4x these numbers to start.

resources:
  limits:
    cpu: "4"
    memory: 2048Mi
  requests:
    cpu: 500m
    memory: 128Mi

Restic

$ kubectl edit daemonset/restic -n velero
 resources:
   limits:
     cpu: "1"
     memory: 1Gi
   requests:
     cpu: 500m
     memory: 512Mi

With the same recommendation as Velero, it is best to start with 4x and then go from there till a backup is able to complete.

resources:
  limits:
    cpu: "4"
    memory: 2048Mi
  requests:
    cpu: 500m
    memory: 128Mi

Additional Resources

[Server] Velero Backups Failing