Symptoms

Case 1

After restarting Replicated with systemctl command, with service command, or with a restart of your service machine, Replicated containers disappeared and eventually CircleCI server crashed.

Case 2

After updating docker-ce on the service machine by apt/apt-get command, Replicated containers disappeared and eventually CircleCI server crashed.

Cause

It happens because 1) Replicated services registered to systemd (i.e. replicated.service, replicated-operator.service, and replicated-ui.service) deletes Docker containers for its own services upon service start-up, and because 2) Docker image names used by Replicated services are not guaranteed to be available anytime.

During their start-up sequence, which will be initiated explicitly by systemctl/service command, or implicitly by system restart or update of docker-ce, Replicated services do the following tasks:

1. Clean up existing Replicated containers.
2. Start a new container with docker run command using corresponding Docker images tagged as :current (such as quay.io/replicated/replicated:current).

However, the :current images are locally tagged Docker images, deriving from other Docker images, and the tag is not used for image pulls. Therefore, if the :current tags are untagged somehow, Replicated services fail to start, as they cannot find Docker images to use.

Note: The situation that the :current tags are untagged can be reproduced quite easily by cleaning up Docker image caches with e.g. docker image prune -a command. We confirmed that docker image prune -a untags :current images for Replicated regardless of whether Replicated is running or not. In other words, if you have ever executed the docker image prune -a command or any equivalent command on your service machine, you are likely to encounter the same symptom.

Solution

You can fix this issue by reinstalling Replicated with the snippet below.

export REPLICATED_VERSION="2.38.6"
export PRIVATE_IP="$(curl http://169.254.169.254/latest/meta-data/local-ipv4)"
sudo curl -sSk -o /tmp/get_replicated.sh "https://get.replicated.com/docker?replicated_tag=$REPLICATED_VERSION&replicated_ui_tag=$REPLICATED_VERSION&replicated_operator_tag=$REPLICATED_VERSION"
sudo bash /tmp/get_replicated.sh local-address="$PRIVATE_IP" no-proxy no-docker