After restarting Replicated with
systemctl command, with
service command, or with a restart of your service machine, Replicated containers disappeared and eventually CircleCI server crashed.
docker-ce on the service machine by
apt-get command, Replicated containers disappeared and eventually CircleCI server crashed.
It happens because 1) Replicated services registered to
replicated-ui.service) deletes Docker containers for its own services upon service start-up, and because 2) Docker image names used by Replicated services are not guaranteed to be available anytime.
During their start-up sequence, which will be initiated explicitly by
service command, or implicitly by system restart or update of
docker-ce, Replicated services do the following tasks:
- Clean up existing Replicated containers.
- Start a new container with
docker runcommand using corresponding Docker images tagged as
:current images are locally tagged Docker images, deriving from other Docker images, and the tag is not used for image pulls. Therefore, if the
:current tags are untagged somehow, Replicated services fail to start, as they cannot find Docker images to use.
Note: The situation that the
:current tags are untagged can be reproduced quite easily by cleaning up Docker image caches with e.g.
docker image prune -a command. We confirmed that
docker image prune -a untags
:current images for Replicated regardless of whether Replicated is running or not. In other words, if you have ever executed the
docker image prune -a command or any equivalent command on your service machine, you are likely to encounter the same symptom.
You can fix this issue by reinstalling Replicated with the snippet below.
export PRIVATE_IP="$(curl http://169.254.169.254/latest/meta-data/local-ipv4)"
sudo curl -sSk -o /tmp/get_replicated.sh "https://get.replicated.com/docker?replicated_tag=$REPLICATED_VERSION&replicated_ui_tag=$REPLICATED_VERSION&replicated_operator_tag=$REPLICATED_VERSION"
sudo bash /tmp/get_replicated.sh local-address="$PRIVATE_IP" no-proxy no-docker