How To Retry an Intermittently Failing Step

Overview

Networking issues can stop your job from succeeding.
You may want to mitigate this issue by implementing a for-loop to retry the failing step until it succeeds with a non-zero exit code or exhausts all retries.

For-Loop Example and Explanation

An example command can be found below:

- run: |
N=5
while [ $N -gt 0 ]
do
if $(docker-compose up -d); then
echo "Spun up services via Docker Compose"
exit 0
fi

echo "Failed to spin up services. Retrying..."
N=$(( $N - 1 ))
sleep 10
done
exit 1

In this example, the main command is docker-compose up -d.

We are looping to retry for up to 5 more times if the command fails (exit non-zero).
Here, we assume it is safe to retry when we receive a non-zero exit.
If the command in question has any side-effects, it may not be best to retry them.

You can replace the sleep 10 with any wait time preferred.

 

IMPORTANT:
Please be aware that this can increase the overall duration of the step.
The credits consumed for this job may be increased as such.
Do consider this impact when implementing this approach.

 

Additional Resources:

Was this article helpful?
4 out of 4 found this helpful

Comments

0 comments

Article is closed for comments.