Dictating a Pass or Fail

Steps will pass or fail based on the error code that is raised from the processes launched within the step and how the shell has been configured to handle errors. You can see some of the linux shell configuration that we apply to each shell that runs a step in the job step output.

We have applied the settings -eo pipefail to the bash shell. The command -e tells bash to exit and raise a failure if it encounters any errors. If you are piping commands together bash will typically only raise an error based on the final command of the pipeline. The -o pipefail tells bash to raise an error if it encounters one at any point within a pipeline sequence.

Overriding Default Settings

Calling commands from within scripts can launch a new shell, which overrides the settings we apply. Or programs can report their error codes incorrectly, and produce successful results when they should not. This can commonly happen when you're running your tests within a multi-line run step, as each step executers in a separate shell. You can apply these configurations more explicitly by adding them to the top of your scripts or to the config.yml directly. This can improve the reliability of what is being reported as a failure. 

Example:

- run:
    name: run tests
    command: |
      source venv/bin/activate
      python manage.py test

Solution

To work around this add the command set -e at the top of the multiline command. This will ensure that errors are not suppressed and are instead passed up to fail the build.

- run:
 name: run tests
 command: |
   set -e
   source venv/bin/activate
   python manage.py test

Resources

Documentation for set

Guide on Bash Strict Mode for debugging