Max Memory Consumption and Memory Usage Over Time

The method of acquiring memory usage data varies between execution environments and is useful in the optimization and debugging of jobs. Below, we have outlined a few methods that will provide this data.

Docker Executor Max Memory Usage

Docker executor users can output the max memory consumed by a job by adding the following as the last step in the job:

- run:
    command: cat /sys/fs/cgroup/memory/memory.max_usage_in_bytes
    when: always

Note: This will be accurate if the job has one Docker image. Multiple Docker image builds will only report the memory usage of the primary image.

Memory usage will be reported in bytes. You can convert to GiB using your favorite search engine to check it against the amount of RAM available to your job, according to its assigned resource class.

This can help troubleshoot out-of-memory (OOM) errors. 

Docker/Machine Executors Memory Usage Over Time

To log memory usage over time for both Docker and machine executors, you can also add this step as the first step in your job:

- run:
    command: |
      while true; do
        sleep 5
        # NOTE: on MacOS, the f argument is not supported.
        # In this case, you can drop the f argument instead.
        ps auxwwf
        echo "======"
      done
    background: true

Alternatively, we can also take advantage of the top command (available on Docker, Machine (Linux) or MacOS executors). This can help show both memory and CPU utilization by individual processes.

- run:
    name: Profile CPU and memory every 5s (background)
    command: |
      while true; do
        sleep 5
        printf "\n\n$(date)\n"
        top -b -c -n 1
        echo "======"
      done
    background: true

Note that you have may have to install these tools, or use an alternative command if these tools are not available in the specific Executor.

Remote Docker

To get the memory usage of the Remote Docker environment, you can pass the ps command through SSH with

ssh remote-docker ps auxwwf

Additional Resources:

• How To Determine if Your Docker Executor Build Is Under-Provisioned
• How to Adjust Your Account Plan