How to record a job's memory usage

Machine and Docker users can output the max memory consumed by a job by adding the following step to their config:

- run:
    command: cat /sys/fs/cgroup/memory/memory.max_usage_in_bytes
    when: always

This step should be added as the last step in your job, to identify the max usage after all of the steps have completed.

Memory usage will be reported in bytes. You can convert to GiB using your favorite search engine to check it against the amount of RAM available to your job, according to its assigned resource class.

This can help troubleshoot out-of-memory (OOM) errors. 

To log memory usage over time, you can also add this step as the first step in your job:

- run:
    command: |
      while true; do
        sleep 5
        ps auxwwf
        echo "======"
      done
    background: true