Overview

There is a hard constraint on the overall file size of a CircleCI config file. 
Over time, as your team sets up more jobs in the project's pipeline, the config file may be at risk of exceeding the file size limits and receiving an error such as "Your compiled config is too large to be processed. Please contact support."

This article aims to share some suggestions on how you can reduce your config file, depending on your use case.

Tip 1: Pack Commands into a Script

Instead of writing a series of commands within a run step, pack these commands into a script.

This way, your run step would be a one-liner that executes the script.
This also means your team can easily execute this same script in other environments, such as during local development.

If you prefer being able to see the exact commands for debugging, you can always make use of set -x within the step.
> Note that using `set -x` may reveal secrets in the build output. Do exercise caution as needed.

As an example:

- run:
    name: Run tests
    command: |
      # set -x will enable debugging; 
      # prints out commands run in the ./run-tests.sh script
      set -x
      sh ./run-tests.sh

Tip 2: Pre-Install Dependencies in a Custom Docker Image

Create a custom Docker image with all required dependencies, for your Docker executor job.
Note that this applies to jobs using the Docker executor.

This way, you do not need to install additional dependencies as part of your build.
An additional benefit is that the overall job duration will also improve.
We also avoid potential network issues from external package registries that may arise when we are installing the dependencies within a CircleCI build.

Take advantage of our convenience images, which come with various tools already pre-installed.
You can, for example, build a custom image based on one of our convenience images.

docker:
  - cimg/python:3.9
steps:
  - run: |
      # NOTE: the following dependencies are already available in cimg/* images.
      which jq 
      which yq
      which psql
  - run: |
      # SUGGESTION: Build a custom image FROM cimg/python:3.9
      # and install the additional tools required instead.
      # After publishing the image, 
      # make sure you update your job's executor to use this custom image then.
      sudo apt-get update
      sudo apt-get install openjdk-13-jdk openjdk-13-jre mysql-client

Tip 3: Trim Imported Orb's Commands and Jobs

Instead of importing Orb commands or jobs as-is, trim out unnecessary steps.

CircleCI's official Orbs try to support all execution environments (e.g., Linux, MacOS, Windows).
The Orb commands or jobs can include steps that may not be relevant for for your job.
For instance, the aws-cli/install command is long, because it covers installation of the AWS CLI for MacOS, Linux, and Windows.

Instead, we can extract these commands, and trim out accordingly.
For example, you can obtain the source code of the aws-cli/install command via the CircleCI CLI and yq:

$ circleci orb source circleci/aws-cli \
| yq -e '.commands.install' -

With this, you can then:

1. Copy this onto your config.
2. Declare it as a command (or job, if you are trying to modify a job).
3. Trim out the unnecessary steps, as needed.
4. Replace the section using the original Orb's command with your custom command in (2).

Alternatively, you can choose to fork our Orbs to modify and publish as your own.
Our Orbs are open-sourced, and you can find them on GitHub here.

Tip 4: Split Config Using Dynamic Configuration

Use Dynamic Configuration to split the pipeline from a .circleci/config.yml file into various config files.

You can leverage Dynamic Configuration to break down various workflows based on conditions.

For example, our Path-filtering Orb can help teams to simplify the overall pipeline to run workflows only based on files modified.
You can then take advantage of Dynamic Configuration to generate a smaller config file, on the fly, for the pipeline's continuation.

Here is an example from one of our colleagues, for the Rails project:
https://github.com/zzak/rails/blob/70dba04faf11eb28ca6761049f8654837dbcbec2/.circleci/config.yml

Tip 5: Break the Project into Multiple Projects

Consider splitting the project into multiple projects instead.

This is applicable or recommended only if your project has the following characteristics:

• is a mono-repo
• contains loosely coupled components that can stand as a project on their own

If so, why not split the mono-repo into separate projects then?
This way, each project can have its own CircleCI config.
Of course, this decision will depend on your team's philosophy as well.

Additional Resources:

• Maximum Size of a CircleCI config file