How to dynamically set job parallelism

There may be cases where you want to dynamically set the parallelism value for a specific job. This can be implemented by using a pipeline parameter, and then passing in a value for that parameter either via the API or a setup workflow.

One example of a use-case for this would be when you want to change the parallelism based on the number of tests defined in your code.

Sample config.yml file

Pipeline parameters can be defined under the parameters section in your config.yml file. Here is a sample:

version: 2.1

parameters:
  parallelism:
    type: integer
    default: 1

jobs:
  build:
    parallelism: << pipeline.parameters.parallelism >>
    docker:
      - image: cimg/base:edge
    steps:
      - checkout
      - run: echo "The parallelism for this job is << pipeline.parameters.parallelism >>"

workflows:
  workflow:
    jobs:
      - build

Triggering a job via the API

Pipeline parameter values can be passed in with an API call. Here is a sample call using cURL:

curl --request POST \
  --url https://circleci.com/api/v2/project/gh/sampleorg/sampleproject/pipeline \
  --header "Circle-Token: $CIRCLECI_TOKEN" \
  --header 'content-type: application/json' \
  --data '{"parameters":{"parallelism":2}}'

where CIRCLECI_TOKEN is a personal api token.

Triggering a job via a setup workflow

Pipeline parameters can also be passed in via a setup workflow. Here is a sample config.yml file showing how to do so:

version: 2.1

setup: true

orbs:
  continuation: circleci/continuation@0.2.0

jobs:
  determine-parallelism:
    docker:
      - image: cimg/base:edge
    resource_class: medium
    steps:
      - checkout
      - run:
          name: Determine parallelism
          command: |
            # Put command/logic here to determine parallelism
            PARALLELISM=2
            echo "{\"parallelism\": $PARALLELISM}" > pipeline-parameters.json
      - continuation/continue:
          configuration_path: .circleci/continue_config.yml
          parameters: pipeline-parameters.json

workflows:
  build-setup:
    jobs:
      - determine-parallelism

In the above config file, it is setting the parallelism value in a json file, which is then being passed to the continuation api via the continuation orb's continue command. You can then use the sample config.yml file from the first section of this article as your continuation config (named continue_config.yml in this example).

Additional Resources: 

• Test splitting and parallelism
• Dynamic Configuration
• circleci/continuation Orb