How to troubleshoot Test Splitting

Troubleshooting Parallelism

There are a number of reasons why test splitting across parallel containers may be behaving unexpectedly. Our documentation on running tests in parallel has a wealth of information to help you, too.

Save Artifacts

The store_test_results step will ensure test timings are saved, but it doesn't allow for easy debugging. You may also want to upload your tests via store_artifacts to visually verify the number of tests being run.

Echo Test Results

Similar to saving artifacts, here is an example of how you can implement a clean test split command and also echo out the test data.

TESTFILES=$(circleci tests glob "test/**_test.rb" | circleci tests split --split-by=timings)
echo ${TESTFILES}
# bundle exec rake knapsack_pro:minitest
bundle exec rspec --format progress \
--format RspecJunitFormatter \
-o test/reports/rspec.xml \
-- ${TESTFILES}


You can find a few examples on our community forum

Varying Parallelism

Different amounts of parallelism may have a great effect on your tests, depending on the way they are written. It may also help you identify particular tests that are causing problems repeatedly.

Timing data

Timing data will be saved upon a successful "green" build. When builds start, we automatically pick the latest job's test results in the same branch jobs. If there is no data in the same branch, we select the latest job's test results in the same project's jobs. You can see which job's test results we picked in the Downloading previous test results step.

You can also inspect your timing data, which can be found in the $CIRCLE_INTERNAL_TASK_DATA/circle-test-results/results.json file.

For example:

- run:
    name: Check out previous test metadata
    command: |
      cat "${CIRCLE_INTERNAL_TASK_DATA}/circle-test-results/results.json" | jq .

Variable Length Tests

You may have a test or suite of tests that will vary greatly in their completion time, either on purpose, or for a number of other reasons. This is common with UI or Unit Testing.
If your test timing varies with every test, the CircleCI splitting system will never be able to determine valid timing data.

Alternative Test Splitting Methods

Our documentation has more information about splitting tests. You can find it here