There are a number of reasons why test splitting across parallel containers may be behaving unexpectedly. Our documentation on running tests in parallel has a wealth of information to help you, too.
store_test_results step will ensure test timings are saved, but it doesn't allow for easy debugging. You may also want to upload your tests via
store_artifacts to visually verify the number of tests being run.
Echo Test Results
Similar to saving artifacts, here is an example of how you can implement a clean test split command and also echo out the test data.
TESTFILES=$(circleci tests glob "test/**_test.rb" | circleci tests split --split-by=timings)
# bundle exec rake knapsack_pro:minitest
bundle exec rspec --format progress \
--format RspecJunitFormatter \
-o test/reports/rspec.xml \
You can find a few examples on our community forum
Vary Your Parallelism
Different amounts of parallelism may have a great effect on your tests, depending on the way they are written. It may also help you identify particular tests that are causing problems repeatedly.
Timing data will be saved upon a successful "green" build. When builds start, we automatically pick the latest job's test results in the same branch jobs. If there is no data in the same branch, we select the latest job's test results in the same project's jobs. You can see which job's test results we picked in the Spin up environment step.
You can also view your timing data in $CIRCLE_INTERNAL_TASK_DATA/circle-test-results if you need to.
Variable Length Tests
You may have a test or suite of tests that will vary greatly in their completion time, either on purpose, or for a number of other reasons. This is common with UI or Unit Testing.
If your test timing varies with every test, the CircleCI splitting system will never be able to determine valid timing data.
Consider Other Ways to Split Tests
Our documentation has more information about splitting tests. You can find it here