Resolving Excessive VM Instances in CircleCI Server with Machine Provisioner

Overview

After upgrading CircleCI Server from version 4.5.8 to 4.6.6, an issue was observed where more Windows VM instances than expected were being launched. This problem was not limited to Windows VMs but also affected Linux VMs, indicating a broader issue with the machine provisioner settings. The root cause was identified as the pre-scaling settings in the machine provisioner, which were causing more instances to be launched than necessary.

Prerequisites

• Access to CircleCI Server configuration files.
• Familiarity with Helm and YAML configuration.
• Access to the CircleCI Server's values.yaml file.

Solution

To address the issue of excessive VM instances being launched, the following steps were taken:

1. Adjust Machine Provisioner Settings: Modify the values.yaml file to adjust the machine provisioner settings. Set the following parameters to zero to disable pre-scaling:

   machine_provisioner:
     fudgeScaleFactor: 0
     demandFudgeFactor: 0
     fudgeConstantTerm: 0

2. Verify Instance Launch: After applying the changes, run a reality check to ensure that only the expected number of EC2 instances are launched. The expected outcome should match the configuration specified in your config.yml.
3. Monitor and Adjust: If users report long queue times for builds, consider re-enabling pre-scaling with adjusted values to balance between resource availability and cost.

By following these steps, you can ensure that your CircleCI Server launches the correct number of VM instances, optimizing resource usage and reducing unnecessary costs.

Additional Resources

• CircleCI Server Installation Reference
• CircleCI Reality Check Repository