Why does the Docker Executor return total CPU Cores?
When calling nproc, lscpu etc on the Docker Executor, you will see it return the total host CPU core count, instead of what is depicted on your resource_class. See below:
This is by design to allow for slight overprovisioning where you may see your CPU reach 100% usage on the resource tab, but not fail.
Typical Use Case
The typical use case for this is typically to utilize a --parallelism flag to auto-scale workers to available CPU cores. However, it is best practice to allow the command to auto-scale or specify the core count based on the resource class you are using.
You can also utilize the Machine Executor which will return the specific CPU core count that your resource class specifies.
Comments
Article is closed for comments.