distributor-internal (and sometimes distributor-external) crash-loops on startup. Jobs do not schedule. The process exits with:
failed to initialise list: list version not ready
This is easy to misdiagnose as object storage being down. The pod has already loaded release.txt successfully. Distributor always loads a second file at the same prefix: canary.txt.
Symptoms
kubectl get podsshowsdistributor-internalinCrashLoopBackOfforError.Distributor logs show a successful fetch of
circleci-agent/release.txt, then the init error above. The error string does not mention canary.Checking
release.txtfrom a debug pod can succeed and still leave distributor crash-looping.Seen on Server 4.x with a customer-owned or air-gapped agent bucket (
distributor.agent_base_url/ Helm object storage). Nomad vs container or machine runner 3 does not matter;canary.txtis required either way.
Cause
Distributor’s release list always requests two objects from the agent base URL:
{base}/release.txt— pinned agent version{base}/canary.txt— extra release type hardcoded ascanary
If canary.txt is missing, returns forbidden, or is not a canary version string, list init fails and the process exits. Agents plus release.txt in the bucket is not enough.
Copy the public canary.txt from CircleCI binary-releases (object circleci-agent/canary.txt). Steps: Copy canary.txt file.
The file body must match n.n.n-canary-xxxxxxxx (digits, then -canary-, then 8 hex characters). Example:
0.5.233529-canary-b3211b23
Confirm
From a pod that can reach the same bucket distributor uses, fetch both files at the same prefix as release.txt:
$AGENT_BASE_URL/release.txt
$AGENT_BASE_URL/canary.txt
release.txtshould return 200 and a normal agent version.canary.txtreturning 404, forbidden, or empty is the bug. 200 plus a line containing-canary-is healthy.
$AGENT_BASE_URL is the prefix already used for release.txt (often the bucket host plus /circleci-agent, or the root of circleci-data).
Fix
Get the file. Download public
canary.txt, or copy it from an environment that already has it.Upload it beside
release.txtat that same prefix (bucket root / agent prefix, not inside the version directory).Re-check both files (200 plus canary version string).
Restart. Delete the crash-looping pod, or wait for backoff. It should become
Running.
Expected object-storage layout:
circleci-data/
├── canary.txt
├── release.txt
├── $CIRCLE_AGENT_VERSION/
└── …
If release.txt itself is forbidden, that is a different problem (egress, bucket policy, wrong URL). Fix connectivity first, then still add canary.txt.
Related
The same error on
radm-externalfetching launch-agentrelease.txtis not this file. That path is runner-admin / launch agent, not distributor canary.