1. CircleCI Support Center
  2. Execution environments
  3. Self-Hosted Runner

Troubleshooting missing execution logs

Avatar
Ben

Overview

You're running workflows with self-hosted runners, but when you check the jobs, there’s no execution logs showing up. This guide breaks down what’s causing the issue and how you can fix it.
 

Why am I unable to see the execution logs?

Version 3.0.7-3109-1cf50bb is over 2 years, making it a very old version of container runner. This issue was raised by some of our customers and it was investigated. At a very high level the problem was identified as a change in the steps service; this problem was fixed in newer versions of container runner. For more information on our previous releases of runner, please see the ChangeLog page which can be found here.
 

Pre-requisites

 

Pre-requisites: Before getting started

Go to ’Self-Hosed Runners’ from the left hand side bar and the following information should be displayed:
 
  • In the ‘Runners’ column it should show container-agent
  • In the ‘Version’ column it should show 3.0.7-3109-1cf50bb

 

Pre-requisites: The upgrade itself

For the upgrade process you will need the following:
 
  • Helm
    • Helm is a package manager for Kubernetes and it's used to upgrade your CircleCI container runner version. It can be installed from here if you don't have it.
  • Access to the Kubernetes cluster
    • You'll need access to this to upgrade make changes to the Helm chart.
  • Access to the CircleCI Container Runner Helm Chart
    • This Helm chart defines the Kubernetes resources (pods, services, etc.) for the self-hosted runner. You use this chart to upgrade the runner.
  • Access to values.yml
    • This is where the changes get made and then applied used Helm

 

Pre-requisites: Post upgrade checks

After the upgrade process there are some checks that can be made. To do these checks you will need the following:
 

 

 
 

How do I fix this issue?

As mentioned above, a newer version of container runner will fix this issue. Here are the steps that can be taken to upgrade container runner to a more recent version. The version being used for this example will be 3.1.2 which was released in March 2025.
 

Upgrading Your Self-Hosted runner

Here are the steps needed to upgrade a specific self-hosted runner:

 

1.  Update the values.yml file 

The values.yml will need to be updated to include the new version of the container runner. The new version can be inserted into the tag property, which can be found under the image: property.  A very simple example can be seen below:

runner:
  name: selfHosted1                       # Found in under the Resource Class column in the UI after the '/'
  resourceClass: CCIsupport/selfHosted1   # The full value from the Resource Class column
  api:
    authToken: YOUR_RUNNER_TOKEN_HERE     # IMPORTANT: If you don't have this, you will need to create a new runner
  image:
    tag: "3.1.2"                          # Update to the version you want

 

2.   Run the Helm upgrade command:

Once values.yml has been updated, the runner can be upgraded. The is done via terminal using this command:
helm upgrade selfHosted1 \
  circleci/container-runner \'/'
  -f values.yml \
    --namespace <your-namespace>
 

NOTES:

  •  In the example above selfHosted1 is the name of the runner. This will be value the from the name property from the runner that was updated in step 1. If in doubt, it can be found by running the helm list command.

  • The <your-namespace> is the namespace where your runner is deployed in Kubernetes (this might bedefault if you're using the default namespace).

Post Upgrade Checks

At this point your runner should be upgraded, however there are a few things that can be done to double-check this:
 

 1. Run the kubectl get pods -n <your-namespace> command in the terminal

This command should return all of the pods in the namespace entered above. When the container runner is upgraded, you should see the old pods terminating and the new ones starting up. The command should return something like this:
NAME                             READY STATUS      RESTARTS AGE
circleci-runner-7d8f75c5bb-xyz12 1/1   Running     0        1m
circleci-runner-7d8f75c5bb-jkld9 1/1   Terminating 0        2h
 

 2. Check the pods from the command above

 Each pod can be inspected using the command kubectl describe pod <pod-name> -n <your-namespace>. Here is an example:
Name:         circleci-runner-7d8f75c5bb-xyz12
Namespace:    default
Priority:     0
Node:         node-02/192.168.1.2
Start Time:   Tue, 11 Apr 2025 10:40:00 UTC
Labels:       app=circleci-runner
              pod-template-hash=7d8f75c5bb
              runner=selfHosted1
              version=3.1.2
Annotations:  kubernetes.io/limit-ranger=limitranger
Status:       Running
IP:           10.244.1.3
Containers:
  circleci-runner:
    Container ID:   docker://abcd1234xyz
    Image:          circleci/container-runner:3.1.2
    Image ID:       docker-pullable://circleci/container-runner@sha256:123456789abc...
    Ports:          8080/TCP, 5000/TCP
    State:          Running
      Started:      Tue, 11 Apr 2025 10:40:02 UTC
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:      500m
      memory:   2Gi
    Requests:
      cpu:      250m
      memory:   1Gi
    Environment:
      RUNNER_TOKEN:  <set to the key 'RUNNER_TOKEN' in secret>
      ...
    Mounts:
      /mnt/data from runner-storage (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-xyz (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  runner-storage:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
  default-token-xyz:
    Type:       Secret (a volume populated by a Secret)
    SecretName: default-token-xyz
    Optional:   false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     <none>
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  1m    default-scheduler  Successfully assigned default/circleci-runner-7d8f75c5bb-xyz12 to node-02
  Normal  Pulling    1m    kubelet            Pulling image "circleci/container-runner:3.1.2"
  Normal  Pulled     1m    kubelet            Successfully pulled image "circleci/container-runner:3.1.2"
  Normal  Created    1m    kubelet            Created container circleci-runner
  Normal  Started    1m    kubelet            Started container circleci-runner
 There is a lot of information here, however the important sections are the containers, status,  and events sections. 
 
In the containers section, there is one container called circleci-runner. That container contains an image property which has circleci/container-runner:3.1.2 as its value, meaning it's running the version of container runner that you specified earlier in this document. 
 
In the status section, it should say running, which means that the pod has started up properly. Anything else means there is a problem with the upgrade.
 
In the events section there should be a breakdown what's happened on that pod. In the example we can see that it's created the container, downloaded the container runner image and applied it to the container. This area is a good place to look if there are errors with the container as it will usually highlight what that error is.
Was this article helpful?
0 out of 0 found this helpful

Comments

0 comments

Article is closed for comments.

Articles in this section