Passing Context or Project Environment Variables to Secondary Docker Containers

Overview

Environment variable interpolation doesn't work for a service(secondary) containers.

For instance, if you write:

docker:
  - image: cimg/base:current
  - image: my.registry/proxy:latest
    environment:
      MY_SECRET_FOO: $MY_SECRET_FOO

The secondary container receives the literal string $MY_SECRET_FOO, not the value from your project settings or context.

Root Cause

This is a limitation of the CircleCI. 

Values in the docker section of your config are processed before job steps run, and CircleCI does not expand project or context variables in secondary container definitions.

While hard-coded environment variable values are passed correctly to secondary or service containers, context and project environment variables are not interpolated for non-primary containers.

Solution

1. Use the machine executor (recommended)

Use the machine executor when you need a real secret in a sidecar container.

The machine executor provides a standalone VM with Docker preinstalled. Job steps run in that VM, where project and context environment variables are available in the shell. You can start sidecar containers with docker run or Docker Compose and pass secrets with -e flags or an env file.

docker:
jobs:
  test:
    machine:
      image: ubuntu-2204:current
    steps:
      - checkout
      - run:
          name: Start sidecar with secrets
          command: |
            docker run -d --name proxy \
              -e MY_SECRET_FOO="$MY_SECRET_FOO" \
              -p 8080:8080 \
              my.registry/proxy:latest
      - run:
          name: Use proxy
          command: curl -f http://127.0.0.1:8080/health

2. Use setup_remote_docker with docker run (Docker executor)

If you must stay on the Docker executor, you can enable Docker Layer Caching / remote Docker and start the sidecar in a run step instead of declaring it under docker: 

docker:
jobs:
  test:
    docker:
      - image: cimg/base:current
    steps:
      - checkout
      - setup_remote_docker
      - run:
          name: Start sidecar with secrets
          command: |
            docker run -d --name proxy \
              -e MY_SECRET_FOO="$MY_SECRET_FOO" \
              -p 8080:8080 \
              my.registry/proxy:latest
      - run:
          name: Use proxy
          command: curl -f http://127.0.0.1:8080/health

Important caveat: 

Containers started with setup_remote_docker run on a remote Docker daemon, not in the same local network as declared secondary containers. Because of this: 

• Port mapping to localhost often does not work as it would with a declared secondary container or on the machine executor. -
• Container-to-container networking can be unreliable or require extra configuration. 

If you use this approach and cannot reach the sidecar from subsequent steps, that networking limitation is the likely cause. In that case, switch to the machine executor