How do I run commands in the remote Docker environment and understanding how this mirrors Docker in Docker?

Introduction

Docker in Docker (DiD) allows you to run Docker commands within a Docker container. In CircleCI, you can't directly run Docker commands in a Docker executor due to security limitations. However, CircleCI's Remote Docker feature solves this by providing a separate, secure virtual machine where Docker commands can execute safely.

This article explains how to use Remote Docker, its current architecture, pricing, and best practices for optimization.

How Remote Docker Works

When you use the Docker executor in CircleCI, you can't run Docker commands directly. Remote Docker creates a separate VM environment where Docker commands execute, while your job runs in the primary container.

Basic Setup

Add setup_remote_docker to your job configuration:

jobs:
  build:
    docker:
      - image: cimg/base:2024.12
    steps:
      - setup_remote_docker:
          docker_layer_caching: true
      - run: docker build -t myapp:latest .

Current Architecture

CircleCI improved Remote Docker architecture for better performance and reliability:

• Single VM: Everything runs in one virtual machine (no more network transfers between containers and remote VMs)
• Better Performance: Faster execution and fewer network-related failures
• Flexible Resources: VM size matches your specified resource class
• UI Labels: Jobs using Remote Docker show "Remote Docker" labels in the CircleCI interface

Resource Classes and Pricing

How Resource Classes Map

Your Remote Docker VM size depends on your Docker executor resource class:

Billing

Since June 2023, billing is based on the VM resource class used (see table above). Check the current pricing page for specific credit costs.

Docker Version Options

Choose your Docker version with the version parameter:

- setup_remote_docker:
    version: default    # Docker 24 (recommended)
    docker_layer_caching: true

Available versions:

• default - Docker 24 (latest supported)
• docker23 - Docker 23 (previous version)
• edge - Latest development version

Running Non-Docker Commands (SSH Access)

Sometimes you need to run non-Docker commands in the remote environment. Use ssh remote-docker to access it.

Basic SSH Usage

- run: |
    ssh remote-docker echo "This runs in the remote Docker environment"

Important: Run setup_remote_docker first, or SSH will fail.

Running Multiple Commands

SSH only accepts one command at a time. For multiple commands, use one of these approaches:

Option 1: Here Document

- run: |
    ssh remote-docker << 'EOF'
    echo "Setting up environment"
    mkdir -p /tmp/workspace
    echo "Setup complete"
    EOF

Option 2: Pipe to Shell

- run: |
    echo 'command1; command2; command3' | ssh remote-docker bash

Optimization Tips

Performance

• Enable Docker Layer Caching: Set docker_layer_caching: true to reuse unchanged layers
• Use Latest Docker: Stick with version: default for best performance
• Right-size Resources: Choose the smallest resource class that meets your needs

Cost Management

• Monitor usage in the CircleCI dashboard
• Consider self-hosted runners for high-volume Docker operations
• Use appropriate resource classes - don't over-provision

Complete Example

version: 2.1

jobs:
  docker-example:
    docker:
      - image: cimg/base:2024.12
    resource_class: medium
    steps:
      - checkout
      - setup_remote_docker:
          docker_layer_caching: true
          version: default
      
      - run:
          name: Build image
          command: docker build -t myapp:$CIRCLE_BUILD_NUM .
      
      - run:
          name: Complex setup via SSH
          command: |
            ssh remote-docker << 'EOF'
            docker system prune -f
            mkdir -p /tmp/build-cache
            EOF
      
      - run:
          name: Push image
          command: |
            echo $DOCKER_PASSWORD | docker login -u $DOCKER_USERNAME --password-stdin
            docker push myapp:$CIRCLE_BUILD_NUM

workflows:
  build:
    jobs:
      - docker-example

Frequently Asked Questions

Q: Can I still use the Small resource class? A: Yes, for regular Docker executor jobs. But when you add setup_remote_docker, it automatically uses a Medium VM to ensure adequate resources for Docker operations.

Q: How is this different from Machine Executor? A: Machine Executor runs your entire job on a VM with Docker pre-installed. Remote Docker runs your job in a container but provides VM access for Docker commands. Use Remote Docker when you want containerized execution with Docker capabilities.

Q: Does this work with Arm builds? A: Yes, Remote Docker supports both x86 and Arm architectures with equivalent resource class mapping.

Q: How do I optimize costs? A: Use the smallest resource class needed, enable Docker Layer Caching, monitor usage in the dashboard, and consider self-hosted runners for high-volume operations.

Q: Why can't I directly access services between my job and Remote Docker? A: They run in separate environments for security. Use Docker networking or SSH to communicate between them, or run services within the same Docker environment.

Additional Resources

CircleCI Documentation

• Run Docker Commands - Official guide to building Docker images
• Configuration Reference - Resource class configuration options
• Remote Docker Images Support Policy - Supported Docker versions and update policy

Pricing and Resource Information

• CircleCI Pricing - Current credit costs and pricing plans
• Resource Classes Product Page - Detailed specifications and pricing

Community Resources

• CircleCI Community Forum - Remote Docker discussions and troubleshooting
• Changes to Remote Docker Pricing Discussion - Community discussion about 2023 architecture changes

External Resources

• Stack Overflow: SSH Command Execution - Advanced SSH techniques for remote command execution