Configuring VPN Connection during a Build

In case you need to connect to a private network during your builds, or if you want to restrict access to your environment to a specific IP address, we suggest configuring a VPN connection as follows.

Prerequisites:

Note that you must use one of the following executors:

• machine

• macOS

• windows

machine [Linux] executor (Available machine images)

• OpenVPN (2.x)

• OpenVPN Connect (OpenVPN 3)

• L2TP

OpenVPN (2.x)

• Base64-encode the OpenVPN client configuration file, and store it as an environment variable.
• If the VPN client authentication is credentials-based (user-locked profile), you'll also need to add the username and password as environment variables (VPN_USER and VPN_PASSWORD).

If Using Ubuntu 20.04 Or Less:

version: 2.1
workflows:
  btd:
    jobs:
      - build
jobs:
  build:
    machine:
      image: ubuntu-2004:202201-02
    steps:
      - run:
          name: Install OpenVPN
          command: |
            sudo apt-get update
            sudo apt-get install openvpn openvpn-systemd-resolved

      - run:
          name: Check IP before VPN connection
          command: |
            ip a
            echo "Public IP before VPN connection is $(curl checkip.amazonaws.com)"

      - run:
          name: VPN Setup
          background: true
          command: |
            echo $VPN_CLIENT_CONFIG | base64 --decode > /tmp/config.ovpn

            if grep -q auth-user-pass /tmp/config.ovpn; then
              if [ -z "${VPN_USER:-}" ] || [ -z "${VPN_PASSWORD:-}" ]; then
                echo "Your VPN client is configured with a user-locked profile. Make sure to set the VPN_USER and VPN_PASSWORD environment variables"
                exit 1
              else
                printf "$VPN_USER\\n$VPN_PASSWORD" > /tmp/vpn.login
              fi
            fi

            vpn_command=(sudo openvpn
              --config /tmp/config.ovpn
              --route 169.254.0.0 255.255.0.0 net_gateway
              --script-security 2
              --up /etc/openvpn/update-systemd-resolved --up-restart
              --down /etc/openvpn/update-systemd-resolved --down-pre
              --dhcp-option DOMAIN-ROUTE .)

            if grep -q auth-user-pass /tmp/config.ovpn; then
              vpn_command+=(--auth-user-pass /tmp/vpn.login)
            fi

            ET_phone_home=$(ss -Hnto state established '( sport = :ssh )' | head -n1 | awk '{ split($4, a, ":"); print a[1] }')
            echo $ET_phone_home

            if [ -n "$ET_phone_home" ]; then
              vpn_command+=(--route $ET_phone_home 255.255.255.255 net_gateway)
            fi

            for IP in $(host runner.circleci.com | awk '{ print $4; }')
              do
                vpn_command+=(--route $IP 255.255.255.255 net_gateway)
                echo $IP
            done

            for SYS_RES_DNS in $(systemd-resolve --status | grep 'DNS Servers'|awk '{print $3}')
              do
                vpn_command+=(--route $SYS_RES_DNS 255.255.255.255 net_gateway)
                echo $SYS_RES_DNS
            done

            "${vpn_command[@]}" > /tmp/openvpn.log

      - run:
          name: Wait for the connection to be established and check IP
          command: |
            counter=1
            until [ -f /tmp/openvpn.log ] && [ "$(grep -c "Initialization Sequence Completed" /tmp/openvpn.log)" != 0 ] || [ "$counter" -ge 5 ]; do
              ((counter++))
              echo "Attempting to connect to VPN server..."
              sleep 1;
            done

            if [ ! -f /tmp/openvpn.log ] || (! grep -iq "Initialization Sequence Completed" /tmp/openvpn.log); then
              printf "\nUnable to establish connection within the allocated time ---> Giving up.\n"
            else
              printf "\nVPN connected\n"
              printf "\nPublic IP is now %s\n" "$(curl -s http://checkip.amazonaws.com)"
            fi

      - run:
          name: Run commands in our infrastructure
          command: |
            # A command
            # Another command

      - run:
          name: Disconnect from OpenVPN
          command: |
            sudo killall openvpn || true
          when: always

If Using Ubuntu 22.04 Or Greater:

version: 2.1
workflows:
  btd:
    jobs:
      - build
jobs:
  build:
    machine:
      image: ubuntu-2004:202201-02
    steps:
      - run:
          name: Install OpenVPN
          command: |
            sudo apt-get update
            sudo apt-get install openvpn openvpn-systemd-resolved

      - run:
          name: Check IP before VPN connection
          command: |
            ip a
            echo "Public IP before VPN connection is $(curl checkip.amazonaws.com)"

      - run:
          name: VPN Setup
          background: true
          command: |
            echo $VPN_CLIENT_CONFIG | base64 --decode > /tmp/config.ovpn

            if grep -q auth-user-pass /tmp/config.ovpn; then
              if [ -z "${VPN_USER:-}" ] || [ -z "${VPN_PASSWORD:-}" ]; then
                echo "Your VPN client is configured with a user-locked profile. Make sure to set the VPN_USER and VPN_PASSWORD environment variables"
                exit 1
              else
                printf "$VPN_USER\\n$VPN_PASSWORD" > /tmp/vpn.login
              fi
            fi

            vpn_command=(sudo openvpn
              --config /tmp/config.ovpn
              --route 169.254.0.0 255.255.0.0 net_gateway
              --script-security 2
              --up /etc/openvpn/update-systemd-resolved --up-restart
              --down /etc/openvpn/update-systemd-resolved --down-pre
              --dhcp-option DOMAIN-ROUTE .)

            if grep -q auth-user-pass /tmp/config.ovpn; then
              vpn_command+=(--auth-user-pass /tmp/vpn.login)
            fi

            ET_phone_home=$(ss -Hnto state established '( sport = :ssh )' | head -n1 | awk '{ split($4, a, ":"); print a[1] }')
            echo $ET_phone_home

            if [ -n "$ET_phone_home" ]; then
              vpn_command+=(--route $ET_phone_home 255.255.255.255 net_gateway)
            fi

            for IP in $(host runner.circleci.com | awk '{ print $4; }')
              do
                vpn_command+=(--route $IP 255.255.255.255 net_gateway)
                echo $IP
            done

            for SYS_RES_DNS in $(resolvectl --status | grep 'DNS Servers'|awk '{print $3}')
              do
                vpn_command+=(--route $SYS_RES_DNS 255.255.255.255 net_gateway)
                echo $SYS_RES_DNS
            done

            "${vpn_command[@]}" > /tmp/openvpn.log

      - run:
          name: Wait for the connection to be established and check IP
          command: |
            counter=1
            until [ -f /tmp/openvpn.log ] && [ "$(grep -c "Initialization Sequence Completed" /tmp/openvpn.log)" != 0 ] || [ "$counter" -ge 5 ]; do
              ((counter++))
              echo "Attempting to connect to VPN server..."
              sleep 1;
            done

            if [ ! -f /tmp/openvpn.log ] || (! grep -iq "Initialization Sequence Completed" /tmp/openvpn.log); then
              printf "\nUnable to establish connection within the allocated time ---> Giving up.\n"
            else
              printf "\nVPN connected\n"
              printf "\nPublic IP is now %s\n" "$(curl -s http://checkip.amazonaws.com)"
            fi

      - run:
          name: Run commands in our infrastructure
          command: |
            # A command
            # Another command

      - run:
          name: Disconnect from OpenVPN
          command: |
            sudo killall openvpn || true
          when: always

OpenVPN Connect (OpenVPN 3)

• Base64-encode the OpenVPN client configuration file, and store it as an environment variable.
• OpenVPN 3 Linux does not support storing user credentials in a text-based file to use when starting a VPN connection. Please refer to this documentation (OpenVPN 3 Linux and --auth-user-pass) to set up a workaround.

version: 2.1
workflows:
  btd:
    jobs:
      - build
jobs:
  build:
    machine:
      image: ubuntu-2004:202201-02
    steps:
      - run:
          name: Install OpenVPN
          command: |
            sudo apt update && sudo apt install apt-transport-https
            sudo wget https://swupdate.openvpn.net/repos/openvpn-repo-pkg-key.pub
            sudo apt-key add openvpn-repo-pkg-key.pub
            
            sudo wget -O /etc/apt/sources.list.d/openvpn3.list https://swupdate.openvpn.net/community/openvpn3/repos/openvpn3-$(sed 's/UBUNTU_CODENAME=//;t;d' /etc/os-release).list
            sudo apt update && sudo apt install openvpn3 openvpn-systemd-resolved

      - run:
          name: Check IP before VPN connection
          command: |
            ip a
            echo "Public IP before VPN connection is $(curl checkip.amazonaws.com)"

      - run:
          name: VPN Setup
          background: true
          command: |
            echo $VPN_CLIENT_CONFIG | base64 --decode > /tmp/config.ovpn

            ET_phone_home=$(ss -Hnto state established '( sport = :ssh )' | head -n1 | awk '{ split($4, a, ":"); print a[1] }')
            ## In case you're using an image with Ubuntu < 20.04, replace the above line with:
            ## ET_phone_home=$(ss -an | grep 'ESTAB .*:22' | head -n1 | awk '{ split($6, a, ":"); print a[1] }')
            echo $ET_phone_home
            
            if [ -n "$ET_phone_home" ]; then
              echo "route $ET_phone_home 255.255.255.255 net_gateway" >> /tmp/config.ovpn
            fi
            
            echo "route 169.254.0.0 255.255.0.0 net_gateway" >> /tmp/config.ovpn
            
            for SYS_RES_DNS in $(systemd-resolve --status | grep 'DNS Servers'|awk '{print $3}')
              do
                echo "route $SYS_RES_DNS 255.255.0.0 net_gateway" >> /tmp/config.ovpn
                echo $SYS_RES_DNS
            done

            for IP in $(host runner.circleci.com | awk '{ print $4; }')
              do 
                echo "route $IP 255.255.255.255 net_gateway" >> /tmp/config.ovpn
                echo $IP
            done

            # This will start the connection
            sudo openvpn3 session-start --config /tmp/config.ovpn > /tmp/openvpn.log

      - run:
          name: Wait for the connection to be established and check
          command: |
            counter=1
            until sudo openvpn3 sessions-list|grep "Client connected" || [ "$counter" -ge 5 ]; do
              ((counter++))
              echo "Attempting to connect to VPN server..."
              sleep 1;
            done
            
            if ( ! sudo openvpn3 sessions-list|grep "Client connected"); then
              printf "\nUnable to establish connection within the allocated time ---> Giving up.\n"
            else
              printf "\nVPN connected\n"
              printf "\nPublic IP is now %s\n" "$(curl -s https://checkip.amazonaws.com)"
            fi

      - run:
          name: Run commands in our infrastructure
          command: |
            # A command
            # Another command

      - run:
          name: Disconnect from OpenVPN
          command: |
            SESSION_PATH=$(sudo openvpn3 sessions-list | grep Path | awk -F': ' '{print $2}')
            echo $SESSION_PATH
            sudo openvpn3 session-manage --session-path $SESSION_PATH --disconnect
          when: always

L2TP

To set up an L2TP VPN connection, we recommend referring to this guide.

We suggest storing VPN_SERVER_IP, VPN_IPSEC_PSK, VPN_USER and VPN_PASSWORD as environment variables. Ideally, you might want to base64-encode VPN_IPSEC_PSK before storing it; you'll need to decode it during the build.

Also, we suggest storing the default gateway IP address in an environment variable:

• DEFAULT_GW_IP=$(ip route show default|awk '{print $3}')

macos executor (Supported Xcode versions)

• Base64-encode the OpenVPN client configuration file, and store it as an environment variable.
• If the VPN client authentication is credentials-based (user-locked profile), you'll also need to add the username and password as environment variables (VPN_USER and VPN_PASSWORD).

      - run:
          name: Install OpenVPN
          command: |
            brew install openvpn
            curl https://raw.githubusercontent.com/andrewgdotcom/openvpn-mac-dns/master/etc/openvpn/update-resolv-conf --output /tmp/update-resolv-conf
            chmod +x /tmp/update-resolv-conf

      - run:
          name: Check IP before VPN connection
          command: |
            echo "Public IP before VPN connection is $(curl checkip.amazonaws.com)"

      - run:
          name: VPN Setup
          command: |
            echo $VPN_CLIENT_CONFIG | base64 --decode | tee /opt/homebrew/etc/openvpn/openvpn.conf 1>/dev/null
            
            if grep auth-user-pass /opt/homebrew/etc/openvpn/openvpn.conf; then
              if [ -z "${VPN_USER:-}" ] || [ -z "${VPN_PASSWORD:-}" ]; then
                echo "Your VPN client is configured with a user-locked profile. Make sure to set the VPN_USER and VPN_PASSWORD environment variables"
                exit 1
              else
                printf "$VPN_USER\\n$VPN_PASSWORD" > /tmp/vpn.login
                sed -i config.bak 's|^auth-user-pass.*|auth-user-pass /tmp/vpn\.login|' /opt/homebrew/etc/openvpn/openvpn.conf
              fi
            fi

            touch /tmp/openvpn.log
            phone_home="$(ifconfig | grep 'inet ' | awk '{print $2}' | grep -v '127.0.0.1' | awk '{ split($1, a, "."); print a[1] "." a[2] "." a[3] "." a[4] }')"
            echo -e "\nroute $phone_home 255.255.255.255 net_gateway" | tee -a /opt/homebrew/etc/openvpn/openvpn.conf
            echo $phone_home

            cat \<< EOF | sudo tee /Library/LaunchDaemons/org.openvpn.plist 1>/dev/null
            <?xml version="1.0" encoding="UTF-8"?>
            <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
            <plist version="1.0">
              <dict>
                <key>Label</key>
                <string>org.openvpn</string>
                <key>Program</key>
                  <string>/opt/homebrew/opt/openvpn/sbin/openvpn</string>
                <key>ProgramArguments</key>
                <array>
                  <string>/opt/homebrew/opt/openvpn/sbin/openvpn</string>
                  <string>--config</string>
                  <string>/opt/homebrew/etc/openvpn/openvpn.conf</string>
                  <string>--route</string>
                  <string>169.254.0.0 255.255.0.0 net_gateway</string>
                  <string>--script-security</string>
                  <string>2</string>
                  <string>--up</string>
                  <string>/tmp/update-resolv-conf</string>
                  <string>--down</string>
                  <string>/tmp/update-resolv-conf</string>
                </array>
                <key>RunAtLoad</key>
                  <false/>
                <key>TimeOut</key>
                  <integer>90</integer>
                <key>StandardErrorPath</key>
                  <string>/tmp/openvpn.log</string>
                <key>StandardOutPath</key>
                  <string>/tmp/openvpn.log</string>
                <key>KeepAlive</key>
                  <false/>
              </dict>
            </plist>
            EOF

      - run:
          name: Connect to VPN
          background: true
          command: |
            echo "Starting VPN session..."
            sudo launchctl load /Library/LaunchDaemons/org.openvpn.plist
            sudo launchctl start org.openvpn


      - run:
          name: Wait for VPN
          command: |
            printf "Attempting to connect to VPN server...\n\n"

            counter=1
            until [ -f /tmp/openvpn.log ] && [ "$(grep -c "Initialization Sequence Completed" /tmp/openvpn.log)" != 0 ] || [ "$counter" -ge 10 ]; do
              ((counter++))
              
              sleep 5
            done

            if [ ! -f /tmp/openvpn.log ] || (! grep -iq "Initialization Sequence Completed" /tmp/openvpn.log); then
              printf "Unable to establish connection within the allocated time ---> Giving up."
            else
              printf "Connected to VPN\nPublic IP is now %s" "$(curl -s http://checkip.amazonaws.com)"
            fi

      - run:
          name: Run Steps
          command: |
            echo "Run steps here"
            sleep 10
      - run:
          name: Disconnect from OpenVPN
          command: sudo launchctl stop org.openvpn
          when: always
      - store_artifacts:
          path: /tmp/openvpn.log

windows executor (Windows executor images)

• Base64-encode the OpenVPN client configuration file, and store it as an environment variable.
• If the VPN client authentication is credentials-based (user-locked profile), you'll also need to add the username and password as environment variables (VPN_USER and VPN_PASSWORD).

version: 2.1

orbs:
  win: circleci/windows@2.2.0
  
workflows:
  btd:
    jobs:
      - build
jobs:
  build:
    executor:
      name: win/default
      shell: bash.exe
      
    steps:
      - run:
          name: Install OpenVPN
          command: |
            choco install openvpn
            
      - run:
          name: Check IP before VPN connection
          command: echo "Public IP before VPN connection is $(curl checkip.amazonaws.com)"
          
      - run:
          name: VPN Setup
          command: |
            echo $VPN_CLIENT_CONFIG | base64 --decode > /C/PROGRA~1/OpenVPN/config/config.ovpn

            if grep auth-user-pass "/C/PROGRA~1/OpenVPN/config/config.ovpn"; then
              if [ -z "${VPN_USER:-}" ] || [ -z "${VPN_PASSWORD:-}" ]; then
                echo "Your VPN client is configured with a user-locked profile. Make sure to set the VPN_USER and VPN_PASSWORD environment variables"
                exit 1
              else
                printf "$VPN_USER\\n$VPN_PASSWORD" > /C/PROGRA~1/OpenVPN/config/vpn.login
                sed -i 's|^auth-user-pass.*|auth-user-pass vpn\.login|' /C/PROGRA~1/OpenVPN/config/config.ovpn
              fi
            fi
            
            ### IMPORTANT: Include the following 3 lines to exclude the connection from CircleCI and the link-local range
            phone_home=$(netstat -an | grep ':22 .*ESTABLISHED' | head -n1 | awk '{ split($3, a, ":"); print a[1] }')
            echo -e "\nroute $phone_home 255.255.255.255 net_gateway" | tee -a "/C/PROGRA~1/OpenVPN/config/config.ovpn"
            echo "route 169.254.0.0 255.255.0.0 net_gateway" | tee -a "/C/PROGRA~1/OpenVPN/config/config.ovpn"
            
            # Create and start the OpenVPN service
            sc.exe create "OpenVPN" binPath= "C:\PROGRA~1\OpenVPN\bin\openvpnserv.exe"
            net start "OpenVPN"
        
            
      - run:
          name: Wait for the connection to be established and check
          command: |
            counter=1
            until [ $(cat /c/progra~1/openvpn/log/config.log|grep -c "Initialization Sequence Completed") == 0 ]  || [ "$counter" -ge 5 ]; do
              ((counter++))
              echo "Attempting to connect..."
              sleep 1;
            done

            if [ ! -f /c/progra~1/openvpn/log/config.log ] || (! grep -iq "Initialization Sequence Completed" /c/progra~1/openvpn/log/config.log); then
              printf "\nUnable to establish connection within the allocated time ---> Giving up.\n"
            else
              printf "\nVPN connected\n"
              printf "\nPublic IP is now %s\n" "$(curl -s http://checkip.amazonaws.com)"
            fi

      - run:
          name: Run commands in our infrastructure
          command: |
            # A command
            # Another command
            
      - run:
          name: Disconnect from OpenVPN
          command: net stop "OpenVPN"
          when: always

Additional Notes:

Please note: any VPN configuration steps are done at your own risk and can break if any changes occur in our underlying infrastructure's network configuration.

In the event, your configuration is met with This job was claimed but has not received a heartbeat in over 5 minutes error. Try to add routing exclusions to unblock your build using our example below which will require some modifications to meet your build needs.

OpenVPN Example:

  ET_phone_home=$(ss -Hnto state established '( sport = :ssh )' | head -n1 | awk '{ split($4, a, ":"); print a[1] }')
  DEFAULT_GW="$(ip route show default|awk '{print $3}')"
  echo "Original default gateway is $DEFAULT_GW"

  if [ -n "$ET_phone_home" ]; then
    sudo ip route add "$ET_phone_home"/32 via "$DEFAULT_GW"
    echo "Added route to $ET_phone_home/32 via default gateway"
  fi

  for IP in $(host runner.circleci.com | awk '{ print $4; }')
    do
      sudo ip route add "$IP"/32 via "$DEFAULT_GW"
      echo "Added route to $IP/32 via default gateway"
  done

  sudo ip route add 169.254.0.0/16 via "$DEFAULT_GW"