Transferring Files Off a CircleCI Self-Hosted Machine Runner

In this guide, we will go over two methods to transfer files off a CircleCI Self-hosted runner: Using CircleCI Artifacts and Using SCP when connected on an SSH Re-run

Option 1: Using CircleCI Artifacts

Artifacts in CircleCI are files that jobs in your pipeline create. These could include log files, binaries, screenshots, or other content.

To use CircleCI Artifacts:

1. In your .circleci/config.yml file, specify the path to the file you want to persist as an artifact in the store_artifacts step.

    steps:
      - run: mkdir -p /tmp/artifacts
      - run: echo 'my artifact file' > /tmp/artifacts/my-artifact
      - store_artifacts:
          path: /tmp/artifacts
          destination: artifact-directory

In the above example, my-artifact is the file created on the CircleCI self-hosted runner, artifact-directory is the location to which the artifact will be uploaded and stored.

2. Once the job has run, you can access the saved artifacts from the CircleCI web interface by navigating to the Artifacts tab on the job page.

Option 2: Using SCP with the -o Flag while re-running with SSH

Some teams and organizations may not wish to use the CircleCI artifacts feature and would prefer to transfer files during runtime where SCP can be used as a means of securely transferring files between hosts on a network. You can use SCP with a CircleCI Self-hosted runner to copy files. The -O flag will use the original SCP protocol for file transfers instead of the SFTP protocol.

Prerequisite:

• Ensure your runner has SSH enabled using the runner.ssh.advertise_addr parameter 

1. Once a job is re-run with SSH and the connection is opened to the host, you can run the following to copy a file over

scp -p -O -P 54782 $HOSTNAME:/path/to/destination ./my_file

In this example:

• my_file is the destination you want to copy to.
•  username@hostname:/path/to/destination is the destination from which you want the file copied.
• We are using the default port (54782) for the runner.ssh.advertise_addr parameter

Additional Notes

• For both solutions, it's crucial to consider the security implications of transferring files. Be aware of the nature of the files, where they are being transferred, and who potentially has access to them.

Additional Resources

• SSH Rerun Job Step Not Showing Up
• Enable SSH on a Machine Runner
• Does Container Runner support SSH Debugging?