Potential Secret Leaks

If you have a reason to think you have been compromised while using our platform, please review this article. Included are questions necessary to provide to help the Security team with the investigation and a checklist you can go through to investigate issues on your side.

We take security issues very seriously and will help with investigations as much as possible. Please open a ticket by emailing security@circleci.com. 

Information required for CircleCI Security Investigation

Please provide:

1. The type of credentials or breach
2. If used within a context or environment variable, the project, context name, and variable name in question.
3. If you have run a workflow or job using the credentials, provide links to them.
4. If you have public code repositories, please provide a list of them and if you allow forks and secrets on those forks.
5. For each event the time in UTC or with timezone
6. A clear description of the issue
7. Explanation why the issue is suspected to be within the CircleCI platform
8. Actions taken to resolve the issue
9. Any information they have that could help us identify the suspicious behavior on our end. Usernames, IP addresses, user agents etc

Checklist for Customer Investigation

Review end users device security to make sure handling of any secure data locally is not compromised. For example keyloggers, info stealers, malware can copy secrets from memory, on disk in configuration files (eg ~/,aws./config or ~/.ssh/) the clipboard or in images/screenshots. Beware of malicious software that is not detected by A/V.
Review CircleCI Audit Logs for access to context and environment variable, for example the Action field of context.secrets.accessed and the last time the context was rotated via Action fields context.env_var.delete and context.env_var.store Project environment variables are always injected inside the running job and you can check the last time they were rotated with Action fields project.env_var.delete and project.env_var.create
Review CircleCI Audit Logs for unexpected workflows or branches. For example filter on Action field workflow.job.start and look for the compromised context in the Payload field, inside that field check for unexpected job_name or vcs_branch using the context
Review who has Admin access to CircleCI and if jobs that contain secrets have been 're-run with SSH'. You can check the CircleCI Audit Logs for tags that contain 'rerun-single-job-with-ssh'
Review who has CircleCI API tokens (project, personal) and rotate them
Review the jobs that use the compromised secrets and look at the console output and their artifacts to check that no secrets are copied over in plain text or exposed to the internet. Secrets printed in plain text to the console output are normally masked, but look for any signs of malicious tampering to circumvent or exfiltrate secrets
Review build dependencies and libraries, and their authenticity to rule out a malicious downstream dependency. This will depend on the languages and frameworks in place and their ecosystem (NPM, Ruby, Python etc)
Review your VCS git repository audit logs and look for unexpected changes to the CircleCI configuration file config.yml and code. Look for any sign of a developer's SSH key or VCS API token being compromised