Potential Secret Leaks

If you need assistance from CircleCI with a security investigation

We take security issues very seriously and will help with your investigation as much as possible. If you need assistance, please open a ticket by emailing security@circleci.com. Please include:

1. Clear description of the issue, including the type of credentials or suspected breach
2. Explanation why the issue is suspected to be within the CircleCI platform
3. Concise timeline of events you observed (with timestamps in UTC or specified timezone)
4. Information that could help us identify the suspicious behavior on our end - e.g. usernames, IP addresses, user agents, etc.
5. Links to all CircleCI workflows or jobs using leaked credentials
6. If the secret is used within a CircleCI context or environment variable, the project name, context name, and variable name
7. If applicable, a list of your public code repositories and if you allow forks and use of secrets on those forks.
8. Any actions you may have taken to investigate and/or remediate the issue

Accidental secret exposure via misconfigured JavaScript build tooling is the most frequent cause of customer-reported secret leaks

Secrets are often leaked when misconfigured JavaScript build tooling causes CI/CD environment variables to be unintentionally included in deployment bundles. This can happen when environment variables are included in frontend assets as part of the build process. When these environment variables include API keys, deployment of the resulting bundles to public websites can unexpectedly make the API keys trivially available to threat actors.

As a rule of thumb, ruling out misconfigured JavaScript build tooling as a root cause should be high priority in case of a suspected secret leak.

To confirm if this is the case, there are a number of free and open source tools that can assist in detection of API keys on web pages. Developer tools in the browser (such as Chrome Developer Tools' "Sources" panel) can also be used to help determine if an API key might be unintentionally leaked in deployed JavaScript.

Checklist for Customer Investigation

Web Frontend Assets

⬜️  Examine deployed web resources to ensure there are no unsecured API keys or tokens in frontend assets. Audit JavaScript files for environment variables or secrets that could be exposed to clients.

CircleCI 

⬜️  Review console output of jobs that use the compromised secrets. Secrets printed in plain text to the console output are normally masked. Look for evidence of secret masking bypass or secret exfiltration. 

⬜️  Review test results and artifacts, especially for open source projects. As documented in https://circleci.com/docs/env-vars/#secrets-masking, CircleCI secrets masking will only prevent values from appearing in job output. If secrets appear elsewhere, such as test results or artifacts, they will not be masked.

⬜️  Rotate CircleCI API tokens (project, personal) and review who has access to them.

⬜️  Review CircleCI Audit Logs for unexpected access to context and project variables. Review the Action field of context.secrets.accessed. Last rotation time can be checked via Action fields context.env_var.delete and context.env_var.store. Project environment variables are always injected inside the running job.You can check the last time they were rotated with Action fields project.env_var.delete and project.env_var.create

⬜️  Review CircleCI Audit Logs for unexpected workflows or branches. Filter on Action field workflow.job.start and look for the compromised context in the Payload field. Inside that field, check for unexpected job_name or vcs_branch using the context

⬜️  Review CircleCI Audit Logs for jobs rerun with SSH. These will be indicated by tags that contain rerun-single-job-with-ssh. Determine if the exposed secrets were available to that job. 

For more information on CircleCI Audit Logs (including how to request audit logs from the web app), see https://circleci.com/docs/audit-logs. 

Dependencies & Libraries

⬜️  Review third-party dependencies and libraries and their authenticity to rule out a malicious or compromised dependency (e.g. via supply-chain attack)

Version Control System (VCS)

⬜️  Review VCS repository commit history for unexpected changes to the CircleCI configuration file config.yml and code. 

⬜️  Review VCS repository audit logs for evidence of a developer's SSH key or VCS API token being compromised

⬜️  Review VCS repositories to ensure that secrets are not exposed. Be sure to include older commits, since deleting the secret from the latest version would not remove it from commit history. 

Docker Images

⬜️  Check Docker images for embedded secrets, including secrets in configuration files within the image, and credentials hardcoded in application code. (Be aware of cached layers; even deleted files can often be recovered from those layers)

Application Artifacts

⬜️  Inspect compiled application artifacts - e.g. Apple apps, Android .apk packages, etc. - for hardcoded secrets. Utilize tools for decompiling and inspecting binaries if necessary.

Cloud Storage

⬜️  Secure public cloud storage buckets (e.g. AWS S3 buckets). Verify that no sensitive files were uploaded to public cloud storage. Implement appropriate bucket policies to enforce access controls.

Build Configuration

⬜️  Review build configuration. Ensure no secrets (API keys, passwords, tokens, etc.) are exposed in build configuration files (e.g. .yml, .json, .env).

Developer End User Devices

⬜️  Review developer end user device security to make sure handling of any secure data locally is not compromised. For example keyloggers, info stealers, malware can copy secrets from memory, on disk, in configuration files (eg ~/.aws, ~/.config or ~/.ssh), the clipboard, or in images/screenshots. Beware of malicious software that is not detected by antivirus software.