Caching Dependencies for optimal performance

Using caching tools for your dependencies can significantly speed up your jobs in CircleCI. Here are some key points to keep in mind.

Saving and Restoring Cache examples

To save a cache, specify the key and paths to the files or directories you want to cache. For example:

steps:
  - save_cache:
     key: my-cache
     paths:
       - my-file.txt
       - my-project/my-dependencies-directory

To restore a cache, provide the corresponding keys and include cascading fallback unless the cache is immutable. For example:

steps:
   - restore_cache:
       keys:
         # Find a cache corresponding to this specific package-lock.json checksum
         # when this file is changed, this key will fail
         - v1-npm-deps-{{ checksum "package-lock.json" }}
         # Find the most recently generated cache used from any branch
         - v1-npm-deps-

You can see more pertaining to these basic examples here along with other resources related to caching configuration for your project.

You can also view our demo apps to see how caching might be implemented for your project type.

Cache Management

Caches cannot be cleared. If you need to generate a new set of caches you can update the cache key, similar to the previous example. You might wish to do this if you have updated language or build management tool versions.

Updating the cache key on save and restore steps in your .circleci/config.yml file will then generate new sets of caches from that point onwards. Please note that older commits using the previous keys may still generate and save cache, so it is recommended that you rebase after the ‘config.yml’ changes when possible.

Cache Size

To ensure optimal performance, it is recommended to keep cache sizes under 500MB. Excessively large caches can result in longer verification and corruption check times. If your cache exceeds this size, consider splitting it into multiple smaller caches to maintain efficiency.

Leveraging Orb Commands

Consider leveraging orb commands, such as `python/install-packages` from the circleci/python orb. The Python orb's commands include `save_cache` and `restore_cache` steps, as seen in the source code at lines 426 and 188 respectively. These steps contribute to improved performance and can be customized if needed.

Alternatively, if caching causes issues with credit utilization, you can install Python packages using `pip install` at runtime without relying on the orb.

Caching Dependencies for Yarn and Node.js Projects

Caching dependencies can significantly improve job performance on CircleCI. This article provides guidance on caching options specifically tailored for Yarn and Node.js dependencies.

Yarn Caching

To maximize caching efficiency when using Yarn, it is recommended to cache the `.yarn/cache` and `.yarn/unplugged` folders instead of the `node_modules` directory directly. This approach aligns with Yarn's caching mechanism. Here's an example configuration for Yarn caching in your CircleCI configuration file:

- yarn/
- .yarn/cache
- .yarn/unplugged

Node Orb Recommendation

For Node.js projects, CircleCI provides the Node Orb, which offers caching options specifically designed for Node.js dependencies. You can utilize the `node-cache` command provided by the Node Orb to cache your `node_modules` directory efficiently. Here's an example configuration:

version: 2.1

orbs:
node: circleci/node@2.1

jobs:
build:
executor: node/default
steps:
- checkout
- node/cache
- run: npm install
- run: npm test

Troubleshooting Checklist

Tips for Optimizing Caching

1. Order jobs for meaningful workflows.
2. Check language-specific caching tips:
- Node (Yarn): Cache `.yarn/cache` and `.yarn/unplugged` folders.
- Node (npm): Cache with lock files (`package-lock.json` or `npm-shrinkwrap.json`).
- Consider using the CircleCI Node Orb for additional caching options.
3. Verify cache restoration and saving.
4. Cache only necessary dependencies.
5. Prune unnecessary dependencies.
6. Optimize cache keys for better performance.
7. Tolerate slightly outdated caches to reduce unnecessary uploads.
8. Explore partial dependency caching strategies for specific package managers.
9. Evaluate tradeoffs between zero and partial cache restores.
10. Use lock files or checksums for precise cache keys.
11. Cache multiple languages separately.
12. Cache expensive steps to improve performance.

Remember to adapt these tips to your specific needs and language requirements.

Youtube resources

Additional Resources:

• CircleCI Documentation: Caching Dependencies