If your jobs are inconsistently failing, and are making requests to external services, you may see a much higher false positive rate. Such requests are typically dependency installs and tests connecting to 3rd party APIs. There are three main reasons for the higher failure rate:
- Your jobs run in a shared environment with other jobs
- Internet network is not 100% reliable
- 3rd party servers can become unavailable.
It's difficult for us to determine which of these is the problem, as it usually only exists for a short time.
While jobs that always integrate with external services may offer additional confidence in your code working as intended, it comes at the cost of false positives (flakiness) in your CI runs. The only way to make jobs more reliable - failing when there's a problem with your code instead of something external -is to minimize network requests. This means using mocked responses and caching dependencies where possible. This isn't to say jobs that install dependencies or talk to a 3rd party API on every run are bad, just that they offer a tradeoff to consider.