times_flaked: definition and timeframe

  1. Flakiness is defined on a rolling 14-day window.
  2. times_flaked is the sum of all flakes detected for a specific test over the last 14 days, qualified by the job it’s been run in, so if Test A is run in Job 1 and Job 2, those are kept as separate counts
  3. Is the `times_flaked` counter collected over a specific timespan? e.g: past day, week? Or is it an all time aggregate? For the count of all the tests that have flaked, is this an all time aggregate?To answer these questions: these are both collected over the most recent 14 day timespan; we delete any records that have a workflow created_at timestamp of older than 14 days
  4. So long as new workflows are created that flake on the same test, that workflow created_at timestamp will be updated

Therefore

If the customer fixes the flaky test, and it doesn’t flake for 14 days, we will no longer flag it as flaky.

If the customer stops running the test, after 14 days, we will no longer flag it as flaky.

Was this article helpful?
0 out of 1 found this helpful

Comments

0 comments

Article is closed for comments.