- Flakiness is defined on a rolling 14-day window.
times_flaked
is the sum of all flakes detected for a specific test over the last 14 days, qualified by the job it’s been run in, so if Test A is run in Job 1 and Job 2, those are kept as separate counts- Is the `times_flaked` counter collected over a specific timespan? e.g: past day, week? Or is it an all time aggregate? For the count of all the tests that have flaked, is this an all time aggregate?To answer these questions: these are both collected over the most recent 14 day timespan; we delete any records that have a workflow
created_at
timestamp of older than 14 days - So long as new workflows are created that flake on the same test, that workflow
created_at
timestamp will be updated
Therefore
If the customer fixes the flaky test, and it doesn’t flake for 14 days, we will no longer flag it as flaky.
If the customer stops running the test, after 14 days, we will no longer flag it as flaky.
Comments
Article is closed for comments.