Flaky tests

What's a flaky test?

It's a test that sometimes fails, but if you retry it enough times, it passes, eventually.

Automatic retries and flaky tests detection

On our CI, we use rspec-retry to automatically retry a failing example a few times (see spec/spec_helper.rb for the precise retries count).

We also use a home-made RspecFlaky::Listener listener which records flaky examples in a JSON report file on master (retrieve-tests-metadata and update-tests-metadata jobs), and warns when a new flaky example is detected in any other branch (flaky-examples-check job). In the future, the flaky-examples-check job will not be allowed to fail.

This was originally implemented in: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/13021.

Problems we had in the past at GitLab

rspec-retry is bitting us when some API specs fail: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/9825
Sporadic RSpec failures due to PG::UniqueViolation: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/9846
- Follow-up: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10688
- Capybara.reset_session! should be called before requests are blocked: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12224
FFaker generates funky data that tests are not ready to handle (and tests should be predictable so that's bad!):

Time-sensitive flaky tests

Array order expectation

https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10148

Feature tests

Capybara viewport size related issues

Transient failure of spec/features/issues/filtered_search/filter_issues_spec.rb: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10411

Capybara JS driver related issues

PhantomJS / WebKit related issues

Memory is through the roof! (TL;DR: Load images but block images requests!): https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12003

Resources

Return to Testing documentation