This page describes the process and technical documentation around reliable tests at GitLab, for both API and UI end-to-end tests located inside the
Reliable tests are executed as a blocking step in the release process. It is vital that these tests are optimized to run quickly and do not have transient failures. Transient failures of
reliable tests will lead to blocking the release team.
A reliable test is an end-to-end test that passes consistently in all pipelines, including merge requests, for at least seven days. Such a test can be given the
If an end-to-end test consistently passes for 7 consecutive days (as mentioned above), it could be considered a reliable test.
Note: the DRI for promoting existing tests to reliable is the author of the test. In case the author of the test isn't available, the counterpart SET of the test's DevOps stage should be the DRI.
These are the steps required to promote a new test to reliable
type: :new, and they should be monitored in all environments that are part of the release process for seven days.
Note: the DRI for promoting new tests to reliable is the author of the MR that adds the new test(s).
A test is no longer considered reliable if it fails in any pipeline, including in merge requests, and the cause of the failure is found to be
In this case, the following process should be followed.
Note: A test is still reliable if it fails due to a bug in the application code, or due to issues with the application infrastructure that the test is not expected to handle.
Note 2: there's a detailed list of possible failures available in the debugging failing tests guideline, on section 4. Classify and triage the failure
The following command is used to run the reliable tests:
bin/qa Test::Instance::All http://localhost:3000 -- --tag reliable
Note: in the above example,
http://localhost:3000 exemplifies how to run the reliable tests against a local GDK environment. This means that this argument can be changed to run the same tests against different environments.
Reliable tests will be run as part of the release process, during every deployment in staging, canary, and production environments.
This is in addition to the
smoke tests that is already run as part of the release process
There is still one open question that needs to be addressed, as described below.
We could potentially use the test suites report to gather such data (e.g., https://ops.gitlab.net/gitlab-org/quality/staging/pipelines/116213/test_report.)
The following issues could be useful as well.
If there are more open questions they can be added here too.