diff options
| author | Jeff Lowdermilk <jeffml@google.com> | 2016-02-02 18:19:00 -0800 |
|---|---|---|
| committer | Jeff Lowdermilk <jeffml@google.com> | 2016-02-02 18:19:00 -0800 |
| commit | 7cba47d83d4941948d4c55557f150a25635ca4f4 (patch) | |
| tree | 8d260a2412a067e1c16cbd0ba311b77a1aba9d2c | |
| parent | c657c0c84c4051e5e2e64b3f0b4bd1fa51080a7d (diff) | |
| parent | dfd29cb1235b2d694d860c7f26a5de9888d1041b (diff) | |
Merge pull request #20453 from ihmccreery/flaky-label-update
Update docs on flaky issues
| -rw-r--r-- | e2e-tests.md | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/e2e-tests.md b/e2e-tests.md index 388e25f0..8d736f25 100644 --- a/e2e-tests.md +++ b/e2e-tests.md @@ -110,7 +110,7 @@ We are working on implementing clearer partitioning of our e2e tests to make run - `[Slow]`: If a test takes more than five minutes to run (by itself or in parallel with many other tests), it is labeled `[Slow]`. This partition allows us to run almost all of our tests quickly in parallel, without waiting for the stragglers to finish. - `[Serial]`: If a test cannot be run in parallel with other tests (e.g. it takes too many resources or restarts nodes), it is labeled `[Serial]`, and should be run in serial as part of a separate suite. - `[Disruptive]`: If a test restarts components that might cause other tests to fail or break the cluster completely, it is labeled `[Disruptive]`. Any `[Disruptive]` test is also assumed to qualify for the `[Serial]` label, but need not be labeled as both. These tests are not run against soak clusters to avoid restarting components. -- `[Flaky]`: If a test is found to be flaky, it receives the `[Flaky]` label until it is fixed. A `[Flaky]` label should be accompanied with a reference to the issue for de-flaking the test, because while a test remains labeled `[Flaky]`, it is not monitored closely in CI. `[Flaky]` tests are by default not run, unless a `focus` or `skip` argument is explicitly given. +- `[Flaky]`: If a test is found to be flaky and we have decided that it's too hard to fix in the short term (e.g. it's going to take a full engineer-week), it receives the `[Flaky]` label until it is fixed. The `[Flaky]` label should be used very sparingly, and should be accompanied with a reference to the issue for de-flaking the test, because while a test remains labeled `[Flaky]`, it is not monitored closely in CI. `[Flaky]` tests are by default not run, unless a `focus` or `skip` argument is explicitly given. - `[Skipped]`: `[Skipped]` is a legacy label that we're phasing out. If a test is marked `[Skipped]`, there should be an issue open to label it properly. `[Skipped]` tests are by default not run, unless a `focus` or `skip` argument is explicitly given. - `[Feature:...]`: If a test has non-default requirements to run or targets some non-core functionality, and thus should not be run as part of the standard suite, it receives a `[Feature:...]` label, e.g. `[Feature:Performance]` or `[Feature:Ingress]`. `[Feature:...]` tests are not run in our core suites, instead running in custom suites. There are a few use-cases for `[Feature:...]` tests: - If a feature is experimental or alpha and is not enabled by default due to being incomplete or potentially subject to breaking changes, it should *not* block the merge-queue, and thus should run in some separate test suites owned by the feature owner(s). |
