summaryrefslogtreecommitdiff
path: root/e2e-tests.md
diff options
context:
space:
mode:
authorIsaac Hollander McCreery <ihmccreery@google.com>2016-01-19 16:45:15 -0800
committerIsaac Hollander McCreery <ihmccreery@google.com>2016-01-19 16:51:18 -0800
commit85f44c87d7606fe8ea59f69a9ccda0214bfeffad (patch)
treeb9108c81f2fdbb10dc38e8e6b3fa62fd2c1e4a0b /e2e-tests.md
parent4843e36ddf99f472c10b9268cb795b0156e21169 (diff)
Add docs about [Feature:...] tests for experimental, beta, and GA features
Diffstat (limited to 'e2e-tests.md')
-rw-r--r--e2e-tests.md4
1 files changed, 3 insertions, 1 deletions
diff --git a/e2e-tests.md b/e2e-tests.md
index 6eae78cf..0fc6bcd7 100644
--- a/e2e-tests.md
+++ b/e2e-tests.md
@@ -112,7 +112,9 @@ We are working on implementing clearer partitioning of our e2e tests to make run
- `[Disruptive]`: If a test restarts components that might cause other tests to fail or break the cluster completely, it is labeled `[Disruptive]`. Any `[Disruptive]` test is also assumed to qualify for the `[Serial]` label, but need not be labeled as both. These tests are not run against soak clusters to avoid restarting components.
- `[Flaky]`: If a test is found to be flaky, it receives the `[Flaky]` label until it is fixed. A `[Flaky]` label should be accompanied with a reference to the issue for de-flaking the test, because while a test remains labeled `[Flaky]`, it is not monitored closely in CI. `[Flaky]` tests are by default not run, unless a `focus` or `skip` argument is explicitly given.
- `[Skipped]`: `[Skipped]` is a legacy label that we're phasing out. If a test is marked `[Skipped]`, there should be an issue open to label it properly. `[Skipped]` tests are by default not run, unless a `focus` or `skip` argument is explicitly given.
-- `[Feature:...]`: If a test has non-default requirements to run or targets some non-core functionality, and thus should not be run as part of the standard suite, it receives a `[Feature:...]` label, e.g. `[Feature:Performance]` or `[Feature:Ingress]`. `[Feature:...]` tests are not run in our core suites, instead running in custom suites.
+- `[Feature:...]`: If a test has non-default requirements to run or targets some non-core functionality, and thus should not be run as part of the standard suite, it receives a `[Feature:...]` label, e.g. `[Feature:Performance]` or `[Feature:Ingress]`. `[Feature:...]` tests are not run in our core suites, instead running in custom suites. There are a few use-cases for `[Feature:...]` tests:
+ - If a feature is experimental (i.e. in the `experimental` API or otherwise experimental), it should *not* block the merge-queue, and thus should run in some separate test suites owned by the feature owner(s).
+ - If a feature is in beta or GA, it *should* block the merge-queue. In moving from experimental to beta or GA, tests that are expected to pass by default should simply remove the `[Feature:...]` label, and will be incorporated into our core suites. If tests are not expected to pass by default, (e.g. they require a special environment such as added quota,) they should remain with the `[Feature:...]` label, and the suites that run them should be incorporated into our merge-queue, owned by the Build Cop.
Finally, `[Conformance]` tests are tests we expect to pass on **any** Kubernetes cluster. The `[Conformance]` label does not supersede any other labels. `[Conformance]` test policies are a work-in-progress; see #18162.