diff options
| author | John Belamaric <jbelamaric@google.com> | 2019-09-27 10:49:35 -0700 |
|---|---|---|
| committer | John Belamaric <jbelamaric@google.com> | 2019-09-30 10:09:33 -0700 |
| commit | 6170f23cbe2b6767b9f4672a1fde2c8de27cd432 (patch) | |
| tree | 7c09d1f121f20fa6a130591e242f29236f2afd1b | |
| parent | 4414db8f8afbf8b2b80fa83a4c512707ac7f7e99 (diff) | |
Some edits and additional notes for writing conformance tests
Signed-off-by: John Belamaric <jbelamaric@google.com>
| -rw-r--r-- | contributors/devel/sig-testing/writing-good-conformance-tests.md | 26 |
1 files changed, 14 insertions, 12 deletions
diff --git a/contributors/devel/sig-testing/writing-good-conformance-tests.md b/contributors/devel/sig-testing/writing-good-conformance-tests.md index 185b8a9a..dbddf4b0 100644 --- a/contributors/devel/sig-testing/writing-good-conformance-tests.md +++ b/contributors/devel/sig-testing/writing-good-conformance-tests.md @@ -2,15 +2,12 @@ The overarching goal of conformance tests is to exercise core Kubernetes functionality in (as much as possible) a provider-agnostic -way. +way. Conformance tests are promoted from existing e2e tests which have been proven to be stable and "non-flaky". -Please read the the -["Writing good e2e -tests"](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-testing/writing-good-e2e-tests.md) -guide as a supplement to this document. +Please read the the ["Writing good e2e tests"](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-testing/writing-good-e2e-tests.md) guide as a supplement to this document. ### Requirements ### @@ -20,7 +17,10 @@ the definitive list of requirements in the [SIG-Arch community docs](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md#conformance-test-requirements). Generally speaking, tests should not target internal behaviors, but -instead target exposed system behaviors. +instead target exposed system behaviors. Care must also be taken to ensure that +the tests do not implicitly test features that are not already subject to +conformance. If a feature is needed for a test, make sure that an existing test +already covers that feature directly. ### Existing Tests ### @@ -34,14 +34,16 @@ Instead, a better approach may be to duplicate the test, and make your modifications, thus preserving the original test as-is. Any existing e2e test should have gone through a standard review -process by a respective [SIG](https://github.com/ii/k8s-community/tree/writing-good-conformance-tests) in order to -have been merged in, which means that the test was most likely +process by a respective [SIG](https://github.com/kubernetes/community/blob/master/sig-list.md) +in order to have been merged in, which means that the test was most likely reviewed by domain-specific experts. This can be useful to note when modifying tests to better suit conformance concerns, or meet -[requirements](#Requirements). +[requirements](#Requirements). That said, existings tests are of varying levels +of quality. It is important to verify that the test actually does validate the +expected behavior, as described in any API documentation, and as specified in +the test description you add for promotion. - -### New Tests ### +### New Tests ### New tests must go through the standard process for e2e tests even when intended for Conformance from the onset. This means working with @@ -50,4 +52,4 @@ the test must be in the test cycle for 2 weeks, to prove its stability. If the test is not slow or flaky, then you may begin the promotion process by following the steps detailed in ["Promoting Tests to -Conformance"](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md#promoting-tests-to-conformance) +Conformance"](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md#promoting-tests-to-conformance). |
