summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorJohn Belamaric <jbelamaric@google.com>2019-09-27 10:49:35 -0700
committerJohn Belamaric <jbelamaric@google.com>2019-09-30 10:09:33 -0700
commit6170f23cbe2b6767b9f4672a1fde2c8de27cd432 (patch)
tree7c09d1f121f20fa6a130591e242f29236f2afd1b
parent4414db8f8afbf8b2b80fa83a4c512707ac7f7e99 (diff)
Some edits and additional notes for writing conformance tests
Signed-off-by: John Belamaric <jbelamaric@google.com>
-rw-r--r--contributors/devel/sig-testing/writing-good-conformance-tests.md26
1 files changed, 14 insertions, 12 deletions
diff --git a/contributors/devel/sig-testing/writing-good-conformance-tests.md b/contributors/devel/sig-testing/writing-good-conformance-tests.md
index 185b8a9a..dbddf4b0 100644
--- a/contributors/devel/sig-testing/writing-good-conformance-tests.md
+++ b/contributors/devel/sig-testing/writing-good-conformance-tests.md
@@ -2,15 +2,12 @@
The overarching goal of conformance tests is to exercise core
Kubernetes functionality in (as much as possible) a provider-agnostic
-way.
+way.
Conformance tests are promoted from existing e2e tests which have
been proven to be stable and "non-flaky".
-Please read the the
-["Writing good e2e
-tests"](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-testing/writing-good-e2e-tests.md)
-guide as a supplement to this document.
+Please read the the ["Writing good e2e tests"](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-testing/writing-good-e2e-tests.md) guide as a supplement to this document.
### Requirements ###
@@ -20,7 +17,10 @@ the definitive list of requirements in the [SIG-Arch community
docs](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md#conformance-test-requirements).
Generally speaking, tests should not target internal behaviors, but
-instead target exposed system behaviors.
+instead target exposed system behaviors. Care must also be taken to ensure that
+the tests do not implicitly test features that are not already subject to
+conformance. If a feature is needed for a test, make sure that an existing test
+already covers that feature directly.
### Existing Tests ###
@@ -34,14 +34,16 @@ Instead, a better approach may be to duplicate the test, and make your
modifications, thus preserving the original test as-is.
Any existing e2e test should have gone through a standard review
-process by a respective [SIG](https://github.com/ii/k8s-community/tree/writing-good-conformance-tests) in order to
-have been merged in, which means that the test was most likely
+process by a respective [SIG](https://github.com/kubernetes/community/blob/master/sig-list.md)
+in order to have been merged in, which means that the test was most likely
reviewed by domain-specific experts. This can be useful to note when
modifying tests to better suit conformance concerns, or meet
-[requirements](#Requirements).
+[requirements](#Requirements). That said, existings tests are of varying levels
+of quality. It is important to verify that the test actually does validate the
+expected behavior, as described in any API documentation, and as specified in
+the test description you add for promotion.
-
-### New Tests ###
+### New Tests ###
New tests must go through the standard process for e2e tests even when
intended for Conformance from the onset. This means working with
@@ -50,4 +52,4 @@ the test must be in the test cycle for 2 weeks, to prove its
stability. If the test is not slow or flaky, then you may begin the
promotion process by following the steps detailed in ["Promoting
Tests to
-Conformance"](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md#promoting-tests-to-conformance)
+Conformance"](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md#promoting-tests-to-conformance).