Merge pull request #4107 from ii/writing-good-conformance-tests

New document "Writing Good Conformance Tests"

1 files changed, 53 insertions, 0 deletions

diff --git a/contributors/devel/sig-testing/writing-good-conformance-tests.md b/contributors/devel/sig-testing/writing-good-conformance-tests.md
new file mode 100644
index 00000000..185b8a9a
--- /dev/null
+++ b/contributors/devel/sig-testing/writing-good-conformance-tests.md
@@ -0,0 +1,53 @@
+# Writing Good Conformance Tests for Kubernetes #
+
+The overarching goal of conformance tests is to exercise core
+Kubernetes functionality in (as much as possible) a provider-agnostic
+way. 
+
+Conformance tests are promoted from existing e2e tests which have
+been proven to be stable and "non-flaky".
+
+Please read the the 
+["Writing good e2e
+tests"](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-testing/writing-good-e2e-tests.md)
+guide as a supplement to this document.
+
+### Requirements  ###
+
+In addition to the "Writing good e2e tests" guide, there are specific
+requirements that all Conformance tests must adhere to. You can find
+the definitive list of requirements in the [SIG-Arch community
+docs](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md#conformance-test-requirements).
+
+Generally speaking, tests should not target internal behaviors, but
+instead target exposed system behaviors. 
+
+### Existing Tests ###
+
+When looking for an existing e2e test which exercises a behaviour
+which is useful for Conformance, you may find that the test does not
+meet all of the [requirements](#Requirements). In these cases, you
+might either modify the test to meet requirements, but sometimes doing
+so will mean losing some of the intent of the test.
+
+Instead, a better approach may be to duplicate the test, and make your
+modifications, thus preserving the original test as-is.
+
+Any existing e2e test should have gone through a standard review
+process by a respective [SIG](https://github.com/ii/k8s-community/tree/writing-good-conformance-tests) in order to
+have been merged in, which means that the test was most likely
+reviewed by domain-specific experts. This can be useful to note when
+modifying tests to better suit conformance concerns, or meet
+[requirements](#Requirements).
+
+
+### New Tests ### 
+
+New tests must go through the standard process for e2e tests even when
+intended for Conformance from the onset. This means working with
+SIG-specific reviewers in order to get your PR accepted. After which,
+the test must be in the test cycle for 2 weeks, to prove its
+stability. If the test is not slow or flaky, then you may begin the
+promotion process by following the steps detailed in ["Promoting
+Tests to
+Conformance"](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md#promoting-tests-to-conformance)