From 418a93350678fc9d4a180f5defe57ab2f6e93c20 Mon Sep 17 00:00:00 2001 From: John Belamaric Date: Tue, 9 Apr 2019 16:30:52 -0700 Subject: Add detail on the conformance review process * Adds the criteria to have an e2e test running for one release before promotion. * Adds a section on the review board describing the types of PRs that require approval through that process. * Updates the link to the review board. --- .../devel/sig-architecture/conformance-tests.md | 28 +++++++++++++++++++--- 1 file changed, 25 insertions(+), 3 deletions(-) diff --git a/contributors/devel/sig-architecture/conformance-tests.md b/contributors/devel/sig-architecture/conformance-tests.md index d726094e..767db367 100644 --- a/contributors/devel/sig-architecture/conformance-tests.md +++ b/contributors/devel/sig-architecture/conformance-tests.md @@ -46,7 +46,8 @@ specifically, a test is eligible for promotion to conformance if: kubernetes releases are built - it passes against the appropriate versions of kubernetes as spelled out in the [conformance test version skew policy] -- it is stable and runs consistently (e.g., no flakes) +- it is stable and runs consistently (e.g., no flakes), and has been running + for at least one release cycle Examples of features which are not currently eligible for conformance tests: @@ -135,6 +136,25 @@ This document will help people understand what features are being tested without having to look through the testcase's code directly. +## Conformance test review board + +The conformance subproject uses the [Conformance Test Review board] to track +progress of PRs through to approval. The following types of PRs must go through +this approval process: + +- promotion of tests to conformance +- demotion of tests from conformance +- changes to existing conformance tests +- changes to the conformance criteria or process +- changes to the conformance infrastructure code + +There are six columns in this board. New PRs should enter in the To Triage +column, and [Conformance test reviewers] will pick it up from there and move it +through the process. New end-to-end tests that are intended to be promoted to +conformance tests in the future may be added to this board, but they will not +move all the way to the Needs Approval column, as that is intended only for the +types of PRs described above. + ## Promoting Tests to Conformance To promote a test to the conformance test suite, open a PR as follows: @@ -152,7 +172,8 @@ To promote a test to the conformance test suite, open a PR as follows: than the `framework.It()` function - adds a comment immediately before the `ConformanceIt()` call that includes all of the required [conformance test comment metadata] -- add the PR to SIG Architecture's [Conformance Test Review board] +- add the PR to SIG Architecture's [Conformance Test Review board] in the To + Triage column ### Conformance Test Comment Metadata @@ -207,7 +228,8 @@ Conformance test results, by provider and releases, can be viewed in the for your provider, please see the [testgrid conformance README] [kubernetes versioning policy]: /contributors/design-proposals/release/versioning.md#supported-releases-and-component-skew -[Conformance Test Review board]: https://github.com/kubernetes-sigs/architecture-tracking/projects/1 +[Conformance Test Review board]: https://github.com/orgs/kubernetes/projects/9 +[Conformance test reviewers]: https://github.com/kubernetes/kubernetes/blob/master/test/conformance/testdata/OWNERS [conformance test requirements]: #conformance-test-requirements [conformance test metadata]: #conformance-test-metadata [conformance test version skew policy]: #conformance-test-version-skew-policy -- cgit v1.2.3 From b5a2d835b0a6944a645cd7c8a312f1cae38f7335 Mon Sep 17 00:00:00 2001 From: John Belamaric Date: Thu, 11 Apr 2019 12:29:11 -0700 Subject: Fix metadata link, add conformance.txt remark --- contributors/devel/sig-architecture/conformance-tests.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/contributors/devel/sig-architecture/conformance-tests.md b/contributors/devel/sig-architecture/conformance-tests.md index 767db367..91ac862b 100644 --- a/contributors/devel/sig-architecture/conformance-tests.md +++ b/contributors/devel/sig-architecture/conformance-tests.md @@ -172,6 +172,7 @@ To promote a test to the conformance test suite, open a PR as follows: than the `framework.It()` function - adds a comment immediately before the `ConformanceIt()` call that includes all of the required [conformance test comment metadata] + - adds the test name to the [conformance.txt] file - add the PR to SIG Architecture's [Conformance Test Review board] in the To Triage column @@ -231,8 +232,9 @@ for your provider, please see the [testgrid conformance README] [Conformance Test Review board]: https://github.com/orgs/kubernetes/projects/9 [Conformance test reviewers]: https://github.com/kubernetes/kubernetes/blob/master/test/conformance/testdata/OWNERS [conformance test requirements]: #conformance-test-requirements -[conformance test metadata]: #conformance-test-metadata +[conformance test comment metadata]: #conformance-test-comment-metadata [conformance test version skew policy]: #conformance-test-version-skew-policy [testgrid conformance dashboard]: https://testgrid.k8s.io/conformance-all [testgrid conformance README]: https://github.com/kubernetes/test-infra/blob/master/testgrid/conformance/README.md [v1.9 conformance doc]: https://github.com/cncf/k8s-conformance/blob/master/docs/KubeConformance-1.9.md +[conformance.txt]: https://github.com/kubernetes/kubernetes/blob/master/test/conformance/testdata/conformance.txt -- cgit v1.2.3