Merge pull request #1743 from yujuhong/cri-testing

Add CRI testing policy

1 files changed, 118 insertions, 0 deletions

diff --git a/contributors/devel/cri-testing-policy.md b/contributors/devel/cri-testing-policy.md
new file mode 100644
index 00000000..fb01b36e
--- /dev/null
+++ b/contributors/devel/cri-testing-policy.md
@@ -0,0 +1,118 @@
+# Container Runtime Interface: Testing Policy
+
+**Owner: SIG-Node**
+
+This document describes testing policy and process for runtimes implementing the
+[Container Runtime Interface (CRI)](/contributors/devel/container-runtime-interface.md)
+to publish test results in a federated dashboard. The objective is to provide
+the Kubernetes community an easy way to track the conformance, stability, and
+supported features of a CRI runtime.
+
+This document focuses on Kubernetes node/cluster end-to-end (E2E) testing
+because many features require integration of runtime, OS, or even the cloud
+provider. A higher-level integration tests provider better signals on vertical
+stack compatibility to the Kubernetes community. On the other hand, runtime
+developers are strongly encouraged to run low-level
+[CRI validation test suite](https://github.com/kubernetes-incubator/cri-tools/blob/master/docs/validation.md)
+for validation as part of their development process.
+
+## Required and optional tests
+
+Runtime maintainers are **required** to submit the tests listed below.
+ 1. Node conformance test suite
+ 2. Node feature test suite
+
+Node E2E tests qualify an OS image with a pre-installed CRI runtime. The
+runtime maintainers are free to choose any OS distribution, packaging, and
+deployment mechanism. Please see the
+[tutorial](https://github.com/kubernetes/community/blob/master/contributors/devel/e2e-node-tests.md)
+to know more about the Node E2E test framework and tests for validating a
+compatible OS image.
+
+The conformance suite is a set of platform-agnostic (e.g., OS, runtime, and
+cloud provider) tests that validate the conformance of the OS image. The feature
+suite allows the runtime to demonstrate what features are supported with the OS
+distribution.
+
+In addition to the required tests, the runtime maintainers are *strongly
+recommended to run and submit results from the Kubernetes conformance test
+suite*. This cluster-level E2E test suite provides extra test signal for areas
+such as Networking, which cannot be covered by CRI, or Node-level
+tests. Because networking requires deep integration between the runtime, the
+cloud provider, and/or other cluster components, runtime maintainers are
+recommended to reach out to other relevant SIGs (e.g., SIG-GCP or SIG-AWS) for
+guidance and/or sponsorship.
+
+## Process for publishing test results
+
+To publish tests results, please submit a proposal in the
+[Kubernetes community repository](https://github.com/kubernetes/community)
+briefly explaining your runtime, providing at least two maintainers, and
+assigning the proposal to the leads of SIG-Node.
+
+These test results should be published under the `sig-node` tab, grouped by the
+runtimes, organized as follows.
+
+```
+sig-node -> sig-node-{Kubernetes-version} -> sig-node-{runtime-name} -> [page containing all test jobs]
+```
+
+The `sig-node` tab only lists up to three most recent Kubernetes versions,
+including the master branch for the current release cycle, e.g.,
+`sig-node-master, sig-node-1.9, sig-node-1.8`.
+
+## Test job maintenance
+
+Tests are required to run at least nightly.
+
+The runtime maintainers are responsible for keeping the tests healthy. If the
+tests are deemed not actively maintained, SIG-Node may remove the tests from
+the test grid at their discretion.
+
+## Process for adding pre-submit testing
+
+If the tests are in good standing (i.e., consistently passing for more than 2
+weeks), the runtime maintainers may request that the tests to be included in the
+pre-submit Pull Request (PR) tests. Please note that the pre-submit tests
+require significantly higher testing capacity, and are held at a higher standard
+since they directly affect the development velocity.
+
+If the tests are flaky or failing, and the maintainers are unable to respond and
+fix the issues in a timely manner, the SIG leads may remove the runtime from
+the presubmit tests until the issues are resolved.
+
+As of now, SIG-Node only accepts promotion of Node conformance tests to
+pre-submit because Kubernetes conformance tests involve a wider scope and may
+need co-sponsorships from other SIGs.
+
+## FAQ
+
+ *1. Can runtime maintainers publish results from other E2E tests?*
+
+Yes, runtime maintainers can publish additional Node E2E tests results. These
+test jobs will be displayed in the `sig-node-{runtime-name}` page. The same
+policy for test maintenance applies.
+
+As for additional Cluster E2E tests, SIG-Node may agree to host the
+results. However, runtime maintainers are strongly encouraged to seek for a more
+appropriate SIG to sponsor or host the results.
+
+ *2. Can these runtime-specific test jobs be considered release blocking?*
+
+This is beyond the authority of SIG-Node, and requires agreement and consensus
+across multiple SIGs (e.g., Release, the relevant cloud provider SIG, etc).
+
+ *3. How to run the aforementioned tests?*
+
+It is hard to keep instructions are even links to them up-to-date in one
+document. Please contact the relevant SIGs for assistance.
+
+ *4. How can I change the test-grid to publish the test results?*
+
+Please contact SIG-Node for the detailed instructions.
+
+ *5. How does this policy apply to Windows containers?*
+
+Windows containers are still in the early development phase and the features
+they support change rapidly. Therefore, it is suggested to treat it as a
+feature with select, whitelisted tests to run.