diff options
| author | Aaron Crickenberger <spiffxp@google.com> | 2018-09-24 13:23:50 -0700 |
|---|---|---|
| committer | Aaron Crickenberger <spiffxp@google.com> | 2018-10-08 15:15:53 -0700 |
| commit | c59aea6fb39a56bd204bedee21eb3966c94cb7f1 (patch) | |
| tree | 1c2d911c431431d3cb04c09ea5e693ec65130760 | |
| parent | 2c134dbb59bb091ddbbff4f4c6122cb1d6a1e547 (diff) | |
Address review comments
| -rw-r--r-- | contributors/devel/conformance-tests.md | 59 | ||||
| -rw-r--r-- | contributors/devel/e2e-tests.md | 6 |
2 files changed, 38 insertions, 27 deletions
diff --git a/contributors/devel/conformance-tests.md b/contributors/devel/conformance-tests.md index 858dc0ca..46ca318d 100644 --- a/contributors/devel/conformance-tests.md +++ b/contributors/devel/conformance-tests.md @@ -8,8 +8,6 @@ expected behavior works as a user might encounter it in the wild. The process to add new conformance tests is intended to decouple the development of useful tests from their promotion to conformance: - Contributors write and submit e2e tests, to be approved by owning SIGs - - Tests should begin by covering the most important and visible aspects of the - function - Tests are proven to meet the [conformance test requirements] by review and by accumulation of data on flakiness and reliability - A follow up PR is submitted to [promote the test to conformance](#promoting-tests-to-conformance) @@ -22,7 +20,7 @@ NB: This should be viewed as a living document in a few key areas: non-optional features or APIs. The list of requirements will be refined over time to the point where it as concrete and complete as possible. - There are currently conformance tests that violate some of the requirements - (ie: require privileged access), we will be categorizing these tests and + (e.g., require privileged access), we will be categorizing these tests and deciding what to do once we have a better understanding of the situation - Once we resolve the above issues, we plan on identifying the appropriate areas to relax requirements to allow for the concept of conformance Profiles that @@ -33,20 +31,22 @@ NB: This should be viewed as a living document in a few key areas: Conformance tests currently test only GA, non-optional features or APIs. More specifically, a test is eligible for promotion to conformance if: -- it tests only GA, non-optional features or APIs (ie: no alpha or beta +- it tests only GA, non-optional features or APIs (e.g., no alpha or beta endpoints, no feature flags required, no deprecated features) -- it works for all providers (ie: no `SkipIfProviderIs`/`SkipUnlessProviderIs` +- it works for all providers (e.g., no `SkipIfProviderIs`/`SkipUnlessProviderIs` calls) -- it is non-privileged (ie: no root on nodes, network, nor cluster) -- it works without access to the public internet +- it is non-privileged (e.g., does not require root on nodes, access to raw + network interfaces, or cluster admin permissions) +- it works without access to the public internet (short of whatever is required + to pre-pull images for conformance tests) - it works without non-standard filesystem permissions granted to pods - it does not rely on any binaries that would not be required for the linux - kernel or kubelet to run (ie: can't rely on git) + kernel or kubelet to run (e.g., can't rely on git) - any container images used within the test support all architectures for which kubernetes releases are built -- it passes against the previous release of kubernetes as well as the current - version of kubernetes per the [conformance test version skew policy] -- it is stable and runs consistently (ie: no flakes) +- it passes against the appropriate versions of kubernetes as spelled out in + the [conformance test version skew policy] +- it is stable and runs consistently (e.g., no flakes) Examples of features which are not currently eligible for conformance tests: @@ -63,6 +63,13 @@ Examples of tests which are not eligible for promotion to conformance: these may change over time (however it is reasonable to verify these fields exist or are non-empty) +Examples of areas we may want to relax these requirements once we have a +sufficient corpus of tests that define out of the box functionality in all +reasonable production worthy environments: +- tests may need to create or set objects or fields that are alpha or beta that + bypass policies that are not yet GA, but which may reasonably be enabled on a + conformant cluster (e.g., pod security policy, non-GA scheduler annotations) + ## Conformance Test Version Skew Policy As each new release of Kubernetes provides new functionality, the subset of @@ -72,16 +79,16 @@ as laid out in the [kubernetes versioning policy] To quote: -> For example, a v1.3 master should work with v1.1, v1.2, and v1.3 nodes, and +> For example, a v1.3 master should work with v1.1, v1.2, and v1.3 nodes, and > should work with v1.2, v1.3, and v1.4 clients. Conformance tests for a given version should be run off of the release branch that corresponds to that version. Thus `v1.2` conformance tests would be run from the head of the `release-1.2` branch. -For example, suppose we're in the midst of developing kubernetes v1.3. The -following clusters must pass conformance tests built from the following -branches: +For example, suppose we're in the midst of developing kubernetes v1.3. Clusters +with the following versions must pass conformance tests built from the +following branches: | cluster version | master | release-1.3 | release-1.2 | release-1.1 | | --------------- | ----- | ----------- | ----------- | ----------- | @@ -91,10 +98,14 @@ branches: ## Running Conformance Tests -Conformance tests are designed to be run with no cloud provider configured. -Conformance tests must be able to be run against clusters that have not been -created with `hack/e2e.go`, just provide a kubeconfig with the appropriate -endpoint and credentials. +Conformance tests are designed to be run even when there is no cloud provider +configured. Conformance tests must be able to be run against clusters that have +not been created with `hack/e2e.go`, just provide a kubeconfig with the +appropriate endpoint and credentials. + +These commands are intended to be run within a kubernetes directory, either +cloned from source, or extracted from release artifacts such as +`kubernetes.tar.gz`. They assume you have a valid golang installation. ```sh # ensure kubetest is installed @@ -127,7 +138,7 @@ having to look through the testcase's code directly. ## Promoting Tests to Conformance To promote a test to the conformance test suite, open a PR as follows: -- is titled "Promte xxx e2e test to Conformance" +- is titled "Promote xxx e2e test to Conformance" - includes information and metadata in the description as follows: - "/area conformance" on a newline - "@kubernetes/sig-architecture-pr-reviews @kubernetes/sig-foo-pr-reviews @@ -136,7 +147,7 @@ To promote a test to the conformance test suite, open a PR as follows: - any necessary information in the description to verify that the test meets [conformance test requirements], such as links to reports or dashboards that prove lack of flakiness -- modifies code as follows: +- contains no other modifications to test source code other than the following: - modifies the testcase to use the `framework.ConformanceIt()` function rather than the `framework.It()` function - adds a comment immediately before the `ConformanceIt()` call that includes @@ -154,7 +165,7 @@ within its associated comment: then those releases should be included as well (comma separated) - `Testname`: a human readable short name of the test - `Description`: a detailed description of the test. This field must describe - the required behaviour of the Kubernetes components being tested using + the required behaviour of the Kubernetes components being tested using [RFC2119](https://tools.ietf.org/html/rfc2119) keywords. This field is meant to be a "specification" of the tested Kubernetes features, as such, it must be detailed enough so that readers can fully understand @@ -184,9 +195,9 @@ The corresponding portion of the Kubernetes Conformance Documentfor this test would then look like this: > ## [Kubelet: log output](https://github.com/kubernetes/kubernetes/tree/release-1.9/test/e2e_node/kubelet_test.go#L47) -> +> > Release : v1.9 -> +> > By default the stdout and stderr from the process being executed in a pod MUST be sent to the pod's logs. ### Reporting Conformance Test Results diff --git a/contributors/devel/e2e-tests.md b/contributors/devel/e2e-tests.md index b1bca351..5bcd8a7c 100644 --- a/contributors/devel/e2e-tests.md +++ b/contributors/devel/e2e-tests.md @@ -122,7 +122,7 @@ GINKGO_PARALLEL=y kubetest --test --test_args="--ginkgo.skip=\[Serial\] --delete kubetest --provider=aws --build --up --test --down # -ctl can be used to quickly call kubectl against your e2e cluster. Useful for -# cleaning up after a failed test or viewing logs. +# cleaning up after a failed test or viewing logs. # kubectl output is default on, you can use --verbose-commands=false to suppress output. kubetest -ctl='get events' kubetest -ctl='delete pod foobar' @@ -569,14 +569,14 @@ breaking changes, it does *not* block the merge-queue, and thus should run in some separate test suites owned by the feature owner(s) (see [Continuous Integration](#continuous-integration) below). - - `[Conformance]`: Designate that is test is included in the Conformance + - `[Conformance]`: Designate that this test is included in the Conformance test suite for [Conformance Testing](conformance-tests.md). This test must meet a number of [requirements](conformance-tests.md#conformance-test-requirements) to be eligible for this tag. This tag does not supersed any other labels. - The following tags are not considered to be exhaustively applied, but are intended to further categorize existing `[Conformance]` tests, or tests that are -being considered as candidate for promotion to `[Conformance]` as we work to +being considered as candidate for promotion to `[Conformance]` as we work to refine requirements: - `[Privileged]`: This is a test that requires privileged access - `[Internet]`: This is a test that assumes access to the public internet |
