summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorAaron Crickenberger <spiffxp@google.com>2018-09-24 13:23:50 -0700
committerAaron Crickenberger <spiffxp@google.com>2018-10-08 15:15:53 -0700
commitc59aea6fb39a56bd204bedee21eb3966c94cb7f1 (patch)
tree1c2d911c431431d3cb04c09ea5e693ec65130760
parent2c134dbb59bb091ddbbff4f4c6122cb1d6a1e547 (diff)
Address review comments
-rw-r--r--contributors/devel/conformance-tests.md59
-rw-r--r--contributors/devel/e2e-tests.md6
2 files changed, 38 insertions, 27 deletions
diff --git a/contributors/devel/conformance-tests.md b/contributors/devel/conformance-tests.md
index 858dc0ca..46ca318d 100644
--- a/contributors/devel/conformance-tests.md
+++ b/contributors/devel/conformance-tests.md
@@ -8,8 +8,6 @@ expected behavior works as a user might encounter it in the wild.
The process to add new conformance tests is intended to decouple the development
of useful tests from their promotion to conformance:
- Contributors write and submit e2e tests, to be approved by owning SIGs
- - Tests should begin by covering the most important and visible aspects of the
- function
- Tests are proven to meet the [conformance test requirements] by review
and by accumulation of data on flakiness and reliability
- A follow up PR is submitted to [promote the test to conformance](#promoting-tests-to-conformance)
@@ -22,7 +20,7 @@ NB: This should be viewed as a living document in a few key areas:
non-optional features or APIs. The list of requirements will be refined over
time to the point where it as concrete and complete as possible.
- There are currently conformance tests that violate some of the requirements
- (ie: require privileged access), we will be categorizing these tests and
+ (e.g., require privileged access), we will be categorizing these tests and
deciding what to do once we have a better understanding of the situation
- Once we resolve the above issues, we plan on identifying the appropriate areas
to relax requirements to allow for the concept of conformance Profiles that
@@ -33,20 +31,22 @@ NB: This should be viewed as a living document in a few key areas:
Conformance tests currently test only GA, non-optional features or APIs. More
specifically, a test is eligible for promotion to conformance if:
-- it tests only GA, non-optional features or APIs (ie: no alpha or beta
+- it tests only GA, non-optional features or APIs (e.g., no alpha or beta
endpoints, no feature flags required, no deprecated features)
-- it works for all providers (ie: no `SkipIfProviderIs`/`SkipUnlessProviderIs`
+- it works for all providers (e.g., no `SkipIfProviderIs`/`SkipUnlessProviderIs`
calls)
-- it is non-privileged (ie: no root on nodes, network, nor cluster)
-- it works without access to the public internet
+- it is non-privileged (e.g., does not require root on nodes, access to raw
+ network interfaces, or cluster admin permissions)
+- it works without access to the public internet (short of whatever is required
+ to pre-pull images for conformance tests)
- it works without non-standard filesystem permissions granted to pods
- it does not rely on any binaries that would not be required for the linux
- kernel or kubelet to run (ie: can't rely on git)
+ kernel or kubelet to run (e.g., can't rely on git)
- any container images used within the test support all architectures for which
kubernetes releases are built
-- it passes against the previous release of kubernetes as well as the current
- version of kubernetes per the [conformance test version skew policy]
-- it is stable and runs consistently (ie: no flakes)
+- it passes against the appropriate versions of kubernetes as spelled out in
+ the [conformance test version skew policy]
+- it is stable and runs consistently (e.g., no flakes)
Examples of features which are not currently eligible for conformance tests:
@@ -63,6 +63,13 @@ Examples of tests which are not eligible for promotion to conformance:
these may change over time (however it is reasonable to verify these fields
exist or are non-empty)
+Examples of areas we may want to relax these requirements once we have a
+sufficient corpus of tests that define out of the box functionality in all
+reasonable production worthy environments:
+- tests may need to create or set objects or fields that are alpha or beta that
+ bypass policies that are not yet GA, but which may reasonably be enabled on a
+ conformant cluster (e.g., pod security policy, non-GA scheduler annotations)
+
## Conformance Test Version Skew Policy
As each new release of Kubernetes provides new functionality, the subset of
@@ -72,16 +79,16 @@ as laid out in the [kubernetes versioning policy]
To quote:
-> For example, a v1.3 master should work with v1.1, v1.2, and v1.3 nodes, and
+> For example, a v1.3 master should work with v1.1, v1.2, and v1.3 nodes, and
> should work with v1.2, v1.3, and v1.4 clients.
Conformance tests for a given version should be run off of the release branch
that corresponds to that version. Thus `v1.2` conformance tests would be run
from the head of the `release-1.2` branch.
-For example, suppose we're in the midst of developing kubernetes v1.3. The
-following clusters must pass conformance tests built from the following
-branches:
+For example, suppose we're in the midst of developing kubernetes v1.3. Clusters
+with the following versions must pass conformance tests built from the
+following branches:
| cluster version | master | release-1.3 | release-1.2 | release-1.1 |
| --------------- | ----- | ----------- | ----------- | ----------- |
@@ -91,10 +98,14 @@ branches:
## Running Conformance Tests
-Conformance tests are designed to be run with no cloud provider configured.
-Conformance tests must be able to be run against clusters that have not been
-created with `hack/e2e.go`, just provide a kubeconfig with the appropriate
-endpoint and credentials.
+Conformance tests are designed to be run even when there is no cloud provider
+configured. Conformance tests must be able to be run against clusters that have
+not been created with `hack/e2e.go`, just provide a kubeconfig with the
+appropriate endpoint and credentials.
+
+These commands are intended to be run within a kubernetes directory, either
+cloned from source, or extracted from release artifacts such as
+`kubernetes.tar.gz`. They assume you have a valid golang installation.
```sh
# ensure kubetest is installed
@@ -127,7 +138,7 @@ having to look through the testcase's code directly.
## Promoting Tests to Conformance
To promote a test to the conformance test suite, open a PR as follows:
-- is titled "Promte xxx e2e test to Conformance"
+- is titled "Promote xxx e2e test to Conformance"
- includes information and metadata in the description as follows:
- "/area conformance" on a newline
- "@kubernetes/sig-architecture-pr-reviews @kubernetes/sig-foo-pr-reviews
@@ -136,7 +147,7 @@ To promote a test to the conformance test suite, open a PR as follows:
- any necessary information in the description to verify that the test meets
[conformance test requirements], such as links to reports or dashboards that
prove lack of flakiness
-- modifies code as follows:
+- contains no other modifications to test source code other than the following:
- modifies the testcase to use the `framework.ConformanceIt()` function rather
than the `framework.It()` function
- adds a comment immediately before the `ConformanceIt()` call that includes
@@ -154,7 +165,7 @@ within its associated comment:
then those releases should be included as well (comma separated)
- `Testname`: a human readable short name of the test
- `Description`: a detailed description of the test. This field must describe
- the required behaviour of the Kubernetes components being tested using
+ the required behaviour of the Kubernetes components being tested using
[RFC2119](https://tools.ietf.org/html/rfc2119) keywords. This field
is meant to be a "specification" of the tested Kubernetes features, as
such, it must be detailed enough so that readers can fully understand
@@ -184,9 +195,9 @@ The corresponding portion of the Kubernetes Conformance Documentfor this test
would then look like this:
> ## [Kubelet: log output](https://github.com/kubernetes/kubernetes/tree/release-1.9/test/e2e_node/kubelet_test.go#L47)
->
+>
> Release : v1.9
->
+>
> By default the stdout and stderr from the process being executed in a pod MUST be sent to the pod's logs.
### Reporting Conformance Test Results
diff --git a/contributors/devel/e2e-tests.md b/contributors/devel/e2e-tests.md
index b1bca351..5bcd8a7c 100644
--- a/contributors/devel/e2e-tests.md
+++ b/contributors/devel/e2e-tests.md
@@ -122,7 +122,7 @@ GINKGO_PARALLEL=y kubetest --test --test_args="--ginkgo.skip=\[Serial\] --delete
kubetest --provider=aws --build --up --test --down
# -ctl can be used to quickly call kubectl against your e2e cluster. Useful for
-# cleaning up after a failed test or viewing logs.
+# cleaning up after a failed test or viewing logs.
# kubectl output is default on, you can use --verbose-commands=false to suppress output.
kubetest -ctl='get events'
kubetest -ctl='delete pod foobar'
@@ -569,14 +569,14 @@ breaking changes, it does *not* block the merge-queue, and thus should run in
some separate test suites owned by the feature owner(s)
(see [Continuous Integration](#continuous-integration) below).
- - `[Conformance]`: Designate that is test is included in the Conformance
+ - `[Conformance]`: Designate that this test is included in the Conformance
test suite for [Conformance Testing](conformance-tests.md). This test must
meet a number of [requirements](conformance-tests.md#conformance-test-requirements)
to be eligible for this tag. This tag does not supersed any other labels.
- The following tags are not considered to be exhaustively applied, but are
intended to further categorize existing `[Conformance]` tests, or tests that are
-being considered as candidate for promotion to `[Conformance]` as we work to
+being considered as candidate for promotion to `[Conformance]` as we work to
refine requirements:
- `[Privileged]`: This is a test that requires privileged access
- `[Internet]`: This is a test that assumes access to the public internet