From d1d4635519415c960c0fa3ffbf4880edc00e9b1d Mon Sep 17 00:00:00 2001 From: Aaron Crickenberger Date: Thu, 2 Aug 2018 11:02:14 -0700 Subject: Tightening up conformance requirements --- contributors/devel/conformance-tests.md | 106 ++++++++++++++++++++------------ 1 file changed, 68 insertions(+), 38 deletions(-) diff --git a/contributors/devel/conformance-tests.md b/contributors/devel/conformance-tests.md index 578fc0c7..9d685f28 100644 --- a/contributors/devel/conformance-tests.md +++ b/contributors/devel/conformance-tests.md @@ -1,58 +1,83 @@ # Conformance Testing in Kubernetes -The Kubernetes conformance test suite is a set of testcases, currently a -subset of the integration/e2e tests, that the Architecture SIG has approved -to define the core set of interoperable features that all Kubernetes -deployments must support. +The Kubernetes Conformance test suite is a subset of e2e tests that +SIG Architecture has approved to define the core set of interoperable +features that all conformant Kubernetes clusters must support. The +tests verify that the expected behavior works as a user might encounter +it in the wild. -Contributors must write and submit e2e tests first (approved by owning Sigs). -Once the new tests prove to be stable in CI runs, later create a follow up PR -to add the test to conformance. This approach also decouples the development -of useful tests from their promotion to conformance. +The process to add new conformance tests is intended to decouple the +development of useful tests from their promotion to conformance: + +- Contributors write and submit e2e tests, to be approved by owning SIGs +- Tests are proven to meet the conformance test requirements +- A follow up PR is submitted to promote the test to conformance -A conformance test verifies the expected functionality works as a user might encounter it in the wild, and tests should begin by covering the most important and visible aspects of the function. ### Conformance Test Requirements -A test is eligible for promotion to conformance if it meets the following requirements: - -- testing GA feature (not alpha or beta APIs, nor deprecated features) -- must be portable (not dependent on provider-specific capabilities or on the public internet) -- cannot test a feature which obviously cannot be supported on a broad range of platforms -(i.e. testing of multiple disk mounts, GPUs, high density) -- cannot test an optional feature (e.g. not policy enforcement) -- should be non-privileged (neither root on nodes, network, nor cluster) -- cannot rely on any particular non-standard file system permissions granted to containers or users -(i.e. sharing writable host /tmp with a container) -- should be stable and run consistently -- cannot skip providers (there should be no Skip like directives added to the test), -especially in the Nucleus or Application layers as described -[here](https://github.com/kubernetes/community/blob/master/contributors/devel/architectural-roadmap.md). -- cannot test cloud provider specific features (i.e. GCE monitoring, S3 Bucketing, ...) -- should work with default settings for all configuration parameters -(example: the default list of admission plugins should not have to be tweaked for passing conformance). -- cannot rely on any binaries that are not required for the -linux kernel or for a kubelet to run (i.e. git) -- any container images used in the test must support all architectures for which kubernetes releases are built +Conformance tests test only GA, non-optional features or APIs. More +specifically, a test is eligible for promotion to conformance if: + +- it tests only GA, non-optional features or APIs (ie: no alpha or beta endpoints, + no feature flags required, no deprecated features) +- it works for all providers (ie: no `SkipIfProviderIs`/`SkipUnlessProviderIs` calls) +- it is non-privileged (ie: no root on nodes, network, nor cluster) ([citation needed] @timothysc what existing tests violate this requirement) +- it works without access to the public internet +- it works without non-standard filesystem permissions granted to containers or users + (ie: can't assume a writable host /tmp shared with container) +- it does not rely on any binaries that would not be required for the linux kernel or + kubelet to run (ie: can't rely on git) +- any container images used within the test support all architectures for which + kubernetes releases are built +- it is stable and runs consistently (ie: no flakes) + +Examples of features which are not eligible for conformance tests: + +- node/platform-reliant features, eg: multiple disk mounts, GPUs, high density, etc. +- optional features, eg: policy enforcement +- cloud-provider-specific features, eg: GCE monitoring, S3 Bucketing, etc. +- anything that requires a non-default admission plugin + +Examples of tests which are not eligible for promotion to conformance: + +- anything that checks specific Events are generated ([citation needed] + @bgrant0607 we don't make guarantee anything about Events) +- anything that checks optional Condition fields, such as Reason or Message, + as these may change over time ([citation needed] @bgrant0607 in some cases + they could be tested to be non-empty however) + +Our intent is to refine the above list of requirements over time to the point +where it is as concrete and complete as possible. Once we reach this point, we +plan on identifying the appropriate areas to relax these requirements to allow +for the concept of conformance Profiles that cover optional or additional +behaviors. ### Conformance Test Version Skew Policy As each new release of Kubernetes provides new functionality, the subset of tests necessary to demonstrate conformance grows with each release. Conformance is thus considered versioned, with the same backwards compatibility guarantees -as laid out in [our versioning policy](/contributors/design-proposals/release/versioning.md#supported-releases-and-component-skew). -Conformance tests for a given version should be run off of the release branch -that corresponds to that version. Thus `v1.2` conformance tests would be run -from the head of the `release-1.2` branch. eg: +as laid out in [the kubernetes versioning policy] - - A v1.3 development cluster should pass v1.1, v1.2 conformance tests +To quote: - - A v1.2 cluster should pass v1.1, v1.2 conformance tests +> For example, a v1.3 master should work with v1.1, v1.2, and v1.3 nodes, and +> should work with v1.2, v1.3, and v1.4 clients. - - A v1.1 cluster should pass v1.0, v1.1 conformance tests, and fail v1.2 -conformance tests +Conformance tests for a given version should be run off of the release branch +that corresponds to that version. Thus `v1.2` conformance tests would be run +from the head of the `release-1.2` branch. + +For example, suppose we're in the midst of developing kubernetes v1.3. The +following clusters must pass conformance tests built from the following branches: +| cluster version | master | release-1.3 | release-1.2 | release-1.1 | +| --------------- | ----- | ----------- | ----------- | ----------- | +| v1.3.0-alpha | yes | yes | yes | no | +| v1.2.x | no | no | yes | yes | +| v1.1.x | no | no | no | yes | ### Running Conformance Tests @@ -92,7 +117,8 @@ the testcase's code directly. To promote a testcase to the conformance test suite, the following steps must be taken: -- as a prerequisite, the test case is already part of e2e and is not flaky +- the test case must already be a part of e2e, and demonstrated to be not flaky + ([citation needed] @spiffxp how do we demonstrate the test isn't flaky) - the testcase must use the `framework.ConformanceIt()` function rather than the `framework.It()` function - the testcase must include a comment immediately before the @@ -102,6 +128,7 @@ steps must be taken: - tag your PR with "/area conformance" label - send your PR to Sig-Architecture for review by adding "@kubernetes/sig-architecture-pr-reviews" also CC the relevant Sig and Sig-Architecture +- add your PR to SIG Architecture's [Conformance Test Review board] ### Test Metadata @@ -157,3 +184,6 @@ federated [Conformance TestGrid dashboard](https://k8s-testgrid.appspot.com/conf If you wish to contribute conformance test results for your provider, please follow this [on-boarding document](https://docs.google.com/document/d/1lGvP89_DdeNO84I86BVAU4qY3h2VCRll45tGrpyx90A/edit#). + +[the kubernetes versioning policy]: /contributors/design-proposals/release/versioning.md#supported-releases-and-component-skew +[Conformance Test Review board]: https://github.com/kubernetes-sigs/architecture-tracking/projects/1 -- cgit v1.2.3 From d1825f5cf78f2a7024bb7771088088f29e12fd0d Mon Sep 17 00:00:00 2001 From: Aaron Crickenberger Date: Fri, 21 Sep 2018 13:17:35 -0700 Subject: Refinements based on google doc iteration --- contributors/devel/conformance-tests.md | 184 +++++++++++++++++--------------- 1 file changed, 100 insertions(+), 84 deletions(-) diff --git a/contributors/devel/conformance-tests.md b/contributors/devel/conformance-tests.md index 9d685f28..858dc0ca 100644 --- a/contributors/devel/conformance-tests.md +++ b/contributors/devel/conformance-tests.md @@ -1,65 +1,74 @@ # Conformance Testing in Kubernetes -The Kubernetes Conformance test suite is a subset of e2e tests that -SIG Architecture has approved to define the core set of interoperable -features that all conformant Kubernetes clusters must support. The -tests verify that the expected behavior works as a user might encounter -it in the wild. - -The process to add new conformance tests is intended to decouple the -development of useful tests from their promotion to conformance: +The Kubernetes Conformance test suite is a subset of e2e tests that SIG +Architecture has approved to define the core set of interoperable features that +all conformant Kubernetes clusters must support. The tests verify that the +expected behavior works as a user might encounter it in the wild. +The process to add new conformance tests is intended to decouple the development +of useful tests from their promotion to conformance: - Contributors write and submit e2e tests, to be approved by owning SIGs -- Tests are proven to meet the conformance test requirements -- A follow up PR is submitted to promote the test to conformance - -and tests should begin by covering the most important and visible aspects of the function. - -### Conformance Test Requirements - -Conformance tests test only GA, non-optional features or APIs. More + - Tests should begin by covering the most important and visible aspects of the + function +- Tests are proven to meet the [conformance test requirements] by review + and by accumulation of data on flakiness and reliability +- A follow up PR is submitted to [promote the test to conformance](#promoting-tests-to-conformance) + +NB: This should be viewed as a living document in a few key areas: +- The desired set of conformant behaviors is not adequately expressed by the + current set of e2e tests, as such this document is currently intended to + guide us in the addition of new e2e tests than can fill this gap +- This document currently focuses solely on the requirements for GA, + non-optional features or APIs. The list of requirements will be refined over + time to the point where it as concrete and complete as possible. +- There are currently conformance tests that violate some of the requirements + (ie: require privileged access), we will be categorizing these tests and + deciding what to do once we have a better understanding of the situation +- Once we resolve the above issues, we plan on identifying the appropriate areas + to relax requirements to allow for the concept of conformance Profiles that + cover optional or additional behaviors + +## Conformance Test Requirements + +Conformance tests currently test only GA, non-optional features or APIs. More specifically, a test is eligible for promotion to conformance if: -- it tests only GA, non-optional features or APIs (ie: no alpha or beta endpoints, - no feature flags required, no deprecated features) -- it works for all providers (ie: no `SkipIfProviderIs`/`SkipUnlessProviderIs` calls) -- it is non-privileged (ie: no root on nodes, network, nor cluster) ([citation needed] @timothysc what existing tests violate this requirement) +- it tests only GA, non-optional features or APIs (ie: no alpha or beta + endpoints, no feature flags required, no deprecated features) +- it works for all providers (ie: no `SkipIfProviderIs`/`SkipUnlessProviderIs` + calls) +- it is non-privileged (ie: no root on nodes, network, nor cluster) - it works without access to the public internet -- it works without non-standard filesystem permissions granted to containers or users - (ie: can't assume a writable host /tmp shared with container) -- it does not rely on any binaries that would not be required for the linux kernel or - kubelet to run (ie: can't rely on git) +- it works without non-standard filesystem permissions granted to pods +- it does not rely on any binaries that would not be required for the linux + kernel or kubelet to run (ie: can't rely on git) - any container images used within the test support all architectures for which kubernetes releases are built +- it passes against the previous release of kubernetes as well as the current + version of kubernetes per the [conformance test version skew policy] - it is stable and runs consistently (ie: no flakes) -Examples of features which are not eligible for conformance tests: +Examples of features which are not currently eligible for conformance tests: -- node/platform-reliant features, eg: multiple disk mounts, GPUs, high density, etc. +- node/platform-reliant features, eg: multiple disk mounts, GPUs, high density, + etc. - optional features, eg: policy enforcement - cloud-provider-specific features, eg: GCE monitoring, S3 Bucketing, etc. - anything that requires a non-default admission plugin Examples of tests which are not eligible for promotion to conformance: +- anything that checks specific Events are generated, as we make no guarantees + about the contents of events, nor their delivery +- anything that checks optional Condition fields, such as Reason or Message, as + these may change over time (however it is reasonable to verify these fields + exist or are non-empty) -- anything that checks specific Events are generated ([citation needed] - @bgrant0607 we don't make guarantee anything about Events) -- anything that checks optional Condition fields, such as Reason or Message, - as these may change over time ([citation needed] @bgrant0607 in some cases - they could be tested to be non-empty however) - -Our intent is to refine the above list of requirements over time to the point -where it is as concrete and complete as possible. Once we reach this point, we -plan on identifying the appropriate areas to relax these requirements to allow -for the concept of conformance Profiles that cover optional or additional -behaviors. - -### Conformance Test Version Skew Policy +## Conformance Test Version Skew Policy As each new release of Kubernetes provides new functionality, the subset of tests necessary to demonstrate conformance grows with each release. Conformance is thus considered versioned, with the same backwards compatibility guarantees -as laid out in [the kubernetes versioning policy] +as laid out in the [kubernetes versioning policy] To quote: @@ -71,7 +80,8 @@ that corresponds to that version. Thus `v1.2` conformance tests would be run from the head of the `release-1.2` branch. For example, suppose we're in the midst of developing kubernetes v1.3. The -following clusters must pass conformance tests built from the following branches: +following clusters must pass conformance tests built from the following +branches: | cluster version | master | release-1.3 | release-1.2 | release-1.1 | | --------------- | ----- | ----------- | ----------- | ----------- | @@ -79,14 +89,17 @@ following clusters must pass conformance tests built from the following branches | v1.2.x | no | no | yes | yes | | v1.1.x | no | no | no | yes | -### Running Conformance Tests +## Running Conformance Tests Conformance tests are designed to be run with no cloud provider configured. -Conformance tests can be run against clusters that have not been created with -`hack/e2e.go`, just provide a kubeconfig with the appropriate endpoint and -credentials. +Conformance tests must be able to be run against clusters that have not been +created with `hack/e2e.go`, just provide a kubeconfig with the appropriate +endpoint and credentials. ```sh +# ensure kubetest is installed +go get -u k8s.io/test-infra/kubetest + # build test binaries, ginkgo, and kubectl first: make WHAT="test/e2e/e2e.test vendor/github.com/onsi/ginkgo/ginkgo cmd/kubectl" @@ -94,44 +107,44 @@ make WHAT="test/e2e/e2e.test vendor/github.com/onsi/ginkgo/ginkgo cmd/kubectl" export KUBECONFIG=/path/to/kubeconfig export KUBERNETES_CONFORMANCE_TEST=y -# run all conformance tests -go run hack/e2e.go -- --provider=skeleton --test --test_args="--ginkgo.focus=\[Conformance\]" - -# run all parallel-safe conformance tests in parallel -GINKGO_PARALLEL=y go run hack/e2e.go -- --provider=skeleton --test --test_args="--ginkgo.focus=\[Conformance\] --ginkgo.skip=\[Serial\]" +# Option A: run all conformance tests serially +kubetest --provider=skeleton --test --test_args="--ginkgo.focus=\[Conformance\]" -# ... and finish up with remaining tests in serial -go run hack/e2e.go -- --provider=skeleton --test --test_args="--ginkgo.focus=\[Serial\].*\[Conformance\]" +# Option B: run parallel conformance tests first, then serial conformance tests serially +GINKGO_PARALLEL=y kubetest --provider=skeleton --test --test_args="--ginkgo.focus=\[Conformance\] --ginkgo.skip=\[Serial\]" +kubetest --provider=skeleton --test --test_args="--ginkgo.focus=\[Serial\].*\[Conformance\]" ``` -### Kubernetes Conformance Document -For each Kubernetes release, a Conformance Document will be generated -that lists all of the tests that comprise the conformance test suite, along -with the formal specification of each test. For example conformance document for -1.9 can be found [here](https://github.com/cncf/k8s-conformance/blob/master/docs/KubeConformance-1.9.md). -This document will help people understand what features are being tested without having to look through -the testcase's code directly. +## Kubernetes Conformance Document +For each Kubernetes release, a Conformance Document will be generated that lists +all of the tests that comprise the conformance test suite, along with the formal +specification of each test. For an example, see the [v1.9 conformance doc]. +This document will help people understand what features are being tested without +having to look through the testcase's code directly. -## Adding New Tests -To promote a testcase to the conformance test suite, the following -steps must be taken: -- the test case must already be a part of e2e, and demonstrated to be not flaky - ([citation needed] @spiffxp how do we demonstrate the test isn't flaky) -- the testcase must use the `framework.ConformanceIt()` function rather - than the `framework.It()` function -- the testcase must include a comment immediately before the - `framework.ConformanceIt()` call that includes all of the required - metadata about the test (see the [Test Metadata](#test-metadata) section) -- use "Promote xxx e2e test to Conformance" as template of your PR title -- tag your PR with "/area conformance" label -- send your PR to Sig-Architecture for review by adding "@kubernetes/sig-architecture-pr-reviews" -also CC the relevant Sig and Sig-Architecture -- add your PR to SIG Architecture's [Conformance Test Review board] +## Promoting Tests to Conformance +To promote a test to the conformance test suite, open a PR as follows: +- is titled "Promte xxx e2e test to Conformance" +- includes information and metadata in the description as follows: + - "/area conformance" on a newline + - "@kubernetes/sig-architecture-pr-reviews @kubernetes/sig-foo-pr-reviews + @kubernetes/cncf-conformance-wg" on a new line, where sig-foo is whichever + sig owns this test + - any necessary information in the description to verify that the test meets + [conformance test requirements], such as links to reports or dashboards that + prove lack of flakiness +- modifies code as follows: + - modifies the testcase to use the `framework.ConformanceIt()` function rather + than the `framework.It()` function + - adds a comment immediately before the `ConformanceIt()` call that includes + all of the required [conformance test comment metadata] +- add the PR to SIG Architecture's [Conformance Test Review board] -### Test Metadata + +### Conformance Test Comment Metadata Each conformance test must include the following piece of metadata within its associated comment: @@ -167,10 +180,9 @@ framework.ConformanceIt("it should print the output to logs", func() { }) ``` -The corresponding portion of the Kubernetes Conformance Documentfor this test would then look -like this: +The corresponding portion of the Kubernetes Conformance Documentfor this test +would then look like this: -> > ## [Kubelet: log output](https://github.com/kubernetes/kubernetes/tree/release-1.9/test/e2e_node/kubelet_test.go#L47) > > Release : v1.9 @@ -179,11 +191,15 @@ like this: ### Reporting Conformance Test Results -Conformance test results, by provider and releases, can be viewed in the -federated [Conformance TestGrid dashboard](https://k8s-testgrid.appspot.com/conformance-all). -If you wish to contribute conformance test results for your provider, -please follow this [on-boarding document](https://docs.google.com/document/d/1lGvP89_DdeNO84I86BVAU4qY3h2VCRll45tGrpyx90A/edit#). - +Conformance test results, by provider and releases, can be viewed in the +[testgrid conformance dashboard]. If you wish to contribute test results +for your provider, please see the [testgrid conformance README] -[the kubernetes versioning policy]: /contributors/design-proposals/release/versioning.md#supported-releases-and-component-skew +[kubernetes versioning policy]: /contributors/design-proposals/release/versioning.md#supported-releases-and-component-skew [Conformance Test Review board]: https://github.com/kubernetes-sigs/architecture-tracking/projects/1 +[conformance test requirements]: #conformance-test-requirements +[conformance test metadata]: #conformance-test-metadata +[conformance test version skew policy]: #conformance-test-version-skew-policy +[testgrid conformance dashboard]: https://testgrid.k8s.io/conformance-all +[testgrid conformance README]: https://github.com/kubernetes/test-infra/blob/master/testgrid/conformance/README.md +[v1.9 conformance doc]: https://github.com/cncf/k8s-conformance/blob/master/docs/KubeConformance-1.9.md -- cgit v1.2.3 From 2c134dbb59bb091ddbbff4f4c6122cb1d6a1e547 Mon Sep 17 00:00:00 2001 From: Aaron Crickenberger Date: Fri, 21 Sep 2018 14:06:13 -0700 Subject: Replace hack/e2e.go with kubetest, add tags for conformance --- contributors/devel/e2e-tests.md | 124 ++++++++++++++++++++-------------------- 1 file changed, 62 insertions(+), 62 deletions(-) diff --git a/contributors/devel/e2e-tests.md b/contributors/devel/e2e-tests.md index c292dba1..b1bca351 100644 --- a/contributors/devel/e2e-tests.md +++ b/contributors/devel/e2e-tests.md @@ -7,7 +7,6 @@ - [Building Kubernetes and Running the Tests](#building-kubernetes-and-running-the-tests) - [Cleaning up](#cleaning-up) - [Advanced testing](#advanced-testing) - - [Installing/updating kubetest](#installingupdating-kubetest) - [Extracting a specific version of kubernetes](#extracting-a-specific-version-of-kubernetes) - [Bringing up a cluster for testing](#bringing-up-a-cluster-for-testing) - [Federation e2e tests](#federation-e2e-tests) @@ -66,7 +65,12 @@ should also read [Writing Good e2e Tests](writing-good-e2e-tests.md) ## Building Kubernetes and Running the Tests There are a variety of ways to run e2e tests, but we aim to decrease the number -of ways to run e2e tests to a canonical way: `hack/e2e.go`. +of ways to run e2e tests to a canonical way: `kubetest`. + +You can install `kubetest` as follows: +```sh +go get -u k8s.io/test-infra/kubetest +``` You can run an end-to-end test which will bring up a master and nodes, perform some tests, and then tear everything down. Make sure you have followed the @@ -81,33 +85,33 @@ you can do so via `make WHAT=test/e2e/e2e.test`, and then re-running the ginkgo To build Kubernetes, up a cluster, run tests, and tear everything down, use: ```sh -go run hack/e2e.go -- --build --up --test --down +kubetest --build --up --test --down ``` If you'd like to just perform one of these steps, here are some examples: ```sh # Build binaries for testing -go run hack/e2e.go -- --build +kubetest --build # Create a fresh cluster. Deletes a cluster first, if it exists -go run hack/e2e.go -- --up +kubetest --up # Run all tests -go run hack/e2e.go -- --test +kubetest --test # Run tests matching the regex "\[Feature:Performance\]" against a local cluster # Specify "--provider=local" flag when running the tests locally -go run hack/e2e.go -- --test --test_args="--ginkgo.focus=\[Feature:Performance\]" --provider=local +kubetest --test --test_args="--ginkgo.focus=\[Feature:Performance\]" --provider=local # Conversely, exclude tests that match the regex "Pods.*env" -go run hack/e2e.go -- --test --test_args="--ginkgo.skip=Pods.*env" +kubetest --test --test_args="--ginkgo.skip=Pods.*env" # Run tests in parallel, skip any that must be run serially -GINKGO_PARALLEL=y go run hack/e2e.go -- --test --test_args="--ginkgo.skip=\[Serial\]" +GINKGO_PARALLEL=y kubetest --test --test_args="--ginkgo.skip=\[Serial\]" # Run tests in parallel, skip any that must be run serially and keep the test namespace if test failed -GINKGO_PARALLEL=y go run hack/e2e.go -- --test --test_args="--ginkgo.skip=\[Serial\] --delete-namespace-on-failure=false" +GINKGO_PARALLEL=y kubetest --test --test_args="--ginkgo.skip=\[Serial\] --delete-namespace-on-failure=false" # Flags can be combined, and their actions will take place in this order: # --build, --up, --test, --down @@ -115,18 +119,18 @@ GINKGO_PARALLEL=y go run hack/e2e.go -- --test --test_args="--ginkgo.skip=\[Seri # You can also specify an alternative provider, such as 'aws' # # e.g.: -go run hack/e2e.go -- --provider=aws --build --up --test --down +kubetest --provider=aws --build --up --test --down # -ctl can be used to quickly call kubectl against your e2e cluster. Useful for # cleaning up after a failed test or viewing logs. # kubectl output is default on, you can use --verbose-commands=false to suppress output. -go run hack/e2e.go -- -ctl='get events' -go run hack/e2e.go -- -ctl='delete pod foobar' +kubetest -ctl='get events' +kubetest -ctl='delete pod foobar' ``` The tests are built into a single binary which can be used to deploy a Kubernetes system or run tests against an already-deployed Kubernetes system. -See `go run hack/e2e.go -- --help` (or the flag definitions in `hack/e2e.go`) for +See `kubetest --help` (or the flag definitions in `hack/e2e.go`) for more options, such as reusing an existing cluster. ### Cleaning up @@ -136,26 +140,11 @@ something goes wrong and you still have some VMs running you can force a cleanup with this command: ```sh -go run hack/e2e.go -- --down +kubetest --down ``` ## Advanced testing -### Installing/updating kubetest - -The logic in `e2e.go` moved out of the main kubernetes repo to test-infra. -The remaining code in `hack/e2e.go` installs `kubetest` and sends it flags. -It now lives in [kubernetes/test-infra/kubetest](https://git.k8s.io/test-infra/kubetest). -By default `hack/e2e.go` updates and installs `kubetest` once per day. -Control the updater behavior with the `--get` and `--old` flags: -The `--` flag separates updater and kubetest flags (kubetest flags on the right). - -```sh -go run hack/e2e.go --get=true --old=1h -- # Update every hour -go run hack/e2e.go --get=false -- # Never attempt to install/update. -go install k8s.io/test-infra/kubetest # Manually install -go get -u k8s.io/test-infra/kubetest # Manually update installation -``` ### Extracting a specific version of kubernetes The `kubetest` binary can download and extract a specific version of kubernetes, @@ -166,28 +155,28 @@ There are a variety of values to pass this flag: ```sh # Official builds: /[-N.N] -go run hack/e2e.go -- --extract=ci/latest --up # Deploy the latest ci build. -go run hack/e2e.go -- --extract=ci/latest-1.5 --up # Deploy the latest 1.5 CI build. -go run hack/e2e.go -- --extract=release/latest --up # Deploy the latest RC. -go run hack/e2e.go -- --extract=release/stable-1.5 --up # Deploy the 1.5 release. +kubetest --extract=ci/latest --up # Deploy the latest ci build. +kubetest --extract=ci/latest-1.5 --up # Deploy the latest 1.5 CI build. +kubetest --extract=release/latest --up # Deploy the latest RC. +kubetest --extract=release/stable-1.5 --up # Deploy the 1.5 release. # A specific version: -go run hack/e2e.go -- --extract=v1.5.1 --up # Deploy 1.5.1 -go run hack/e2e.go -- --extract=v1.5.2-beta.0 --up # Deploy 1.5.2-beta.0 -go run hack/e2e.go -- --extract=gs://foo/bar --up # --stage=gs://foo/bar +kubetest --extract=v1.5.1 --up # Deploy 1.5.1 +kubetest --extract=v1.5.2-beta.0 --up # Deploy 1.5.2-beta.0 +kubetest --extract=gs://foo/bar --up # --stage=gs://foo/bar # Whatever GKE is using (gke, gke-staging, gke-test): -go run hack/e2e.go -- --extract=gke --up # Deploy whatever GKE prod uses +kubetest --extract=gke --up # Deploy whatever GKE prod uses # Using a GCI version: -go run hack/e2e.go -- --extract=gci/gci-canary --up # Deploy the version for next gci release -go run hack/e2e.go -- --extract=gci/gci-57 # Deploy the version bound to gci m57 -go run hack/e2e.go -- --extract=gci/gci-57/ci/latest # Deploy the latest CI build using gci m57 for the VM image +kubetest --extract=gci/gci-canary --up # Deploy the version for next gci release +kubetest --extract=gci/gci-57 # Deploy the version bound to gci m57 +kubetest --extract=gci/gci-57/ci/latest # Deploy the latest CI build using gci m57 for the VM image # Reuse whatever is already built -go run hack/e2e.go -- --up # Most common. Note, no extract flag -go run hack/e2e.go -- --build --up # Most common. Note, no extract flag -go run hack/e2e.go -- --build --stage=gs://foo/bar --extract=local --up # Extract the staged version +kubetest --up # Most common. Note, no extract flag +kubetest --build --up # Most common. Note, no extract flag +kubetest --build --stage=gs://foo/bar --extract=local --up # Extract the staged version ``` ### Bringing up a cluster for testing @@ -327,7 +316,7 @@ Next, specify the docker repository where your ci images will be pushed. * Compile the binaries and build container images: ```sh - $ KUBE_RELEASE_RUN_TESTS=n KUBE_FASTBUILD=true go run hack/e2e.go -- -build + $ KUBE_RELEASE_RUN_TESTS=n KUBE_FASTBUILD=true kubetest -build ``` * Push the federation container images @@ -342,7 +331,7 @@ The following command will create the underlying Kubernetes clusters in each of federation control plane in the cluster occupying the last zone in the `E2E_ZONES` list. ```sh -$ go run hack/e2e.go -- --up +$ kubetest --up ``` #### Run the Tests @@ -350,13 +339,13 @@ $ go run hack/e2e.go -- --up This will run only the `Feature:Federation` e2e tests. You can omit the `ginkgo.focus` argument to run the entire e2e suite. ```sh -$ go run hack/e2e.go -- --test --test_args="--ginkgo.focus=\[Feature:Federation\]" +$ kubetest --test --test_args="--ginkgo.focus=\[Feature:Federation\]" ``` #### Teardown ```sh -$ go run hack/e2e.go -- --down +$ kubetest --down ``` #### Shortcuts for test developers @@ -417,20 +406,20 @@ In order to run an E2E test against a locally running cluster, first make sure to have a local build of the tests: ```sh -go run hack/e2e.go -- --build +kubetest --build ``` Then point the tests at a custom host directly: ```sh export KUBECONFIG=/path/to/kubeconfig -go run hack/e2e.go -- --provider=local --test +kubetest --provider=local --test ``` To control the tests that are run: ```sh -go run hack/e2e.go -- --provider=local --test --test_args="--ginkgo.focus=Secrets" +kubetest --provider=local --test --test_args="--ginkgo.focus=Secrets" ``` You will also likely need to specify `minStartupPods` to match the number of @@ -438,7 +427,7 @@ nodes in your cluster. If you're testing against a cluster set up by `local-up-cluster.sh`, you will need to do the following: ```sh -go run hack/e2e.go -- --provider=local --test --test_args="--minStartupPods=1 --ginkgo.focus=Secrets" +kubetest --provider=local --test --test_args="--minStartupPods=1 --ginkgo.focus=Secrets" ``` ### Version-skewed and upgrade testing @@ -470,7 +459,7 @@ export CLUSTER_API_VERSION=${OLD_VERSION} # Deploy a cluster at the old version; see above for more details cd ./kubernetes_old -go run ./hack/e2e.go -- --up +kubetest --up # Upgrade the cluster to the new version # @@ -478,11 +467,11 @@ go run ./hack/e2e.go -- --up # # You can target Feature:MasterUpgrade or Feature:ClusterUpgrade cd ../kubernetes -go run ./hack/e2e.go -- --provider=gke --test --check-version-skew=false --test_args="--ginkgo.focus=\[Feature:MasterUpgrade\]" +kubetest --provider=gke --test --check-version-skew=false --test_args="--ginkgo.focus=\[Feature:MasterUpgrade\]" # Run old tests with new kubectl cd ../kubernetes_old -go run ./hack/e2e.go -- --provider=gke --test --test_args="--kubectl-path=$(pwd)/../kubernetes/cluster/kubectl.sh" +kubetest --provider=gke --test --test_args="--kubectl-path=$(pwd)/../kubernetes/cluster/kubectl.sh" ``` If you are just testing version-skew, you may want to just deploy at one @@ -494,14 +483,14 @@ upgrade process: # Deploy a cluster at the new version cd ./kubernetes -go run ./hack/e2e.go -- --up +kubetest --up # Run new tests with old kubectl -go run ./hack/e2e.go -- --test --test_args="--kubectl-path=$(pwd)/../kubernetes_old/cluster/kubectl.sh" +kubetest --test --test_args="--kubectl-path=$(pwd)/../kubernetes_old/cluster/kubectl.sh" # Run old tests with new kubectl cd ../kubernetes_old -go run ./hack/e2e.go -- --test --test_args="--kubectl-path=$(pwd)/../kubernetes/cluster/kubectl.sh" +kubetest --test --test_args="--kubectl-path=$(pwd)/../kubernetes/cluster/kubectl.sh" ``` #### Test jobs naming convention @@ -580,6 +569,21 @@ breaking changes, it does *not* block the merge-queue, and thus should run in some separate test suites owned by the feature owner(s) (see [Continuous Integration](#continuous-integration) below). + - `[Conformance]`: Designate that is test is included in the Conformance +test suite for [Conformance Testing](conformance-tests.md). This test must +meet a number of [requirements](conformance-tests.md#conformance-test-requirements) +to be eligible for this tag. This tag does not supersed any other labels. + + - The following tags are not considered to be exhaustively applied, but are +intended to further categorize existing `[Conformance]` tests, or tests that are +being considered as candidate for promotion to `[Conformance]` as we work to +refine requirements: + - `[Privileged]`: This is a test that requires privileged access + - `[Internet]`: This is a test that assumes access to the public internet + - `[Deprecated]`: This is a test that exercises a deprecated feature + - `[Alpha]`: This is a test that exercises an alpha feature + - `[Beta]`: This is a test that exercises a beta feature + Every test should be owned by a [SIG](/sig-list.md), and have a corresponding `[sig-]` label. @@ -599,10 +603,6 @@ In time, it is our intent to add or autogenerate a sample viper configuration th ### Conformance tests -Finally, `[Conformance]` tests represent a subset of the e2e-tests we expect to -pass on **any** Kubernetes cluster. The `[Conformance]` label does not supersede -any other labels. - For more information on Conformance tests please see the [Conformance Testing](conformance-tests.md) ## Continuous Integration -- cgit v1.2.3 From c59aea6fb39a56bd204bedee21eb3966c94cb7f1 Mon Sep 17 00:00:00 2001 From: Aaron Crickenberger Date: Mon, 24 Sep 2018 13:23:50 -0700 Subject: Address review comments --- contributors/devel/conformance-tests.md | 59 +++++++++++++++++++-------------- contributors/devel/e2e-tests.md | 6 ++-- 2 files changed, 38 insertions(+), 27 deletions(-) diff --git a/contributors/devel/conformance-tests.md b/contributors/devel/conformance-tests.md index 858dc0ca..46ca318d 100644 --- a/contributors/devel/conformance-tests.md +++ b/contributors/devel/conformance-tests.md @@ -8,8 +8,6 @@ expected behavior works as a user might encounter it in the wild. The process to add new conformance tests is intended to decouple the development of useful tests from their promotion to conformance: - Contributors write and submit e2e tests, to be approved by owning SIGs - - Tests should begin by covering the most important and visible aspects of the - function - Tests are proven to meet the [conformance test requirements] by review and by accumulation of data on flakiness and reliability - A follow up PR is submitted to [promote the test to conformance](#promoting-tests-to-conformance) @@ -22,7 +20,7 @@ NB: This should be viewed as a living document in a few key areas: non-optional features or APIs. The list of requirements will be refined over time to the point where it as concrete and complete as possible. - There are currently conformance tests that violate some of the requirements - (ie: require privileged access), we will be categorizing these tests and + (e.g., require privileged access), we will be categorizing these tests and deciding what to do once we have a better understanding of the situation - Once we resolve the above issues, we plan on identifying the appropriate areas to relax requirements to allow for the concept of conformance Profiles that @@ -33,20 +31,22 @@ NB: This should be viewed as a living document in a few key areas: Conformance tests currently test only GA, non-optional features or APIs. More specifically, a test is eligible for promotion to conformance if: -- it tests only GA, non-optional features or APIs (ie: no alpha or beta +- it tests only GA, non-optional features or APIs (e.g., no alpha or beta endpoints, no feature flags required, no deprecated features) -- it works for all providers (ie: no `SkipIfProviderIs`/`SkipUnlessProviderIs` +- it works for all providers (e.g., no `SkipIfProviderIs`/`SkipUnlessProviderIs` calls) -- it is non-privileged (ie: no root on nodes, network, nor cluster) -- it works without access to the public internet +- it is non-privileged (e.g., does not require root on nodes, access to raw + network interfaces, or cluster admin permissions) +- it works without access to the public internet (short of whatever is required + to pre-pull images for conformance tests) - it works without non-standard filesystem permissions granted to pods - it does not rely on any binaries that would not be required for the linux - kernel or kubelet to run (ie: can't rely on git) + kernel or kubelet to run (e.g., can't rely on git) - any container images used within the test support all architectures for which kubernetes releases are built -- it passes against the previous release of kubernetes as well as the current - version of kubernetes per the [conformance test version skew policy] -- it is stable and runs consistently (ie: no flakes) +- it passes against the appropriate versions of kubernetes as spelled out in + the [conformance test version skew policy] +- it is stable and runs consistently (e.g., no flakes) Examples of features which are not currently eligible for conformance tests: @@ -63,6 +63,13 @@ Examples of tests which are not eligible for promotion to conformance: these may change over time (however it is reasonable to verify these fields exist or are non-empty) +Examples of areas we may want to relax these requirements once we have a +sufficient corpus of tests that define out of the box functionality in all +reasonable production worthy environments: +- tests may need to create or set objects or fields that are alpha or beta that + bypass policies that are not yet GA, but which may reasonably be enabled on a + conformant cluster (e.g., pod security policy, non-GA scheduler annotations) + ## Conformance Test Version Skew Policy As each new release of Kubernetes provides new functionality, the subset of @@ -72,16 +79,16 @@ as laid out in the [kubernetes versioning policy] To quote: -> For example, a v1.3 master should work with v1.1, v1.2, and v1.3 nodes, and +> For example, a v1.3 master should work with v1.1, v1.2, and v1.3 nodes, and > should work with v1.2, v1.3, and v1.4 clients. Conformance tests for a given version should be run off of the release branch that corresponds to that version. Thus `v1.2` conformance tests would be run from the head of the `release-1.2` branch. -For example, suppose we're in the midst of developing kubernetes v1.3. The -following clusters must pass conformance tests built from the following -branches: +For example, suppose we're in the midst of developing kubernetes v1.3. Clusters +with the following versions must pass conformance tests built from the +following branches: | cluster version | master | release-1.3 | release-1.2 | release-1.1 | | --------------- | ----- | ----------- | ----------- | ----------- | @@ -91,10 +98,14 @@ branches: ## Running Conformance Tests -Conformance tests are designed to be run with no cloud provider configured. -Conformance tests must be able to be run against clusters that have not been -created with `hack/e2e.go`, just provide a kubeconfig with the appropriate -endpoint and credentials. +Conformance tests are designed to be run even when there is no cloud provider +configured. Conformance tests must be able to be run against clusters that have +not been created with `hack/e2e.go`, just provide a kubeconfig with the +appropriate endpoint and credentials. + +These commands are intended to be run within a kubernetes directory, either +cloned from source, or extracted from release artifacts such as +`kubernetes.tar.gz`. They assume you have a valid golang installation. ```sh # ensure kubetest is installed @@ -127,7 +138,7 @@ having to look through the testcase's code directly. ## Promoting Tests to Conformance To promote a test to the conformance test suite, open a PR as follows: -- is titled "Promte xxx e2e test to Conformance" +- is titled "Promote xxx e2e test to Conformance" - includes information and metadata in the description as follows: - "/area conformance" on a newline - "@kubernetes/sig-architecture-pr-reviews @kubernetes/sig-foo-pr-reviews @@ -136,7 +147,7 @@ To promote a test to the conformance test suite, open a PR as follows: - any necessary information in the description to verify that the test meets [conformance test requirements], such as links to reports or dashboards that prove lack of flakiness -- modifies code as follows: +- contains no other modifications to test source code other than the following: - modifies the testcase to use the `framework.ConformanceIt()` function rather than the `framework.It()` function - adds a comment immediately before the `ConformanceIt()` call that includes @@ -154,7 +165,7 @@ within its associated comment: then those releases should be included as well (comma separated) - `Testname`: a human readable short name of the test - `Description`: a detailed description of the test. This field must describe - the required behaviour of the Kubernetes components being tested using + the required behaviour of the Kubernetes components being tested using [RFC2119](https://tools.ietf.org/html/rfc2119) keywords. This field is meant to be a "specification" of the tested Kubernetes features, as such, it must be detailed enough so that readers can fully understand @@ -184,9 +195,9 @@ The corresponding portion of the Kubernetes Conformance Documentfor this test would then look like this: > ## [Kubelet: log output](https://github.com/kubernetes/kubernetes/tree/release-1.9/test/e2e_node/kubelet_test.go#L47) -> +> > Release : v1.9 -> +> > By default the stdout and stderr from the process being executed in a pod MUST be sent to the pod's logs. ### Reporting Conformance Test Results diff --git a/contributors/devel/e2e-tests.md b/contributors/devel/e2e-tests.md index b1bca351..5bcd8a7c 100644 --- a/contributors/devel/e2e-tests.md +++ b/contributors/devel/e2e-tests.md @@ -122,7 +122,7 @@ GINKGO_PARALLEL=y kubetest --test --test_args="--ginkgo.skip=\[Serial\] --delete kubetest --provider=aws --build --up --test --down # -ctl can be used to quickly call kubectl against your e2e cluster. Useful for -# cleaning up after a failed test or viewing logs. +# cleaning up after a failed test or viewing logs. # kubectl output is default on, you can use --verbose-commands=false to suppress output. kubetest -ctl='get events' kubetest -ctl='delete pod foobar' @@ -569,14 +569,14 @@ breaking changes, it does *not* block the merge-queue, and thus should run in some separate test suites owned by the feature owner(s) (see [Continuous Integration](#continuous-integration) below). - - `[Conformance]`: Designate that is test is included in the Conformance + - `[Conformance]`: Designate that this test is included in the Conformance test suite for [Conformance Testing](conformance-tests.md). This test must meet a number of [requirements](conformance-tests.md#conformance-test-requirements) to be eligible for this tag. This tag does not supersed any other labels. - The following tags are not considered to be exhaustively applied, but are intended to further categorize existing `[Conformance]` tests, or tests that are -being considered as candidate for promotion to `[Conformance]` as we work to +being considered as candidate for promotion to `[Conformance]` as we work to refine requirements: - `[Privileged]`: This is a test that requires privileged access - `[Internet]`: This is a test that assumes access to the public internet -- cgit v1.2.3