Merge pull request #2459 from spiffxp/conformance-guidelines-edits

Refine conformance requirements

2 files changed, 208 insertions, 151 deletions

diff --git a/contributors/devel/conformance-tests.md b/contributors/devel/conformance-tests.md
index 578fc0c7..46ca318d 100644
--- a/contributors/devel/conformance-tests.md
+++ b/contributors/devel/conformance-tests.md
@@ -1,67 +1,116 @@
 # Conformance Testing in Kubernetes
 
-The Kubernetes conformance test suite is a set of testcases, currently a
-subset of the integration/e2e tests, that the Architecture SIG has approved
-to define the core set of interoperable features that all Kubernetes
-deployments must support.
-
-Contributors must write and submit e2e tests first (approved by owning Sigs). 
-Once the new tests prove to be stable in CI runs, later create a follow up PR 
-to add the test to conformance. This approach also decouples the development 
-of useful tests from their promotion to conformance.
-
-A conformance test verifies the expected functionality works as a user might encounter it in the wild, 
-and tests should begin by covering the most important and visible aspects of the function.
-
-### Conformance Test Requirements
-
-A test is eligible for promotion to conformance if it meets the following requirements:
-
-- testing GA feature (not alpha or beta APIs, nor deprecated features)
-- must be portable (not dependent on provider-specific capabilities or on the public internet)
-- cannot test a feature which obviously cannot be supported on a broad range of platforms 
-(i.e. testing of multiple disk mounts, GPUs, high density)
-- cannot test an optional feature (e.g. not policy enforcement)
-- should be non-privileged (neither root on nodes, network, nor cluster)
-- cannot rely on any particular non-standard file system permissions granted to containers or users 
-(i.e. sharing writable host /tmp with a container)
-- should be stable and run consistently
-- cannot skip providers (there should be no Skip like directives added to the test), 
-especially in the Nucleus or Application layers as described 
-[here](https://github.com/kubernetes/community/blob/master/contributors/devel/architectural-roadmap.md).
-- cannot test cloud provider specific features (i.e. GCE monitoring, S3 Bucketing, ...)
-- should work with default settings for all configuration parameters 
-(example: the default list of admission plugins should not have to be tweaked for passing conformance).
-- cannot rely on any binaries that are not required for the
-linux kernel or for a kubelet to run (i.e. git)
-- any container images used in the test must support all architectures for which kubernetes releases are built
-
-### Conformance Test Version Skew Policy
+The Kubernetes Conformance test suite is a subset of e2e tests that SIG
+Architecture has approved to define the core set of interoperable features that
+all conformant Kubernetes clusters must support. The tests verify that the
+expected behavior works as a user might encounter it in the wild.
+
+The process to add new conformance tests is intended to decouple the development
+of useful tests from their promotion to conformance:
+- Contributors write and submit e2e tests, to be approved by owning SIGs
+- Tests are proven to meet the [conformance test requirements] by review
+  and by accumulation of data on flakiness and reliability
+- A follow up PR is submitted to [promote the test to conformance](#promoting-tests-to-conformance)
+
+NB: This should be viewed as a living document in a few key areas:
+- The desired set of conformant behaviors is not adequately expressed by the
+  current set of e2e tests, as such this document is currently intended to
+  guide us in the addition of new e2e tests than can fill this gap
+- This document currently focuses solely on the requirements for GA,
+  non-optional features or APIs. The list of requirements will be refined over
+  time to the point where it as concrete and complete as possible.
+- There are currently conformance tests that violate some of the requirements
+  (e.g., require privileged access), we will be categorizing these tests and
+  deciding what to do once we have a better understanding of the situation
+- Once we resolve the above issues, we plan on identifying the appropriate areas
+  to relax requirements to allow for the concept of conformance Profiles that
+  cover optional or additional behaviors
+
+## Conformance Test Requirements
+
+Conformance tests currently test only GA, non-optional features or APIs. More
+specifically, a test is eligible for promotion to conformance if:
+
+- it tests only GA, non-optional features or APIs (e.g., no alpha or beta
+  endpoints, no feature flags required, no deprecated features)
+- it works for all providers (e.g., no `SkipIfProviderIs`/`SkipUnlessProviderIs`
+  calls)
+- it is non-privileged (e.g., does not require root on nodes, access to raw
+  network interfaces, or cluster admin permissions)
+- it works without access to the public internet (short of whatever is required
+  to pre-pull images for conformance tests)
+- it works without non-standard filesystem permissions granted to pods
+- it does not rely on any binaries that would not be required for the linux
+  kernel or kubelet to run (e.g., can't rely on git)
+- any container images used within the test support all architectures for which
+  kubernetes releases are built
+- it passes against the appropriate versions of kubernetes as spelled out in
+  the [conformance test version skew policy]
+- it is stable and runs consistently (e.g., no flakes)
+
+Examples of features which are not currently eligible for conformance tests:
+
+- node/platform-reliant features, eg: multiple disk mounts, GPUs, high density,
+  etc.
+- optional features, eg: policy enforcement
+- cloud-provider-specific features, eg: GCE monitoring, S3 Bucketing, etc.
+- anything that requires a non-default admission plugin
+
+Examples of tests which are not eligible for promotion to conformance:
+- anything that checks specific Events are generated, as we make no guarantees
+  about the contents of events, nor their delivery
+- anything that checks optional Condition fields, such as Reason or Message, as
+  these may change over time (however it is reasonable to verify these fields
+  exist or are non-empty)
+
+Examples of areas we may want to relax these requirements once we have a
+sufficient corpus of tests that define out of the box functionality in all
+reasonable production worthy environments:
+- tests may need to create or set objects or fields that are alpha or beta that
+  bypass policies that are not yet GA, but which may reasonably be enabled on a
+  conformant cluster (e.g., pod security policy, non-GA scheduler annotations)
+
+## Conformance Test Version Skew Policy
 
 As each new release of Kubernetes provides new functionality, the subset of
 tests necessary to demonstrate conformance grows with each release. Conformance
 is thus considered versioned, with the same backwards compatibility guarantees
-as laid out in [our versioning policy](/contributors/design-proposals/release/versioning.md#supported-releases-and-component-skew).
+as laid out in the [kubernetes versioning policy]
+
+To quote:
+
+> For example, a v1.3 master should work with v1.1, v1.2, and v1.3 nodes, and
+> should work with v1.2, v1.3, and v1.4 clients.
+
 Conformance tests for a given version should be run off of the release branch
 that corresponds to that version. Thus `v1.2` conformance tests would be run
-from the head of the `release-1.2` branch. eg:
+from the head of the `release-1.2` branch.
 
- - A v1.3 development cluster should pass v1.1, v1.2 conformance tests
+For example, suppose we're in the midst of developing kubernetes v1.3. Clusters
+with the following versions must pass conformance tests built from the
+following branches:
 
- - A v1.2 cluster should pass v1.1, v1.2 conformance tests
+| cluster version | master | release-1.3 | release-1.2 | release-1.1 |
+| --------------- | -----  | ----------- | ----------- | ----------- |
+| v1.3.0-alpha    | yes    | yes         | yes         | no          |
+| v1.2.x          | no     | no          | yes         | yes         |
+| v1.1.x          | no     | no          | no          | yes         |
 
- - A v1.1 cluster should pass v1.0, v1.1 conformance tests, and fail v1.2
-conformance tests
+## Running Conformance Tests
 
+Conformance tests are designed to be run even when there is no cloud provider
+configured. Conformance tests must be able to be run against clusters that have
+not been created with `hack/e2e.go`, just provide a kubeconfig with the
+appropriate endpoint and credentials.
 
-### Running Conformance Tests
-
-Conformance tests are designed to be run with no cloud provider configured.
-Conformance tests can be run against clusters that have not been created with
-`hack/e2e.go`, just provide a kubeconfig with the appropriate endpoint and
-credentials.
+These commands are intended to be run within a kubernetes directory, either
+cloned from source, or extracted from release artifacts such as
+`kubernetes.tar.gz`. They assume you have a valid golang installation.
 
 ```sh
+# ensure kubetest is installed
+go get -u k8s.io/test-infra/kubetest
+
 # build test binaries, ginkgo, and kubectl first:
 make WHAT="test/e2e/e2e.test vendor/github.com/onsi/ginkgo/ginkgo cmd/kubectl"
 
@@ -69,42 +118,44 @@ make WHAT="test/e2e/e2e.test vendor/github.com/onsi/ginkgo/ginkgo cmd/kubectl"
 export KUBECONFIG=/path/to/kubeconfig
 export KUBERNETES_CONFORMANCE_TEST=y
 
-# run all conformance tests
-go run hack/e2e.go -- --provider=skeleton --test --test_args="--ginkgo.focus=\[Conformance\]"
-
-# run all parallel-safe conformance tests in parallel
-GINKGO_PARALLEL=y go run hack/e2e.go -- --provider=skeleton --test --test_args="--ginkgo.focus=\[Conformance\] --ginkgo.skip=\[Serial\]"
+# Option A: run all conformance tests serially
+kubetest --provider=skeleton --test --test_args="--ginkgo.focus=\[Conformance\]"
 
-# ... and finish up with remaining tests in serial
-go run hack/e2e.go -- --provider=skeleton --test --test_args="--ginkgo.focus=\[Serial\].*\[Conformance\]"
+# Option B: run parallel conformance tests first, then serial conformance tests serially
+GINKGO_PARALLEL=y kubetest --provider=skeleton --test --test_args="--ginkgo.focus=\[Conformance\] --ginkgo.skip=\[Serial\]"
+kubetest --provider=skeleton --test --test_args="--ginkgo.focus=\[Serial\].*\[Conformance\]"
 ```
 
-### Kubernetes Conformance Document
-For each Kubernetes release, a Conformance Document will be generated
-that lists all of the tests that comprise the conformance test suite, along
-with the formal specification of each test. For example conformance document for 
-1.9 can be found [here](https://github.com/cncf/k8s-conformance/blob/master/docs/KubeConformance-1.9.md).
-This document will help people understand what features are being tested without having to look through
-the testcase's code directly.
+## Kubernetes Conformance Document
 
+For each Kubernetes release, a Conformance Document will be generated that lists
+all of the tests that comprise the conformance test suite, along with the formal
+specification of each test. For an example, see the [v1.9 conformance doc].
+This document will help people understand what features are being tested without
+having to look through the testcase's code directly.
 
-## Adding New Tests
 
-To promote a testcase to the conformance test suite, the following
-steps must be taken:
-- as a prerequisite, the test case is already part of e2e and is not flaky 
-- the testcase must use the `framework.ConformanceIt()` function rather
-  than the `framework.It()` function
-- the testcase must include a comment immediately before the
-  `framework.ConformanceIt()` call that includes all of the required
-  metadata about the test (see the [Test Metadata](#test-metadata) section)
-- use "Promote xxx e2e test to Conformance" as template of your PR title
-- tag your PR with "/area conformance" label
-- send your PR to Sig-Architecture for review by adding "@kubernetes/sig-architecture-pr-reviews" 
-also CC the relevant Sig and Sig-Architecture
+## Promoting Tests to Conformance
 
+To promote a test to the conformance test suite, open a PR as follows:
+- is titled "Promote xxx e2e test to Conformance"
+- includes information and metadata in the description as follows:
+  - "/area conformance" on a newline
+  - "@kubernetes/sig-architecture-pr-reviews @kubernetes/sig-foo-pr-reviews
+    @kubernetes/cncf-conformance-wg" on a new line, where sig-foo is whichever
+    sig owns this test
+  - any necessary information in the description to verify that the test meets
+    [conformance test requirements], such as links to reports or dashboards that
+    prove lack of flakiness
+- contains no other modifications to test source code other than the following:
+  - modifies the testcase to use the `framework.ConformanceIt()` function rather
+    than the `framework.It()` function
+  - adds a comment immediately before the `ConformanceIt()` call that includes
+    all of the required [conformance test comment metadata]
+- add the PR to SIG Architecture's [Conformance Test Review board]
 
-### Test Metadata
+
+### Conformance Test Comment Metadata
 
 Each conformance test must include the following piece of metadata
 within its associated comment:
@@ -114,7 +165,7 @@ within its associated comment:
   then those releases should be included as well (comma separated)
 - `Testname`: a human readable short name of the test
 - `Description`: a detailed description of the test. This field must describe
-  the required behaviour of the Kubernetes components being tested using 
+  the required behaviour of the Kubernetes components being tested using
   [RFC2119](https://tools.ietf.org/html/rfc2119) keywords. This field
   is meant to be a "specification" of the tested Kubernetes features, as
   such, it must be detailed enough so that readers can fully understand
@@ -140,20 +191,26 @@ framework.ConformanceIt("it should print the output to logs", func() {
 })
 ```
 
-The corresponding portion of the Kubernetes Conformance Documentfor this test would then look
-like this:
+The corresponding portion of the Kubernetes Conformance Documentfor this test
+would then look like this:
 
->
 > ## [Kubelet: log output](https://github.com/kubernetes/kubernetes/tree/release-1.9/test/e2e_node/kubelet_test.go#L47)
-> 
+>
 > Release : v1.9
-> 
+>
 > By default the stdout and stderr from the process being executed in a pod MUST be sent to the pod's logs.
 
 ### Reporting Conformance Test Results
 
-Conformance test results, by provider and releases, can be viewed in the 
-federated [Conformance TestGrid dashboard](https://k8s-testgrid.appspot.com/conformance-all). 
-If you wish to contribute conformance test results for your provider, 
-please follow this [on-boarding document](https://docs.google.com/document/d/1lGvP89_DdeNO84I86BVAU4qY3h2VCRll45tGrpyx90A/edit#).
-
+Conformance test results, by provider and releases, can be viewed in the
+[testgrid conformance dashboard]. If you wish to contribute test results
+for your provider, please see the [testgrid conformance README]
+
+[kubernetes versioning policy]: /contributors/design-proposals/release/versioning.md#supported-releases-and-component-skew
+[Conformance Test Review board]: https://github.com/kubernetes-sigs/architecture-tracking/projects/1
+[conformance test requirements]: #conformance-test-requirements
+[conformance test metadata]: #conformance-test-metadata
+[conformance test version skew policy]: #conformance-test-version-skew-policy
+[testgrid conformance dashboard]: https://testgrid.k8s.io/conformance-all
+[testgrid conformance README]: https://github.com/kubernetes/test-infra/blob/master/testgrid/conformance/README.md
+[v1.9 conformance doc]: https://github.com/cncf/k8s-conformance/blob/master/docs/KubeConformance-1.9.md
diff --git a/contributors/devel/e2e-tests.md b/contributors/devel/e2e-tests.md
index 190cd0e7..861c6001 100644
--- a/contributors/devel/e2e-tests.md
+++ b/contributors/devel/e2e-tests.md
@@ -7,7 +7,6 @@
   - [Building Kubernetes and Running the Tests](#building-kubernetes-and-running-the-tests)
     - [Cleaning up](#cleaning-up)
   - [Advanced testing](#advanced-testing)
-    - [Installing/updating kubetest](#installingupdating-kubetest)
     - [Extracting a specific version of kubernetes](#extracting-a-specific-version-of-kubernetes)
     - [Bringing up a cluster for testing](#bringing-up-a-cluster-for-testing)
     - [Federation e2e tests](#federation-e2e-tests)
@@ -66,7 +65,12 @@ should also read [Writing Good e2e Tests](writing-good-e2e-tests.md)
 ## Building Kubernetes and Running the Tests
 
 There are a variety of ways to run e2e tests, but we aim to decrease the number
-of ways to run e2e tests to a canonical way: `hack/e2e.go`.
+of ways to run e2e tests to a canonical way: `kubetest`.
+
+You can install `kubetest` as follows:
+```sh
+go get -u k8s.io/test-infra/kubetest
+```
 
 You can run an end-to-end test which will bring up a master and nodes, perform
 some tests, and then tear everything down. Make sure you have followed the
@@ -81,33 +85,33 @@ you can do so via `make WHAT=test/e2e/e2e.test`, and then re-running the ginkgo
 To build Kubernetes, up a cluster, run tests, and tear everything down, use:
 
 ```sh
-go run hack/e2e.go -- --build --up --test --down
+kubetest --build --up --test --down
 ```
 
 If you'd like to just perform one of these steps, here are some examples:
 
 ```sh
 # Build binaries for testing
-go run hack/e2e.go -- --build
+kubetest --build
 
 # Create a fresh cluster.  Deletes a cluster first, if it exists
-go run hack/e2e.go -- --up
+kubetest --up
 
 # Run all tests
-go run hack/e2e.go -- --test
+kubetest --test
 
 # Run tests matching the regex "\[Feature:Performance\]" against a local cluster
 # Specify "--provider=local" flag when running the tests locally
-go run hack/e2e.go -- --test --test_args="--ginkgo.focus=\[Feature:Performance\]" --provider=local
+kubetest --test --test_args="--ginkgo.focus=\[Feature:Performance\]" --provider=local
 
 # Conversely, exclude tests that match the regex "Pods.*env"
-go run hack/e2e.go -- --test --test_args="--ginkgo.skip=Pods.*env"
+kubetest --test --test_args="--ginkgo.skip=Pods.*env"
 
 # Run tests in parallel, skip any that must be run serially
-GINKGO_PARALLEL=y go run hack/e2e.go -- --test --test_args="--ginkgo.skip=\[Serial\]"
+GINKGO_PARALLEL=y kubetest --test --test_args="--ginkgo.skip=\[Serial\]"
 
 # Run tests in parallel, skip any that must be run serially and keep the test namespace if test failed
-GINKGO_PARALLEL=y go run hack/e2e.go -- --test --test_args="--ginkgo.skip=\[Serial\] --delete-namespace-on-failure=false"
+GINKGO_PARALLEL=y kubetest --test --test_args="--ginkgo.skip=\[Serial\] --delete-namespace-on-failure=false"
 
 # Flags can be combined, and their actions will take place in this order:
 # --build, --up, --test, --down
@@ -115,18 +119,18 @@ GINKGO_PARALLEL=y go run hack/e2e.go -- --test --test_args="--ginkgo.skip=\[Seri
 # You can also specify an alternative provider, such as 'aws'
 #
 # e.g.:
-go run hack/e2e.go -- --provider=aws --build --up --test --down
+kubetest --provider=aws --build --up --test --down
 
 # -ctl can be used to quickly call kubectl against your e2e cluster. Useful for
-# cleaning up after a failed test or viewing logs. 
+# cleaning up after a failed test or viewing logs.
 # kubectl output is default on, you can use --verbose-commands=false to suppress output.
-go run hack/e2e.go -- -ctl='get events'
-go run hack/e2e.go -- -ctl='delete pod foobar'
+kubetest -ctl='get events'
+kubetest -ctl='delete pod foobar'
 ```
 
 The tests are built into a single binary which can be used to deploy a
 Kubernetes system or run tests against an already-deployed Kubernetes system.
-See `go run hack/e2e.go -- --help` (or the flag definitions in `hack/e2e.go`) for
+See `kubetest --help` (or the flag definitions in `hack/e2e.go`) for
 more options, such as reusing an existing cluster.
 
 ### Cleaning up
@@ -136,26 +140,11 @@ something goes wrong and you still have some VMs running you can force a cleanup
 with this command:
 
 ```sh
-go run hack/e2e.go -- --down
+kubetest --down
 ```
 
 ## Advanced testing
 
-### Installing/updating kubetest
-
-The logic in `e2e.go` moved out of the main kubernetes repo to test-infra.
-The remaining code in `hack/e2e.go` installs `kubetest` and sends it flags.
-It now lives in [kubernetes/test-infra/kubetest](https://git.k8s.io/test-infra/kubetest).
-By default `hack/e2e.go` updates and installs `kubetest` once per day.
-Control the updater behavior with the `--get` and `--old` flags:
-The `--` flag separates updater and kubetest flags (kubetest flags on the right).
-
-```sh
-go run hack/e2e.go --get=true --old=1h -- # Update every hour
-go run hack/e2e.go --get=false -- # Never attempt to install/update.
-go install k8s.io/test-infra/kubetest  # Manually install
-go get -u k8s.io/test-infra/kubetest  # Manually update installation
-```
 ### Extracting a specific version of kubernetes
 
 The `kubetest` binary can download and extract a specific version of kubernetes,
@@ -166,28 +155,28 @@ There are a variety of values to pass this flag:
 
 ```sh
 # Official builds: <ci|release>/<latest|stable>[-N.N]
-go run hack/e2e.go -- --extract=ci/latest --up  # Deploy the latest ci build.
-go run hack/e2e.go -- --extract=ci/latest-1.5 --up  # Deploy the latest 1.5 CI build.
-go run hack/e2e.go -- --extract=release/latest --up  # Deploy the latest RC.
-go run hack/e2e.go -- --extract=release/stable-1.5 --up  # Deploy the 1.5 release.
+kubetest --extract=ci/latest --up  # Deploy the latest ci build.
+kubetest --extract=ci/latest-1.5 --up  # Deploy the latest 1.5 CI build.
+kubetest --extract=release/latest --up  # Deploy the latest RC.
+kubetest --extract=release/stable-1.5 --up  # Deploy the 1.5 release.
 
 # A specific version:
-go run hack/e2e.go -- --extract=v1.5.1 --up  # Deploy 1.5.1
-go run hack/e2e.go -- --extract=v1.5.2-beta.0  --up  # Deploy 1.5.2-beta.0
-go run hack/e2e.go -- --extract=gs://foo/bar  --up  # --stage=gs://foo/bar
+kubetest --extract=v1.5.1 --up  # Deploy 1.5.1
+kubetest --extract=v1.5.2-beta.0  --up  # Deploy 1.5.2-beta.0
+kubetest --extract=gs://foo/bar  --up  # --stage=gs://foo/bar
 
 # Whatever GKE is using (gke, gke-staging, gke-test):
-go run hack/e2e.go -- --extract=gke  --up  # Deploy whatever GKE prod uses
+kubetest --extract=gke  --up  # Deploy whatever GKE prod uses
 
 # Using a GCI version:
-go run hack/e2e.go -- --extract=gci/gci-canary --up  # Deploy the version for next gci release
-go run hack/e2e.go -- --extract=gci/gci-57  # Deploy the version bound to gci m57
-go run hack/e2e.go -- --extract=gci/gci-57/ci/latest  # Deploy the latest CI build using gci m57 for the VM image
+kubetest --extract=gci/gci-canary --up  # Deploy the version for next gci release
+kubetest --extract=gci/gci-57  # Deploy the version bound to gci m57
+kubetest --extract=gci/gci-57/ci/latest  # Deploy the latest CI build using gci m57 for the VM image
 
 # Reuse whatever is already built
-go run hack/e2e.go -- --up  # Most common. Note, no extract flag
-go run hack/e2e.go -- --build --up  # Most common. Note, no extract flag
-go run hack/e2e.go -- --build --stage=gs://foo/bar --extract=local --up  # Extract the staged version
+kubetest --up  # Most common. Note, no extract flag
+kubetest --build --up  # Most common. Note, no extract flag
+kubetest --build --stage=gs://foo/bar --extract=local --up  # Extract the staged version
 ```
 
 ### Bringing up a cluster for testing
@@ -327,7 +316,7 @@ Next, specify the docker repository where your ci images will be pushed.
 * Compile the binaries and build container images:
 
   ```sh
-  $ KUBE_RELEASE_RUN_TESTS=n KUBE_FASTBUILD=true go run hack/e2e.go -- -build
+  $ KUBE_RELEASE_RUN_TESTS=n KUBE_FASTBUILD=true kubetest -build
   ```
 
 * Push the federation container images
@@ -342,7 +331,7 @@ The following command will create the underlying Kubernetes clusters in each of
 federation control plane in the cluster occupying the last zone in the `E2E_ZONES` list.
 
 ```sh
-$ go run hack/e2e.go -- --up
+$ kubetest --up
 ```
 
 #### Run the Tests
@@ -350,13 +339,13 @@ $ go run hack/e2e.go -- --up
 This will run only the `Feature:Federation` e2e tests. You can omit the `ginkgo.focus` argument to run the entire e2e suite.
 
 ```sh
-$ go run hack/e2e.go -- --test --test_args="--ginkgo.focus=\[Feature:Federation\]"
+$ kubetest --test --test_args="--ginkgo.focus=\[Feature:Federation\]"
 ```
 
 #### Teardown
 
 ```sh
-$ go run hack/e2e.go -- --down
+$ kubetest --down
 ```
 
 #### Shortcuts for test developers
@@ -417,20 +406,20 @@ In order to run an E2E test against a locally running cluster, first make sure
 to have a local build of the tests:
 
 ```sh
-go run hack/e2e.go -- --build
+kubetest --build
 ```
 
 Then point the tests at a custom host directly:
 
 ```sh
 export KUBECONFIG=/path/to/kubeconfig
-go run hack/e2e.go -- --provider=local --test
+kubetest --provider=local --test
 ```
 
 To control the tests that are run:
 
 ```sh
-go run hack/e2e.go -- --provider=local --test --test_args="--ginkgo.focus=Secrets"
+kubetest --provider=local --test --test_args="--ginkgo.focus=Secrets"
 ```
 
 You will also likely need to specify `minStartupPods` to match the number of
@@ -438,7 +427,7 @@ nodes in your cluster. If you're testing against a cluster set up by
 `local-up-cluster.sh`, you will need to do the following:
 
 ```sh
-go run hack/e2e.go -- --provider=local --test --test_args="--minStartupPods=1 --ginkgo.focus=Secrets"
+kubetest --provider=local --test --test_args="--minStartupPods=1 --ginkgo.focus=Secrets"
 ```
 
 ### Version-skewed and upgrade testing
@@ -470,7 +459,7 @@ export CLUSTER_API_VERSION=${OLD_VERSION}
 
 # Deploy a cluster at the old version; see above for more details
 cd ./kubernetes_old
-go run ./hack/e2e.go -- --up
+kubetest --up
 
 # Upgrade the cluster to the new version
 #
@@ -478,11 +467,11 @@ go run ./hack/e2e.go -- --up
 #
 # You can target Feature:MasterUpgrade or Feature:ClusterUpgrade
 cd ../kubernetes
-go run ./hack/e2e.go -- --provider=gke --test --check-version-skew=false --test_args="--ginkgo.focus=\[Feature:MasterUpgrade\]"
+kubetest --provider=gke --test --check-version-skew=false --test_args="--ginkgo.focus=\[Feature:MasterUpgrade\]"
 
 # Run old tests with new kubectl
 cd ../kubernetes_old
-go run ./hack/e2e.go -- --provider=gke --test --test_args="--kubectl-path=$(pwd)/../kubernetes/cluster/kubectl.sh"
+kubetest --provider=gke --test --test_args="--kubectl-path=$(pwd)/../kubernetes/cluster/kubectl.sh"
 ```
 
 If you are just testing version-skew, you may want to just deploy at one
@@ -494,14 +483,14 @@ upgrade process:
 
 # Deploy a cluster at the new version
 cd ./kubernetes
-go run ./hack/e2e.go -- --up
+kubetest --up
 
 # Run new tests with old kubectl
-go run ./hack/e2e.go -- --test --test_args="--kubectl-path=$(pwd)/../kubernetes_old/cluster/kubectl.sh"
+kubetest --test --test_args="--kubectl-path=$(pwd)/../kubernetes_old/cluster/kubectl.sh"
 
 # Run old tests with new kubectl
 cd ../kubernetes_old
-go run ./hack/e2e.go -- --test --test_args="--kubectl-path=$(pwd)/../kubernetes/cluster/kubectl.sh"
+kubetest --test --test_args="--kubectl-path=$(pwd)/../kubernetes/cluster/kubectl.sh"
 ```
 
 #### Test jobs naming convention
@@ -580,6 +569,21 @@ breaking changes, it does *not* block PR merges, and thus should run in
 some separate test suites owned by the feature owner(s)
 (see [Continuous Integration](#continuous-integration) below).
 
+  - `[Conformance]`: Designate that this test is included in the Conformance
+test suite for [Conformance Testing](conformance-tests.md). This test must
+meet a number of [requirements](conformance-tests.md#conformance-test-requirements)
+to be eligible for this tag. This tag does not supersed any other labels.
+
+  - The following tags are not considered to be exhaustively applied, but are
+intended to further categorize existing `[Conformance]` tests, or tests that are
+being considered as candidate for promotion to `[Conformance]` as we work to
+refine requirements:
+    - `[Privileged]`: This is a test that requires privileged access
+    - `[Internet]`: This is a test that assumes access to the public internet
+    - `[Deprecated]`: This is a test that exercises a deprecated feature
+    - `[Alpha]`: This is a test that exercises an alpha feature
+    - `[Beta]`: This is a test that exercises a beta feature
+
 Every test should be owned by a [SIG](/sig-list.md),
 and have a corresponding `[sig-<name>]` label.
 
@@ -599,10 +603,6 @@ In time, it is our intent to add or autogenerate a sample viper configuration th
 
 ### Conformance tests
 
-Finally, `[Conformance]` tests represent a subset of the e2e-tests we expect to
-pass on **any** Kubernetes cluster. The `[Conformance]` label does not supersede
-any other labels.
-
 For more information on Conformance tests please see the [Conformance Testing](conformance-tests.md)
 
 ## Continuous Integration