Replace hack/e2e.go with kubetest, add tags for conformance

1 files changed, 62 insertions, 62 deletions

diff --git a/contributors/devel/e2e-tests.md b/contributors/devel/e2e-tests.md
index c292dba1..b1bca351 100644
--- a/contributors/devel/e2e-tests.md
+++ b/contributors/devel/e2e-tests.md
@@ -7,7 +7,6 @@
   - [Building Kubernetes and Running the Tests](#building-kubernetes-and-running-the-tests)
     - [Cleaning up](#cleaning-up)
   - [Advanced testing](#advanced-testing)
-    - [Installing/updating kubetest](#installingupdating-kubetest)
     - [Extracting a specific version of kubernetes](#extracting-a-specific-version-of-kubernetes)
     - [Bringing up a cluster for testing](#bringing-up-a-cluster-for-testing)
     - [Federation e2e tests](#federation-e2e-tests)
@@ -66,7 +65,12 @@ should also read [Writing Good e2e Tests](writing-good-e2e-tests.md)
 ## Building Kubernetes and Running the Tests
 
 There are a variety of ways to run e2e tests, but we aim to decrease the number
-of ways to run e2e tests to a canonical way: `hack/e2e.go`.
+of ways to run e2e tests to a canonical way: `kubetest`.
+
+You can install `kubetest` as follows:
+```sh
+go get -u k8s.io/test-infra/kubetest
+```
 
 You can run an end-to-end test which will bring up a master and nodes, perform
 some tests, and then tear everything down. Make sure you have followed the
@@ -81,33 +85,33 @@ you can do so via `make WHAT=test/e2e/e2e.test`, and then re-running the ginkgo
 To build Kubernetes, up a cluster, run tests, and tear everything down, use:
 
 ```sh
-go run hack/e2e.go -- --build --up --test --down
+kubetest --build --up --test --down
 ```
 
 If you'd like to just perform one of these steps, here are some examples:
 
 ```sh
 # Build binaries for testing
-go run hack/e2e.go -- --build
+kubetest --build
 
 # Create a fresh cluster.  Deletes a cluster first, if it exists
-go run hack/e2e.go -- --up
+kubetest --up
 
 # Run all tests
-go run hack/e2e.go -- --test
+kubetest --test
 
 # Run tests matching the regex "\[Feature:Performance\]" against a local cluster
 # Specify "--provider=local" flag when running the tests locally
-go run hack/e2e.go -- --test --test_args="--ginkgo.focus=\[Feature:Performance\]" --provider=local
+kubetest --test --test_args="--ginkgo.focus=\[Feature:Performance\]" --provider=local
 
 # Conversely, exclude tests that match the regex "Pods.*env"
-go run hack/e2e.go -- --test --test_args="--ginkgo.skip=Pods.*env"
+kubetest --test --test_args="--ginkgo.skip=Pods.*env"
 
 # Run tests in parallel, skip any that must be run serially
-GINKGO_PARALLEL=y go run hack/e2e.go -- --test --test_args="--ginkgo.skip=\[Serial\]"
+GINKGO_PARALLEL=y kubetest --test --test_args="--ginkgo.skip=\[Serial\]"
 
 # Run tests in parallel, skip any that must be run serially and keep the test namespace if test failed
-GINKGO_PARALLEL=y go run hack/e2e.go -- --test --test_args="--ginkgo.skip=\[Serial\] --delete-namespace-on-failure=false"
+GINKGO_PARALLEL=y kubetest --test --test_args="--ginkgo.skip=\[Serial\] --delete-namespace-on-failure=false"
 
 # Flags can be combined, and their actions will take place in this order:
 # --build, --up, --test, --down
@@ -115,18 +119,18 @@ GINKGO_PARALLEL=y go run hack/e2e.go -- --test --test_args="--ginkgo.skip=\[Seri
 # You can also specify an alternative provider, such as 'aws'
 #
 # e.g.:
-go run hack/e2e.go -- --provider=aws --build --up --test --down
+kubetest --provider=aws --build --up --test --down
 
 # -ctl can be used to quickly call kubectl against your e2e cluster. Useful for
 # cleaning up after a failed test or viewing logs. 
 # kubectl output is default on, you can use --verbose-commands=false to suppress output.
-go run hack/e2e.go -- -ctl='get events'
-go run hack/e2e.go -- -ctl='delete pod foobar'
+kubetest -ctl='get events'
+kubetest -ctl='delete pod foobar'
 ```
 
 The tests are built into a single binary which can be used to deploy a
 Kubernetes system or run tests against an already-deployed Kubernetes system.
-See `go run hack/e2e.go -- --help` (or the flag definitions in `hack/e2e.go`) for
+See `kubetest --help` (or the flag definitions in `hack/e2e.go`) for
 more options, such as reusing an existing cluster.
 
 ### Cleaning up
@@ -136,26 +140,11 @@ something goes wrong and you still have some VMs running you can force a cleanup
 with this command:
 
 ```sh
-go run hack/e2e.go -- --down
+kubetest --down
 ```
 
 ## Advanced testing
 
-### Installing/updating kubetest
-
-The logic in `e2e.go` moved out of the main kubernetes repo to test-infra.
-The remaining code in `hack/e2e.go` installs `kubetest` and sends it flags.
-It now lives in [kubernetes/test-infra/kubetest](https://git.k8s.io/test-infra/kubetest).
-By default `hack/e2e.go` updates and installs `kubetest` once per day.
-Control the updater behavior with the `--get` and `--old` flags:
-The `--` flag separates updater and kubetest flags (kubetest flags on the right).
-
-```sh
-go run hack/e2e.go --get=true --old=1h -- # Update every hour
-go run hack/e2e.go --get=false -- # Never attempt to install/update.
-go install k8s.io/test-infra/kubetest  # Manually install
-go get -u k8s.io/test-infra/kubetest  # Manually update installation
-```
 ### Extracting a specific version of kubernetes
 
 The `kubetest` binary can download and extract a specific version of kubernetes,
@@ -166,28 +155,28 @@ There are a variety of values to pass this flag:
 
 ```sh
 # Official builds: <ci|release>/<latest|stable>[-N.N]
-go run hack/e2e.go -- --extract=ci/latest --up  # Deploy the latest ci build.
-go run hack/e2e.go -- --extract=ci/latest-1.5 --up  # Deploy the latest 1.5 CI build.
-go run hack/e2e.go -- --extract=release/latest --up  # Deploy the latest RC.
-go run hack/e2e.go -- --extract=release/stable-1.5 --up  # Deploy the 1.5 release.
+kubetest --extract=ci/latest --up  # Deploy the latest ci build.
+kubetest --extract=ci/latest-1.5 --up  # Deploy the latest 1.5 CI build.
+kubetest --extract=release/latest --up  # Deploy the latest RC.
+kubetest --extract=release/stable-1.5 --up  # Deploy the 1.5 release.
 
 # A specific version:
-go run hack/e2e.go -- --extract=v1.5.1 --up  # Deploy 1.5.1
-go run hack/e2e.go -- --extract=v1.5.2-beta.0  --up  # Deploy 1.5.2-beta.0
-go run hack/e2e.go -- --extract=gs://foo/bar  --up  # --stage=gs://foo/bar
+kubetest --extract=v1.5.1 --up  # Deploy 1.5.1
+kubetest --extract=v1.5.2-beta.0  --up  # Deploy 1.5.2-beta.0
+kubetest --extract=gs://foo/bar  --up  # --stage=gs://foo/bar
 
 # Whatever GKE is using (gke, gke-staging, gke-test):
-go run hack/e2e.go -- --extract=gke  --up  # Deploy whatever GKE prod uses
+kubetest --extract=gke  --up  # Deploy whatever GKE prod uses
 
 # Using a GCI version:
-go run hack/e2e.go -- --extract=gci/gci-canary --up  # Deploy the version for next gci release
-go run hack/e2e.go -- --extract=gci/gci-57  # Deploy the version bound to gci m57
-go run hack/e2e.go -- --extract=gci/gci-57/ci/latest  # Deploy the latest CI build using gci m57 for the VM image
+kubetest --extract=gci/gci-canary --up  # Deploy the version for next gci release
+kubetest --extract=gci/gci-57  # Deploy the version bound to gci m57
+kubetest --extract=gci/gci-57/ci/latest  # Deploy the latest CI build using gci m57 for the VM image
 
 # Reuse whatever is already built
-go run hack/e2e.go -- --up  # Most common. Note, no extract flag
-go run hack/e2e.go -- --build --up  # Most common. Note, no extract flag
-go run hack/e2e.go -- --build --stage=gs://foo/bar --extract=local --up  # Extract the staged version
+kubetest --up  # Most common. Note, no extract flag
+kubetest --build --up  # Most common. Note, no extract flag
+kubetest --build --stage=gs://foo/bar --extract=local --up  # Extract the staged version
 ```
 
 ### Bringing up a cluster for testing
@@ -327,7 +316,7 @@ Next, specify the docker repository where your ci images will be pushed.
 * Compile the binaries and build container images:
 
   ```sh
-  $ KUBE_RELEASE_RUN_TESTS=n KUBE_FASTBUILD=true go run hack/e2e.go -- -build
+  $ KUBE_RELEASE_RUN_TESTS=n KUBE_FASTBUILD=true kubetest -build
   ```
 
 * Push the federation container images
@@ -342,7 +331,7 @@ The following command will create the underlying Kubernetes clusters in each of
 federation control plane in the cluster occupying the last zone in the `E2E_ZONES` list.
 
 ```sh
-$ go run hack/e2e.go -- --up
+$ kubetest --up
 ```
 
 #### Run the Tests
@@ -350,13 +339,13 @@ $ go run hack/e2e.go -- --up
 This will run only the `Feature:Federation` e2e tests. You can omit the `ginkgo.focus` argument to run the entire e2e suite.
 
 ```sh
-$ go run hack/e2e.go -- --test --test_args="--ginkgo.focus=\[Feature:Federation\]"
+$ kubetest --test --test_args="--ginkgo.focus=\[Feature:Federation\]"
 ```
 
 #### Teardown
 
 ```sh
-$ go run hack/e2e.go -- --down
+$ kubetest --down
 ```
 
 #### Shortcuts for test developers
@@ -417,20 +406,20 @@ In order to run an E2E test against a locally running cluster, first make sure
 to have a local build of the tests:
 
 ```sh
-go run hack/e2e.go -- --build
+kubetest --build
 ```
 
 Then point the tests at a custom host directly:
 
 ```sh
 export KUBECONFIG=/path/to/kubeconfig
-go run hack/e2e.go -- --provider=local --test
+kubetest --provider=local --test
 ```
 
 To control the tests that are run:
 
 ```sh
-go run hack/e2e.go -- --provider=local --test --test_args="--ginkgo.focus=Secrets"
+kubetest --provider=local --test --test_args="--ginkgo.focus=Secrets"
 ```
 
 You will also likely need to specify `minStartupPods` to match the number of
@@ -438,7 +427,7 @@ nodes in your cluster. If you're testing against a cluster set up by
 `local-up-cluster.sh`, you will need to do the following:
 
 ```sh
-go run hack/e2e.go -- --provider=local --test --test_args="--minStartupPods=1 --ginkgo.focus=Secrets"
+kubetest --provider=local --test --test_args="--minStartupPods=1 --ginkgo.focus=Secrets"
 ```
 
 ### Version-skewed and upgrade testing
@@ -470,7 +459,7 @@ export CLUSTER_API_VERSION=${OLD_VERSION}
 
 # Deploy a cluster at the old version; see above for more details
 cd ./kubernetes_old
-go run ./hack/e2e.go -- --up
+kubetest --up
 
 # Upgrade the cluster to the new version
 #
@@ -478,11 +467,11 @@ go run ./hack/e2e.go -- --up
 #
 # You can target Feature:MasterUpgrade or Feature:ClusterUpgrade
 cd ../kubernetes
-go run ./hack/e2e.go -- --provider=gke --test --check-version-skew=false --test_args="--ginkgo.focus=\[Feature:MasterUpgrade\]"
+kubetest --provider=gke --test --check-version-skew=false --test_args="--ginkgo.focus=\[Feature:MasterUpgrade\]"
 
 # Run old tests with new kubectl
 cd ../kubernetes_old
-go run ./hack/e2e.go -- --provider=gke --test --test_args="--kubectl-path=$(pwd)/../kubernetes/cluster/kubectl.sh"
+kubetest --provider=gke --test --test_args="--kubectl-path=$(pwd)/../kubernetes/cluster/kubectl.sh"
 ```
 
 If you are just testing version-skew, you may want to just deploy at one
@@ -494,14 +483,14 @@ upgrade process:
 
 # Deploy a cluster at the new version
 cd ./kubernetes
-go run ./hack/e2e.go -- --up
+kubetest --up
 
 # Run new tests with old kubectl
-go run ./hack/e2e.go -- --test --test_args="--kubectl-path=$(pwd)/../kubernetes_old/cluster/kubectl.sh"
+kubetest --test --test_args="--kubectl-path=$(pwd)/../kubernetes_old/cluster/kubectl.sh"
 
 # Run old tests with new kubectl
 cd ../kubernetes_old
-go run ./hack/e2e.go -- --test --test_args="--kubectl-path=$(pwd)/../kubernetes/cluster/kubectl.sh"
+kubetest --test --test_args="--kubectl-path=$(pwd)/../kubernetes/cluster/kubectl.sh"
 ```
 
 #### Test jobs naming convention
@@ -580,6 +569,21 @@ breaking changes, it does *not* block the merge-queue, and thus should run in
 some separate test suites owned by the feature owner(s)
 (see [Continuous Integration](#continuous-integration) below).
 
+  - `[Conformance]`: Designate that is test is included in the Conformance
+test suite for [Conformance Testing](conformance-tests.md). This test must
+meet a number of [requirements](conformance-tests.md#conformance-test-requirements)
+to be eligible for this tag. This tag does not supersed any other labels.
+
+  - The following tags are not considered to be exhaustively applied, but are
+intended to further categorize existing `[Conformance]` tests, or tests that are
+being considered as candidate for promotion to `[Conformance]` as we work to 
+refine requirements:
+    - `[Privileged]`: This is a test that requires privileged access
+    - `[Internet]`: This is a test that assumes access to the public internet
+    - `[Deprecated]`: This is a test that exercises a deprecated feature
+    - `[Alpha]`: This is a test that exercises an alpha feature
+    - `[Beta]`: This is a test that exercises a beta feature
+
 Every test should be owned by a [SIG](/sig-list.md),
 and have a corresponding `[sig-<name>]` label.
 
@@ -599,10 +603,6 @@ In time, it is our intent to add or autogenerate a sample viper configuration th
 
 ### Conformance tests
 
-Finally, `[Conformance]` tests represent a subset of the e2e-tests we expect to
-pass on **any** Kubernetes cluster. The `[Conformance]` label does not supersede
-any other labels.
-
 For more information on Conformance tests please see the [Conformance Testing](conformance-tests.md)
 
 ## Continuous Integration