Merge pull request #3233 from mattjmcnaughton/mattjmcnaughton/fix-broken-testing-commands

Fix and update testing documentation

1 files changed, 50 insertions, 11 deletions

diff --git a/contributors/devel/sig-testing/testing.md b/contributors/devel/sig-testing/testing.md
index 22f91c55..f35c516f 100644
--- a/contributors/devel/sig-testing/testing.md
+++ b/contributors/devel/sig-testing/testing.md
@@ -42,13 +42,23 @@ passing, so it is often a good idea to make sure the e2e tests work as well.
 ### Run all unit tests
 
 `make test` is the entrypoint for running the unit tests that ensures that
-`GOPATH` is set up correctly.  If you have `GOPATH` set up correctly, you can
-also just use `go test` directly.
+`GOPATH` is set up correctly.
 
 ```sh
 cd kubernetes
 make test  # Run all unit tests.
 ```
+If you have `GOPATH` set up correctly, you can
+also just use `go test` directly.
+
+```sh
+cd kubernetes
+go test ./... # Run all unit tests
+```
+
+The remainder of this documentation presumes that you use `Make` as an
+entry point, but remember that the ability to use `go test` exists should you
+desire.
 
 If any unit test fails with a timeout panic (see [#1594](https://github.com/kubernetes/community/issues/1594)) on the testing package, you can increase the `KUBE_TIMEOUT` value as shown below.
 
@@ -67,19 +77,33 @@ You can set [go flags](https://golang.org/cmd/go/) by setting the
 added automatically to these:
 
 ```sh
-make test WHAT=./pkg/api                # run tests for pkg/api
+make test WHAT=./pkg/kubelet                # run tests for pkg/kubelet
+```
+
+Expressed strictly with `go test`, the above command is equivalent to the following:
+
+```sh
+go test ./pkg/kubelet
+```
+
+To run tests for a package and all of its subpackages, you need to append `...`
+to the package path:
+
+```sh
+make test WHAT=./pkg/api/...  # run tests for pkg/api and all its subpackages
 ```
 
 To run multiple targets you need quotes:
 
 ```sh
-make test WHAT="./pkg/api ./pkg/kubelet"  # run tests for pkg/api and pkg/kubelet
+make test WHAT="./pkg/kubelet ./pkg/scheduler"  # run tests for pkg/kubelet and pkg/scheduler
 ```
 
 In a shell, it's often handy to use brace expansion:
 
 ```sh
-make test WHAT=./pkg/{api,kubelet}  # run tests for pkg/api and pkg/kubelet
+make test WHAT=./pkg/{kubelet,scheduler}  # run tests for pkg/kubelet and
+pkg/scheduler
 ```
 
 ### Run specific unit test cases in a package
@@ -90,10 +114,16 @@ regular expression for the name of the test that should be run.
 
 ```sh
 # Runs TestValidatePod in pkg/api/validation with the verbose flag set
-make test WHAT=./pkg/api/validation GOFLAGS="-v" KUBE_TEST_ARGS='-run ^TestValidatePod$'
+make test WHAT=./pkg/apis/core/validation GOFLAGS="-v" KUBE_TEST_ARGS='-run ^TestValidatePod$'
 
 # Runs tests that match the regex ValidatePod|ValidateConfigMap in pkg/api/validation
-make test WHAT=./pkg/api/validation GOFLAGS="-v" KUBE_TEST_ARGS="-run ValidatePod\|ValidateConfigMap$"
+make test WHAT=./pkg/apis/core/validation GOFLAGS="-v" KUBE_TEST_ARGS="-run ValidatePod\|ValidateConfigMap$"
+```
+
+Or if we are using `go test` as our entry point, we could run:
+
+```sh
+go test ./pkg/apis/core/validation -v -run ^TestValidatePods$
 ```
 
 For other supported test flags, see the [golang
@@ -139,14 +169,23 @@ combined for all tests run.
 To run benchmark tests, you'll typically use something like:
 
 ```sh
-go test ./pkg/apiserver -benchmem -run=XXX -bench=BenchmarkWatch
+make test WHAT=./pkg/scheduler/internal/cache KUBE_TEST_ARGS='-benchmem -run=XXX -bench=BenchmarkExpirePods'
+```
+
+Alternatively, to express in pure Go, you could write the following:
+
+```sh
+go test ./pkg/scheduler/internal/cache -benchmem -run=XXX -bench=Benchmark
 ```
 
 This will do the following:
 
-1. `-run=XXX` is a regular expression filter on the name of test cases to run
-2. `-bench=BenchmarkWatch` will run test methods with BenchmarkWatch in the name
-  * See `grep -nr BenchmarkWatch .` for examples
+1. `-run=XXX` is a regular expression filter on the name of test cases to run.
+   Go will execute both the tests matching the `-bench` regex and the `-run`
+   regex. Since we only want to execute benchmark tests, we set the `-run` regex
+   to XXX, which will not match any tests.
+2. `-bench=Benchmark` will run test methods with Benchmark in the name
+  * See `grep -nr Benchmark .` for examples
 3. `-benchmem` enables memory allocation stats
 
 See `go help test` and `go help testflag` for additional info.