Merge pull request #1012 from thockin/master

Automatic merge from submit-queue.

Rewrite docs on godep

This corresponds to https://github.com/kubernetes/kubernetes/pull/51766 - do not merge until we get confirmation on that PR.

2 files changed, 136 insertions, 97 deletions

diff --git a/contributors/devel/development.md b/contributors/devel/development.md
index 841cbe71..6bf66ef1 100644
--- a/contributors/devel/development.md
+++ b/contributors/devel/development.md
@@ -152,24 +152,11 @@ images.
 
 #### Dependency management
 
-Kubernetes build/test scripts use [`godep`](https://github.com/tools/godep) to
-manage dependencies.
-
-```sh
-go get -u github.com/tools/godep
-```
-
-The Godep version that Kubernetes is using is listed in `Godep/Godep.json` (in
-the kubernetes repo root). See what version you are running with this command:
-
-```sh
-godep version
-```
-
-Developers planning to manage dependencies in the `vendor/` tree may want to
-explore alternative environment setups. See [using godep to manage
-dependencies](godep.md).
+Kubernetes uses [`godep`](https://github.com/tools/godep) to manage
+dependencies.
 
+Developers who need to manage dependencies in the `vendor/` tree should read
+the docs on [using godep to manage dependencies](godep.md).
 
 
 ## Build with Bazel/Gazel
@@ -178,7 +165,6 @@ Building with Bazel is currently experimental.  For more information,
 see [Build with Bazel].
 
 
-
 ## Workflow
 
 ![Git workflow](git_workflow.png)
diff --git a/contributors/devel/godep.md b/contributors/devel/godep.md
index 3b9ca7fd..cc741f66 100644
--- a/contributors/devel/godep.md
+++ b/contributors/devel/godep.md
@@ -1,128 +1,181 @@
 # Using godep to manage dependencies
 
 This document is intended to show a way for managing `vendor/` tree dependencies
-in Kubernetes. If you are not planning on managing `vendor` dependencies go here
-[Godep dependency management](development.md#godep-dependency-management).
+in Kubernetes. If you do not need to manage vendored dependencies, you probably
+do not need to read this.
 
-## Alternate GOPATH for installing and using godep
+## Background
 
-There are many ways to build and host Go binaries. Here is one way to get
-utilities like `godep` installed:
+As a tool, `godep` leaves much to be desired.  It builds on `go get`, and adds
+the ability to pin dependencies to exact git version. The `go get` tool itself
+doesn't have any concept of versions, and tends to blow up if it finds a git
+repo synced to anything but `master`, but that is exactly the state that
+`godep` leaves repos.  This is a recipe for frustration when people try to use
+the tools.
 
-Create a new GOPATH just for your go tools and install godep:
+This doc will focus on predictability and reproducibility.
+
+## Theory of operation
+
+The `go` toolchain assumes a global workspace that hosts all of your Go code.
+
+The `godep` tool operates by first "restoring" dependencies into your `$GOPATH`.
+This reads the `Godeps.json` file, downloads all of the dependencies from the
+internet, and syncs them to the specified revisions.  You can then make
+changes - sync to different revisions or edit Kubernetes code to use new
+dependencies (and satisfy them with `go get`).  When ready, you tell `godep` to
+"save" everything, which it does by walking the Kubernetes code, finding all
+required dependencies, copying them from `$GOPATH` into the `vendor/` directory,
+and rewriting `Godeps.json`.
+
+This does not work well, when combined with a global Go workspace.  Instead, we
+will set up a private workspace for this process.
+
+The Kubernetes build process uses this same technique, and offers a tool called
+`run-in-gopath.sh` which sets up and switches to a local, private workspace,
+including setting up `$GOPATH` and `$PATH`.  If you wrap commands with this
+tool, they will use the private workspace, which will not conflict with other
+projects and is easily cleaned up and recreated.
+
+To see this in action, you can run an interactive shell in this environment:
 
 ```sh
-export GOPATH=$HOME/go-tools
-mkdir -p $GOPATH
-go get -u github.com/tools/godep
+# Run a shell, but don't run your own shell initializations.
+hack/run-in-gopath.sh bash --norc --noprofile
 ```
 
-Add this $GOPATH/bin to your path. Typically you'd add this to your ~/.profile:
+## Restoring deps
+
+To extract and download dependencies into `$GOPATH` we provide a script:
+`hack/godep-restore.sh`.  If you run this tool, it will restore into your own
+`$GOPATH`.  If you wrap it in `run-in-gopath.sh` it will restore into your
+`_output/` directory.
 
 ```sh
-export GOPATH=$HOME/go-tools
-export PATH=$PATH:$GOPATH/bin
+hack/run-in-gopath.sh hack/godep-restore.sh
 ```
 
-## Using godep
+This script will try to optimize what it needs to download, and if it seems the
+dependencies are all present already, it will return very quickly.
+
+If there's every any doubt about the correctness of your dependencies, you can
+simply `make clean` or `rm -rf _output`, and run it again.
+
+Now you should have a clean copy of all of the Kubernetes dependencies.
+
+## Making changes
+
+The most common things people need to do with deps are add and update them.
+These are similar but different.
 
-Here's a quick walkthrough of one way to use godeps to add or update a
-Kubernetes dependency into `vendor/`. For more details, please see the
-instructions in [godep's documentation](https://github.com/tools/godep).
+### Adding a dep
 
-1) Devote a directory to this endeavor:
+For the sake of examples, consider that we have discovered a wonderful Go
+library at `example.com/go/frob`.  The first thing you need to do is get that
+code into your workspace:
 
-_Devoting a separate directory is not strictly required, but it is helpful to
-separate dependency updates from other changes._
+```sh
+hack/run-in-gopath.sh go get -d example.com/go/frob
+```
+
+This will fetch, but not compile (omit the `-d` if you want to compile it now),
+the library into your private `$GOPATH`.  It will pull whatever the default
+revision of that library is, typically the `master` branch for git repositories.
+If this is not the revision you need, you can change it, for example to
+`v1.0.0`:
 
 ```sh
-export KPATH=$HOME/code/kubernetes
-mkdir -p $KPATH/src/k8s.io
-cd $KPATH/src/k8s.io
-git clone https://github.com/$YOUR_GITHUB_USERNAME/kubernetes.git # assumes your fork is 'kubernetes'
-# Or copy your existing local repo here. IMPORTANT: making a symlink doesn't work.
+hack/run-in-gopath.sh bash -c 'git -C $GOPATH/src/example.com/go/frob checkout v1.0.0'
 ```
 
-2) Set up your GOPATH.
+Now that the code is present, you can start to use it in Kubernetes code.
+Because it is in your private workspace's `$GOPATH`, it might not be part of
+your own `$GOPATH`, so tools like `goimports` might not find it.  This is an
+unfortunate side-effect of this process.  You can either add the whole private
+workspace to your own `$GOPATH` or you can `go get` the library into your own
+`$GOPATH` until it is properly vendored into Kubernetes.
+
+Another possible complication is a dep that uses `gopdep` itself.  In that case,
+you need to restore its dependencies, too:
 
 ```sh
-# This will *not* let your local builds see packages that exist elsewhere on your system.
-export GOPATH=$KPATH
+hack/run-in-gopath.sh bash -c 'cd $GOPATH/src/example.com/go/frob && godep restore'
 ```
 
-3) Populate your new GOPATH.
+If the transitive deps collide with Kubernetes deps, you may have to manually
+resolve things.  This is where the ability to run a shell in this environment
+comes in handy:
 
 ```sh
-cd $KPATH/src/k8s.io/kubernetes
-./hack/godep-restore.sh
+hack/run-in-gopath.sh bash --norc --noprofile
 ```
 
-4) Next, you can either add a new dependency or update an existing one.
+### Updating a dep
 
-To add a new dependency is simple (if a bit slow):
+Sometimes we already have a dep, but the version of it is wrong.  Because of the
+way that `godep` and `go get` interact (badly) it's generally easiest to hit it
+with a big hammer:
 
 ```sh
-cd $KPATH/src/k8s.io/kubernetes
-DEP=example.com/path/to/dependency
-godep get $DEP/...
-# Now change code in Kubernetes to use the dependency.
-./hack/godep-save.sh
+hack/run-in-gopath.sh bash -c 'rm -rf $GOPATH/src/example.com/go/frob'
+hack/run-in-gopath.sh go get -d example.com/go/frob
+hack/run-in-gopath.sh bash -c 'git -C $GOPATH/src/example.com/go/frob checkout v2.0.0'
 ```
 
-To update an existing dependency is a bit more complicated.  Godep has an
-`update` command, but none of us can figure out how to actually make it work.
-Instead, this procedure seems to work reliably:
+This will remove the code, re-fetch it, and sync to your desired version.
+
+### Removing a dep
+
+This happens almost for free.  If you edit Kubernetes code and remove the last
+use of a given dependency, you only need to restore and save the deps, and the
+`godep` tool will figure out that you don't need that dep any more:
+
+## Saving deps
+
+Now that you have made your changes - adding, updating, or removing the use of a
+dep - you need to rebuild the dependency database and make changes to the
+`vendor/` directory.
 
 ```sh
-cd $KPATH/src/k8s.io/kubernetes
-DEP=example.com/path/to/dependency
-# NB: For the next step, $DEP is assumed be the repo root.  If it is actually a
-# subdir of the repo, use the repo root here.  This is required to keep godep
-# from getting angry because `godep restore` left the tree in a "detached head"
-# state.
-rm -rf $KPATH/src/$DEP # repo root
-godep get $DEP/...
-# Change code in Kubernetes, if necessary.
-rm -rf Godeps
-rm -rf vendor
-./hack/godep-save.sh
-# Regenerate removed BUILD, licenses.
-touch vendor/BUILD
-./hack/update-bazel.sh
-./hack/update-godep-licenses.sh
-# If you haven't followed this doc step-by-step and haven't created a dedicated GOPATH,
-# make sure there is no client-go or other staging repo in $GOPATH before running the next command.
-./hack/update-staging-godeps.sh
-git checkout -- $(git status -s | grep "^ D" | awk '{print $2}' | grep ^Godeps)
+hack/run-in-gopath.sh hack/godep-save.sh
 ```
 
-_If `go get -u path/to/dependency` fails with compilation errors, instead try
-`go get -d -u path/to/dependency` to fetch the dependencies without compiling
-them. This is unusual, but has been observed._
+This will run through all of the primary targets for the Kubernetes project,
+calculate which deps are needed, and rebuild the database.  It will also
+regenerate other metadata files which the project needs, such as BUILD files and
+the LICENSE database.
 
 After all of this is done, `git status` should show you what files have been
-modified and added/removed.  Make sure to `git add` and `git rm` them.  It is
-commonly advised to make one `git commit` which includes just the dependency
-update and Godeps files, and another `git commit` that includes changes to
-Kubernetes code to use the new/updated dependency.  These commits can go into a
-single pull request.
+modified and added/removed.  Make sure to sanity-check them with `git diff`, and
+to `git add` and `git rm` them, as needed.  It is commonly advised to make one
+`git commit` which includes just the dependencies and Godeps files, and
+another `git commit` that includes changes to Kubernetes code to use (or stop
+using) the new/updated/removed dependency.  These commits can go into a single
+pull request.
+
+## Sanity checking
 
-5) Before sending your PR, it's a good idea to sanity check that your
-Godeps.json file and the contents of `vendor/ `are ok by running `hack/verify-godeps.sh`
+Before sending your PR, it's a good idea to sanity check that your
+Godeps.json file and the contents of `vendor/ `are ok:
 
-_If `hack/verify-godeps.sh` fails after a `godep update`, it is possible that a
-transitive dependency was added or removed but not updated by godeps. It then
-may be necessary to perform a `hack/godep-save.sh` to pick up the transitive
-dependency changes._
+```sh
+hack/run-in-gopath.sh hack/verify-godeps.sh
+```
 
-It is sometimes expedient to manually fix the /Godeps/Godeps.json file to
-minimize the changes. However without great care this can lead to failures
-with `hack/verify-godeps.sh`. This must pass for every PR.
+All this script will do is a restore, followed by a save, and then look for
+changes.  If you followed the above instructions, it should be clean.  If it is
+not, you get to figure out why.
 
-6) If you updated the Godeps, please also update `Godeps/LICENSES` by running
-`hack/update-godep-licenses.sh`.
+## Manual updates
 
+It is sometimes expedient to manually fix the `Godeps.json` file to
+minimize the changes. However, without great care this can lead to failures
+with the verifier scripts.  The kubernetes codebase does "interesting things"
+with symlinks between `vendor/` and `staging/` to allow multiple Go import
+paths to coexist in the same git repo.
 
+The verifiers, including `hack/verify-godeps.sh` *must* pass for every pull
+request.
 
 
 <!-- BEGIN MUNGE: GENERATED_ANALYTICS -->