Merge pull request #2912 from ixdy/bazel-docs

Update bazel documentation with current known issues and limitations

1 files changed, 116 insertions, 27 deletions

diff --git a/contributors/devel/bazel.md b/contributors/devel/bazel.md
index de80b4b2..991a0ac2 100644
--- a/contributors/devel/bazel.md
+++ b/contributors/devel/bazel.md
@@ -2,6 +2,12 @@
 
 Building and testing Kubernetes with  Bazel is supported but not yet default.
 
+Bazel is used to run all Kubernetes PRs on [Prow](https://prow.k8s.io),
+as remote caching enables significantly reduced build and test times.
+
+Some repositories (such as kubernetes/test-infra) have switched to using Bazel
+exclusively for all build, test, and release workflows.
+
 Go rules are managed by the [`gazelle`](https://github.com/bazelbuild/rules_go/tree/master/go/tools/gazelle)
 tool, with some additional rules managed by the [`kazel`](https://git.k8s.io/repo-infra/kazel) tool.
 These tools are called via the `hack/update-bazel.sh` script.
@@ -9,13 +15,16 @@ These tools are called via the `hack/update-bazel.sh` script.
 Instructions for installing Bazel
 can be found [here](https://www.bazel.io/versions/master/docs/install.html).
 
-Several `make` rules have been created for common operations:
+Several convenience `make` rules have been created for common operations:
 
-* `make bazel-build`: builds all binaries in tree
-* `make bazel-test`: runs all unit tests
-* `make bazel-test-integration`: runs all integration tests
+* `make bazel-build`: builds all binaries in tree (`bazel build -- //...
+  -//vendor/...`)
+* `make bazel-test`: runs all unit tests (`bazel test --config=unit -- //...
+  //hack:verify-all -//build/... -//vendor/...`)
+* `make bazel-test-integration`: runs all integration tests (`bazel test
+  --config integration //test/integration/...`)
 * `make bazel-release`: builds release tarballs, Docker images (for server
-  components), and Debian images
+  components), and Debian images (`bazel build //build/release-tars`)
 
 You can also interact with Bazel directly; for example, to run all `kubectl` unit
 tests, run
@@ -46,26 +55,6 @@ There are several bazel CI jobs:
 Similar jobs are run on all PRs; additionally, several of the e2e jobs use
 Bazel-built binaries when launching and testing Kubernetes clusters.
 
-## Known issues
-
-[Cross-compilation is not currently supported](https://github.com/bazelbuild/rules_go/issues/70),
-so all binaries will be built for the host OS and architecture running Bazel.
-(For example, you can't currently target linux/amd64 from macOS or linux/s390x
-from an amd64 machine.)
-
-Additionally, native macOS support is still a work in progress. Using Planter is
-a possible workaround in the interim.
-
-[Bazel does not validate build environment](https://github.com/kubernetes/kubernetes/issues/51623), thus make sure that needed
-tools and development packages are installed in the system. Bazel builds require presence of `make`, `gcc`, `g++`, `glibc and libstdc++ development headers` and `glibc static development libraries`. Please check your distribution for exact names of the packages. Examples for some commonly used distributions are below:
-
-|     Dependency        | Debian/Ubuntu                 | CentOS                         | OpenSuSE                                |
-|:---------------------:|-------------------------------|--------------------------------|-----------------------------------------|
-| Build essentials      | `apt install build-essential` | `yum groupinstall development` | `zypper install -t pattern devel_C_C++` |
-| GCC C++               | `apt install g++`             | `yum install gcc-c++`          | `zypper install gcc-c++`                |
-| GNU Libc static files | `apt install libc6-dev`       | `yum install glibc-static`     | `zypper install glibc-devel-static`     |
-
-
 ## Updating `BUILD` files
 
 To update `BUILD` files, run:
@@ -77,10 +66,10 @@ $ ./hack/update-bazel.sh
 To prevent Go rules from being updated, consult the [gazelle
 documentation](https://github.com/bazelbuild/rules_go/tree/master/go/tools/gazelle).
 
-Note that much like Go files and `gofmt`, BUILD files have standardized,
+Note that much like Go files and `gofmt`, `BUILD` files have standardized,
 opinionated style rules, and running `hack/update-bazel.sh` will format them for you.
 
-If you want to auto-format BUILD files in your editor, using something like
+If you want to auto-format `BUILD` files in your editor, use of
 [Buildifier](https://github.com/bazelbuild/buildtools/blob/master/buildifier/README.md)
 is recommended.
 
@@ -90,6 +79,106 @@ Updating the `BUILD` file for a package will be required when:
 * A `BUILD` file has been updated and needs to be reformatted
 * A new `BUILD` file has been added (parent `BUILD` files will be updated)
 
+## Known issues and limitations
+
+### [Cross-compilation of cgo is not currently natively supported](https://github.com/bazelbuild/rules_go/issues/1020)
+All binaries are currently built for the host OS and architecture running Bazel.
+(For example, you can't currently target linux/amd64 from macOS or linux/s390x
+from an amd64 machine.)
+
+The Go rules support cross-compilation of pure Go code using the `--platforms`
+flag, and this is being used successfully in the kubernetes/test-infra repo.
+
+It may already be possible to cross-compile cgo code if a custom CC toolchain is
+set up, possibly reusing the kube-cross Docker image, but this area needs
+further exploration.
+
+### The CC toolchain is not fully hermetic
+Bazel requires several tools and development packages to be installed in the system, including `gcc`, `g++`, `glibc and libstdc++ development headers` and `glibc static development libraries`. Please check your distribution for exact names of the packages. Examples for some commonly used distributions are below:
+
+|     Dependency        | Debian/Ubuntu                 | CentOS                         | OpenSuSE                                |
+|:---------------------:|-------------------------------|--------------------------------|-----------------------------------------|
+| Build essentials      | `apt install build-essential` | `yum groupinstall development` | `zypper install -t pattern devel_C_C++` |
+| GCC C++               | `apt install g++`             | `yum install gcc-c++`          | `zypper install gcc-c++`                |
+| GNU Libc static files | `apt install libc6-dev`       | `yum install glibc-static`     | `zypper install glibc-devel-static`     |
+
+If any of these packages change, they may also cause spurious build failures
+as described in [this issue](https://github.com/bazelbuild/bazel/issues/4907).
+
+An example error might look something like
+```
+ERROR: undeclared inclusion(s) in rule '//vendor/golang.org/x/text/cases:go_default_library.cgo_c_lib':
+this rule is missing dependency declarations for the following files included by 'vendor/golang.org/x/text/cases/linux_amd64_stripped/go_default_library.cgo_codegen~/_cgo_export.c':
+  '/usr/lib/gcc/x86_64-linux-gnu/7/include/stddef.h'
+```
+
+The only way to recover from this error is to force Bazel to regenerate its
+automatically-generated CC toolchain configuration by running `bazel clean
+--expunge`.
+
+Improving cgo cross-compilation may help with all of this.
+
+### Changes to Go imports requires updating BUILD files
+The Go rules in `BUILD` and `BUILD.bazel` files must be updated any time files
+are added or removed or Go imports are changed. These rules are automatically
+maintained by `gazelle`, which is run via `hack/update-bazel.sh`, but this is
+still a source of friction.
+
+[Autogazelle](https://github.com/bazelbuild/bazel-gazelle/tree/master/cmd/autogazelle)
+is a new experimental tool which may reduce or remove the need for developers
+to run `hack/update-bazel.sh`, but no work has yet been done to support it in
+kubernetes/kubernetes.
+
+### Code coverage support is incomplete for Go
+Bazel and the Go rules have limited support for code coverage. Running something
+like `bazel coverage -- //... -//vendor/...` will run tests in coverage mode,
+but no report summary is currently generated. It may be possible to combine
+`bazel coverage` with
+[Gopherage](https://github.com/kubernetes/test-infra/tree/master/gopherage),
+however.
+
+### Kubernetes code generators are not fully supported
+The make-based build system in kubernetes/kubernetes runs several code
+generators at build time:
+* [conversion-gen](https://github.com/kubernetes/code-generator/tree/master/cmd/conversion-gen)
+* [deepcopy-gen](https://github.com/kubernetes/code-generator/tree/master/cmd/deepcopy-gen)
+* [defaulter-gen](https://github.com/kubernetes/code-generator/tree/master/cmd/defaulter-gen)
+* [openapi-gen](https://github.com/kubernetes/kube-openapi/tree/master/cmd/openapi-gen)
+* [go-bindata](https://github.com/jteeuwen/go-bindata/tree/master/go-bindata)
+
+Of these, only `openapi-gen` and `go-bindata` are currently supported when
+building Kubernetes with Bazel.
+
+The `go-bindata` generated code is produced by hand-written genrules.
+
+The other code generators use special build tags of the form `//
++k8s:generator-name=arg`; for example, input files to the openapi-gen tool are
+specified with `// +k8s:openapi-gen=true`.
+
+`kazel` is used to find all packages that require OpenAPI generation, and then a
+handwritten genrule consumes this list of packages to run `openapi-gen`.
+
+For `openapi-gen`, a single output file is produced in a single Go package, which
+makes this fairly compatible with Bazel.
+All other Kubernetes code generators generally produce one output file per input
+package, which is less compatible with the Bazel workflow.
+
+The make-based build system batches up all input packages into one call to the
+code generator binary, but this is inefficient for Bazel's incrementality, as a
+change in one package may result in unnecessarily recompiling many other
+packages.
+On the other hand, calling the code generator binary multiple times is less
+efficient than calling it once, since many of the generators parse the tree for
+Go type information and other metadata.
+
+One additional challenge is that many of the code generators add additional
+Go imports which `gazelle` (and `autogazelle`) cannot infer, and so they must be
+explicitly added as dependencies in the `BUILD` files.
+
+Kubernetes has even more code generators than this limited list, but the rest
+are generally run as `hack/update-*.sh` scripts and checked into the repository,
+and so are not immediately needed for Bazel parity.
+
 ## Contacts
 For help or discussion, join the [#bazel](https://kubernetes.slack.com/messages/bazel)
 channel on Kubernetes Slack.