summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorLv Lv <github@lvlv.email>2017-02-04 17:22:02 +0800
committerLv Lv <github@lvlv.email>2017-02-04 17:22:02 +0800
commit9a631fff444a968dd2dbf30fc8c976aa35d994b8 (patch)
tree903e337b7d6afdc7a607c80884b541c09645360e
parent5fd2d14b294e80f59c1e2e1853e501d99d90af26 (diff)
parenta9a3996aff21d772ba96d612ea0614ecafd5919c (diff)
Merge branch 'master' of https://github.com/kubernetes/community
-rw-r--r--CLA.md4
-rw-r--r--README.md32
-rw-r--r--community/K8sYoutubeCollaboration.md39
-rw-r--r--community/developer-summit-2016/KubDevSummitVoting.md2
-rw-r--r--community/developer-summit-2016/application_service_definition_notes.md2
-rw-r--r--community/developer-summit-2016/cluster_lifecycle_notes.md132
-rw-r--r--contributors/design-proposals/admission_control_limit_range.md4
-rw-r--r--contributors/design-proposals/aggregated-api-servers.md277
-rw-r--r--contributors/design-proposals/all-in-one-volume.md297
-rw-r--r--contributors/design-proposals/api-group.md2
-rw-r--r--contributors/design-proposals/apiserver-watch.md4
-rw-r--r--contributors/design-proposals/bootstrap-discovery.md6
-rw-r--r--contributors/design-proposals/client-package-structure.md4
-rw-r--r--contributors/design-proposals/cluster-deployment.md4
-rw-r--r--contributors/design-proposals/container-runtime-interface-v1.md6
-rw-r--r--contributors/design-proposals/control-plane-resilience.md6
-rw-r--r--contributors/design-proposals/controller-ref.md4
-rw-r--r--contributors/design-proposals/core-metrics-pipeline.md156
-rw-r--r--contributors/design-proposals/cri-dockershim-checkpoint.md127
-rw-r--r--contributors/design-proposals/custom-metrics-api.md329
-rw-r--r--contributors/design-proposals/daemon.md10
-rw-r--r--contributors/design-proposals/daemonset-update.md315
-rw-r--r--contributors/design-proposals/deploy.md2
-rwxr-xr-xcontributors/design-proposals/disk-accounting.md128
-rw-r--r--contributors/design-proposals/downward_api_resources_limits_requests.md22
-rw-r--r--contributors/design-proposals/dramatically-simplify-cluster-creation.md2
-rw-r--r--contributors/design-proposals/enhance-pluggable-policy.md8
-rw-r--r--contributors/design-proposals/expansion.md4
-rw-r--r--contributors/design-proposals/external-lb-source-ip-preservation.md24
-rw-r--r--contributors/design-proposals/federated-api-servers.md205
-rw-r--r--contributors/design-proposals/federated-replicasets.md42
-rw-r--r--contributors/design-proposals/federated-services.md2
-rw-r--r--contributors/design-proposals/federation-phase-1.md18
-rw-r--r--contributors/design-proposals/garbage-collection.md32
-rw-r--r--contributors/design-proposals/gpu-support.md2
-rw-r--r--contributors/design-proposals/ha_master.md4
-rw-r--r--contributors/design-proposals/high-availability.md2
-rw-r--r--contributors/design-proposals/initial-resources.md4
-rw-r--r--contributors/design-proposals/kubectl-create-from-env-file.md84
-rw-r--r--contributors/design-proposals/kubelet-cri-logging.md2
-rw-r--r--contributors/design-proposals/kubelet-rootfs-distribution.md165
-rw-r--r--contributors/design-proposals/kubemark.md12
-rw-r--r--contributors/design-proposals/monitoring_architecture.md6
-rw-r--r--contributors/design-proposals/multi-platform.md6
-rw-r--r--contributors/design-proposals/network-policy.md2
-rw-r--r--contributors/design-proposals/networking.md11
-rw-r--r--contributors/design-proposals/node-allocatable.md2
-rw-r--r--contributors/design-proposals/optional-configmap.md174
-rw-r--r--contributors/design-proposals/performance-related-monitoring.md4
-rw-r--r--contributors/design-proposals/pod-pid-namespace.md77
-rw-r--r--contributors/design-proposals/pod-resource-management.md8
-rw-r--r--contributors/design-proposals/propagation.md2
-rw-r--r--contributors/design-proposals/release-test-signal.md130
-rw-r--r--contributors/design-proposals/rescheduling-for-critical-pods.md8
-rw-r--r--contributors/design-proposals/resource-qos.md6
-rw-r--r--contributors/design-proposals/resource-quota-scoping.md6
-rw-r--r--contributors/design-proposals/runtimeconfig.md4
-rw-r--r--contributors/design-proposals/secret-configmap-downwarapi-file-mode.md2
-rw-r--r--contributors/design-proposals/security-context-constraints.md27
-rw-r--r--contributors/design-proposals/self-hosted-final-cluster.pngbin0 -> 22888 bytes
-rw-r--r--contributors/design-proposals/self-hosted-kubernetes.md103
-rw-r--r--contributors/design-proposals/self-hosted-layers.pngbin0 -> 17774 bytes
-rw-r--r--contributors/design-proposals/self-hosted-moving-parts.pngbin0 -> 36201 bytes
-rw-r--r--contributors/design-proposals/service-discovery.md4
-rw-r--r--contributors/design-proposals/service-external-name.md14
-rw-r--r--contributors/design-proposals/service_accounts.md2
-rw-r--r--contributors/design-proposals/sig-cli/template.md38
-rw-r--r--contributors/design-proposals/stateful-apps.md20
-rw-r--r--contributors/design-proposals/synchronous-garbage-collection.md2
-rw-r--r--contributors/design-proposals/templates.md22
-rw-r--r--contributors/design-proposals/versioning.md2
-rw-r--r--contributors/design-proposals/volume-hostpath-qualifiers.md2
-rw-r--r--contributors/design-proposals/volume-ownership-management.md2
-rw-r--r--contributors/design-proposals/volume-provisioning.md2
-rw-r--r--contributors/design-proposals/volume-snapshotting.md6
-rw-r--r--contributors/devel/README.md14
-rw-r--r--contributors/devel/adding-an-APIGroup.md39
-rwxr-xr-xcontributors/devel/api_changes.md2
-rw-r--r--contributors/devel/bazel.md60
-rw-r--r--contributors/devel/cli-roadmap.md11
-rw-r--r--contributors/devel/controllers.md24
-rw-r--r--contributors/devel/development.md13
-rw-r--r--contributors/devel/e2e-tests.md13
-rw-r--r--contributors/devel/faster_reviews.md12
-rw-r--r--contributors/devel/instrumentation.md204
-rw-r--r--contributors/devel/kubectl-conventions.md16
-rw-r--r--contributors/devel/kubelet-cri-networking.md16
-rwxr-xr-xcontributors/devel/kubemark-guide.md172
-rw-r--r--contributors/devel/local-cluster/local.md4
-rw-r--r--contributors/devel/on-call-build-cop.md18
-rw-r--r--contributors/devel/pull-request-commands.md80
-rw-r--r--contributors/devel/pull-requests.md2
-rw-r--r--contributors/devel/release/README.md78
-rw-r--r--contributors/devel/running-locally.md3
-rwxr-xr-xcontributors/devel/scheduler.md2
-rwxr-xr-xcontributors/devel/scheduler_algorithm.md4
-rw-r--r--contributors/devel/testing.md14
-rw-r--r--governance.md121
-rw-r--r--project-managers/README.md2
-rw-r--r--sig-api-machinery/README.md4
-rw-r--r--sig-apps/README.md2
-rw-r--r--sig-apps/minutes/2016-05-18.md2
-rw-r--r--sig-apps/minutes/2016-06-08.md2
-rw-r--r--sig-aws/README.md19
-rw-r--r--sig-big-data/README.md27
-rw-r--r--sig-cli/README.md26
-rw-r--r--sig-cli/contributing.md329
-rw-r--r--sig-contribx/README.md35
-rw-r--r--sig-creation-procedure.md104
-rw-r--r--sig-federation/README.md4
-rw-r--r--sig-network/README.md44
-rw-r--r--sig-on-prem/README.md21
-rw-r--r--sig-openstack/README.md6
-rw-r--r--sig-scheduling/README.md9
-rw-r--r--sig-service-catalog/README.md2
-rw-r--r--sig-storage/1.3-retrospective/README.md8
-rw-r--r--sig-storage/README.md31
-rw-r--r--sig-storage/contributing.md2
-rw-r--r--sig-testing/README.md6
-rw-r--r--sig-ui/README.md2
-rw-r--r--sig-windows/README.md6
121 files changed, 4048 insertions, 787 deletions
diff --git a/CLA.md b/CLA.md
index dabd1c37..2f327ffc 100644
--- a/CLA.md
+++ b/CLA.md
@@ -25,6 +25,6 @@
**Step 5**: The status on your old PRs will be updated when any new comment is made on it.
-### I’m having issues with signing the CLA.
+### I'm having issues with signing the CLA.
-If you’re facing difficulty with signing the CNCF CLA, please explain your case on https://github.com/kubernetes/kubernetes/issues/27796 and we (@sarahnovotny and @foxish), along with the CNCF will help sort it out.
+If you're facing difficulty with signing the CNCF CLA, please explain your case on https://github.com/kubernetes/kubernetes/issues/27796 and we (@sarahnovotny and @foxish), along with the CNCF will help sort it out.
diff --git a/README.md b/README.md
index 72e00942..89647907 100644
--- a/README.md
+++ b/README.md
@@ -45,41 +45,27 @@ The community meeting calendar is available as an [iCal to subscribe to] (https:
| [Apps](sig-apps/README.md) | [@michelleN (Michelle Noorali, Deis)](https://github.com/michelleN)<br>[@mattfarina (Matt Farina, HPE)](https://github.com/mattfarina) | [Group](https://groups.google.com/forum/#!forum/kubernetes-sig-apps) | [#sig-apps](https://kubernetes.slack.com/messages/sig-apps) | [Mondays 9:00AM PST](https://zoom.us/j/4526666954) |
| [Auth](sig-auth/README.md) | [@ erictune (Eric Tune, Google)](https://github.com/erictune)<br> [@ericchiang (Eric Chiang, CoreOS)](https://github.com/orgs/kubernetes/people/ericchiang)<br> [@liggitt (Jordan Liggitt, Red Hat)] (https://github.com/orgs/kubernetes/people/liggitt) <br> [@deads2k (David Eads, Red Hat)] (https://github.com/orgs/kubernetes/people/deads2k) | [Group](https://groups.google.com/forum/#!forum/kubernetes-sig-auth) | [#sig-auth](https://kubernetes.slack.com/messages/sig-auth/) | Biweekly [Wednesdays at 1100 to 1200 PT](https://zoom.us/my/k8s.sig.auth) |
| [Autoscaling](sig-autoscaling/README.md) | [@fgrzadkowski (Filip Grządkowski, Google)](https://github.com/fgrzadkowski)<br> [@directxman12 (Solly Ross, Red Hat)](https://github.com/directxman12) | [Group](https://groups.google.com/forum/#!forum/kubernetes-sig-autoscaling) | [#sig-autoscaling](https://kubernetes.slack.com/messages/sig-autoscaling/) | Biweekly (or triweekly) on [Thurs at 0830 PT](https://plus.google.com/hangouts/_/google.com/k8s-autoscaling) |
-| [AWS](sig-aws/README.md) | [@justinsb (Justin Santa Barbara)](https://github.com/justinsb)<br>[@kris-nova (Kris Childress )](https://github.com/kris-nova)<br>[@chrislovecnm (Chris Love)](https://github.com/chrislovecnm)<br>[@mfburnett (Mackenzie Burnett)](https://github.com/mfburnett) | [Group](https://groups.google.com/forum/#!forum/kubernetes-sig-aws) | [#sig-aws](https://kubernetes.slack.com/messages/sig-aws/) | 1st/3rd Friday at 9:00AM PST on [Zoom](https://zoom.us/my/k8ssigaws) |
-| [Big Data](sig-big-data/README.md) | [@zmerlynn (Zach Loafman, Google)](https://github.com/zmerlynn)<br>[@timothysc (Timothy St. Clair, Red Hat)](https://github.com/timothysc)<br>[@wattsteve (Steve Watt, Red Hat)](https://github.com/wattsteve) | [Group](https://groups.google.com/forum/#!forum/kubernetes-sig-big-data) | [#sig-big-data](https://kubernetes.slack.com/messages/sig-big-data/) | Suspended |
+| [AWS](sig-aws/README.md) | [@justinsb (Justin Santa Barbara)](https://github.com/justinsb)<br>[@kris-nova (Kris Nova)](https://github.com/kris-nova)<br>[@chrislovecnm (Chris Love)](https://github.com/chrislovecnm)<br>[@mfburnett (Mackenzie Burnett)](https://github.com/mfburnett) | [Group](https://groups.google.com/forum/#!forum/kubernetes-sig-aws) | [#sig-aws](https://kubernetes.slack.com/messages/sig-aws/) | We meet on [Zoom](https://zoom.us/my/k8ssigaws), and the calls are scheduled via the official [group mailing list](https://groups.google.com/forum/#!forum/kubernetes-sig-aws) |
+| [Big Data](sig-big-data/README.md) | [@foxish (Anirudh Ramanathan, Google)](https://github.com/foxish)| [Group](https://groups.google.com/forum/#!forum/kubernetes-sig-big-data) | [#sig-big-data](https://kubernetes.slack.com/messages/sig-big-data/) | Wednesdays at 10am PST, link posted in [the official group](https://groups.google.com/forum/#!forum/kubernetes-sig-big-data). |
| [CLI](sig-cli/README.md) | [@fabianofranz (Fabiano Franz, Red Hat)](https://github.com/fabianofranz)<br>[@pwittrock (Phillip Wittrock, Google)](https://github.com/pwittrock)<br>[@AdoHe (Tony Ado, Alibaba)](https://github.com/AdoHe) | [Group](https://groups.google.com/forum/#!forum/kubernetes-sig-cli) | [#sig-cli](https://kubernetes.slack.com/messages/sig-cli) | Bi-weekly Wednesdays at 9:00 AM PT on [Zoom](https://zoom.us/my/sigcli) |
| [Cluster Lifecycle](sig-cluster-lifecycle/README.md) | [@lukemarsden (Luke Marsden, Weave)] (https://github.com/lukemarsden) | [Group](https://groups.google.com/forum/#!forum/kubernetes-sig-cluster-lifecycle) | [#sig-cluster-lifecycle](https://kubernetes.slack.com/messages/sig-cluster-lifecycle) | Tuesdays at 09:00 AM PST on [Zoom](https://zoom.us/j/166836​624) |
| [Cluster Ops](sig-cluster-ops/README.md) | [@zehicle (Rob Hirschfeld, RackN)](https://github.com/zehicle) <br> [@mikedanese (Mike Danese, Google] (https://github.com/mikedanese) | [Group](https://groups.google.com/forum/#!forum/kubernetes-sig-cluster-ops) | [#sig-cluster-ops](https://kubernetes.slack.com/messages/sig-cluster-ops) | Thursdays at 1:00 PM PST on [hangouts](https://plus.google.com/hangouts/_/google.com/sig-cluster-ops)|
-| [Contributor Experience](sig-contribx/README.md) | [@grodrigues3 (Garrett Rodrigues, Google)](https://github.com/Grodrigues3) <br> [@pwittrock (Phillip Witrock, Google)] (https://github.com/pwittrock) | [Group](https://groups.google.com/forum/#!forum/kubernetes-wg-contribex) | [#wg-contribex] (https://kubernetes.slack.com/messages/wg-contribex) | Biweekly Wednesdays 9:30 AM PST on [zoom] (https://zoom.us/j/4730809290) |
+| [Contributor Experience](sig-contribx/README.md) | [@grodrigues3 (Garrett Rodrigues, Google)](https://github.com/Grodrigues3) <br> [@pwittrock (Phillip Witrock, Google)] (https://github.com/pwittrock) <br> [@Phillels (Elsie Phillips, CoreOS)](https://github.com/Phillels) | [Group](https://groups.google.com/forum/#!forum/kubernetes-wg-contribex) | [#wg-contribex] (https://kubernetes.slack.com/messages/wg-contribex) | Biweekly Wednesdays 9:30 AM PST on [zoom] (https://zoom.us/j/4730809290) |
| [Docs] (sig-docs/README.md) | [@pwittrock (Philip Wittrock, Google)] (https://github.com/pwittrock) <br> [@devin-donnelly (Devin Donnelly, Google)] (https://github.com/devin-donnelly) <br> [@jaredbhatti (Jared Bhatti, Google)] (https://github.com/jaredbhatti)| [Group] (https://groups.google.com/forum/#!forum/kubernetes-sig-docs) | [#sig-docs] (https://kubernetes.slack.com/messages/sig-docs) | Tuesdays @ 10:30AM PST on [Zoom](https://zoom.us/j/4730809290) |
-| [Federation](sig-federation/README.md) | [@quinton-hoole (Quinton Hoole, Google)](https://github.com/quinton-hoole) | [Group](https://groups.google.com/forum/#!forum/kubernetes-sig-federation) | [#sig-federation](https://kubernetes.slack.com/messages/sig-federation/) | Bi-weekly on Thursdays at 9:30 AM PT on [hangouts](https://plus.google.com/hangouts/_/google.com/ubernetes) |
+| [Federation](sig-federation/README.md) | [@csbell (Christian Bell, Google)](https://github.com/csbell) <br> [@quinton-hoole (Quinton Hoole, Huawei)](https://github.com/quinton-hoole) | [Group](https://groups.google.com/forum/#!forum/kubernetes-sig-federation) | [#sig-federation](https://kubernetes.slack.com/messages/sig-federation/) | Bi-weekly on Monday at 9:00 AM PST on [hangouts](https://plus.google.com/hangouts/_/google.com/ubernetes) |
| [Instrumentation](sig-instrumentation/README.md) | [@piosz (Piotr Szczesniak, Google)](https://github.com/piosz) <br> [@fabxc (Fabian Reinartz, CoreOS)](https://github.com/fabxc) | [Group](https://groups.google.com/forum/#!forum/kubernetes-sig-instrumentation) | [#sig-instrumentation](https://kubernetes.slack.com/messages/sig-instrumentation) | [Thursdays at 9.30 AM PST](https://zoom.us/j/5342565819) |
| [Network](sig-network/README.md) | [@thockin (Tim Hockin, Google)](https://github.com/thockin)<br> [@dcbw (Dan Williams, Red Hat)](https://github.com/dcbw)<br> [@caseydavenport (Casey Davenport, Tigera)](https://github.com/caseydavenport) | [Group](https://groups.google.com/forum/#!forum/kubernetes-sig-network) | [#sig-network](https://kubernetes.slack.com/messages/sig-network/) | Thursdays at 2:00 PM PST on [Zoom](https://zoom.us/j/5806599998) |
| [Node](sig-node/README.md) | [@dchen1107 (Dawn Chen, Google)](https://github.com/dchen1107)<br>[@euank (Euan Kemp, CoreOS)](https://github.com/orgs/kubernetes/people/euank)<br>[@derekwaynecarr (Derek Carr, Red Hat)](https://github.com/derekwaynecarr) | [Group](https://groups.google.com/forum/#!forum/kubernetes-sig-node) | [#sig-node](https://kubernetes.slack.com/messages/sig-node/) | [Tuesdays at 10:00 PT](https://plus.google.com/hangouts/_/google.com/sig-node-meetup?authuser=0) |
-| [On Prem](sig-onprem/README.md) | [@josephjacks (Joseph Jacks, Apprenda)] (https://github.com/josephjacks) <br> [@zen (Tomasz Napierala, Mirantis)] (https://github.com/zen)| [Group](https://groups.google.com/forum/#!forum/kubernetes-sig-on-prem) | [#sig-onprem](https://kubernetes.slack.com/messages/sig-onprem/) | Every second Wednesday at 8 PM PST / 11 PM EST |
+| [On Prem](sig-on-prem/README.md) | [@josephjacks (Joseph Jacks, Apprenda)] (https://github.com/josephjacks) <br> [@zen (Tomasz Napierala, Mirantis)] (https://github.com/zen)| [Group](https://groups.google.com/forum/#!forum/kubernetes-sig-on-prem) | [#sig-onprem](https://kubernetes.slack.com/messages/sig-onprem/) | Every two weeks on Wednesday at 9 PM PST / 12 PM EST |
| [OpenStack](sig-openstack/README.md) | [@idvoretskyi (Ihor Dvoretskyi, Mirantis)] (https://github.com/idvoretskyi) <br> [@xsgordon (Steve Gordon, Red Hat)] (https://github.com/xsgordon)| [Group](https://groups.google.com/forum/#!forum/kubernetes-sig-openstack) | [#sig-openstack](https://kubernetes.slack.com/messages/sig-openstack/) | Every second Wednesday at 5 PM PDT / 2 PM EDT |
| [PM](project-managers/README.md) | [] ()| [Group](https://groups.google.com/forum/#!forum/kubernetes-pm) | []() | TBD|
| [Rktnetes](sig-rktnetes/README.md) | [@euank (Euan Kemp, CoreOS)] (https://github.com/euank) <br> [@tmrts (Tamer Tas)] (https://github.com/tmrts) <br> [@yifan-gu (Yifan Gu, CoreOS)] (https://github.com/yifan-gu) | [Group](https://groups.google.com/forum/#!forum/kubernetes-sig-rktnetes) | [#sig-rktnetes](https://kubernetes.slack.com/messages/sig-rktnetes/) | [As needed (ad-hoc)](https://zoom.us/j/830298957) |
| [Scalability](sig-scalability/README.md) | [@lavalamp (Daniel Smith, Google)](https://github.com/lavalamp)<br>[@countspongebob (Bob Wise, Samsung SDS)](https://github.com/countspongebob)<br>[@jbeda (Joe Beda)](https://github.com/jbeda) | [Group](https://groups.google.com/forum/#!forum/kubernetes-sig-scale) | [#sig-scale](https://kubernetes.slack.com/messages/sig-scale/) | [Thursdays at 09:00 PT](https://zoom.us/j/989573207) |
| [Scheduling](sig-scheduling/README.md) | [@davidopp (David Oppenheimer, Google)](https://github.com/davidopp)<br>[@timothysc (Timothy St. Clair, Red Hat)](https://github.com/timothysc) | [Group](https://groups.google.com/forum/#!forum/kubernetes-sig-scheduling) | [#sig-scheduling](https://kubernetes.slack.com/messages/sig-scheduling/) | Alternate between Mondays at 1 PM PT and Wednesdays at 12:30 AM PT on [Zoom](https://zoom.us/zoomconference?m=rN2RrBUYxXgXY4EMiWWgQP6Vslgcsn86) |
-| [Service Catalog](sig-service-catalog/README.md) | [@pmorie (Paul Morie, Red Hat)](https://github.com/pmorie) <br> [@dutronlabs (Chris Dutra, Apprenda)](github.com/dutronlabs) | [Group](https://groups.google.com/forum/#!forum/kubernetes-sig-service-catalog) | [#sig-service-catalog](https://kubernetes.slack.com/messages/sig-service-catalog/) | [Mondays at 1 PM PST](https://zoom.us/j/7201225346) |
+| [Service Catalog](sig-service-catalog/README.md) | [@pmorie (Paul Morie, Red Hat)](https://github.com/pmorie) <br> [@arschles (Aaron Schlesinger, Deis)](github.com/arschles) <br> [@bmelville (Brendan Melville, Google)](https://github.com/bmelville) <br> [@duglin (Doug Davis, IBM)](https://github.com/duglin)| [Group](https://groups.google.com/forum/#!forum/kubernetes-sig-service-catalog) | [#sig-service-catalog](https://kubernetes.slack.com/messages/sig-service-catalog/) | [Mondays at 1 PM PST](https://zoom.us/j/7201225346) |
| [Storage](sig-storage/README.md) | [@saad-ali (Saad Ali, Google)](https://github.com/saad-ali)<br>[@childsb (Brad Childs, Red Hat)](https://github.com/childsb) | [Group](https://groups.google.com/forum/#!forum/kubernetes-sig-storage) | [#sig-storage](https://kubernetes.slack.com/messages/sig-storage/) | Bi-weekly Thursdays 9 AM PST (or more frequently) on [Zoom](https://zoom.us/j/614261834) |
| [Testing](sig-testing/README.md) | [@spiffxp (Aaron Crickenberger, Samsung)](https://github.com/spiffxp)<br>[@ixdy (Jeff Grafton, Google)](https://github.com/ixdy) | [Group](https://groups.google.com/forum/#!forum/kubernetes-sig-testing) | [#sig-testing](https://kubernetes.slack.com/messages/sig-testing/) | [Tuesdays at 9:30 AM PT](https://zoom.us/j/553910341) |
| [UI](sig-ui/README.md) | [@romlein (Dan Romlein, Apprenda)](https://github.com/romlein)<br> [@bryk (Piotr Bryk, Google)](https://github.com/bryk) | [Group](https://groups.google.com/forum/#!forum/kubernetes-sig-ui) | [#sig-ui](https://kubernetes.slack.com/messages/sig-ui/) | Wednesdays at 4:00 PM CEST |
-| [Windows](sig-windows/README.md) | [@michmike77 (Michael Michael, Apprenda)](https://github.com/michmike)| [Group](https://groups.google.com/forum/#!forum/kubernetes-sig-windows) | [#sig-windows](https://kubernetes.slack.com/messages/sig-windows) | Bi-weekly Tuesdays at 10:00 AM PT |
-
-### How to start a SIG
-
-* Propose the new SIG publicly, including a brief mission statement, by emailing kubernetes-dev@googlegroups.com and kubernetes-users@googlegroups.com, then wait a couple of days for feedback
-* Create a group with at least 2 owners (and one of which should be sarahnovotny at google.com): kubernetes-sig-foo@googlegroups.com
- * Put a description of the topic in the welcome message
- * Set "Join the group" and "Post" to "Public"
-* Ask a repo maintainer to create a github label, if one doesn't already exist: sig/foo
-* If you wish, request a new [kubernetes.slack.com](http://kubernetes.slack.com) channel (#sig-foo) from @sarahnovotny. New users can join at [slack.kubernetes.io](http://slack.kubernetes.io).
-* Organize video meetings as needed. No need to wait for the [Weekly Community Video Conference](community/README.md) to discuss. Please report summary of SIG activities there.
- * Request a Zoom account from @sarahnovotny if you expect more than 30 attendees or attendees from China.
- * Add the meeting to the community meeting calendar by inviting cgnt364vd8s86hr2phapfjc6uk@group.calendar.google.com.
-* Use existing proposal and PR process
-* Announce new SIG on kubernetes-dev@googlegroups.com and ask a repo maintainer to create a kubernetes/community directory and github team for the new group
-* Submit a PR to add any SIG-related docs, schedules, roadmaps, etc. to your new kubernetes/community/SIG-foo directory.
-* Slack activity is archived at [kubernetes.slackarchive.io](http://kubernetes.slackarchive.io). To start archiving a new channel invite the slackarchive bot to the channel via `/invite @slackarchive`
+| [Windows](sig-windows/README.md) | [@michmike77 (Michael Michael, Apprenda)](https://github.com/michmike)| [Group](https://groups.google.com/forum/#!forum/kubernetes-sig-windows) | [#sig-windows](https://kubernetes.slack.com/messages/sig-windows) | Bi-weekly Tuesdays at 9:30 AM PT |
+
+### [How to start a SIG](sig-creation-procedure.md)
diff --git a/community/K8sYoutubeCollaboration.md b/community/K8sYoutubeCollaboration.md
new file mode 100644
index 00000000..b1c7de37
--- /dev/null
+++ b/community/K8sYoutubeCollaboration.md
@@ -0,0 +1,39 @@
+
+### Kubernetes Youtube Channel Collaboration
+###
+
+#### Meeting Playlists
+####
+The [Kubernetes Youtube Channel](https://www.youtube.com/channel/UCZ2bu0qutTOM0tHYa_jkIwg) has separate playlists for each SIG’s meeting recordings, as well as recordings for other meetings (i.e. Cloud Native, Kubernetes Community meetings).
+
+To better serve the community, we have setup [collaboration](https://support.google.com/youtube/answer/6109639) on these playlists, so that anyone with the appropriate link to the particular playlist can upload videos *to that particular playlist* (links & playlists are 1:1).
+
+Each SIG playlist’s link will be shared with the SIG’s leadership, and other playlists' links (i.e. Cloud Native) will be shared with the appropriate point(s) of contact. The SIG playlist links will be sent to the official SIG lead Google Groups.
+
+#### Uploading Guidelines for Collaborators
+####
+Collaboration should simplify things for everyone, but with privilege comes responsibility :). We assume all playlist collaborators in the community will use things fairly and appropriately, subject to the guidelines below:
+
+1. Once collaboration is setup for each meeting recording playlist, the upload responsibility will fall on the SIG leaders or other appropriate point(s) of contact. Community managers (Sarah and Cameron) *will only be escalation for issues with those playlists*
+
+2. Please post only related content (mostly meeting recordings) in the appropriate playlists
+ - Posting of any exceedingly inappropriate content (i.e. NSFW content) will result in ***immediate*** suspension of privileges
+
+3. Please ensure all videos have the same naming format, which is:
+ - Kubernetes [Name of Playlist’s Group] YYYYMMDD
+ - i.e. Kubernetes SIG Service Catalog 20161129
+
+4. All playlists need to be organized chronologically for ease of use, which is most easily done by selected “date added, newest” as the “Ordering” option in the playlist settings
+
+5. Please do not remove any already-published content from the playlists without checking with the community managers
+
+6. For any small issues that arise (i.e. improper naming / ordering), you may be asked by the community managers to attempt to resolve the issue yourselves
+
+7. Any egregious or habitual violation* of the above rules will result in suspension of collaboration privileges for the particular individual, or for the entire playlist if the individual can’t be identified
+ - If an individual is suspended, the playlist link will be remade and the new link will be shared with the non-offending individuals
+ - If *playlist* collaboration is suspended, uploading by community managers to the playlist of interest will ***not*** be a priority, and will likely occur on a delayed basis
+ - *Note that "habitual violation" means "more than 3 issues per quarter"
+
+Your community managers are happy to help with any questions that you may have and will do their best to help if anything goes wrong. Please get in touch via sarahnovotny@google.com and czahedi@google.com.
+
+
diff --git a/community/developer-summit-2016/KubDevSummitVoting.md b/community/developer-summit-2016/KubDevSummitVoting.md
index daa4c5e3..46909c02 100644
--- a/community/developer-summit-2016/KubDevSummitVoting.md
+++ b/community/developer-summit-2016/KubDevSummitVoting.md
@@ -1,4 +1,4 @@
-## #Kubernetes Developer's Summit Discussion Topics Voting
+###Kubernetes Developer's Summit Discussion Topics Voting
###
A voting poll for discussion topic proposals has been created, and the link to the poll can be found [here][poll].
diff --git a/community/developer-summit-2016/application_service_definition_notes.md b/community/developer-summit-2016/application_service_definition_notes.md
index 54b13192..e8f4c0c5 100644
--- a/community/developer-summit-2016/application_service_definition_notes.md
+++ b/community/developer-summit-2016/application_service_definition_notes.md
@@ -36,7 +36,7 @@ Is this different from what the PAASes already do? It's not that different, we
Being able to view an application as a single unifying concept is a major desire. Want to click "my app" and see all of the objects associated with it. It would be an overlay on top of Kubernetes, not something in core.
-One pending feature is that you can't look up different types of controllers in the API, that's going to be fixed. Another one is that we can't trace the depenences; helm doesn't label all of the components deployed with the app.
+One pending feature is that you can't look up different types of controllers in the API, that's going to be fixed. Another one is that we can't trace the dependencies; helm doesn't label all of the components deployed with the app.
Need to identify things which are missing in core kubernetes, if there are any.
diff --git a/community/developer-summit-2016/cluster_lifecycle_notes.md b/community/developer-summit-2016/cluster_lifecycle_notes.md
new file mode 100644
index 00000000..f42df9f6
--- /dev/null
+++ b/community/developer-summit-2016/cluster_lifecycle_notes.md
@@ -0,0 +1,132 @@
+# Cluster Lifecycle Deployment & Upgrade Roadmap
+
+Moderator: Mike Danese
+
+Note taker: Robert Bailey
+
+Date: 2016-11-10
+
+## Goals
+
+Discuss HA, upgrades, and config management beyond kubeadm/kops & try to identify things that are currently underserved (upgrade testing, version skew policy, security upgrades)
+
+## Discussion
+
+kubeadm - not destined for production?
+* Doing resource provisioning (cloud VMs) is out of scope
+* Should be a toolbox that does the common parts of cluster lifecycle
+ * And should be able to break out just the pieces that you want
+* Found a bunch of common parts of existing cluster deployment and want to build more of it into the core
+
+AI (luke): Create an intro guide to cluster lifecyle
+
+If anyone wants to work on Upgrades the Hard Way with Rob this afternoon.
+
+Wishlist
+* HA
+* Upgrades
+* Config Management
+* Toolbox vs. guided flow
+* Documentation
+* Conformance Testing
+* PKI
+
+### HA
+
+* Story hasn't changed in a long time:
+ * People set up the cluster, run them in production
+ * Lack of documentation
+* What haven't we been focused on?
+ * Some day there may be apiservers that may move, but there aren't today
+ * If you've misconfigure it, it's really hard to debug
+ * E.g. If you just launch 2 apiservers they fight over the endpoint. It requires insider knowledge to recognize this and know how to fix it
+ * Forces an ip address on the endpoint, which isn't compatible with AWS
+ * AI (claytonc): Fix the flag to take a host instead of just an ip address
+* There are things that are command line flags that are going to be a pain to synchronize
+ * Move more configuration into etcd
+
+### Upgrades
+
+* Minor and patch upgrades, look at them separately
+* Do we have skew requirements that are different for minor vs patch upgrades?
+ * E.g. Can you upgrade nodes before master for a patch version?
+* AI: Socialize the existing version skew documentation
+* AI: Clarify the documentation skew
+* Do we want to support 4-part version numbers?
+ * Chris love: please don't do this
+* Mike Rubin: Patch releases shouldn't create surprises
+* Distribution of Kubernetes?
+* Jordan Liggitt: Would like to see upgrade documentation on every install guide
+ * At least for patch releases
+ * AI (?): File issues against owners for the current getting started guides to add a section on upgrades
+* Luke Marsden: I would like to lead an effort to support self hosting in some of the user flows (in particular a kubeadm flow) in an attempt to make it really easy to deploy a patch upgrade
+ * Assume single master, external etcd
+ * Joe Beda: The nice thing about self hosting upgrades is that there isn't anything cloud platform specific which allows us to build more general tooling
+
+### Distribution
+
+* In 1.5 we've begun experimenting with OS package management tooling
+* Should we push further on this?
+* The release tarball is getting bigger
+ * Should be ameliorated in 1.5 by breaking it into arch-specific bundles
+* Jordan Liggitt: We can't tell people to script their install against the tarball because the structure of the tarball becomes an api
+
+### Config Management
+
+* Config is currently command line flags. Work is being done to convert into structured api types (outstanding PRs from ncdc@ and mtoffen@).
+* How much should we use config maps vs flags vs something else?
+* Joe Beda: Definitely an issue for the kubelet - specifically setting the DNS IP flag
+ * Vish: Unless/until the kubelet has local checkpointing it's dangerous to use the apiserver for checkpointing configuration since the apiserver may become unavailable
+* Need to figure out how we deal with config that points to other files (e.g. certs)
+* Rob: We may want to split the discussion about configuring the kubelet vs the control plane
+* Mike Rubin: The kubelet will eventually need to be able to run standalone. Need to think about packaging and configuration as distinct.
+ * The Kubelet has a lot of value if it can work without an apiserver
+ * The node effort takes Docker + Linux and productizes it
+* Mike Danese: The same type of config could also benefit other components
+* Jordan Liggitt: Have client cert bootstrap stuff in Kubelet
+
+### Toolbox vs guided flow
+
+* [mostly skipped for time]
+* Chris Love: Need to document our compartmentelizing each thing
+* Luke: This isn't a "vs" it's an "and"
+
+### Documentation
+
+* What is lacking?
+* Joe Beda: The fact that Kelsey had to write "Kubernetes the hard way" shows that we don't have documentation
+* Chris Love: HA Upgrades
+* Jordan Liggitt: Docs should look like a tree
+ * Start at the high level, if you want more detail, then you can drill down into each piece
+ * If you expand to all of the leaves, then you end up back at k8s the hard way
+* Mike Rubin: Questions from support/users are less about setting up and more about tearing down
+ * What will still be around after destroying a cluster
+ * E.g. Deleting a namespace, deleting a cluster from a federation, deleting a node from the cluster
+* Need an introduction to the SIG (Luke already volunteered to write one)
+* Mike Rubin: Rollbacks and rollback documentation
+ * When you add a new feature (say in 1.4) and we roll back to an earlier version what happens to those resources
+ * Chris Love: elephant is what happens when you roll back from etcd3 → etcd2
+
+### Conformance Testing
+
+* What can we do in 2017 to make progress on this?
+* Jordan Liggitt: Need to categorize conformance tests into ones that you could run against a production cluster vs those that you shouldn't
+* Lucas: Three levels of validation: node, k8s standard base, deep/destructive testing. Want to make these all easy through kubeadm
+* Is performance testing out of scope?
+ * Clayton: Misconfiguration is often caught through performance testing so we shouldn't remove it from scope
+
+### PKI
+
+* Jordan Liggitt: Have client cert bootstrap stuff in Kubelet
+* Chris Love: Need to loop in sig-auth. Need to use TLS certs for etcd clusters.
+* Aaron Levy: Plan to add CSR into the etcd operation similar to what is going into the k8s api.
+* Joe Beda: Two modes right now: can have the apiserver act as a CA; many serious users will want to use their own CA
+* Jordan Liggitt: Things that need certs should be able to take them or components should be able to generate (if appropriate)
+ * Rotation depends on whether we are using the built-in CA or an external CA
+* Mike Rubin: Why not do both rotation and revocation
+ * Rob: Many applications don't respect revocation so it's generally considered weaker
+* Clayton: If you can rotate then you may not need to revoke
+* Jordan Liggitt: Tied to config management
+* Joe Beda: Part of the discovery info is the root CA and many people don't realize that it can be a bundle instead of a single CA — this enables rotation
+* Clayton: In a secured cluster, etcd is the core. Have to think about it as the inner circle of security that the apiserver is outside of. If you are extremely cautious then you should use client certs in the apiserver. You can collapse the rings if you want.
+
diff --git a/contributors/design-proposals/admission_control_limit_range.md b/contributors/design-proposals/admission_control_limit_range.md
index 06cce2cb..4821dfce 100644
--- a/contributors/design-proposals/admission_control_limit_range.md
+++ b/contributors/design-proposals/admission_control_limit_range.md
@@ -154,9 +154,9 @@ Per container, the following must hold true:
Supported Defaults:
-1. Default - if the named resource has no enumerated value, the Limit is equal
+1. Default - if the named resource has no enumerated values, the Limit is equal
to the Default
-2. DefaultRequest - if the named resource has no enumerated value, the Request
+2. DefaultRequest - if the named resource has no enumerated values, the Request
is equal to the DefaultRequest
**Type: Pod**
diff --git a/contributors/design-proposals/aggregated-api-servers.md b/contributors/design-proposals/aggregated-api-servers.md
new file mode 100644
index 00000000..5b3a10c3
--- /dev/null
+++ b/contributors/design-proposals/aggregated-api-servers.md
@@ -0,0 +1,277 @@
+# Aggregated API Servers
+
+## Abstract
+
+We want to divide the single monolithic API server into multiple aggregated
+servers. Anyone should be able to write their own aggregated API server to expose APIs they want.
+Cluster admins should be able to expose new APIs at runtime by bringing up new
+aggregated servers.
+
+## Motivation
+
+* Extensibility: We want to allow community members to write their own API
+ servers to expose APIs they want. Cluster admins should be able to use these
+ servers without having to require any change in the core kubernetes
+ repository.
+* Unblock new APIs from core kubernetes team review: A lot of new API proposals
+ are currently blocked on review from the core kubernetes team. By allowing
+ developers to expose their APIs as a separate server and enabling the cluster
+ admin to use it without any change to the core kubernetes repository, we
+ unblock these APIs.
+* Place for staging experimental APIs: New APIs can be developed in separate
+ aggregated servers, and installed only by those willing to take the risk of
+ installing an experimental API. One they are stable, it is then easy to
+ package them up for installation in other clusters.
+* Ensure that new APIs follow kubernetes conventions: Without the mechanism
+ proposed here, community members might be forced to roll their own thing which
+ may or may not follow kubernetes conventions.
+
+## Goal
+
+* Developers should be able to write their own API server and cluster admins
+ should be able to add them to their cluster, exposing new APIs at runtime. All
+ of this should not require any change to the core kubernetes API server.
+* These new APIs should be seamless extension of the core kubernetes APIs (ex:
+ they should be operated upon via kubectl).
+
+## Non Goals
+
+The following are related but are not the goals of this specific proposal:
+* Make it easy to write a kubernetes API server.
+
+## High Level Architecture
+
+There will be a new component in the cluster, `kube-aggregator`, which has these
+responsibilities:
+* Provide an API for registering API servers.
+* Summarize discovery information from all the servers.
+* Proxy client requests to individual servers.
+
+The reverse proxy is provided for convenience. Clients can discover server URLs
+using the summarized discovery information and contact them directly. Simple
+clients can always use the proxy and don't need to know that under the hood
+multiple apiservers are running.
+
+Wording note: When we say "API servers" we really mean groups of apiservers,
+since any individual apiserver is horizontally replicatable. Similarly,
+kube-aggregator itself is horizontally replicatable.
+
+## Operational configurations
+
+There are two configurations in which it makes sense to run `kube-aggregator`.
+ 1. In **test mode**/**single-user mode**. An individual developer who wants to test
+ their own apiserver could run their own private copy of `kube-aggregator`,
+ configured such that only they can interact with it. This allows for testing
+ both `kube-aggregator` and any custom apiservers without the potential for
+ causing any collateral damage in the rest of the cluster. Unfortunately, in
+ this configuration, `kube-aggregator`'s built in proxy will lack the client
+ cert that allows it to perform authentication that the rest of the cluster
+ will trust, so its functionality will be somewhat limited.
+ 2. In **gateway mode**. Each cluster should run kube-aggregator as the official
+ gateway to the cluster, where it aggregates all of the apiservers the cluster
+ administer wishes to provide. This configuration is the intended usage of
+ `kube-aggregator`.
+
+### Constraints
+
+* Unique API group versions across servers: Each API server (and groups of servers, in HA)
+ should expose unique API group versions. So, for example, you can serve
+ `api.mycompany.com/v1` from one apiserver and the replacement
+ `api.mycompany.com/v2` from another apiserver while you update clients. But
+ you can't serve `api.mycompany.com/v1/frobbers` and
+ `api.mcompany.com/v1/grobinators` from different apiservers. This restriction
+ allows us to limit the scope of `kube-aggregator` to a managable level.
+* Follow API conventions: APIs exposed by every API server should adhere to [kubernetes API
+ conventions](../devel/api-conventions.md).
+* Support discovery API: Each API server should support the kubernetes discovery API
+ (list the suported groupVersions at `/apis` and list the supported resources
+ at `/apis/<groupVersion>/`)
+* No bootstrap problem: The core kubernetes apiserver must not depend on any
+ other aggregated server to come up. Non-core apiservers may use other non-core
+ apiservers, but must not fail in their absence.
+
+## Component Dependency Order
+
+`kube-aggregator` is not part of the core `kube-apiserver`.
+The dependency order (for the cluster gateway configuration) looks like this:
+ 1. `etcd`
+ 2. `kube-apiserver`
+ 3. core scheduler, kubelet, service proxy (enough stuff to create a pod, run it on a node, and find it via service)
+ 4. `kubernetes-aggregator` as a pod/service - default summarizer and proxy
+ 5. controllers
+ 6. other API servers and their controllers
+ 7. clients, web consoles, etc
+
+Nothing below the `kubernetes-aggregator` can rely on the aggregator or proxy
+being present. `kubernetes-aggregator` should be runnable as a pod backing a
+service in a well-known location. Something like `api.kube-public.svc` or
+similar seems appropriate since we'll want to allow network traffic to it from
+every other namespace in the cluster. We recommend using a dedicated namespace,
+since compromise of that namespace will expose the entire cluster: the
+proxy has the power to act as any user against any API server.
+
+## Implementation Details
+
+### Summarizing discovery information
+
+We can have a very simple Go program to summarize discovery information from all
+servers. Cluster admins will register each aggregated API server (its baseURL and swagger
+spec path) with the proxy. The proxy will summarize the list of all group versions
+exposed by all registered API servers with their individual URLs at `/apis`.
+
+### Reverse proxy
+
+We can use any standard reverse proxy server like nginx or extend the same Go program that
+summarizes discovery information to act as reverse proxy for all aggregated servers.
+
+Cluster admins are also free to use any of the multiple open source API management tools
+(for example, there is [Kong](https://getkong.org/), which is written in lua and there is
+[Tyk](https://tyk.io/), which is written in Go). These API management tools
+provide a lot more functionality like: rate-limiting, caching, logging,
+transformations and authentication.
+In future, we can also use ingress. That will give cluster admins the flexibility to
+easily swap out the ingress controller by a Go reverse proxy, nginx, haproxy
+or any other solution they might want.
+
+`kubernetes-aggregator` uses a simple proxy implementation alongside its discovery information
+which supports connection upgrade (for `exec`, `attach`, etc) and runs with delegated
+authentication and authorization against the core `kube-apiserver`. As a proxy, it adds
+complete user information, including user, groups, and "extra" for backing API servers.
+
+### Storage
+
+Each API server is responsible for storing their resources. They can have their
+own etcd or can use kubernetes server's etcd using [third party
+resources](../design/extending-api.md#adding-custom-resources-to-the-kubernetes-api-server).
+
+### Health check
+
+Kubernetes server's `/api/v1/componentstatuses` will continue to report status
+of master components that it depends on (scheduler and various controllers).
+Since clients have access to server URLs, they can use that to do
+health check of individual servers.
+In future, if a global health check is required, we can expose a health check
+endpoint in the proxy that will report the status of all aggregated api servers
+in the cluster.
+
+### Auth
+
+Since the actual server which serves client's request can be opaque to the client,
+all API servers need to have homogeneous authentication and authorisation mechanisms.
+All API servers will handle authn and authz for their resources themselves.
+The current authentication infrastructure allows token authentication delegation to the
+core `kube-apiserver` and trust of an authentication proxy, which can be fullfilled by
+`kubernetes-aggregator`.
+
+#### Server Role Bootstrapping
+
+External API servers will often have to provide roles for the resources they
+provide to other API servers in the cluster. This will usually be RBAC
+clusterroles, RBAC clusterrolebindings, and apiaggregation types to describe
+their API server. The external API server should *never* attempt to
+self-register these since the power to mutate those resources provides the
+power to destroy the cluster. Instead, there are two paths:
+ 1. the easy path - In this flow, the API server supports a `/bootstrap/<group>` endpoint
+ which provides the resources that can be piped to a `kubectl create -f` command a cluster-admin
+ can use those endpoints to prime other servers.
+ 2. the reliable path - In a production cluster, you generally want to know, audit, and
+ track the resources required to make your cluster work. In these scenarios, you want
+ to have the API resource list ahead of time. API server authors can provide a template.
+
+Nothing stops an external API server from supporting both.
+
+### kubectl
+
+kubectl will talk to `kube-aggregator`'s discovery endpoint and use the discovery API to
+figure out the operations and resources supported in the cluster.
+We will need to make kubectl truly generic. Right now, a lot of operations
+(like get, describe) are hardcoded in the binary for all resources. A future
+proposal will provide details on moving those operations to server.
+
+Note that it is possible for kubectl to talk to individual servers directly in
+which case proxy will not be required at all, but this requires a bit more logic
+in kubectl. We can do this in future, if desired.
+
+### Handling global policies
+
+Now that we have resources spread across multiple API servers, we need to
+be careful to ensure that global policies (limit ranges, resource quotas, etc) are enforced.
+Future proposals will improve how this is done across the cluster.
+
+#### Namespaces
+
+When a namespaced resource is created in any of the aggregated server, that
+server first needs to check with the kubernetes server that:
+
+* The namespace exists.
+* User has authorization to create resources in that namespace.
+* Resource quota for the namespace is not exceeded.
+
+To prevent race conditions, the kubernetes server might need to expose an atomic
+API for all these operations.
+
+While deleting a namespace, kubernetes server needs to ensure that resources in
+that namespace maintained by other servers are deleted as well. We can do this
+using resource [finalizers](../design/namespaces.md#finalizers). Each server
+will add themselves in the set of finalizers before they create a resource in
+the corresponding namespace and delete all their resources in that namespace,
+whenever it is to be deleted (kubernetes API server already has this code, we
+will refactor it into a library to enable reuse).
+
+Future proposal will talk about this in more detail and provide a better
+mechanism.
+
+#### Limit ranges and resource quotas
+
+kubernetes server maintains [resource quotas](../admin/resourcequota/README.md) and
+[limit ranges](../admin/limitrange/README.md) for all resources.
+Aggregated servers will need to check with the kubernetes server before creating any
+resource.
+
+## Methods for running on hosted kubernetes clusters
+
+Where "hosted" means the cluster users have very limited or no permissions to
+change the control plane installation, for example on GKE, where it is managed
+by Google. There are three ways of running on such a cluster:.
+
+ 1. `kube-aggregator` will run in the single-user / test configuration on any
+ installation of Kubernetes, even if the user starting it only has permissions
+ in one namespace.
+ 2. Just like 1 above, if all of the users can agree on a location, then they
+ can make a public namespace and run a copy of `kube-aggregator` in that
+ namespace for everyone. The downside of running like this is that none of the
+ cluster components (controllers, nodes, etc) would be going through this
+ kube-aggregator.
+ 3. The hosted cluster provider can integrate `kube-aggregator` into the
+ cluster. This is the best configuration, but it may take a quarter or two after
+ `kube-aggregator` is ready to go for providers to complete this integration.
+
+## Alternatives
+
+There were other alternatives that we had discussed.
+
+* Instead of adding a proxy in front, let the core kubernetes server provide an
+ API for other servers to register themselves. It can also provide a discovery
+ API which the clients can use to discover other servers and then talk to them
+ directly. But this would have required another server API a lot of client logic as well.
+* Validating aggregated servers: We can validate new servers when they are registered
+ with the proxy, or keep validating them at regular intervals, or validate
+ them only when explicitly requested, or not validate at all.
+ We decided that the proxy will just assume that all the servers are valid
+ (conform to our api conventions). In future, we can provide conformance tests.
+
+## Future Work
+
+* Validate servers: We should have some conformance tests that validate that the
+ servers follow kubernetes api-conventions.
+* Provide centralised auth service: It is very hard to ensure homogeneous auth
+ across multiple aggregated servers, especially in case of hosted clusters
+ (where different people control the different servers). We can fix it by
+ providing a centralised authentication and authorization service which all of
+ the servers can use.
+
+
+
+<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
+[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/aggregated-api-servers.md?pixel)]()
+<!-- END MUNGE: GENERATED_ANALYTICS -->
diff --git a/contributors/design-proposals/all-in-one-volume.md b/contributors/design-proposals/all-in-one-volume.md
new file mode 100644
index 00000000..504a10bf
--- /dev/null
+++ b/contributors/design-proposals/all-in-one-volume.md
@@ -0,0 +1,297 @@
+## Abstract
+
+Describes a proposal for a new volume type that can project secrets,
+configmaps, and downward API items.
+
+## Motivation
+
+Users often need to build directories that contain multiple types of
+configuration and secret data. For example, a configuration directory for some
+software package may contain both config files and credentials. Currently, there
+is no way to achieve this in Kubernetes without scripting inside of a container.
+
+## Constraints and Assumptions
+
+1. The volume types must remain unchanged for backward compatability
+2. There will be a new volume type for this proposed functionality, but no
+ other API changes
+3. The new volume type should support atomic updates in the event of an input
+ change
+
+## Use Cases
+
+1. As a user, I want to automatically populate a single volume with the keys
+ from multiple secrets, configmaps, and with downward API information, so
+ that I can synthesize a single directory with various sources of
+ information
+2. As a user, I want to populate a single volume with the keys from multiple
+ secrets, configmaps, and with downward API information, explicitly
+ specifying paths for each item, so that I can have full control over the
+ contents of that volume
+
+### Populating a single volume without pathing
+
+A user should be able to map any combination of resources mentioned above into a
+single directory. There are plenty of examples of software that needs to be
+configured both with config files and secret data. The combination of having
+that data not only accessible, but in the same location provides for an easier
+user experience.
+
+### Populating a single volume with pathing
+
+Currently it is possible to define the path within a volume for specific
+resources. Therefore the same is true for each resource contained within the
+new single volume.
+
+## Current State Overview
+
+The only way of utilizing secrets, configmaps, and downward API (while
+maintaining atomic updates) currently is to access the data using separate mount
+paths as shown in the volumeMounts section below:
+
+```yaml
+apiVersion: v1
+kind: Pod
+metadata:
+ name: volume-test
+spec:
+ containers:
+ - name: container-test
+ image: busybox
+ volumeMounts:
+ - name: mysecret
+ mountPath: "/secrets"
+ readOnly: true
+ - name: podInfo
+ mountPath: "/podinfo"
+ readOnly: true
+ - name: config-volume
+ mountPath: "/config"
+ readOnly: true
+ volumes:
+ - name: mysecret
+ secret:
+ secretName: jpeeler-db-secret
+ items:
+ - key: username
+ path: my-group/my-username
+ - name: podInfo
+ downwardAPI:
+ items:
+ - path: "labels"
+ fieldRef:
+ fieldPath: metadata.labels
+ - path: "annotations"
+ fieldRef:
+ fieldPath: metadata.annotations
+ - name: config-volume
+ configMap:
+ name: special-config
+ items:
+ - key: special.how
+ path: path/to/special-key
+```
+
+## Analysis
+
+There are several combinations of resources that can be used at once, which
+all warrant consideration. The combinations are listed with one instance of
+each resource, but real world usage will support multiple instances of a
+specific resource too. Each example was written with the expectation that all
+of the resources are to be projected to the same directory (or with the same
+non-root parent), though it is not strictly required.
+
+### ConfigMap + Secrets + Downward API
+
+The user wishes to deploy containers with configuration data that includes
+passwords. An application using these resources could be deploying OpenStack
+on Kubernetes. The configuration data may need to be assembled differently
+depending on if the services are going to be used for production or for
+testing. If a pod is labeled with production or testing, the downward API
+selector metadata.labels can be used to produce the correct OpenStack configs.
+
+### ConfigMap + Secrets
+
+Again, the user wishes to deploy containers involving configuration data and
+passwords. This time the user is executing an Ansible playbook stored as a
+configmap, with some sensitive encrypted tasks that are decrypted using a vault
+password file.
+
+### ConfigMap + Downward API
+
+In this case, the user wishes to generate a config including the pod’s name
+(available via the metadata.name selector). This application may then pass the
+pod name along with requests in order to easily determine the source without
+using IP tracking.
+
+### Secrets + Downward API
+
+A user may wish to use a secret as a public key to encrypt the namespace of
+the pod (available via the metadata.namespace selector). This example may be
+the most contrived, but perhaps the operator wishes to use the application to
+deliver the namespace information securely without using an encrypted
+transport.
+
+### Collisions between keys when configured paths are identical
+
+In the event the user specifies any keys with the same path, the pod spec will
+not be accepted as valid. Note the specified path for mysecret and myconfigmap
+are the same:
+
+```yaml
+apiVersion: v1
+kind: Pod
+metadata:
+ name: volume-test
+spec:
+ containers:
+ - name: container-test
+ image: busybox
+ volumeMounts:
+ - name: all-in-one
+ mountPath: "/projected-volume"
+ readOnly: true
+ volumes:
+ - name: all-in-one
+ projected:
+ sources:
+ - secret:
+ name: mysecret
+ items:
+ - key: username
+ path: my-group/data
+ - configMap:
+ name: myconfigmap
+ items:
+ - key: config
+ path: my-group/data
+```
+
+
+### Collisions between keys without configured paths
+
+The only run time validation can occur is when all the paths are known at pod
+creation, similar to the above scenario. Otherwise, when a conflict occurs the
+most recent specified resource will overwrite anything preceding it (this is
+true for resources that are updated after pod creation as well).
+
+### Collisions when one path is explicit and the other is automatically projected
+
+In the event that there is a collision due to a user specified path matching
+data that is automatically projected, the latter resource will overwrite
+anything preceding it as before.
+
+## Code changes
+
+### Proposed API objects
+
+```go
+type Projections struct {
+ Sources []VolumeProjection `json:"sources"`
+ DefaultMode *int32 `json:"defaultMode,omitempty"`
+}
+
+type VolumeProjection struct {
+ Secret *SecretVolumeSource `json:"secret,omitempty"`
+ ConfigMap *ConfigMapVolumeSource `json:"configMap,omitempty"`
+ DownwardAPI *DownwardAPIVolumeSource `json:"downwardAPI,omitempty"`
+}
+```
+
+### Additional required modifications
+
+Add to the VolumeSource struct:
+
+```go
+Projected *Projections `json:"projected,omitempty"`
+// (other existing fields omitted for brevity)
+```
+
+Add to the SecretVolumeSource struct:
+
+```go
+LocalObjectReference `json:"name,omitempty"`
+// (other existing fields omitted for brevity)
+```
+
+The appropriate conversion code would need to be generated for v1, validations
+written, and the new volume plugin code produced as well.
+
+## Examples
+
+### Sample pod spec with a secret, a downward API, and a configmap
+
+```yaml
+apiVersion: v1
+kind: Pod
+metadata:
+ name: volume-test
+spec:
+ containers:
+ - name: container-test
+ image: busybox
+ volumeMounts:
+ - name: all-in-one
+ mountPath: "/projected-volume"
+ readOnly: true
+ volumes:
+ - name: all-in-one
+ projected:
+ sources:
+ - secret:
+ name: mysecret
+ items:
+ - key: username
+ path: my-group/my-username
+ - downwardAPI:
+ items:
+ - path: "labels"
+ fieldRef:
+ fieldPath: metadata.labels
+ - path: "cpu_limit"
+ resourceFieldRef:
+ containerName: container-test
+ resource: limits.cpu
+ - configMap:
+ name: myconfigmap
+ items:
+ - key: config
+ path: my-group/my-config
+```
+
+### Sample pod spec with multiple secrets with a non-default permission mode set
+
+```yaml
+apiVersion: v1
+kind: Pod
+metadata:
+ name: volume-test
+spec:
+ containers:
+ - name: container-test
+ image: busybox
+ volumeMounts:
+ - name: all-in-one
+ mountPath: "/projected-volume"
+ readOnly: true
+ volumes:
+ - name: all-in-one
+ projected:
+ sources:
+ - secret:
+ name: mysecret
+ items:
+ - key: username
+ path: my-group/my-username
+ - secret:
+ name: mysecret2
+ items:
+ - key: password
+ path: my-group/my-password
+ mode: 511
+```
+
+
+<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
+[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/all-in-one-volume.md?pixel)]()
+<!-- END MUNGE: GENERATED_ANALYTICS -->
diff --git a/contributors/design-proposals/api-group.md b/contributors/design-proposals/api-group.md
index 435994fe..586cc7d6 100644
--- a/contributors/design-proposals/api-group.md
+++ b/contributors/design-proposals/api-group.md
@@ -49,7 +49,7 @@
* for OpenShift v1 API, because it's currently registered at /oapi/v1, to be backward compatible, OpenShift may set prefix="oapi", group="".
- * for other new third-party API, they should also use the prefix="apis" and choose the group and version. This can be done through the thirdparty API plugin mechanism in [13000](http://pr.k8s.io/13000).
+ * for other new third-party API, they should also use the prefix="apis" and choose the group and version. This can be done through the third-party API plugin mechanism in [13000](http://pr.k8s.io/13000).
2. supporting API discovery:
diff --git a/contributors/design-proposals/apiserver-watch.md b/contributors/design-proposals/apiserver-watch.md
index 9764768f..2fc8900a 100644
--- a/contributors/design-proposals/apiserver-watch.md
+++ b/contributors/design-proposals/apiserver-watch.md
@@ -63,7 +63,7 @@ The rest of this section describes the concrete steps that need to be done
to implement the proposal.
1. Since we want the watch in apiserver to be optional for different resource
-types, this needs to be self-contained and hidden behind a well defined API.
+types, this needs to be self-contained and hidden behind a well-defined API.
This should be a layer very close to etcd - in particular all registries:
"pkg/registry/generic/registry" should be built on top of it.
We will solve it by turning tools.EtcdHelper by extracting its interface
@@ -80,7 +80,7 @@ we will store two things:
- the resourceVersion of the object (being an etcdIndex)
- the object watched from etcd itself (in a deserialized form)
- This should be as simple as having an array an treating it as a cyclic buffer.
+ This should be as simple as having an array and treating it as a cyclic buffer.
Obviously resourceVersion of objects watched from etcd will be increasing, but
they are necessary for registering a new watcher that is interested in all the
changes since a given etcdIndex.
diff --git a/contributors/design-proposals/bootstrap-discovery.md b/contributors/design-proposals/bootstrap-discovery.md
index dbcf793c..11a912e2 100644
--- a/contributors/design-proposals/bootstrap-discovery.md
+++ b/contributors/design-proposals/bootstrap-discovery.md
@@ -36,7 +36,7 @@ Additions include:
* In an HA world, API servers may come and go and it is necessary to make sure we are talking to the same cluster as we thought we were talking to.
* A _set_ of addresses for finding the cluster.
* It is implied that all of these are equivalent and that a client can try multiple until an appropriate target is found.
- * Initially I’m proposing a flat set here. In the future we can introduce more structure that hints to the user which addresses to try first.
+ * Initially I'm proposing a flat set here. In the future we can introduce more structure that hints to the user which addresses to try first.
* Better documentation and exposure of:
* The root certificates can be a bundle to enable rotation.
* If no root certificates are given (and the insecure bit isn't set) then the client trusts the system managed list of CAs.
@@ -45,7 +45,7 @@ Additions include:
**This is to be implemented in a later phase**
-Any client of the cluster will want to have this information. As the configuration of the cluster changes we need the client to keep this information up to date. It is assumed that the information here won’t drift so fast that clients won’t be able to find *some* way to connect.
+Any client of the cluster will want to have this information. As the configuration of the cluster changes we need the client to keep this information up to date. It is assumed that the information here won't drift so fast that clients won't be able to find *some* way to connect.
In exceptional circumstances it is possible that this information may be out of date and a client would be unable to connect to a cluster. Consider the case where a user has kubectl set up and working well and then doesn't run kubectl for quite a while. It is possible that over this time (a) the set of servers will have migrated so that all endpoints are now invalid or (b) the root certificates will have rotated so that the user can no longer trust any endpoint.
@@ -83,7 +83,7 @@ If the user requires some auth to the HTTPS server (to keep the ClusterInfo obje
### Method: Bootstrap Token
-There won’t always be a trusted external endpoint to talk to and transmitting
+There won't always be a trusted external endpoint to talk to and transmitting
the locator file out of band is a pain. However, we want something more secure
than just hitting HTTP and trusting whatever we get back. In this case, we
assume we have the following:
diff --git a/contributors/design-proposals/client-package-structure.md b/contributors/design-proposals/client-package-structure.md
index 2d30021d..9cbd6aa8 100644
--- a/contributors/design-proposals/client-package-structure.md
+++ b/contributors/design-proposals/client-package-structure.md
@@ -151,7 +151,7 @@ Where:
resource.
The types returned/consumed by such functions will be e.g. api/v1, NOT the
-current version inspecific types. The current internal-versioned client is
+current version in specific types. The current internal-versioned client is
inconvenient for users, as it does not protect them from having to recompile
their code with every minor update. (We may continue to generate an
internal-versioned client for our own use for a while, but even for our own
@@ -235,7 +235,7 @@ We do this to:
against their cluster, but we need to make guarantees that the clients we
shipped with v1.X.0 will work with v1.X+1.0, and vice versa. That's not
practical unless we "bless" a particular version of each API group and ship an
- official client set with earch release. (If the server supports 15 groups with
+ official client set with each release. (If the server supports 15 groups with
2 versions each, that's 2^15 different possible client sets. We don't want to
test all of them.)
diff --git a/contributors/design-proposals/cluster-deployment.md b/contributors/design-proposals/cluster-deployment.md
index c6466f89..e9ca8d54 100644
--- a/contributors/design-proposals/cluster-deployment.md
+++ b/contributors/design-proposals/cluster-deployment.md
@@ -74,7 +74,7 @@ TODO: Enumerate certificates which have to be generated.
**v1.2 simplifications**:
1. kubelet-runner.sh - we will provide a custom docker image to run kubelet; it will contain
-kubelet binary and will run it using ```nsenter``` to workaround problem with mount propagation
+kubelet binary and will run it using ```nsenter``` to workaround problems with mount propagation
1. kubelet config file - we will read kubelet configuration file from disk instead of apiserver; it will
be generated locally and copied to all nodes.
@@ -125,7 +125,7 @@ network solutions.
1. flannel daemon will run as a standalone binary (not in docker container)
2. flannel server will assign CIDRs to nodes outside of kubernetes; this will require restarting kubelet
-after reconfiguring network bridge on local machine; this will also require running master nad node differently
+after reconfiguring network bridge on local machine; this will also require running master and node differently
(```--configure-cbr0=false``` on node and ```--allocate-node-cidrs=false``` on master), which breaks encapsulation
between nodes
diff --git a/contributors/design-proposals/container-runtime-interface-v1.md b/contributors/design-proposals/container-runtime-interface-v1.md
index 36592727..d305aaaa 100644
--- a/contributors/design-proposals/container-runtime-interface-v1.md
+++ b/contributors/design-proposals/container-runtime-interface-v1.md
@@ -86,7 +86,7 @@ container setup that are not currently trackable as Pod constraints, e.g.,
filesystem setup, container image pulling, etc.*
A container in a PodSandbox maps to an application in the Pod Spec. For Linux
-containers, they are expected to share at least network and IPC namespaces,
+containers, they are expected to share at least network, PID and IPC namespaces,
with sharing more namespaces discussed in [#1615](https://issues.k8s.io/1615).
@@ -130,7 +130,7 @@ All functions listed above are expected to be thread-safe.
### Pod/Container Lifecycle
-The PodSandbox’s lifecycle is decoupled from the containers, i.e., a sandbox
+The PodSandbox's lifecycle is decoupled from the containers, i.e., a sandbox
is created before any containers, and can exist after all containers in it have
terminated.
@@ -225,7 +225,7 @@ applicable to its platform, and should return an error otherwise.
### Keep it minimal
The proposed interface is experimental, i.e., it will go through (many) changes
-until it stabilizes. The principle is to to keep the interface minimal and
+until it stabilizes. The principle is to keep the interface minimal and
extend it later if needed. This includes a several features that are still in
discussion and may be achieved alternatively:
diff --git a/contributors/design-proposals/control-plane-resilience.md b/contributors/design-proposals/control-plane-resilience.md
index 7b93cba8..95a1d93c 100644
--- a/contributors/design-proposals/control-plane-resilience.md
+++ b/contributors/design-proposals/control-plane-resilience.md
@@ -10,9 +10,9 @@
Some amount of confusion exists around how we currently, and in future
want to ensure resilience of the Kubernetes (and by implication
-Kubernetes Cluster Federation) control plane. This document is an attempt to capture that
+Kubernetes Cluster Federation) control plane. This document is an attempt to capture that
definitively. It covers areas including self-healing, high
-availability, bootstrapping and recovery. Most of the information in
+availability, bootstrapping and recovery. Most of the information in
this document already exists in the form of github comments,
PR's/proposals, scattered documents, and corridor conversations, so
document is primarily a consolidation and clarification of existing
@@ -23,7 +23,7 @@ ideas.
* **Self-healing:** automatically restarting or replacing failed
processes and machines without human intervention
* **High availability:** continuing to be available and work correctly
- even if some components are down or uncontactable. This typically
+ even if some components are down or uncontactable. This typically
involves multiple replicas of critical services, and a reliable way
to find available replicas. Note that it's possible (but not
desirable) to have high
diff --git a/contributors/design-proposals/controller-ref.md b/contributors/design-proposals/controller-ref.md
index 09dfd684..349f9b4b 100644
--- a/contributors/design-proposals/controller-ref.md
+++ b/contributors/design-proposals/controller-ref.md
@@ -22,7 +22,7 @@ Approvers:
Main goal of `ControllerReference` effort is to solve a problem of overlapping controllers that fight over some resources (e.g. `ReplicaSets` fighting with `ReplicationControllers` over `Pods`), which cause serious [problems](https://github.com/kubernetes/kubernetes/issues/24433) such as exploding memory of Controller Manager.
-We don’t want to have (just) an in-memory solution, as we don’t want a Controller Manager crash to cause massive changes in object ownership in the system. I.e. we need to persist the information about "owning controller".
+We don't want to have (just) an in-memory solution, as we don’t want a Controller Manager crash to cause massive changes in object ownership in the system. I.e. we need to persist the information about "owning controller".
Secondary goal of this effort is to improve performance of various controllers and schedulers, by removing the need for expensive lookup for all matching "controllers".
@@ -75,7 +75,7 @@ and
By design there are possible races during adoption if multiple controllers can own a given object.
-To prevent re-adoption of an object during deletion the `DeletionTimestamp` will be set when deletion is starting. When a controller has a non-nil `DeletionTimestamp` it won’t take any actions except updating its `Status` (in particular it won’t adopt any objects).
+To prevent re-adoption of an object during deletion the `DeletionTimestamp` will be set when deletion is starting. When a controller has a non-nil `DeletionTimestamp` it won't take any actions except updating its `Status` (in particular it won't adopt any objects).
# Implementation plan (sketch):
diff --git a/contributors/design-proposals/core-metrics-pipeline.md b/contributors/design-proposals/core-metrics-pipeline.md
new file mode 100644
index 00000000..4f7c69ad
--- /dev/null
+++ b/contributors/design-proposals/core-metrics-pipeline.md
@@ -0,0 +1,156 @@
+# Core Metrics in kubelet
+
+**Author**: David Ashpole (@dashpole)
+
+**Last Updated**: 1/31/2017
+
+**Status**: Proposal
+
+This document proposes a design for the set of metrics included in an eventual Core Metrics Pipeline.
+
+<!-- BEGIN MUNGE: GENERATED_TOC -->
+
+- [Core Metrics in kubelet](#core-metrics-in-kubelet)
+ - [Introduction](#introduction)
+ - [Definitions](#definitions)
+ - [Background](#background)
+ - [Motivations](#motivations)
+ - [Proposal](#proposal)
+ - [Non Goals](#non-goals)
+ - [Design](#design)
+ - [Metric Requirements:](#metric-requirements)
+ - [Proposed Core Metrics:](#proposed-core-metrics)
+ - [On-Demand Design:](#on-demand-design)
+ - [Future Work](#future-work)
+
+<!-- END MUNGE: GENERATED_TOC -->
+
+## Introduction
+
+### Definitions
+"Kubelet": The daemon that runs on every kubernetes node and controls pod and container lifecycle, among many other things.
+["cAdvisor":](https://github.com/google/cadvisor) An open source container monitoring solution which only monitors containers, and has no concept of kubernetes constructs like pods or volumes.
+["Summary API":](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/api/v1alpha1/stats/types.go) A kubelet API which currently exposes node metrics for use by both system components and monitoring systems.
+["CRI":](https://github.com/kubernetes/community/blob/master/contributors/devel/container-runtime-interface.md) The Container Runtime Interface designed to provide an abstraction over runtimes (docker, rkt, etc).
+"Core Metrics": A set of metrics described in the [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md) whose purpose is to provide metrics for first-class resource isolation and untilization features, including [resource feasibility checking](https://github.com/eBay/Kubernetes/blob/master/docs/design/resources.md#the-resource-model) and node resource management.
+"Resource": A consumable element of a node (e.g. memory, disk space, CPU time, etc).
+"First-class Resource": A resource critical for scheduling, whose requests and limits can be (or soon will be) set via the Pod/Container Spec.
+"Metric": A measure of consumption of a Resource.
+
+### Background
+The [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md) proposal contains a blueprint for a set of metrics referred to as "Core Metrics". The purpose of this proposal is to specify what those metrics are, to enable work relating to the collection, by the kubelet, of the metrics.
+
+Kubernetes vendors cAdvisor into its codebase, and the kubelet uses cAdvisor as a library that enables it to collect metrics on containers. The kubelet can then combine container-level metrics from cAdvisor with the kubelet's knowledge of kubernetes constructs (e.g. pods) to produce the kubelet Summary statistics, which provides metrics for use by the kubelet, or by users through the [Summary API](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/api/v1alpha1/stats/types.go). cAdvisor works by collecting metrics at an interval (10 seconds, by default), and the kubelet then simply queries these cached metrics whenever it has a need for them.
+
+Currently, cAdvisor collects a large number of metrics related to system and container performance. However, only some of these metrics are consumed by the kubelet summary API, and many are not used. The kubelet [Summary API](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/api/v1alpha1/stats/types.go) is published to the kubelet summary API endpoint (stats/summary). Some of the metrics provided by the summary API are consumed by kubernetes system components, but many are included for the sole purpose of providing metrics for monitoring.
+
+### Motivations
+The [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md) proposal explains why a separate monitoring pipeline is required.
+
+By publishing core metrics, the kubelet is relieved of its responsibility to provide metrics for monitoring.
+The third party monitoring pipeline also is relieved of any responsibility to provide these metrics to system components.
+
+cAdvisor is structured to collect metrics on an interval, which is appropriate for a stand-alone metrics collector. However, many functions in the kubelet are latency-sensitive (eviction, for example), and would benifit from a more "On-Demand" metrics collection design.
+
+### Proposal
+This proposal is to use this set of core metrics, collected by the kubelet, and used solely by kubernetes system components to support "First-Class Resource Isolation and Utilization Features". This proposal is not designed to be an API published by the kubelet, but rather a set of metrics collected by the kubelet that will be transformed, and published in the future.
+
+The target "Users" of this set of metrics are kubernetes components (though not neccessarily directly). This set of metrics itself is not designed to be user-facing, but is designed to be general enough to support user-facing components.
+
+### Non Goals
+Everything covered in the [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md) design doc will not be covered in this proposal. This includes the third party metrics pipeline, and the methods by which the metrics found in this proposal are provided to other kubernetes components.
+
+Integration with CRI will not be covered in this proposal. In future proposals, integrating with CRI may provide a better abstraction of information required by the core metrics pipeline to collect metrics.
+
+The kubelet API endpoint, including the format, url pattern, versioning strategy, and name of the API will be the topic of a follow-up proposal to this proposal.
+
+## Design
+This design covers only metrics to be included in the Core Metrics Pipeline.
+
+High level requirements for the design are as follows:
+ - The kubelet collects the minimum possible number of metrics to provide "First-Class Resource Isolation and Utilization Features".
+ - Metrics can be fetched "On Demand", giving the kubelet more up-to-date stats.
+
+This proposal purposefully omits many metrics that may eventually become core metrics. This is by design. Once metrics are needed to support First-Class Resource Isolation and Utilization Features, they can be added to the core metrics API.
+
+### Metric Requirements
+The core metrics api is designed to provide metrics for "First Class Resource Isolation and Utilization Features" within kubernetes.
+
+Many kubernetes system components currently support these features. Many more components that support these features are in development.
+The following is not meant to be an exhaustive list, but gives the current set of use cases for these metrics.
+
+Metrics requirements for "First Class Resource Isolation and Utilization Features", based on kubernetes component needs, are as follows:
+
+ - Kubelet
+ - Node-level usage metrics for Filesystems, CPU, and Memory
+ - Pod-level usage metrics for Filesystems and Memory
+ - Metrics Server (outlined in [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md)), which exposes the [Resource Metrics API](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-metrics-api.md) to the following system components:
+ - Scheduler
+ - Node-level usage metrics for Filesystems, CPU, and Memory
+ - Pod-level usage metrics for Filesystems, CPU, and Memory
+ - Vertical-Pod-Autoscaler
+ - Node-level usage metrics for Filesystems, CPU, and Memory
+ - Pod-level usage metrics for Filesystems, CPU, and Memory
+ - Container-level usage metrics for Filesystems, CPU, and Memory
+ - Horizontal-Pod-Autoscaler
+ - Node-level usage metrics for CPU and Memory
+ - Pod-level usage metrics for CPU and Memory
+ - Cluster Federation
+ - Node-level usage metrics for Filesystems, CPU, and Memory
+ - kubectl top and Kubernetes Dashboard
+ - Node-level usage metrics for Filesystems, CPU, and Memory
+ - Pod-level usage metrics for Filesystems, CPU, and Memory
+ - Container-level usage metrics for Filesystems, CPU, and Memory
+
+### Proposed Core Metrics:
+This section defines "usage metrics" for filesystems, CPU, and Memory.
+As stated in Non-Goals, this proposal does not attempt to define the specific format by which these are exposed. For convenience, it may be neccessary to include static information such as start time, node capacities for CPU, Memory, or filesystems, and more.
+
+```go
+// CpuUsage holds statistics about the amount of cpu time consumed
+type CpuUsage struct {
+ // The time at which these Metrics were updated.
+ Timestamp metav1.Time
+ // Cumulative CPU usage (sum of all cores) since object creation.
+ CumulativeUsageNanoSeconds *uint64
+}
+
+// MemoryUsage holds statistics about the quantity of memory consumed
+type MemoryUsage struct {
+ // The time at which these metrics were updated.
+ Timestamp metav1.Time
+ // The amount of "working set" memory. This includes recently accessed memory,
+ // dirty memory, and kernel memory.
+ UsageBytes *uint64
+}
+
+// FilesystemUsage holds statistics about the quantity of local storage (e.g. disk) resources consumed
+type FilesystemUsage struct {
+ // The time at which these metrics were updated.
+ Timestamp metav1.Time
+ // StorageIdentifier must uniquely identify the node-level storage resource that is consumed.
+ // It may utilize device, partition, filesystem id, or other identifiers.
+ StorageIdentifier string
+ // UsedBytes represents the disk space consumed, in bytes.
+ UsedBytes *uint64
+ // UsedInodes represents the inodes consumed
+ UsedInodes *uint64
+}
+```
+
+### On-Demand Design
+Interface:
+The interface for exposing these metrics within the kubelet contains methods for fetching each relevant metric. These methods contains a "recency" parameter which specifies how recently the metrics must have been computed. Kubelet components which require very up-to-date metrics (eviction, for example), use very low values. Other components use higher values.
+
+Implementation:
+To keep performance bounded while still offering metrics "On-Demand", all calls to get metrics are cached, and a minimum recency is established to prevent repeated metrics computation. Before computing new metrics, the previous metrics are checked to see if they meet the recency requirements of the caller. If the age of the metrics meet the recency requirements, then the cached metrics are returned. If not, then new metrics are computed and cached.
+
+## Future work
+Suggested, tentative future work, which may be covered by future proposals:
+ - Decide on the format, name, and kubelet endpoint for publishing these metrics.
+ - Integrate with the CRI to allow compatibility with a greater number of runtimes, and to create a better runtime abstraction.
+
+
+<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
+[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/core-metrics-pipeline.md?pixel)]()
+<!-- END MUNGE: GENERATED_ANALYTICS -->
diff --git a/contributors/design-proposals/cri-dockershim-checkpoint.md b/contributors/design-proposals/cri-dockershim-checkpoint.md
new file mode 100644
index 00000000..85db4c89
--- /dev/null
+++ b/contributors/design-proposals/cri-dockershim-checkpoint.md
@@ -0,0 +1,127 @@
+# CRI: Dockershim PodSandbox Checkpoint
+
+## Umbrella Issue
+[#34672](https://github.com/kubernetes/kubernetes/issues/34672)
+
+## Background
+[Container Runtime Interface (CRI)](../devel/container-runtime-interface.md)
+is an ongoing project to allow container runtimes to integrate with
+kubernetes via a newly-defined API.
+[Dockershim](https://github.com/kubernetes/kubernetes/blob/release-1.5/pkg/kubelet/dockershim)
+is the Docker CRI implementation. This proposal aims to introduce
+checkpoint mechanism in dockershim.
+
+## Motivation
+### Why do we need checkpoint?
+
+
+With CRI, Kubelet only passes configurations (SandboxConfig,
+ContainerConfig and ImageSpec) when creating sandbox, container and
+image, and only use the reference id to manage them after creation.
+However, information in configuration is not only needed during creation.
+
+In the case of dockershim with CNI network plugin, CNI plugins needs
+the same information from PodSandboxConfig at creation and deletion.
+
+```
+Kubelet ---------------------------------
+ | RunPodSandbox(PodSandboxConfig)
+ | StopPodSandbox(PodSandboxID)
+ V
+Dockershim-------------------------------
+ | SetUpPod
+ | TearDownPod
+ V
+Network Plugin---------------------------
+ | ADD
+ | DEL
+ V
+CNI plugin-------------------------------
+```
+
+
+In addition, checkpoint helps to improve the reliability of dockershim.
+With checkpoints, critical information for disaster recovery could be
+preserved. Kubelet makes decisions based on the reported pod states
+from runtime shims. Dockershim currently gathers states from docker
+engine. However, in case of disaster, docker engine may lose all
+container information, including the reference ids. Without necessary
+information, kubelet and dockershim could not conduct proper clean up.
+For example, if docker containers are removed underneath kubelet, reference
+to the allocated IPs and iptables setup for the pods are also lost.
+This leads to resource leak and potential iptables rule conflict.
+
+### Why checkpoint in dockershim?
+- CNI specification does not require CNI plugins to be stateful. And CNI
+specification does not provide interface to retrieve states from CNI plugins.
+- Currently there is no uniform checkpoint requirements across existing runtime shims.
+- Need to preserve backward compatibility for kubelet.
+- Easier to maintain backward compatibility by checkpointing at a lower level.
+
+## PodSandbox Checkpoint
+Checkpoint file will be created for each PodSandbox. Files will be
+placed under `/var/lib/dockershim/sandbox/`. File name will be the
+corresponding `PodSandboxID`. File content will be json encoded.
+Data structure is as follows:
+
+```go
+const schemaVersion = "v1"
+
+type Protocol string
+
+// PortMapping is the port mapping configurations of a sandbox.
+type PortMapping struct {
+ // Protocol of the port mapping.
+ Protocol *Protocol `json:"protocol,omitempty"`
+ // Port number within the container.
+ ContainerPort *int32 `json:"container_port,omitempty"`
+ // Port number on the host.
+ HostPort *int32 `json:"host_port,omitempty"`
+}
+
+// CheckpointData contains all types of data that can be stored in the checkpoint.
+type CheckpointData struct {
+ PortMappings []*PortMapping `json:"port_mappings,omitempty"`
+}
+
+// PodSandboxCheckpoint is the checkpoint structure for a sandbox
+type PodSandboxCheckpoint struct {
+ // Version of the pod sandbox checkpoint schema.
+ Version string `json:"version"`
+ // Pod name of the sandbox. Same as the pod name in the PodSpec.
+ Name string `json:"name"`
+ // Pod namespace of the sandbox. Same as the pod namespace in the PodSpec.
+ Namespace string `json:"namespace"`
+ // Data to checkpoint for pod sandbox.
+ Data *CheckpointData `json:"data,omitempty"`
+}
+```
+
+
+## Workflow Changes
+
+
+`RunPodSandbox` creates checkpoint:
+```
+() --> Pull Image --> Create Sandbox Container --> (Create Sandbox Checkpoint) --> Start Sandbox Container --> Set Up Network --> ()
+```
+
+`RemovePodSandbox` removes checkpoint:
+```
+() --> Remove Sandbox --> (Remove Sandbox Checkpoint) --> ()
+```
+
+`ListPodSandbox` need to include all PodSandboxes as long as their
+checkpoint files exist. If sandbox checkpoint exists but sandbox
+container could not be found, the PodSandbox object will include
+PodSandboxID, namespace and name. PodSandbox state will be `PodSandboxState_SANDBOX_NOTREADY`.
+
+`StopPodSandbox` and `RemovePodSandbox` need to conduct proper error handling to ensure idempotency.
+
+
+
+## Future extensions
+This proposal is mainly driven by networking use cases. More could be added into checkpoint.
+
+
+
diff --git a/contributors/design-proposals/custom-metrics-api.md b/contributors/design-proposals/custom-metrics-api.md
new file mode 100644
index 00000000..e01848e3
--- /dev/null
+++ b/contributors/design-proposals/custom-metrics-api.md
@@ -0,0 +1,329 @@
+Custom Metrics API
+==================
+
+The new [metrics monitoring vision](monitoring_architecture.md) proposes
+an API that the Horizontal Pod Autoscaler can use to access arbitrary
+metrics.
+
+Similarly to the [master metrics API](resource-metrics-api.md), the new
+API should be structured around accessing metrics by referring to
+Kubernetes objects (or groups thereof) and a metric name. For this
+reason, the API could be useful for other consumers (most likely
+controllers) that want to consume custom metrics (similarly to how the
+master metrics API is generally useful to multiple cluster components).
+
+The HPA can refer to metrics describing all pods matching a label
+selector, as well as an arbitrary named object.
+
+API Paths
+---------
+
+The root API path will look like `/apis/custom-metrics/v1alpha1`. For
+brevity, this will be left off below.
+
+- `/{object-type}/{object-name}/{metric-name...}`: retrieve the given
+ metric for the given non-namespaced object (e.g. Node, PersistentVolume)
+
+- `/{object-type}/*/{metric-name...}`: retrieve the given metric for all
+ non-namespaced objects of the given type
+
+- `/{object-type}/*/{metric-name...}?labelSelector=foo`: retrieve the
+ given metric for all non-namespaced objects of the given type matching
+ the given label selector
+
+- `/namespaces/{namespace-name}/{object-type}/{object-name}/{metric-name...}`:
+ retrieve the given metric for the given namespaced object
+
+- `/namespaces/{namespace-name}/{object-type}/*/{metric-name...}`: retrieve the given metric for all
+ namespaced objects of the given type
+
+- `/namespaces/{namespace-name}/{object-type}/*/{metric-name...}?labelSelector=foo`: retrieve the given
+ metric for all namespaced objects of the given type matching the
+ given label selector
+
+- `/namespaces/{namespace-name}/metrics/{metric-name}`: retrieve the given
+ metric which describes the given namespace.
+
+For example, to retrieve the custom metric "hits-per-second" for all
+ingress objects matching "app=frontend` in the namespaces "webapp", the
+request might look like:
+
+```
+GET /apis/custom-metrics/v1alpha1/namespaces/webapp/ingress.extensions/*/hits-per-second?labelSelector=app%3Dfrontend`
+
+---
+
+Verb: GET
+Namespace: webapp
+APIGroup: custom-metrics
+APIVersion: v1alpha1
+Resource: ingress.extensions
+Subresource: hits-per-second
+Name: ResourceAll(*)
+```
+
+Notice that getting metrics which describe a namespace follows a slightly
+different pattern from other resources; Since namespaces cannot feasibly
+have unbounded subresource names (due to collision with resource names,
+etc), we introduce a pseudo-resource named "metrics", which represents
+metrics describing namespaces, where the resource name is the metric name:
+
+```
+GET /apis/custom-metrics/v1alpha1/namespaces/webapp/metrics/queue-length
+
+---
+
+Verb: GET
+Namespace: webapp
+APIGroup: custom-metrics
+APIVersion: v1alpha1
+Resource: metrics
+Name: queue-length
+```
+
+NB: the branch-node LIST operations (e.g. `LIST
+/apis/custom-metrics/v1alpha1/namespaces/webapp/pods/`) are unsupported in
+v1alpha1. They may be defined in a later version of the API.
+
+API Path Design, Discovery, and Authorization
+---------------------------------------------
+
+The API paths in this proposal are designed to a) resemble normal
+Kubernetes APIs, b) facilitate writing authorization rules, and c)
+allow for discovery.
+
+Since the API structure follows the same structure as other Kubernetes
+APIs, it allows for fine grained control over access to metrics. Access
+can be controlled on a per-metric basic (each metric is a subresource, so
+metrics may be whitelisted by allowing access to a particular
+resource-subresource pair), or granted in general for a namespace (by
+allowing access to any resource in the `custom-metrics` API group).
+
+Similarly, since metrics are simply subresources, a normal Kubernetes API
+discovery document can be published by the adapter's API server, allowing
+clients to discover the available metrics.
+
+Note that we introduce the syntax of having a name of ` * ` here since
+there is no current syntax for getting the output of a subresource on
+multiple objects.
+
+API Objects
+-----------
+
+The request URLs listed above will return the `MetricValueList` type described
+below (when a name is given that is not ` * `, the API should simply return a
+list with a single element):
+
+```go
+
+// a list of values for a given metric for some set of objects
+type MetricValueList struct {
+ metav1.TypeMeta`json:",inline"`
+ metav1.ListMeta`json:"metadata,omitempty"`
+
+ // the value of the metric across the described objects
+ Items []MetricValue `json:"items"`
+}
+
+// a metric value for some object
+type MetricValue struct {
+ metav1.TypeMeta`json:",inline"`
+
+ // a reference to the described object
+ DescribedObject ObjectReference `json:"describedObject"`
+
+ // the name of the metric
+ MetricName string `json:"metricName"`
+
+ // indicates the time at which the metrics were produced
+ Timestamp unversioned.Time `json:"timestamp"`
+
+ // indicates the window ([Timestamp-Window, Timestamp]) from
+ // which these metrics were calculated, when returning rate
+ // metrics calculated from cumulative metrics (or zero for
+ // non-calculated instantaneous metrics).
+ WindowSeconds *int64 `json:"window,omitempty"`
+
+ // the value of the metric for this
+ Value resource.Quantity
+}
+```
+
+For instance, the example request above would yield the following object:
+
+```json
+{
+ "kind": "MetricValueList",
+ "apiVersion": "custom-metrics/v1alpha1",
+ "items": [
+ {
+ "metricName": "hits-per-second",
+ "describedObject": {
+ "kind": "Ingress",
+ "apiVersion": "extensions",
+ "name": "server1",
+ "namespace": "webapp"
+ },
+ "timestamp": SOME_TIMESTAMP_HERE,
+ "windowSeconds": "10",
+ "value": "10"
+ },
+ {
+ "metricName": "hits-per-second",
+ "describedObject": {
+ "kind": "Ingress",
+ "apiVersion": "extensions",
+ "name": "server2",
+ "namespace": "webapp"
+ },
+ "timestamp": ANOTHER_TIMESTAMP_HERE,
+ "windowSeconds": "10",
+ "value": "15"
+ }
+ ]
+}
+```
+
+Semantics
+---------
+
+### Object Types ###
+
+In order to properly identify resources, we must use resource names
+qualified with group names (since the group for the requests will always
+be `custom-metrics`).
+
+The `object-type` parameter should be the string form of
+`unversioned.GroupResource`. Note that we do not include version in this;
+we simply wish to uniquely identify all the different types of objects in
+Kubernetes. For example, the pods resource (which exists in the un-named
+legacy API group) would be represented simply as `pods`, while the jobs
+resource (which exists in the `batch` API group) would be represented as
+`jobs.batch`.
+
+In the case of cross-group object renames, the adapter should maintain
+a list of "equivalent versions" that the monitoring system uses. This is
+monitoring-system dependent (for instance, the monitoring system might
+record all HorizontalPodAutoscalers as in `autoscaling`, but should be
+aware that HorizontalPodAutoscaler also exist in `extensions`).
+
+Note that for namespace metrics, we use a pseudo-resource called
+`metrics`. Since there is no resource in the legacy API group, this will
+not clash with any existing resources.
+
+### Metric Names ###
+
+Metric names must be able to appear as a single subresource. In particular,
+metric names, *as passed to the API*, may not contain the characters '%', '/',
+or '?', and may not be named '.' or '..' (but may contain these sequences).
+Note, specifically, that URL encoding is not acceptable to escape the forbidden
+characters, due to issues in the Go URL handling libraries. Otherwise, metric
+names are open-ended.
+
+### Metric Values and Timing ###
+
+There should be only one metric value per object requested. The returned
+metrics should be the most recently available metrics, as with the resource
+metrics API. Implementers *should* attempt to return all metrics with roughly
+identical timestamps and windows (when appropriate), but consumers should also
+verify that any differences in timestamps are within tolerances for
+a particular application (e.g. a dashboard might simply display the older
+metric with a note, while the horizontal pod autoscaler controller might choose
+to pretend it did not receive that metric value).
+
+### Labeled Metrics (or lack thereof) ###
+
+For metrics systems that support differentiating metrics beyond the
+Kubernetes object hierarchy (such as using additional labels), the metrics
+systems should have a metric which represents all such series aggregated
+together. Additionally, implementors may choose to identify the individual
+"sub-metrics" via the metric name, but this is expected to be fairly rare,
+since it most likely requires specific knowledge of individual metrics.
+For instance, suppose we record filesystem usage by filesystem inside the
+container. There should then be a metric `filesystem/usage`, and the
+implementors of the API may choose to expose more detailed metrics like
+`filesystem/usage/my-first-filesystem`.
+
+### Resource Versions ###
+
+API implementors should set the `resourceVersion` field based on the
+scrape time of the metric. The resource version is expected to increment
+when the scrape/collection time of the returned metric changes. While the
+API does not support writes, and does not currently support watches,
+populating resource version preserves the normal expected Kubernetes API
+semantics.
+
+Relationship to HPA v2
+----------------------
+
+The URL paths in this API are designed to correspond to different source
+types in the [HPA v2](hpa-v2.md). Specifially, the `pods` source type
+corresponds to a URL of the form
+`/namespaces/$NS/pods/*/$METRIC_NAME?labelSelector=foo`, while the
+`object` source type corresponds to a URL of the form
+`/namespaces/$NS/$RESOURCE.$GROUP/$OBJECT_NAME/$METRIC_NAME`.
+
+The HPA then takes the results, aggregates them together (in the case of
+the former source type), and uses the resulting value to produce a usage
+ratio.
+
+The resource source type is taken from the API provided by the
+"metrics" API group (the master/resource metrics API).
+
+The HPA will consume the API as a federated API server.
+
+Relationship to Resource Metrics API
+------------------------------------
+
+The metrics presented by this API may be a superset of those present in the
+resource metrics API, but this is not guaranteed. Clients that need the
+information in the resource metrics API should use that to retrieve those
+metrics, and supplement those metrics with this API.
+
+Mechanical Concerns
+-------------------
+
+This API is intended to be implemented by monitoring pipelines (e.g.
+inside Heapster, or as an adapter on top of a solution like Prometheus).
+It shares many mechanical requirements with normal Kubernetes APIs, such
+as the need to support encoding different versions of objects in both JSON
+and protobuf, as well as acting as a discoverable API server. For these
+reasons, it is expected that implemenators will make use of the Kubernetes
+genericapiserver code. If implementors choose not to use this, they must
+still follow all of the Kubernetes API server conventions in order to work
+properly with consumers of the API.
+
+Specifically, they must support the semantics of the GET verb in
+Kubernetes, including outputting in different API versions and formats as
+requested by the client. They must support integrating with API discovery
+(including publishing a discovery document, etc).
+
+Location
+--------
+
+The types and clients for this API will live in a separate repository
+under the Kubernetes organization (e.g. `kubernetes/metrics`). This
+repository will most likely also house other metrics-related APIs for
+Kubernetes (e.g. historical metrics API definitions, the resource metrics
+API definitions, etc).
+
+Note that there will not be a canonical implemenation of the custom
+metrics API under Kubernetes, just the types and clients. Implementations
+will be left up to the monitoring pipelines.
+
+Alternative Considerations
+--------------------------
+
+### Quantity vs Float ###
+
+In the past, custom metrics were represented as floats. In general,
+however, Kubernetes APIs are not supposed to use floats. The API proposed
+above thus uses `resource.Quantity`. This adds a bit of encoding
+overhead, but makes the API line up nicely with other Kubernetes APIs.
+
+### Labeled Metrics ###
+
+Many metric systems support labeled metrics, allowing for dimenisionality
+beyond the Kubernetes object hierarchy. Since the HPA currently doesn't
+support specifying metric labels, this is not supported via this API. We
+may wish to explore this in the future.
diff --git a/contributors/design-proposals/daemon.md b/contributors/design-proposals/daemon.md
index 2c306056..58735728 100644
--- a/contributors/design-proposals/daemon.md
+++ b/contributors/design-proposals/daemon.md
@@ -46,7 +46,7 @@ For other uses, see the related [feature request](https://issues.k8s.io/1518)
The DaemonSet supports standard API features:
- create
- The spec for DaemonSets has a pod template field.
- - Using the pod’s nodeSelector field, DaemonSets can be restricted to operate
+ - Using the pod's nodeSelector field, DaemonSets can be restricted to operate
over nodes that have a certain label. For example, suppose that in a cluster
some nodes are labeled ‘app=database’. You can use a DaemonSet to launch a
datastore pod on exactly those nodes labeled ‘app=database’.
@@ -81,7 +81,7 @@ nodes, preempting other pods if necessary.
nodeSelector:
app: datastore-node
containers:
- name: datastore-shard
+ - name: datastore-shard
image: kubernetes/sharded
ports:
- containerPort: 9042
@@ -118,7 +118,7 @@ replica of the daemon pod on the node.
- When a new node is added to the cluster, the DaemonSet controller starts
daemon pods on the node for DaemonSets whose pod template nodeSelectors match
-the node’s labels.
+the node's labels.
- Suppose the user launches a DaemonSet that runs a logging daemon on all
nodes labeled “logger=fluentd”. If the user then adds the “logger=fluentd” label
to a node (that did not initially have the label), the logging daemon will
@@ -179,7 +179,7 @@ expapi/v1/register.go
#### Daemon Manager
- Creates new DaemonSets when requested. Launches the corresponding daemon pod
-on all nodes with labels matching the new DaemonSet’s selector.
+on all nodes with labels matching the new DaemonSet's selector.
- Listens for addition of new nodes to the cluster, by setting up a
framework.NewInformer that watches for the creation of Node API objects. When a
new node is added, the daemon manager will loop through each DaemonSet. If the
@@ -193,7 +193,7 @@ via its hostname.)
- Does not need to be modified, but health checking will occur for the daemon
pods and revive the pods if they are killed (we set the pod restartPolicy to
-Always). We reject DaemonSet objects with pod templates that don’t have
+Always). We reject DaemonSet objects with pod templates that don't have
restartPolicy set to Always.
## Open Issues
diff --git a/contributors/design-proposals/daemonset-update.md b/contributors/design-proposals/daemonset-update.md
new file mode 100644
index 00000000..a6cc98cb
--- /dev/null
+++ b/contributors/design-proposals/daemonset-update.md
@@ -0,0 +1,315 @@
+# DaemonSet Updates
+
+**Author**: @madhusudancs, @lukaszo, @janetkuo
+
+**Status**: Proposal
+
+## Abstract
+
+A proposal for adding the update feature to `DaemonSet`. This feature will be
+implemented on server side (in `DaemonSet` API).
+
+Users already can update a `DaemonSet` today (Kubernetes release 1.5), which will
+not cause changes to its subsequent pods, until those pods are killed. In this
+proposal, we plan to add a "RollingUpdate" strategy which allows DaemonSet to
+downstream its changes to pods.
+
+## Requirements
+
+In this proposal, we design DaemonSet updates based on the following requirements:
+
+- Users can trigger a rolling update of DaemonSet at a controlled speed, which
+ is achieved by:
+ - Only a certain number of DaemonSet pods can be down at the same time during
+ an update
+ - A DaemonSet pod needs to be ready for a specific amount of time before it's
+ considered up
+- Users can monitor the status of a DaemonSet update (e.g. the number of pods
+ that are updated and healthy)
+- A broken DaemonSet update should not continue, but one can still update the
+ DaemonSet again to fix it
+- Users should be able to update a DaemonSet even during an ongoing DaemonSet
+ upgrade -- in other words, rollover (e.g. update the DaemonSet to fix a broken
+ DaemonSet update)
+
+Here are some potential requirements that haven't been covered by this proposal:
+
+- Users should be able to view the history of previous DaemonSet updates
+- Users can figure out the revision of a DaemonSet's pod (e.g. which version is
+ this DaemonSet pod?)
+- DaemonSet should provide at-most-one guarantee per node (i.e. at most one pod
+ from a DaemonSet can exist on a node at any time)
+- Uptime is critical for each pod of a DaemonSet during an upgrade (e.g. the time
+ from a DaemonSet pods being killed to recreated and healthy should be < 5s)
+- Each DaemonSet pod can still fit on the node after being updated
+- Some DaemonSets require the node to be drained before the DeamonSet's pod on it
+ is updated (e.g. logging daemons)
+- DaemonSet's pods are implicitly given higher priority than non-daemons
+- DaemonSets can only be operated by admins (i.e. people who manage nodes)
+ - This is required if we allow DaemonSet controllers to drain, cordon,
+ uncordon nodes, evict pods, or allow DaemonSet pods to have higher priority
+
+## Implementation
+
+### API Object
+
+To enable DaemonSet upgrades, `DaemonSet` related API object will have the following
+changes:
+
+```go
+type DaemonSetUpdateStrategy struct {
+ // Type of daemon set update. Can be "RollingUpdate" or "OnDelete".
+ // Default is OnDelete.
+ Type DaemonSetUpdateStrategyType
+
+ // Rolling update config params. Present only if DaemonSetUpdateStrategy =
+ // RollingUpdate.
+ //---
+ // TODO: Update this to follow our convention for oneOf, whatever we decide it
+ // to be. Same as DeploymentStrategy.RollingUpdate.
+ // See https://github.com/kubernetes/kubernetes/issues/35345
+ RollingUpdate *RollingUpdateDaemonSet
+}
+
+type DaemonSetUpdateStrategyType string
+
+const (
+ // Replace the old daemons by new ones using rolling update i.e replace them on each node one after the other.
+ RollingUpdateDaemonSetStrategyType DaemonSetUpdateStrategyType = "RollingUpdate"
+
+ // Replace the old daemons only when it's killed
+ OnDeleteDaemonSetStrategyType DaemonSetUpdateStrategyType = "OnDelete"
+)
+
+// Spec to control the desired behavior of daemon set rolling update.
+type RollingUpdateDaemonSet struct {
+ // The maximum number of DaemonSet pods that can be unavailable during
+ // the update. Value can be an absolute number (ex: 5) or a percentage of total
+ // number of DaemonSet pods at the start of the update (ex: 10%). Absolute
+ // number is calculated from percentage by rounding up.
+ // This must be greater than 0.
+ // Default value is 1.
+ // Example: when this is set to 30%, 30% of the currently running DaemonSet
+ // pods can be stopped for an update at any given time. The update starts
+ // by stopping at most 30% of the currently running DaemonSet pods and then
+ // brings up new DaemonSet pods in their place. Once the new pods are ready,
+ // it then proceeds onto other DaemonSet pods, thus ensuring that at least
+ // 70% of original number of DaemonSet pods are available at all times
+ // during the update.
+ MaxUnavailable intstr.IntOrString
+}
+
+// DaemonSetSpec is the specification of a daemon set.
+type DaemonSetSpec struct {
+ // Note: Existing fields, including Selector and Template are ommitted in
+ // this proposal.
+
+ // Update strategy to replace existing DaemonSet pods with new pods.
+ UpdateStrategy DaemonSetUpdateStrategy `json:"updateStrategy,omitempty"`
+
+ // Minimum number of seconds for which a newly created DaemonSet pod should
+ // be ready without any of its container crashing, for it to be considered
+ // available. Defaults to 0 (pod will be considered available as soon as it
+ // is ready).
+ MinReadySeconds int32 `json:"minReadySeconds,omitempty"`
+}
+
+const (
+ // DefaultDaemonSetUniqueLabelKey is the default key of the labels that is added
+ // to daemon set pods to distinguish between old and new pod templates during
+ // DaemonSet update.
+ DefaultDaemonSetUniqueLabelKey string = "pod-template-hash"
+)
+
+// DaemonSetStatus represents the current status of a daemon set.
+type DaemonSetStatus struct {
+ // Note: Existing fields, including CurrentNumberScheduled, NumberMissscheduled,
+ // DesiredNumberScheduled, NumberReady, and ObservedGeneration are ommitted in
+ // this proposal.
+
+ // UpdatedNumberScheduled is the total number of nodes that are running updated
+ // daemon pod
+ UpdatedNumberScheduled int32 `json:"updatedNumberScheduled"`
+
+ // NumberAvailable is the number of nodes that should be running the
+ // daemon pod and have one or more of the daemon pod running and
+ // available (ready for at least minReadySeconds)
+ NumberAvailable int32 `json:"numberAvailable"`
+
+ // NumberUnavailable is the number of nodes that should be running the
+ // daemon pod and have non of the daemon pod running and available
+ // (ready for at least minReadySeconds)
+ NumberUnavailable int32 `json:"numberUnavailable"`
+}
+```
+
+### Controller
+
+#### DaemonSet Controller
+
+The DaemonSet Controller will make DaemonSet updates happen. It will watch
+DaemonSets on the apiserver.
+
+For each pending DaemonSet updates, it will:
+
+1. Find all pods whose labels are matched by DaemonSet `.spec.selector`.
+ - If `OwnerReference` is implemented for DaemonSets, filter out pods that
+ aren't controlled by this DaemonSet too.
+1. Check `DaemonSetUpdateStrategy`:
+ - If `OnDelete`: do nothing
+ - If `RollingUpdate`:
+ - Compare spec of the daemon pods from step 1 with DaemonSet
+ `.spec.template.spec` to see if DaemonSet spec has changed.
+ - If DaemonSet spec has changed, compare `MaxUnavailable` with DaemonSet
+ `.status.numberUnavailable` to see how many old daemon pods can be
+ killed. Then, kill those pods in the order that unhealthy pods (failed,
+ pending, not ready) are killed first.
+1. Find all nodes that should run these pods created by this DaemonSet.
+1. Create daemon pods on nodes when they should have those pods running but not
+ yet. Otherwise, delete running daemon pods that shouldn't be running on nodes.
+1. Cleanup, update DaemonSet status
+ - `.status.numberAvailable` = the total number of DaemonSet pods that have
+ become `Ready` for `MinReadySeconds`
+ - `.status.numberUnavailable` = `.status.desiredNumberScheduled` -
+ `.status.numberAvailable`
+
+If DaemonSet Controller crashes during an update, it can still recover.
+
+### kubectl
+
+#### kubectl rollout
+
+Users can use `kubectl rollout` to monitor DaemonSet updates:
+
+- `kubectl rollout status daemonset/<DaemonSet-Name>`: to see the DaemonSet
+ upgrade status
+
+## Updating DaemonSets mid-way
+
+Users can update an updated DaemonSet before its rollout completes.
+In this case, the existing daemon pods will not continue rolling out and the new
+one will begin rolling out.
+
+
+## Deleting DaemonSets
+
+Deleting a DaemonSet (with cascading) will delete all its pods.
+
+
+## DaemonSet Strategies
+
+DaemonSetStrategy specifies how the new daemon pods should replace existing ones.
+To begin with, we will support 2 types:
+
+* On delete: Do nothing, until existing daemon pods are killed (for backward
+ compatibility).
+ - Other alternative names: No-op, External
+* Rolling update: We gradually kill existing ones while creating the new one.
+
+
+## Tests
+
+- Updating a RollingUpdate DaemonSet will trigger updates to its daemon pods.
+- Updating an OnDelete DaemonSet will not trigger updates, until the pods are
+ killed.
+- Users can use node labels to choose which nodes this DaemonSet should target.
+ DaemonSet updates only affect pods on those nodes.
+ - For example, some nodes may be running manifest pods, and other nodes will
+ be running daemon pods
+- DaemonSets can be updated while already being updated (i.e. rollover updates)
+- Broken rollout can be rolled back (by applying old config)
+- If a daemon pod can no long fit on the node after rolling update, the users
+ can manually evict or delete other pods on the node to make room for the
+ daemon pod, and the DaemonSet rollout will eventually succeed (DaemonSet
+ controller will recreate the failed daemon pod if it can't be scheduled)
+
+
+## Future Plans
+
+In the future, we may:
+
+- Implement at-most-one and/or at-least-one guarantees for DaemonSets (i.e. at
+ most/at least one pod from a DaemonSet can exist on a node at any time)
+ - At-most-one would use a deterministic name for the pod (e.g. use node name
+ as daemon pod name suffix)
+- Support use cases where uptime is critical for each pod of a DaemonSet during
+ an upgrade
+ - One approach is to use dummy pods to pre-pull images to reduce down time
+- Support use cases that each DaemonSet pod can still fit on the node after
+ being updated (unless it becomes larger than the node). Some possible
+ approaches include:
+ - Make DaemonSet pods (daemons) have higher priority than non-daemons, and
+ kubelet will evict pods with lower priority to make room for higher priority
+ ones
+ - The DaemonSet controller will evict pods when daemons can't fit on the node
+ - The DaemonSet controller will cordon the node before upgrading the daemon on
+ it, and uncordon the node once it's done
+- Support use cases that require the node to be drained before the daemons on it
+ can updated (e.g. logging daemons)
+ - The DaemonSet controller will drain the node before upgrading the daemon on
+ it, and uncordon the node once it's done
+- Make DaemonSets admin-only resources (admin = people who manage nodes). Some
+ possible approaches include:
+ - Remove namespace from DaemonSets (DaemonSets become node-level resources)
+ - Modify RBAC bootstrap policy to make DaemonSets admin-only
+ - Delegation or impersonation
+- Support more DaemonSet update strategies
+- Allow user-defined DaemonSet unique label key
+- Support pausing DaemonSet rolling update
+- Support auto-rollback DaemonSets
+
+### DaemonSet History
+
+In the future, we will support DaemonSet history, so that users can view
+previous revisions and roll back when necessary.
+
+One possible approach is to create a new resource called `DaemonSetRevision` to
+store DaemonSet history.
+
+Another way to implement DaemonSet history is through creating `PodTemplates` as
+snapshots of DaemonSet templates, and then create them in DaemonSet controller:
+
+- Find existing PodTemplates whose labels are matched by DaemonSet
+ `.spec.selector`
+ - Sort those PodTemplates by creation timestamp and only retain at most
+ `.spec.revisionHistoryLimit` latest PodTemplates (remove the rest)
+ - Find the PodTemplate whose `.template` is the same as DaemonSet
+ `.spec.template`. If not found, create a new PodTemplate from DaemonSet
+ `.spec.template`:
+ - The name will be `<DaemonSet-Name>-<Hash-of-pod-template>`
+ - PodTemplate `.metadata.labels` will have a "pod-template-hash" label,
+ value be the hash of PodTemplate `.template` (note: don't include the
+ "pod-template-hash" label when calculating hash)
+ - PodTemplate `.metadata.annotations` will be copied from DaemonSet
+ `.metadata.annotations`
+
+Note that when the DaemonSet controller creates pods, those pods will be created
+with the "pod-template-hash" label.
+
+PodTemplate may need to be made an admin-only or read only resource if it's used
+to store DaemonSet history.
+
+#### History Clean Up Policy
+
+We also need to specify a clean up policy for the number of history to keep in
+the system, and keep only limited number of history by default (for example, 3).
+
+#### kubectl
+
+`kubectl` should also support DaemonSet history:
+
+- `kubectl rollout undo daemonset/<DaemonSet-Name>` to roll back
+- `kubectl rollout history daemonset/<DaemonSet-Name>`: to view the history of
+ DaemonSet updates
+
+#### API
+
+Implement a subresource for DaemonSet history (e.g. `daemonsets/foo/history`)
+that summarizes the information in the history.
+
+### Tests
+
+- DaemonSet should support at most one daemon pod per node guarantee.
+ - Adding or deleting nodes won't break that.
+- Users should be able to specify acceptable downtime of their daemon pods, and
+ DaemonSet updates should respect that.
diff --git a/contributors/design-proposals/deploy.md b/contributors/design-proposals/deploy.md
index a27fb01f..8185757d 100644
--- a/contributors/design-proposals/deploy.md
+++ b/contributors/design-proposals/deploy.md
@@ -121,7 +121,7 @@ See issue [#17164](https://github.com/kubernetes/kubernetes/issues/17164).
We store previous deployment version information in annotations `rollout.kubectl.kubernetes.io/change-source` and `rollout.kubectl.kubernetes.io/version` of replication controllers of the deployment, to support rolling back changes as well as for the users to view previous changes with `kubectl rollout history`.
- `rollout.kubectl.kubernetes.io/change-source`, which is optional, records the kubectl command of the last mutation made to this rollout. Users may use `--record` in `kubectl` to record current command in this annotation.
- `rollout.kubectl.kubernetes.io/version` records a version number to distinguish the change sequence of a deployment's
-replication controllers. A deployment obtains the largest version number from its replication controllers and increments the number by 1 upon update or creation of the deployment, and update the version annotation of its new replication controller.
+replication controllers. A deployment obtains the largest version number from its replication controllers and increments the number by 1 upon update or creation of the deployment, and updates the version annotation of its new replication controller.
When the users perform a rollback, i.e. `kubectl rollout undo`, the deployment first looks at its existing replication controllers, regardless of their number of replicas. Then it finds the one with annotation `rollout.kubectl.kubernetes.io/version` that either contains the specified rollback version number or contains the second largest version number among all the replication controllers (current new replication controller should obtain the largest version number) if the user didn't specify any version number (the user wants to rollback to the last change). Lastly, it
starts scaling up that replication controller it's rolling back to, and scaling down the current ones, and then update the version counter and the rollout annotations accordingly.
diff --git a/contributors/design-proposals/disk-accounting.md b/contributors/design-proposals/disk-accounting.md
index 19782356..d11c7a6f 100755
--- a/contributors/design-proposals/disk-accounting.md
+++ b/contributors/design-proposals/disk-accounting.md
@@ -8,7 +8,7 @@ This proposal is an attempt to come up with a means for accounting disk usage in
### Why is disk accounting necessary?
-As of kubernetes v1.1 clusters become unusable over time due to the local disk becoming full. The kubelets on the node attempt to perform garbage collection of old containers and images, but that doesn’t prevent running pods from using up all the available disk space.
+As of kubernetes v1.1 clusters become unusable over time due to the local disk becoming full. The kubelets on the node attempt to perform garbage collection of old containers and images, but that doesn't prevent running pods from using up all the available disk space.
Kubernetes users have no insight into how the disk is being consumed.
@@ -42,13 +42,13 @@ Disk can be consumed for:
1. Container images
-2. Container’s writable layer
+2. Container's writable layer
-3. Container’s logs - when written to stdout/stderr and default logging backend in docker is used.
+3. Container's logs - when written to stdout/stderr and default logging backend in docker is used.
4. Local volumes - hostPath, emptyDir, gitRepo, etc.
-As of Kubernetes v1.1, kubelet exposes disk usage for the entire node and the container’s writable layer for aufs docker storage driver.
+As of Kubernetes v1.1, kubelet exposes disk usage for the entire node and the container's writable layer for aufs docker storage driver.
This information is made available to end users via the heapster monitoring pipeline.
#### Image layers
@@ -86,7 +86,7 @@ In addition to this, the changes introduced by a pod on the source of a hostPath
### Docker storage model
-Before we start exploring solutions, let’s get familiar with how docker handles storage for images, writable layer and logs.
+Before we start exploring solutions, let's get familiar with how docker handles storage for images, writable layer and logs.
On all storage drivers, logs are stored under `<docker root dir>/containers/<container-id>/`
@@ -123,7 +123,7 @@ Everything under `/var/lib/docker/overlay/<id>` are files required for running
Disk accounting is dependent on the storage driver in docker. A common solution that works across all storage drivers isn't available.
-I’m listing a few possible solutions for disk accounting below along with their limitations.
+I'm listing a few possible solutions for disk accounting below along with their limitations.
We need a plugin model for disk accounting. Some storage drivers in docker will require special plugins.
@@ -136,7 +136,7 @@ But isolated usage isn't of much use because image layers are shared between con
Continuing to use the entire partition availability for garbage collection purposes in kubelet, should not affect reliability.
We might garbage collect more often.
-As long as we do not expose features that require persisting old containers, computing image layer usage wouldn’t be necessary.
+As long as we do not expose features that require persisting old containers, computing image layer usage wouldn't be necessary.
Main goals for images are
1. Capturing total image disk usage
@@ -181,7 +181,7 @@ Another option is to expose disk usage of all images together as a first-class f
##### Overlayfs and Aufs
-####### `du`
+###### `du`
We can list all the image layer specific directories, excluding container directories, and run `du` on each of those directories.
@@ -200,7 +200,7 @@ We can list all the image layer specific directories, excluding container direct
* Can block container deletion by keeping file descriptors open.
-####### Linux gid based Disk Quota
+###### Linux gid based Disk Quota
[Disk quota](https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Storage_Administration_Guide/ch-disk-quotas.html) feature provided by the linux kernel can be used to track the usage of image layers. Ideally, we need `project` support for disk quota, which lets us track usage of directory hierarchies using `project ids`. Unfortunately, that feature is only available for zfs filesystems. Since most of our distributions use `ext4` by default, we will have to use either `uid` or `gid` based quota tracking.
@@ -208,7 +208,7 @@ Both `uids` and `gids` are meant for security. Overloading that concept for disk
Kubelet needs to define a gid for tracking image layers and make that gid or group the owner of `/var/lib/docker/[aufs | overlayfs]` recursively. Once this is done, the quota sub-system in the kernel will report the blocks being consumed by the storage driver on the underlying partition.
-Since this number also includes the container’s writable layer, we will have to somehow subtract that usage from the overall usage of the storage driver directory. Luckily, we can use the same mechanism for tracking container’s writable layer. Once we apply a different `gid` to the container’s writable layer, which is located under `/var/lib/docker/<storage_driver>/diff/<container_id>`, the quota subsystem will not include the container’s writable layer usage.
+Since this number also includes the container's writable layer, we will have to somehow subtract that usage from the overall usage of the storage driver directory. Luckily, we can use the same mechanism for tracking container’s writable layer. Once we apply a different `gid` to the container's writable layer, which is located under `/var/lib/docker/<storage_driver>/diff/<container_id>`, the quota subsystem will not include the container's writable layer usage.
Xfs on the other hand support project quota which lets us track disk usage of arbitrary directories using a project. Support for this feature in ext4 is being reviewed. So on xfs, we can use quota without having to clobber the writable layer's uid and gid.
@@ -219,7 +219,7 @@ Xfs on the other hand support project quota which lets us track disk usage of ar
**Cons**
-* Requires updates to default ownership on docker’s internal storage driver directories. We will have to deal with storage driver implementation details in any approach that is not docker native.
+* Requires updates to default ownership on docker's internal storage driver directories. We will have to deal with storage driver implementation details in any approach that is not docker native.
* Requires additional node configuration - quota subsystem needs to be setup on the node. This can either be automated or made a requirement for the node.
@@ -238,11 +238,11 @@ Project Quota support for ext4 is currently being reviewed upstream. If that fea
Devicemapper storage driver will setup two volumes, metadata and data, that will be used to store image layers and container writable layer. The volumes can be real devices or loopback. A Pool device is created which uses the underlying volume for real storage.
-A new thinly-provisioned volume, based on the pool, will be created for running container’s.
+A new thinly-provisioned volume, based on the pool, will be created for running container's.
-The kernel tracks the usage of the pool device at the block device layer. The usage here includes image layers and container’s writable layers.
+The kernel tracks the usage of the pool device at the block device layer. The usage here includes image layers and container's writable layers.
-Since the kubelet has to track the writable layer usage anyways, we can subtract the aggregated root filesystem usage from the overall pool device usage to get the image layer’s disk usage.
+Since the kubelet has to track the writable layer usage anyways, we can subtract the aggregated root filesystem usage from the overall pool device usage to get the image layer's disk usage.
Linux quota and `du` will not work with device mapper.
@@ -253,7 +253,7 @@ A docker dry run option (mentioned above) is another possibility.
###### Overlayfs / Aufs
-Docker creates a separate directory for the container’s writable layer which is then overlayed on top of read-only image layers.
+Docker creates a separate directory for the container's writable layer which is then overlayed on top of read-only image layers.
Both the previously mentioned options of `du` and `Linux Quota` will work for this case as well.
@@ -268,14 +268,14 @@ If local disk becomes a schedulable resource, `linux quota` can be used to impos
FIXME: How to calculate writable layer usage with devicemapper?
-To enforce `limits` the volume created for the container’s writable layer filesystem can be dynamically [resized](https://jpetazzo.github.io/2014/01/29/docker-device-mapper-resize/), to not use more than `limit`. `request` will have to be enforced by the kubelet.
+To enforce `limits` the volume created for the container's writable layer filesystem can be dynamically [resized](https://jpetazzo.github.io/2014/01/29/docker-device-mapper-resize/), to not use more than `limit`. `request` will have to be enforced by the kubelet.
#### Container logs
Container logs are not storage driver specific. We can use either `du` or `quota` to track log usage per container. Log files are stored under `/var/lib/docker/containers/<container-id>`.
-In the case of quota, we can create a separate gid for tracking log usage. This will let users track log usage and writable layer’s usage individually.
+In the case of quota, we can create a separate gid for tracking log usage. This will let users track log usage and writable layer's usage individually.
For the purposes of enforcing limits though, kubelet will use the sum of logs and writable layer.
@@ -340,9 +340,9 @@ In this milestone, we will add support for quota and make it opt-in. There shoul
* Configure linux quota automatically on startup. Do not set any limits in this phase.
-* Allocate gids for pod volumes, container’s writable layer and logs, and also for image layers.
+* Allocate gids for pod volumes, container's writable layer and logs, and also for image layers.
-* Update the docker runtime plugin in kubelet to perform the necessary `chown’s` and `chmod’s` between container creation and startup.
+* Update the docker runtime plugin in kubelet to perform the necessary `chown's` and `chmod's` between container creation and startup.
* Pass the allocated gids as supplementary gids to containers.
@@ -363,7 +363,7 @@ In this milestone, we will make local disk a schedulable resource.
* Quota plugin sets hard limits equal to user specified `limits`.
-* Devicemapper plugin resizes writable layer to not exceed the container’s disk `limit`.
+* Devicemapper plugin resizes writable layer to not exceed the container's disk `limit`.
* Disk manager evicts pods based on `usage` - `request` delta instead of just QoS class.
@@ -417,15 +417,12 @@ Tested on Debian jessie
8. Check usage using quota and group ‘x’
- ```shell
- $ quota -g x -v
-
- Disk quotas for group x (gid 9000):
-
- Filesystem **blocks** quota limit grace files quota limit grace
-
- /dev/sda1 **10248** 0 0 3 0 0
- ```
+ ```shell
+ $ quota -g x -v
+ Disk quotas for group x (gid 9000):
+ Filesystem blocks quota limit grace files quota limit grace
+ /dev/sda1 10248 0 0 3 0 0
+ ```
Using the same workflow, we can add new sticky group IDs to emptyDir volumes and account for their usage against pods.
@@ -451,7 +448,7 @@ Track the space occupied by images after it has been pulled locally as follows.
3. Any new images pulled or containers created will be accounted to the `docker-images` group by default.
-4. Once we update the group ownership on newly created containers to a different gid, the container writable layer’s specific disk usage gets dropped from this group.
+4. Once we update the group ownership on newly created containers to a different gid, the container writable layer's specific disk usage gets dropped from this group.
#### Overlayfs
@@ -484,29 +481,24 @@ Overlayfs works similar to Aufs. The path to the writable directory for containe
* Check quota before and after running the container.
```shell
- $ quota -g x -v
-
- Disk quotas for group x (gid 9000):
-
- Filesystem blocks quota limit grace files quota limit grace
-
- /dev/sda1 48 0 0 19 0 0
- ```
+ $ quota -g x -v
+ Disk quotas for group x (gid 9000):
+ Filesystem blocks quota limit grace files quota limit grace
+ /dev/sda1 48 0 0 19 0 0
+ ```
* Start the docker container
* `docker start b8`
- * ```shell
- quota -g x -v
-
- Disk quotas for group x (gid 9000):
-
- Filesystem **blocks** quota limit grace files quota limit grace
+ Notice the **blocks** has changed
- /dev/sda1 **10288** 0 0 20 0 0
-
- ```
+ ```sh
+ $ quota -g x -v
+ Disk quotas for group x (gid 9000):
+ Filesystem blocks quota limit grace files quota limit grace
+ /dev/sda1 10288 0 0 20 0 0
+ ```
##### Device mapper
@@ -518,60 +510,41 @@ These devices can be loopback or real storage devices.
The base device has a maximum storage capacity. This means that the sum total of storage space occupied by images and containers cannot exceed this capacity.
-By default, all images and containers are created from an initial filesystem with a 10GB limit.
+By default, all images and containers are created from an initial filesystem with a 10GB limit.
A separate filesystem is created for each container as part of start (not create).
It is possible to [resize](https://jpetazzo.github.io/2014/01/29/docker-device-mapper-resize/) the container filesystem.
-For the purposes of image space tracking, we can
-
-####Testing notes:
+For the purposes of image space tracking, we can
-* ```shell
+#### Testing notes:
+Notice the **Pool Name**
+```shell
$ docker info
-
...
-
Storage Driver: devicemapper
-
- Pool Name: **docker-8:1-268480-pool**
-
+ Pool Name: docker-8:1-268480-pool
Pool Blocksize: 65.54 kB
-
Backing Filesystem: extfs
-
Data file: /dev/loop0
-
Metadata file: /dev/loop1
-
Data Space Used: 2.059 GB
-
Data Space Total: 107.4 GB
-
Data Space Available: 48.45 GB
-
Metadata Space Used: 1.806 MB
-
Metadata Space Total: 2.147 GB
-
Metadata Space Available: 2.146 GB
-
Udev Sync Supported: true
-
Deferred Removal Enabled: false
-
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
-
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
-
Library Version: 1.02.99 (2015-06-20)
```
```shell
-$ dmsetup table docker-8\:1-268480-pool
-
-0 209715200 thin-pool 7:1 7:0 **128** 32768 1 skip_block_zeroing
+$ dmsetup table docker-8\:1-268480-pool
+0 209715200 thin-pool 7:1 7:0 128 32768 1 skip_block_zeroing
```
128 is the data block size
@@ -579,9 +552,8 @@ $ dmsetup table docker-8\:1-268480-pool
Usage from kernel for the primary block device
```shell
-$ dmsetup status docker-8\:1-268480-pool
-
-0 209715200 thin-pool 37 441/524288 **31424/1638400** - rw discard_passdown queue_if_no_space -
+$ dmsetup status docker-8\:1-268480-pool
+0 209715200 thin-pool 37 441/524288 31424/1638400 - rw discard_passdown queue_if_no_space -
```
Usage/Available - 31424/1638400
@@ -602,7 +574,7 @@ Capacity in MB = 1638400 * 512 * 128 bytes = 100 GB
##### Testing titbits
-* Ubuntu 15.10 doesn’t ship with the quota module on virtual machines. [Install ‘linux-image-extra-virtual’](http://askubuntu.com/questions/109585/quota-format-not-supported-in-kernel) package to get quota to work.
+* Ubuntu 15.10 doesn't ship with the quota module on virtual machines. [Install ‘linux-image-extra-virtual’](http://askubuntu.com/questions/109585/quota-format-not-supported-in-kernel) package to get quota to work.
* Overlay storage driver needs kernels >= 3.18. I used Ubuntu 15.10 to test Overlayfs.
diff --git a/contributors/design-proposals/downward_api_resources_limits_requests.md b/contributors/design-proposals/downward_api_resources_limits_requests.md
index ab17c321..bebf05ab 100644
--- a/contributors/design-proposals/downward_api_resources_limits_requests.md
+++ b/contributors/design-proposals/downward_api_resources_limits_requests.md
@@ -141,7 +141,7 @@ Volumes are pod scoped, so a selector must be specified with a container name.
Full json path selectors will use existing `type ObjectFieldSelector`
to extend the current implementation for resources requests and limits.
-```
+```go
// ObjectFieldSelector selects an APIVersioned field of an object.
type ObjectFieldSelector struct {
APIVersion string `json:"apiVersion"`
@@ -154,7 +154,7 @@ type ObjectFieldSelector struct {
These examples show how to use full selectors with environment variables and volume plugin.
-```
+```yaml
apiVersion: v1
kind: Pod
metadata:
@@ -178,7 +178,7 @@ spec:
fieldPath: spec.containers[?(@.name=="test-container")].resources.limits.cpu
```
-```
+```yaml
apiVersion: v1
kind: Pod
metadata:
@@ -221,7 +221,7 @@ relative to the container spec. These will be implemented by introducing a
`ContainerSpecFieldSelector` (json: `containerSpecFieldRef`) to extend the current
implementation for `type DownwardAPIVolumeFile struct` and `type EnvVarSource struct`.
-```
+```go
// ContainerSpecFieldSelector selects an APIVersioned field of an object.
type ContainerSpecFieldSelector struct {
APIVersion string `json:"apiVersion"`
@@ -300,7 +300,7 @@ Volumes are pod scoped, the container name must be specified as part of
These examples show how to use partial selectors with environment variables and volume plugin.
-```
+```yaml
apiVersion: v1
kind: Pod
metadata:
@@ -337,7 +337,7 @@ spec:
resources:
requests:
memory: "64Mi"
- cpu: "250m"
+ cpu: "250m"
limits:
memory: "128Mi"
cpu: "500m"
@@ -388,7 +388,7 @@ For example, if requests.cpu is `250m` (250 millicores) and the divisor by defau
exposed value will be `1` core. It is because 250 millicores when converted to cores will be 0.25 and
the ceiling of 0.25 is 1.
-```
+```go
type ResourceFieldSelector struct {
// Container name
ContainerName string `json:"containerName,omitempty"`
@@ -462,7 +462,7 @@ Volumes are pod scoped, the container name must be specified as part of
These examples show how to use magic keys approach with environment variables and volume plugin.
-```
+```yaml
apiVersion: v1
kind: Pod
metadata:
@@ -493,7 +493,7 @@ spec:
In the above example, the exposed values of CPU_LIMIT and MEMORY_LIMIT will be 1 (in cores) and 128 (in Mi), respectively.
-```
+```yaml
apiVersion: v1
kind: Pod
metadata:
@@ -578,7 +578,7 @@ in a shell script, and then export `JAVA_OPTS` (assuming your container image su
and GOMAXPROCS environment variables inside the container image. The spec file for the
application pod could look like:
-```
+```yaml
apiVersion: v1
kind: Pod
metadata:
@@ -609,7 +609,7 @@ spec:
Note that the value of divisor by default is `1`. Now inside the container,
the HEAP_SIZE (in bytes) and GOMAXPROCS (in cores) could be exported as:
-```
+```sh
export JAVA_OPTS="$JAVA_OPTS -Xmx:$(HEAP_SIZE)"
and
diff --git a/contributors/design-proposals/dramatically-simplify-cluster-creation.md b/contributors/design-proposals/dramatically-simplify-cluster-creation.md
index d5bc8a38..78a1089a 100644
--- a/contributors/design-proposals/dramatically-simplify-cluster-creation.md
+++ b/contributors/design-proposals/dramatically-simplify-cluster-creation.md
@@ -221,7 +221,7 @@ Unable to join mesh network. Check your token.
* @jbeda & @philips?
-1. Documentation - so that new users can see this in 1.4 (even if it’s caveated with alpha/experimental labels and flags all over it)
+1. Documentation - so that new users can see this in 1.4 (even if it's caveated with alpha/experimental labels and flags all over it)
* @lukemarsden
diff --git a/contributors/design-proposals/enhance-pluggable-policy.md b/contributors/design-proposals/enhance-pluggable-policy.md
index ecc908ee..14fff3dd 100644
--- a/contributors/design-proposals/enhance-pluggable-policy.md
+++ b/contributors/design-proposals/enhance-pluggable-policy.md
@@ -141,7 +141,7 @@ accepts creates. The caller POSTs a SubjectAccessReview to this URL and he gets
a SubjectAccessReviewResponse back. Here is an example of a call and its
corresponding return:
-```
+```json
// input
{
"kind": "SubjectAccessReview",
@@ -172,7 +172,7 @@ only accepts creates. The caller POSTs a PersonalSubjectAccessReview to this URL
and he gets a SubjectAccessReviewResponse back. Here is an example of a call and
its corresponding return:
-```
+```json
// input
{
"kind": "PersonalSubjectAccessReview",
@@ -202,7 +202,7 @@ accepts creates. The caller POSTs a LocalSubjectAccessReview to this URL and he
gets a LocalSubjectAccessReviewResponse back. Here is an example of a call and
its corresponding return:
-```
+```json
// input
{
"kind": "LocalSubjectAccessReview",
@@ -353,7 +353,7 @@ accepts creates. The caller POSTs a ResourceAccessReview to this URL and he gets
a ResourceAccessReviewResponse back. Here is an example of a call and its
corresponding return:
-```
+```json
// input
{
"kind": "ResourceAccessReview",
diff --git a/contributors/design-proposals/expansion.md b/contributors/design-proposals/expansion.md
index ace1faf0..4f2f6f40 100644
--- a/contributors/design-proposals/expansion.md
+++ b/contributors/design-proposals/expansion.md
@@ -275,7 +275,7 @@ func (r *objectRecorderImpl) Event(reason, message string) {
}
func ObjectEventRecorderFor(object runtime.Object, recorder EventRecorder) ObjectEventRecorder {
- return &objectRecorderImpl{object, recorder}
+ return &objectRecorderImpl{object, recorder}
}
```
@@ -367,7 +367,7 @@ No other variables are defined.
| `"--$($($($($--"` | `"--$($($($($--"` |
| `"$($($($($--foo$("` | `"$($($($($--foo$("` |
| `"foo0--$($($($("` | `"foo0--$($($($("` |
-| `"$(foo$$var)` | `$(foo$$var)` |
+| `"$(foo$$var)"` | `"$(foo$$var)"` |
#### In a pod: building a URL
diff --git a/contributors/design-proposals/external-lb-source-ip-preservation.md b/contributors/design-proposals/external-lb-source-ip-preservation.md
index e1450e64..38f9c646 100644
--- a/contributors/design-proposals/external-lb-source-ip-preservation.md
+++ b/contributors/design-proposals/external-lb-source-ip-preservation.md
@@ -194,27 +194,27 @@ The cases we should test are:
1. Core Functionality Tests
-1.1 Source IP Preservation
+ 1.1 Source IP Preservation
-Test the main intent of this change, source ip preservation - use the all-in-one network tests container
-with new functionality that responds with the client IP. Verify the container is seeing the external IP
-of the test client.
+ Test the main intent of this change, source ip preservation - use the all-in-one network tests container
+ with new functionality that responds with the client IP. Verify the container is seeing the external IP
+ of the test client.
-1.2 Health Check responses
+ 1.2 Health Check responses
-Testcases use pods explicitly pinned to nodes and delete/add to nodes randomly. Validate that healthchecks succeed
-and fail on the expected nodes as endpoints move around. Gather LB response times (time from pod declares ready to
-time for Cloud LB to declare node healthy and vice versa) to endpoint changes.
+ Testcases use pods explicitly pinned to nodes and delete/add to nodes randomly. Validate that healthchecks succeed
+ and fail on the expected nodes as endpoints move around. Gather LB response times (time from pod declares ready to
+ time for Cloud LB to declare node healthy and vice versa) to endpoint changes.
2. Inter-Operability Tests
-Validate that internal cluster communications are still possible from nodes without local endpoints. This change
-is only for externally sourced traffic.
+ Validate that internal cluster communications are still possible from nodes without local endpoints. This change
+ is only for externally sourced traffic.
3. Backward Compatibility Tests
-Validate that old and new functionality can simultaneously exist in a single cluster. Create services with and without
-the annotation, and validate datapath correctness.
+ Validate that old and new functionality can simultaneously exist in a single cluster. Create services with and without
+ the annotation, and validate datapath correctness.
# Beta Design
diff --git a/contributors/design-proposals/federated-api-servers.md b/contributors/design-proposals/federated-api-servers.md
index 1d2d5ba1..2a6000e4 100644
--- a/contributors/design-proposals/federated-api-servers.md
+++ b/contributors/design-proposals/federated-api-servers.md
@@ -1,208 +1,7 @@
# Federated API Servers
-## Abstract
-
-We want to divide the single monolithic API server into multiple federated
-servers. Anyone should be able to write their own federated API server to expose APIs they want.
-Cluster admins should be able to expose new APIs at runtime by bringing up new
-federated servers.
-
-## Motivation
-
-* Extensibility: We want to allow community members to write their own API
- servers to expose APIs they want. Cluster admins should be able to use these
- servers without having to require any change in the core kubernetes
- repository.
-* Unblock new APIs from core kubernetes team review: A lot of new API proposals
- are currently blocked on review from the core kubernetes team. By allowing
- developers to expose their APIs as a separate server and enabling the cluster
- admin to use it without any change to the core kubernetes repository, we
- unblock these APIs.
-* Place for staging experimental APIs: New APIs can remain in separate
- federated servers until they become stable, at which point, they can be moved
- to the core kubernetes master, if appropriate.
-* Ensure that new APIs follow kubernetes conventions: Without the mechanism
- proposed here, community members might be forced to roll their own thing which
- may or may not follow kubernetes conventions.
-
-## Goal
-
-* Developers should be able to write their own API server and cluster admins
- should be able to add them to their cluster, exposing new APIs at runtime. All
- of this should not require any change to the core kubernetes API server.
-* These new APIs should be seamless extension of the core kubernetes APIs (ex:
- they should be operated upon via kubectl).
-
-## Non Goals
-
-The following are related but are not the goals of this specific proposal:
-* Make it easy to write a kubernetes API server.
-
-## High Level Architecture
-
-There will be 2 new components in the cluster:
-* A simple program to summarize discovery information from all the servers.
-* A reverse proxy to proxy client requests to individual servers.
-
-The reverse proxy is optional. Clients can discover server URLs using the
-summarized discovery information and contact them directly. Simple clients, can
-always use the proxy.
-The same program can provide both discovery summarization and reverse proxy.
-
-### Constraints
-
-* Unique API groups across servers: Each API server (and groups of servers, in HA)
- should expose unique API groups.
-* Follow API conventions: APIs exposed by every API server should adhere to [kubernetes API
- conventions](../devel/api-conventions.md).
-* Support discovery API: Each API server should support the kubernetes discovery API
- (list the suported groupVersions at `/apis` and list the supported resources
- at `/apis/<groupVersion>/`)
-* No bootstrap problem: The core kubernetes server should not depend on any
- other federated server to come up. Other servers can only depend on the core
- kubernetes server.
-
-## Implementation Details
-
-### Summarizing discovery information
-
-We can have a very simple Go program to summarize discovery information from all
-servers. Cluster admins will register each federated API server (its baseURL and swagger
-spec path) with the proxy. The proxy will summarize the list of all group versions
-exposed by all registered API servers with their individual URLs at `/apis`.
-
-### Reverse proxy
-
-We can use any standard reverse proxy server like nginx or extend the same Go program that
-summarizes discovery information to act as reverse proxy for all federated servers.
-
-Cluster admins are also free to use any of the multiple open source API management tools
-(for example, there is [Kong](https://getkong.org/), which is written in lua and there is
-[Tyk](https://tyk.io/), which is written in Go). These API management tools
-provide a lot more functionality like: rate-limiting, caching, logging,
-transformations and authentication.
-In future, we can also use ingress. That will give cluster admins the flexibility to
-easily swap out the ingress controller by a Go reverse proxy, nginx, haproxy
-or any other solution they might want.
-
-### Storage
-
-Each API server is responsible for storing their resources. They can have their
-own etcd or can use kubernetes server's etcd using [third party
-resources](../design/extending-api.md#adding-custom-resources-to-the-kubernetes-api-server).
-
-### Health check
-
-Kubernetes server's `/api/v1/componentstatuses` will continue to report status
-of master components that it depends on (scheduler and various controllers).
-Since clients have access to server URLs, they can use that to do
-health check of individual servers.
-In future, if a global health check is required, we can expose a health check
-endpoint in the proxy that will report the status of all federated api servers
-in the cluster.
-
-### Auth
-
-Since the actual server which serves client's request can be opaque to the client,
-all API servers need to have homogeneous authentication and authorisation mechanisms.
-All API servers will handle authn and authz for their resources themselves.
-In future, we can also have the proxy do the auth and then have apiservers trust
-it (via client certs) to report the actual user in an X-something header.
-
-For now, we will trust system admins to configure homogeneous auth on all servers.
-Future proposals will refine how auth is managed across the cluster.
-
-### kubectl
-
-kubectl will talk to the discovery endpoint (or proxy) and use the discovery API to
-figure out the operations and resources supported in the cluster.
-Today, it uses RESTMapper to determine that. We will update kubectl code to populate
-RESTMapper using the discovery API so that we can add and remove resources
-at runtime.
-We will also need to make kubectl truly generic. Right now, a lot of operations
-(like get, describe) are hardcoded in the binary for all resources. A future
-proposal will provide details on moving those operations to server.
-
-Note that it is possible for kubectl to talk to individual servers directly in
-which case proxy will not be required at all, but this requires a bit more logic
-in kubectl. We can do this in future, if desired.
-
-### Handling global policies
-
-Now that we have resources spread across multiple API servers, we need to
-be careful to ensure that global policies (limit ranges, resource quotas, etc) are enforced.
-Future proposals will improve how this is done across the cluster.
-
-#### Namespaces
-
-When a namespaced resource is created in any of the federated server, that
-server first needs to check with the kubernetes server that:
-
-* The namespace exists.
-* User has authorization to create resources in that namespace.
-* Resource quota for the namespace is not exceeded.
-
-To prevent race conditions, the kubernetes server might need to expose an atomic
-API for all these operations.
-
-While deleting a namespace, kubernetes server needs to ensure that resources in
-that namespace maintained by other servers are deleted as well. We can do this
-using resource [finalizers](../design/namespaces.md#finalizers). Each server
-will add themselves in the set of finalizers before they create a resource in
-the corresponding namespace and delete all their resources in that namespace,
-whenever it is to be deleted (kubernetes API server already has this code, we
-will refactor it into a library to enable reuse).
-
-Future proposal will talk about this in more detail and provide a better
-mechanism.
-
-#### Limit ranges and resource quotas
-
-kubernetes server maintains [resource quotas](../admin/resourcequota/README.md) and
-[limit ranges](../admin/limitrange/README.md) for all resources.
-Federated servers will need to check with the kubernetes server before creating any
-resource.
-
-## Running on hosted kubernetes cluster
-
-This proposal is not enough for hosted cluster users, but allows us to improve
-that in the future.
-On a hosted kubernetes cluster, for e.g. on GKE - where Google manages the kubernetes
-API server, users will have to bring up and maintain the proxy and federated servers
-themselves.
-Other system components like the various controllers, will not be aware of the
-proxy and will only talk to the kubernetes API server.
-
-One possible solution to fix this is to update kubernetes API server to detect when
-there are federated servers in the cluster and then change its advertise address to
-the IP address of the proxy.
-Future proposal will talk about this in more detail.
-
-## Alternatives
-
-There were other alternatives that we had discussed.
-
-* Instead of adding a proxy in front, let the core kubernetes server provide an
- API for other servers to register themselves. It can also provide a discovery
- API which the clients can use to discover other servers and then talk to them
- directly. But this would have required another server API a lot of client logic as well.
-* Validating federated servers: We can validate new servers when they are registered
- with the proxy, or keep validating them at regular intervals, or validate
- them only when explicitly requested, or not validate at all.
- We decided that the proxy will just assume that all the servers are valid
- (conform to our api conventions). In future, we can provide conformance tests.
-
-## Future Work
-
-* Validate servers: We should have some conformance tests that validate that the
- servers follow kubernetes api-conventions.
-* Provide centralised auth service: It is very hard to ensure homogeneous auth
- across multiple federated servers, especially in case of hosted clusters
- (where different people control the different servers). We can fix it by
- providing a centralised authentication and authorization service which all of
- the servers can use.
-
-
+Moved to [aggregated-api-servers.md](./aggregated-api-servers.md) since cluster
+federation stole the word "federation" from this effort and it was very confusing.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/federated-api-servers.md?pixel)]()
diff --git a/contributors/design-proposals/federated-replicasets.md b/contributors/design-proposals/federated-replicasets.md
index 3443912d..83a4e1ed 100644
--- a/contributors/design-proposals/federated-replicasets.md
+++ b/contributors/design-proposals/federated-replicasets.md
@@ -18,7 +18,7 @@ control whether he has both enough application replicas running
locally in each of the clusters (so that, for example, users are
handled by a nearby cluster, with low latency) and globally (so that
there is always enough capacity to handle all traffic). If one of the
-clusters has issues or hasn’t enough capacity to run the given set of
+clusters has issues or hasn't enough capacity to run the given set of
replicas the replicas should be automatically moved to some other
cluster to keep the application responsive.
@@ -71,7 +71,7 @@ A component that checks how many replicas are actually running in each
of the subclusters and if the number matches to the
FederatedReplicaSet preferences (by default spread replicas evenly
across the clusters but custom preferences are allowed - see
-below). If it doesn’t and the situation is unlikely to improve soon
+below). If it doesn't and the situation is unlikely to improve soon
then the replicas should be moved to other subclusters.
### API and CLI
@@ -96,7 +96,7 @@ be passed as annotations.
The preferences are expressed by the following structure, passed as a
serialized json inside annotations.
-```
+```go
type FederatedReplicaSetPreferences struct {
// If set to true then already scheduled and running replicas may be moved to other clusters to
// in order to bring cluster replicasets towards a desired state. Otherwise, if set to false,
@@ -104,7 +104,7 @@ type FederatedReplicaSetPreferences struct {
Rebalance bool `json:"rebalance,omitempty"`
// Map from cluster name to preferences for that cluster. It is assumed that if a cluster
- // doesn’t have a matching entry then it should not have local replica. The cluster matches
+ // doesn't have a matching entry then it should not have local replica. The cluster matches
// to "*" if there is no entry with the real cluster name.
Clusters map[string]LocalReplicaSetPreferences
}
@@ -126,7 +126,7 @@ How this works in practice:
**Scenario 1**. I want to spread my 50 replicas evenly across all available clusters. Config:
-```
+```go
FederatedReplicaSetPreferences {
Rebalance : true
Clusters : map[string]LocalReplicaSet {
@@ -146,7 +146,7 @@ Example:
**Scenario 2**. I want to have only 2 replicas in each of the clusters.
-```
+```go
FederatedReplicaSetPreferences {
Rebalance : true
Clusters : map[string]LocalReplicaSet {
@@ -157,7 +157,7 @@ FederatedReplicaSetPreferences {
Or
-```
+```go
FederatedReplicaSetPreferences {
Rebalance : true
Clusters : map[string]LocalReplicaSet {
@@ -169,7 +169,7 @@ FederatedReplicaSetPreferences {
Or
-```
+```go
FederatedReplicaSetPreferences {
Rebalance : true
Clusters : map[string]LocalReplicaSet {
@@ -182,7 +182,7 @@ There is a global target for 50, however if there are 3 clusters there will be o
**Scenario 3**. I want to have 20 replicas in each of 3 clusters.
-```
+```go
FederatedReplicaSetPreferences {
Rebalance : true
Clusters : map[string]LocalReplicaSet {
@@ -194,9 +194,9 @@ FederatedReplicaSetPreferences {
There is a global target for 50, however clusters require 60. So some clusters will have less replicas.
Replica layout: A=20 B=20 C=10.
-**Scenario 4**. I want to have equal number of replicas in clusters A,B,C, however don’t put more than 20 replicas to cluster C.
+**Scenario 4**. I want to have equal number of replicas in clusters A,B,C, however don't put more than 20 replicas to cluster C.
-```
+```go
FederatedReplicaSetPreferences {
Rebalance : true
Clusters : map[string]LocalReplicaSet {
@@ -217,7 +217,7 @@ Example:
**Scenario 5**. I want to run my application in cluster A, however if there are troubles FRS can also use clusters B and C, equally.
-```
+```go
FederatedReplicaSetPreferences {
Clusters : map[string]LocalReplicaSet {
“A” : LocalReplicaSet{ Weight: 1000000}
@@ -236,7 +236,7 @@ Example:
**Scenario 6**. I want to run my application in clusters A, B and C. Cluster A gets twice the QPS than other clusters.
-```
+```go
FederatedReplicaSetPreferences {
Clusters : map[string]LocalReplicaSet {
“A” : LocalReplicaSet{ Weight: 2}
@@ -249,7 +249,7 @@ FederatedReplicaSetPreferences {
**Scenario 7**. I want to spread my 50 replicas evenly across all available clusters, but if there
are already some replicas, please do not move them. Config:
-```
+```go
FederatedReplicaSetPreferences {
Rebalance : false
Clusters : map[string]LocalReplicaSet {
@@ -312,7 +312,7 @@ enumerated the key idea elements:
+ [E4] LRS is manually deleted from the local cluster. In this case
a new LRS should be created. It is the same case as
[[E1]](#heading=h.wn3dfsyc4yuh). Any pods that were left behind
- won’t be killed and will be adopted after the LRS is recreated.
+ won't be killed and will be adopted after the LRS is recreated.
+ [E5] LRS fails to create (not necessary schedule) the desired
number of pods due to master troubles, admission control
@@ -341,7 +341,7 @@ elsewhere. For that purpose FRSC will maintain a data structure
where for each FRS controlled LRS we store a list of pods belonging
to that LRS along with their current status and status change timestamp.
-+ [I5] If a new cluster is added to the federation then it doesn’t
++ [I5] If a new cluster is added to the federation then it doesn't
have a LRS and the situation is equal to
[[E1]](#heading=h.wn3dfsyc4yuh)/[[E4]](#heading=h.vlyovyh7eef).
@@ -350,7 +350,7 @@ to that LRS along with their current status and status change timestamp.
a cluster is lost completely then the cluster is removed from the
the cluster list (or marked accordingly) so
[[E6]](#heading=h.in6ove1c1s8f) and [[E7]](#heading=h.37bnbvwjxeda)
- don’t need to be handled.
+ don't need to be handled.
+ [I7] All ToBeChecked FRS are browsed every 1 min (configurable),
checked against the current list of clusters, and all missing LRS
@@ -449,7 +449,7 @@ goroutines (however if needed the function can be parallelized for
different FRS). It takes data only from store maintained by GR2_* and
GR3_*. The external communication is only required to:
-+ Create LRS. If a LRS doesn’t exist it is created after the
++ Create LRS. If a LRS doesn't exist it is created after the
rescheduling, when we know how much replicas should it have.
+ Update LRS replica targets.
@@ -470,7 +470,7 @@ as events.
## Workflow
Here is the sequence of tasks that need to be done in order for a
-typical FRS to be split into a number of LRS’s and to be created in
+typical FRS to be split into a number of LRS's and to be created in
the underlying federated clusters.
Note a: the reason the workflow would be helpful at this phase is that
@@ -489,7 +489,7 @@ Note c: federation-apiserver populates the clusterid field in the FRS
before persisting it into the federation etcd
Step 3: the federation-level “informer” in FRSC watches federation
-etcd for new/modified FRS’s, with empty clusterid or clusterid equal
+etcd for new/modified FRS's, with empty clusterid or clusterid equal
to federation ID, and if detected, it calls the scheduling code
Step 4.
@@ -503,7 +503,7 @@ distribution, i.e., equal weights for all of the underlying clusters
Step 5. As soon as the scheduler function returns the control to FRSC,
the FRSC starts a number of cluster-level “informer”s, one per every
target cluster, to watch changes in every target cluster etcd
-regarding the posted LRS’s and if any violation from the scheduled
+regarding the posted LRS's and if any violation from the scheduled
number of replicase is detected the scheduling code is re-called for
re-scheduling purposes.
diff --git a/contributors/design-proposals/federated-services.md b/contributors/design-proposals/federated-services.md
index aabdf6f7..8ec9ca29 100644
--- a/contributors/design-proposals/federated-services.md
+++ b/contributors/design-proposals/federated-services.md
@@ -473,11 +473,13 @@ underlying clusters, to make up the total of 6 replicas required. To handle
entire cluster failures, various approaches are possible, including:
1. **simple overprovisioning**, such that sufficient replicas remain even if a
cluster fails. This wastes some resources, but is simple and reliable.
+
2. **pod autoscaling**, where the replication controller in each
cluster automatically and autonomously increases the number of
replicas in its cluster in response to the additional traffic
diverted from the failed cluster. This saves resources and is relatively
simple, but there is some delay in the autoscaling.
+
3. **federated replica migration**, where the Cluster Federation
control system detects the cluster failure and automatically
increases the replica count in the remaining clusters to make up
diff --git a/contributors/design-proposals/federation-phase-1.md b/contributors/design-proposals/federation-phase-1.md
index e6c54bf6..3aa16dd8 100644
--- a/contributors/design-proposals/federation-phase-1.md
+++ b/contributors/design-proposals/federation-phase-1.md
@@ -59,7 +59,7 @@ clusters.
## SCOPE
-It’s difficult to have a perfect design with one click that implements
+It's difficult to have a perfect design with one click that implements
all the above requirements. Therefore we will go with an iterative
approach to design and build the system. This document describes the
phase one of the whole work. In phase one we will cover only the
@@ -95,7 +95,7 @@ Some design principles we are following in this architecture:
1. Keep the Ubernetes API interface compatible with K8S API as much as
possible.
1. Re-use concepts from K8S as much as possible. This reduces
-customers’ learning curve and is good for adoption. Below is a brief
+customers' learning curve and is good for adoption. Below is a brief
description of each module contained in above diagram.
## Ubernetes API Server
@@ -105,7 +105,7 @@ Server in K8S. It talks to a distributed key-value store to persist,
retrieve and watch API objects. This store is completely distinct
from the kubernetes key-value stores (etcd) in the underlying
kubernetes clusters. We still use `etcd` as the distributed
-storage so customers don’t need to learn and manage a different
+storage so customers don't need to learn and manage a different
storage system, although it is envisaged that other storage systems
(consol, zookeeper) will probably be developedand supported over
time.
@@ -200,7 +200,7 @@ $version.clusterSpec
<td style="padding:5px;">Credential<br>
</td>
<td style="padding:5px;">the type (e.g. bearer token, client
-certificate etc) and data of the credential used to access cluster. It’s used for system routines (not behalf of users)<br>
+certificate etc) and data of the credential used to access cluster. It's used for system routines (not behalf of users)<br>
</td>
<td style="padding:5px;">yes<br>
</td>
@@ -263,7 +263,7 @@ $version.clusterStatus
</tbody>
</table>
-**For simplicity we didn’t introduce a separate “cluster metrics” API
+**For simplicity we didn't introduce a separate “cluster metrics” API
object here**. The cluster resource metrics are stored in cluster
status section, just like what we did to nodes in K8S. In phase one it
only contains available CPU resources and memory resources. The
@@ -295,7 +295,7 @@ cases it may be complex. For example:
+ This workload can only be scheduled to cluster Foo. It cannot be
scheduled to any other clusters. (use case: sensitive workloads).
+ This workload prefers cluster Foo. But if there is no available
- capacity on cluster Foo, it’s OK to be scheduled to cluster Bar
+ capacity on cluster Foo, it's OK to be scheduled to cluster Bar
(use case: workload )
+ Seventy percent of this workload should be scheduled to cluster Foo,
and thirty percent should be scheduled to cluster Bar (use case:
@@ -306,7 +306,7 @@ cases it may be complex. For example:
Below is a sample of the YAML to create such a replication controller.
-```
+```yaml
apiVersion: v1
kind: ReplicationController
metadata:
@@ -373,7 +373,7 @@ plane:
1. Each cluster control is watching the sub RCs bound to its
corresponding cluster. It picks up the newly created sub RC.
1. The cluster controller issues requests to the underlying cluster
-API Server to create the RC. In phase one we don’t support complex
+API Server to create the RC. In phase one we don't support complex
distribution policies. The scheduling rule is basically:
1. If a RC does not specify any nodeSelector, it will be scheduled
to the least loaded K8S cluster(s) that has enough available
@@ -388,7 +388,7 @@ the cluster is working independently it still accepts workload
requests from other K8S clients or even another Cluster Federation control
plane. The Cluster Federation scheduling decision is based on this data of
available resources. However when the actual RC creation happens to
-the cluster at time _T2_, the cluster may don’t have enough resources
+the cluster at time _T2_, the cluster may don't have enough resources
at that time. We will address this problem in later phases with some
proposed solutions like resource reservation mechanisms.
diff --git a/contributors/design-proposals/garbage-collection.md b/contributors/design-proposals/garbage-collection.md
index b24a7f21..0bf42e5d 100644
--- a/contributors/design-proposals/garbage-collection.md
+++ b/contributors/design-proposals/garbage-collection.md
@@ -33,9 +33,9 @@ Non-goals include:
## API Changes
-```
+```go
type ObjectMeta struct {
- ...
+ ...
OwnerReferences []OwnerReference
}
```
@@ -43,7 +43,7 @@ type ObjectMeta struct {
**ObjectMeta.OwnerReferences**:
List of objects depended by this object. If ***all*** objects in the list have been deleted, this object will be garbage collected. For example, a replica set `R` created by a deployment `D` should have an entry in ObjectMeta.OwnerReferences pointing to `D`, set by the deployment controller when `R` is created. This field can be updated by any client that has the privilege to both update ***and*** delete the object. For safety reasons, we can add validation rules to restrict what resources could be set as owners. For example, Events will likely be banned from being owners.
-```
+```go
type OwnerReference struct {
// Version of the referent.
APIVersion string
@@ -83,7 +83,7 @@ The Garbage Collector consists of a scanner, a garbage processor, and a propagat
* Worker:
* Dequeues an item from the *Event Queue*.
* If the item is an creation or update, then updates the DAG accordingly.
- * If the object has an owner and the owner doesn’t exist in the DAG yet, then apart from adding the object to the DAG, also enqueues the object to the *Dirty Queue*.
+ * If the object has an owner and the owner doesn't exist in the DAG yet, then apart from adding the object to the DAG, also enqueues the object to the *Dirty Queue*.
* If the item is a deletion, then removes the object from the DAG, and enqueues all its dependent objects to the *Dirty Queue*.
* The propagator shouldn't need to do any RPCs, so a single worker should be sufficient. This makes locking easier.
* With the Propagator, we *only* need to run the Scanner when starting the GC to populate the DAG and the *Dirty Queue*.
@@ -96,7 +96,7 @@ Users may want to delete an owning object (e.g., a replicaset) while orphaning t
## API changes
-```
+```go
type ObjectMeta struct {
Finalizers []string
@@ -133,7 +133,7 @@ type ObjectMeta struct {
## API changes
-```
+```go
type DeleteOptions struct {
OrphanDependents bool
@@ -162,8 +162,8 @@ Adding a fourth component to the Garbage Collector, the"orphan" finalizer:
## Orphan adoption
Controllers are responsible for adopting orphaned dependent resources. To do so, controllers
-* Checks a potential dependent object’s OwnerReferences to determine if it is orphaned.
-* Fills the OwnerReferences if the object matches the controller’s selector and is orphaned.
+* Checks a potential dependent object's OwnerReferences to determine if it is orphaned.
+* Fills the OwnerReferences if the object matches the controller's selector and is orphaned.
There is a potential race between the "orphan" finalizer removing an owner reference and the controllers adding it back during adoption. Imagining this case: a user deletes an owning object and intends to orphan the dependent objects, so the GC removes the owner from the dependent object's OwnerReferences list, but the controller of the owner resource hasn't observed the deletion yet, so it adopts the dependent again and adds the reference back, resulting in the mistaken deletion of the dependent object. This race can be avoided by implementing Status.ObservedGeneration in all resources. Before updating the dependent Object's OwnerReferences, the "orphan" finalizer checks Status.ObservedGeneration of the owning object to ensure its controller has already observed the deletion.
@@ -173,7 +173,7 @@ For the master, after upgrading to a version that supports cascading deletion, t
For nodes, cascading deletion does not affect them.
-For kubectl, we will keep the kubectl’s cascading deletion logic for one more release.
+For kubectl, we will keep the kubectl's cascading deletion logic for one more release.
# End-to-End Examples
@@ -243,7 +243,7 @@ This section presents an example of all components working together to enforce t
## API Changes
-```
+```go
type DeleteOptions struct {
OrphanChildren bool
@@ -252,16 +252,16 @@ type DeleteOptions struct {
**DeleteOptions.OrphanChildren**: allows a user to express whether the child objects should be orphaned.
-```
+```go
type ObjectMeta struct {
- ...
+ ...
ParentReferences []ObjectReference
}
```
**ObjectMeta.ParentReferences**: links the resource to the parent resources. For example, a replica set `R` created by a deployment `D` should have an entry in ObjectMeta.ParentReferences pointing to `D`. The link should be set when the child object is created. It can be updated after the creation.
-```
+```go
type Tombstone struct {
unversioned.TypeMeta
ObjectMeta
@@ -299,7 +299,7 @@ The only new component is the Garbage Collector, which consists of a scanner, a
* Worker:
* Dequeues an item from the *Event Queue*.
* If the item is an creation or update, then updates the DAG accordingly.
- * If the object has a parent and the parent doesn’t exist in the DAG yet, then apart from adding the object to the DAG, also enqueues the object to the *Dirty Queue*.
+ * If the object has a parent and the parent doesn't exist in the DAG yet, then apart from adding the object to the DAG, also enqueues the object to the *Dirty Queue*.
* If the item is a deletion, then removes the object from the DAG, and enqueues all its children to the *Dirty Queue*.
* The propagator shouldn't need to do any RPCs, so a single worker should be sufficient. This makes locking easier.
* With the Propagator, we *only* need to run the Scanner when starting the Propagator to populate the DAG and the *Dirty Queue*.
@@ -310,14 +310,14 @@ The only new component is the Garbage Collector, which consists of a scanner, a
* API Server: when handling a deletion request, if DeleteOptions.OrphanChildren is true, then the API Server either creates a tombstone with TTL if the tombstone doesn't exist yet, or updates the TTL of the existing tombstone. The API Server deletes the object after the tombstone is created.
-* Controllers: when creating child objects, controllers need to fill up their ObjectMeta.ParentReferences field. Objects that don’t have a parent should have the namespace object as the parent.
+* Controllers: when creating child objects, controllers need to fill up their ObjectMeta.ParentReferences field. Objects that don't have a parent should have the namespace object as the parent.
## Comparison with the selected design
The main difference between the two designs is when to update the ParentReferences. In design #1, because a tombstone is created to indicate "orphaning" is desired, the updates to ParentReferences can be deferred until the deletion of the tombstone. In design #2, the updates need to be done before the parent object is deleted from the registry.
* Advantages of "Tombstone + GC" design
- * Faster to free the resource name compared to using finalizers. The original object can be deleted to free the resource name once the tombstone is created, rather than waiting for the finalizers to update all children’s ObjectMeta.ParentReferences.
+ * Faster to free the resource name compared to using finalizers. The original object can be deleted to free the resource name once the tombstone is created, rather than waiting for the finalizers to update all children's ObjectMeta.ParentReferences.
* Advantages of "Finalizer Framework + GC"
* The finalizer framework is needed for other purposes as well.
diff --git a/contributors/design-proposals/gpu-support.md b/contributors/design-proposals/gpu-support.md
index 604f64bd..19aa1dac 100644
--- a/contributors/design-proposals/gpu-support.md
+++ b/contributors/design-proposals/gpu-support.md
@@ -203,7 +203,7 @@ linked directly into kubelet. A partial list of tradeoffs:
| Reliability | Need to handle the binary disappearing at any time | Fewer headeaches |
| (Un)Marshalling | Need to talk over JSON | None |
| Administration cost | One more daemon to install, configure and monitor | No extra work required, other than perhaps configuring flags |
-| Releases | Potentially on its own schedule | Tied to Kubernetes' |
+| Releases | Potentially on its own schedule | Tied to Kubernetes |
## Implementation plan
diff --git a/contributors/design-proposals/ha_master.md b/contributors/design-proposals/ha_master.md
index 2575aaf7..b225e667 100644
--- a/contributors/design-proposals/ha_master.md
+++ b/contributors/design-proposals/ha_master.md
@@ -49,7 +49,7 @@ It will be covered in a separate doc.
All etcd instances will be clustered together and one of them will be an elected master.
In order to commit any change quorum of the cluster will have to confirm it. Etcd will be
configured in such a way that all writes and reads will go through the master (requests
-will be forwarded by the local etcd server such that it’s invisible for the user). It will
+will be forwarded by the local etcd server such that it's invisible for the user). It will
affect latency for all operations, but it should not increase by much more than the network
latency between master replicas (latency between GCE zones with a region is < 10ms).
@@ -57,7 +57,7 @@ Currently etcd exposes port only using localhost interface. In order to allow cl
and inter-VM communication we will also have to use public interface. To secure the
communication we will use SSL (as described [here](https://coreos.com/etcd/docs/latest/security.html)).
-When generating command line for etcd we will always assume it’s part of a cluster
+When generating command line for etcd we will always assume it's part of a cluster
(initially of size 1) and list all existing kubernetes master replicas.
Based on that, we will set the following flags:
* `-initial-cluster` - list of all hostnames/DNS names for master replicas (including the new one)
diff --git a/contributors/design-proposals/high-availability.md b/contributors/design-proposals/high-availability.md
index da2f4fc9..ecaa4a99 100644
--- a/contributors/design-proposals/high-availability.md
+++ b/contributors/design-proposals/high-availability.md
@@ -1,7 +1,7 @@
# High Availability of Scheduling and Controller Components in Kubernetes
This document is deprecated. For more details about running a highly available
-cluster master, please see the [admin instructions document](../../docs/admin/high-availability.md).
+cluster master, please see the [admin instructions document](https://github.com/kubernetes/kubernetes/blob/master/docs/admin/high-availability.md).
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/high-availability.md?pixel)]()
diff --git a/contributors/design-proposals/initial-resources.md b/contributors/design-proposals/initial-resources.md
index f383f14a..432f6021 100644
--- a/contributors/design-proposals/initial-resources.md
+++ b/contributors/design-proposals/initial-resources.md
@@ -5,7 +5,7 @@ and set them before the container is run. This document describes design of the
## Motivation
-Since we want to make Kubernetes as simple as possible for its users we don’t want to require setting [Resources](../design/resource-qos.md) for container by its owner.
+Since we want to make Kubernetes as simple as possible for its users we don't want to require setting [Resources](../design/resource-qos.md) for container by its owner.
On the other hand having Resources filled is critical for scheduling decisions.
Current solution to set up Resources to hardcoded value has obvious drawbacks.
We need to implement a component which will set initial Resources to a reasonable value.
@@ -22,7 +22,7 @@ InitialResources will set only [request](../design/resource-qos.md#requests-and-
To make the component work with LimitRanger the estimated value will be capped by min and max possible values if defined.
It will prevent from situation when the pod is rejected due to too low or too high estimation.
-The container won’t be marked as managed by this component in any way, however appropriate event will be exported.
+The container won't be marked as managed by this component in any way, however appropriate event will be exported.
The predicting algorithm should have very low latency to not increase significantly e2e pod startup latency
[#3954](https://github.com/kubernetes/kubernetes/pull/3954).
diff --git a/contributors/design-proposals/kubectl-create-from-env-file.md b/contributors/design-proposals/kubectl-create-from-env-file.md
new file mode 100644
index 00000000..adc235db
--- /dev/null
+++ b/contributors/design-proposals/kubectl-create-from-env-file.md
@@ -0,0 +1,84 @@
+# Kubectl create configmap/secret --env-file
+
+## Goals
+
+Allow a Docker environment file (.env) to populate an entire `ConfigMap` or `Secret`.
+The populated `ConfigMap` or `Secret` can be referenced by a pod to load all
+the data contained within.
+
+## Design
+
+The `create configmap` subcommand would add a new option called
+`--from-env-file`. The option will accept a single file. The option may not be
+used in conjunction with `--from-file` or `--from-literal`.
+
+The `create secret generic` subcommand would add a new option called
+`--from-env-file`. The option will accept a single file. The option may not be
+used in conjunction with `--from-file` or `--from-literal`.
+
+### Environment file specification
+
+An environment file consists of lines to be in VAR=VAL format. Lines beginning
+with # (i.e. comments) are ignored, as are blank lines. Any whitespace in
+front of the VAR is removed. VAR must be a valid C_IDENTIFIER. If the line
+consists of just VAR, then the VAL will be given a value from the current
+environment.
+
+Any ill-formed line will be flagged as an error and will prevent the
+`ConfigMap` or `Secret` from being created.
+
+[Docker's environment file processing](https://github.com/docker/docker/blob/master/runconfig/opts/envfile.go)
+
+## Examples
+
+```
+$ cat game.env
+enemies=aliens
+lives=3
+enemies_cheat=true
+enemies_cheat_level=noGoodRotten
+secret_code_passphrase=UUDDLRLRBABAS
+secret_code_allowed=true
+secret_code_lives=30
+```
+
+Create configmap from an env file:
+```
+kubectl create configmap game-config --from-env-file=./game.env
+```
+
+The populated configmap would look like:
+```
+$ kubectl get configmaps game-config -o yaml
+
+apiVersion: v1
+data:
+ enemies: aliens
+ lives: 3
+ enemies_cheat: true
+ enemies_cheat_level: noGoodRotten
+ secret_code_passphrase: UUDDLRLRBABAS
+ secret_code_allowed: true
+ secret_code_lives: 30
+```
+
+Create secret from an env file:
+```
+kubectl create secret generic game-config --from-env-file=./game.env
+```
+
+The populated secret would look like:
+```
+$ kubectl get secret game-config -o yaml
+
+apiVersion: v1
+type: Opaque
+data:
+ enemies: YWxpZW5z
+ enemies_cheat: dHJ1ZQ==
+ enemies_cheat_level: bm9Hb29kUm90dGVu
+ lives: Mw==
+ secret_code_allowed: dHJ1ZQ==
+ secret_code_lives: MzA=
+ secret_code_passphrase: VVVERExSTFJCQUJBUw==
+```
diff --git a/contributors/design-proposals/kubelet-cri-logging.md b/contributors/design-proposals/kubelet-cri-logging.md
index 0881c42c..0cda9901 100644
--- a/contributors/design-proposals/kubelet-cri-logging.md
+++ b/contributors/design-proposals/kubelet-cri-logging.md
@@ -160,7 +160,7 @@ arbitrary container logs.
**Who should rotate the logs?**
We assume that a separate task (e.g., cron job) will be configured on the node
-to rotate the logs periodically, similar to today’s implementation.
+to rotate the logs periodically, similar to today's implementation.
We do not rule out the possibility of letting kubelet or a per-node daemon
(`DaemonSet`) to take up the responsibility, or even declare rotation policy
diff --git a/contributors/design-proposals/kubelet-rootfs-distribution.md b/contributors/design-proposals/kubelet-rootfs-distribution.md
new file mode 100644
index 00000000..00ddf07f
--- /dev/null
+++ b/contributors/design-proposals/kubelet-rootfs-distribution.md
@@ -0,0 +1,165 @@
+# Running Kubelet in a Chroot
+
+Authors: Vishnu Kannan \<vishh@google.com\>, Euan Kemp \<euan.kemp@coreos.com\>, Brandon Philips \<brandon.philips@coreos.com\>
+
+## Introduction
+
+The Kubelet is a critical component of Kubernetes that must be run on every node in a cluster.
+
+However, right now it's not always easy to run it *correctly*. The Kubelet has
+a number of dependencies that must exist in its filesystem, including various
+mount and network utilities. Missing any of these can lead to unexpected
+differences between Kubernetes hosts. For example, the Google Container VM
+image (GCI) is missing various mount commands even though the Kernel supports
+those filesystem types. Similarly, CoreOS Container Linux intentionally doesn't ship with
+many mount utilities or socat in the base image. Other distros have a related
+problem of ensuring these dependencies are present and versioned appropriately
+for the Kubelet.
+
+In order to solve this problem, it's proposed that running the Kubelet in a
+prepackaged chroot should be a supported, recommended, way of running a fully
+functioning Kubelet.
+
+## The Kubelet Chroot
+
+The easiest way to express all filesystem dependencies of the Kubelet comprehensively is to ship a filesystem image and run the Kubelet within it. The [hyperkube image](../../cluster/images/hyperkube/) already provides such a filesystem.
+
+Even though the hyperkube image is distributed as a container, this method of
+running the Kubelet intentionally is using a chroot and is neither a container nor pod.
+
+The kubelet chroot will essentially operate as follows:
+
+```
+container-download-and-extract gcr.io/google_containers/hyperkube:v1.4.0 /path/to/chroot
+mount --make-shared /var/lib/kubelet
+mount --rbind /var/lib/kubelet /path/to/chroot/var/lib/kubelet
+# And many more mounts, omitted
+...
+chroot /path/to/kubelet /usr/bin/hyperkube kubelet
+```
+
+Note: Kubelet might need access to more directories on the host and we intend to identity mount all those directories into the chroot. A partial list can be found in the CoreOS Container Linux kubelet-wrapper script.
+This logic will also naturally be abstracted so it's no more difficult for the user to run the Kubelet.
+
+Currently, the Kubelet does not need access to arbitrary paths on the host (as
+hostPath volumes are managed entirely by the docker daemon process, including
+SELinux context applying), so Kubelet makes no operations at those paths).
+
+This will likely change in the future, at which point a shared bindmount of `/`
+will be made available at a known path in the Kubelet chroot. This change will
+necessarily be more intrusive since it will require the kubelet to behave
+differently (use the shared rootfs mount's path) when running within the
+chroot.
+
+## Current Use
+
+This method of running the Kubelet is already in use by users of CoreOS Container Linux. The details of this implementation are found in the [kubelet wrapper documentation](https://coreos.com/kubernetes/docs/latest/kubelet-wrapper.html).
+
+## Implementation
+
+### Target Distros
+
+The two distros which benefit the most from this change are GCI and CoreOS Container Linux. Initially, these changes will only be implemented for those distros.
+
+This work will also only initially target the GCE provider and `kube-up` method of deployment.
+
+#### Hyperkube Image Packaging
+
+The Hyperkube image is distributed as part of an official release to the `gcr.io/google_containers` registry, but is not included along with the `kube-up` artifacts used for deployment.
+
+This will need to be remediated in order to complete this proposal.
+
+### Testing & Rollout
+
+In order to ensure the paths remain complete, e2e tests *must* be run against a
+Kubelet operating in this manner as part of the submit queue.
+
+To ensure that this feature does not unduly impact others, it will be added to
+GCI, but gated behind a feature-flag for a sort confidence-building period
+(e.g. `KUBE_RUN_HYPERKUBE_IMAGE=false`). A temporary non-blocking e2e job will
+be added with that option. If the results look clean after a week, the
+deployment option can be removed and the GCI image can completely switch over.
+
+Once that testing is in place, it can be rolled out across other distros as
+desired.
+
+
+#### Everything else
+
+In the initial implementation, rkt or docker can be used to extract the rootfs of the hyperkube image. rkt fly or a systemd unit (using [`RootDirectory`](https://www.freedesktop.org/software/systemd/man/systemd.exec.html#RootDirectory=)) can be used to perform the needed setup, chroot, and execution of the kubelet within that rootfs.
+
+
+
+## FAQ
+
+#### Will this replace or break other installation options?
+
+Other installation options include using RPMs, DEBs, and simply running the statically compiled Kubelet binary.
+
+All of these methods will continue working as they do now. In the future they may choose to also run the kubelet in this manner, but they don't necessarily have to.
+
+
+#### Is this running the kubelet as a pod?
+
+This is different than running the Kubelet as a pod. Rather than using namespaces, it uses only a chroot and shared bind mounts.
+
+## Alternatives
+
+#### Container + Shared bindmounts
+
+Instead of using a chroot with shared bindmounts, a proper pod or container could be used if the container supported shared bindmounts.
+
+This introduces some additional complexity in requiring something more than just the bare minimum. It also relies on having a container runtime available and puts said runtime in the critical path for the Kubelet.
+
+#### "Dependency rootfs" aware kubelet
+
+The Kubelet could be made aware of the rootfs containing all its dependencies, but not chrooted into it (e.g. started with a `--dependency-root-dir=/path/to/extracted/container` flag).
+
+The Kubelet could then always search for the binary it wishes to run in that path first and prefer it, as well as preferring libraries in that path. It would effectively run all dependencies similar to the following:
+
+```bash
+export PATH=${dep_root}/bin:${dep_root}/usr/bin:...
+export LD_LIBRARY_PATH=${dep_root}/lib:${dep_root}/usr/lib:...
+# Run 'mount':
+$ ${dep_root}/lib/x86_64-linux-gnu/ld.so --inhibit-cache mount $args
+```
+
+**Downsides**:
+
+This adds significant complexity and, due to the dynamic library hackery, might require some container-specific knowledge of the Kubelet or a rootfs of a predetermined form.
+
+This solution would also have to still solve the packaging of that rootfs, though the solution would likely be identical to the solution for distributing the chroot-kubelet-rootfs.
+
+#### Waiting for Flexv2 + port-forwarding changes
+
+The CRI effort plans to change how [port-forward](https://github.com/kubernetes/kubernetes/issues/29579) works, towards a method which will not depend explicitly on socat or other networking utilities.
+
+Similarly, for the mount utilities, the [Flex Volume v2](https://github.com/kubernetes/features/issues/93) feature is aiming to solve this utility.
+
+
+**Downsides**:
+
+This requires waiting on other features which might take a signficant time to land. It also could end up not fully fixing the problem (e.g. pushing down port-forwarding to the runtime doesn't ensure the runtime doesn't rely on host utilities).
+
+The Flex Volume feature is several releases out from fully replacing the current volumes as well.
+
+Finally, there are dependencies that neither of these proposals cover. An
+effort to identify these is underway [here](https://issues.k8s.io/26093).
+
+## Non-Alternatives
+
+#### Pod + containerized flag
+
+Currently, there's a `--containerized` flag. This flag doesn't actually remove the dependency on mount utilities on the node though, so does not solve the problem described here. It also is under consideration for [removal](https://issues.k8s.io/26093).
+
+## Open Questions
+
+#### Why not a mount namespace?
+
+#### Timeframe
+
+During the 1.6 timeframe, the changes mentioned in implementation will be undergone for the CoreOS Container Linux and GCI distros.
+
+Based on the test results and additional problems that may arise, rollout will
+be determined from there. Hopefully the rollout can also occur in the 1.6
+timeframe.
diff --git a/contributors/design-proposals/kubemark.md b/contributors/design-proposals/kubemark.md
index 1f28e2b0..95888c94 100644
--- a/contributors/design-proposals/kubemark.md
+++ b/contributors/design-proposals/kubemark.md
@@ -10,7 +10,7 @@ detailed discussion.
Currently performance testing happens on ‘live’ clusters of up to 100 Nodes. It takes quite a while to start such cluster or to push
updates to all Nodes, and it uses quite a lot of resources. At this scale the amount of wasted time and used resources is still acceptable.
-In the next quarter or two we’re targeting 1000 Node cluster, which will push it way beyond ‘acceptable’ level. Additionally we want to
+In the next quarter or two we're targeting 1000 Node cluster, which will push it way beyond ‘acceptable’ level. Additionally we want to
enable people without many resources to run scalability tests on bigger clusters than they can afford at given time. Having an ability to
cheaply run scalability tests will enable us to run some set of them on "normal" test clusters, which in turn would mean ability to run
them on every PR.
@@ -18,7 +18,7 @@ them on every PR.
This means that we need a system that will allow for realistic performance testing on (much) smaller number of “real” machines. First
assumption we make is that Nodes are independent, i.e. number of existing Nodes do not impact performance of a single Node. This is not
entirely true, as number of Nodes can increase latency of various components on Master machine, which in turn may increase latency of Node
-operations, but we’re not interested in measuring this effect here. Instead we want to measure how number of Nodes and the load imposed by
+operations, but we're not interested in measuring this effect here. Instead we want to measure how number of Nodes and the load imposed by
Node daemons affects the performance of Master components.
## Kubemark architecture overview
@@ -30,7 +30,7 @@ initial version). To teach Hollow components replaying recorded traffic they wil
should die (e.g. observed lifetime). Such data can be extracted e.g. from etcd Raft logs, or it can be reconstructed from Events. In the
initial version we only want them to be able to fool Master components and put some configurable (in what way TBD) load on them.
-When we have Hollow Node ready, we’ll be able to test performance of Master Components by creating a real Master Node, with API server,
+When we have Hollow Node ready, we'll be able to test performance of Master Components by creating a real Master Node, with API server,
Controllers, etcd and whatnot, and create number of Hollow Nodes that will register to the running Master.
To make Kubemark easier to maintain when system evolves Hollow components will reuse real "production" code for Kubelet and KubeProxy, but
@@ -83,8 +83,8 @@ Pod on each Node that exports logs to Elasticsearch (or Google Cloud Logging). B
cluster so do not add any load on a Master components by themselves. There can be other systems that scrape Heapster through proxy running
on Master, which adds additional load, but they're not the part of default setup, so in the first version we won't simulate this behavior.
-In the first version we’ll assume that all started Pods will run indefinitely if not explicitly deleted. In the future we can add a model
-of short-running batch jobs, but in the initial version we’ll assume only serving-like Pods.
+In the first version we'll assume that all started Pods will run indefinitely if not explicitly deleted. In the future we can add a model
+of short-running batch jobs, but in the initial version we'll assume only serving-like Pods.
### Heapster
@@ -138,7 +138,7 @@ don't need to solve this problem now.
- new HollowNode combining the two,
- make sure that Master can talk to two HollowKubelets running on the same machine
- Make sure that we can run Hollow cluster on top of Kubernetes [option 2](#option-2)
-- Write a player that will automatically put some predefined load on Master, <- this is the moment when it’s possible to play with it and is useful by itself for
+- Write a player that will automatically put some predefined load on Master, <- this is the moment when it's possible to play with it and is useful by itself for
scalability tests. Alternatively we can just use current density/load tests,
- Benchmark our machines - see how many Watch clients we can have before everything explodes,
- See how many HollowNodes we can run on a single machine by attaching them to the real master <- this is the moment it starts to useful
diff --git a/contributors/design-proposals/monitoring_architecture.md b/contributors/design-proposals/monitoring_architecture.md
index b819eeca..e19c52a1 100644
--- a/contributors/design-proposals/monitoring_architecture.md
+++ b/contributors/design-proposals/monitoring_architecture.md
@@ -99,12 +99,12 @@ version of today's Heapster. metrics-server stores locally only latest values an
metrics-server exposes the master metrics API. (The configuration described here is similar
to the current Heapster in “standalone” mode.)
[Discovery summarizer](../../docs/proposals/federated-api-servers.md)
-makes the master metrics API available to external clients such that from the client’s perspective
+makes the master metrics API available to external clients such that from the client's perspective
it looks the same as talking to the API server.
Core (system) metrics are handled as described above in all deployment environments. The only
easily replaceable part is resource estimator, which could be replaced by power users. In
-theory, metric-server itself can also be substituted, but it’d be similar to substituting
+theory, metric-server itself can also be substituted, but it'd be similar to substituting
apiserver itself or controller-manager - possible, but not recommended and not supported.
Eventually the core metrics pipeline might also collect metrics from Kubelet and Docker daemon
@@ -170,7 +170,7 @@ cAdvisor + Heapster + InfluxDB (or any other sink)
* snapd + SNAP cluster-level agent
* Sysdig
-As an example we’ll describe a potential integration with cAdvisor + Prometheus.
+As an example we'll describe a potential integration with cAdvisor + Prometheus.
Prometheus has the following metric sources on a node:
* core and non-core system metrics from cAdvisor
diff --git a/contributors/design-proposals/multi-platform.md b/contributors/design-proposals/multi-platform.md
index 36eacefa..145c0ab9 100644
--- a/contributors/design-proposals/multi-platform.md
+++ b/contributors/design-proposals/multi-platform.md
@@ -277,7 +277,7 @@ A great blog post [that is describing this](https://medium.com/@rakyll/go-1-5-cr
Before Go 1.5, the whole Go project had to be cross-compiled from source for **all** platforms that _might_ be used, and that was quite a slow process:
```console
-# From build-tools/build-image/cross/Dockerfile when we used Go 1.4
+# From build/build-image/cross/Dockerfile when we used Go 1.4
$ cd /usr/src/go/src
$ for platform in ${PLATFORMS}; do GOOS=${platform%/*} GOARCH=${platform##*/} ./make.bash --no-clean; done
```
@@ -288,7 +288,7 @@ If you cross-compile multiple times, Go will build parts of `std`, throw it away
However, there is an easy way of cross-compiling all `std` packages in advance with Go 1.5+:
```console
-# From build-tools/build-image/cross/Dockerfile when we're using Go 1.5+
+# From build/build-image/cross/Dockerfile when we're using Go 1.5+
$ for platform in ${PLATFORMS}; do GOOS=${platform%/*} GOARCH=${platform##*/} go install std; done
```
@@ -377,7 +377,7 @@ In order to dynamically compile a go binary with `cgo`, we need `gcc` installed
The only Kubernetes binary that is using C code is the `kubelet`, or in fact `cAdvisor` on which `kubelet` depends. `hyperkube` is also dynamically linked as long as `kubelet` is. We should aim to make `kubelet` statically linked.
-The normal `x86_64-linux-gnu` can't cross-compile binaries, so we have to install gcc cross-compilers for every platform. We do this in the [`kube-cross`](../../build-tools/build-image/cross/Dockerfile) image,
+The normal `x86_64-linux-gnu` can't cross-compile binaries, so we have to install gcc cross-compilers for every platform. We do this in the [`kube-cross`](https://github.com/kubernetes/kubernetes/blob/master/build/build-image/cross/Dockerfile) image,
and depend on the [`emdebian.org` repository](https://wiki.debian.org/CrossToolchains). Depending on `emdebian` isn't ideal, so we should consider using the latest `gcc` cross-compiler packages from the `ubuntu` main repositories in the future.
Here's an example when cross-compiling plain C code:
diff --git a/contributors/design-proposals/network-policy.md b/contributors/design-proposals/network-policy.md
index ff75aa57..e4f09283 100644
--- a/contributors/design-proposals/network-policy.md
+++ b/contributors/design-proposals/network-policy.md
@@ -96,7 +96,7 @@ The user needs a way to explicitly declare which connections are allowed into po
This is accomplished through ingress rules on `NetworkPolicy`
objects (of which there can be multiple in a single namespace). Pods selected by
one or more NetworkPolicy objects should allow any incoming connections that match any
-ingress rule on those NetworkPolicy objects, per the network plugin’s capabilities.
+ingress rule on those NetworkPolicy objects, per the network plugin's capabilities.
NetworkPolicy objects and the above namespace isolation both act on _connections_ rather than individual packets. That is to say that if traffic from pod A to pod B is allowed by the configured
policy, then the return packets for that connection from B -> A are also allowed, even if the policy in place would not allow B to initiate a connection to A. NetworkPolicy objects act on a broad definition of _connection_ which includes both TCP and UDP streams. If new network policy is applied that would block an existing connection between two endpoints, the enforcer of policy
diff --git a/contributors/design-proposals/networking.md b/contributors/design-proposals/networking.md
index 6e269481..df8c8f82 100644
--- a/contributors/design-proposals/networking.md
+++ b/contributors/design-proposals/networking.md
@@ -35,7 +35,7 @@ among other problems.
## Container to container
All containers within a pod behave as if they are on the same host with regard
-to networking. They can all reach each other’s ports on localhost. This offers
+to networking. They can all reach each other's ports on localhost. This offers
simplicity (static ports know a priori), security (ports bound to localhost
are visible within the pod but never outside it), and performance. This also
reduces friction for applications moving from the world of uncontainerized apps
@@ -180,10 +180,11 @@ and the network plugin architecture Kubernetes uses needs to allow returning
IPv6 addresses too [CNI issue #245](https://github.com/containernetworking/cni/issues/245).
Kubernetes code that deals with IP addresses must then be audited and fixed to
support both IPv4 and IPv6 addresses and not assume IPv4.
-Additionally, direct ipv6 assignment to instances doesn't appear to be supported
-by major cloud providers (e.g., AWS EC2, GCE) yet. We'd happily take pull
-requests from people running Kubernetes on bare metal, though. :-)
-
+AWS started rolling out basic
+[ipv6 support](https://aws.amazon.com/about-aws/whats-new/2016/12/announcing-internet-protocol-version-6-support-for-ec2-instances-in-amazon-virtual-private-cloud/),
+but direct ipv6 assignment to instances doesn't appear to be supported by other
+major cloud providers (e.g. GCE) yet. We'd happily take pull requests from people
+running Kubernetes on bare metal, though. :-)
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/design/networking.md?pixel)]()
diff --git a/contributors/design-proposals/node-allocatable.md b/contributors/design-proposals/node-allocatable.md
index ac9f46c4..12f244eb 100644
--- a/contributors/design-proposals/node-allocatable.md
+++ b/contributors/design-proposals/node-allocatable.md
@@ -99,7 +99,7 @@ behavior is equivalent to the 1.1 behavior with scheduling based on Capacity.
#### System-Reserved
In the initial implementation, `SystemReserved` will be functionally equivalent to
-[`KubeReserved`](#system-reserved), but with a different semantic meaning. While KubeReserved
+[`KubeReserved`](#kube-reserved), but with a different semantic meaning. While KubeReserved
designates resources set aside for kubernetes components, SystemReserved designates resources set
aside for non-kubernetes components (currently this is reported as all the processes lumped
together in the `/system` raw container).
diff --git a/contributors/design-proposals/optional-configmap.md b/contributors/design-proposals/optional-configmap.md
new file mode 100644
index 00000000..715ac6a7
--- /dev/null
+++ b/contributors/design-proposals/optional-configmap.md
@@ -0,0 +1,174 @@
+# Optional ConfigMaps and Secrets
+
+## Goal
+
+Allow the ConfigMaps or Secrets that are used to populate the environment variables of a
+container and files within a Volume to be optional.
+
+## Use Cases
+
+When deploying an application to multiple environments like development, test,
+and production, there may be certain environment variables that must reflect
+the values that are relevant to said environment. One way to do so would be to
+have a well named ConfigMap which contains all the environment variables
+needed. With the introduction of optional ConfigMaps, one could instead define a required
+ConfigMap which contains all the environment variables for any environment
+with a set of initialized or default values. An additional optional ConfigMap
+can also be specified which allows the deployer to provide any overrides for
+the current environment.
+
+An application developer can populate a volume with files defined from a
+ConfigMap. The developer may have some required files to be created and have
+optional additional files at a different target. The developer can specify on
+the Pod that there is an optional ConfigMap that will provide these additional
+files if the ConfigMap exists.
+
+## Design Points
+
+A container can specify an entire ConfigMap to be populated as environment
+variables via `EnvFrom`. When required, the container fails to start if the
+ConfigMap does not exist. If the ConfigMap is optional, the container will
+skip the non-existant ConfigMap and proceed as normal.
+
+A container may also specify a single environment variable to retrieve its
+value from a ConfigMap via `Env`. If the key does not exist in the ConfigMap
+during container start, the container will fail to start. If however, the
+ConfigMap is marked optional, during container start, a non-existant ConfigMap
+or a missing key in the ConfigMap will not prevent the container from
+starting. Any previous value for the given key will be used.
+
+Any changes to the ConfigMap will not affect environment variables of running
+containers. If the Container is restarted, the set of environment variables
+will be re-evaluated.
+
+The same processing rules applies to Secrets.
+
+A pod can specify a set of Volumes to mount. A ConfigMap can represent the
+files to populate the volume. The ConfigMaps can be marked as optional. The
+default is to require the ConfigMap existence. If the ConfigMap is required
+and does not exist, the volume creation will fail. If the ConfigMap is marked
+as optional, the volume will be created regardless, and the files will be
+populated only if the ConfigMap exists and has content. If the ConfigMap is
+changed, the volume will eventually reflect the new set of data available from
+the ConfigMap.
+
+## Proposed Design
+
+To support an optional ConfigMap either as a ConfigMapKeySelector, ConfigMapEnvSource or a
+ConfigMapVolumeSource, a boolean will be added to specify whether it is
+optional. The default will be required.
+
+To support an optional Secret either as a SecretKeySelector, or a
+SecretVolumeSource, a boolean will be added to specify whether it is optional.
+The default will be required.
+
+### Kubectl updates
+
+The `describe` command will display the additional optional field of the
+ConfigMap and Secret for both the environment variables and volume sources.
+
+### API Resource
+
+A new `Optional` field of type boolean will be added.
+
+```go
+type ConfigMapKeySelector struct {
+ // Specify whether the ConfigMap must be defined
+ // +optional
+ Optional *bool `json:"optional,omitempty" protobuf:"varint,3,opt,name=optional"`
+}
+
+type ConfigMapEnvSource struct {
+ // Specify whether the ConfigMap must be defined
+ // +optional
+ Optional *bool `json:"optional,omitempty" protobuf:"varint,2,opt,name=optional"`
+}
+
+type ConfigMapVolumeSource struct {
+ // Specify whether the ConfigMap must be defined
+ // +optional
+ Optional *bool `json:"optional,omitempty" protobuf:"varint,4,opt,name=optional"`
+}
+
+type SecretKeySelector struct {
+ // Specify whether the ConfigMap must be defined
+ // +optional
+ Optional *bool `json:"optional,omitempty" protobuf:"varint,3,opt,name=optional"`
+}
+
+type SecretVolumeSource struct {
+ // Specify whether the Secret must be defined
+ // +optional
+ Optional *bool `json:"optional,omitempty" protobuf:"varint,4,opt,name=optional"`
+}
+```
+
+### Examples
+
+Optional `ConfigMap` as Environment Variables
+
+```yaml
+apiVersion: v1
+kind: Pod
+metadata:
+ name: config-env-example
+spec:
+ containers:
+ - name: etcd
+ image: openshift/etcd-20-centos7
+ ports:
+ - containerPort: 2379
+ protocol: TCP
+ - containerPort: 2380
+ protocol: TCP
+ env:
+ - name: foo
+ valueFrom:
+ configMapKeyRef:
+ name: etcd-env-config
+ key: port
+ optional: true
+```
+
+```yaml
+apiVersion: v1
+kind: Pod
+metadata:
+ name: config-env-example
+spec:
+ containers:
+ - name: etcd
+ image: openshift/etcd-20-centos7
+ ports:
+ - containerPort: 2379
+ protocol: TCP
+ - containerPort: 2380
+ protocol: TCP
+ envFrom:
+ - configMap:
+ name: etcd-env-config
+ optional: true
+```
+
+Optional `ConfigMap` as a VolumeSource
+
+```yaml
+apiVersion: v1
+kind: Pod
+metadata:
+ name: config-env-example
+spec:
+ volumes:
+ - name: pod-configmap-volume
+ configMap:
+ name: configmap-test-volume
+ optional: true
+ containers:
+ - name: etcd
+ image: openshift/etcd-20-centos7
+ ports:
+ - containerPort: 2379
+ protocol: TCP
+ - containerPort: 2380
+ protocol: TCP
+```
diff --git a/contributors/design-proposals/performance-related-monitoring.md b/contributors/design-proposals/performance-related-monitoring.md
index f70da39b..7937dbdd 100644
--- a/contributors/design-proposals/performance-related-monitoring.md
+++ b/contributors/design-proposals/performance-related-monitoring.md
@@ -20,13 +20,13 @@ lost because of this issue.
### Adding per-pod probe-time, which increased the number of PodStatus updates, causing major slowdown
In September 2015 we tried to add per-pod probe times to the PodStatus. It caused (https://github.com/kubernetes/kubernetes/issues/14273) a massive increase in both number and
-total volume of object (PodStatus) changes. It drastically increased the load on API server which wasn’t able to handle new number of requests quickly enough, violating our
+total volume of object (PodStatus) changes. It drastically increased the load on API server which wasn't able to handle new number of requests quickly enough, violating our
response time SLO. We had to revert this change.
### Late Ready->Running PodPhase transition caused test failures as it seemed like slowdown
In late September we encountered a strange problem (https://github.com/kubernetes/kubernetes/issues/14554): we observed an increased observed latencies in small clusters (few
-Nodes). It turned out that it’s caused by an added latency between PodRunning and PodReady phases. This was not a real regression, but our tests thought it were, which shows
+Nodes). It turned out that it's caused by an added latency between PodRunning and PodReady phases. This was not a real regression, but our tests thought it were, which shows
how careful we need to be.
### Huge number of handshakes slows down API server
diff --git a/contributors/design-proposals/pod-pid-namespace.md b/contributors/design-proposals/pod-pid-namespace.md
new file mode 100644
index 00000000..43c38f22
--- /dev/null
+++ b/contributors/design-proposals/pod-pid-namespace.md
@@ -0,0 +1,77 @@
+# Shared PID Namespace
+
+Pods share namespaces where possible, but a requirement for sharing the PID
+namespace has not been defined due to lack of support in Docker. Docker began
+supporting a shared PID namespace in 1.12, and other Kubernetes runtimes (rkt,
+cri-o, hyper) have already implemented a shared PID namespace.
+
+This proposal defines a shared PID namespace as a requirement of the Container
+Runtime Interface and links its rollout in Docker to that of the CRI.
+
+## Motivation
+
+Sharing a PID namespace between containers in a pod is discussed in
+[#1615](https://issues.k8s.io/1615), and enables:
+
+ 1. signaling between containers, which is useful for side cars (e.g. for
+ signaling a daemon process after rotating logs).
+ 2. easier troubleshooting of pods.
+ 3. addressing [Docker's zombie problem][1] by reaping orphaned zombies in the
+ infra container.
+
+## Goals and Non-Goals
+
+Goals include:
+ - Changing default behavior in the Docker runtime as implemented by the CRI
+ - Making Docker behavior compatible with the other Kubernetes runtimes
+
+Non-goals include:
+ - Creating an init solution that works for all runtimes
+ - Supporting isolated PID namespace indefinitely
+
+## Modification to the Docker Runtime
+
+We will modify the Docker implementation of the CRI to use a shared PID
+namespace when running with a version of Docker >= 1.12. The legacy
+`dockertools` implementation will not be changed.
+
+Linking this change to the CRI means that Kubernetes users who care to test such
+changes can test the combined changes at once. Users who do not care to test
+such changes will be insulated by Kubernetes not recommending Docker >= 1.12
+until after switching to the CRI.
+
+Other changes that must be made to support this change:
+
+1. Add a test to verify all containers restart if the infra container
+ responsible for the PodSandbox dies. (Note: With Docker 1.12 if the source
+ of the PID namespace dies all containers sharing that namespace are killed
+ as well.)
+2. Modify the Infra container used by the Docker runtime to reap orphaned
+ zombies ([#36853](https://pr.k8s.io/36853)).
+
+## Rollout Plan
+
+SIG Node is planning to switch to the CRI as a default in 1.6, at which point
+users with Docker >= 1.12 will receive a shared PID namespace by default.
+Cluster administrators will be able to disable this behavior by providing a flag
+to the kubelet which will cause the dockershim to revert to previous behavior.
+
+The ability to disable shared PID namespaces is intended as a way to roll back
+to prior behavior in the event of unforeseen problems. It won't be possible to
+configure the behavior per-pod. We believe this is acceptable because:
+
+* We have not identified a concrete use case requiring isolated PID namespaces.
+* Making PID namespace configurable requires changing the CRI, which we would
+ like to avoid since there are no use cases.
+
+In a future release, SIG Node will recommend docker >= 1.12. Unless a compelling
+use case for isolated PID namespaces is discovered, we will remove the ability
+to disable the shared PID namespace in the subsequent release.
+
+
+[1]: https://blog.phusion.nl/2015/01/20/docker-and-the-pid-1-zombie-reaping-problem/
+
+
+<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
+[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/pod-pid-namespace.md?pixel)]()
+<!-- END MUNGE: GENERATED_ANALYTICS -->
diff --git a/contributors/design-proposals/pod-resource-management.md b/contributors/design-proposals/pod-resource-management.md
index 39f939e3..fc66a0dc 100644
--- a/contributors/design-proposals/pod-resource-management.md
+++ b/contributors/design-proposals/pod-resource-management.md
@@ -350,7 +350,7 @@ Two top level cgroups for `Bu` and `BE` QoS classes are created when Kubelet sta
#### Pod level Cgroup creation and deletion (Docker runtime)
- When a new pod is brought up, its QoS class is firstly determined.
-- We add an interface to Kubelet’s ContainerManager to create and delete pod level cgroups under the cgroup that matches the pod’s QoS class.
+- We add an interface to Kubelet's ContainerManager to create and delete pod level cgroups under the cgroup that matches the pod's QoS class.
- This interface will be pluggable. Kubelet will support both systemd and raw cgroups based __cgroup__ drivers. We will be using the --cgroup-driver flag proposed in the [Systemd Node Spec](kubelet-systemd.md) to specify the cgroup driver.
- We inject creation and deletion of pod level cgroups into the pod workers.
- As new pods are added QoS class cgroup parameters are updated to match the resource requests by the Pod.
@@ -365,7 +365,7 @@ We want to have rkt create pods under a root QoS class that kubelet specifies, a
#### Add Pod level metrics to Kubelet's metrics provider
-Update Kubelet’s metrics provider to include Pod level metrics. Use cAdvisor's cgroup subsystem information to determine various Pod level usage metrics.
+Update Kubelet's metrics provider to include Pod level metrics. Use cAdvisor's cgroup subsystem information to determine various Pod level usage metrics.
`Note: Changes to cAdvisor might be necessary.`
@@ -393,7 +393,7 @@ Updating QoS limits needs to happen before pod cgroups values are updated. When
Other smaller work items that we would be good to have before the release of this feature.
- [ ] Add Pod UID to the downward api which will help simplify the e2e testing logic.
-- [ ] Check if parent cgroup exist and error out if they don’t.
+- [ ] Check if parent cgroup exist and error out if they don't.
- [ ] Set top level cgroup limit to resource allocatable until we support QoS level cgroup updates. If cgroup root is not `/` then set node resource allocatable as the cgroup resource limits on cgroup root.
- [ ] Add a NodeResourceAllocatableProvider which returns the amount of allocatable resources on the nodes. This interface would be used both by the Kubelet and ContainerManager.
- [ ] Add top level feasibility check to ensure that pod can be admitted on the node by estimating left over resources on the node.
@@ -403,7 +403,7 @@ Other smaller work items that we would be good to have before the release of thi
To better support our requirements we needed to make some changes/add features to Libcontainer as well
- [x] Allowing or denying all devices by writing 'a' to devices.allow or devices.deny is
-not possible once the device cgroups has children. Libcontainer doesn’t have the option of skipping updates on parent devices cgroup. opencontainers/runc/pull/958
+not possible once the device cgroups has children. Libcontainer doesn't have the option of skipping updates on parent devices cgroup. opencontainers/runc/pull/958
- [x] To use libcontainer for creating and managing cgroups in the Kubelet, I would like to just create a cgroup with no pid attached and if need be apply a pid to the cgroup later on. But libcontainer did not support cgroup creation without attaching a pid. opencontainers/runc/pull/956
diff --git a/contributors/design-proposals/propagation.md b/contributors/design-proposals/propagation.md
index 7b1d4dfb..76949d5c 100644
--- a/contributors/design-proposals/propagation.md
+++ b/contributors/design-proposals/propagation.md
@@ -37,7 +37,7 @@ The slave mount namespace is the correct solution for this AFAICS. Until this
becomes available in k8s, we will have to have operations restart containers manually
based on monitoring alerts.
-1. (From @victorgp) When using CoreOS that does not provides external fuse systems
+1. (From @victorgp) When using CoreOS Container Linux that does not provides external fuse systems
like, in our case, GlusterFS, and you need a container to do the mounts. The only
way to see those mounts in the host, hence also visible by other containers, is by
sharing the mount propagation.
diff --git a/contributors/design-proposals/release-test-signal.md b/contributors/design-proposals/release-test-signal.md
new file mode 100644
index 00000000..6975629d
--- /dev/null
+++ b/contributors/design-proposals/release-test-signal.md
@@ -0,0 +1,130 @@
+# Overview
+
+Describes the process and tooling (`find_green_build`) used to find a
+binary signal from the Kubernetes testing framework for the purposes of
+selecting a release candidate. Currently this process is used to gate
+all Kubernetes releases.
+
+## Motivation
+
+Previously, the guidance in the [(now deprecated) release document](https://github.com/kubernetes/kubernetes/blob/fc3ef9320eb9d8211d85fbc404e4bbdd751f90af/docs/devel/releasing.md)
+was to "look for green tests". That is, of course, decidedly insufficient.
+
+Software releases should have the goal of being primarily automated and
+having a gating binary test signal is a key component to that ultimate goal.
+
+## Design
+
+### General
+
+The idea is to capture and automate the existing manual methods of
+finding a green signal for testing.
+
+* Identify a green run from the primary job `ci-kubernetes-e2e-gce`
+* Identify matching green runs from the secondary jobs
+
+The tooling should also have a simple and common interface whether using it
+for a dashboard, to gate a release within anago or for an individual to use it
+to check the state of testing at any time.
+
+Output looks like this:
+
+```
+$ find_green_build
+find_green_build: BEGIN main on djmm Mon Dec 19 16:28:15 PST 2016
+
+Checking for a valid github API token: OK
+Checking required system packages: OK
+Checking/setting cloud tools: OK
+
+Getting ci-kubernetes-e2e-gce build results from Jenkins...
+Getting ci-kubernetes-e2e-gce-serial build results from Jenkins...
+Getting ci-kubernetes-e2e-gce-slow build results from Jenkins...
+Getting ci-kubernetes-kubemark-5-gce build results from Jenkins...
+Getting ci-kubernetes-e2e-gce-reboot build results from Jenkins...
+Getting ci-kubernetes-e2e-gce-scalability build results from Jenkins...
+Getting ci-kubernetes-test-go build results from Jenkins...
+Getting ci-kubernetes-cross-build build results from Jenkins...
+Getting ci-kubernetes-e2e-gke-serial build results from Jenkins...
+Getting ci-kubernetes-e2e-gke build results from Jenkins...
+Getting ci-kubernetes-e2e-gke-slow build results from Jenkins...
+
+(*) Primary job (-) Secondary jobs
+
+Jenkins Job Run # Build # Time/Status
+= ================================= ====== ======= ===========
+* ci-kubernetes-e2e-gce #1668 #2347 [14:46 12/19]
+* (--buildversion=v1.6.0-alpha.0.2347+9925b68038eacc)
+- ci-kubernetes-e2e-gce-serial -- -- GIVE UP
+
+* ci-kubernetes-e2e-gce #1666 #2345 [13:23 12/19]
+* (--buildversion=v1.6.0-alpha.0.2345+523ff93471b052)
+- ci-kubernetes-e2e-gce-serial -- -- GIVE UP
+
+* ci-kubernetes-e2e-gce #1664 #2341 [09:38 12/19]
+* (--buildversion=v1.6.0-alpha.0.2341+def802272904c0)
+- ci-kubernetes-e2e-gce-serial -- -- GIVE UP
+
+* ci-kubernetes-e2e-gce #1662 #2339 [08:45 12/19]
+* (--buildversion=v1.6.0-alpha.0.2339+ce67a03b81dee5)
+- ci-kubernetes-e2e-gce-serial -- -- GIVE UP
+
+* ci-kubernetes-e2e-gce #1653 #2335 [07:42 12/19]
+* (--buildversion=v1.6.0-alpha.0.2335+d6046aab0e0678)
+- ci-kubernetes-e2e-gce-serial #192 #2335 PASSED
+- ci-kubernetes-e2e-gce-slow #989 #2335 PASSED
+- ci-kubernetes-kubemark-5-gce #2602 #2335 PASSED
+- ci-kubernetes-e2e-gce-reboot #1523 #2335 PASSED
+- ci-kubernetes-e2e-gce-scalability #460 #2335 PASSED
+- ci-kubernetes-test-go #1266 #2335 PASSED
+- ci-kubernetes-cross-build -- -- GIVE UP
+
+* ci-kubernetes-e2e-gce #1651 #2330 [06:43 12/19]
+* (--buildversion=v1.6.0-alpha.0.2330+75dfb21018a7c3)
+- ci-kubernetes-e2e-gce-serial #191 #2319 PASSED
+- ci-kubernetes-e2e-gce-slow #988 #2330 PASSED
+- ci-kubernetes-kubemark-5-gce #2599 #2330 PASSED
+- ci-kubernetes-e2e-gce-reboot #1521 #2330 PASSED
+- ci-kubernetes-e2e-gce-scalability #459 #2321 PASSED
+- ci-kubernetes-test-go #1264 #2330 PASSED
+- ci-kubernetes-cross-build #320 #2330 PASSED
+- ci-kubernetes-e2e-gke-serial #233 #2319 PASSED
+- ci-kubernetes-e2e-gke #1834 #2330 PASSED
+- ci-kubernetes-e2e-gke-slow #1041 #2330 PASSED
+
+JENKINS_BUILD_VERSION=v1.6.0-alpha.0.2330+75dfb21018a7c3
+RELEASE_VERSION[alpha]=v1.6.0-alpha.1
+RELEASE_VERSION_PRIME=v1.6.0-alpha.1
+```
+
+### v1
+
+The initial release of this analyzer did everything on the client side.
+This was slow to grab 100s of individual test results from GCS.
+This was mitigated somewhat by building a local cache, but for those that
+weren't using it regularly, the cache building step was a significant
+(~1 minute) hit when just trying to check the test status.
+
+### v2
+
+Building and storing that local cache on the jenkins server at build time
+was the way to speed things up. Getting the cache from GCS is now consistent
+for all users at ~10 seconds. After that the analyzer is running.
+
+
+## Uses
+
+`find_green_build` and its functions are used in 3 ways:
+
+1. During the release process itself via `anago`.
+1. When creating a pending release notes report via `relnotes --preview`,
+ used in creating dashboards
+1. By an individual to get a quick check on the binary signal status of jobs
+
+## Future work
+
+1. There may be other ways to improve the performance of this check by
+ doing more work server side.
+1. Using the `relnotes --preview` output to generate an external dashboard
+ will give more real-time visibility to both candidate release notes and
+ testing state.
diff --git a/contributors/design-proposals/rescheduling-for-critical-pods.md b/contributors/design-proposals/rescheduling-for-critical-pods.md
index 1d2d80ee..525e0805 100644
--- a/contributors/design-proposals/rescheduling-for-critical-pods.md
+++ b/contributors/design-proposals/rescheduling-for-critical-pods.md
@@ -9,7 +9,7 @@ by evicting a critical addon (either manually or as a side effect of an other op
which possibly can become pending (for example when the cluster is highly utilized).
To avoid such situation we want to have a mechanism which guarantees that
critical addons are scheduled assuming the cluster is big enough.
-This possibly may affect other pods (including production user’s applications).
+This possibly may affect other pods (including production user's applications).
## Design
@@ -33,13 +33,13 @@ Later we may want to introduce some heuristic:
* minimize number of evicted pods with violation of disruption budget or shortened termination grace period
* minimize number of affected pods by choosing a node on which we have to evict less pods
* increase probability of scheduling of evicted pods by preferring a set of pods with the smallest total sum of requests
-* avoid nodes which are ‘non-drainable’ (according to drain logic), for example on which there is a pod which doesn’t belong to any RC/RS/Deployment
+* avoid nodes which are ‘non-drainable’ (according to drain logic), for example on which there is a pod which doesn't belong to any RC/RS/Deployment
#### Evicting pods
There are 2 mechanism which possibly can delay a pod eviction: Disruption Budget and Termination Grace Period.
-While removing a pod we will try to avoid violating Disruption Budget, though we can’t guarantee it
+While removing a pod we will try to avoid violating Disruption Budget, though we can't guarantee it
since there is a chance that it would block this operation for longer period of time.
We will also try to respect Termination Grace Period, though without any guarantee.
In case we have to remove a pod with termination grace period longer than 10s it will be shortened to 10s.
@@ -70,7 +70,7 @@ This situation would be rare and usually an extra node would be anyway needed fo
In the worst case CA will add and then remove the node.
To not complicate architecture by introducing interaction between those 2 components we accept this overlap.
-We want to ensure that CA won’t remove nodes with critical addons by adding appropriate logic there.
+We want to ensure that CA won't remove nodes with critical addons by adding appropriate logic there.
### Rescheduler control loop
diff --git a/contributors/design-proposals/resource-qos.md b/contributors/design-proposals/resource-qos.md
index 9e8ba4ee..7c84e778 100644
--- a/contributors/design-proposals/resource-qos.md
+++ b/contributors/design-proposals/resource-qos.md
@@ -167,7 +167,7 @@ Under system memory pressure, these containers are more likely to be killed once
Pod OOM score configuration
- Note that the OOM score of a process is 10 times the % of memory the process consumes, adjusted by OOM_SCORE_ADJ, barring exceptions (e.g. process is launched by root). Processes with higher OOM scores are killed.
-- The base OOM score is between 0 and 1000, so if process A’s OOM_SCORE_ADJ - process B’s OOM_SCORE_ADJ is over a 1000, then process A will always be OOM killed before B.
+- The base OOM score is between 0 and 1000, so if process A's OOM_SCORE_ADJ - process B's OOM_SCORE_ADJ is over a 1000, then process A will always be OOM killed before B.
- The final OOM score of a process is also between 0 and 1000
*Best-effort*
@@ -194,7 +194,7 @@ Pod OOM score configuration
- OOM_SCORE_ADJ: -998
*Kubelet, Docker*
- - OOM_SCORE_ADJ: -999 (won’t be OOM killed)
+ - OOM_SCORE_ADJ: -999 (won't be OOM killed)
- Hack, because these critical tasks might die if they conflict with guaranteed containers. In the future, we should place all user-pods into a separate cgroup, and set a limit on the memory they can consume.
## Known issues and possible improvements
@@ -203,7 +203,7 @@ The above implementation provides for basic oversubscription with protection, bu
#### Support for Swap
-- The current QoS policy assumes that swap is disabled. If swap is enabled, then resource guarantees (for pods that specify resource requirements) will not hold. For example, suppose 2 guaranteed pods have reached their memory limit. They can continue allocating memory by utilizing disk space. Eventually, if there isn’t enough swap space, processes in the pods might get killed. The node must take into account swap space explicitly for providing deterministic isolation behavior.
+- The current QoS policy assumes that swap is disabled. If swap is enabled, then resource guarantees (for pods that specify resource requirements) will not hold. For example, suppose 2 guaranteed pods have reached their memory limit. They can continue allocating memory by utilizing disk space. Eventually, if there isn't enough swap space, processes in the pods might get killed. The node must take into account swap space explicitly for providing deterministic isolation behavior.
## Alternative QoS Class Policy
diff --git a/contributors/design-proposals/resource-quota-scoping.md b/contributors/design-proposals/resource-quota-scoping.md
index ac977d4e..5cffb998 100644
--- a/contributors/design-proposals/resource-quota-scoping.md
+++ b/contributors/design-proposals/resource-quota-scoping.md
@@ -121,7 +121,7 @@ tracking the following resources:
## Data Model Impact
-```
+```go
// The following identify resource constants for Kubernetes object types
const (
// CPU request, in cores. (500m = .5 cores)
@@ -241,7 +241,7 @@ The cluster-admin wants to restrict the following:
This would require the following quotas to be added to the namespace:
-```
+```sh
$ cat quota-best-effort
apiVersion: v1
kind: ResourceQuota
@@ -279,7 +279,7 @@ spec:
cpu.limit: 4
scopes:
- NotTerminating
- - NotBestEffort
+ - NotBestEffort
$ cat quota
apiVersion: v1
diff --git a/contributors/design-proposals/runtimeconfig.md b/contributors/design-proposals/runtimeconfig.md
index 896ca130..b2ed83dd 100644
--- a/contributors/design-proposals/runtimeconfig.md
+++ b/contributors/design-proposals/runtimeconfig.md
@@ -31,7 +31,7 @@ which loads a `config.ConfigurationMap`:
- kube-dns
(Note kubelet is omitted, it's dynamic config story is being addressed
-by #29459). Alpha features that are not accessed via an alpha API
+by [#29459](https://issues.k8s.io/29459)). Alpha features that are not accessed via an alpha API
group should define an `enableFeatureName` flag and use it to toggle
activation of the feature in each system component that the feature
uses.
@@ -60,7 +60,7 @@ not be altered in a running cluster.
## Future work
1. The eventual plan is for component config to be managed by versioned
-APIs and not flags (#12245). When that is added, toggling of features
+APIs and not flags ([#12245](https://issues.k8s.io/12245)). When that is added, toggling of features
could be handled by versioned component config and the component flags
deprecated.
diff --git a/contributors/design-proposals/secret-configmap-downwarapi-file-mode.md b/contributors/design-proposals/secret-configmap-downwarapi-file-mode.md
index 42def9bf..f9b3b5af 100644
--- a/contributors/design-proposals/secret-configmap-downwarapi-file-mode.md
+++ b/contributors/design-proposals/secret-configmap-downwarapi-file-mode.md
@@ -94,7 +94,7 @@ and are not affected by this setting.
In other words, the fields will look like this:
-```
+```go
type SecretVolumeSource struct {
// Name of the secret in the pod's namespace to use.
SecretName string `json:"secretName,omitempty"`
diff --git a/contributors/design-proposals/security-context-constraints.md b/contributors/design-proposals/security-context-constraints.md
index ae966e21..a61d2f3b 100644
--- a/contributors/design-proposals/security-context-constraints.md
+++ b/contributors/design-proposals/security-context-constraints.md
@@ -1,7 +1,8 @@
## Abstract
PodSecurityPolicy allows cluster administrators to control the creation and validation of a security
-context for a pod and containers.
+context for a pod and containers. The intent of PodSecurityPolicy is to protect the cluster from the
+pod and containers, not to protect a pod or containers from a user.
## Motivation
@@ -17,7 +18,7 @@ granting the user themselves an elevated set of permissions.
## Goals
-1. Associate [service accounts](../design/service_accounts.md), groups, and users with
+1. Associate [service accounts](../design-proposals/service_accounts.md), groups, and users with
a set of constraints that dictate how a security context is established for a pod and the pod's containers.
1. Provide the ability for users and infrastructure components to run pods with elevated privileges
on behalf of another user or within a namespace where privileges are more restrictive.
@@ -49,8 +50,8 @@ pods and service accounts within a project
1. Provide a set of restrictions that controls how a security context is created for pods and containers
as a new cluster-scoped object called `PodSecurityPolicy`.
1. User information in `user.Info` must be available to admission controllers. (Completed in
-https://github.com/GoogleCloudPlatform/kubernetes/pull/8203)
-1. Some authorizers may restrict a user’s ability to reference a service account. Systems requiring
+https://github.com/kubernetes/kubernetes/pull/8203)
+1. Some authorizers may restrict a user's ability to reference a service account. Systems requiring
the ability to secure service accounts on a user level must be able to add a policy that enables
referencing specific service accounts themselves.
1. Admission control must validate the creation of Pods against the allowed set of constraints.
@@ -126,24 +127,24 @@ type HostPortRange struct {
// VolumeSecurityPolicy allows and disallows the use of different types of volume plugins.
type VolumeSecurityPolicy struct {
// HostPath allows or disallows the use of the HostPath volume plugin.
- // More info: http://kubernetes.io/docs/user-guide/volumes#hostpath
+ // More info: http://kubernetes.io/docs/user-guide/volumes/#hostpath
HostPath bool `json:"hostPath,omitempty"`
// EmptyDir allows or disallows the use of the EmptyDir volume plugin.
- // More info: http://kubernetes.io/docs/user-guide/volumes#emptydir
+ // More info: http://kubernetes.io/docs/user-guide/volumes/#emptydir
EmptyDir bool `json:"emptyDir,omitempty"`
// GCEPersistentDisk allows or disallows the use of the GCEPersistentDisk volume plugin.
- // More info: http://kubernetes.io/docs/user-guide/volumes#gcepersistentdisk
+ // More info: http://kubernetes.io/docs/user-guide/volumes/#gcepersistentdisk
GCEPersistentDisk bool `json:"gcePersistentDisk,omitempty"`
// AWSElasticBlockStore allows or disallows the use of the AWSElasticBlockStore volume plugin.
- // More info: http://kubernetes.io/docs/user-guide/volumes#awselasticblockstore
+ // More info: http://kubernetes.io/docs/user-guide/volumes/#awselasticblockstore
AWSElasticBlockStore bool `json:"awsElasticBlockStore,omitempty"`
// GitRepo allows or disallows the use of the GitRepo volume plugin.
GitRepo bool `json:"gitRepo,omitempty"`
// Secret allows or disallows the use of the Secret volume plugin.
- // More info: http://kubernetes.io/docs/user-guide/volumes#secrets
+ // More info: http://kubernetes.io/docs/user-guide/volumes/#secret
Secret bool `json:"secret,omitempty"`
// NFS allows or disallows the use of the NFS volume plugin.
- // More info: http://kubernetes.io/docs/user-guide/volumes#nfs
+ // More info: http://kubernetes.io/docs/user-guide/volumes/#nfs
NFS bool `json:"nfs,omitempty"`
// ISCSI allows or disallows the use of the ISCSI volume plugin.
// More info: http://releases.k8s.io/HEAD/examples/volumes/iscsi/README.md
@@ -152,7 +153,7 @@ type VolumeSecurityPolicy struct {
// More info: http://releases.k8s.io/HEAD/examples/volumes/glusterfs/README.md
Glusterfs bool `json:"glusterfs,omitempty"`
// PersistentVolumeClaim allows or disallows the use of the PersistentVolumeClaim volume plugin.
- // More info: http://kubernetes.io/docs/user-guide/persistent-volumes#persistentvolumeclaims
+ // More info: http://kubernetes.io/docs/user-guide/persistent-volumes/#persistentvolumeclaims
PersistentVolumeClaim bool `json:"persistentVolumeClaim,omitempty"`
// RBD allows or disallows the use of the RBD volume plugin.
// More info: http://releases.k8s.io/HEAD/examples/volumes/rbd/README.md
@@ -221,7 +222,9 @@ const (
As reusable objects in the root scope, PodSecurityPolicy follows the lifecycle of the
cluster itself. Maintenance of constraints such as adding, assigning, or changing them is the
-responsibility of the cluster administrator.
+responsibility of the cluster administrator. Deleting is not considered in PodSecurityPolicy,
+It's important for controllers without the ability to use psps (like the namespace controller)
+to be able to delete pods.
Creating a new user within a namespace should not require the cluster administrator to
define the user's PodSecurityPolicy. They should receive the default set of policies
diff --git a/contributors/design-proposals/self-hosted-final-cluster.png b/contributors/design-proposals/self-hosted-final-cluster.png
new file mode 100644
index 00000000..e5302b07
--- /dev/null
+++ b/contributors/design-proposals/self-hosted-final-cluster.png
Binary files differ
diff --git a/contributors/design-proposals/self-hosted-kubernetes.md b/contributors/design-proposals/self-hosted-kubernetes.md
new file mode 100644
index 00000000..2693ad84
--- /dev/null
+++ b/contributors/design-proposals/self-hosted-kubernetes.md
@@ -0,0 +1,103 @@
+# Proposal: Self-hosted Control Plane
+
+Author: Brandon Philips <brandon.philips@coreos.com>
+
+## Motivations
+
+> Running our components in pods would solve many problems, which we'll otherwise need to implement other, less portable, more brittle solutions to, and doesn't require much that we don't need to do for other reasons. Full self-hosting is the eventual goal.
+>
+> - Brian Grant ([ref](https://github.com/kubernetes/kubernetes/issues/4090#issuecomment-74890508))
+
+### What is self-hosted?
+
+Self-hosted Kubernetes runs all required and optional components of a Kubernetes cluster on top of Kubernetes itself.
+
+The advantages of a self-hosted Kubernetes cluster are:
+
+1. **Small Dependencies:** self-hosted should reduce the number of components required, on host, for a Kubernetes cluster to be deployed to a Kubelet (ideally running in a container). This should greatly simplify the perceived complexity of Kubernetes installation.
+2. **Deployment consistency:** self-hosted reduces the number of files that are written to disk or managed via configuration management or manual installation via SSH. Our hope is to reduce the number of moving parts relying on the host OS to make deployments consistent in all environments.
+3. **Introspection:** internal components can be debugged and inspected by users using existing Kubernetes APIs like `kubectl logs`
+4. **Cluster Upgrades:** Related to introspection the components of a Kubernetes cluster are now subject to control via Kubernetes APIs. Upgrades of Kubelet's are possible via new daemon sets, API servers can be upgraded using daemon sets and potentially deployments in the future, and flags of add-ons can be changed by updating deployments, etc.
+5. **Easier Highly-Available Configurations:** Using Kubernetes APIs will make it easier to scale up and monitor an HA environment without complex external tooling. Because of the complexity of these configurations tools that create them without self-hosted often implement significant complex logic.
+
+However, there is a spectrum of ways that a cluster can be self-hosted. To do this we are going to divide the Kubernetes cluster into a variety of layers beginning with the Kubelet (level 0) and going up to the add-ons (Level 4). A cluster can self-host all of these levels 0-4 or only partially self-host.
+
+![](self-hosted-layers.png)
+
+For example, a 0-4 self-hosted cluster means that the kubelet is a daemon set, the API server runs as a pod and is exposed as a service, and so on. While a 1-4 self-hosted cluster would have a system installed Kubelet. And a 02-4 system would have everything except etcd self-hosted.
+
+It is also important to point out that self-hosted stands alongside other methods to install and configure Kubernetes, including scripts like kube-up.sh, configuration management tools, and deb/rpm/etc packages. A non-goal of this self-hosted proposal is replacing or introducing anything that might impede these installation and management methods. In fact it is likely that by dogfooding Kubernetes APIs via self-hosted improvements will be made to Kubernetes components that will simplify other installation and management methods.
+
+## Practical Implementation Overview
+
+This document outlines the current implementation of "self-hosted Kubernetes" installation and upgrade of Kubernetes clusters based on the work that the teams at CoreOS and Google have been doing. The work is motivated by the early ["Self-hosted Proposal"](https://github.com/kubernetes/kubernetes/issues/246#issuecomment-64533959) by Brian Grant.
+
+The entire system is working today and is used by Bootkube, a Kubernetes Incubator project, to create 2-4 and 1-4 self-hosted clusters. All Tectonic clusters created since July 2016 are 2-4 self-hosted and will be moving to 1-4 early in 2017 as the self-hosted etcd work becomes stable in bootkube. This document outlines the implementation, not the experience. The experience goal is that users not know all of these details and instead get a working Kubernetes cluster out the other end that can be upgraded using the Kubernetes APIs.
+
+The target audience are projects in SIG Cluster Lifecycle thinking about and building the way forward for install and upgrade of Kubernetes. We hope to inspire direction in various Kubernetes components like kubelet and [kubeadm](https://github.com/kubernetes/kubernetes/pull/38407) to make self-hosted a compelling mainstream installation method. If you want a higher level demonstration of "Self-Hosted" and the value see this [video and blog](https://coreos.com/blog/self-hosted-kubernetes.html).
+
+### Bootkube
+
+Today, the first component of the installation of a self-hosted cluster is [`bootkube`](https://github.com/kubernetes-incubator/bootkube). Bootkube provides a temporary Kubernetes control plane that tells a kubelet to execute all of the components necessary to run a full blown Kubernetes control plane. When the kubelet connects to this temporary API server it will deploy the required Kubernetes components as pods. This diagram shows all of the moving parts:
+
+![](self-hosted-moving-parts.png)
+
+Note: In the future this temporary control plane may be replaced with a kubelet API that will enable injection of this state directly into the kubelet without a temporary Kubernetes API server.
+
+At the end of this process the bootkube can be shut down and the system kubelet will coordinate, through a POSIX lock (see `kubelet --exit-on-lock-contention`), to let the self-hosted kubelet take over lifecycle and management of the control plane components. The final cluster state looks like this:
+
+![](self-hosted-final-cluster.png)
+
+There are a few things to note. First, generally, the control components like the API server, etc will be pinned to a set of dedicated control nodes. For security policy, service discovery, and scaling reasons it is easiest to assume that control nodes will always exist on N nodes.
+
+Another challenge is load balancing the API server. Bedrock for the API server will be DNS, TLS, and a load balancer that live off cluster and that load balancer will want to only healthcheck a handful of servers for the API server port liveness probe.
+
+### Bootkube Challenges
+
+This process has a number of moving parts. Most notably the hand off of control from the "host system" to the Kubernetes self-hosted system. And things can go wrong:
+
+1) The self-hosted Kubelet is in a precarious position as there is no one around to restart the process if it crashes. The high level is that the system init system will watch for the Kubelet POSIX lock and start the system Kubelet if the lock is missing. Once the system Kubelet starts it will launch the self-hosted Kubelet.
+
+2) Recovering from reboots of single-master installations is a challenge as the Kubelet won't have an API server to talk to to restart the self-hosted components. We are solving this today with "[user space checkpointing](https://github.com/kubernetes-incubator/bootkube/tree/master/cmd/checkpoint#checkpoint)" container in the Kubelet pod that will periodically check the pod manifests and persist them to the static pod manifest directory. Longer term we would like for the kubelet to be able to checkpoint itself without external code.
+
+## Long Term Goals
+
+Ideally bootkube disappears over time and is replaced by a [Kubelet pod API](https://github.com/kubernetes/kubernetes/issues/28138). The write API would enable an external installation program to setup the control plane of a self-hosted Kubernetes cluster without requiring an existing API server.
+
+[Checkpointing](https://github.com/kubernetes/kubernetes/issues/489) is also required to make for a reliable system that can survive a number of normal operations like full down scenarios of the control plane. Today, we can sufficiently do checkpointing external of the Kubelet process, but checkpointing inside of the Kubelet would be ideal.
+
+A simple updater can take care of helping users update from v1.3.0 to v1.3.1, etc over time.
+
+### Self-hosted Cluster Upgrades
+
+#### Kubelet upgrades
+
+The kubelet could be upgraded in a very similar process to that outlined in the self-hosted proposal.
+
+However, because of the challenges around the self-hosted Kubelet (see above) Tectonic currently has a 1-4 self-hosted cluster with an alternative Kubelet update scheme which side-steps the self-hosted Kubelet issues. First, a kubelet system service is launched that uses the [chrooted kubelet](https://github.com/kubernetes/community/pull/131) implemented by the [kubelet-wrapper](https://coreos.com/kubernetes/docs/latest/kubelet-wrapper.html). Then, when an update is required, a node annotation is made which is read by a long-running daemonset that updates the kubelet-wrapper configuration. This makes Kubelet versions updateable from the cluster API.
+
+#### API Server, Scheduler, and Controller Manager
+
+Upgrading these components is fairly straightforward. They are stateless, easily run in containers, and can be modeled as pods and services. Upgrades are simply a matter of deploying new versions, health checking them, and changing the service label selectors.
+
+In HA configurations the API servers should be able to be upgraded in-place one-by-one and rely on external load balancing or client retries to recover from the temporary downtime. This is not well tested upstream and something we need to fix (see known issues).
+
+#### etcd self-hosted
+
+As the primary data store of Kubernetes etcd plays an important role. Today, etcd does not run on top of the self-hosted cluster. However, progress is being made with the introduction of the [etcd Operator](https://coreos.com/blog/introducing-the-etcd-operator.html) and integration into [bootkube](https://github.com/kubernetes-incubator/bootkube/blob/848cf581451425293031647b5754b528ec5bf2a0/cmd/bootkube/start.go#L37).
+
+### Highly-available Clusters
+
+Self-hosted will make operating highly-available clusters even easier. For internal critical components like the scheduler and controller manager, which already know how to leader elect using the Kubernetes leader election API, creating HA instances will be a simple matter of `kubectl scale` for most administrators. For the data store, etcd, the etcd Operator will ease much of the scaling concern.
+
+However, the API server will be a slightly trickier matter for most deployments as the API server relies on either external load balancing or external DNS in most common HA configurations. But, with the addition of Kubernetes label metadata on the [Node API](https://github.com/kubernetes/kubernetes/pull/39112) self-hosted may make it easier for systems administrators to create glue code that finds the appropriate Node IPs and adds them to these external systems.
+
+### Conclusions
+
+Kubernetes self-hosted is working today. Bootkube is an implementation of the "temporary control plane" and this entire process has been used by [`bootkube`](https://github.com/kubernetes-incubator/bootkube) users and Tectonic since the Kubernetes v1.4 release. We are excited to give users a simpler installation flow and sustainable cluster lifecycle upgrade/management.
+
+## Known Issues
+
+- [Health check endpoints for components don't work correctly](https://github.com/kubernetes-incubator/bootkube/issues/64#issuecomment-228144345)
+- [kubeadm does do self-hosted, but isn't tested yet](https://github.com/kubernetes/kubernetes/pull/40075)
+- The Kubernetes [versioning policy](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/versioning.md) allows for version skew of kubelet and control plane but not skew between control plane components themselves. We must add testing and validation to Kubernetes that this skew works. Otherwise the work to make Kubernetes HA is rather pointless if it can't be upgraded in an HA manner as well.
diff --git a/contributors/design-proposals/self-hosted-layers.png b/contributors/design-proposals/self-hosted-layers.png
new file mode 100644
index 00000000..1dc3e06a
--- /dev/null
+++ b/contributors/design-proposals/self-hosted-layers.png
Binary files differ
diff --git a/contributors/design-proposals/self-hosted-moving-parts.png b/contributors/design-proposals/self-hosted-moving-parts.png
new file mode 100644
index 00000000..423add2e
--- /dev/null
+++ b/contributors/design-proposals/self-hosted-moving-parts.png
Binary files differ
diff --git a/contributors/design-proposals/service-discovery.md b/contributors/design-proposals/service-discovery.md
index 28d1f8d4..66e497e7 100644
--- a/contributors/design-proposals/service-discovery.md
+++ b/contributors/design-proposals/service-discovery.md
@@ -20,7 +20,7 @@ If a user and/or password is required then this information can be passed using
`Service Scheme` - Services can be deployed using different schemes. Some popular schemes include `http`,`https`,`file`,`ftp` and `jdbc`.
-`Service Protocol` - Services use different protocols that clients need to speak in order to communicate with the service, some examples of service level protocols are SOAP, REST (Yes, technically REST isn’t a protocol but an architectural style). For service consumers it can be hard to tell what protocol is expected.
+`Service Protocol` - Services use different protocols that clients need to speak in order to communicate with the service, some examples of service level protocols are SOAP, REST (Yes, technically REST isn't a protocol but an architectural style). For service consumers it can be hard to tell what protocol is expected.
## Service Description
@@ -37,7 +37,7 @@ Kubernetes allows the creation of Service Annotations. Here we propose the use o
* `api.service.kubernetes.io/path` - the path part of the service endpoint url. An example value could be `cxfcdi`,
* `api.service.kubernetes.io/scheme` - the scheme part of the service endpoint url. Some values could be `http` or `https`.
* `api.service.kubernetes.io/protocol` - the protocol of the service. Known values are `SOAP`, `XML-RPC` and `REST`,
-* `api.service.kubernetes.io/description-path` - the path part of the service description document’s endpoint. It is a pretty safe assumption that the service self-documents. An example value for a swagger 2.0 document can be `cxfcdi/swagger.json`,
+* `api.service.kubernetes.io/description-path` - the path part of the service description document's endpoint. It is a pretty safe assumption that the service self-documents. An example value for a swagger 2.0 document can be `cxfcdi/swagger.json`,
* `api.kubernetes.io/description-language` - the type of Description Language used. Known values are `WSDL`, `WADL`, `SwaggerJSON`, `SwaggerYAML`.
The fragment below is taken from the service section of the kubernetes.json were these annotations are used
diff --git a/contributors/design-proposals/service-external-name.md b/contributors/design-proposals/service-external-name.md
index 798da87f..eaab4c51 100644
--- a/contributors/design-proposals/service-external-name.md
+++ b/contributors/design-proposals/service-external-name.md
@@ -15,7 +15,7 @@ RDS resource `something.rds.aws.amazon.com`. No proxying is involved.
# Motivation
There were many related issues, but we'll try to summarize them here. More info
-is on GitHub issues/PRs: #13748, #11838, #13358, #23921
+is on GitHub issues/PRs: [#13748](https://issues.k8s.io/13748), [#11838](https://issues.k8s.io/11838), [#13358](https://issues.k8s.io/13358), [#23921](https://issues.k8s.io/23921)
One motivation is to present as native cluster services, services that are
hosted externally. Some cloud providers, like AWS, hand out hostnames (IPs are
@@ -60,7 +60,7 @@ with DNS TTL and more. One imperfect approach was to only resolve the hostname
upon creation, but this was considered not a great idea. A better approach
would be at a higher level, maybe a service type.
-There are more ideas described in #13748, but all raised further issues,
+There are more ideas described in [#13748](https://issues.k8s.io/13748), but all raised further issues,
ranging from using another upstream DNS server to creating a Name object
associated with DNSs.
@@ -81,7 +81,7 @@ https://github.com/kubernetes/kubernetes/issues/13748#issuecomment-230397975
Currently a ServiceSpec looks like this, with comments edited for clarity:
-```
+```go
type ServiceSpec struct {
Ports []ServicePort
@@ -105,7 +105,7 @@ type ServiceSpec struct {
The proposal is to change it to:
-```
+```go
type ServiceSpec struct {
Ports []ServicePort
@@ -135,7 +135,7 @@ type ServiceSpec struct {
For example, it can be used like this:
-```
+```yaml
apiVersion: v1
kind: Service
metadata:
@@ -143,8 +143,8 @@ metadata:
spec:
ports:
- port: 12345
-type: ExternalName
-externalName: myapp.rds.whatever.aws.says
+ type: ExternalName
+ externalName: myapp.rds.whatever.aws.says
```
There is one issue to take into account, that no other alternative considered
diff --git a/contributors/design-proposals/service_accounts.md b/contributors/design-proposals/service_accounts.md
index 89a3771b..79902140 100644
--- a/contributors/design-proposals/service_accounts.md
+++ b/contributors/design-proposals/service_accounts.md
@@ -112,7 +112,7 @@ another, and another, until its behavior cannot be attributed to a single human.
**TODO**: consider getting rid of separate serviceAccount object and just
rolling its parts into the SecurityContext or Pod Object.
-The `secrets` field is a list of references to /secret objects that an process
+The `secrets` field is a list of references to /secret objects that a process
started as that service account should have access to be able to assert that
role.
diff --git a/contributors/design-proposals/sig-cli/template.md b/contributors/design-proposals/sig-cli/template.md
new file mode 100644
index 00000000..ccfcbcf3
--- /dev/null
+++ b/contributors/design-proposals/sig-cli/template.md
@@ -0,0 +1,38 @@
+# <Title>
+
+Status: Pending
+
+Version: Alpha | Beta | GA
+
+Assignee: <github handle>
+
+## Motivation
+
+<2-6 sentences about why this is needed>
+
+## Proposal
+
+<4-6 description of the proposed solution>
+
+## User Experience
+
+### Use Cases
+
+<enumerated list of use cases for this feature>
+
+<in depth description of user experience>
+
+<*include full examples*>
+
+## Implementation
+
+<in depth description of how the feature will be implemented. in some cases this may be very simple.>
+
+### Client/Server Backwards/Forwards compatibility
+
+<define behavior when using a kubectl client with an older or newer version of the apiserver (+-1 version)>
+
+## Alternatives considered
+
+<short description of alternative solutions to be considered>
+
diff --git a/contributors/design-proposals/stateful-apps.md b/contributors/design-proposals/stateful-apps.md
index c5196f2a..ee9f9cd5 100644
--- a/contributors/design-proposals/stateful-apps.md
+++ b/contributors/design-proposals/stateful-apps.md
@@ -64,7 +64,7 @@ While it is possible to emulate the guarantees described above by leveraging mul
(for distinct pod templates and pod identities) and multiple services (for stable network identity), the
resulting objects are hard to maintain and must be copied manually in order to scale a cluster.
-By constrast, a DaemonSet *can* offer some of the guarantees above, by leveraging Nodes as stable, long-lived
+By contrast, a DaemonSet *can* offer some of the guarantees above, by leveraging Nodes as stable, long-lived
entities. An administrator might choose a set of nodes, label them a particular way, and create a
DaemonSet that maps pods to each node. The storage of the node itself (which could be network attached
storage, or a local SAN) is the persistent storage. The network identity of the node is the stable
@@ -108,7 +108,7 @@ if a previous pod has been fully terminated (reached its graceful termination li
A StatefulSet has 0..N **members**, each with a unique **identity** which is a name that is unique within the
set.
-```
+```go
type StatefulSet struct {
ObjectMeta
@@ -204,7 +204,7 @@ match a volume mount within the pod template.
Future work:
-* In the future other resources may be added that must also be templated - for instance, secrets (unique secret per member), config data (unique config per member), and in the futher future, arbitrary extension resources.
+* In the future other resources may be added that must also be templated - for instance, secrets (unique secret per member), config data (unique config per member), and in the further future, arbitrary extension resources.
* Consider allowing the identity value itself to be passed as an environment variable via the downward API
* Consider allowing per identity values to be specified that are passed to the pod template or volume claim.
@@ -225,7 +225,7 @@ fashion via DNS by leveraging information written to the endpoints by the endpoi
The end result might be DNS resolution as follows:
-```
+```sh
# service mongo pointing to pods created by StatefulSet mdb, with identities mdb-1, mdb-2, mdb-3
dig mongodb.namespace.svc.cluster.local +short A
@@ -244,9 +244,9 @@ dig mdb-3.mongodb.namespace.svc.cluster.local +short A
This is currently implemented via an annotation on pods, which is surfaced to endpoints, and finally
surfaced as DNS on the service that exposes those pods.
-```
-// The pods created by this StatefulSet will have the DNS names "mysql-0.NAMESPACE.svc.cluster.local"
-// and "mysql-1.NAMESPACE.svc.cluster.local"
+```yaml
+# The pods created by this StatefulSet will have the DNS names "mysql-0.NAMESPACE.svc.cluster.local"
+# and "mysql-1.NAMESPACE.svc.cluster.local"
kind: StatefulSet
metadata:
name: mysql
@@ -276,7 +276,7 @@ The StatefulSet controller is expected to execute like other controllers, as a s
considering designing for safety first, the possibility of the controller running concurrently cannot
be overlooked, and so it is important to ensure that duplicate pod identities are not achieved.
-There are two mechanisms to acheive this at the current time. One is to leverage unique names for pods
+There are two mechanisms to achieve this at the current time. One is to leverage unique names for pods
that carry the identity of the pod - this prevents duplication because etcd 2 can guarantee single
key transactionality. The other is to use the status field of the StatefulSet to coordinate membership
information. It is possible to leverage both at this time, and encourage users to not assume pod
@@ -300,7 +300,7 @@ and in the future provide more tools to reduce the amount of work necessary to m
state.
The first mechanism is that the StatefulSet controller blocks creation of new pods until all previous pods
-are reporting a healthy status. The StatefulSet controller uses the strong serializability of the underyling
+are reporting a healthy status. The StatefulSet controller uses the strong serializability of the underlying
etcd storage to ensure that it acts on a consistent view of the cluster membership (the pods and their)
status, and serializes the creation of pods based on the health state of other pods. This simplifies
reasoning about how to initialize a StatefulSet, but is not sufficient to guarantee split brain does not
@@ -327,7 +327,7 @@ Criteria for advancing to beta:
Criteria for advancing to GA:
-* StatefulSets solve 80% of clustered software configuraton with minimal input from users and are safe from common split brain problems
+* StatefulSets solve 80% of clustered software configuration with minimal input from users and are safe from common split brain problems
* Several representative examples of StatefulSets from the community have been proven/tested to be "correct" for a variety of partition problems (possibly via Jepsen or similar)
* Sufficient testing and soak time has been in place (like for Deployments) to ensure the necessary features are in place.
* StatefulSets are considered easy to use for deploying clustered software for common cases
diff --git a/contributors/design-proposals/synchronous-garbage-collection.md b/contributors/design-proposals/synchronous-garbage-collection.md
index c5157408..6f2a9be5 100644
--- a/contributors/design-proposals/synchronous-garbage-collection.md
+++ b/contributors/design-proposals/synchronous-garbage-collection.md
@@ -106,7 +106,7 @@ When validating the ownerReference, API server needs to query the `Authorizer` t
**Modifications to processEvent()**
-Currently `processEvent()` manages GC’s internal owner-dependency relationship graph, `uidToNode`. It updates `uidToNode` according to the Add/Update/Delete events in the cluster. To support synchronous GC, it has to:
+Currently `processEvent()` manages GC's internal owner-dependency relationship graph, `uidToNode`. It updates `uidToNode` according to the Add/Update/Delete events in the cluster. To support synchronous GC, it has to:
* handle Add or Update events where `obj.Finalizers.Has(GCFinalizer) && obj.DeletionTimestamp != nil`. The object will be added into the `dirtyQueue`. The object will be marked as “GC in progress” in `uidToNode`.
* Upon receiving the deletion event of an object, put its owner into the `dirtyQueue` if the owner node is marked as "GC in progress". This is to force the `processItem()` (described next) to re-check if all dependents of the owner is deleted.
diff --git a/contributors/design-proposals/templates.md b/contributors/design-proposals/templates.md
index 2d58fbd5..50712932 100644
--- a/contributors/design-proposals/templates.md
+++ b/contributors/design-proposals/templates.md
@@ -157,7 +157,7 @@ Template definition.
**Template Object**
-```
+```go
// Template contains the inputs needed to produce a Config.
type Template struct {
unversioned.TypeMeta
@@ -179,7 +179,7 @@ type Template struct {
**Parameter Object**
-```
+```go
// Parameter defines a name/value variable that is to be processed during
// the Template to Config transformation.
type Parameter struct {
@@ -194,7 +194,7 @@ type Parameter struct {
Description string
// Optional: Value holds the Parameter data.
- // The value replaces all occurrences of the Parameter $(Name) or
+ // The value replaces all occurrences of the Parameter $(Name) or
// $((Name)) expression during the Template to Config transformation.
Value string
@@ -208,26 +208,26 @@ type Parameter struct {
```
As seen above, parameters allow for metadata which can be fed into client implementations to display information about the
-parameter’s purpose and whether a value is required. In lieu of type information, two reference styles are offered: `$(PARAM)`
+parameter's purpose and whether a value is required. In lieu of type information, two reference styles are offered: `$(PARAM)`
and `$((PARAM))`. When the single parens option is used, the result of the substitution will remain quoted. When the double
parens option is used, the result of the substitution will not be quoted. For example, given a parameter defined with a value
of "BAR", the following behavior will be observed:
-```
+```go
somefield: "$(FOO)" -> somefield: "BAR"
somefield: "$((FOO))" -> somefield: BAR
```
-// for concatenation, the result value reflects the type of substitution (quoted or unquoted):
+for concatenation, the result value reflects the type of substitution (quoted or unquoted):
-```
+```go
somefield: "prefix_$(FOO)_suffix" -> somefield: "prefix_BAR_suffix"
somefield: "prefix_$((FOO))_suffix" -> somefield: prefix_BAR_suffix
```
-// if both types of substitution exist, quoting is performed:
+if both types of substitution exist, quoting is performed:
-```
+```go
somefield: "prefix_$((FOO))_$(FOO)_suffix" -> somefield: "prefix_BAR_BAR_suffix"
```
@@ -243,7 +243,7 @@ Illustration of a template which defines a service and replication controller wi
the name of the top level objects, the number of replicas, and several environment variables defined on the
pod template.
-```
+```json
{
"kind": "Template",
"apiVersion": "v1",
@@ -402,7 +402,7 @@ The api endpoint will then:
1. Validate the template including confirming “required” parameters have an explicit value.
2. Walk each api object in the template.
-3. Adding all labels defined in the template’s ObjectLabels field.
+3. Adding all labels defined in the template's ObjectLabels field.
4. For each field, check if the value matches a parameter name and if so, set the value of the field to the value of the parameter.
* Partial substitutions are accepted, such as `SOME_$(PARAM)` which would be transformed into `SOME_XXXX` where `XXXX` is the value
of the `$(PARAM)` parameter.
diff --git a/contributors/design-proposals/versioning.md b/contributors/design-proposals/versioning.md
index ae724b12..2164df3a 100644
--- a/contributors/design-proposals/versioning.md
+++ b/contributors/design-proposals/versioning.md
@@ -154,7 +154,7 @@ version changes, not new major nor minor versions).
* Users can upgrade from any Kube 1.x release to any other Kube 1.x release as a
rolling upgrade across their cluster. (Rolling upgrade means being able to
-upgrade the master first, then one node at a time. See #4855 for details.)
+upgrade the master first, then one node at a time. See [#4855](https://issues.k8s.io/4855) for details.)
* However, we do not recommend upgrading more than two minor releases at a
time (see [Supported releases](#supported-releases)), and do not recommend
running non-latest patch releases of a given minor release.
diff --git a/contributors/design-proposals/volume-hostpath-qualifiers.md b/contributors/design-proposals/volume-hostpath-qualifiers.md
index cd0902ec..b207253c 100644
--- a/contributors/design-proposals/volume-hostpath-qualifiers.md
+++ b/contributors/design-proposals/volume-hostpath-qualifiers.md
@@ -98,7 +98,7 @@ One alternative is to augment Host Volumes with a `MustExist` bool and provide
no further granularity. This would allow toggling between the `auto` and
`exists` behaviors described above. This would likely cover the "90%" use-case
and would be a simpler API. It would be sufficient for all of the examples
-linked above in my opionion.
+linked above in my opinion.
## Kubelet implementation
diff --git a/contributors/design-proposals/volume-ownership-management.md b/contributors/design-proposals/volume-ownership-management.md
index d08c491c..a9fb1cfe 100644
--- a/contributors/design-proposals/volume-ownership-management.md
+++ b/contributors/design-proposals/volume-ownership-management.md
@@ -58,7 +58,7 @@ type Builder interface {
Each volume plugin will have to change to support the new `SetUp` signature. The existing
ownership management code will be refactored into a library that volume plugins can use:
-```
+```go
package volume
func ManageOwnership(path string, fsGroup int64) error {
diff --git a/contributors/design-proposals/volume-provisioning.md b/contributors/design-proposals/volume-provisioning.md
index f8202fbe..ff68d280 100644
--- a/contributors/design-proposals/volume-provisioning.md
+++ b/contributors/design-proposals/volume-provisioning.md
@@ -464,7 +464,7 @@ provisioner and to favor existing volumes before provisioning a new one.
This example shows two storage classes, "aws-fast" and "aws-slow".
-```
+```yaml
apiVersion: v1
kind: StorageClass
metadata:
diff --git a/contributors/design-proposals/volume-snapshotting.md b/contributors/design-proposals/volume-snapshotting.md
index e92ed3d1..0d569618 100644
--- a/contributors/design-proposals/volume-snapshotting.md
+++ b/contributors/design-proposals/volume-snapshotting.md
@@ -190,7 +190,7 @@ Open questions:
* Can the API call methods on VolumePlugins? Yeah via controller
- * The scheduler gives users functionality that doesn’t already exist, but required adding an entirely new controller
+ * The scheduler gives users functionality that doesn't already exist, but required adding an entirely new controller
* Should the list and restore operations be part of v1?
@@ -446,7 +446,7 @@ Users will specify a snapshotting schedule for particular volumes, which Kuberne
17. If the pod dies do we continue creating snapshots?
- 18. How to communicate errors (PD doesn’t support snapshotting, time period unsupported)
+ 18. How to communicate errors (PD doesn't support snapshotting, time period unsupported)
19. Off schedule snapshotting like before an application upgrade
@@ -456,7 +456,7 @@ Options, pros, cons, suggestion/recommendation
Example 1b
-During pod creation, a user can specify a pod definition in a yaml file. As part of this specification, users should be able to denote a [list of] times at which an existing snapshot command can be executed on the pod’s associated volume.
+During pod creation, a user can specify a pod definition in a yaml file. As part of this specification, users should be able to denote a [list of] times at which an existing snapshot command can be executed on the pod's associated volume.
For a simple example, take the definition of a [pod using a GCE PD](http://kubernetes.io/docs/user-guide/volumes/#example-pod-2):
diff --git a/contributors/devel/README.md b/contributors/devel/README.md
index cf29f3b4..a5ef3804 100644
--- a/contributors/devel/README.md
+++ b/contributors/devel/README.md
@@ -2,8 +2,8 @@
The developer guide is for anyone wanting to either write code which directly accesses the
Kubernetes API, or to contribute directly to the Kubernetes project.
-It assumes some familiarity with concepts in the [User Guide](../user-guide/README.md) and the [Cluster Admin
-Guide](../admin/README.md).
+It assumes some familiarity with concepts in the [User Guide](http://kubernetes.io/docs/user-guide/) and the [Cluster Admin
+Guide](http://kubernetes.io/docs/admin/).
## The process of developing and contributing code to the Kubernetes project
@@ -49,10 +49,10 @@ Guide](../admin/README.md).
## Developing against the Kubernetes API
-* The [REST API documentation](../api-reference/README.md) explains the REST
+* The [REST API documentation](http://kubernetes.io/docs/reference/) explains the REST
API exposed by apiserver.
-* **Annotations** ([docs/user-guide/annotations.md](../user-guide/annotations.md)): are for attaching arbitrary non-identifying metadata to objects.
+* **Annotations** ([Annotations](http://kubernetes.io/docs/concepts/object-metadata/annotations/)): are for attaching arbitrary non-identifying metadata to objects.
Programs that automate Kubernetes objects may use annotations to store small amounts of their state.
* **API Conventions** ([api-conventions.md](api-conventions.md)):
@@ -64,14 +64,14 @@ Guide](../admin/README.md).
## Writing plugins
-* **Authentication Plugins** ([docs/admin/authentication.md](../admin/authentication.md)):
+* **Authentication** ([Authentication](http://kubernetes.io/docs/admin/authentication/)):
The current and planned states of authentication tokens.
-* **Authorization Plugins** ([docs/admin/authorization.md](../admin/authorization.md)):
+* **Authorization Plugins** ([Authorization](http://kubernetes.github.io/docs/admin/authorization/)):
Authorization applies to all HTTP requests on the main apiserver port.
This doc explains the available authorization implementations.
-* **Admission Control Plugins** ([admission_control](../design/admission_control.md))
+* **Admission Control Plugins** ([admission_control](https://github.com/kubernetes/kubernetes/blob/master/docs/design/admission_control.md))
## Building releases
diff --git a/contributors/devel/adding-an-APIGroup.md b/contributors/devel/adding-an-APIGroup.md
index 5832be23..2c090b37 100644
--- a/contributors/devel/adding-an-APIGroup.md
+++ b/contributors/devel/adding-an-APIGroup.md
@@ -21,21 +21,38 @@ in your group;
2. Create pkg/apis/`<group>`/{register.go, `<version>`/register.go} to register
this group's API objects to the encoding/decoding scheme (e.g.,
-[pkg/apis/authentication/register.go](../../pkg/apis/authentication/register.go) and
-[pkg/apis/authentication/v1beta1/register.go](../../pkg/apis/authentication/v1beta1/register.go);
-
-3. Add a pkg/apis/`<group>`/install/install.go, which is responsible for adding
-the group to the `latest` package, so that other packages can access the group's
-meta through `latest.Group`. You probably only need to change the name of group
-and version in the [example](../../pkg/apis/authentication/install/install.go)). You
-need to import this `install` package in {pkg/master,
-pkg/client/unversioned}/import_known_versions.go, if you want to make your group
-accessible to other packages in the kube-apiserver binary, binaries that uses
-the client package.
+[pkg/apis/authentication/register.go](https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/authentication/register.go)
+and
+[pkg/apis/authentication/v1beta1/register.go](https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/authentication/v1beta1/register.go);
+The register files must have a var called SchemeBuilder for the generated code
+to reference. There must be an AddToScheme method for the installer to
+reference. You can look at a group under `pkg/apis/...` for example register.go
+files to use as a template, but do not copy the register.go files under
+`pkg/api/...`--they are not general.
+
+3. Add a pkg/apis/`<group>`/install/install.go, You probably only need to change
+the name of group and version in the
+[example](https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/authentication/install/install.go)). This
+package must be imported by the server along with
+`k8s.io/kubernetes/pkg/api/install`. Import these packages with the blank
+identifier as they do not have user callable code and exist solely for their
+initialization side-effects.
Step 2 and 3 are mechanical, we plan on autogenerate these using the
cmd/libs/go2idl/ tool.
+### Type definitions in `types.go`
+
+Each type should be an exported struct (have a capitalized name). The struct
+should have the `TypeMeta` and `ObjectMeta` embeds. There should be a `Spec` and
+a `Status` field. If the object is soley a data storage object, and will not be
+modified by a controller, the status field can be left off and the fields inside
+the `Spec` can be inlined directly into the struct.
+
+For each top-level type there should also be a `List` struct. The `List` struct should
+have the `TypeMeta` and `ListMeta` embeds. There should be an `Items` field that
+is a slice of the defined type.
+
### Scripts changes and auto-generated code:
1. Generate conversions and deep-copies:
diff --git a/contributors/devel/api_changes.md b/contributors/devel/api_changes.md
index 963deb7c..c63e12fa 100755
--- a/contributors/devel/api_changes.md
+++ b/contributors/devel/api_changes.md
@@ -292,7 +292,7 @@ unacceptable. Compatibility for experimental or alpha APIs is not strictly
required, but breaking compatibility should not be done lightly, as it disrupts
all users of the feature. Experimental APIs may be removed. Alpha and beta API
versions may be deprecated and eventually removed wholesale, as described in the
-[versioning document](../design/versioning.md). Document incompatible changes
+[versioning document](../design-proposals/versioning.md). Document incompatible changes
across API versions under the appropriate
[{v? conversion tips tag in the api.md doc](../api.md).
diff --git a/contributors/devel/bazel.md b/contributors/devel/bazel.md
index e6a4e9c5..8662d984 100644
--- a/contributors/devel/bazel.md
+++ b/contributors/devel/bazel.md
@@ -1,43 +1,59 @@
# Build with Bazel
-Building with bazel is currently experimental. Automanaged BUILD rules have the
+Building with Bazel is currently experimental. Automanaged `BUILD` rules have the
tag "automanaged" and are maintained by
-[gazel](https://github.com/mikedanese/gazel). Instructions for installing bazel
+[gazel](https://github.com/mikedanese/gazel). Instructions for installing Bazel
can be found [here](https://www.bazel.io/versions/master/docs/install.html).
-To build docker images for the components, run:
+Several `make` rules have been created for common operations:
-```
-$ bazel build //build-tools/...
-```
+* `make bazel-build`: builds all binaries in tree
+* `make bazel-test`: runs all unit tests
+* `make bazel-release`: builds release tarballs, Docker images (for server
+ components), and Debian images
-To run many of the unit tests, run:
+You can also interact with Bazel directly; for example, to run all `kubectl` unit
+tests, run
-```
-$ bazel test //cmd/... //build-tools/... //pkg/... //federation/... //plugin/...
+```console
+$ bazel test //pkg/kubectl/...
```
-To update automanaged build files, run:
+## Continuous Integration
-```
-$ ./hack/update-bazel.sh
-```
+The [Bazel CI job](http://k8s-testgrid.appspot.com/google-unit#bazel) runs
+`make bazel-build`, `make bazel-test`, and (transitively) `make bazel-release`.
+A similar job is run on all PRs.
-**NOTES**: `update-bazel.sh` only works if check out directory of Kubernetes is "$GOPATH/src/k8s.io/kubernetes".
+Many steps are cached, so the Bazel job usually executes fairly quickly.
-To update a single build file, run:
+## Known issues
-```
-$ # get gazel
-$ go get -u github.com/mikedanese/gazel
-$ # .e.g. ./pkg/kubectl/BUILD
-$ gazel -root="${YOUR_KUBE_ROOT_PATH}" ./pkg/kubectl
+[Cross-compilation is not currently supported](https://github.com/bazelbuild/rules_go/issues/70),
+so all binaries will be built for the host architecture running Bazel.
+Additionally, Go build tags are not supported. This means that builds on macOS may not work.
+
+[Binaries produced by Bazel are not statically linked](https://github.com/bazelbuild/rules_go/issues/161),
+and they are not currently tagged with version information.
+
+## Updating `BUILD` files
+
+To update `BUILD` files, run:
+
+```console
+$ ./hack/update-bazel.sh
```
-Updating BUILD file for a package will be required when:
+**NOTE**: `update-bazel.sh` only works if check out directory of Kubernetes is `$GOPATH/src/k8s.io/kubernetes`.
+
+Only rules which are automanaged will be updated, but all rules will be
+auto-formatted.
+
+Updating the `BUILD` file for a package will be required when:
* Files are added to or removed from a package
* Import dependencies change for a package
-
+* A `BUILD` file has been updated and needs to be reformatted
+* A new `BUILD` file has been added (parent `BUILD` files will be updated)
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/bazel.md?pixel)]()
diff --git a/contributors/devel/cli-roadmap.md b/contributors/devel/cli-roadmap.md
deleted file mode 100644
index cd21da08..00000000
--- a/contributors/devel/cli-roadmap.md
+++ /dev/null
@@ -1,11 +0,0 @@
-# Kubernetes CLI/Configuration Roadmap
-
-See github issues with the following labels:
-* [area/app-config-deployment](https://github.com/kubernetes/kubernetes/labels/area/app-config-deployment)
-* [component/kubectl](https://github.com/kubernetes/kubernetes/labels/component/kubectl)
-* [component/clientlib](https://github.com/kubernetes/kubernetes/labels/component/clientlib)
-
-
-<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
-[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/cli-roadmap.md?pixel)]()
-<!-- END MUNGE: GENERATED_ANALYTICS -->
diff --git a/contributors/devel/controllers.md b/contributors/devel/controllers.md
index 057d74ee..c3dcebdf 100644
--- a/contributors/devel/controllers.md
+++ b/contributors/devel/controllers.md
@@ -18,10 +18,10 @@ Watches, etc, are all merely optimizations of this logic.
## Guidelines
-When you’re writing controllers, there are few guidelines that will help make sure you get the results and performance
-you’re looking for.
+When you're writing controllers, there are few guidelines that will help make sure you get the results and performance
+you're looking for.
-1. Operate on one item at a time. If you use a `workqueue.Interface`, you’ll be able to queue changes for a
+1. Operate on one item at a time. If you use a `workqueue.Interface`, you'll be able to queue changes for a
particular resource and later pop them in multiple “worker” gofuncs with a guarantee that no two gofuncs will
work on the same item at the same time.
@@ -37,11 +37,11 @@ you’re looking for.
resourceB/Y”, your controller could observe “created resourceB/Y” and “created resourceA/X”.
-1. Level driven, not edge driven. Just like having a shell script that isn’t running all the time, your controller
+1. Level driven, not edge driven. Just like having a shell script that isn't running all the time, your controller
may be off for an indeterminate amount of time before running again.
- If an API object appears with a marker value of `true`, you can’t count on having seen it turn from `false` to `true`,
- only that you now observe it being `true`. Even an API watch suffers from this problem, so be sure that you’re not
+ If an API object appears with a marker value of `true`, you can't count on having seen it turn from `false` to `true`,
+ only that you now observe it being `true`. Even an API watch suffers from this problem, so be sure that you're not
counting on seeing a change unless your controller is also marking the information it last made the decision on in
the object's status.
@@ -61,18 +61,18 @@ you’re looking for.
1. Never mutate original objects! Caches are shared across controllers, this means that if you mutate your "copy"
- (actually a reference or shallow copy) of an object, you’ll mess up other controllers (not just your own).
+ (actually a reference or shallow copy) of an object, you'll mess up other controllers (not just your own).
The most common point of failure is making a shallow copy, then mutating a map, like `Annotations`. Use
`api.Scheme.Copy` to make a deep copy.
1. Wait for your secondary caches. Many controllers have primary and secondary resources. Primary resources are the
- resources that you’ll be updating `Status` for. Secondary resources are resources that you’ll be managing
+ resources that you'll be updating `Status` for. Secondary resources are resources that you'll be managing
(creating/deleting) or using for lookups.
Use the `framework.WaitForCacheSync` function to wait for your secondary caches before starting your primary sync
- functions. This will make sure that things like a Pod count for a ReplicaSet isn’t working off of known out of date
+ functions. This will make sure that things like a Pod count for a ReplicaSet isn't working off of known out of date
information that results in thrashing.
@@ -87,14 +87,14 @@ you’re looking for.
1. Percolate errors to the top level for consistent re-queuing. We have a `workqueue.RateLimitingInterface` to allow
simple requeuing with reasonable backoffs.
- Your main controller func should return an error when requeuing is necessary. When it isn’t, it should use
+ Your main controller func should return an error when requeuing is necessary. When it isn't, it should use
`utilruntime.HandleError` and return nil instead. This makes it very easy for reviewers to inspect error handling
- cases and to be confident that your controller doesn’t accidentally lose things it should retry for.
+ cases and to be confident that your controller doesn't accidentally lose things it should retry for.
1. Watches and Informers will “sync”. Periodically, they will deliver every matching object in the cluster to your
`Update` method. This is good for cases where you may need to take additional action on the object, but sometimes you
- know there won’t be more work to do.
+ know there won't be more work to do.
In cases where you are *certain* that you don't need to requeue items when there are no new changes, you can compare the
resource version of the old and new objects. If they are the same, you skip requeuing the work. Be careful when you
diff --git a/contributors/devel/development.md b/contributors/devel/development.md
index 8248839e..5a1f1262 100644
--- a/contributors/devel/development.md
+++ b/contributors/devel/development.md
@@ -15,7 +15,7 @@ branch, but release branches of Kubernetes should not change.
Official releases are built using Docker containers. To build Kubernetes using
Docker please follow [these instructions]
-(http://releases.k8s.io/HEAD/build-tools/README.md).
+(http://releases.k8s.io/HEAD/build/README.md).
## Building Kubernetes on a local OS/shell environment
@@ -30,8 +30,9 @@ some of which may be incompatible in subtle ways, so we recommend
Kubernetes is written in the [Go](http://golang.org) programming language.
To build Kubernetes without using Docker containers, you'll need a Go
-development environment. Builds for Kubernetes 1.0 - 1.2 require Go version
-1.4.2. Builds for Kubernetes 1.3 and higher require Go version 1.6.0. If you
+development environment. Builds for Kubernetes 1.0 - 1.2 require Go version 1.4.2.
+Builds for Kubernetes 1.3 and 1.4 require Go version 1.6. Builds for Kubernetes 1.5
+and higher require Go version 1.7. If you
haven't set up a Go development environment, please follow [these
instructions](http://golang.org/doc/code.html) to install the go tools.
@@ -107,10 +108,10 @@ bump to a minor release version for security updates.
Since kubernetes is mostly built and tested in containers, there are a few
unique places you need to update the go version.
-- The image for cross compiling in [build-tools/build-image/cross/](../../build-tools/build-image/cross/). The `VERSION` file and `Dockerfile`.
+- The image for cross compiling in [build/build-image/cross/](https://github.com/kubernetes/kubernetes/blob/master/build/build-image/cross/). The `VERSION` file and `Dockerfile`.
- Update [dockerized-e2e-runner.sh](https://github.com/kubernetes/test-infra/blob/master/jenkins/dockerized-e2e-runner.sh) to run a kubekins-e2e with the desired go version, which requires pushing [e2e-image](https://github.com/kubernetes/test-infra/tree/master/jenkins/e2e-image) and [test-image](https://github.com/kubernetes/test-infra/tree/master/jenkins/test-image) images that are `FROM` the desired go version.
-- The docker image being run in [gotest-dockerized.sh](https://github.com/kubernetes/test-infra/tree/master/jenkins/gotest-dockerized.sh).
-- The cross tag `KUBE_BUILD_IMAGE_CROSS_TAG` in [build-tools/common.sh](../../build-tools/common.sh)
+- The docker image being run in [gotest-dockerized.sh](https://github.com/kubernetes/test-infra/blob/master/jenkins/gotest-dockerized.sh).
+- The cross tag `KUBE_BUILD_IMAGE_CROSS_TAG` in [build/common.sh](https://github.com/kubernetes/kubernetes/blob/master/build/common.sh)
## Workflow
diff --git a/contributors/devel/e2e-tests.md b/contributors/devel/e2e-tests.md
index fc8f1995..78176f44 100644
--- a/contributors/devel/e2e-tests.md
+++ b/contributors/devel/e2e-tests.md
@@ -271,7 +271,7 @@ Next, specify the docker repository where your ci images will be pushed.
* Push the federation container images
```sh
- $ build-tools/push-federation-images.sh
+ $ build/push-federation-images.sh
```
#### Deploy federation control plane
@@ -330,7 +330,7 @@ running:
```
cluster/log-dump.sh <directory>
-````
+```
will ssh to the master and all nodes and download a variety of useful logs to
the provided directory (which should already exist).
@@ -480,6 +480,15 @@ breaking changes, it does *not* block the merge-queue, and thus should run in
some separate test suites owned by the feature owner(s)
(see [Continuous Integration](#continuous-integration) below).
+In order to simplify running component-specific test suites, it may also be
+necessary to tag tests with a component label. The component may include
+standard and non-standard tests, so the `[Feature:.+]` label is not sufficient for
+this purpose. These component labels have no impact on the standard e2e test
+suites. The following component labels have been defined:
+
+ - `[Volume]`: All tests related to volumes and storage: volume plugins,
+attach/detatch controller, persistent volume controller, etc.
+
### Viper configuration and hierarchichal test parameters.
The future of e2e test configuration idioms will be increasingly defined using viper, and decreasingly via flags.
diff --git a/contributors/devel/faster_reviews.md b/contributors/devel/faster_reviews.md
index 85568d3f..0ad9b540 100644
--- a/contributors/devel/faster_reviews.md
+++ b/contributors/devel/faster_reviews.md
@@ -101,14 +101,14 @@ something (or that you won't remember what you yourself did), comment it. If
you think there's something pretty obvious that we could follow up on, add a
TODO. Many code-review comments are about this exact issue.
-## 5. Tests are almost always required
+## 6. Tests are almost always required
Nothing is more frustrating than doing a review, only to find that the tests are
inadequate or even entirely absent. Very few PRs can touch code and NOT touch
tests. If you don't know how to test Feature-X - ask! We'll be happy to help
you design things for easy testing or to suggest appropriate test cases.
-## 6. Look for opportunities to generify
+## 7. Look for opportunities to generify
If you find yourself writing something that touches a lot of modules, think hard
about the dependencies you are introducing between packages. Can some of what
@@ -121,7 +121,7 @@ month and it happens to exactly duplicate some tricky stuff from Feature-W,
consider prefactoring core logic out and using it in both Feature-W and
Feature-X. But do that in a different commit or PR, please.
-## 7. Fix feedback in a new commit
+## 8. Fix feedback in a new commit
Your reviewer has finally sent you some feedback on Feature-X. You make a bunch
of changes and ... what? You could patch those into your commits with git
@@ -157,7 +157,7 @@ description paragraph describing in more detail the change intended. Do not link
pull requests by `#` in a commit description, because GitHub creates lots of
spam. Instead, reference other PRs via the PR your commit is in.
-## 8. KISS, YAGNI, MVP, etc
+## 9. KISS, YAGNI, MVP, etc
Sometimes we need to remind each other of core tenets of software design - Keep
It Simple, You Aren't Gonna Need It, Minimum Viable Product, and so on. Adding
@@ -165,7 +165,7 @@ features "because we might need it later" is antithetical to software that
ships. Add the things you need NOW and (ideally) leave room for things you
might need later - but don't implement them now.
-## 9. Push back
+## 10. Push back
We understand that it is hard to imagine, but sometimes we make mistakes. It's
OK to push back on changes requested during a review. If you have a good reason
@@ -173,7 +173,7 @@ for doing something a certain way, you are absolutely allowed to debate the
merits of a requested change. You might be overruled, but you might also
prevail. We're mostly pretty reasonable people. Mostly.
-## 10. I'm still getting stalled - help?!
+## 11. I'm still getting stalled - help?!
So, you've done all that and you still aren't getting any PR love? Here's some
things you can do that might help kick a stalled process along:
diff --git a/contributors/devel/instrumentation.md b/contributors/devel/instrumentation.md
index b73221a9..66992e40 100644
--- a/contributors/devel/instrumentation.md
+++ b/contributors/devel/instrumentation.md
@@ -1,11 +1,28 @@
-## Instrumenting Kubernetes with a new metric
+## Instrumenting Kubernetes
-The following is a step-by-step guide for adding a new metric to the Kubernetes
-code base.
+The following references and outlines general guidelines for metric instrumentation
+in Kubernetes components. Components are instrumented using the
+[Prometheus Go client library](https://github.com/prometheus/client_golang). For non-Go
+components. [Libraries in other languages](https://prometheus.io/docs/instrumenting/clientlibs/)
+are available.
-We use the Prometheus monitoring system's golang client library for
-instrumenting our code. Once you've picked out a file that you want to add a
-metric to, you should:
+The metrics are exposed via HTTP in the
+[Prometheus metric format](https://prometheus.io/docs/instrumenting/exposition_formats/),
+which is open and well-understood by a wide range of third party applications and vendors
+outside of the Prometheus eco-system.
+
+The [general instrumentation advice](https://prometheus.io/docs/practices/instrumentation/)
+from the Prometheus documentation applies. This document reiterates common pitfalls and some
+Kubernetes specific considerations.
+
+Prometheus metrics are cheap as they have minimal internal memory state. Set and increment
+operations are thread safe and take 10-25 nanoseconds (Go &amp; Java).
+Thus, instrumentation can and should cover all operationally relevant aspects of an application,
+internal and external.
+
+## Quick Start
+
+The following describes the basic steps required to add a new metric (in Go).
1. Import "github.com/prometheus/client_golang/prometheus".
@@ -22,29 +39,180 @@ the values.
labels on the metric. If so, add "Vec" to the name of the type of metric you
want and add a slice of the label names to the definition.
- https://github.com/kubernetes/kubernetes/blob/cd3299307d44665564e1a5c77d0daa0286603ff5/pkg/apiserver/apiserver.go#L53
- https://github.com/kubernetes/kubernetes/blob/cd3299307d44665564e1a5c77d0daa0286603ff5/pkg/kubelet/metrics/metrics.go#L31
+ [Example](https://github.com/kubernetes/kubernetes/blob/cd3299307d44665564e1a5c77d0daa0286603ff5/pkg/apiserver/apiserver.go#L53)
+ ```go
+ requestCounter = prometheus.NewCounterVec(
+ prometheus.CounterOpts{
+ Name: "apiserver_request_count",
+ Help: "Counter of apiserver requests broken out for each verb, API resource, client, and HTTP response code.",
+ },
+ []string{"verb", "resource", "client", "code"},
+ )
+ ```
3. Register the metric so that prometheus will know to export it.
- https://github.com/kubernetes/kubernetes/blob/cd3299307d44665564e1a5c77d0daa0286603ff5/pkg/kubelet/metrics/metrics.go#L74
- https://github.com/kubernetes/kubernetes/blob/cd3299307d44665564e1a5c77d0daa0286603ff5/pkg/apiserver/apiserver.go#L78
+ [Example](https://github.com/kubernetes/kubernetes/blob/cd3299307d44665564e1a5c77d0daa0286603ff5/pkg/apiserver/apiserver.go#L78)
+ ```go
+ func init() {
+ prometheus.MustRegister(requestCounter)
+ prometheus.MustRegister(requestLatencies)
+ prometheus.MustRegister(requestLatenciesSummary)
+ }
+ ```
4. Use the metric by calling the appropriate method for your metric type (Set,
Inc/Add, or Observe, respectively for Gauge, Counter, or Histogram/Summary),
first calling WithLabelValues if your metric has any labels
- https://github.com/kubernetes/kubernetes/blob/3ce7fe8310ff081dbbd3d95490193e1d5250d2c9/pkg/kubelet/kubelet.go#L1384
- https://github.com/kubernetes/kubernetes/blob/cd3299307d44665564e1a5c77d0daa0286603ff5/pkg/apiserver/apiserver.go#L87
+ [Example](https://github.com/kubernetes/kubernetes/blob/cd3299307d44665564e1a5c77d0daa0286603ff5/pkg/apiserver/apiserver.go#L87)
+ ```go
+ requestCounter.WithLabelValues(*verb, *resource, client, strconv.Itoa(*httpCode)).Inc()
+ ```
+
+
+## Instrumentation types
+
+Components have metrics capturing events and states that are inherent to their
+application logic. Examples are request and error counters, request latency
+histograms, or internal garbage collection cycles. Those metrics are instrumented
+directly in the application code.
+
+Secondly, there are business logic metrics. Those are not about observed application
+behavior but abstract system state, such as desired replicas for a deployment.
+They are not directly instrumented but collected from otherwise exposed data.
+
+In Kubernetes they are generally captured in the [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics)
+component, which reads them from the API server.
+For this types of metric exposition, the
+[exporter guidelines](https://prometheus.io/docs/instrumenting/writing_exporters/)
+apply additionally.
+
+## Naming
+
+Metrics added directly by application or package code should have a unique name.
+This avoids collisions of metrics added via dependencies. They also clearly
+distinguish metrics collected with different semantics. This is solved through
+prefixes:
+
+```
+<component_name>_<metric>
+```
+
+For example, suppose the kubelet instrumented its HTTP requests but also uses
+an HTTP router providing its own implementation. Both expose metrics on total
+http requests. They should be distinguishable as in:
+
+```
+kubelet_http_requests_total{path=”/some/path”,status=”200”}
+routerpkg_http_requests_total{path=”/some/path”,status=”200”,method=”GET”}
+```
+
+As we can see they expose different labels and thus a naming collision would
+not have been possible to resolve even if both metrics counted the exact same
+requests.
+
+Resource objects that occur in names should inherit the spelling that is used
+in kubectl, i.e. daemon sets are `daemonset` rather than `daemon_set`.
+
+## Dimensionality & Cardinality
+
+Metrics can often replace more expensive logging as they are time-aggregated
+over a sampling interval. The [multidimensional data model](https://prometheus.io/docs/concepts/data_model/)
+enables deep insights and all metrics should use those label dimensions
+where appropriate.
+
+A common error that often causes performance issues in the ingesting metric
+system is considering dimensions that inhibit or eliminate time aggregation
+by being too specific. Typically those are user IDs or error messages.
+More generally: one should know a comprehensive list of all possible values
+for a label at instrumentation time.
+
+Notable exceptions are exporters like kube-state-metrics, which expose per-pod
+or per-deployment metrics, which are theoretically unbound over time as one could
+constantly create new ones, with new names. However, they have
+a reasonable upper bound for a given size of infrastructure they refer to and
+its typical frequency of changes.
+
+In general, “external” labels like pod or node name do not belong into the
+instrumentation itself. They are to be attached to metrics by the collecting
+system that has the external knowledge ([blog post](https://www.robustperception.io/target-labels-are-for-life-not-just-for-christmas/)).
+
+## Normalization
+
+Metrics should be normalized with respect to their dimensions. They should
+expose the minimal set of labels, each of which provides additional information.
+Labels that are composed from values of different labels are not desirable.
+For example:
+
+```
+example_metric{pod=”abc”,container=”proxy”,container_long=”abc/proxy”}
+```
+
+It often seems feasible to add additional meta information about an object
+to all metrics about that object, e.g.:
+
+```
+kube_pod_container_restarts{namespace=...,pod=...,container=...}
+```
+
+A common use case is wanting to look at such metrics w.r.t to the node the
+pod is scheduled on. So it seems convenient to add a “node” label.
+
+```
+kube_pod_container_restarts{namespace=...,pod=...,container=...,node=...}
+```
+
+This however only caters to one specific query use case. There are many more
+pieces of metadata that could be added, effectively blowing up the instrumentation.
+They are also not guaranteed to be stable over time. What if pods at some
+point can be live migrated?
+Those pieces of information should be normalized into an info-level metric
+([blog post](https://www.robustperception.io/exposing-the-software-version-to-prometheus/)),
+which is always set to 1. For example:
+
+```
+kube_pod_info{pod=...,namespace=...,pod_ip=...,host_ip=..,node=..., ...}
+```
+
+The metric system can later denormalize those along the identifying labels
+“pod” and “namespace” labels. This leads to...
+
+## Resource Referencing
+
+It is often desirable to correlate different metrics about a common object,
+such as a pod. Label dimensions can be used to match up different metrics.
+This is most easy if label names and values are following a common pattern.
+For metrics exposed by the same application, that often happens naturally.
+
+For a system composed of several independent, and also pluggable components,
+it makes sense to set cross-component standards to allow easy querying in
+metric systems without extensive post-processing of data.
+In Kubernetes, those are the resource objects such as deployments,
+pods, or services and the namespace they belong to.
+
+The following should be consistently used:
+
+```
+example_metric_ccc{pod=”example-app-5378923”, namespace=”default”}
+```
+
+An object is referenced by its unique name in a label named after the resource
+itself (i.e. `pod`/`deployment`/... and not `pod_name`/`deployment_name`)
+and the namespace it belongs to in the `namespace` label.
+Note: namespace/name combinations are only unique at a certain point in time.
+For time series this is given by the timestamp associated with any data point.
+UUIDs are truly unique but not convenient to use in user-facing time series
+queries.
+They can still be incorporated using an info level metric as described above for
+`kube_pod_info`. A query to a metric system selecting by UUID via a the info level
+metric could look as follows:
-These are the metric type definitions if you're curious to learn about them or
-need more information:
+```
+kube_pod_restarts and on(namespace, pod) kube_pod_info{uuid=”ABC”}
+```
-https://github.com/prometheus/client_golang/blob/master/prometheus/gauge.go
-https://github.com/prometheus/client_golang/blob/master/prometheus/counter.go
-https://github.com/prometheus/client_golang/blob/master/prometheus/histogram.go
-https://github.com/prometheus/client_golang/blob/master/prometheus/summary.go
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
diff --git a/contributors/devel/kubectl-conventions.md b/contributors/devel/kubectl-conventions.md
index 1e94b3ba..2bb5f49d 100644
--- a/contributors/devel/kubectl-conventions.md
+++ b/contributors/devel/kubectl-conventions.md
@@ -13,6 +13,7 @@ Updated: 8/27/2015
- [Flag conventions](#flag-conventions)
- [Output conventions](#output-conventions)
- [Documentation conventions](#documentation-conventions)
+ - [kubectl Factory conventions](#kubectl-Factory-conventions)
- [Command implementation conventions](#command-implementation-conventions)
- [Generators](#generators)
@@ -245,6 +246,21 @@ rather than "RESOURCE" or "KIND"
* Use "NAME" for resource names
+## kubectl Factory conventions
+
+The kubectl `Factory` is a large interface which is used to provide access to clients,
+polymorphic inspection, and polymorphic mutation. The `Factory` is layered in
+"rings" in which one ring may reference inner rings, but not peers or outer rings.
+This is done to allow composition by extenders.
+
+In order for composers to be able to provide alternative factory implementations
+they need to provide low level pieces of *certain* functions so that when the factory
+calls back into itself it uses the custom version of the function. Rather than try
+to enumerate everything that someone would want to override we split the factory into
+rings, where each ring can depend on methods an earlier ring, but cannot depend upon
+peer methods in its own ring.
+
+
## Command implementation conventions
For every command there should be a `NewCmd<CommandName>` function that creates
diff --git a/contributors/devel/kubelet-cri-networking.md b/contributors/devel/kubelet-cri-networking.md
index 90b33220..123a0cc4 100644
--- a/contributors/devel/kubelet-cri-networking.md
+++ b/contributors/devel/kubelet-cri-networking.md
@@ -41,19 +41,19 @@ background on k8s networking could be found
[here](http://kubernetes.io/docs/admin/networking/)
## Requirements
-1. Kubelet expects the runtime shim to manage pod’s network life cycle. Pod
+1. Kubelet expects the runtime shim to manage pod's network life cycle. Pod
networking should be handled accordingly along with pod sandbox operations.
- * `RunPodSandbox` must set up pod’s network. This includes, but is not limited
-to allocating a pod IP, configuring the pod’s network interfaces and default
+ * `RunPodSandbox` must set up pod's network. This includes, but is not limited
+to allocating a pod IP, configuring the pod's network interfaces and default
network route. Kubelet expects the pod sandbox to have an IP which is
routable within the k8s cluster, if `RunPodSandbox` returns successfully.
-`RunPodSandbox` must return an error if it fails to set up the pod’s network.
-If the pod’s network has already been set up, `RunPodSandbox` must skip
+`RunPodSandbox` must return an error if it fails to set up the pod's network.
+If the pod's network has already been set up, `RunPodSandbox` must skip
network setup and proceed.
- * `StopPodSandbox` must tear down the pod’s network. The runtime shim
-must return error on network tear down failure. If pod’s network has
+ * `StopPodSandbox` must tear down the pod's network. The runtime shim
+must return error on network tear down failure. If pod's network has
already been torn down, `StopPodSandbox` must skip network tear down and proceed.
- * `RemovePodSandbox` may tear down pod’s network, if the networking has
+ * `RemovePodSandbox` may tear down pod's network, if the networking has
not been torn down already. `RemovePodSandbox` must return error on
network tear down failure.
* Response from `PodSandboxStatus` must include pod sandbox network status.
diff --git a/contributors/devel/kubemark-guide.md b/contributors/devel/kubemark-guide.md
index e914226d..cb699a88 100755
--- a/contributors/devel/kubemark-guide.md
+++ b/contributors/devel/kubemark-guide.md
@@ -13,27 +13,38 @@ and how to use it.
## Architecture
-On a very high level Kubemark cluster consists of two parts: real master
-components and a set of “Hollow” Nodes. The prefix “Hollow” means an
-implementation/instantiation of a component with all “moving” parts mocked out.
-The best example is HollowKubelet, which pretends to be an ordinary Kubelet, but
-does not start anything, nor mount any volumes - it just lies it does. More
-detailed design and implementation details are at the end of this document.
-
-Currently master components run on a dedicated machine(s), and HollowNodes run
-on an ‘external’ Kubernetes cluster. This design has a slight advantage, over
-running master components on external cluster, of completely isolating master
-resources from everything else.
+On a very high level, Kubemark cluster consists of two parts: a real master
+and a set of “Hollow” Nodes. The prefix “Hollow” to any component means an
+implementation/instantiation of the actual component with all “moving”
+parts mocked out. The best example is HollowKubelet, which pretends to be an
+ordinary Kubelet, but does not start anything, nor mount any volumes - it just
+lies it does. More detailed design and implementation details are at the end
+of this document.
+
+Currently, master components run on a dedicated machine as pods that are
+created/managed by kubelet, which itself runs as either a systemd or a supervisord
+service on the master VM depending on the VM distro (though currently it is
+only systemd as we use a GCI image). Having a dedicated machine for the master
+has a slight advantage over running the master components on an external cluster,
+which is being able to completely isolate master resources from everything else.
+The HollowNodes on the other hand are run on an ‘external’ Kubernetes cluster
+as pods in an isolated namespace (named kubemark). This idea of using pods on a
+real cluster behave (or act) as nodes on the kubemark cluster lies at the heart of
+kubemark's design.
## Requirements
To run Kubemark you need a Kubernetes cluster (called `external cluster`)
for running all your HollowNodes and a dedicated machine for a master.
-Master machine has to be directly routable from HollowNodes. You also need an
-access to some Docker repository.
+Master machine has to be directly routable from HollowNodes. You also need
+access to a Docker repository (which is gcr.io in the case of GCE) that has the
+container images for etcd, hollow-node and node-problem-detector.
-Currently scripts are written to be easily usable by GCE, but it should be
+Currently, scripts are written to be easily usable by GCE, but it should be
relatively straightforward to port them to different providers or bare metal.
+There is an ongoing effort to refactor kubemark code into provider-specific (gce)
+and provider-independent code, which should make it relatively simple to run
+kubemark clusters on other cloud providers as well.
## Common use cases and helper scripts
@@ -43,8 +54,10 @@ Common workflow for Kubemark is:
- monitoring test execution and debugging problems
- turning down Kubemark cluster
-Included in descriptions there will be comments helpful for anyone who’ll want to
-port Kubemark to different providers.
+(For now) Included in descriptions there will be comments helpful for anyone who’ll
+want to port Kubemark to different providers.
+(Later) When the refactoring mentioned in the above section finishes, we would replace
+these comments with a clean API that would allow kubemark to run on top of any provider.
### Starting a Kubemark cluster
@@ -52,46 +65,32 @@ To start a Kubemark cluster on GCE you need to create an external kubernetes
cluster (it can be GCE, GKE or anything else) by yourself, make sure that kubeconfig
points to it by default, build a kubernetes release (e.g. by running
`make quick-release`) and run `test/kubemark/start-kubemark.sh` script.
-This script will create a VM for master components, Pods for HollowNodes
-and do all the setup necessary to let them talk to each other. It will use the
-configuration stored in `cluster/kubemark/config-default.sh` - you can tweak it
-however you want, but note that some features may not be implemented yet, as
-implementation of Hollow components/mocks will probably be lagging behind ‘real’
-one. For performance tests interesting variables are `NUM_NODES` and
-`MASTER_SIZE`. After start-kubemark script is finished you’ll have a ready
-Kubemark cluster, a kubeconfig file for talking to the Kubemark cluster is
-stored in `test/kubemark/kubeconfig.kubemark`.
-
-Currently we're running HollowNode with limit of 0.05 a CPU core and ~60MB or
-memory, which taking into account default cluster addons and fluentD running on
-an 'external' cluster, allows running ~17.5 HollowNodes per core.
+This script will create a VM for master (along with mounted PD and firewall rules set),
+then start kubelet and run the pods for the master components. Following this, it
+sets up the HollowNodes as Pods on the external cluster and do all the setup necessary
+to let them talk to the kubemark apiserver. It will use the configuration stored in
+`cluster/kubemark/config-default.sh` - you can tweak it however you want, but note that
+some features may not be implemented yet, as implementation of Hollow components/mocks
+will probably be lagging behind ‘real’ one. For performance tests interesting variables
+are `NUM_NODES` and `KUBEMARK_MASTER_SIZE`. After start-kubemark script is finished,
+you’ll have a ready Kubemark cluster, and a kubeconfig file for talking to the Kubemark
+cluster is stored in `test/kubemark/resources/kubeconfig.kubemark`.
+
+Currently we're running HollowNode with a limit of 0.09 CPU core/pod and 220MB of memory.
+However, if we also take into account the resources absorbed by default cluster addons
+and fluentD running on the 'external' cluster, this limit becomes ~0.1 CPU core/pod,
+thus allowing ~10 HollowNodes to run per core (on an "n1-standard-8" VM node).
#### Behind the scene details:
-Start-kubemark script does quite a lot of things:
+start-kubemark.sh script does quite a lot of things:
-- Creates a master machine called hollow-cluster-master and PD for it (*uses
-gcloud, should be easy to do outside of GCE*)
-
-- Creates a firewall rule which opens port 443\* on the master machine (*uses
-gcloud, should be easy to do outside of GCE*)
-
-- Builds a Docker image for HollowNode from the current repository and pushes it
-to the Docker repository (*GCR for us, using scripts from
-`cluster/gce/util.sh` - it may get tricky outside of GCE*)
-
-- Generates certificates and kubeconfig files, writes a kubeconfig locally to
-`test/kubemark/kubeconfig.kubemark` and creates a Secret which stores kubeconfig for
-HollowKubelet/HollowProxy use (*used gcloud to transfer files to Master, should
-be easy to do outside of GCE*).
-
-- Creates a ReplicationController for HollowNodes and starts them up. (*will
-work exactly the same everywhere as long as MASTER_IP will be populated
-correctly, but you’ll need to update docker image address if you’re not using
-GCR and default image name*)
-
-- Waits until all HollowNodes are in the Running phase (*will work exactly the
-same everywhere*)
+- Prepare a master machine named MASTER_NAME (this variable's value should be set by this point):
+ (*the steps below use gcloud, and should be easy to do outside of GCE*)
+ 1. Creates a Persistent Disk for use by the master (one more for etcd-events, if flagged)
+ 2. Creates a static IP address for the master in the cluster and assign it to variable MASTER_IP
+ 3. Creates a VM instance for the master, configured with the PD and IP created above.
+ 4. Set firewall rule in the master to open port 443\* for all TCP traffic by default.
<sub>\* Port 443 is a secured port on the master machine which is used for all
external communication with the API server. In the last sentence *external*
@@ -99,6 +98,40 @@ means all traffic coming from other machines, including all the Nodes, not only
from outside of the cluster. Currently local components, i.e. ControllerManager
and Scheduler talk with API server using insecure port 8080.</sub>
+- [Optional to read] Establish necessary certs/keys required for setting up the PKI for kubemark cluster:
+ (*the steps below are independent of GCE and work for all providers*)
+ 1. Generate a randomly named temporary directory for storing PKI certs/keys which is delete-trapped on EXIT.
+ 2. Create a bearer token for 'admin' in master.
+ 3. Generate certificate for CA and (certificate + private-key) pair for each of master, kubelet and kubecfg.
+ 4. Generate kubelet and kubeproxy tokens for master.
+ 5. Write a kubeconfig locally to `test/kubemark/resources/kubeconfig.kubemark` for enabling local kubectl use.
+
+- Set up environment and start master components (through `start-kubemark-master.sh` script):
+ (*the steps below use gcloud for SSH and SCP to master, and should be easy to do outside of GCE*)
+ 1. SSH to the master machine and create a new directory (`/etc/srv/kubernetes`) and write all the
+ certs/keys/tokens/passwords to it.
+ 2. SCP all the master pod manifests, shell scripts (`start-kubemark-master.sh`, `configure-kubectl.sh`, etc),
+ config files for passing env variables (`kubemark-master-env.sh`) from the local machine to the master.
+ 3. SSH to the master machine and run the startup script `start-kubemark-master.sh` (and possibly others).
+
+ Note: The directory structure and the functions performed by the startup script(s) can vary based on master distro.
+ We currently support the GCI image `gci-dev-56-8977-0-0` in GCE.
+
+- Set up and start HollowNodes (as pods) on the external cluster:
+ (*the steps below (except 2nd and 3rd) are independent of GCE and work for all providers*)
+ 1. Identify the right kubemark binary from the current kubernetes repo for the platform linux/amd64.
+ 2. Create a Docker image for HollowNode using this binary and upload it to a remote Docker repository.
+ (We use gcr.io/ as our remote docker repository in GCE, should be different for other providers)
+ 3. [One-off] Create and upload a Docker image for NodeProblemDetector (see kubernetes/node-problem-detector repo),
+ which is one of the containers in the HollowNode pod, besides HollowKubelet and HollowProxy. However we
+ use it with a hollow config that esentially has an empty set of rules and conditions to be detected.
+ This step is required only for other cloud providers, as the docker image for GCE already exists on GCR.
+ 4. Create secret which stores kubeconfig for use by HollowKubelet/HollowProxy, addons, and configMaps
+ for the HollowNode and the HollowNodeProblemDetector.
+ 5. Create a ReplicationController for HollowNodes that starts them up, after replacing all variables in
+ the hollow-node_template.json resource.
+ 6. Wait until all HollowNodes are in the Running phase.
+
### Running e2e tests on Kubemark cluster
To run standard e2e test on your Kubemark cluster created in the previous step
@@ -129,7 +162,7 @@ Master machine (currently) differs from the ordinary one.
If you need to debug master machine you can do similar things as you do on your
ordinary master. The difference between Kubemark setup and ordinary setup is
that in Kubemark etcd is run as a plain docker container, and all master
-components are run as normal processes. There’s no Kubelet overseeing them. Logs
+components are run as normal processes. There's no Kubelet overseeing them. Logs
are stored in exactly the same place, i.e. `/var/logs/` directory. Because
binaries are not supervised by anything they won't be restarted in the case of a
crash.
@@ -145,7 +178,7 @@ one of them you need to learn which hollow-node pod corresponds to a given
HollowNode known by the Master. During self-registeration HollowNodes provide
their cluster IPs as Names, which means that if you need to find a HollowNode
named `10.2.4.5` you just need to find a Pod in external cluster with this
-cluster IP. There’s a helper script
+cluster IP. There's a helper script
`test/kubemark/get-real-pod-for-hollow-node.sh` that does this for you.
When you have a Pod name you can use `kubectl logs` on external cluster to get
@@ -156,7 +189,7 @@ E.g. you want to see the logs of HollowKubelet on which pod `my-pod` is running.
To do so you can execute:
```
-$ kubectl kubernetes/test/kubemark/kubeconfig.kubemark describe pod my-pod
+$ kubectl kubernetes/test/kubemark/resources/kubeconfig.kubemark describe pod my-pod
```
Which outputs pod description and among it a line:
@@ -190,21 +223,36 @@ All those things should work exactly the same on all cloud providers.
On GCE you just need to execute `test/kubemark/stop-kubemark.sh` script, which
will delete HollowNode ReplicationController and all the resources for you. On
-other providers you’ll need to delete all this stuff by yourself.
+other providers you’ll need to delete all this stuff by yourself. As part of
+the effort mentioned above to refactor kubemark into provider-independent and
+provider-specific parts, the resource deletion logic specific to the provider
+would move out into a clean API.
-## Some current implementation details
+## Some current implementation details and future roadmap
Kubemark master uses exactly the same binaries as ordinary Kubernetes does. This
means that it will never be out of date. On the other hand HollowNodes use
existing fake for Kubelet (called SimpleKubelet), which mocks its runtime
manager with `pkg/kubelet/dockertools/fake_manager.go`, where most logic sits.
-Because there’s no easy way of mocking other managers (e.g. VolumeManager), they
-are not supported in Kubemark (e.g. we can’t schedule Pods with volumes in them
+Because there's no easy way of mocking other managers (e.g. VolumeManager), they
+are not supported in Kubemark (e.g. we can't schedule Pods with volumes in them
yet).
-As the time passes more fakes will probably be plugged into HollowNodes, but
-it’s crucial to make it as simple as possible to allow running a big number of
-Hollows on a single core.
+We currently plan to extend kubemark along the following directions:
+- As you would have noticed at places above, we aim to make kubemark more structured
+ and easy to run across various providers without having to tweak the setup scripts,
+ using a well-defined kubemark-provider API.
+- Allow kubemark to run on various distros (GCI, debian, redhat, etc) for any
+ given provider.
+- Make Kubemark performance on ci-tests mimic real cluster ci-tests on metrics such as
+ CPU, memory and network bandwidth usage and realizing this goal through measurable
+ objectives (like the kubemark metric should vary no more than X% with the real
+ cluster metric). We could also use metrics reported by Prometheus.
+- Improve logging of CI-test metrics (such as aggregated API call latencies, scheduling
+ call latencies, %ile for CPU/mem usage of different master components in density/load
+ tests) by packing them into well-structured artifacts instead of the (current) dumping
+ to logs.
+- Create a Dashboard that lets easy viewing and comparison of these metrics across tests.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
diff --git a/contributors/devel/local-cluster/local.md b/contributors/devel/local-cluster/local.md
index 60bd5a8f..c3ae7293 100644
--- a/contributors/devel/local-cluster/local.md
+++ b/contributors/devel/local-cluster/local.md
@@ -30,6 +30,10 @@ and run the following (since one needs sudo access to start/stop Kubernetes daem
cd kubernetes
hack/local-up-cluster.sh
```
+If you want to enable RBAC on local cluster, you can run the following:
+```shell
+ENABLE_RBAC=true hack/local-up-cluster.sh
+```
This will build and start a lightweight local cluster, consisting of a master
and a single node. Type Control-C to shut it down.
diff --git a/contributors/devel/on-call-build-cop.md b/contributors/devel/on-call-build-cop.md
index 15c71e5d..92895b26 100644
--- a/contributors/devel/on-call-build-cop.md
+++ b/contributors/devel/on-call-build-cop.md
@@ -7,9 +7,9 @@
### Traffic sources and responsibilities
-* GitHub Kubernetes [issues](https://github.com/kubernetes/kubernetes/issues)
-and [pulls](https://github.com/kubernetes/kubernetes/pulls): Your job is to be
-the first responder to all new issues and PRs. If you are not equipped to do
+* GitHub Kubernetes [issues](https://github.com/kubernetes/kubernetes/issues):
+Your job is to be
+the first responder to all new issues. If you are not equipped to do
this (which is fine!), it is your job to seek guidance!
* Support issues should be closed and redirected to Stackoverflow (see example
@@ -35,18 +35,12 @@ This is the only situation in which you should add a priority/* label
* Assign any issues related to Vagrant to @derekwaynecarr (and @mention him
in the issue)
- * All incoming PRs should be assigned a reviewer.
-
- * unless it is a WIP (Work in Progress), RFC (Request for Comments), or design proposal.
- * An auto-assigner [should do this for you] (https://github.com/kubernetes/kubernetes/pull/12365/files)
- * When in doubt, choose a TL or team maintainer of the most relevant team; they can delegate
-
- * Keep in mind that you can @ mention people in an issue/PR to bring it to
+ * Keep in mind that you can @ mention people in an issue to bring it to
their attention without assigning it to them. You can also @ mention github
teams, such as @kubernetes/goog-ux or @kubernetes/kubectl
- * If you need help triaging an issue or PR, consult with (or assign it to)
-@brendandburns, @thockin, @bgrant0607, @quinton-hoole, @davidopp, @dchen1107,
+ * If you need help triaging an issue, consult with (or assign it to)
+@brendandburns, @thockin, @bgrant0607, @davidopp, @dchen1107,
@lavalamp (all U.S. Pacific Time) or @fgrzadkowski (Central European Time).
* At the beginning of your shift, please add team/* labels to any issues that
diff --git a/contributors/devel/pull-request-commands.md b/contributors/devel/pull-request-commands.md
new file mode 100644
index 00000000..d33411bc
--- /dev/null
+++ b/contributors/devel/pull-request-commands.md
@@ -0,0 +1,80 @@
+# Overview
+
+There are several steps to authoring or reviewing a pull request in Kubernetes.
+Due to limitations on how GitHub permissions can be expressed, the Kubernetes
+authors developed a workflow using comments to run tests and set labels
+used for merging PRs. Merging a PR requires the following steps to be
+completed before the PR will automatically be merged:
+
+- Signing a CLA
+- Setting release notes
+- Having all e2e tests pass
+- Getting an LGTM from a reviewer
+
+# Master Branch Workflow
+
+## Sign the CLA
+
+If you have not already, the `k8s-ci-robot` will leave a comment with
+instructions on how to sign the CLA.
+
+**Important** the email you sign the CLA with must be the same email
+attached to the commits in your PR. If it is not, you may need to change
+the email and repush the commits.
+
+## Set Release Notes
+
+Every PR must be labeled either `release-note` or `release-note-none`.
+This can be done by adding a release-note section to the pr description:
+
+For PRs with a release note:
+
+ ```release-note
+ Your release note here
+ ```
+
+
+
+For PRs without a release note:
+
+ ```release-note
+ NONE
+ ```
+
+Release notes should be present for any PRs with user visible changes such as
+bug-fixes, feature additions, and output format changes.
+
+Additionally, commenting either `/release-note` or `/release-note-none`
+will also set the `release-note` or `release-note-none` labels respectively.
+
+## Run e2e Tests
+
+End-to-end tests are run and post the status results to the PR. PRs authored by
+regular contributors have the tests run automatically. PRs authored by new
+community members require the reviewer to mark the tests as safe to run by
+commenting `@k8s-bot ok to test`.
+
+If an e2e test fails, `k8s-ci-robot` will comment on the PR with the test history
+and the comment-command to re-run that test. e.g.
+
+>
+The magic incantation to run this job again is @k8s-bot unit test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.
+
+## LGTM and Approval
+
+A reviewer will be automatically assigned to your PR by the `k8s-merge-robot`. The
+reviewer will leave comments on your PR. Once all comments have been addressed,
+squash the commits and the reviewer will mark the PR as looking good. This
+can be done with the `/lgtm` command.
+
+## PR merge
+
+After all of the checks have passed, the PR will enter the merge queue:
+[http://submit-queue.k8s.io](http://submit-queue.k8s.io).
+The merge queue re-runs the tests for PRs and then merges them if they pass. The
+merge queue is needed to make sure no incompatible changes have been introduced by other
+PRs since the tests were last run on your PR.
+
+# Comment Commands Reference
+
+Documented [here](https://github.com/kubernetes/test-infra/blob/master/prow/commands.md)
diff --git a/contributors/devel/pull-requests.md b/contributors/devel/pull-requests.md
index 888d7320..fdb91f4c 100644
--- a/contributors/devel/pull-requests.md
+++ b/contributors/devel/pull-requests.md
@@ -93,6 +93,8 @@ request that subsequently needs to be reopened. We want to limit the total numbe
* Remove old PRs that would be difficult to rebase as the underlying code has changed over time
* Encourage code velocity
+For pull requests that are in progress but not ready for review, prefix the PR title with "WIP" and
+track any remaining TODOs in a checklist in the pull request description.
# Automation
diff --git a/contributors/devel/release/README.md b/contributors/devel/release/README.md
new file mode 100644
index 00000000..c3ba6922
--- /dev/null
+++ b/contributors/devel/release/README.md
@@ -0,0 +1,78 @@
+This document captures the requirements and duties of the individuals responsible for Kubernetes releases.
+
+As documented in the [Kubernetes Versioning doc](https://github.com/kubernetes/kubernetes/blob/master/docs/design/versioning.md), there are 3 types of Kubernetes releases:
+* Major (x.0.0)
+* Minor (x.x.0)
+* Patch (x.x.x)
+
+Major and minor releases are managed by a **Kubernetes Release Management Team**, and patch releases are managed by the **Patch Release Manager**. Exact roles and duties are defined below.
+
+# Patch Release Manager
+
+Patch releases are managed by the **Patch Release Manager**. Duties of the patch release manager include:
+* Ensuring the release branch (e.g. `release-1.5`) remains in a healthy state.
+ * If the build breaks or any CI for the release branch becomes unhealthy due to a bad merge or infrastructure issue, ensure that actions are taken ASAP to bring it back to a healthy state.
+* Reviewing and approving [cherry picks](https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md) to the release branch.
+ * Patch releases should not contain new features, so ensure that cherry-picks are for bug/security fixes only.
+ * Cherry picks should not destabilize the branch, so ensure that either the PR has had time to stabilize in master or will have time to stabilize in the release branch before the next patch release is cut.
+* Setting the exact schedule (and cadence) for patch releases and actually cutting the [releases](https://github.com/kubernetes/kubernetes/releases).
+
+Current and past patch release managers are listed [here](https://github.com/kubernetes/community/wiki).
+
+# Kubernetes Release Management Team for Major/Minor Releases
+
+Major and Minor releases are managed by the **Kubernetes Release Management Team** which is responsible for ensuring Kubernetes releases go out on time (as scheduled) and with high quality (stable, with no major bugs).
+
+Roles and responsibilities within the Kubernetes Release Management Team are as follows.
+
+#### Release Management Team Lead
+The Release Management Team Lead is the person ultimately responsible for ensuring the release goes out on-time with high-quality. All the roles defined below report to the Release Management Team Lead.
+* Establishes and communicates responsibilities and deadlines to release management team members, developers/feature owners, SIG leads, etc.
+* Escalates and unblocks any issue that may jeopardise the release schedule or quality as quickly as possible.
+* Finds people to take ownership of any release blocking issues that are not getting adequate attention.
+* Keeps track of, and widely communicates, the status of the release (including status of all sub-leads, all release blockers, etc) and all deadlines leading up to release.
+* Manages [exception](https://github.com/kubernetes/features/blob/master/EXCEPTIONS.md) process for features that want to merge after code freeze.
+
+#### Release Branch Manager
+* Manages (initiates and enforces) code freeze on main branch as scheduled for the release.
+ * Ensures no new features are merged after code complete, unless they've been approved by the [exception process](https://github.com/kubernetes/features/blob/master/EXCEPTIONS.md).
+* Cuts the `release-x.x` branch at the appropriate time during the milestone.
+* Ensures release branch (e.g. `release-1.5`) remains in a healthy state for the duration of the major or minor release.
+ * If the build breaks, or any CI for the release branch becomes unhealthy due to a bad merge or infrastructure issue, ensures that actions are taken ASAP to bring it back to a healthy state.
+* Initiates automatic fast-forwards of the release branch to pick up all changes from master branch, when appropriate.
+* Reviews and approves [cherry picks](https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md) to the release branch.
+ * Ensures onlyl bug/security fixes (but no new features) are cherry-picked after code complete unless approved by the [exception process](https://github.com/kubernetes/features/blob/master/EXCEPTIONS.md).
+ * Ensures that cherry-picks do not destabilize the branch by either giving the PR enough time to stabilize in master or giving it enough time to stabilize in the release branch before cutting the release.
+* Cuts the actual [release](https://github.com/kubernetes/kubernetes/releases).
+
+#### Release Docs Lead
+* Sets release docs related deadlines for developers and works with Release Management Team Lead to ensure they are widely communicated.
+* Sets up release branch for docs.
+* Pings feature owners to ensure that release docs are created on time.
+* Reviews/merges release doc PRs.
+* Merges the docs release branch to master to make release docs live as soon as the release is official.
+
+#### Features Lead
+* Compiles the major themes, new features, known issues, actions required, notable changes to existing behavior, deprecations, etc. and edits them into a release doc checked in to the feature repository (ready to go out with the release).
+* Collects and prepares the release notes
+
+#### Bug Triage Lead
+* Figures out which bugs (whether manually created or automatically generated) should be tracked for the release.
+* Ensures all bugs being tracked for the release have owners that are responsive.
+* Ensures all bugs are triaged as blocking or non-blocking.
+* Ensures all bugs that are blocking are being actively worked on, esp after code complete.
+
+#### Test Infra Lead
+* Sets up and maintains all CI for the release branch.
+
+#### Automated Upgrade Testing Lead
+* Ensures that automated upgrade tests provide a clear go/no-go signal for the release.
+* Tracks and finds owners for all issues with automated upgrade tests.
+
+#### Manual Upgrade Testing Lead
+* Ensures that any gaps in automated upgrade testing are covered by manual upgrade testing.
+* Organizes the manual upgrade testing efforts, including setting up instructions for manual testing, finding manual testing volunteers, and ensuring any issues discovered are communicated widely and fixed quickly.
+
+#### Testing Lead
+* Ensures that all non-upgrade test CI provides a clear go/no-go signal for the release.
+* Tracks and finds owners to fix any issues with any (non-upgrade) tests.
diff --git a/contributors/devel/running-locally.md b/contributors/devel/running-locally.md
index 60fb3c74..64714ce9 100644
--- a/contributors/devel/running-locally.md
+++ b/contributors/devel/running-locally.md
@@ -169,8 +169,7 @@ KUBE_DNS_SERVER_IP="10.0.0.10"
KUBE_DNS_DOMAIN="cluster.local"
```
-To know more on DNS service you can look [here](http://issue.k8s.io/6667). Related documents can be found [here](../../build-tools/kube-dns/#how-do-i-configure-it)
-
+To know more on DNS service you can check out the [docs](http://kubernetes.io/docs/admin/dns/).
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/running-locally.md?pixel)]()
diff --git a/contributors/devel/scheduler.md b/contributors/devel/scheduler.md
index 562297a7..9230535e 100755
--- a/contributors/devel/scheduler.md
+++ b/contributors/devel/scheduler.md
@@ -84,7 +84,7 @@ scheduling policies to apply, and can add new ones.
The policies that are applied when scheduling can be chosen in one of two ways.
The default policies used are selected by the functions `defaultPredicates()` and `defaultPriorities()` in
[plugin/pkg/scheduler/algorithmprovider/defaults/defaults.go](http://releases.k8s.io/HEAD/plugin/pkg/scheduler/algorithmprovider/defaults/defaults.go).
-However, the choice of policies can be overridden by passing the command-line flag `--policy-config-file` to the scheduler, pointing to a JSON file specifying which scheduling policies to use. See [examples/scheduler-policy-config.json](../../examples/scheduler-policy-config.json) for an example
+However, the choice of policies can be overridden by passing the command-line flag `--policy-config-file` to the scheduler, pointing to a JSON file specifying which scheduling policies to use. See [examples/scheduler-policy-config.json](http://releases.k8s.io/HEAD/examples/scheduler-policy-config.json) for an example
config file. (Note that the config file format is versioned; the API is defined in [plugin/pkg/scheduler/api](http://releases.k8s.io/HEAD/plugin/pkg/scheduler/api/)).
Thus to add a new scheduling policy, you should modify [plugin/pkg/scheduler/algorithm/predicates/predicates.go] (http://releases.k8s.io/HEAD/plugin/pkg/scheduler/algorithm/predicates/predicates.go) or add to the directory [plugin/pkg/scheduler/algorithm/priorities](http://releases.k8s.io/HEAD/plugin/pkg/scheduler/algorithm/priorities/), and either register the policy in `defaultPredicates()` or `defaultPriorities()`, or use a policy config file.
diff --git a/contributors/devel/scheduler_algorithm.md b/contributors/devel/scheduler_algorithm.md
index 28c6c2bc..1053d11e 100755
--- a/contributors/devel/scheduler_algorithm.md
+++ b/contributors/devel/scheduler_algorithm.md
@@ -6,9 +6,9 @@ For each unscheduled Pod, the Kubernetes scheduler tries to find a node across t
The purpose of filtering the nodes is to filter out the nodes that do not meet certain requirements of the Pod. For example, if the free resource on a node (measured by the capacity minus the sum of the resource requests of all the Pods that already run on the node) is less than the Pod's required resource, the node should not be considered in the ranking phase so it is filtered out. Currently, there are several "predicates" implementing different filtering policies, including:
-- `NoDiskConflict`: Evaluate if a pod can fit due to the volumes it requests, and those that are already mounted. Currently supported volumes are: AWS EBS, GCE PD, and Ceph RBD. Only Persistent Volume Claims for those supported types are checked. Persistent Volumes added directly to pods are not evaluated and are not constrained by this policy.
+- `NoDiskConflict`: Evaluate if a pod can fit due to the volumes it requests, and those that are already mounted. Currently supported volumes are: AWS EBS, GCE PD, ISCSI and Ceph RBD. Only Persistent Volume Claims for those supported types are checked. Persistent Volumes added directly to pods are not evaluated and are not constrained by this policy.
- `NoVolumeZoneConflict`: Evaluate if the volumes a pod requests are available on the node, given the Zone restrictions.
-- `PodFitsResources`: Check if the free resource (CPU and Memory) meets the requirement of the Pod. The free resource is measured by the capacity minus the sum of requests of all Pods on the node. To learn more about the resource QoS in Kubernetes, please check [QoS proposal](../design/resource-qos.md).
+- `PodFitsResources`: Check if the free resource (CPU and Memory) meets the requirement of the Pod. The free resource is measured by the capacity minus the sum of requests of all Pods on the node. To learn more about the resource QoS in Kubernetes, please check [QoS proposal](../design-proposals/resource-qos.md).
- `PodFitsHostPorts`: Check if any HostPort required by the Pod is already occupied on the node.
- `HostName`: Filter out all nodes except the one specified in the PodSpec's NodeName field.
- `MatchNodeSelector`: Check if the labels of the node match the labels specified in the Pod's `nodeSelector` field and, as of Kubernetes v1.2, also match the `scheduler.alpha.kubernetes.io/affinity` pod annotation if present. See [here](../user-guide/node-selection/) for more details on both.
diff --git a/contributors/devel/testing.md b/contributors/devel/testing.md
index 45848f3b..e67d81de 100644
--- a/contributors/devel/testing.md
+++ b/contributors/devel/testing.md
@@ -40,7 +40,7 @@ passing, so it is often a good idea to make sure the e2e tests work as well.
* All packages and any significant files require unit tests.
* The preferred method of testing multiple scenarios or input is
[table driven testing](https://github.com/golang/go/wiki/TableDrivenTests)
- - Example: [TestNamespaceAuthorization](../../test/integration/auth/auth_test.go)
+ - Example: [TestNamespaceAuthorization](https://github.com/kubernetes/kubernetes/blob/master/test/integration/auth/auth_test.go)
* Unit tests must pass on OS X and Windows platforms.
- Tests using linux-specific features must be skipped or compiled out.
- Skipped is better, compiled out is required when it won't compile.
@@ -161,9 +161,9 @@ See `go help test` and `go help testflag` for additional info.
- This includes kubectl commands
* The preferred method of testing multiple scenarios or inputs
is [table driven testing](https://github.com/golang/go/wiki/TableDrivenTests)
- - Example: [TestNamespaceAuthorization](../../test/integration/auth/auth_test.go)
+ - Example: [TestNamespaceAuthorization](https://github.com/kubernetes/kubernetes/blob/master/test/integration/auth/auth_test.go)
* Each test should create its own master, httpserver and config.
- - Example: [TestPodUpdateActiveDeadlineSeconds](../../test/integration/pods/pods_test.go)
+ - Example: [TestPodUpdateActiveDeadlineSeconds](https://github.com/kubernetes/kubernetes/blob/master/test/integration/pods/pods_test.go)
* See [coding conventions](coding-conventions.md).
### Install etcd dependency
@@ -180,7 +180,7 @@ hack/install-etcd.sh # Installs in ./third_party/etcd
echo export PATH="\$PATH:$(pwd)/third_party/etcd" >> ~/.profile # Add to PATH
# Option b) install manually
-grep -E "image.*etcd" cluster/saltbase/etcd/etcd.manifest # Find version
+grep -E "image.*etcd" cluster/saltbase/salt/etcd/etcd.manifest # Find version
# Install that version using yum/apt-get/etc
echo export PATH="\$PATH:<LOCATION>" >> ~/.profile # Add to PATH
```
@@ -206,12 +206,12 @@ make test-integration # Run all integration tests.
```
This script runs the golang tests in package
-[`test/integration`](../../test/integration/).
+[`test/integration`](https://github.com/kubernetes/kubernetes/tree/master/test/integration).
### Run a specific integration test
-You can use also use the `KUBE_TEST_ARGS` environment variable with the `hack
-/test-integration.sh` script to run a specific integration test case:
+You can also use the `KUBE_TEST_ARGS` environment variable with the `hack/test-integration.sh`
+script to run a specific integration test case:
```sh
# Run integration test TestPodUpdateActiveDeadlineSeconds with the verbose flag set.
diff --git a/governance.md b/governance.md
index 5f4787f1..8ec5eeb2 100644
--- a/governance.md
+++ b/governance.md
@@ -1,10 +1,12 @@
+This is a Work in Progress, documenting approximately how we have been operating up to this point.
+
# Principles
The Kubernetes community adheres to the following principles:
* Open: Kubernetes is open source. See repository guidelines and CLA, below.
* Welcoming and respectful: See Code of Conduct, below.
* Transparent and accessible: Work and collaboration should be done in public. See SIG governance, below.
-* Merit: Ideas and contributions are accepted according to their technical merit and alignment with project objectives, [scope](kubernetes.io/docs/whatisk8s/), and [design principles](contributors/design-proposals/principles.md).
+* Merit: Ideas and contributions are accepted according to their technical merit and alignment with project objectives, [scope](http://kubernetes.io/docs/whatisk8s/), and [design principles](contributors/design-proposals/principles.md).
# Code of Conduct
@@ -29,31 +31,44 @@ use the [Apache Licence version 2.0](LICENSE). Documentation repositories should
Kubernetes is a large project. It is necessarily a group effort.
There are many ways to participate and contribute.
-We value all forms of constructive contributions, no matter how small, even if not
+We value all forms of constructive contribution, no matter how small, even if not
explicitly described below.
It is intended that contributors have the opportunity to grow in responsibilities,
privileges, and authority corresponding to the scope, quality, quantity, and duration
-of their contributions. Definition of criteria and process is in progress.
+of their contributions. Definition of criteria and process is in progress, with preliminary
+requirements below.
Why would someone want to perform and be accepted into a particular role?
- - To work more efficiently; more permissions reduces development friction
+ - To work more efficiently; more permissions reduce development friction
- Status (have the Kubernetes badge on his/her profile and/or contributions)
- Ownership
- etc...
-Roles that are currently assumed by project participants are described below.
+Roles that are currently assumed by project participants are described below,
+with a focus on the `kubernetes/kubernetes` repo.
+
+## Code and documentation contributors
+
+New community members:
-Code and documentation contributors:
- [**New Contributor**](https://github.com/kubernetes/contrib/issues/1090): a
couple of PRs; should be welcomed to the community, helped with PR workflow, and
directed to relevant documentation
-- **Active Contributor**: at least 3 merged PRs (which could include documentation
+- **Active Contributor**: at least 3 merged and/or assigned PRs (which could include documentation
contributions as well as code), including one in the past month; we have
[expectations](contributors/devel/community-expectations.md)
that frequent contributors will assist in our code-review process and with project
maintenance
-- **Org Member**: an active contributor for at least 3 months; active enough to be useful
+
+Established community members:
+
+Established community members are expected to demonstrate their adherence to the principles in this
+document, familiarity with project organization, roles, policies, procedures, conventions, etc.,
+and technical and/or writing ability. Role-specific expectations, responsibilities, and requirements
+are enumerated below.
+
+- **Member**: an active contributor for at least 3 months; at least 10 merged and/or assigned PRs; active enough to be useful
to assign issues to them and add them to a github team (e.g., for a SIG) for notification
purposes; trusted enough to run tests on their PRs automatically; can issue "@k8s-bot ok to test"
for other contributors; if they choose public membership, they get a badge on their github profile;
@@ -61,72 +76,103 @@ Code and documentation contributors:
project organization, roles, policies, procedures, etc.; should read the [developer
guide](contributors/devel/README.md); must enable
[two-factor authentication](https://help.github.com/articles/about-two-factor-authentication/)
-- **Reviewer**: org member for at least 3 months; familiar enough with some part of the codebase to be in some
+- **Reviewer**: org member for at least 3 months; at least 20 merged and/or assigned PRs, including
+ at least 3 as the primary reviewer; familiar enough with some part of the codebase to be in some
[OWNERS](contributors/devel/owners.md) file as a reviewer (in repos using the bot),
- assigned related PRs, assigned relevant test bugs; expected to be responsive to
+ assigned related PRs, assigned relevant test bugs; responsible for project quality control via
+ [code reviews](contributors/devel/collab.md); expected to be responsive to
review requests as per [community expectations](contributors/devel/community-expectations.md);
can champion incubator repos; must be nominated by an approver for that part of the codebase,
with no objections from other approvers; should be added to
[`kubernetes-reviewers`](https://github.com/orgs/kubernetes/teams/kubernetes-reviewers);
- "read access" to kubernetes repo; get a badge on PR and issue comments
+ "read access" to kubernetes repo; get a badge on PR and issue comments; may be asked to
+ become a reviewer as a precondition for accepting a large code contribution
- **Approver**: in some [OWNERS](contributors/devel/owners.md) file as an approver, which
will be needed to get code merged; previously a reviewer for that part of the
- codebase for at least 3 months; expected to be responsive to review requests as per
+ codebase for at least 3 months; at least 30 merged and/or assigned PRs, including at least 10 as
+ the primary reviewer; expected to be responsive to review requests as per
[community expectations](contributors/devel/community-expectations.md); expected to
mentor contributors and reviewers; demonstrated sound technical judgement; nominated
- by an area/component owner, with no objections from other owners
+ by an area/component owner, with no objections from other owners; may be asked to
+ become an approver as a precondition for accepting a large code contribution
- **Area/Component Owner**: in top-level [OWNERS](contributors/devel/owners.md) file for
some area/component as an approver; design/proposal approval authority for some area
of the project, though escalation is still possible; expected to mentor and guide approvers,
- reviewers, and other contributors
+ reviewers, and other contributors; may be asked to become an area/component owner as a precondition
+ for accepting the contribution of a new component or other major function
- [**kubernetes-maintainers**](https://github.com/orgs/kubernetes/teams/kubernetes-maintainers):
+ approver for some part of the codebase for at least 3 months; on project for at least 1 year;
+ at least 50 merged and/or assigned PRs, including at least 20 as the primary reviewer;
write access to repo (assign issues/PRs, add/remove labels and milestones, edit issues and PRs, edit wiki,
create/delete labels and milestones); technically can lgtm any PR and cause it
- to be merged by the submit queue; expected to review PRs, fix bugs, maintain and
+ to be merged by the submit queue, but expected to respect OWNERS files; expected to review PRs, fix bugs, maintain and
improve health and quality of the project, provide user support, mentor and guide approvers,
- reviewers, and other contributors; approver for some part of the codebase for at least 3
- months; on project for at least 1 year
+ reviewers, and other contributors; must apply to `kubernetes-maintainers@googlegroups.com`, with a
+ [Champion](https://github.com/kubernetes/community/blob/master/incubator.md#faq) from the existing
+ kubernetes-maintainers members and a Sponsor from Project Approvers, with a summary
+ of contributions to the project, current project responsibilities, and links to merged and assigned PRs;
+ at least 3 of the maintainers must approve the application, with no objections; the application
+ expires after 2 weeks if not enough approvals are granted
- **Project Approvers**: approver in [top-level OWNERS file in kubernetes repo](https://github.com/kubernetes/kubernetes/blob/master/OWNERS);
de-facto project decision makers; technically can
- approve virtually any PRs; can sponsor incubator repos
+ approve virtually any PRs; can sponsor incubator repos; can sponsor maintainers;
+ maintainer in good standing for at least 1 year; strong technical vision;
+ committed to project's mission and culture; nomination/application process TBD
- [**API Approver**](https://github.com/orgs/kubernetes/teams/api-approvers):
lead designers of the project, who are familiar with the
design, requirements, mechanics, conventions, style, scope, gotchas, etc.
of the API; most beta/GA API changes are vetted by the API approvers
-
-SIG roles:
+- [**API Reviewer**](https://github.com/orgs/kubernetes/teams/api-reviewers):
+ contributors familiar with design, requirements, mechanics, conventions, style,
+ scope, gotchas, etc. of the API; have written and/or reviewed Kubernetes APIs
+
+## SIG roles
- **SIG Participant**: active in one or more areas of the project; wide
variety of roles are represented
- **SIG Lead**: SIG organizer
-Management roles:
+## Management roles
- **Team Lead**: tech lead or manager of some team at some company working on
K8s; can influence priorities of their team members; pragmatically,
probably want label/assignment powers
- [**kubernetes-pm**](https://github.com/orgs/kubernetes/teams/kubernetes-pm): help to [manage and maintain the project](project-managers/README.md) in
ways other than just writing code (e.g. managing issues); should subscribe to kubernetes-pm@googlegroups.com
-Rotations:
+## Rotations
- [**Build Cop**](contributors/devel/on-call-build-cop.md): ensure tests pass, submit queue is working, rollback PRs,
- manually merge as necessary to fix build
+ manually merge as necessary to fix build; should be members of appropriate repo's build-cops github team
+ (e.g., [kubernetes-build-cops](https://github.com/orgs/kubernetes/teams/kubernetes-build-cops))
- [**User-Support Rotation**](contributors/devel/on-call-user-support.md): answer questions on stackoverflow, googlegroups,
slack, twitter, etc. full time while on duty
-Release roles:
-- **Minor Release Team**: help with minor release
-- **Minor Release Manager**: drive minor release
-- **Patch Release Team**: help with patch release
-- **Patch Release Manager**: drive patch release
+## Release roles
+- The roles of the individuals/team responsible for major, minor, and patch releases is documented
+ [here](https://github.com/kubernetes/community/tree/master/contributors/devel/release). Should be
+ members of the appropriate release-managers github team (e.g.,
+ [kubernetes-release-managers](https://github.com/orgs/kubernetes/teams/kubernetes-release-managers)).
-Duty-specific github roles:
-- [**kubernetes-admin**](https://github.com/orgs/kubernetes/teams/kubernetes-admin): direct code write/merge access; for build cops and
- release czars only.
-- **K8s Org Owner**: can create repos, do ~any github action; the number of
+## Other duty-specific github roles:
+- [**Github Org Owner**](https://github.com/orgs/kubernetes/people?utf8=%E2%9C%93&query=%20role%3Aowner):
+ can create repos, do ~any github action; the number of
owners shouldn't scale with the organization's growth, O(1), and optimally it
- should be less than 10 people who are very familiar with project workings and
+ should be less than 20 people who are very familiar with project workings and
distributed across a few time zones and organizations The other repos will
have distinct sets of people filling some of the above roles, also.
-
+
+## Other repositories
+
+Guidelines for roles in other repositories are TBD. New subprojects/repositories need to be
+able to add reviewers, approvers, and maintainers more rapidly than more mature subprojects.
+Subprojects less than 1 year old will have relaxed time and PR requirements.
+
+## Removal from roles
+
+Most of the above roles require continuous, significant involvement in the project. If someone
+becomes unable or unwilling to continue in their roles, they may retire. If someone doesn't fulfill
+their role for 90 days or violates the code of conduct, they may be removed from the role (escalation/vote
+process TBD). If they wish to resume their role in the future, they may request to return to it by asking
+the current members filling that role.
+
# SIG Governance
In order to standardize Special Interest Group efforts, create maximum transparency, and route contributors to the appropriate SIG, SIGs should follow the guidelines stated below:
@@ -140,6 +186,13 @@ In order to standardize Special Interest Group efforts, create maximum transpare
* Participate in release planning meetings and retrospectives, and burndown meetings, as needed
* Ensure related work happens in a project-owned github org and repository, with code and tests explicitly owned and supported by the SIG, including issue triage, PR reviews, test-failure response, bug fixes, etc.
* Use the above forums as the primary means of working, communicating, and collaborating, as opposed to private emails and meetings
+* Represent the SIG for the PM group:
+ * identify all features in the current release from the SIG
+ * track all features (in the repo with all the fields complete)
+ * attend your SIG meetings
+ * attend the PM group meetings which occur 3-5 times per release
+ * identify the annual roadmap
+ * advise their SIG as needed
# CLA
diff --git a/project-managers/README.md b/project-managers/README.md
index a9418fbe..8879bb70 100644
--- a/project-managers/README.md
+++ b/project-managers/README.md
@@ -17,7 +17,7 @@ They should **NOT**:
* Apply priority labels to PRs
* Apply cherrypick labels to PRs
* Edit text of other people's PRs and issues, including deleting comments
-* Modify anyone else’s release note
+* Modify anyone else's release note
* Create, edit, delete labels
* Create, edit, close, delete milestones
* Create, edit, delete releases
diff --git a/sig-api-machinery/README.md b/sig-api-machinery/README.md
index ec5f8b07..e1a6f147 100644
--- a/sig-api-machinery/README.md
+++ b/sig-api-machinery/README.md
@@ -1,7 +1,7 @@
-Kubernetes Special Interest Group for Authentication and Authorization (SIG API Machinery)
+Kubernetes Special Interest Group for API Machinery (SIG API Machinery)
Links/Info:
-* Github team: https://github.com/orgs/kubernetes/teams/sig-api-machinery (Use ‘@kubernetes/sig-api-machinery’ on github to notify team members.)
+* Github team: https://github.com/orgs/kubernetes/teams/sig-api-machinery-misc (Use ‘@kubernetes/sig-api-machinery-misc’ on github to notify team members.)
* Mailing list: https://groups.google.com/forum/#!forum/kubernetes-sig-api-machinery
* Slack channel: https://kubernetes.slack.com/messages/sig-api-machinery/
* [Agenda/Mission Doc](https://goo.gl/x5nWrF)
diff --git a/sig-apps/README.md b/sig-apps/README.md
index 9d61df32..01e92917 100644
--- a/sig-apps/README.md
+++ b/sig-apps/README.md
@@ -18,7 +18,7 @@ We focus on the developer and devops experience of running applications in Kuber
* Show early features/demos of tools that make running apps easier
## Non-goals:
-* Our job is not to go implement stacks. We’re helping people to help themselves. We will help connect people to the right folks * but we do not want to own a set of examples (as a group)
+* Our job is not to go implement stacks. We're helping people to help themselves. We will help connect people to the right folks * but we do not want to own a set of examples (as a group)
* Do not endorse one particular tool
* Do not pick which apps to run on top of the platform
* Do not recommend one way to do things
diff --git a/sig-apps/minutes/2016-05-18.md b/sig-apps/minutes/2016-05-18.md
index 405588fa..66f9db88 100644
--- a/sig-apps/minutes/2016-05-18.md
+++ b/sig-apps/minutes/2016-05-18.md
@@ -4,7 +4,7 @@
* Intro from Michelle
* Discussion on the future of the SIG:
* Mike from Rackspace offered to do a demo of the recursive functionality ([issue](https://github.com/kubernetes/kubernetes/pull/25110))
- * Idea: solicit the community for cases where their use cases aren’t met.
+ * Idea: solicit the community for cases where their use cases aren't met.
* Demo from Prashanth B on PetSets ([issue](https://github.com/kubernetes/kubernetes/issues/260))
* Supposed to make deploying and managing stateful apps easier. Will be alpha in 1.3.
* Zookeeper, mysql, cassandra are example apps to run in this
diff --git a/sig-apps/minutes/2016-06-08.md b/sig-apps/minutes/2016-06-08.md
index d273d4c6..3c6674cf 100644
--- a/sig-apps/minutes/2016-06-08.md
+++ b/sig-apps/minutes/2016-06-08.md
@@ -2,7 +2,7 @@
- Intro by Michelle Noorali
- Adnan Abdulhussein, Stacksmith lead at Bitnami, did a demo of Stacksmith
- - In the container world, updates to your application’s stack or environment are rolled out by bringing down outdated containers and replacing them with an updated container image. Tools like Docker and Kubernetes make it incredibly easy to do this, however, knowing when your stack is outdated or vulnerable and starting the upgrade process is still a manual step. Stacksmith is a service that aims to solve this by maintaining your base Dockerfiles and proactively keeping them up-to-date and secure. This demo walked through how you can use Stacksmith with your application on GitHub to provide continuous delivery of your application container images.
+ - In the container world, updates to your application's stack or environment are rolled out by bringing down outdated containers and replacing them with an updated container image. Tools like Docker and Kubernetes make it incredibly easy to do this, however, knowing when your stack is outdated or vulnerable and starting the upgrade process is still a manual step. Stacksmith is a service that aims to solve this by maintaining your base Dockerfiles and proactively keeping them up-to-date and secure. This demo walked through how you can use Stacksmith with your application on GitHub to provide continuous delivery of your application container images.
- Adnan is available as @prydonius on the Kubernetes slack as well as on [twitter](https://twitter.com/prydonius) for questions and feedback.
- Feel free to leave feedback on the [Stacksmith](https://stacksmith.bitnami.com/) feedback tab.
- Matt Farina gave an update on the SIG-Apps survey.
diff --git a/sig-aws/README.md b/sig-aws/README.md
index bb0ecf04..fbfe363a 100644
--- a/sig-aws/README.md
+++ b/sig-aws/README.md
@@ -1,10 +1,9 @@
# SIG AWS
A Special Interest Group for maintaining, supporting, and using Kubernetes on AWS.
-All things K8s on AWS. We meet twice a month.
## Meeting:
-- Meetings: First and Third Friday of the month at 9:00AM PST for one hour
+- Meetings: Scheduled via the official [group mailing list](https://groups.google.com/forum/#!forum/kubernetes-sig-aws)
- Zoom Link: [SIG AWS](https://zoom.us/my/k8ssigaws)
## Organizers:
@@ -13,19 +12,7 @@ All things K8s on AWS. We meet twice a month.
| ------------- | ------------- |
| Justin Santa Barbara | [justinsb](https://github.com/justinsb) |
| Chris Love | [chrislovecnm](https://github.com/chrislovecnm) |
-| Kris Childress | [kris-nova](https://github.com/kris-nova) |
+| Kris Nova | [kris-nova](https://github.com/kris-nova) |
| Mackenzie Burnett | [mfburnett](https://github.com/mfburnett) |
-The meeting is open to all and we encourage you to join. Feel free to dial into the zoom call at your convenience.
-
-If you would like to be on the official calendar invite ping one of the organizers.
-
-## Agenda
-
-### October 21st, 2016
-
-- Introductions
-- Future of Kubernetes on AWS: a quick overview of where K8s is going
-- HA Kubernetes on AWS: a quick demo and roundtable
-- Setting our future: what do we want to talk about
-- Kubernetes Blockers on AWS: what is blocking you today on AWS
+The meeting is open to all and we encourage you to join. Feel free to join the zoom call at your convenience. \ No newline at end of file
diff --git a/sig-big-data/README.md b/sig-big-data/README.md
index 3a099918..c7dbe442 100644
--- a/sig-big-data/README.md
+++ b/sig-big-data/README.md
@@ -1,3 +1,26 @@
-# NOTE: THE BIG DATA SIG IS INDEFINITELY SUSPENDED, IN FAVOR OF THE ["APPS" SIG](https://github.com/kubernetes/community/blob/master/sig-apps/README.md).
+# SIG Big Data
-[Old Meeting Notes](https://docs.google.com/document/d/1YhNLN39f5oZ4AHn_g7vBp0LQd7k37azL7FkWG8CEDrE/edit)
+A Special Interest Group for deploying and operating big data applications (Spark, Kafka, Hadoop, Flink, Storm, etc) on Kubernetes. We focus on integrations with big data applications and architecting the best ways to run them on Kubernetes.
+
+## Meeting:
+
+* Meetings: Wednesdays 10:00 AM PST
+* Video Conference Link: updated in [the official group](https://groups.google.com/forum/#!forum/kubernetes-sig-big-data)
+* Check out the [Agenda and Minutes](https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/edit)! Note: this SIG was operational briefly in 2015. Minutes for those meetings are in [their prior location](https://docs.google.com/document/u/1/d/1YhNLN39f5oZ4AHn_g7vBp0LQd7k37azL7FkWG8CEDrE/edit).
+* Slack: https://kubernetes.slack.com/messages/sig-big-data/
+
+## Goals:
+
+* Design and architect ways to run big data applications effectively on Kubernetes
+* Discuss ongoing implementation efforts
+* Discuss resource sharing and multi-tenancy (in the context of big data applications)
+* Suggest Kubernetes features where we see a need
+
+## Non-goals:
+
+* Endorsing any particular tool/framework
+
+## Organizers:
+
+* [Anirudh Ramanathan](https://github.com/foxish), Google
+* [Kenneth Owens](https://github.com/kow3ns), Google
diff --git a/sig-cli/README.md b/sig-cli/README.md
index b4354394..abbce854 100644
--- a/sig-cli/README.md
+++ b/sig-cli/README.md
@@ -14,6 +14,30 @@ We focus on the development and standardization of the CLI [framework](https://g
* Google Group: <https://groups.google.com/forum/#!forum/kubernetes-sig-cli>
## Organizers:
+
+**Note:** Escalate to these folks if you cannot get help from slack or the Google group
+
* Fabiano Franz <ffranz@redhat.com>, Red Hat
+ - slack / github: @fabianofranz
* Phillip Wittrock <pwittroc@google.com>, Google
-* Tony Ado <coolhzb@gmail.com>, Alibaba \ No newline at end of file
+ - slack / github: @pwittrock
+* Tony Ado <coolhzb@gmail.com>, Alibaba
+ - slack / github: @adohe
+
+## Contributing
+
+See [this document](https://github.com/kubernetes/community/blob/master/sig-cli/contributing.md) for contributing instructions.
+
+## Sig-cli teams
+
+Mention one or more of
+
+| Name | Description |
+|------------------------------------|--------------------------------------------------|
+|@kubernetes/sig-cli-bugs | For bugs in kubectl |
+|@kubernetes/sig-cli-feature-requests| For initial discussion of new feature requests |
+|@kubernetes/sig-cli-proposals | For in depth discussion of new feature proposals |
+|@kubernetes/sig-cli-pr-reviews | For PR code reviews |
+|@kubernetes/sig-test-failures | For e2e test flakes |
+|@kubernetes/sig-cli-misc | For general discussion and escalation |
+
diff --git a/sig-cli/contributing.md b/sig-cli/contributing.md
new file mode 100644
index 00000000..a7ce03b2
--- /dev/null
+++ b/sig-cli/contributing.md
@@ -0,0 +1,329 @@
+# Contributing to sig-Cli
+
+This document explains the process for contributing code to the Kubernetes
+repo through the sig-cli community.
+
+## TL;DR
+
+- New contributors should reach out on slack sig-cli - [link](https://github.com/kubernetes/community/tree/master/sig-cli) for an issue to work on that has an approved design
+- New contributors should attend the next sig-cli meeting and introduce themselves
+- Write tests
+
+## Important
+
+The [sig-cli community page](https://github.com/kubernetes/community/tree/master/sig-cli)
+contains the GitHub handles for the sig-cli leads, slack channels,
+email discussion group, and bi-weekly meeting time.
+
+## Message to new contributors
+
+Welcome to the Kubernetes sig-cli contributing guide. We are excited
+about the prospect of you joining our community. Please understand
+that all contributions to Kubernetes require time and commitment from
+the project maintainers to review the ux, software design, and code. We
+recommend that you reach out to one of the sig leads
+and ask the best way to get started. It is likely that there a couple
+issues perfect for a new contributor to get started on.
+
+sig leads can be reached in the sig-cli slack channel or at the sig-cli
+bi-weekly meeting. See the [sig-cli community page](https://github.com/kubernetes/community/tree/master/sig-cli)
+for a link to the slack channel, list of sig leads, and the community meeting time and location.
+
+Example message for asking how to become a contributor:
+
+ Hello, my name is <your name>. I would like to get involved in
+ contributing to the Kubernetes project. What is the best way to
+ get started?
+
+Please understand that mentoring and on boarding new contributors takes
+a lot time and energy from maintainers who have many other
+responsibilities.
+
+## Before you begin
+
+We are so glad you are ready to get started. Please complete the following steps
+to join the community.
+
+- Read the Kubectl user facing documentation to make sure you understand the tool
+ - Complete the [Kubernetes Basics Tutorial](https://kubernetes.io/docs/tutorials/kubernetes-basics/)
+ - Read the concept guides starting with the [Overview](https://kubernetes.io/docs/concepts/tools/kubectl/object-management-overview/)
+ - This is really important so that you have a good understanding of the tool before you start trying to work on it
+- Visit the [sig-cli community page](https://github.com/kubernetes/community/tree/master/sig-cli)
+ - Join the `kubernetes-sig-cli@googlegroups.com` so you get email updates
+ - Join the Kubernetes slack channel `sig-cli`
+- Introduce yourself to the community by sending an email to kubernetes-sig-cli@googlegroups.com and let us know you want to be a contributor
+
+Completing the following steps will make sure you are ready to immediately
+get started once you have been assigned a piece of work. Do these right
+away.
+
+- Setup your development environment so you can build and run Kubernetes. See [this guide for details](https://github.com/kubernetes/community/blob/master/contributors/devel/development.md)
+- Starting taking a look at the code:
+ - `kubernetes/cmd/kubectl`: This is the entry point
+ - `kubernetes/pkg/kubectl`: This contains the implementation
+ - Look at how some of the other commands are implemented
+- Try adding a new command to do something simple:
+ - Add `kubectl hello-world`: print "Hello World"
+ - Add `kubectl hello-kubernetes -f file`: Print "Hello <kind of resource> <name of resource>"
+ - Add `kubectl hello-kubernetes type/name`: Print "Hello <kind of resource> <name of resource> <creation time>"
+
+## Finding an existing issue to work on
+
+The preferred way to own a piece of work is to directly reach out
+to one of the sig-cli leads. This will ensure that the issue is
+high-priority enough that it will receive PR review bandwidth from
+the sig-cli community.
+
+1. In the Kubernetes sig-cli slack channel, mention @pwittrock, @adohe, or @fabianofranz and ask if there are any issues you could pick up
+2. Send an email to the `kubernetes-sig-cli@googlegroups.com`
+
+ Subject: `New Sig-Cli Contributor <Your Name>`
+ Body: Hello, my name is <your name>. I would like to get involved in
+ contributing to the Kubernetes project. I have read all of the
+ user documentation listed on the community contributring page.
+ What should I do next to get started?
+
+3. Attend the sig-cli [bi-weekly meeting](https://github.com/kubernetes/community/tree/master/sig-cli).
+ - Introduce yourself at the beginning of the meeting
+
+## Expectations
+
+If a sig-cli identifies a bug or feature to you to work on,
+they will need your help to ensure that continual progress is
+made and the fix / feature makes it into a Kubernetes release.
+While you are working on the issue, you must leave a weekly update
+including:
+
+1. What has been finished
+2. What is being worked on
+3. What if anything is blocking progress
+
+## Life of a sig-cli bug overview
+
+1. File a GitHub issue
+ - Mention `@kubernetes/sig-cli-bugs`
+ - Describe steps to reproduce the issue including client / server version
+2. Implement the fix
+ - Send a PR
+ - Add `Closes #<Issue Number>` to the description
+ - Mention `@kubernetes/sig-cli-pr-reviews` in a comment
+ - Incorporate review feedback
+ - **Note:** Include unit and e2e tests
+7. Release
+ - Wait for your feature to appear in the next Kubernetes release!
+
+## Life of a sig-cli feature overview
+
+Picking up an issue and implementing it without getting approval for
+the user experience and software design often results in wasted time
+and effort due to decisions such as flag-names, command names, specific
+command behavior.
+
+In order to minimize wasted work and improve communication across efforts,
+the user experience and software design must be agreed upon before
+any PRs are sent for code review.
+
+1. Identify a problem - GitHub issue with basic description
+2. Propose a solution - GitHub design proposal PR
+ - **Action:** Write a design proposal using the [template](contributors/design-proposals/sig-cli/template.md). Once
+ complete, create a pull request and mention `@kubernetes/sig-cli-misc`
+ - Avoid "Work In Progress" PRs (WIP), only send the PR after you
+ consider it complete. To get early feedback, consider messaging the
+ email discussion group.
+ - Expect feedback from 2-3 different sig-cli community members in the form of PR comments
+ - Incorporate feedback from sig-cli community members and comment "PTAL" (please take another look)
+ - Proposal accepted by at least one sig-cli lead
+ - Proposal merged
+3. Communicate what will be done - Discussion at sig-cli bi-weekly meeting - See the [sig-cli community page](https://github.com/kubernetes/community/tree/master/sig-cli) for meeting time and location.
+ - **Note:** This should be done *before* the proposal has been merged, and after an initial round of feedback - e.g. several folks have looked at it and left comments.
+ - **Action:** Add the design proposal as an item to the [bi-weekly meeting agenda](https://docs.google.com/document/d/1r0YElcXt6G5mOWxwZiXgGu_X6he3F--wKwg-9UBc29I/edit)
+ - Make sure the sig-cli community is aware of the work that is being done and can comment on it
+ - Should be included in meeting notes sent to the sig-cli googlegroup
+4. Add feature to Kubernetes release feature repo in [kubernetes/features](https://github.com/kubernetes/features/)
+ - **Action:** *Done by sig-cli lead* - add the feature to the feature tracking repo tag
+ to a release. This is done to ensure release related decisions are
+ properly made and communicated. Such as defining alpha / beta / ga
+ release, proper testing, and documentation are completed.
+5. Implement the code - GitHub implementation PR
+ - **Action:** Implement the solution described in the proposal, then
+ send a PR - **include a link to the design proposal in the description**
+ - Incorporate review feedback
+ - **Note:** Include unit and e2e tests
+6. Update related documentation
+ - [Concept Docs](https://github.com/kubernetes/kubernetes.github.io/tree/master/docs/concepts/tools/kubectl)
+ - [Tasks Docs](https://github.com/kubernetes/kubernetes.github.io/tree/master/docs/tasks/tools/kubectl)
+7. Release
+ - Wait for your feature to appear in the next Kubernetes release!
+
+# New feature development
+
+## Creating a GitHub issue
+
+**Note:** For new contributors, it is recommended that you pick up an
+existing issue with an approved design proposal by reaching out a sig-lead.
+See the [Message to new contributors](#message-to-new-contributors).
+
+In order to start a discussion around a feature, create a new GitHub issue
+in the https://github.com/kubernetes/kubernetes repo and mention `@kubernetes/sig-cli-misc`.
+
+Keep the issue to 2-4 sentence description of the basic description of
+the problem and proposed solution.
+
+**Note:** You must mention `@kubernetes/sig-cli-feature-requests` in the description of the
+issue or in a comment for someone from the sig-cli community to look
+at the issue.
+
+## Creating a design proposal
+
+**Note:** For new contributors, it is recommended that you pick up an
+existing issue with an approved design proposal by reaching out to a sig-lead.
+See the [Message to new contributors](#message-to-new-contributors).
+
+Once at least one sig-lead has agreed that design and code review resources
+can be allocated to tackle a particular issue, the fine details
+of the user experience and design should be agreed upon.
+
+A PR is filed against the https://github.com/kubernetes/community repo.
+This will provide a chance for community members to comment on the
+user experience and software design.
+
+**Note:** This step is important to prevent a lot of code churn and
+thrashing around issues like flag names, command names, etc.
+
+**Note:** It is normal for sig-cli community members to push back on
+feature requests and proposals. sig-cli development and review resources are extremely constrained.
+If your proposal is not high-impact or urgent it may not be accepted
+in favor of more pressing needs. sig-cli community members should
+be free to say: "No, not this release." or "No, not this year." or "No, not right now because although this
+is desirable we need help on these other concrete issues before we can
+tackle this." or "No, this problem should be solved in another way."
+
+Create a design proposal PR under
+https://github.com/kubernetes/community/tree/master/contributors/design-proposals/sig-cli/
+by copying template.md.
+
+Mention `@kubernetes/sig-cli-proposals` on the proposal and link to
+the proposal from the original issue. Expect to get feedback
+from 2-3 community members. At least
+one sig lead must approve the design proposal for it to be accepted.
+
+## Discussing at sig-cli
+
+Finally, when a contributor picks up a design proposals and is ready
+to begin implementing it, we should put it as an item on the next
+sig-cli agenda. This ensures that everyone in sig-cli community
+is aware that work is being started.
+
+## Updating the feature repo
+
+The kubernetes/feature repo exists to help PMs, folks managing the release,
+and sig leads coordinate the work going into a given release. This
+includes items like:
+
+- ensuring all items slated for the release have been completed, or have
+ been disabled and pushed into the next release.
+- support level for items has been properly defined - alpha / beta / ga
+- api changes have been properly vetted by the appropriate parties
+- user facing documentation has been updated
+
+**Note:** in most cases this will be done by sig leads on behalf of the
+other community members.
+
+## Implementing the feature
+
+Contributors can begin implementing a feature before any of the above
+steps have been completed, but should not send a PR until
+the design proposal has been approved / merged.
+
+Go [here](https://github.com/kubernetes/community/blob/master/contributors/devel/development.md)
+for instructions on setting up the Kubernetes development environment.
+
+**Note:** All new features must have automated tests in the same PR.
+Small features and flag changes require only unit/integration tests,
+while larger changes require both unit/integration tests and e2e tests.
+
+If you get stuck or need help, reach out in the Kubernetes slack
+channel and mention @jess and @adohe.
+
+## Updating documentation
+
+Most new features should include user facing documentation. This is
+important to make sure that users are aware of the cool new thing that
+was added. Depending on the contributor and size of the feature, this
+may be done either by the same contributor that implemented the feature,
+or another contributor who is more familiar with the existing docs
+templates.
+
+## Notes on release
+
+Several weeks before a Kubernetes release, development enters a stabilization
+period where no new features are merged. For a feature to be accepted
+into a release, it must be fully merged and tested by this time. If
+your feature is not fully complete - including tests - it will have
+to wait until the next release.
+
+# Escalation
+
+## What to do if your bug issue is stuck
+
+If an issue isn't getting any attention and is unresolved, mention
+@kubernetes/sig-cli-bugs.
+Highlight the severity and urgency of the issue. For severe issues
+escalate by contacting sig leads and attending the bi-weekly sig-cli
+meeting.
+
+## What to do if your feature request issue is stuck
+
+If an issue isn't getting any attention and is unresolved, mention
+@kubernetes/sig-cli-feature-requests.
+If a particular issue has a high impact for you or your business,
+make sure this is clear on the bug, and reach out to the sig leads
+directly. Consider attending the sig meeting to discuss over video
+conference.
+
+## What to do if your PR is stuck
+
+It may happen that your PR seems to be stuck without clear actionable
+feedback for a week or longer. If your PR is based off a design proposal,
+the chances of the PR getting stuck for a long period of time
+are much smaller. However, if it happens do the following:
+
+- If your PR is stuck for a week or more because it has never gotten any
+ comments, mention @kubernetes/sig-cli-pr-reviews and ask for attention.
+ If this is not successful in a day or two, escalate to sig leads or
+ attend the bi-weekly meeting.
+- If your PR is stuck for a week or more after it got comments, but
+ the attention has died down. Mention the reviewer and comment with
+ "PTAL" (please take another look). If you are still not able to
+ get any attention after a couple days, escalate to sig leads by
+ mentioning them.
+
+## What to do if your design docs is stuck
+
+It may happen that your design doc gets stuck without getting merged
+or additional feedback. This may happen for a number of reasons. If you believe that your
+design is important and has been dropped, or it is not moving forward,
+please add it to the sig cli bi-weekly meeting agenda and email
+kubernetes-sig-cli@googlegroups.com notifying the community you would
+like to discuss it at the next meeting.
+
+Merge state meanings:
+
+- Merged:
+ - Ready to be implemented.
+- Unmerged:
+ - Experience and design still being worked out.
+ - Not a high priority issue but may implement in the future: revisit
+ in 6 months.
+ - Unintentionally dropped.
+- Closed:
+ - Not something we plan to implement in the proposed manner.
+ - Not something we plan to revisit in the next 12 months.
+
+## General escalation instructions if you get stuck
+
+See the [sig-cli community page](https://github.com/kubernetes/community/tree/master/sig-cli) for points of contact and meeting times:
+
+- attend the sig-cli bi-weekly meeting
+- message one of the sig leads [on slack](https://kubernetes.slack.com/messages/sig-cli/)
+- send an email to the [kubernetes-sig-cli@googlegroups.com](https://groups.google.com/forum/#!forum/kubernetes-sig-cli)
diff --git a/sig-contribx/README.md b/sig-contribx/README.md
index 6aa88503..9f84bfa6 100644
--- a/sig-contribx/README.md
+++ b/sig-contribx/README.md
@@ -1,13 +1,38 @@
-#SIG Contribx
+#SIG Contributor Experience
+
+We need your help!!!
+
+Developing and sustaining a healthy community of contributors is critical to scaling the project
+and growing the ecosystem. We need to ensure our contributors are happy and productive, we need
+to break the [commit-rate ceiling we hit in July 2015](https://github.com/kubernetes/kubernetes/graphs/contributors),
+and we need to get the monotonically growing PR merge latency and numbers of open PRs and issues under control.
+
##Meeting:
* Meetings: Wednesdays 9:30AM PST (biweekly and changing soon)
* Zoom Link: https://zoom.us/j/7658488911
* Check out the [Agenda and Minutes](https://docs.google.com/document/d/1qf-02B7EOrItQgwXFxgqZ5qjW0mtfu5qkYIF1Hl4ZLI/ )!
##Goals:
-* Improve the experience for contributors working on Kubernetes
-* Help people get involved in the kubernetes community
+* Ensure contributors are happy and productive.
+* Owns project PR and issue automation.
+* Help people get involved in the kubernetes community.
##Organizers:
-* Phillip Wittrock pwittroc@google.com, Google
-* Garrett Rodrigues grod@google.com, Google
+* Garrett Rodrigues grod@google.com, @grodrigues3, Google
+* Elsie Phillips elsie.phillips@coreos.com, @Phillels, CoreOS
+
+##Issues:
+* [Detailed backlog](https://github.com/kubernetes/contrib/projects/1)
+* [velocity-improvement issues](https://github.com/kubernetes/contrib/labels/kind%2Fvelocity-improvement)
+* Too many notifications, which are insufficiently targeted and have too little context, as well as missing
+ notifications for important events
+* [Too-coarse ACLs](https://github.com/kubernetes/contrib/issues/1908)
+* Inadequate PR reviewer assignment and processes, with too small a reviewer pool
+* Lack of a sustainable approach to issue management (we have thousands of open issues, with over 100 new ones each week)
+* Lack of processes for growing leadership within the community
+* Trying to maintain a [large monorepo in github](https://github.com/kubernetes/kubernetes/issues/24343), which is intended for small repos and teams
+* Insufficient project-wide (cross-SIG) communication and coordination
+ * Need to make [slackarchive](http://kubernetes.slackarchive.io/kubernetes-dev) more discoverable
+* [Too many flaky tests](https://github.com/kubernetes/kubernetes/issues/24753)
+* Excessive PR rebases due to checking in generated files
+* Inadequate and out-of-date contributor documentation
diff --git a/sig-creation-procedure.md b/sig-creation-procedure.md
new file mode 100644
index 00000000..897cd1b6
--- /dev/null
+++ b/sig-creation-procedure.md
@@ -0,0 +1,104 @@
+### SIG creation procedure
+
+* Propose the new SIG publicly, including a brief mission statement, by emailing kubernetes-dev@googlegroups.com and kubernetes-users@googlegroups.com, then wait a couple of days for feedback
+* Ask a repo maintainer to create a github label, if one doesn't already exist: sig/foo
+* Request a new [kubernetes.slack.com](http://kubernetes.slack.com) channel (#sig-foo) from @sarahnovotny. New users can join at [slack.kubernetes.io](http://slack.kubernetes.io).
+* Slack activity is archived at [kubernetes.slackarchive.io](http://kubernetes.slackarchive.io). To start archiving a new channel invite the slackarchive bot to the channel via `/invite @slackarchive`
+* Organize video meetings as needed. No need to wait for the [Weekly Community Video Conference](community/README.md) to discuss. Please report summary of SIG activities there.
+ * Request a Zoom account from @sarahnovotny if you expect more than 30 attendees or attendees from China.
+ * Add the meeting to the community meeting calendar by inviting cgnt364vd8s86hr2phapfjc6uk@group.calendar.google.com.
+* Use existing proposal and PR process (to be documented)
+* Announce new SIG on kubernetes-dev@googlegroups.com
+* Submit a PR to add a row for the SIG to the table in the kubernetes/community README.md file, to create a kubernetes/community directory, and to add any SIG-related docs, schedules, roadmaps, etc. to your new kubernetes/community/SIG-foo directory.
+
+####
+**Google Groups creation**
+
+Create Google Groups at [https://groups.google.com/forum/#!creategroup](https://groups.google.com/forum/#!creategroup), following the procedure:
+
+* Each SIG should have one discussion groups, and a number of groups for mirroring relevant github notificaitons;
+* Create groups using the name conventions below;
+* Groups should be created as e-mail lists with at least three owners (including sarahnovotny at google.com and ihor.dvoretskyi at gmail.com);
+* To add the owners, visit the Group Settings (drop-down menu on the right side), select Direct Add Members on the left side and add Sarah and Ihor via email address (with a suitable welcome message); in Members/All Members select Ihor and Sarah and assign to an "owner role";
+* Set "View topics", "Post", "Join the Group" permissions to be "Public"
+
+Name convention:
+
+* kubernetes-sig-foo (the discussion group)
+* kubernetes-sig-foo-misc
+* kubernetes-sig-foo-test-failures
+* kubernetes-sig-foo-bugs
+* kubernetes-sig-foo-feature-requests
+* kubernetes-sig-foo-proposals
+* kubernetes-sig-foo-pr-reviews
+* kubernetes-sig-foo-api-reviews
+
+Example:
+
+* kubernetes-sig-onprem
+* kubernetes-sig-onprem-misc
+* kubernetes-sig-onprem-test-failures
+* kubernetes-sig-onprem-bugs
+* kubernetes-sig-onprem-feature-requests
+* kubernetes-sig-onprem-proposals
+* kubernetes-sig-onprem-pr-reviews
+* kubernetes-sig-onprem-api-reviews
+
+####
+**GitHub users creation**
+
+Create the GitHub users at [https://github.com/join](https://github.com/join), using the name convention below.
+
+As an e-mail address, please, use the Google Group e-mail address of the respective Google Group, created before (i.e. - for user ‘k8s-mirror-foo-misc’ use ‘[kubernetes-sig-foo-misc@googlegroups.com](mailto:kubernetes-sig-foo-misc@googlegroups.com)’). After creating the GitHub users, please, request @idvoretskyi (backup person - @sarahnovotny) to add these users to the Kubernetes organization. If github contacts you about having too many robot accounts, please let us know.
+
+
+Name convention:
+
+* k8s-mirror-foo-misc
+* k8s-mirror-foo-test-failures
+* k8s-mirror-foo-bugs
+* k8s-mirror-foo-feature-requests
+* k8s-mirror-foo-proposals
+* k8s-mirror-foo-pr-reviews
+* k8s-mirror-foo-api-reviews
+
+There is no need for a k8s-mirro-foo user.
+
+Example:
+
+* k8s-mirror-onprem-misc
+* k8s-mirror-onprem-test-failures
+* k8s-mirror-onprem-bugs
+* k8s-mirror-onprem-feature-requests
+* k8s-mirror-onprem-proposals
+* k8s-mirror-onprem-pr-reviews
+* k8s-mirror-onprem-api-reviews
+
+NOTE: We have found that Github's notification autocompletion finds the users before the corresponding teams. This is the reason we recommend naming the users `k8s-mirror-foo-*` instead of `k8s-sig-foo-*`. If you previously created users named `k8s-sig-foo-*`, we recommend you rename them.
+
+####
+**Create the GitHub teams**
+
+Create the GitHub teams at [https://github.com/orgs/kubernetes/new-team](https://github.com/orgs/kubernetes/new-team), using the name convention below. Please, add the GitHub users (created before) to the GitHub teams respectively.
+
+Name convention:
+
+* sig-foo-misc
+* sig-foo-test-failures
+* sig-foo-bugs
+* sig-foo-feature-requests
+* sig-foo-proposals
+* sig-foo-pr-reviews
+* sig-foo-api-reviews
+
+Note that there should not be a sig-foo team. We want to encourage contributors to select the most appropriate team to notify.
+
+Example:
+
+* sig-onprem-misc
+* sig-onprem-test-failures
+* sig-onprem-bugs
+* sig-onprem-feature-requests
+* sig-onprem-proposals
+* sig-onprem-pr-reviews
+* sig-onprem-api-reviews
diff --git a/sig-federation/README.md b/sig-federation/README.md
index 431c9a4a..fec94361 100644
--- a/sig-federation/README.md
+++ b/sig-federation/README.md
@@ -4,12 +4,12 @@ This is a SIG focused on Federation of Kubernetes Clusters ("Ubernetes") and rel
* Hybrid clouds
* Spanning of multiple could providers
* Application migration from private to public clouds (and vice versa)
-* ... and other similar subjects.
+* ... and other similar subjects.
## Meetings:
* Bi-weekly on Mondays @ 9am [America/Los_Angeles](http://time.is/Los_Angeles) (check [the calendar](https://calendar.google.com/calendar/embed?src=cgnt364vd8s86hr2phapfjc6uk%40group.calendar.google.com&ctz=America/Los_Angeles))
* Hangouts link: <https://plus.google.com/hangouts/_/google.com/ubernetes>
-* [Working Group Notes](https://docs.google.com/document/d/1r0YElcXt6G5mOWxwZiXgGu_X6he3F--wKwg-9UBc29I/edit?usp=sharing)
+* [Working Group Notes](https://docs.google.com/document/d/18mk62nOXE_MCSSnb4yJD_8UadtzJrYyJxFwbrgabHe8/edit)
## Communication:
* Slack: <https://kubernetes.slack.com/messages/sig-federation> ([archive](http://kubernetes.slackarchive.io/sig-federation))
diff --git a/sig-network/README.md b/sig-network/README.md
index 3ef3658d..3c4ff612 100644
--- a/sig-network/README.md
+++ b/sig-network/README.md
@@ -1 +1,43 @@
-[This wiki has moved](https://github.com/kubernetes/community/wiki/SIG-Network)
+# SIG Network
+
+A Special Interest Group for networking in Kubernetes.
+
+## Meetings
+
+* Meetings: Every other Thursday @ 14:00 Pacific time
+* [SIG Network Calendar](https://calendar.google.com/calendar/embed?src=ODhmZTFsM3FmbjJiNnIxMWs4dW01YW03NmNAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ)
+* Zoom Link: [https://zoom.us/j/5806599998](https://zoom.us/j/5806599998)
+* [Agenda and Minutes](https://docs.google.com/document/d/1_w77-zG_Xj0zYvEMfQZTQ-wPP4kXkpGD8smVtW_qqWM/edit)
+* Recorded meetings are on the [SIG Network YouTube channel](https://www.youtube.com/playlist?list=PL69nYSiGNLP2E8vmnqo5MwPOY25sDWIxb)
+
+## Google Group / Mailing List
+
+Join the [SIG network mailing list](https://groups.google.com/forum/#!forum/kubernetes-sig-network)
+
+Most discussion happens on the Google Group and the weekly meetings help more quickly resolve issues
+that come up in PRs or on the group.
+
+## Areas of Responsibility
+
+SIG Network is responsible for the following Kubernetes subsystems:
+
+- DNS
+- Ingress
+- Network plugins / CNI
+- Network Policy
+- Services / kube-proxy
+
+SIG Network is responsible for a number of issues and PRs. A summary can be found through GitHub search:
+
+* [SIG Network PRs](https://github.com/issues?utf8=%E2%9C%93&q=team%3Akubernetes%2Fsig-network+is%3Aopen+is%3Apr+)
+* [SIG Network Issues](https://github.com/issues?utf8=%E2%9C%93&q=team%3A%22kubernetes%2Fsig-network%22+is%3Aopen+is%3Aissue)
+
+## Documents
+
+* [2017 Planning](https://docs.google.com/document/d/1fBxC36UCBnqY_w3m3TjdnXFsIT--GS6HmKb5o0nhkTk/edit#)
+
+## Organizers:
+
+* Casey Davenport <casey@tigera.io>, Tigera
+* Dan Williams <dcbw@redhat.com>, Red Hat
+* Tim Hockin <thockin@google.com>, Google
diff --git a/sig-on-prem/README.md b/sig-on-prem/README.md
new file mode 100644
index 00000000..3faaff1f
--- /dev/null
+++ b/sig-on-prem/README.md
@@ -0,0 +1,21 @@
+# SIG On-Prem
+
+This is the wiki page of the Kubernetes SIG On-Prem: a special interest group
+that brings together member of Kubernetes community interested in running
+Kubernetes on premise, on bare metal or more generally beyond cloud
+providers.
+
+We use **sig-onprem** label to track on premise efforts in PRs and issues:
+
+[Issues](https://github.com/kubernetes/kubernetes/issues?utf8=%E2%9C%93&q=is%3Aopen%20label%3Asig%2Fonprem%20is%3Aissue%20)
+[Pull Requests](https://github.com/kubernetes/kubernetes/pulls?q=is%3Aopen%20is%3Apr%20label%3Asig%2Fonprem)
+
+**Leads:** Joseph Jacks (@josephjacks) and Tomasz Napierala (@zen)
+
+**Slack Channel:** [#sig-onprem](https://kubernetes.slack.com/messages/sig-onprem/). [Archive](http://kubernetes.slackarchive.io/sig-onprem/)
+
+**Mailing List:** [kubernetes-sig-on-prem](https://groups.google.com/forum/#!forum/kubernetes-sig-on-prem)
+
+**Effort tracking document** [On-Prem related projects](https://docs.google.com/spreadsheets/d/1Ca9ZpGXM4PfycYv0Foi7Y4vmN4KVXrGYcJipbH8_xLY/edit#gid=0)
+
+**Meetings:** Every second Wednesday at 0800 PST (11 AM ET / 5 PM CET) - [Connect using Zoom](https://zoom.us/my/k8s.sig.onprem), [Agenda/Notes](https://docs.google.com/document/d/1AHF1a8ni7iMOpUgDMcPKrLQCML5EMZUAwP4rro3P6sk/edit#)
diff --git a/sig-openstack/README.md b/sig-openstack/README.md
index 973f2bfa..c68c27ce 100644
--- a/sig-openstack/README.md
+++ b/sig-openstack/README.md
@@ -13,5 +13,9 @@ are tagged with the **sig-openstack** label.
**Mailing List:** [kubernetes-sig-openstack](https://groups.google.com/forum/#!forum/kubernetes-sig-openstack)
-**Meetings:** Every second Wednesday at 2100 UTC (2 PM PDT / 5 PM EDT) - [Connect using Zoom](https://zoom.us/j/417251241), [Agenda/Notes](https://docs.google.com/document/d/1iAQ3LSF_Ky6uZdFtEZPD_8i6HXeFxIeW4XtGcUJtPyU/edit#
+**Meetings:** Meetings are held every second Wednesday. The meetings occur at
+1500 UTC or 2100 UTC, alternating. To check which time is being used for the
+upcoming meeting refer to the [Agenda/Notes](https://docs.google.com/document/d/1iAQ3LSF_Ky6uZdFtEZPD_8i6HXeFxIeW4XtGcUJtPyU/edit#).
+Meeting reminders are also sent to the mailing list linked above. Meetings are
+held on [Zoom](https://zoom.us) in the room at [https://zoom.us/j/417251241](https://zoom.us/j/417251241).
)
diff --git a/sig-scheduling/README.md b/sig-scheduling/README.md
index efa2c8e6..9d65ba96 100644
--- a/sig-scheduling/README.md
+++ b/sig-scheduling/README.md
@@ -1,7 +1,5 @@
# sig-scheduling
-[Google group](https://groups.google.com/forum/#!forum/kubernetes-sig-scheduling)
-
## Organizers
- [David Oppenheimer](https://github.com/davidopp) / Google
@@ -14,4 +12,9 @@
for joining the video conference. _NOTE: Since meetings are occasionally
canceled due to lack of content, if you have topics to discuss please also
write a message to the list._
-- [Meeting link](https://zoom.us/zoomconference?m=rN2RrBUYxXgXY4EMiWWgQP6Vslgcsn86) \ No newline at end of file
+- [Meeting link](https://zoom.us/zoomconference?m=rN2RrBUYxXgXY4EMiWWgQP6Vslgcsn86)
+
+## Communication
+
+- [Slack](https://kubernetes.slack.com/messages/sig-scheduling) ([archive](http://kubernetes.slackarchive.io/sig-scheduling))
+- [Google Group](https://groups.google.com/forum/#!forum/kubernetes-sig-scheduling)
diff --git a/sig-service-catalog/README.md b/sig-service-catalog/README.md
index 88bd89aa..0fe2c188 100644
--- a/sig-service-catalog/README.md
+++ b/sig-service-catalog/README.md
@@ -1,6 +1,6 @@
# SIG Service Catalog
-**Leads:** Paul Morie (@pmorie, Red Hat), Chris Dutra (@dutronlabs, Apprenda)
+**Leads:** Paul Morie (@pmorie, Red Hat), Aaron Schlesinger (@arschles, Deis), Brendan Melville (@bmelville, Google), Doug Davis (@duglin, IBM)
**Slack Channel:** [#sig-service-catalog](https://kubernetes.slack.com/messages/sig-service-catalog/). [Archive](http://kubernetes.slackarchive.io/sig-service-catalog/)
diff --git a/sig-storage/1.3-retrospective/README.md b/sig-storage/1.3-retrospective/README.md
index 62546b1f..52e28490 100644
--- a/sig-storage/1.3-retrospective/README.md
+++ b/sig-storage/1.3-retrospective/README.md
@@ -25,7 +25,7 @@ A characteristic list of issues (as not all of them were well captured in GitHub
5. External object bleeding. Much of the logic was centered on a state machine that lived in the kubelet. Other kube components had to be aware of the state machine and other aspects of the binding framework to use Volumes.
6. Maintenance was difficult as this work was implemented in three different controllers that spread the logic for provisioning, binding, and recycling Volumes.
7. Kubelet failures on the Node could “strand” storage. Requiring users to manually unmount storage.
-8. A pod’s long running detach routine could impact other pods as the operations run synchronously in the kubelet sync loop.
+8. A pod's long running detach routine could impact other pods as the operations run synchronously in the kubelet sync loop.
9. Nodes required elevated privileges to be able to trigger attach/detach. Ideally attach/detach should be triggered from master which is considered more secure (see Issue [#12399](https://github.com/kubernetes/kubernetes/issues/12399)).
Below are the Github Issues that were filed for this area:
@@ -41,7 +41,7 @@ Below are the Github Issues that were filed for this area:
## How Did We Solve the Problem?
Addressing these issues was the main deliverable for storage in 1.3. This required an in depth rewrite of several components.
-Early in the 1.3 development cycle (March 28 to April 1, 2016) several community members in the Storage SIG met at a week long face-to-face summit at Google’s office in Mountain View to address these issues. A plan was established to approach the attach/detach/mount/unmount issues as a deliberate effort with contributors already handling the design. Since that work was already in flight and a plan established, the majority of the summit was devoted to resolving the PV/PVC controller issues. Meeting notes were captured [in this document](https://github.com/kubernetes/community/blob/master/sig-storage/1.3-retrospective/2016-03-28_Storage-SIG-F2F_Notes.pdf).
+Early in the 1.3 development cycle (March 28 to April 1, 2016) several community members in the Storage SIG met at a week long face-to-face summit at Google's office in Mountain View to address these issues. A plan was established to approach the attach/detach/mount/unmount issues as a deliberate effort with contributors already handling the design. Since that work was already in flight and a plan established, the majority of the summit was devoted to resolving the PV/PVC controller issues. Meeting notes were captured [in this document](https://github.com/kubernetes/community/blob/master/sig-storage/1.3-retrospective/2016-03-28_Storage-SIG-F2F_Notes.pdf).
Three projects were planned to fix the issues outlined above:
* PV/PVC Controller Redesign (a.k.a. Provisioner/Binder/Recycler controller)
@@ -64,7 +64,7 @@ The Kubelet Volume Redesign involved changing fundamental assumptions of data fl
1. **Release delay**
* The large amount of churn so late in the release with little stabilization time resulted in the delay of the release by one week: The Kubernetes 1.3 release [was targeted](https://github.com/kubernetes/features/blob/master/release-1.3/release-1.3.md) for June 20 to June 24, 2016. It ended up [going out on July 1, 2016](https://github.com/kubernetes/kubernetes/releases/tag/v1.3.0). This was mostly due to the time to resolve a data corruption issue on ungracefully terminated pods caused by detaching of mounted volumes ([#27691](https://github.com/kubernetes/kubernetes/issues/27691)). A large number of the bugs introduced in the release were fixed in the 1.3.4 release which [was cut on August 1, 2016](https://github.com/kubernetes/kubernetes/releases/tag/v1.3.4).
-2. **Instability in 1.3’s Storage stack**
+2. **Instability in 1.3's Storage stack**
* The Kubelet volume redesign shipped in 1.3.0 with several bugs. These were mostly due to unexpected interactions between the new functionality and other Kubernetes components. For example, secrets were handled serially not in parallel, namespace dependencies were not well understood, etc. Most of these issues were quickly identified and addressed but waited for 1.3 patch releases.
* Issues related to this include:
* PVC Volume will not detach if PVC or PV is deleted before pod ([#29051](https://github.com/kubernetes/kubernetes/issues/29051))
@@ -93,4 +93,4 @@ The value of the feature freeze date is to ensure the release has time to stabil
2. Establish a formal exception process for merging large changes after feature complete dates.
* Status: [Drafted as of 1.4](https://github.com/kubernetes/features/blob/master/EXCEPTIONS.md)
-Kubernetes is an incredibly fast moving project, with hundreds of active contributors creating a solution that thousands of organization rely on. Stability, trust, and openness are paramount in both the product and the community around Kubernetes. We undertook this retrospective effort to learn from the 1.3 release’s shipping delay. These action items and other work in the upcoming releases are part of our commitment to continually improve our project, our community, and our ability to deliver production-grade infrastructure platform software.
+Kubernetes is an incredibly fast moving project, with hundreds of active contributors creating a solution that thousands of organization rely on. Stability, trust, and openness are paramount in both the product and the community around Kubernetes. We undertook this retrospective effort to learn from the 1.3 release's shipping delay. These action items and other work in the upcoming releases are part of our commitment to continually improve our project, our community, and our ability to deliver production-grade infrastructure platform software.
diff --git a/sig-storage/README.md b/sig-storage/README.md
index 6bb5f375..85fb704b 100644
--- a/sig-storage/README.md
+++ b/sig-storage/README.md
@@ -13,10 +13,29 @@ Interested in contributing to storage features in Kubernetes? [Please read our g
### Links
* Public Slack Channel: https://kubernetes.slack.com/messages/sig-storage/details/
* Get invite to join here: http://slack.k8s.io/
-* Google Group: https://groups.google.com/forum/#!forum/kubernetes-sig-storage
-* Github team: https://github.com/orgs/kubernetes/teams/sig-storage
-* Github issues: [link](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Aarea%2Fstorage)
+* Google Groups
+ * General storage SIG related discussions.
+ * [kubernetes-sig-storage](https://groups.google.com/forum/#!forum/kubernetes-sig-storage)
+ * Archive of Github team notifications:
+ * [kubernetes-sig-storage-api-reviews](https://groups.google.com/forum/#!forum/kubernetes-sig-storage-api-reviews)
+ * [kubernetes-sig-storage-bugs](https://groups.google.com/forum/#!forum/kubernetes-sig-storage-bugs)
+ * [kubernetes-sig-storage-feature-requests](https://groups.google.com/forum/#!forum/kubernetes-sig-storage-feature-requests)
+ * [kubernetes-sig-storage-misc](https://groups.google.com/forum/#!forum/kubernetes-sig-storage-misc)
+ * [kubernetes-sig-storage-pr-reviews](https://groups.google.com/forum/#!forum/kubernetes-sig-storage-pr-reviews)
+ * [kubernetes-sig-storage-proposals](https://groups.google.com/forum/#!forum/kubernetes-sig-storage-proposals)
+ * [kubernetes-sig-storage-test-failures](https://groups.google.com/forum/#!forum/kubernetes-sig-storage-test-failures)
+* Github Teams - These are the teams that should be mentioned on Github PRs and Issues:
+ * [kubernetes-sig-storage-api-reviews](https://groups.google.com/forum/#!forum/kubernetes-sig-storage-api-reviews)
+ * [kubernetes-sig-storage-bugs](https://groups.google.com/forum/#!forum/kubernetes-sig-storage-bugs)
+ * [kubernetes-sig-storage-feature-requests](https://groups.google.com/forum/#!forum/kubernetes-sig-storage-feature-requests)
+ * [kubernetes-sig-storage-misc](https://groups.google.com/forum/#!forum/kubernetes-sig-storage-misc)
+ * [kubernetes-sig-storage-pr-reviews](https://groups.google.com/forum/#!forum/kubernetes-sig-storage-pr-reviews)
+ * [kubernetes-sig-storage-proposals](https://groups.google.com/forum/#!forum/kubernetes-sig-storage-proposals)
+ * [kubernetes-sig-storage-test-failures](https://groups.google.com/forum/#!forum/kubernetes-sig-storage-test-failures)
+* Github Issues
+ * [link](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Fstorage)
* Documentation for currently supported volume plugins: http://kubernetes.io/docs/user-guide/volumes/
-* Code for Volume plugins can be found [here](https://github.com/kubernetes/kubernetes/tree/master/pkg/volume).
-* Code for volume controllers can be found [here](https://github.com/kubernetes/kubernetes/tree/master/pkg/controller/volume/).
-* Code for Kubelet volume manager can be found [here](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/volumemanager/).
+* Code
+ * [Volume plugins](https://github.com/kubernetes/kubernetes/tree/master/pkg/volume)
+ * [Volume controllers](https://github.com/kubernetes/kubernetes/tree/master/pkg/controller/volume/)
+ * [Kubelet volume manager](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/volumemanager/)
diff --git a/sig-storage/contributing.md b/sig-storage/contributing.md
index c22d9aff..141cb91c 100644
--- a/sig-storage/contributing.md
+++ b/sig-storage/contributing.md
@@ -25,7 +25,7 @@ what you are proposing. We are really trying to improve our test coverage and do
and documentation in your implementation PR.
### Helping with Issues
-A great way to get involved is to pick an issue and help address it. We would love help here. Storage related issues are [listed here](https://github.com/kubernetes/kubernetes/labels/area%2Fstorage)
+A great way to get involved is to pick an issue and help address it. We would love help here. Storage related issues are [listed here](https://github.com/kubernetes/kubernetes/labels/sig%2Fstorage)
### Adding support for a new storage platform in Kubernetes
For folks looking to add support for a new storage platform in Kubernetes, you have several options:
diff --git a/sig-testing/README.md b/sig-testing/README.md
index fbe33994..8110d87e 100644
--- a/sig-testing/README.md
+++ b/sig-testing/README.md
@@ -1,6 +1,6 @@
# sig-testing
-The Kubernetes Testing SIG (sig-testing) is a working group within the Kubernetes contributor community interested in how we can most effectively test Kubernetes. We’re interested specifically in making it easier for the community to run tests and contribute test results, to ensure Kubernetes is stable across a variety of cluster configurations and cloud providers.
+The Kubernetes Testing SIG (sig-testing) is a working group within the Kubernetes contributor community interested in how we can most effectively test Kubernetes. We're interested specifically in making it easier for the community to run tests and contribute test results, to ensure Kubernetes is stable across a variety of cluster configurations and cloud providers.
## video conference
@@ -10,7 +10,7 @@ We meet weekly on Tuesdays at 9:30am PDT (16:30 UTC) at [this zoom room](https:/
We use [a public google doc](https://docs.google.com/document/d/1z8MQpr_jTwhmjLMUaqQyBk1EYG_Y_3D4y4YdMJ7V1Kk) to track proposed agenda items, as well as take notes during meetings.
-The agenda is open for comment. Please contact the organizers listed below if you’d like to propose a topic. Typically in the absence of anything formal we poll attendees for topics, and discuss tactical work.
+The agenda is open for comment. Please contact the organizers listed below if you'd like to propose a topic. Typically in the absence of anything formal we poll attendees for topics, and discuss tactical work.
## slack
@@ -22,7 +22,7 @@ Signup for access at http://slack.kubernetes.io/
- [our github team: @kubernetes/sig-testing](https://github.com/orgs/kubernetes/teams/sig-testing)
- [issues mentioning @kubernetes/sig-testing](https://github.com/issues?q=is%3Aopen+team%3Akubernetes%2Fsig-testing)
-We use the @kubernetes/sig-testing team to notify SIG members of particular issues or PR’s of interest. If you would like to be added to this team, please contact the organizers listed below.
+We use the @kubernetes/sig-testing team to notify SIG members of particular issues or PR's of interest. If you would like to be added to this team, please contact the organizers listed below.
## google group
diff --git a/sig-ui/README.md b/sig-ui/README.md
index b112de7a..9341df09 100644
--- a/sig-ui/README.md
+++ b/sig-ui/README.md
@@ -5,7 +5,7 @@ Efforts are centered around [Kubernetes Dashboard](https://github.com/kubernetes
####Leads:
-* Piotr Bryk (@bryk, _Google_) and Dan Romlein (@romlein, _Apprenda_)
+* Piotr Bryk (@bryk, _Google_) and Dan Romlein (@danielromlein, _Apprenda_)
####Meetings:
* Wednesdays 4PM CEST
diff --git a/sig-windows/README.md b/sig-windows/README.md
index 36b460de..0291e2a5 100644
--- a/sig-windows/README.md
+++ b/sig-windows/README.md
@@ -3,9 +3,9 @@
A special interest group for bringing Kubernetes support to Windows.
## Meeting
-* Bi-weekly: Tuesday 1:00 PM EST (10:00 AM PST)
+* Bi-weekly: Tuesday 12:30 PM EST (9:30 AM PST)
* Zoom link: [https://zoom.us/my/sigwindows](https://zoom.us/my/sigwindows)
+* Recorded Meetings Playlist on Youtube: https://www.youtube.com/playlist?list=PL69nYSiGNLP2OH9InCcNkWNu2bl-gmIU4&jct=LZ9EIvD4DGrhr2h4r0ItaBmco7gTgw
* Meeting Notes: https://docs.google.com/document/d/1Tjxzjjuy4SQsFSUVXZbvqVb64hjNAG5CQX8bK7Yda9w/edit#heading=h.kbz22d1yc431
-
-The meeting agenda and notes can be found [here](https://docs.google.com/document/d/1Tjxzjjuy4SQsFSUVXZbvqVb64hjNAG5CQX8bK7Yda9w/edit)
+* Find us on Slack at https://kubernetes.slack.com/messages/sig-windows