Merge pull request #2275 from cheftako/ccm-build

Fixing cloud provider KEP.

2 files changed, 530 insertions, 5 deletions

diff --git a/keps/sig-cloud-provider/0002-cloud-controller-manager.md b/keps/sig-cloud-provider/0002-cloud-controller-manager.md
index c3ce25c2..8545d962 100644
--- a/keps/sig-cloud-provider/0002-cloud-controller-manager.md
+++ b/keps/sig-cloud-provider/0002-cloud-controller-manager.md
@@ -55,6 +55,27 @@ replaces:
          - [Process Goals](#process-goals)
    - [Implementation History](#implementation-history)
    - [Alternatives](#alternatives)
+   
+## Terms
+
+- **CCM**: Cloud Controller Manager - The controller manager responsible for running cloud provider dependent logic, 
+such as the service and route controllers.
+- **KCM**: Kubernetes Controller Manager - The controller manager responsible for running generic Kubernetes logic, 
+such as job and node_lifecycle controllers.
+- **KAS**: Kubernetes API Server - The core api server responsible for handling all API requests for the Kubernetes 
+control plane. This includes things like namespace, node, pod and job resources.
+- **K8s/K8s**: The core kubernetes github repository.
+- **K8s/cloud-provider**: Any or all of the repos for each cloud provider. Examples include [cloud-provider-gcp](https://github.com/kubernetes/cloud-provider-gcp), 
+[cloud-provider-aws](https://github.com/kubernetes/cloud-provider-aws) and [cloud-provider-azure](https://github.com/kubernetes/cloud-provider-azure). 
+We have created these repos for each of the in-tree cloud providers. This document assumes in various places that the 
+cloud providers will place the relevant code in these repos. Whether this is a long-term solution to which additional 
+cloud providers will be added, or an incremental step toward moving out of the Kubernetes org is out of scope of this 
+document, and merits discussion in a broader forum and input from SIG-Architecture and Steering Committee. 
+- **K8s SIGs/library**: Any SIG owned repository. 
+- **Staging**: Staging: Separate repositories which are currently visible under the K8s/K8s repo, which contain code
+considered to be safe to be vendored outside of the K8s/K8s repo and which should eventually be fully separated from 
+the K8s/K8s repo. Contents of Staging are prevented from depending on code in K8s/K8s which are not in Staging. 
+Controlled by [publishing kubernetes-rules-configmap](https://github.com/kubernetes/publishing-bot/blob/master/configs/kubernetes-rules-configmap.yaml)
 
 ## Summary
 
@@ -154,7 +175,8 @@ easily, therefore, this will be addressed with Flex Volumes (Discussed under a s
 
 The kube-controller-manager has many controller loops. [See NewControllerInitializers](https://github.com/kubernetes/kubernetes/blob/release-1.9/cmd/kube-controller-manager/app/controllermanager.go#L332)
 
- - [nodeController](https://github.com/kubernetes/kubernetes/tree/release-1.9/pkg/controller/node)
+ - [nodeIpamController](https://github.com/kubernetes/kubernetes/tree/release-1.10/pkg/controller/nodeipam)
+ - [nodeLifecycleController](https://github.com/kubernetes/kubernetes/tree/release-1.10/pkg/controller/nodelifecycle)
  - [volumeController](https://github.com/kubernetes/kubernetes/tree/release-1.9/pkg/controller/volume)
  - [routeController](https://github.com/kubernetes/kubernetes/tree/release-1.9/pkg/controller/route)
  - [serviceController](https://github.com/kubernetes/kubernetes/tree/release-1.9/pkg/controller/service)
@@ -167,14 +189,23 @@ The kube-controller-manager has many controller loops. [See NewControllerInitial
 
 Among these controller loops, the following are cloud provider dependent.
 
- - [nodeController](https://github.com/kubernetes/kubernetes/tree/release-1.9/pkg/controller/node)
+ - [nodeIpamController](https://github.com/kubernetes/kubernetes/tree/release-1.10/pkg/controller/nodeipam)
+ - [nodeLifecycleController](https://github.com/kubernetes/kubernetes/tree/release-1.10/pkg/controller/nodelifecycle)
  - [volumeController](https://github.com/kubernetes/kubernetes/tree/release-1.9/pkg/controller/volume)
  - [routeController](https://github.com/kubernetes/kubernetes/tree/release-1.9/pkg/controller/route)
  - [serviceController](https://github.com/kubernetes/kubernetes/tree/release-1.9/pkg/controller/service)
 
-The nodeController uses the cloudprovider to check if a node has been deleted from the cloud. If cloud provider reports
-a node as deleted, then this controller immediately deletes the node from kubernetes. This check removes the need to
-wait for a specific amount of time to conclude that an inactive node is actually dead.
+The nodeIpamController uses the cloudprovider to handle cloud specific CIDR assignment of a node. Currently the only
+cloud provider using this functionality is GCE. So the current plan is to break this functionality out of the common 
+verion of the nodeIpamController. Most cloud providers can just run the default version of this controller. However any
+cloud provider which needs cloud specific version of this functionality and disable the default version running in the 
+KCM and run their own version in the CCM.
+
+The nodeLifecycleController uses the cloudprovider to check if a node has been deleted from/exists in the cloud. 
+If cloud provider reports a node as deleted, then this controller immediately deletes the node from kubernetes. 
+This check removes the need to wait for a specific amount of time to conclude that an inactive node is actually dead.
+The current plan is to move this functionality into its own controller, allowing the nodeIpamController to remain in
+K8s/K8s and the Kube Controller Manager.
 
 The volumeController uses the cloudprovider to create, delete, attach and detach volumes to nodes. For instance, the
 logic for provisioning, attaching, and detaching a EBS volume resides in the AWS cloudprovider. The volumeController
diff --git a/keps/sig-cloud-provider/0013-build-deploy-ccm.md b/keps/sig-cloud-provider/0013-build-deploy-ccm.md
new file mode 100644
index 00000000..ff0dd415
--- /dev/null
+++ b/keps/sig-cloud-provider/0013-build-deploy-ccm.md
@@ -0,0 +1,494 @@
+---
+kep-number: 13
+title: Switching To Cloud Provider Repo And Builds
+authors:
+  - "@cheftako"
+  - "@calebamiles"
+  - "@nckturner"
+owning-sig: sig-apimachinery
+participating-sigs:
+  - sig-apps
+  - sig-aws
+  - sig-azure
+  - sig-cloud-provider
+  - sig-gcp
+  - sig-network
+  - sig-openstack
+  - sig-storage
+reviewers:
+  - "@andrewsykim"
+  - "@calebamiles"
+  - "@nckturner"
+approvers:
+  - "@thockin"
+editor: TBD
+status: provisional
+---
+
+# Switching To Cloud Provider Repo And Builds 
+
+## How To Remove Cloud Provider Code From Kubernetes Core
+
+## Table of Contents
+
+- [Switching To Cloud Provider Repo And Builds](#switching-to-cloud-provider-repo-and-builds)
+   - [Table of Contents](#table-of-contents)
+   - [Terms](#terms)
+   - [Summary](#summary)
+   - [Motivation](#motivation)
+      - [Goals](#goals)
+      - [Intermediary Goals](#intermediary-goals)
+      - [Non-Goals](#non-goals)
+   - [Proposal](#proposal)
+   - [Building CCM For Cloud Providers](#building-ccm-for-cloud-providers)
+      - [Background](#background)
+      - [Design Options](#design-options)
+          - [Staging](#staging)
+          - [Cloud Provider Instances](#cloud-provider-instances)
+          - [Build Targets](#build-targets)
+          - [K8s/K8s Releases and K8s/CP Releases](#k8s/k8s-releases-and-k8s/cp-releases)
+          - [Migrating to a CCM Build](#migrating-to-a-ccm-build)
+          - [Flags, Service Accounts, etc](#flags,-service-accounts,-etc)
+          - [Deployment Scripts](#deployment-scripts)
+          - [kubeadm](#kubeadm)
+          - [CI, TestGrid and Other Testing Issues](#ci,-testgrid-and-other-testing-issues)
+   - [Alternatives](#alternatives)
+      - [Staging Alternatives](#staging-alternatives)
+          - [Git Filter-Branch](#Git Filter-Branch)
+      - [Build Location Alternatives](#Build Location Alternatives)
+          - [Build K8s/K8s from within K8s/Cloud-provider](#build-k8s/k8s-from-within-k8s/cloud-provider)
+          - [Build K8s/Cloud-provider within K8s/K8s](#build-k8s/cloud-provider-within-k8s/k8s)
+      - [Config Alternatives](#config-alternatives)
+          - [Use component config to determine where controllers run](#use-component-config-to-determine-where-controllers-run)
+
+## Terms
+
+- **CCM**: Cloud Controller Manager - The controller manager responsible for running cloud provider dependent logic, 
+such as the service and route controllers.
+- **KCM**: Kubernetes Controller Manager - The controller manager responsible for running generic Kubernetes logic, 
+such as job and node_lifecycle controllers.
+- **KAS**: Kubernetes API Server - The core api server responsible for handling all API requests for the Kubernetes 
+control plane. This includes things like namespace, node, pod and job resources.
+- **K8s/K8s**: The core kubernetes github repository.
+- **K8s/cloud-provider**: Any or all of the repos for each cloud provider. Examples include [cloud-provider-gcp](https://github.com/kubernetes/cloud-provider-gcp), 
+[cloud-provider-aws](https://github.com/kubernetes/cloud-provider-aws) and [cloud-provider-azure](https://github.com/kubernetes/cloud-provider-azure). 
+We have created these repos for each of the in-tree cloud providers. This document assumes in various places that the 
+cloud providers will place the relevant code in these repos. Whether this is a long-term solution to which additional 
+cloud providers will be added, or an incremental step toward moving out of the Kubernetes org is out of scope of this 
+document, and merits discussion in a broader forum and input from SIG-Architecture and Steering Committee. 
+- **K8s SIGs/library**: Any SIG owned repository. 
+- **Staging**: Staging: Separate repositories which are currently visible under the K8s/K8s repo, which contain code
+considered to be safe to be vendored outside of the K8s/K8s repo and which should eventually be fully separated from 
+the K8s/K8s repo. Contents of Staging are prevented from depending on code in K8s/K8s which are not in Staging. 
+Controlled by [publishing kubernetes-rules-configmap](https://github.com/kubernetes/publishing-bot/blob/master/configs/kubernetes-rules-configmap.yaml)
+
+
+## Summary
+
+We want to remove any cloud provider specific logic from the kubernetes/kubernetes repo. We want to restructure the code
+to make it easy for any cloud provider to extend the kubernetes core in a consistent manner for their cloud. New cloud
+providers should look at the [Creating a Custom Cluster from Scratch](https://kubernetes.io/docs/getting-started-guides/scratch/#cloud-provider)
+and the [cloud provider interface](https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/cloud.go#L31)
+which will need to be implemented.
+
+## Motivation
+
+We are trying to remove any dependencies from Kubernetes Core to any specific cloud provider. Currently we have seven
+such dependencies. To prevent this number from growing we have locked Kubernetes Core to the addition of any new
+dependencies. This means all new cloud providers have to implement all their pieces outside of the Core.
+However everyone still ends up consuming the current set of seven in repo dependencies. For the seven in repo cloud
+providers any changes to their specific cloud provider code requires OSS PR approvals and a deployment to get those
+changes in to an official build. The relevant dependencies require changes in the following areas.
+
+- [Kube Controller Manager](https://kubernetes.io/docs/reference/generated/kube-controller-manager/) - Track usages of [CMServer.CloudProvider](https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-controller-manager/app/options/options.go)
+- [API Server](https://kubernetes.io/docs/reference/generated/kube-apiserver/) - Track usages of [ServerRunOptions.CloudProvider](https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-apiserver/app/options/options.go)
+- [kubelet](https://kubernetes.io/docs/reference/generated/kubelet/) - Track usages of [KubeletFlags.CloudProvider](https://github.com/kubernetes/kubernetes/blob/master/cmd/kubelet/app/options/options.go)
+- [How Cloud Provider Functionality is deployed to and enabled in the cluster](https://kubernetes.io/docs/setup/pick-right-solution/#hosted-solutions) - Track usage from [PROVIDER_UTILS](https://github.com/kubernetes/kubernetes/blob/master/cluster/kube-util.sh)
+
+For the cloud providers who are in repo, moving out would allow them to more quickly iterate on their solution and
+decouple cloud provider fixes from open source releases. Moving the cloud provider code out of the open source
+processes means that these processes do not need to load/run unnecessary code for the environment they are in.
+We would like to abstract a core controller manager library so help standardize the behavior of the cloud
+controller managers produced by each cloud provider. We would like to minimize the number and scope of controllers
+running in the cloud controller manager so as to minimize the surface area for per cloud provider deviation.
+
+### Goals
+
+- Get to a point where we do not load the cloud interface for any of kubernetes core processes.
+- Remove all cloud provider specific code from kubernetes/kubernetes.
+- Have a generic controller manager library available for use by the per cloud provider controller managers.
+- Move the cloud provider specific controller manager logic into repos appropriate for those cloud providers.
+
+### Intermediary Goals
+
+Have a cloud controller manager in the kubernetes main repo which hosts all of
+the controller loops for the in repo cloud providers.
+Do not run any cloud provider logic in the kube controller manager, the kube apiserver or the kubelet.
+At intermediary points we may just move some of the cloud specific controllers out. (Eg. volumes may be later than the rest)
+
+### Non-Goals
+
+Forcing cloud providers to use the generic cloud manager.
+
+## Proposal
+
+## Building CCM For Cloud Providers
+
+### Background
+
+The CCM in K8s/K8s links in all eight of the in-tree providers (aws, azure, cloudstack, gce, openstack, ovirt, photon 
+and vsphere). Each of these providers has an implementation of the Cloud Provider Interface in the K8s/K8s repo. CNCF 
+has created a repo for each of the cloud providers to extract their cloud specific code into. The assumption is that 
+this will include the CCM executable, their Cloud Provider Implementation and various build and deploy scripts. Until 
+we have extracted every in-tree provider and removed cloud provider dependencies from other binaries, we need to 
+maintain the Cloud Provider Implementation in the K8s/K8s repo. After the Cloud Provider specific code has been 
+extracted Golang imports for things like the service controller, prometheus code, utilities etc will still require each 
+cloud provider specific repository to vendor in a significant portion of the code in k8s/k8s. 
+
+We need a solution which meets the objective in both the short and long term. In the short term we cannot delete CCM or 
+cloud provider code from the K8s/K8s repo. We need to keep this code in K8s/K8s while we still support cloud provider 
+deployments from K8s/K8s. In the long term this code should be part of each cloud providers repo and that code should 
+be removed from K8s/K8s. This suggests that in the short term that code should have one source of truth. However it 
+should probably not end up in the vendor directory as that is not its intended final home. Other code such as the 
+controllers should end up in the vendor directory. Additionally each provider will need their own copy of 
+pkg/cloudprovider/providers/providers.go and related build file to properly control which Cloud Provider Implementation 
+get linked in.
+
+We also need to be able to package a combination of binaries from K8s/K8s and K8s/cloud-provider-<cp> into a deployable 
+package. The code for this will need to accommodate things like differing side cars for each cloud provider’s CSI 
+implementation and possible desire to run additional controller managers or extension api servers. As such it seems 
+better to have this code live in the cloud provider specific repo. This also allows this code to be simpler as it does 
+not have to attempt to support all the different cloud provider configurations. This separate from things like the 
+local deployment option, which K8s/K8s should continue to support.
+
+Lastly there are specific flags which need to be set on various binaries for this to work. Kubernetes API-Server, 
+Kubernetes Controller-Manager and kubelet should all have the --cloud-provider flag set to external. For the Cloud 
+Controller-Manager the --cloud-provider flag should be set appropriately for that cloud provider. In addition we need 
+to set the set of controllers running in the Kubernetes Controller-Manager. More on that later.
+
+For further background material please look at [running cloud controller](https://kubernetes.io/docs/tasks/administer-cluster/running-cloud-controller/).
+
+### Design Options
+
+For the short term problem of sharing code between K8s/K8s repo, the K8s/cloud-provider repos and K8s SIGs/library repos; 
+there are 2 fundamental solutions. We can push more code into Staging to make the code available to the 
+K8s/cloud-provider repos. Currently code which needs to be shared between K8s/K8s itself and other projects is put 
+the code in Staging. This allows the build machinery to detect things like attempts to make the shared code depend on 
+code which is not shared (i.e. disallowing code in Staging from depending Iton code in K8s/K8s but not in Staging) 
+In addition using Staging means that we can benefit from work to properly break out the libraries/code which exists in 
+Staging. Most of the repositories we are adding to staging should end up as K8s SIGs/library repos. The K8s/cloud-provider 
+repos should be the top of any build dependency trees. Code which needs to be consumed by other repos (Eg CSI plugins, 
+shared controllers, ...) should not be in the K8s/cloud-provider repo but in an appropriate K8s SIGs/library repo.
+ 
+The code which needs to be shared can be broken into several types. 
+
+There is code which properly belongs in the various cloud-provider repos. The classic example of this would be the 
+implementations of the cloud provider interface. (Eg. [gce](https://github.com/kubernetes/kubernetes/tree/master/pkg/cloudprovider/providers/gce) 
+or [aws](https://github.com/kubernetes/kubernetes/tree/master/pkg/cloudprovider/providers/aws)) These sections of code 
+need to be shared until we can remove all cloud provider dependencies from K8s/K8s. (i.e. When KAS, KCM and kubelet no 
+longer contain a cloud-provider flag and no longer depend on either the cloud provider interface or any cloud provider 
+implementations) At that point they should be permanently moved to the individual provider repos. I would suggest that 
+as long as the code is shared it be in vendor for K8s/cloud-provider. We would want to create a separate Staging repo 
+in K8s/K8s for each cloud-provider. 
+
+The majority of the controller-manager framework and its dependencies also needs to be shared. This is code that would 
+be shared even after cloud providers are removed from K8s/K8s. As such it probably makes sense to make the controller 
+manager framework its own K8s/K8s Staging repo. 
+
+It should be generally possible for cloud providers to determine where a controller runs and even over-ride specific 
+controller functionality. Please note that if a cloud provider exercises this possibility it is up to that cloud provider 
+to keep their custom controller conformant to the K8s/K8s standard. This means any controllers may be run in either KCM 
+or CCM. As an example the NodeIpamController, will be shared acrosss K8s/K8s and K8s/cloud-provider-gce, both in the 
+short and long term. Currently it needs to take a cloud provider to allow it to do GCE CIDR management. We could handle 
+this by leaving the cloud provider interface with the controller manager framework code. The GCE controller manager could 
+then inject the cloud provider for that controller. For everyone else (especially the KCM) NodeIpamController is 
+interesting because currently everyone needs its generic behavior for things like ranges. However Google then wires in 
+the cloud provider to provide custom functionality on things like CIDRs. The thought then is that in the short term we 
+allow it to be run in either KCM or CCM. Most cloud providers would run it in the KCM, Google would run it in the CCM. 
+When we are ready to move cloud provider code out of K8s/K8s, we remove the cloud provider code from the version which 
+is in K8s/K8s and continue to have flags to control if it runs. K8s/Cloud-Provider-Google could then have an enhanced 
+version which is run in the CCM. Other controllers such as Route and Service needs to run in either the KCM or CCM. For 
+things like K8s/K8s e2e tests we will always want these controllers in the K8s/K8s repo. Having it in the K8s/K8s repo 
+is also useful for keeping the behavior of these sort of core systems consistent.
+
+#### Staging
+
+There are several sections of code which need to be shared between the K8s/K8s repo and the K8s/Cloud-provider repos. 
+The plan for doing that sharing is to move the relevant code into the Staging directory as that is where we share code 
+today. The current Staging repo has the following packages in it.
+- Api
+- Apiextensions-apiserver
+- Apimachinery
+- Apiserver
+- Client-go
+- Code-generator
+- Kube-aggregator
+- Metrics
+- Sample-apiserver
+- Sample-Controller
+
+With the additions needed in the short term to make this work; the Staging area would now need to look as follows. 
+- Api
+- Apiextensions-apiserver
+- Apimachinery
+- Apiserver
+- Client-go
+- **Controller**
+  - **Cloud**
+  - **Service**
+  - **NodeIpam**
+  - **Route**
+  - **?Volume?**
+- **Controller-manager**
+- **Cloud-provider-aws**
+- **Cloud-provider-azure**
+- **Cloud-provider-cloudstack**
+- **Cloud-provider-gce**
+- **Cloud-provider-openstack**
+- **Cloud-provider-ovirt**
+- **Cloud-provider-photon**
+- **Cloud-provider-vsphere**
+- Code-generator
+- Kube-aggregator
+- **Kube-utilities**
+- Metrics
+- Sample-apiserver
+- Sample-Controller
+
+When we complete the cloud provider work, several of the new modules in staging should be moving to their permanent new 
+home in the appropriate K8s/Cloud-provider repoas they will no longer be needed in the K8s/K8s repo. There are however 
+other new modules we will add which continue to be needed by both K8s/K8s and K8s/Cloud-provider. Those modules will 
+remain in Staging until the Staging initiative completes and they are moved into some other Kubernetes shared code repo.
+- Api
+- Apiextensions-apiserver
+- Apimachinery
+- Apiserver
+- Client-go
+- **Controller**
+  - **Cloud**
+  - **Service**
+  - **NodeIpam**
+  - **Route**
+  - **?Volume?**
+- **Controller-manager**
+- ~~Cloud-provider-aws~~ 
+- ~~Cloud-provider-azure~~
+- ~~Cloud-provider-cloudstack~~
+- ~~Cloud-provider-gce~~
+- ~~Cloud-provider-openstack~~
+- ~~Cloud-provider-ovirt~~
+- ~~Cloud-provider-photon~~
+- ~~Cloud-provider-vsphere~~
+- Code-generator
+- Kube-aggregator
+- **Kube-utilities**
+- Metrics
+- Sample-apiserver
+- Sample-Controller
+
+#### Cloud Provider Instances
+
+Currently in K8s/K8s the cloud providers are actually included by including [providers.go](https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/providers.go)
+file which then includes each of the in-tree cloud providers. In the short term we would leave that file where it is 
+and adjust it to point at the new homes under Staging. For the K8s/cloud-provider repo, would have the following CCM 
+wrapper file. (Essentially a modified copy of cmd/cloud-controller-manager/controller-manager.go) The wrapper for each 
+cloud provider would import just their vendored cloud-provider implementation rather than providers.go file. 
+
+k8s/k8s: pkg/cloudprovider/providers/providers.go
+```package cloudprovider
+
+import (
+  // Prior to cloud providers having been moved to Staging
+  _ "k8s.io/cloudprovider-aws"
+  _ "k8s.io/cloudprovider-azure"
+  _ "k8s.io/cloudprovider-cloudstack"
+  _ "k8s.io/cloudprovider-gce"
+  _ "k8s.io/cloudprovider-openstack"
+  _ "k8s.io/cloudprovider-ovirt"
+  _ "k8s.io/cloudprovider-photon"
+  _ "k8s.io/cloudprovider-vsphere"
+)
+```
+
+k8s/cloud-provider-gcp: pkg/cloudprovider/providers/providers.go
+```package cloudprovider
+
+import (
+  // Cloud providers
+  _ "k8s.io/cloudprovider-gce"
+)
+```
+
+#### Build Targets
+
+We then get to the issue of creating a deployable artifact for each cloud provider. There are several artifacts beyond 
+the CCM which are cloud provider specific. These artifacts include things like the deployment scripts themselves, the 
+contents of the add-on manager and the sidecar needed to get CSI/cloud-specific persistent storage to work. Ideally 
+these would then be packaged with a version of the Kubernetes core components (KAS, KCM, kubelet, …) which have not 
+been statically linked against the cloud provider libraries. However in the short term the K8s/K8s deployable builds 
+will still need to link these binaries against all of the in-tree plugins. We need a way for the K8s/cloud-provider 
+repo to consume artifacts generated by the K8s/K8s repo. For official releases these artifacts should be published to a 
+package repository. From there the cloud providers can pull the cloud agnostic Kubernetes artifact and decorate it 
+appropriate to their cloud. The K8s/Cloud-Provider should be using some sort of manifest file to determine which 
+official K8s/K8s artifact to pull. This allows for things like repeatability in builds, for hot fix builds and also a 
+mechanism for rolling out changes which span K8s/K8s and K8s/Cloud-Provider. For local cloud releases we can use a 
+build convention. We can expect the K8s/K8s and K8s/Cloud-Provider repos to be checked out under the same GOPATH. 
+The K8s/Cloud-Provider can then have a local build target which looks for the K8s/K8s artifacts to be in the 
+appropriate place under the GOPATH (or at a location pointed to be a K8s home environment variable) This allows for 
+both official builds and for developers to easily work at running changes which span K8s/K8s and K8s/Cloud-Provider.
+
+#### K8s/K8s Releases and K8s/CP Releases
+
+One of the goals for this project is to free cloud providers to generate releases as they want. This implies that 
+K8s/CP releases are largely decoupled from K8s/K8s releases. The cadence of K8s/K8s releases should not change. We are 
+assuming that K8s/CP releases will be on a similar or faster cadence as needed. It is desirable for the community for 
+all cloud providers to support recent releases. As such it would be good for them to minimize the lag between K8s/K8s 
+releases and K8s/CP releases. A K8s/CP cannot however release a Kubernetes version prior to that version having been 
+released by K8s/K8s. (So for example the cloud provider cannot make a release with a 1.13 Kubernetes core, prior to 
+K8s/K8s having released 1.13) The ownership and responsibility of publishing K8s/K8s will not change with this project. 
+The K8s/CP releases must necessarily move to being the responsibility of that cloud provider or a set of owners 
+delegated by that cloud provider.
+
+#### Migrating to a CCM Build
+
+Migrating to a CCM build requires some thought. When dealing with a new cluster, things are fairly simple; we install 
+everything at the same version and then the customer begins to customize. Migrating an existing cluster which is a 
+generic K8s/K8s cluster with the cloud-provider flag set to a K8s/Cloud-Provider built is a bit trickier. The system 
+needs to work during the migration where disparate pieces can be on different versions. While we specify that the exact 
+upgrade steps are cloud provider specific, we do provide guidance that the control plane (master) should be upgraded 
+first (master version >= kubelet version) and that the system should be able to handle up to a 2 revision difference 
+between the control plane and the kubelets. In addition with disruptions budgets etc, there will not be a consistent 
+version of the kubelets, until the upgrade completes. So we need to ensure that our cloud provider/CCM builds work with 
+existing clusters. This means that we need to account for things like older kubelets having the cloud provider enabled 
+and using it for things like direct volume mount/unmount, IP discovery, … We can even expect that scaling events such 
+as increases in the size of a replica set may cause us to deploy old kubelet images which directly use the cloud 
+provider implementation in clusters which are controlled by a CCM build. We need to make sure we test these sort of 
+scenarios and ensure they work (get their IP, can mount cloud specific volume types, …) 
+
+HA migrations presents some special issues. HA masters are composed of multiple master nodes in which components like 
+the controller managers use leader election to determine which is the currently operating instance. Before the 
+migration begins there is no guarantee which instances will be leading or that all the leaders will be on the same 
+master instance. So we will begin by taking down 1 master instance and upgrading it. At this point there will be some 
+well known conditions. The Kube-Controller-Master in the lead will be one of the old build instances. If the leader had 
+been the instance running on that master it will lose its lease to one of the older instances. At the same time there 
+will only be one Cloud-Controller-Manager which is the running on the master running the new code. This implies that we 
+will have a few controllers running in both the new lead cloud-controller-manager and the old kube-controller-manager. 
+I would suggest that as an initial part of a HA upgrade we disable these controllers in the kube-controller-managers. 
+This can be accomplished by providing the controllers flag. If the only controller we wanted to disable were service 
+and route then we would set the flag as follows
+ ```
+kube-controller-manager --controllers=\"*,-service,-route\"
+``` 
+This assumes that upgrading the first master instance can be accomplished inside of the SLO for these controllers being 
+down.
+
+#### Flags, Service Accounts, etc
+
+The correct set of flags, service accounts etc, which will be needed for each cloud provider, is expected to be 
+different and is at some level, left as an exercise for each cloud provider. That having been said, there are a few 
+common guidelines which are worth mentioning. It is expected that all the core components (kube-apiserver, 
+kube-controller-manager, kubelet) should have their --cloud-provider flag set to “external”. Cloud-providers, who have 
+their own volume type (eg. gce-pd) but do not yet have the CSI plugin (& side car) enabled in their CCM build, will 
+need to set the --external-cloud-volume-plugin to their cloud provider implementation key. (eg. gce) There are also a 
+considerable number of roles and bindings which are needed to get the CCM working. For core components this is handled 
+through a bootstrapping process inside of the kube-apiserver. However the CCM and cloud-provider pieces are not 
+considered core components. The expectation then is that the appropriate objects will be created by deploying yaml 
+files for them in the addons directory. The add on manager (or cloud provider equivalent system) will then cause the 
+objects to be created as the system comes up. The set of objects which we know we will need include :-
+- ServiceAccount
+  - cloud-controller-manager
+- User
+  - system:cloud-controller-manager
+- Role
+  - system::leader-locking-cloud-controller-manager
+- ClusterRole
+  - system:controller:cloud-node-controller
+  - system:cloud-controller-manager
+  - system:controller:pvl-controller
+- RoleBinding
+  - cloud-controller-manager to system::leader-locking-cloud-controller-manager
+- ClusterRoleBinding
+  - cloud-node-controller to system:controller:cloud-node-controller
+
+#### Deployment Scripts
+
+Currently there is a lot of common code in the K8s/K8s cluster directory. Each of the cloud providers today build on 
+top of that common code to do their deployment. The code running in K8s/Cloud-provider will necessarily be different. 
+We have new files (addon) and executables (CCM and CSI) to be deployed. The existing executables need to be started 
+with different flags (--cloud-provider=external). We also have additional executables which need to be started. This 
+may then result in different resource requirements for the master and the kubelets. So it is clear that there will need 
+to be at least some changes between the deployment scripts going from K8s/K8s to K8s/Cloud-provider. There is also 
+likely to be some desire to stream-line and simplify these scripts in K8s/Cloud-provider. A lot of the generic handling 
+and flexibility in K8s/K8s is not needed in K8s/Cloud-provider. It is also worth looking at 
+[CCM Repo Requirements](#repository-requirements) for some suggestions on common K8s/Cloud-provider 
+layout suggestions. These include an installer directory for custom installer code.
+
+#### kubeadm [WIP]
+
+kubeadm is a tool for creating clusters. For reference see [creating cluster with kubeadm](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/). 
+Need to determine how kubeadm and K8s/Cloud-providers should interact. More planning clearly needs to be done on 
+cloud-provider and kubeadm planning.
+
+#### CI, TestGrid and Other Testing Issues [WIP]
+
+As soon as we have more than one repo involved in a product/build we get some interesting problems in running tests. A 
+K8s/Cloud-provider deployment is the result of a combination of code from K8s/K8s and K8s/Cloud-provider. If something 
+is broken it could be the result of a code change in either, and modifications by an engineer testing against one cloud 
+provider could have unintended consequences in another. 
+
+To address this issue, SIG-Testing has invested in a process by which participating cloud providers can report results 
+from their own CI testing processes back to TestGrid, the Kubernetes community tool for tracking project health. 
+Results become part of a central dashboard to display test results across multiple cloud providers. 
+
+More thought should be put into managing multiple, independently evolving components. 
+
+## Alternatives
+
+### Staging Alternatives
+
+#### Git Filter-Branch
+
+One possible alternative is to make use of a Git Filter Branch to extract a sub-directory into a virtual repo. The repo 
+needs to be sync'd in an ongoing basis with K8s/K8s as we want one source of truth until K8s/K8s does not pull in the 
+code. This has issues such as not giving K8s/K8s developers any indications of what the dependencies various 
+K8s/Cloud-providers have. Without that information it becomes very easy to accidentally break various cloud providers 
+and time you change dependencies in the K8s/K8s repo. With staging the dependency line is simple and [automatically 
+enforced](https://github.com/kubernetes/kubernetes/blob/master/hack/verify-no-vendor-cycles.sh). Things in Staging are 
+not allowed to depend on things outside of Staging. If you want to add such a dependency you need to add the dependent 
+code to Staging. The act of doing this means that code should get synced and solve the problem. In addition the usage 
+of a second different library and repo movement mechanism will make things more difficult for everyone.
+
+“Trying to share code through the git filter will not provide this protection. In addition it means that we now have 
+two sharing code mechanisms which increases complexity on the community and build tooling. As such I think it is better 
+to continue to use the Staging mechanisms. ”
+
+### Build Location Alternatives
+
+#### Build K8s/K8s from within K8s/Cloud-provider
+
+The idea here is to avoid having to add a new build target to K8s/K8s. The K8s/Cloud-provider could have their own 
+custom targets for building things like KAS without other cloud-providers implementations linked in. It would also 
+allow other customizations of the standard binaries to be created. While a powerful tool, this mechanism seems to 
+encourage customization of these core binaries and as such to be discouraged. Providing the appropriate generic 
+binaries cuts down on the need to duplicate build logic for these core components and allow each optimization of build. 
+Download prebuilt images at a version and then just build the appropriate addons.
+
+#### Build K8s/Cloud-provider within K8s/K8s
+
+The idea here would be to treat the various K8s/Cloud-provider repos as libraries. You would specify a build flavor and 
+we would pull in the relevant code based on what you specified. This would put tight restrictions on how the 
+K8s/Cloud-provider repos would work as they would need to be consumed by the K8s/K8s build system. This seems less 
+extensible and removes the nice loose coupling which the other systems have. It also makes it difficult for the cloud 
+providers to control their release cadence. 
+
+### Config Alternatives
+
+#### Use component config to determine where controllers run
+
+Currently KCM and CCM have their configuration passed in as command line flags. If their configuration were obtained 
+from a configuration server (component config) then we could have a single source of truth about where each controller 
+should be run. This both solves the HA migration issue and other concerns about making sure that a controller only runs 
+in 1 controller manager. Rather than having the controllers as on or off, controllers would now be configured to state 
+where they should run, KCM, CCM, Nowhere, … If the KCM could handle this as a run-time change nothing would need to 
+change. Otherwise it becomes a slight variant of the proposed solution. This is probably the correct long term 
+solution. However for the timeline we are currently working with we should use the proposed solution.