From 4bb732b8eaf0143546065ca3fac54753c5c54374 Mon Sep 17 00:00:00 2001 From: Ryan Phillips Date: Thu, 17 Aug 2017 16:46:29 -0500 Subject: add apiserver-count-fix proposal --- .../design-proposals/apiserver-count-fix.md | 98 ++++++++++++++++++++++ 1 file changed, 98 insertions(+) create mode 100644 contributors/design-proposals/apiserver-count-fix.md diff --git a/contributors/design-proposals/apiserver-count-fix.md b/contributors/design-proposals/apiserver-count-fix.md new file mode 100644 index 00000000..3f9b735a --- /dev/null +++ b/contributors/design-proposals/apiserver-count-fix.md @@ -0,0 +1,98 @@ +# apiserver-count fix proposal + +Authors: @rphillips + +## Table of Contents + +1. [Overview](#overview) +2. [Known Issues](#known-issues) +3. [Proposal](#proposal) +4. [Prior Art](#prior-art) + +## Overview + +Proposal to fix Issue [#22609](https://github.com/kubernetes/kubernetes/issues/22609) + +`kube-apiserver` currently has a command-line argument `--apiserver-count` +specifying the number of api masters. This masterCount is used in the +MasterCountEndpointReconciler on a 10 second interval to potentially cleanup +stale API Endpoints. The issue is when the number of kube-apiserver instances +gets below masterCount. If this case happens, the stale instances within the +Endpoints does not get cleaned up. + +## Known Issues + +Each apiserver’s reconciler only cleans up for its own IP. If a new server +is spun up at a new IP, then the old IP in the Endpoints list is only +reclaimed if the number of apiservers becomes greater-than or equal to the +masterCount. For example: + +* If the masterCount = 3, and there are 3 API servers running (named: A, B, and +C) +* ‘B’ API server is terminated for any reason +* The IP for endpoint ‘B’ is not +removed from the Endpoints list + +There is logic within the [MasterCountEndpointReconciler](https://github.com/kubernetes/kubernetes/blob/68814c0203c4b8abe59812b1093844a1f9bdac05/pkg/master/controller.go#L293) to attempt to make +the Endpoints eventually consistent, but the code relies on the Endpoints +count becoming equal to or greater than masterCount. When the apiservers +become greater than the masterCount the Endpoints tend to flap. + +If the number endpoints were scaled down from automation, then the Endpoints +would never become consistent. + +## Proposal + +### Create New Reconciler + +| Kubernetes Release | Quality | Description | +| ------------- | ------------- | ----------- | +| 1.9 | alpha | +| 1.10 | beta | +| 1.11 | stable | + +The MasterCountEndpointReconciler does not meet the current needs for durability of API Endpoint creation, deletion, or failure cases. + +Create a new MasterEndpointReconciler within master/controller.go. + +Add a standard `kube-apiserver-endpoints` ConfigMap in the `default` namespace. The ConfigMap would be formed such that: + +*Key Format*: ip-[IP String Formatted]-[port] + +```go +ConfigMap{ + "ip-2001-4860-4860--8888-443": "serialized JSON ControllerEndpointData", + "ip-192-168-0-3-443": "serialized JSON ControllerEndpointData", +} + +type ControllerEndpointData struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:",inline"` + api.EndpointPort + CreationTimestamp Time `json:"creationTimestamp,omitempty" protobuf:"bytes,8,opt,name=creationTimestamp"` + UpdateTimestamp Time `json:"creationTimestamp,omitempty" protobuf:"bytes,8,opt,name=updateTimestamp"` +} +``` + +configmap.yml: + +```yaml +kind: ConfigMap +apiVersion: v1 +metadata: + name: kube-apiserver-endpoints + namespace: default +data: +``` + +### Refactor Old Reconciler + +| Kubernetes Release | Quality | Description | +| ------------- | ------------- | ----------- | +| 1.9 | stable | + +We could potentially reuse the old reconciler, but ignore the masterCount and change the logic to use the proposal from the previous section. + +## Prior Art + +[Security Labeller](https://github.com/coreos-inc/security-labeller/issues/18#issuecomment-320791878) \ No newline at end of file -- cgit v1.2.3 From 81b6c5845aca4e268ba662f4f70e4570fc58562c Mon Sep 17 00:00:00 2001 From: Ryan Phillips Date: Fri, 18 Aug 2017 12:18:12 -0500 Subject: add reconcile loop logic --- .../design-proposals/apiserver-count-fix.md | 23 ++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) diff --git a/contributors/design-proposals/apiserver-count-fix.md b/contributors/design-proposals/apiserver-count-fix.md index 3f9b735a..7ed6a39b 100644 --- a/contributors/design-proposals/apiserver-count-fix.md +++ b/contributors/design-proposals/apiserver-count-fix.md @@ -22,7 +22,7 @@ Endpoints does not get cleaned up. ## Known Issues -Each apiserver’s reconciler only cleans up for its own IP. If a new server +Each apiserver’s reconciler only cleans up for it's own IP. If a new server is spun up at a new IP, then the old IP in the Endpoints list is only reclaimed if the number of apiservers becomes greater-than or equal to the masterCount. For example: @@ -53,7 +53,15 @@ would never become consistent. The MasterCountEndpointReconciler does not meet the current needs for durability of API Endpoint creation, deletion, or failure cases. -Create a new MasterEndpointReconciler within master/controller.go. +Create a new `MasterEndpointReconciler` within master/controller.go. + +Add a `kube-apiserver-endpoints-config` ConfigMap in the `default` namespace. The duration found within the map would be configurable by admins without a recompile. The ConfigMap would include the following: + +```go +ConfigMap{ + "expire-duration": "1m", // golang duration (ns,us,ms,s,m,h) +} +``` Add a standard `kube-apiserver-endpoints` ConfigMap in the `default` namespace. The ConfigMap would be formed such that: @@ -74,6 +82,14 @@ type ControllerEndpointData struct { } ``` +On each reconcile loop (defaults to every 10 seconds currently): + +1. Retrieve `kube-apiserver-endpoints-config` ConfigMap (as configMap) +1. Retrieve `kube-apiserver-endpoints` ConfigMap (as endpointMap) +1. Update the `UpdateTimestamp` for the currently running API server +1. Remove all endpoints where the UpdateTimestamp is greater than `expire-duration` from the configMap. +1. Write endpointMap back to Kubernetes ConfigMap API + configmap.yml: ```yaml @@ -82,7 +98,6 @@ apiVersion: v1 metadata: name: kube-apiserver-endpoints namespace: default -data: ``` ### Refactor Old Reconciler @@ -95,4 +110,4 @@ We could potentially reuse the old reconciler, but ignore the masterCount and ch ## Prior Art -[Security Labeller](https://github.com/coreos-inc/security-labeller/issues/18#issuecomment-320791878) \ No newline at end of file +[Security Labeller](https://github.com/coreos-inc/security-labeller/issues/18#issuecomment-320791878) -- cgit v1.2.3 From 82e7ae00a7091b6dcd7b35e3a0181af3563a8a7f Mon Sep 17 00:00:00 2001 From: Ryan Phillips Date: Fri, 18 Aug 2017 12:21:24 -0500 Subject: make the stanza clear on what it does --- contributors/design-proposals/apiserver-count-fix.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/contributors/design-proposals/apiserver-count-fix.md b/contributors/design-proposals/apiserver-count-fix.md index 7ed6a39b..f822c7d9 100644 --- a/contributors/design-proposals/apiserver-count-fix.md +++ b/contributors/design-proposals/apiserver-count-fix.md @@ -82,7 +82,8 @@ type ControllerEndpointData struct { } ``` -On each reconcile loop (defaults to every 10 seconds currently): +The reconcile loop will expire endpoints that do not meet the duration. On +each reconcile loop (the loop runs every 10 seconds currently): 1. Retrieve `kube-apiserver-endpoints-config` ConfigMap (as configMap) 1. Retrieve `kube-apiserver-endpoints` ConfigMap (as endpointMap) -- cgit v1.2.3 From 71c6ab2c2dcadd3641e542a973253522bf510785 Mon Sep 17 00:00:00 2001 From: Ryan Phillips Date: Tue, 22 Aug 2017 13:06:48 -0500 Subject: update with comments addressed --- .../design-proposals/apiserver-count-fix.md | 108 ++++++++++++--------- 1 file changed, 61 insertions(+), 47 deletions(-) diff --git a/contributors/design-proposals/apiserver-count-fix.md b/contributors/design-proposals/apiserver-count-fix.md index f822c7d9..89775bf2 100644 --- a/contributors/design-proposals/apiserver-count-fix.md +++ b/contributors/design-proposals/apiserver-count-fix.md @@ -13,33 +13,34 @@ Authors: @rphillips Proposal to fix Issue [#22609](https://github.com/kubernetes/kubernetes/issues/22609) -`kube-apiserver` currently has a command-line argument `--apiserver-count` -specifying the number of api masters. This masterCount is used in the -MasterCountEndpointReconciler on a 10 second interval to potentially cleanup -stale API Endpoints. The issue is when the number of kube-apiserver instances -gets below masterCount. If this case happens, the stale instances within the -Endpoints does not get cleaned up. +`kube-apiserver` currently has a command-line argument +`--apiserver-count` specifying the number of api masters. This +masterCount is used in the MasterCountEndpointReconciler on a 10 second +interval to potentially cleanup stale API Endpoints. The issue is when +the number of kube-apiserver instances gets below masterCount. If this +case happens, the stale instances within the Endpoints does not get +cleaned up. ## Known Issues -Each apiserver’s reconciler only cleans up for it's own IP. If a new server -is spun up at a new IP, then the old IP in the Endpoints list is only -reclaimed if the number of apiservers becomes greater-than or equal to the -masterCount. For example: +Each apiserver’s reconciler only cleans up for it's own IP. If a new +server is spun up at a new IP, then the old IP in the Endpoints list is +only reclaimed if the number of apiservers becomes greater-than or equal +to the masterCount. For example: -* If the masterCount = 3, and there are 3 API servers running (named: A, B, and -C) +* If the masterCount = 3, and there are 3 API servers running (named: A, B, and C) * ‘B’ API server is terminated for any reason * The IP for endpoint ‘B’ is not removed from the Endpoints list There is logic within the [MasterCountEndpointReconciler](https://github.com/kubernetes/kubernetes/blob/68814c0203c4b8abe59812b1093844a1f9bdac05/pkg/master/controller.go#L293) to attempt to make -the Endpoints eventually consistent, but the code relies on the Endpoints -count becoming equal to or greater than masterCount. When the apiservers -become greater than the masterCount the Endpoints tend to flap. +the Endpoints eventually consistent, but the code relies on the +Endpoints count becoming equal to or greater than masterCount. When the +apiservers become greater than the masterCount the Endpoints tend to +flap. -If the number endpoints were scaled down from automation, then the Endpoints -would never become consistent. +If the number endpoints were scaled down from automation, then the +Endpoints would never become consistent. ## Proposal @@ -47,49 +48,54 @@ would never become consistent. | Kubernetes Release | Quality | Description | | ------------- | ------------- | ----------- | -| 1.9 | alpha |
  • Add a new reconciler
  • Add a command-line switch --new-reconciler
  • Add a command-line switch --old-reconciler
-| 1.10 | beta |
  • Turn on the new reconciler by default |
-| 1.11 | stable |
  • Remove code for old reconciler
  • Remove --old-reconciler
  • Remove --new-reconciler
  • Remove --apiserver-count
+| 1.9 | alpha |
  • Add a new reconciler
  • Add a command-line type `--apiserver-endpoint-reconciler-type`
    • configmap
    • default
+| 1.10 | beta |
  • Turn on the `configmap` type by default
+| 1.11 | stable |
  • Remove code for old reconciler
  • Remove --apiserver-count
-The MasterCountEndpointReconciler does not meet the current needs for durability of API Endpoint creation, deletion, or failure cases. +The MasterCountEndpointReconciler does not meet the current needs for +durability of API Endpoint creation, deletion, or failure cases. -Create a new `MasterEndpointReconciler` within master/controller.go. +Create a new `MasterEndpointConfigMapReconciler` within +master/controller.go. -Add a `kube-apiserver-endpoints-config` ConfigMap in the `default` namespace. The duration found within the map would be configurable by admins without a recompile. The ConfigMap would include the following: +Add a `kube-apiserver-config` ConfigMap in the `kube-system` +namespace. The duration found within the Coordination map would be configurable by +admins without a recompile. The ConfigMap would include the following: ```go -ConfigMap{ - "expire-duration": "1m", // golang duration (ns,us,ms,s,m,h) +KubeApiServerConfigMap{ + "expiration-duration": "1m", + ... [other flags potentially] } ``` -Add a standard `kube-apiserver-endpoints` ConfigMap in the `default` namespace. The ConfigMap would be formed such that: - -*Key Format*: ip-[IP String Formatted]-[port] +The Coordination ConfigMap would be formed such that: ```go -ConfigMap{ - "ip-2001-4860-4860--8888-443": "serialized JSON ControllerEndpointData", - "ip-192-168-0-3-443": "serialized JSON ControllerEndpointData", +CoordinationConfigMap{ + "[namespace]/[name]": "[KubeAPIServerEndpoint]", } +``` + +*Key Format*: ip-[IP String Formatted]-[port] -type ControllerEndpointData struct { - metav1.TypeMeta `json:",inline"` - metav1.ObjectMeta `json:",inline"` - api.EndpointPort - CreationTimestamp Time `json:"creationTimestamp,omitempty" protobuf:"bytes,8,opt,name=creationTimestamp"` - UpdateTimestamp Time `json:"creationTimestamp,omitempty" protobuf:"bytes,8,opt,name=updateTimestamp"` +```go +KubeAPIServerEndpointConfigMap{ + "master-count": "0", + "ip-2001-4860-4860--8888-443": "time.Time", + "ip-192-168-0-3-443": "time.Time", } ``` -The reconcile loop will expire endpoints that do not meet the duration. On -each reconcile loop (the loop runs every 10 seconds currently): +The reconcile loop will expire endpoints that do not meet the duration. +On each reconcile loop (the loop runs every 10 seconds currently, +but interval will be changed to 80% of the `expiration-duration`): -1. Retrieve `kube-apiserver-endpoints-config` ConfigMap (as configMap) 1. Retrieve `kube-apiserver-endpoints` ConfigMap (as endpointMap) -1. Update the `UpdateTimestamp` for the currently running API server -1. Remove all endpoints where the UpdateTimestamp is greater than `expire-duration` from the configMap. -1. Write endpointMap back to Kubernetes ConfigMap API +1. Update the timestamp for the currently running API server +1. Remove all endpoints where the timestamp is greater than `expiration-duration` from the configMap. +1. Do a GET on the `kube-apiserver-endpoints` +1. Construct a PATCH operation with the deleted endpoints configmap.yml: @@ -101,13 +107,21 @@ metadata: namespace: default ``` +### Custom Resource Definitions + +CRD's were considered for this proposal, but were not proposed due to +constraints of having CRDs within core has not been clearly defined. +Layering could also be an issue, so the proposal defined ConfigMaps as +the current path forward. + ### Refactor Old Reconciler -| Kubernetes Release | Quality | Description | -| ------------- | ------------- | ----------- | -| 1.9 | stable |
  • Change the logic in the current reconciler
+| Release | Quality | Description | +| ------- | ------- | ------------------------------------------------------------ | +| 1.9 | stable |
  • Change the logic in the current reconciler
| -We could potentially reuse the old reconciler, but ignore the masterCount and change the logic to use the proposal from the previous section. +We could potentially reuse the old reconciler by changing the reconciler to count +the endpoints and set the `masterCount` (with a RWLock) to the count. ## Prior Art -- cgit v1.2.3 From f6021c8a3de5e03f19caf00b7e19054af3085ead Mon Sep 17 00:00:00 2001 From: Ryan Phillips Date: Tue, 22 Aug 2017 13:21:05 -0500 Subject: more updates to address lucas' comments --- .../design-proposals/apiserver-count-fix.md | 41 +++++++++++----------- 1 file changed, 21 insertions(+), 20 deletions(-) diff --git a/contributors/design-proposals/apiserver-count-fix.md b/contributors/design-proposals/apiserver-count-fix.md index 89775bf2..df9ca7f3 100644 --- a/contributors/design-proposals/apiserver-count-fix.md +++ b/contributors/design-proposals/apiserver-count-fix.md @@ -5,22 +5,23 @@ Authors: @rphillips ## Table of Contents 1. [Overview](#overview) -2. [Known Issues](#known-issues) -3. [Proposal](#proposal) -4. [Prior Art](#prior-art) +1. [Known Issues](#known-issues) +1. [Proposal](#proposal) +1. [Alternate Proposals](#alternate-proposals) + 1. [Custom Resource Definitions](#custom-resource-definitions) + 1. [Refactor Old Reconciler](#refactor-old-reconciler) ## Overview Proposal to fix Issue [#22609](https://github.com/kubernetes/kubernetes/issues/22609) -`kube-apiserver` currently has a command-line argument -`--apiserver-count` specifying the number of api masters. This -masterCount is used in the MasterCountEndpointReconciler on a 10 second -interval to potentially cleanup stale API Endpoints. The issue is when -the number of kube-apiserver instances gets below masterCount. If this -case happens, the stale instances within the Endpoints does not get -cleaned up. - +`kube-apiserver` currently has a command-line argument `--apiserver-count` +specifying the number of api servers. This masterCount is used in the +MasterCountEndpointReconciler on a 10 second interval to potentially cleanup +stale API Endpoints. The issue is when the number of kube-apiserver instances +gets below or above the masterCount. If the below case happens, the stale +instances within the Endpoints does not get cleaned up, or in the latter case +the endpoints start to flap. ## Known Issues Each apiserver’s reconciler only cleans up for it's own IP. If a new @@ -48,7 +49,7 @@ Endpoints would never become consistent. | Kubernetes Release | Quality | Description | | ------------- | ------------- | ----------- | -| 1.9 | alpha |
  • Add a new reconciler
  • Add a command-line type `--apiserver-endpoint-reconciler-type`
    • configmap
    • default
+| 1.9 | alpha |
  • Add a new reconciler
  • Add a command-line type `--alpha-apiserver-endpoint-reconciler-type`
    • configmap
    • default
| 1.10 | beta |
  • Turn on the `configmap` type by default
| 1.11 | stable |
  • Remove code for old reconciler
  • Remove --apiserver-count
@@ -87,11 +88,13 @@ KubeAPIServerEndpointConfigMap{ } ``` +***TODO: should it a serialized struct, or the raw time value?*** + The reconcile loop will expire endpoints that do not meet the duration. On each reconcile loop (the loop runs every 10 seconds currently, but interval will be changed to 80% of the `expiration-duration`): -1. Retrieve `kube-apiserver-endpoints` ConfigMap (as endpointMap) +1. GET `kube-apiserver-endpoints` ConfigMap (as endpointMap) 1. Update the timestamp for the currently running API server 1. Remove all endpoints where the timestamp is greater than `expiration-duration` from the configMap. 1. Do a GET on the `kube-apiserver-endpoints` @@ -107,22 +110,20 @@ metadata: namespace: default ``` -### Custom Resource Definitions +### Alternate Proposals + +#### Custom Resource Definitions CRD's were considered for this proposal, but were not proposed due to constraints of having CRDs within core has not been clearly defined. Layering could also be an issue, so the proposal defined ConfigMaps as the current path forward. -### Refactor Old Reconciler +#### Refactor Old Reconciler | Release | Quality | Description | | ------- | ------- | ------------------------------------------------------------ | -| 1.9 | stable |
  • Change the logic in the current reconciler
| +| 1.9 | stable | Change the logic in the current reconciler We could potentially reuse the old reconciler by changing the reconciler to count the endpoints and set the `masterCount` (with a RWLock) to the count. - -## Prior Art - -[Security Labeller](https://github.com/coreos-inc/security-labeller/issues/18#issuecomment-320791878) -- cgit v1.2.3 From 3b7d2b8083d7fda48129ebb768f3c9d34a00bdea Mon Sep 17 00:00:00 2001 From: Ryan Phillips Date: Tue, 22 Aug 2017 13:25:41 -0500 Subject: fix sub list formatting --- contributors/design-proposals/apiserver-count-fix.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/contributors/design-proposals/apiserver-count-fix.md b/contributors/design-proposals/apiserver-count-fix.md index df9ca7f3..500c052d 100644 --- a/contributors/design-proposals/apiserver-count-fix.md +++ b/contributors/design-proposals/apiserver-count-fix.md @@ -5,11 +5,11 @@ Authors: @rphillips ## Table of Contents 1. [Overview](#overview) -1. [Known Issues](#known-issues) -1. [Proposal](#proposal) -1. [Alternate Proposals](#alternate-proposals) - 1. [Custom Resource Definitions](#custom-resource-definitions) - 1. [Refactor Old Reconciler](#refactor-old-reconciler) +2. [Known Issues](#known-issues) +3. [Proposal](#proposal) +4. [Alternate Proposals](#alternate-proposals) + 1. [Custom Resource Definitions](#custom-resource-definitions) + 2. [Refactor Old Reconciler](#refactor-old-reconciler) ## Overview -- cgit v1.2.3 From 59a7e480c4fa483ee0d208240341869dc1ba8d60 Mon Sep 17 00:00:00 2001 From: Ryan Phillips Date: Mon, 28 Aug 2017 10:45:35 -0500 Subject: leverage OpenShift's scheduler --- .../design-proposals/apiserver-count-fix.md | 85 +++++----------------- 1 file changed, 19 insertions(+), 66 deletions(-) diff --git a/contributors/design-proposals/apiserver-count-fix.md b/contributors/design-proposals/apiserver-count-fix.md index 500c052d..a9150914 100644 --- a/contributors/design-proposals/apiserver-count-fix.md +++ b/contributors/design-proposals/apiserver-count-fix.md @@ -22,6 +22,7 @@ stale API Endpoints. The issue is when the number of kube-apiserver instances gets below or above the masterCount. If the below case happens, the stale instances within the Endpoints does not get cleaned up, or in the latter case the endpoints start to flap. + ## Known Issues Each apiserver’s reconciler only cleans up for it's own IP. If a new @@ -34,11 +35,11 @@ to the masterCount. For example: * The IP for endpoint ‘B’ is not removed from the Endpoints list -There is logic within the [MasterCountEndpointReconciler](https://github.com/kubernetes/kubernetes/blob/68814c0203c4b8abe59812b1093844a1f9bdac05/pkg/master/controller.go#L293) to attempt to make -the Endpoints eventually consistent, but the code relies on the -Endpoints count becoming equal to or greater than masterCount. When the -apiservers become greater than the masterCount the Endpoints tend to -flap. +There is logic within the +[MasterCountEndpointReconciler](https://github.com/kubernetes/kubernetes/blob/68814c0203c4b8abe59812b1093844a1f9bdac05/pkg/master/controller.go#L293) +to attempt to make the Endpoints eventually consistent, but the code relies on +the Endpoints count becoming equal to or greater than masterCount. When the +apiservers become greater than the masterCount the Endpoints tend to flap. If the number endpoints were scaled down from automation, then the Endpoints would never become consistent. @@ -49,75 +50,27 @@ Endpoints would never become consistent. | Kubernetes Release | Quality | Description | | ------------- | ------------- | ----------- | -| 1.9 | alpha |
  • Add a new reconciler
  • Add a command-line type `--alpha-apiserver-endpoint-reconciler-type`
    • configmap
    • default
-| 1.10 | beta |
  • Turn on the `configmap` type by default
+| 1.9 | alpha |
  • Add a new reconciler
  • Add a command-line type `--alpha-apiserver-endpoint-reconciler-type`
    • storage
    • default
+| 1.10 | beta |
  • Turn on the `storage` type by default
| 1.11 | stable |
  • Remove code for old reconciler
  • Remove --apiserver-count
-The MasterCountEndpointReconciler does not meet the current needs for -durability of API Endpoint creation, deletion, or failure cases. - -Create a new `MasterEndpointConfigMapReconciler` within -master/controller.go. - -Add a `kube-apiserver-config` ConfigMap in the `kube-system` -namespace. The duration found within the Coordination map would be configurable by -admins without a recompile. The ConfigMap would include the following: - -```go -KubeApiServerConfigMap{ - "expiration-duration": "1m", - ... [other flags potentially] -} -``` - -The Coordination ConfigMap would be formed such that: - -```go -CoordinationConfigMap{ - "[namespace]/[name]": "[KubeAPIServerEndpoint]", -} -``` - -*Key Format*: ip-[IP String Formatted]-[port] - -```go -KubeAPIServerEndpointConfigMap{ - "master-count": "0", - "ip-2001-4860-4860--8888-443": "time.Time", - "ip-192-168-0-3-443": "time.Time", -} -``` - -***TODO: should it a serialized struct, or the raw time value?*** - -The reconcile loop will expire endpoints that do not meet the duration. -On each reconcile loop (the loop runs every 10 seconds currently, -but interval will be changed to 80% of the `expiration-duration`): - -1. GET `kube-apiserver-endpoints` ConfigMap (as endpointMap) -1. Update the timestamp for the currently running API server -1. Remove all endpoints where the timestamp is greater than `expiration-duration` from the configMap. -1. Do a GET on the `kube-apiserver-endpoints` -1. Construct a PATCH operation with the deleted endpoints +The MasterCountEndpointReconciler does not meet the current needs for durability +of API Endpoint creation, deletion, or failure cases. -configmap.yml: +Custom Resource Definitions and ConfigMaps were proposed, but since they are +watched globally, liveliness updates could be overly chatty. -```yaml -kind: ConfigMap -apiVersion: v1 -metadata: - name: kube-apiserver-endpoints - namespace: default -``` +By porting OpenShift's +[LeaseEndpointReconciler](https://github.com/openshift/origin/blob/master/pkg/cmd/server/election/lease_endpoint_reconciler.go) +to Kubernetes we can use use the Storage API directly to store Endpoints +dynamically within the system. ### Alternate Proposals -#### Custom Resource Definitions +#### Custom Resource Definitions and ConfigMaps -CRD's were considered for this proposal, but were not proposed due to -constraints of having CRDs within core has not been clearly defined. -Layering could also be an issue, so the proposal defined ConfigMaps as -the current path forward. +CRD's and ConfigMaps were considered for this proposal. They were not adopted +for this proposal by the community due to tecnical issues explained earlier. #### Refactor Old Reconciler -- cgit v1.2.3 From 3cf30ed472ca666295ba06dcd0840b317d61ee31 Mon Sep 17 00:00:00 2001 From: Ryan Phillips Date: Mon, 28 Aug 2017 11:06:43 -0500 Subject: include Xiang's comments --- contributors/design-proposals/apiserver-count-fix.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/contributors/design-proposals/apiserver-count-fix.md b/contributors/design-proposals/apiserver-count-fix.md index a9150914..0b17772f 100644 --- a/contributors/design-proposals/apiserver-count-fix.md +++ b/contributors/design-proposals/apiserver-count-fix.md @@ -57,8 +57,12 @@ Endpoints would never become consistent. The MasterCountEndpointReconciler does not meet the current needs for durability of API Endpoint creation, deletion, or failure cases. -Custom Resource Definitions and ConfigMaps were proposed, but since they are -watched globally, liveliness updates could be overly chatty. +Custom Resource Definitions were proposed, but they do not have clean layering. +Additionally, liveness and locking would be a nice to have feature for a long +term solution. + +ConfigMaps were proposed, but since they are watched globally, liveliness +updates could be overly chatty. By porting OpenShift's [LeaseEndpointReconciler](https://github.com/openshift/origin/blob/master/pkg/cmd/server/election/lease_endpoint_reconciler.go) -- cgit v1.2.3