From baf0dcb0bdae7734302cb8e86bf819e910e3ee70 Mon Sep 17 00:00:00 2001 From: Jan Safranek Date: Tue, 1 Aug 2017 12:46:13 +0200 Subject: Redesign mount propagation --- contributors/design-proposals/propagation.md | 55 ++++++++++++++++++++++++---- 1 file changed, 47 insertions(+), 8 deletions(-) diff --git a/contributors/design-proposals/propagation.md b/contributors/design-proposals/propagation.md index 03d4124c..6bfc595f 100644 --- a/contributors/design-proposals/propagation.md +++ b/contributors/design-proposals/propagation.md @@ -65,6 +65,12 @@ and references to network namespaces persist. The new `VolumeMount` will look like: ```go +const ( + PropagationRShared PropagationMode = "RShared" + PropagationRSlave PropagationMode = "RSlave" + PropagationPrivate PropagationMode = "Private" +) + type VolumeMount struct { // Required: This must match the Name of a Volume [above]. Name string `json:"name"` @@ -73,13 +79,18 @@ type VolumeMount struct { // Required. MountPath string `json:"mountPath"` // Optional. - Propagation string `json:"propagation"` + Propagation PropagationMode `json:"propagation,omitempty"` } ``` +Default would be `RSlave`, which should not break backward compatibility, +`RShared` must be explicitly requested. + Opinion against this: -1. This will affect all volumes, while only HostPath need this. +1. This will affect all volumes, while only HostPath need this. It could be +checked during validation and any non-HostPath volumes with non-default +propagation could be rejected. 1. This need API change, which is discouraged. @@ -89,8 +100,8 @@ The new `HostPathVolumeSource` will look like: ```go const ( - PropagationShared PropagationMode = "Shared" - PropagationSlave PropagationMode = "Slave" + PropagationRShared PropagationMode = "RShared" + PropagationRSlave PropagationMode = "RSlave" PropagationPrivate PropagationMode = "Private" ) @@ -142,13 +153,41 @@ distros. 1. (From @euank) Changing those mountflags may make docker even less stable, this may lock up kernel accidentally or potentially leak mounts. +1. (From @jsafrane) Typical container that needs to mount something needs to +see host's `/dev` and `/sys` as HostPath volumes. This would make them shared +without any way to opt-out. Docker creates a new `/dev/shm` in the +container, which gets propagated to the host, shadowing host's `/dev/shm`. +Similarly, systemd running in a container is very picky about `/sys/fs/cgroup` +and something prevents it from starting if `/sys` is shared. ## Decision -We will take 'Make HostPath shared for privileged containers, slave for -non-privileged', an environment check and an WARNING log will be emitted about -whether propagation mode is supported. - +* We will take 'Add an option in VolumeMount API' (with an annotation during + alpha instead of real VolumeMount field): + * With validation that it can be used only with HostPath volumes. + * With validation that shared propagation can be used only in privileged + containers. + * kubernetes/kubernetes#46444 +* Kubelet will make sure that at least `/var/lib/kubelet` can be share-able into + containers and it will refuse to start if it's unsuccessful + * kubernetes/kubernetes#45724 +* Kubelet's Docker shim layer will check that it is able to run a container with + shared mount propagation on `/var/lib/kubelet` during startup and refuse to + start otherwise. This ensures that both Docker and kubelet see the same + `/var/lib/kubelet` and it can be shared into containers. + E.g. Google COS-58 runs Docker in a separate mount namespace with slave + propagation and thus can't run a container with shared propagation on + anything. Other container engines should follow the suit. +* Node conformance suite will check that mount propagation in /var/lib/kubelet + works. +* During alpha, all the behavior above must be explicitly enabled by + `kubelet --experimental-enable-mount-propagation` + It will be used only for testing of volume plugins in e2e tests and + it will be marked as deprecated from the beginning. + Developers / testers can enable it in their clusters manually. + Mount propagation may be redesigned or even removed in any future release. +* The default mount propagation will be `rslave`, which is different to current + `private`. Extensive testing is needed! ## Extra Concerns -- cgit v1.2.3 From 67cd5443727ed90b98bcc2dfb5d7e46000c4a671 Mon Sep 17 00:00:00 2001 From: Jan Safranek Date: Tue, 1 Aug 2017 12:46:34 +0200 Subject: [squash] Add feature gate --- contributors/design-proposals/propagation.md | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/contributors/design-proposals/propagation.md b/contributors/design-proposals/propagation.md index 6bfc595f..6bd880e2 100644 --- a/contributors/design-proposals/propagation.md +++ b/contributors/design-proposals/propagation.md @@ -162,8 +162,8 @@ and something prevents it from starting if `/sys` is shared. ## Decision -* We will take 'Add an option in VolumeMount API' (with an annotation during - alpha instead of real VolumeMount field): +* We will take 'Add an option in VolumeMount API' + * With an alpha feature gate in 1.8. * With validation that it can be used only with HostPath volumes. * With validation that shared propagation can be used only in privileged containers. @@ -181,13 +181,11 @@ and something prevents it from starting if `/sys` is shared. * Node conformance suite will check that mount propagation in /var/lib/kubelet works. * During alpha, all the behavior above must be explicitly enabled by - `kubelet --experimental-enable-mount-propagation` + `kubelet --feature-gates=MountPropagation=true` It will be used only for testing of volume plugins in e2e tests and - it will be marked as deprecated from the beginning. - Developers / testers can enable it in their clusters manually. Mount propagation may be redesigned or even removed in any future release. -* The default mount propagation will be `rslave`, which is different to current - `private`. Extensive testing is needed! + * When the feature is enabled the default mount propagation will be `rslave`, + which is different to current `private`. Extensive testing is needed! ## Extra Concerns @@ -215,7 +213,7 @@ if a pod does not create any new mountpoints under its hostpath bindmount, it's not hard to reach multiplicative explosions with shared bindmounts and so the change in default + no cleanup could result in existing workloads knocking the node over. - + These concerns are valid and we decide to limit the propagation mode to HostPath volume only, in HostPath, we expect any runtime should NOT perform any additional actions (such as clean up). This behavior is also consistent with current HostPath -- cgit v1.2.3 From fb3e2f97ddfc86f47105e113ad46ee9764368ea6 Mon Sep 17 00:00:00 2001 From: Jan Safranek Date: Thu, 3 Aug 2017 16:42:50 +0200 Subject: Move Propagation to HostPathVolumeSource --- contributors/design-proposals/propagation.md | 34 ++++++++++++++++++---------- 1 file changed, 22 insertions(+), 12 deletions(-) diff --git a/contributors/design-proposals/propagation.md b/contributors/design-proposals/propagation.md index 6bd880e2..352b40b8 100644 --- a/contributors/design-proposals/propagation.md +++ b/contributors/design-proposals/propagation.md @@ -66,9 +66,9 @@ The new `VolumeMount` will look like: ```go const ( - PropagationRShared PropagationMode = "RShared" - PropagationRSlave PropagationMode = "RSlave" - PropagationPrivate PropagationMode = "Private" + MountPropagationRShared MountPropagationMode = "RShared" + MountPropagationRSlave MountPropagationMode = "RSlave" + MountPropagationPrivate MountPropagationMode = "Private" ) type VolumeMount struct { @@ -100,18 +100,23 @@ The new `HostPathVolumeSource` will look like: ```go const ( - PropagationRShared PropagationMode = "RShared" - PropagationRSlave PropagationMode = "RSlave" - PropagationPrivate PropagationMode = "Private" + MountPropagationRShared MountPropagationMode = "RShared" + MountPropagationRSlave MountPropagationMode = "RSlave" + MountPropagationPrivate MountPropagationMode = "Private" ) type HostPathVolumeSource struct { Path string `json:"path"` // Mount the host path with propagation mode specified. Docker only. - Propagation PropagationMode `json:"propagation,omitempty"` + MountPropagation MountPropagationMode `json:"propagation,omitempty"` } ``` +The default mount propagation is `rslave`. Any HostPath can ask for `private`. +Only privileged containers can use HostPath with `rshared` mount propagation - +kubelet silently downgrades the propagation to `rslave` when running `rshared` +HostPath in a non-privileged container. + Opinion against this: 1. This need API change, which is discouraged. @@ -120,6 +125,12 @@ Opinion against this: 1. (From @jonboulle) May cause cross-runtime compatibility issue. +1. It's not possible to validate a pod + mount propagation. Mount propagation + is stored in a HostPath PersistentVolume object, while privileged mode is + stored in Pod object. Validator sees only one object and we don't do + cross-object validation and can't reject non-provileged pod that uses a PV + with shared mount propagation. + ### Make HostPath shared for privileged containers, slave for non-privileged. Given only HostPath needs this feature, and CAP_SYS_ADMIN access is needed when @@ -162,12 +173,11 @@ and something prevents it from starting if `/sys` is shared. ## Decision -* We will take 'Add an option in VolumeMount API' +* We will take 'Add an option in HostPathVolumeSource API' * With an alpha feature gate in 1.8. - * With validation that it can be used only with HostPath volumes. - * With validation that shared propagation can be used only in privileged - containers. - * kubernetes/kubernetes#46444 + * Only privileged containers can use `rshared` mount propagation. + * When non-privileged container uses `rshared` HostPath, it silently + downgrades it to `rslave`. * Kubelet will make sure that at least `/var/lib/kubelet` can be share-able into containers and it will refuse to start if it's unsuccessful * kubernetes/kubernetes#45724 -- cgit v1.2.3 From 99c31f9cb88bd1c7759644c2d59a9e1df2e91525 Mon Sep 17 00:00:00 2001 From: Jan Safranek Date: Thu, 10 Aug 2017 14:16:48 +0200 Subject: Rename fields and enhance their documentation --- contributors/design-proposals/propagation.md | 82 +++++++++++++++++++++------- 1 file changed, 61 insertions(+), 21 deletions(-) diff --git a/contributors/design-proposals/propagation.md b/contributors/design-proposals/propagation.md index 352b40b8..3473790e 100644 --- a/contributors/design-proposals/propagation.md +++ b/contributors/design-proposals/propagation.md @@ -65,10 +65,25 @@ and references to network namespaces persist. The new `VolumeMount` will look like: ```go +type MountPropagationMode string + const ( - MountPropagationRShared MountPropagationMode = "RShared" - MountPropagationRSlave MountPropagationMode = "RSlave" - MountPropagationPrivate MountPropagationMode = "Private" + // MountPropagationPrivate means that the volume in a container won't receive + // new mounts from host and also no mounts in this volume will be propagated + // to the host. + MountPropagationPrivate MountPropagationMode = "private" + // MountPropagationSlave means that the volume in a container will receive + // new mounts from the host or other containers, but its own mounts won't + // be propagated to the host or other containers. + // Note that this mode is recursively applied to all mounts in the volume + // ("rslave" in Linux terminology) + MountPropagationSlave MountPropagationMode = "slave" + // MountPropagationShared means that the volume in a container will receive + // new mounts from the host or other containers, and its own mounts will + // be propagated from the container to the host or other containers. + // Note that this mode is recursively applied to all mounts in the volume + // ("rshared" in Linux terminology) + MountPropagationShared MountPropagationMode = "shared" ) type VolumeMount struct { @@ -78,13 +93,18 @@ type VolumeMount struct { ReadOnly bool `json:"readOnly,omitempty"` // Required. MountPath string `json:"mountPath"` + // mountPropagation is the propagation mode how is the directory + // made available to a container in a pod. See description of individual + // modes for details. + // This field is alpha in 1.8 and can be reworked or removed in a future + // release. // Optional. - Propagation PropagationMode `json:"propagation,omitempty"` + MountPropagation MountPropagationMode `json:"mountPropagation,omitempty"` } ``` -Default would be `RSlave`, which should not break backward compatibility, -`RShared` must be explicitly requested. +Default would be `Slave`, which should not break backward compatibility, +`Shared` must be explicitly requested. Opinion against this: @@ -99,22 +119,42 @@ propagation could be rejected. The new `HostPathVolumeSource` will look like: ```go +type MountPropagationMode string + const ( - MountPropagationRShared MountPropagationMode = "RShared" - MountPropagationRSlave MountPropagationMode = "RSlave" - MountPropagationPrivate MountPropagationMode = "Private" + // MountPropagationPrivate means that the volume in a container won't receive + // new mounts from host and also no mounts in this volume will be propagated + // to the host. + MountPropagationPrivate MountPropagationMode = "private" + // MountPropagationSlave means that the volume in a container will receive + // new mounts from the host or other containers, but its own mounts won't + // be propagated to the host or other containers. + // Note that this mode is recursively applied to all mounts in the volume + // ("rslave" in Linux terminology) + MountPropagationSlave MountPropagationMode = "slave" + // MountPropagationShared means that the volume in a container will receive + // new mounts from the host or other containers, and its own mounts will + // be propagated from the container to the host or other containers. + // Note that this mode is recursively applied to all mounts in the volume + // ("rshared" in Linux terminology) + MountPropagationShared MountPropagationMode = "shared" ) type HostPathVolumeSource struct { Path string `json:"path"` - // Mount the host path with propagation mode specified. Docker only. - MountPropagation MountPropagationMode `json:"propagation,omitempty"` + // mountPropagation is the propagation mode how is the directory + // made available to a container in a pod. See description of individual + // modes for details. + // This field is alpha in 1.8 and can be reworked or removed in a future + // release. + // Optional. + MountPropagation MountPropagationMode `json:"mountPropagation,omitempty"` } ``` -The default mount propagation is `rslave`. Any HostPath can ask for `private`. -Only privileged containers can use HostPath with `rshared` mount propagation - -kubelet silently downgrades the propagation to `rslave` when running `rshared` +The default mount propagation is `slave`. Any HostPath can ask for `private`. +Only privileged containers can use HostPath with `shared` mount propagation - +kubelet silently downgrades the propagation to `slave` when running `shared` HostPath in a non-privileged container. Opinion against this: @@ -140,14 +180,14 @@ privileged, or we can introduce a new option in SecurityContext to control this. The propagation mode could be determined by the following logic: ```go -// Environment check to ensure "rshared" is supported. +// Environment check to ensure "shared" is supported. if !dockerNewerThanV110 || !mountPathIsShared { return "" } if container.SecurityContext.Privileged { - return "rshared" + return "shared" } else { - return "rslave" + return "slave" } ``` @@ -175,9 +215,9 @@ and something prevents it from starting if `/sys` is shared. * We will take 'Add an option in HostPathVolumeSource API' * With an alpha feature gate in 1.8. - * Only privileged containers can use `rshared` mount propagation. - * When non-privileged container uses `rshared` HostPath, it silently - downgrades it to `rslave`. + * Only privileged containers can use `shared` mount propagation. + * When non-privileged container uses `shared` HostPath, it silently + downgrades it to `slave`. * Kubelet will make sure that at least `/var/lib/kubelet` can be share-able into containers and it will refuse to start if it's unsuccessful * kubernetes/kubernetes#45724 @@ -194,7 +234,7 @@ and something prevents it from starting if `/sys` is shared. `kubelet --feature-gates=MountPropagation=true` It will be used only for testing of volume plugins in e2e tests and Mount propagation may be redesigned or even removed in any future release. - * When the feature is enabled the default mount propagation will be `rslave`, + * When the feature is enabled the default mount propagation will be `slave`, which is different to current `private`. Extensive testing is needed! ## Extra Concerns -- cgit v1.2.3 From 7a2b2f7d88b50ee7e83dcca79dc96fac96988084 Mon Sep 17 00:00:00 2001 From: Jan Safranek Date: Tue, 15 Aug 2017 09:50:22 +0200 Subject: Final version --- contributors/design-proposals/propagation.md | 161 +++++++++++++++++---------- 1 file changed, 101 insertions(+), 60 deletions(-) diff --git a/contributors/design-proposals/propagation.md b/contributors/design-proposals/propagation.md index 3473790e..09f68edd 100644 --- a/contributors/design-proposals/propagation.md +++ b/contributors/design-proposals/propagation.md @@ -57,6 +57,16 @@ survive we need to bind mount a dir from host mount namespace to container one with shared flag, so that all bind mounts are propagated across mount namespaces and references to network namespaces persist. +1. (From https://github.com/kubernetes/kubernetes/issues/46643) I expect the + container to start and any fuse mounts it creates in a volume that exists on + other containers in the pod (that are using :slave) are available to those + other containers. + + In other words, two containers in the same pod share an EmptyDir. One + container mounts something in it and the other one can see it. The first + container must have (r)shared mount propagation to the EmptyDir, the second + one can have (r)slave. + ## Implementation Alternatives @@ -68,22 +78,19 @@ The new `VolumeMount` will look like: type MountPropagationMode string const ( - // MountPropagationPrivate means that the volume in a container won't receive - // new mounts from host and also no mounts in this volume will be propagated - // to the host. - MountPropagationPrivate MountPropagationMode = "private" - // MountPropagationSlave means that the volume in a container will receive - // new mounts from the host or other containers, but its own mounts won't - // be propagated to the host or other containers. + // MountPropagationHostToContainer means that the volume in a container will + // receive new mounts from the host or other containers, but filesystems + // mounted inside the container won't be propagated to the host or other + // containers. // Note that this mode is recursively applied to all mounts in the volume - // ("rslave" in Linux terminology) - MountPropagationSlave MountPropagationMode = "slave" - // MountPropagationShared means that the volume in a container will receive - // new mounts from the host or other containers, and its own mounts will - // be propagated from the container to the host or other containers. + // ("rslave" in Linux terminology). + MountPropagationHostToContainer MountPropagationMode = "HostToContainer" + // MountPropagationBidirectional means that the volume in a container will + // receive new mounts from the host or other containers, and its own mounts + // will be propagated from the container to the host or other containers. // Note that this mode is recursively applied to all mounts in the volume - // ("rshared" in Linux terminology) - MountPropagationShared MountPropagationMode = "shared" + // ("rshared" in Linux terminology). + MountPropagationBidirectional MountPropagationMode = "Bidirectional" ) type VolumeMount struct { @@ -93,18 +100,24 @@ type VolumeMount struct { ReadOnly bool `json:"readOnly,omitempty"` // Required. MountPath string `json:"mountPath"` - // mountPropagation is the propagation mode how is the directory - // made available to a container in a pod. See description of individual - // modes for details. + // mountPropagation is the mode how are mounts in the volume propagated from + // the host to the container and from the container to the host. + // When not set, MountPropagationHostToContainer is used. // This field is alpha in 1.8 and can be reworked or removed in a future // release. // Optional. - MountPropagation MountPropagationMode `json:"mountPropagation,omitempty"` + MountPropagation *MountPropagationMode `json:"mountPropagation,omitempty"` } ``` -Default would be `Slave`, which should not break backward compatibility, -`Shared` must be explicitly requested. +Default would be `HostToContainer`, i.e. `rslave`, which should not break +backward compatibility, `Bidirectional` must be explicitly requested. +Using enum instead of simple `PropagateMounts bool` allows us to extend the +modes to `private` or non-recursive `shared` and `slave` if we need so in +future. + +Only privileged containers are allowed to use `Bidirectional` for their volumes. +This will be enforced during validation. Opinion against this: @@ -122,40 +135,45 @@ The new `HostPathVolumeSource` will look like: type MountPropagationMode string const ( - // MountPropagationPrivate means that the volume in a container won't receive - // new mounts from host and also no mounts in this volume will be propagated - // to the host. - MountPropagationPrivate MountPropagationMode = "private" - // MountPropagationSlave means that the volume in a container will receive - // new mounts from the host or other containers, but its own mounts won't - // be propagated to the host or other containers. + // MountPropagationHostToContainer means that the volume in a container will + // receive new mounts from the host or other containers, but filesystems + // mounted inside the container won't be propagated to the host or other + // containers. // Note that this mode is recursively applied to all mounts in the volume - // ("rslave" in Linux terminology) - MountPropagationSlave MountPropagationMode = "slave" - // MountPropagationShared means that the volume in a container will receive - // new mounts from the host or other containers, and its own mounts will - // be propagated from the container to the host or other containers. + // ("rslave" in Linux terminology). + MountPropagationHostToContainer MountPropagationMode = "HostToContainer" + // MountPropagationBidirectional means that the volume in a container will + // receive new mounts from the host or other containers, and its own mounts + // will be propagated from the container to the host or other containers. // Note that this mode is recursively applied to all mounts in the volume - // ("rshared" in Linux terminology) - MountPropagationShared MountPropagationMode = "shared" + // ("rshared" in Linux terminology). + MountPropagationBidirectional MountPropagationMode = "Bidirectional" ) type HostPathVolumeSource struct { Path string `json:"path"` - // mountPropagation is the propagation mode how is the directory - // made available to a container in a pod. See description of individual - // modes for details. + // mountPropagation is the mode how are mounts in the volume propagated from + // the host to the container and from the container to the host. + // When not set, MountPropagationHostToContainer is used. // This field is alpha in 1.8 and can be reworked or removed in a future // release. // Optional. - MountPropagation MountPropagationMode `json:"mountPropagation,omitempty"` + MountPropagation *MountPropagationMode `json:"mountPropagation,omitempty"` } ``` -The default mount propagation is `slave`. Any HostPath can ask for `private`. -Only privileged containers can use HostPath with `shared` mount propagation - -kubelet silently downgrades the propagation to `slave` when running `shared` -HostPath in a non-privileged container. +Default would be `HostToContainer`, i.e. `rslave`, which should not break +backward compatibility, `Bidirectional` must be explicitly requested. +Using enum instead of simple `PropagateMounts bool` allows us to extend the +modes to `private` or non-recursive `shared` and `slave` if we need so in +future. + +Only privileged containers can use HostPath with `Bidirectional` mount +propagation - kubelet silently downgrades the propagation to `HostToContainer` +when running `Bidirectional` HostPath in a non-privileged container. This allows +us to use the same `HostPathVolumeSource` in a pod with two containers, one +non-privileged with `HostToContainer` propagation and second privileged with +`Bidirectional` that mounts stuff for the first one. Opinion against this: @@ -163,6 +181,9 @@ Opinion against this: 1. All containers use this volume will share the same propagation mode. +1. Silent downgrade from `Bidirectional` to `HostToContainer` for non-privileged + containers. + 1. (From @jonboulle) May cause cross-runtime compatibility issue. 1. It's not possible to validate a pod + mount propagation. Mount propagation @@ -213,29 +234,49 @@ and something prevents it from starting if `/sys` is shared. ## Decision -* We will take 'Add an option in HostPathVolumeSource API' +* We will take 'Add an option in VolumeMount API' * With an alpha feature gate in 1.8. - * Only privileged containers can use `shared` mount propagation. - * When non-privileged container uses `shared` HostPath, it silently - downgrades it to `slave`. -* Kubelet will make sure that at least `/var/lib/kubelet` can be share-able into - containers and it will refuse to start if it's unsuccessful - * kubernetes/kubernetes#45724 -* Kubelet's Docker shim layer will check that it is able to run a container with - shared mount propagation on `/var/lib/kubelet` during startup and refuse to - start otherwise. This ensures that both Docker and kubelet see the same - `/var/lib/kubelet` and it can be shared into containers. - E.g. Google COS-58 runs Docker in a separate mount namespace with slave - propagation and thus can't run a container with shared propagation on - anything. Other container engines should follow the suit. -* Node conformance suite will check that mount propagation in /var/lib/kubelet - works. + * Only privileged containers can use `rshared` (`Bidirectional`) mount + propagation (with a validator). + * During alpha, all the behavior above must be explicitly enabled by `kubelet --feature-gates=MountPropagation=true` It will be used only for testing of volume plugins in e2e tests and Mount propagation may be redesigned or even removed in any future release. - * When the feature is enabled the default mount propagation will be `slave`, - which is different to current `private`. Extensive testing is needed! + + When the feature is enabled: + + * The default mount propagation of **all** volumes (incl. GCE, AWS, Cinder, + Gluster, Flex, ...) will be `slave`, which is different to current + `private`. Extensive testing is needed! We may restrict it to HostPath + + EmptyDir in Beta. + + * **Any** volume in a privileged container can be `Bidirectional`. We may + restrict it to HostPath + EmptyDir in Beta. + + * Kubelet's Docker shim layer will check that it is able to run a container + with shared mount propagation on `/var/lib/kubelet` during startup and log + a warning otherwise. This ensures that both Docker and kubelet see the same + `/var/lib/kubelet` and it can be shared into containers. + E.g. Google COS-58 runs Docker in a separate mount namespace with slave + propagation and thus can't run a container with shared propagation on + anything. + + This will be done via simple docker version check (1.13 is required) when + the feature gate is enabled. + + * Node conformance suite will check that mount propagation in /var/lib/kubelet + works. + + * When running on a distro with `private` as default mount propagation + (probably anything that does not run systemd, such as Debian Wheezy), + Kubelet will make `/var/lib/kubelet` share-able into containers and it will + refuse to start if it's unsuccessful. + + It sounds complicated, but it's simple + `mount --bind --rshared /var/lib/kubelet /var/lib/kubelet`. See + kubernetes/kubernetes#45724 + ## Extra Concerns -- cgit v1.2.3