From 216e4d444b202c183949f74fe92f1e7f5c5e7a0c Mon Sep 17 00:00:00 2001 From: Jing Xu Date: Tue, 7 Aug 2018 16:39:04 -0700 Subject: Add data source for volume operations propose to add data source for volume operations including provisoning volume from snapshot, and cloning volumes --- contributors/design-proposals/storage/data-source.md | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 contributors/design-proposals/storage/data-source.md diff --git a/contributors/design-proposals/storage/data-source.md b/contributors/design-proposals/storage/data-source.md new file mode 100644 index 00000000..108502d2 --- /dev/null +++ b/contributors/design-proposals/storage/data-source.md @@ -0,0 +1,10 @@ +# Add DataSource for Volume Operations + +## Goal + + +## Design + + +## Use cases + -- cgit v1.2.3 From 0bdaa171386b65dc30b56eb8ef7df4e474d4dab6 Mon Sep 17 00:00:00 2001 From: Jing Xu Date: Tue, 7 Aug 2018 16:49:11 -0700 Subject: Update data-source.md --- .../design-proposals/storage/data-source.md | 80 +++++++++++++++++++++- 1 file changed, 79 insertions(+), 1 deletion(-) diff --git a/contributors/design-proposals/storage/data-source.md b/contributors/design-proposals/storage/data-source.md index 108502d2..d3f5038d 100644 --- a/contributors/design-proposals/storage/data-source.md +++ b/contributors/design-proposals/storage/data-source.md @@ -1,10 +1,88 @@ # Add DataSource for Volume Operations -## Goal +Note: this proposal is part of [Volume Snapshot](https://github.com/kubernetes/community/pull/2335) feature design, and also relevant to recently proposed Volume Clone feature. +## Goal +In Volume Snapshot proposal, a snapshot is now represented as first-class CRD objects and an external snapshot controller is responsible for managing its lifecycle. With Snapshot API available, users could provision volumes from snapshot and data will be prepopulated to the volumes. Also considering clone and other possible storage operations, there could be many different types of sources used for populating the data to the volumes. In this proposal, we add a general "DataSource" which could be used to represent different types of data sources. ## Design +The following are the APIs we propose to add + +``` + +type PersistentVolumeClaimSpec struct { + // If specified, volume will be prepopulated with data from the DataSource. + // +optional + DataSource *TypedLocalObjectReference `json:"dataSource" protobuf:"bytes,2,opt,name=dataSource"` +} + +// TypedLocalObjectReference contains enough information to let you locate the referenced object inside the same namespace. +type TypedLocalObjectReference struct { + // Name of the object reference. + Name string + // Kind indicates the type of the object reference. + Kind string +} +``` ## Use cases +* Use snapshot to backup data: Alice wants to take a snapshot of his Mongo database, and accidentally delete her tables, she wants to restore her volumes from the snapshot. +To create a snapshot for a volume (represented by PVC), use the snapshot.yaml + +``` +apiVersion: snapshot.storage.k8s.io/v1alpha1 +kind: VolumeSnapshot +metadata: + name: snapshot-pd-1 + namespace: myns +spec: + source: + Kind: PersistentVolumeClaim + Name: podpvc + snapshotClassName: snapshot-class + + ``` + After snapshot is ready, create a new volume from the snapshot + +``` +kind: PersistentVolumeClaim +apiVersion: v1 +metadata: + name: snapshot-pvc + Namespace: myns +spec: + accessModes: + - ReadWriteOnce + storageClassName: csi-gce-pd + dataSource: + type: VolumeSnapshot + name: snapshot-pd-1 + resources: + requests: + storage: 6Gi + +``` + +* Clone volume: Bob want to copy the data from one volume to another by cloning the volume. + +``` +kind: PersistentVolumeClaim +apiVersion: v1 +metadata: + name: clone-pvc + Namespace: myns +spec: + accessModes: + - ReadWriteOnce + storageClassName: csi-gce-pd + dataSource: + type: PersistentVolumeClaim + name: pvc-1 + resources: + requests: + storage: 10Gi + +``` + -- cgit v1.2.3 From 4c89dc52fc8673306375413e1697cfc20c150b5f Mon Sep 17 00:00:00 2001 From: Jing Xu Date: Thu, 9 Aug 2018 11:24:41 -0700 Subject: Update data-source.md --- .../design-proposals/storage/data-source.md | 36 +++++++++++++++++++--- 1 file changed, 32 insertions(+), 4 deletions(-) diff --git a/contributors/design-proposals/storage/data-source.md b/contributors/design-proposals/storage/data-source.md index d3f5038d..f21cf468 100644 --- a/contributors/design-proposals/storage/data-source.md +++ b/contributors/design-proposals/storage/data-source.md @@ -6,7 +6,9 @@ Note: this proposal is part of [Volume Snapshot](https://github.com/kubernetes/c In Volume Snapshot proposal, a snapshot is now represented as first-class CRD objects and an external snapshot controller is responsible for managing its lifecycle. With Snapshot API available, users could provision volumes from snapshot and data will be prepopulated to the volumes. Also considering clone and other possible storage operations, there could be many different types of sources used for populating the data to the volumes. In this proposal, we add a general "DataSource" which could be used to represent different types of data sources. ## Design -The following are the APIs we propose to add +A new DataSource field is proposed to add to both PVC and PV to represent the source of the data which is prepopulated to the provisioned volume. If an external-provisioner does not understand the new DataSource field and cannot populate the data to the volume, PV/PVC controller should be able to detect that by comparing DataSource field in PV and PVC (i.e., PVC has DataSource but PV does not) and fail the operation. + +For DataSource, we propose to define a new type “TypedLocalObjectReference”. It is similar to “LocalObjectReference” type with additional Kind field in order to support multiple data source types. In the alpha version, this data source is restricted in the same namespace of the PVC. The following are the APIs we propose to add. ``` @@ -16,6 +18,12 @@ type PersistentVolumeClaimSpec struct { DataSource *TypedLocalObjectReference `json:"dataSource" protobuf:"bytes,2,opt,name=dataSource"` } +type PersistentVolumeSpec struct { + // If specified, volume will be prepopulated with data from the PVCDataSourceRef. + // +optional + DataSourceRef *ypedLocalObjectReference `json:"dataSourceRef" protobuf:"bytes,2,opt,name=dataSourceRef"` +} + // TypedLocalObjectReference contains enough information to let you locate the referenced object inside the same namespace. type TypedLocalObjectReference struct { // Name of the object reference. @@ -61,7 +69,6 @@ spec: resources: requests: storage: 6Gi - ``` * Clone volume: Bob want to copy the data from one volume to another by cloning the volume. @@ -81,8 +88,29 @@ spec: name: pvc-1 resources: requests: - storage: 10Gi - + storage: 10Gi ``` +* Import data from Github repo: Alice want to import data from a github repo to her volume. The github repo is represented by a PVC (gitrepo-1). Compare with the user case 2 is that the data source should be the same kind of volume as the provisioned volume for cloning. + +``` +kind: PersistentVolumeClaim +apiVersion: v1 +metadata: + name: clone-pvc + Namespace: myns +spec: + accessModes: + - ReadWriteOnce + storageClassName: csi-gce-pd + dataSource: + kind: PersistentVolumeClaim + name: gitrepo-1 + resources: + requests: + storage: 100Gi +``` + + + -- cgit v1.2.3 From 29c02bc1d61247f039de39eacaccfb4356f8d534 Mon Sep 17 00:00:00 2001 From: Jing Xu Date: Fri, 10 Aug 2018 11:19:28 -0700 Subject: Update data-source.md --- contributors/design-proposals/storage/data-source.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/contributors/design-proposals/storage/data-source.md b/contributors/design-proposals/storage/data-source.md index f21cf468..d5feba0e 100644 --- a/contributors/design-proposals/storage/data-source.md +++ b/contributors/design-proposals/storage/data-source.md @@ -13,15 +13,15 @@ For DataSource, we propose to define a new type “TypedLocalObjectReference”. ``` type PersistentVolumeClaimSpec struct { - // If specified, volume will be prepopulated with data from the DataSource. + // If specified, volume will be pre-populated with data from the specified data source. // +optional DataSource *TypedLocalObjectReference `json:"dataSource" protobuf:"bytes,2,opt,name=dataSource"` } type PersistentVolumeSpec struct { - // If specified, volume will be prepopulated with data from the PVCDataSourceRef. + // If specified, volume was pre-populated with data from the specified data source. // +optional - DataSourceRef *ypedLocalObjectReference `json:"dataSourceRef" protobuf:"bytes,2,opt,name=dataSourceRef"` + DataSource *ypedLocalObjectReference `json:"dataSourceRef" protobuf:"bytes,2,opt,name=dataSourceRef"` } // TypedLocalObjectReference contains enough information to let you locate the referenced object inside the same namespace. @@ -35,7 +35,7 @@ type TypedLocalObjectReference struct { ``` ## Use cases -* Use snapshot to backup data: Alice wants to take a snapshot of his Mongo database, and accidentally delete her tables, she wants to restore her volumes from the snapshot. +* Use snapshot to backup data: Alice wants to take a snapshot of her Mongo database, and accidentally delete her tables, she wants to restore her volumes from the snapshot. To create a snapshot for a volume (represented by PVC), use the snapshot.yaml ``` @@ -46,8 +46,8 @@ metadata: namespace: myns spec: source: - Kind: PersistentVolumeClaim - Name: podpvc + kind: PersistentVolumeClaim + name: podpvc snapshotClassName: snapshot-class ``` @@ -64,7 +64,7 @@ spec: - ReadWriteOnce storageClassName: csi-gce-pd dataSource: - type: VolumeSnapshot + kind: VolumeSnapshot name: snapshot-pd-1 resources: requests: @@ -84,7 +84,7 @@ spec: - ReadWriteOnce storageClassName: csi-gce-pd dataSource: - type: PersistentVolumeClaim + kind: PersistentVolumeClaim name: pvc-1 resources: requests: -- cgit v1.2.3 From bac19f0b0dc8d0edb850fb527efabf40670b5c44 Mon Sep 17 00:00:00 2001 From: Jing Xu Date: Mon, 20 Aug 2018 11:06:07 -0700 Subject: Update data-source.md --- contributors/design-proposals/storage/data-source.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/contributors/design-proposals/storage/data-source.md b/contributors/design-proposals/storage/data-source.md index d5feba0e..64c9fc83 100644 --- a/contributors/design-proposals/storage/data-source.md +++ b/contributors/design-proposals/storage/data-source.md @@ -21,7 +21,7 @@ type PersistentVolumeClaimSpec struct { type PersistentVolumeSpec struct { // If specified, volume was pre-populated with data from the specified data source. // +optional - DataSource *ypedLocalObjectReference `json:"dataSourceRef" protobuf:"bytes,2,opt,name=dataSourceRef"` + DataSource *TypedLocalObjectReference `json:"dataSourceRef" protobuf:"bytes,2,opt,name=dataSourceRef"` } // TypedLocalObjectReference contains enough information to let you locate the referenced object inside the same namespace. @@ -43,7 +43,7 @@ apiVersion: snapshot.storage.k8s.io/v1alpha1 kind: VolumeSnapshot metadata: name: snapshot-pd-1 - namespace: myns + namespace: mynamespace spec: source: kind: PersistentVolumeClaim @@ -58,7 +58,7 @@ kind: PersistentVolumeClaim apiVersion: v1 metadata: name: snapshot-pvc - Namespace: myns + Namespace: mynamespace spec: accessModes: - ReadWriteOnce @@ -78,7 +78,7 @@ kind: PersistentVolumeClaim apiVersion: v1 metadata: name: clone-pvc - Namespace: myns + Namespace: mynamespace spec: accessModes: - ReadWriteOnce @@ -98,7 +98,7 @@ kind: PersistentVolumeClaim apiVersion: v1 metadata: name: clone-pvc - Namespace: myns + Namespace: mynamespace spec: accessModes: - ReadWriteOnce -- cgit v1.2.3 From 22021f22144d689ca11589d2f9340e4cb76310d3 Mon Sep 17 00:00:00 2001 From: Jing Xu Date: Mon, 20 Aug 2018 14:45:51 -0700 Subject: Update data-source.md --- contributors/design-proposals/storage/data-source.md | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/contributors/design-proposals/storage/data-source.md b/contributors/design-proposals/storage/data-source.md index 64c9fc83..d26bb0cf 100644 --- a/contributors/design-proposals/storage/data-source.md +++ b/contributors/design-proposals/storage/data-source.md @@ -6,7 +6,7 @@ Note: this proposal is part of [Volume Snapshot](https://github.com/kubernetes/c In Volume Snapshot proposal, a snapshot is now represented as first-class CRD objects and an external snapshot controller is responsible for managing its lifecycle. With Snapshot API available, users could provision volumes from snapshot and data will be prepopulated to the volumes. Also considering clone and other possible storage operations, there could be many different types of sources used for populating the data to the volumes. In this proposal, we add a general "DataSource" which could be used to represent different types of data sources. ## Design -A new DataSource field is proposed to add to both PVC and PV to represent the source of the data which is prepopulated to the provisioned volume. If an external-provisioner does not understand the new DataSource field and cannot populate the data to the volume, PV/PVC controller should be able to detect that by comparing DataSource field in PV and PVC (i.e., PVC has DataSource but PV does not) and fail the operation. +A new DataSource field is proposed to add to PVC to represent the source of the data which is prepopulated to the provisioned volume. If an external-provisioner does not understand the new DataSource field and cannot populate the data to the volume, there is a WIP proposal to use readiness condition to detect this situation and fail the operation. For DataSource, we propose to define a new type “TypedLocalObjectReference”. It is similar to “LocalObjectReference” type with additional Kind field in order to support multiple data source types. In the alpha version, this data source is restricted in the same namespace of the PVC. The following are the APIs we propose to add. @@ -18,12 +18,6 @@ type PersistentVolumeClaimSpec struct { DataSource *TypedLocalObjectReference `json:"dataSource" protobuf:"bytes,2,opt,name=dataSource"` } -type PersistentVolumeSpec struct { - // If specified, volume was pre-populated with data from the specified data source. - // +optional - DataSource *TypedLocalObjectReference `json:"dataSourceRef" protobuf:"bytes,2,opt,name=dataSourceRef"` -} - // TypedLocalObjectReference contains enough information to let you locate the referenced object inside the same namespace. type TypedLocalObjectReference struct { // Name of the object reference. -- cgit v1.2.3 From 095154f6cd233044f4b6363ee952aaae4b3862c6 Mon Sep 17 00:00:00 2001 From: Jing Xu Date: Mon, 20 Aug 2018 15:50:13 -0700 Subject: Update data-source.md --- .../design-proposals/storage/data-source.md | 44 ++++++++++++++++++++-- 1 file changed, 41 insertions(+), 3 deletions(-) diff --git a/contributors/design-proposals/storage/data-source.md b/contributors/design-proposals/storage/data-source.md index d26bb0cf..e68c38b5 100644 --- a/contributors/design-proposals/storage/data-source.md +++ b/contributors/design-proposals/storage/data-source.md @@ -6,9 +6,8 @@ Note: this proposal is part of [Volume Snapshot](https://github.com/kubernetes/c In Volume Snapshot proposal, a snapshot is now represented as first-class CRD objects and an external snapshot controller is responsible for managing its lifecycle. With Snapshot API available, users could provision volumes from snapshot and data will be prepopulated to the volumes. Also considering clone and other possible storage operations, there could be many different types of sources used for populating the data to the volumes. In this proposal, we add a general "DataSource" which could be used to represent different types of data sources. ## Design -A new DataSource field is proposed to add to PVC to represent the source of the data which is prepopulated to the provisioned volume. If an external-provisioner does not understand the new DataSource field and cannot populate the data to the volume, there is a WIP proposal to use readiness condition to detect this situation and fail the operation. - -For DataSource, we propose to define a new type “TypedLocalObjectReference”. It is similar to “LocalObjectReference” type with additional Kind field in order to support multiple data source types. In the alpha version, this data source is restricted in the same namespace of the PVC. The following are the APIs we propose to add. +### API Change +A new DataSource field is proposed to add to PVC to represent the source of the data which is pre-populated to the provisioned volume. For DataSource field, we propose to define a new type “TypedLocalObjectReference”. It is similar to “LocalObjectReference” type with additional Kind field in order to support multiple data source types. In the alpha version, this data source is restricted in the same namespace of the PVC. The following are the APIs we propose to add. ``` @@ -26,6 +25,45 @@ type TypedLocalObjectReference struct { Kind string } +``` +### Error Handling +If an external-provisioner does not understand the new DataSource field and cannot populate the data to the volume, we propose to add a PV condition to detect this situation and fail the operation. The idea is simlar to the design of ["Pod Ready++"]https://github.com/kubernetes/community/blob/master/keps/sig-network/0007-pod-ready%2B%2B.md + +First we introduce a condition called "VolumeDataPopulated" into PV. With the feature of snapshot and data source, volume might be created from snapshot source with data populated into the volume instead of an empty one. The external provision needs to understand the data source and also mark the condition "VolumeDataPopualted" to true. The PV controller will check this condition if the data source of PVC is specified. Only if the condition is true, the PVC can be marked as "Bound" phase. + +``` +// PersistentVolumeStatus is the current status of a persistent volume. +type PersistentVolumeStatus struct { + // Phase indicates if a volume is available, bound to a claim, or released by a claim. + // More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#phase + // +optional + Phase PersistentVolumePhase `json:"phase,omitempty" protobuf:"bytes,1,opt,name=phase,casttype=PersistentVolumePhase"` + ... + + // +optional + Conditions []PersistentVolumeCondition +} + +type PersistentVolumeConditionType string + +// These are valid conditions of Pvc +const ( + // An user trigger resize of pvc has been started + PersistentVolumeDataPopulated PersistentVolumeConditionType = "dataPopulated" +) + +type PersistentVolumeCondition struct { + Type PersistentVolumeConditionType + Status ConditionStatus + // +optional + LastProbeTime metav1.Time + // +optional + LastTransitionTime metav1.Time + // +optional + Reason string + // +optional + Message string +} ``` ## Use cases -- cgit v1.2.3 From c900a728c33bdc6752dd60d61c5369719f589f7e Mon Sep 17 00:00:00 2001 From: Jing Xu Date: Mon, 20 Aug 2018 15:55:26 -0700 Subject: Update data-source.md --- contributors/design-proposals/storage/data-source.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/contributors/design-proposals/storage/data-source.md b/contributors/design-proposals/storage/data-source.md index e68c38b5..796be7c1 100644 --- a/contributors/design-proposals/storage/data-source.md +++ b/contributors/design-proposals/storage/data-source.md @@ -27,9 +27,9 @@ type TypedLocalObjectReference struct { ``` ### Error Handling -If an external-provisioner does not understand the new DataSource field and cannot populate the data to the volume, we propose to add a PV condition to detect this situation and fail the operation. The idea is simlar to the design of ["Pod Ready++"]https://github.com/kubernetes/community/blob/master/keps/sig-network/0007-pod-ready%2B%2B.md +If an external-provisioner does not understand the new DataSource field and cannot populate the data to the volume, we propose to add a PV condition to detect this situation and fail the operation. The idea is simlar to the design of ["Pod Ready++"](https://github.com/kubernetes/community/blob/master/keps/sig-network/0007-pod-ready%2B%2B.md) -First we introduce a condition called "VolumeDataPopulated" into PV. With the feature of snapshot and data source, volume might be created from snapshot source with data populated into the volume instead of an empty one. The external provision needs to understand the data source and also mark the condition "VolumeDataPopualted" to true. The PV controller will check this condition if the data source of PVC is specified. Only if the condition is true, the PVC can be marked as "Bound" phase. +First we introduce a condition called "VolumeDataPopulated" into PV. With the feature of snapshot and data source, volume might be created from snapshot source with data populated into the volume instead of an empty one. The external provision needs to understand the data source and also mark the condition "VolumeDataPopualted" to true. The PV controller will check this condition if the data source of PVC is specified. Only if the condition is true, the PVC can be marked as "Bound" phase. In case of an old version of provisioner is in use and it cannot recognize the data source field in PVC, an empty Volume will be provisioned. In this situation, since the "VolumeDataPouplated" condition is not set to true by the provisioner, the PV controller will not mark PVC as "Bound" phase so that user cannot use this PVC yet. We can also add this condition into PVC to tell the users that the PVC's data is not populated. ``` // PersistentVolumeStatus is the current status of a persistent volume. -- cgit v1.2.3 From 12fc30dadbf700150a8553ffb764405c12c74b48 Mon Sep 17 00:00:00 2001 From: Jing Xu Date: Tue, 21 Aug 2018 15:52:01 -0700 Subject: Update data-source.md --- contributors/design-proposals/storage/data-source.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/contributors/design-proposals/storage/data-source.md b/contributors/design-proposals/storage/data-source.md index 796be7c1..fa03b553 100644 --- a/contributors/design-proposals/storage/data-source.md +++ b/contributors/design-proposals/storage/data-source.md @@ -23,6 +23,8 @@ type TypedLocalObjectReference struct { Name string // Kind indicates the type of the object reference. Kind string + // APIGroup is the group for the resource being referenced + APIGroup string } ``` @@ -46,18 +48,16 @@ type PersistentVolumeStatus struct { type PersistentVolumeConditionType string -// These are valid conditions of Pvc +// These are valid conditions of PersistentVolume const ( // An user trigger resize of pvc has been started - PersistentVolumeDataPopulated PersistentVolumeConditionType = "dataPopulated" + PersistentVolumeDataPopulated PersistentVolumeConditionType = "DataPopulated" ) type PersistentVolumeCondition struct { Type PersistentVolumeConditionType Status ConditionStatus // +optional - LastProbeTime metav1.Time - // +optional LastTransitionTime metav1.Time // +optional Reason string -- cgit v1.2.3 From 7e303357bf07a645bfb4ea597d4bcd1042cb5bb7 Mon Sep 17 00:00:00 2001 From: Jing Xu Date: Thu, 23 Aug 2018 11:39:16 -0700 Subject: Update data-source.md --- .../design-proposals/storage/data-source.md | 45 +++++----------------- 1 file changed, 9 insertions(+), 36 deletions(-) diff --git a/contributors/design-proposals/storage/data-source.md b/contributors/design-proposals/storage/data-source.md index fa03b553..5a55639f 100644 --- a/contributors/design-proposals/storage/data-source.md +++ b/contributors/design-proposals/storage/data-source.md @@ -3,7 +3,11 @@ Note: this proposal is part of [Volume Snapshot](https://github.com/kubernetes/community/pull/2335) feature design, and also relevant to recently proposed Volume Clone feature. ## Goal -In Volume Snapshot proposal, a snapshot is now represented as first-class CRD objects and an external snapshot controller is responsible for managing its lifecycle. With Snapshot API available, users could provision volumes from snapshot and data will be prepopulated to the volumes. Also considering clone and other possible storage operations, there could be many different types of sources used for populating the data to the volumes. In this proposal, we add a general "DataSource" which could be used to represent different types of data sources. +Currently in Kuberentes, volume plugin only supports to provision an empty volume. With the new storage features (including [Volume Snapshot](https://github.com/kubernetes/community/pull/2335) and [volume clone](https://github.com/erinboyd/community/blob/patch-3/contributors/design-proposals/storage/cloning.md)) being proposed, there is a need to support data population for volume provisioning. For example, volume can be created from a snapshot source, or volume could be cloned from another volume source. Depending on the sources for creating the volume, there are two scenarios +1. Volume provisioner can recoginize the source and be able to create the volume from the source directly (e.g., restore snapshot to a volume or clone volume). +2. Volume provisioner does not recognize the volume source, and create an empty volume. Another external component (data populator) could watch the volume creation and implement the logic to populate/import the data to the volume provisioned. Only after data is populated to the volume, the PVC is ready for use. + +There could be many different types of sources used for populating the data to the volumes. In this proposal, we propose to add a generic "DataSource" field to PersistentVolumeClaimSpec to represent different types of data sources. ## Design ### API Change @@ -28,43 +32,12 @@ type TypedLocalObjectReference struct { } ``` -### Error Handling -If an external-provisioner does not understand the new DataSource field and cannot populate the data to the volume, we propose to add a PV condition to detect this situation and fail the operation. The idea is simlar to the design of ["Pod Ready++"](https://github.com/kubernetes/community/blob/master/keps/sig-network/0007-pod-ready%2B%2B.md) - -First we introduce a condition called "VolumeDataPopulated" into PV. With the feature of snapshot and data source, volume might be created from snapshot source with data populated into the volume instead of an empty one. The external provision needs to understand the data source and also mark the condition "VolumeDataPopualted" to true. The PV controller will check this condition if the data source of PVC is specified. Only if the condition is true, the PVC can be marked as "Bound" phase. In case of an old version of provisioner is in use and it cannot recognize the data source field in PVC, an empty Volume will be provisioned. In this situation, since the "VolumeDataPouplated" condition is not set to true by the provisioner, the PV controller will not mark PVC as "Bound" phase so that user cannot use this PVC yet. We can also add this condition into PVC to tell the users that the PVC's data is not populated. +### Design Details +In the first Alphal version, we only support data source from Snapshot. So the expected Kind in DataSource has to be "VolumeSnapshot". In this case, provisioner should provision volume and populate data in one step. There is no need for external data populator yet. -``` -// PersistentVolumeStatus is the current status of a persistent volume. -type PersistentVolumeStatus struct { - // Phase indicates if a volume is available, bound to a claim, or released by a claim. - // More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#phase - // +optional - Phase PersistentVolumePhase `json:"phase,omitempty" protobuf:"bytes,1,opt,name=phase,casttype=PersistentVolumePhase"` - ... - - // +optional - Conditions []PersistentVolumeCondition -} +For other types of data sources that require external data populator, volume creation and data population are two seperate steps. Only when data is ready, PVC/PV can be marked as ready (Bound) so that users can start use them. We are working on a seperate proposal to address this using similar idea from ["Pod Ready++"](https://github.com/kubernetes/community/blob/master/keps/sig-network/0007-pod-ready%2B%2B.md). -type PersistentVolumeConditionType string - -// These are valid conditions of PersistentVolume -const ( - // An user trigger resize of pvc has been started - PersistentVolumeDataPopulated PersistentVolumeConditionType = "DataPopulated" -) - -type PersistentVolumeCondition struct { - Type PersistentVolumeConditionType - Status ConditionStatus - // +optional - LastTransitionTime metav1.Time - // +optional - Reason string - // +optional - Message string -} -``` +Note: In order to use this data source feature, user/admin needs to update to the new external provisioner which can recognize snapshot data source. Otherwise, data source will be ignoed and an empty volume will be created ## Use cases * Use snapshot to backup data: Alice wants to take a snapshot of her Mongo database, and accidentally delete her tables, she wants to restore her volumes from the snapshot. -- cgit v1.2.3 From 4effd1a6309520ebf6b67604d9f57f7601d6b4d4 Mon Sep 17 00:00:00 2001 From: Jing Xu Date: Fri, 24 Aug 2018 11:08:26 -0700 Subject: Update data-source.md --- contributors/design-proposals/storage/data-source.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/contributors/design-proposals/storage/data-source.md b/contributors/design-proposals/storage/data-source.md index 5a55639f..58697072 100644 --- a/contributors/design-proposals/storage/data-source.md +++ b/contributors/design-proposals/storage/data-source.md @@ -1,10 +1,10 @@ # Add DataSource for Volume Operations -Note: this proposal is part of [Volume Snapshot](https://github.com/kubernetes/community/pull/2335) feature design, and also relevant to recently proposed Volume Clone feature. +Note: this proposal is part of [Volume Snapshot](https://github.com/kubernetes/community/pull/2335) feature design, and also relevant to recently proposed [Volume Clone](https://github.com/kubernetes/community/pull/2533) feature. ## Goal -Currently in Kuberentes, volume plugin only supports to provision an empty volume. With the new storage features (including [Volume Snapshot](https://github.com/kubernetes/community/pull/2335) and [volume clone](https://github.com/erinboyd/community/blob/patch-3/contributors/design-proposals/storage/cloning.md)) being proposed, there is a need to support data population for volume provisioning. For example, volume can be created from a snapshot source, or volume could be cloned from another volume source. Depending on the sources for creating the volume, there are two scenarios -1. Volume provisioner can recoginize the source and be able to create the volume from the source directly (e.g., restore snapshot to a volume or clone volume). +Currently in Kuberentes, volume plugin only supports to provision an empty volume. With the new storage features (including [Volume Snapshot](https://github.com/kubernetes/community/pull/2335) and [volume clone](https://github.com/kubernetes/community/pull/2533)) being proposed, there is a need to support data population for volume provisioning. For example, volume can be created from a snapshot source, or volume could be cloned from another volume source. Depending on the sources for creating the volume, there are two scenarios +1. Volume provisioner can recognize the source and be able to create the volume from the source directly (e.g., restore snapshot to a volume or clone volume). 2. Volume provisioner does not recognize the volume source, and create an empty volume. Another external component (data populator) could watch the volume creation and implement the logic to populate/import the data to the volume provisioned. Only after data is populated to the volume, the PVC is ready for use. There could be many different types of sources used for populating the data to the volumes. In this proposal, we propose to add a generic "DataSource" field to PersistentVolumeClaimSpec to represent different types of data sources. @@ -35,9 +35,9 @@ type TypedLocalObjectReference struct { ### Design Details In the first Alphal version, we only support data source from Snapshot. So the expected Kind in DataSource has to be "VolumeSnapshot". In this case, provisioner should provision volume and populate data in one step. There is no need for external data populator yet. -For other types of data sources that require external data populator, volume creation and data population are two seperate steps. Only when data is ready, PVC/PV can be marked as ready (Bound) so that users can start use them. We are working on a seperate proposal to address this using similar idea from ["Pod Ready++"](https://github.com/kubernetes/community/blob/master/keps/sig-network/0007-pod-ready%2B%2B.md). +For other types of data sources that require external data populator, volume creation and data population are two seperate steps. Only when data is ready, PVC/PV can be marked as ready (Bound) so that users can start to use them. We are working on a seperate proposal to address this using similar idea from ["Pod Ready++"](https://github.com/kubernetes/community/blob/master/keps/sig-network/0007-pod-ready%2B%2B.md). -Note: In order to use this data source feature, user/admin needs to update to the new external provisioner which can recognize snapshot data source. Otherwise, data source will be ignoed and an empty volume will be created +Note: In order to use this data source feature, user/admin needs to update to the new external provisioner which can recognize snapshot data source. Otherwise, data source will be ignored and an empty volume will be created ## Use cases * Use snapshot to backup data: Alice wants to take a snapshot of her Mongo database, and accidentally delete her tables, she wants to restore her volumes from the snapshot. -- cgit v1.2.3 From 8c6cea4e2e2bdd9167a08921563a9c6f0ec44091 Mon Sep 17 00:00:00 2001 From: Jing Xu Date: Fri, 24 Aug 2018 11:12:16 -0700 Subject: Update data-source.md --- contributors/design-proposals/storage/data-source.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/contributors/design-proposals/storage/data-source.md b/contributors/design-proposals/storage/data-source.md index 58697072..80db936b 100644 --- a/contributors/design-proposals/storage/data-source.md +++ b/contributors/design-proposals/storage/data-source.md @@ -11,7 +11,7 @@ There could be many different types of sources used for populating the data to t ## Design ### API Change -A new DataSource field is proposed to add to PVC to represent the source of the data which is pre-populated to the provisioned volume. For DataSource field, we propose to define a new type “TypedLocalObjectReference”. It is similar to “LocalObjectReference” type with additional Kind field in order to support multiple data source types. In the alpha version, this data source is restricted in the same namespace of the PVC. The following are the APIs we propose to add. +A new DataSource field is proposed to be added to PVC to represent the source of the data which is pre-populated to the provisioned volume. For DataSource field, we propose to define a new type “TypedLocalObjectReference”. It is similar to “LocalObjectReference” type with additional Kind field in order to support multiple data source types. In the alpha version, this data source is restricted in the same namespace of the PVC. The following are the APIs we propose to add. ``` @@ -33,9 +33,9 @@ type TypedLocalObjectReference struct { ``` ### Design Details -In the first Alphal version, we only support data source from Snapshot. So the expected Kind in DataSource has to be "VolumeSnapshot". In this case, provisioner should provision volume and populate data in one step. There is no need for external data populator yet. +In the first alpha version, we only support data source from Snapshot. So the expected Kind in DataSource has to be "VolumeSnapshot". In this case, provisioner should provision volume and populate data in one step. There is no need for external data populator yet. -For other types of data sources that require external data populator, volume creation and data population are two seperate steps. Only when data is ready, PVC/PV can be marked as ready (Bound) so that users can start to use them. We are working on a seperate proposal to address this using similar idea from ["Pod Ready++"](https://github.com/kubernetes/community/blob/master/keps/sig-network/0007-pod-ready%2B%2B.md). +For other types of data sources that require external data populator, volume creation and data population are two separate steps. Only when data is ready, PVC/PV can be marked as ready (Bound) so that users can start to use them. We are working on a separate proposal to address this using similar idea from ["Pod Ready++"](https://github.com/kubernetes/community/blob/master/keps/sig-network/0007-pod-ready%2B%2B.md). Note: In order to use this data source feature, user/admin needs to update to the new external provisioner which can recognize snapshot data source. Otherwise, data source will be ignored and an empty volume will be created -- cgit v1.2.3