Blog: Update on Volume Snapshot Alpha for Kubernetes
-
Authors: Jing Xu (Google), Xing Yang (Huawei), Saad Ali (Google)
Volume snapshotting support was introduced in Kubernetes v1.12 as an alpha feature. In Kubernetes v1.13, it remains an alpha feature, but a few enhancements were added and some breaking changes were made. This post summarizes the changes.
Breaking Changes
CSI spec v1.0 introduced a few breaking changes to the volume snapshot feature. CSI driver maintainers should be aware of these changes as they upgrade their drivers to support v1.0.
SnapshotStatus replaced with Boolean ReadyToUse
CSI v0.3.0, defined a
SnapshotStatus
enum inCreateSnapshotResponse
which indicates whether the snapshot isREADY
,UPLOADING
, orERROR_UPLOADING
. In CSI v1.0,SnapshotStatus
has been removed fromCreateSnapshotResponse
and replaced with aboolean ReadyToUse
. AReadyToUse
value oftrue
indicates that post snapshot processing (such as uploading) is complete and the snapshot is ready to be used as a source to create a volume.Storage systems that need to do post snapshot processing (such as uploading after the snapshot is cut) should return a successful
CreateSnapshotResponse
with theReadyToUse
field set tofalse
as soon as the snapshot has been taken. This indicates that the Container Orchestration System (CO) can resume any workload that was quiesced for the snapshot to be taken. The CO can then repeatedly callCreateSnapshot
until theReadyToUse
field is set totrue
or the call returns an error indicating a problem in processing. The CSIListSnapshot
call could be used along withsnapshot_id
filtering to determine if the snapshot is ready to use, but is not recommended because it provides no way to detect errors during processing (theReadyToUse
field simply remainsfalse
indefinitely).The v1.x.x releases of the CSI external-snapshotter sidecar container already handle this change by calling
CreateSnapshot
instead ofListSnapshots
to check if a snapshot is ready to use. When upgrading their drivers to CSI 1.0, driver maintainers should use the appropriate 1.0 compatible sidecar container.To be consistent with the change in the CSI spec, the
Ready
field in theVolumeSnapshot
API object has been renamed toReadyToUse
. This change is visible to the user when runningkubectl describe volumesnapshot
to view the details of a snapshot.Timestamp Data Type
The creation time of a snapshot is available to Kubernetes admins as part of the
VolumeSnapshotContent
API object. This field is populated using thecreation_time
field in the CSICreateSnapshotResponse
. In CSI v1.0, thiscreation_time
field type was changed to.google.protobuf.Timestamp
instead ofint64
. When upgrading drivers to CSI 1.0, driver maintainers must make changes accordingly. The v1.x.x releases of the CSI external-snapshotter sidecar container has been updated to handle this change.Deprecations
The following
VolumeSnapshotClass
parameters are deprecated and will be removed in a future release. They will be replaced with parameters listed in theReplacement
section below.Deprecated Replacement csiSnapshotterSecretName csi.storage.k8s.io/snapshotter-secret-name csiSnapshotterSecretNameSpace csi.storage.k8s.io/snapshotter-secret-namespace
New Features
SnapshotContent Deletion/Retain Policy
As described in the initial blog post announcing the snapshot alpha, the Kubernetes snapshot APIs are similar to the PV/PVC APIs: just like a volume is represented by a bound PVC and PV pair, a snapshot is represented by a bound
VolumeSnapshot
andVolumeSnapshotContent
pair.With PV/PVC pairs, when a user is done with a volume, they can delete the PVC. And the reclaim policy on the PV determines what happens to the PV (whether it is also deleted or retained).
In the initial alpha release, snapshots did not support the ability to specify a reclaim policy. Instead when a snapshot object was deleted it always resulted in the snapshot being deleted. In Kubernetes v1.13, a snapshot content
DeletionPolicy
was added. It enables an admin to configure what what happens to aVolumeSnapshotContent
after theVolumeSnapshot
object it is bound to is deleted. TheDeletionPolicy
of a volume snapshot can either beRetain
orDelete
. If the value is not specified, the default depends on whether theSnapshotContent
object was created via static binding or dynamic provisioning.Retain
The
Retain
policy allows for manual reclamation of the resource. If aVolumeSnapshotContent
is statically created and bound, the defaultDeletionPolicy
isRetain
. When theVolumeSnapshot
is deleted, theVolumeSnapshotContent
continues to exist and theVolumeSnapshotContent
is considered “released”. But it is not available for binding to otherVolumeSnapshot
objects because it contains data. It is up to an administrator to decide how to handle the remaining API object and resource cleanup.Delete
A
Delete
policy enables automatic deletion of the boundVolumeSnapshotContent
object from Kubernetes and the associated storage asset in the external infrastructure (such as an AWS EBS snapshot or GCE PD snapshot, etc.). Snapshots that are dynamically provisioned inherit the deletion policy of theirVolumeSnapshotClass
, which defaults toDelete
. The administrator should configure theVolumeSnapshotClass
with the desired retention policy. The policy may be changed for individualVolumeSnapshotContent
after it is created by patching the object.The following example demonstrates how to check the deletion policy of a dynamically provisioned
VolumeSnapshotContent
.$ kubectl create -f ./examples/kubernetes/demo-defaultsnapshotclass.yaml $ kubectl create -f ./examples/kubernetes/demo-snapshot.yaml $ kubectl get volumesnapshots demo-snapshot-podpvc -o yaml apiVersion: snapshot.storage.k8s.io/v1alpha1 kind: VolumeSnapshot metadata: creationTimestamp: "2018-11-27T23:57:09Z" ... spec: snapshotClassName: default-snapshot-class snapshotContentName: snapcontent-26cd0db3-f2a0-11e8-8be6-42010a800002 source: apiGroup: null kind: PersistentVolumeClaim name: podpvc status: … $ kubectl get volumesnapshotcontent snapcontent-26cd0db3-f2a0-11e8-8be6-42010a800002 -o yaml apiVersion: snapshot.storage.k8s.io/v1alpha1 kind: VolumeSnapshotContent … spec: csiVolumeSnapshotSource: creationTime: 1546469777852000000 driver: pd.csi.storage.gke.io restoreSize: 6442450944 snapshotHandle: projects/jing-k8s-dev/global/snapshots/snapshot-26cd0db3-f2a0-11e8-8be6-42010a800002 deletionPolicy: Delete persistentVolumeRef: apiVersion: v1 kind: PersistentVolume name: pvc-853622a4-f28b-11e8-8be6-42010a800002 resourceVersion: "21117" uid: ae400e9f-f28b-11e8-8be6-42010a800002 snapshotClassName: default-snapshot-class volumeSnapshotRef: apiVersion: snapshot.storage.k8s.io/v1alpha1 kind: VolumeSnapshot name: demo-snapshot-podpvc namespace: default resourceVersion: "6948065" uid: 26cd0db3-f2a0-11e8-8be6-42010a800002
User can change the deletion policy by using patch:
$ kubectl patch volumesnapshotcontent snapcontent-26cd0db3-f2a0-11e8-8be6-42010a800002 -p '{"spec":{"deletionPolicy":"Retain"}}' --type=merge $ kubectl get volumesnapshotcontent snapcontent-26cd0db3-f2a0-11e8-8be6-42010a800002 -o yaml apiVersion: snapshot.storage.k8s.io/v1alpha1 kind: VolumeSnapshotContent ... spec: csiVolumeSnapshotSource: ... deletionPolicy: Retain persistentVolumeRef: apiVersion: v1 kind: PersistentVolume name: pvc-853622a4-f28b-11e8-8be6-42010a800002 ...
Snapshot Object in Use Protection
The purpose of the Snapshot Object in Use Protection feature is to ensure that in-use snapshot API objects are not removed from the system (as this may result in data loss). There are two cases that require “in-use” protection:
- If a volume snapshot is in active use by a persistent volume claim as a source to create a volume.
- If a
VolumeSnapshotContent
API object is bound to a VolumeSnapshot API object, the content object is considered in use.
If a user deletes a
VolumeSnapshot
API object in active use by a PVC, theVolumeSnapshot
object is not removed immediately. Instead, removal of theVolumeSnapshot
object is postponed until theVolumeSnapshot
is no longer actively used by any PVCs. Similarly, if an admin deletes aVolumeSnapshotContent
that is bound to aVolumeSnapshot
, theVolumeSnapshotContent
is not removed immediately. Instead, theVolumeSnapshotContent
removal is postponed until theVolumeSnapshotContent
is not bound to theVolumeSnapshot
object.Which volume plugins support Kubernetes Snapshots?
Snapshots are only supported for CSI drivers (not for in-tree or Flexvolume). To use the Kubernetes snapshots feature, ensure that a CSI Driver that implements snapshots is deployed on your cluster.
As of the publishing of this blog post, the following CSI drivers support snapshots:
- GCE Persistent Disk CSI Driver
- OpenSDS CSI Driver
- Ceph RBD CSI Driver
- Portworx CSI Driver
- GlusterFS CSI Driver
- Digital Ocean CSI Driver
- Ember CSI Driver
- Cinder CSI Driver
- Datera CSI Driver
- NexentaStor CSI Driver
Snapshot support for other drivers is pending, and should be available soon. Read the “Container Storage Interface (CSI) for Kubernetes GA” blog post to learn more about CSI and how to deploy CSI drivers.
What’s next?
Depending on feedback and adoption, the Kubernetes team plans to push the CSI Snapshot implementation to beta in either 1.15 or 1.16. Some of the features we are interested in supporting include consistency groups, application consistent snapshots, workload quiescing, in-place restores, and more.
How can I learn more?
The code repository for snapshot APIs and controller is here: https://github.com/kubernetes-csi/external-snapshotter
Check out additional documentation on the snapshot feature here: http://k8s.io/docs/concepts/storage/volume-snapshots and https://kubernetes-csi.github.io/docs/
How do I get involved?
This project, like all of Kubernetes, is the result of hard work by many contributors from diverse backgrounds working together.
Special thanks to all the contributors that helped add CSI v1.0 support and improve the snapshot feature in this release, including Saad Ali (saadali), Michelle Au (msau42), Deep Debroy (ddebroy), James DeFelice (jdef), John Griffith (j-griffith), Julian Hjortshoj (julian-hj), Tim Hockin (thockin), Patrick Ohly (pohly), Luis Pabon (lpabon), Cheng Xing (verult), Jing Xu (jingxu97), Shiwei Xu (wackxu), Xing Yang (xing-yang), Jie Yu (jieyu), David Zhu (davidz627).
Those interested in getting involved with the design and development of CSI or any part of the Kubernetes Storage system, join the Kubernetes Storage Special Interest Group (SIG). We’re rapidly growing and always welcome new contributors.
We also hold regular SIG-Storage Snapshot Working Group meetings. New attendees are welcome to join for design and development discussions.
https://kubernetes.io/blog/2019/01/17/update-on-volume-snapshot-alpha-for-kubernetes/
© Lightnetics 2024