Expanding a Volume with vSphere Container Storage Plug-in
vSphere
Container Storage Plug-in
supports volume expansion for block volumes that are created dynamically or statically. Kubernetes supports offline and online modes
of volume expansion. When the PVC is used by a pod and is mounted on a node, the volume
expansion operation is categorized as online. In all other cases, it is an offline
expansion. For information about vSphere versions that support volume expansion, see
Supported Kubernetes Functionality.
Expanding volume is not supported when volume
snapshot is present, and when a Node VM snapshot is present with the volume attached to
it.
The following considerations apply when you
use volume expansion:
- Feature Gate
- Expand CSI Volumes feature is enabled by default since it was promoted to beta in Kubernetes 1.16. For Kubernetes releases before 1.16, enable Expand CSI Volumes feature gate to support volume expansion invSphere Container Storage Plug-in.
- Sidecar Container
- An external resizer sidecar container implements the logic of watching the Kubernetes API for PVC edits, issuing the ControllerExpandVolume RPC call against a CSI endpoint, and updating thePersistentVolumeobject to reflect the new size. This container is already deployed as part of the vsphere-csi-controller pod.
- Volume Resize Limitations
- Kubernetes does not support expansion of a volume backed up by Statefulset.
- vSphere Container Storage Plug-indoes not support expansion of a volume backed up byvSANfile service.
Expand a Volume in Online Mode
vSphere
Container Storage Plug-in
supports volume expansion for block volumes in online mode. - Create a PVC and a pod to use this PVC.
- Create a new StorageClass or edit the existing StorageClass to setallowVolumeExpansionto true.kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: example-block-sc provisioner: csi.vsphere.vmware.com allowVolumeExpansion: true
- Create or edit a PVC that references the StorageClass.Ensure that the PVC is in bound state. If you are using a statically provisioned PVC, ensure that the PVC and the PV specifications have thestorageClassNameparameter pointing to the StorageClass with theallowVolumeExpansionset to true.
- Deploy the PVC and create a pod to use this PVC.$ kubectl get pvc,pv,pod NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE persistentvolumeclaim/example-block-pvc Bound pvc-84c89bf9-8455-4633-a8c8-cd623e155dbd 1Gi RWO example-block-sc 8m5s NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE persistentvolume/pvc-84c89bf9-8455-4633-a8c8-cd623e155dbd 1Gi RWO Delete Bound default/example-block-pvc example-block-sc 7m59s NAME READY STATUS RESTARTS AGE pod/example-block-pod 1/1 Running 0 7m1s
- Expand the PVC in online mode.
- Patch the PVC to increase the storage size, in this example, to2Gi.$ kubectl patch pvc example-block-pvc -p '{"spec": {"resources": {"requests": {"storage": "2Gi"}}}}' persistentvolumeclaim/example-block-pvc patchedThis action triggers an expansion in the volume associated with the PVC in vSphere Cloud Native Storage.
- Describe the PVC.The output looks similar to the following. The PVC shows the increase in size after the volume underneath is expanded.$ kubectl describe pvc example-block-pvc Name: example-block-pvc Namespace: default StorageClass: example-block-sc Status: Bound Volume: pvc-84c89bf9-8455-4633-a8c8-cd623e155dbd Labels: <none> Annotations: pv.kubernetes.io/bind-completed: yes pv.kubernetes.io/bound-by-controller: yes volume.beta.kubernetes.io/storage-provisioner: csi.vsphere.vmware.com Finalizers: [kubernetes.io/pvc-protection] Capacity: 2Gi Access Modes: RWO VolumeMode: Filesystem Mounted By: example-block-pod Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ExternalProvisioning 19m persistentvolume-controller waiting for a volume to be created, either by external provisioner "csi.vsphere.vmware.com" or manually created by system administrator Normal Provisioning 19m csi.vsphere.vmware.com_vsphere-csi-controller-5d8c5c7d6-9r9kv_7adc4efc-10a6-4615-b90b-790032cc4569 External provisioner is provisioning volume for claim "default/example-block-pvc" Normal ProvisioningSucceeded 19m csi.vsphere.vmware.com_vsphere-csi-controller-5d8c5c7d6-9r9kv_7adc4efc-10a6-4615-b90b-790032cc4569 Successfully provisioned volume pvc-84c89bf9-8455-4633-a8c8-cd623e155dbd Warning ExternalExpanding 75s volume_expand Ignoring the PVC: didn't find a plugin capable of expanding the volume; waiting for an external controller to process this PVC. Normal Resizing 75s external-resizer csi.vsphere.vmware.com External resizer is resizing volume pvc-84c89bf9-8455-4633-a8c8-cd623e155dbd Normal FileSystemResizeRequired 69s external-resizer csi.vsphere.vmware.com Require file system resize of volume on node Normal FileSystemResizeSuccessful 6s kubelet, k8s-node-072 MountVolume.NodeExpandVolume succeeded for volume "pvc-84c89bf9-8455-4633-a8c8-cd623e155dbd"The PVC goes through eventsResizingtoFileSystemResizeRequiredtoFileSystemResizeSuccessful.
- Display the PV to verify the expanded size.$ kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-84c89bf9-8455-4633-a8c8-cd623e155dbd 2Gi RWO Delete Bound default/example-block-pvc example-block-sc 25m
Expand a Volume in Offline Mode
vSphere
Container Storage Plug-in
supports volume expansion for block volumes in offline mode. - Create a PVC.
- Create a new StorageClass or edit the existing StorageClass to setallowVolumeExpansionto true.kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: example-block-sc provisioner: csi.vsphere.vmware.com allowVolumeExpansion: true
- Create or edit a PVC that references the StorageClass.Ensure that the PVC is in bound state. If you are using a statically provisioned PVC, ensure that the PVC and the PV specifications have thestorageClassNameparameter pointing to the StorageClass with theallowVolumeExpansionset to true.
- Deploy the PVC.$ kubectl get pvc,pv NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE persistentvolumeclaim/example-block-pvc Bound pvc-9e9a325d-ee1c-11e9-a223-005056ad1fc1 1Gi RWO example-block-sc 5m5s NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE persistentvolume/pvc-9e9a325d-ee1c-11e9-a223-005056ad1fc1 1Gi RWO Delete Bound default/example-block-pvc example-block-sc 5m18s
- Expand the PVC in offline mode.
- Patch the PVC to increase the storage size, in this example, to2Gi.
This action triggers an expansion in the volume associated with the PVC. The capacity of the corresponding PV object changes. However, the capacity of the PVC does not change until the PVC is used by a pod and mounted on a node.$ kubectl patch pvc example-block-pvc -p '{"spec": {"resources": {"requests": {"storage": "2Gi"}}}}' persistentvolumeclaim/example-block-pvc patched$ kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-9e9a325d-ee1c-11e9-a223-005056ad1fc1 2Gi RWO Delete Bound default/example-block-pvc example-block-sc 6m44s $ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE example-block-pvc Bound pvc-9e9a325d-ee1c-11e9-a223-005056ad1fc1 1Gi RWO example-block-sc 6m57sYou can also see aFilesystemResizePendingcondition applied on the PVC when you describe it. - Create a pod to use the PVC.apiVersion: v1 kind: Pod metadata: name: example-block-pod spec: containers: - name: test-container image: gcr.io/google_containers/busybox:1.24 command: ["/bin/sh", "-c", "echo 'hello' > /mnt/volume1/index.html && chmod o+rX /mnt /mnt/volume1/index.html && while true ; do sleep 2 ; done"] volumeMounts: - name: test-volume mountPath: /mnt/volume1 restartPolicy: Never volumes: - name: test-volume persistentVolumeClaim: claimName: example-block-pvc$ kubectl create -f example-pod.yaml pod/example-block-pod createdThe Kubelet on the node triggers the filesystem expansion on the volume when the PVC is attached to the pod.$ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE example-block-pvc Bound pvc-24114458-9753-428e-9c90-9f568cb25788 2Gi RWO example-block-sc 2m12s $ kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-24114458-9753-428e-9c90-9f568cb25788 2Gi RWO Delete Bound default/example-block-pvc example-block-sc 2m3s$ kubectl get pod NAME READY STATUS RESTARTS AGE example-block-pod 1/1 Running 0 65sThe capacity of PVC is modified and theFilesystemResizePendingcondition is removed from the PVC. Offline volume expansion is complete.