Volume Health Monitoring Alpha Update
Author: Xing Yang (VMware)
The CSI Volume Health Monitoring feature, originally introduced in 1.19 has undergone a large update for the 1.21 release.
Why add Volume Health Monitoring to Kubernetes?
Without Volume Health Monitoring, Kubernetes has no knowledge of the state of the underlying volumes of a storage system after a PVC is provisioned and used by a Pod. Many things could happen to the underlying storage system after a volume is provisioned in Kubernetes. For example, the volume could be deleted by accident outside of Kubernetes, the disk that the volume resides on could fail, it could be out of capacity, the disk may be degraded which affects its performance, and so on. Even when the volume is mounted on a pod and used by an application, there could be problems later on such as read/write I/O errors, file system corruption, accidental unmounting of the volume outside of Kubernetes, etc. It is very hard to debug and detect root causes when something happened like this.
Volume health monitoring can be very beneficial to Kubernetes users. It can communicate with the CSI driver to retrieve errors detected by the underlying storage system. PVC events can be reported up to the user to take action. For example, if the volume is out of capacity, they could request a volume expansion to get more space.
What is Volume Health Monitoring?
CSI Volume Health Monitoring allows CSI Drivers to detect abnormal volume conditions from the underlying storage systems and report them as events on PVCs or Pods.
The Kubernetes components that monitor the volumes and report events with volume health information include the following:
- Kubelet, in addition to gathering the existing volume stats will watch the volume health of the PVCs on that node. If a PVC has an abnormal health condition, an event will be reported on the pod object using the PVC. If multiple pods are using the same PVC, events will be reported on all pods using that PVC.
- An
Note that the node side volume health monitoring logic was an external agent when this feature was first introduced in the Kubernetes 1.19 release. In Kubernetes 1.21, the node side volume health monitoring logic was moved from the external agent into the Kubelet, to avoid making duplicate CSI function calls. With this change in 1.21, a new alpha feature gate CSIVolumeHealth was introduced for the volume health monitoring logic in Kubelet.
Currently the Volume Health Monitoring feature is informational only as it only reports abnormal volume health events on PVCs or Pods. Users will need to check these events and manually fix the problems. This feature serves as a stepping stone towards programmatic detection and resolution of volume health issues by Kubernetes in the future.
How do I use Volume Health on my Kubernetes Cluster?
To use the Volume Health feature, first make sure the CSI driver you are using supports this feature. Refer to this
To enable Volume Health Monitoring from the node side, the alpha feature gate If a CSI driver supports the Volume Health Monitoring feature from the controller side, events regarding abnormal volume conditions will be recorded on PVCs. If a CSI driver supports the Volume Health Monitoring feature from the controller side, user can also get events regarding node failures if the If a CSI driver supports the Volume Health Monitoring feature from the node side, events regarding abnormal volume conditions will be recorded on pods using the PVCs. Volume Health Monitoring includes two parts: For details, see the . There is a sample implementation for volume health in . To learn how to deploy the External Volume Health Monitoring controller, see
The External Health Monitor Controller calls either To implement the volume health feature from the controller side, a CSI driver must add support for the new controller capabilities. If a CSI driver supports If a CSI driver supports If a CSI driver supports Kubelet calls To implement the volume health feature from the node side, a CSI driver must add support for the new node capabilities. If a CSI driver supports Depending on feedback and adoption, the Kubernetes team plans to push the CSI volume health implementation to beta in either 1.22 or 1.23. We are also exploring how to use volume health information for programmatic detection and automatic reconcile in Kubernetes. To learn the design details for Volume Health Monitoring, read the
The Volume Health Monitor controller source code is at
There are also more details about volume health checks in the . The
We offer a huge thank you to the contributors who helped release this feature in 1.21. We want to thank Yuquan Ren (
Those interested in getting involved with the design and development of CSI or any part of the Kubernetes Storage system, join the
CSIVolumeHealth needs to be enabled.enable-node-watcher flag is set to true when deploying the External Health Monitor Controller. When a node failure event is detected, an event will be reported on the PVC to indicate that pods using this PVC are on a failed node.As a storage vendor, how do I add support for volume health to my CSI driver?
Controller Side Volume Health Monitoring
ListVolumes or ControllerGetVolume CSI RPC and reports VolumeConditionAbnormal events with messages on PVCs if abnormal volume conditions are detected. Only CSI drivers with LIST_VOLUMES and VOLUME_CONDITION controller capability or GET_VOLUME and VOLUME_CONDITION controller capability support Volume Health Monitoring in the external controller.LIST_VOLUMES and VOLUME_CONDITION controller capabilities, it must implement controller RPC ListVolumes and report the volume condition in the response.GET_VOLUME and VOLUME_CONDITION controller capability, it must implement controller PRC ControllerGetVolume and report the volume condition in the response.LIST_VOLUMES, GET_VOLUME, and VOLUME_CONDITION controller capabilities, only ListVolumes CSI RPC will be invoked by the External Health Monitor Controller.Node Side Volume Health Monitoring
NodeGetVolumeStats CSI RPC and reports VolumeConditionAbnormal events with messages on Pods if abnormal volume conditions are detected. Only CSI drivers with VOLUME_CONDITION node capability support Volume Health Monitoring in Kubelet.VOLUME_CONDITION node capability, it must report the volume condition in node RPC NodeGetVoumeStats.What’s next?
How can I learn more?
How do I get involved?