CSI Common Issues
Issues when provisioning volumes with the Ceph CSI driver can happen for many reasons such as:
- Network connectivity between CSI pods and ceph
- Cluster health issues
- Slow operations
- Kubernetes issues
- Ceph-CSI configuration or bugs
The following troubleshooting steps can help identify a number of issues.
Block (RBD)¶
If you are mounting block volumes (usually RWO), these are referred to as RBD
volumes in Ceph. See the sections below for RBD if you are having block volume issues.
Shared Filesystem (CephFS)¶
If you are mounting shared filesystem volumes (usually RWX), these are referred to as CephFS
volumes in Ceph. See the sections below for CephFS if you are having filesystem volume issues.
Network Connectivity¶
The Ceph monitors are the most critical component of the cluster to check first. Retrieve the mon endpoints from the services:
If host networking is enabled in the CephCluster CR, you will instead need to find the node IPs for the hosts where the mons are running.
The clusterIP
is the mon IP and 3300
is the port that will be used by Ceph-CSI to connect to the ceph cluster. These endpoints must be accessible by all clients in the cluster, including the CSI driver.
If you are seeing issues provisioning the PVC then you need to check the network connectivity from the provisioner pods.
- For CephFS PVCs, check network connectivity from the
csi-cephfsplugin
container of thecsi-cephfsplugin-provisioner
pods - For Block PVCs, check network connectivity from the
csi-rbdplugin
container of thecsi-rbdplugin-provisioner
pods
For redundancy, there are two provisioner pods for each type. Make sure to test connectivity from all provisioner pods.
Connect to the provisioner pods and verify the connection to the mon endpoints such as the following:
If you see the response "ceph v2", the connection succeeded. If there is no response then there is a network issue connecting to the ceph cluster.
Check network connectivity for all monitor IP’s and ports which are passed to ceph-csi.
Ceph Health¶
Sometimes an unhealthy Ceph cluster can contribute to the issues in creating or mounting the PVC. Check that your Ceph cluster is healthy by connecting to the Toolbox and running the ceph
commands:
Slow Operations¶
Even slow ops in the ceph cluster can contribute to the issues. In the toolbox, make sure that no slow ops are present and the ceph cluster is healthy
If Ceph is not healthy, check the following health for more clues:
- The Ceph monitor logs for errors
- The OSD logs for errors
- Disk Health
- Network Health
Ceph Troubleshooting¶
Check if the RBD Pool exists¶
Make sure the pool you have specified in the storageclass.yaml
exists in the ceph cluster.
Suppose the pool name mentioned in the storageclass.yaml
is replicapool
. It can be verified to exist in the toolbox:
If the pool is not in the list, create the CephBlockPool
CR for the pool if you have not already. If you have already created the pool, check the Rook operator log for errors creating the pool.
Check if the Filesystem exists¶
For the shared filesystem (CephFS), check that the filesystem and pools you have specified in the storageclass.yaml
exist in the Ceph cluster.
Suppose the fsName
name mentioned in the storageclass.yaml
is myfs
. It can be verified in the toolbox:
Now verify the pool
mentioned in the storageclass.yaml
exists, such as the example myfs-data0
.
The pool for the filesystem will have the suffix -data0
compared the filesystem name that is created by the CephFilesystem CR.
subvolumegroups¶
If the subvolumegroup is not specified in the ceph-csi configmap (where you have passed the ceph monitor information), Ceph-CSI creates the default subvolumegroup with the name csi. Verify that the subvolumegroup exists:
If you don’t see any issues with your Ceph cluster, the following sections will start debugging the issue from the CSI side.
Provisioning Volumes¶
At times the issue can also exist in the Ceph-CSI or the sidecar containers used in Ceph-CSI.
Ceph-CSI has included number of sidecar containers in the provisioner pods such as: csi-attacher
, csi-resizer
, csi-provisioner
, csi-cephfsplugin
, csi-snapshotter
, and liveness-prometheus
.
The CephFS provisioner core CSI driver container name is csi-cephfsplugin
as one of the container names. For the RBD (Block) provisioner you will see csi-rbdplugin
as the container name.
Here is a summary of the sidecar containers:
csi-provisioner¶
The external-provisioner is a sidecar container that dynamically provisions volumes by calling ControllerCreateVolume()
and ControllerDeleteVolume()
functions of CSI drivers. More details about external-provisioner can be found here.
If there is an issue with PVC Create or Delete, check the logs of the csi-provisioner
sidecar container.
csi-resizer¶
The CSI external-resizer
is a sidecar container that watches the Kubernetes API server for PersistentVolumeClaim updates and triggers ControllerExpandVolume
operations against a CSI endpoint if the user requested more storage on the PersistentVolumeClaim object. More details about external-provisioner can be found here.
If any issue exists in PVC expansion you can check the logs of the csi-resizer
sidecar container.
csi-snapshotter¶
The CSI external-snapshotter sidecar only watches for VolumeSnapshotContent
create/update/delete events. It will talk to ceph-csi containers to create or delete snapshots. More details about external-snapshotter can be found here.
In Kubernetes 1.17 the volume snapshot feature was promoted to beta. In Kubernetes 1.20, the feature gate is enabled by default on standard Kubernetes deployments and cannot be turned off.
Make sure you have installed the correct snapshotter CRD version. If you have not installed the snapshotter controller, see the Snapshots guide.
The above CRDs must have the matching version in your snapshotclass.yaml
or snapshot.yaml
. Otherwise, the VolumeSnapshot
and VolumesnapshotContent
will not be created.
The snapshot controller is responsible for creating both VolumeSnapshot
and VolumesnapshotContent
object. If the objects are not getting created, you may need to check the logs of the snapshot-controller container.
Rook only installs the snapshotter sidecar container, not the controller. It is recommended that Kubernetes distributors bundle and deploy the controller and CRDs as part of their Kubernetes cluster management process (independent of any CSI Driver).
If your Kubernetes distribution does not bundle the snapshot controller, you may manually install these components.
If any issue exists in the snapshot Create/Delete operation you can check the logs of the csi-snapshotter sidecar container.
If you see an error about a volume already existing such as:
The issue typically is in the Ceph cluster or network connectivity. If the issue is in Provisioning the PVC Restarting the Provisioner pods help(for CephFS issue restart csi-cephfsplugin-provisioner-xxxxxx
CephFS Provisioner. For RBD, restart the csi-rbdplugin-provisioner-xxxxxx
pod. If the issue is in mounting the PVC, restart the csi-rbdplugin-xxxxx
pod (for RBD) and the csi-cephfsplugin-xxxxx
pod for CephFS issue.
Mounting the volume to application pods¶
When a user requests to create the application pod with PVC, there is a three-step process
- CSI driver registration
- Create volume attachment object
- Stage and publish the volume
csi-driver registration¶
csi-cephfsplugin-xxxx
or csi-rbdplugin-xxxx
is a daemonset pod running on all the nodes where your application gets scheduled. If the plugin pods are not running on the node where your application is scheduled might cause the issue, make sure plugin pods are always running.
Each plugin pod has two important containers: one is driver-registrar
and csi-rbdplugin
or csi-cephfsplugin
. Sometimes there is also a liveness-prometheus
container.
driver-registrar¶
The node-driver-registrar is a sidecar container that registers the CSI driver with Kubelet. More details can be found here.
If any issue exists in attaching the PVC to the application pod check logs from driver-registrar sidecar container in plugin pod where your application pod is scheduled.
You should see the response RegistrationStatus{PluginRegistered:true,Error:,}
in the logs to confirm that plugin is registered with kubelet.
If you see a driver not found an error in the application pod describe output. Restarting the csi-xxxxplugin-xxx
pod on the node may help.
Volume Attachment¶
Each provisioner pod also has a sidecar container called csi-attacher
.
csi-attacher¶
The external-attacher is a sidecar container that attaches volumes to nodes by calling ControllerPublish
and ControllerUnpublish
functions of CSI drivers. It is necessary because the internal Attach/Detach controller running in Kubernetes controller-manager does not have any direct interfaces to CSI drivers. More details can be found here.
If any issue exists in attaching the PVC to the application pod first check the volumeattachment object created and also log from csi-attacher sidecar container in provisioner pod.
CephFS Stale operations¶
Check for any stale mount commands on the csi-cephfsplugin-xxxx
pod on the node where your application pod is scheduled.
You need to exec in the csi-cephfsplugin-xxxx
pod and grep for stale mount operators.
Identify the csi-cephfsplugin-xxxx
pod running on the node where your application is scheduled with kubectl get po -o wide
and match the node names.
If any commands are stuck check the dmesg logs from the node. Restarting the csi-cephfsplugin
pod may also help sometimes.
If you don’t see any stuck messages, confirm the network connectivity, Ceph health, and slow ops.
RBD Stale operations¶
Check for any stale map/mkfs/mount
commands on the csi-rbdplugin-xxxx
pod on the node where your application pod is scheduled.
You need to exec in the csi-rbdplugin-xxxx
pod and grep for stale operators like (rbd map, rbd unmap, mkfs, mount
and umount
).
Identify the csi-rbdplugin-xxxx
pod running on the node where your application is scheduled with kubectl get po -o wide
and match the node names.
If any commands are stuck check the dmesg logs from the node. Restarting the csi-rbdplugin
pod also may help sometimes.
If you don’t see any stuck messages, confirm the network connectivity, Ceph health, and slow ops.
dmesg logs¶
Check the dmesg logs on the node where pvc mounting is failing or the csi-rbdplugin
container of the csi-rbdplugin-xxxx
pod on that node.
RBD Commands¶
If nothing else helps, get the last executed command from the ceph-csi pod logs and run it manually inside the provisioner or plugin pod to see if there are errors returned even if they couldn't be seen in the logs.
Where -m
is one of the mon endpoints and the --key
is the key used by the CSI driver for accessing the Ceph cluster.
Node Loss¶
When a node is lost, you will see application pods on the node stuck in the Terminating
state while another pod is rescheduled and is in the ContainerCreating
state.
Important
For clusters with Kubernetes version 1.26 or greater, see the improved automation to recover from the node loss. If using K8s 1.25 or older, continue with these instructions.
Force deleting the pod¶
To force delete the pod stuck in the Terminating
state:
After the force delete, wait for a timeout of about 8-10 minutes. If the pod still not in the running state, continue with the next section to blocklist the node.
Blocklisting a node¶
To shorten the timeout, you can mark the node as "blocklisted" from the Rook toolbox so Rook can safely failover the pod sooner.
After running the above command within a few minutes the pod will be running.
Removing a node blocklist¶
After you are absolutely sure the node is permanently offline and that the node no longer needs to be blocklisted, remove the node from the blocklist.