Skip to content

NVMe-oF Block Storage

This feature is experimental

NVMe over Fabrics (NVMe-oF) allows RBD volumes to be exposed and accessed via the NVMe/TCP protocol. This enables both Kubernetes pods within the cluster and external clients outside the cluster to connect to Ceph block storage using standard NVMe-oF initiators, providing high-performance block storage access over the network.

Goals

The NVMe-oF integration in Rook serves two primary purposes:

  1. In-Cluster Consumption: Pods inside the Kubernetes cluster can consume storage via the NVMe-oF protocol, providing an alternative to traditional RBD mounts with potential performance benefits for certain workloads.

  2. External Client Access: Rook serves as a backend for external clients outside the cluster, enabling non-Kubernetes workloads to access Ceph block storage through standard NVMe-oF initiators. This allows organizations to leverage their Ceph storage infrastructure for both containerized and traditional workloads.

Both use cases share a common foundation (gateway, CSI driver, and data pool) and then diverge. Follow the Common Setup first, then continue with either In-Cluster Deployment or External Client Deployment (or both).

For more background and design details, see the NVMe-oF gateway design doc. For the Ceph-CSI NVMe-oF design proposal, see the ceph-csi NVMe-oF proposal.

Prerequisites

This guide assumes a Rook cluster as explained in the Quickstart Guide.

Requirements

  • Ceph Version: Ceph v20 (Tentacle) or later

Common Setup

These steps are required for both in-cluster and external deployments.

Step 1: Create the NVMe-oF Gateway

The CephNVMeOFGateway CRD manages the NVMe-oF gateway infrastructure. The operator will automatically create the following resources:

  • Service: One per gateway instance for service discovery
  • Deployment: One per gateway instance running the NVMe-oF gateway daemon

NVMe-oF uses two separate pools:

  • Metadata pool (.nvmeof): An internal pool that stores gateway state (subsystem and namespace definitions). Created with spec.name: .nvmeof.
  • Data pool: A standard CephBlockPool that holds the actual RBD image data. See Step 3.

Create the gateway and the .nvmeof metadata pool:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  name: builtin-nvmeof
  namespace: rook-ceph
spec:
  name: .nvmeof
  failureDomain: host
  replicated:
    size: 3
---
apiVersion: ceph.rook.io/v1
kind: CephNVMeOFGateway
metadata:
  name: nvmeof
  namespace: rook-ceph
spec:
  # ANA (Asymmetric Namespace Access) group name
  group: group-a
  # Number of gateway instances to run (2 recommended for HA)
  instances: 2
  hostNetwork: false

Apply the gateway configuration:

1
kubectl create -f deploy/examples/nvmeof.yaml

Verify the gateway is running:

1
kubectl get pod -n rook-ceph -l app=rook-ceph-nvmeof

Example Output

1
2
NAME                                         READY   STATUS    RESTARTS   AGE
rook-ceph-nvmeof-nvmeof-a-85844ff6b8-4r8gj   1/1     Running   0          91s

Step 2: Deploy the NVMe-oF CSI Driver via CSI Operator

The NVMe-oF CSI driver is deployed via the ceph-csi operator.

Apply the Driver CR for NVMe-oF that will trigger the creation of the Ceph-CSI/NVMe-oF deployment and daemonset:

1
kubectl create -f deploy/examples/csi/nvmeof/driver.yaml

Verify the CSI operator created the controller and node plugins:

1
kubectl get pods -n rook-ceph | grep nvmeof

Example Output

1
2
rook-ceph.nvmeof.csi.ceph.com-ctrlplugin-d9d77fb7c-kkl28   5/5     Running   0          60s
rook-ceph.nvmeof.csi.ceph.com-nodeplugin-xvt5g              2/2     Running   0          60s

Step 3: Create the Data Pool

Create a CephBlockPool that will be used by the StorageClass to store data:

1
2
3
4
5
6
7
8
9
apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  name: nvmeof
  namespace: rook-ceph
spec:
  failureDomain: host
  replicated:
    size: 3

1
kubectl create -f deploy/examples/csi/nvmeof/nvmeof-pool.yaml

In-Cluster Deployment (for Pods)

This section covers consuming NVMe-oF volumes from pods inside the Kubernetes cluster via the NVMe-oF CSI driver.

Step 4: Create the StorageClass

Create a StorageClass that uses the NVMe-oF CSI driver. The subsystemNQN parameter places all in-cluster PVCs into a shared subsystem, and the listeners list tells the CSI node plugin which gateways to connect to for I/O.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ceph-nvmeof
parameters:
  clusterID: rook-ceph
  pool: nvmeof
  subsystemNQN: nqn.2016-06.io.spdk:cnode1.rook-ceph
  nvmeofGatewayAddress: "rook-ceph-nvmeof-nvmeof-a.rook-ceph.svc.cluster.local"
  nvmeofGatewayPort: "5500"
  listeners: |
    [
      {
        "hostname": "rook-ceph-nvmeof-nvmeof-a"
      },
      {
        "hostname": "rook-ceph-nvmeof-nvmeof-b"
      }
    ]
  csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
  csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
  csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
  csi.storage.k8s.io/controller-modify-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/controller-modify-secret-namespace: rook-ceph
  csi.storage.k8s.io/node-expand-secret-name: rook-csi-rbd-node
  csi.storage.k8s.io/node-expand-secret-namespace: rook-ceph
  imageFormat: "2"
  imageFeatures: layering,deep-flatten,exclusive-lock,object-map,fast-diff
provisioner: rook-ceph.nvmeof.csi.ceph.com
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true

Note

The provisioner name rook-ceph.nvmeof.csi.ceph.com is prefixed with the operator namespace.

1
kubectl create -f deploy/examples/csi/nvmeof/storageclass.yaml

Step 5: Create a PersistentVolumeClaim

Create a PVC using the NVMe-oF storage class:

1
kubectl create -f deploy/examples/csi/nvmeof/pvc.yaml

Verify the PVC is bound:

1
kubectl get pvc nvmeof-volume

Example Output

1
2
NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
nvmeof-volume    Bound    pvc-b4108580-5cfa-46d3-beff-320088a5bf3c   1Gi        RWO            ceph-nvmeof    20m

Step 6: Create a Pod

Create a pod that consumes the NVMe-oF volume:

1
kubectl create -f deploy/examples/csi/nvmeof/pod.yaml

Verify the pod is running:

1
kubectl get pods -n default nvmeof-test-pod

Example Output

1
2
NAME              READY   STATUS    RESTARTS   AGE
nvmeof-test-pod   1/1     Running   0          60s

External Client Deployment (via LoadBalancer)

This section covers exposing NVMe-oF volumes to clients outside the Kubernetes cluster. External clients connect using standard NVMe-oF initiators (nvme-cli) through LoadBalancer services.

Prerequisites for External Clients

  • nvme-cli: The nvme command-line tool must be installed on the client
  • Network Access: The client must be able to reach the gateway LoadBalancer IP
  • Host NQN: The client's host NQN must be listed in the VolumeAttributesClass

Install nvme-cli and load the required kernel module on the external client:

1
2
3
4
5
# Fedora / RHEL
sudo dnf install nvme-cli

# Load NVMe-oF/TCP transport
sudo modprobe nvme-tcp

Step 4: Create a VolumeAttributesClass

A VolumeAttributesClass defines mutable volume parameters such as host access control. It specifies which host NQNs are allowed to access the volume.

Create the VolumeAttributesClass with the allowed host NQNs:

1
kubectl create -f deploy/examples/csi/nvmeof/volume-attributes-class.yaml

Step 5: Create the External StorageClass and PVC

The external StorageClass does not include subsystemNQN. Each PVC is automatically placed in its own subsystem, and external clients discover the subsystem NQN dynamically via nvme discover.

Create the external StorageClass and PVC:

1
2
kubectl create -f deploy/examples/csi/nvmeof/storageclass-external.yaml
kubectl create -f deploy/examples/csi/nvmeof/pvc-external.yaml

Verify the PVC is bound:

1
kubectl get pvc nvmeof-external-volume

Example Output

1
2
NAME                     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS           VOLUMEATTRIBUTESCLASS   AGE
nvmeof-external-volume   Bound    pvc-bbad520d-fa7d-476c-904e-a8da2b6476ab   1Gi        RWO            ceph-nvmeof-external   nvmeof-external-hosts   19s

Step 6: Create LoadBalancer Services

External clients need LoadBalancer services to reach the NVMe-oF gateways from outside the cluster. Create one LoadBalancer per gateway instance so each gateway has a dedicated external endpoint.

Create the services:

1
kubectl create -f deploy/examples/csi/nvmeof/nvmeof-lb.yaml

Get the external IPs:

1
kubectl get service -n rook-ceph nvmeof-lb-a nvmeof-lb-b

Example Output

1
2
3
NAME           TYPE           CLUSTER-IP      EXTERNAL-IP       PORT(S)                         AGE
nvmeof-lb-a    LoadBalancer   10.96.100.50    192.168.1.100     4420:31334/TCP,8009:31235/TCP    60s
nvmeof-lb-b    LoadBalancer   10.96.100.51    192.168.1.101     4420:31336/TCP,8009:31237/TCP    60s

Step 7: Connect from the External Client

The connection process has two stages: first discover the available subsystems via the discovery port (8009), then connect to the discovered subsystem via the I/O port (4420).

Discover subsystems

Run a discovery against either gateway LoadBalancer address using the discovery port (8009):

1
sudo nvme discover -t tcp -a <GW-A-EXTERNAL-IP> -s 8009

Replace <GW-A-EXTERNAL-IP> with one of the LoadBalancer external IPs from the service output above.

The discovery log returns the target subsystem NQN and available transport addresses:

1
2
3
4
5
6
7
Discovery Log Number of Records 2, Generation counter 7
=====Discovery Log Entry 0======
trtype:  tcp
subtype: nvme subsystem
trsvcid: 4420
subnqn:  nqn.2016-06.io.ceph:subsystem.0001-0009-rook-ceph-0000000000000003-...
traddr:  0.0.0.0

Connect to the LoadBalancer

Use the subsystem NQN from the discovery output to connect through all gateway LoadBalancers in the gateway group. The host must connect to every gateway to see all namespaces:

1
2
3
4
5
6
7
sudo nvme connect -t tcp \
  -n <SUBSYSTEM-NQN> \
  -a <GW-A-EXTERNAL-IP> -s 4420

sudo nvme connect -t tcp \
  -n <SUBSYSTEM-NQN> \
  -a <GW-B-EXTERNAL-IP> -s 4420

Verify the connection:

1
sudo nvme list-subsys

Format and Mount

1
2
3
4
sudo mkfs.ext4 /dev/nvme1n1
sudo mkdir -p /mnt/nvmeof
sudo mount /dev/nvme1n1 /mnt/nvmeof
sudo chown $(whoami):$(whoami) /mnt/nvmeof

Teardown

Warning

Deleting a PVC will also delete the underlying RBD image and NVMe namespace. Ensure you have backups if needed.

In-Cluster Teardown

1
2
3
4
5
6
7
8
# Delete the test pod
kubectl delete -f deploy/examples/csi/nvmeof/pod.yaml

# Delete the PVC
kubectl delete pvc nvmeof-volume

# Delete the StorageClass
kubectl delete storageclass ceph-nvmeof

External Teardown

Disconnect the NVMe-oF volume from the external client:

1
2
sudo umount /mnt/nvmeof
sudo nvme disconnect-all

Then delete the Kubernetes resources:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# Delete the external PVC
kubectl delete pvc nvmeof-external-volume

# Delete the VolumeAttributesClass
kubectl delete volumeattributesclass nvmeof-external-hosts

# Delete the LoadBalancer services
kubectl delete service -n rook-ceph nvmeof-lb-a nvmeof-lb-b

# Delete the external StorageClass
kubectl delete storageclass ceph-nvmeof-external

Common Teardown

After removing the mode-specific resources above, clean up the shared infrastructure:

1
2
3
4
5
6
7
8
# Delete the NVMe-oF CSI driver
kubectl delete -f deploy/examples/csi/nvmeof/driver.yaml

# Delete the NVMe-oF gateway and its metadata pool
kubectl delete -f deploy/examples/nvmeof.yaml

# Delete the data pool
kubectl delete -f deploy/examples/csi/nvmeof/nvmeof-pool.yaml

References