Skip to content

Filesystem Mirroring

Ceph filesystem mirroring is a process of asynchronous replication of snapshots to a remote CephFS file system. Snapshots are synchronized by mirroring snapshot data followed by creating a snapshot with the same name (for a given directory on the remote file system) as the snapshot being synchronized. It is generally useful when planning for Disaster Recovery. Mirroring is for clusters that are geographically distributed and stretching a single cluster is not possible due to high latencies.

Prerequisites

This guide assumes you have created a Rook cluster as explained in the main quickstart guide

Create the Filesystem with Mirroring enabled

The following will enable mirroring on the filesystem:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
apiVersion: ceph.rook.io/v1
kind: CephFilesystem
metadata:
  name: myfs
  namespace: rook-ceph
spec:
  metadataPool:
    failureDomain: host
    replicated:
      size: 3
  dataPools:
    - name: replicated
      failureDomain: host
      replicated:
        size: 3
  preserveFilesystemOnDelete: true
  metadataServer:
    activeCount: 1
    activeStandby: true
  mirroring:
    enabled: true
    # list of Kubernetes Secrets containing the peer token
    # for more details see: https://docs.ceph.com/en/latest/dev/cephfs-mirroring/#bootstrap-peers
    # Add the secret name if it already exists else specify the empty list here.
    peers:
      secretNames:
        #- secondary-cluster-peer
    # specify the schedule(s) on which snapshots should be taken
    # see the official syntax here https://docs.ceph.com/en/latest/cephfs/snap-schedule/#add-and-remove-schedules
    snapshotSchedules:
      - path: /
        interval: 24h # daily snapshots
        # The startTime should be mentioned in the format YYYY-MM-DDTHH:MM:SS
        # If startTime is not specified, then by default the start time is considered as midnight UTC.
        # see usage here https://docs.ceph.com/en/latest/cephfs/snap-schedule/#usage
        # startTime: 2022-07-15T11:55:00
    # manage retention policies
    # see syntax duration here https://docs.ceph.com/en/latest/cephfs/snap-schedule/#add-and-remove-retention-policies
    snapshotRetention:
      - path: /
        duration: "h 24"

Create the cephfs-mirror daemon

Launch the rook-ceph-fs-mirror pod on the source storage cluster, which deploys the cephfs-mirror daemon in the cluster: 

1
kubectl create -f deploy/examples/filesystem-mirror.yaml

Please refer to Filesystem Mirror CRD for more information.

Configuring mirroring peers

Once mirroring is enabled, Rook will by default create its own bootstrap peer token so that it can be used by another cluster. The bootstrap peer token can be found in a Kubernetes Secret. The name of the Secret is present in the Status field of the CephFilesystem CR:

1
2
3
status:
  info:
    fsMirrorBootstrapPeerSecretName: fs-peer-token-myfs

This secret can then be fetched like so:

1
2
# kubectl get secret -n rook-ceph fs-peer-token-myfs -o jsonpath='{.data.token}'|base64 -d
eyJmc2lkIjoiOTFlYWUwZGQtMDZiMS00ZDJjLTkxZjMtMTMxMWM5ZGYzODJiIiwiY2xpZW50X2lkIjoicmJkLW1pcnJvci1wZWVyIiwia2V5IjoiQVFEN1psOWZ3V1VGRHhBQWdmY0gyZi8xeUhYeGZDUTU5L1N0NEE9PSIsIm1vbl9ob3N0IjoiW3YyOjEwLjEwMS4xOC4yMjM6MzMwMCx2MToxMC4xMDEuMTguMjIzOjY3ODldIn0=

Import the token in the Destination cluster

The decoded secret must be saved in a file before importing.

1
# ceph fs snapshot mirror peer_bootstrap import <fs_name> <token file path>

See the CephFS mirror documentation on how to add a bootstrap peer.

Further refer to CephFS mirror documentation to configure a directory for snapshot mirroring.

Verify that the snapshots have synced

To check the mirror daemon status, please run the following command from the toolbox pod.

For example :

1
# ceph fs snapshot mirror daemon status | jq

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
[
  {
    "daemon_id": 906790,
    "filesystems": [
      {
        "filesystem_id": 1,
        "name": "myfs",
        "directory_count": 1,
        "peers": [
          {
            "uuid": "a24a3366-8130-4d55-aada-95fa9d3ff94d",
            "remote": {
              "client_name": "client.mirror",
              "cluster_name": "91046889-a6aa-4f74-9fb0-f7bb111666b4",
              "fs_name": "myfs"
            },
            "stats": {
              "failure_count": 0,
              "recovery_count": 0
            }
          }
        ]
      }
    ]
  }
]

Please refer to the --admin-daemon socket commands from the CephFS mirror documentation to verify mirror status and peer synchronization status and run the commands from the rook-ceph-fs-mirror pod:

1
# kubectl -n rook-ceph exec -it deploy/rook-ceph-fs-mirror -- bash

Fetch the ceph-client.fs-mirror daemon admin socket file from the /var/run/ceph directory:

1
# ls -lhsa /var/run/ceph/

1
# ceph --admin-daemon /var/run/ceph/ceph-client.fs-mirror.1.93989418120648.asok fs mirror status myfs@1

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
{
    "rados_inst": "X.X.X.X:0/2286593433",
    "peers": {
        "a24a3366-8130-4d55-aada-95fa9d3ff94d": {
            "remote": {
                "client_name": "client.mirror",
                "cluster_name": "91046889-a6aa-4f74-9fb0-f7bb111666b4",
                "fs_name": "myfs"
            }
        }
    },
    "snap_dirs": {
        "dir_count": 1
    }
}

For getting peer synchronization status:

1
# ceph --admin-daemon /var/run/ceph/ceph-client.fs-mirror.1.93989418120648.asok fs mirror peer status myfs@1 a24a3366-8130-4d55-aada-95fa9d3ff94d

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
{
    "/volumes/_nogroup/subvol-1": {
        "state": "idle",
        "last_synced_snap": {
            "id": 4,
            "name": "snap2"
        },
        "snaps_synced": 0,
        "snaps_deleted": 0,
        "snaps_renamed": 0
    }
}