Ceph

PLEASE NOTE: This document applies to v1.0 version and not to the latest stable release v1.9

Using Rook-Ceph with Pod Security Policies (PSPs)

See the Rook overall PSP document before continuing on here with Ceph specifics.

PodSecurityPolicy

You need at least one PodSecurityPolicy that allows privileged Pod execution. Here is an example that is reasonably pared down for Ceph, though more work to minimize permissions can be done:

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: privileged
spec:
  fsGroup:
    rule: RunAsAny
  privileged: true
  runAsUser:
    rule: RunAsAny
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  volumes:
    - 'configMap'
    - 'emptyDir'
    - 'projected'
    - 'secret'
    - 'downwardAPI'
    - 'hostPath'
    - 'flexVolume'
  hostPID: true
  # hostNetwork is required for using host networking
  hostNetwork: false

Hint: Allowing hostNetwork usage is required when using hostNetwork: true in the Cluster Resource Definition! You are then also required to allow the usage of hostPorts in the PodSecurityPolicy. The given port range is a minimal working recommendation for a Rook Ceph cluster:

   hostPorts:
     # Ceph msgr2 port
     - min: 3300
       max: 3300
     # Ceph ports
     - min: 6789
       max: 7300
     # Ceph MGR Prometheus Metrics
     - min: 9283
       max: 9283

ClusterRole and ClusterRoleBinding

Next up you require a ClusterRole and a corresponding ClusterRoleBinding, which enables the Rook Agent ServiceAccount to run the rook-ceph-agent Pods on all nodes with privileged rights. Here are the definitions:

# privilegedPSP grants access to use the privileged PSP.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: psp:rook
rules:
- apiGroups:
  - policy
  resources:
  - podsecuritypolicies
  resourceNames:
  - privileged
  verbs:
  - use

and

apiVersion: v1
kind: Namespace
metadata:
  name: rook-ceph-system
---
# Allow the rook-ceph-system serviceAccount to use the privileged PSP
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: rook-ceph-system-psp
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: psp:rook
subjects:
- kind: ServiceAccount
  name: rook-ceph-system
  namespace: rook-ceph

Save these definitions to one or multiple yaml files and create them by executing kubectl apply -f <nameOfYourFile>.yaml

You will also require two more RoleBindings for each Rook Cluster you deploy: Create these two RoleBindings in the Namespace you plan to deploy your Rook Cluster into (default is “rook” namespace):

apiVersion: v1
kind: Namespace
metadata:
  name: rook-ceph
---
# Allow the default serviceAccount to use the privileged PSP
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: rook-default-psp
  namespace: rook-ceph
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: psp:rook
subjects:
- kind: ServiceAccount
  name: default
  namespace: rook-ceph
---
# Allow the rook-ceph-osd serviceAccount to use the privileged PSP
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: rook-ceph-osd-psp
  namespace: rook-ceph
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: psp:rook
subjects:
- kind: ServiceAccount
  name: rook-ceph-osd
  namespace: rook-ceph
---
# Allow the rook-ceph-mgr serviceAccount to use the privileged PSP
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: rook-ceph-mgr-psp
  namespace: rook-ceph
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: psp:rook
subjects:
- kind: ServiceAccount
  name: rook-ceph-mgr
  namespace: rook-ceph