Skip to content

Key Management System

Rook has the ability to encrypt OSDs of clusters running on PVC via the flag (encrypted: true) in your storageClassDeviceSets template. Rook also has the ability to rotate encryption keys of OSDs using a cron job per OSD. By default, the Key Encryption Keys (also known as Data Encryption Keys) are stored in a Kubernetes Secret. However, if a Key Management System exists Rook is capable of using it.

The security section contains settings related to encryption of the cluster.

  • security:
    • kms: Key Management System settings
      • connectionDetails: the list of parameters representing kms connection details
      • tokenSecretName: the name of the Kubernetes Secret containing the kms authentication token
    • keyRotation: Key Rotation settings
      • enabled: whether key rotation is enabled or not, default is false
      • schedule: the schedule, written in cron format, with which key rotation CronJob is created, default value is "@weekly".

Note

Currently key rotation is supported when the Key Encryption Keys are stored in a Kubernetes Secret or Vault KMS.

Supported KMS providers:

Vault

Rook supports storing OSD encryption keys in HashiCorp Vault KMS.

Authentication methods

Rook support two authentication methods:

  • token-based: a token is provided by the user and is stored in a Kubernetes Secret. It's used to authenticate the KMS by the Rook operator. This has several pitfalls such as:
    • when the token expires it must be renewed, so the secret holding it must be updated
    • no token automatic rotation
  • Kubernetes Service Account uses Vault Kubernetes native authentication mechanism and alleviate some of the limitations from the token authentication such as token automatic renewal. This method is generally recommended over the token-based authentication.

Token-based authentication

When using the token-based authentication, a Kubernetes Secret must be created to hold the token. This is governed by the tokenSecretName parameter.

Note: Rook supports all the Vault environment variables.

The Kubernetes Secret rook-vault-token should contain:

1
2
3
4
5
6
7
apiVersion: v1
kind: Secret
metadata:
  name: rook-vault-token
  namespace: rook-ceph
data:
  token: <TOKEN> # base64 of a token to connect to Vault, for example: cy5GWXpsbzAyY2duVGVoRjhkWG5Bb3EyWjkK

You can create a token in Vault by running the following command:

1
vault token create -policy=rook

Refer to the official vault document for more details on how to create a token. For which policy to apply see the next section.

In order for Rook to connect to Vault, you must configure the following in your CephCluster template:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
security:
  kms:
    # name of the k8s config map containing all the kms connection details
    connectionDetails:
      KMS_PROVIDER: vault
      VAULT_ADDR: https://vault.default.svc.cluster.local:8200
      VAULT_BACKEND_PATH: rook
      VAULT_SECRET_ENGINE: kv
      VAULT_AUTH_METHOD: token
    # name of the k8s secret containing the kms authentication token
    tokenSecretName: rook-vault-token

Kubernetes-based authentication

In order to use the Kubernetes Service Account authentication method, the following must be run to properly configure Vault:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
ROOK_NAMESPACE=rook-ceph
ROOK_VAULT_SA=rook-vault-auth
ROOK_SYSTEM_SA=rook-ceph-system
ROOK_OSD_SA=rook-ceph-osd
VAULT_POLICY_NAME=rook

# create service account for vault to validate API token
kubectl -n "$ROOK_NAMESPACE" create serviceaccount "$ROOK_VAULT_SA"

# create the RBAC for this SA
kubectl -n "$ROOK_NAMESPACE" create clusterrolebinding vault-tokenreview-binding --clusterrole=system:auth-delegator --serviceaccount="$ROOK_NAMESPACE":"$ROOK_VAULT_SA"

# get the service account common.yaml created earlier
VAULT_SA_SECRET_NAME=$(kubectl -n "$ROOK_NAMESPACE" get sa "$ROOK_VAULT_SA" -o jsonpath="{.secrets[*]['name']}")

# Set SA_JWT_TOKEN value to the service account JWT used to access the TokenReview API
SA_JWT_TOKEN=$(kubectl -n "$ROOK_NAMESPACE" get secret "$VAULT_SA_SECRET_NAME" -o jsonpath="{.data.token}" | base64 --decode)

# Set SA_CA_CRT to the PEM encoded CA cert used to talk to Kubernetes API
SA_CA_CRT=$(kubectl -n "$ROOK_NAMESPACE" get secret "$VAULT_SA_SECRET_NAME" -o jsonpath="{.data['ca\.crt']}" | base64 --decode)

# get kubernetes endpoint
K8S_HOST=$(kubectl config view --minify --flatten -o jsonpath="{.clusters[0].cluster.server}")

# enable kubernetes auth
vault auth enable kubernetes

# To fetch the service account issuer
kubectl proxy &
proxy_pid=$!

# configure the kubernetes auth
vault write auth/kubernetes/config \
    token_reviewer_jwt="$SA_JWT_TOKEN" \
    kubernetes_host="$K8S_HOST" \
    kubernetes_ca_cert="$SA_CA_CRT" \
    issuer="$(curl --silent http://127.0.0.1:8001/.well-known/openid-configuration | jq -r .issuer)"

kill $proxy_pid

# configure a role for rook
vault write auth/kubernetes/role/"$ROOK_NAMESPACE" \
    bound_service_account_names="$ROOK_SYSTEM_SA","$ROOK_OSD_SA" \
    bound_service_account_namespaces="$ROOK_NAMESPACE" \
    policies="$VAULT_POLICY_NAME" \
    ttl=1440h

Once done, your CephCluster CR should look like:

1
2
3
4
5
6
7
8
9
security:
  kms:
    connectionDetails:
        KMS_PROVIDER: vault
        VAULT_ADDR: https://vault.default.svc.cluster.local:8200
        VAULT_BACKEND_PATH: rook
        VAULT_SECRET_ENGINE: kv
        VAULT_AUTH_METHOD: kubernetes
        VAULT_AUTH_KUBERNETES_ROLE: rook-ceph

Note

The VAULT_ADDR value above assumes that Vault is accessible within the cluster itself on the default port (8200). If running elsewhere, please update the URL accordingly.

General Vault configuration

As part of the token, here is an example of a policy that can be used:

1
2
3
4
5
6
path "rook/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}
path "sys/mounts" {
capabilities = ["read"]
}

You can write the policy like so and then create a token:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
$ vault policy write rook /tmp/rook.hcl
$ vault token create -policy=rook
Key                  Value
---                  -----
token                s.FYzlo02cgnTehF8dXnAoq2Z9
token_accessor       oMo7sAXQKbYtxU4HtO8k3pko
token_duration       768h
token_renewable      true
token_policies       ["default" "rook"]
identity_policies    []
policies             ["default" "rook"]

In the above example, Vault's secret backend path name is rook. It must be enabled with the following:

1
vault secrets enable -path=rook kv

If a different path is used, the VAULT_BACKEND_PATH key in connectionDetails must be changed.

TLS configuration

This is an advanced but recommended configuration for production deployments, in this case the vault-connection-details will look like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
security:
  kms:
    # name of the k8s config map containing all the kms connection details
    connectionDetails:
      KMS_PROVIDER: vault
      VAULT_ADDR: https://vault.default.svc.cluster.local:8200
      VAULT_CACERT: <name of the k8s secret containing the PEM-encoded CA certificate>
      VAULT_CLIENT_CERT: <name of the k8s secret containing the PEM-encoded client certificate>
      VAULT_CLIENT_KEY: <name of the k8s secret containing the PEM-encoded private key>
    # name of the k8s secret containing the kms authentication token
    tokenSecretName: rook-vault-token

Each secret keys are expected to be:

  • VAULT_CACERT: cert
  • VAULT_CLIENT_CERT: cert
  • VAULT_CLIENT_KEY: key

For instance VAULT_CACERT Secret named vault-tls-ca-certificate will look like:

1
2
3
4
5
6
7
apiVersion: v1
kind: Secret
metadata:
  name: vault-tls-ca-certificate
  namespace: rook-ceph
data:
  cert: <PEM base64 encoded CA certificate>

Note: if you are using self-signed certificates (not known/approved by a proper CA) you must pass VAULT_SKIP_VERIFY: true. Communications will remain encrypted but the validity of the certificate will not be verified.

IBM Key Protect

Rook supports storing OSD encryption keys in IBM Key Protect. The current implementation stores OSD encryption keys as Standard Keys using the Bring Your Own Key (BYOK) method. This means that the Key Protect instance policy must have Standard Imported Key enabled.

Configuration

First, you need to provision the Key Protect service on the IBM Cloud. Once completed, retrieve the instance ID. Make a record of it; we need it in the CRD.

On the IBM Cloud, the user must create a Service ID, then assign an Access Policy to this service. Ultimately, a Service API Key needs to be generated. All the steps are summarized in the official documentation.

The Service ID must be granted access to the Key Protect Service. Once the Service API Key is generated, store it in a Kubernetes Secret.

1
2
3
4
5
6
7
apiVersion: v1
kind: Secret
metadata:
  name: ibm-kp-svc-api-key
  namespace: rook-ceph
data:
  IBM_KP_SERVICE_API_KEY: <service API Key>

In order for Rook to connect to IBM Key Protect, you must configure the following in your CephCluster template:

1
2
3
4
5
6
7
8
security:
  kms:
    # name of the k8s config map containing all the kms connection details
    connectionDetails:
      KMS_PROVIDER: ibmkeyprotect
      IBM_KP_SERVICE_INSTANCE_ID: <instance ID that was retrieved in the first paragraph>
    # name of the k8s secret containing the service API Key
    tokenSecretName: ibm-kp-svc-api-key

More options are supported such as:

  • IBM_BASE_URL: the base URL of the Key Protect instance, depending on your region. Defaults to https://us-south.kms.cloud.ibm.com.
  • IBM_TOKEN_URL: the URL of the Key Protect instance to retrieve the token. Defaults to https://iam.cloud.ibm.com/oidc/token. Only needed for private instances.

Key Management Interoperability Protocol

Rook supports storing OSD encryption keys in Key Management Interoperability Protocol (KMIP) supported KMS. The current implementation stores OSD encryption keys using the Register operation. Key is fetched and deleted using Get and Destroy operations respectively.

Configuration

The Secret with credentials for the KMIP KMS is expected to contain the following.

1
2
3
4
5
6
7
8
9
apiVersion: v1
kind: Secret
metadata:
  name: kmip-credentials
  namespace: rook-ceph
stringData:
  CA_CERT: <ca certificate>
  CLIENT_CERT: <client certificate>
  CLIENT_KEY: <client key>

In order for Rook to connect to KMIP, you must configure the following in your CephCluster template:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
security:
  kms:
    # name of the k8s config map containing all the kms connection details
    connectionDetails:
      KMS_PROVIDER: kmip
      KMIP_ENDPOINT: <KMIP endpoint address>
      # (optional) The endpoint server name. Useful when the KMIP endpoint does not have a DNS entry.
      TLS_SERVER_NAME: <tls server name>
      # (optional) Network read timeout, in seconds. The default value is 10.
      READ_TIMEOUT: <read timeout>
      # (optional) Network write timeout, in seconds. The default value is 10.
      WRITE_TIMEOUT: <write timeout>
    # name of the k8s secret containing the credentials.
    tokenSecretName: kmip-credentials

Azure Key Vault

Rook supports storing OSD encryption keys in Azure Key vault

Client Authentication

Different methods are available in Azure to authenticate a client. Rook supports Azure recommended method of authentication with Service Principal and a certificate. Refer the following Azure documentation to set up key vault and authenticate it via service principal and certtificate

  • Create Azure Key Vault

    • AZURE_VAULT_URL can be retrieved at this step

  • Create Service Principal

    • AZURE_CLIENT_ID and AZURE_TENANT_ID can be obtained after creating the service principal
    • Ensure that the service principal is authenticated with a certificate and not with a client secret.

  • Set Azure Key Vault RBAC

    • Ensure that the role assigned to the key vault should be able to create, retrieve and delete secrets in the key vault.

Provide the following KMS connection details in order to connect with Azure Key Vault.

1
2
3
4
5
6
7
8
security:
  kms:
    connectionDetails:
      KMS_PROVIDER: azure-kv
      AZURE_VAULT_URL: https://<key-vault name>.vault.azure.net
      AZURE_CLIENT_ID: Application ID of an Azure service principal
      AZURE_TENANT_ID: ID of the application's Microsoft Entra tenant
      AZURE_CERT_SECRET_NAME: <name of the k8s secret containing the certificate along with the private key (without password protection)>
  • AZURE_CERT_SECRET_NAME should hold the name of the k8s secret. The secret data should be base64 encoded certificate along with private key (without password protection)