Specification

Spec represents the specification of a Ceph COSI Driver

Spec represents the specification of a Ceph Client

Status represents the status of a Ceph Client

Spec represents the specification of a Ceph Filesystem SubVolumeGroup

Status represents the status of a CephFilesystem SubvolumeGroup

The URI of the AMQP endpoint to push notification to

Name of the exchange that is used to route messages based on topics

Indicate whether the server certificate is validated by the client or not

The ack level required for this topic (none/broker/routeable)

SubPath defines the sub-path (subdirectory) of the directory root where the volumeSource will be mounted. All files/keys in the volume source’s volume will be mounted to the subdirectory. This is not the same as the Kubernetes subPath volume mount option. Each subPath definition must be unique and must not contain ‘:’.

VolumeSource accepts a pared down version of the standard Kubernetes VolumeSource for the additional file(s) like what is normally used to configure Volumes for a Pod. Fore example, a ConfigMap, Secret, or HostPath. Each VolumeSource adds one or more additional files to the container <directory-root>/<subPath> directory. Be aware that some files may need to have a specific file mode like 0600 due to application requirements. For example, CA or TLS certificates.

Public defines a list of CIDRs to use for Ceph public network communication.

Cluster defines a list of CIDRs to use for Ceph cluster network communication.

The spec for Keystone

The name of the topic associated with this notification

List of events that should trigger the notification

Spec of notification filter

The name of the object store on which to define the topic

The namespace of the object store on which to define the topic

Data which is sent in each event

Indication whether notifications to this endpoint are persistent or not

Contains the endpoint spec of the topic

The ARN of the topic generated by the RGW

ObservedGeneration is the latest generation observed by the controller.

"Always"

Always means the Ceph COSI driver will be deployed even if the object store is not present

"Auto"

Auto means the Ceph COSI driver will be deployed automatically if object store is present

"Never"

Never means the Ceph COSI driver will never deployed

KernelMountOptions defines the mount options for kernel mounter.

FuseMountOptions defines the mount options for ceph fuse mounter.

ReadAffinity defines the read affinity settings for CSI driver.

CephFS defines CSI Driver settings for CephFS driver.

Spec represents the specification of a Ceph BlockPool Rados Namespace

Status represents the status of a CephBlockPool Rados Namespace

The name of the CephBlockPoolRadosNamespaceSpec namespace. If not set, the default is the name of the CR.

BlockPoolName is the name of Ceph BlockPool. Typically it’s the name of the CephBlockPool CR.

Mirroring configuration of CephBlockPoolRadosNamespace

optional

ObservedGeneration is the latest generation observed by the controller.

Image is the container image to run the Ceph COSI driver

ObjectProvisionerImage is the container image to run the COSI driver sidecar

DeploymentStrategy is the strategy to use to deploy the COSI driver.

Placement is the placement strategy to use for the COSI driver

Resources is the resource requirements for the COSI driver

ObservedGeneration is the latest generation observed by the controller.

DaemonHealth is the health check for a given daemon

LivenessProbe allows changing the livenessProbe configuration for a given daemon

StartupProbe allows changing the startupProbe configuration for a given daemon

Mon shows Mon Ceph version

Mgr shows Mgr Ceph version

Osd shows Osd Ceph version

Rgw shows Rgw Ceph version

Mds shows Mds Ceph version

RbdMirror shows RbdMirror Ceph version

CephFSMirror shows CephFSMirror Ceph version

Overall shows overall Ceph version

Only performance counters greater than or equal to this option are fetched

Time to wait before sending requests again to exporter server (seconds)

Use only info and put mirroringStatus in it?

MirroringStatus is the filesystem mirroring status

ObservedGeneration is the latest generation observed by the controller.

The name of the subvolume group. If not set, the default is the name of the subvolumeGroup CR.

FilesystemName is the name of Ceph Filesystem SubVolumeGroup volume name. Typically it’s the name of the CephFilesystem CR. If not coming from the CephFilesystem CR, it can be retrieved from the list of Ceph Filesystem volumes with ceph fs volume ls. To learn more about Ceph Filesystem abstractions see https://docs.ceph.com/en/latest/cephfs/fs-volumes/#fs-volumes-and-subvolumes

Pinning configuration of CephFilesystemSubVolumeGroup, reference https://docs.ceph.com/en/latest/cephfs/fs-volumes/#pinning-subvolumes-and-subvolume-groups only one out of (export, distributed, random) can be set at a time

Quota size of the Ceph Filesystem subvolume group.

The data pool name for the Ceph Filesystem subvolume group layout, if the default CephFS pool is not desired.

ObservedGeneration is the latest generation observed by the controller.

"cluster"

"public"

Image is the container image used to launch the ceph daemons, such as quay.io/ceph/ceph: The full list of images can be found at https://quay.io/repository/ceph/ceph?tab=tags

Whether to allow unsupported versions (do not set to true in production)

ImagePullPolicy describes a policy for if/when to pull a container image One of Always, Never, IfNotPresent.

"yes-really-destroy-data"

DeleteDataDirOnHostsConfirmation represents the validation to destroy dataDirHostPath

Confirmation represents the cleanup confirmation

SanitizeDisks represents way we sanitize disks

AllowUninstallWithVolumes defines whether we can proceed with the uninstall if they are RBD images still present

The version information that instructs Rook to orchestrate a particular version of Ceph.

A spec for available storage in the cluster and how it should be used

The annotations-related configuration to add/set on each Pod related object.

The labels-related configuration to add/set on each Pod related object.

The placement-related configuration to pass to kubernetes (affinity, node selector, tolerations).

Network related configuration

Resources set resource requests and limits

PriorityClassNames sets priority classes on components

The path on the host where config and data can be persisted

SkipUpgradeChecks defines if an upgrade should be forced even if one of the check fails

ContinueUpgradeAfterChecksEvenIfNotHealthy defines if an upgrade should continue even if PGs are not clean

WaitTimeoutForHealthyOSDInMinutes defines the time the operator would wait before an OSD can be stopped for upgrade or restart. If the timeout exceeds and OSD is not ok to stop, then the operator would skip upgrade for the current OSD and proceed with the next one if continueUpgradeAfterChecksEvenIfNotHealthy is false. If continueUpgradeAfterChecksEvenIfNotHealthy is true, then operator would continue with the upgrade of an OSD even if its not ok to stop after the timeout. This timeout won’t be applied if skipUpgradeChecks is true. The default wait timeout is 10 minutes.

UpgradeOSDRequiresHealthyPGs defines if OSD upgrade requires PGs are clean. If set to true OSD upgrade process won’t start until PGs are healthy. This configuration will be ignored if skipUpgradeChecks is true. Default is false.

A spec for configuring disruption management.

A spec for mon related options

A spec for the crash controller

Dashboard settings

Prometheus based Monitoring settings

Whether the Ceph Cluster is running external to this Kubernetes cluster mon, mgr, osd, mds, and discover daemons will not be created for external clusters.

A spec for mgr related options

Remove the OSD that is out and safe to remove only if this option is true

Indicates user intent when deleting a cluster; blocks orchestration and should not be set if cluster deletion is not imminent.

Internal daemon healthchecks and liveness probe

Security represents security settings

Logging represents loggings settings

CSI Driver Options applied per cluster.

Ceph Config options

"Connected"

ClusterStateConnected represents the Connected state of a Ceph Cluster

"Connecting"

ClusterStateConnecting represents the Connecting state of a Ceph Cluster

"Created"

ClusterStateCreated represents the Created state of a Ceph Cluster

"Creating"

ClusterStateCreating represents the Creating state of a Ceph Cluster

"Error"

ClusterStateError represents the Error state of a Ceph Cluster

"Updating"

ClusterStateUpdating represents the Updating state of a Ceph Cluster

ObservedGeneration is the latest generation observed by the controller.

Whether to compress the data in transit across the wire. The default is not set.

"ClusterConnected"

ClusterConnectedReason is cluster connected reason

"ClusterConnecting"

ClusterConnectingReason is cluster connecting reason

"ClusterCreated"

ClusterCreatedReason is cluster created reason

"ClusterDeleting"

ClusterDeletingReason is cluster deleting reason

"ClusterProgressing"

ClusterProgressingReason is cluster progressing reason

"Deleting"

DeletingReason represents when Rook has detected a resource object should be deleted.

"ObjectHasDependents"

ObjectHasDependentsReason represents when a resource object has dependents that are blocking deletion.

"ObjectHasNoDependents"

ObjectHasNoDependentsReason represents when a resource object has no dependents that are blocking deletion.

"ReconcileFailed"

ReconcileFailed represents when a resource reconciliation failed.

"ReconcileStarted"

ReconcileStarted represents when a resource reconciliation started.

"ReconcileSucceeded"

ReconcileSucceeded represents when a resource reconciliation was successful.

"Connected"

ConditionConnected represents Connected state of an object

"Connecting"

ConditionConnecting represents Connecting state of an object

"Deleting"

ConditionDeleting represents Deleting state of an object

"DeletionIsBlocked"

ConditionDeletionIsBlocked represents when deletion of the object is blocked.

"Failure"

ConditionFailure represents Failure state of an object

"Progressing"

ConditionProgressing represents Progressing state of an object

"Ready"

ConditionReady represents Ready state of an object

hostPath represents a pre-existing file or directory on the host machine that is directly exposed to the container. This is generally used for system agents or other privileged things that are allowed to see the host machine. Most containers will NOT need this. More info: https://kubernetes.io/docs/concepts/storage/volumes#hostpath

emptyDir represents a temporary directory that shares a pod’s lifetime. More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir

secret represents a secret that should populate this volume. More info: https://kubernetes.io/docs/concepts/storage/volumes#secret

persistentVolumeClaimVolumeSource represents a reference to a PersistentVolumeClaim in the same namespace. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims

configMap represents a configMap that should populate this volume

projected items for all in one resources secrets, configmaps, and downward API

Encryption settings for the network connections.

Compression settings for the network connections.

Whether to require msgr2 (port 3300) even if compression or encryption are not enabled. If true, the msgr1 port (6789) will be disabled. Requires a kernel that supports msgr2 (kernel 5.11 or CentOS 8.4 or newer).

Disable determines whether we should enable the crash collector

DaysToRetain represents the number of days to retain crash until they get pruned

Status represents the health check settings for the Ceph health

Monitor represents the health check settings for the Ceph monitor

ObjectStorageDaemon represents the health check settings for the Ceph OSDs

Enabled determines whether to enable the dashboard

URLPrefix is a prefix for all URLs to use the dashboard with a reverse proxy

Port is the dashboard webserver port

SSL determines whether SSL should be used

Endpoint for the Prometheus host

Whether to verify the ssl endpoint for prometheus. Set to false for a self-signed cert.

This enables management of poddisruptionbudgets

OSDMaintenanceTimeout sets how many additional minutes the DOWN/OUT interval is for drained failure domains it only works if managePodBudgets is true. the default is 30 minutes

PGHealthCheckTimeout is the time (in minutes) that the operator will wait for the placement groups to become healthy (active+clean) after a drain was completed and OSDs came back up. Rook will continue with the next drain if the timeout exceeds. It only works if managePodBudgets is true. No values or 0 means that the operator will wait until the placement groups are healthy before unblocking the next drain.

PgHealthyRegex is the regular expression that is used to determine which PG states should be considered healthy. The default is ^(active\+clean|active\+clean\+scrubbing|active\+clean\+scrubbing\+deep)$

Deprecated. This enables management of machinedisruptionbudgets.

Deprecated. Namespace to look for MDBs by the machineDisruptionBudgetController

Whether to encrypt the data in transit across the wire to prevent eavesdropping the data on the network. The default is not set. Even if encryption is not enabled, clients still establish a strong initial authentication for the connection and data integrity is still validated with a crc check. When encryption is enabled, all communication between clients and Ceph daemons, or between Ceph daemons will be encrypted.

The IP of this endpoint. As a legacy behavior, this supports being given a DNS-addressable hostname as well.

The DNS-addressable Hostname of this endpoint. This field will be preferred over IP if both are given.

Number of coding chunks per object in an erasure coded storage pool (required for erasure-coded pool type). This is the number of OSDs that can be lost simultaneously before data cannot be recovered.

Number of data chunks per object in an erasure coded storage pool (required for erasure-coded pool type). The number of chunks required to recover an object when any single OSD is lost is the same as dataChunks so be aware that the larger the number of data chunks, the higher the cost of recovery.

The algorithm for erasure coding

Enable determines whether external mode is enabled or not

Enabled whether this filesystem is mirrored or not

Peers represents the peers spec

SnapshotSchedules is the scheduling of snapshot for mirrored filesystems

Retention is the retention policy for a snapshot schedule One path has exactly one retention policy. A policy can however contain multiple count-time period pairs in order to specify complex retention policies

UUID is the peer unique identifier

Remote are the remote cluster information

Stats are the stat a peer mirror

DaemonID is the cephfs-mirror name

Filesystems is the list of filesystems managed by a given cephfs-mirror daemon

PoolMirroringStatus is the mirroring status of a filesystem

LastChecked is the last time time the status was checked

LastChanged is the last time time the status last changed

Details contains potential status errors

The affinity to place the rgw pods (default is to place on any available node)

The annotations-related configuration to add/set on each Pod related object.

The labels-related configuration to add/set on each Pod related object.

The resource requirements for the cephfs-mirror pods

PriorityClassName sets priority class on the cephfs-mirror pods

Start is when the snapshot schedule starts

Created is when the snapshot schedule was created

First is when the first snapshot schedule was taken

Last is when the last snapshot schedule was taken

LastPruned is when the last snapshot schedule was pruned

CreatedCount is total amount of snapshots

PrunedCount is total amount of pruned snapshots

Active is whether the scheduled is active or not

SnapshotSchedules is the list of snapshots scheduled

LastChecked is the last time time the status was checked

LastChanged is the last time time the status last changed

Details contains potential status errors

Fs is the name of the Ceph Filesystem

Subvol is the name of the sub volume

Path is the path on the filesystem

The metadata pool settings

The data pool settings, with optional predefined pool name.

Preserve pools on filesystem deletion

Preserve the fs in the cluster on CephFilesystem CR deletion. Setting this to true automatically implies PreservePoolsOnDelete is true.

The mds pod info

The mirroring settings

The mirroring statusCheck

FilesystemID is the filesystem identifier

Name is name of the filesystem

DirectoryCount is the number of directories in the filesystem

Peers represents the mirroring peers

The Ceph pool used store the shared configuration for NFS-Ganesha daemons. This setting is deprecated, as it is internally required to be “.nfs”.

The namespace inside the Ceph pool (set by ‘pool’) where shared NFS-Ganesha config is stored. This setting is deprecated as it is internally set to the name of the CephNFS.

The number of active Ganesha servers

The affinity to place the ganesha pods

The annotations-related configuration to add/set on each Pod related object.

The labels-related configuration to add/set on each Pod related object.

Resources set resource requests and limits

PriorityClassName sets the priority class on the pods

LogLevel set logging level

Whether host networking is enabled for the Ganesha server. If not set, the network settings from the cluster CR will be applied.

A liveness-probe to verify that Ganesha server has valid run-time state. If LivenessProbe.Disabled is false and LivenessProbe.Probe is nil uses default probe.

The port the rgw service will be listening on (http)

The port the rgw service will be listening on (https)

The number of pods in the rgw replicaset.

The name of the secret that stores the ssl certificate for secure rgw connections

The name of the secret that stores custom ca-bundle with root and intermediate certificates.

The affinity to place the rgw pods (default is to place on any available node)

DisableMultisiteSyncTraffic, when true, prevents this object store’s gateways from transmitting multisite replication data. Note that this value does not affect whether gateways receive multisite replication traffic: see ObjectZone.spec.customEndpoints for that. If false or unset, this object store’s gateways will be able to transmit multisite replication data.

The annotations-related configuration to add/set on each Pod related object.

The labels-related configuration to add/set on each Pod related object.

The resource requirements for the rgw pods

PriorityClassName sets priority classes on the rgw pods

ExternalRgwEndpoints points to external RGW endpoint(s). Multiple endpoints can be given, but for stability of ObjectBucketClaims, we highly recommend that users give only a single external RGW endpoint that is a load balancer that sends requests to the multiple RGWs.

The configuration related to add/set on each rgw service.

Whether host networking is enabled for the rgw daemon. If not set, the network settings from the cluster CR will be applied.

Whether rgw dashboard is enabled for the rgw daemon. If not set, the rgw dashboard will be enabled.

AdditionalVolumeMounts allows additional volumes to be mounted to the RGW pod. The root directory for each additional volume mount is /var/rgw. Example: for an additional mount at subPath ldap, mounted from a secret that has key bindpass.secret, the file would reside at /var/rgw/ldap/bindpass.secret.

The URI of the HTTP endpoint to push notification to

Indicate whether the server certificate is validated by the client or not

Send the notifications with the CloudEvents header: https://github.com/cloudevents/spec/blob/main/cloudevents/adapters/aws-s3.md

Interval is the internal in second or minute for the health check to run like 60s for 60 seconds

PrimaryDeviceClass represents high performance tier (for example SSD or NVME) for Primary OSD

SecondaryDeviceClass represents low performance tier (for example HDDs) for remaining OSDs

"IPv4"

IPv4 internet protocol version

"IPv6"

IPv6 internet protocol version

""

"false"

"s3"

"swift"

"true"

The URI of the Kafka endpoint to push notification to

Indicate whether to use SSL when communicating with the broker

Indicate whether the server certificate is validated by the client or not

The ack level required for this topic (none/broker)

VolumeSource accepts a pared down version of the standard Kubernetes VolumeSource for Kerberos configuration files like what is normally used to configure Volumes for a Pod. For example, a ConfigMap, Secret, or HostPath. The volume may contain multiple files, all of which will be loaded.

VolumeSource accepts a pared down version of the standard Kubernetes VolumeSource for the Kerberos keytab file like what is normally used to configure Volumes for a Pod. For example, a Secret or HostPath. There are two requirements for the source’s content: 1. The config file must be mountable via subPath: krb5.keytab. For example, in a Secret, the data item must be named krb5.keytab, or items must be defined to select the key and give it path krb5.keytab. A HostPath directory must have the krb5.keytab file. 2. The volume or config file must have mode 0600.

PrincipalName corresponds directly to NFS-Ganesha’s NFS_KRB5:PrincipalName config. In practice, this is the service prefix of the principal name. The default is “nfs”. This value is combined with (a) the namespace and name of the CephNFS (with a hyphen between) and (b) the Realm configured in the user-provided krb5.conf to determine the full principal name: /-@. e.g., nfs/rook-ceph-my-nfs@example.net. See https://github.com/nfs-ganesha/nfs-ganesha/wiki/RPCSEC_GSS for more detail.

DomainName should be set to the Kerberos Realm.

ConfigFiles defines where the Kerberos configuration should be sourced from. Config files will be placed into the /etc/krb5.conf.rook/ directory.

If this is left empty, Rook will not add any files. This allows you to manage the files yourself however you wish. For example, you may build them into your custom Ceph container image or use the Vault agent injector to securely add the files via annotations on the CephNFS spec (passed to the NFS server pods).

Rook configures Kerberos to log to stderr. We suggest removing logging sections from config files to avoid consuming unnecessary disk space from logging to files.

KeytabFile defines where the Kerberos keytab should be sourced from. The keytab file will be placed into /etc/krb5.keytab. If this is left empty, Rook will not add the file. This allows you to manage the krb5.keytab file yourself however you wish. For example, you may build it into your custom Ceph container image or use the Vault agent injector to securely add the file via annotations on the CephNFS spec (passed to the NFS server pods).

ConnectionDetails contains the KMS connection details (address, port etc)

TokenSecretName is the kubernetes secret containing the KMS token

Enabled represents whether the key rotation is enabled.

Schedule represents the cron schedule for key rotation.

"exporter"

"cleanup"

"clusterMetadata"

"cmdreporter"

"crashcollector"

"dashboard"

"mds"

"mgr"

"mon"

"arbiter"

"monitoring"

"osd"

"prepareosd"

"rgw"

"keyrotation"

The URL for the Keystone server.

The name of the secret containing the credentials for the service user account used by RGW. It has to be in the same namespace as the object store resource.

The roles requires to serve requests.

Create new users in their own tenants of the same name. Possible values are true, false, swift and s3. The latter have the effect of splitting the identity space such that only the indicated protocol will use implicit tenants.

The maximum number of entries in each Keystone token cache.

The number of seconds between token revocation checks.

Enabled represents whether the log collector is enabled

Periodicity is the periodicity of the log rotation.

MaxLogSize is the maximum size of the log per ceph daemons. Must be at least 1M.

The number of metadata servers that are active. The remaining servers in the cluster will be in standby mode.

Whether each active MDS instance will have an active standby with a warm metadata cache for faster failover. If false, standbys will still be available, but will not have a warm metadata cache.

The affinity to place the mds pods (default is to place on all available node) with a daemonset

The annotations-related configuration to add/set on each Pod related object.

The labels-related configuration to add/set on each Pod related object.

The resource requirements for the mds pods

PriorityClassName sets priority classes on components

Count is the number of manager daemons to run

AllowMultiplePerNode allows to run multiple managers on the same node (not recommended)

Modules is the list of ceph manager modules to enable/disable

Mode is the mirroring mode

SiteName is the current site name

Peers are the list of peer sites connected to that cluster

(Members of MirroringInfo are embedded into this type.)

SecretNames represents the Kubernetes Secret names to add rbd-mirror or cephfs-mirror peers

Enabled whether this pool is mirrored or not

Mode is the mirroring mode: either pool or image

SnapshotSchedules is the scheduling of snapshot for mirrored images/pools

Peers represents the peers spec

Summary is the mirroring status summary

(Members of MirroringStatus are embedded into this type.)

MirroringStatus is the mirroring status of a pool/radosNamespace

LastChecked is the last time time the status was checked

LastChanged is the last time time the status last changed

Details contains potential status errors

Health is the mirroring health

DaemonHealth is the health of the mirroring daemon

ImageHealth is the health of the mirrored image

States is the various state for all mirrored images

Name is the name of the ceph manager module

Enabled determines whether a module should be enabled or not

Settings to further configure the module

BalancerMode sets the balancer module with different modes like upmap, crush-compact etc

Count is the number of Ceph monitors

AllowMultiplePerNode determines if we can run multiple monitors on the same node (not recommended)

Zones are specified when we want to provide zonal awareness to mons

StretchCluster is the stretch cluster specification

VolumeClaimTemplate is the PVC definition

Name is the name of the zone

Arbiter determines if the zone contains the arbiter used for stretch cluster mode

VolumeClaimTemplate is the PVC template

Enabled determines whether to create the prometheus rules for the ceph cluster. If true, the prometheus types must exist or the creation will fail. Default is false.

Whether to disable the metrics reported by Ceph. If false, the prometheus mgr module and Ceph exporter are enabled. If true, the prometheus mgr module and Ceph exporter are both disabled. Default is false.

ExternalMgrEndpoints points to an existing Ceph prometheus exporter endpoint

ExternalMgrPrometheusPort Prometheus exporter port

Port is the prometheus server port

Interval determines prometheus scrape interval

Ceph exporter configuration

Enable multiClusterService to export the mon and OSD services to peer cluster. Ensure that peer clusters are connected using an MCS API compatible application, like Globalnet Submariner.

ClusterID uniquely identifies a cluster. It is used as a prefix to nslookup exported services. For example: ...svc.clusterset.local

RADOS is the Ganesha RADOS specification

Server is the Ganesha Server specification

Security allows specifying security configurations for the NFS cluster

SSSD enables integration with System Security Services Daemon (SSSD). SSSD can be used to provide user ID mapping from a number of sources. See https://sssd.io for more information about the SSSD project.

Kerberos configures NFS-Ganesha to secure NFS client connections with Kerberos.

The desired name of the pool if different from the CephBlockPool CR name.

(Members of PoolSpec are embedded into this type.)

The core pool configuration

Name of the pool

(Members of PoolSpec are embedded into this type.)

PoolSpec represents the spec of ceph pool

""

"host"

"multus"

Provider is what provides network connectivity to the cluster e.g. “host” or “multus”. If the Provider is updated from being empty to “host” on a running cluster, then the operator will automatically fail over all the mons to apply the “host” network settings.

Selectors define NetworkAttachmentDefinitions to be used for Ceph public and/or cluster networks when the “multus” network provider is used. This config section is not used for other network providers.

Valid keys are “public” and “cluster”. Refer to Ceph networking documentation for more: https://docs.ceph.com/en/latest/rados/configuration/network-config-ref/

Refer to Multus network annotation documentation for help selecting values: https://github.com/k8snetworkplumbingwg/multus-cni/blob/master/docs/how-to-use.md#run-pod-with-network-annotation

Rook will make a best-effort attempt to automatically detect CIDR address ranges for given network attachment definitions. Rook’s methods are robust but may be imprecise for sufficiently complicated networks. Rook’s auto-detection process obtains a new IP address lease for each CephCluster reconcile. If Rook fails to detect, incorrectly detects, only partially detects, or if underlying networks do not support reusing old IP addresses, it is best to use the ‘addressRanges’ config section to specify CIDR ranges for the Ceph cluster.

As a contrived example, one can use a theoretical Kubernetes-wide network for Ceph client traffic and a theoretical Rook-only network for Ceph replication traffic as shown: selectors: public: “default/cluster-fast-net” cluster: “rook-ceph/ceph-backend-net”

AddressRanges specify a list of CIDRs that Rook will apply to Ceph’s ‘public_network’ and/or ‘cluster_network’ configurations. This config section may be used for the “host” or “multus” network providers.

Settings for network connections such as compression and encryption across the wire.

HostNetwork to enable host network. If host networking is enabled or disabled on a running cluster, then the operator will automatically fail over all the mons to apply the new network settings.

IPFamily is the single stack IPv6 or IPv4 protocol

DualStack determines whether Ceph daemons should listen on both IPv4 and IPv6

Enable multiClusterService to export the Services between peer clusters

(Members of Selection are embedded into this type.)

Name of the metadata or tag

Value to filter on

Filters based on the object’s key

Filters based on the object’s metadata

Filters based on the object’s tags

Name of the filter - prefix/suffix/regex

Value to filter on

StoreType is a mapping between the OSD backend stores and number of OSDs using these stores

Type of backend storage to be used while creating OSDs. If empty, then bluestore will be used

UpdateStore updates the backend store for existing OSDs. It destroys each OSD one at a time, cleans up the backing disk and prepares same OSD on that disk

DnsName is the DNS name (in RFC-1123 format) of the endpoint. If the DNS name corresponds to an endpoint with DNS wildcard support, do not include the wildcard itself in the list of hostnames. E.g., use “mystore.example.com” instead of “*.mystore.example.com”.

Port is the port on which S3 connections can be made for this endpoint.

UseTls defines whether the endpoint uses TLS (HTTPS) or not (HTTP).

The metadata pool used for creating RADOS namespaces in the object store

The data pool used for creating RADOS namespaces in the object store

Whether the RADOS namespaces should be preserved on deletion of the object store

PoolPlacements control which Pools are associated with a particular RGW bucket. Once PoolPlacements are defined, RGW client will be able to associate pool with ObjectStore bucket by providing “” during s3 bucket creation or “X-Storage-Policy” header during swift container creation. See: https://docs.ceph.com/en/latest/radosgw/placement/#placement-targets PoolPlacement with name: “default” will be used as a default pool if no option is provided during bucket creation. If default placement is not provided, spec.sharedPools.dataPoolName and spec.sharedPools.MetadataPoolName will be used as default pools. If spec.sharedPools are also empty, then RGW pools (spec.dataPool and spec.metadataPool) will be used as defaults.

AdvertiseEndpoint is the default endpoint Rook will return for resources dependent on this object store. This endpoint will be returned to CephObjectStoreUsers, Object Bucket Claims, and COSI Buckets/Accesses. By default, Rook returns the endpoint for the object store’s Kubernetes service using HTTPS with gateway.securePort if it is defined (otherwise, HTTP with gateway.port).

A list of DNS host names on which object store gateways will accept client S3 connections. When specified, object store gateways will reject client S3 connections to hostnames that are not present in this list, so include all endpoints. The object store’s advertiseEndpoint and Kubernetes service endpoint, plus CephObjectZone customEndpoints are automatically added to the list but may be set here again if desired. Each DNS name must be valid according RFC-1123. If the DNS name corresponds to an endpoint with DNS wildcard support, do not include the wildcard itself in the list of hostnames. E.g., use “mystore.example.com” instead of “*.mystore.example.com”.

The settings for supporting AWS-SSE:S3 with RGW

The metadata pool settings

The data pool settings

The pool information when configuring RADOS namespaces in existing pools.

Preserve pools on object store deletion

The rgw pod info

The protocol specification

The authentication configuration

The multisite info

The RGW health probes

Security represents security settings

The list of allowed namespaces in addition to the object store namespace where ceph object store users may be created. Specify “*” to allow all namespaces, otherwise list individual namespaces that are to be allowed. This is useful for applications that need object store credentials to be created in their own namespace, where neither OBCs nor COSI is being used to create buckets. The default is empty.

Hosting settings for the object store. A common use case for hosting configuration is to inform Rook of endpoints that support DNS wildcards, which in turn allows virtual host-style bucket addressing.

ObservedGeneration is the latest generation observed by the controller.

The store the user will be created in

The display name for the ceph users

The namespace where the parent CephCluster and CephObjectStore are found

ObservedGeneration is the latest generation observed by the controller.

Admin capabilities to read/write Ceph object store users. Documented in https://docs.ceph.com/en/latest/radosgw/admin/?#add-remove-admin-capabilities

Admin capabilities to read/write Ceph object store users. Documented in https://docs.ceph.com/en/latest/radosgw/admin/?#add-remove-admin-capabilities

Admin capabilities to read/write Ceph object store buckets. Documented in https://docs.ceph.com/en/latest/radosgw/admin/?#add-remove-admin-capabilities

Admin capabilities to read/write Ceph object store buckets. Documented in https://docs.ceph.com/en/latest/radosgw/admin/?#add-remove-admin-capabilities

Admin capabilities to read/write Ceph object store metadata. Documented in https://docs.ceph.com/en/latest/radosgw/admin/?#add-remove-admin-capabilities

Admin capabilities to read/write Ceph object store usage. Documented in https://docs.ceph.com/en/latest/radosgw/admin/?#add-remove-admin-capabilities

Admin capabilities to read/write Ceph object store zones. Documented in https://docs.ceph.com/en/latest/radosgw/admin/?#add-remove-admin-capabilities

Admin capabilities to read/write roles for user. Documented in https://docs.ceph.com/en/latest/radosgw/admin/?#add-remove-admin-capabilities

Admin capabilities to read/write information about the user. Documented in https://docs.ceph.com/en/latest/radosgw/admin/?#add-remove-admin-capabilities

Add capabilities for user to send request to RGW Cache API header. Documented in https://docs.ceph.com/en/latest/radosgw/rgw-cache/#cache-api

Add capabilities for user to change bucket index logging. Documented in https://docs.ceph.com/en/latest/radosgw/admin/?#add-remove-admin-capabilities

Add capabilities for user to change metadata logging. Documented in https://docs.ceph.com/en/latest/radosgw/admin/?#add-remove-admin-capabilities

Add capabilities for user to change data logging. Documented in https://docs.ceph.com/en/latest/radosgw/admin/?#add-remove-admin-capabilities

Add capabilities for user to change user policies. Documented in https://docs.ceph.com/en/latest/radosgw/admin/?#add-remove-admin-capabilities

Add capabilities for user to change oidc provider. Documented in https://docs.ceph.com/en/latest/radosgw/admin/?#add-remove-admin-capabilities

Add capabilities for user to set rate limiter for user and bucket. Documented in https://docs.ceph.com/en/latest/radosgw/admin/?#add-remove-admin-capabilities

Maximum bucket limit for the ceph user

Maximum size limit of all objects across all the user’s buckets See https://pkg.go.dev/k8s.io/apimachinery/pkg/api/resource#Quantity for more info.

Maximum number of objects across all the user’s buckets

The display name for the ceph users

The display name for the ceph users

The metadata pool settings

The data pool settings

The pool information when configuring RADOS namespaces in existing pools.

If this zone cannot be accessed from other peer Ceph clusters via the ClusterIP Service endpoint created by Rook, you must set this to the externally reachable endpoint(s). You may include the port in the definition. For example: “https://my-object-store.my-domain.net:443”. In many cases, you should set this to the endpoint of the ingress resource that makes the CephObjectStore associated with this CephObjectStoreZone reachable to peer clusters. The list can have one or more endpoints pointing to different RGW servers in the zone.

If a CephObjectStore endpoint is omitted from this list, that object store’s gateways will not receive multisite replication data (see CephObjectStore.spec.gateway.disableMultisiteSyncTraffic).

Preserve pools on object zone deletion

ClientName is cephx name

ClusterName is the name of the cluster

FsName is the filesystem name

FailureCount is the number of mirroring failure

RecoveryCount is the number of recovery attempted after failures

UUID is the peer UUID

Direction is the peer mirroring direction

SiteName is the current site name

MirrorUUID is the mirror UUID

ClientName is the CephX user used to connect to the peer

NodeAffinity is a group of node affinity scheduling rules

PodAffinity is a group of inter pod affinity scheduling rules

PodAntiAffinity is a group of inter pod anti affinity scheduling rules

The pod this Toleration is attached to tolerates any taint that matches the triple using the matching operator

TopologySpreadConstraints specifies how to spread matching pods among the given topology

Name is the StorageClass name. Ceph allows arbitrary name for StorageClasses, however most clients/libs insist on AWS names so it is recommended to use one of the valid x-amz-storage-class values for better compatibility: REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING | GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE See AWS docs: https://aws.amazon.com/de/s3/storage-classes/

DataPoolName is the data pool used to store ObjectStore objects data.

Pool placement name. Name can be arbitrary. Placement with name “default” will be used as default.

Sets given placement as default. Only one placement in the list can be marked as default. Default is false.

The metadata pool used to store ObjectStore bucket index.

The data pool used to store ObjectStore objects data.

The data pool used to store ObjectStore data that cannot use erasure coding (ex: multi-part uploads). If dataPoolName is not erasure coded, then there is no need for dataNonECPoolName.

StorageClasses can be selected by user to override dataPoolName during object creation. Each placement has default STANDARD StorageClass pointing to dataPoolName. This list allows defining additional StorageClasses on top of default STANDARD storage class.

The failure domain: osd/host/(region or zone if available) - technically also any type in the crush map

The root of the crush hierarchy utilized by the pool

The device class the OSD should set to for use in the pool

Allow rook operator to change the pool CRUSH tunables once the pool is created

DEPRECATED: use Parameters instead, e.g., Parameters[“compression_mode”] = “force” The inline compression mode in Bluestore OSD to set to (options are: none, passive, aggressive, force) Do NOT set a default value for kubebuilder as this will override the Parameters

The replication settings

The erasure code settings

Parameters is a list of properties to enable on a given pool

EnableRBDStats is used to enable gathering of statistics for all RBD images in the pool

The mirroring settings

The mirroring statusCheck

The quota settings

The application name to set on the pool. Only expected to be set for rgw pools.

Disabled determines whether probe is disable or not

Probe describes a health check to be performed against a container to determine whether it is alive or ready to receive traffic.

The spec for S3

The spec for Swift

MaxBytes represents the quota in bytes Deprecated in favor of MaxSize

MaxSize represents the quota in bytes as a string

MaxObjects represents the quota in objects

Count represents the number of rbd mirror instance to run

Peers represents the peers spec

The affinity to place the rgw pods (default is to place on any available node)

The annotations-related configuration to add/set on each Pod related object.

The labels-related configuration to add/set on each Pod related object.

The resource requirements for the rbd mirror pods

PriorityClassName sets priority class on the rbd mirror pods

The annotations-related configuration to add/set on each rgw service. nullable optional

RemoteNamespace is the name of the CephBlockPoolRadosNamespace on the secondary cluster CephBlockPool

Mode is the mirroring mode; either pool or image

SnapshotSchedules is the scheduling of snapshot for mirrored images

"image"

RadosNamespaceMirroringModeImage represents the image mode

"pool"

RadosNamespaceMirroringModePool represents the pool mode

Enables read affinity for CSI driver.

CrushLocationLabels defines which node labels to use as CRUSH location. This should correspond to the values set in the CRUSH map.

Size - Number of copies per object in a replicated storage pool, including the object itself (required for replicated pool type)

TargetSizeRatio gives a hint (%) to Ceph in terms of expected consumption of the total cluster capacity

RequireSafeReplicaSize if false allows you to set replica 1

ReplicasPerFailureDomain the number of replica in the specified failure domain

SubFailureDomain the name of the sub-failure domain

HybridStorage represents hybrid storage tier settings

Whether to enable S3. This defaults to true (even if protocols.s3 is not present in the CRD). This maintains backwards compatibility – by default S3 is enabled.

Whether to use Keystone for authentication. This option maps directly to the rgw_s3_auth_use_keystone option. Enabling it allows generating S3 credentials via an OpenStack API call, see the docs. If not given, the defaults of the corresponding RGW option apply.

Image defines the container image that should be used for the SSSD sidecar.

SSSDConfigFile defines where the SSSD configuration should be sourced from. The config file will be placed into /etc/sssd/sssd.conf. If this is left empty, Rook will not add the file. This allows you to manage the sssd.conf file yourself however you wish. For example, you may build it into your custom Ceph container image or use the Vault agent injector to securely add the file via annotations on the CephNFS spec (passed to the NFS server pods).

AdditionalFiles defines any number of additional files that should be mounted into the SSSD sidecar with a directory root of /etc/sssd/rook-additional/. These files may be referenced by the sssd.conf config file.

Resources allow specifying resource requests/limits on the SSSD sidecar container.

DebugLevel sets the debug level for SSSD. If unset or set to 0, Rook does nothing. Otherwise, this may be a value between 1 and 10. See SSSD docs for more info: https://sssd.io/troubleshooting/basics.html#sssd-debug-logs

VolumeSource accepts a pared down version of the standard Kubernetes VolumeSource for the SSSD configuration file like what is normally used to configure Volumes for a Pod. For example, a ConfigMap, Secret, or HostPath. There are two requirements for the source’s content: 1. The config file must be mountable via subPath: sssd.conf. For example, in a ConfigMap, the data item must be named sssd.conf, or items must be defined to select the key and give it path sssd.conf. A HostPath directory must have the sssd.conf file. 2. The volume or config file must have mode 0600.