Chapter 1. The basics of Ceph configuration

All Red Hat Ceph Storage clusters have a configuration, which defines:

A deployment tool such as Ansible will typically create an initial Ceph configuration file for you. However, you can create one yourself if you prefer to bootstrap a Red Hat Ceph Storage cluster without using a deployment tool.

The Ceph Monitor manages a configuration database of Ceph options which centralizes configuration management by storing configuration options for the entire storage cluster. By centralizing the Ceph configuration in a database, this helps with storage cluster administration. There are still a handful of Ceph options that can be defined in the local Ceph configuration file, by default, /etc/ceph/ceph.conf. These few Ceph configuration options control how other Ceph components connect to the Ceph Monitors to authenticate, and fetch the configuration information from the database.

Ceph allows you to make changes to the configuration of a daemon at runtime. This capability can be useful for increasing or decreasing the logging output, by enabling or disabling debug settings, and can even be used for runtime optimization.

Masks have two forms:

The Ceph configuration file configures the Ceph daemons at start time, which will override their default values.

The location of Ceph’s default configuration file is /etc/ceph/ceph.conf. You can change that location by setting a different path by:

Ceph configuration files use an ini style syntax. You can add comments by preceding comments with a pound sign (#) or a semi-colon (;).

# <--A pound sign (#) sign precedes a comment.
# Comments always follow a semi-colon (;) or a pound (#) on each line.
# The end of the line terminates a comment.
# We recommend that you provide comments in your configuration file(s).
; A comment may be anything.

The configuration file can configure all Ceph daemons in a Ceph storage cluster or all Ceph daemons of a particular type at start time. To configure a series of daemons, the settings must be included under the processes that will receive the configuration as follows:

    [mon.host01]
     `host = host01`
     `mon_addr = 10.0.0.101`
    [mon.host02]
     `host = host02`
     `mon_addr = 10.0.0.102`

Global settings affect all instances of all daemon in the Ceph storage cluster. Use the [global] heading for values that are common for all daemons in the Ceph storage cluster. You can override each [global] option by:

OR

Overriding a global setting affects all child processes, except those that you specifically override in a particular daemon.

A typical global setting involves activating authentication.

[global]
#Enable authentication between hosts within the cluster.
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx

You can specify settings that apply to a particular type of daemon. When you specify settings under [osd] or [mon] without specifying a particular instance, the setting will apply to all OSD or monitor daemons respectively. One example of a daemon-wide setting is the osd memory target.

[osd]
osd_memory_target = 5368709120

You can specify settings for particular instances of a daemon. You might specify an instance by entering its type, delimited by a period (.) and by the instance ID. The instance ID for a Ceph OSD daemons is always numeric, but it may be alphanumeric for Ceph monitors.

[osd.1]
# settings affect osd.1 only.

[mon.a]
# settings affect mon.a only.

A typical Ceph configuration file has at least the following settings:

[global]
fsid = UNIQUE_CLUSTER_ID
mon_initial_members = NODE_NAME[, NODE_NAME]
mon_host = IP_ADDRESS[, IP_ADDRESS]

#All clusters have a front-side public network.
#If you have two NICs, you can configure a back side cluster
#network for OSD object replication, heart beats, backfilling,
#recovery, and so on
public_network = PUBLIC_NET[, PUBLIC_NET]
#cluster_network = PRIVATE_NET[, PRIVATE_NET]

#Clusters require authentication by default.
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx

#Choose reasonable numbers for your number of replicas
#and placement groups.
osd_pool_default_size = NUM  # Write an object n times.
osd_pool_default_min_size = NUM # Allow writing n copy in a degraded state.
osd_pool_default_pg_num = NUM
osd_pool_default_pgp_num = NUM

#Choose a reasonable crush leaf type.
#0 for a 1-node cluster.
#1 for a multi node cluster in a single rack
#2 for a multi node, multi chassis cluster with multiple hosts in a chassis
#3 for a multi node cluster with hosts across racks, and so on
osd_crush_chooseleaf_type = NUM

[global]
cluster network = 10.74.250.101/21
fsid = 3e07d43f-688e-4284-bfb7-3e6ed5d3b77b
mon host = [v2:10.0.0.101:3300/0,v1:10.0.0.101:6789/0] [v2:10.0.0.102:3300/0,v1:10.0.0.102:6789/0] [v2:10.0.0.103:3300/0,v1:10.0.0.103:6789/0]
mon initial members = host01, host02, host03
osd pool default crush rule = -1
public network = 10.74.250.101/21

[osd]
osd memory target = 4294967296

[mon]
    [mon.host01]
     host = host01
     mon_addr = 10.0.0.101
    [mon.host02]
     host = host02
     mon_addr = 10.0.0.102

Metavariables simplify Ceph storage cluster configuration dramatically. When a metavariable is set in a configuration value, Ceph expands the metavariable into a concrete value.

Metavariables are very powerful when used within the [global], [osd], [mon], or [client] sections of the Ceph configuration file. However, you can also use them with the administration socket. Ceph metavariables are similar to Bash shell expansion.

Ceph supports the following metavariables:

The Ceph configuration files can be viewed at boot time and run time.

Configuration settings for Red Hat Ceph Storage can be viewed at runtime from the Ceph Monitor node.

There are two general ways to set a runtime configuration.

You can set a Ceph runtime configuration option by contacting the monitor using the tell and injectargs command.

BlueStore keeps OSD heap memory usage under a designated target size with the osd_memory_target configuration option.

The option osd_memory_target sets OSD memory based upon the available RAM in the system. By default, Ansible sets the value to 4 GB. You can change the value, expressed in bytes, in the /usr/share/ceph-ansible/group_vars/all.yml file when deploying the daemon. You can also use the Ceph overrides in the ceph.conf file to manually set the osd memory target, for example to 6 GB.

ceph_conf_overrides:
  osd:
    osd memory target: 6442450944

Ceph OSD memory caching is more important when the block device is slow, for example, traditional hard drives, because the benefit of a cache hit is much higher than it would be with a solid state drive. However, this has to be weighed-in to co-locate OSDs with other services, such as in a hyper-converged infrastructure (HCI), or other applications.

MDS servers keep their metadata in a separate storage pool, named cephfs_metadata, and are the users of Ceph OSDs. For Ceph File Systems, MDS servers have to support an entire Red Hat Ceph Storage cluster, not just a single storage device within the storage cluster, so their memory requirements can be significant, particularly if the workload consists of small-to-medium-size files, where the ratio of metadata to data is much higher.

Example: Set the mds_cache_memory_limit to 2 GiB

ceph_conf_overrides:
  osd:
    mds_cache_memory_limit: 2147483648