Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 2. Installing and Configuring Ceph Metadata Servers (MDS)

The Ceph Metadata Server (MDS) node runs the MDS daemon (ceph-mds), which manages metadata related to files stored on the Ceph File System. The MDS daemon also coordinates access to the shared Ceph Storage Cluster.

2.1. Prerequisites

The following procedure assumes that:

2.2. Installing a Ceph Metadata Server

Use the Ansible automation application to install a Ceph Metadata Server. Perform the following steps on the Ansible administration server:

  1. Add a new section [mdss] to the /etc/ansible/hosts file:

    Copy to Clipboard Toggle word wrap 
    [mdss]
<mdss-hostname>

    Replace <mdss-hostname> with the host name of the node where you want to install the Ceph Metadata Server.

  2. Navigate to the Ansible configuration directory, /usr/share/ceph-ansible/:

    Copy to Clipboard Toggle word wrap 
    $ cd /usr/share/ceph-ansible

  3. Run the Ansible playbook:

    Copy to Clipboard Toggle word wrap 
    $ ansible-playbook site.yml

2.3. Configuring a Ceph Metadata Server

The Ceph Metadata Servers (MDS) have two modes:

  • active
  • standby

The first MDS that you started becomes active. The rest of the MDS are in standby mode.

When the active MDS becomes unresponsive, the monitor will wait the number of seconds specified by the mds_beacon_grace option. Then the monitor marks the MDS as laggy. When this happens, one of the standby servers becomes active depending on your configuration. See Section 2.3.2, “Configuring Standby Daemons” for details.

To change the value of mds_beacon_grace, add this option to the Ceph configuration file and specify the new value.

2.3.1. Terminology

FSCID

A Ceph cluster can have zero or more Ceph File Systems. Ceph File Systems have a human readable name (set by the fs new command) and an integer ID. The ID is called the File System Cluster ID, or FSCID.

Ranks

Each Ceph File System has a number of ranks, one by default, which start at zero.

Ranks are how the metadata workload is shared between multiple MDS (ceph-mds) daemons. The number of ranks is the maximum number of MDS daemons that may be active at one time. Each MDS handles the subset of the file system metadata that is assigned to that rank.

Each MDS daemon initially starts without a rank. The monitor cluster assigns a rank to the daemon. An MDS daemon can only hold one rank at a time. Daemons only lose ranks when they are stopped.

Ranks can be:

  • Up - A rank is up once it is assigned to a daemon.
  • Failed - A rank is failed if it is not associated with an instance of the MDS daemon.
  • Damaged - A rank is damaged when its metadata is corrupted or missing. Damaged ranks will not be assigned to any MDS daemon until you fix the problem and use the ceph mds repaired command on the damaged rank.

The max_mds setting controls how many ranks will be created.

The actual number of ranks in the file system is only increased if a spare daemon is available to accept the new rank.

Daemon name

Each daemon has a static name that is set by the administrator when configuring the daemon for the first time. Usually, the host name of the host where the daemon runs is used as the daemon name.

GID

Each time a daemon starts, it is also assigned a GID, which is unique to the process lifetime of the daemon.

Referring to MDS daemons

Most of the administrative commands that refer to MDS daemons accept a flexible argument format.

A rank can be optionally qualified with a leading file system name or ID. If a daemon is in standby mode (meaning that it does not currently have a rank assigned), it can only be referred to by GID or name.

For example, an MDS daemon is called myhost and has GID 5446. It was assigned rank 0 in the file system myfs, which has FSCID 3. The following examples show possible forms of the fail command:

Copy to Clipboard Toggle word wrap 
ceph mds fail 5446     # GID
ceph mds fail myhost   # Daemon name
ceph mds fail 0        # Unqualified rank
ceph mds fail 3:0      # FSCID and rank
ceph mds fail myfs:0   # File system name and rank

2.3.2. Configuring Standby Daemons

There are four configuration settings that control how daemons behave in standby mode:

These settings can be set in the Ceph configuration file (ceph.conf by default) on the host where the MDS daemon runs as opposed to the one on the monitor node. The MDS daemon loads these settings when it starts and sends them to the monitor node.

By default, if none of these settings are used, all MDS daemons that do not hold a rank will be used as standby daemons for any rank.

Standby Replay

When the mds_standby_replay option is set to true for a daemon, this daemon will continuously read the metadata journal of a rank associated with another MDS daemon (the up rank). This behavior gives the standby replay daemon a more recent metadata cache and makes the failover process faster if the daemon serving the rank fails.

An up rank can only have one standby replay daemon assigned to it. If two daemons are both set to be standby replay then one of them becomes a normal non-replay standby daemon.

If the mon_force_standby_active option is set to false, then a standby replay daemon is only used as a standby for the rank that it is following. If another rank fails, the standby replay daemon will not be used as a replacement, even if no other standby daemons are available. By default, mon_force_standby_active is set to true.

Standby for Name

When setting the mds_standby_for_name option, the standby daemon only takes over a failed rank if the name of the daemon that previously held the rank matches the given name.

Standby for Rank

Set the mds_standby_for_rank option to configure the standby daemon to only take over the specified rank. If another rank fails, this daemon will not replace it.

If you have multiple file systems, use this option in conjunction with the mds_standby_for_fscid option to specify which file system rank you target.

Standby for FSCID

If the mds_standby_for_fscid option is used in conjunction with mds_standby_for_rank it only specifies which file system rank is referred to.

If mds_standby_for_rank is not set, then setting mds_standby_for_fscid causes the standby daemon to target any rank in the specified FSCID.

Use mds_standby_for_fscid if you want to use the standby daemon for any rank, but only within a particular file system.

For more information about MSD configuration options, see Configuration Reference.

Configuration Examples

The following examples of parts of the Ceph configuration file can be:

  • in the main Ceph configuration file present on all servers
  • in different configuration files on each server that contain just configuration related to that server

Simple pair

Two MDS daemons 'a' and 'b' acting as a pair, where whichever one has not currently assigned a rank will be the standby replay follower of the other:

Copy to Clipboard Toggle word wrap 
  [mds.a]
  mds_standby_replay = true
  mds_standby_for_rank = 0

  [mds.b]
  mds_standby_replay = true
  mds_standby_for_rank = 0

Two MDS clusters

There are two file systems and four MDS daemons, each file has a pair of daemons:

Copy to Clipboard Toggle word wrap 
  [mds.a]
  mds_standby_for_fscid = 1

  [mds.b]
  mds_standby_for_fscid = 1

  [mds.c]
  mds_standby_for_fscid = 2

  [mds.d]
  mds_standby_for_fscid = 2
Back to top
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2025 Red Hat, Inc.