Red Hat Summit 红帽全球峰会支持控制台（Console）开发人员开始试用联系人

此内容没有您所选择的语言版本。

Chapter 6. Managing Cluster Size

Managing cluster size generally involves adding or removing Ceph Monitors or OSDs. If you are bootstrapping a cluster for the first time, see the Red Hat Ceph Storage 2 Installation Guide for Red Hat Enterprise Linux or Ubuntu.

6.1. Adding a Monitor
复制链接

Ceph Monitors are light-weight processes that maintain a master copy of the cluster map. All Ceph clients contact a Ceph Monitor and retrieve the current copy of the cluster map, enabling clients to bind to a pool and read and write data.

Ceph Monitors use a variation of the Paxos protocol to establish consensus about maps and other critical information across the cluster. Due to the nature of Paxos, Ceph requires a majority of Monitors running to establish a quorum thus establishing consensus.

Note

Red Hat requires at least three Monitors on separate hosts to receive support for a production cluster.

Red Hat recommends deploying an odd number of Monitors, but it is not mandatory. An odd number of Monitors has a higher resiliency to failures than an even number of Monitors. For example, to maintain a quorum on a two Monitor deployment, Ceph cannot tolerate any failures; with three Monitors, one failure; with four Monitors, one failure; with five Monitors, two failures. This is why an odd number is advisable. Summarizing, Ceph needs a majority of Monitors to be running and to be able to communicate with each other, two out of three, three out of four, and so on.

For an initial deployment of a multi-node Ceph cluster, Red Hat requires three Monitors, increasing the number two at a time if a valid need for more than three Monitors exists.

Since Monitors are light-weight, it is possible to run them on the same host as OpenStack nodes. However, Red Hat recommends running Monitors on separate hosts. Co-locating Monitors and OSDs on the same node can impair performance and is not supported.

Note

A majority of Monitors in the storage cluster must be able to reach each other in order to establish a quorum.

6.1.1. Configuring a Host
复制链接

When adding Ceph Monitors to a cluster, deploy them on separate nodes. Running Ceph Monitors on the same node does not provide any additional high availability assurance if a node fails. Ideally, the node hardware should be uniform for all Monitor nodes in the storage cluster.

For details on the minimum recommendations for Ceph Monitor hardware, see the Red Hat Ceph Storage Hardware Guide.

Before doing a new installation, review the requirements listed in the Prerequisites section in the Red Hat Ceph Storage Installation Guide for Red Hat Enterprise Linux or Ubuntu.

Add a new Monitor node to a rack in the cluster, connect it to the network and ensure that the Monitor has network connectivity.

Important

You must install NTP and open port 6789.

6.1.2. Adding a Monitor with Ansible
复制链接

Before You Start…​

Red Hat recommends adding two Monitors at a time to maintain an odd number of Monitors. For example, if you have a three Monitor cluster, Red Hat recommends expanding it to a five Monitor cluster.

To add a new Ceph Monitor to an existing Ansible cluster, simply open the /etc/ansible/hosts file and add another Monitor node under the [mons] section. Next, run the Ansible playbook again.

  1. As root, add the Ceph Monitor hosts to the /etc/ansible/hosts file. Add Monitors under a [mons] section: 

    [mons]
<monitor-host-name>
<monitor-host-name>
<monitor-host-name>
<new-monitor-host-name>
<new-monitor-host-name>
    Copy to Clipboard Toggle word wrap

  2. Ensure that Ansible can reach the Ceph nodes: 

    # ansible all -m ping
    Copy to Clipboard Toggle word wrap

  3. Navigate to the Ansible configuration directory: 

    # cd /usr/share/ceph-ansible
    Copy to Clipboard Toggle word wrap

  4. Run the Ansible playbook: 

    $ ansible-playbook site.yml
    Copy to Clipboard Toggle word wrap

    If adding a monitor in Ceph deployed in a container: 

    $ ansible-playbook site-docker.yml
    Copy to Clipboard Toggle word wrap

    ++ NOTE: If adding a node after upgrading ceph-ansible from a previous version, be sure to use the corresponding version of the site-docker.yml and site.yml files with respect to ceph-ansible. Updating ceph-ansible replaces site-docker.yml sample and site.yml sample, but not site-docker.yml and site.yml.

Ansible will add the new Monitors to the Ceph storage cluster.

6.1.3. Adding a Monitor with the Command Line Interface
复制链接

This procedure creates a ceph-mon data directory, retrieves the Monitor map and Monitor keyring, and adds a ceph-mon daemon to the cluster. Add more Ceph Monitors by repeating this procedure until you have a sufficient number of ceph-mon daemons to achieve a quorum.

By convention, Ceph Monitors use the host name of the node where the ceph-mon daemon is running, and Red Hat recommends only running one Monitor daemon per node. The Ceph Monitor identifier in the Ceph configuration file can be defined as you see fit. For the purpose of this document, the Monitor identifier, <mon_id>, is the host name of the Monitor node.

  1. Add the Red Hat Ceph Storage 2 Monitor repository.

    1. Red Hat Enterprise Linux
    2. Ubuntu

  2. As root, install the ceph-mon package on the new Ceph Monitor node:

    Red Hat Enterprise Linux

    # yum install ceph-mon
    Copy to Clipboard Toggle word wrap

    Ubuntu

    $ sudo apt-get install ceph-mon
    Copy to Clipboard Toggle word wrap

  3. To ensure the cluster identifies the Monitor on start or restart, add the Monitor IP address to the Ceph configuration file.

    To add the new Monitor in the [mon] or [global] section of the Ceph configuration file on an existing Monitor node in the storage cluster. The mon_host setting, which is a list of DNS-resolvable host names or IP addresses, separated by "," or ";" or " ". Optionally, you can create a specific section in the Ceph configuration file for the new Monitor node:

    Syntax

    [mon]
mon host = <mon_ip:port> <mon_ip:port> ... <new_mon_ip:port>

[mon.<mon_id>]
host = <mon_id>
mon addr = <mon_ip>
    Copy to Clipboard Toggle word wrap

    If you want to make the Monitor part of the initial quorum, you must also add the host name to the mon_initial_members parameter in the [global] section of the Ceph configuration file.

    Important

    If you are adding a Monitor to a cluster that has only one Monitor, you MUST add the next two Monitors to mon_initial_members and mon_host. Production clusters REQUIRE at least three Monitors set in mon_initial_members and mon_host to ensure high availability. If a cluster with only one initial Monitor adds two more Monitors but does not add them to mon_initial_members and mon_host, the failure of the initial Monitor will cause the cluster to lock up. If the Monitor you are adding is replacing a Monitor that is part of mon_initial_members and mon_host, the new Monitor must be added to mon_initial_members and mon_host too.

    Example

    [global]
mon initial members = node1 node2 node3 node4
...
[mon]
mon host = 192.168.0.1:6789 192.168.0.2:6789 192.168.0.3:6789 192.168.0.4:6789
...
[mon.node4]
host = node4
mon addr = 192.168.0.4
    Copy to Clipboard Toggle word wrap

    Finally, send a new copy of the Ceph configuration file to the Ceph nodes and Ceph clients.

    Syntax

    # scp /etc/ceph/<cluster_name>.conf <target_host_name>:/etc/ceph
    Copy to Clipboard Toggle word wrap

    Example

    # scp /etc/ceph/ceph.conf node4:/etc/ceph
    Copy to Clipboard Toggle word wrap

  4. Create the default Monitor directory on the new node:

    Syntax

    # mkdir /var/lib/ceph/mon/<cluster_name>-<mon_id>
    Copy to Clipboard Toggle word wrap

    Example

    # mkdir /var/lib/ceph/mon/ceph-node4
    Copy to Clipboard Toggle word wrap

  5. Create a temporary directory <temp_directory> to keep the files needed during this process. This directory should be different from the Monitor’s default directory created in the previous step, and can be removed after all the steps are executed:

    Syntax

    # mkdir <temp_directory>
    Copy to Clipboard Toggle word wrap

    Exmaple

    # mkdir /tmp/ceph
    Copy to Clipboard Toggle word wrap

  6. Copy the admin key from a running Monitor node to the new Monitor node so that you can run ceph commands:

    Syntax

    # scp /etc/ceph/<cluster_name>.client.admin.keyring <target_host_name>:/etc/ceph
    Copy to Clipboard Toggle word wrap

    Example

    # scp /etc/ceph/ceph.client.admin.keyring node4:/etc/ceph
    Copy to Clipboard Toggle word wrap

  7. Retrieve the Monitor keyring, where <temp_directory> is the path to the retrieved keyring, and <key_file_name> is the name of the file containing the retrieved Monitor keyring:

    Syntax

    # ceph auth get mon. -o /<temp_directory>/<key_file_name>
    Copy to Clipboard Toggle word wrap

    Example

    # ceph auth get mon. -o /tmp/ceph/ceph_keyring.out
    Copy to Clipboard Toggle word wrap

  8. Retrieve the Monitor map, where <tmp> is the path to the retrieved Monitor map, and <mon_map_file_name> is the name of the file containing the retrieved Monitor map:

    Syntax

    # ceph mon getmap -o /<temp_directory>/<mon_map_filename>
    Copy to Clipboard Toggle word wrap

    Example

    # ceph mon getmap -o /tmp/ceph/ceph_mon_map.out
    Copy to Clipboard Toggle word wrap

  9. Prepare the Monitor’s data directory created in the fourth step. You must specify the path to the Monitor map so that you can retrieve the information about a quorum of Monitors and their fsid. You must also specify a path to the Monitor keyring:

    Syntax

    # ceph-mon -i <mon_id> --mkfs --monmap /<temp_directory>/<mon_map_filename> --keyring /<temp_directory>/<key_file_name>
    Copy to Clipboard Toggle word wrap

    Example

    # ceph-mon -i node4 --mkfs --monmap /tmp/ceph/ceph_mon_map.out --keyring /tmp/ceph/ceph_keyring.out
    Copy to Clipboard Toggle word wrap

  10. For storage clusters with custom names, as root, add the the following line to the appropriate file:

    Red Hat Enterprise Linux

    # echo "CLUSTER=<custom_cluster_name>" >> /etc/sysconfig/ceph
    Copy to Clipboard Toggle word wrap

    Ubuntu

    $ sudo echo "CLUSTER=<custom_cluster_name>" >> /etc/default/ceph
    Copy to Clipboard Toggle word wrap

  11. As root, update the owner and group permissions:

    Syntax

    # chown -R <owner>:<group> <path_to_directory>
    Copy to Clipboard Toggle word wrap

    Example

    # chown -R ceph:ceph /var/lib/ceph/mon
# chown -R ceph:ceph /var/log/ceph
# chown -R ceph:ceph /var/run/ceph
# chown -R ceph:ceph /etc/ceph
    Copy to Clipboard Toggle word wrap

  12. As root, enable and start the ceph-mon process on the new Monitor node:

    Syntax

    # systemctl enable ceph-mon.target
# systemctl enable ceph-mon@<monitor_host_name>
# systemctl start ceph-mon@<monitor_host_name>
    Copy to Clipboard Toggle word wrap

    Example

    # systemctl enable ceph-mon.target
# systemctl enable ceph-mon@node4
# systemctl start ceph-mon@node4
    Copy to Clipboard Toggle word wrap

6.2. Removing a Monitor
复制链接

When you remove monitors from a cluster, consider that Ceph monitors use PAXOS to establish consensus about the master cluster map. You must have a sufficient number of monitors to establish a quorum for consensus about the cluster map.

6.2.1. Removing a Monitor with the Command Line Interface
复制链接

This procedure removes a ceph-mon daemon from the storage cluster. If this procedure results in only two monitor daemons, you may add or remove another monitor until you have a number of ceph-mon daemons that can achieve a quorum.

  1. As root, stop the monitor:

    Syntax

    # systemctl stop ceph-mon@<monitor_hostname>
    Copy to Clipboard Toggle word wrap

    Example

    # systemctl stop ceph-mon@node1
    Copy to Clipboard Toggle word wrap

  2. Remove the monitor from the cluster: 

    # ceph mon remove <mon_id>
    Copy to Clipboard Toggle word wrap
  3. On the Ceph Monitor node, remove the Monitor entry from the Ceph configuration file.

  4. Redistribute the Ceph configuration file: 

    # scp /etc/ceph/ceph.conf <user>@<target_hostname>:/etc/ceph/
    Copy to Clipboard Toggle word wrap

  5. Optionally, archive the monitor data: 

    # mv /var/lib/ceph/mon/<cluster>-<daemon_id> /var/lib/ceph/mon/removed-<cluster>-<daemon_id>
    Copy to Clipboard Toggle word wrap

  6. Remove the monitor data, only if the previous archive step is executed: 

    # rm -r /var/lib/ceph/mon/<cluster>-<daemon_id>
    Copy to Clipboard Toggle word wrap

6.2.2. Removing Monitors from an Unhealthy Cluster
复制链接

This procedure removes a ceph-mon daemon from an unhealhty cluster. For example, a cluster that has placement groups persistently not active + clean.

  1. Identify a surviving monitor and log in to that host. : 

    # ceph mon dump
# ssh <monitor_hostname>
    Copy to Clipboard Toggle word wrap

  2. As root, stop the ceph-mon daemon and extract a copy of the monap file. :

    Syntax

    # systemctl stop ceph-mon@<monitor_hostname>
# ceph-mon -i <monitor_id> --extract-monmap <map_path>
    Copy to Clipboard Toggle word wrap

    Example

    # systemctl stop ceph-mon@node1
# ceph-mon -i a --extract-monmap /tmp/monmap
    Copy to Clipboard Toggle word wrap

  3. Remove the non-surviving monitors. For example, if you have three monitors, mon.a, mon.b, and mon.c, where only mon.a will survive, follow the example below:

    Syntax

    # monmaptool <map_path> --rm <monitor_id>
    Copy to Clipboard Toggle word wrap

    Example

    # monmaptool /tmp/monmap --rm b
# monmaptool /tmp/monmap --rm c
    Copy to Clipboard Toggle word wrap

  4. Inject the surviving map with the removed monitors into the surviving monitors. For example, to inject a map into monitor mon.a, follow the example below:

    Syntax

    # ceph-mon -i <monitor_id> --inject-monmap <map_path>
    Copy to Clipboard Toggle word wrap

    Example

    # ceph-mon -i a --inject-monmap /tmp/monmap
    Copy to Clipboard Toggle word wrap

6.3. Adding an OSD
复制链接

When a cluster is up and running, you can add OSDs or remove OSDs from the cluster at runtime.

A Ceph OSD generally consists of one ceph-osd daemon for one storage drive and its associated journal within a host machine. If a host has multiple storage drives, map one ceph-osd daemon for each drive.

Red Hat recommends checking the capacity of a cluster regularly to see if it is reaching the upper end of its storage capacity. As a cluster reaches its near full ratio, add one or more OSDs to expand the cluster’s capacity.

Important

Do not let a cluster reach the full ratio before adding an OSD. OSD failures that occur after the cluster reaches the near full ratio can cause the cluster to exceed the full ratio. Ceph blocks write access to protect the data until you resolve the storage capacity issues. Do not remove OSDs without considering the impact on the full ratio first.

6.3.1. Configuring a Host
复制链接

OSDs and their supporting hardware should be similarly configured as a storage strategy for the pool(s) that will use the OSDs. Ceph prefers uniform hardware across pools for a consistent performance profile. For best performance, consider a CRUSH hierarchy with drives of the same type or size. View the Storage Strategies guide for details.

If you add drives of dissimilar size, adjust their weights. When you add the OSD to the CRUSH map, consider the weight for the new OSD. Hard drive capacity grows approximately 40% per year, so newer OSD hosts might have larger hard drives than older hosts in the cluster, that is, they might have greater weight.

Before doing a new OSD installation, review the requirements listed in the Prerequisites section in the Red Hat Ceph Storage Installation Guide for Red Hat Enterprise Linux or Ubuntu.

6.3.2. Adding an OSD with Ansible
复制链接

Before You Start…​

Using the Same Disk Topology

To add a new Ceph OSD node(s) to an existing storage cluster, under Ansible’s control, open the /etc/ansible/hosts file and add the new OSD node under the [osds] section.

An OSD node(s) should have the same number of OSD data drives and the same drive paths as other OSD nodes in the storage cluster. Ansible will add the same number of OSDs as other OSD nodes using the same paths specified in the devices: section of the /usr/share/ceph-ansible/group_vars/osds file.

  1. As root, add the Ceph OSD host(s) to the /etc/ansible/hosts file, under the [osds] section: 

    [osds]
...
<existing_osd-host-name>
<new_osd-host-name>
    Copy to Clipboard Toggle word wrap

  2. Ensure that Ansible can reach the Ceph hosts: 

    $ ansible all -m ping
    Copy to Clipboard Toggle word wrap

  3. Navigate to the Ansible configuration directory: 

    $ cd /usr/share/ceph-ansible
    Copy to Clipboard Toggle word wrap

  4. Run the Ansible playbook: 

    $ ansible-playbook site.yml
    Copy to Clipboard Toggle word wrap

    If adding an OSD in Ceph deployed in a container: 

    $ ansible-playbook site-docker.yml
    Copy to Clipboard Toggle word wrap

    ++ NOTE: If adding a node after upgrading ceph-ansbile from a previous version, be sure to use the corresponding version of the site-docker.yml and site.yml files with respect to ceph-ansible. Updating ceph-ansible replaces site-docker.yml sample and site.yml sample, but not site-docker.yml and site.yml.

Ansible will add the new OSD node(s) to the Ceph storage cluster.

Note

The new OSD node(s) will have the same configuration as the rest of the OSDs.

Using Different Disk Topologies

If the new Ceph OSD node(s) is(are) using a different disk topology, then there are two approaches for adding the new OSD node(s) to an existing storage cluster.

  • First Approach

    1. As root, add the new Ceph OSD node(s) to the /etc/ansible/hosts file, under the [osds] section. Append the new OSD node name to the end of the OSD list. For example, adding the OSD node new_osdnode10: 

      [osds]
...
existing_osdnode09
new_osdnode10
      Copy to Clipboard Toggle word wrap

    2. As root, create a new file for each new Ceph OSD node added to the storage cluster. Name the new file the same as the new Ceph OSD node name. Create this new file under the /etc/ansible/host_vars/ directory, for example: 

      # touch /etc/ansible/host_vars/new_osdnode10
      Copy to Clipboard Toggle word wrap

      1. As root, edit the new file, and add the devices: and raw_journal_devices: sections to the file. Under each of these sections add a -, space, then the full path to the block device names for this OSD node. For example: 

        devices:
  - /dev/sdc
  - /dev/sdd
  - /dev/sde
  - /dev/sdf

raw_journal_devices:
  - /dev/sda
  - /dev/sda
  - /dev/sdb
  - /dev/sdb
        Copy to Clipboard Toggle word wrap

    3. Ensure that Ansible can reach the Ceph hosts: 

      $ ansible all -m ping
      Copy to Clipboard Toggle word wrap

    4. Navigate to the Ansible configuration directory: 

      $ cd /usr/share/ceph-ansible
      Copy to Clipboard Toggle word wrap

    5. Run the Ansible playbook: 

      $ ansible-playbook site.yml
      Copy to Clipboard Toggle word wrap

Ansible will add the new OSD node(s) to the Ceph storage cluster.

  • Second Approach

    1. As root, modify the existing /etc/ansible/hosts file. Under the [osds] section, add the new OSD node name, and using the devices and raw_journal_devices settings, specify the different disk topology. For example: 

      [osds]
...
osdnode10 devices="['/dev/sdc', '/dev/sdd', '/dev/sde', '/dev/sdf']" raw_journal_devices="['/dev/sda', '/dev/sda', '/dev/sdb', '/dev/sdb']"
      Copy to Clipboard Toggle word wrap

    2. Ensure that Ansible can reach the Ceph hosts: 

      $ ansible all -m ping
      Copy to Clipboard Toggle word wrap

    3. Navigate to the Ansible configuration directory: 

      $ cd /usr/share/ceph-ansible
      Copy to Clipboard Toggle word wrap

    4. Run the Ansible playbook: 

      $ ansible-playbook site.yml
      Copy to Clipboard Toggle word wrap

Ansible will add the new OSD node(s) to the Ceph storage cluster.

See Also..

For more details on the different OSD settings, see the Configuring Ceph OSD Settings table in the Red Hat Ceph Storage 2 Installation Guide.

6.3.3. Adding an OSD with the Command Line Interface
复制链接

To add a new OSD node manually:

  1. Install the ceph-osd package and create a new OSD instance.
  2. Prepare and mount the OSD data and journal drives.
  3. Initialize the OSD data directory and register the OSD authentication key.
  4. Add the new OSD node to the CRUSH map.
  5. Update the owner and group permissions.
  6. Enable and start the ceph-osd daemon.
Note

Enter the commands that start with the number sign (#) as root or with the sudo utility.

In addition, if you use a custom cluster name, use the --cluster <cluster-name> option with the ceph and ceph-osd commands.

Before you Start

  1. Add the Red Hat Ceph Storage 2 OSD repository:

    1. Red Hat Enterprise Linux
    2. Ubuntu

  2. Create the /etc/ceph/ directory: 

    # mkdir /etc/ceph
    Copy to Clipboard Toggle word wrap

  3. On the new OSD node, copy the Ceph administration keyring and configuration file from the initial Monitor node: 

    scp <user_name>@<monitor_host_name>:/etc/ceph/<cluster_name>.client.admin.keyring /etc/ceph
scp <user_name>@<monitor_host_name>:/etc/ceph/<cluster_name>.conf /etc/ceph
    Copy to Clipboard Toggle word wrap

    For example: 

    # scp root@node1:/etc/ceph/ceph.client.admin.keyring /etc/ceph/
# scp root@node1:/etc/ceph/ceph.conf /etc/ceph/
    Copy to Clipboard Toggle word wrap

Installing ceph-osd and Creating a New OSD Instance

  1. Install the ceph-osd package on the new Ceph OSD node:

    1. For Red Hat Enterprise Linux nodes: 

      # yum install ceph-osd
      Copy to Clipboard Toggle word wrap

    2. For Ubuntu nodes: 

      $ sudo apt-get install ceph-osd
      Copy to Clipboard Toggle word wrap

  2. Generate the Universally Unique Identifier (UUID) for the new OSD: 

    $ uuidgen
ebd24968-0f3a-499c-a5b7-929c71ac8c1a
    Copy to Clipboard Toggle word wrap

  3. Create the OSD instance. Note the OSD number identifier because you will need it for subsequent procedures. 

    ceph osd create <uuid> [<osd_id>]
    Copy to Clipboard Toggle word wrap

    For example: 

    # ceph osd create ebd24968-0f3a-499c-a5b7-929c71ac8c1a
4
    Copy to Clipboard Toggle word wrap

Preparing the OSD Data and Journal Drives

  1. Create the default directory for the new OSD: 

    mkdir /var/lib/ceph/osd/<cluster_name>-<osd_id>
    Copy to Clipboard Toggle word wrap

    For example: 

    # mkdir /var/lib/ceph/osd/ceph-4
    Copy to Clipboard Toggle word wrap

  2. Create a new GUID partition table on the drive to be used as an OSD: 

    parted <path_to_disk> mklabel gpt
    Copy to Clipboard Toggle word wrap

    For example: 

    # parted /dev/sdd mklabel gpt
    Copy to Clipboard Toggle word wrap

    If you want to use a separate drive for the journal, create a new GUID partition table on it, too, for example: 

    # parted /dev/sde mklabel gpt
    Copy to Clipboard Toggle word wrap

  3. Create the data and journal partitions: 

    parted <path_to_disk> mkpart primary <start> <end>
    Copy to Clipboard Toggle word wrap

    For example: 

    # parted /dev/sdd mkpart primary 1 10000
# parted /dev/sdd mkpart primary 10001 16000
    Copy to Clipboard Toggle word wrap

    If you use a separate drive for the journal, create the journal partition on it, for example: 

    # parted /dev/sde mkpart primary 1 6000
    Copy to Clipboard Toggle word wrap
    Note

    The minimum size of a journal partition is 5 GB. Journal partitions bigger than 10 GB are not usually needed.

  4. Create a file system on the data partition: 

    mkfs -t <fstype> <path_to_partition>
    Copy to Clipboard Toggle word wrap

    For example: 

    # mkfs -t xfs /dev/sdd1
    Copy to Clipboard Toggle word wrap

  5. Set the type code of the partitions using the proper GUID value: 

    # sgdisk --typecode=<partnum:{GUID}> -- <path to the device>
    Copy to Clipboard Toggle word wrap

    The GUID for Ceph OSD is 4fbd7e29-9d25-41b8-afd0-062c0ceff05d, and the GUID for Ceph Journal is 45B0969E-9B03-4F30-B4C6-B4B80CEFF106. For example: 

    # sgdisk --typecode=1:4fbd7e29-9d25-41b8-afd0-062c0ceff05d -- /dev/sdd
# sgdisk --typecode=2:45b0969e-9b03-4f30-b4c6-b4b80ceff106 -- /dev/sdd
    Copy to Clipboard Toggle word wrap

  6. Mount the data partition to the directory created in the first step: 

    mount -o noatime <path_to_partition> /var/lib/ceph/osd/<cluster_name>-<osd_id>
    Copy to Clipboard Toggle word wrap

    For example: 

    # mount -o noatime /dev/sdd1 /var/lib/ceph/osd/ceph-4
    Copy to Clipboard Toggle word wrap

  7. Mount the data partition permanently by editing the /etc/fstab file: 

    echo "<path_to_partition>  /var/lib/ceph/osd/<cluster_name>-<osd_id>   xfs defaults,noatime 1 2" >> /etc/fstab
    Copy to Clipboard Toggle word wrap

    For example: 

    # echo "/dev/sdd1 /var/lib/ceph/osd/ceph-4  xfs defaults,noatime 1 2" >> /etc/fstab
    Copy to Clipboard Toggle word wrap

Initializing the OSD Data and Journal Directory and Registering the OSD Authentication Key

  1. Initialize the OSD data directory: 

    ceph-osd -i <osd_id> --mkfs --mkkey --osd-uuid <uuid>
    Copy to Clipboard Toggle word wrap

    For example: 

    # ceph-osd -i 4 --mkfs --mkkey --osd-uuid ebd24968-0f3a-499c-a5b7-929c71ac8c1a
... auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory
... created new key in keyring /var/lib/ceph/osd/ceph-4/keyring
    Copy to Clipboard Toggle word wrap

    If you use a separate drive for the journal, use the --osd-journal option to specify the journal drive: 

    ceph-osd -i <osd_id> --mkfs --mkkey --osd-uuid <uuid> --osd-journal <path_to_journal>
    Copy to Clipboard Toggle word wrap

    For example: 

    # ceph-osd -i 4 --mkfs --mkkey --osd-uuid ebd24968-0f3a-499c-a5b7-929c71ac8c1a --osd-journal /dev/sde1
    Copy to Clipboard Toggle word wrap
    Important

    The directory must be empty before you run ceph-osd with the --mkkey option.

  2. If you use a separate drive for the journal, perform the following steps:

    1. Determine the UUID of the journal drive: 

      $ ls -l /dev/disk/by-partuuid/*
      Copy to Clipboard Toggle word wrap

    2. Create a symbolic link between the journal drive and the journal file: 

      ln -s /dev/disk/by-partuuid/<uuid> /var/lib/ceph/osd/<cluster_name>-<osd_id>/journal
      Copy to Clipboard Toggle word wrap

      For example: 

      # ln -s /dev/disk/by-partuuid/62bc7076-0533-4440-9024-ddaeea4f7a7b
/var/lib/ceph/osd/ceph-4/journal
      Copy to Clipboard Toggle word wrap

    3. Create the journal on the journal disk: 

      ceph-osd -i <osd-id> --mkjournal
      Copy to Clipboard Toggle word wrap

      For example: 

      # ceph-osd -i 4 --mkjournal
      Copy to Clipboard Toggle word wrap

  3. Register the OSD authentication key. If your cluster name differs from ceph, insert the custom cluster name instead: 

    ceph auth add osd.<osd_id> osd 'allow *' mon 'allow profile osd' -i /var/lib/ceph/osd/<cluster_name>-<osd_id>/keyring
    Copy to Clipboard Toggle word wrap

    For example: 

    # ceph auth add osd.4 osd 'allow *' mon 'allow profile osd' -i /var/lib/ceph/osd/ceph-4/keyring
added key for osd.4
    Copy to Clipboard Toggle word wrap

Adding the New OSD Node to the CRUSH map

Add the OSD to the CRUSH map so that the OSD can start receiving data: 

ceph osd crush add <osd_id> <weight> [<bucket_type>=<bucket_name> ...]
Copy to Clipboard Toggle word wrap

For example: 

# ceph osd crush add 4 1 host=node4
Copy to Clipboard Toggle word wrap

If you specify more than one bucket, the command places the OSD into the most specific bucket out of those you specified, and it moves the bucket underneath any other buckets you specified.

Important

If you specify only the root bucket, the command attaches the OSD directly to the root, but CRUSH rules expect OSDs to be inside of the host bucket.

Note

You can also edit the CRUSH map manually. See the Editing a CRUSH map section in the Storage Strategies Guide for Red Hat Ceph Storage 2.

Updating the Owner and Group Permissions

Update the owner and group permissions for the newly created directories: 

chown -R <owner>:<group> <path_to_directory>
Copy to Clipboard Toggle word wrap

For example: 

# chown -R ceph:ceph /var/lib/ceph/osd
# chown -R ceph:ceph /var/log/ceph
# chown -R ceph:ceph /var/run/ceph
# chown -R ceph:ceph /etc/ceph
Copy to Clipboard Toggle word wrap

Enabling and Starting the ceph-osd Daemon

  1. If you use clusters with custom names, add the following line to the appropriate file:

    1. For Red Hat Enterprise Linux nodes: 

      # echo "CLUSTER=<custom_cluster_name>" >> /etc/sysconfig/ceph
      Copy to Clipboard Toggle word wrap

    2. For Ubuntu nodes: 

      $ sudo echo "CLUSTER=<custom_cluster_name>" >> /etc/default/ceph
      Copy to Clipboard Toggle word wrap

  2. To ensure that the new OSD is up and ready to receive data, enable and start the OSD process: 

    systemctl enable ceph-osd@<osd_id>
systemctl start ceph-osd@<osd_id>
    Copy to Clipboard Toggle word wrap

    For example: 

    # systemctl enable ceph-osd@4
# systemctl start ceph-osd@4
    Copy to Clipboard Toggle word wrap

6.3.4. Observing Data Migration
复制链接

When you add an OSD to the CRUSH map, Ceph begins rebalacing the server by migrating placement groups to the new OSD. To observe this process, use the ceph command line utility: 

# ceph -w
Copy to Clipboard Toggle word wrap

The placement group states change from active+clean to active, some degraded objects, and finally active+clean when migration completes. To exit the utility, press Ctrl + C.

6.4. Removing an OSD
复制链接

When you want to reduce the size of a cluster or replace hardware, you may remove an OSD at runtime. With Ceph, an OSD is generally one Ceph ceph-osd daemon for one storage drive within a host machine. If the node has multiple storage drives, you may need to remove one ceph-osd daemon for each drive. Generally, it’s a good idea to check the capacity of the storage cluster to see if you are reaching the upper end of its capacity. Ensure that when you remove an OSD that the storage cluster is not at its near full ratio.

Warning

Do not let the storage cluster reach its full ratio when removing an OSD. Removing OSDs could cause the cluster to reach or exceed its full ratio.

6.4.1. Removing an OSD with the Command Line Interface
复制链接

This procedure takes out an OSD, removes an OSD from a cluster map, removes its authentication key, removes the OSD from the OSD map, and removes the OSD from the ceph.conf file. If the node has multiple drives, you may need to remove an OSD for each drive by repeating this procedure.

  1. After you take an OSD out of the cluster, it may still be running. That is, the OSD may be up and out. You must stop the OSD before you remove it from the configuration. As root, disable and stop the OSD process:

    Syntax

    # systemctl disable ceph-osd@<osd_id>
# systemctl stop ceph-osd@<osd_id>
    Copy to Clipboard Toggle word wrap

    Example

    # systemctl disable ceph-osd@4
# systemctl stop ceph-osd@4
    Copy to Clipboard Toggle word wrap

    Once the OSD is stopped, it is down.

  2. Before removing the OSD, the OSD needs to be taken out of the storage cluster:

    Syntax

    # ceph osd out <osd_id>
    Copy to Clipboard Toggle word wrap

    Example

    # ceph osd out 4
    Copy to Clipboard Toggle word wrap

    Important

    Once the OSD is out, Ceph will start rebalancing and copying data to other OSDs in the storage cluster. Red Hat recommends waiting until the storage cluster becomes active+clean before proceeding to the next step. To observe the data migration, run the following command: 

    # ceph -w
    Copy to Clipboard Toggle word wrap

  3. Remove the OSD from the CRUSH map so that it no longer receives data. You may also decompile the CRUSH map, remove the OSD from the device list, remove the device as an item in the host bucket or remove the host bucket, if it’s in the CRUSH map and you intend to remove the host, recompile the map and set it. See the Storage Strategies Guide for details.

    Syntax

    # ceph osd crush remove <osd_name>
    Copy to Clipboard Toggle word wrap

    Example

    # ceph osd crush remove osd.4
    Copy to Clipboard Toggle word wrap

  4. Remove the OSD authentication key:

    Syntax

    # ceph auth del osd.<osd_id>
    Copy to Clipboard Toggle word wrap

    Example

    # ceph auth del osd.4
    Copy to Clipboard Toggle word wrap

  5. Remove the OSD:

    Syntax

    # ceph osd rm <osd_id>
    Copy to Clipboard Toggle word wrap

    Example

    # ceph osd rm 4
    Copy to Clipboard Toggle word wrap

  6. Navigate to the host where you keep the master copy of the storage cluster’s ceph.conf file: 

    # cd /etc/ceph
# vim ceph.conf
    Copy to Clipboard Toggle word wrap

  7. Remove the OSD entry from the ceph.conf file, if it exists: 

    [osd.4]
host = <hostname>
    Copy to Clipboard Toggle word wrap
  8. Remove the reference to the OSD in the /etc/fstab file, if the OSD was added manually.
  9. From the node where you keep the master copy of the storage cluster’s ceph.conf file, copy the updated ceph.conf file to the /etc/ceph/ directory of other nodes in the storage cluster.
返回顶部
Red Hat logoGithubredditYoutubeTwitter

学习

尝试、购买和销售

社区

关于红帽文档

通过我们的产品和服务，以及可以信赖的内容，帮助红帽用户创新并实现他们的目标。 了解我们当前的更新.

让开源更具包容性

红帽致力于替换我们的代码、文档和 Web 属性中存在问题的语言。欲了解更多详情，请参阅红帽博客.

關於紅帽

我们提供强化的解决方案，使企业能够更轻松地跨平台和环境（从核心数据中心到网络边缘）工作。

Theme

© 2025 Red Hat