Installation Guide for Red Hat Enterprise Linux

1. Register the node. When prompted, enter your Red Hat Customer Portal credentials:

   # subscription-manager register

2. Pull the latest subscription data from the CDN:

   # subscription-manager refresh

3. List all available subscriptions for Red Hat Ceph Storage:

   # subscription-manager list --available --all --matches="*Ceph*"

   Identify the appropriate subscription and retrieve its Pool ID.

4. Attach the subscription:

   # subscription-manager attach --pool=$POOL_ID

   Replace

   • $POOL_ID with the Pool ID identified in the previous step.

5. Disable the default software repositories. Then, enable the Red Hat Enterprise Linux 7 Server, Red Hat Enterprise Linux 7 Server Extras, and RHCS repositories:

   # subscription-manager repos --disable=*
   # subscription-manager repos --enable=rhel-7-server-rpms
   # subscription-manager repos --enable=rhel-7-server-extras-rpms
   # subscription-manager repos --enable=rhel-7-server-rhceph-3-mon-els-rpms
   # subscription-manager repos --enable=rhel-7-server-rhceph-3-osd-els-rpms
   # subscription-manager repos --enable=rhel-7-server-rhceph-3-tools-els-rpms

6. Update the system to receive the latest packages:

   # yum update

1. Log in to the Red Hat Customer Portal.
2. Click Downloads to visit the Software & Download center.
3. In the Red Hat Ceph Storage area, click Download Software to download the latest version of the software.

1. Verify the following settings are in the /etc/sysconfig/network-scripts/ifcfg-* file corresponding the public-facing network interface card:

   1. The BOOTPROTO parameter is set to none for static IP addresses.

   2. The ONBOOT parameter must be set to yes.

      If it is set to no, the Ceph storage cluster might fail to peer on reboot.

   3. If you intend to use IPv6 addressing, you must set the IPv6 parameters such as IPV6INIT to yes, except the IPV6_FAILURE_FATAL parameter.

      Also, edit the Ceph configuration file, /etc/ceph/ceph.conf, to instruct Ceph to use IPv6, otherwise, Ceph will use IPv4.

1. On all RHCS nodes, start the firewalld service. Enable it to run on boot, and ensure that it is running:

   # systemctl enable firewalld
   # systemctl start firewalld
   # systemctl status firewalld

2. On all Monitor nodes, open port 6789 on the public network:

   [root@monitor ~]# firewall-cmd --zone=public --add-port=6789/tcp
   [root@monitor ~]# firewall-cmd --zone=public --add-port=6789/tcp --permanent

   To limit access based on the source address:

   firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
   source address="IP_address/netmask_prefix" port protocol="tcp" \
   port="6789" accept"

   firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
   source address="IP_address/netmask_prefix" port protocol="tcp" \
   port="6789" accept" --permanent

   Replace

   • IP_address with the network address of the Monitor node.
   • netmask_prefix with the netmask in CIDR notation.

   Example

   [root@monitor ~]# firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
   source address="192.168.0.11/24" port protocol="tcp" \
   port="6789" accept"

   [root@monitor ~]# firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
   source address="192.168.0.11/24" port protocol="tcp" \
   port="6789" accept" --permanent

3. On all OSD nodes, open ports 6800-7300 on the public network:

   [root@osd ~]# firewall-cmd --zone=public --add-port=6800-7300/tcp
   [root@osd ~]# firewall-cmd --zone=public --add-port=6800-7300/tcp --permanent

   If you have a separate cluster network, repeat the commands with the appropriate zone.

4. On all Ceph Manager (ceph-mgr) nodes (usually the same nodes as Monitor ones), open ports 6800-7300 on the public network:

   [root@monitor ~]# firewall-cmd --zone=public --add-port=6800-7300/tcp
   [root@monitor ~]# firewall-cmd --zone=public --add-port=6800-7300/tcp --permanent

   If you have a separate cluster network, repeat the commands with the appropriate zone.

5. On all Ceph Metadata Server (ceph-mds) nodes, open port 6800 on the public network:

   [root@monitor ~]# firewall-cmd --zone=public --add-port=6800/tcp
   [root@monitor ~]# firewall-cmd --zone=public --add-port=6800/tcp --permanent

   If you have a separate cluster network, repeat the commands with the appropriate zone.

6. On all Ceph Object Gateway nodes, open the relevant port or ports on the public network.

   1. To open the default Ansible configured port of 8080:

      [root@gateway ~]# firewall-cmd --zone=public --add-port=8080/tcp
      [root@gateway ~]# firewall-cmd --zone=public --add-port=8080/tcp --permanent

      To limit access based on the source address:

      firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
      source address="IP_address/netmask_prefix" port protocol="tcp" \
      port="8080" accept"

      firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
      source address="IP_address/netmask_prefix" port protocol="tcp" \
      port="8080" accept" --permanent

      Replace

      • IP_address with the network address of the object gateway node.
      • netmask_prefix with the netmask in CIDR notation.

      Example

      [root@gateway ~]# firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
      source address="192.168.0.31/24" port protocol="tcp" \
      port="8080" accept"

      [root@gateway ~]# firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
      source address="192.168.0.31/24" port protocol="tcp" \
      port="8080" accept" --permanent

   2. Optional. If you installed Ceph Object Gateway using Ansible and changed the default port that Ansible configures Ceph Object Gateway to use from 8080, for example, to port 80, open this port:

      [root@gateway ~]# firewall-cmd --zone=public --add-port=80/tcp
      [root@gateway ~]# firewall-cmd --zone=public --add-port=80/tcp --permanent

      To limit access based on the source address, run the following commands:

      firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
      source address="IP_address/netmask_prefix" port protocol="tcp" \
      port="80" accept"

      firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
      source address="IP_address/netmask_prefix" port protocol="tcp" \
      port="80" accept" --permanent

      Replace

      • IP_address with the network address of the object gateway node.
      • netmask_prefix with the netmask in CIDR notation.

      Example

      [root@gateway ~]# firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
      source address="192.168.0.31/24" port protocol="tcp" \
      port="80" accept"

      [root@gateway ~]# firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
      source address="192.168.0.31/24" port protocol="tcp" \
      port="80" accept" --permanent

   3. Optional. To use SSL/TLS, open port 443:

      [root@gateway ~]# firewall-cmd --zone=public --add-port=443/tcp
      [root@gateway ~]# firewall-cmd --zone=public --add-port=443/tcp --permanent

      To limit access based on the source address, run the following commands:

      firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
      source address="IP_address/netmask_prefix" port protocol="tcp" \
      port="443" accept"

      firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
      source address="IP_address/netmask_prefix" port protocol="tcp" \
      port="443" accept" --permanent

      Replace

      • IP_address with the network address of the object gateway node.
      • netmask_prefix with the netmask in CIDR notation.

      Example

      [root@gateway ~]# firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
      source address="192.168.0.31/24" port protocol="tcp" \
      port="443" accept"
      [root@gateway ~]# firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
      source address="192.168.0.31/24" port protocol="tcp" \
      port="443" accept" --permanent

Procedure

1. Log in to a Ceph node as the root user:

   ssh root@$HOST_NAME

   Replace

   • $HOST_NAME with the host name of the Ceph node.

   Example

   # ssh root@mon01

   Enter the root password when prompted.

2. Create a new Ansible user:

   adduser $USER_NAME

   Replace

   • $USER_NAME with the new user name for the Ansible user.

   Example

   # adduser admin

   Important

   Do not use ceph as the user name. The ceph user name is reserved for the Ceph daemons. A uniform user name across the cluster can improve ease of use, but avoid using obvious user names, because intruders typically use them for brute-force attacks.

3. Set a new password for this user:

   # passwd $USER_NAME

   Replace

   • $USER_NAME with the new user name for the Ansible user.

   Example

   # passwd admin

   Enter the new password twice when prompted.

4. Configure sudo access for the newly created user:

   cat << EOF >/etc/sudoers.d/$USER_NAME
   $USER_NAME ALL = (root) NOPASSWD:ALL
   EOF

   Replace

   • $USER_NAME with the new user name for the Ansible user.

   Example

   # cat << EOF >/etc/sudoers.d/admin
   admin ALL = (root) NOPASSWD:ALL
   EOF

5. Assign the correct file permissions to the new file:

   chmod 0440 /etc/sudoers.d/$USER_NAME

   Replace

   • $USER_NAME with the new user name for the Ansible user.

   Example

   # chmod 0440 /etc/sudoers.d/admin

1. Generate the SSH key pair, accept the default file name and leave the passphrase empty:

   [user@admin ~]$ ssh-keygen

2. Copy the public key to all nodes in the storage cluster:

   ssh-copy-id $USER_NAME@$HOST_NAME

   Replace

   • $USER_NAME with the new user name for the Ansible user.
   • $HOST_NAME with the host name of the Ceph node.

   Example

   [user@admin ~]$ ssh-copy-id admin@ceph-mon01

3. Create and edit the ~/.ssh/config file.

   Important

   By creating and editing the ~/.ssh/config file you do not have to specify the -u $USER_NAME option each time you execute the ansible-playbook command.

   1. Create the SSH config file:

      [user@admin ~]$ touch ~/.ssh/config

   2. Open the config file for editing. Set the Hostname and User options for each node in the storage cluster:

      Host node1
         Hostname $HOST_NAME
         User $USER_NAME
      Host node2
         Hostname $HOST_NAME
         User $USER_NAME
      ...

      Replace

      • $HOST_NAME with the host name of the Ceph node.
      • $USER_NAME with the new user name for the Ansible user.

      Example

      Host node1
         Hostname monitor
         User admin
      Host node2
         Hostname osd
         User admin
      Host node3
         Hostname gateway
         User admin

4. Set the correct file permissions for the ~/.ssh/config file:

   [admin@admin ~]$ chmod 600 ~/.ssh/config

1. As the Ansible user, create the ceph-ansible-keys directory where Ansible stores temporary values generated by the ceph-ansible playbook.

   [user@admin ~]$ mkdir ~/ceph-ansible-keys

2. As root, create a symbolic link to the /usr/share/ceph-ansible/group_vars directory in the /etc/ansible/ directory:

   [root@admin ~]# ln -s /usr/share/ceph-ansible/group_vars /etc/ansible/group_vars

3. Navigate to the /usr/share/ceph-ansible/ directory:

   [root@admin ~]$ cd /usr/share/ceph-ansible

4. Create new copies of the yml.sample files:

   [root@admin ceph-ansible]# cp group_vars/all.yml.sample group_vars/all.yml
   [root@admin ceph-ansible]# cp group_vars/osds.yml.sample group_vars/osds.yml
   [root@admin ceph-ansible]# cp site.yml.sample site.yml

5. Edit the copied files.

   1. Edit the group_vars/all.yml file. See the table below for the most common required and optional parameters to uncomment. Note that the table does not include all parameters.

      Important

      Do not set the cluster: ceph parameter to any value other than ceph because using custom cluster names is not supported.

      Table 3.1. General Ansible Settings

      Option	Value	Required	Notes
      ceph_origin	repository or distro or local	Yes	The repository value means Ceph will be installed through a new repository. The distro value means that no separate repository file will be added, and you will get whatever version of Ceph that is included with the Linux distribution. The local value means the Ceph binaries will be copied from the local machine.
      ceph_repository_type	cdn or iso	Yes
      ceph_rhcs_version	3	Yes
      ceph_rhcs_iso_path	The path to the ISO image	Yes if using an ISO image
      monitor_interface	The interface that the Monitor nodes listen to	monitor_interface, monitor_address, or monitor_address_block is required
      monitor_address	The address that the Monitor nodes listen to
      monitor_address_block	The subnet of the Ceph public network		Use when the IP addresses of the nodes are unknown, but the subnet is known
      ip_version	ipv6	Yes if using IPv6 addressing
      public_network	The IP address and netmask of the Ceph public network, or the corresponding IPv6 address if using IPv6	Yes	Section 2.8, “Verifying the Network Configuration for Red Hat Ceph Storage”
      cluster_network	The IP address and netmask of the Ceph cluster network	No, defaults to public_network
      configure_firewall	Ansible will try to configure the appropriate firewall rules	No. Either set the value to true or false.

      An example of the all.yml file can look like:

      ceph_origin: distro
      ceph_repository: rhcs
      ceph_repository_type: cdn
      ceph_rhcs_version: 3
      monitor_interface: eth0
      public_network: 192.168.0.0/24

      Note

      Be sure to set ceph_origin to distro in the all.yml file. This ensures that the installation process uses the correct download repository.

      Note

      Having the ceph_rhcs_version option set to 3 will pull in the latest version of Red Hat Ceph Storage 3.

      Warning

      By default, Ansible attempts to restart an installed, but masked firewalld service, which can cause the Red Hat Ceph Storage deployment to fail. To work around this issue, set the configure_firewall option to false in the all.yml file. If you are running the firewalld service, then there is no requirement to use the configure_firewall option in the all.yml file.

      For additional details, see the all.yml file.

   2. Edit the group_vars/osds.yml file. See the table below for the most common required and optional parameters to uncomment. Note that the table does not include all parameters.

      Important

      Use a different physical device to install an OSD than the device where the operating system is installed. Sharing the same device between the operating system and OSDs causes performance issues.

      Table 3.2. OSD Ansible Settings

      Option	Value	Required	Notes
      osd_scenario	collocated to use the same device for write-ahead logging and key/value data (BlueStore) or journal (FileStore) and OSD data non-collocated to use a dedicated device, such as SSD or NVMe media to store write-ahead log and key/value data (BlueStore) or journal data (FileStore) lvm to use the Logical Volume Manager to store OSD data	Yes	When using osd_scenario: non-collocated, ceph-ansible expects the numbers of variables in devices and dedicated_devices to match. For example, if you specify 10 disks in devices, you must specify 10 entries in dedicated_devices.
      osd_auto_discovery	true to automatically discover OSDs	Yes if using osd_scenario: collocated	Cannot be used when devices setting is used
      devices	List of devices where ceph data is stored	Yes to specify the list of devices	Cannot be used when osd_auto_discovery setting is used. When using lvm as the osd_scenario and setting the devices option, ceph-volume lvm batch mode creates the optimized OSD configuration.
      dedicated_devices	List of dedicated devices for non-collocated OSDs where ceph journal is stored	Yes if osd_scenario: non-collocated	Should be nonpartitioned devices
      dmcrypt	true to encrypt OSDs	No	Defaults to false
      lvm_volumes	A list of FileStore or BlueStore dictionaries	Yes if using osd_scenario: lvm and storage devices are not defined using devices	Each dictionary must contain a data, journal and data_vg keys. Any logical volume or volume group must be the name and not the full path. The data, and journal keys can be a logical volume (LV) or partition, but do not use one journal for multiple data LVs. The data_vg key must be the volume group containing the data LV. Optionally, the journal_vg key can be used to specify the volume group containing the journal LV, if applicable. See the examples below for various supported configurations.
      osds_per_device	The number of OSDs to create per device.	No	Defaults to 1
      osd_objectstore	The Ceph object store type for the OSDs.	No	Defaults to bluestore. The other option is filestore. Required for upgrades.

      The following are examples of the osds.yml file when using the three OSD scenarios: collocated, non-collocated, and lvm. The default OSD object store format is BlueStore, if not specified.

      Collocated

      osd_objectstore: filestore
      osd_scenario: collocated
      devices:
        - /dev/sda
        - /dev/sdb

      Non-collocated - BlueStore

      osd_objectstore: bluestore
      osd_scenario: non-collocated
      devices:
       - /dev/sda
       - /dev/sdb
       - /dev/sdc
       - /dev/sdd
      dedicated_devices:
       - /dev/nvme0n1
       - /dev/nvme0n1
       - /dev/nvme1n1
       - /dev/nvme1n1

      This non-collocated example will create four BlueStore OSDs, one per device. In this example, the traditional hard drives (sda, sdb, sdc, sdd) are used for object data, and the solid state drives (SSDs) (/dev/nvme0n1, /dev/nvme1n1) are used for the BlueStore databases and write-ahead logs. This configuration pairs the /dev/sda and /dev/sdb devices with the /dev/nvme0n1 device, and pairs the /dev/sdc and /dev/sdd devices with the /dev/nvme1n1 device.

      Non-collocated - FileStore

      osd_objectstore: filestore
      osd_scenario: non-collocated
      devices:
        - /dev/sda
        - /dev/sdb
        - /dev/sdc
        - /dev/sdd
      dedicated_devices:
         - /dev/nvme0n1
         - /dev/nvme0n1
         - /dev/nvme1n1
         - /dev/nvme1n1

      LVM simple

      osd_objectstore: bluestore
      osd_scenario: lvm
      devices:
        - /dev/sda
        - /dev/sdb

      or

      osd_objectstore: bluestore
      osd_scenario: lvm
      devices:
        - /dev/sda
        - /dev/sdb
        - /dev/nvme0n1

      With these simple configurations ceph-ansible uses batch mode (ceph-volume lvm batch) to create the OSDs.

      In the first scenario, if the devices are traditional hard drives or SSDs, then one OSD per device is created.

      In the second scenario, when there is a mix of traditional hard drives and SSDs, the data is placed on the traditional hard drives (sda, sdb) and the BlueStore database (block.db) is created as large as possible on the SSD (nvme0n1).

      LVM advance

      osd_objectstore: filestore
      osd_scenario: lvm
      lvm_volumes:
         - data: data-lv1
           data_vg: vg1
           journal: journal-lv1
           journal_vg: vg2
         - data: data-lv2
           journal: /dev/sda
           data_vg: vg1

      or

      osd_objectstore: bluestore
      osd_scenario: lvm
      lvm_volumes:
        - data: data-lv1
          data_vg: data-vg1
          db: db-lv1
          db_vg: db-vg1
          wal: wal-lv1
          wal_vg: wal-vg1
        - data: data-lv2
          data_vg: data-vg2
          db: db-lv2
          db_vg: db-vg2
          wal: wal-lv2
          wal_vg: wal-vg2

      With these advance scenario examples, the volume groups and logical volumes must be created beforehand. They will not be created by ceph-ansible.

      Note

      If using all NVMe SSDs set the osd_scenario: lvm and osds_per_device: 4 options. For more information, see Configuring OSD Ansible settings for all NVMe Storage for Red Hat Enterprise Linux or Configuring OSD Ansible settings for all NVMe Storage for Ubuntu in the Red Hat Ceph Storage Installation Guides.

      For additional details, see the comments in the osds.yml file.

6. Edit the Ansible inventory file located by default at /etc/ansible/hosts. Remember to comment out example hosts.

   1. Add the Monitor nodes under the [mons] section:

      [mons]
      MONITOR_NODE_NAME1
      MONITOR_NODE_NAME2
      MONITOR_NODE_NAME3

   2. Add OSD nodes under the [osds] section. If the nodes have sequential naming, consider using a range:

      [osds]
      OSD_NODE_NAME1[1:10]

      Note

      For OSDs in a new installation, the default object store format is BlueStore.

      1. Optionally, use the devices and dedicated_devices options to specify devices that the OSD nodes will use. Use a comma-separated list to list multiple devices.

         Syntax

         [osds]
         CEPH_NODE_NAME devices="['DEVICE_1', 'DEVICE_2']" dedicated_devices="['DEVICE_3', 'DEVICE_4']"

         Example

         [osds]
         ceph-osd-01 devices="['/dev/sdc', '/dev/sdd']" dedicated_devices="['/dev/sda', '/dev/sdb']"
         ceph-osd-02 devices="['/dev/sdc', '/dev/sdd', '/dev/sde']" dedicated_devices="['/dev/sdf', '/dev/sdg']"

         When specifying no devices, set the osd_auto_discovery option to true in the osds.yml file.

         Note

         Using the devices and dedicated_devices parameters is useful when OSDs use devices with different names or when one of the devices failed on one of the OSDs.

7. Optionally, if you want to use host specific parameters, for all deployments, bare-metal or in containers, create host files in the host_vars directory to include any parameters specific to hosts.

   1. Create a new file for each new Ceph OSD node added to the storage cluster, under the /etc/ansible/host_vars/ directory:

      Syntax

      touch /etc/ansible/host_vars/OSD_NODE_NAME

      Example

      [root@admin ~]# touch /etc/ansible/host_vars/osd07

   2. Update the file with any host specific parameters. In bare-metal deployments, you can add the devices: and dedicated_devices: sections to the file.

      Example

      devices:
        - /dev/sdc
        - /dev/sdd
        - /dev/sde
        - /dev/sdf
      
      dedicated_devices:
        - /dev/sda
        - /dev/sdb

8. Optionally, for all deployments, bare-metal or in containers, you can create a custom CRUSH hierarchy using ansible-playbook:

   1. Setup your Ansible inventory file. Specify where you want the OSD hosts to be in the CRUSH map’s hierarchy by using the osd_crush_location parameter. You must specify at least two CRUSH bucket types to specify the location of the OSD, and one bucket type must be host. By default, these include root, datacenter, room, row, pod, pdu, rack, chassis and host.

      Syntax

      [osds]
      CEPH_OSD_NAME osd_crush_location="{ 'root': ROOT_BUCKET', 'rack': 'RACK_BUCKET', 'pod': 'POD_BUCKET', 'host': 'CEPH_HOST_NAME' }"

      Example

      [osds]
      ceph-osd-01 osd_crush_location="{ 'root': 'default', 'rack': 'rack1', 'pod': 'monpod', 'host': 'ceph-osd-01' }"

   2. Set the crush_rule_config and create_crush_tree parameters to True, and create at least one CRUSH rule if you do not want to use the default CRUSH rules. For example, if you are using HDD devices, edit the paramters as follows:

      crush_rule_config: True
      crush_rule_hdd:
          name: replicated_hdd_rule
          root: root-hdd
          type: host
          class: hdd
          default: True
      crush_rules:
        - "{{ crush_rule_hdd }}"
      create_crush_tree: True

      If you are using SSD devices, then edit the parameters as follows:

      crush_rule_config: True
      crush_rule_ssd:
          name: replicated_ssd_rule
          root: root-ssd
          type: host
          class: ssd
          default: True
      crush_rules:
        - "{{ crush_rule_ssd }}"
      create_crush_tree: True

      Note

      The default CRUSH rules fail if both ssd and hdd OSDs are not deployed because the default rules now include the class parameter, which must be defined.

      Note

      Additionally, add the custom CRUSH hierarchy to the OSD files in the host_vars directory as described in a step above to make this configuration work.

   3. Create pools, with created crush_rules in group_vars/clients.yml file.

      Example

      copy_admin_key: True
      user_config: True
      pool1:
        name: "pool1"
        pg_num: 128
        pgp_num: 128
        rule_name: "HDD"
        type: "replicated"
        device_class: "hdd"
      pools:
        - "{{ pool1 }}"

   4. View the tree.

      [root@mon ~]# ceph osd tree

   5. Validate the pools.

      # for i in $(rados lspools);do echo "pool: $i"; ceph osd pool get $i crush_rule;done
      
      pool: pool1
      crush_rule: HDD

9. For all deployments, bare-metal or in containers, open for editing the Ansible inventory file, by default the /etc/ansible/hosts file. Comment out the example hosts.

   1. Add the Ceph Manager (ceph-mgr) nodes under the [mgrs] section. Colocate the Ceph Manager daemon with Monitor nodes.

      [mgrs]
      <monitor-host-name>
      <monitor-host-name>
      <monitor-host-name>

10. As the Ansible user, ensure that Ansible can reach the Ceph hosts:

    [user@admin ~]$ ansible all -m ping

11. Add the following line to the /etc/ansible/ansible.cfg file:

    retry_files_save_path = ~/

12. As root, create the /var/log/ansible/ directory and assign the appropriate permissions for the ansible user:

    [root@admin ~]# mkdir /var/log/ansible
    [root@admin ~]# chown ansible:ansible  /var/log/ansible
    [root@admin ~]# chmod 755 /var/log/ansible

    1. Edit the /usr/share/ceph-ansible/ansible.cfg file, updating the log_path value as follows:

       log_path = /var/log/ansible/ansible.log

13. As the Ansible user, change to the /usr/share/ceph-ansible/ directory:

    [user@admin ~]$ cd /usr/share/ceph-ansible/

14. Run the ceph-ansible playbook:

    [user@admin ceph-ansible]$ ansible-playbook site.yml

    Note

    To increase the deployment speed, use the --forks option to ansible-playbook. By default, ceph-ansible sets forks to 20. With this setting, up to twenty nodes will be installed at the same time. To install up to thirty nodes at a time, run ansible-playbook --forks 30 PLAYBOOK FILE. The resources on the admin node must be monitored to ensure they are not overused. If they are, lower the number passed to --forks.

15. Using the root account on a Monitor node, verify the status of the Ceph cluster:

    [root@monitor ~]# ceph health
    HEALTH_OK

16. Verify the cluster is functioning using rados.

    1. From a monitor node, create a test pool with eight placement groups:

       Syntax

       [root@monitor ~]# ceph osd pool create <pool-name> <pg-number>

       Example

       [root@monitor ~]# ceph osd pool create test 8

    2. Create a file called hello-world.txt:

       Syntax

       [root@monitor ~]# vim <file-name>

       Example

       [root@monitor ~]# vim hello-world.txt

    3. Upload hello-world.txt to the test pool using the object name hello-world:

       Syntax

       [root@monitor ~]# rados --pool <pool-name> put <object-name> <object-file>

       Example

       [root@monitor ~]# rados --pool test put hello-world hello-world.txt

    4. Download hello-world from the test pool as file name fetch.txt:

       Syntax

       [root@monitor ~]# rados --pool <pool-name> get <object-name> <object-file>

       Example

       [root@monitor ~]# rados --pool test get hello-world fetch.txt

    5. Check the contents of fetch.txt:

       [root@monitor ~]# cat fetch.txt

       The output should be:

       "Hello World!"

       Note

       In addition to verifying the cluster status, you can use the ceph-medic utility to overall diagnose the Ceph Storage Cluster. See the Using ceph-medic to diagnose a Ceph Storage Cluster chapter in the Red Hat Ceph Storage 3 Administration Guide.

Procedure

1. Set osd_scenario: lvm and osds_per_device: 4 in group_vars/osds.yml:

   osd_scenario: lvm
   osds_per_device: 4

2. List the NVMe devices under devices:

   devices:
     - /dev/nvme0n1
     - /dev/nvme1n1
     - /dev/nvme2n1
     - /dev/nvme3n1

3. The settings in group_vars/osds.yml will look similar to this example:

   osd_scenario: lvm
   osds_per_device: 4
   devices:
     - /dev/nvme0n1
     - /dev/nvme1n1
     - /dev/nvme2n1
     - /dev/nvme3n1

1. Add a new section [mdss] to the /etc/ansible/hosts file:

   [mdss]
   hostname
   hostname
   hostname

   Replace hostname with the host names of the nodes where you want to install the Ceph Metadata Servers.

2. Navigate to the /usr/share/ceph-ansible directory:

   [root@admin ~]# cd /usr/share/ceph-ansible

3. Optional. Change the default variables.

   1. Create a copy of the group_vars/mdss.yml.sample file named mdss.yml:

      [root@admin ceph-ansible]# cp group_vars/mdss.yml.sample group_vars/mdss.yml

   2. Optionally, edit parameters in mdss.yml. See mdss.yml for details.

4. As the Ansible user, run the Ansible playbook:

   [user@admin ceph-ansible]$ ansible-playbook site.yml --limit mdss

5. After installing Metadata Servers, configure them. For details, see the Configuring Metadata Server Daemons chapter in the Ceph File System Guide for Red Hat Ceph Storage 3.

1. Add a new section [clients] to the /etc/ansible/hosts file:

   [clients]
   <client-hostname>

   Replace <client-hostname> with the host name of the node where you want to install the ceph-client role.

2. Navigate to the /usr/share/ceph-ansible directory:

   [root@admin ~]# cd /usr/share/ceph-ansible

3. Create a new copy of the clients.yml.sample file named clients.yml:

   [root@admin ceph-ansible ~]# cp group_vars/clients.yml.sample group_vars/clients.yml

4. Open the group_vars/clients.yml file, and uncomment the following lines:

   keys:
     - { name: client.test, caps: { mon: "allow r", osd: "allow class-read object_prefix rbd_children, allow rwx pool=test" },  mode: "{{ ceph_keyring_permissions }}" }

   1. Replace client.test with the real client name, and add the client key to the client definition line, for example:

      key: "ADD-KEYRING-HERE=="

      Now the whole line example would look similar to this:

      - { name: client.test, key: "AQAin8tUMICVFBAALRHNrV0Z4MXupRw4v9JQ6Q==", caps: { mon: "allow r", osd: "allow class-read object_prefix rbd_children, allow rwx pool=test" },  mode: "{{ ceph_keyring_permissions }}" }

      Note

      The ceph-authtool --gen-print-key command can generate a new client key.

5. Optionally, instruct ceph-client to create pools and clients.

   1. Update clients.yml.

      • Uncomment the user_config setting and set it to true.
      • Uncomment the pools and keys sections and update them as required. You can define custom pools and client names altogether with the cephx capabilities.

   2. Add the osd_pool_default_pg_num setting to the ceph_conf_overrides section in the all.yml file:

      ceph_conf_overrides:
         global:
            osd_pool_default_pg_num: <number>

      Replace <number> with the default number of placement groups.

6. Run the Ansible playbook:

   [user@admin ceph-ansible]$ ansible-playbook site.yml --limit clients

1. Add gateway hosts to the /etc/ansible/hosts file under the [rgws] section to identify their roles to Ansible. If the hosts have sequential naming, use a range, for example:

   [rgws]
   <rgw_host_name_1>
   <rgw_host_name_2>
   <rgw_host_name[3..10]>

2. Navigate to the Ansible configuration directory:

   [root@ansible ~]# cd /usr/share/ceph-ansible

3. Create the rgws.yml file from the sample file:

   [root@ansible ~]# cp group_vars/rgws.yml.sample group_vars/rgws.yml

4. Open and edit the group_vars/rgws.yml file. To copy the administrator key to the Ceph Object Gateway node, uncomment the copy_admin_key option:

   copy_admin_key: true

5. The rgws.yml file may specify a different default port than the default port 7480. For example:

   ceph_rgw_civetweb_port: 80

6. The all.yml file MUST specify a radosgw_interface. For example:

   radosgw_interface: eth0

   Specifying the interface prevents Civetweb from binding to the same IP address as another Civetweb instance when running multiple instances on the same host.

7. Generally, to change default settings, uncomment the settings in the rgw.yml file, and make changes accordingly. To make additional changes to settings that are not in the rgw.yml file, use ceph_conf_overrides: in the all.yml file. For example, set the rgw_dns_name: with the host of the DNS server and ensure the cluster’s DNS server to configure it for wild cards to enable S3 subdomains.

   ceph_conf_overrides:
      client.rgw.rgw1:
         rgw_dns_name: <host_name>
         rgw_override_bucket_index_max_shards: 16
         rgw_bucket_default_quota_max_objects: 1638400

   For advanced configuration details, see the Red Hat Ceph Storage 3 Ceph Object Gateway for Production guide. Advanced topics include:

   • Configuring Ansible Groups

   • Developing Storage Strategies. See the Creating the Root Pool, Creating System Pools, and Creating Data Placement Strategies sections for additional details on how create and configure the pools.

     See Bucket Sharding for configuration details on bucket sharding.

8. Uncomment the radosgw_interface parameter in the group_vars/all.yml file.

   radosgw_interface: <interface>

   Replace:

   • <interface> with the interface that the Ceph Object Gateway nodes listen to

   For additional details, see the all.yml file.

9. Run the Ansible playbook:

   [user@admin ceph-ansible]$ ansible-playbook site.yml --limit rgws

1. Create the nfss file from the sample file:

   [root@ansible ~]# cd /usr/share/ceph-ansible/group_vars
   [root@ansible ~]# cp nfss.yml.sample nfss.yml

2. Add gateway hosts to the /etc/ansible/hosts file under an [nfss] group to identify their group membership to Ansible. If the hosts have sequential naming, use a range. For example:

   [nfss]
   <nfs_host_name_1>
   <nfs_host_name_2>
   <nfs_host_name[3..10]>

3. Navigate to the Ansible configuration directory, /etc/ansible/:

   [root@ansible ~]# cd /usr/share/ceph-ansible

4. To copy the administrator key to the Ceph Object Gateway node, uncomment the copy_admin_key setting in the /usr/share/ceph-ansible/group_vars/nfss.yml file:

   copy_admin_key: true

5. Configure the FSAL (File System Abstraction Layer) sections of the /usr/share/ceph-ansible/group_vars/nfss.yml file. Provide an ID, S3 user ID, S3 access key and secret. For NFSv4, it should look something like this:

   ###################
   # FSAL RGW Config #
   ###################
   #ceph_nfs_rgw_export_id: <replace-w-numeric-export-id>
   #ceph_nfs_rgw_pseudo_path: "/"
   #ceph_nfs_rgw_protocols: "3,4"
   #ceph_nfs_rgw_access_type: "RW"
   #ceph_nfs_rgw_user: "cephnfs"
   # Note: keys are optional and can be generated, but not on containerized, where
   # they must be configered.
   #ceph_nfs_rgw_access_key: "<replace-w-access-key>"
   #ceph_nfs_rgw_secret_key: "<replace-w-secret-key>"

   Warning

   Access and secret keys are optional, and can be generated.

6. Run the Ansible playbook:

   [user@admin ceph-ansible]$ ansible-playbook site-docker.yml --limit nfss

1. As the root user, navigate to the /usr/share/ceph-ansible/ directory:

   [root@admin ~]# cd /usr/share/ceph-ansible/

2. Skip this step when upgrading from Red Hat Ceph Storage version 3.x to the latest version. Back up the group_vars/all.yml and group_vars/osds.yml files.

   [root@admin ceph-ansible]# cp group_vars/all.yml group_vars/all_old.yml
   [root@admin ceph-ansible]# cp group_vars/osds.yml group_vars/osds_old.yml
   [root@admin ceph-ansible]# cp group_vars/clients.yml group_vars/clients_old.yml

3. Skip this step when upgrading from Red Hat Ceph Storage version 3.x to the latest version. When upgrading from Red Hat Ceph Storage 2.x to 3.x, create new copies of the group_vars/all.yml.sample, group_vars/osds.yml.sample and group_vars/clients.yml.sample files, and rename them to group_vars/all.yml, group_vars/osds.yml, and group_vars/clients.yml respectively. Open and edit them accordingly. For details, see Appendix H, Changes in Ansible Variables Between Version 2 and 3 and Section 3.2, “Installing a Red Hat Ceph Storage Cluster” .

   [root@admin ceph-ansible]# cp group_vars/all.yml.sample group_vars/all.yml
   [root@admin ceph-ansible]# cp group_vars/osds.yml.sample group_vars/osds.yml
   [root@admin ceph-ansible]# cp group_vars/clients.yml.sample group_vars/clients.yml

4. Skip this step when upgrading from Red Hat Ceph Storage version 3.x to the latest version. When upgrading from Red Hat Ceph Storage 2.x to 3.x, open the group_vars/clients.yml file, and uncomment the following lines:

   keys:
     - { name: client.test, caps: { mon: "allow r", osd: "allow class-read object_prefix rbd_children, allow rwx pool=test" },  mode: "{{ ceph_keyring_permissions }}" }

   1. Replace client.test with the real client name, and add the client key to the client definition line, for example:

      key: "ADD-KEYRING-HERE=="

      Now the whole line example would look similar to this:

      - { name: client.test, key: "AQAin8tUMICVFBAALRHNrV0Z4MXupRw4v9JQ6Q==", caps: { mon: "allow r", osd: "allow class-read object_prefix rbd_children, allow rwx pool=test" },  mode: "{{ ceph_keyring_permissions }}" }

      Note

      To get the client key, run the ceph auth get-or-create command to view the key for the named client.

5. In the group_vars/all.yml file, uncomment the upgrade_ceph_packages option and set it to True.

   upgrade_ceph_packages: True

6. In the group_vars/all.yml file, set ceph_rhcs_version to 3.

   ceph_rhcs_version: 3

   Note

   Having the ceph_rhcs_version option set to 3 will pull in the latest version of Red Hat Ceph Storage 3.

7. Set the ceph_origin parameter to distro in the group_vars/all.yml file:

   ceph_origin: distro

8. Add the fetch_directory parameter to the group_vars/all.yml file.

   fetch_directory: <full_directory_path>

   Replace:

   • <full_directory_path> with a writable location, such as the Ansible user’s home directory. Provide the existing path that was used for the initial storage cluster installation.

   If the existing path is lost or missing, then do the following first:

   1. Add the following options to the existing group_vars/all.yml file:

      fsid: <add_the_fsid>
      generate_fsid: false

   2. Run the take-over-existing-cluster.yml Ansible playbook:

      [user@admin ceph-ansible]$ cp infrastructure-playbooks/take-over-existing-cluster.yml .
      [user@admin ceph-ansible]$ ansible-playbook take-over-existing-cluster.yml

9. If the cluster you want to upgrade contains any Ceph Object Gateway nodes, add the radosgw_interface parameter to the group_vars/all.yml file.

   radosgw_interface: <interface>

   Replace:

   • <interface> with the interface that the Ceph Object Gateway nodes listen to.

10. Starting with Red Hat Ceph Storage 3.2, the default OSD object store is BlueStore. To keep the traditional OSD object store, you must explicitly set the osd_objectstore option to filestore in the group_vars/all.yml file.

    osd_objectstore: filestore

    Note

    With the osd_objectstore option set to filestore, replacing an OSD will use FileStore, instead of BlueStore.

11. In the Ansible inventory file located at /etc/ansible/hosts, add the Ceph Manager (ceph-mgr) nodes under the [mgrs] section. Colocate the Ceph Manager daemon with Monitor nodes. Skip this step when upgrading from version 3.x to the latest version.

    [mgrs]
    <monitor-host-name>
    <monitor-host-name>
    <monitor-host-name>

12. Copy rolling_update.yml from the infrastructure-playbooks directory to the current directory.

    [root@admin ceph-ansible]# cp infrastructure-playbooks/rolling_update.yml .

    Important

    Do not use the limit ansible option with the rolling_update.yml playbook.

13. Create the /var/log/ansible/ directory and assign the appropriate permissions for the ansible user:

    [root@admin ceph-ansible]# mkdir /var/log/ansible
    [root@admin ceph-ansible]# chown ansible:ansible  /var/log/ansible
    [root@admin ceph-ansible]# chmod 755 /var/log/ansible

    1. Edit the /usr/share/ceph-ansible/ansible.cfg file, updating the log_path value as follows:

       log_path = /var/log/ansible/ansible.log

14. As the Ansible user, run the playbook:

    [user@admin ceph-ansible]$ ansible-playbook rolling_update.yml

15. While logged in as the root user on the RBD mirroring daemon node, upgrade rbd-mirror manually:

    # yum upgrade rbd-mirror

    Restart the daemon:

    # systemctl restart  ceph-rbd-mirror@<client-id>

16. Verify that the cluster health is OK. ..Log into a monitor node as the root user and run the ceph status command.

1. If working in an OpenStack environment, update all the cephx users to use the RBD profile for pools. The following commands must be run as the root user:

   • Glance users

     ceph auth caps client.glance mon 'profile rbd' osd 'profile rbd pool=<glance-pool-name>'

     Example

     [root@monitor ~]# ceph auth caps client.glance mon 'profile rbd' osd 'profile rbd pool=images'

   • Cinder users

     ceph auth caps client.cinder mon 'profile rbd' osd 'profile rbd pool=<cinder-volume-pool-name>, profile rbd pool=<nova-pool-name>, profile rbd-read-only pool=<glance-pool-name>'

     Example

     [root@monitor ~]# ceph auth caps client.cinder mon 'profile rbd' osd 'profile rbd pool=volumes, profile rbd pool=vms, profile rbd-read-only pool=images'

   • OpenStack general users

     ceph auth caps client.openstack mon 'profile rbd' osd 'profile rbd-read-only pool=<cinder-volume-pool-name>, profile rbd pool=<nova-pool-name>, profile rbd-read-only pool=<glance-pool-name>'

     Example

     [root@monitor ~]# ceph auth caps client.openstack mon 'profile rbd' osd 'profile rbd-read-only pool=volumes, profile rbd pool=vms, profile rbd-read-only pool=images'

     Important

     Do these CAPS updates before performing any live client migrations. This allows clients to use the new libraries running in memory, causing the old CAPS settings to drop from cache and applying the new RBD profile settings.

Procedure

1. As the root user, update the cephmetrics-ansible package from the Ansible administration node:

   [root@admin ~]# yum update cephmetrics-ansible

2. Change to the /usr/share/cephmetrics-ansible directory:

   [root@admin ~]# cd /usr/share/cephmetrics-ansible

3. Install the updated Red Hat Ceph Storage Dashboard:

   [root@admin cephmetrics-ansible]# ansible-playbook -v playbook.yml