Installation Guide

Procedure

1. For container deployments only, when the Red Hat Ceph Storage nodes do NOT have access to the Internet during deployment. You must follow these steps first on a node with Internet access:

   1. Start a local container registry:

      Red Hat Enterprise Linux 7

      # docker run -d -p 5000:5000 --restart=always --name registry registry:2

      Red Hat Enterprise Linux 8

      # podman run -d -p 5000:5000 --restart=always --name registry registry:2

   2. Verify registry.redhat.io is in the container registry search path.

      Open for editing the /etc/containers/registries.conf file:

      [registries.search]
      registries = [ 'registry.access.redhat.com', 'registry.fedoraproject.org', 'registry.centos.org', 'docker.io']

      If registry.redhat.io is not included in the file, add it:

      [registries.search]
      registries = ['registry.redhat.io', 'registry.access.redhat.com', 'registry.fedoraproject.org', 'registry.centos.org', 'docker.io']

   3. Pull the Red Hat Ceph Storage 4 image, Prometheus image, and Dashboard image from the Red Hat Customer Portal:

      Red Hat Enterprise Linux 7

      # docker pull registry.redhat.io/rhceph/rhceph-4-rhel8:latest
      # docker pull registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.6
      # docker pull registry.redhat.io/rhceph/rhceph-4-dashboard-rhel8:latest
      # docker pull registry.redhat.io/openshift4/ose-prometheus:v4.6
      # docker pull registry.redhat.io/openshift4/ose-prometheus-alertmanager:v4.6

      Red Hat Enterprise Linux 8

      # podman pull registry.redhat.io/rhceph/rhceph-4-rhel8:latest
      # podman pull registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.6
      # podman pull registry.redhat.io/rhceph/rhceph-4-dashboard-rhel8:latest
      # podman pull registry.redhat.io/openshift4/ose-prometheus:v4.6
      # podman pull registry.redhat.io/openshift4/ose-prometheus-alertmanager:v4.6

      Note

      Red Hat Enterprise Linux 7 and 8 both use the same container image, based on Red Hat Enterprise Linux 8.

   4. Tag the image:

      The Prometheus image tag version is v4.6 for Red Hat Ceph Storage 4.2.

      Red Hat Enterprise Linux 7

       # docker tag registry.redhat.io/rhceph/rhceph-4-rhel8:latest LOCAL_NODE_FQDN:5000/rhceph/rhceph-4-rhel8:latest
       # docker tag registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.6 LOCAL_NODE_FQDN:5000/openshift4/ose-prometheus-node-exporter:v4.6
       # docker tag registry.redhat.io/rhceph/rhceph-4-dashboard-rhel8:latest LOCAL_NODE_FQDN:5000/rhceph/rhceph-4-dashboard-rhel8:latest
       # docker tag registry.redhat.io/openshift4/ose-prometheus-alertmanager:v4.6 LOCAL_NODE_FQDN:5000/openshift4/ose-prometheus-alertmanager:v4.6
       # docker tag registry.redhat.io/openshift4/ose-prometheus:v4.6 LOCAL_NODE_FQDN:5000/openshift4/ose-prometheus:v4.6

      Replace

      • LOCAL_NODE_FQDN with your local host FQDN.

      Red Hat Enterprise Linux 8

       # podman tag registry.redhat.io/rhceph/rhceph-4-rhel8:latest LOCAL_NODE_FQDN:5000/rhceph/rhceph-4-rhel8:latest
       # podman tag registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.6 LOCAL_NODE_FQDN:5000/openshift4/ose-prometheus-node-exporter:v4.6
       # podman tag registry.redhat.io/rhceph/rhceph-4-dashboard-rhel8:latest LOCAL_NODE_FQDN:5000/rhceph/rhceph-4-dashboard-rhel8:latest
       # podman tag registry.redhat.io/openshift4/ose-prometheus-alertmanager:v4.6 LOCAL_NODE_FQDN:5000/openshift4/ose-prometheus-alertmanager:v4.6
       # podman tag registry.redhat.io/openshift4/ose-prometheus:v4.6 LOCAL_NODE_FQDN:5000/openshift4/ose-prometheus:v4.6

      Replace

      • LOCAL_NODE_FQDN with your local host FQDN.

   5. Edit the /etc/containers/registries.conf file and add the node’s FQDN with the port in the file, and save:

      [registries.insecure]
      registries = ['LOCAL_NODE_FQDN:5000']

      Note

      This step must be done on all storage cluster nodes that access the local Docker registry.

   6. Push the image to the local Docker registry you started:

      Red Hat Enterprise Linux 7

       # docker push --remove-signatures LOCAL_NODE_FQDN:5000/rhceph/rhceph-4-rhel8
       # docker push --remove-signatures LOCAL_NODE_FQDN:5000/openshift4/ose-prometheus-node-exporter:v4.6
       # docker push --remove-signatures LOCAL_NODE_FQDN:5000/rhceph/rhceph-4-dashboard-rhel8
       # docker push --remove-signatures LOCAL_NODE_FQDN:5000/openshift4/ose-prometheus-alertmanager:v4.6
       # docker push --remove-signatures LOCAL_NODE_FQDN:5000/openshift4/ose-prometheus:v4.6

      Replace

      • LOCAL_NODE_FQDN with your local host FQDN.

      Red Hat Enterprise Linux 8

       # podman push --remove-signatures LOCAL_NODE_FQDN:5000/rhceph/rhceph-4-rhel8
       # podman push --remove-signatures LOCAL_NODE_FQDN:5000/openshift4/ose-prometheus-node-exporter:v4.6
       # podman push --remove-signatures LOCAL_NODE_FQDN:5000/rhceph/rhceph-4-dashboard-rhel8
       # podman push --remove-signatures LOCAL_NODE_FQDN:5000/openshift4/ose-prometheus-alertmanager:v4.6
       # podman push --remove-signatures LOCAL_NODE_FQDN:5000/openshift4/ose-prometheus:v4.6

      Replace

      • LOCAL_NODE_FQDN with your local host FQDN.

   7. For Red Hat Enterprise Linux 7, restart the docker service:

      # systemctl restart docker

      Note

      See the Installing a Red Hat Ceph Storage cluster for an example of the all.yml file when the Red Hat Ceph Storage nodes do NOT have access to the Internet during deployment.

2. For all deployments, bare-metal or in containers:

   1. Register the node, and when prompted, enter the appropriate Red Hat Customer Portal credentials:

      # subscription-manager register

   2. Pull the latest subscription data from the CDN:

      # subscription-manager refresh

   3. List all available subscriptions for Red Hat Ceph Storage:

      # subscription-manager list --available --all --matches="*Ceph*"

      Copy the Pool ID from the list of available subscriptions for Red Hat Ceph Storage.

   4. Attach the subscription:

      # subscription-manager attach --pool=POOL_ID

      Replace

      • POOL_ID with the Pool ID identified in the previous step.

   5. Disable the default software repositories, and enable the server and the extras repositories on the respective version of Red Hat Enterprise Linux:

      Red Hat Enterprise Linux 7

      # subscription-manager repos --disable=*
      # subscription-manager repos --enable=rhel-7-server-rpms
      # subscription-manager repos --enable=rhel-7-server-extras-rpms

      Red Hat Enterprise Linux 8

      # subscription-manager repos --disable=*
      # subscription-manager repos --enable=rhel-8-for-x86_64-baseos-rpms
      # subscription-manager repos --enable=rhel-8-for-x86_64-appstream-rpms

3. Update the system to receive the latest packages.

   1. For Red Hat Enterprise Linux 7:

      # yum update

   2. For Red Hat Enterprise Linux 8:

      # dnf update

1. Verify the following settings are in the /etc/sysconfig/network-scripts/ifcfg-* file corresponding the public-facing network interface card:

   1. The BOOTPROTO parameter is set to none for static IP addresses.

   2. The ONBOOT parameter must be set to yes.

      If it is set to no, the Ceph storage cluster might fail to peer on reboot.

   3. If you intend to use IPv6 addressing, you must set the IPv6 parameters such as IPV6INIT to yes, except the IPV6_FAILURE_FATAL parameter.

      Also, edit the Ceph configuration file, /etc/ceph/ceph.conf, to instruct Ceph to use IPv6, otherwise, Ceph uses IPv4.

Procedure

1. On all nodes in the storage cluster, start the firewalld service. Enable it to run on boot, and ensure that it is running:

   # systemctl enable firewalld
   # systemctl start firewalld
   # systemctl status firewalld

2. On all monitor nodes, open port 3300 and 6789 on the public network:

   [root@monitor ~]# firewall-cmd --zone=public --add-port=3300/tcp
   [root@monitor ~]# firewall-cmd --zone=public --add-port=3300/tcp --permanent
   [root@monitor ~]# firewall-cmd --zone=public --add-port=6789/tcp
   [root@monitor ~]# firewall-cmd --zone=public --add-port=6789/tcp --permanent
   [root@monitor ~]# firewall-cmd --permanent --add-service=ceph-mon
   [root@monitor ~]# firewall-cmd --add-service=ceph-mon

   To limit access based on the source address:

   firewall-cmd --zone=public --add-rich-rule='rule family=ipv4 \
   source address=IP_ADDRESS/NETMASK_PREFIX port protocol=tcp \
   port=6789 accept' --permanent

   Replace

   • IP_ADDRESS with the network address of the Monitor node.

   • NETMASK_PREFIX with the netmask in CIDR notation.

     Example

     [root@monitor ~]# firewall-cmd --zone=public --add-rich-rule='rule family=ipv4 \
     source address=192.168.0.11/24 port protocol=tcp \
     port=6789 accept' --permanent

3. On all OSD nodes, open ports 6800-7300 on the public network:

   [root@osd ~]# firewall-cmd --zone=public --add-port=6800-7300/tcp
   [root@osd ~]# firewall-cmd --zone=public --add-port=6800-7300/tcp --permanent
   [root@osd ~]# firewall-cmd --permanent --add-service=ceph
   [root@osd ~]# firewall-cmd --add-service=ceph

   If you have a separate cluster network, repeat the commands with the appropriate zone.

4. On all Ceph Manager (ceph-mgr) nodes, open ports 6800-7300 on the public network:

   [root@monitor ~]# firewall-cmd --zone=public --add-port=6800-7300/tcp
   [root@monitor ~]# firewall-cmd --zone=public --add-port=6800-7300/tcp --permanent

   If you have a separate cluster network, repeat the commands with the appropriate zone.

5. On all Ceph Metadata Server (ceph-mds) nodes, open ports 6800-7300 on the public network:

   [root@monitor ~]# firewall-cmd --zone=public --add-port=6800-7300/tcp
   [root@monitor ~]# firewall-cmd --zone=public --add-port=6800-7300/tcp --permanent

   If you have a separate cluster network, repeat the commands with the appropriate zone.

6. On all Ceph Object Gateway nodes, open the relevant port or ports on the public network.

   1. To open the default Ansible configured port of 8080:

      [root@gateway ~]# firewall-cmd --zone=public --add-port=8080/tcp
      [root@gateway ~]# firewall-cmd --zone=public --add-port=8080/tcp --permanent

      To limit access based on the source address:

      firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
      source address="IP_ADDRESS/NETMASK_PREFIX" port protocol="tcp" \
      port="8080" accept"

      firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
      source address="IP_ADDRESS/NETMASK_PREFIX" port protocol="tcp" \
      port="8080" accept" --permanent

      Replace

      • IP_ADDRESS with the network address of the Monitor node.

      • NETMASK_PREFIX with the netmask in CIDR notation.

        Example

        [root@gateway ~]# firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
        source address="192.168.0.31/24" port protocol="tcp" \
        port="8080" accept"

        [root@gateway ~]# firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
        source address="192.168.0.31/24" port protocol="tcp" \
        port="8080" accept" --permanent

   2. Optionally, if you installed Ceph Object Gateway using Ansible and changed the default port that Ansible configures the Ceph Object Gateway to use from 8080, for example, to port 80, then open this port:

      [root@gateway ~]# firewall-cmd --zone=public --add-port=80/tcp
      [root@gateway ~]# firewall-cmd --zone=public --add-port=80/tcp --permanent

      To limit access based on the source address, run the following commands:

      firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
      source address="IP_ADDRESS/NETMASK_PREFIX" port protocol="tcp" \
      port="80" accept"

      firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
      source address="IP_ADDRESS/NETMASK_PREFIX" port protocol="tcp" \
      port="80" accept" --permanent

      Replace

      • IP_ADDRESS with the network address of the Monitor node.
      • NETMASK_PREFIX with the netmask in CIDR notation.

      Example

      [root@gateway ~]# firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
      source address="192.168.0.31/24" port protocol="tcp" \
      port="80" accept"

      [root@gateway ~]# firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
      source address="192.168.0.31/24" port protocol="tcp" \
      port="80" accept" --permanent

   3. Optional. To use SSL/TLS, open port 443:

      [root@gateway ~]# firewall-cmd --zone=public --add-port=443/tcp
      [root@gateway ~]# firewall-cmd --zone=public --add-port=443/tcp --permanent

      To limit access based on the source address, run the following commands:

      firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
      source address="IP_ADDRESS/NETMASK_PREFIX" port protocol="tcp" \
      port="443" accept"

      firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
      source address="IP_ADDRESS/NETMASK_PREFIX" port protocol="tcp" \
      port="443" accept" --permanent

      Replace

      • IP_ADDRESS with the network address of the Monitor node.
      • NETMASK_PREFIX with the netmask in CIDR notation.

      Example

      [root@gateway ~]# firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
      source address="192.168.0.31/24" port protocol="tcp" \
      port="443" accept"
      [root@gateway ~]# firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
      source address="192.168.0.31/24" port protocol="tcp" \
      port="443" accept" --permanent

Procedure

1. Log into the node as the root user:

   ssh root@HOST_NAME

   Replace

   • HOST_NAME with the host name of the Ceph node.

     Example

     # ssh root@mon01

     Enter the root password when prompted.

2. Create a new Ansible user:

   adduser USER_NAME

   Replace

   • USER_NAME with the new user name for the Ansible user.

     Example

     # adduser admin

     Important

     Do not use ceph as the user name. The ceph user name is reserved for the Ceph daemons. A uniform user name across the cluster can improve ease of use, but avoid using obvious user names, because intruders typically use them for brute-force attacks.

3. Set a new password for this user:

   # passwd USER_NAME

   Replace

   • USER_NAME with the new user name for the Ansible user.

     Example

     # passwd admin

     Enter the new password twice when prompted.

4. Configure sudo access for the newly created user:

   cat << EOF >/etc/sudoers.d/USER_NAME
   $USER_NAME ALL = (root) NOPASSWD:ALL
   EOF

   Replace

   • USER_NAME with the new user name for the Ansible user.

     Example

     # cat << EOF >/etc/sudoers.d/admin
     admin ALL = (root) NOPASSWD:ALL
     EOF

5. Assign the correct file permissions to the new file:

   chmod 0440 /etc/sudoers.d/USER_NAME

   Replace

   • USER_NAME with the new user name for the Ansible user.

     Example

     # chmod 0440 /etc/sudoers.d/admin

Procedure

1. Generate the SSH key pair, accept the default file name and leave the passphrase empty:

   [ansible@admin ~]$ ssh-keygen

2. Copy the public key to all nodes in the storage cluster:

   ssh-copy-id USER_NAME@HOST_NAME

   Replace

   • USER_NAME with the new user name for the Ansible user.

   • HOST_NAME with the host name of the Ceph node.

     Example

     [ansible@admin ~]$ ssh-copy-id ceph-admin@ceph-mon01

3. Create the user’s SSH config file:

   [ansible@admin ~]$ touch ~/.ssh/config

4. Open for editing the config file. Set values for the Hostname and User options for each node in the storage cluster:

   Host node1
      Hostname HOST_NAME
      User USER_NAME
   Host node2
      Hostname HOST_NAME
      User USER_NAME
   ...

   Replace

   • HOST_NAME with the host name of the Ceph node.

   • USER_NAME with the new user name for the Ansible user.

     Example

     Host node1
        Hostname monitor
        User admin
     Host node2
        Hostname osd
        User admin
     Host node3
        Hostname gateway
        User admin

     Important

     By configuring the ~/.ssh/config file you do not have to specify the -u USER_NAME option each time you execute the ansible-playbook command.

5. Set the correct file permissions for the ~/.ssh/config file:

   [admin@admin ~]$ chmod 600 ~/.ssh/config

Procedure

1. Verify Cockpit is installed.

   $ rpm -q cockpit

   Example:

   [admin@jb-ceph4-admin ~]$ rpm -q cockpit
   cockpit-196.3-1.el8.x86_64

   If you see similar output to the example above, skip to the step Verify Cockpit is running. If the output is package cockpit is not installed, continue to the step Install Cockpit.

2. Optional: Install Cockpit.

   1. For Red Hat Enterprise Linux 8:

      # dnf install cockpit

   2. For Red Hat Enterprise Linux 7:

      # yum install cockpit

3. Verify Cockpit is running.

   # systemctl status cockpit.socket

   If you see Active: active (listening) in the output, skip to the step Install the Cockpit plugin for Red Hat Ceph Storage. If instead you see Active: inactive (dead), continue to the step Enable Cockpit.

4. Optional: Enable Cockpit.

   1. Use the systemctl command to enable Cockpit:

      # systemctl enable --now cockpit.socket

      You will see a line like the following:

      Created symlink /etc/systemd/system/sockets.target.wants/cockpit.socket → /usr/lib/systemd/system/cockpit.socket.

   2. Verify Cockpit is running:

      # systemctl status cockpit.socket

      You will see a line like the following:

      Active: active (listening) since Tue 2020-01-07 18:49:07 EST; 7min ago

5. Install the Cockpit Ceph Installer for Red Hat Ceph Storage.

   1. For Red Hat Enterprise Linux 8:

      # dnf install cockpit-ceph-installer

   2. For Red Hat Enterprise Linux 7:

      # yum install cockpit-ceph-installer

6. As the Ansible user, log in to the container catalog using sudo:

   Note

   By default, the Cockpit Ceph Installer uses the root user to install Ceph. To use the Ansible user created as a part of the prerequisites to install Ceph, run the rest of the commands in this procedure with sudo as the Ansible user.

   Red Hat Enterprise Linux 7

   $ sudo docker login -u CUSTOMER_PORTAL_USERNAME https://registry.redhat.io

   Example

   [admin@jb-ceph4-admin ~]$ sudo docker login -u myusername https://registry.redhat.io
   Password:
   Login Succeeded!

   Red Hat Enterprise Linux 8

   $ sudo podman login -u CUSTOMER_PORTAL_USERNAME https://registry.redhat.io

   Example

   [admin@jb-ceph4-admin ~]$ sudo podman login -u myusername https://registry.redhat.io
   Password:
   Login Succeeded!

7. Verify registry.redhat.io is in the container registry search path.

   1. Open for editing the /etc/containers/registries.conf file:

      [registries.search]
      registries = [ 'registry.access.redhat.com', 'registry.fedoraproject.org', 'registry.centos.org', 'docker.io']

      If registry.redhat.io is not included in the file, add it:

      [registries.search]
      registries = ['registry.redhat.io', 'registry.access.redhat.com', 'registry.fedoraproject.org', 'registry.centos.org', 'docker.io']

8. As the Ansible user, start the ansible-runner-service using sudo.

   $ sudo ansible-runner-service.sh -s

   Example

   [admin@jb-ceph4-admin ~]$ sudo ansible-runner-service.sh -s
   Checking environment is ready
   Checking/creating directories
   Checking SSL certificate configuration
   Generating RSA private key, 4096 bit long modulus (2 primes)
   ..................................................................................................................................................................................................................................++++
   ......................................................++++
   e is 65537 (0x010001)
   Generating RSA private key, 4096 bit long modulus (2 primes)
   ........................................++++
   ..............................................................................................................................................................................++++
   e is 65537 (0x010001)
   writing RSA key
   Signature ok
   subject=C = US, ST = North Carolina, L = Raleigh, O = Red Hat, OU = RunnerServer, CN = jb-ceph4-admin
   Getting CA Private Key
   Generating RSA private key, 4096 bit long modulus (2 primes)
   .....................................................................................................++++
   ..++++
   e is 65537 (0x010001)
   writing RSA key
   Signature ok
   subject=C = US, ST = North Carolina, L = Raleigh, O = Red Hat, OU = RunnerClient, CN = jb-ceph4-admin
   Getting CA Private Key
   Setting ownership of the certs to your user account(admin)
   Setting target user for ansible connections to admin
   Applying SELINUX container_file_t context to '/etc/ansible-runner-service'
   Applying SELINUX container_file_t context to '/usr/share/ceph-ansible'
   Ansible API (runner-service) container set to rhceph/ansible-runner-rhel8:latest
   Fetching Ansible API container (runner-service). Please wait...
   Trying to pull registry.redhat.io/rhceph/ansible-runner-rhel8:latest...Getting image source signatures
   Copying blob c585fd5093c6 done
   Copying blob 217d30c36265 done
   Copying blob e61d8721e62e done
   Copying config b96067ea93 done
   Writing manifest to image destination
   Storing signatures
   b96067ea93c8d6769eaea86854617c63c61ea10c4ff01ecf71d488d5727cb577
   Starting Ansible API container (runner-service)
   Started runner-service container
   Waiting for Ansible API container (runner-service) to respond
   The Ansible API container (runner-service) is available and responding to requests
   
   Login to the cockpit UI at https://jb-ceph4-admin:9090/cockpit-ceph-installer to start the install

   The last line of output includes the URL to the Cockpit Ceph Installer. In the example above the URL is https://jb-ceph4-admin:9090/cockpit-ceph-installer. Take note of the URL printed in your environment.

Procedure

1. Log in to the Ansible administration node as the Ansible user.

   ssh ANSIBLE_USER@HOST_NAME

   Example:

   $ ssh admin@jb-ceph4-admin

2. Copy the SSH public key to the first node:

   sudo ssh-copy-id -f -i /usr/share/ansible-runner-service/env/ssh_key.pub _ANSIBLE_USER_@_HOST_NAME_

   Example:

   $ sudo ssh-copy-id -f -i /usr/share/ansible-runner-service/env/ssh_key.pub admin@jb-ceph4-mon
   /bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/usr/share/ansible-runner-service/env/ssh_key.pub"
   admin@192.168.122.182's password:
   
   Number of key(s) added: 1
   
   Now try logging into the machine, with:   "ssh 'admin@jb-ceph4-mon'"
   and check to make sure that only the key(s) you wanted were added.

   Repeat this step for all nodes in the cluster

Procedure

1. Open the URL in a web browser.

   

2. Enter the Ansible user name and its password.

   

3. Click the radio button for Reuse my password for privileged tasks.

   

4. Click Log In.

   

5. Review the welcome page to understand how the installer works and the overall flow of the installation process.

   

   Click the Environment button at the bottom right corner of the web page after you have reviewed the information in the welcome page.

Procedure

1. Select the Installation Source. Choose Red Hat to use repositories from Red Hat Subscription Manager, or ISO to use a CD image downloaded from the Red Hat Customer Portal.

   

   If you choose Red Hat, Target Version will be set to RHCS 4 without any other options. If you choose ISO, Target Version will be set to the ISO image file.

   Important

   If you choose ISO, the image file must be in the /usr/share/ansible-runner-service/iso directory and its SELinux context must be set to container_file_t.

   Important

   The Community and Distribution options for Installation Source are not supported.

2. Select the Cluster Type. The Production selection prohibits the install from proceeding if certain resource requirements like CPU number and memory size are not met. To allow the cluster installation to proceed even if the resource requirements are not met, select Development/POC.

   
   Important

   Do not use Development/POC mode to install a Ceph cluster that will be used in production.

3. Set the Service Account Login and Service Account Token. If you do not have a Red Hat Registry Service Account, create one using the Registry Service Account webpage.

   

4. Set Configure Firewall to ON to apply rules to firewalld to open ports for Ceph services. Use the OFF setting if you are not using firewalld.

   

5. Currently, the Cockpit Ceph Installer only supports IPv4. If you require IPv6 support, discountinue use of the Cockpit Ceph Installer and proceed with installing Ceph using the Ansible scripts directly.

   

6. Set OSD Type to BlueStore or FileStore.

   
   Important

   BlueStore is the default OSD type. Previously, Ceph used FileStore as the object store. This format is deprecated for new Red Hat Ceph Storage 4.0 installs because BlueStore offers more features and improved performance. It is still possible to use FileStore, but using it requires a support exception. For more information on BlueStore, see Ceph BlueStore in the Architecture Guide.

7. Set Flash Configuration to Journal/Logs or OSD data. If you have Solid State Drives (SSDs), whether they use NVMe or a traditional SATA/SAS interface, you can choose to use them just for write journaling and logs while the actual data goes on Hard Disk Drives (HDDs), or you can use the SSDs for journaling, logs, and data, and not use HDDs for any Ceph OSD functions.

   

8. Set Encryption to None or Encrypted. This refers to at rest encryption of storage devices using the LUKS1 format.

   

9. Set Installation type to Container or RPM. Traditionally, Red Hat Package Manager (RPM) was used to install software on Red Hat Enterprise Linux. Now, you can install Ceph using RPM or containers. Installing Ceph using containers can provide improved hardware utilization since services can be isolated and collocated.

   

10. Review all the Environment settings and click the Hosts button at the bottom right corner of the webpage.

    

Procedure

1. Click the Add Host(s) button.

   

2. Enter the hostname for a Ceph OSD node, check the box for OSD, and click the Add button.

   

   The first Ceph OSD node is added.

   

   For production clusters, repeat this step until you have added at least three Ceph OSD nodes.

3. Optional: Use a host name pattern to define a range of nodes. For example, to add jb-ceph4-osd2 and jb-ceph4-osd3 at the same time, enter jb-ceph4-osd[2-3].

   

   Both jb-ceph4-osd2 and jb-ceph4-ods3 are added.

   

4. Repeat the above steps for the other nodes in your cluster.

   1. For production clusters, add at least three Ceph Monitor nodes. In the dialog, the role is listed as MON.
   2. Add a node with the Metrics role. The Metrics role installs Grafana and Prometheus to provide real-time insights into the performance of the Ceph cluster. These metrics are presented in the Ceph Dashboard, which allows you to monitor and manage the cluster. The installation of the dashboard, Grafana, and Prometheus are required. You can colocate the metrics functions on the Ansible Administration node. If you do, ensure the system resources of the node are greater than what is required for a stand alone metrics node.
   3. Optional: Add a node with the MDS role. The MDS role installs the Ceph Metadata Server (MDS). Metadata Server daemons are necessary for deploying a Ceph File System.
   4. Optional: Add a node with the RGW role. The RGW role installs the Ceph Object Gateway, also know as the RADOS gateway, which is an object storage interface built on top of the librados API to provide applications with a RESTful gateway to Ceph storage clusters. It supports the Amazon S3 and OpenStack Swift APIs.
   5. Optional: Add a node with the iSCSI role. The iSCSI role installs an iSCSI gateway so you can share Ceph Block Devices over iSCSI. To use iSCSI with Ceph, you must install the iSCSI gateway on at least two nodes for multipath I/O.

5. Optional: Colocate more than one service on the same node by selecting multiple roles when adding the node.

   

   For more information on colocating daemons, see Colocation of containerized Ceph daemons in the Installation Guide.

6. Optional: Modify the roles assigned to a node by checking or unchecking roles in the table.

   

7. Optional: To delete a node, on the far right side of the row of the node you want to delete, click the kebab icon and then click Delete.

   

8. Click the Validate button at the bottom right corner of the page after you have added all the nodes in your cluster and set all the required roles.

   

Procedure

1. Click the Probe Hosts button.

   

   To continue you must select at least three hosts which have an OK Status.

2. Optional: If warnings or errors were generated for hosts, click the arrow to the left of the check mark for the host to view the issues.

   
   
   Important

   If you set Cluster Type to Production, any errors generated will cause Status to be NOTOK and you will not be able to select them for installation. Read the next step for information on how to resolve errors.

   Important

   If you set Cluster Type to Development/POC, any errors generated will be listed as warnings so Status is always OK. This allows you to select the hosts and install Ceph on them regardless of whether the hosts meet the requirements or suggestions. You can still resolve warnings if you want to. Read the next step for information on how to resolve warnings.

3. Optional: To resolve errors and warnings use one or more of the following methods.

   1. The easiest way to resolve errors or warnings is to disable certain roles completely or to disable a role on one host and enable it on another host which has the required resources.

      Experiment with enabling or disabling roles until you find a combination where, if you are installing a Development/POC cluster, you are comfortable proceeding with any remaining warnings, or if you are installing a Production cluster, at least three hosts have all the resources required for the roles assigned to them and you are comfortable proceeding with any remaining warnings.

   2. You can also use a new host which meets the requirements for the roles required. First go back to the Hosts page and delete the hosts with issues.

      

      Then, add the new hosts.

   3. If you want to upgrade the hardware on a host or modify it in some other way so it will meet the requirements or suggestions, first make the desired changes to the host, and then click Probe Hosts again. If you have to reinstall the operating system you will have to copy the SSH key again.

4. Select the hosts to install Red Hat Ceph Storage on by checking the box next to the host.

   
   Important

   If installing a production cluster, you must resolve any errors before you can select them for installation.

5. Click the Network button at the bottom right corner of the page to review and configure networking for the cluster.

   

Procedure

1. Take note of the network types you can configure on the Network page. Each type has its own column. Columns for Cluster Network and Public Network are always displayed. If you are installing hosts with the RADOS Gateway role, the S3 Network column will be displayed. If you are installing hosts with the iSCSI role, the iSCSI Network column will be displayed. In the example below, columns for Cluster Network, Public Network, and S3 Network are shown.

   

2. Take note of the networks you can select for each network type. Only the networks which are available on all hosts that make up a particular network type are shown. In the example below, there are three networks which are available on all hosts in the cluster. Because all three networks are available on every set of hosts which make up a network type, each network type lists the same three networks.

   

   The three networks available are 192.168.122.0/24, 192.168.123.0/24, and 192.168.124.0/24.

3. Take note of the speed each network operates at. This is the speed of the NICs used for the particular network. In the example below, 192.168.123.0/24, and 192.168.124.0/24 are at 1,000 mbps. The Cockpit Ceph Installer could not determine the speed for the 192.168.122.0/24 network.

   

4. Select the networks you want to use for each network type. For production clusters, you must select separate networks for Cluster Network and Public Network. For development/POC clusters, you can select the same network for both types, or if you only have one network configured on all hosts, only that network will be displayed and you will not be able to select other networks.

   

   The 192.168.122.0/24 network will be used for the Public Network, the 192.168.123.0/24 network will be used for the Cluster Network, and the 192.168.124.0/24 network will be used for the S3 Network.

5. Click the Review button at the bottom right corner of the page to review the entire cluster configuration before installation.

   

Procedure

1. View the review page.

   

2. Verify the information from each previous page is as you expect it as shown on the Review page. A summary of information from the Environment page is at 1, followed by the Hosts page at 2, the Validate page at 3, the Network page at 4, and details about the hosts, including some additional details which were not included in previous pages, are at 5.

   

3. Click the Deploy button at the bottom right corner of the page to go to the Deploy page where you can finalize and start the actual installation process.

   

Procedure

1. Click the Save button at the bottom right corner of the page to save the installation settings to the Ansible playbooks that will be used by Ansible to perform the actual install.

   
2. Optional: View or further customize the settings in the Ansible playbooks located on the Ansible administration node. The playbooks are located in /usr/share/ceph-ansible. For more information about the Ansible playbooks and how to use them to customize the install, see Installing a Red Hat Ceph Storage cluster.
3. Secure the default user names and passwords for Grafana and dashboard. Starting with Red Hat Ceph Storage 4.1, you must uncomment or set dashboard_admin_password and grafana_admin_password in /usr/share/ceph-ansible/group_vars/all.yml. Set secure passwords for each. Also set custom user names for dashboard_admin_user and grafana_admin_user.

4. Click the Deploy button at the bottom right corner of the page to start the install.

   

5. Observe the installation progress while it is running.

   The information at 1 shows whether the install is running or not, the start time, and elapsed time. The information at 2 shows a summary of the Ansible tasks that have been attempted. The information at 3 shows which roles have been installed or are installing. Green represents a role where all hosts that were assigned that role have had that role installed on them. Blue represents a role where hosts that have that role assigned to them are still being installed. At 4 you can view details about the current task or view failed tasks. Use the Filter by menu to switch between current task and failed tasks.

   

   The role names come from the Ansible inventory file. The equivalency is: mons are Monitors, mgrs are Managers, note the Manager role is installed alongside the Monitor role, osds are Object Storage Devices, mdss are Metadata Servers, rgws are RADOS Gateways, metrics are Grafana and Prometheus services for dashboard metrics. Not shown in the example screenshot: iscsigws are iSCSI Gateways.

6. After the installation finishes, click the Complete button at the bottom right corner of the page. This opens a window which displays the output of the command ceph status, as well as dashboard access information.

   

7. Compare cluster status information in the example below with the cluster status information on your cluster. The example shows a healthy cluster, with all OSDs up and in, and all services active. PGs are in the active+clean state. If some aspects of your cluster are not the same, refer to the Troubleshoting Guide for information on how to resolve the issues.

   

8. At the bottom of the Ceph Cluster Status window, the dashboard access information is displayed, including the URL, user name, and password. Take note of this information.

   

9. Use the information from the previous step along with the Dashboard Guide to access the dashboard.

   

   The dashboard provides a web interface so you can administer and monitor the Red Hat Ceph Storage cluster. For more information, see the Dashboard Guide.

10. Optional: View the cockpit-ceph-installer.log file. This file records a log of the selections made and any associated warnings the probe process generated. It is located in the home directory of the user that ran the installer script, ansible-runner-service.sh.

Procedure

1. Log in as the root user account on the Ansible administration node.

2. For all deployments, bare-metal or in containers, install the ceph-ansible package:

   Red Hat Enterprise Linux 7

   [root@admin ~]# yum install ceph-ansible

   Red Hat Enterprise Linux 8

   [root@admin ~]# dnf install ceph-ansible

3. Navigate to the /usr/share/ceph-ansible/ directory:

   [root@admin ~]# cd /usr/share/ceph-ansible

4. Create new yml files:

   [root@admin ceph-ansible]# cp group_vars/all.yml.sample group_vars/all.yml
   [root@admin ceph-ansible]# cp group_vars/osds.yml.sample group_vars/osds.yml

   1. Bare-metal deployments:

      [root@admin ceph-ansible]# cp site.yml.sample site.yml

   2. Container deployments:

      [root@admin ceph-ansible]# cp site-container.yml.sample site-container.yml

5. Edit the new files.

   1. Open for editing the group_vars/all.yml file.

      Important

      Using a custom storage cluster name is not supported. Do not set the cluster parameter to any value other than ceph. Using a custom storage cluster name is only supported with Ceph clients, such as: librados, the Ceph Object Gateway, and RADOS block device mirroring.

      Warning

      By default, Ansible attempts to restart an installed, but masked firewalld service, which can cause the Red Hat Ceph Storage deployment to fail. To work around this issue, set the configure_firewall option to false in the all.yml file. If you are running the firewalld service, then there is no requirement to use the configure_firewall option in the all.yml file.

      Note

      Having the ceph_rhcs_version option set to 4 will pull in the latest version of Red Hat Ceph Storage 4.

      Note

      Red Hat recommends leaving the dashboard_enabled option set to True in the group_vars/all.yml file, and not changing it to False. If you want to disable the dashboard, see Disabling the Ceph Dashboard.

      Note

      Dashboard related components are containerized. Therefore, for Bare-metal or Container deployment, ceph_docker_registry_username and ceph_docker_registry_password parameters have to be included so that ceph-ansible can fetch container images required for the dashboard.

      Note

      If you do not have a Red Hat Registry Service Account, create one using the Registry Service Account webpage. See the Red Hat Container Registry Authentication Knowledgebase article for details on how to create and manage tokens.

      Note

      In addition to using a Service Account for the ceph_docker_registry_username and ceph_docker_registry_password parameters, you can also use your Customer Portal credentials, but to ensure security, encrypt the ceph_docker_registry_password parameter. For more information, see Encrypting Ansible password variables with ansible-vault.

      1. Bare-metal example of the all.yml file for CDN installation:

         fetch_directory: ~/ceph-ansible-keys
         ceph_origin: repository
         ceph_repository: rhcs
         ceph_repository_type: cdn
         ceph_rhcs_version: 4
         monitor_interface: eth0 1
         public_network: 192.168.0.0/24
         ceph_docker_registry: registry.redhat.io
         ceph_docker_registry_auth: true
         ceph_docker_registry_username: SERVICE_ACCOUNT_USER_NAME
         ceph_docker_registry_password: TOKEN
         dashboard_admin_user:
         dashboard_admin_password:
         node_exporter_container_image: registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.6
         grafana_admin_user:
         grafana_admin_password:
         grafana_container_image: registry.redhat.io/rhceph/rhceph-4-dashboard-rhel8
         prometheus_container_image: registry.redhat.io/openshift4/ose-prometheus:v4.6
         alertmanager_container_image: registry.redhat.io/openshift4/ose-prometheus-alertmanager:v4.6

         1

         This is the interface on the public network.

         Important

         Starting with Red Hat Ceph Storage 4.1, you must uncomment or set dashboard_admin_password and grafana_admin_password in /usr/share/ceph-ansible/group_vars/all.yml. Set secure passwords for each. Also set custom user names for dashboard_admin_user and grafana_admin_user.

         Note

         For Red Hat Ceph Storage 4.2, if you have used local registry for installation, use 4.6 for Prometheus image tags.

      2. Bare-metal example of the all.yml file for ISO installation:

         fetch_directory: ~/ceph-ansible-keys
         ceph_origin: repository
         ceph_repository: rhcs
         ceph_repository_type: iso
         ceph_rhcs_iso_path: /home/rhceph-4-rhel-8-x86_64.iso
         ceph_rhcs_version: 4
         monitor_interface: eth0 1
         public_network: 192.168.0.0/24
         ceph_docker_registry: registry.redhat.io
         ceph_docker_registry_auth: true
         ceph_docker_registry_username: SERVICE_ACCOUNT_USER_NAME
         ceph_docker_registry_password: TOKEN
         dashboard_admin_user:
         dashboard_admin_password:
         node_exporter_container_image: registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.6
         grafana_admin_user:
         grafana_admin_password:
         grafana_container_image: registry.redhat.io/rhceph/rhceph-4-dashboard-rhel8
         prometheus_container_image: registry.redhat.io/openshift4/ose-prometheus:v4.6
         alertmanager_container_image: registry.redhat.io/openshift4/ose-prometheus-alertmanager:v4.6

         1

         This is the interface on the public network.

      3. Containers example of the all.yml file:

         fetch_directory: ~/ceph-ansible-keys
         monitor_interface: eth0 1
         public_network: 192.168.0.0/24
         ceph_docker_image: rhceph/rhceph-4-rhel8
         ceph_docker_image_tag: latest
         containerized_deployment: true
         ceph_docker_registry: registry.redhat.io
         ceph_docker_registry_auth: true
         ceph_docker_registry_username: SERVICE_ACCOUNT_USER_NAME
         ceph_docker_registry_password: TOKEN
         ceph_origin: repository
         ceph_repository: rhcs
         ceph_repository_type: cdn
         ceph_rhcs_version: 4
         dashboard_admin_user:
         dashboard_admin_password:
         node_exporter_container_image: registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.6
         grafana_admin_user:
         grafana_admin_password:
         grafana_container_image: registry.redhat.io/rhceph/rhceph-4-dashboard-rhel8
         prometheus_container_image: registry.redhat.io/openshift4/ose-prometheus:v4.6
         alertmanager_container_image: registry.redhat.io/openshift4/ose-prometheus-alertmanager:v4.6

         1

         This is the interface on the public network.

         Important

         Look up the latest container images tags on the Red Hat Ecosystem Catalog to install the latest container images with all the latest patches applied.

      4. Containers example of the all.yml file, when the Red Hat Ceph Storage nodes do NOT have access to the Internet during deployment:

         fetch_directory: ~/ceph-ansible-keys
         monitor_interface: eth0 1
         public_network: 192.168.0.0/24
         ceph_docker_image: rhceph/rhceph-4-rhel8
         ceph_docker_image_tag: latest
         containerized_deployment: true
         ceph_docker_registry: LOCAL_NODE_FQDN:5000
         ceph_docker_registry_auth: false
         ceph_origin: repository
         ceph_repository: rhcs
         ceph_repository_type: cdn
         ceph_rhcs_version: 4
         dashboard_admin_user:
         dashboard_admin_password:
         node_exporter_container_image: LOCAL_NODE_FQDN:5000/openshift4/ose-prometheus-node-exporter:v4.6
         grafana_admin_user:
         grafana_admin_password:
         grafana_container_image: LOCAL_NODE_FQDN:5000/rhceph/rhceph-4-dashboard-rhel8
         prometheus_container_image: LOCAL_NODE_FQDN:5000/openshift4/ose-prometheus:4.6
         alertmanager_container_image: LOCAL_NODE_FQDN:5000/openshift4/ose-prometheus-alertmanager:4.6

         1

         This is the interface on the public network.

         Replace

         • LOCAL_NODE_FQDN with your local host FQDN.

      5. From Red Hat Ceph Storage 4.2, dashboard_protocol is set to https and Ansible generates the dashboard and grafana keys and certificates. For custom certificates, in the all.yml file, update the path at Ansible installer host for dashboard_crt, dashboard_key, grafana_crt, and grafana_key for bare-metal or container deployment.

         Syntax

         dashboard_protocol: https
         dashboard_port: 8443
         dashboard_crt: 'DASHBOARD_CERTIFICATE_PATH'
         dashboard_key: 'DASHBOARD_KEY_PATH'
         dashboard_tls_external: false
         dashboard_grafana_api_no_ssl_verify: "{{ True if dashboard_protocol == 'https' and not grafana_crt and not grafana_key else False }}"
         grafana_crt: 'GRAFANA_CERTIFICATE_PATH'
         grafana_key: 'GRAFANA_KEY_PATH'

   2. To install Red Hat Ceph Storage using a container registry reachable with a http or https proxy, set the ceph_docker_http_proxy or ceph_docker_https_proxy variables in the group_vars/all.yml file.

      Example

      ceph_docker_http_proxy: http://192.168.42.100:8080
      ceph_docker_https_proxy: https://192.168.42.100:8080

      If you need to exclude some host for the proxy configuration, use the ceph_docker_no_proxy variable in the group_vars/all.yml file.

      Example

      ceph_docker_no_proxy: "localhost,127.0.0.1"

   3. In addition to editing the all.yml file for proxy installation of Red Hat Ceph Storage, edit the /etc/environment file:

      Example

      HTTP_PROXY: http://192.168.42.100:8080
      HTTPS_PROXY: https://192.168.42.100:8080
      NO_PROXY: "localhost,127.0.0.1"

      This triggers the podman to start the containerized services such as prometheus, grafana-server, alertmanager, and node-exporter, and download the required images.

   4. For all deployments, bare-metal or in containers, edit the group_vars/osds.yml file.

      Important

      Do not install an OSD on the device the operating system is installed on. Sharing the same device between the operating system and OSDs causes performance issues.

      Ceph-ansible uses the ceph-volume tool to prepare storage devices for Ceph usage. You can configure osds.yml to use your storage devices in different ways to optimize performance for your particular workload.

      Important

      All the examples below use the BlueStore object store, which is the format Ceph uses to store data on devices. Previously, Ceph used FileStore as the object store. This format is deprecated for new Red Hat Ceph Storage 4.0 installs because BlueStore offers more features and improved performance. It is still possible to use FileStore, but using it requires a Red Hat support exception. For more information on BlueStore, see Ceph BlueStore in the Red Hat Ceph Storage Architecture Guide.

      1. Auto discovery

         osd_auto_discovery: true

         The above example uses all empty storage devices on the system to create the OSDs, so you do not have to specify them explicitly. The ceph-volume tool checks for empty devices, so devices which are not empty will not be used.

         Note

         If you later decide to remove the cluster using purge-docker-cluster.yml or purge-cluster.yml, you must comment out osd_auto_discovery and declare the OSD devices in the osds.yml file. For more information, see Purging storage clusters deployed by Ansible.

      2. Simple configuration

         First Scenario

         devices:
           - /dev/sda
           - /dev/sdb

         or

         Second Scenario

         devices:
           - /dev/sda
           - /dev/sdb
           - /dev/nvme0n1
           - /dev/sdc
           - /dev/sdd
           - /dev/nvme1n1

         or

         Third Scenario

         lvm_volumes:
            - data: /dev/sdb
            - data: /dev/sdc

         or

         Fourth Scenario

         lvm_volumes:
             - data: /dev/sdb
             - data:/dev/nvme0n1

         When using the devices option alone, ceph-volume lvm batch mode automatically optimizes OSD configuration.

         In the first scenario, if the devices are traditional hard drives or SSDs, then one OSD per device is created.

         In the second scenario, when there is a mix of traditional hard drives and SSDs, the data is placed on the traditional hard drives (sda, sdb) and the BlueStore database is created as large as possible on the SSD (nvme0n1). Similarly, the data is placed on the traditional hard drives (sdc, sdd), and the BlueStore database is created on the SSD nvme1n1 irrespective of the order of devices mentioned.

         Note

         By default ceph-ansible does not override the default values of bluestore_block_db_size and bluestore_block_wal_size. You can set bluestore_block_db_size using ceph_conf_overrides in the group_vars/all.yml file. The value of bluestore_block_db_size should be greater than 2 GB.

         In the third scenario, data is placed on the traditional hard drives (sdb, sdc), and the BlueStore database is collocated on the same devices.

         In the fourth scenario, data is placed on the traditional hard drive (sdb) and on the SSD (nvme1n1), and the BlueStore database is collocated on the same devices. This is different from using the devices directive, where the BlueStore database is placed on the SSD.

         Important

         The ceph-volume lvm batch mode command creates the optimized OSD configuration by placing data on the traditional hard drives and the BlueStore database on the SSD. If you want to specify the logical volumes and volume groups to use, you can create them directly by following the Advanced configuration scenarios below.

      3. Advanced configuration

         First Scenario

         devices:
           - /dev/sda
           - /dev/sdb
         dedicated_devices:
           - /dev/sdx
           - /dev/sdy

         or

         Second Scenario

         devices:
           - /dev/sda
           - /dev/sdb
         dedicated_devices:
           - /dev/sdx
           - /dev/sdy
         bluestore_wal_devices:
           - /dev/nvme0n1
           - /dev/nvme0n2

         In the first scenario, there are two OSDs. The sda and sdb devices each have their own data segments and write-ahead logs. The additional dictionary dedicated_devices is used to isolate their databases, also known as block.db, on sdx and sdy, respectively.

         In the second scenario, another additional dictionary, bluestore_wal_devices, is used to isolate the write-ahead log on NVMe devices nvme0n1 and nvme0n2. Using the devices, dedicated_devices, and bluestore_wal_devices, options together, this allows you to isolate all components of an OSD onto separate devices. Laying out the OSDs like this can increase overall performance.

      4. Pre-created logical volumes

         First Scenario

         lvm_volumes:
           - data: data-lv1
             data_vg: data-vg1
             db: db-lv1
             db_vg: db-vg1
             wal: wal-lv1
             wal_vg: wal-vg1
           - data: data-lv2
             data_vg: data-vg2
             db: db-lv2
             db_vg: db-vg2
             wal: wal-lv2
             wal_vg: wal-vg2

         or

         Second Scenario

         lvm_volumes:
           - data: /dev/sdb
             db:    db-lv1
             db_vg: db-vg1
             wal: wal-lv1
             wal_vg: wal-vg1

         By default, Ceph uses Logical Volume Manager to create logical volumes on the OSD devices. In the Simple configuration and Advanced configuration examples above, Ceph creates logical volumes on the devices automatically. You can use previously created logical volumes with Ceph by specifying the lvm_volumes dictionary.

         In the first scenario, the data is placed on dedicated logical volumes, database, and WAL. You can also specify just data, data and WAL, or data and database. The data: line must specify the logical volume name where data is to be stored, and data_vg: must specify the name of the volume group the data logical volume is contained in. Similarly, db: is used to specify the logical volume the database is stored on and db_vg: is used to specify the volume group its logical volume is in. The wal: line specifies the logical volume the WAL is stored on and the wal_vg: line specifies the volume group that contains it.

         In the second scenario, the actual device name is set for the data: option, and doing so, does not require specifying the data_vg: option. You must specify the logical volume name and the volume group details for the BlueStore database and WAL devices.

         Important

         With lvm_volumes:, the volume groups and logical volumes must be created beforehand. The volume groups and logical volumes will not be created by ceph-ansible.

         Note

         If using all NVMe SSDs, then set osds_per_device: 2. For more information, see Configuring OSD Ansible settings for all NVMe Storage in the Red Hat Ceph Storage Installation Guide.

         Note

         After rebooting a Ceph OSD node, there is a possibility that the block device assignments will change. For example, sdc might become sdd. You can use persistent naming devices, such as the /dev/disk/by-path/ device path, instead of the traditional block device name.

6. For all deployments, bare-metal or in containers, create the Ansible inventory file and then open it for editing:

   [root@admin ~]# cd /usr/share/ceph-ansible/
   [root@admin ceph-ansible]# touch hosts

   Edit the hosts file accordingly.

   Note

   For information about editing the Ansible inventory location, see Configuring Ansible inventory location.

   1. Add a node under [grafana-server]. This role installs Grafana and Prometheus to provide real-time insights into the performance of the Ceph cluster. These metrics are presented in the Ceph Dashboard, which allows you to monitor and manage the cluster. The installation of the dashboard, Grafana, and Prometheus are required. You can colocate the metrics functions on the Ansible Administration node. If you do, ensure the system resources of the node are greater than than what is required for a stand alone metrics node.

      [grafana-server]
      GRAFANA-SERVER_NODE_NAME

   2. Add the monitor nodes under the [mons] section:

      [mons]
      MONITOR_NODE_NAME_1
      MONITOR_NODE_NAME_2
      MONITOR_NODE_NAME_3

   3. Add OSD nodes under the [osds] section:

      [osds]
      OSD_NODE_NAME_1
      OSD_NODE_NAME_2
      OSD_NODE_NAME_3

      Note

      You can add a range specifier ([1:10]) to the end of the node name, if the node names are numerically sequential. For example:

      [osds]
      example-node[1:10]

      Note

      For OSDs in a new installation, the default object store format is BlueStore.

   4. Optionally, in container deployments, colocate Ceph Monitor daemons with the Ceph OSD daemons on one node by adding the same node under the [mon] and [osd] sections. In the Additional Resources section below, see the link on colocating Ceph daemons for more information.

   5. Add the Ceph Manager (ceph-mgr) nodes under the [mgrs] section. This is colocating the Ceph Manager daemon with Ceph Monitor daemon.

      [mgrs]
      MONITOR_NODE_NAME_1
      MONITOR_NODE_NAME_2
      MONITOR_NODE_NAME_3

7. Optionally, if you want to use host specific parameters, for all deployments, bare-metal or in containers, create the host_vars directory with host files to include any parameters specific to hosts.

   1. Create the host_vars directory:

      [ansible@admin ~]$ mkdir /usr/share/ceph-ansible/host_vars

   2. Change to the host_vars directory:

      [ansible@admin ~]$ cd /usr/share/ceph-ansible/host_vars

   3. Create the host files. Use the host-name-short-name format for the name of the files, for example:

      [ansible@admin host_vars]$ touch tower-osd6

   4. Update the file with any host specific parameters, for example:

      1. In bare-metal deployments use the devices parameter to specify devices that the OSD nodes will use. Using devices is useful when OSDs use devices with different names or when one of the devices failed on one of the OSDs.

         devices:
             DEVICE_1
             DEVICE_2

         Example

         devices:
             /dev/sdb
             /dev/sdc

         Note

         When specifying no devices, set the osd_auto_discovery parameter to true in the group_vars/osds.yml file.

8. Optionally, for all deployments, bare-metal or in containers, you can create a custom CRUSH hierarchy using Ceph Ansible:

   1. Setup your Ansible inventory file. Specify where you want the OSD hosts to be in the CRUSH map’s hierarchy by using the osd_crush_location parameter. You must specify at least two CRUSH bucket types to specify the location of the OSD, and one bucket type must be host. By default, these include root, datacenter, room, row, pod, pdu, rack, chassis and host.

      Syntax

      [osds]
      CEPH_OSD_NAME osd_crush_location="{ 'root': ROOT_BUCKET_', 'rack': 'RACK_BUCKET', 'pod': 'POD_BUCKET', 'host': 'CEPH_HOST_NAME' }"

      Example

      [osds]
      ceph-osd-01 osd_crush_location="{ 'root': 'default', 'rack': 'rack1', 'pod': 'monpod', 'host': 'ceph-osd-01' }"

   2. Edit the group_vars/osds.yml file, and set the crush_rule_config and create_crush_tree parameters to True. Create at least one CRUSH rule if you do not want to use the default CRUSH rules, for example:

      crush_rule_config: True
      crush_rule_hdd:
          name: replicated_hdd_rule
          root: root-hdd
          type: host
          class: hdd
          default: True
      crush_rules:
        - "{{ crush_rule_hdd }}"
      create_crush_tree: True

      If you are using faster SSD devices, then edit the parameters as follows:

      crush_rule_config: True
      crush_rule_ssd:
          name: replicated_ssd_rule
          root: root-ssd
          type: host
          class: ssd
          default: True
      crush_rules:
        - "{{ crush_rule_ssd }}"
      create_crush_tree: True

      Note

      The default CRUSH rules fail if both ssd and hdd OSDs are not deployed because the default rules now include the class parameter, which must be defined.

   3. Create pools, with created crush_rules in group_vars/clients.yml file:

      Example

      copy_admin_key: True
      user_config: True
      pool1:
        name: "pool1"
        pg_num: 128
        pgp_num: 128
        rule_name: "HDD"
        type: "replicated"
        device_class: "hdd"
      pools:
        - "{{ pool1 }}"

   4. View the tree:

      [root@mon ~]# ceph osd tree

   5. Validate the pools:

      [root@mon ~]# for i in $(rados lspools); do echo "pool: $i"; ceph osd pool get $i crush_rule; done
      
      pool: pool1
      crush_rule: HDD

9. For all deployments, bare-metal or in containers, log in with or switch to the ansible user.

   1. Create the ceph-ansible-keys directory where Ansible stores temporary values generated by the ceph-ansible playbook:

      [ansible@admin ~]$ mkdir ~/ceph-ansible-keys

   2. Change to the /usr/share/ceph-ansible/ directory:

      [ansible@admin ~]$ cd /usr/share/ceph-ansible/

   3. Verify that Ansible can reach the Ceph nodes:

      [ansible@admin ceph-ansible]$ ansible all -m ping -i hosts

10. Run the ceph-ansible playbook.

    1. Bare-metal deployments:

       [ansible@admin ceph-ansible]$ ansible-playbook site.yml -i hosts

    2. Container deployments:

       [ansible@admin ceph-ansible]$ ansible-playbook site-container.yml -i hosts

       Note

       If you deploy Red Hat Ceph Storage to Red Hat Enterprise Linux Atomic Host hosts, use the --skip-tags=with_pkg option:

       [user@admin ceph-ansible]$ ansible-playbook site-container.yml --skip-tags=with_pkg -i hosts

       Note

       To increase the deployment speed, use the --forks option to ansible-playbook. By default, ceph-ansible sets forks to 20. With this setting, up to twenty nodes will be installed at the same time. To install up to thirty nodes at a time, run ansible-playbook --forks 30 PLAYBOOK FILE -i hosts. The resources on the admin node must be monitored to ensure they are not overused. If they are, lower the number passed to --forks.

11. Wait for the Ceph deployment to finish.

    Example output

    INSTALLER STATUS *******************************
    Install Ceph Monitor           : Complete (0:00:30)
    Install Ceph Manager           : Complete (0:00:47)
    Install Ceph OSD               : Complete (0:00:58)
    Install Ceph RGW               : Complete (0:00:34)
    Install Ceph Dashboard         : Complete (0:00:58)
    Install Ceph Grafana           : Complete (0:00:50)
    Install Ceph Node Exporter     : Complete (0:01:14)

12. Verify the status of the Ceph storage cluster.

    1. Bare-metal deployments:

       [root@mon ~]# ceph health
       HEALTH_OK

    2. Container deployments:

       Red Hat Enterprise Linux 7

       [root@mon ~]# docker exec ceph-mon-ID ceph health

       Red Hat Enterprise Linux 8

       [root@mon ~]# podman exec ceph-mon-ID ceph health

       Replace

       • ID with the host name of the Ceph Monitor node:

         Example

         [root@mon ~]# podman exec ceph-mon-mon0 ceph health
         HEALTH_OK

13. For all deployments, bare-metal or in containers, verify the storage cluster is functioning using rados.

    1. From a Ceph Monitor node, create a test pool with eight placement groups (PG):

       Syntax

       [root@mon ~]# ceph osd pool create POOL_NAME PG_NUMBER

       Example

       [root@mon ~]# ceph osd pool create test 8

    2. Create a file called hello-world.txt:

       Syntax

       [root@mon ~]# vim FILE_NAME

       Example

       [root@mon ~]# vim hello-world.txt

    3. Upload hello-world.txt to the test pool using the object name hello-world:

       Syntax

       [root@mon ~]# rados --pool POOL_NAME put OBJECT_NAME OBJECT_FILE_NAME

       Example

       [root@mon ~]# rados --pool test put hello-world hello-world.txt

    4. Download hello-world from the test pool as file name fetch.txt:

       Syntax

       [root@mon ~]# rados --pool POOL_NAME get OBJECT_NAME OBJECT_FILE_NAME

       Example

       [root@mon ~]# rados --pool test get hello-world fetch.txt

    5. Check the contents of fetch.txt:

       [root@mon ~]# cat fetch.txt
       "Hello World!"

       Note

       In addition to verifying the storage cluster status, you can use the ceph-medic utility to overall diagnose the Ceph Storage cluster. See the Installing and Using ceph-medic to Diagnose a Ceph Storage Cluster chapter in the Red Hat Ceph Storage 4 Troubleshooting Guide.

Procedure

1. Set osds_per_device: 2 in group_vars/osds.yml:

   osds_per_device: 2

2. List the NVMe devices under devices:

   devices:
     - /dev/nvme0n1
     - /dev/nvme1n1
     - /dev/nvme2n1
     - /dev/nvme3n1

3. The settings in group_vars/osds.yml will look similar to this example:

   osds_per_device: 2
   devices:
     - /dev/nvme0n1
     - /dev/nvme1n1
     - /dev/nvme2n1
     - /dev/nvme3n1

1. Add a new section [mdss] to the /etc/ansible/hosts file:

   [mdss]
   MDS_NODE_NAME1
   MDS_NODE_NAME2
   MDS_NODE_NAME3

   Replace MDS_NODE_NAME with the host names of the nodes where you want to install the Ceph Metadata servers.

   Alternatively, you can colocate the Metadata server with the OSD daemon on one node by adding the same node under the [osds] and [mdss] sections.

2. Navigate to the /usr/share/ceph-ansible directory:

   [root@admin ~]# cd /usr/share/ceph-ansible

3. Optionally, you can change the default variables.

   1. Create a copy of the group_vars/mdss.yml.sample file named mdss.yml:

      [root@admin ceph-ansible]# cp group_vars/mdss.yml.sample group_vars/mdss.yml

   2. Optionally, edit the parameters in mdss.yml. See mdss.yml for details.

4. As the ansible user, run the Ansible playbook:

   • Bare-metal deployments:

     [user@admin ceph-ansible]$ ansible-playbook site.yml --limit mdss -i hosts

   • Container deployments:

     [ansible@admin ceph-ansible]$ ansible-playbook site-container.yml --limit mdss -i hosts

5. After installing the Metadata servers, you can now configure them. For details, see the The Ceph File System Metadata Server chapter in the Red Hat Ceph Storage File System Guide.

1. Add a new section [clients] to the /etc/ansible/hosts file:

   [clients]
   CLIENT_NODE_NAME

   Replace CLIENT_NODE_NAME with the host name of the node where you want to install the ceph-client role.

2. Navigate to the /usr/share/ceph-ansible directory:

   [root@admin ~]# cd /usr/share/ceph-ansible

3. Create a new copy of the clients.yml.sample file named clients.yml:

   [root@admin ceph-ansible ~]# cp group_vars/clients.yml.sample group_vars/clients.yml

4. Open the group_vars/clients.yml file, and uncomment the following lines:

   keys:
     - { name: client.test, caps: { mon: "allow r", osd: "allow class-read object_prefix rbd_children, allow rwx pool=test" },  mode: "{{ ceph_keyring_permissions }}" }

   1. Replace client.test with the real client name, and add the client key to the client definition line, for example:

      key: "ADD-KEYRING-HERE=="

      Now the whole line example would look similar to this:

      - { name: client.test, key: "AQAin8tUMICVFBAALRHNrV0Z4MXupRw4v9JQ6Q==", caps: { mon: "allow r", osd: "allow class-read object_prefix rbd_children, allow rwx pool=test" },  mode: "{{ ceph_keyring_permissions }}" }

      Note

      The ceph-authtool --gen-print-key command can generate a new client key.

5. Optionally, instruct ceph-client to create pools and clients.

   1. Update clients.yml.

      • Uncomment the user_config setting and set it to true.
      • Uncomment the pools and keys sections and update them as required. You can define custom pools and client names altogether with the cephx capabilities.

   2. Add the osd_pool_default_pg_num setting to the ceph_conf_overrides section in the all.yml file:

      ceph_conf_overrides:
         global:
            osd_pool_default_pg_num: NUMBER

      Replace NUMBER with the default number of placement groups.

6. As the ansible user, run the Ansible playbook:

   1. Bare-metal deployments:

      [ansible@admin ceph-ansible]$ ansible-playbook site.yml --limit clients -i hosts

   2. Container deployments:

      [ansible@admin ceph-ansible]$ ansible-playbook site-container.yml --limit clients -i hosts

1. Add gateway hosts to the /etc/ansible/hosts file under the [rgws] section to identify their roles to Ansible. If the hosts have sequential naming, use a range, for example:

   [rgws]
   <rgw_host_name_1>
   <rgw_host_name_2>
   <rgw_host_name[3..10]>

2. Navigate to the Ansible configuration directory:

   [root@ansible ~]# cd /usr/share/ceph-ansible

3. Create the rgws.yml file from the sample file:

   [root@ansible ~]# cp group_vars/rgws.yml.sample group_vars/rgws.yml

4. Open and edit the group_vars/rgws.yml file. To copy the administrator key to the Ceph Object Gateway node, uncomment the copy_admin_key option:

   copy_admin_key: true

5. In the all.yml file, you MUST specify a radosgw_interface.

   radosgw_interface: <interface>

   Replace:

   • <interface> with the interface that the Ceph Object Gateway nodes listen to

   For example:

   radosgw_interface: eth0

   Specifying the interface prevents Civetweb from binding to the same IP address as another Civetweb instance when running multiple instances on the same host.

   For additional details, see the all.yml file.

6. Generally, to change default settings, uncomment the settings in the rgws.yml file, and make changes accordingly. To make additional changes to settings that are not in the rgws.yml file, use ceph_conf_overrides: in the all.yml file.

   ceph_conf_overrides:
       client.rgw.rgw1:
         rgw_override_bucket_index_max_shards: 16
         rgw_bucket_default_quota_max_objects: 1638400

   For advanced configuration details, see the Red Hat Ceph Storage 4 Ceph Object Gateway for Production guide. Advanced topics include:

   • Configuring Ansible Groups

   • Developing Storage Strategies. See the Creating the Root Pool, Creating System Pools, and Creating Data Placement Strategies sections for additional details on how create and configure the pools.

     See Bucket Sharding for configuration details on bucket sharding.

7. Run the Ansible playbook:

   Warning

   Do not run the Ansible playbook if you intend to set up multisite. Proceed to the Configuring multisite Ceph Object Gateways section to set up multisite.

   1. Bare-metal deployments:

      [user@admin ceph-ansible]$ ansible-playbook site.yml --limit rgws -i hosts

   2. Container deployments:

      [user@admin ceph-ansible]$ ansible-playbook site-container.yml --limit rgws -i hosts

Procedure

1. Create crush_rules in the group_vars/mons.yml file:

   Example

   crush_rule_config: true
   crush_rule_hdd:
       name: HDD
       root: default
       type: host
       class: hdd
       default: true
   crush_rule_ssd:
       name: SSD
       root: default
       type: host
       class: ssd
       default: true
   crush_rules:
         - "{{ crush_rule_hdd }}"
         - "{{ crush_rule_ssd }}"
   create_crush_tree: true

   Note

   If you are not using SSD or HDD devices in the cluster, do not define the crush_rules for that device.

2. Create pools, with created crush_rules in group_vars/clients.yml file.

   Example

   copy_admin_key: True
   user_config: True
   pool1:
     name: "pool1"
     pg_num: 128
     pgp_num: 128
     rule_name: "HDD"
     type: "replicated"
     device_class: "hdd"
   pools:
     - "{{ pool1 }}"

3. Sample the inventory file to assign roots to OSDs:

   Example

   [mons]
   mon1
   
   [osds]
   osd1 osd_crush_location="{ 'root': 'default', 'rack': 'rack1', 'host': 'osd1' }"
   osd2 osd_crush_location="{ 'root': 'default', 'rack': 'rack1', 'host': 'osd2' }"
   osd3 osd_crush_location="{ 'root': 'default', 'rack': 'rack2', 'host': 'osd3' }"
   osd4 osd_crush_location="{ 'root': 'default', 'rack': 'rack2', 'host': 'osd4' }"
   osd5 devices="['/dev/sda', '/dev/sdb']" osd_crush_location="{ 'root': 'default', 'rack': 'rack3', 'host': 'osd5' }"
   osd6 devices="['/dev/sda', '/dev/sdb']" osd_crush_location="{ 'root': 'default', 'rack': 'rack3', 'host': 'osd6' }"
   
   [mgrs]
   mgr1
   
   [clients]
   client1

4. View the tree.

   Syntax

   [root@mon ~]# ceph osd tree

   Example

   TYPE NAME
   
   root default
        rack rack1
           host osd1
                osd.0
                osd.10
           host osd2
                osd.3
                osd.7
                osd.12
        rack rack2
           host osd3
                osd.1
                osd.6
                osd.11
           host osd4
                osd.4
                osd.9
                osd.13
        rack rack3
            host osd5
                osd.2
                osd.8
            host osd6
                osd.14
                osd.15

5. Validate the pools.

   Example

   # for i in $(rados lspools);do echo "pool: $i"; ceph osd pool get $i crush_rule;done
   
   pool: pool1
   crush_rule: HDD

1. Create the nfss.yml file from the sample file:

   [root@ansible ~]# cd /usr/share/ceph-ansible/group_vars
   [root@ansible ~]# cp nfss.yml.sample nfss.yml

2. Add gateway hosts to the /etc/ansible/hosts file under an [nfss] group to identify their group membership to Ansible.

   [nfss]
   NFS_HOST_NAME_1
   NFS_HOST_NAME_2
   NFS_HOST_NAME[3..10]

   If the hosts have sequential naming, then you can use a range specifier, for example: [3..10].

3. Navigate to the Ansible configuration directory:

   [root@ansible ~]# cd /usr/share/ceph-ansible

4. To copy the administrator key to the Ceph Object Gateway node, uncomment the copy_admin_key setting in the /usr/share/ceph-ansible/group_vars/nfss.yml file:

   copy_admin_key: true

5. Configure the FSAL (File System Abstraction Layer) sections of the /usr/share/ceph-ansible/group_vars/nfss.yml file. Provide an export ID (NUMERIC_EXPORT_ID), S3 user ID (S3_USER), S3 access key (ACCESS_KEY) and secret key (SECRET_KEY):

   # FSAL RGW Config #
   
   ceph_nfs_rgw_export_id: NUMERIC_EXPORT_ID
   #ceph_nfs_rgw_pseudo_path: "/"
   #ceph_nfs_rgw_protocols: "3,4"
   #ceph_nfs_rgw_access_type: "RW"
   ceph_nfs_rgw_user: "S3_USER"
   ceph_nfs_rgw_access_key: "ACCESS_KEY"
   ceph_nfs_rgw_secret_key: "SECRET_KEY"

   Warning

   Access and secret keys are optional, and can be generated.

6. Run the Ansible playbook:

   1. Bare-metal deployments:

      [ansible@admin ceph-ansible]$ ansible-playbook site.yml --limit nfss -i hosts

   2. Container deployments:

      [ansible@admin ceph-ansible]$ ansible-playbook site-container.yml --limit nfss -i hosts

Procedure

1. To change the default CPU limit for a daemon, set the ceph_daemon-type_docker_cpu_limit parameter in the appropriate .yml configuration file when deploying the daemon. See the following table for details.

   Daemon	Parameter	Configuration file
   OSD	ceph_osd_docker_cpu_limit	osds.yml
   MDS	ceph_mds_docker_cpu_limit	mdss.yml
   RGW	ceph_rgw_docker_cpu_limit	rgws.yml

   For example, to change the default CPU limit to 2 for the Ceph Object Gateway, edit the /usr/share/ceph-ansible/group_vars/rgws.yml file as follows:

   ceph_rgw_docker_cpu_limit: 2

2. To change the default RAM for OSD daemons, set the osd_memory_target in the /usr/share/ceph-ansible/group_vars/all.yml file when deploying the daemon. For example, to limit the OSD RAM to 6 GB:

   ceph_conf_overrides:
     osd:
       osd_memory_target=6000000000

   Important

   In an hyperconverged infrastructure (HCI) configuration, you can also use the ceph_osd_docker_memory_limit parameter in the osds.yml configuration file to change the Docker memory CGroup limit. In this case, set ceph_osd_docker_memory_limit to 50% higher than osd_memory_target, so that the CGroup limit is more constraining than it is by default for an HCI configuration. For example, if osd_memory_target is set to 6 GB, set ceph_osd_docker_memory_limit to 9 GB:

   ceph_osd_docker_memory_limit: 9g

Procedure

1. Log in as the root user on all nodes in the storage cluster.
2. If the Ceph nodes are not connected to the Red Hat Content Delivery Network (CDN), you can use an ISO image to upgrade Red Hat Ceph Storage by updating the local repository with the latest version of Red Hat Ceph Storage.

3. If upgrading Red Hat Ceph Storage from version 3 to version 4, remove an existing Ceph dashboard installation.

   1. On the Ansible administration node, change to the cephmetrics-ansible directory:

      [root@admin ~]# cd /usr/share/cephmetrics-ansible

   2. Run the purge.yml playbook to remove an existing Ceph dashboard installation:

      [root@admin cephmetrics-ansible]# ansible-playbook -v purge.yml

4. If upgrading Red Hat Ceph Storage from version 3 to version 4, enable the Ceph and Ansible repositories on the Ansible administration node:

   Red Hat Enterprise Linux 7

   [root@admin ~]# subscription-manager repos --enable=rhel-7-server-rhceph-4-tools-rpms --enable=rhel-7-server-ansible-2.9-rpms

   Red Hat Enterprise Linux 8

   [root@admin ~]# subscription-manager repos --enable=rhceph-4-tools-for-rhel-8-x86_64-rpms --enable=ansible-2.9-for-rhel-8-x86_64-rpms

5. On the Ansible administration node, ensure the latest versions of the ansible and ceph-ansible packages are installed.

   Red Hat Enterprise Linux 7

   [root@admin ~]# yum update ansible ceph-ansible

   Red Hat Enterprise Linux 8

   [root@admin ~]# dnf update ansible ceph-ansible

6. Edit the infrastructure-playbooks/rolling_update.yml playbook and change the health_osd_check_retries and health_osd_check_delay values to 50 and 30 respectively:

   health_osd_check_retries: 50
   health_osd_check_delay: 30

   For each OSD node, these values cause Ansible to wait for up to 25 minutes, and will check the storage cluster health every 30 seconds, waiting before continuing the upgrade process.

   Note

   Adjust the health_osd_check_retries option value up or down based on the used storage capacity of the storage cluster. For example, if you are using 218 TB out of 436 TB, basically using 50% of the storage capacity, then set the health_osd_check_retries option to 50.

7. If the storage cluster you want to upgrade contains Ceph Block Device images that use the exclusive-lock feature, ensure that all Ceph Block Device users have permissions to blacklist clients:

   ceph auth caps client.ID mon 'allow r, allow command "osd blacklist"' osd 'EXISTING_OSD_USER_CAPS'

8. If the storage cluster was originally installed using Cockpit, create a symbolic link in the /usr/share/ceph-ansible directory to the inventory file where Cockpit created it, at /usr/share/ansible-runner-service/inventory/hosts:

   1. Change to the /usr/share/ceph-ansible directory:

      # cd /usr/share/ceph-ansible

   2. Create the symbolic link:

      # ln -s /usr/share/ansible-runner-service/inventory/hosts hosts

9. To upgrade the cluster using ceph-ansible, create the symbolic link in the etc/ansible/hosts directory to the hosts inventory file:

   # ln -s /etc/ansible/hosts hosts

10. If the storage cluster was originally installed using Cockpit, copy the Cockpit generated SSH keys to the Ansible user’s ~/.ssh directory:

    1. Copy the keys:

       # cp /usr/share/ansible-runner-service/env/ssh_key.pub /home/ANSIBLE_USERNAME/.ssh/id_rsa.pub
       # cp /usr/share/ansible-runner-service/env/ssh_key /home/ANSIBLE_USERNAME/.ssh/id_rsa

       Replace ANSIBLE_USERNAME with the username for Ansible, usually admin.

       Example

       # cp /usr/share/ansible-runner-service/env/ssh_key.pub /home/admin/.ssh/id_rsa.pub
       # cp /usr/share/ansible-runner-service/env/ssh_key /home/admin/.ssh/id_rsa

    2. Set the appropriate owner, group, and permissions on the key files:

       # chown ANSIBLE_USERNAME:_ANSIBLE_USERNAME_ /home/ANSIBLE_USERNAME/.ssh/id_rsa.pub
       # chown ANSIBLE_USERNAME:_ANSIBLE_USERNAME_ /home/ANSIBLE_USERNAME/.ssh/id_rsa
       # chmod 644 /home/ANSIBLE_USERNAME/.ssh/id_rsa.pub
       # chmod 600 /home/ANSIBLE_USERNAME/.ssh/id_rsa

       Replace ANSIBLE_USERNAME with the username for Ansible, usually admin.

       Example

       # chown admin:admin /home/admin/.ssh/id_rsa.pub
       # chown admin:admin /home/admin/.ssh/id_rsa
       # chmod 644 /home/admin/.ssh/id_rsa.pub
       # chmod 600 /home/admin/.ssh/id_rsa

Procedure

1. Navigate to the /usr/share/ceph-ansible/ directory:

   Example

   [root@admin ~]# cd /usr/share/ceph-ansible/

2. If upgrading from Red Hat Ceph Storage 3 to Red Hat Ceph Storage 4, make backup copies of the group_vars/all.yml, group_vars/osds.yml, and group_vars/clients.yml files:

   [root@admin ceph-ansible]# cp group_vars/all.yml group_vars/all_old.yml
   [root@admin ceph-ansible]# cp group_vars/osds.yml group_vars/osds_old.yml
   [root@admin ceph-ansible]# cp group_vars/clients.yml group_vars/clients_old.yml

3. If upgrading from Red Hat Ceph Storage 3 to Red Hat Ceph Storage 4, create new copies of the group_vars/all.yml.sample, group_vars/osds.yml.sample and group_vars/clients.yml.sample files, and rename them to group_vars/all.yml, group_vars/osds.yml, and group_vars/clients.yml respectively. Open and edit them accordingly, basing the changes on your previously backed up copies.

   [root@admin ceph-ansible]# cp group_vars/all.yml.sample group_vars/all.yml
   [root@admin ceph-ansible]# cp group_vars/osds.yml.sample group_vars/osds.yml
   [root@admin ceph-ansible]# cp group_vars/clients.yml.sample group_vars/clients.yml

4. Edit the group_vars/osds.yml file. Add and set the following options:

   nb_retry_wait_osd_up: 60
   delay_wait_osd_up: 10

   Note

   These are the default values; you can modify the values as per your use case.

5. If upgrading to a new minor version of Red Hat Ceph Storage 4, verify the value for grafana_container_image in group_vars/all.yml is the same as in group_vars/all.yml.sample. If it is not the same, edit it so it is.

   Example

   grafana_container_image: registry.redhat.io/rhceph/rhceph-4-dashboard-rhel8:4

   Note

   The image path shown is included in ceph-ansible version 4.0.23-1.

6. Copy the latest site.yml or site-container.yml file from the sample files:

   1. For bare-metal deployments:

      [root@admin ceph-ansible]# cp site.yml.sample site.yml

   2. For container deployments:

      [root@admin ceph-ansible]# cp site-container.yml.sample site-container.yml

7. Open the group_vars/all.yml file and edit the following options.

   1. Add the fetch_directory option:

      fetch_directory: FULL_DIRECTORY_PATH

      Replace

      • FULL_DIRECTORY_PATH with a writable location, such as the Ansible user’s home directory.

   2. If the cluster you want to upgrade contains any Ceph Object Gateway nodes, add the radosgw_interface option:

      radosgw_interface: INTERFACE

      Replace

      • INTERFACE with the interface that the Ceph Object Gateway nodes listen to.

   3. If your current setup has SSL certificates configured, you need to edit the following:

      radosgw_frontend_ssl_certificate: /etc/pki/ca-trust/extracted/CERTIFICATE_NAME
      radosgw_frontend_port: 443

   4. The default OSD object store is BlueStore. To keep the traditional OSD object store, you must explicitly set the osd_objectstore option to filestore:

      osd_objectstore: filestore

      Note

      With the osd_objectstore option set to filestore, replacing an OSD will use FileStore, instead of BlueStore.

      Important

      Starting with Red Hat Ceph Storage 4, FileStore is a deprecated feature. Red Hat recommends migrating the FileStore OSDs to BlueStore OSDs.

   5. Starting with Red Hat Ceph Storage 4.1, you must uncomment or set dashboard_admin_password and grafana_admin_password in /usr/share/ceph-ansible/group_vars/all.yml. Set secure passwords for each. Also set custom user names for dashboard_admin_user and grafana_admin_user.

   6. For both bare-metal and containers deployments:

      1. Uncomment the upgrade_ceph_packages option and set it to True:

         upgrade_ceph_packages: True

      2. Set the ceph_rhcs_version option to 4:

         ceph_rhcs_version: 4

         Note

         Setting the ceph_rhcs_version option to 4 will pull in the latest version of Red Hat Ceph Storage 4.

      3. Add the ceph_docker_registry information to all.yml:

         Syntax

         ceph_docker_registry: registry.redhat.io
         ceph_docker_registry_username: SERVICE_ACCOUNT_USER_NAME
         ceph_docker_registry_password: TOKEN

         Note

         If you do not have a Red Hat Registry Service Account, create one using the Registry Service Account webpage. See the Red Hat Container Registry Authentication Knowledgebase article for more details.

         Note

         In addition to using a Service Account for the ceph_docker_registry_username and ceph_docker_registry_password parameters, you can also use your Customer Portal credentials, but to ensure security, encrypt the ceph_docker_registry_password parameter. For more information, see Encrypting Ansible password variables with ansible-vault.

   7. For containers deployments:

      1. Change the ceph_docker_image option to point to the Ceph 4 container version:

         ceph_docker_image: rhceph/rhceph-4-rhel8

      2. Change the ceph_docker_image_tag option to point to the latest version of rhceph/rhceph-4-rhel8:

         ceph_docker_image_tag: latest

8. If upgrading from Red Hat Ceph Storage 3 to Red Hat Ceph Storage 4, open the Ansible inventory file for editing, /etc/ansible/hosts by default, and add the Ceph dashboard node name or IP address under the [grafana-server] section. If this section does not exist, then also add this section along with the node name or IP address.

9. Switch to or log in as the Ansible user, then run the rolling_update.yml playbook:

   [ansible@admin ceph-ansible]$ ansible-playbook infrastructure-playbooks/rolling_update.yml -i hosts

   Important

   Using the --limit Ansible option with the rolling_update.yml playbook is not supported.

10. As the root user on the RBD mirroring daemon node, upgrade the rbd-mirror package manually:

    [root@rbd ~]# yum upgrade rbd-mirror

11. Restart the rbd-mirror daemon:

    systemctl restart ceph-rbd-mirror@CLIENT_ID

12. Verify the health status of the storage cluster.

    1. For bare-metal deployments, log into a monitor node as the root user and run the Ceph status command:

       [root@mon ~]# ceph -s

    2. For container deployments, log into a Ceph Monitor node as the root user.

       1. List all running containers:

          Red Hat Enterprise Linux 7

          [root@mon ~]# docker ps

          Red Hat Enterprise Linux 8

          [root@mon ~]# podman ps

       2. Check health status:

          Red Hat Enterprise Linux 7

          [root@mon ~]# docker exec ceph-mon-MONITOR_NAME ceph -s

          Red Hat Enterprise Linux 8

          [root@mon ~]# podman exec ceph-mon-MONITOR_NAME ceph -s

          Replace

          • MONITOR_NAME with the name of the Ceph Monitor container found in the previous step.

            Example

            [root@mon ~]# podman exec ceph-mon-mon01 ceph -s

13. Optional: If upgrading from Red Hat Ceph Storage 3.x to Red Hat Ceph Storage 4.x, you might see this health warning: Legacy BlueStore stats reporting detected on 336 OSD(s). This is caused by newer code calculating pool stats differently. You can resolve this by setting the bluestore_fsck_quick_fix_on_mount parameter.

    1. Set bluestore_fsck_quick_fix_on_mount to true:

       Example

       [root@mon ~]# ceph config set osd bluestore_fsck_quick_fix_on_mount true

    2. Set the noout and norebalance flags to prevent data movement while OSDs are down:

       Example

       [root@mon ~]# ceph osd set noout
       [root@mon ~]# ceph osd set norebalance

    3. For bare-metal deployment, restart ceph-osd.target on every OSD node of the storage cluster:

       Example

       [root@osd ~]# systemctl restart ceph-osd.target

    4. For containerized deployment, restart the individual OSDs one after the other and wait for all the placement groups to be in active+clean state.

       Syntax

       systemctl restart ceph-osd@OSD_ID.service

       Example

       [root@osd ~]# systemctl restart ceph-osd@0.service

    5. When all the OSDs are repaired, unset the nout and norebalance flags:

       Example

       [root@mon ~]# ceph osd unset noout
       [root@mon ~]# ceph osd unset norebalance

    6. Set the bluestore_fsck_quick_fix_on_mount to false once all the OSDs are repaired:

       Example

       [root@mon ~]# ceph config set osd bluestore_fsck_quick_fix_on_mount false

    7. Optional: An alternate method for bare-metal deployment is to stop the OSD service, run the repair function on the OSD using the ceph-bluestore-tool command, and then start the OSD service:

       1. Stop the OSD service:

          [root@osd ~]# systemctl stop ceph-osd.target

       2. Run the repair function on the OSD, specifying its actual OSD ID:

          Syntax

          ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-OSDID repair

          Example

          [root@osd ~]# ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-2 repair

       3. Start the OSD service:

          [root@osd ~]# systemctl start ceph-osd.target

14. Once the upgrade finishes, you can migrate the FileStore OSDs to BlueStore OSDs, by running the Ansible playbook:

    Syntax

    ansible-playbook infrastructure-playbooks/filestore-to-bluestore.yml --limit OSD_NODE_TO_MIGRATE

    Example

    [ansible@admin ceph-ansible]$ ansible-playbook infrastructure-playbooks/filestore-to-bluestore.yml --limit osd01

    Once the migration completes do the following sub steps.

    1. Open for editing the group_vars/osds.yml file, and set the osd_objectstore option to bluestore, for example:

       osd_objectstore: bluestore

    2. If you are using the lvm_volumes variable, then change the journal and journal_vg options to db and db_vg respectively, for example:

       Before

       lvm_volumes:
         - data: /dev/sdb
           journal: /dev/sdc1
         - data: /dev/sdd
           journal: journal1
           journal_vg: journals

       After converting to Bluestore

       lvm_volumes:
         - data: /dev/sdb
           db: /dev/sdc1
         - data: /dev/sdd
           db: journal1
           db_vg: journals

15. If working in an OpenStack environment, update all the cephx users to use the RBD profile for pools. The following commands must be run as the root user:

    1. Glance users:

       Syntax

       ceph auth caps client.glance mon 'profile rbd' osd 'profile rbd pool=GLANCE_POOL_NAME'

       Example

       [root@mon ~]# ceph auth caps client.glance mon 'profile rbd' osd 'profile rbd pool=images'

    2. Cinder users:

       Syntax

       ceph auth caps client.cinder mon 'profile rbd' osd 'profile rbd pool=CINDER_VOLUME_POOL_NAME, profile rbd pool=NOVA_POOL_NAME, profile rbd-read-only pool=GLANCE_POOL_NAME'

       Example

       [root@mon ~]# ceph auth caps client.cinder mon 'profile rbd' osd 'profile rbd pool=volumes, profile rbd pool=vms, profile rbd-read-only pool=images'

    3. OpenStack general users:

       Syntax

       ceph auth caps client.openstack mon 'profile rbd' osd 'profile rbd-read-only pool=CINDER_VOLUME_POOL_NAME, profile rbd pool=NOVA_POOL_NAME, profile rbd-read-only pool=GLANCE_POOL_NAME'

       Example

       [root@mon ~]# ceph auth caps client.openstack mon 'profile rbd' osd 'profile rbd-read-only pool=volumes, profile rbd pool=vms, profile rbd-read-only pool=images'

       Important

       Do these CAPS updates before performing any live client migrations. This allows clients to use the new libraries running in memory, causing the old CAPS settings to drop from cache and applying the new RBD profile settings.

16. Optional: On client nodes, restart any applications that depend on the Ceph client-side libraries.

    Note

    If you are upgrading OpenStack Nova compute nodes that have running QEMU or KVM instances or use a dedicated QEMU or KVM client, stop and start the QEMU or KVM instance because restarting the instance does not work in this case.

Procedure

1. Ensure that the cluster has completed at least one full scrub of all PGs while running Red Hat Ceph Storage 3. Failure to do so can cause your monitor daemons to refuse to join the quorum on start, leaving them non-functional. To ensure the cluster has completed at least one full scrub of all PGs, execute the following:

   # ceph osd dump | grep ^flags

   To proceed with an upgrade from Red Hat Ceph Storage 3 to Red Hat Ceph Storage 4, the OSD map must include the recovery_deletes and purged_snapdirs flags.

2. Ensure the cluster is in a healthy and clean state.

   ceph health
   HEALTH_OK

3. For nodes running ceph-mon and ceph-manager, execute:

   # subscription-manager repos --enable=rhel-7-server-rhceph-4-mon-rpms

   Once the Red Hat Ceph Storage 4 package is enabled, execute the following on each of the ceph-mon and ceph-manager nodes:

   # firewall-cmd --add-port=3300/tcp
   # firewall-cmd --add-port=3300/tcp --permanent
   # yum update -y
   # systemctl restart ceph-mon@<mon-hostname>
   # systemctl restart ceph-mgr@<mgr-hostname>

   Replace <mon-hostname> and <mgr-hostname> with the hostname of the target host.

4. Before upgrading OSDs, set the noout and nodeep-scrub flags on a Ceph Monitor node to prevent OSDs from rebalancing during upgrade.

   # ceph osd set noout
   # ceph osd det nodeep-scrub

5. On each OSD node, execute:

   # subscription-manager repos --enable=rhel-7-server-rhceph-4-osd-rpms

   Once the Red Hat Ceph Storage 4 package is enabled, update the OSD node:

   # yum update -y

   For each OSD daemon running on the node, execute:

   # systemctl restart ceph-osd@<osd-num>

   Replace <osd-num> with the osd number to restart. Ensure all OSDs on the node have restarted before proceeding to the next OSD node.

6. If there are any OSDs in the storage cluster deployed with ceph-disk, instruct ceph-volume to start the daemons.

   # ceph-volume simple scan
   # ceph-volume simple activate --all

7. Enable the Nautilus only functionality:

   # ceph osd require-osd-release nautilus

   Important

   Failure to execute this step will make it impossible for OSDs to communicate after msgr2 is enabled.

8. After upgrading all OSD nodes, unset the noout and nodeep-scrub flags on a Ceph Monitor node.

   # ceph osd unset noout
   # ceph osd unset nodeep-scrub

9. Switch any existing CRUSH buckets to the latest bucket type straw2.

   # ceph osd getcrushmap -o backup-crushmap
   # ceph osd crush set-all-straw-buckets-to-straw2

10. Once all the daemons are updated after upgrading from Red Hat Ceph Storage 3 to Red Hat Ceph Storage 4, run the following steps:

    1. Enable the messenger v2 protocol, msgr2:

       ceph mon enable-msgr2

       This instructs all Ceph Monitors that bind to the old default port of 6789, to also bind to the new port of 3300.

    2. Verify the status of the monitor:

       ceph mon dump

       Note

       Running nautilus OSDs does not bind to their v2 address automatically. They must be restarted.

11. For each host upgraded from Red Hat Ceph Storage 3 to Red Hat Ceph Storage 4, update the ceph.conf file to either not specify any monitor port or reference both the v2 and v1 addresses and ports.

12. Import any configuration options in ceph.conf file into the storage cluster’s configuration database.

    Example

    [root@mon ~]# ceph config assimilate-conf -i /etc/ceph/ceph.conf

    1. Check the storage cluster’s configuration database.

       Example

       [root@mon ~]# ceph config dump

    2. Optional: After upgrading to Red Hat Ceph Storage 4, create a minimal ceph.conf file for each host:

       Example

       [root@mon ~]# ceph config generate-minimal-conf > /etc/ceph/ceph.conf.new
       [root@mon ~]# mv /etc/ceph/ceph.conf.new /etc/ceph/ceph.conf

13. On Ceph Object Gateway nodes, execute:

    # subscription-manager repos --enable=rhel-7-server-rhceph-4-tools-rpms

    Once the Red Hat Ceph Storage 4 package is enabled, update the node and restart the ceph-rgw daemon:

    # yum update -y
    # systemctl restart ceph-rgw@<rgw-target>

    Replace <rgw-target> with the rgw target to restart.

14. For the administration node, execute:

    # subscription-manager repos --enable=rhel-7-server-rhceph-4-tools-rpms
    # yum update -y

15. Ensure the cluster is in a healthy and clean state.

    # ceph health
    HEALTH_OK

16. Optional: On client nodes, restart any applications that depend on the Ceph client-side libraries.

    Note

    If you are upgrading OpenStack Nova compute nodes that have running QEMU or KVM instances or use a dedicated QEMU or KVM client, stop and start the QEMU or KVM instance because restarting the instance does not work in this case.

Procedure

1. Reduce the number of active MDS ranks to 1:

   Syntax

   ceph fs set FILE_SYSTEM_NAME max_mds 1

   Example

   [root@mds ~]# ceph fs set fs1 max_mds 1

2. Wait for the cluster to stop all of the MDS ranks. When all of the MDS have stopped, only rank 0 should be active. The rest should be in standby mode. Check the status of the file system:

   [root@mds ~]# ceph status

3. Use systemctl to take all standby MDS offline:

   [root@mds ~]# systemctl stop ceph-mds.target

4. Confirm that only one MDS is online, and that it has rank 0 for your file system:

   [root@mds ~]# ceph status

5. If you are upgrading from Red Hat Ceph Storage 3 on RHEL 7, disable the Red Hat Ceph Storage 3 tools repository and enable the Red Hat Ceph Storage 4 tools repository:

   [root@mds ~]# subscription-manager repos --disable=rhel-7-server-rhceph-3-tools-rpms
   [root@mds ~]# subscription-manager repos --enable=rhel-7-server-rhceph-4-tools-rpms

6. Update the node and restart the ceph-mds daemon:

   [root@mds ~]# yum update -y
   [root@mds ~]# systemctl restart ceph-mds.target

7. Follow the same processes for the standby daemons. Disable and enable the tools repositories, and then upgrade and restart each standby MDS:

   [root@mds ~]# subscription-manager repos --disable=rhel-7-server-rhceph-3-tools-rpms
   [root@mds ~]# subscription-manager repos --enable=rhel-7-server-rhceph-4-tools-rpms
   [root@mds ~]# yum update -y
   [root@mds ~]# systemctl restart ceph-mds.target

8. When you have finished restarting all of the MDS in standby, restore the previous value of max_mds for the storage cluster:

   Syntax

   ceph fs set FILE_SYSTEM_NAME max_mds ORIGINAL_VALUE

   Example

   [root@mds ~]# ceph fs set fs1 max_mds 5

Procedure

1. Stop the monitor service:

   Syntax

   systemctl stop ceph-mon@MONITOR_ID

   Replace MONITOR_ID with the Monitor’s ID number.

2. If using Red Hat Ceph Storage 3, disable the Red Hat Ceph Storage 3 repositories.

   1. Disable the tools repository:

      [root@mon ~]# subscription-manager repos --disable=rhel-7-server-rhceph-3-tools-rpms

   2. Disable the mon repository:

      [root@mon ~]# subscription-manager repos --disable=rhel-7-server-rhceph-3-mon-rpms

3. If using Red Hat Ceph Storage 4, disable the Red Hat Ceph Storage 4 repositories.

   1. Disable the tools repository:

      [root@mon ~]# subscription-manager repos --disable=rhel-7-server-rhceph-4-tools-rpms

   2. Disable the mon repository:

      [root@mon ~]# subscription-manager repos --disable=rhel-7-server-rhceph-4-mon-rpms

4. Install the leapp utility. See Upgrading from Red Hat Enterprise Linux 7 to Red Hat Enterprise Linux 8.
5. Run through the leapp preupgrade checks. See Assessing upgradability from the command line.
6. Set PermitRootLogin yes in /etc/ssh/sshd_config.

7. Restart the OpenSSH SSH daemon:

   [root@mon ~]# systemctl restart sshd.service

8. Remove the iSCSI module from the Linux kernel:

   [root@mon ~]# modprobe -r iscsi

9. Perform the upgrade by following Performing the upgrade from RHEL 7 to RHEL 8.
10. Reboot the node.

11. Enable the repositories for Red Hat Ceph Storage 4 for Red Hat Enterprise Linux 8.

    1. Enable the tools repository:

       [root@mon ~]# subscription-manager repos --enable=rhceph-4-tools-for-rhel-8-x86_64-rpms

    2. Enable the mon repository:

       [root@mon ~]# subscription-manager repos --enable=rhceph-4-mon-for-rhel-8-x86_64-rpms

12. Install the ceph-mon package:

    [root@mon ~]# dnf install ceph-mon

13. If the manager service is colocated with the monitor service, install the ceph-mgr package:

    [root@mon ~]# dnf install ceph-mgr

14. Restore the ceph-client-admin.keyring and ceph.conf files from a Monitor node which has not been upgraded yet or from a node that has already had those files restored.

15. Switch any existing CRUSH buckets to the latest bucket type straw2.

    # ceph osd getcrushmap -o backup-crushmap
    # ceph osd crush set-all-straw-buckets-to-straw2

16. Once all the daemons are updated after upgrading from Red Hat Ceph Storage 3 to Red Hat Ceph Storage 4, run the following steps:

    1. Enable the messenger v2 protocol, msgr2:

       ceph mon enable-msgr2

       This instructs all Ceph Monitors that bind to the old default port of 6789, to also bind to the new port of 3300.

       Important

       Ensure all the Ceph Monitors are upgraded from Red Hat Ceph Storage 3 to Red Hat Ceph Storage 4 before performing any further Ceph Monitor configuration.

    2. Verify the status of the monitor:

       ceph mon dump

       Note

       Running nautilus OSDs does not bind to their v2 address automatically. They must be restarted.

17. For each host upgraded from Red Hat Ceph Storage 3 to Red Hat Ceph Storage 4, update the ceph.conf file to either not specify any monitor port or reference both the v2 and v1 addresses and ports. Import any configuration options in ceph.conf file into the storage cluster’s configuration database.

    Example

    [root@mon ~]# ceph config assimilate-conf -i /etc/ceph/ceph.conf

    1. Check the storage cluster’s configuration database.

       Example

       [root@mon ~]# ceph config dump

    2. Optional: After upgrading to Red Hat Ceph Storage 4, create a minimal ceph.conf file for each host:

       Example

       [root@mon ~]# ceph config generate-minimal-conf > /etc/ceph/ceph.conf.new
       [root@mon ~]# mv /etc/ceph/ceph.conf.new /etc/ceph/ceph.conf

18. Install the leveldb package:

    [root@mon ~]# dnf install leveldb

19. Start the monitor service:

    [root@mon ~]# systemctl start ceph-mon.target

20. If the manager service is colocated with the monitor service, start the manager service too:

    [root@mon ~]# systemctl start ceph-mgr.target

21. Verify the monitor service came back up and is in quorum.

    [root@mon ~]# ceph -s

    On the mon: line under services:, ensure the node is listed as in quorum and not as out of quorum.

    Example

    mon: 3 daemons, quorum ceph4-mon,ceph4-mon2,ceph4-mon3 (age 2h)

22. If the manager service is colocated with the monitor service, verify it is up too:

    [root@mon ~]# ceph -s

    Look for the manager’s node name on the mgr: line under services.

    Example

    mgr: ceph4-mon(active, since 2h), standbys: ceph4-mon3, ceph4-mon2

23. Repeat the above steps on all Monitor nodes until they have all been upgraded.

Procedure

1. Set the OSD noout flag to prevent OSDs from getting marked down during the migration:

   ceph osd set noout

2. Set the OSD nobackfill, norecover, norrebalance, noscrub and nodeep-scrub flags to avoid unnecessary load on the cluster and to avoid any data reshuffling when the node goes down for migration:

   ceph osd set nobackfill
   ceph osd set norecover
   ceph osd set norebalance
   ceph osd set noscrub
   ceph osd set nodeep-scrub

3. Gracefully shut down all the OSD processes on the node:

   [root@mon ~]# systemctl stop ceph-osd.target

4. If using Red Hat Ceph Storage 3, disable the Red Hat Ceph Storage 3 repositories.

   1. Disable the tools repository:

      [root@mon ~]# subscription-manager repos --disable=rhel-7-server-rhceph-3-tools-rpms

   2. Disable the osd repository:

      [root@mon ~]# subscription-manager repos --disable=rhel-7-server-rhceph-3-osd-rpms

5. If using Red Hat Ceph Storage 4, disable the Red Hat Ceph Storage 4 repositories.

   1. Disable the tools repository:

      [root@mon ~]# subscription-manager repos --disable=rhel-7-server-rhceph-4-tools-rpms

   2. Disable the osd repository:

      [root@mon ~]# subscription-manager repos --disable=rhel-7-server-rhceph-4-osd-rpms

6. Install the leapp utility. See Upgrading from Red Hat Enterprise Linux 7 to Red Hat Enterprise Linux 8.
7. Run through the leapp preupgrade checks. See Assessing upgradability from the command line.
8. Set PermitRootLogin yes in /etc/ssh/sshd_config.

9. Restart the OpenSSH SSH daemon:

   [root@mon ~]# systemctl restart sshd.service

10. Remove the iSCSI module from the Linux kernel:

    [root@mon ~]# modprobe -r iscsi

11. Perform the upgrade by following Performing the upgrade from RHEL 7 to RHEL 8.
12. Reboot the node.

13. Enable the repositories for Red Hat Ceph Storage 4 for Red Hat Enterprise Linux 8.

    1. Enable the tools repository:

       [root@mon ~]# subscription-manager repos --enable=rhceph-4-tools-for-rhel-8-x86_64-rpms

    2. Enable the osd repository:

       [root@mon ~]# subscription-manager repos --enable=rhceph-4-osd-for-rhel-8-x86_64-rpms

14. Install the ceph-osd package:

    [root@mon ~]# dnf install ceph-osd

15. Install the leveldb package:

    [root@mon ~]# dnf install leveldb

16. Restore the ceph.conf file from a node which has not been upgraded yet or from a node that has already had those files restored.

17. Unset the noout, nobackfill, norecover, norrebalance, noscrub and nodeep-scrub flags:

    # ceph osd unset noout
    # ceph osd unset nobackfill
    # ceph osd unset norecover
    # ceph osd unset norebalance
    # ceph osd unset noscrub
    # ceph osd unset nodeep-scrub

18. Switch any existing CRUSH buckets to the latest bucket type straw2.

    # ceph osd getcrushmap -o backup-crushmap
    # ceph osd crush set-all-straw-buckets-to-straw2

19. Optional: If the OSDs were created using ceph-disk, and are still managed by ceph-disk, you must use ceph-volume to take over management of them.

    1. Mount each object storage device:

       Syntax

       /dev/DRIVE /var/lib/ceph/osd/ceph-OSD_ID

       Replace DRIVE with the storage device name and partition number.

       Replace OSD_ID with the OSD ID.

       Example

       [root@mon ~]# mount /dev/sdb1 /var/lib/ceph/osd/ceph-0

       Verify the ID_NUMBER is correct.

       Syntax

       cat /var/lib/ceph/osd/ceph-OSD_ID/whoami

       Replace OSD_ID with the OSD ID.

       Example

       [root@mon ~]# cat /var/lib/ceph/osd/ceph-0/whoami
       0

       Repeat the above steps for any additional object store devices.

    2. Scan the newly mounted devices:

       Syntax

       ceph-volume simple scan /var/lib/ceph/osd/ceph-OSD_ID

       Replace OSD_ID with the OSD ID.

       Example

       [root@mon ~]# ceph-volume simple scan /var/lib/ceph/osd/ceph-0
        stderr: lsblk: /var/lib/ceph/osd/ceph-0: not a block device
        stderr: lsblk: /var/lib/ceph/osd/ceph-0: not a block device
        stderr: Unknown device, --name=, --path=, or absolute path in /dev/ or /sys expected.
       Running command: /usr/sbin/cryptsetup status /dev/sdb1
       --> OSD 0 got scanned and metadata persisted to file: /etc/ceph/osd/0-0c9917f7-fce8-42aa-bdec-8c2cf2d536ba.json
       --> To take over management of this scanned OSD, and disable ceph-disk and udev, run:
       -->     ceph-volume simple activate 0 0c9917f7-fce8-42aa-bdec-8c2cf2d536ba

       Repeat the above step for any additional object store devices.

    3. Activate the device:

       Syntax

       ceph-volume simple activate OSD_ID UUID

       Replace OSD_ID with the OSD ID and UUID with the UUID printed in the scan output from earlier.

       Example

       [root@mon ~]# ceph-volume simple activate 0 0c9917f7-fce8-42aa-bdec-8c2cf2d536ba
       Running command: /usr/bin/ln -snf /dev/sdb2 /var/lib/ceph/osd/ceph-0/journal
       Running command: /usr/bin/chown -R ceph:ceph /dev/sdb2
       Running command: /usr/bin/systemctl enable ceph-volume@simple-0-0c9917f7-fce8-42aa-bdec-8c2cf2d536ba
        stderr: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-volume@simple-0-0c9917f7-fce8-42aa-bdec-8c2cf2d536ba.service → /usr/lib/systemd/system/ceph-volume@.service.
       Running command: /usr/bin/ln -sf /dev/null /etc/systemd/system/ceph-disk@.service
       --> All ceph-disk systemd units have been disabled to prevent OSDs getting triggered by UDEV events
       Running command: /usr/bin/systemctl enable --runtime ceph-osd@0
        stderr: Created symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd@0.service → /usr/lib/systemd/system/ceph-osd@.service.
       Running command: /usr/bin/systemctl start ceph-osd@0
       --> Successfully activated OSD 0 with FSID 0c9917f7-fce8-42aa-bdec-8c2cf2d536ba

       Repeat the above step for any additional object store devices.

20. Optional: If your OSDs were created with ceph-volume and you did not complete the previous step, start the OSD service now:

    [root@mon ~]# systemctl start ceph-osd.target

21. Activate the OSDs:

    BlueStore

    [root@mon ~]# ceph-volume lvm activate --all

22. Verify that the OSDs are up and in, and that they are in the active+clean state.

    [root@mon ~]# ceph -s

    On the osd: line under services:, ensure that all OSDs are up and in:

    Example

    osd: 3 osds: 3 up (since 8s), 3 in (since 3M)

23. Repeat the above steps on all OSD nodes until they have all been upgraded.

24. If upgrading from Red Hat Ceph Storage 3, disallow pre-Nautilus OSDs and enable the Nautilus-only functionality:

    [root@mon ~]# ceph osd require-osd-release nautilus

    Note

    Failure to execute this step makes it impossible for OSDs to communicate after msgrv2 is enabled.

25. Once all the daemons are updated after upgrading from Red Hat Ceph Storage 3 to Red Hat Ceph Storage 4, run the following steps:

    1. Enable the messenger v2 protocol, msgr2:

       [root@mon ~]# ceph mon enable-msgr2

       This instructs all Ceph Monitors that bind to the old default port of 6789, to also bind to the new port of 3300.

    2. On every node, import any configuration options in ceph.conf file into the storage cluster’s configuration database:

       Example

       [root@mon ~]# ceph config assimilate-conf -i /etc/ceph/ceph.conf

       Note

       When you assimilate a config into your monitors, for example, if you have different config values set for the same set of options, the end result depends on the order in which the files are assimilated.

    3. Check the storage cluster’s configuration database:

       Example

       [root@mon ~]# ceph config dump

Procedure

1. Stop the Ceph Object Gateway service:

   # systemctl stop ceph-radosgw.target

2. If using Red Hat Ceph Storage 3, disable the Red Hat Ceph Storage 3 tool repository:

   # subscription-manager repos --disable=rhel-7-server-rhceph-3-tools-rpms

3. If using Red Hat Ceph Storage 4, disable the Red Hat Ceph Storage 4 tools repository:

   # subscription-manager repos --disable=rhel-7-server-rhceph-4-tools-rpms

4. Install the leapp utility. See Upgrading from Red Hat Enterprise Linux 7 to Red Hat Enterprise Linux 8.
5. Run through the leapp preupgrade checks. See Assessing upgradability from the command line.
6. Set PermitRootLogin yes in /etc/ssh/sshd_config.

7. Restart the OpenSSH SSH daemon:

   # systemctl restart sshd.service

8. Remove the iSCSI module from the Linux kernel:

   # modprobe -r iscsi

9. Perform the upgrade by following Performing the upgrade from RHEL 7 to RHEL 8.
10. Reboot the node.

11. Enable the tools repositories for Red Hat Ceph Storage 4 for Red Hat Enterprise Linux 8.

    # subscription-manager repos --enable=rhceph-4-tools-for-rhel-8-x86_64-rpms

12. Install the ceph-radosgw package:

    # dnf install ceph-radosgw

13. Optional: Install the packages for any Ceph services that are colocated on this node. Enable additional Ceph repositories if needed.

14. Optional: Install the leveldb package which is needed by other Ceph services.

    # dnf install leveldb

15. Restore the ceph-client-admin.keyring and ceph.conf files from a node which has not been upgraded yet or from a node that has already had those files restored.

16. Start the RGW service:

    # systemctl start ceph-radosgw.target

17. Switch any existing CRUSH buckets to the latest bucket type straw2.

    # ceph osd getcrushmap -o backup-crushmap
    # ceph osd crush set-all-straw-buckets-to-straw2

18. Verify the daemon is active:

    # ceph -s

    There is an rgw: line under services:.

    Example

    rgw: 1 daemon active (jb-ceph4-rgw.rgw0)

19. Repeat the above steps on all Ceph Object Gateway nodes until they have all been upgraded.

Procedure

1. Uninstall the existing dashboard from the cluster.

   1. Change to the /usr/share/cephmetrics-ansible directory:

      # cd /usr/share/cephmetrics-ansible

   2. Run the purge.yml Ansible playbook:

      # ansible-playbook -v purge.yml

2. If using Red Hat Ceph Storage 3, disable the Red Hat Ceph Storage 3 tools repository:

   # subscription-manager repos --disable=rhel-7-server-rhceph-3-tools-rpms

3. If using Red Hat Ceph Storage 4, disable the Red Hat Ceph Storage 4 tools repository:

   # subscription-manager repos --disable=rhel-7-server-rhceph-4-tools-rpms

4. Install the leapp utility. See Upgrading from Red Hat Enterprise Linux 7 to Red Hat Enterprise Linux 8.
5. Run through the leapp preupgrade checks. See Assessing upgradability from the command line.
6. Set PermitRootLogin yes in /etc/ssh/sshd_config.

7. Restart the OpenSSH SSH daemon:

   # systemctl restart sshd.service

8. Remove the iSCSI module from the Linux kernel:

   # modprobe -r iscsi

9. Perform the upgrade by following Performing the upgrade from RHEL 7 to RHEL 8.
10. Reboot the node.

11. Enable the tools repository for Red Hat Ceph Storage 4 for Red Hat Enterprise Linux 8:

    # subscription-manager repos --enable=rhceph-4-tools-for-rhel-8-x86_64-rpms

12. Enable the Ansible repository:

    # subscription-manager repos --enable=ansible-2.9-for-rhel-8-x86_64-rpms

13. Configure ceph-ansible to manage the cluster. It will install the dashboard. Follow the instructions in Installing Red Hat Ceph Storage using Ansible, including the prerequisites.
14. After you run ansible-playbook site.yml as a part of the above procedures, the URL for the dashboard will be printed. See Installing dashboard using Ansible in the Dashboard guide for more information on locating the URL and accessing the dashboard.

Procedure

1. Enable the tools repository for Red Hat Ceph Storage 4 for Red Hat Enterprise Linux 8:

   [root@dashboard ~]# subscription-manager repos --enable=rhceph-4-tools-for-rhel-8-x86_64-rpms

2. Enable the Ansible repository:

   [root@dashboard ~]# subscription-manager repos --enable=ansible-2.9-for-rhel-8-x86_64-rpms

3. Configure ceph-ansible to manage the storage cluster. It will install the dashboard. Follow the instructions in Installing Red Hat Ceph Storage using Ansible, including the prerequisites.
4. After you run ansible-playbook site.yml as a part of the above procedures, the URL for the dashboard will be printed. See Installing dashboard using Ansible in the Dashboard guide for more information on locating the URL and accessing the dashboard.

Procedure

1. Reduce the number of active MDS ranks to 1:

   Syntax

   ceph fs set FILE_SYSTEM_NAME max_mds 1

   Example

   [root@mds ~]# ceph fs set fs1 max_mds 1

2. Wait for the cluster to stop all of the MDS ranks. When all of the MDS have stopped, only rank 0 should be active. The rest should be in standby mode. Check the status of the file system:

   [root@mds ~]# ceph status

3. Use systemctl to take all standby MDS offline:

   [root@mds ~]# systemctl stop ceph-mds.target

4. Confirm that only one MDS is online, and that it has rank 0 for the file system:

   [root@mds ~]# ceph status

5. Disable the tools repository for the operating system version:

   1. If you are upgrading from Red Hat Ceph Storage 3 on RHEL 7, disable the Red Hat Ceph Storage 3 tools repository:

      [root@mds ~]# subscription-manager repos --disable=rhel-7-server-rhceph-3-tools-rpms

   2. If you are using Red Hat Ceph Storage 4, disable the Red Hat Ceph Storage 4 tools repository:

      [root@mds ~]# subscription-manager repos --disable=rhel-7-server-rhceph-4-tools-rpms

6. Install the leapp utility. For more information about leapp, refer to Upgrading from Red Hat Enterprise Linux 7 to Red Hat Enterprise Linux 8.
7. Run through the leapp preupgrade checks. For more information, refer to Assessing upgradability from the command line.
8. Edit /etc/ssh/sshd_config and set PermitRootLogin to yes.

9. Restart the OpenSSH SSH daemon:

   [root@mds ~]# systemctl restart sshd.service

10. Remove the iSCSI module from the Linux kernel:

    [root@mds ~]# modprobe -r iscsi

11. Perform the upgrade. See Performing the upgrade from RHEL 7 to RHEL 8.
12. Reboot the MDS node.

13. Enable the tools repositories for Red Hat Ceph Storage 4 for Red Hat Enterprise Linux 8:

    [root@mds ~]# subscription-manager repos --enable=rhceph-4-tools-for-rhel-8-x86_64-rpms

14. Install the ceph-mds package:

    [root@mds ~]# dnf install ceph-mds -y

15. Optional: Install the packages for any Ceph services that are colocated on this node. Enable additional Ceph repositories, if needed.

16. Optional: Install the leveldb package, which is needed by other Ceph services:

    [root@mds ~]# dnf install leveldb

17. Restore the ceph-client-admin.keyring and ceph.conf files from a node that has not been upgraded yet, or from a node that has already had those files restored.

18. Switch any existing CRUSH buckets to the latest bucket type straw2.

    # ceph osd getcrushmap -o backup-crushmap
    # ceph osd crush set-all-straw-buckets-to-straw2

19. Start the MDS service:

    [root@mds ~]# systemctl restart ceph-mds.target

20. Verify that the daemon is active:

    [root@mds ~]# ceph -s

21. Follow the same processes for the standby daemons.

22. When you have finished restarting all of the MDS in standby, restore the previous value of max_mds for your cluster:

    Syntax

    ceph fs set FILE_SYSTEM_NAME max_mds ORIGINAL_VALUE

    Example

    [root@mds ~]# ceph fs set fs1 max_mds 5

Procedure

1. Perform a standard installation of Red Hat Enterprise Linux 8.4 on the failed node and enable the Red Hat Enterprise Linux repositories.

   • Performing a standard RHEL installation

2. Enable the repositories for Red Hat Ceph Storage 4 for Red Hat Enterprise Linux 8.

   1. Enable the tools repository:

      # subscription-manager repos --enable=rhceph-4-tools-for-rhel-8-x86_64-rpms

   2. Enable the osd repository:

      # subscription-manager repos --enable=rhceph-4-osd-for-rhel-8-x86_64-rpms

3. Install the ceph-osd package:

   # dnf install ceph-osd

4. Restore the ceph.conf file to /etc/ceph from a node which has not been upgraded yet or from a node that has already had those files restored.

5. Start the OSD service:

   # systemctl start ceph-osd.target

6. Activate the object store devices:

   ceph-volume lvm activate --all

7. Watch the recovery of the OSDs and cluster backfill writes to recovered OSDs:

   # ceph -w

   Monitor the output until all PGs are in state active+clean.