Chapter 3. Requirements for Installing Red Hat Ceph Storage

Procedure

1. For container deployments only, when the Red Hat Ceph Storage nodes do NOT have access to the Internet during deployment. You must follow these steps first on a node with Internet access:

   1. Start a local container registry:

      Red Hat Enterprise Linux 7

      # docker run -d -p 5000:5000 --restart=always --name registry registry:2

      Copy to Clipboard Toggle word wrap

      Red Hat Enterprise Linux 8

      # podman run -d -p 5000:5000 --restart=always --name registry registry:2

      Copy to Clipboard Toggle word wrap

   2. Verify registry.redhat.io is in the container registry search path.

      Open for editing the /etc/containers/registries.conf file:

      [registries.search]
      registries = [ 'registry.access.redhat.com', 'registry.fedoraproject.org', 'registry.centos.org', 'docker.io']

      Copy to Clipboard Toggle word wrap

      If registry.redhat.io is not included in the file, add it:

      [registries.search]
      registries = ['registry.redhat.io', 'registry.access.redhat.com', 'registry.fedoraproject.org', 'registry.centos.org', 'docker.io']

      Copy to Clipboard Toggle word wrap

   3. Pull the Red Hat Ceph Storage 4 image, Prometheus image, and Dashboard image from the Red Hat Customer Portal:

      Red Hat Enterprise Linux 7

      # docker pull registry.redhat.io/rhceph/rhceph-4-rhel8:latest
      # docker pull registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.6
      # docker pull registry.redhat.io/rhceph/rhceph-4-dashboard-rhel8:latest
      # docker pull registry.redhat.io/openshift4/ose-prometheus:v4.6
      # docker pull registry.redhat.io/openshift4/ose-prometheus-alertmanager:v4.6

      Copy to Clipboard Toggle word wrap

      Red Hat Enterprise Linux 8

      # podman pull registry.redhat.io/rhceph/rhceph-4-rhel8:latest
      # podman pull registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.6
      # podman pull registry.redhat.io/rhceph/rhceph-4-dashboard-rhel8:latest
      # podman pull registry.redhat.io/openshift4/ose-prometheus:v4.6
      # podman pull registry.redhat.io/openshift4/ose-prometheus-alertmanager:v4.6

      Copy to Clipboard Toggle word wrap

      Note

      Red Hat Enterprise Linux 7 and 8 both use the same container image, based on Red Hat Enterprise Linux 8.

   4. Tag the image:

      The Prometheus image tag version is v4.6 for Red Hat Ceph Storage 4.2.

      Red Hat Enterprise Linux 7

       # docker tag registry.redhat.io/rhceph/rhceph-4-rhel8:latest LOCAL_NODE_FQDN:5000/rhceph/rhceph-4-rhel8:latest
       # docker tag registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.6 LOCAL_NODE_FQDN:5000/openshift4/ose-prometheus-node-exporter:v4.6
       # docker tag registry.redhat.io/rhceph/rhceph-4-dashboard-rhel8:latest LOCAL_NODE_FQDN:5000/rhceph/rhceph-4-dashboard-rhel8:latest
       # docker tag registry.redhat.io/openshift4/ose-prometheus-alertmanager:v4.6 LOCAL_NODE_FQDN:5000/openshift4/ose-prometheus-alertmanager:v4.6
       # docker tag registry.redhat.io/openshift4/ose-prometheus:v4.6 LOCAL_NODE_FQDN:5000/openshift4/ose-prometheus:v4.6

      Copy to Clipboard Toggle word wrap

      Replace

      • LOCAL_NODE_FQDN with your local host FQDN.

      Red Hat Enterprise Linux 8

       # podman tag registry.redhat.io/rhceph/rhceph-4-rhel8:latest LOCAL_NODE_FQDN:5000/rhceph/rhceph-4-rhel8:latest
       # podman tag registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.6 LOCAL_NODE_FQDN:5000/openshift4/ose-prometheus-node-exporter:v4.6
       # podman tag registry.redhat.io/rhceph/rhceph-4-dashboard-rhel8:latest LOCAL_NODE_FQDN:5000/rhceph/rhceph-4-dashboard-rhel8:latest
       # podman tag registry.redhat.io/openshift4/ose-prometheus-alertmanager:v4.6 LOCAL_NODE_FQDN:5000/openshift4/ose-prometheus-alertmanager:v4.6
       # podman tag registry.redhat.io/openshift4/ose-prometheus:v4.6 LOCAL_NODE_FQDN:5000/openshift4/ose-prometheus:v4.6

      Copy to Clipboard Toggle word wrap

      Replace

      • LOCAL_NODE_FQDN with your local host FQDN.

   5. Edit the /etc/containers/registries.conf file and add the node’s FQDN with the port in the file, and save:

      [registries.insecure]
      registries = ['LOCAL_NODE_FQDN:5000']

      Copy to Clipboard Toggle word wrap
      Note

      This step must be done on all storage cluster nodes that access the local Docker registry.

   6. Push the image to the local Docker registry you started:

      Red Hat Enterprise Linux 7

       # docker push --remove-signatures LOCAL_NODE_FQDN:5000/rhceph/rhceph-4-rhel8
       # docker push --remove-signatures LOCAL_NODE_FQDN:5000/openshift4/ose-prometheus-node-exporter:v4.6
       # docker push --remove-signatures LOCAL_NODE_FQDN:5000/rhceph/rhceph-4-dashboard-rhel8
       # docker push --remove-signatures LOCAL_NODE_FQDN:5000/openshift4/ose-prometheus-alertmanager:v4.6
       # docker push --remove-signatures LOCAL_NODE_FQDN:5000/openshift4/ose-prometheus:v4.6

      Copy to Clipboard Toggle word wrap

      Replace

      • LOCAL_NODE_FQDN with your local host FQDN.

      Red Hat Enterprise Linux 8

       # podman push --remove-signatures LOCAL_NODE_FQDN:5000/rhceph/rhceph-4-rhel8
       # podman push --remove-signatures LOCAL_NODE_FQDN:5000/openshift4/ose-prometheus-node-exporter:v4.6
       # podman push --remove-signatures LOCAL_NODE_FQDN:5000/rhceph/rhceph-4-dashboard-rhel8
       # podman push --remove-signatures LOCAL_NODE_FQDN:5000/openshift4/ose-prometheus-alertmanager:v4.6
       # podman push --remove-signatures LOCAL_NODE_FQDN:5000/openshift4/ose-prometheus:v4.6

      Copy to Clipboard Toggle word wrap

      Replace

      • LOCAL_NODE_FQDN with your local host FQDN.

   7. For Red Hat Enterprise Linux 7, restart the docker service:

      # systemctl restart docker

      Copy to Clipboard Toggle word wrap
      Note

      See the Installing a Red Hat Ceph Storage cluster for an example of the all.yml file when the Red Hat Ceph Storage nodes do NOT have access to the Internet during deployment.

2. For all deployments, bare-metal or in containers:

   1. Register the node, and when prompted, enter the appropriate Red Hat Customer Portal credentials:

      # subscription-manager register

      Copy to Clipboard Toggle word wrap

   2. Pull the latest subscription data from the CDN:

      # subscription-manager refresh

      Copy to Clipboard Toggle word wrap

   3. List all available subscriptions for Red Hat Ceph Storage:

      # subscription-manager list --available --all --matches="*Ceph*"

      Copy to Clipboard Toggle word wrap

      Copy the Pool ID from the list of available subscriptions for Red Hat Ceph Storage.

   4. Attach the subscription:

      # subscription-manager attach --pool=POOL_ID

      Copy to Clipboard Toggle word wrap

      Replace

      • POOL_ID with the Pool ID identified in the previous step.

   5. Disable the default software repositories, and enable the server and the extras repositories on the respective version of Red Hat Enterprise Linux:

      Red Hat Enterprise Linux 7

      # subscription-manager repos --disable=*
      # subscription-manager repos --enable=rhel-7-server-rpms
      # subscription-manager repos --enable=rhel-7-server-extras-rpms

      Copy to Clipboard Toggle word wrap

      Red Hat Enterprise Linux 8

      # subscription-manager repos --disable=*
      # subscription-manager repos --enable=rhel-8-for-x86_64-baseos-rpms
      # subscription-manager repos --enable=rhel-8-for-x86_64-appstream-rpms

      Copy to Clipboard Toggle word wrap

3. Update the system to receive the latest packages.

   1. For Red Hat Enterprise Linux 7:

      # yum update

      Copy to Clipboard Toggle word wrap

   2. For Red Hat Enterprise Linux 8:

      # dnf update

      Copy to Clipboard Toggle word wrap

1. Verify the following settings are in the /etc/sysconfig/network-scripts/ifcfg-* file corresponding the public-facing network interface card:

   1. The BOOTPROTO parameter is set to none for static IP addresses.

   2. The ONBOOT parameter must be set to yes.

      If it is set to no, the Ceph storage cluster might fail to peer on reboot.

   3. If you intend to use IPv6 addressing, you must set the IPv6 parameters such as IPV6INIT to yes, except the IPV6_FAILURE_FATAL parameter.

      Also, edit the Ceph configuration file, /etc/ceph/ceph.conf, to instruct Ceph to use IPv6, otherwise, Ceph uses IPv4.

Procedure

1. On all nodes in the storage cluster, start the firewalld service. Enable it to run on boot, and ensure that it is running:

   # systemctl enable firewalld
   # systemctl start firewalld
   # systemctl status firewalld

   Copy to Clipboard Toggle word wrap

2. On all monitor nodes, open port 3300 and 6789 on the public network:

   [root@monitor ~]# firewall-cmd --zone=public --add-port=3300/tcp
   [root@monitor ~]# firewall-cmd --zone=public --add-port=3300/tcp --permanent
   [root@monitor ~]# firewall-cmd --zone=public --add-port=6789/tcp
   [root@monitor ~]# firewall-cmd --zone=public --add-port=6789/tcp --permanent
   [root@monitor ~]# firewall-cmd --permanent --add-service=ceph-mon
   [root@monitor ~]# firewall-cmd --add-service=ceph-mon

   Copy to Clipboard Toggle word wrap

   To limit access based on the source address:

   firewall-cmd --zone=public --add-rich-rule='rule family=ipv4 \
   source address=IP_ADDRESS/NETMASK_PREFIX port protocol=tcp \
   port=6789 accept' --permanent

   Copy to Clipboard Toggle word wrap

   Replace

   • IP_ADDRESS with the network address of the Monitor node.

   • NETMASK_PREFIX with the netmask in CIDR notation.

     Example

     [root@monitor ~]# firewall-cmd --zone=public --add-rich-rule='rule family=ipv4 \
     source address=192.168.0.11/24 port protocol=tcp \
     port=6789 accept' --permanent

     Copy to Clipboard Toggle word wrap

3. On all OSD nodes, open ports 6800-7300 on the public network:

   [root@osd ~]# firewall-cmd --zone=public --add-port=6800-7300/tcp
   [root@osd ~]# firewall-cmd --zone=public --add-port=6800-7300/tcp --permanent
   [root@osd ~]# firewall-cmd --permanent --add-service=ceph
   [root@osd ~]# firewall-cmd --add-service=ceph

   Copy to Clipboard Toggle word wrap

   If you have a separate cluster network, repeat the commands with the appropriate zone.

4. On all Ceph Manager (ceph-mgr) nodes, open ports 6800-7300 on the public network:

   [root@monitor ~]# firewall-cmd --zone=public --add-port=6800-7300/tcp
   [root@monitor ~]# firewall-cmd --zone=public --add-port=6800-7300/tcp --permanent

   Copy to Clipboard Toggle word wrap

   If you have a separate cluster network, repeat the commands with the appropriate zone.

5. On all Ceph Metadata Server (ceph-mds) nodes, open ports 6800-7300 on the public network:

   [root@monitor ~]# firewall-cmd --zone=public --add-port=6800-7300/tcp
   [root@monitor ~]# firewall-cmd --zone=public --add-port=6800-7300/tcp --permanent

   Copy to Clipboard Toggle word wrap

   If you have a separate cluster network, repeat the commands with the appropriate zone.

6. On all Ceph Object Gateway nodes, open the relevant port or ports on the public network.

   1. To open the default Ansible configured port of 8080:

      [root@gateway ~]# firewall-cmd --zone=public --add-port=8080/tcp
      [root@gateway ~]# firewall-cmd --zone=public --add-port=8080/tcp --permanent

      Copy to Clipboard Toggle word wrap

      To limit access based on the source address:

      firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
      source address="IP_ADDRESS/NETMASK_PREFIX" port protocol="tcp" \
      port="8080" accept"

      Copy to Clipboard Toggle word wrap

      firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
      source address="IP_ADDRESS/NETMASK_PREFIX" port protocol="tcp" \
      port="8080" accept" --permanent

      Copy to Clipboard Toggle word wrap

      Replace

      • IP_ADDRESS with the network address of the Monitor node.

      • NETMASK_PREFIX with the netmask in CIDR notation.

        Example

        [root@gateway ~]# firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
        source address="192.168.0.31/24" port protocol="tcp" \
        port="8080" accept"

        Copy to Clipboard Toggle word wrap

        [root@gateway ~]# firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
        source address="192.168.0.31/24" port protocol="tcp" \
        port="8080" accept" --permanent

        Copy to Clipboard Toggle word wrap

   2. Optionally, if you installed Ceph Object Gateway using Ansible and changed the default port that Ansible configures the Ceph Object Gateway to use from 8080, for example, to port 80, then open this port:

      [root@gateway ~]# firewall-cmd --zone=public --add-port=80/tcp
      [root@gateway ~]# firewall-cmd --zone=public --add-port=80/tcp --permanent

      Copy to Clipboard Toggle word wrap

      To limit access based on the source address, run the following commands:

      firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
      source address="IP_ADDRESS/NETMASK_PREFIX" port protocol="tcp" \
      port="80" accept"

      Copy to Clipboard Toggle word wrap

      firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
      source address="IP_ADDRESS/NETMASK_PREFIX" port protocol="tcp" \
      port="80" accept" --permanent

      Copy to Clipboard Toggle word wrap

      Replace

      • IP_ADDRESS with the network address of the Monitor node.
      • NETMASK_PREFIX with the netmask in CIDR notation.

      Example

      [root@gateway ~]# firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
      source address="192.168.0.31/24" port protocol="tcp" \
      port="80" accept"

      Copy to Clipboard Toggle word wrap

      [root@gateway ~]# firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
      source address="192.168.0.31/24" port protocol="tcp" \
      port="80" accept" --permanent

      Copy to Clipboard Toggle word wrap

   3. Optional. To use SSL/TLS, open port 443:

      [root@gateway ~]# firewall-cmd --zone=public --add-port=443/tcp
      [root@gateway ~]# firewall-cmd --zone=public --add-port=443/tcp --permanent

      Copy to Clipboard Toggle word wrap

      To limit access based on the source address, run the following commands:

      firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
      source address="IP_ADDRESS/NETMASK_PREFIX" port protocol="tcp" \
      port="443" accept"

      Copy to Clipboard Toggle word wrap

      firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
      source address="IP_ADDRESS/NETMASK_PREFIX" port protocol="tcp" \
      port="443" accept" --permanent

      Copy to Clipboard Toggle word wrap

      Replace

      • IP_ADDRESS with the network address of the Monitor node.
      • NETMASK_PREFIX with the netmask in CIDR notation.

      Example

      [root@gateway ~]# firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
      source address="192.168.0.31/24" port protocol="tcp" \
      port="443" accept"
      [root@gateway ~]# firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
      source address="192.168.0.31/24" port protocol="tcp" \
      port="443" accept" --permanent

      Copy to Clipboard Toggle word wrap

Procedure

1. Log into the node as the root user:

   ssh root@HOST_NAME

   Copy to Clipboard Toggle word wrap

   Replace

   • HOST_NAME with the host name of the Ceph node.

     Example

     # ssh root@mon01

     Copy to Clipboard Toggle word wrap

     Enter the root password when prompted.

2. Create a new Ansible user:

   adduser USER_NAME

   Copy to Clipboard Toggle word wrap

   Replace

   • USER_NAME with the new user name for the Ansible user.

     Example

     # adduser admin

     Copy to Clipboard Toggle word wrap

     Important

     Do not use ceph as the user name. The ceph user name is reserved for the Ceph daemons. A uniform user name across the cluster can improve ease of use, but avoid using obvious user names, because intruders typically use them for brute-force attacks.

3. Set a new password for this user:

   # passwd USER_NAME

   Copy to Clipboard Toggle word wrap

   Replace

   • USER_NAME with the new user name for the Ansible user.

     Example

     # passwd admin

     Copy to Clipboard Toggle word wrap

     Enter the new password twice when prompted.

4. Configure sudo access for the newly created user:

   cat << EOF >/etc/sudoers.d/USER_NAME
   $USER_NAME ALL = (root) NOPASSWD:ALL
   EOF

   Copy to Clipboard Toggle word wrap

   Replace

   • USER_NAME with the new user name for the Ansible user.

     Example

     # cat << EOF >/etc/sudoers.d/admin
     admin ALL = (root) NOPASSWD:ALL
     EOF

     Copy to Clipboard Toggle word wrap

5. Assign the correct file permissions to the new file:

   chmod 0440 /etc/sudoers.d/USER_NAME

   Copy to Clipboard Toggle word wrap

   Replace

   • USER_NAME with the new user name for the Ansible user.

     Example

     # chmod 0440 /etc/sudoers.d/admin

     Copy to Clipboard Toggle word wrap

Procedure

1. Generate the SSH key pair, accept the default file name and leave the passphrase empty:

   [ansible@admin ~]$ ssh-keygen

   Copy to Clipboard Toggle word wrap

2. Copy the public key to all nodes in the storage cluster:

   ssh-copy-id USER_NAME@HOST_NAME

   Copy to Clipboard Toggle word wrap

   Replace

   • USER_NAME with the new user name for the Ansible user.

   • HOST_NAME with the host name of the Ceph node.

     Example

     [ansible@admin ~]$ ssh-copy-id ceph-admin@ceph-mon01

     Copy to Clipboard Toggle word wrap

3. Create the user’s SSH config file:

   [ansible@admin ~]$ touch ~/.ssh/config

   Copy to Clipboard Toggle word wrap

4. Open for editing the config file. Set values for the Hostname and User options for each node in the storage cluster:

   Host node1
      Hostname HOST_NAME
      User USER_NAME
   Host node2
      Hostname HOST_NAME
      User USER_NAME
   ...

   Copy to Clipboard Toggle word wrap

   Replace

   • HOST_NAME with the host name of the Ceph node.

   • USER_NAME with the new user name for the Ansible user.

     Example

     Host node1
        Hostname monitor
        User admin
     Host node2
        Hostname osd
        User admin
     Host node3
        Hostname gateway
        User admin

     Copy to Clipboard Toggle word wrap

     Important

     By configuring the ~/.ssh/config file you do not have to specify the -u USER_NAME option each time you execute the ansible-playbook command.

5. Set the correct file permissions for the ~/.ssh/config file:

   [admin@admin ~]$ chmod 600 ~/.ssh/config

   Copy to Clipboard Toggle word wrap