Installation Guide

Example 1

1. Media: Full flash systems (SSDs)
2. Use case: Block (RBD) and File (CephFS), or Object (Ceph Object Gateway)
3. Number of nodes: 3
4. Replication scheme: 2

Example 2

1. Media: Full flash systems (SSDs) or spinning devices (HDDs)
2. Use case: Block (RBD), File (CephFS), and Object (Ceph Object Gateway)
3. Number of nodes: 4
4. Replication scheme: 3

Example 3

1. Media: Full flash systems (SSDs) or spinning devices (HDDs)
2. Use case: Block (RBD), Object (Ceph Object Gateway), and NFS for Ceph Object Gateway
3. Number of nodes: 4
4. Replication scheme: 3

Procedure

1. Register the node, and when prompted, enter your Red Hat Customer Portal credentials:

   Syntax

   subscription-manager register

2. Pull the latest subscription data from the CDN:

   Syntax

   subscription-manager refresh

3. List all available subscriptions for Red Hat Ceph Storage:

   Syntax

   subscription-manager list --available --matches 'Red Hat Ceph Storage'

4. Identify the appropriate subscription and retrieve its Pool ID.

5. Attach a pool ID to gain access to the software entitlements. Use the Pool ID you identified in the previous step.

   Syntax

   subscription-manager attach --pool=POOL_ID

6. Disable the default software repositories, and then enable the server and the extras repositories on the respective version of Red Hat Enterprise Linux:

   Red Hat Enterprise Linux 9

   subscription-manager repos --disable=*
   subscription-manager repos --enable=rhel-9-for-x86_64-baseos-rpms
   subscription-manager repos --enable=rhel-9-for-x86_64-appstream-rpms

7. Update the system to receive the latest packages for Red Hat Enterprise Linux:

   Syntax

   # dnf update

8. Subscribe to Red Hat Ceph Storage 8 content. Follow the instructions in How to Register Ceph with Red Hat Satellite 6.

9. Enable the ceph-tools repository:

   Red Hat Enterprise Linux 9

   subscription-manager repos --enable=rhceph-8-tools-for-rhel-9-x86_64-rpms

10. Repeat the above steps on all nodes you are adding to the cluster.

11. Install cephadm-ansible:

    Syntax

    dnf install cephadm-ansible

Procedure

1. Navigate to the /usr/share/cephadm-ansible/ directory:

   [root@admin ~]# cd /usr/share/cephadm-ansible

2. Optional: Create subdirectories for staging and production:

   [root@admin cephadm-ansible]# mkdir -p inventory/staging inventory/production

3. Optional: Edit the ansible.cfg file and add the following line to assign a default inventory location:

   [defaults]
   inventory = ./inventory/staging

4. Optional: Create an inventory hosts file for each environment:

   [root@admin cephadm-ansible]# touch inventory/staging/hosts
   [root@admin cephadm-ansible]# touch inventory/production/hosts

5. Open and edit each hosts file and add the nodes and [admin] group:

   NODE_NAME_1
   NODE_NAME_2
   
   [admin]
   ADMIN_NODE_NAME_1

   • Replace NODE_NAME_1 and NODE_NAME_2 with the Ceph nodes such as monitors, OSDs, MDSs, and gateway nodes.

   • Replace ADMIN_NODE_NAME_1 with the name of the node where the admin keyring is stored.

     Example

     host02
     host03
     host04
     
     [admin]
     host01

     Note

     If you set the inventory location in the ansible.cfg file to staging, you need to run the playbooks in the staging environment as follows:

     Syntax

     ansible-playbook -i inventory/staging/hosts PLAYBOOK.yml

     To run the playbooks in the production environment:

     Syntax

     ansible-playbook -i inventory/production/hosts PLAYBOOK.yml

Procedure

1. Open the etc/ssh/sshd_config file and set the PermitRootLogin to yes:

   Example

   [root@admin ~]# echo 'PermitRootLogin yes' >> /etc/ssh/sshd_config.d/01-permitrootlogin.conf

2. Restart the SSH service:

   Example

   [root@admin ~]# systemctl restart sshd.service

3. Login to the node as the root user:

   Syntax

   ssh root@HOST_NAME

   Replace HOST_NAME with the host name of the Ceph node.

   Example

   [root@admin ~]# ssh root@host01

   Enter the root password when prompted.

Procedure

1. Log in to the node as the root user:

   Syntax

   ssh root@HOST_NAME

   Replace HOST_NAME with the host name of the Ceph node.

   Example

   [root@admin ~]# ssh root@host01

   Enter the root password when prompted.

2. Create a new Ansible user:

   Syntax

   adduser USER_NAME

   Replace USER_NAME with the new user name for the Ansible user.

   Example

   [root@host01 ~]# adduser ceph-admin

   Important

   Do not use ceph as the user name. The ceph user name is reserved for the Ceph daemons. A uniform user name across the cluster can improve ease of use, but avoid using obvious user names, because intruders typically use them for brute-force attacks.

3. Set a new password for this user:

   Syntax

   passwd USER_NAME

   Replace USER_NAME with the new user name for the Ansible user.

   Example

   [root@host01 ~]# passwd ceph-admin

   Enter the new password twice when prompted.

4. Configure sudo access for the newly created user:

   Syntax

   cat << EOF >/etc/sudoers.d/USER_NAME
   $USER_NAME ALL = (root) NOPASSWD:ALL
   EOF

   Replace USER_NAME with the new user name for the Ansible user.

   Example

   [root@host01 ~]# cat << EOF >/etc/sudoers.d/ceph-admin
   ceph-admin ALL = (root) NOPASSWD:ALL
   EOF

5. Assign the correct file permissions to the new file:

   Syntax

   chmod 0440 /etc/sudoers.d/USER_NAME

   Replace USER_NAME with the new user name for the Ansible user.

   Example

   [root@host01 ~]# chmod 0440 /etc/sudoers.d/ceph-admin

6. Repeat the above steps on all nodes in the storage cluster.

Procedure

1. Generate the SSH key pair, accept the default file name and leave the passphrase empty:

   [ceph-admin@admin ~]$ ssh-keygen

2. Copy the public key to all nodes in the storage cluster:

   ssh-copy-id USER_NAME@HOST_NAME

   Replace USER_NAME with the new user name for the Ansible user. Replace HOST_NAME with the host name of the Ceph node.

   Example

   [ceph-admin@admin ~]$ ssh-copy-id ceph-admin@host01

3. Create the user’s SSH config file:

   [ceph-admin@admin ~]$ touch ~/.ssh/config

4. Open for editing the config file. Set values for the Hostname and User options for each node in the storage cluster:

   Host host01
      Hostname HOST_NAME
      User USER_NAME
   Host host02
      Hostname HOST_NAME
      User USER_NAME
   ...

   Replace HOST_NAME with the host name of the Ceph node. Replace USER_NAME with the new user name for the Ansible user.

   Example

   Host host01
      Hostname host01
      User ceph-admin
   Host host02
      Hostname host02
      User ceph-admin
   Host host03
      Hostname host03
      User ceph-admin

   Important

   By configuring the ~/.ssh/config file you do not have to specify the -u USER_NAME option each time you execute the ansible-playbook command.

5. Set the correct file permissions for the ~/.ssh/config file:

   [ceph-admin@admin ~]$ chmod 600 ~/.ssh/config

Procedure

1. Navigate to the the /usr/share/cephadm-ansible directory.

2. Open and edit the hosts file and add your nodes:

   Example

   host02
   host03
   host04
   
   [admin]
   host01

3. Run the preflight playbook:

   Syntax

   ansible-playbook -i INVENTORY_FILE cephadm-preflight.yml --extra-vars "ceph_origin=rhcs"

   Example

   [ceph-admin@admin cephadm-ansible]$ ansible-playbook -i hosts cephadm-preflight.yml --extra-vars "ceph_origin=rhcs"

   After installation is complete, cephadm resides in the /usr/sbin/ directory.

   • Use the --limit option to run the preflight playbook on a selected set of hosts in the storage cluster:

     Syntax

     ansible-playbook -i INVENTORY_FILE cephadm-preflight.yml --extra-vars "ceph_origin=rhcs" --limit GROUP_NAME|NODE_NAME

     Replace GROUP_NAME with a group name from your inventory file. Replace NODE_NAME with a specific node name from your inventory file.

     Note

     Optionally, you can group your nodes in your inventory file by group name such as [mons], [osds], and [mgrs]. However, admin nodes must be added to the [admin] group and clients must be added to the [clients] group.

     Example

     [ceph-admin@admin cephadm-ansible]$ ansible-playbook -i hosts cephadm-preflight.yml --extra-vars "ceph_origin=rhcs" --limit clients
     [ceph-admin@admin cephadm-ansible]$ ansible-playbook -i hosts cephadm-preflight.yml --extra-vars "ceph_origin=rhcs" --limit host01

   • When you run the preflight playbook, cephadm-ansible automatically installs chrony and ceph-common on the client nodes.

     The preflight playbook installs chrony but configures it for a single NTP source. If you want to configure multiple sources or if you have a disconnected environment, see the following documentation for more information:

     • How to configure chrony?
     • Best practices for NTP.
     • Basic chrony NTP troubleshooting.

Procedure

1. Bootstrap a storage cluster:

   Syntax

   cephadm bootstrap --cluster-network NETWORK_CIDR --mon-ip IP_ADDRESS --registry-url registry.redhat.io --registry-username USER_NAME --registry-password PASSWORD --yes-i-know

   Example

   [root@host01 ~]# cephadm bootstrap --cluster-network 10.10.128.0/24 --mon-ip 10.10.128.68 --registry-url registry.redhat.io --registry-username myuser1 --registry-password mypassword1 --yes-i-know

   Note

   If you want internal cluster traffic routed over the public network, you can omit the --cluster-network NETWORK_CIDR option.

   The script takes a few minutes to complete. Once the script completes, it provides the credentials to the Red Hat Ceph Storage Dashboard URL, a command to access the Ceph command-line interface (CLI), and a request to enable telemetry.

   Ceph Dashboard is now available at:
   
                URL: https://host01:8443/
               User: admin
           Password: i8nhu7zham
   
   Enabling client.admin keyring and conf on hosts with "admin" label
   You can access the Ceph CLI with:
   
           sudo /usr/sbin/cephadm shell --fsid 266ee7a8-2a05-11eb-b846-5254002d4916 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring
   
   Please consider enabling telemetry to help improve Ceph:
   
           ceph telemetry on
   
   For more information see:
   
           https://docs.ceph.com/docs/master/mgr/telemetry/
   
   Bootstrap complete.

Procedure

1. Navigate to the /usr/share/cephadm-ansible directory on the Ansible administration node:

   Example

   [ansible@admin ~]$ cd /usr/share/cephadm-ansible

2. From the Ansible administration node, distribute the SSH keys. The optional cephadm_pubkey_path parameter is the full path name of the SSH public key file on the ansible controller host.

   Note

   If cephadm_pubkey_path is not specified, the playbook gets the key from the cephadm get-pub-key command. This implies that you have at least bootstrapped a minimal cluster.

   Syntax

   ansible-playbook -i INVENTORY_HOST_FILE cephadm-distribute-ssh-key.yml -e cephadm_ssh_user=USER_NAME -e cephadm_pubkey_path= home/cephadm/ceph.key -e admin_node=ADMIN_NODE_NAME_1

   Example

   [ansible@admin cephadm-ansible]$ ansible-playbook -i hosts cephadm-distribute-ssh-key.yml -e cephadm_ssh_user=ceph-admin -e cephadm_pubkey_path=/home/cephadm/ceph.key -e admin_node=host01
   
   [ansible@admin cephadm-ansible]$ ansible-playbook -i hosts cephadm-distribute-ssh-key.yml -e cephadm_ssh_user=ceph-admin -e admin_node=host01

1. From the node that contains the admin keyring, install the storage cluster’s public SSH key in the root user’s authorized_keys file on the new host:

   Syntax

   ssh-copy-id -f -i /etc/ceph/ceph.pub user@NEWHOST

   Example

   [root@host01 ~]# ssh-copy-id -f -i /etc/ceph/ceph.pub root@host02
   [root@host01 ~]# ssh-copy-id -f -i /etc/ceph/ceph.pub root@host03

2. Navigate to the /usr/share/cephadm-ansible directory on the Ansible administration node.

   Example

   [ceph-admin@admin ~]$ cd /usr/share/cephadm-ansible

3. From the Ansible administration node, add the new host to the Ansible inventory file. The default location for the file is /usr/share/cephadm-ansible/hosts. The following example shows the structure of a typical inventory file:

   Example

   [ceph-admin@admin ~]$ cat hosts
   
   host02
   host03
   host04
   
   [admin]
   host01

   Note

   If you have previously added the new host to the Ansible inventory file and run the preflight playbook on the host, skip to step 4.

4. Run the preflight playbook with the --limit option:

   Syntax

   ansible-playbook -i INVENTORY_FILE cephadm-preflight.yml --extra-vars "ceph_origin=rhcs" --limit NEWHOST

   Example

   [ceph-admin@admin cephadm-ansible]$ ansible-playbook -i hosts cephadm-preflight.yml --extra-vars "ceph_origin=rhcs" --limit host02

   The preflight playbook installs podman, lvm2, chrony, and cephadm on the new host. After installation is complete, cephadm resides in the /usr/sbin/ directory.

5. From the bootstrap node, use the cephadm orchestrator to add the new host to the storage cluster:

   Syntax

   ceph orch host add NEWHOST

   Example

   [ceph: root@host01 /]# ceph orch host add host02
   Added host 'host02' with addr '10.10.128.69'
   [ceph: root@host01 /]# ceph orch host add host03
   Added host 'host03' with addr '10.10.128.70'

6. Optional: You can also add nodes by IP address, before and after you run the preflight playbook. If you do not have DNS configured in your storage cluster environment, you can add the hosts by IP address, along with the host names.

   Syntax

   ceph orch host add HOSTNAME IP_ADDRESS

   Example

   [ceph: root@host01 /]# ceph orch host add host02 10.10.128.69
   Added host 'host02' with addr '10.10.128.69'

   Verification

   • View the status of the storage cluster and verify that the new host has been added. The STATUS of the hosts is blank, in the output of the ceph orch host ls command.

     Example

     [ceph: root@host01 /]# ceph orch host ls

Procedure

1. Apply the five Monitor daemons to five random hosts in the storage cluster:

   ceph orch apply mon 5

2. Disable automatic Monitor deployment:

   ceph orch apply mon --unmanaged

Procedure

1. Navigate to the cephadm-ansible directory.

2. Generate a new SSH key:

   Example

   [ceph-admin@admin cephadm-ansible]$ ceph cephadm generate-key

3. Retrieve the public portion of the SSH key:

   Example

   [ceph-admin@admin cephadm-ansible]$ ceph cephadm get-pub-key

4. Delete the currently stored SSH key:

   Example

   [ceph-admin@admin cephadm-ansible]$ceph cephadm clear-key

5. Restart the mgr daemon to reload the configuration:

   Example

   [ceph-admin@admin cephadm-ansible]$ ceph mgr fail

Procedure

1. Use ceph orch host ls to view the hosts in your storage cluster:

   Example

   [root@host01 ~]# ceph orch host ls
   HOST   ADDR   LABELS  STATUS
   host01        mon,mgr,_admin
   host02        mon
   host03        mon,mgr
   host04
   host05
   host06

2. Use the _admin label to designate the admin host in your storage cluster. For best results, this host should have both Monitor and Manager daemons running.

   Syntax

   ceph orch host label add HOSTNAME _admin

   Example

   [root@host01 ~]#  ceph orch host label add host03 _admin

3. Verify that the admin host has the _admin label.

   Example

   [root@host01 ~]#  ceph orch host ls
   HOST   ADDR   LABELS  STATUS
   host01        mon,mgr,_admin
   host02        mon
   host03        mon,mgr,_admin
   host04
   host05
   host06

4. Log in to the admin node to manage the storage cluster.

Procedure

1. List the available devices to deploy OSDs:

   Syntax

   ceph orch device ls [--hostname=HOSTNAME1 HOSTNAME2] [--wide] [--refresh]

   Example

   [ceph: root@host01 /]# ceph orch device ls --wide --refresh

2. You can either deploy the OSDs on specific hosts or on all the available devices:

   • To create an OSD from a specific device on a specific host:

     Syntax

     ceph orch daemon add osd HOSTNAME:DEVICE_PATH

     Example

     [ceph: root@host01 /]# ceph orch daemon add osd host02:/dev/sdb

   • To deploy OSDs on any available and unused devices, use the --all-available-devices option.

     Example

     [ceph: root@host01 /]# ceph orch apply osd --all-available-devices

Procedure

1. Navigate to the /usr/share/cephadm-ansible directory.

2. Run the cephadm-clients.yml playbook on the initial host in the group of clients. Use the full path name to the admin keyring on the admin host for PATH_TO_KEYRING. Optional: If you want to specify an existing configuration file to use, specify the full path to the configuration file for CONFIG-FILE. Use the Ansible group name for the group of clients for ANSIBLE_GROUP_NAME. Use the FSID of the cluster where the admin keyring and configuration files are stored for FSID. The default path for the FSID is /var/lib/ceph/.

   Syntax

   ansible-playbook -i hosts cephadm-clients.yml -extra-vars '{"fsid":"FSID", "client_group":"ANSIBLE_GROUP_NAME", "keyring":"PATH_TO_KEYRING", "conf":"CONFIG_FILE"}'

   Example

   [ceph-admin@admin cephadm-ansible]$ ansible-playbook -i hosts cephadm-clients.yml --extra-vars '{"fsid":"be3ca2b2-27db-11ec-892b-005056833d58","client_group":"fs_clients","keyring":"/etc/ceph/fs.keyring", "conf": "/etc/ceph/ceph.conf"}'

Procedure

1. Disable cephadm to stop all the orchestration operations to avoid deploying new daemons:

   Example

   [ceph: root#host01 /]# ceph mgr module disable cephadm

2. Get the FSID of the cluster:

   Example

   [ceph: root#host01 /]# ceph fsid

3. Exit the cephadm shell.

   Example

   [ceph: root@host01 /]# exit

4. Purge the Ceph daemons from all hosts in the cluster:

   Syntax

   cephadm rm-cluster --force --zap-osds  --fsid FSID

   Example

   [root@host01 ~]# cephadm rm-cluster --force --zap-osds  --fsid a6ca415a-cde7-11eb-a41a-002590fc2544

Procedure

1. As an Ansible user, navigate to the /usr/share/cephadm-ansible directory on the Ansible administration node:

   Example

    [ceph-admin@admin ~]$ cd /usr/share/cephadm-ansible

2. Open and edit the hosts inventory file and add the [clients] group and clients to your inventory:

   Example

   host02
   host03
   host04
   
   [admin]
   host01
   
   [clients]
   client01
   client02
   client03

3. Run the cephadm-preflight.yml playbook to install the prerequisites on the clients:

   Syntax

   ansible-playbook -i INVENTORY_FILE cephadm-preflight.yml --limit CLIENT_GROUP_NAME|CLIENT_NODE_NAME

   Example

   [ceph-admin@admin cephadm-ansible]$ ansible-playbook -i hosts cephadm-preflight.yml --limit clients

4. Run the cephadm-clients.yml playbook to distribute the keyring and Ceph configuration files to a set of clients.

   1. To copy the keyring with a custom destination keyring name:

      Syntax

      ansible-playbook -i INVENTORY_FILE cephadm-clients.yml --extra-vars '{"fsid":"FSID","keyring":"KEYRING_PATH","client_group":"CLIENT_GROUP_NAME","conf":"CEPH_CONFIGURATION_PATH","keyring_dest":"KEYRING_DESTINATION_PATH"}'

      • Replace INVENTORY_FILE with the Ansible inventory file name.
      • Replace FSID with the FSID of the cluster.
      • Replace KEYRING_PATH with the full path name to the keyring on the admin host that you want to copy to the client.
      • Optional: Replace CLIENT_GROUP_NAME with the Ansible group name for the clients to set up.
      • Optional: Replace CEPH_CONFIGURATION_PATH with the full path to the Ceph configuration file on the admin node.

      • Optional: Replace KEYRING_DESTINATION_PATH with the full path name of the destination where the keyring will be copied.

        Note

        If you do not specify a configuration file with the conf option when you run the playbook, the playbook generates and distributes a minimal configuration file. By default, the generated file is located at /etc/ceph/ceph.conf.

        Example

        [ceph-admin@host01 cephadm-ansible]$ ansible-playbook -i hosts cephadm-clients.yml --extra-vars '{"fsid":"266ee7a8-2a05-11eb-b846-5254002d4916","keyring":"/etc/ceph/ceph.client.admin.keyring","client_group":"clients","conf":"/etc/ceph/ceph.conf","keyring_dest":"/etc/ceph/custom.name.ceph.keyring"}'

   2. To copy a keyring with the default destination keyring name of ceph.keyring and using the default group of clients:

      Syntax

      ansible-playbook -i INVENTORY_FILE cephadm-clients.yml --extra-vars '{"fsid":"FSID","keyring":"KEYRING_PATH","conf":"CONF_PATH"}'

Procedure

1. Log in to the Ansible administration node.

2. Navigate to the /usr/share/cephadm-ansible directory on the Ansible administration node:

   Example

   [ceph-admin@admin ~]$ cd /usr/share/cephadm-ansible

3. Create the hosts file and add hosts, labels, and monitor IP address of the first host in the storage cluster:

   Syntax

   sudo vi INVENTORY_FILE
   
   HOST1 labels="['LABEL1', 'LABEL2']"
   HOST2 labels="['LABEL1', 'LABEL2']"
   HOST3 labels="['LABEL1']"
   
   [admin]
   ADMIN_HOST monitor_address=MONITOR_IP_ADDRESS labels="['ADMIN_LABEL', 'LABEL1', 'LABEL2']"

   Example

   [ceph-admin@admin cephadm-ansible]$ sudo vi hosts
   
   host02 labels="['mon', 'mgr']"
   host03 labels="['mon', 'mgr']"
   host04 labels="['osd']"
   host05 labels="['osd']"
   host06 labels="['osd']"
   
   [admin]
   host01 monitor_address=10.10.128.68 labels="['_admin', 'mon', 'mgr']"

4. Run the preflight playbook:

   Syntax

   ansible-playbook -i INVENTORY_FILE cephadm-preflight.yml --extra-vars "ceph_origin=rhcs"

   Example

   [ceph-admin@admin cephadm-ansible]$ ansible-playbook -i hosts cephadm-preflight.yml --extra-vars "ceph_origin=rhcs"

5. Create a playbook to bootstrap your cluster:

   Syntax

   sudo vi PLAYBOOK_FILENAME.yml
   
   ---
   - name: NAME_OF_PLAY
     hosts: BOOTSTRAP_HOST
     become: USE_ELEVATED_PRIVILEGES
     gather_facts: GATHER_FACTS_ABOUT_REMOTE_HOSTS
     tasks:
       -name: NAME_OF_TASK
        cephadm_registry_login:
          state: STATE
          registry_url: REGISTRY_URL
          registry_username: REGISTRY_USER_NAME
          registry_password: REGISTRY_PASSWORD
   
       - name: NAME_OF_TASK
         cephadm_bootstrap:
           mon_ip: "{{ monitor_address }}"
           dashboard_user: DASHBOARD_USER
           dashboard_password: DASHBOARD_PASSWORD
           allow_fqdn_hostname: ALLOW_FQDN_HOSTNAME
           cluster_network: NETWORK_CIDR

   Example

   [ceph-admin@admin cephadm-ansible]$ sudo vi bootstrap.yml
   
   ---
   - name: bootstrap the cluster
     hosts: host01
     become: true
     gather_facts: false
     tasks:
       - name: login to registry
         cephadm_registry_login:
           state: login
           registry_url: registry.redhat.io
           registry_username: user1
           registry_password: mypassword1
   
       - name: bootstrap initial cluster
         cephadm_bootstrap:
           mon_ip: "{{ monitor_address }}"
           dashboard_user: mydashboarduser
           dashboard_password: mydashboardpassword
           allow_fqdn_hostname: true
           cluster_network: 10.10.128.0/28

6. Run the playbook:

   Syntax

   ansible-playbook -i INVENTORY_FILE PLAYBOOK_FILENAME.yml -vvv

   Example

   [ceph-admin@admin cephadm-ansible]$ ansible-playbook -i hosts bootstrap.yml -vvv

Procedure

1. Use the following procedure to add new hosts to the cluster:

   1. Log in to the Ansible administration node.

   2. Navigate to the /usr/share/cephadm-ansible directory on the Ansible administration node:

      Example

      [ceph-admin@admin ~]$ cd /usr/share/cephadm-ansible

   3. Add the new hosts and labels to the Ansible inventory file.

      Syntax

      sudo vi INVENTORY_FILE
      
      NEW_HOST1 labels="['LABEL1', 'LABEL2']"
      NEW_HOST2 labels="['LABEL1', 'LABEL2']"
      NEW_HOST3 labels="['LABEL1']"
      
      [admin]
      ADMIN_HOST monitor_address=MONITOR_IP_ADDRESS labels="['ADMIN_LABEL', 'LABEL1', 'LABEL2']"

      Example

      [ceph-admin@admin cephadm-ansible]$ sudo vi hosts
      
      host02 labels="['mon', 'mgr']"
      host03 labels="['mon', 'mgr']"
      host04 labels="['osd']"
      host05 labels="['osd']"
      host06 labels="['osd']"
      
      [admin]
      host01 monitor_address= 10.10.128.68 labels="['_admin', 'mon', 'mgr']"

   4. Run the preflight playbook with the --limit option:

      Syntax

      ansible-playbook -i INVENTORY_FILE cephadm-preflight.yml --extra-vars "ceph_origin=rhcs" --limit NEWHOST

      Example

      [ceph-admin@admin cephadm-ansible]$ ansible-playbook -i hosts cephadm-preflight.yml --extra-vars "ceph_origin=rhcs" --limit host02

      The preflight playbook installs podman, lvm2, chrony, and cephadm on the new host. After installation is complete, cephadm resides in the /usr/sbin/ directory.

   5. Create a playbook to add the new hosts to the cluster:

      Syntax

      sudo vi PLAYBOOK_FILENAME.yml
      
      ---
      - name: PLAY_NAME
        hosts: HOSTS_OR_HOST_GROUPS
        become: USE_ELEVATED_PRIVILEGES
        gather_facts: GATHER_FACTS_ABOUT_REMOTE_HOSTS
        tasks:
          - name: NAME_OF_TASK
            ceph_orch_host:
              name: "{{ ansible_facts['hostname'] }}"
              address: "{{ ansible_facts['default_ipv4']['address'] }}"
              labels: "{{ labels }}"
            delegate_to: HOST_TO_DELEGATE_TASK_TO
      
          - name: NAME_OF_TASK
            when: inventory_hostname in groups['admin']
            ansible.builtin.shell:
              cmd: CEPH_COMMAND_TO_RUN
            register: REGISTER_NAME
      
          - name: NAME_OF_TASK
            when: inventory_hostname in groups['admin']
            debug:
              msg: "{{ REGISTER_NAME.stdout }}"

      Note

      By default, Ansible executes all tasks on the host that matches the hosts line of your playbook. The ceph orch commands must run on the host that contains the admin keyring and the Ceph configuration file. Use the delegate_to keyword to specify the admin host in your cluster.

      Example

      [ceph-admin@admin cephadm-ansible]$ sudo vi add-hosts.yml
      
      ---
      - name: add additional hosts to the cluster
        hosts: all
        become: true
        gather_facts: true
        tasks:
          - name: add hosts to the cluster
            ceph_orch_host:
              name: "{{ ansible_facts['hostname'] }}"
              address: "{{ ansible_facts['default_ipv4']['address'] }}"
              labels: "{{ labels }}"
            delegate_to: host01
      
          - name: list hosts in the cluster
            when: inventory_hostname in groups['admin']
            ansible.builtin.shell:
              cmd: ceph orch host ls
            register: host_list
      
          - name: print current list of hosts
            when: inventory_hostname in groups['admin']
            debug:
              msg: "{{ host_list.stdout }}"

      In this example, the playbook adds the new hosts to the cluster and displays a current list of hosts.

   6. Run the playbook to add additional hosts to the cluster:

      Syntax

      ansible-playbook -i INVENTORY_FILE PLAYBOOK_FILENAME.yml

      Example

      [ceph-admin@admin cephadm-ansible]$ ansible-playbook -i hosts add-hosts.yml

2. Use the following procedure to remove hosts from the cluster:

   1. Log in to the Ansible administration node.

   2. Navigate to the /usr/share/cephadm-ansible directory on the Ansible administration node:

      Example

      [ceph-admin@admin ~]$ cd /usr/share/cephadm-ansible

   3. Create a playbook to remove a host or hosts from the cluster:

      Syntax

      sudo vi PLAYBOOK_FILENAME.yml
      
      ---
      - name: NAME_OF_PLAY
        hosts: ADMIN_HOST
        become: USE_ELEVATED_PRIVILEGES
        gather_facts: GATHER_FACTS_ABOUT_REMOTE_HOSTS
        tasks:
          - name: NAME_OF_TASK
            ceph_orch_host:
              name: HOST_TO_REMOVE
              state: STATE
      
          - name: NAME_OF_TASK
            ceph_orch_host:
              name: HOST_TO_REMOVE
              state: STATE
            retries: NUMBER_OF_RETRIES
            delay: DELAY
            until: CONTINUE_UNTIL
            register: REGISTER_NAME
      
          - name: NAME_OF_TASK
            ansible.builtin.shell:
              cmd: ceph orch host ls
            register: REGISTER_NAME
      
          - name: NAME_OF_TASK
              debug:
                msg: "{{ REGISTER_NAME.stdout }}"

      Example

      [ceph-admin@admin cephadm-ansible]$ sudo vi remove-hosts.yml
      
      ---
      - name: remove host
        hosts: host01
        become: true
        gather_facts: true
        tasks:
          - name: drain host07
            ceph_orch_host:
              name: host07
              state: drain
      
          - name: remove host from the cluster
            ceph_orch_host:
              name: host07
              state: absent
            retries: 20
            delay: 1
            until: result is succeeded
            register: result
      
           - name: list hosts in the cluster
             ansible.builtin.shell:
               cmd: ceph orch host ls
             register: host_list
      
           - name: print current list of hosts
             debug:
               msg: "{{ host_list.stdout }}"

      In this example, the playbook tasks drain all daemons on host07, removes the host from the cluster, and displays a current list of hosts.

   4. Run the playbook to remove host from the cluster:

      Syntax

      ansible-playbook -i INVENTORY_FILE PLAYBOOK_FILENAME.yml

      Example

      [ceph-admin@admin cephadm-ansible]$ ansible-playbook -i hosts remove-hosts.yml

Procedure

1. Log in to the Ansible administration node.

2. Navigate to the /usr/share/cephadm-ansible directory on the Ansible administration node:

   Example

   [ceph-admin@admin ~]$ cd /usr/share/cephadm-ansible

3. Create a playbook with configuration changes:

   Syntax

   sudo vi PLAYBOOK_FILENAME.yml
   
   ---
   - name: PLAY_NAME
     hosts: ADMIN_HOST
     become: USE_ELEVATED_PRIVILEGES
     gather_facts: GATHER_FACTS_ABOUT_REMOTE_HOSTS
     tasks:
       - name: NAME_OF_TASK
         ceph_config:
           action: GET_OR_SET
           who: DAEMON_TO_SET_CONFIGURATION_TO
           option: CEPH_CONFIGURATION_OPTION
           value: VALUE_OF_PARAMETER_TO_SET
   
       - name: NAME_OF_TASK
         ceph_config:
           action: GET_OR_SET
           who: DAEMON_TO_SET_CONFIGURATION_TO
           option: CEPH_CONFIGURATION_OPTION
         register: REGISTER_NAME
   
       - name: NAME_OF_TASK
         debug:
           msg: "MESSAGE_TO_DISPLAY {{ REGISTER_NAME.stdout }}"

   Example

   [ceph-admin@admin cephadm-ansible]$ sudo vi change_configuration.yml
   
   ---
   - name: set pool delete
     hosts: host01
     become: true
     gather_facts: false
     tasks:
       - name: set the allow pool delete option
         ceph_config:
           action: set
           who: mon
           option: mon_allow_pool_delete
           value: true
   
       - name: get the allow pool delete setting
         ceph_config:
           action: get
           who: mon
           option: mon_allow_pool_delete
         register: verify_mon_allow_pool_delete
   
       - name: print current mon_allow_pool_delete setting
         debug:
           msg: "the value of 'mon_allow_pool_delete' is {{ verify_mon_allow_pool_delete.stdout }}"

   In this example, the playbook first sets the mon_allow_pool_delete option to false. The playbook then gets the current mon_allow_pool_delete setting and displays the value in the Ansible output.

4. Run the playbook:

   Syntax

   ansible-playbook -i INVENTORY_FILE _PLAYBOOK_FILENAME.yml

   Example

   [ceph-admin@admin cephadm-ansible]$ ansible-playbook -i hosts change_configuration.yml

Procedure

1. Log in to the Ansible administration node.

2. Navigate to the /usr/share/cephadm-ansible directory on the Ansible administration node:

   Example

   [ceph-admin@admin ~]$ cd /usr/share/cephadm-ansible

3. Create a playbook with the service specifications:

   Syntax

   sudo vi PLAYBOOK_FILENAME.yml
   
   ---
   - name: PLAY_NAME
     hosts: HOSTS_OR_HOST_GROUPS
     become: USE_ELEVATED_PRIVILEGES
     gather_facts: GATHER_FACTS_ABOUT_REMOTE_HOSTS
     tasks:
       - name: NAME_OF_TASK
         ceph_orch_apply:
           spec: |
             service_type: SERVICE_TYPE
             service_id: UNIQUE_NAME_OF_SERVICE
             placement:
               host_pattern: 'HOST_PATTERN_TO_SELECT_HOSTS'
               label: LABEL
             spec:
               SPECIFICATION_OPTIONS:

   Example

   [ceph-admin@admin cephadm-ansible]$ sudo vi deploy_osd_service.yml
   
   ---
   - name: deploy osd service
     hosts: host01
     become: true
     gather_facts: true
     tasks:
       - name: apply osd spec
         ceph_orch_apply:
           spec: |
             service_type: osd
             service_id: osd
             placement:
               host_pattern: '*'
               label: osd
             spec:
               data_devices:
                 all: true

   In this example, the playbook deploys the Ceph OSD service on all hosts with the label osd.

4. Run the playbook:

   Syntax

   ansible-playbook -i INVENTORY_FILE _PLAYBOOK_FILENAME.yml

   Example

   [ceph-admin@admin cephadm-ansible]$ ansible-playbook -i hosts deploy_osd_service.yml

Procedure

1. Log in to the Ansible administration node.

2. Navigate to the /usr/share/cephadm-ansible directory on the Ansible administration node:

   Example

   [ceph-admin@admin ~]$ cd /usr/share/cephadm-ansible

3. Create a playbook with daemon state changes:

   Syntax

   sudo vi PLAYBOOK_FILENAME.yml
   
   ---
   - name: PLAY_NAME
     hosts: ADMIN_HOST
     become: USE_ELEVATED_PRIVILEGES
     gather_facts: GATHER_FACTS_ABOUT_REMOTE_HOSTS
     tasks:
       - name: NAME_OF_TASK
         ceph_orch_daemon:
           state: STATE_OF_SERVICE
           daemon_id: DAEMON_ID
           daemon_type: TYPE_OF_SERVICE

   Example

   [ceph-admin@admin cephadm-ansible]$ sudo vi restart_services.yml
   
   ---
   - name: start and stop services
     hosts: host01
     become: true
     gather_facts: false
     tasks:
       - name: start osd.0
         ceph_orch_daemon:
           state: started
           daemon_id: 0
           daemon_type: osd
   
       - name: stop mon.host02
         ceph_orch_daemon:
           state: stopped
           daemon_id: host02
           daemon_type: mon

   In this example, the playbook starts the OSD with an ID of 0 and stops a Ceph Monitor with an id of host02.

4. Run the playbook:

   Syntax

   ansible-playbook -i INVENTORY_FILE _PLAYBOOK_FILENAME.yml

   Example

   [ceph-admin@admin cephadm-ansible]$ ansible-playbook -i hosts restart_services.yml