Deploying an overcloud with containerized Red Hat Ceph

Procedure

1. After you create the inventory file, save the file to the home directory of the stack user (/home/stack/instackenv.json).

2. Initialize the stack user, then import the instackenv.json inventory file into the director:

   $ source ~/stackrc
   $ openstack overcloud node import ~/instackenv.json

   The openstack overcloud node import command imports the inventory file and registers each node with the director.

3. Assign the kernel and ramdisk images to each node:

1. Trigger hardware introspection to retrieve the hardware attributes of each node:

   $ openstack overcloud node introspect --all-manageable --provide

   • The --all-manageable option introspects only the nodes that are in a managed state. In this example, all nodes are in a managed state.

   • The --provide option resets all nodes to an active state after introspection.

     Important

     Ensure that this process completes successfully. This process usually takes 15 minutes for bare metal nodes.

2. Retrieve a list of your nodes to identify their UUIDs:

   $ openstack baremetal node list

3. Add a profile option to the properties/capabilities parameter for each node to manually tag a node to a specific profile. The addition of the profile option tags the nodes into each respective profile.

   Note

   As an alternative to manual tagging, use the Automated Health Check (AHC) Tools to automatically tag larger numbers of nodes based on benchmarking data.

   For example, a typical deployment contains three profiles: control, compute, and ceph-storage. Run the following commands to tag three nodes for each profile:

   $ ironic node-update 1a4e30da-b6dc-499d-ba87-0bd8a3819bc0 add properties/capabilities='profile:control,boot_option:local'
   $ ironic node-update 6faba1a9-e2d8-4b7c-95a2-c7fbdc12129a add properties/capabilities='profile:control,boot_option:local'
   $ ironic node-update 5e3b2f50-fcd9-4404-b0a2-59d79924b38e add properties/capabilities='profile:control,boot_option:local'
   $ ironic node-update 484587b2-b3b3-40d5-925b-a26a2fa3036f add properties/capabilities='profile:compute,boot_option:local'
   $ ironic node-update d010460b-38f2-4800-9cc4-d69f0d067efe add properties/capabilities='profile:compute,boot_option:local'
   $ ironic node-update d930e613-3e14-44b9-8240-4f3559801ea6 add properties/capabilities='profile:compute,boot_option:local'
   $ ironic node-update da0cc61b-4882-45e0-9f43-fab65cf4e52b add properties/capabilities='profile:ceph-storage,boot_option:local'
   $ ironic node-update b9f70722-e124-4650-a9b1-aade8121b5ed add properties/capabilities='profile:ceph-storage,boot_option:local'
   $ ironic node-update 68bf8f29-7731-4148-ba16-efb31ab8d34f add properties/capabilities='profile:ceph-storage,boot_option:local'

   Tip

   You can also configure a new custom profile that you can use to tag a node for the Ceph MON and Ceph MDS services. See Chapter 3, Deploying Ceph services on dedicated nodes for details.

Procedure

1. Check the disk information from the hardware introspection of each node. Run the following command to display the disk information of a node:

   (undercloud) $ openstack baremetal introspection data save 1a4e30da-b6dc-499d-ba87-0bd8a3819bc0 | jq ".inventory.disks"

   For example, the data for one node might show three disks:

   [
     {
       "size": 299439751168,
       "rotational": true,
       "vendor": "DELL",
       "name": "/dev/sda",
       "wwn_vendor_extension": "0x1ea4dcc412a9632b",
       "wwn_with_extension": "0x61866da04f3807001ea4dcc412a9632b",
       "model": "PERC H330 Mini",
       "wwn": "0x61866da04f380700",
       "serial": "61866da04f3807001ea4dcc412a9632b"
     }
     {
       "size": 299439751168,
       "rotational": true,
       "vendor": "DELL",
       "name": "/dev/sdb",
       "wwn_vendor_extension": "0x1ea4e13c12e36ad6",
       "wwn_with_extension": "0x61866da04f380d001ea4e13c12e36ad6",
       "model": "PERC H330 Mini",
       "wwn": "0x61866da04f380d00",
       "serial": "61866da04f380d001ea4e13c12e36ad6"
     }
     {
       "size": 299439751168,
       "rotational": true,
       "vendor": "DELL",
       "name": "/dev/sdc",
       "wwn_vendor_extension": "0x1ea4e31e121cfb45",
       "wwn_with_extension": "0x61866da04f37fc001ea4e31e121cfb45",
       "model": "PERC H330 Mini",
       "wwn": "0x61866da04f37fc00",
       "serial": "61866da04f37fc001ea4e31e121cfb45"
     }
   ]

2. Run the openstack baremetal node set --property root_device= command to set the root disk for a node. Include the most appropriate hardware attribute value to define the root disk.

   (undercloud) $ openstack baremetal node set --property root_device=’{“serial”:”<serial_number>”} <node-uuid>

   For example, to set the root device to disk 2, which has the serial number 61866da04f380d001ea4e13c12e36ad6 run the following command:

   (undercloud) $ openstack baremetal node set --property root_device='{"serial": "61866da04f380d001ea4e13c12e36ad6"}' 1a4e30da-b6dc-499d-ba87-0bd8a3819bc0

Procedure

1. To configure director to use the overcloud-minimal image, create an environment file that contains the following image definition:

   parameter_defaults:
     <roleName>Image: overcloud-minimal

2. Replace <roleName> with the name of the role and append Image to the name of the role. The following example shows an overcloud-minimal image for Ceph storage nodes:

   parameter_defaults:
     CephStorageImage: overcloud-minimal

3. Pass the environment file to the openstack overcloud deploy command.

Procedure

1. Make a copy of the roles_data.yaml file in /home/stack/templates/ so that you can add custom roles:

   $ cp /usr/share/openstack-tripleo-heat-templates/roles_data.yaml /home/stack/templates/roles_data_custom.yaml

2. Include the new custom role file in the openstack overcloud deploy command.

Procedure

1. Open the /home/stack/templates/roles_data_custom.yaml file.
2. Remove the service entry for the Ceph MON service (namely, OS::TripleO::Services::CephMon) from the Controller role.

3. Add the OS::TripleO::Services::CephClient service to the Controller role:

   [...]
   - name: Controller # the 'primary' role goes first
     CountDefault: 1
     ServicesDefault:
       - OS::TripleO::Services::CACerts
       - OS::TripleO::Services::CephMds
       - OS::TripleO::Services::CephClient
       - OS::TripleO::Services::CephExternal
       - OS::TripleO::Services::CephRbdMirror
       - OS::TripleO::Services::CephRgw
       - OS::TripleO::Services::CinderApi
   [...]

4. At the end of the roles_data_custom.yaml file, add a custom CephMon role that contains the Ceph MON service and all the other required node services:

   - name: CephMon
     ServicesDefault:
       # Common Services
       - OS::TripleO::Services::AuditD
       - OS::TripleO::Services::CACerts
       - OS::TripleO::Services::CertmongerUser
       - OS::TripleO::Services::Collectd
       - OS::TripleO::Services::Docker
       - OS::TripleO::Services::FluentdClient
       - OS::TripleO::Services::Kernel
       - OS::TripleO::Services::Ntp
       - OS::TripleO::Services::ContainersLogrotateCrond
       - OS::TripleO::Services::SensuClient
       - OS::TripleO::Services::Snmp
       - OS::TripleO::Services::Timezone
       - OS::TripleO::Services::TripleoFirewall
       - OS::TripleO::Services::TripleoPackages
       - OS::TripleO::Services::Tuned
       # Role-Specific Services
       - OS::TripleO::Services::CephMon

5. Run the openstack flavor create command to define a new flavor named ceph-mon for the CephMon role:

   $ openstack flavor create --id auto --ram 6144 --disk 40 --vcpus 4 ceph-mon

   Note

   For details about this command, run openstack flavor create --help.

6. Map this flavor to a new profile, also named ceph-mon:

   $ openstack flavor set --property "cpu_arch"="x86_64" --property "capabilities:boot_option"="local" --property "capabilities:profile"="ceph-mon" ceph-mon

   Note

   For details about this command, run openstack flavor set --help.

7. Tag nodes into the new ceph-mon profile:

   $ ironic node-update UUID add properties/capabilities='profile:ceph-mon,boot_option:local'

8. Add the following configuration to the node-info.yaml file to associate the ceph-mon flavor with the CephMon role:

   parameter_defaults:
     OvercloudCephMonFlavor: CephMon
     CephMonCount: 3

Procedure

1. Open the /home/stack/templates/roles_data_custom.yaml file.

2. Remove the service entry for the Ceph MDS service (namely, OS::TripleO::Services::CephMds) from the Controller role:

   [...]
   - name: Controller # the 'primary' role goes first
     CountDefault: 1
     ServicesDefault:
       - OS::TripleO::Services::CACerts
       # - OS::TripleO::Services::CephMds 1
       - OS::TripleO::Services::CephMon
       - OS::TripleO::Services::CephExternal
       - OS::TripleO::Services::CephRbdMirror
       - OS::TripleO::Services::CephRgw
       - OS::TripleO::Services::CinderApi
   [...]

   1

   Comment out this line. In the next step, you add this service to the new custom role.

3. At the end of the roles_data_custom.yaml file, add a custom CephMDS role containing the Ceph MDS service and all the other required node services:

   - name: CephMDS
     ServicesDefault:
       # Common Services
       - OS::TripleO::Services::AuditD
       - OS::TripleO::Services::CACerts
       - OS::TripleO::Services::CertmongerUser
       - OS::TripleO::Services::Collectd
       - OS::TripleO::Services::Docker
       - OS::TripleO::Services::FluentdClient
       - OS::TripleO::Services::Kernel
       - OS::TripleO::Services::Ntp
       - OS::TripleO::Services::ContainersLogrotateCrond
       - OS::TripleO::Services::SensuClient
       - OS::TripleO::Services::Snmp
       - OS::TripleO::Services::Timezone
       - OS::TripleO::Services::TripleoFirewall
       - OS::TripleO::Services::TripleoPackages
       - OS::TripleO::Services::Tuned
       # Role-Specific Services
       - OS::TripleO::Services::CephMds
       - OS::TripleO::Services::CephClient 1

   1

   The Ceph MDS service requires the admin keyring, which you can set with either the Ceph MON or Ceph Client service. If you deploy Ceph MDS on a dedicated node without the Ceph MON service, you must also include the Ceph Client service in the new CephMDS role.

4. Run the openstack flavor create command to define a new flavor named ceph-mds for this role:

   $ openstack flavor create --id auto --ram 6144 --disk 40 --vcpus 4 ceph-mds

   Note

   For details about this command, run openstack flavor create --help.

5. Map the new ceph-mds flavor to a new profile, also named ceph-mds:

   $ openstack flavor set --property "cpu_arch"="x86_64" --property "capabilities:boot_option"="local" --property "capabilities:profile"="ceph-mds" ceph-mds

   Note

   For details about this command, run openstack flavor set --help.

6. Tag nodes into the new ceph-mds profile:

1. Remove the first two lines (parameter_defaults: and NodeDataLookup:) from the file temporarily.
2. Run cat node-spec-overrides.yaml | jq .

Procedure

1. Update the CephAnsibleExtraConfig parameter to change the default delay and retry values:

   parameter_defaults:
     CephAnsibleExtraConfig:
       health_osd_check_delay: 40
       health_osd_check_retries: 30
       health_mon_check_delay: 20
       health_mon_check_retries: 10

   This example makes the cluster check 30 times and wait 40 seconds between each check for the Ceph OSDs, and check 20 times and wait 10 seconds between each check for the Ceph MONs.

2. To incorporate the changes, pass the updated yaml file with -e using openstack overcloud deploy.

Procedure

1. Declare the OSDs/journal mapping:

   parameter_defaults:
     CephAnsibleDisksConfig:
      devices:
      - /dev/sda
      - /dev/sdb
      dedicated_devices:
      - /dev/sdc
      - /dev/sdc
      osd_scenario: non-collocated
      journal_size: 8192

Procedure

1. Set the default replication factor to two. This example splits four nodes into two different roots.

   parameter_defaults:
     CephPoolDefaultSize: 2

Procedure

1. Run the following commands to retrieve the UUIDs of the four nodes:

   for ((x=1; x<=4; x++)); \
   { echo "Node overcloud-ceph0${x}"; \
   openstack baremetal introspection data save overcloud-ceph0${x} | jq .extra.system.product.uuid; }
   Node overcloud-ceph01
   "32C2BC31-F6BB-49AA-971A-377EFDFDB111"
   Node overcloud-ceph02
   "76B4C69C-6915-4D30-AFFD-D16DB74F64ED"
   Node overcloud-ceph03
   "FECF7B20-5984-469F-872C-732E3FEF99BF"
   Node overcloud-ceph04
   "5FFEFA5F-69E4-4A88-B9EA-62811C61C8B3"

   Note

   In the example, overcloud-ceph0[1-4] are the Ironic nodes names; they will be deployed as lab-ceph0[1–4] (via HostnameMap.yaml).

2. Specify the node placement as follows:

   Root	Rack	Node
   standard_root	rack1_std	overcloud-ceph01 (lab-ceph01)
   	rack2_std	overcloud-ceph02 (lab-ceph02)
   fast_root	rack1_fast	overcloud-ceph03 (lab-ceph03)
   	rack2_fast	overcloud-ceph04 (lab-ceph04)

   Note

   You cannot have two buckets with the same name. Even if lab-ceph01 and lab-ceph03 are in the same physical rack, you cannot have two buckets called rack1. Therefore, we named them rack1_std and rack1_fast.

   Note

   This example demonstrates how to create a specific route called “standard_root” to illustrate multiple custom roots. However, you could have kept the HDDs OSD nodes in the default root.

3. Use the following NodeDataLookup syntax:

   NodeDataLookup: {"SYSTEM_UUID": {"osd_crush_location": {"root": "$MY_ROOT", "rack": "$MY_RACK", "host": "$OVERCLOUD_NODE_HOSTNAME"}}}

   Note

   You must specify the system UUID and then the CRUSH hierarchy from top to bottom. Also, the host parameter must point to the node’s overcloud host name, not the Bare Metal service (ironic) node name. To match the example configuration, enter the following:

   parameter_defaults:
     NodeDataLookup: {"32C2BC31-F6BB-49AA-971A-377EFDFDB111": {"osd_crush_location": {"root": "standard_root", "rack": "rack1_std", "host": "lab-ceph01"}},
        "76B4C69C-6915-4D30-AFFD-D16DB74F64ED": {"osd_crush_location": {"root": "standard_root", "rack": "rack2_std", "host": "lab-ceph02"}},
        "FECF7B20-5984-469F-872C-732E3FEF99BF": {"osd_crush_location": {"root": "fast_root", "rack": "rack1_fast", "host": "lab-ceph03"}},
        "5FFEFA5F-69E4-4A88-B9EA-62811C61C8B3": {"osd_crush_location": {"root": "fast_root", "rack": "rack2_fast", "host": "lab-ceph04"}}}

4. Enable CRUSH map management at the ceph-ansible level:

   parameter_defaults:
     CephAnsibleExtraConfig:
       create_crush_tree: true

5. Use scheduler hints to ensure the Bare Metal service node UUIDs correctly map to the hostnames:

   parameter_defaults:
     CephStorageCount: 4
     OvercloudCephStorageFlavor: ceph-storage
     CephStorageSchedulerHints:
       'capabilities:node': 'ceph-%index%'

6. Tag the Bare Metal service nodes with the corresponding hint:

   openstack baremetal node set --property capabilities='profile:ceph-storage,node:ceph-0,boot_option:local' overcloud-ceph01
   
   openstack baremetal node set --property capabilities=profile:ceph-storage,'node:ceph-1,boot_option:local' overcloud-ceph02
   
   openstack baremetal node set --property capabilities='profile:ceph-storage,node:ceph-2,boot_option:local' overcloud-ceph03
   
   openstack baremetal node set --property capabilities='profile:ceph-storage,node:ceph-3,boot_option:local' overcloud-ceph04

   Note

   For more information about predictive placement, see Assigning Specific Node IDs in the Advanced Overcloud Customization guide.

Procedure

1. Use the following syntax to define the CRUSH rules:

   parameter_defaults:
     CephAnsibleExtraConfig:
       crush_rules:
         - name: $RULE_NAME
           root: $ROOT_NAME
           type: $REPLICAT_DOMAIN
           default: true/false

   Note

   Setting the default parameter to true means that this rule will be used when you create a new pool without specifying any rule. There may only be one default rule.

   In the following example, rule standard points to the OSD nodes hosted on the standard_root with one replicate per rack. Rule fast points to the OSD nodes hosted on the standard_root with one replicate per rack:

   parameter_defaults:
     CephAnsibleExtraConfig:
       crush_rule_config: true
       crush_rules:
         - name: standard
           root: standard_root
           type: rack
           default: true
         - name: fast
           root: fast_root
           type: rack
           default: false

   Note

   You must set crush_rule_config to true.

Procedure

1. Use the following syntax to define or change a pool property:

   - name: $POOL_NAME
        pg_num: $PG_COUNT
        rule_name: $RULE_NAME
        application: rbd

2. List all OSP pools and set the appropriate rule (standard, in this case), and create a new pool called tier2 that uses the fast rule. This pool will be used by Block Storage (cinder).

   parameter_defaults:
     CephPools:
       - name: tier2
         pg_num: 64
         rule_name: fast
         application: rbd
   
       - name: volumes
         pg_num: 64
         rule_name: standard
         application: rbd
   
       - name: vms
         pg_num: 64
         rule_name: standard
         application: rbd
   
       - name: backups
         pg_num: 64
         rule_name: standard
         application: rbd
   
       - name: images
         pg_num: 64
         rule_name: standard
         application: rbd
   
       - name: metrics
         pg_num: 64
         rule_name: standard
         application: openstack_gnocchi

Procedure

1. Update cinder.conf as follows:

   parameter_defaults:
     CinderRbdExtraPools:
       - tier2

Procedure

1. Connect to a Ceph monitor node and run the following command:

   # ceph osd tree
   ID WEIGHT  TYPE NAME               UP/DOWN REWEIGHT PRIMARY-AFFINITY
   -7 0.39996 root standard_root
   -6 0.19998     rack rack1_std
   -5 0.19998         host lab-ceph02
    1 0.09999             osd.1            up  1.00000          1.00000
    4 0.09999             osd.4            up  1.00000          1.00000
   -9 0.19998     rack rack2_std
   -8 0.19998         host lab-ceph03
    0 0.09999             osd.0            up  1.00000          1.00000
    3 0.09999             osd.3            up  1.00000          1.00000
   -4 0.19998 root fast_root
   -3 0.19998     rack rack1_fast
   -2 0.19998         host lab-ceph01
    2 0.09999             osd.2            up  1.00000          1.00000
    5 0.09999             osd.5            up  1.00000          1.00000

1. Log in to a Controller node as the heat-admin user:

   $ nova list
   $ ssh heat-admin@192.168.0.25

2. Check the health of the cluster:

   $ sudo podman exec ceph-mon-$HOSTNAME ceph health

   If the cluster has no issues, the command reports back HEALTH_OK. This means the cluster is safe to use.

3. Log in to an overcloud node that runs the Ceph monitor service and check the status of all OSDs in the cluster:

   $ sudo podman exec ceph-mon-$HOSTNAME ceph osd tree

4. Check the status of the Ceph Monitor quorum:

   $ sudo ceph quorum_status

   This shows the monitors participating in the quorum and which one is the leader.

5. Verify that all Ceph OSDs are running:

   $ ceph osd stat

Procedure

1. Log in to a Ceph MON or Controller node and disable Ceph Storage cluster rebalancing temporarily:

   $ sudo podman exec -it ceph-mon-controller-0 ceph osd set noout
   $ sudo podman exec -it ceph-mon-controller-0 ceph osd set norebalance

2. Select the first Ceph Storage node to reboot and log into the node.

3. Reboot the node:

   $ sudo reboot

4. Wait until the node boots.

5. Log in to the node and check the cluster status:

   $ sudo podman exec -it ceph-mon-controller-0 ceph status

   Check the pgmap reports all pgs as normal (active+clean).

6. Log out of the node, reboot the next node, and check its status. Repeat this process until you have rebooted all Ceph storage nodes.

7. When complete, log into a Ceph MON or Controller node and enable cluster rebalancing again:

   $ sudo podman exec -it ceph-mon-controller-0 ceph osd unset noout
   $ sudo podman exec -it ceph-mon-controller-0 ceph osd unset norebalance

8. Perform a final status check to verify the cluster reports HEALTH_OK:

   $ sudo podman exec -it ceph-mon-controller-0 ceph status

1. Log into the director host as the stack user and initialize your director configuration:

   $ source ~/stackrc

2. Define the hardware and power management details for the new nodes in a new node definition template; for example, instackenv-scale.json.

3. Import this file to the OpenStack director:

   $ openstack overcloud node import ~/instackenv-scale.json

   Importing the node definition template registers each node defined there to the director.

4. Assign the kernel and ramdisk images to all nodes:

   $ openstack overcloud node configure

1. Trigger hardware introspection to retrieve the hardware attributes of each node:

   $ openstack overcloud node introspect --all-manageable --provide

   • The --all-manageable option introspects only the nodes that are in a managed state. In this example, all nodes are in a managed state.

   • The --provide option resets all nodes to an active state after introspection.

     Important

     Ensure that this process completes successfully. This process usually takes 15 minutes for bare metal nodes.

2. Retrieve a list of your nodes to identify their UUIDs:

   $ openstack baremetal node list

3. Add a profile option to the properties/capabilities parameter for each node to manually tag a node to a specific profile. The addition of the profile option tags the nodes into each respective profile.

   Note

   As an alternative to manual tagging, use the Automated Health Check (AHC) Tools to automatically tag larger numbers of nodes based on benchmarking data.

   For example, the following commands tag three additional nodes with the ceph-storage profile:

   $ ironic node-update 551d81f5-4df2-4e0f-93da-6c5de0b868f7 add properties/capabilities='profile:ceph-storage,boot_option:local'
   $ ironic node-update 5e735154-bd6b-42dd-9cc2-b6195c4196d7 add properties/capabilities='profile:ceph-storage,boot_option:local'
   $ ironic node-update 1a2b090c-299d-4c20-a25d-57dd21a7085b add properties/capabilities='profile:ceph-storage,boot_option:local'

Procedure

1. Log in to a Controller node as the heat-admin user. The director stack user has an SSH key to access the heat-admin user.

2. List the OSD tree and find the OSDs for your node. For example, the node you want to remove might contain the following OSDs:

   -2 0.09998     host overcloud-cephstorage-0
   0 0.04999         osd.0                         up  1.00000          1.00000
   1 0.04999         osd.1                         up  1.00000          1.00000

3. Disable the OSDs on the Ceph Storage node. In this case, the OSD IDs are 0 and 1.

   [heat-admin@overcloud-controller-0 ~]$ sudo podman exec ceph-mon-<HOSTNAME> ceph osd out 0
   [heat-admin@overcloud-controller-0 ~]$ sudo podman exec ceph-mon-<HOSTNAME> ceph osd out 1

4. The Ceph Storage cluster begins rebalancing. Wait for this process to complete. Follow the status by using the following command:

   [heat-admin@overcloud-controller-0 ~]$ sudo podman exec ceph-mon-<HOSTNAME> ceph -w

5. After the Ceph cluster completes rebalancing, log in to the Ceph Storage node you are removing, in this case overcloud-cephstorage-0, as the heat-admin user and stop the node.

   [heat-admin@overcloud-cephstorage-0 ~]$ sudo systemctl disable ceph-osd@0
   [heat-admin@overcloud-cephstorage-0 ~]$ sudo systemctl disable ceph-osd@1

6. Stop the OSDs.

   [heat-admin@overcloud-cephstorage-0 ~]$ sudo systemctl stop ceph-osd@0
   [heat-admin@overcloud-cephstorage-0 ~]$ sudo systemctl stop ceph-osd@1

7. While logged in to the Controller node, remove the OSDs from the CRUSH map so that they no longer receive data.

   [heat-admin@overcloud-controller-0 ~]$ sudo podman exec ceph-mon-<HOSTNAME> ceph osd crush remove osd.0
   [heat-admin@overcloud-controller-0 ~]$ sudo podman exec ceph-mon-<HOSTNAME> ceph osd crush remove osd.1

8. Remove the OSD authentication key.

   [heat-admin@overcloud-controller-0 ~]$ sudo podman exec ceph-mon-<HOSTNAME> ceph auth del osd.0
   [heat-admin@overcloud-controller-0 ~]$ sudo podman exec ceph-mon-<HOSTNAME> ceph auth del osd.1

9. Remove the OSD from the cluster.

   [heat-admin@overcloud-controller-0 ~]$ sudo podman exec ceph-mon-<HOSTNAME> ceph osd rm 0
   [heat-admin@overcloud-controller-0 ~]$ sudo podman exec ceph-mon-<HOSTNAME> ceph osd rm 1

10. Leave the node and return to the undercloud as the stack user.

    [heat-admin@overcloud-controller-0 ~]$ exit
    [stack@director ~]$

11. Disable the Ceph Storage node so that director does not reprovision it.

    [stack@director ~]$ openstack baremetal node list
    [stack@director ~]$ openstack baremetal node maintenance set UUID

12. Removing a Ceph Storage node requires an update to the overcloud stack in director with the local template files. First identify the UUID of the overcloud stack:

    $ openstack stack list

13. Identify the UUIDs of the Ceph Storage node you want to delete:

    $ openstack server list

14. Delete the node from the stack and update the plan accordingly:

    $ openstack overcloud node delete --stack overcloud <NODE_UUID>

1. Wait until the stack completes its update. Use the heat stack-list --show-nested command to monitor the stack update.

2. Add new nodes to the director node pool and deploy them as Ceph Storage nodes. Use the CephStorageCount parameter in the parameter_defaults section of your environment file, in this case, ~/templates/storage-config.yaml, to define the total number of Ceph Storage nodes in the overcloud.

   parameter_defaults:
     ControllerCount: 3
     OvercloudControlFlavor: control
     ComputeCount: 3
     OvercloudComputeFlavor: compute
     CephStorageCount: 3
     OvercloudCephStorageFlavor: ceph-storage
     CephMonCount: 3
     OvercloudCephMonFlavor: ceph-mon

1. After you update your environment file, redeploy the overcloud:

   $ openstack overcloud deploy --templates -e <ENVIRONMENT_FILES>

   Director provisions the new node and updates the entire stack with the details of the new node.

2. Log in to a Controller node as the heat-admin user and check the status of the Ceph Storage node:

   [heat-admin@overcloud-controller-0 ~]$ sudo ceph status

3. Confirm that the value in the osdmap section matches the number of nodes in your cluster that you want. The Ceph Storage node that you removed is replaced with a new node.

1. Notice the following heat template deploys Ceph Storage with three OSD devices:

   parameter_defaults:
     CephAnsibleDisksConfig:
       devices:
         - /dev/sdb
         - /dev/sdc
         - /dev/sdd
       osd_scenario: lvm
       osd_objectstore: bluestore

2. To add an OSD, update the node disk layout as described in Section 5.2, “Mapping the Ceph Storage node disk layout”. In this example, add /dev/sde to the template:

   parameter_defaults:
     CephAnsibleDisksConfig:
       devices:
         - /dev/sdb
         - /dev/sdc
         - /dev/sdd
         - /dev/sde
       osd_scenario: lvm
       osd_objectstore: bluestore

3. Run openstack overcloud deploy to update the overcloud.

1. SSH into ceph-storage0 and log in as root.

2. Disable and stop the OSD service:

   [root@ceph-storage0 ~]# systemctl disable ceph-osd@4
   [root@ceph-stoarge0 ~]# systemctl stop ceph-osd@4

3. Disconnect from ceph-storage0.
4. SSH into controller0 and log in as root.

5. Identify the name of the Ceph monitor container:

   [root@controller0 ~]# podman ps | grep ceph-mon
   ceph-mon-controller0
   [root@controller0 ~]#

6. Enable the Ceph monitor container to mark the undesired OSD as out:

   [root@controller0 ~]# podman exec ceph-mon-controller0 ceph osd out 4

   Note

   This command causes Ceph to rebalance the storage cluster and copy data to other OSDs in the cluster. The cluster temporarily leaves the active+clean state until rebalancing is complete.

7. Run the following command and wait for the storage cluster state to become active+clean:

   [root@controller0 ~]# podman exec ceph-mon-controller0 ceph -w

8. Remove the OSD from the CRUSH map so that it no longer receives data:

   [root@controller0 ~]# podman exec ceph-mon-controller0 ceph osd crush remove osd.4

9. Remove the OSD authentication key:

   [root@controller0 ~]# podman exec ceph-mon-controller0 ceph auth del osd.4

10. Remove the OSD:

    [root@controller0 ~]# podman exec ceph-mon-controller0 ceph osd rm 4

11. Disconnect from controller0.
12. SSH into the undercloud as the stack user and locate the heat environment file in which you defined the CephAnsibleDisksConfig parameter.

13. Notice the heat template contains four OSDs:

    parameter_defaults:
      CephAnsibleDisksConfig:
        devices:
          - /dev/sdb
          - /dev/sdc
          - /dev/sdd
          - /dev/sde
        osd_scenario: lvm
        osd_objectstore: bluestore

14. Modify the template to remove /dev/sde.

    parameter_defaults:
      CephAnsibleDisksConfig:
        devices:
          - /dev/sdb
          - /dev/sdc
          - /dev/sdd
        osd_scenario: lvm
        osd_objectstore: bluestore

15. Run openstack overcloud deploy to update the overcloud.

    Note

    This example assumes that you removed the /dev/sde device from all hosts with OSDs. If you do not remove the same device from all nodes, update the heat template as shown and see Section 5.4, “Mapping the disk layout to non-homogeneous Ceph Storage nodes” for information about how to define hosts with a differing devices list.