Deploying an overcloud with containerized Red Hat Ceph

Procedure

1. After you create the inventory file, save the file to the home directory of the stack user (/home/stack/instackenv.json).

2. Initialize the stack user, then import the instackenv.json inventory file into the director:

   $ source ~/stackrc
   $ openstack overcloud node import ~/instackenv.json

   Copy to Clipboard Toggle word wrap

   The openstack overcloud node import command imports the inventory file and registers each node with the director.

3. Assign the kernel and ramdisk images to each node:

1. Trigger hardware introspection to retrieve the hardware attributes of each node:

   $ openstack overcloud node introspect --all-manageable --provide

   Copy to Clipboard Toggle word wrap
   • The --all-manageable option introspects only the nodes that are in a managed state. In this example, all nodes are in a managed state.

   • The --provide option resets all nodes to an active state after introspection.

     Important

     Ensure that this process completes successfully. This process usually takes 15 minutes for bare metal nodes.

2. Retrieve a list of your nodes to identify their UUIDs:

   $ openstack baremetal node list

   Copy to Clipboard Toggle word wrap

3. Add a profile option to the properties/capabilities parameter for each node to manually tag a node to a specific profile. The addition of the profile option tags the nodes into each respective profile.

   Note

   As an alternative to manual tagging, use the Automated Health Check (AHC) Tools to automatically tag larger numbers of nodes based on benchmarking data.

   For example, a typical deployment contains three profiles: control, compute, and ceph-storage. Enter the following commands to tag three nodes for each profile:

   $ openstack baremetal node set 1a4e30da-b6dc-499d-ba87-0bd8a3819bc0 --property capabilities="profile:control,boot_option:local"
   $ openstack baremetal node set 6faba1a9-e2d8-4b7c-95a2-c7fbdc12129a --property capabilities="profile:control,boot_option:local"
   $ openstack baremetal node set 5e3b2f50-fcd9-4404-b0a2-59d79924b38e --property capabilities="profile:control,boot_option:local"
   $ openstack baremetal node set 484587b2-b3b3-40d5-925b-a26a2fa3036f --property capabilities="profile:compute,boot_option:local"
   $ openstack baremetal node set d010460b-38f2-4800-9cc4-d69f0d067efe --property capabilities="profile:compute,boot_option:local"
   $ openstack baremetal node set d930e613-3e14-44b9-8240-4f3559801ea6 --property capabilities="profile:compute,boot_option:local"
   $ openstack baremetal node set da0cc61b-4882-45e0-9f43-fab65cf4e52b --property capabilities="profile:ceph-storage,boot_option:local"
   $ openstack baremetal node set b9f70722-e124-4650-a9b1-aade8121b5ed --property capabilities="profile:ceph-storage,boot_option:local"
   $ openstack baremetal node set 68bf8f29-7731-4148-ba16-efb31ab8d34f --property capabilities="profile:ceph-storage,boot_option:local"

   Copy to Clipboard Toggle word wrap
   Tip

   You can also configure a new custom profile to tag a node for the Ceph MON and Ceph MDS services, see Chapter 3, Deploying Ceph services on dedicated nodes.

Procedure

1. Check the disk information from the hardware introspection of each node. Run the following command to display the disk information of a node:

   (undercloud) $ openstack baremetal introspection data save 1a4e30da-b6dc-499d-ba87-0bd8a3819bc0 | jq ".inventory.disks"

   Copy to Clipboard Toggle word wrap

   For example, the data for one node might show three disks:

   [
     {
       "size": 299439751168,
       "rotational": true,
       "vendor": "DELL",
       "name": "/dev/sda",
       "wwn_vendor_extension": "0x1ea4dcc412a9632b",
       "wwn_with_extension": "0x61866da04f3807001ea4dcc412a9632b",
       "model": "PERC H330 Mini",
       "wwn": "0x61866da04f380700",
       "serial": "61866da04f3807001ea4dcc412a9632b"
     }
     {
       "size": 299439751168,
       "rotational": true,
       "vendor": "DELL",
       "name": "/dev/sdb",
       "wwn_vendor_extension": "0x1ea4e13c12e36ad6",
       "wwn_with_extension": "0x61866da04f380d001ea4e13c12e36ad6",
       "model": "PERC H330 Mini",
       "wwn": "0x61866da04f380d00",
       "serial": "61866da04f380d001ea4e13c12e36ad6"
     }
     {
       "size": 299439751168,
       "rotational": true,
       "vendor": "DELL",
       "name": "/dev/sdc",
       "wwn_vendor_extension": "0x1ea4e31e121cfb45",
       "wwn_with_extension": "0x61866da04f37fc001ea4e31e121cfb45",
       "model": "PERC H330 Mini",
       "wwn": "0x61866da04f37fc00",
       "serial": "61866da04f37fc001ea4e31e121cfb45"
     }
   ]

   Copy to Clipboard Toggle word wrap

2. Run the openstack baremetal node set --property root_device= command to set the root disk for a node. Include the most appropriate hardware attribute value to define the root disk.

   (undercloud) $ openstack baremetal node set --property root_device=’{“serial”:”<serial_number>”}' <node-uuid>

   Copy to Clipboard Toggle word wrap

   For example, to set the root device to disk 2, which has the serial number 61866da04f380d001ea4e13c12e36ad6 run the following command:

Procedure

1. To configure director to use the overcloud-minimal image, create an environment file that contains the following image definition:

   parameter_defaults:
     <roleName>Image: overcloud-minimal

   Copy to Clipboard Toggle word wrap

2. Replace <roleName> with the name of the role and append Image to the name of the role. The following example shows an overcloud-minimal image for Ceph storage nodes:

   parameter_defaults:
     CephStorageImage: overcloud-minimal

   Copy to Clipboard Toggle word wrap
3. Pass the environment file to the openstack overcloud deploy command.

Procedure

1. Make a copy of the roles_data.yaml file in /home/stack/templates/ so that you can add custom roles:

   $ cp /usr/share/openstack-tripleo-heat-templates/roles_data.yaml /home/stack/templates/roles_data_custom.yaml

   Copy to Clipboard Toggle word wrap
2. Include the new custom role file in the openstack overcloud deploy command.

Procedure

1. Open the /home/stack/templates/roles_data_custom.yaml file.
2. Remove the service entry for the Ceph MON service (namely, OS::TripleO::Services::CephMon) from the Controller role.

3. Add the OS::TripleO::Services::CephClient service to the Controller role:

   [...]
   - name: Controller # the 'primary' role goes first
     CountDefault: 1
     ServicesDefault:
       - OS::TripleO::Services::CACerts
       - OS::TripleO::Services::CephMds
       - OS::TripleO::Services::CephClient
       - OS::TripleO::Services::CephExternal
       - OS::TripleO::Services::CephRbdMirror
       - OS::TripleO::Services::CephRgw
       - OS::TripleO::Services::CinderApi
   [...]

   Copy to Clipboard Toggle word wrap

4. At the end of the roles_data_custom.yaml file, add a custom CephMon role that contains the Ceph MON service and all the other required node services:

   - name: CephMon
     ServicesDefault:
       # Common Services
       - OS::TripleO::Services::AuditD
       - OS::TripleO::Services::CACerts
       - OS::TripleO::Services::CertmongerUser
       - OS::TripleO::Services::Collectd
       - OS::TripleO::Services::Docker
       - OS::TripleO::Services::FluentdClient
       - OS::TripleO::Services::Kernel
       - OS::TripleO::Services::Ntp
       - OS::TripleO::Services::ContainersLogrotateCrond
       - OS::TripleO::Services::SensuClient
       - OS::TripleO::Services::Snmp
       - OS::TripleO::Services::Timezone
       - OS::TripleO::Services::TripleoFirewall
       - OS::TripleO::Services::TripleoPackages
       - OS::TripleO::Services::Tuned
       # Role-Specific Services
       - OS::TripleO::Services::CephMon

   Copy to Clipboard Toggle word wrap

5. Run the openstack flavor create command to define a new flavor named ceph-mon for the CephMon role:

   $ openstack flavor create --id auto --ram 6144 --disk 40 --vcpus 4 ceph-mon

   Copy to Clipboard Toggle word wrap
   Note

   For details about this command, run openstack flavor create --help.

6. Map this flavor to a new profile, also named ceph-mon:

   $ openstack flavor set --property "cpu_arch"="x86_64" --property "capabilities:boot_option"="local" --property "capabilities:profile"="ceph-mon" ceph-mon

   Copy to Clipboard Toggle word wrap
   Note

   For details about this command, run openstack flavor set --help.

7. Tag nodes into the new ceph-mon profile:

   $ openstack baremetal node set UUID --property capabilities="profile:ceph-mon,boot_option:local"

   Copy to Clipboard Toggle word wrap

8. Add the following configuration to the node-info.yaml file to associate the ceph-mon flavor with the CephMon role:

   parameter_defaults:
     OvercloudCephMonFlavor: CephMon
     CephMonCount: 3

   Copy to Clipboard Toggle word wrap

Procedure

1. Open the /home/stack/templates/roles_data_custom.yaml file.

2. Remove the service entry for the Ceph MDS service (namely, OS::TripleO::Services::CephMds) from the Controller role:

   [...]
   - name: Controller # the 'primary' role goes first
     CountDefault: 1
     ServicesDefault:
       - OS::TripleO::Services::CACerts
       # - OS::TripleO::Services::CephMds 

   1

   
       - OS::TripleO::Services::CephMon
       - OS::TripleO::Services::CephExternal
       - OS::TripleO::Services::CephRbdMirror
       - OS::TripleO::Services::CephRgw
       - OS::TripleO::Services::CinderApi
   [...]

   Copy to Clipboard Toggle word wrap

   1

   Comment out this line. In the next step, you add this service to the new custom role.

3. At the end of the roles_data_custom.yaml file, add a custom CephMDS role containing the Ceph MDS service and all the other required node services:

   - name: CephMDS
     ServicesDefault:
       # Common Services
       - OS::TripleO::Services::AuditD
       - OS::TripleO::Services::CACerts
       - OS::TripleO::Services::CertmongerUser
       - OS::TripleO::Services::Collectd
       - OS::TripleO::Services::Docker
       - OS::TripleO::Services::FluentdClient
       - OS::TripleO::Services::Kernel
       - OS::TripleO::Services::Ntp
       - OS::TripleO::Services::ContainersLogrotateCrond
       - OS::TripleO::Services::SensuClient
       - OS::TripleO::Services::Snmp
       - OS::TripleO::Services::Timezone
       - OS::TripleO::Services::TripleoFirewall
       - OS::TripleO::Services::TripleoPackages
       - OS::TripleO::Services::Tuned
       # Role-Specific Services
       - OS::TripleO::Services::CephMds
       - OS::TripleO::Services::CephClient 

   1

   Copy to Clipboard Toggle word wrap

   1

   The Ceph MDS service requires the admin keyring, which you can set with either the Ceph MON or Ceph Client service. If you deploy Ceph MDS on a dedicated node without the Ceph MON service, you must also include the Ceph Client service in the new CephMDS role.

4. Run the openstack flavor create command to define a new flavor named ceph-mds for this role:

   $ openstack flavor create --id auto --ram 6144 --disk 40 --vcpus 4 ceph-mds

   Copy to Clipboard Toggle word wrap
   Note

   For details about this command, run openstack flavor create --help.

5. Map the new ceph-mds flavor to a new profile, also named ceph-mds:

   $ openstack flavor set --property "cpu_arch"="x86_64" --property "capabilities:boot_option"="local" --property "capabilities:profile"="ceph-mds" ceph-mds

   Copy to Clipboard Toggle word wrap
   Note

   For details about this command, run openstack flavor set --help.

6. Tag nodes into the new ceph-mds profile:

1. Remove the first two lines (parameter_defaults: and NodeDataLookup:) from the file temporarily.
2. Enter cat node-spec-overrides.yaml | jq .

Procedure

1. Update the CephAnsibleExtraConfig parameter to change the default delay and retry values:

   parameter_defaults:
     CephAnsibleExtraConfig:
       health_osd_check_delay: 40
       health_osd_check_retries: 30
       health_mon_check_delay: 20
       health_mon_check_retries: 10

   Copy to Clipboard Toggle word wrap

   This example makes the cluster check 30 times and wait 40 seconds between each check for the Ceph OSDs, and check 20 times and wait 10 seconds between each check for the Ceph MONs.

2. To incorporate the changes, pass the updated yaml file with -e using openstack overcloud deploy.

Procedure

1. Declare the OSDs/journal mapping:

   parameter_defaults:
     CephAnsibleDisksConfig:
      devices:
      - /dev/sda
      - /dev/sdb
      dedicated_devices:
      - /dev/sdc
      - /dev/sdc
      osd_scenario: non-collocated
      journal_size: 8192

   Copy to Clipboard Toggle word wrap

Procedure

1. Set the default replication factor to two. This example splits four nodes into two different roots.

   parameter_defaults:
     CephPoolDefaultSize: 2

   Copy to Clipboard Toggle word wrap

Procedure

1. Run the following commands to retrieve the UUIDs of the four nodes:

   for ((x=1; x<=4; x++)); \
   { echo "Node overcloud-ceph0${x}"; \
   openstack baremetal introspection data save overcloud-ceph0${x} | jq .extra.system.product.uuid; }
   Node overcloud-ceph01
   "32C2BC31-F6BB-49AA-971A-377EFDFDB111"
   Node overcloud-ceph02
   "76B4C69C-6915-4D30-AFFD-D16DB74F64ED"
   Node overcloud-ceph03
   "FECF7B20-5984-469F-872C-732E3FEF99BF"
   Node overcloud-ceph04
   "5FFEFA5F-69E4-4A88-B9EA-62811C61C8B3"

   Copy to Clipboard Toggle word wrap
   Note

   In the example, overcloud-ceph0[1-4] are the Ironic nodes names; they are deployed as lab-ceph0[1–4] (through HostnameMap.yaml).

2. Specify the node placement as follows:

   Expand

   Root	Rack	Node
   standard_root	rack1_std	overcloud-ceph01 (lab-ceph01)
   	rack2_std	overcloud-ceph02 (lab-ceph02)
   fast_root	rack1_fast	overcloud-ceph03 (lab-ceph03)
   	rack2_fast	overcloud-ceph04 (lab-ceph04)

   Show more
   Note

   You cannot have two buckets with the same name. Even if lab-ceph01 and lab-ceph03 are in the same physical rack, you cannot have two buckets called rack1. Therefore, we named them rack1_std and rack1_fast.

   Note

   This example demonstrates how to create a specific route called “standard_root” to illustrate multiple custom roots. However, you could have kept the HDDs OSD nodes in the default root.

3. Use the following NodeDataLookup syntax:

   NodeDataLookup: {"SYSTEM_UUID": {"osd_crush_location": {"root": "$MY_ROOT", "rack": "$MY_RACK", "host": "$OVERCLOUD_NODE_HOSTNAME"}}}

   Copy to Clipboard Toggle word wrap
   Note

   You must specify the system UUID and then the CRUSH hierarchy from top to bottom. Also, the host parameter must point to the node’s overcloud host name, not the Bare Metal service (ironic) node name. To match the example configuration, enter the following:

   parameter_defaults:
     NodeDataLookup: {"32C2BC31-F6BB-49AA-971A-377EFDFDB111": {"osd_crush_location": {"root": "standard_root", "rack": "rack1_std", "host": "lab-ceph01"}},
        "76B4C69C-6915-4D30-AFFD-D16DB74F64ED": {"osd_crush_location": {"root": "standard_root", "rack": "rack2_std", "host": "lab-ceph02"}},
        "FECF7B20-5984-469F-872C-732E3FEF99BF": {"osd_crush_location": {"root": "fast_root", "rack": "rack1_fast", "host": "lab-ceph03"}},
        "5FFEFA5F-69E4-4A88-B9EA-62811C61C8B3": {"osd_crush_location": {"root": "fast_root", "rack": "rack2_fast", "host": "lab-ceph04"}}}

   Copy to Clipboard Toggle word wrap

4. Enable CRUSH map management at the ceph-ansible level:

   parameter_defaults:
     CephAnsibleExtraConfig:
       create_crush_tree: true

   Copy to Clipboard Toggle word wrap

5. Use scheduler hints to ensure the Bare Metal service node UUIDs correctly map to the hostnames:

   parameter_defaults:
     CephStorageCount: 4
     OvercloudCephStorageFlavor: ceph-storage
     CephStorageSchedulerHints:
       'capabilities:node': 'ceph-%index%'

   Copy to Clipboard Toggle word wrap

6. Tag the Bare Metal service nodes with the corresponding hint:

   openstack baremetal node set --property capabilities='profile:ceph-storage,node:ceph-0,boot_option:local' overcloud-ceph01
   
   openstack baremetal node set --property capabilities=profile:ceph-storage,'node:ceph-1,boot_option:local' overcloud-ceph02
   
   openstack baremetal node set --property capabilities='profile:ceph-storage,node:ceph-2,boot_option:local' overcloud-ceph03
   
   openstack baremetal node set --property capabilities='profile:ceph-storage,node:ceph-3,boot_option:local' overcloud-ceph04

   Copy to Clipboard Toggle word wrap
   Note

   For more information about predictive placement, see Assigning Specific Node IDs in the Advanced Overcloud Customization guide.

Procedure

1. Use the following syntax to define the CRUSH rules:

   parameter_defaults:
     CephAnsibleExtraConfig:
       crush_rules:
         - name: $RULE_NAME
           root: $ROOT_NAME
           type: $REPLICAT_DOMAIN
           default: true/false

   Copy to Clipboard Toggle word wrap
   Note

   Setting the default parameter to true means that this rule will be used when you create a new pool without specifying any rule. There may only be one default rule.

   In the following example, rule standard points to the OSD nodes hosted on the standard_root with one replicate per rack. Rule fast points to the OSD nodes hosted on the standard_root with one replicate per rack:

   parameter_defaults:
     CephAnsibleExtraConfig:
       crush_rule_config: true
       crush_rules:
         - name: standard
           root: standard_root
           type: rack
           default: true
         - name: fast
           root: fast_root
           type: rack
           default: false

   Copy to Clipboard Toggle word wrap
   Note

   You must set crush_rule_config to true.

Procedure

1. Use the following syntax to define or change a pool property:

   - name: $POOL_NAME
        pg_num: $PG_COUNT
        rule_name: $RULE_NAME
        application: rbd

   Copy to Clipboard Toggle word wrap

2. List all OSP pools and set the appropriate rule (standard, in this case), and create a new pool called tier2 that uses the fast rule. This pool will be used by Block Storage (cinder).

   parameter_defaults:
     CephPools:
       - name: tier2
         pg_num: 64
         rule_name: fast
         application: rbd
   
       - name: volumes
         pg_num: 64
         rule_name: standard
         application: rbd
   
       - name: vms
         pg_num: 64
         rule_name: standard
         application: rbd
   
       - name: backups
         pg_num: 64
         rule_name: standard
         application: rbd
   
       - name: images
         pg_num: 64
         rule_name: standard
         application: rbd
   
       - name: metrics
         pg_num: 64
         rule_name: standard
         application: openstack_gnocchi

   Copy to Clipboard Toggle word wrap

Procedure

1. Update cinder.conf as follows:

   parameter_defaults:
     CinderRbdExtraPools:
       - tier2

   Copy to Clipboard Toggle word wrap

Procedure

1. Connect to a Ceph monitor node and run the following command:

   # ceph osd tree
   ID WEIGHT  TYPE NAME               UP/DOWN REWEIGHT PRIMARY-AFFINITY
   -7 0.39996 root standard_root
   -6 0.19998     rack rack1_std
   -5 0.19998         host lab-ceph02
    1 0.09999             osd.1            up  1.00000          1.00000
    4 0.09999             osd.4            up  1.00000          1.00000
   -9 0.19998     rack rack2_std
   -8 0.19998         host lab-ceph03
    0 0.09999             osd.0            up  1.00000          1.00000
    3 0.09999             osd.3            up  1.00000          1.00000
   -4 0.19998 root fast_root
   -3 0.19998     rack rack1_fast
   -2 0.19998         host lab-ceph01
    2 0.09999             osd.2            up  1.00000          1.00000
    5 0.09999             osd.5            up  1.00000          1.00000

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to your undercloud host as the stack user.

2. Generate the default container image preparation file:

   $ openstack tripleo container image prepare default \
     --local-push-destination \
     --output-env-file containers-prepare-parameter.yaml

   Copy to Clipboard Toggle word wrap

3. Edit the containers-prepare-parameter.yaml file and make the modifications to suit your requirements. The following example containers-prepare-parameter.yaml file contains the image locations and tags related to the Dashboard services including Grafana, Prometheus, Alertmanager, and Node Exporter. Edit the values depending on your specific scenario:

   parameter_defaults:
       ContainerImagePrepare:
       - push_destination: true
           set:
               ceph_alertmanager_image: ose-prometheus-alertmanager
               ceph_alertmanager_namespace: registry.redhat.io/openshift4
               ceph_alertmanager_tag: v4.1
               ceph_grafana_image: rhceph-3-dashboard-rhel7
               ceph_grafana_namespace: registry.redhat.io/rhceph
               ceph_grafana_tag: 3
               ceph_image: rhceph-4-rhel8
               ceph_namespace: registry.redhat.io/rhceph
               ceph_node_exporter_image: ose-prometheus-node-exporter
               ceph_node_exporter_namespace: registry.redhat.io/openshift4
               ceph_node_exporter_tag: v4.1
               ceph_prometheus_image: ose-prometheus
               ceph_prometheus_namespace: registry.redhat.io/openshift4
               ceph_prometheus_tag: v4.1
               ceph_tag: latest

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to the undercloud node as the stack user.

2. Include the following environment files, with all environment files that are part of your existing deployment, in the openstack overcloud deploy command:

   $ openstack overcloud deploy \
     --templates \
     -e <existing_overcloud_environment_files> \
     -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
     -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-dashboard.yaml

   Copy to Clipboard Toggle word wrap

   Replace <existing_overcloud_environment_files> with the list of environment files that are part of your existing deployment.

   Result

   The resulting deployment comprises an external stack with the grafana, prometheus, alertmanager, and node-exporter containers. The Ceph Dashboard manager module is the back end for this stack and embeds the grafana layouts to provide ceph cluster specific metrics to the end users.

Procedure

1. Log in to the undercloud as the stack user.

2. Create the following ceph_dashboard_admin.yaml environment file:

     parameter_defaults:
        CephDashboardAdminRO: false

   Copy to Clipboard Toggle word wrap

3. Run the overcloud deploy command to update the existing stack and include the environment file you created with all other environment files that are part of your existing deployment:

   $ openstack overcloud deploy \
   --templates \
   -e <existing_overcloud_environment_files> \
   -e ceph_dashboard_admin.yml

   Copy to Clipboard Toggle word wrap

   Replace <existing_overcloud_environment_files> with the list of environment files that are part of your existing deployment.

Procedure

1. Log in to the undercloud node as the stack user.

2. Retrieve the dashboard admin login credentials:

   [stack@undercloud ~]$ grep dashboard_admin_password /var/lib/mistral/overcloud/ceph-ansible/group_vars/all.yml

   Copy to Clipboard Toggle word wrap

3. Retrieve the VIP address to access the Ceph Dashboard:

   [stack@undercloud-0 ~]$ grep dashboard_frontend /var/lib/mistral/overcloud/ceph-ansible/group_vars/mgrs.yml

   Copy to Clipboard Toggle word wrap

4. Use a web browser to point to the front end VIP and access the Dashboard. Director configures and exposes the Dashboard on the provisioning network, so you can use the VIP that you retrieved in step 2 to access the dashboard directly on TCP port 8444. Ensure that the following conditions are met:

   • The Web client host is layer 2 connected to the provisioning network.

   • The provisioning network is properly routed or proxied, and it can be reached from the web client host. If these conditions are not met, you can still open a SSH tunnel to reach the Dashboard VIP on the overcloud:

     client_host$ ssh -L 8444:<dashboard vip>:8444 stack@<your undercloud>

     Copy to Clipboard Toggle word wrap

     Replace <dashboard vip> with the IP address of the control plane VIP that you retrieved in step 3.

5. Access the Dashboard by pointing your web browser to http://localhost:8444. The default user that ceph-ansible creates is admin. You can retrieve the password in /var/lib/mistral/overcloud/ceph-ansible/group_vars/all.yml.

   Results

   • You can access the Ceph Dashboard.
   • The numbers and graphs that the Dashboard displays reflect the same cluster status that the CLI command, ceph -s, returns.

Procedure

1. Log in to a Controller node as the heat-admin user:

   $ nova list
   $ ssh heat-admin@192.168.0.25

   Copy to Clipboard Toggle word wrap

2. Check the health of the cluster:

   $ sudo podman exec ceph-mon-<HOSTNAME> ceph health

   Copy to Clipboard Toggle word wrap

   If the cluster has no issues, the command reports back HEALTH_OK. This means the cluster is safe to use.

3. Log in to an overcloud node that runs the Ceph monitor service and check the status of all OSDs in the cluster:

   $ sudo podman exec ceph-mon-<HOSTNAME> ceph osd tree

   Copy to Clipboard Toggle word wrap

4. Check the status of the Ceph Monitor quorum:

   $ sudo podman exec ceph-mon-<HOSTNAME> ceph quorum_status

   Copy to Clipboard Toggle word wrap

   This shows the monitors participating in the quorum and which one is the leader.

5. Verify that all Ceph OSDs are running:

   $ sudo podman exec ceph-mon-<HOSTNAME> ceph osd stat

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to a Ceph MON or Controller node and disable Ceph Storage cluster rebalancing temporarily:

   $ sudo podman exec -it ceph-mon-controller-0 ceph osd set noout
   $ sudo podman exec -it ceph-mon-controller-0 ceph osd set norebalance

   Copy to Clipboard Toggle word wrap
2. Select the first Ceph Storage node that you want to reboot and log in to the node.

3. Reboot the node:

   $ sudo reboot

   Copy to Clipboard Toggle word wrap
4. Wait until the node boots.

5. Log in to the node and check the cluster status:

   $ sudo podman exec -it ceph-mon-controller-0 ceph status

   Copy to Clipboard Toggle word wrap

   Check that the pgmap reports all pgs as normal (active+clean).

6. Log out of the node, reboot the next node, and check its status. Repeat this process until you have rebooted all Ceph storage nodes.

7. When complete, log into a Ceph MON or Controller node and re-enable cluster rebalancing:

   $ sudo podman exec -it ceph-mon-controller-0 ceph osd unset noout
   $ sudo podman exec -it ceph-mon-controller-0 ceph osd unset norebalance

   Copy to Clipboard Toggle word wrap

8. Perform a final status check to verify that the cluster reports HEALTH_OK:

   $ sudo podman exec -it ceph-mon-controller-0 ceph status

   Copy to Clipboard Toggle word wrap

1. Log in to the undercloud as the stack user and initialize your director configuration:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap
2. Define the hardware and power management details for the new nodes in a new node definition template; for example, instackenv-scale.json.

3. Import this file to the OpenStack director:

   $ openstack overcloud node import ~/instackenv-scale.json

   Copy to Clipboard Toggle word wrap

   Importing the node definition template registers each node defined there to the director.

4. Assign the kernel and ramdisk images to all nodes:

   $ openstack overcloud node configure

   Copy to Clipboard Toggle word wrap

1. Trigger hardware introspection to retrieve the hardware attributes of each node:

   $ openstack overcloud node introspect --all-manageable --provide

   Copy to Clipboard Toggle word wrap
   • The --all-manageable option introspects only the nodes that are in a managed state. In this example, all nodes are in a managed state.

   • The --provide option resets all nodes to an active state after introspection.

     Important

     Ensure that this process completes successfully. This process usually takes 15 minutes for bare metal nodes.

2. Retrieve a list of your nodes to identify their UUIDs:

   $ openstack baremetal node list

   Copy to Clipboard Toggle word wrap

3. Add a profile option to the properties/capabilities parameter for each node to manually tag a node to a specific profile. The addition of the profile option tags the nodes into each respective profile.

   Note

   As an alternative to manual tagging, use the Automated Health Check (AHC) Tools to automatically tag larger numbers of nodes based on benchmarking data.

   For example, the following commands tag three additional nodes with the ceph-storage profile:

   $ openstack baremetal node set 551d81f5-4df2-4e0f-93da-6c5de0b868f7 --property capabilities="profile:ceph-storage,boot_option:local"
   $ openstack baremetal node set 5e735154-bd6b-42dd-9cc2-b6195c4196d7 --property capabilities="profile:ceph-storage,boot_option:local"
   $ openstack baremetal node set 1a2b090c-299d-4c20-a25d-57dd21a7085b --property capabilities="profile:ceph-storage,boot_option:local"

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to a Controller node as the heat-admin user. The director’s stack user has an SSH key to access the heat-admin user.

2. List the OSD tree and find the OSDs for your node. For example, the node you want to remove might contain the following OSDs:

   -2 0.09998     host overcloud-cephstorage-0
   0 0.04999         osd.0                         up  1.00000          1.00000
   1 0.04999         osd.1                         up  1.00000          1.00000

   Copy to Clipboard Toggle word wrap

3. Disable the OSDs on the Ceph Storage node. In this case, the OSD IDs are 0 and 1.

   [heat-admin@overcloud-controller-0 ~]$ sudo podman exec ceph-mon-<HOSTNAME> ceph osd out 0
   [heat-admin@overcloud-controller-0 ~]$ sudo podman exec ceph-mon-<HOSTNAME> ceph osd out 1

   Copy to Clipboard Toggle word wrap

4. The Ceph Storage cluster begins rebalancing. Wait for this process to complete. Follow the status by using the following command:

   [heat-admin@overcloud-controller-0 ~]$ sudo podman exec ceph-mon-<HOSTNAME> ceph -w

   Copy to Clipboard Toggle word wrap

5. After the Ceph cluster completes rebalancing, log in to the Ceph Storage node you are removing, in this case overcloud-cephstorage-0, as the heat-admin user and stop the node.

   [heat-admin@overcloud-cephstorage-0 ~]$ sudo systemctl disable ceph-osd@0
   [heat-admin@overcloud-cephstorage-0 ~]$ sudo systemctl disable ceph-osd@1

   Copy to Clipboard Toggle word wrap

6. Stop the OSDs.

   [heat-admin@overcloud-cephstorage-0 ~]$ sudo systemctl stop ceph-osd@0
   [heat-admin@overcloud-cephstorage-0 ~]$ sudo systemctl stop ceph-osd@1

   Copy to Clipboard Toggle word wrap

7. While logged in to the Controller node, remove the OSDs from the CRUSH map so that they no longer receive data.

   [heat-admin@overcloud-controller-0 ~]$ sudo podman exec ceph-mon-<HOSTNAME> ceph osd crush remove osd.0
   [heat-admin@overcloud-controller-0 ~]$ sudo podman exec ceph-mon-<HOSTNAME> ceph osd crush remove osd.1

   Copy to Clipboard Toggle word wrap

8. Remove the OSD authentication key.

   [heat-admin@overcloud-controller-0 ~]$ sudo podman exec ceph-mon-<HOSTNAME> ceph auth del osd.0
   [heat-admin@overcloud-controller-0 ~]$ sudo podman exec ceph-mon-<HOSTNAME> ceph auth del osd.1

   Copy to Clipboard Toggle word wrap

9. Remove the OSD from the cluster.

   [heat-admin@overcloud-controller-0 ~]$ sudo podman exec ceph-mon-<HOSTNAME> ceph osd rm 0
   [heat-admin@overcloud-controller-0 ~]$ sudo podman exec ceph-mon-<HOSTNAME> ceph osd rm 1

   Copy to Clipboard Toggle word wrap

10. Leave the node and return to the undercloud as the stack user.

    [heat-admin@overcloud-controller-0 ~]$ exit
    [stack@director ~]$

    Copy to Clipboard Toggle word wrap

11. Disable the Ceph Storage node so that director does not reprovision it.

    [stack@director ~]$ openstack baremetal node list
    [stack@director ~]$ openstack baremetal node maintenance set UUID

    Copy to Clipboard Toggle word wrap

12. Removing a Ceph Storage node requires an update to the overcloud stack in director with the local template files. First identify the UUID of the overcloud stack:

    $ openstack stack list

    Copy to Clipboard Toggle word wrap

13. Identify the UUIDs of the Ceph Storage node you want to delete:

    $ openstack server list

    Copy to Clipboard Toggle word wrap

14. Delete the node from the stack and update the plan accordingly:

    $ openstack overcloud node delete --stack overcloud <NODE_UUID>

    Copy to Clipboard Toggle word wrap
    Important

    If you passed any extra environment files when you created the overcloud, pass them again here using the -e option to avoid making undesired changes to the overcloud. For more information, see Modifying the overcloud environment in the Director Installation and Usage guide.

15. Wait until the stack completes its update. Use the heat stack-list --show-nested command to monitor the stack update.

16. Add new nodes to the director node pool and deploy them as Ceph Storage nodes. Use the CephStorageCount parameter in the parameter_defaults of your environment file, in this case, ~/templates/storage-config.yaml, to define the total number of Ceph Storage nodes in the overcloud.

    parameter_defaults:
      ControllerCount: 3
      OvercloudControlFlavor: control
      ComputeCount: 3
      OvercloudComputeFlavor: compute
      CephStorageCount: 3
      OvercloudCephStorageFlavor: ceph-storage
      CephMonCount: 3
      OvercloudCephMonFlavor: ceph-mon

    Copy to Clipboard Toggle word wrap
    Note

    For more information about how to define the number of nodes per role, see Section 7.1, “Assigning nodes and flavors to roles”.

17. After you update your environment file, redeploy the overcloud:

    $ openstack overcloud deploy --templates -e <ENVIRONMENT_FILE>

    Copy to Clipboard Toggle word wrap

    Director provisions the new node and updates the entire stack with the details of the new node.

18. Log in to a Controller node as the heat-admin user and check the status of the Ceph Storage node:

    [heat-admin@overcloud-controller-0 ~]$ sudo ceph status

    Copy to Clipboard Toggle word wrap
19. Confirm that the value in the osdmap section matches the number of nodes in your cluster that you want. The Ceph Storage node that you removed is replaced with a new node.

1. Notice the following heat template deploys Ceph Storage with three OSD devices:

   parameter_defaults:
     CephAnsibleDisksConfig:
       devices:
         - /dev/sdb
         - /dev/sdc
         - /dev/sdd
       osd_scenario: lvm
       osd_objectstore: bluestore

   Copy to Clipboard Toggle word wrap

2. To add an OSD, update the node disk layout as described in Section 5.3, “Mapping the Ceph Storage node disk layout”. In this example, add /dev/sde to the template:

   parameter_defaults:
     CephAnsibleDisksConfig:
       devices:
         - /dev/sdb
         - /dev/sdc
         - /dev/sdd
         - /dev/sde
       osd_scenario: lvm
       osd_objectstore: bluestore

   Copy to Clipboard Toggle word wrap
3. Run openstack overcloud deploy to update the overcloud.

1. SSH into ceph-storage0 and log in as root.

2. Disable and stop the OSD service:

   [root@ceph-storage0 ~]# systemctl disable ceph-osd@4
   [root@ceph-stoarge0 ~]# systemctl stop ceph-osd@4

   Copy to Clipboard Toggle word wrap
3. Disconnect from ceph-storage0.
4. SSH into controller0 and log in as root.

5. Identify the name of the Ceph monitor container:

   [root@controller0 ~]# podman ps | grep ceph-mon
   ceph-mon-controller0
   [root@controller0 ~]#

   Copy to Clipboard Toggle word wrap

6. Enable the Ceph monitor container to mark the undesired OSD as out:

   [root@controller0 ~]# podman exec ceph-mon-controller0 ceph osd out 4

   Copy to Clipboard Toggle word wrap
   Note

   This command causes Ceph to rebalance the storage cluster and copy data to other OSDs in the cluster. The cluster temporarily leaves the active+clean state until rebalancing is complete.

7. Run the following command and wait for the storage cluster state to become active+clean:

   [root@controller0 ~]# podman exec ceph-mon-controller0 ceph -w

   Copy to Clipboard Toggle word wrap

8. Remove the OSD from the CRUSH map so that it no longer receives data:

   [root@controller0 ~]# podman exec ceph-mon-controller0 ceph osd crush remove osd.4

   Copy to Clipboard Toggle word wrap

9. Remove the OSD authentication key:

   [root@controller0 ~]# podman exec ceph-mon-controller0 ceph auth del osd.4

   Copy to Clipboard Toggle word wrap

10. Remove the OSD:

    [root@controller0 ~]# podman exec ceph-mon-controller0 ceph osd rm 4

    Copy to Clipboard Toggle word wrap
11. Disconnect from controller0.
12. SSH into the undercloud as the stack user and locate the heat environment file in which you defined the CephAnsibleDisksConfig parameter.

13. Notice the heat template contains four OSDs:

    parameter_defaults:
      CephAnsibleDisksConfig:
        devices:
          - /dev/sdb
          - /dev/sdc
          - /dev/sdd
          - /dev/sde
        osd_scenario: lvm
        osd_objectstore: bluestore

    Copy to Clipboard Toggle word wrap

14. Modify the template to remove /dev/sde.

    parameter_defaults:
      CephAnsibleDisksConfig:
        devices:
          - /dev/sdb
          - /dev/sdc
          - /dev/sdd
        osd_scenario: lvm
        osd_objectstore: bluestore

    Copy to Clipboard Toggle word wrap

15. Run openstack overcloud deploy to update the overcloud.

    Note

    This example assumes that you removed the /dev/sde device from all hosts with OSDs. If you do not remove the same device from all nodes, update the heat template. For for information about how to define hosts with a differing devices list, see Section 5.5, “Mapping the disk layout to non-homogeneous Ceph Storage nodes”.

Procedure

1. Identify the name of the running Ceph monitor container and store it in an environment variable called MON:

   MON=$(podman ps | grep ceph-mon | awk {'print $1'})

   Copy to Clipboard Toggle word wrap

2. Alias the ceph command so that it executes within the running Ceph monitor container:

   alias ceph="podman exec $MON ceph"

   Copy to Clipboard Toggle word wrap

3. Use the new alias to verify that the OSD that you want to replace is down:

   [root@overcloud-controller-0 ~]# ceph osd tree | grep 27
   27   hdd 0.04790         osd.27                    down  1.00000 1.00000

   Copy to Clipboard Toggle word wrap

4. Destroy the OSD. The following example command destroys OSD 27:

   [root@overcloud-controller-0 ~]# ceph osd destroy 27 --yes-i-really-mean-it
   destroyed osd.27

   Copy to Clipboard Toggle word wrap

Procedure

1. Ensure that the failed OSD is not running:

   systemctl stop ceph-osd@27

   Copy to Clipboard Toggle word wrap

2. Identify the image ID of the ceph container image and store it in an environment variable called IMG:

   IMG=$(podman images | grep ceph | awk {'print $3'})

   Copy to Clipboard Toggle word wrap

3. Alias the ceph-volume command so that it runs inside the $IMG Ceph container, with the ceph-volume entry point and relevant directories:

   alias ceph-volume="podman run --rm --privileged --net=host --ipc=host -v /run/lock/lvm:/run/lock/lvm:z -v /var/run/udev/:/var/run/udev/:z -v /dev:/dev -v /etc/ceph:/etc/ceph:z -v /var/lib/ceph/:/var/lib/ceph/:z -v /var/log/ceph/:/var/log/ceph/:z --entrypoint=ceph-volume $IMG --cluster ceph"

   Copy to Clipboard Toggle word wrap

4. Verify that the aliased command runs successfully:

   ceph-volume lvm list

   Copy to Clipboard Toggle word wrap

5. Check that your new OSD device is not already part of LVM. Use the pvdisplay command to inspect the device, and ensure that the VG Name field is empty. Replace <NEW_DEVICE> with the /dev/* path of your new OSD device:

   [root@overcloud-computehci-2 ~]# pvdisplay <NEW_DEVICE>
     --- Physical volume ---
     PV Name               /dev/sdj
     VG Name               ceph-0fb0de13-fc8e-44c8-99ea-911e343191d2
     PV Size               50.00 GiB / not usable 1.00 GiB
     Allocatable           yes (but full)
     PE Size               1.00 GiB
     Total PE              49
     Free PE               0
     Allocated PE          49
     PV UUID               kOO0If-ge2F-UH44-6S1z-9tAv-7ypT-7by4cp
   [root@overcloud-computehci-2 ~]#

   Copy to Clipboard Toggle word wrap

   If the VG Name field is not empty, then the device belongs to a volume group that you must remove.

6. If the device belongs to a volume group, use the lvdisplay command to check if there is a logical volume in the volume group. Replace <VOLUME_GROUP> with the value of the VG Name field that you retrieved from the pvdisplay command:

   [root@overcloud-computehci-2 ~]# lvdisplay | grep <VOLUME_GROUP>
     LV Path                /dev/ceph-0fb0de13-fc8e-44c8-99ea-911e343191d2/osd-data-a0810722-7673-43c7-8511-2fd9db1dbbc6
     VG Name                ceph-0fb0de13-fc8e-44c8-99ea-911e343191d2
   [root@overcloud-computehci-2 ~]#

   Copy to Clipboard Toggle word wrap

   If the LV Path field is not empty, then the device contains a logical volume that you must remove.

7. If the new device is part of a logical volume or volume group, remove the logical volume, volume group, and the device association as a physical volume within the LVM system.

   • Replace <LV_PATH> with the value of the LV Path field.
   • Replace <VOLUME_GROUP> with the value of the VG Name field.

   • Replace <NEW_DEVICE> with the /dev/* path of your new OSD device.

     [root@overcloud-computehci-2 ~]# lvremove --force <LV_PATH>
       Logical volume "osd-data-a0810722-7673-43c7-8511-2fd9db1dbbc6" successfully removed

     Copy to Clipboard Toggle word wrap

     [root@overcloud-computehci-2 ~]# vgremove --force <VOLUME_GROUP>
       Volume group "ceph-0fb0de13-fc8e-44c8-99ea-911e343191d2" successfully removed

     Copy to Clipboard Toggle word wrap

     [root@overcloud-computehci-2 ~]# pvremove <NEW_DEVICE>
       Labels on physical volume "/dev/sdj" successfully wiped.

     Copy to Clipboard Toggle word wrap

8. Ensure that the new OSD device is clean. In the following example, the device is /dev/sdj:

   [root@overcloud-computehci-2 ~]# ceph-volume lvm zap /dev/sdj
   --> Zapping: /dev/sdj
   --> --destroy was not specified, but zapping a whole device will remove the partition table
   Running command: /usr/sbin/wipefs --all /dev/sdj
   Running command: /bin/dd if=/dev/zero of=/dev/sdj bs=1M count=10
    stderr: 10+0 records in
   10+0 records out
   10485760 bytes (10 MB, 10 MiB) copied, 0.010618 s, 988 MB/s
   --> Zapping successful for: <Raw Device: /dev/sdj>
   [root@overcloud-computehci-2 ~]#

   Copy to Clipboard Toggle word wrap

9. Create the new OSD with the existing OSD ID by using the new device but pass --no-systemd so that ceph-volume does not attempt to start the OSD. This is not possible from within the container:

   ceph-volume lvm create --osd-id 27 --data /dev/sdj --no-systemd

   Copy to Clipboard Toggle word wrap

10. Start the OSD outside of the container:

    systemctl start ceph-osd@27

    Copy to Clipboard Toggle word wrap

Procedure

1. Check if the device name changed, update the devices list according to the naming method you used to deploy Ceph. For more information, see Section 12.1, “Determining if there is a device name change”.
2. To ensure that the change did not introduce any inconsistencies, re-run the overcloud deploy command to perform a stack update.

3. In cases where you have hosts that have different device lists, you might have to define an exception. For example, you might use the following example heat environment file to deploy a node with three OSD devices.

   parameter_defaults:
     CephAnsibleDisksConfig:
       devices:
         - /dev/sdb
         - /dev/sdc
         - /dev/sdd
       osd_scenario: lvm
       osd_objectstore: bluestore

   Copy to Clipboard Toggle word wrap

   The CephAnsibleDisksConfig parameter applies to all nodes that host OSDs, so you cannot update the devices parameter with the new device list. Instead, you must define an exception for the new host that has a different device list. For more information about defining an exception, see Section 5.5, “Mapping the disk layout to non-homogeneous Ceph Storage nodes”.