Deploying an Overcloud with Containerized Red Hat Ceph

1. Trigger hardware introspection to retrieve the hardware attributes of each node:

   $ openstack overcloud node introspect --all-manageable --provide

   Copy to Clipboard Toggle word wrap
   • The --all-manageable option introspects only nodes in a managed state. In this example, it is all of them.

   • The --provide option resets all nodes to an active state after introspection.

     Important

     Make sure this process runs to completion. This process usually takes 15 minutes for bare metal nodes.

2. Retrieve a list of your nodes to identify their UUIDs:

   $ openstack baremetal node list

   Copy to Clipboard Toggle word wrap

3. Add a profile option to the properties/capabilities parameter for each node to manually tag a node to a specific profile.

   For example, a typical deployment will use three profiles: control, compute, and ceph-storage. The following commands tag three nodes for each profile:

   $ ironic node-update 1a4e30da-b6dc-499d-ba87-0bd8a3819bc0 add properties/capabilities='profile:control,boot_option:local'
   $ ironic node-update 6faba1a9-e2d8-4b7c-95a2-c7fbdc12129a add properties/capabilities='profile:control,boot_option:local'
   $ ironic node-update 5e3b2f50-fcd9-4404-b0a2-59d79924b38e add properties/capabilities='profile:control,boot_option:local'
   $ ironic node-update 484587b2-b3b3-40d5-925b-a26a2fa3036f add properties/capabilities='profile:compute,boot_option:local'
   $ ironic node-update d010460b-38f2-4800-9cc4-d69f0d067efe add properties/capabilities='profile:compute,boot_option:local'
   $ ironic node-update d930e613-3e14-44b9-8240-4f3559801ea6 add properties/capabilities='profile:compute,boot_option:local'
   $ ironic node-update da0cc61b-4882-45e0-9f43-fab65cf4e52b add properties/capabilities='profile:ceph-storage,boot_option:local'
   $ ironic node-update b9f70722-e124-4650-a9b1-aade8121b5ed add properties/capabilities='profile:ceph-storage,boot_option:local'
   $ ironic node-update 68bf8f29-7731-4148-ba16-efb31ab8d34f add properties/capabilities='profile:ceph-storage,boot_option:local'

   Copy to Clipboard Toggle word wrap
   Tip

   You can also configure a new custom profile to tag a node for the Ceph MON and Ceph MDS services. See Chapter 3, Deploying Other Ceph Services on dedicated nodes for details.

   The addition of the profile option tags the nodes into each respective profiles.

Procedure

1. Check the disk information from the hardware introspection of each node. Run the following command to display the disk information of a node:

   (undercloud) $ openstack baremetal introspection data save 1a4e30da-b6dc-499d-ba87-0bd8a3819bc0 | jq ".inventory.disks"

   Copy to Clipboard Toggle word wrap

   For example, the data for one node might show three disks:

   [
     {
       "size": 299439751168,
       "rotational": true,
       "vendor": "DELL",
       "name": "/dev/sda",
       "wwn_vendor_extension": "0x1ea4dcc412a9632b",
       "wwn_with_extension": "0x61866da04f3807001ea4dcc412a9632b",
       "model": "PERC H330 Mini",
       "wwn": "0x61866da04f380700",
       "serial": "61866da04f3807001ea4dcc412a9632b"
     }
     {
       "size": 299439751168,
       "rotational": true,
       "vendor": "DELL",
       "name": "/dev/sdb",
       "wwn_vendor_extension": "0x1ea4e13c12e36ad6",
       "wwn_with_extension": "0x61866da04f380d001ea4e13c12e36ad6",
       "model": "PERC H330 Mini",
       "wwn": "0x61866da04f380d00",
       "serial": "61866da04f380d001ea4e13c12e36ad6"
     }
     {
       "size": 299439751168,
       "rotational": true,
       "vendor": "DELL",
       "name": "/dev/sdc",
       "wwn_vendor_extension": "0x1ea4e31e121cfb45",
       "wwn_with_extension": "0x61866da04f37fc001ea4e31e121cfb45",
       "model": "PERC H330 Mini",
       "wwn": "0x61866da04f37fc00",
       "serial": "61866da04f37fc001ea4e31e121cfb45"
     }
   ]

   Copy to Clipboard Toggle word wrap

2. Change to the root_device parameter for the node definition. The following example shows how to set the root device to disk 2, which has 61866da04f380d001ea4e13c12e36ad6 as the serial number:

   (undercloud) $ openstack baremetal node set --property root_device='{"serial": "61866da04f380d001ea4e13c12e36ad6"}' 1a4e30da-b6dc-499d-ba87-0bd8a3819bc0

   Copy to Clipboard Toggle word wrap
   Note

   Ensure that you configure the BIOS of each node to include booting from the root disk that you choose. Configure the boot order to boot from the network first, then to boot from the root disk.

Procedure

1. To configure director to use the overcloud-minimal image, create an environment file that contains the following image definition:

   parameter_defaults:
     <roleName>Image: overcloud-minimal

   Copy to Clipboard Toggle word wrap

2. Replace <roleName> with the name of the role and append Image to the name of the role. The following example shows an overcloud-minimal image for Ceph storage nodes:

   parameter_defaults:
     CephStorageImage: overcloud-minimal

   Copy to Clipboard Toggle word wrap
3. Pass the environment file to the openstack overcloud deploy command.

1. Open the /home/stack/templates/roles_data_custom.yaml file.
2. Remove the service entry for the Ceph MON service (namely, OS::TripleO::Services::CephMon) from under the Controller role.

3. Add the OS::TripleO::Services::CephClient service to the Controller role:

   [...]
   - name: Controller # the 'primary' role goes first
     CountDefault: 1
     ServicesDefault:
       - OS::TripleO::Services::CACerts
       - OS::TripleO::Services::CephMds
       - OS::TripleO::Services::CephClient
       - OS::TripleO::Services::CephExternal
       - OS::TripleO::Services::CephRbdMirror
       - OS::TripleO::Services::CephRgw
       - OS::TripleO::Services::CinderApi
   [...]

   Copy to Clipboard Toggle word wrap

4. At the end of roles_data_custom.yaml, add a custom CephMon role containing the Ceph MON service and all the other required node services. For example:

   - name: CephMon
     ServicesDefault:
       # Common Services
       - OS::TripleO::Services::AuditD
       - OS::TripleO::Services::CACerts
       - OS::TripleO::Services::CertmongerUser
       - OS::TripleO::Services::Collectd
       - OS::TripleO::Services::Docker
       - OS::TripleO::Services::Fluentd
       - OS::TripleO::Services::Kernel
       - OS::TripleO::Services::Ntp
       - OS::TripleO::Services::ContainersLogrotateCrond
       - OS::TripleO::Services::SensuClient
       - OS::TripleO::Services::Snmp
       - OS::TripleO::Services::Timezone
       - OS::TripleO::Services::TripleoFirewall
       - OS::TripleO::Services::TripleoPackages
       - OS::TripleO::Services::Tuned
       # Role-Specific Services
       - OS::TripleO::Services::CephMon

   Copy to Clipboard Toggle word wrap

5. Using the openstack flavor create command, define a new flavor named ceph-mon for this role:

   $ openstack flavor create --id auto --ram 6144 --disk 40 --vcpus 4 ceph-mon

   Copy to Clipboard Toggle word wrap
   Note

   For more details about this command, run openstack flavor create --help.

6. Map this flavor to a new profile, also named ceph-mon:

   $ openstack flavor set --property "cpu_arch"="x86_64" --property "capabilities:boot_option"="local" --property "capabilities:profile"="ceph-mon" ceph-mon

   Copy to Clipboard Toggle word wrap
   Note

   For more details about this command, run openstack flavor set --help.

7. Tag nodes into the new ceph-mon profile:

   $ ironic node-update UUID add properties/capabilities='profile:ceph-mon,boot_option:local'

   Copy to Clipboard Toggle word wrap

8. Add the following configuration to the node-info.yaml file to associate the ceph-mon flavor with the CephMon role:

   parameter_defaults:
     OvercloudCephMonFlavor: CephMon
     CephMonCount: 3

   Copy to Clipboard Toggle word wrap

1. Open the /home/stack/templates/roles_data_custom.yaml file.

2. Remove the service entry for the Ceph MDS service (namely, OS::TripleO::Services::CephMds) from under the Controller role:

   [...]
   - name: Controller # the 'primary' role goes first
     CountDefault: 1
     ServicesDefault:
       - OS::TripleO::Services::CACerts
       # - OS::TripleO::Services::CephMds 

   1

   
       - OS::TripleO::Services::CephMon
       - OS::TripleO::Services::CephExternal
       - OS::TripleO::Services::CephRbdMirror
       - OS::TripleO::Services::CephRgw
       - OS::TripleO::Services::CinderApi
   [...]

   Copy to Clipboard Toggle word wrap

   1

   Comment out this line. This same service will be added to a custom role in the next step.

3. At the end of roles_data_custom.yaml, add a custom CephMDS role containing the Ceph MDS service and all the other required node services. For example:

   - name: CephMDS
     ServicesDefault:
       # Common Services
       - OS::TripleO::Services::AuditD
       - OS::TripleO::Services::CACerts
       - OS::TripleO::Services::CertmongerUser
       - OS::TripleO::Services::Collectd
       - OS::TripleO::Services::Docker
       - OS::TripleO::Services::Fluentd
       - OS::TripleO::Services::Kernel
       - OS::TripleO::Services::Ntp
       - OS::TripleO::Services::ContainersLogrotateCrond
       - OS::TripleO::Services::SensuClient
       - OS::TripleO::Services::Snmp
       - OS::TripleO::Services::Timezone
       - OS::TripleO::Services::TripleoFirewall
       - OS::TripleO::Services::TripleoPackages
       - OS::TripleO::Services::Tuned
       # Role-Specific Services
       - OS::TripleO::Services::CephMds
       - OS::TripleO::Services::CephClient 

   1

   Copy to Clipboard Toggle word wrap

   1

   The Ceph MDS service requires the admin keyring, which can be set by either Ceph MON or Ceph Client service. As we are deploying Ceph MDS on a dedicated node (without the Ceph MON service), include the Ceph Client service on the role as well.

4. Using the openstack flavor create command, define a new flavor named ceph-mds for this role:

   $ openstack flavor create --id auto --ram 6144 --disk 40 --vcpus 4 ceph-mds

   Copy to Clipboard Toggle word wrap
   Note

   For more details about this command, run openstack flavor create --help.

5. Map this flavor to a new profile, also named ceph-mds:

   $ openstack flavor set --property "cpu_arch"="x86_64" --property "capabilities:boot_option"="local" --property "capabilities:profile"="ceph-mds" ceph-mds

   Copy to Clipboard Toggle word wrap
   Note

   For more details about this command, run openstack flavor set --help.

Procedure

1. Add the following parameter_defaults to a custom environment file, for example, swift-external-params.yaml, and adjust the values to suit your deployment:

   parameter_defaults:
      ExternalPublicUrl: 'http://<Public RGW endpoint or loadbalancer>:8080/swift/v1/AUTH_%(project_id)s'
      ExternalInternalUrl: 'http://<Internal RGW endpoint>:8080/swift/v1/AUTH_%(project_id)s'
      ExternalAdminUrl: 'http://<Admin RGW endpoint>:8080/swift/v1/AUTH_%(project_id)s'
      ExternalSwiftUserTenant: 'service'
      SwiftPassword: 'choose_a_random_password'

   Copy to Clipboard Toggle word wrap
   Note

   The example code snippet contains parameter values that might differ from values that you use in your environment:

   • The default port where the remote RGW instance listens is 8080. The port might be different depending on how the external RGW is configured.
   • The swift user created in the overcloud uses the password defined by the SwiftPassword parameter. You must configure the external RGW instance to use the same password to authenticate with the Identity service by using the rgw_keystone_admin_password.

2. Add the following code to the Ceph config file to configure RGW to use the Identity service. Adjust the variable values to suit your environment.

       rgw_keystone_api_version: 3
       rgw_keystone_url: http://<public Keystone endpoint>:5000/
       rgw_keystone_accepted_roles: 'member, Member, admin'
       rgw_keystone_accepted_admin_roles: ResellerAdmin, swiftoperator
       rgw_keystone_admin_domain: default
       rgw_keystone_admin_project: service
       rgw_keystone_admin_user: swift
       rgw_keystone_admin_password: <Password as defined in the environment parameters>
       rgw_keystone_implicit_tenants: 'true'
       rgw_keystone_revocation_interval: '0'
       rgw_s3_auth_use_keystone: 'true'
       rgw_swift_versioning_enabled: 'true'
       rgw_swift_account_in_url: 'true'

   Copy to Clipboard Toggle word wrap
   Note

   Director creates the following roles and users in the Identity service by default:

   • rgw_keystone_accepted_admin_roles: ResellerAdmin, swiftoperator
   • rgw_keystone_admin_domain: default
   • rgw_keystone_admin_project: service
   • rgw_keystone_admin_user: swift

3. Deploy the overcloud with the additional environment files:

   openstack overcloud deploy --templates \
   -e <your environment files>
   -e /usr/share/openstack-tripleo-heat-templates/environments/swift-external.yaml
   -e swift-external-params.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Declare the OSDs/journal mapping:

   parameter_defaults:
     CephAnsibleDisksConfig:
      devices:
      - /dev/sda
      - /dev/sdb
      dedicated_devices:
      - /dev/sdc
      - /dev/sdc
      osd_scenario: non-collocated
      journal_size: 8192

   Copy to Clipboard Toggle word wrap

Procedure

1. Set the default replication factor to two. This example splits four nodes into two different roots.

   parameter_defaults:
     CephPoolDefaultSize: 2

   Copy to Clipboard Toggle word wrap

Procedure

1. Run the following commands to retrieve the UUIDs of the four nodes:

   for ((x=1; x<=4; x++)); \
   { echo "Node overcloud-ceph0${x}"; \
   openstack baremetal introspection data save overcloud-ceph0${x} | jq .extra.system.product.uuid; }
   Node overcloud-ceph01
   "32C2BC31-F6BB-49AA-971A-377EFDFDB111"
   Node overcloud-ceph02
   "76B4C69C-6915-4D30-AFFD-D16DB74F64ED"
   Node overcloud-ceph03
   "FECF7B20-5984-469F-872C-732E3FEF99BF"
   Node overcloud-ceph04
   "5FFEFA5F-69E4-4A88-B9EA-62811C61C8B3"

   Copy to Clipboard Toggle word wrap
   Note

   In the example, overcloud-ceph0[1-4] are the Ironic nodes names; they will be deployed as lab-ceph0[1–4] (via HostnameMap.yaml).

2. Specify the node placement as follows:

   Expand

   Root	Rack	Node
   standard_root	rack1_std	overcloud-ceph01 (lab-ceph01)
   	rack2_std	overcloud-ceph02 (lab-ceph02)
   fast_root	rack1_fast	overcloud-ceph03 (lab-ceph03)
   	rack2_fast	overcloud-ceph04 (lab-ceph04)

   Show more
   Note

   You cannot have two buckets with the same name. Even if lab-ceph01 and lab-ceph03 are in the same physical rack, you cannot have two buckets called rack1. Therefore, we named them rack1_std and rack1_fast.

   Note

   This example demonstrates how to create a specific route called “standard_root” to illustrate multiple custom roots. However, you could have kept the HDDs OSD nodes in the default root.

3. Use the following NodeDataLookup syntax:

   NodeDataLookup: {"SYSTEM_UUID": {"osd_crush_location": {"root": "$MY_ROOT", "rack": "$MY_RACK", "host": "$OVERCLOUD_NODE_HOSTNAME"}}}

   Copy to Clipboard Toggle word wrap
   Note

   You must specify the system UUID and then the CRUSH hierarchy from top to bottom. Also, the host parameter must point to the node’s overcloud host name, not the Bare Metal service (ironic) node name. To match the example configuration, enter the following:

   parameter_defaults:
     NodeDataLookup: {"32C2BC31-F6BB-49AA-971A-377EFDFDB111": {"osd_crush_location": {"root": "standard_root", "rack": "rack1_std", "host": "lab-ceph01"}},
        "76B4C69C-6915-4D30-AFFD-D16DB74F64ED": {"osd_crush_location": {"root": "standard_root", "rack": "rack2_std", "host": "lab-ceph02"}},
        "FECF7B20-5984-469F-872C-732E3FEF99BF": {"osd_crush_location": {"root": "fast_root", "rack": "rack1_fast", "host": "lab-ceph03"}},
        "5FFEFA5F-69E4-4A88-B9EA-62811C61C8B3": {"osd_crush_location": {"root": "fast_root", "rack": "rack2_fast", "host": "lab-ceph04"}}}

   Copy to Clipboard Toggle word wrap

4. Enable CRUSH map management at the ceph-ansible level:

   parameter_defaults:
     CephAnsibleExtraConfig:
       create_crush_tree: true

   Copy to Clipboard Toggle word wrap

5. Use scheduler hints to ensure the Bare Metal service node UUIDs correctly map to the hostnames:

   parameter_defaults:
     CephStorageCount: 4
     OvercloudCephStorageFlavor: ceph-storage
     CephStorageSchedulerHints:
       'capabilities:node': 'ceph-%index%'

   Copy to Clipboard Toggle word wrap

6. Tag the Bare Metal service nodes with the corresponding hint:

   openstack baremetal node set --property capabilities='profile:ceph-storage,node:ceph-0,boot_option:local' overcloud-ceph01
   
   openstack baremetal node set --property capabilities=profile:ceph-storage,'node:ceph-1,boot_option:local' overcloud-ceph02
   
   openstack baremetal node set --property capabilities='profile:ceph-storage,node:ceph-2,boot_option:local' overcloud-ceph03
   
   openstack baremetal node set --property capabilities='profile:ceph-storage,node:ceph-3,boot_option:local' overcloud-ceph04

   Copy to Clipboard Toggle word wrap
   Note

   For more information about predictive placement, see Assigning Specific Node IDs in the Advanced Overcloud Customization guide.

Procedure

1. Use the following syntax to define the CRUSH rules:

   parameter_defaults:
     CephAnsibleExtraConfig:
       crush_rules:
         - name: $RULE_NAME
           root: $ROOT_NAME
           type: $REPLICAT_DOMAIN
           default: true/false

   Copy to Clipboard Toggle word wrap
   Note

   Setting the default parameter to true means that this rule will be used when you create a new pool without specifying any rule. There may only be one default rule.

   In the following example, rule standard points to the OSD nodes hosted on the standard_root with one replicate per rack. Rule fast points to the OSD nodes hosted on the standard_root with one replicate per rack:

   parameter_defaults:
     CephAnsibleExtraConfig:
       crush_rule_config: true
       crush_rules:
         - name: standard
           root: standard_root
           type: rack
           default: true
         - name: fast
           root: fast_root
           type: rack
           default: false

   Copy to Clipboard Toggle word wrap
   Note

   You must set crush_rule_config to true.

Procedure

1. Use the following syntax to define or change a pool property:

   - name: $POOL_NAME
        pg_num: $PG_COUNT
        rule_name: $RULE_NAME
        application: rbd

   Copy to Clipboard Toggle word wrap

2. List all OSP pools and set the appropriate rule (standard, in this case), and create a new pool called tier2 that uses the fast rule. This pool will be used by Block Storage (cinder).

   parameter_defaults:
     CephPools:
       - name: tier2
         pg_num: 64
         rule_name: fast
         application: rbd
   
       - name: volumes
         pg_num: 64
         rule_name: standard
         application: rbd
   
       - name: vms
         pg_num: 64
         rule_name: standard
         application: rbd
   
       - name: backups
         pg_num: 64
         rule_name: standard
         application: rbd
   
       - name: images
         pg_num: 64
         rule_name: standard
         application: rbd
   
       - name: metrics
         pg_num: 64
         rule_name: standard
         application: openstack_gnocchi

   Copy to Clipboard Toggle word wrap

Procedure

1. Update cinder.conf as follows:

   parameter_defaults:
     CinderRbdExtraPools:
       - tier2

   Copy to Clipboard Toggle word wrap

Procedure

1. Connect to a Ceph monitor node and run the following command:

   # ceph osd tree
   ID WEIGHT  TYPE NAME               UP/DOWN REWEIGHT PRIMARY-AFFINITY
   -7 0.39996 root standard_root
   -6 0.19998     rack rack1_std
   -5 0.19998         host lab-ceph02
    1 0.09999             osd.1            up  1.00000          1.00000
    4 0.09999             osd.4            up  1.00000          1.00000
   -9 0.19998     rack rack2_std
   -8 0.19998         host lab-ceph03
    0 0.09999             osd.0            up  1.00000          1.00000
    3 0.09999             osd.3            up  1.00000          1.00000
   -4 0.19998 root fast_root
   -3 0.19998     rack rack1_fast
   -2 0.19998         host lab-ceph01
    2 0.09999             osd.2            up  1.00000          1.00000
    5 0.09999             osd.5            up  1.00000          1.00000

   Copy to Clipboard Toggle word wrap

1. Log in to a Controller node as the heat-admin user:

   $ nova list
   $ ssh heat-admin@192.168.0.25

   Copy to Clipboard Toggle word wrap

2. Check the health of the cluster:

   $ sudo docker exec ceph-mon-$HOSTNAME ceph health

   Copy to Clipboard Toggle word wrap

   If the cluster has no issues, the command reports back HEALTH_OK. This means the cluster is safe to use.

3. Log in to an overcloud node that runs the Ceph monitor service and check the status of all OSDs in the cluster:

   sudo docker exec ceph-mon-$HOSTNAME ceph osd tree

   Copy to Clipboard Toggle word wrap

4. Check the status of the Ceph Monitor quorum:

   $ sudo ceph quorum_status

   Copy to Clipboard Toggle word wrap

   This shows the monitors participating in the quorum and which one is the leader.

5. Verify that all Ceph OSDs are running:

   $ ceph osd stat

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to a Ceph MON or Controller node and disable Ceph Storage cluster rebalancing temporarily:

   $ sudo ceph osd set noout
   $ sudo ceph osd set norebalance

   Copy to Clipboard Toggle word wrap
2. Select the first Ceph Storage node to reboot and log into it.

3. Reboot the node:

   $ sudo reboot

   Copy to Clipboard Toggle word wrap
4. Wait until the node boots.

5. Log in to a Ceph MON or Controller node and check the cluster status:

   $ sudo ceph -s

   Copy to Clipboard Toggle word wrap

   Check that the pgmap reports all pgs as normal (active+clean).

6. Log out of the Ceph MON or Controller node, reboot the next Ceph Storage node, and check its status. Repeat this process until you have rebooted all Ceph storage nodes.

7. When complete, log into a Ceph MON or Controller node and enable cluster rebalancing again:

   $ sudo ceph osd unset noout
   $ sudo ceph osd unset norebalance

   Copy to Clipboard Toggle word wrap

8. Perform a final status check to verify the cluster reports HEALTH_OK:

   $ sudo ceph status

   Copy to Clipboard Toggle word wrap

1. Log into the director host as the stack user and initialize your director configuration:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap
2. Define the hardware and power management details for the new nodes in a new node definition template; for example, instackenv-scale.json.

3. Import this file to the OpenStack director:

   $ openstack overcloud node import ~/instackenv-scale.json

   Copy to Clipboard Toggle word wrap

   Importing the node definition template registers each node defined there to the director.

4. Assign the kernel and ramdisk images to all nodes:

   $ openstack overcloud node configure

   Copy to Clipboard Toggle word wrap

1. Trigger hardware introspection to retrieve the hardware attributes of each node:

   $ openstack overcloud node introspect --all-manageable --provide

   Copy to Clipboard Toggle word wrap
   • The --all-manageable option introspects only nodes in a managed state. In this example, it is all of them.

   • The --provide option resets all nodes to an active state after introspection.

     Important

     Make sure this process runs to completion. This process usually takes 15 minutes for bare metal nodes.

2. Retrieve a list of your nodes to identify their UUIDs:

   $ openstack baremetal node list

   Copy to Clipboard Toggle word wrap

3. Add a profile option to the properties/capabilities parameter for each node to manually tag a node to a specific profile.

   For example, the following commands tag three additional nodes with the ceph-storage profile:

   $ ironic node-update 551d81f5-4df2-4e0f-93da-6c5de0b868f7 add properties/capabilities='profile:ceph-storage,boot_option:local'
   $ ironic node-update 5e735154-bd6b-42dd-9cc2-b6195c4196d7 add properties/capabilities='profile:ceph-storage,boot_option:local'
   $ ironic node-update 1a2b090c-299d-4c20-a25d-57dd21a7085b add properties/capabilities='profile:ceph-storage,boot_option:local'

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to a Controller node as the heat-admin user. The director stack user has an SSH key to access the heat-admin user.

2. List the OSD tree and find the OSDs for your node. For example, the node you want to remove might contain the following OSDs:

   -2 0.09998     host overcloud-cephstorage-0
   0 0.04999         osd.0                         up  1.00000          1.00000
   1 0.04999         osd.1                         up  1.00000          1.00000

   Copy to Clipboard Toggle word wrap

3. Disable the OSDs on the Ceph Storage node. In this case, the OSD IDs are 0 and 1.

   [heat-admin@overcloud-controller-0 ~]$ sudo docker exec ceph-mon-<HOSTNAME> ceph osd out 0
   [heat-admin@overcloud-controller-0 ~]$ sudo docker exec ceph-mon-<HOSTNAME> ceph osd out 1

   Copy to Clipboard Toggle word wrap

4. The Ceph Storage cluster begins rebalancing. Wait for this process to complete. Follow the status by using the following command:

   [heat-admin@overcloud-controller-0 ~]$ sudo docker exec ceph-mon-<HOSTNAME> ceph -w

   Copy to Clipboard Toggle word wrap

5. After the Ceph cluster completes rebalancing, log in to the Ceph Storage node you are removing, in this case, overcloud-cephstorage-0, as the heat-admin user, and stop and disable the node.

   [heat-admin@overcloud-cephstorage-0 ~]$ sudo systemctl stop ceph-osd@0
   [heat-admin@overcloud-cephstorage-0 ~]$ sudo systemctl stop ceph-osd@1
   [heat-admin@overcloud-cephstorage-0 ~]$ sudo systemctl disable ceph-osd@0
   [heat-admin@overcloud-cephstorage-0 ~]$ sudo systemctl disable ceph-osd@1

   Copy to Clipboard Toggle word wrap

6. Stop the OSDs.

   [heat-admin@overcloud-cephstorage-0 ~]$ sudo systemctl stop ceph-osd@0
   [heat-admin@overcloud-cephstorage-0 ~]$ sudo systemctl stop ceph-osd@1

   Copy to Clipboard Toggle word wrap

7. While logged in to the Controller node, remove the OSDs from the CRUSH map so that they no longer receive data.

   [heat-admin@overcloud-controller-0 ~]$ sudo docker exec ceph-mon-<HOSTNAME> ceph osd crush remove osd.0
   [heat-admin@overcloud-controller-0 ~]$ sudo docker exec ceph-mon-<HOSTNAME> ceph osd crush remove osd.1

   Copy to Clipboard Toggle word wrap

8. Remove the OSD authentication key.

   [heat-admin@overcloud-controller-0 ~]$ sudo docker exec ceph-mon-<HOSTNAME> ceph auth del osd.0
   [heat-admin@overcloud-controller-0 ~]$ sudo docker exec ceph-mon-<HOSTNAME> ceph auth del osd.1

   Copy to Clipboard Toggle word wrap

9. Remove the OSD from the cluster.

   [heat-admin@overcloud-controller-0 ~]$ sudo docker exec ceph-mon-<HOSTNAME> ceph osd rm 0
   [heat-admin@overcloud-controller-0 ~]$ sudo docker exec ceph-mon-<HOSTNAME> ceph osd rm 1

   Copy to Clipboard Toggle word wrap

10. Remove the Storage node from the CRUSH map:

    [heat-admin@overcloud-controller-0 ~]$ sudo docker exec ceph-mon-<HOSTNAME> ceph osd crush rm <NODE>
    [heat-admin@overcloud-controller-0 ~]$ sudo ceph osd crush remove <NODE>

    Copy to Clipboard Toggle word wrap

    You can confirm the <NODE> name as defined in the CRUSH map by searching the CRUSH tree:

    [heat-admin@overcloud-controller-0 ~]$ sudo docker exec ceph-mon-<HOSTNAME> ceph osd crush tree | grep overcloud-osd-compute-3 -A 4
                    "name": "overcloud-osd-compute-3",
                    "type": "host",
                    "type_id": 1,
                    "items": []
                },
    [heat-admin@overcloud-controller-0 ~]$

    Copy to Clipboard Toggle word wrap

    In the CRUSH tree, ensure that the items list is empty. If the list is not empty, revisit step 7.

11. Leave the node and return to the undercloud as the stack user.

    [heat-admin@overcloud-controller-0 ~]$ exit
    [stack@director ~]$

    Copy to Clipboard Toggle word wrap

12. Disable the Ceph Storage node so the director does not reprovision it.

    [stack@director ~]$ openstack baremetal node list
    [stack@director ~]$ openstack baremetal node maintenance set UUID

    Copy to Clipboard Toggle word wrap

13. Removing a Ceph Storage node requires an update to the overcloud stack in director with the local template files. First identify the UUID of the overcloud stack:

    $ openstack stack list

    Copy to Clipboard Toggle word wrap

14. Identify the UUIDs of the Ceph Storage node you want to delete:

    $ openstack server list

    Copy to Clipboard Toggle word wrap

15. Delete the node from the stack and update the plan accordingly:

    Important

    If you passed any extra environment files when you created the overcloud, pass them again here by using the -e option to avoid making undesired changes to the overcloud. For more information, see Modifying the Overcloud Environment in the Director Installation and Usage guide.

    $ openstack overcloud node delete /
    --stack <stack-name> /
    --templates /
    -e <other-environment-files> /
    <node_UUID>

    Copy to Clipboard Toggle word wrap
16. Wait until the stack completes its update. Use the heat stack-list --show-nested command to monitor the stack update.

17. Add new nodes to the director node pool and deploy them as Ceph Storage nodes. Use the CephStorageCount parameter in parameter_defaults of your environment file, in this case, ~/templates/storage-config.yaml, to define the total number of Ceph Storage nodes in the overcloud.

    parameter_defaults:
      ControllerCount: 3
      OvercloudControlFlavor: control
      ComputeCount: 3
      OvercloudComputeFlavor: compute
      CephStorageCount: 3
      OvercloudCephStorageFlavor: ceph-storage
      CephMonCount: 3
      OvercloudCephMonFlavor: ceph-mon

    Copy to Clipboard Toggle word wrap
    Note

    For more information about how to define the number of nodes per role, see Section 7.1, “Assigning nodes and flavors to roles”.

18. After you update your environment file, re-deploy the overcloud:

    $ openstack overcloud deploy --templates -e <ENVIRONMENT_FILES>

    Copy to Clipboard Toggle word wrap

    Director provisions the new node and updates the entire stack with the details of the new node.

19. Log in to a Controller node as the heat-admin user and check the status of the Ceph Storage node:

    [heat-admin@overcloud-controller-0 ~]$ sudo ceph status

    Copy to Clipboard Toggle word wrap
20. Confirm that the value in the osdmap section matches the number of nodes in your cluster that you want. The Ceph Storage node that you removed is replaced with a new node.

1. Notice the following heat template deploys Ceph Storage with three OSD devices:

   parameter_defaults:
     CephAnsibleDisksConfig:
       devices:
         - /dev/sdb
         - /dev/sdc
         - /dev/sdd
       osd_scenario: lvm
       osd_objectstore: bluestore

   Copy to Clipboard Toggle word wrap

2. To add an OSD, update the node disk layout as described in Section 5.2, “Mapping the Ceph Storage node disk layout”. In this example, add /dev/sde to the template:

   parameter_defaults:
     CephAnsibleDisksConfig:
       devices:
         - /dev/sdb
         - /dev/sdc
         - /dev/sdd
         - /dev/sde
       osd_scenario: lvm
       osd_objectstore: bluestore

   Copy to Clipboard Toggle word wrap
3. Run openstack overcloud deploy to update the overcloud.

1. SSH into ceph-storage0 and log in as root.

2. Disable and stop the OSD service:

   [root@ceph-storage0 ~]# systemctl disable ceph-osd@4
   [root@ceph-stoarge0 ~]# systemctl stop ceph-osd@4

   Copy to Clipboard Toggle word wrap
3. Disconnect from ceph-storage0.
4. SSH into controller0 and log in as root.

5. Identify the name of the Ceph monitor container:

   [root@controller0 ~]# docker ps | grep ceph-mon
   ceph-mon-controller0
   [root@controller0 ~]#

   Copy to Clipboard Toggle word wrap

6. Enable the Ceph monitor container to mark the undesired OSD as out:

   [root@controller0 ~]# docker exec ceph-mon-controller0 ceph osd out 4

   Copy to Clipboard Toggle word wrap
   Note

   This command causes Ceph to rebalance the storage cluster and copy data to other OSDs in the cluster. The cluster temporarily leaves the active+clean state until rebalancing is complete.

7. Run the following command and wait for the storage cluster state to become active+clean:

   [root@controller0 ~]# docker exec ceph-mon-controller0 ceph -w

   Copy to Clipboard Toggle word wrap

8. Remove the OSD from the CRUSH map so that it no longer receives data:

   [root@controller0 ~]# docker exec ceph-mon-controller0 ceph osd crush remove osd.4

   Copy to Clipboard Toggle word wrap

9. Remove the OSD authentication key:

   [root@controller0 ~]# docker exec ceph-mon-controller0 ceph auth del osd.4

   Copy to Clipboard Toggle word wrap

10. Remove the OSD:

    [root@controller0 ~]# docker exec ceph-mon-controller0 ceph osd rm 4

    Copy to Clipboard Toggle word wrap
11. Disconnect from controller0.
12. SSH into the undercloud as the stack user and locate the heat environment file in which you defined the CephAnsibleDisksConfig parameter.

13. Notice the heat template contains four OSDs:

    parameter_defaults:
      CephAnsibleDisksConfig:
        devices:
          - /dev/sdb
          - /dev/sdc
          - /dev/sdd
          - /dev/sde
        osd_scenario: lvm
        osd_objectstore: bluestore

    Copy to Clipboard Toggle word wrap

14. Modify the template to remove /dev/sde.

    parameter_defaults:
      CephAnsibleDisksConfig:
        devices:
          - /dev/sdb
          - /dev/sdc
          - /dev/sdd
        osd_scenario: lvm
        osd_objectstore: bluestore

    Copy to Clipboard Toggle word wrap

15. Run openstack overcloud deploy to update the overcloud.

    Note

    This example assumes that you removed the /dev/sde device from all hosts with OSDs. If you do not remove the same device from all nodes, update the heat template as shown and see Section 5.2.5, “Mapping the Disk Layout to Non-Homogeneous Ceph Storage Nodes” for information about how to define hosts with a differing devices list.

Procedure

1. Identify the name of the running Ceph monitor container and store it in an environment variable called MON:

   MON=$(docker ps | grep ceph-mon | awk {'print $1'})

   Copy to Clipboard Toggle word wrap

2. Alias the ceph command so that it executes within the running Ceph monitor container:

   alias ceph="docker exec $MON ceph"

   Copy to Clipboard Toggle word wrap

3. Use the new alias to verify that the OSD that you want to replace is down:

   [root@overcloud-controller-0 ~]# ceph osd tree | grep 27
   27   hdd 0.04790         osd.27                    down  1.00000 1.00000

   Copy to Clipboard Toggle word wrap

4. Destroy the OSD. The following example command destroys OSD 27:

   [root@overcloud-controller-0 ~]# ceph osd destroy 27 --yes-i-really-mean-it
   destroyed osd.27

   Copy to Clipboard Toggle word wrap

Procedure

1. Ensure that the failed OSD is not running:

   systemctl stop ceph-osd@27

   Copy to Clipboard Toggle word wrap

2. Identify the image ID of the ceph container image and store it in an environment variable called IMG:

   IMG=$(docker images | grep ceph | awk {'print $3'})

   Copy to Clipboard Toggle word wrap

3. Alias the ceph-volume command so that it runs inside the $IMG Ceph container, with the ceph-volume entry point and relevant directories:

   alias ceph-volume="docker run --rm --privileged --net=host --ipc=host -v /run/lock/lvm:/run/lock/lvm:z -v /var/run/udev/:/var/run/udev/:z -v /dev:/dev -v /etc/ceph:/etc/ceph:z -v /var/lib/ceph/:/var/lib/ceph/:z -v /var/log/ceph/:/var/log/ceph/:z --entrypoint=ceph-volume $IMG --cluster ceph"

   Copy to Clipboard Toggle word wrap

4. Verify that the aliased command runs successfully:

   ceph-volume lvm list

   Copy to Clipboard Toggle word wrap

5. Check that your new OSD device is not already part of LVM. Use the pvdisplay command to inspect the device, and ensure that the VG Name field is empty. Replace <NEW_DEVICE> with the /dev/* path of your new OSD device:

   [root@overcloud-computehci-2 ~]# pvdisplay <NEW_DEVICE>
     --- Physical volume ---
     PV Name               /dev/sdj
     VG Name               ceph-0fb0de13-fc8e-44c8-99ea-911e343191d2
     PV Size               50.00 GiB / not usable 1.00 GiB
     Allocatable           yes (but full)
     PE Size               1.00 GiB
     Total PE              49
     Free PE               0
     Allocated PE          49
     PV UUID               kOO0If-ge2F-UH44-6S1z-9tAv-7ypT-7by4cp
   [root@overcloud-computehci-2 ~]#

   Copy to Clipboard Toggle word wrap

   If the VG Name field is not empty, then the device belongs to a volume group that you must remove.

6. If the device belongs to a volume group, use the lvdisplay command to check if there is a logical volume in the volume group. Replace <VOLUME_GROUP> with the value of the VG Name field that you retrieved from the pvdisplay command:

   [root@overcloud-computehci-2 ~]# lvdisplay | grep <VOLUME_GROUP>
     LV Path                /dev/ceph-0fb0de13-fc8e-44c8-99ea-911e343191d2/osd-data-a0810722-7673-43c7-8511-2fd9db1dbbc6
     VG Name                ceph-0fb0de13-fc8e-44c8-99ea-911e343191d2
   [root@overcloud-computehci-2 ~]#

   Copy to Clipboard Toggle word wrap

   If the LV Path field is not empty, then the device contains a logical volume that you must remove.

7. If the new device is part of a logical volume or volume group, remove the logical volume, volume group, and the device association as a physical volume within the LVM system.

   • Replace <LV_PATH> with the value of the LV Path field.
   • Replace <VOLUME_GROUP> with the value of the VG Name field.

   • Replace <NEW_DEVICE> with the /dev/* path of your new OSD device.

     [root@overcloud-computehci-2 ~]# lvremove --force <LV_PATH>
       Logical volume "osd-data-a0810722-7673-43c7-8511-2fd9db1dbbc6" successfully removed

     Copy to Clipboard Toggle word wrap

     [root@overcloud-computehci-2 ~]# vgremove --force <VOLUME_GROUP>
       Volume group "ceph-0fb0de13-fc8e-44c8-99ea-911e343191d2" successfully removed

     Copy to Clipboard Toggle word wrap

     [root@overcloud-computehci-2 ~]# pvremove <NEW_DEVICE>
       Labels on physical volume "/dev/sdj" successfully wiped.

     Copy to Clipboard Toggle word wrap

8. Ensure that the new OSD device is clean. In the following example, the device is /dev/sdj:

   [root@overcloud-computehci-2 ~]# ceph-volume lvm zap /dev/sdj
   --> Zapping: /dev/sdj
   --> --destroy was not specified, but zapping a whole device will remove the partition table
   Running command: /usr/sbin/wipefs --all /dev/sdj
   Running command: /bin/dd if=/dev/zero of=/dev/sdj bs=1M count=10
    stderr: 10+0 records in
   10+0 records out
   10485760 bytes (10 MB, 10 MiB) copied, 0.010618 s, 988 MB/s
   --> Zapping successful for: <Raw Device: /dev/sdj>
   [root@overcloud-computehci-2 ~]#

   Copy to Clipboard Toggle word wrap

9. Create the new OSD with the existing OSD ID by using the new device but pass --no-systemd so that ceph-volume does not attempt to start the OSD. This is not possible from within the container:

   ceph-volume lvm create --osd-id 27 --data /dev/sdj --no-systemd

   Copy to Clipboard Toggle word wrap

10. Start the OSD outside of the container:

    systemctl start ceph-osd@27

    Copy to Clipboard Toggle word wrap

Procedure

1. Check if the device name changed, update the devices list according to the naming method you used to deploy Ceph. For more information, see Section 11.1, “Determining if there is a device name change”.
2. To ensure that the change did not introduce any inconsistencies, re-run the overcloud deploy command to perform a stack update.

3. In cases where you have hosts that have different device lists, you might have to define an exception. For example, you might use the following example heat environment file to deploy a node with three OSD devices.

   parameter_defaults:
     CephAnsibleDisksConfig:
       devices:
         - /dev/sdb
         - /dev/sdc
         - /dev/sdd
       osd_scenario: lvm
       osd_objectstore: bluestore

   Copy to Clipboard Toggle word wrap

   The CephAnsibleDisksConfig parameter applies to all nodes that host OSDs, so you cannot update the devices parameter with the new device list. Instead, you must define an exception for the new host that has a different device list. For more information about defining an exception, see Section 5.2.5, “Mapping the Disk Layout to Non-Homogeneous Ceph Storage Nodes”.