Deploying Red Hat Ceph Storage and Red Hat OpenStack Platform together with director

Procedure

1. Open /home/stack/undercloud.conf and add the following parameter:

   clean_nodes=true

2. Save /home/stack/undercloud.conf.

3. Update the undercloud configuration.

   openstack undercloud install

Procedure

1. Create a node inventory JSON file in /home/stack.

2. Enter hardware and power management details for each node.

   For example:

   {
       "nodes":[
           {
               "mac":[
                   "b1:b1:b1:b1:b1:b1"
               ],
               "cpu":"4",
               "memory":"6144",
               "disk":"40",
               "arch":"x86_64",
               "pm_type":"ipmi",
               "pm_user":"admin",
               "pm_password":"p@55w0rd!",
               "pm_addr":"192.0.2.205"
           },
           {
               "mac":[
                   "b2:b2:b2:b2:b2:b2"
               ],
               "cpu":"4",
               "memory":"6144",
               "disk":"40",
               "arch":"x86_64",
               "pm_type":"ipmi",
               "pm_user":"admin",
               "pm_password":"p@55w0rd!",
               "pm_addr":"192.0.2.206"
           },
           {
               "mac":[
                   "b3:b3:b3:b3:b3:b3"
               ],
               "cpu":"4",
               "memory":"6144",
               "disk":"40",
               "arch":"x86_64",
               "pm_type":"ipmi",
               "pm_user":"admin",
               "pm_password":"p@55w0rd!",
               "pm_addr":"192.0.2.207"
           },
           {
               "mac":[
                   "c1:c1:c1:c1:c1:c1"
               ],
               "cpu":"4",
               "memory":"6144",
               "disk":"40",
               "arch":"x86_64",
               "pm_type":"ipmi",
               "pm_user":"admin",
               "pm_password":"p@55w0rd!",
               "pm_addr":"192.0.2.208"
           },
           {
               "mac":[
                   "c2:c2:c2:c2:c2:c2"
               ],
               "cpu":"4",
               "memory":"6144",
               "disk":"40",
               "arch":"x86_64",
               "pm_type":"ipmi",
               "pm_user":"admin",
               "pm_password":"p@55w0rd!",
               "pm_addr":"192.0.2.209"
           },
           {
               "mac":[
                   "c3:c3:c3:c3:c3:c3"
               ],
               "cpu":"4",
               "memory":"6144",
               "disk":"40",
               "arch":"x86_64",
               "pm_type":"ipmi",
               "pm_user":"admin",
               "pm_password":"p@55w0rd!",
               "pm_addr":"192.0.2.210"
           },
           {
               "mac":[
                   "d1:d1:d1:d1:d1:d1"
               ],
               "cpu":"4",
               "memory":"6144",
               "disk":"40",
               "arch":"x86_64",
               "pm_type":"ipmi",
               "pm_user":"admin",
               "pm_password":"p@55w0rd!",
               "pm_addr":"192.0.2.211"
           },
           {
               "mac":[
                   "d2:d2:d2:d2:d2:d2"
               ],
               "cpu":"4",
               "memory":"6144",
               "disk":"40",
               "arch":"x86_64",
               "pm_type":"ipmi",
               "pm_user":"admin",
               "pm_password":"p@55w0rd!",
               "pm_addr":"192.0.2.212"
           },
           {
               "mac":[
                   "d3:d3:d3:d3:d3:d3"
               ],
               "cpu":"4",
               "memory":"6144",
               "disk":"40",
               "arch":"x86_64",
               "pm_type":"ipmi",
               "pm_user":"admin",
               "pm_password":"p@55w0rd!",
               "pm_addr":"192.0.2.213"
           }
       ]
   }

3. Save the new file.

4. Initialize the stack user:

   $ source ~/stackrc

5. Import the JSON inventory file into director and register nodes

   $ openstack overcloud node import <inventory_file>

   Replace <inventory_file> with the name of the file created in the first step.

6. Assign the kernel and ramdisk images to each node:

   $ openstack overcloud node configure <node>

Procedure

1. Verify the disk information from the hardware introspection of each node:

   (undercloud)$ openstack baremetal introspection data save <node_uuid> --file <output_file_name>

   • Replace <node_uuid> with the UUID of the node.

   • Replace <output_file_name> with the name of the file that contains the output of the node introspection.

     For example, the data for one node might show three disks:

     [
       {
         "size": 299439751168,
         "rotational": true,
         "vendor": "DELL",
         "name": "/dev/sda",
         "wwn_vendor_extension": "0x1ea4dcc412a9632b",
         "wwn_with_extension": "0x61866da04f3807001ea4dcc412a9632b",
         "model": "PERC H330 Mini",
         "wwn": "0x61866da04f380700",
         "serial": "61866da04f3807001ea4dcc412a9632b"
       }
       {
         "size": 299439751168,
         "rotational": true,
         "vendor": "DELL",
         "name": "/dev/sdb",
         "wwn_vendor_extension": "0x1ea4e13c12e36ad6",
         "wwn_with_extension": "0x61866da04f380d001ea4e13c12e36ad6",
         "model": "PERC H330 Mini",
         "wwn": "0x61866da04f380d00",
         "serial": "61866da04f380d001ea4e13c12e36ad6"
       }
       {
         "size": 299439751168,
         "rotational": true,
         "vendor": "DELL",
         "name": "/dev/sdc",
         "wwn_vendor_extension": "0x1ea4e31e121cfb45",
         "wwn_with_extension": "0x61866da04f37fc001ea4e31e121cfb45",
         "model": "PERC H330 Mini",
         "wwn": "0x61866da04f37fc00",
         "serial": "61866da04f37fc001ea4e31e121cfb45"
       }
     ]

2. Set the root disk for the node by using a unique hardware property:

   (undercloud)$ openstack baremetal node set --property root_device='{<property_value>}' <node-uuid>

   • Replace <property_value> with the unique hardware property value from the introspection data to use to set the root disk.

   • Replace <node_uuid> with the UUID of the node.

     Note

     A unique hardware property is any property from the hardware introspection step that uniquely identifies the disk. For example, the following command uses the disk serial number to set the root disk:

     (undercloud)$ openstack baremetal node set --property root_device='{"serial": "61866da04f380d001ea4e13c12e36ad6"}' 1a4e30da-b6dc-499d-ba87-0bd8a3819bc0

3. Configure the BIOS of each node to first boot from the network and then the root disk.

Procedure

1. Open your /home/stack/templates/overcloud-baremetal-deploy.yaml file.

2. Add or update the image property for the nodes that you want to use the overcloud-minimal image. You can set the image to overcloud-minimal on specific nodes, or for all nodes for a role.

   Note

   The overcloud minimal image is not a whole disk image. The kernel and ramdisk must be specified in the /home/stack/templates/overcloud-baremetal-deploy.yaml file.

   Specific nodes

   - name: Ceph
     count: 3
     instances:
     - hostname: overcloud-ceph-0
       name: node00
       image:
         href: file:///var/lib/ironic/images/overcloud-minimal.raw
         kernel: file://var/lib/ironic/images/overcloud-minimal.vmlinuz
         ramdisk: file://var/lib/ironic/images/overcloud-minimal.initrd
     - hostname: overcloud-ceph-1
       name: node01
       image:
         href: file:///var/lib/ironic/images/overcloud-minimal.raw
         kernel: file://var/lib/ironic/images/overcloud-minimal.vmlinuz
         ramdisk: file://var/lib/ironic/images/overcloud-minimal.initrd
     - hostname: overcloud-ceph-2
       name: node02
       image:
         href: file:///var/lib/ironic/images/overcloud-minimal.raw
         kernel: file://var/lib/ironic/images/overcloud-minimal.vmlinuz
         ramdisk: file://var/lib/ironic/images/overcloud-minimal.initrd

   All nodes for a specific role

   - name: Ceph
     count: 3
     defaults:
       image:
         href: file:///var/lib/ironic/images/overcloud-minimal.raw
         kernel: file://var/lib/ironic/images/overcloud-minimal.vmlinuz
         ramdisk: file://var/lib/ironic/images/overcloud-minimal.initrd
     instances:
     - hostname: overcloud-ceph-0
       name: node00
     - hostname: overcloud-ceph-1
       name: node01
     - hostname: overcloud-ceph-2
       name: node02

3. In the roles_data.yaml role definition file, set the rhsm_enforce parameter to False.

   rhsm_enforce: False

4. Run the provisioning command:

   (undercloud)$ openstack overcloud node provision \
   --stack overcloud \
   --output /home/stack/templates/overcloud-baremetal-deployed.yaml \
   /home/stack/templates/overcloud-baremetal-deploy.yaml

5. Pass the overcloud-baremetal-deployed.yaml environment file to the openstack overcloud ceph deploy command.

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   [stack@director ~]$ source ~/stackrc

3. Generate a new roles data file named roles_data.yaml that includes the Controller, Compute, and CephStorage roles:

   (undercloud)$ openstack overcloud roles \
    generate Controller Compute CephStorage -o /home/stack/templates/roles_data.yaml \

4. Open roles_data.yaml and ensure it has the following parameters and sections:

   Section/Parameter	Value
   Role comment	Role: CephStorage
   Role name	name: CephStorage
   description	Ceph node role
   HostnameFormatDefault	%stackname%-novaceph-%index%
   deprecated_nic_config_name	ceph.yaml

5. Register the Ceph nodes for the overcloud by adding them to your node definition template.

6. Inspect the node hardware:

   (undercloud)$ openstack overcloud node introspect --all-manageable --provide

7. Tag each bare metal node that you want to designate for Ceph with a custom Ceph resource class:

   (undercloud)$ openstack baremetal node set \
    --resource-class baremetal.CEPH <node>

   Replace <node> with the ID of the bare metal node.

8. Add the CephStorage role to your overcloud-baremetal-deploy.yaml file, and define any predictive node placements, resource classes, or other attributes that you want to assign to your nodes:

   - name: Controller
     	  count: 3
   - name: Compute
    	  count: 3
   - name: CephStorage
   	  count: 5
       defaults:
   	    resource_class: baremetal.CEPH

9. Run the provisioning command:

   (undercloud)$ openstack overcloud node provision \
   --stack stack \
   --output /home/stack/templates/overcloud-baremetal-deployed.yaml \
   /home/stack/templates/overcloud-baremetal-deploy.yaml

10. Monitor the provisioning progress in a separate terminal. When provisioning is successful, the node state changes from available to active:

    (undercloud)$ watch openstack baremetal node list

Procedure

1. Log in to the undercloud node as the stack user.

2. Configure the top level domain suffix:

     openstack overcloud ceph deploy \
             --tld "<domain_name>"

   Replace <domain_name> with the required domain name.

     openstack overcloud ceph deploy \
             --tld "example.local"

Procedure

1. Log in to the undercloud node as the stack user.

2. Configure the name of the Ceph Storage cluster by using the following command:

   openstack overcloud ceph deploy \ --cluster <cluster_name>

   $ openstack overcloud ceph deploy \ --cluster central \

1. Log in to the undercloud node as the stack user.

2. Create a YAML format file that defines the custom network attributes called network_data.yaml.

   Important

   Using network isolation, the standard network deployment consists of two storage networks which map to the two Ceph networks:

   • The storage network, storage, maps to the Ceph network, public_network. This network handles storage traffic such as the RBD traffic from the Compute nodes to the Ceph cluster.
   • The storage network, storage_mgmt, maps to the Ceph network, cluster_network. This network handles storage management traffic such as data replication between Ceph OSDs.

3. Use the openstack overcloud ceph deploy command with the --crush-hierarchy option to deploy the configuration.

   openstack overcloud ceph deploy \
           deployed_metal.yaml \
           -o deployed_ceph.yaml \
           --network-data network_data.yaml

   Important

   The openstack overcloud ceph deploy command uses the network data file specified by the --network-data option to determine the networks to be used as the public_network and cluster_network. The command assumes these networks are named storage and storage_mgmt in network data file unless a different name is specified by the --public-network-name and --cluster-network-name options.

   You must use the --network-data option when deploying with network isolation. The default undercloud (192.168.24.0/24) will be used for both the public_network and cluster_network if you do not use this option.

Procedure

1. Log in to the undercloud node as the stack user.
2. Create a standard format initialization file to configure the Ceph cluster. If you have already created a file to include other configuration options, you can add the network configuration to it.

3. Add the following parameters to the [global] section of the file:

   • public_network
   • cluster_network

   • ms_bind_ipv4

     Important

     Ensure the public_network and cluster_network map to the same networks as storage and storage_mgmt.

     The following is an example of a configuration file entry for a network configuration with multiple subnets and custom networking names:

     [global]
     public_network = 172.16.14.0/24,172.16.15.0/24
     cluster_network = 172.16.12.0/24,172.16.13.0/24
     ms_bind_ipv4 = True
     ms_bind_ipv6 = False

4. Use the command openstack overcloud ceph deploy with the --config option to deploy the configuration file.

   $ openstack overcloud ceph deploy \
     --config initial-ceph.conf --network-data network_data.yaml

Procedure

1. Log in to the undercloud node as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Create a configuration file to define the custom CRUSH hierarchy, for example, crush_hierarchy.yaml.

4. Add the following configuration to the file:

   <osd_host>:
     root: default
     rack: <rack_num>
   <osd_host>:
     root: default
     rack: <rack_num>
   <osd_host>:
     root: default
     rack: <rack_num>

   • Replace <osd_host> with the hostnames of the nodes where the OSDs are deployed, for example, ceph-0.
   • Replace <rack_num> with the number of the rack where the OSDs are deployed, for example, r0.

5. Deploy the Ceph cluster with your custom OSD layout:

   openstack overcloud ceph deploy \
           deployed_metal.yaml \
           -o deployed_ceph.yaml \
           --osd-spec osd_spec.yaml \
           --crush-hierarchy crush_hierarchy.yaml

Procedure

1. Log in to the undercloud node as the stack user.
2. Create a YAML format file that defines the custom roles.

3. Deploy the configuration file:

   $ openstack overcloud ceph deploy \
           deployed_metal.yaml \
           -o deployed_ceph.yaml \
           --roles-data custom_roles.yaml

Procedure

1. Log in to the undercloud node as the stack user.

2. Optional: Use a standard format initialization (ini) file to configure the Ceph cluster.

   1. Create the file with configuration options.

      The following is an example of a simple configuration file:

      [global]
      osd_crush_chooseleaf type = 0
      log_file = /var/log/ceph/$cluster-$type.$id.log
      
      [mon]
      mon_cluster_log_to_syslog = true

   2. Save the configuration file.

   3. Use the openstack overcloud ceph deploy --config <configuration_file_name> command to deploy the configuration.

      Replace <configuration_file_name> with the name of the file you created.

      $ openstack overcloud ceph deploy --config initial-ceph.conf

3. Optional: Send configuration values to the cephadm bootstrap command:

   openstack overcloud ceph deploy --force \ --cephadm-extra-args '<optional_arguments>' \

   Replace <optional_arguments> with the configuration values to provide to the underlying command.

   Note

   When using the arguments --log-to-file and --skip-prepare-host, the command openstack overcloud ceph deploy --force \ --cephadm-extra-args '--log-to-file --skip-prepare-host' \ is used.

Procedure

1. Log in to the undercloud node as the stack user.

2. Generate the specification file:

   openstack overcloud ceph spec deployed_metal.yaml -o <specification_file>

   • Replace <specification_file> with the name of the file to generate with the current service specification.

     Note

     The deployed_metal.yaml comes from the output of the openstack overcloud node provision command.

3. Edit the generated file with the required configuration.

4. Deploy the custom service specification:

   openstack overcloud ceph deploy \
      deployed_metal.yaml \
      -o deployed_ceph.yaml \
      --ceph-spec <specification_file>

   • Replace <specification_file> with the name of the custom service specification file.

Procedure

1. Log in to the undercloud node as the stack user.

2. Create a YAML format file that defines the advanced OSD specification.

   The following is an example of a custom OSD specification.

   data_devices:
     rotational: 1
   db_devices:
     rotational: 0

   This example would create an OSD specification where all rotating devices will be data devices and all non-rotating devices will be used as shared devices. When the dynamic Ceph service specification is built, whatever is in the specification file is appended to the section of the specification if the service_type is osd.

3. Save the specification file.

4. Deploy the specification:

   openstack overcloud ceph deploy \ --osd-spec <osd_specification_file>

   Replace <osd_specification_file> with the name of the specification file you created.

   $ openstack overcloud ceph deploy \ --osd-spec osd_spec.yaml \

Procedure

1. Log in to the undercloud as the stack user.

2. Create a file to define the custom configuration.

   vi /home/stack/templates/storage-config.yaml

3. Add a parameter_defaults section to the file.

4. Add the custom configuration parameters. For more information about parameter definitions, see Overcloud parameters.

   parameter_defaults:
       CinderEnableIscsiBackend: false
       CinderEnableRbdBackend: true
       CinderBackupBackend: ceph
       NovaEnableRbdBackend: true
       GlanceBackend: rbd

   Note

   Parameters defined in a custom configuration file override any corresponding default settings in /usr/share/openstack-tripleo-heat-templates/environments/cephadm/cephadm.yaml.

5. Save the file.

Procedure

1. Log in to the undercloud node as the stack user.
2. Create an environment file, such as /home/stack/templates/ceph-config.yaml, to contain the Ceph config parameters and the device classes variables. Alternatively, you can add the following configurations to an existing environment file.

3. Add the CephCrushRules parameters. The CephCrushRules parameter must contain a rule for each class that you define or that Ceph detects automatically. When you create a new pool, if no rule is specified, the rule that you want Ceph to use as the default is selected.

   CephCrushRules:
   
     - name: HDD
       root: default
       type: host
       class: hdd
       default: true
     - name: SSD
       root: default
       type: host
       class: ssd
       default: false

4. Add the CephPools parameter:

   • Use the rule_name parameter to specify the tier for each pool that does not use the default rule. In the following example, the fastpool pool uses the SSD device class that is configured as a fast tier, to manage Block Storage volumes.

   • Use the CinderRbdExtraPools parameter to configure fastpool as a Block Storage back end.

     CephPools:
       - name: fastpool
         rule_name: SSD
         application: rbd
     CinderRbdExtraPools: fastpool

5. Use the following example to ensure that your environment file contains the correct values:

   parameter_defaults:
   CephCrushRules:
   
     - name: replicated_hdd
       default: true
       class: hdd
       root: default
       type: host
       CinderRbdExtraPools: fastpool
       CephPools:
     - name: fastpool
       rule_name: SSD
       application: rbd

6. Include the new environment file in the openstack overcloud deploy command.

   $ openstack overcloud deploy \
   --templates \
   …
   -e <other_overcloud_environment_files> \
   -e /home/stack/templates/ceph-config.yaml  \
   …

   Replace <other_overcloud_environment_files> with the list of other environment files that are part of your deployment.

Procedure

1. Log in to the overcloud Controller node as the tripleo-admin user.

2. To verify that your OSD tiers are successfully set, enter the following command.

   $ sudo cephadm shell ceph osd tree

3. In the resulting tree view, verify that the CLASS column displays the correct device class for each OSD that you set.

4. Also verify that the OSDs are correctly assigned to the device classes with the following command.

   $ sudo cephadm shell ceph osd crush tree --show-shadow

5. Compare the resulting hierarchy with the results of the following command to ensure that the same values apply for each rule.

   $ sudo cephadm shell ceph osd crush rule dump <rule_name>

   • Replace <rule_name> with the name of the rule you want to check.

6. Verify that the rules name and ID that you created are correct according to the crush_rules parameter that you used during deployment.

   $ sudo cephadm shell ceph osd crush rule dump | grep -E "rule_(id|name)"

7. Verify that the Ceph pools are tied to the correct CRUSH rule ID that you retrieved in Step 3.

   $ sudo cephadm shell -- ceph osd dump | grep pool

8. For each pool, ensure that the rule ID matches the rule name that you expect.

Procedure

1. Log in to your undercloud host as the stack user.

2. Generate the default container image preparation file:

   $ sudo openstack tripleo container image prepare default \
     --local-push-destination \
     --output-env-file containers-prepare-parameter.yaml

3. Edit the containers-prepare-parameter.yaml file and make the modifications to suit your requirements. The following example containers-prepare-parameter.yaml file contains the image locations and tags related to the Dashboard services including Grafana, Prometheus, Alertmanager, and Node Exporter. Edit the values depending on your specific scenario:

   parameter_defaults:
       ContainerImagePrepare:
       - push_destination: true
           set:
               ceph_alertmanager_image: ose-prometheus-alertmanager
               ceph_alertmanager_namespace: registry.redhat.io/openshift4
               ceph_alertmanager_tag: v4.12
               ceph_grafana_image: rhceph-6-dashboard-rhel9
               ceph_grafana_namespace: registry.redhat.io/rhceph
               ceph_grafana_tag: 6
               ceph_image: rhceph-6-rhel9
               ceph_namespace: registry.redhat.io/rhceph
               ceph_node_exporter_image: ose-prometheus-node-exporter
               ceph_node_exporter_namespace: registry.redhat.io/openshift4
               ceph_node_exporter_tag: v4.12
               ceph_prometheus_image: ose-prometheus
               ceph_prometheus_namespace: registry.redhat.io/openshift4
               ceph_prometheus_tag: v4.12
               ceph_tag: latest

Procedure

1. Log in to the undercloud node as the stack user.

2. Optional: The Ceph Dashboard network is set by default to the provisioning network. If you want to deploy the Ceph Dashboard and access it through a different network, create an environment file, for example: ceph_dashboard_network_override.yaml. Set CephDashboardNetwork to one of the existing overcloud routed networks, for example external:

     parameter_defaults:
       ServiceNetMap:
         CephDashboardNetwork: external

   Important

   Changing the CephDashboardNetwork value to access the Ceph Dashboard from a different network is not supported after the initial deployment.

3. Include the following environment files in the openstack overcloud deploy command. Include all environment files that are part of your deployment, and the ceph_dashboard_network_override.yaml file if you chose to change the default network:

   $ openstack overcloud deploy \
    --templates \
    -e <overcloud_environment_files> \
    -e /usr/share/openstack-tripleo-heat-templates/environments/cephadm/cephadm.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/cephadm/ceph-dashboard.yaml \
    -e ceph_dashboard_network_override.yaml

   Replace <overcloud_environment_files> with the list of environment files that are part of your deployment.

Procedure

1. Log in to the undercloud as the stack user.

2. Generate the Controller specific role to include the Dashboard composable network:

   $ openstack overcloud roles generate -o /home/stack/roles_data_dashboard.yaml ControllerStorageDashboard Compute BlockStorage ObjectStorage CephStorage

   • A new ControllerStorageDashboard role is generated inside the YAML file defined as the output of the command. You must include this YAML file in the template list when you use the overcloud deploy command. The ControllerStorageDashboard role does not contain CephNFS or network_data_dashboard.yaml.
   • Director provides a network environment file where the composable network is defined. The default location of this file is /usr/share/openstack-tripleo-heat-templates/network_data_dashboard.yaml. You must include this file in the overcloud template list when you use the overcloud deploy command.

3. Include the following environment files, with all environment files that are part of your deployment, in the openstack overcloud deploy command:

   $ openstack overcloud deploy \
     --templates \
     -r /home/stack/roles_data.yaml \
     -n /usr/share/openstack-tripleo-heat-templates/network_data_dashboard.yaml \
     -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
     -e /usr/share/openstack-tripleo-heat-templates/environments/network-environment.yaml \
     -e <overcloud_environment_files> \
     -e /usr/share/openstack-tripleo-heat-templates/environments/cephadm/cephadm.yaml \
     -e /usr/share/openstack-tripleo-heat-templates/environments/cephadm/ceph-dashboard.yaml

   Replace <overcloud_environment_files> with the list of environment files that are part of your deployment.

Procedure

1. Log in to the undercloud as the stack user.

2. Create the following ceph_dashboard_admin.yaml environment file:

     parameter_defaults:
        CephDashboardAdminRO: false

3. Run the overcloud deploy command to update the existing stack and include the environment file you created with all other environment files that are part of your existing deployment:

   $ openstack overcloud deploy \
   --templates \
   -e <existing_overcloud_environment_files> \
   -e ceph_dashboard_admin.yml

   Replace <existing_overcloud_environment_files> with the list of environment files that are part of your existing deployment.

Procedure

1. Log in to the undercloud node as the stack user.

2. Retrieve the dashboard admin login credentials:

   [stack@undercloud ~]$ grep tripleo_cephadm_dashboard_admin_password <config-download>/<stack>/cephadm/cephadm-extra-vars-heat.yml

3. Retrieve the VIP address to access the Ceph Dashboard:

   [stack@undercloud-0 ~]$ grep tripleo_cephadm_dashboard_frontend_vip <config-download>/<stack>/cephadm/cephadm-extra-vars-ansible.yml

4. Use a web browser to point to the front end VIP and access the Dashboard. Director configures and exposes the Dashboard on the provisioning network, so you can use the VIP that you retrieved to access the Dashboard directly on TCP port 8444. Ensure that the following conditions are met:

   • The Web client host is layer 2 connected to the provisioning network.

   • The provisioning network is properly routed or proxied, and it can be reached from the web client host. If these conditions are not met, you can still open a SSH tunnel to reach the Dashboard VIP on the overcloud:

     client_host$ ssh -L 8444:<dashboard_vip>:8444 stack@<your undercloud>

     Replace <dashboard_vip> with the IP address of the control plane VIP that you retrieved.

5. To access the Dashboard, go to: http://localhost:8444 in a web browser and log in with the following details:

   • The default user that cephadm creates: admin.
   • The password in <config-download>/<stack>/cephadm/cephadm-extra-vars-heat.yml.

Procedure

1. Log in to the undercloud node as the stack user.
2. Open the file deployed_ceph.yaml or the file you use in your environment to define the Ceph Storage cluster configuration.
3. Locate the ApplyCephConfigOverridesOnUpdate parameter.
4. Change the ApplyCephConfigOverridesOnUpdate parameter value to false.
5. Save the file.

Procedure

1. Run the following command to source the file:

   $ source ~/overcloudrc

   This loads the necessary environment variables to interact with your overcloud from the undercloud CLI.

2. To return to interacting with the undercloud, run the following command:

   $ source ~/stackrc

Procedure

1. Log in to a Controller node as the tripleo-admin user:

   $ nova list
   $ ssh tripleo-admin@192.168.0.25

2. Check the health of the Ceph cluster:

   $ sudo cephadm shell -- ceph health

   If the Ceph cluster has no issues, the command reports back HEALTH_OK. This means the Ceph cluster is safe to use.

3. Log in to an overcloud node that runs the Ceph monitor service and check the status of all OSDs in the Ceph cluster:

   $ sudo cephadm shell -- ceph osd tree

4. Check the status of the Ceph Monitor quorum:

   $ sudo cephadm shell -- ceph quorum_status

   This shows the monitors participating in the quorum and which one is the leader.

5. Verify that all Ceph OSDs are running:

   $ sudo cephadm shell -- ceph osd stat

Procedure

1. Log in to the undercloud node as the stack user.

2. Source the overcloudrc file:

   $ source overcloudrc

3. Check the Block Storage volume existing types:

   $ cinder type-list

4. Create the new Block Storage volume fast_tier:

   $ cinder type-create fast_tier

5. Check that the Block Storage type is created:

   $ cinder type-list

6. When the fast_tier Block Storage type is available, set the fastpool as the Block Storage volume back end for the new tier that you created:

   $ cinder type-key fast_tier set volume_backend_name=tripleo_ceph_fastpool

7. Use the new tier to create new volumes:

   $ cinder create 1 --volume-type fast_tier --name fastdisk

1. Source the overcloud credentials file:

   $ source ~/<credentials_file>

   • Replace <credentials_file> with the name of your credentials file, for example, overcloudrc.

2. On an undercloud node, create the storage network:

   (overcloud) [stack@undercloud-0 ~]$ openstack network create Storage --provider-network-type vlan --provider-physical-network datacentre --provider-segment 30

   You can enter this command with the following options:

   • For the --provider-physical-network option, use the default value datacentre, unless you set another tag for the br-isolated bridge through NeutronBridgeMappings in your tripleo-heat-templates.
   • For the --provider-segment option, use the value set for the Storage isolated network in your network environment file. If this was not customized, the default environment file is /usr/share/openstack-tripleo-heat-templates/network_data.yaml. The VLAN associated with the Storage network value is 30 unless you modified the isolated network definitions.
   • For the --provider-network-type option, use the value vlan.

Procedure

1. From an undercloud node, enter the following command:

   [stack@undercloud ~]$ source ~/overcloudrc

2. Use the sample command to provision the network. Update the values to suit your environment.

   (overcloud) [stack@undercloud-0 ~]$ openstack subnet create \
   --allocation-pool start=172.17.3.10,end=172.17.3.149 \
   --dhcp \
   --network Storage \
   --subnet-range 172.17.3.0/24 \
   --gateway none StorageSubnet

   • For the --allocation-pool option, replace the start=172.17.3.10,end=172.17.3.149 IP values with the IP values for your network.
   • For the --subnet-range option, replace the 172.17.3.0/24 subnet range with the subnet range for your network.

Procedure

1. From an undercloud node, enter the following command:

   [stack@undercloud ~]$ source ~/overcloudrc

2. Identify the projects that require access:

   (overcloud) [stack@undercloud-0 ~]$ openstack project list
   +----------------------------------+---------+
   | ID                               | Name    |
   +----------------------------------+---------+
   | 06f1068f79d2400b88d1c2c33eacea87 | demo    |
   | 5038dde12dfb44fdaa0b3ee4bfe487ce | service |
   | 820e2d9c956644c2b1530b514127fd0d | admin   |
   +----------------------------------+---------+

3. Create network RBAC rules with the desired projects:

   (overcloud) [stack@undercloud-0 ~]$ openstack network rbac create \
   --action access_as_shared Storage \
   --type network \
   --target-project demo

   Repeat this step for all of the projects that require access to the storage network.

Procedure

1. Source the overcloud credentials file:

   $ source ~/<credentials_file>

   • Replace <credentials_file> with the name of your credentials file, for example, overcloudrc.

2. Set an extra specification on the share type:

   (overcloud) [stack@undercloud-0 ~]$ manila type-create default false
   (overcloud) [stack@undercloud-0 ~]$ manila type-key default set share_backend_name='s!= cephfs'

3. Create a private share type and provide trusted tenants with access to this share type:

   (overcloud) [stack@undercloud-0 ~]$ manila type-create --is-public false nativecephfstype false
   (overcloud) [stack@undercloud-0 ~]$ manila type-key nativecephfstype set share_backend_name='cephfs'
   (overcloud) [stack@undercloud-0 ~]$ manila type-access-add nativecephfstype <trusted_tenant_project_id>

   • Replace <trusted_tenant_project_id> with the ID of the trusted tenant.

Procedure

1. Log in to one of the Controller nodes in the overcloud.

2. Check the connected networks and verify the existence of the VLAN as set in the network_data.yaml file:

   $ ip a
   8: vlan30: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
       link/ether 52:9c:82:7a:d4:75 brd ff:ff:ff:ff:ff:ff
       inet 172.17.3.144/24 brd 172.17.3.255 scope global vlan30
          valid_lft forever preferred_lft forever
       inet6 fe80::509c:82ff:fe7a:d475/64 scope link
          valid_lft forever preferred_lft forever

Procedure

1. Log in to any Controller node.

2. From the Ceph monitor daemon, enter the following command:

   $ sudo podman exec ceph-mon-controller-0 ceph -s
     cluster:
       id:     670dc288-cd36-4772-a4fc-47287f8e2ebf
       health: HEALTH_OK
   
     services:
       mon: 3 daemons, quorum controller-1,controller-2,controller-0 (age 14h)
       mgr: controller-1(active, since 8w), standbys: controller-0, controller-2
       mds: cephfs:1 {0=controller-2=up:active} 2 up:standby
       osd: 15 osds: 15 up (since 8w), 15 in (since 8w)
   
     task status:
       scrub status:
           mds.controller-2: idle
   
     data:
       pools: 6 pools, 192 pgs
       objects: 309 objects, 1.6 GiB
       usage: 21 GiB used, 144 GiB / 165 GiB avail
       pgs: 192 active+clean

   Note

   There is one active MDS and two MDSs on standby.

3. To see a detailed status of the Ceph File System, enter the following command:

   $ sudo ceph fs ls
   
   name: cephfs metadata pool: manila_metadata, data pools: [manila_data]

   Note

   In this example output, cephfs is the name of Ceph File System that director creates to host CephFS shares that users create through the Shared File Systems service.

Procedure

1. From one of the Controller nodes, confirm that openstack-manila-share started:

   $ sudo pcs status resources | grep manila
   
   * Container bundle: openstack-manila-share [cluster.common.tag/rhosp16-openstack-manila-share:pcmklatest]:
   * openstack-manila-share-podman-0	(ocf::heartbeat:podman):	Started controller-0

Procedure

1. Log in to the undercloud.

2. Enter the following command:

   $ source /home/stack/overcloudrc

3. Enter the following command to confirm manila-scheduler and manila-share are enabled:

   $ manila service-list
   
   | Id | Binary          | Host             | Zone | Status | State | Updated_at |
   
   | 2 | manila-scheduler | hostgroup        | nova | enabled | up | 2018-08-08T04:15:03.000000 |
   | 5 | manila-share | hostgroup@cephfs | nova | enabled | up | 2018-08-08T04:15:03.000000 |

1. Source the overcloud credentials file:

   $ source ~/<credentials_file>

   • Replace <credentials_file> with the name of your credentials file, for example, overcloudrc.

2. On an undercloud node, create the StorageNFS network:

   (overcloud) [stack@undercloud-0 ~]$ openstack network create StorageNFS --share  --provider-network-type vlan --provider-physical-network datacentre --provider-segment 70

   You can enter this command with the following options:

   • For the --provider-physical-network option, use the default value datacentre, unless you set another tag for the br-isolated bridge through NeutronBridgeMappings in your tripleo-heat-templates.
   • For the --provider-segment option, use the VLAN value set for the StorageNFS isolated network in the heat template, /usr/share/openstack-tripleo-heat-templates/network_data_ganesha.yaml. This value is 70, unless the deployer modified the isolated network definitions.
   • For the --provider-network-type option, use the value vlan.

Procedure

1. Log in to one of the controllers in the overcloud.

2. Enter the following command to check the connected networks and verify the existence of the VLAN as set in network_data_ganesha.yaml:

   $ ip a
   15: vlan310: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
       link/ether 32:80:cf:0e:11:ca brd ff:ff:ff:ff:ff:ff
       inet 172.16.4.4/24 brd 172.16.4.255 scope global vlan310
          valid_lft forever preferred_lft forever
       inet 172.16.4.7/32 brd 172.16.4.255 scope global vlan310
          valid_lft forever preferred_lft forever
       inet6 fe80::3080:cfff:fe0e:11ca/64 scope link
          valid_lft forever preferred_lft forever

Procedure

1. Log in to the active Controller node.

2. Enter the following command:

   $ sudo ceph -s
   
   cluster:
       id: 3369e280-7578-11e8-8ef3-801844eeec7c
       health: HEALTH_OK
   
     services:
       mon: 3 daemons, quorum overcloud-controller-1,overcloud-controller-2,overcloud-controller-0
       mgr: overcloud-controller-1(active), standbys: overcloud-controller-2, overcloud-controller-0
       mds: cephfs-1/1/1 up  {0=overcloud-controller-0=up:active}, 2 up:standby
       osd: 6 osds: 6 up, 6 in

   There is one active MDS and two MDSs on standby.

3. To check the status of the Ceph file system in more detail, enter the following command and replace <cephfs> with the name of the Ceph file system:

   $ sudo ceph fs ls
   
   name: cephfs, metadata pool: manila_metadata, data pools: [manila_data]

Procedure

1. Enter the following command from one of the Controller nodes to confirm that ceph-nfs and openstack-manila-share started:

   $ pcs status
   
   ceph-nfs       (systemd:ceph-nfs@pacemaker):   Started overcloud-controller-1
   
   podman container: openstack-manila-share [192.168.24.1:8787/rhosp-rhel8/openstack-manila-share:pcmklatest]
      openstack-manila-share-podman-0      (ocf::heartbeat:podman):        Started overcloud-controller-1

Procedure

1. Log in to the undercloud.

2. Enter the following command:

   $ source /home/stack/overcloudrc

3. Enter the following command to confirm manila-scheduler and manila-share are enabled:

   $ manila service-list
   
   | Id | Binary          | Host             | Zone | Status | State | Updated_at |
   
   | 2 | manila-scheduler | hostgroup        | nova | enabled | up | 2018-08-08T04:15:03.000000 |
   | 5 | manila-share | hostgroup@cephfs | nova | enabled | up | 2018-08-08T04:15:03.000000 |

Procedure

1. Log in to a Ceph Monitor or Controller node that is running the ceph-mon service, and disable Ceph Storage cluster rebalancing temporarily:

   $ sudo cephadm shell -- ceph osd set noout
   $ sudo cephadm shell -- ceph osd set norebalance

   Note

   If you have a multistack or distributed compute node (DCN) architecture, you must specify the Ceph cluster name when you set the noout and norebalance flags. For example: sudo cephadm shell -c /etc/ceph/<cluster>.conf -k /etc/ceph/<cluster>.client.keyring.

2. Select the first Ceph Storage node that you want to reboot and log in to the node.

3. Reboot the node:

   $ sudo reboot

4. Wait until the node boots.

5. Log in to the node and check the Ceph cluster status:

   $ sudo cephadm -- shell ceph status

   Check that the pgmap reports all pgs as normal (active+clean).

6. Log out of the node, reboot the next node, and check its status. Repeat this process until you have rebooted all Ceph Storage nodes.

7. When complete, log in to a Ceph Monitor or Controller node that is running the ceph-mon service and enable Ceph cluster rebalancing:

   $ sudo cephadm shell -- ceph osd unset noout
   $ sudo cephadm shell -- ceph osd unset norebalance

   Note

   If you have a multistack or distributed compute node (DCN) architecture, you must specify the Ceph cluster name when you unset the noout and norebalance flags. For example: sudo cephadm shell -c /etc/ceph/<cluster>.conf -k /etc/ceph/<cluster>.client.keyring

8. Perform a final status check to verify that the cluster reports HEALTH_OK:

   $ sudo cephadm shell ceph status

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Modify the ~/overcloud-baremetal-deploy.yaml to add the CephStorage nodes to the deployment.

   The following example file represents an original deployment with three CephStorage nodes.

   - name: CephStorage
     count: 3
     instances:
       - hostname: ceph-0
         name: ceph-0
       - hostname: ceph-1
         name: ceph-2
       - hostname: ceph-2
         name: ceph-2

   The following example modifies this file to add three additional nodes.

   - name: CephStorage
     count: 6
     instances:
       - hostname: ceph-0
         name: ceph-0
       - hostname: ceph-1
         name: ceph-2
       - hostname: ceph-2
         name: ceph-2
       - hostname: ceph-3
         name: ceph-3
       - hostname: ceph-4
         name: ceph-4
       - hostname: ceph-5
         name: ceph-5

4. Use the openstack overcloud node provision command with the updated ~/overcloud-baremetal-deploy.yaml file.

   $ openstack overcloud node provision \
     --stack overcloud \
     --network-config \
     --output ~/overcloud-baremetal-deployed.yaml \
     ~/overcloud-baremetal-deploy.yaml

   Note

   This command will provision the configured nodes and and output an updated copy of ~/overcloud-baremetal-deployed.yaml. The new version updates the CephStorage role. The DeployedServerPortMap and HostnameMap also contains the new storage nodes.

5. Use the openstack overcloud ceph spec command to generate a Ceph specification file.

   $ openstack overcloud ceph spec ~/overcloud-baremetal-deployed.yaml \
       --osd-spec osd_spec.yaml \
       --roles-data roles_data.yaml \
       -o ceph_spec.yaml

   Note

   The files used in the openstack overcloud ceph spec should already be available for use. They are created in the following locations:

   • The overcloud-baremetal-deployed.yaml file was created in the previous step of this procedure.
   • The osd_spec.yaml file was created in Configuring advanced OSD specifications. Providing the OSD specification with the --osd-spec parameter is optional.
   • The roles_data.yaml file was created in Designating nodes for Red Hat Ceph Storage. It is assumed the new nodes are assigned to one of the roles in this file.

   The output of this command will be the ceph_spec.yaml file.

6. Use the openstack overcloud ceph user enable command to create the ceph-admin user on all nodes in the cluster. The ceph-admin user must be present on all nodes to enable SSH access to a node by the Ceph orchestrator.

   $ openstack overcloud ceph user enable ceph_spec.yaml

   Note

   Use the ceph_spec.yaml file created in the previous step.

7. Use Deploying the Ceph daemons using the service specification in the Red Hat Ceph Storage Operations Guide to apply the specification file to the Red Hat Ceph Storage cluster. This specification file now describes the operational state of cluster with the new nodes added.

8. Use the openstack overcloud deploy command with the updated ~/overcloud-baremetal-deployed.yaml file.

   $ openstack overcloud deploy --templates \
     -e /usr/share/openstack-tripleo-heat-templates/environments/cephadm/cephadm.yaml \
     -e deployed_ceph.yaml
     -e overcloud-baremetal-deploy.yaml

1. Log in to the overcloud Controller node as the tripleo-admin user.
2. Use the sudo cephadm shell command to start a Ceph shell.

3. Use the ceph osd tree command to identify OSDs to be removed by server.

   In the following example we want to identify the OSDs of ceph-2 host.

   [ceph: root@oc0-controller-0 /]# ceph osd tree
   ID  CLASS  WEIGHT   TYPE NAME            STATUS  REWEIGHT  PRI-AFF
   -1         0.58557  root default
   -7         0.19519  host ceph-2
    5    hdd  0.04880       osd.5           up      1.00000  1.00000
    7    hdd  0.04880       osd.7           up      1.00000  1.00000
    9    hdd  0.04880       osd.9           up      1.00000  1.00000
   11    hdd  0.04880       osd.11          up      1.00000  1.00000

4. Export the Ceph cluster specification to a YAML file.

   [ceph: root@oc0-controller-0 /]# ceph orch ls --export > spec.yml

5. Edit the exported specification file so that the applicable hosts are removed from the service-type: osd hosts list and the applicable hosts have the placement: hosts value removed.
6. Save the edited file.

7. Apply the modified Ceph specification file.

   [ceph: root@oc0-controller-0 /]# ceph orch apply -i spec.yml

   Important

   If you do not export and edit the Ceph specification file before removing the OSDs, the Ceph Manager will attempt to recreate the OSDs.

8. Use the command ceph orch osd rm --zap <osd_list> to remove the OSDs.

   [ceph: root@oc0-controller-0 /]# ceph orch osd rm --zap 5 7 9 11
   Scheduled OSD(s) for removal
   [ceph: root@oc0-controller-0 /]# ceph orch osd rm status
   OSD_ID HOST   STATE    PG_COUNT REPLACE  FORCE  DRAIN_STARTED_AT
   7      ceph-2 draining 27       False    False  2021-04-23 21:35:51.215361
   9      ceph-2 draining 8        False    False  2021-04-23 21:35:49.111500
   11     ceph-2 draining 14       False    False  2021-04-23 21:35:50.243762

9. Use the command ceph orch osd status to check the status of OSD removal.

   [ceph: root@oc0-controller-0 /]# ceph orch osd rm status
   OSD_ID HOST   STATE    PG_COUNT REPLACE FORCE DRAIN_STARTED_AT
   7      ceph-2 draining 34       False   False 2021-04-23 21:35:51.215361
   11     ceph-2 draining 14       False   False 2021-04-23 21:35:50.243762

   Warning

   Do not proceed with the next step until this command returns no results.

10. Use the command ceph orch host drain <HOST> to drain any remaining daemons.

    [ceph: root@oc0-controller-0 /]# ceph orch host drain ceph-2

11. Use the command ceph orch host rm <HOST> to remove the host.

    [ceph: root@oc0-controller-0 /]# ceph orch host rm ceph-2

    Note

    This node is no longer used by the Ceph cluster but is still managed by director as a bare-metal node.

12. End the Ceph shell session.

    Note

    If scaling down the Ceph cluster is temporary and the nodes removed will be restored later, the scaling up action can increment the count and set provisioned: true on nodes that were previously set provisioned: false. If the node will never reused, it can be set provisioned: false indefinitely and the scaling up action can specify a new instances entry.

    The following file sample provides some examples of each instance.

    - name: Compute
      count: 2
      instances:
      - hostname: overcloud-compute-0
        name: node10
        # Removed from deployment due to disk failure
        provisioned: false
      - hostname: overcloud-compute-1
        name: node11
      - hostname: overcloud-compute-2
        name: node12

13. To remove the node from director, see Scaling down bare-metal nodes in Installing and managing Red Hat OpenStack Platform with director.