Deploying a hyperconverged infrastructure

Procedure

1. Open /home/stack/undercloud.conf and add the following parameter:

   clean_nodes=true

   Copy to Clipboard Toggle word wrap
2. Save /home/stack/undercloud.conf.

3. Update the undercloud configuration.

   openstack undercloud install

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a node inventory JSON file in /home/stack.

2. Enter hardware and power management details for each node.

   For example:

   {
       "nodes":[
           {
               "mac":[
                   "b1:b1:b1:b1:b1:b1"
               ],
               "cpu":"4",
               "memory":"6144",
               "disk":"40",
               "arch":"x86_64",
               "pm_type":"ipmi",
               "pm_user":"admin",
               "pm_password":"p@55w0rd!",
               "pm_addr":"192.0.2.205"
           },
           {
               "mac":[
                   "b2:b2:b2:b2:b2:b2"
               ],
               "cpu":"4",
               "memory":"6144",
               "disk":"40",
               "arch":"x86_64",
               "pm_type":"ipmi",
               "pm_user":"admin",
               "pm_password":"p@55w0rd!",
               "pm_addr":"192.0.2.206"
           },
           {
               "mac":[
                   "b3:b3:b3:b3:b3:b3"
               ],
               "cpu":"4",
               "memory":"6144",
               "disk":"40",
               "arch":"x86_64",
               "pm_type":"ipmi",
               "pm_user":"admin",
               "pm_password":"p@55w0rd!",
               "pm_addr":"192.0.2.207"
           },
           {
               "mac":[
                   "c1:c1:c1:c1:c1:c1"
               ],
               "cpu":"4",
               "memory":"6144",
               "disk":"40",
               "arch":"x86_64",
               "pm_type":"ipmi",
               "pm_user":"admin",
               "pm_password":"p@55w0rd!",
               "pm_addr":"192.0.2.208"
           },
           {
               "mac":[
                   "c2:c2:c2:c2:c2:c2"
               ],
               "cpu":"4",
               "memory":"6144",
               "disk":"40",
               "arch":"x86_64",
               "pm_type":"ipmi",
               "pm_user":"admin",
               "pm_password":"p@55w0rd!",
               "pm_addr":"192.0.2.209"
           },
           {
               "mac":[
                   "c3:c3:c3:c3:c3:c3"
               ],
               "cpu":"4",
               "memory":"6144",
               "disk":"40",
               "arch":"x86_64",
               "pm_type":"ipmi",
               "pm_user":"admin",
               "pm_password":"p@55w0rd!",
               "pm_addr":"192.0.2.210"
           },
           {
               "mac":[
                   "d1:d1:d1:d1:d1:d1"
               ],
               "cpu":"4",
               "memory":"6144",
               "disk":"40",
               "arch":"x86_64",
               "pm_type":"ipmi",
               "pm_user":"admin",
               "pm_password":"p@55w0rd!",
               "pm_addr":"192.0.2.211"
           },
           {
               "mac":[
                   "d2:d2:d2:d2:d2:d2"
               ],
               "cpu":"4",
               "memory":"6144",
               "disk":"40",
               "arch":"x86_64",
               "pm_type":"ipmi",
               "pm_user":"admin",
               "pm_password":"p@55w0rd!",
               "pm_addr":"192.0.2.212"
           },
           {
               "mac":[
                   "d3:d3:d3:d3:d3:d3"
               ],
               "cpu":"4",
               "memory":"6144",
               "disk":"40",
               "arch":"x86_64",
               "pm_type":"ipmi",
               "pm_user":"admin",
               "pm_password":"p@55w0rd!",
               "pm_addr":"192.0.2.213"
           }
       ]
   }

   Copy to Clipboard Toggle word wrap
3. Save the new file.

4. Initialize the stack user:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap

5. Import the JSON inventory file into director and register nodes

   $ openstack overcloud node import <inventory_file>

   Copy to Clipboard Toggle word wrap

   Replace <inventory_file> with the name of the file created in the first step.

6. Assign the kernel and ramdisk images to each node:

   $ openstack overcloud node configure <node>

   Copy to Clipboard Toggle word wrap

Procedure

1. Open your /home/stack/templates/overcloud-baremetal-deploy.yaml file.

2. Add or update the image property for nodes that require the overcloud-hardened-uefi-full image. You can set the image to be used on specific nodes, or for all nodes that use a specific role:

   Specific nodes

   - name: Ceph
     count: 3
     instances:
     - hostname: overcloud-ceph-0
       name: node00
       image:
         href: file:///var/lib/ironic/images/overcloud-minimal.qcow2
     - hostname: overcloud-ceph-1
       name: node01
       image:
         href: file:///var/lib/ironic/images/overcloud-hardened-uefi-full.qcow2
     - hostname: overcloud-ceph-2
       name: node02
       image:
         href: file:///var/lib/ironic/images/overcloud-hardened-uefi-full.qcow2

   Copy to Clipboard Toggle word wrap

   All nodes configured for a specific role

   - name: ComputeHCI
     count: 3
     defaults:
       image:
         href: file:///var/lib/ironic/images/overcloud-hardened-uefi-full.qcow2
     instances:
     - hostname: overcloud-ceph-0
       name: node00
     - hostname: overcloud-ceph-1
       name: node01
     - hostname: overcloud-ceph-2
       name: node02

   Copy to Clipboard Toggle word wrap

3. In the roles_data.yaml role definition file, set the rhsm_enforce parameter to False.

   rhsm_enforce: False

   Copy to Clipboard Toggle word wrap

4. Run the provisioning command:

   (undercloud)$ openstack overcloud node provision \
   --stack overcloud \
   --output /home/stack/templates/overcloud-baremetal-deployed.yaml \
   /home/stack/templates/overcloud-baremetal-deploy.yaml

   Copy to Clipboard Toggle word wrap
5. Pass the overcloud-baremetal-deployed.yaml environment file to the openstack overcloud deploy command.

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc credentials file:

   [stack@director ~]$ source ~/stackrc

   Copy to Clipboard Toggle word wrap

3. Generate a new roles data file named roles_data.yaml that includes the Controller and ComputeHCI roles:

   (undercloud)$ openstack overcloud roles generate Controller ComputeHCI -o ~/roles_data.yaml

   Copy to Clipboard Toggle word wrap

4. Open roles_data.yaml and ensure that it has the following parameters and sections:

   Expand

   Section/Parameter	Value
   Role comment	Role: ComputeHCI
   Role name	name: ComputeHCI
   description	HCI role
   HostnameFormatDefault	%stackname%-novaceph-%index%
   deprecated_nic_config_name	ceph.yaml

   Show more
5. Register the ComputeHCI nodes for the overcloud by adding them to your node definition template, node.json or node.yaml.

6. Inspect the node hardware:

   (undercloud)$ openstack overcloud node introspect --all-manageable --provide

   Copy to Clipboard Toggle word wrap

7. Tag each bare metal node that you want to designate for HCI with a custom HCI resource class:

   (undercloud)$ openstack baremetal node set \
    --resource-class baremetal.HCI <node>

   Copy to Clipboard Toggle word wrap

   Replace <node> with the ID of the bare metal node.

8. Add the ComputeHCI role to your /home/stack/templates/overcloud-baremetal-deploy.yaml file, and define any predictive node placements, resource classes, or other attributes that you want to assign to your nodes:

   - name: Controller
     count: 3
   - name: ComputeHCI
     count: 1
     defaults:
       resource_class: baremetal.HCI

   Copy to Clipboard Toggle word wrap

9. Open the baremetal.yaml file and ensure that it contains the network configuration necessary for HCI. The following is an example configuration:

   - name: ComputeHCI
     count: 3
     hostname_format: compute-hci-%index%
     defaults:
       profile: ComputeHCI
       network_config:
         template: /home/stack/templates/three-nics-vlans/compute-hci.j2
       networks:
       - network: ctlplane
         vif: true
       - network: external
         subnet: external_subnet
       - network: internalapi
         subnet: internal_api_subnet01
       - network: storage
         subnet: storage_subnet01
       - network: storage_mgmt
         subnet: storage_mgmt_subnet01
       - network: tenant
         subnet: tenant_subnet01

   Copy to Clipboard Toggle word wrap
   Note

   Network configuration in the ComputeHCI role contains the storage_mgmt network. CephOSD nodes use this network to make redundant copies of data. The network configuration for the Compute role does not contain this network.

   See Configuring the Bare Metal Provisioning service for more information.

10. Run the provisioning command:

    (undercloud)$ openstack overcloud node provision \
    --stack overcloud \
    --network_config \
    --output /home/stack/templates/overcloud-baremetal-deployed.yaml \
    /home/stack/templates/overcloud-baremetal-deploy.yaml

    Copy to Clipboard Toggle word wrap

11. Monitor the provisioning progress in a separate terminal.

    (undercloud)$ watch openstack baremetal node list

    Copy to Clipboard Toggle word wrap
    Note

    The watch command renews every 2 seconds by default. The -n option sets the renewal timer to a different value.

12. To stop the watch process, enter Ctrl-c.
13. Verification: When provisioning is successful, the node state changes from available to active.

Procedure

1. Verify the disk information from the hardware introspection of each node:

   (undercloud)$ openstack baremetal introspection data save <node_uuid> --file <output_file_name>

   Copy to Clipboard Toggle word wrap
   • Replace <node_uuid> with the UUID of the node.

   • Replace <output_file_name> with the name of the file that contains the output of the node introspection.

     For example, the data for one node might show three disks:

     [
       {
         "size": 299439751168,
         "rotational": true,
         "vendor": "DELL",
         "name": "/dev/sda",
         "wwn_vendor_extension": "0x1ea4dcc412a9632b",
         "wwn_with_extension": "0x61866da04f3807001ea4dcc412a9632b",
         "model": "PERC H330 Mini",
         "wwn": "0x61866da04f380700",
         "serial": "61866da04f3807001ea4dcc412a9632b"
       }
       {
         "size": 299439751168,
         "rotational": true,
         "vendor": "DELL",
         "name": "/dev/sdb",
         "wwn_vendor_extension": "0x1ea4e13c12e36ad6",
         "wwn_with_extension": "0x61866da04f380d001ea4e13c12e36ad6",
         "model": "PERC H330 Mini",
         "wwn": "0x61866da04f380d00",
         "serial": "61866da04f380d001ea4e13c12e36ad6"
       }
       {
         "size": 299439751168,
         "rotational": true,
         "vendor": "DELL",
         "name": "/dev/sdc",
         "wwn_vendor_extension": "0x1ea4e31e121cfb45",
         "wwn_with_extension": "0x61866da04f37fc001ea4e31e121cfb45",
         "model": "PERC H330 Mini",
         "wwn": "0x61866da04f37fc00",
         "serial": "61866da04f37fc001ea4e31e121cfb45"
       }
     ]

     Copy to Clipboard Toggle word wrap

2. Set the root disk for the node by using a unique hardware property:

   (undercloud)$ openstack baremetal node set --property root_device='{<property_value>}' <node-uuid>

   • Replace <property_value> with the unique hardware property value from the introspection data to use to set the root disk.

   • Replace <node_uuid> with the UUID of the node.

     Note

     A unique hardware property is any property from the hardware introspection step that uniquely identifies the disk. For example, the following command uses the disk serial number to set the root disk:

     (undercloud)$ openstack baremetal node set --property root_device='{"serial": "61866da04f380d001ea4e13c12e36ad6"}' 1a4e30da-b6dc-499d-ba87-0bd8a3819bc0

3. Configure the BIOS of each node to first boot from the network and then the root disk.

Procedure

1. Log in to the undercloud node as the stack user.

2. Configure the top level domain suffix:

     openstack overcloud ceph deploy \
             --tld "<domain_name>"

   Copy to Clipboard Toggle word wrap

   Replace <domain_name> with the required domain name.

     openstack overcloud ceph deploy \
             --tld "example.local"

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to the undercloud node as the stack user.

2. Configure the name of the Ceph Storage cluster by using the following command:

   openstack overcloud ceph deploy \ --cluster <cluster_name>

   $ openstack overcloud ceph deploy \ --cluster central \

1. Log in to the undercloud node as the stack user.

2. Create a YAML format file that defines the custom network attributes called network_data.yaml.

   Important

   Using network isolation, the standard network deployment consists of two storage networks which map to the two Ceph networks:

   • The storage network, storage, maps to the Ceph network, public_network. This network handles storage traffic such as the RBD traffic from the Compute nodes to the Ceph cluster.
   • The storage network, storage_mgmt, maps to the Ceph network, cluster_network. This network handles storage management traffic such as data replication between Ceph OSDs.

3. Use the openstack overcloud ceph deploy command with the --crush-hierarchy option to deploy the configuration.

   openstack overcloud ceph deploy \
           deployed_metal.yaml \
           -o deployed_ceph.yaml \
           --network-data network_data.yaml

   Copy to Clipboard Toggle word wrap
   Important

   The openstack overcloud ceph deploy command uses the network data file specified by the --network-data option to determine the networks to be used as the public_network and cluster_network. The command assumes these networks are named storage and storage_mgmt in network data file unless a different name is specified by the --public-network-name and --cluster-network-name options.

   You must use the --network-data option when deploying with network isolation. The default undercloud (192.168.24.0/24) will be used for both the public_network and cluster_network if you do not use this option.

Procedure

1. Log in to the undercloud node as the stack user.
2. Create a standard format initialization file to configure the Ceph cluster. If you have already created a file to include other configuration options, you can add the network configuration to it.

3. Add the following parameters to the [global] section of the file:

   • public_network
   • cluster_network

   • ms_bind_ipv4

     Important

     Ensure the public_network and cluster_network map to the same networks as storage and storage_mgmt.

     The following is an example of a configuration file entry for a network configuration with multiple subnets and custom networking names:

     [global]
     public_network = 172.16.14.0/24,172.16.15.0/24
     cluster_network = 172.16.12.0/24,172.16.13.0/24
     ms_bind_ipv4 = True
     ms_bind_ipv6 = False

     Copy to Clipboard Toggle word wrap

4. Use the command openstack overcloud ceph deploy with the --config option to deploy the configuration file.

   $ openstack overcloud ceph deploy \
     --config initial-ceph.conf --network-data network_data.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to the undercloud node as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Create a configuration file to define the custom CRUSH hierarchy, for example, crush_hierarchy.yaml.

4. Add the following configuration to the file:

   <osd_host>:
     root: default
     rack: <rack_num>
   <osd_host>:
     root: default
     rack: <rack_num>
   <osd_host>:
     root: default
     rack: <rack_num>

   Copy to Clipboard Toggle word wrap
   • Replace <osd_host> with the hostnames of the nodes where the OSDs are deployed, for example, ceph-0.
   • Replace <rack_num> with the number of the rack where the OSDs are deployed, for example, r0.

5. Deploy the Ceph cluster with your custom OSD layout:

   openstack overcloud ceph deploy \
           deployed_metal.yaml \
           -o deployed_ceph.yaml \
           --osd-spec osd_spec.yaml \
           --crush-hierarchy crush_hierarchy.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to the undercloud node as the stack user.
2. Create a YAML format file that defines the custom roles.

3. Deploy the configuration file:

   $ openstack overcloud ceph deploy \
           deployed_metal.yaml \
           -o deployed_ceph.yaml \
           --roles-data custom_roles.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to the undercloud node as the stack user.

2. Optional: Use a standard format initialization (ini) file to configure the Ceph cluster.

   1. Create the file with configuration options.

      The following is an example of a simple configuration file:

      [global]
      osd_crush_chooseleaf type = 0
      log_file = /var/log/ceph/$cluster-$type.$id.log
      
      [mon]
      mon_cluster_log_to_syslog = true

      Copy to Clipboard Toggle word wrap
   2. Save the configuration file.

   3. Use the openstack overcloud ceph deploy --config <configuration_file_name> command to deploy the configuration.

      Replace <configuration_file_name> with the name of the file you created.

      $ openstack overcloud ceph deploy --config initial-ceph.conf

      Copy to Clipboard Toggle word wrap

3. Optional: Send configuration values to the cephadm bootstrap command:

   openstack overcloud ceph deploy --force \ --cephadm-extra-args '<optional_arguments>' \

   Replace <optional_arguments> with the configuration values to provide to the underlying command.

   Note

   When using the arguments --log-to-file and --skip-prepare-host, the command openstack overcloud ceph deploy --force \ --cephadm-extra-args '--log-to-file --skip-prepare-host' \ is used.

Procedure

1. Log in to the undercloud node as the stack user.

2. Generate the specification file:

   openstack overcloud ceph spec deployed_metal.yaml -o <specification_file>

   • Replace <specification_file> with the name of the file to generate with the current service specification.

     Note

     The deployed_metal.yaml comes from the output of the openstack overcloud node provision command.

3. Edit the generated file with the required configuration.

4. Deploy the custom service specification:

   openstack overcloud ceph deploy \
      deployed_metal.yaml \
      -o deployed_ceph.yaml \
      --ceph-spec <specification_file>

   Copy to Clipboard Toggle word wrap
   • Replace <specification_file> with the name of the custom service specification file.

Procedure

1. Log in to the undercloud node as the stack user.

2. Create a YAML format file that defines the advanced OSD specification.

   The following is an example of a custom OSD specification.

   data_devices:
     rotational: 1
   db_devices:
     rotational: 0

   Copy to Clipboard Toggle word wrap

   This example would create an OSD specification where all rotating devices will be data devices and all non-rotating devices will be used as shared devices. When the dynamic Ceph service specification is built, whatever is in the specification file is appended to the section of the specification if the service_type is osd.

   Note

   Custom OSD specification files should not have a full OSD specification.

   The following is an example of a full OSD specification:

   service_type: osd
   service_id: osd_spec_hdd
   placement:
     host_pattern: 'storage-*'
   data_devices:
     paths:
       - /dev/sda
       - /dev/sdb

   Copy to Clipboard Toggle word wrap

   Custom OSD specification files should not have multiple YAML documents in the same file.

   The following is an example of multiple YAML documents in the same file:

   data_devices:
     paths:
     - /dev/sda
     - /dev/sdb
     - /dev/sdc
   ---
   data_devices:
     paths:
     - /dev/sdk
     - /dev/sdl
     - /dev/sdm

   Copy to Clipboard Toggle word wrap

   Configure OSD specifications with these requirements using the Red Hat Ceph Storage command line tools after the initial deployment of your environment. For more information, see Deploying Ceph OSDs using advanced service specifications in the Red Hat Ceph Storage Operations Guide.

3. Save the specification file.

4. Deploy the specification:

   openstack overcloud ceph deploy \ --osd-spec <osd_specification_file>

   Replace <osd_specification_file> with the name of the specification file you created.

   $ openstack overcloud ceph deploy \ --osd-spec osd_spec.yaml \

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap

3. Add the NovaReservedHostMemory parameter to the ceph-overrides.yaml file. The following is a usage example.

   parameter_defaults:
     ComputeHCIParameters:
       NovaReservedHostMemory: 75000

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to the undercloud as the stack user.

2. Create a file to define the custom configuration.

   vi /home/stack/templates/storage-config.yaml

3. Add a parameter_defaults section to the file.

4. Add the custom configuration parameters. For more information about parameter definitions, see Overcloud parameters.

   parameter_defaults:
       CinderEnableIscsiBackend: false
       CinderEnableRbdBackend: true
       CinderBackupBackend: ceph
       NovaEnableRbdBackend: true
       GlanceBackend: rbd

   Copy to Clipboard Toggle word wrap
   Note

   Parameters defined in a custom configuration file override any corresponding default settings in /usr/share/openstack-tripleo-heat-templates/environments/cephadm/cephadm.yaml.

5. Save the file.

Procedure

1. Start a ceph shell.

2. Confirm NUMA and memory target configuration:

   [ceph: root@oc0-controller-0 /]# ceph config dump | grep numa
     osd                                             advanced  osd_numa_auto_affinity                 true
   [ceph: root@oc0-controller-0 /]# ceph config dump | grep autotune
     osd                                             advanced  osd_memory_target_autotune             true
   [ceph: root@oc0-controller-0 /]# ceph config get mgr mgr/cephadm/autotune_memory_target_ratio
   0.200000

   Copy to Clipboard Toggle word wrap

3. Confirm specific OSD configuration:

   [ceph: root@oc0-controller-0 /]# ceph config get osd.11 osd_memory_target
   4294967296
   [ceph: root@oc0-controller-0 /]# ceph config get osd.11 osd_memory_target_autotune
   true
   [ceph: root@oc0-controller-0 /]# ceph config get osd.11 osd_numa_auto_affinity
   true

   Copy to Clipboard Toggle word wrap

4. Confirm specific OSD backfill configuration:

   [ceph: root@oc0-controller-0 /]# ceph config get osd.11 osd_recovery_op_priority
   3
   [ceph: root@oc0-controller-0 /]# ceph config get osd.11 osd_max_backfills
   1
   [ceph: root@oc0-controller-0 /]# ceph config get osd.11 osd_recovery_max_active_hdd
   3
   [ceph: root@oc0-controller-0 /]# ceph config get osd.11 osd_recovery_max_active_ssd
   10

   Copy to Clipboard Toggle word wrap

5. Confirm the reserved_host_memory_mb configuration on the Compute node.

   $ sudo podman exec -ti nova_compute /bin/bash
   bash-5.1$ grep reserved_host_memory_mb /etc/nova/nova.conf

   Copy to Clipboard Toggle word wrap

Procedure

1. Disable and rebalance the Ceph OSD services on the HCI node. This step is necessary because director does not automatically rebalance the Red Hat Ceph Storage cluster when you remove HCI or Red Hat Ceph Storage nodes. For more information, see Scaling the Ceph Storage cluster in Deploying Red Hat Ceph Storage and Red Hat OpenStack Platform together with director for more information.
2. Migrate the instances from the HCI nodes. For more information, see Migrating virtual machines between Compute nodes in the Configuring the Compute service for instance creation guide.
3. Remove the Compute nodes from the overcloud. For more information, see Removing Compute nodes.