Chapter 3. Configuring a Hyperconverged Infrastructure environment

Procedure

1. Create an OpenStackDataPlaneNodeSet CRD file to represent the data plane nodes.

   Note

   Do not create the CR in Red Hat OpenShift yet.

2. Add the ceph-hci-pre service to the list before the configure-os service and remove all other service listings after run-os.

   The following is an example of the edited list:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneNodeSet
   spec:
     ...
     services:
       - download-cache
       - bootstrap
       - configure-network
       - validate-network
       - install-os
       - ceph-hci-pre
       - configure-os
       - ssh-known-hosts
       - run-os
       - reboot-os

   Note

   Note the services that you remove from the list. You add them back to the list later.

3. (Optional) The ceph-hci-pre service prepares EDPM nodes to host Red Hat Ceph Storage services after network configuration using the edpm_ceph_hci_pre edpm-ansible role. By default, the edpm_ceph_hci_pre_enabled_services parameter of this role only contains RBD, RGW, and NFS services. If other services, such as the Dashboard, are deployed with HCI nodes; they must be added to the edpm_ceph_hci_pre_enabled_services parameter list. For more information about this role, see edpm_ceph_hci_pre role.

4. Configure the Red Hat Ceph Storage cluster_network for storage management traffic between OSDs. Modify the CR to set edpm-ansible variables so that the edpm_network_config role configures a storage management network which Ceph uses as the cluster_network.

   The following example has 3 nodes. It assumes the storage management network range is 172.20.0.0/24 and that it is on VLAN23. The bolded lines are additions for the cluster_network:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneNodeSet
   metadata:
     name: openstack-edpm
     namespace: openstack
   spec:
     env:
     - name: ANSIBLE_FORCE_COLOR
       value: "True"
     networkAttachments:
     - ctlplane
     nodeTemplate:
       ansible:
         ansiblePort: 22
         ansibleUser: cloud-admin
         ansibleVars:
           edpm_ceph_hci_pre_enabled_services:
           - ceph_mon
           - ceph_mgr
           - ceph_osd
           - ceph_rgw
           - ceph_nfs
           - ceph_rgw_frontend
           - ceph_nfs_frontend
           edpm_fips_mode: check
           edpm_iscsid_image: {{ registry_url }}/openstack-iscsid:{{ image_tag }}
           edpm_logrotate_crond_image: {{ registry_url }}/openstack-cron:{{ image_tag }}
           edpm_network_config_hide_sensitive_logs: false
           edpm_network_config_os_net_config_mappings:
             edpm-compute-0:
               nic1: 52:54:00:1e:af:6b
               nic2: 52:54:00:d9:cb:f4
             edpm-compute-1:
               nic1: 52:54:00:f2:bc:af
               nic2: 52:54:00:f1:c7:dd
             edpm-compute-2:
               nic1: 52:54:00:dd:33:14
               nic2: 52:54:00:50:fb:c3
           edpm_network_config_template: |
             ---
             {% set mtu_list = [ctlplane_mtu] %}
             {% for network in nodeset_networks %}
             {{ mtu_list.append(lookup(vars, networks_lower[network] ~ _mtu)) }}
             {%- endfor %}
             {% set min_viable_mtu = mtu_list | max %}
             network_config:
             - type: ovs_bridge
               name: {{ neutron_physical_bridge_name }}
               mtu: {{ min_viable_mtu }}
               use_dhcp: false
               dns_servers: {{ ctlplane_dns_nameservers }}
               domain: {{ dns_search_domains }}
               addresses:
               - ip_netmask: {{ ctlplane_ip }}/{{ ctlplane_cidr }}
               routes: {{ ctlplane_host_routes }}
               members:
               - type: interface
                 name: nic2
                 mtu: {{ min_viable_mtu }}
                 # force the MAC address of the bridge to this interface
                 primary: true
             {% for network in nodeset_networks %}
               - type: vlan
                 mtu: {{ lookup(vars, networks_lower[network] ~ _mtu) }}
                 vlan_id: {{ lookup(vars, networks_lower[network] ~ _vlan_id) }}
                 addresses:
                 - ip_netmask:
                     {{ lookup(vars, networks_lower[network] ~ _ip) }}/{{ lookup(vars, networks_lower[network] ~ _cidr) }}
                 routes: {{ lookup(vars, networks_lower[network] ~ _host_routes) }}
             {% endfor %}
           edpm_neutron_metadata_agent_image: {{ registry_url }}/openstack-neutron-metadata-agent-ovn:{{ image_tag }}
           edpm_nodes_validation_validate_controllers_icmp: false
           edpm_nodes_validation_validate_gateway_icmp: false
           edpm_selinux_mode: enforcing
           edpm_sshd_allowed_ranges:
           - 192.168.122.0/24
           - 192.168.111.0/24
           edpm_sshd_configure_firewall: true
           enable_debug: false
           gather_facts: false
           image_tag: current-podified
           neutron_physical_bridge_name: br-ex
           neutron_public_interface_name: eth0
           service_net_map:
             nova_api_network: internalapi
             nova_libvirt_network: internalapi
           storage_mgmt_cidr: "24"
           storage_mgmt_host_routes: []
           storage_mgmt_mtu: 9000
           storage_mgmt_vlan_id: 23
           storage_mtu: 9000
           timesync_ntp_servers:
           - hostname: pool.ntp.org
       ansibleSSHPrivateKeySecret: dataplane-ansible-ssh-private-key-secret
       managementNetwork: ctlplane
       networks:
       - defaultRoute: true
         name: ctlplane
         subnetName: subnet1
       - name: internalapi
         subnetName: subnet1
       - name: storage
         subnetName: subnet1
       - name: tenant
         subnetName: subnet1
     nodes:
       edpm-compute-0:
         ansible:
           host: 192.168.122.100
         hostName: compute-0
         networks:
         - defaultRoute: true
           fixedIP: 192.168.122.100
           name: ctlplane
           subnetName: subnet1
         - name: internalapi
           subnetName: subnet1
         - name: storage
           subnetName: subnet1
         - name: storagemgmt
           subnetName: subnet1
         - name: tenant
           subnetName: subnet1
       edpm-compute-1:
         ansible:
           host: 192.168.122.101
         hostName: compute-1
         networks:
         - defaultRoute: true
           fixedIP: 192.168.122.101
           name: ctlplane
           subnetName: subnet1
         - name: internalapi
           subnetName: subnet1
         - name: storage
           subnetName: subnet1
         - name: storagemgmt
           subnetName: subnet1
         - name: tenant
           subnetName: subnet1
       edpm-compute-2:
         ansible:
           host: 192.168.122.102
         hostName: compute-2
         networks:
         - defaultRoute: true
           fixedIP: 192.168.122.102
           name: ctlplane
           subnetName: subnet1
         - name: internalapi
           subnetName: subnet1
         - name: storage
           subnetName: subnet1
         - name: storagemgmt
           subnetName: subnet1
         - name: tenant
           subnetName: subnet1
     preProvisioned: true
     services:
     - bootstrap
     - configure-network
     - validate-network
     - install-os
     - ceph-hci-pre
     - configure-os
     - ssh-known-hosts
     - run-os
     - reboot-os

   Note

   It is not necessary to add the storage management network to the networkAttachments key.

5. Apply the CR:

   $ oc apply -f <dataplane_cr_file>

   • Replace <dataplane_cr_file> with the name of your file.

     Note

     Ansible does not configure or validate the networks until the OpenStackDataPlaneDeployment CRD is created.

6. Create an OpenStackDataPlaneDeployment CRD, as described in Deploying the data plane, which has the OpenStackDataPlaneNodeSet CRD file defined above to have Ansible configure the services on the data plane nodes.

7. To confirm the network is configured, complete the following steps:

   1. SSH into a data plane node.
   2. Use the ip a command to display configured networks.
   3. Confirm the storage networks are in the list of configured networks.

Procedure

1. Edit the OpenStackDataPlaneNodeSet CR.

2. To define the cephx key and configuration file for the Compute service (nova), use the extraMounts parameter.

   The following is an example of using the extraMounts parameter for this purpose:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlane
   spec:
     roles:
       edpm-compute:
         nodeTemplate:
           extraMounts:
           - extraVolType: Ceph
             volumes:
             - name: ceph
               secret:
                 secretName: ceph-conf-files
             mounts:
             - name: ceph
               mountPath: "/etc/ceph"
               readOnly: true

3. Locate the services list in the CR.

4. Edit the services list to restore all of the services removed in Configuring the data plane node networks. Restoring the full services list allows the remaining jobs to be run that complete the configuration of the HCI environment.

   The following is an example of a full services list with the additional services in bold:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneNodeSet
   spec:
     ...
     services:
       - bootstrap
       - configure-network
       - validate-network
       - install-os
       - ceph-hci-pre
       - configure-os
       - ssh-known-hosts
       - run-os
       - reboot-os
       - install-certs
       - ceph-client
       - ovn
       - neutron-metadata
       - libvirt
       - nova-custom-ceph

   Note

   In addition to restoring the default service list, the ceph-client `service is added after the run-os service. The `ceph-client service configures EDPM nodes as clients of a Red Hat Ceph Storage server. This service distributes the files necessary for the clients to connect to the Red Hat Ceph Storage server.

5. Create a ConfigMap to set the reserved_host_memory_mb parameter to a value appropriate for your configuration.

   The following is an example of a ConfigMap used for this purpose:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: reserved-memory-nova
   data:
     04-reserved-memory-nova.conf: |
       [DEFAULT]
       reserved_host_memory_mb=75000

   Note

   The value for the reserved_host_memory_mb parameter may be set so that the Compute service scheduler does not give memory to a virtual machine that a Ceph OSD on the same server needs. The example reserves 5 GB per OSD for 10 OSDs per host in addition to the default reserved memory for the hypervisor. In an IOPS-optimized cluster, you can improve performance by reserving more memory for each OSD. The 5 GB number is provided as a starting point which can be further tuned if necessary.

6. Add reserved-memory-nova to the configMaps list by editing the OpenStackDataPlaneService/nova-custom-ceph file:

   kind: OpenStackDataPlaneService
   <...>
   spec:
     configMaps:
     - ceph-nova
     - reserved-memory-nova

7. Apply the CR changes.

   $ oc apply -f <dataplane_cr_file>

   • Replace <dataplane_cr_file> with the name of your file.

     Note

     Ansible does not configure or validate the networks until the OpenStackDataPlaneDeployment CRD is created.

8. Create an OpenStackDataPlaneDeployment CRD, as described in Deploying the data plane, which has the OpenStackDataPlaneNodeSet CRD file defined above to have Ansible configure the services on the data plane nodes.