Deploying an overcloud in a Red Hat OpenShift Container Platform cluster with director Operator

Procedure

1. Create the OSPdO project:

   $ oc new-project openstack

2. Obtain the latest osp-director-operator-bundle image from https://catalog.redhat.com/software/containers/search.
3. Download the Operator Package Manager (opm) tool from https://console.redhat.com/openshift/downloads.

4. Use the opm tool to create an index image:

   $ BUNDLE_IMG="registry.redhat.io/rhosp-rhel9/osp-director-operator-bundle:1.3.1"
   $ INDEX_IMG="quay.io/<account>/osp-director-operator-index:x.y.z-a"
   $ opm index add --bundles ${BUNDLE_IMG} --tag ${INDEX_IMG} -u podman --pull-tool podman

5. Push the index image to your registry:

   $ podman push ${INDEX_IMG}

6. Create an environment file to configure the CatalogSource, OperatorGroup, and Subscription CRs required to install OSPdO, for example, osp-director-operator.yaml.

7. To configure the CatalogSource CR, add the following configuration to osp-director-operator.yaml:

   apiVersion: operators.coreos.com/v1alpha1
   kind: CatalogSource
   metadata:
     name: osp-director-operator-index
     namespace: openstack
   spec:
     sourceType: grpc
     image: quay.io/<account>/osp-director-operator-index:x.y.z-a

   For information about how to apply the Quay authentication so that the Operator deployment can pull the image, see Accessing images for Operators from private registries.

8. To configure the OperatorGroup CR, add the following configuration to osp-director-operator.yaml:

   ---
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: "osp-director-operator-group"
     namespace: openstack
   spec:
     targetNamespaces:
     - openstack

9. To configure the Subscription CR, add the following configuration to osp-director-operator.yaml:

   ---
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: osp-director-operator-subscription
     namespace: openstack
   spec:
     config:
       env:
       - name: WATCH_NAMESPACE
         value: openstack,openshift-machine-api,openshift-sriov-network-operator
     source: osp-director-operator-index
     sourceNamespace: openstack
     name: osp-director-operator

10. Create the new CatalogSource, OperatorGroup, and Subscription CRs within the openstack namespace:

    $ oc apply -f osp-director-operator.yaml

11. Confirm that you have installed OSPdO, osp-director-operator.openstack, by listing the installed operators:

    $ oc get operators
    NAME                                     AGE
    osp-director-operator.openstack          5m

Procedure

1. Download a RHEL 9.2 QCOW2 image from the Product Download section of the Red Hat Customer Portal to your workstation.

2. Optional: Add a custom CA certificate:

   $ sudo -s
   $ export LIBGUESTFS_BACKEND=direct
   $ virt-copy-in -a <local_path_to_image> <ca_certificate>.pem /etc/pki/ca-trust/source/anchors/

   You might want to add a custom CA certificate to secure LDAP communication for the Identity service, or to communicate with any non-RHOSP system.

3. Create a script to customize the image to assign predictable network interface names:

   #!/bin/bash
   set -eux
   
   if [ -e /etc/kernel/cmdline ]; then
     echo 'Updating /etc/kernel/cmdline'
     sed -i -e "s/^\(.*\)net\.ifnames=0\s*\(.*\)/\1\2/" /etc/kernel/cmdline
   fi
   
   source /etc/default/grub
   if grep -q "net.ifnames=0" <<< "$GRUB_CMDLINE_LINUX"; then
     echo 'Updating /etc/default/grub'
     sed -i -e "s/^\(GRUB_CMDLINE_LINUX=.*\)net\.ifnames=0\s*\(.*\)/\1\2/" /etc/default/grub
   fi
   if [ "$GRUB_ENABLE_BLSCFG" == "true" ]; then
     echo 'Fixing BLS entries'
     find /boot/loader/entries -type f -exec sed -i -e "s/^\(.*\)net\.ifnames=0\s*\(.*\)/\1\2/" {} \;
   fi
   # Always do this, on RHEL8 with BLS we still need it as the BLS entry uses $kernelopts from grubenv
   echo 'Running grub2-mkconfig'
   grub2-mkconfig -o /etc/grub2.cfg
   grub2-mkconfig -o /etc/grub2-efi.cfg
   rm -f /etc/sysconfig/network-scripts/ifcfg-ens* /etc/sysconfig/network-scripts/ifcfg-eth*
   update-ca-trust extract

4. Run the image customization script:

   $ sudo -s
   $ export LIBGUESTFS_BACKEND=direct
   $ chmod 755 customize_image.sh
   $ virt-customize -a <local_path_to_image> --run customize_image.sh --truncate /etc/machine-id

5. Use virtctl to upload the image to OpenShift Virtualization:

   $ virtctl image-upload dv <datavolume_name> -n openstack \
    --size=<size> --image-path=<local_path_to_image> \
    --storage-class <storage_class> --access-mode <access_mode> --insecure

   • Replace <datavolume_name> with the name of the data volume, for example, openstack-base-img.
   • Replace <size> with the size of the data volume required for your environment, for example, 500Gi. The minimum size is 500GB.

   • Replace <storage_class> with the required storage class from your cluster. Use the following command to retrieve the available storage classes:

     $ oc get storageclass

   • Replace <access_mode> with the access mode for the PVC. The default value is ReadWriteOnce.

Procedure

1. Create the git-secret Secret resource:

   $ oc create secret generic <secret_name> -n <namespace> \
    --from-file=git_ssh_identity=<path_to_private_SSH_key> \
    --from-literal=git_url=<git_server_URL>

   • Replace <secret_name> with the name of the secret, in this case, git-secret.
   • Replace <namespace> with the name of the namespace to create the secret in, for example, openstack.
   • Replace <path_to_private_SSH_key> with the path to the private key to access the Git repository.
   • Replace <git_server_URL> with the SSH URL of the git repository that stores the OSPdO configuration, for example, ssh://<user>@<server>:2202/repo.git.

2. Verify that the Secret resource is created:

   $ oc get secret/git-secret -n openstack

Procedure

1. Convert your chosen password to a base64 value:

   $ echo -n "p@ssw0rd!" | base64
   cEBzc3cwcmQh

   Important

   The -n option removes the trailing newline from the echo output.

2. Create a file named openstack-userpassword.yaml on your workstation. Include the following resource specification for the Secret in the file:

   apiVersion: v1
   kind: Secret
   metadata:
     name: <secret_name>
     namespace: openstack
   data:
     NodeRootPassword: "<password>"

   • Replace <secret_name> with the name of this Secret resource, for example, userpassword.
   • Replace <password> with your base64 encoded password.

3. Create the userpassword Secret:

   $ oc create -f openstack-userpassword.yaml -n openstack

Procedure

1. Create a file named openstacknetconfig.yaml on your workstation.

2. Add the following configuration to openstacknetconfig.yaml to create the OpenStackNetConfig custom resource (CR):

   apiVersion: osp-director.openstack.org/v1beta1
   kind: OpenStackNetConfig
   metadata:
     name: openstacknetconfig

3. Configure network attachment definitions for the bridges you require for your network. For example, add the following configuration to openstacknetconfig.yaml to create the RHOSP bridge network attachment definition br-osp, and set the nodeNetworkConfigurationPolicy option to create a Linux bridge:

   apiVersion: osp-director.openstack.org/v1beta1
   kind: OpenStackNetConfig
   metadata:
     name: openstacknetconfig
   spec:
     attachConfigurations:
       br-osp: 1
         nodeNetworkConfigurationPolicy:
           nodeSelector:
             node-role.kubernetes.io/worker: ""
           desiredState:
             interfaces:
             - bridge:
                 options:
                   stp:
                     enabled: false
                 port:
                 - name: enp6s0 2
               description: Linux bridge with enp6s0 as a port
               name: br-osp 3
               state: up
               type: linux-bridge
               mtu: 1500 4
     # optional DnsServers list
     dnsServers:
     - 192.168.25.1
     # optional DnsSearchDomains list
     dnsSearchDomains:
     - osptest.test.metalkube.org
     - some.other.domain
     # DomainName of the OSP environment
     domainName: osptest.test.metalkube.org

   1

   The network attachment definition for br-osp.

   2

   The NIC / Ethernet device to attach to on each host.

   3

   The interface name.

   4

   The maximum amount of data that can be transferred in a single network packet.

4. Optional: To use Jumbo Frames for a bridge, configure the bridge interface to use Jumbo Frames and update the value of mtu for the bridge:

   apiVersion: osp-director.openstack.org/v1beta1
   kind: OpenStackNetConfig
   metadata:
     name: openstacknetconfig
   spec:
     attachConfigurations:
       br-osp:
         nodeNetworkConfigurationPolicy:
           nodeSelector:
             node-role.kubernetes.io/worker: ""
           desiredState:
             interfaces:
             - bridge:
                 options:
                   stp:
                     enabled: false
                 port:
                 - name: enp6s0
               description: Linux bridge with enp6s0 as a port
               name: br-osp
               state: up
               type: linux-bridge
               mtu: 9000 1
             - name: enp6s0 2
               description: Configuring enp6s0 on workers
               type: ethernet
               state: up
               mtu: 9000
     ...

   1

   Update the maximum amount of data that can be transferred in a single network packet with Jumbo Frames.

   2

   Configure the bridge interface to use Jumbo Frames.

5. Define each overcloud network. The following example creates a control plane network and an isolated network for InternalAPI traffic:

   spec:
     ...
     networks:
     - name: Control 1
       nameLower: ctlplane 2
       subnets: 3
       - name: ctlplane 4
         ipv4: 5
           allocationEnd: 172.22.0.250
           allocationStart: 172.22.0.100
           cidr: 172.22.0.0/24
           gateway: 172.22.0.1
         attachConfiguration: br-osp 6
     - name: InternalApi 7
       nameLower: internal_api
       mtu: 1350
       subnets:
       - name: internal_api
         attachConfiguration: br-osp
         vlan: 20 8
         ipv4:
           allocationEnd: 172.17.0.250
           allocationStart: 172.17.0.10
           cidr: 172.17.0.0/24
     ...

   1

   The name of the network, for example, Control.

   2

   The lowercase version of the network name, for example, ctlplane.

   3

   The subnet specifications.

   4

   The name of the subnet, for example, ctlplane.

   5

   Details of the IPv4 subnet with allocationStart, allocationEnd, cidr, gateway, and an optional list of routes with destination and nexthop.

   6

   The network attachment definition to connect the network to. In this example, the RHOSP bridge, br-osp, is connected to a NIC on each worker.

   7

   The network definition for a composable network. To use the default RHOSP networks, you must create an OpenStackNetConfig resource for each network. For information on the default RHOSP networks, see Default Red Hat OpenStack Platform networks. To use different networks, you must create a custom network_data.yaml file. For information on creating a custom network_data.yaml file, see Configuring overcloud networking.

   8

   The network VLAN. For information on the default RHOSP networks, see Default Red Hat OpenStack Platform networks. For more information on virtual machine bridging with the OpenStackNetConfig CRD, see Understanding virtual machine bridging with the OpenStackNetConfig CRD.

6. Optional: Reserve static IP addresses for networks on specific nodes:

   spec:
     ...
     reservations:
       controller-0:
         ipReservations:
           ctlplane: 172.22.0.120
       compute-0:
         ipReservations:
           ctlplane: 172.22.0.140
           internal_api: 172.17.0.40
           storage: 172.18.0.40
           tenant: 172.20.0.40

   Note

   Reservations have precedence over any autogenerated IP addresses.

7. Save the openstacknetconfig.yaml definition file.

8. Create the overcloud network:

   $ oc create -f openstacknetconfig.yaml -n openstack

9. To verify that the overcloud network is created, view the resources for the overcloud network:

   $ oc get openstacknetconfig/openstacknetconfig

10. View the OpenStackNetConfig API and child resources:

    $ oc get openstacknetconfig/openstacknetconfig -n openstack
    $ oc get openstacknetattachment -n openstack
    $ oc get openstacknet -n openstack

    If you see errors, check the underlying network-attach-definition and node network configuration policies:

    $ oc get network-attachment-definitions -n openstack
    $ oc get nncp

Procedure

1. Navigate to the location of your custom templates:

   $ cd ~/custom_templates

2. Archive the templates into a gzipped tarball:

   $ tar -cvzf custom-config.tar.gz *.yaml

3. Create the tripleo-tarball-config ConfigMap CR and use the tarball as data:

   $ oc create configmap tripleo-tarball-config --from-file=custom-config.tar.gz -n openstack

4. Verify that the ConfigMap CR is created:

   $ oc get configmap/tripleo-tarball-config -n openstack

Procedure

1. Create the heat-env-config ConfigMap object:

   $ oc create configmap -n openstack heat-env-config \
    --from-file=~/<dir_custom_environment_files>/ \
    --dry-run=client -o yaml | oc apply -f -

   • Replace <dir_custom_environment_files> with the directory that contains the environment files you want to use in your overcloud deployment. The ConfigMap object stores these as individual data entries.

2. Verify that the heat-env-config ConfigMap object contains all the required environment files:

   $ oc get configmap/heat-env-config -n openstack

Procedure

1. Create a file named openstack-controller.yaml on your workstation. Include the resource specification for the Controller nodes. The following example defines a specification for a control plane that consists of 3 Controller nodes:

   apiVersion: osp-director.openstack.org/v1beta2
   kind: OpenStackControlPlane
   metadata:
     name: overcloud 1
     namespace: openstack 2
   spec: 3
     openStackClientNetworks:
           - ctlplane
           - internal_api
           - external
     openStackClientStorageClass: host-nfs-storageclass
     passwordSecret: userpassword 4
     virtualMachineRoles:
       Controller:
         roleName: Controller
         roleCount: 3
         networks:
           - ctlplane
           - internal_api
           - external
           - tenant
           - storage
           - storage_mgmt
         cores: 12
         memory: 64
         rootDisk:
           diskSize: 500
           baseImageVolumeName: openstack-base-img 5
           storageClass: host-nfs-storageclass 6
           storageAccessMode:  ReadWriteMany
           storageVolumeMode: Filesystem
         # optional configure additional discs to be attached to the VMs,
         # need to be configured manually inside the VMs where to be used.
         additionalDisks:
           - name: datadisk
             diskSize: 500
             storageClass: host-nfs-storageclass
             storageAccessMode:  ReadWriteMany
             storageVolumeMode: Filesystem
     openStackRelease: "17.1"

   1

   The name of the overcloud control plane, for example, overcloud.

   2

   The OSPdO namespace, for example, openstack.

   3

   The configuration for the control plane.

   4

   Optional: The Secret resource that provides root access on each node to users with the password.

   5

   The name of the data volume that stores the base operating system image for your Controller VMs. For more information on creating the data volume, see Creating a data volume for the base operating system.

   6

   For information on configuring Red Hat OpenShift Container Platform (RHOCP) storage, see Dynamic provisioning.

2. Save the openstack-controller.yaml file.

3. Create the control plane:

   $ oc create -f openstack-controller.yaml -n openstack

4. Wait until RHOCP creates the resources related to OpenStackControlPlane CR. OSPdO also creates an OpenStackClient pod that you can access through a remote shell to run RHOSP commands.

Verification

1. View the resource for the control plane:

   $ oc get openstackcontrolplane/overcloud -n openstack

2. View the OpenStackVMSet resources to verify the creation of the control plane VM set:

   $ oc get openstackvmsets -n openstack

3. View the VMs to verify the creation of the control plane OpenShift Virtualization VMs:

   $ oc get virtualmachines -n openstack

4. Test access to the openstackclient remote shell:

   $ oc rsh -n openstack openstackclient

Procedure

1. Create a file named openstack-compute.yaml on your workstation. Include the resource specification for the Compute nodes. The following example defines a specification for 1 Compute node:

   apiVersion: osp-director.openstack.org/v1beta1
   kind: OpenStackBaremetalSet
   metadata:
     name: compute 1
     namespace: openstack 2
   spec: 3
     count: 1
     baseImageUrl: http://<source_host>/rhel-9.2-x86_64-kvm.qcow2
     deploymentSSHSecret: osp-controlplane-ssh-keys
     # If you manually created an OpenStackProvisionServer, you can use it here,
     # otherwise director Operator will create one for you (with `baseImageUrl` as the image that it server)
     # to use with this OpenStackBaremetalSet
     # provisionServerName: openstack-provision-server
     ctlplaneInterface: enp2s0
     networks:
       - ctlplane
       - internal_api
       - tenant
       - storage
     roleName: Compute
     passwordSecret: userpassword 4

   1

   The name of the Compute node bare-metal set, for example, compute.

   2

   The OSPdO namespace, for example, openstack.

   3

   The configuration for the Compute nodes.

   4

   Optional: The Secret resource that provides root access on each node to users with the password.

2. Save the openstack-compute.yaml file.

3. Create the Compute nodes:

   $ oc create -f openstack-compute.yaml -n openstack

Verification

1. View the resource for the Compute nodes:

   $ oc get openstackbaremetalset/compute -n openstack

2. View the bare-metal machines that RHOCP manages to verify the creation of the Compute nodes:

   $ oc get baremetalhosts -n openshift-machine-api

Procedure

1. Create a file named openstack-provision.yaml on your workstation. Include the resource specification for the Provisioning server. The following example defines a specification for a Provisioning server using a specific RHEL 9.2 QCOW2 images:

   apiVersion: osp-director.openstack.org/v1beta1
   kind: OpenStackProvisionServer
   metadata:
     name: openstack-provision-server 1
     namespace: openstack 2
   spec:
     baseImageUrl: http://<source_host>/rhel-9.2-x86_64-kvm.qcow2 3
     port: 8080 4

   1

   The name that identifies the OpenStackProvisionServer CR.

   2

   The OSPdO namespace, for example, openstack.

   3

   The initial source of the RHEL QCOW2 image for the Provisioning server. The image is downloaded from this remote source when the server is created.

   4

   The Provisioning server port, set to 8080 by default. You can change it for a specific port configuration.

   For further descriptions of the values you can use to configure your OpenStackProvisionServer CR, view the OpenStackProvisionServer CRD specification schema:

   $ oc describe crd openstackprovisionserver

2. Save the openstack-provision.yaml file.

3. Create the Provisioning Server:

   $ oc create -f openstack-provision.yaml -n openstack

4. Verify that the resource for the Provisioning server is created:

   $ oc get openstackprovisionserver/openstack-provision-server -n openstack

Procedure

1. Create a file named openstack-config-generator.yaml on your workstation. Include the resource specification to generate the Ansible playbooks. The following example defines a specification to generate the playbooks:

   apiVersion: osp-director.openstack.org/v1beta1
   kind: OpenStackConfigGenerator
   metadata:
     name: default 1
     namespace: openstack
   spec:
     enableFencing: true 2
     gitSecret: git-secret 3
     imageURL: registry.redhat.io/rhosp-rhel8/openstack-tripleoclient:17.1
     heatEnvConfigMap: heat-env-config 4
     # List of heat environment files to include from tripleo-heat-templates/environments
     heatEnvs: 5
     - ssl/tls-endpoints-public-dns.yaml
     - ssl/enable-tls.yaml
     tarballConfigMap: tripleo-tarball-config 6

   1

   The name of the config generator, which is default by default.

   2

   Set to true to enable the automatic creation of required heat environment files to enable fencing. Production RHOSP environments must have fencing enabled. Virtual machines running Pacemaker require the fence-agents-kubevirt package.

   3

   Set to the ConfigMap object that contains the Git authentication credentials, by default git-secret.

   4

   The ConfigMap object that contains your custom environment files, by default heat-env-config.

   5

   A list of the default heat environment files, provided by TripleO in the tripleo-heat-templates/environments directory, to use to generate the playbooks.

   6

   The ConfigMap object that contains the tarball with your custom heat templates, by default tripleo-tarball-config.

2. Optional: To change the location of the container images the OpenStackConfigGenerator CR uses to create the ephemeral heat service, add the following configuration to your openstack-config-generator.yaml file:

   spec:
     ...​
     ephemeralHeatSettings:
       heatAPIImageURL: <heat_api_image_location>
       heatEngineImageURL: <heat_engine_image_location>
       mariadbImageURL: <mariadb_image_location>
       rabbitImageURL: <rabbitmq_image_location>

   • Replace <heat_api_image_location> with the path to the directory where you host your heat API image, openstack-heat-api.
   • Replace <heat_engine_image_location> with the path to the directory where you host your heat engine image, openstack-heat-engine.
   • Replace <mariadb_image_location> with the path to the directory where you host your MariaDB image, openstack-mariadb.
   • Replace <rabbitmq_image_location> with the path to the directory where you host your RabbitMQ image, openstack-rabbitmq.

3. Optional: To create the Ansible playbooks for configuration generation in debug mode, add the following configuration to your openstack-config-generator.yaml file:

   spec:
     ...​
     interactive: true

   For more information on debugging an OpenStackConfigGenerator pod in interactive mode, see Debugging configuration generation.

4. Save the openstack-config-generator.yaml file.

5. Create the Ansible config generator:

   $ oc create -f openstack-config-generator.yaml -n openstack

6. Verify that the resource for the config generator is created:

   $ oc get openstackconfiggenerator/default -n openstack

Procedure

1. Open an RSH connection to the OpenStackClient pod:

   $ oc rsh -n openstack openstackclient

2. Change to the cloud-admin home directory:

   $ cd /home/cloud-admin

3. Create a playbook that uses the redhat_subscription modules to register your nodes. For example, the following playbook registers Controller nodes:

   ---
   - name: Register Controller nodes
     hosts: Controller
     become: yes
     vars:
       repos:
         - rhel-9-for-x86_64-baseos-eus-rpms
         - rhel-9-for-x86_64-appstream-eus-rpms
         - rhel-9-for-x86_64-highavailability-eus-rpms
         - openstack-17.1-for-rhel-9-x86_64-rpms
         - fast-datapath-for-rhel-9-x86_64-rpms
         - rhceph-6-tools-for-rhel-9-x86_64-rpms
     tasks:
       - name: Register system 1
         redhat_subscription:
           username: myusername
           password: p@55w0rd!
           org_id: 1234567
           release: 9.2
           pool_ids: 1a85f9223e3d5e43013e3d6e8ff506fd
       - name: Disable all repos 2
         command: "subscription-manager repos --disable *"
       - name: Enable Controller node repos 3
         command: "subscription-manager repos --enable {{ item }}"
         with_items: "{{ repos }}"

   1

   Task that registers the node.

   2

   Task that disables any auto-enabled repositories.

   3

   Task that enables only the repositories relevant to the Controller node. The repositories are listed with the repos variable.

4. Register the overcloud nodes to the required repositories:

   $ ansible-playbook -i /home/cloud-admin/ctlplane-ansible-inventory ./rhsm.yaml

Procedure

1. Retrieve the hash/digest of the latest OpenStackConfigVersion object, which represents the Ansible playbooks that should be used to configure the overcloud:

   $ oc get -n openstack --sort-by {.metadata.creationTimestamp} openstackconfigversion -o json

2. Create a file named openstack-deployment.yaml on your workstation and include the resource specification to the Ansible playbooks:

   apiVersion: osp-director.openstack.org/v1beta1
   kind: OpenStackDeploy
   metadata:
     name: default
   spec:
     configVersion: <config_version>
     configGenerator: default

   • Replace <config_version> with the Ansible playbooks hash/digest retrieved in step 1, for example, n5fch96h548h75hf4hbdhb8hfdh676h57bh96h5c5h59hf4h88h....
3. Save the openstack-deployment.yaml file.

4. Create the OpenStackDeploy resource:

   $ oc create -f openstack-deployment.yaml -n openstack

   As the deployment runs, it creates a Kubernetes job to execute the Ansible playbooks. You can view the logs of the job to watch the Ansible playbooks running:

   $ oc logs -f jobs/deploy-openstack-default

   You can also manually access the executed Ansible playbooks by logging into the openstackclient pod. You can find the ansible playbooks and the ansible.log file for the current deployment in /home/cloud-admin/work/directory.

Procedure

1. Open a Remote Shell (RSH) connection to the OpenStackConfigGenerator pod:

   $ oc rsh $(oc get pod -o name -l job-name=generate-config-default)

2. Inspect the files and playbook rendering:

   $ ls -la /home/cloud-admin/
   ...
   config 1
   config-custom 2
   config-passwords 3
   create-playbooks.sh 4
   process-heat-environment.py 5
   tht-tars 6

Procedure

1. Access the remote shell for openstackclient:

   $ oc rsh -n openstack openstackclient

2. Unset the OS_CLOUD environment variable:

   $ unset OS_CLOUD

3. Change to the cloud-admin directory:

   $ cd /home/cloud-admin/

4. Generate a new roles_data.yaml file with the Controller and ComputeHCI roles:

   $ openstack overcloud roles generate -o roles_data.yaml Controller ComputeHCI

5. Exit the openstackclient pod:

   $ exit

6. Copy the custom roles_data.yaml file from the openstackclient pod to your custom templates directory:

   $ oc cp openstackclient:/home/cloud-admin/roles_data.yaml custom_templates/roles_data.yaml -n openstack

Procedure

1. Create a directory for your custom templates:

   $ mkdir custom_templates

2. Create a custom template file named multiple_nics_vlans_dvr.j2 in your custom_templates directory.
3. Add configuration for the NICs of your bare-metal nodes to your multiple_nics_vlans_dvr.j2 file. For an example NIC configuration file, see Custom NIC heat template for HCI Compute nodes.

4. Create a directory for your custom environment files:

   $ mkdir custom_environment_files

5. Map the NIC template for your overcloud role in the network-environment.yaml environment file in your custom_environment_files directory:

   parameter_defaults:
     ComputeHCINetworkConfigTemplate: 'multiple_nics_vlans_dvr.j2'

Procedure

1. Navigate to the location of your custom templates:

   $ cd ~/custom_templates

2. Archive the templates into a gzipped tarball:

   $ tar -cvzf custom-config.tar.gz *.yaml

3. Create the tripleo-tarball-config ConfigMap CR and use the tarball as data:

   $ oc create configmap tripleo-tarball-config --from-file=custom-config.tar.gz -n openstack

4. Verify that the ConfigMap CR is created:

   $ oc get configmap/tripleo-tarball-config -n openstack

Procedure

1. Create the heat-env-config ConfigMap object:

   $ oc create configmap -n openstack heat-env-config \
    --from-file=~/<dir_custom_environment_files>/ \
    --dry-run=client -o yaml | oc apply -f -

   • Replace <dir_custom_environment_files> with the directory that contains the environment files you want to use in your overcloud deployment. The ConfigMap object stores these as individual data entries.

2. Verify that the heat-env-config ConfigMap object contains all the required environment files:

   $ oc get configmap/heat-env-config -n openstack

Procedure

1. Create a file named openstack-hcicompute.yaml on your workstation. Include the resource specification for the HCI Compute nodes. For example, the specification for 3 HCI Compute nodes is as follows:

   apiVersion: osp-director.openstack.org/v1beta1
   kind: OpenStackBaremetalSet
   metadata:
     name: computehci 1
     namespace: openstack 2
   spec: 3
     count: 3
     baseImageUrl: http://<source_host>/rhel-9.2-x86_64-kvm.qcow2
     deploymentSSHSecret: osp-controlplane-ssh-keys
     ctlplaneInterface: enp8s0
     networks:
       - ctlplane
       - internal_api
       - tenant
       - storage
       - storage_mgmt
     roleName: ComputeHCI
     passwordSecret: userpassword 4

   1

   The name of the HCI Compute node bare metal set, for example, computehci.

   2

   The OSPdO namespace, for example, openstack.

   3

   The configuration for the HCI Compute nodes.

   4

   Optional: The Secret resource that provides root access on each node to users with the password.

2. Save the openstack-hcicompute.yaml file.

3. Create the HCI Compute nodes:

   $ oc create -f openstack-hcicompute.yaml -n openstack

4. Verify that the resource for the HCI Compute nodes is created:

   $ oc get openstackbaremetalset/computehci -n openstack

5. To verify the creation of the HCI Compute nodes, view the bare-metal machines that RHOCP manages:

   $ oc get baremetalhosts -n openshift-machine-api

6. Create the Ansible playbooks for overcloud configuration with the OpenStackConfigGenerator CRD. For more information, see Creating Ansible playbooks for overcloud configuration with the OpenStackConfigGenerator CRD.
7. Register the operating system of your overcloud. For more information, see Registering the operating system of your overcloud.
8. Apply the overcloud configuration. For more information, see Applying overcloud configuration with director Operator.

Procedure

1. Create a directory for your custom templates:

   $ mkdir custom_templates

2. Create a custom template file named multiple_nics_vlans_dvr.j2 in your custom_templates directory.
3. Add configuration for the NICs of your bare-metal Compute nodes to your multiple_nics_vlans_dvr.j2 file. For an example NIC configuration file, see Custom NIC heat template for Compute nodes.

4. Create a directory for your custom environment files:

   $ mkdir custom_environment_files

5. Map the NIC template for your overcloud role in the network-environment.yaml environment file in your custom_environment_files directory:

   parameter_defaults:
     ComputeNetworkConfigTemplate: 'multiple_nics_vlans_dvr.j2'

Procedure

1. Navigate to the location of your custom templates:

   $ cd ~/custom_templates

2. Archive the templates into a gzipped tarball:

   $ tar -cvzf custom-config.tar.gz *.yaml

3. Create the tripleo-tarball-config ConfigMap CR and use the tarball as data:

   $ oc create configmap tripleo-tarball-config --from-file=custom-config.tar.gz -n openstack

4. Verify that the ConfigMap CR is created:

   $ oc get configmap/tripleo-tarball-config -n openstack

Procedure

1. Create the heat-env-config ConfigMap object:

   $ oc create configmap -n openstack heat-env-config \
    --from-file=~/<dir_custom_environment_files>/ \
    --dry-run=client -o yaml | oc apply -f -

   • Replace <dir_custom_environment_files> with the directory that contains the environment files you want to use in your overcloud deployment. The ConfigMap object stores these as individual data entries.

2. Verify that the heat-env-config ConfigMap object contains all the required environment files:

   $ oc get configmap/heat-env-config -n openstack

Procedure

1. Create your Compute nodes by using the OpenStackBaremetalSet CRD. For more information, see Creating Compute nodes with the OpenStackBaremetalSet CRD.
2. Create the Ansible playbooks for overcloud configuration with the OpenStackConfigGenerator CRD. For more information, see Creating Ansible playbooks for overcloud configuration with the OpenStackConfigGenerator CRD.
3. Register the operating system of your overcloud. For more information, see Registering the operating system of your overcloud.
4. Apply the overcloud configuration. For more information, see Applying overcloud configuration with director Operator.

Procedure

1. Access the remote shell for openstackclient:

   $ oc rsh -n openstack openstackclient

2. Change to the cloud-admin home directory:

   $ cd /home/cloud-admin

3. Run your openstack commands. For example, you can create a default network with the following command:

   $ openstack network create default

Procedure

1. Optional: To login as the admin user, obtain the admin password from the AdminPassword parameter in the tripleo-passwords secret:

   $ oc get secret tripleo-passwords -o jsonpath='{.data.tripleo-overcloud-passwords\.yaml}' | base64 -d

2. Retrieve the IP address reserved for the control plane from your OpenStackNetConfig CR:

   spec:
     ...
     reservations:
       controlplane:
         ipReservations:
           ctlplane: 172.22.0.110
           external: 10.0.0.10
           internal_api: 172.17.0.10
           storage: 172.18.0.10
           storage_mgmt: 172.19.0.10

3. Open a web browser.
4. Enter the IP address for the control plane in the URL field.
5. Log in to the dashboard with your username and password.

Procedure

1. Check that you have enough hosts in a ready state in the openshift-machine-api namespace:

   $ oc get baremetalhosts -n openshift-machine-api

   For more information on managing your bare-metal hosts, see Managing bare metal hosts.

2. Increase the count parameter for the compute OpenStackBaremetalSet resource:

   $ oc patch openstackbaremetalset compute --type=merge --patch '{"spec":{"count":3}}' -n openstack

   The OpenStackBaremetalSet resource automatically provisions the new nodes with the Red Hat Enterprise Linux base operating system.

3. Wait until the provisioning process completes. Check the nodes periodically to determine the readiness of the nodes:

   $ oc get baremetalhosts -n openshift-machine-api
   $ oc get openstackbaremetalset

4. Optional: Reserve static IP addresses for networks on the new Compute nodes. For more information, see Reserving static IP addresses for added Compute nodes with the OpenStackNetConfig CRD.
5. Generate the Ansible playbooks by using OpenStackConfigGenerator. For more information, see Creating Ansible playbooks for overcloud configuration with the OpenStackConfigGenerator CRD.
6. Apply the overcloud configuration. For more information, see Applying overcloud configuration with director Operator.

Procedure

1. Open the openstacknetconfig.yaml file for the overcloud on your workstation.

2. Add the following configuration to openstacknetconfig.yaml to create the OpenStackNetConfig custom resource (CR):

   apiVersion: osp-director.openstack.org/v1beta1
   kind: OpenStackNetConfig
   metadata:
     name: openstacknetconfig

3. Reserve static IP addresses for networks on specific nodes:

   spec:
     ...
     reservations:
       controller-0:
         ipReservations:
           ctlplane: 172.22.0.120
       compute-0:
         ipReservations:
           ctlplane: 172.22.0.140
           internal_api: 172.17.0.40
           storage: 172.18.0.40
           tenant: 172.20.0.40
       ...
       //The key for the ctlplane VIPs
       controlplane:
         ipReservations:
           ctlplane: 172.22.0.110
           external: 10.0.0.10
           internal_api: 172.17.0.10
           storage: 172.18.0.10
           storage_mgmt: 172.19.0.10
         macReservations: {}

   Note

   Reservations have precedence over any autogenerated IP addresses.

4. Save the openstacknetconfig.yaml definition file.

5. Create the overcloud network configuration:

   $ oc create -f openstacknetconfig.yaml -n openstack

Verification

1. To verify that the overcloud network configuration is created, view the resources for the overcloud network configuration:

   $ oc get openstacknetconfig/openstacknetconfig

2. View the OpenStackNetConfig API and child resources:

   $ oc get openstacknetconfig/openstacknetconfig -n openstack
   $ oc get openstacknetattachment -n openstack
   $ oc get openstacknet -n openstack

   If you see errors, check the underlying network-attach-definition and node network configuration policies:

   $ oc get network-attachment-definitions -n openstack
   $ oc get nncp

Procedure

1. Access the remote shell for openstackclient:

   $ oc rsh -n openstack openstackclient

2. Identify the Compute node that you want to remove:

   $ openstack compute service list

3. Disable the Compute service on the node to prevent the node from scheduling new instances:

   $ openstack compute service set <hostname> nova-compute --disable

4. Annotate the bare-metal node to prevent Metal³ from starting the node:

   $ oc annotate baremetalhost <node> baremetalhost.metal3.io/detached=true
   $ oc logs --since=1h <metal3-pod> metal3-baremetal-operator | grep -i detach
   $ oc get baremetalhost <node> -o json | jq .status.operationalStatus
   "detached"

   • Replace <node> with the name of the BareMetalHost resource.
   • Replace <metal3-pod> with the name of your metal3 pod.

5. Log in to the Compute node as the root user and shut down the bare-metal node:

   [root@compute-0 ~]# shutdown -h now

   If the Compute node is not accessible, complete the following steps:

   1. Log in to a Controller node as the root user.

   2. If Instance HA is enabled, disable the STONITH device for the Compute node:

      [root@controller-0 ~]# pcs stonith disable <stonith_resource_name>

      • Replace <stonith_resource_name> with the name of the STONITH resource that corresponds to the node. The resource name uses the format <resource_agent>-<host_mac>. You can find the resource agent and the host MAC address in the FencingConfig section of the fencing.yaml file.
   3. Use IPMI to power off the bare-metal node. For more information, see your hardware vendor documentation.

6. Retrieve the BareMetalHost resource that corresponds to the node that you want to remove:

   $ oc get openstackbaremetalset compute -o json | jq '.status.baremetalHosts | to_entries[] | "\(.key) => \(.value | .hostRef)"'
   "compute-0, openshift-worker-3"
   "compute-1, openshift-worker-4"

7. To change the status of the annotatedForDeletion parameter to true in the OpenStackBaremetalSet resource, annotate the BareMetalHost resource with osp-director.openstack.org/delete-host=true:

   $ oc annotate -n openshift-machine-api bmh/openshift-worker-3 osp-director.openstack.org/delete-host=true --overwrite

8. Optional: Confirm that the annotatedForDeletion status has changed to true in the OpenStackBaremetalSet resource:

   $ oc get openstackbaremetalset compute -o json -n openstack | jq .status
   {
     "baremetalHosts": {
       "compute-0": {
         "annotatedForDeletion": true,
         "ctlplaneIP": "192.168.25.105/24",
         "hostRef": "openshift-worker-3",
         "hostname": "compute-0",
         "networkDataSecretName": "compute-cloudinit-networkdata-openshift-worker-3",
         "provisioningState": "provisioned",
         "userDataSecretName": "compute-cloudinit-userdata-openshift-worker-3"
       },
       "compute-1": {
         "annotatedForDeletion": false,
         "ctlplaneIP": "192.168.25.106/24",
         "hostRef": "openshift-worker-4",
         "hostname": "compute-1",
         "networkDataSecretName": "compute-cloudinit-networkdata-openshift-worker-4",
         "provisioningState": "provisioned",
         "userDataSecretName": "compute-cloudinit-userdata-openshift-worker-4"
       }
     },
     "provisioningStatus": {
       "readyCount": 2,
       "reason": "All requested BaremetalHosts have been provisioned",
       "state": "provisioned"
     }
   }

9. Decrease the count parameter for the compute OpenStackBaremetalSet resource:

   $ oc patch openstackbaremetalset compute --type=merge --patch '{"spec":{"count":1}}' -n openstack

   When you reduce the resource count of the OpenStackBaremetalSet resource, you trigger the corresponding controller to handle the resource deletion, which causes the following actions:

   • Director Operator deletes the corresponding IP reservations from OpenStackIPSet and OpenStackNetConfig for the deleted node.

   • Director Operator flags the IP reservation entry in the OpenStackNet resource as deleted.

     $ oc get osnet ctlplane -o json -n openstack | jq .reservations
     {
       "compute-0": {
         "deleted": true,
         "ip": "172.22.0.140"
       },
       "compute-1": {
         "deleted": false,
         "ip": "172.22.0.100"
       },
       "controller-0": {
         "deleted": false,
         "ip": "172.22.0.120"
       },
       "controlplane": {
         "deleted": false,
         "ip": "172.22.0.110"
       },
       "openstackclient-0": {
         "deleted": false,
         "ip": "172.22.0.251"
       }

10. Optional: To make the IP reservations of the deleted OpenStackBaremetalSet resource available for other roles to use, set the value of the spec.preserveReservations parameter to false in the OpenStackNetConfig object.

11. Access the remote shell for openstackclient:

    $ oc rsh openstackclient -n openstack

12. Remove the Compute service entries from the overcloud:

    $ openstack compute service list
    $ openstack compute service delete <service-id>

13. Check the Compute network agents entries in the overcloud and remove them if they exist:

    $ openstack network agent list
    $ for AGENT in $(openstack network agent list --host <scaled-down-node> -c ID -f value) ; do openstack network agent delete $AGENT ; done

14. Exit from openstackclient:

    $ exit

1. Lock the RHOSP environment to a Red Hat Enterprise Linux (RHEL) release.
2. Update RHOSP repositories.
3. Update the container image preparation file.
4. Disable fencing in the overcloud.

Procedure

1. Create an OpenStackConfigGenerator resource called osconfiggenerator-update-prepare.yaml:

   $ cat <<EOF > osconfiggenerator-update-prepare.yaml
   apiVersion: osp-director.openstack.org/v1beta1
   kind: OpenStackConfigGenerator
   metadata:
     name: "update"
     namespace: openstack
   spec:
     gitSecret: git-secret
     enableFencing: false
     heatEnvs:
       - lifecycle/update-prepare.yaml
     heatEnvConfigMap: heat-env-config-update
     tarballConfigMap: tripleo-tarball-config-update
   EOF

2. Apply the configuration:

   $ oc apply -f osconfiggenerator-update-prepare.yaml

3. Wait until the update preparation process completes.

Procedure

1. Create an OpenStackDeploy custom resource (CR) named osdeploy-ovn-update.yaml:

   apiVersion: osp-director.openstack.org/v1beta1
   kind: OpenStackDeploy
   metadata:
     name: ovn-update
   spec:
     configVersion: <config_version>
     configGenerator: update
     mode: externalUpdate
     advancedSettings:
       tags:
         - ovn

2. Apply the updated configuration:

   $ oc apply -f osdeploy-ovn-update.yaml

3. Wait until the ovn-controller container update completes.

Procedure

1. Create an OpenStackDeploy custom resource (CR) named osdeploy-controller-update.yaml:

   apiVersion: osp-director.openstack.org/v1beta1
   kind: OpenStackDeploy
   metadata:
     name: controller-update
   spec:
     configVersion: <config_version>
     configGenerator: update
     mode: update
     advancedSettings:
       limit: Controller

2. Apply the updated configuration:

   $ oc apply -f osdeploy-controller-update.yaml

3. Wait until the Controller node update completes.

Procedure

1. Create an OpenStackDeploy CR named osdeploy-compute-update.yaml:

   apiVersion: osp-director.openstack.org/v1beta1
   kind: OpenStackDeploy
   metadata:
     name: compute-update
   spec:
     configVersion: <config_version>
     configGenerator: update
     mode: update
     advancedSettings:
       limit: Compute

2. Apply the updated configuration:

   $ oc apply -f osdeploy-compute-update.yaml

3. Wait until the Compute node update completes.

Procedure

1. Create an OpenStackDeploy CR named osdeploy-computehci-update.yaml:

   apiVersion: osp-director.openstack.org/v1beta1
   kind: OpenStackDeploy
   metadata:
     name: computehci-update
   spec:
     configVersion: <config_version>
     configGenerator: update
     mode: update
     advancedSettings:
       limit: ComputeHCI

2. Apply the updated configuration:

   $ oc apply -f osdeploy-computehci-update.yaml

3. Wait until the ComputeHCI node update completes.

4. Create an OpenStackDeploy CR named osdeploy-ceph-update.yaml:

   apiVersion: osp-director.openstack.org/v1beta1
   kind: OpenStackDeploy
   metadata:
     name: ceph-update
   spec:
     configVersion: <config_version>
     configGenerator: update
     mode: external-update
     advancedSettings:
       tags:
         - ceph

5. Apply the updated configuration:

   $ oc apply -f osdeploy-ceph-update.yaml

6. Wait until the Red Hat Ceph Storage node update completes.

Procedure

1. Create an OpenStackDeploy custom resource (CR) named osdeploy-cephstorage-update.yaml:

   apiVersion: osp-director.openstack.org/v1beta1
   kind: OpenStackDeploy
   metadata:
     name: cephstorage-update
   spec:
     configVersion: <config_version>
     configGenerator: update
     mode: externalUpdate
     advancedSettings:
       limit: CephStorage

2. Apply the updated configuration:

   $ oc apply -f osdeploy-cephstorage-update.yaml

3. Wait until the Red Hat Ceph Storage node update completes.

4. Create an OpenStackDeploy CR named osdeploy-ceph-update.yaml:

   apiVersion: osp-director.openstack.org/v1beta1
   kind: OpenStackDeploy
   metadata:
     name: ceph-update
   spec:
     configVersion: <config_version>
     configGenerator: update
     mode: externalUpdate
     advancedSettings:
       tags:
         - ceph

5. Apply the updated configuration:

   $ oc apply -f osdeploy-ceph-update.yaml

6. Wait until the Red Hat Ceph Storage node update completes.

Procedure

1. Access the remote shell for the openstackclient pod:

   $ oc rsh openstackclient

2. Log in to a Controller node:

   $ ssh <controller-0.ctlplane>

   • Replace <controller-0.ctlplane> with the name of your Controller node.

3. Log into the cephadm shell:

   [cloud-admin@controller-0 ~]$ sudo cephadm shell

4. Upgrade your Red Hat Ceph Storage cluster by using cephadm. For more information, see Upgrade a Red Hat Ceph Storage cluster using cephadm in the Red Hat Ceph Storage 6 Upgrade Guide.

5. Exit the remote shell for the openstackclient pod:

   $ exit

Procedure

1. Create an OpenStackDeploy custom resource (CR) named osdeploy-online-migration.yaml:

   apiVersion: osp-director.openstack.org/v1beta1
   kind: OpenStackDeploy
   metadata:
     name: online-migration
   spec:
     configVersion: <config_version>
     configGenerator: update
     mode: external-update
     advancedSettings:
       tags:
         - online_upgrade

2. Apply the updated configuration:

   $ oc apply -f osdeploy-online-migration.yaml

Procedure

1. Access the remote shell for the openstackclient pod:

   $ oc rsh openstackclient

2. Log in to a Controller node and run the Pacemaker command to enable fencing:

   $ ssh <controller-0.ctlplane> "sudo pcs property set stonith-enabled=true"

   • Replace <controller-0.ctlplane> with the name of your Controller node.

3. Exit the remote shell for the openstackclient pod:

   $ exit

Procedure

1. Create a ConfigMap resource to store the CA certificates:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cacerts
     namespace: openstack
   data:
    local_CA: |
       -----BEGIN CERTIFICATE-----
       …
      -----END CERTIFICATE-----
     another_CA: |
       -----BEGIN CERTIFICATE-----
       …
      -----END CERTIFICATE-----

2. Create the OpenStackControlPlane resource and reference the ConfigMap resource:

   apiVersion: osp-director.openstack.org/v1beta2
   kind: OpenStackControlPlane
   metadata:
     name: <overcloud>
     namespace: openstack
   spec:
     caConfigMap: cacerts

   • Replace <overcloud> with the name of your overcloud control plane.
3. Create a file in the ~/custom_environment_files directory named tls-certs.yaml, that specifies the generated certificates for the deployment by using the SSLCertificate, SSLIntermediateCertificate, SSLKey, and CAMap parameters.

4. Update the heatEnvConfigMap to add the tls-certs.yaml file:

   $ oc create configmap -n openstack heat-env-config --from-file=~/custom_environment_files/ --dry-run=client -o yaml | oc apply -f -

5. Create an OpenStackConfigGenerator resource and add the required heatEnvs configuration files to configure TLS for public endpoint IPs:

   apiVersion: osp-director.openstack.org/v1beta1
   kind: OpenStackConfigGenerator
   …
   spec:
     …
     heatEnvs:
       - ssl/tls-endpoints-public-ip.yaml
       - ssl/enable-tls.yaml
     …
     heatEnvConfigMap: heat-env-config
     tarballConfigMap: tripleo-tarball-config

6. Generate the Ansible playbooks by using OpenStackConfigGenerator and apply the overcloud configuration. For more information, see Configuring and deploying the overcloud with director Operator.

Procedure

1. Create a ConfigMap resource to store the CA certificates:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cacerts
     namespace: openstack
   data:
    local_CA: |
       -----BEGIN CERTIFICATE-----
       …
      -----END CERTIFICATE-----
     another_CA: |
       -----BEGIN CERTIFICATE-----
       …
      -----END CERTIFICATE-----

2. Create the OpenStackControlPlane resource and reference the ConfigMap resource:

   apiVersion: osp-director.openstack.org/v1beta2
   kind: OpenStackControlPlane
   metadata:
     name: <overcloud>
     namespace: openstack
   spec:
     caConfigMap: cacerts

   • Replace <overcloud> with the name of your overcloud control plane.
3. Create a file in the ~/custom_environment_files directory named tls-certs.yaml, that specifies the generated certificates for the deployment by using the SSLCertificate, SSLIntermediateCertificate, SSLKey, and CAMap parameters.

4. Update the heatEnvConfigMap to add the tls-certs.yaml file:

   $ oc create configmap -n openstack heat-env-config --from-file=~/custom_environment_files/ --dry-run=client -o yaml | oc apply -f -

5. Create an OpenStackConfigGenerator resource and add the required heatEnvs configuration files to configure TLS for public endpoint DNS names:

   apiVersion: osp-director.openstack.org/v1beta1
   kind: OpenStackConfigGenerator
   …
   spec:
     …
     heatEnvs:
       - ssl/tls-endpoints-public-dns.yaml
       - ssl/enable-tls.yaml
     …
     heatEnvConfigMap: heat-env-config
     tarballConfigMap: tripleo-tarball-config

6. Generate the Ansible playbooks by using OpenStackConfigGenerator and apply the overcloud configuration. For more information, see Configuring and deploying the overcloud with director Operator.

Procedure

1. Create a backup of the current tripleo-passwords secret:

   $ oc get secret tripleo-passwords -n openstack -o yaml > tripleo-passwords_backup.yaml

2. Create a plain text file named tripleo-overcloud-passwords_preserve_list to specify that the passwords for the following services should not be rotated:

   parameter_defaults
   BarbicanSimpleCryptoKek
   KeystoneCredential0
   KeystoneCredential1
   KeystoneFernetKey0
   KeystoneFernetKey1
   KeystoneFernetKeys
   CephClientKey
   CephClusterFSID
   CephManilaClientKey
   CephRgwKey
   HeatAuthEncryptionKey
   MysqlClustercheckPassword
   MysqlMariabackupPassword
   PacemakerRemoteAuthkey
   PcsdPassword

   You can add additional services to this list if there are other services for which you want to preserve the password.

3. Create a password parameter file, tripleo-overcloud-passwords.yaml, that lists the passwords that should not be modified:

   $ oc get secret tripleo-passwords -n openstack \
   -o jsonpath='{.data.tripleo-overcloud-passwords\.yaml}' \
   | base64 -d | grep -f ./tripleo-overcloud-passwords_preserve_list > tripleo-overcloud-passwords.yaml

4. Validate that the tripleo-overcloud-passwords.yaml file contains the passwords that you do not want to rotate.

5. Update the tripleo-password secret:

   $ oc create secret generic tripleo-passwords -n openstack \
   --from-file=./tripleo-overcloud-passwords.yaml \
   --dry-run=client -o yaml | oc apply -f -

6. Create Ansible playbooks to configure the overcloud with the OpenStackConfigGenerator CRD. For more information, see Creating Ansible playbooks for overcloud configuration with the OpenStackConfigGenerator CRD.
7. Apply the updated configuration. For more information, see Applying overcloud configuration with director Operator.

1. Get the password from the updated secret:

   $ oc get secret tripleo-passwords -n openstack -o jsonpath='{.data.tripleo-overcloud-passwords\.yaml}' | base64 -d | grep NovaPassword

   Example output:

   NovaPassword: hp4xpt7t2p79ktqjjnxpqwbp6

2. Retrieve the password for the Compute service (nova) running on the Controller nodes:

   1. Access the openstackclient remote shell:

      $ oc rsh openstackclient -n openstack

   2. Ensure that you are in the home directory:

      $ cd

   3. Retrieve the Compute service password:

      $ ansible -i /home/cloud-admin/ctlplane-ansible-inventory Controller -b -a "grep ^connection /var/lib/config-data/puppet-generated/nova/etc/nova/nova.conf"

      Example output:

      172.22.0.120 | CHANGED | rc=0 >> connection=mysql+pymysql://nova_api:hp4xpt7t2p79ktqjjnxpqwbp6@172.17.0.10/nova_api?read_default_file=/etc/my.cnf.d/tripleo.cnf&read_default_group=tripleo connection=mysql+pymysql://nova:hp4xpt7t2p79ktqjjnxpqwbp6@172.17.0.10/nova?read_default_file=/etc/my.cnf.d/tripleo.cnf&read_default_group=tripleo

Procedure

1. Create a configuration file named openstacknetconfig.yaml:

   apiVersion: osp-director.openstack.org/v1beta1
   kind: OpenStackNetConfig
   metadata:
     name: openstacknetconfig
   spec:
     attachConfigurations:
       br-osp:
         nodeNetworkConfigurationPolicy:
           nodeSelector:
             node-role.kubernetes.io/worker: ""
           desiredState:
             interfaces:
             - bridge:
                 options:
                   stp:
                     enabled: false
                 port:
                 - name: enp7s0
               description: Linux bridge with enp7s0 as a port
               name: br-osp
               state: up
               type: linux-bridge
               mtu: 1500
       br-ex:
         nodeNetworkConfigurationPolicy:
           nodeSelector:
             node-role.kubernetes.io/worker: ""
           desiredState:
             interfaces:
             - bridge:
                 options:
                   stp:
                     enabled: false
                 port:
                 - name: enp6s0
               description: Linux bridge with enp6s0 as a port
               name: br-ex
               state: up
               type: linux-bridge
               mtu: 1500
     # optional DnsServers list
     dnsServers:
     - 192.168.25.1
     # optional DnsSearchDomains list
     dnsSearchDomains:
     - osptest.test.metalkube.org
     - some.other.domain
     # DomainName of the OSP environment
     domainName: osptest.test.metalkube.org
     networks:
     - name: Control
       nameLower: ctlplane
       subnets:
       - name: ctlplane
         ipv4:
           allocationEnd: 192.168.25.250
           allocationStart: 192.168.25.100
           cidr: 192.168.25.0/24
           gateway: 192.168.25.1
         attachConfiguration: br-osp
     - name: InternalApi
       nameLower: internal_api
       mtu: 1350
       subnets:
       - name: internal_api
         ipv4:
           allocationEnd: 172.17.0.250
           allocationStart: 172.17.0.10
           cidr: 172.17.0.0/24
           routes:
           - destination: 172.17.1.0/24
             nexthop: 172.17.0.1
           - destination: 172.17.2.0/24
             nexthop: 172.17.0.1
         vlan: 20
         attachConfiguration: br-osp
       - name: internal_api_leaf1
         ipv4:
           allocationEnd: 172.17.1.250
           allocationStart: 172.17.1.10
           cidr: 172.17.1.0/24
           routes:
           - destination: 172.17.0.0/24
             nexthop: 172.17.1.1
           - destination: 172.17.2.0/24
             nexthop: 172.17.1.1
         vlan: 21
         attachConfiguration: br-osp
       - name: internal_api_leaf2
         ipv4:
           allocationEnd: 172.17.2.250
           allocationStart: 172.17.2.10
           cidr: 172.17.2.0/24
           routes:
           - destination: 172.17.1.0/24
             nexthop: 172.17.2.1
           - destination: 172.17.0.0/24
             nexthop: 172.17.2.1
         vlan: 22
         attachConfiguration: br-osp
     - name: External
       nameLower: external
       subnets:
       - name: external
         ipv4:
           allocationEnd: 10.0.0.250
           allocationStart: 10.0.0.10
           cidr: 10.0.0.0/24
           gateway: 10.0.0.1
         attachConfiguration: br-ex
     - name: Storage
       nameLower: storage
       mtu: 1350
       subnets:
       - name: storage
         ipv4:
           allocationEnd: 172.18.0.250
           allocationStart: 172.18.0.10
           cidr: 172.18.0.0/24
           routes:
           - destination: 172.18.1.0/24
             nexthop: 172.18.0.1
           - destination: 172.18.2.0/24
             nexthop: 172.18.0.1
         vlan: 30
         attachConfiguration: br-osp
       - name: storage_leaf1
         ipv4:
           allocationEnd: 172.18.1.250
           allocationStart: 172.18.1.10
           cidr: 172.18.1.0/24
           routes:
           - destination: 172.18.0.0/24
             nexthop: 172.18.1.1
           - destination: 172.18.2.0/24
             nexthop: 172.18.1.1
         vlan: 31
         attachConfiguration: br-osp
       - name: storage_leaf2
         ipv4:
           allocationEnd: 172.18.2.250
           allocationStart: 172.18.2.10
           cidr: 172.18.2.0/24
           routes:
           - destination: 172.18.0.0/24
             nexthop: 172.18.2.1
           - destination: 172.18.1.0/24
             nexthop: 172.18.2.1
         vlan: 32
         attachConfiguration: br-osp
     - name: StorageMgmt
       nameLower: storage_mgmt
       mtu: 1350
       subnets:
       - name: storage_mgmt
         ipv4:
           allocationEnd: 172.19.0.250
           allocationStart: 172.19.0.10
           cidr: 172.19.0.0/24
           routes:
           - destination: 172.19.1.0/24
             nexthop: 172.19.0.1
           - destination: 172.19.2.0/24
             nexthop: 172.19.0.1
         vlan: 40
         attachConfiguration: br-osp
       - name: storage_mgmt_leaf1
         ipv4:
           allocationEnd: 172.19.1.250
           allocationStart: 172.19.1.10
           cidr: 172.19.1.0/24
           routes:
           - destination: 172.19.0.0/24
             nexthop: 172.19.1.1
           - destination: 172.19.2.0/24
             nexthop: 172.19.1.1
         vlan: 41
         attachConfiguration: br-osp
       - name: storage_mgmt_leaf2
         ipv4:
           allocationEnd: 172.19.2.250
           allocationStart: 172.19.2.10
           cidr: 172.19.2.0/24
           routes:
           - destination: 172.19.0.0/24
             nexthop: 172.19.2.1
           - destination: 172.19.1.0/24
             nexthop: 172.19.2.1
         vlan: 42
         attachConfiguration: br-osp
     - name: Tenant
       nameLower: tenant
       vip: False
       mtu: 1350
       subnets:
       - name: tenant
         ipv4:
           allocationEnd: 172.20.0.250
           allocationStart: 172.20.0.10
           cidr: 172.20.0.0/24
           routes:
           - destination: 172.20.1.0/24
             nexthop: 172.20.0.1
           - destination: 172.20.2.0/24
             nexthop: 172.20.0.1
         vlan: 50
         attachConfiguration: br-osp
       - name: tenant_leaf1
         ipv4:
           allocationEnd: 172.20.1.250
           allocationStart: 172.20.1.10
           cidr: 172.20.1.0/24
           routes:
           - destination: 172.20.0.0/24
             nexthop: 172.20.1.1
           - destination: 172.20.2.0/24
             nexthop: 172.20.1.1
         vlan: 51
         attachConfiguration: br-osp
       - name: tenant_leaf2
         ipv4:
           allocationEnd: 172.20.2.250
           allocationStart: 172.20.2.10
           cidr: 172.20.2.0/24
           routes:
           - destination: 172.20.0.0/24
             nexthop: 172.20.2.1
           - destination: 172.20.1.0/24
             nexthop: 172.20.2.1
         vlan: 52
         attachConfiguration: br-osp

2. Create the internal API network:

   $ oc create -f openstacknetconfig.yaml -n openstack

3. Verify that the resources and child resources for the OpenStackNetConfig resource are created:

   $ oc get openstacknetconfig/openstacknetconfig -n openstack
   $ oc get openstacknetattachment -n openstack
   $ oc get openstacknet -n openstack

Procedure

1. Update the roles_data.yaml file:

   ...
   ###############################################################################
   # Role: ComputeLeaf1                                                          #
   ###############################################################################
   - name: ComputeLeaf1
     description: |
       Basic ComputeLeaf1 Node role
     # Create external Neutron bridge (unset if using ML2/OVS without DVR)
     tags:
       - compute
       - external_bridge
     networks:
       InternalApi:
         subnet: internal_api_leaf1
       Tenant:
         subnet: tenant_leaf1
       Storage:
         subnet: storage_leaf1
     HostnameFormatDefault: '%stackname%-novacompute-leaf1-%index%'
   ...
   ###############################################################################
   # Role: ComputeLeaf2                                                          #
   ###############################################################################
   - name: ComputeLeaf2
     description: |
       Basic ComputeLeaf1 Node role
     # Create external Neutron bridge (unset if using ML2/OVS without DVR)
     tags:
       - compute
       - external_bridge
     networks:
       InternalApi:
         subnet: internal_api_leaf2
       Tenant:
         subnet: tenant_leaf2
       Storage:
         subnet: storage_leaf2
     HostnameFormatDefault: '%stackname%-novacompute-leaf2-%index%'
   ...

2. Create a NIC template for each Compute role. For example Ansible NIC templates, see https://github.com/openstack/tripleo-ansible/tree/stable/wallaby/tripleo_ansible/roles/tripleo_network_config/templates.

3. Add the NIC templates for the new nodes to an environment file:

   parameter_defaults:
     ComputeNetworkConfigTemplate: 'multiple_nics_vlans_dvr.j2'
     ComputeLeaf1NetworkConfigTemplate: 'multiple_nics_vlans_dvr.j2'
     ComputeLeaf2NetworkConfigTemplate: 'multiple_nics_compute_leaf_2_vlans_dvr.j2'

4. In the ~/custom_environment_files directory, archive the roles_data.yaml file, the environment file, and the NIC templates into a tarball:

   $ tar -cvzf custom-spine-leaf-config.tar.gz *.yaml

5. Create the tripleo-tarball-config ConfigMap resource:

   $ oc create configmap tripleo-tarball-config --from-file=custom-spine-leaf-config.tar.gz -n openstack

Procedure

1. Create a file named openstack-controller.yaml on your workstation. Include the resource specification for the Controller nodes. The following example shows a specification for a control plane that consists of three Controller nodes:

   apiVersion: osp-director.openstack.org/v1beta2
   kind: OpenStackControlPlane
   metadata:
     name: overcloud
     namespace: openstack
   spec:
     gitSecret: git-secret
     openStackClientImageURL: registry.redhat.io/rhosp-rhel9/openstack-tripleoclient:17.1
     openStackClientNetworks:
       - ctlplane
       - external
       - internal_api
       - internal_api_leaf1 # optionally the openstackclient can also be connected to subnets
     openStackClientStorageClass: host-nfs-storageclass
     passwordSecret: userpassword
     domainName: ostest.test.metalkube.org
     virtualMachineRoles:
       Controller:
         roleName: Controller
         roleCount: 1
         networks:
           - ctlplane
           - internal_api
           - external
           - tenant
           - storage
           - storage_mgmt
         cores: 6
         memory: 20
         rootDisk:
           diskSize: 500
           baseImageVolumeName: openstack-base-img
           storageClass: host-nfs-storageclass
           storageAccessMode: ReadWriteMany
           storageVolumeMode: Filesystem
     enableFencing: False

2. Create the control plane:

   $ oc create -f openstack-controller.yaml -n openstack

3. Wait until Red Hat OpenShift Container Platform (RHOCP) creates the resources related to the OpenStackControlPlane resource.

4. Create a file on your workstation for each Compute leaf, for example, openstack-computeleaf1.yaml. Include the resource specification for the Compute nodes for the leaf. The following example shows a specification for one Compute leaf that includes one Compute node:

   apiVersion: osp-director.openstack.org/v1beta1
   kind: OpenStackBaremetalSet
   metadata:
     name: computeleaf1
     namespace: openstack
   spec:
     # How many nodes to provision
     count: 1
     # The image to install on the provisioned nodes
     baseImageUrl: http://<source_host>/rhel-9.2-x86_64-kvm.qcow2
     # The secret containing the SSH pub key to place on the provisioned nodes
     deploymentSSHSecret: osp-controlplane-ssh-keys
     # The interface on the nodes that will be assigned an IP from the mgmtCidr
     ctlplaneInterface: enp7s0
     # Networks to associate with this host
     networks:
           - ctlplane
           - internal_api_leaf1
           - external
           - tenant_leaf1
           - storage_leaf1
     roleName: ComputeLeaf1
     passwordSecret: userpassword

5. Create the Compute nodes for each leaf:

   $ oc create -f openstack-computeleaf1.yaml -n openstack

6. Generate the Ansible playbooks by using OpenStackConfigGenerator and apply the overcloud configuration. For more information, see Configuring and deploying the overcloud with director Operator.

Verification

1. View the resource for the control plane:

   $ oc get openstackcontrolplane/overcloud -n openstack

2. View the OpenStackVMSet resources to verify the creation of the control plane virtual machine (VM) set:

   $ oc get openstackvmsets -n openstack

3. View the VM resources to verify the creation of the control plane VMs in OpenShift Virtualization:

   $ oc get virtualmachines -n openstack

4. Test access to the openstackclient pod remote shell:

   $ oc rsh -n openstack openstackclient

5. View the resource for each Compute leaf:

   $ oc get openstackbaremetalset/computeleaf1 -n openstack

6. View the bare-metal machines managed by RHOCP to verify the creation of the Compute nodes:

   $ oc get baremetalhosts -n openshift-machine-api

Procedure

1. Create a file named openstack_backup.yaml on your workstation.

2. Add the following configuration to your openstack_backup.yaml file to create the OpenStackBackupRequest custom resource (CR):

   apiVersion: osp-director.openstack.org/v1beta1
   kind: OpenStackBackupRequest
   metadata:
     name: openstackbackupsave
     namespace: openstack
   spec:
     mode: save 1
     additionalConfigMaps: [] 2
     additionalSecrets: [] 3

   1

   Set the mode to save to request creation of an OpenStackBackup CR.

   2

   Optional: Include any ConfigMap resources that you created manually.

   3

   Optional: Include any Secret resources that you created manually.

   Note

   OSPdO attempts to include all ConfigMap and Secret objects associated with the OSPdO CRs in the namespace, such as OpenStackControlPlane and OpenStackBaremetalSet. You do not need to include those in the additional lists.

3. Save the openstack_backup.yaml file.

4. Create the OpenStackBackupRequest CR:

   $ oc create -f openstack_backup.yaml -n openstack

5. Monitor the creation status of the OpenStackBackupRequest CR:

   $ oc get openstackbackuprequest openstackbackupsave -n openstack

   • The Quiescing state indicates that OSPdO is waiting for the CRs to reach their finished state. The number of CRs can affect how long it takes to finish creating the backup.

     NAME                     OPERATION   SOURCE   STATUS      COMPLETION TIMESTAMP
     openstackbackupsave      save                 Quiescing

     If the status remains in the Quiescing state for longer than expected, you can investigate the OSPdO logs to check progress:

     $ oc logs <operator_pod> -c manager -f
     2022-01-11T18:26:15.180Z        INFO    controllers.OpenStackBackupRequest      Quiesce for save for OpenStackBackupRequest openstackbackupsave is waiting for: [OpenStackBaremetalSet: compute, OpenStackControlPlane: overcloud, OpenStackVMSet: controller]

     • Replace <operator_pod> with the name of the Operator pod.

   • The Saved state indicates that the OpenStackBackup CR is created.

     NAME                     OPERATION   SOURCE   STATUS   COMPLETION TIMESTAMP
     openstackbackupsave      save                 Saved    2022-01-11T19:12:58Z

   • The Error state indicates the backup has failed to create. Review the request contents to find the error:

     $ oc get openstackbackuprequest openstackbackupsave -o yaml -n openstack

6. View the OpenStackBackup resource to confirm it exists:

   $ oc get openstackbackup -n openstack
   NAME                                AGE
   openstackbackupsave-1641928378      6m7s

Procedure

1. List the available backups:

   $ oc get osbackup

2. Inspect the details of a specific backup:

   $ oc get backup <name> -o yaml

   • Replace <name> with the name of the backup you want to inspect.
3. Create a file named openstack_restore.yaml on your workstation.

4. Add the following configuration to your openstack_restore.yaml file to create the OpenStackBackupRequest custom resource (CR):

   apiVersion: osp-director.openstack.org/v1beta1
   kind: OpenStackBackupRequest
   metadata:
     name: openstackbackuprestore
     namespace: openstack
   spec:
     mode: <mode>
     restoreSource: <restore_source>

   • Replace <mode> with one of the following options:

     • restore: Requests a restore from an existing OpenStackBackup.
     • cleanRestore: Completely wipes the existing OSPdO resources within the namespace before restoring and creating new resources from the existing OpenStackBackup.
   • Replace <restore_source> with the ID of the OpenStackBackup to restore, for example, openstackbackupsave-1641928378.
5. Save the openstack_restore.yaml file.

6. Create the OpenStackBackupRequest CR:

   $ oc create -f openstack_restore.yaml -n openstack

7. Monitor the creation status of the OpenStackBackupRequest CR:

   $ oc get openstackbackuprequest openstackbackuprestore -n openstack

   • The Loading state indicates that all resources from the OpenStackBackup are being applied against the cluster.

     NAME                     OPERATION  SOURCE                           STATUS     COMPLETION TIMESTAMP
     openstackbackuprestore   restore    openstackbackupsave-1641928378   Loading

   • The Reconciling state indicates that all resources are loaded and OSPdO has begun reconciling to attempt to provision all resources.

     NAME                     OPERATION  SOURCE                           STATUS       COMPLETION TIMESTAMP
     openstackbackuprestore   restore    openstackbackupsave-1641928378   Reconciling

   • The Restored state indicates that the OpenStackBackup CR has been restored.

     NAME                     OPERATION  SOURCE                           STATUS     COMPLETION TIMESTAMP
     openstackbackuprestore   restore    openstackbackupsave-1641928378   Restored   2022-01-12T13:48:57Z

   • The Error state indicates the restoration has failed. Review the request contents to find the error:

     $ oc get openstackbackuprequest openstackbackuprestore -o yaml -n openstack

Procedure

1. Change the number of Controller virtualMachineRole cores to 8:

   $ oc patch -n openstack osctlplane overcloud --type='json' -p='[{"op": "add", "path": "/spec/virtualMachineRoles/controller/cores", "value": 8 }]'

2. Change the Controller virtualMachineRole RAM size to 22GB:

   $ oc patch -n openstack osctlplane overcloud --type='json' -p='[{"op": "add", "path": "/spec/virtualMachineRoles/controller/memory", "value": 22 }]'

3. Validate the virtualMachineRole resource:

   $ oc get osvmset
   NAME         CORES   RAM   DESIRED   READY   STATUS        REASON
   controller   8       22    1         1       Provisioned   All requested VirtualMachines have been provisioned

4. From inside the virtual machine do a graceful shutdown. Shutdown each updated virtual machine one by one.

5. Power on the virtual machine:

   $ `virtctl start <VM>` to power on the virtual machine.

   • Replace <VM> with the name of your virtual machine.

Procedure

1. Add or update the additionalDisks parameter in the OpenStackControlPlane object:

   spec:
     ...
     virtualMachineRoles:
       Controller:
         ...
         additionalDisks:
         - baseImageVolumeName: openstack-base-img
           dedicatedIOThread: false
           diskSize: 10
           name: "data-disk1"
           storageAccessMode: ReadWriteMany
           storageClass: host-nfs-storageclass
           storageVolumeMode: Filesystem

2. Apply the patch:

   $ oc patch -n openstack osctlplane overcloud --patch-file controller_add_data_disk1.yaml

3. Validate the virtualMachineRole resource:

   $ oc get osvmset controller -o json | jq .spec.additionalDisks
   [
     {
       "baseImageVolumeName": "openstack-base-img",
       "dedicatedIOThread": false,
       "diskSize": 10,
       "name": "data-disk1",
       "storageAccessMode": "ReadWriteMany",
       "storageClass": "host-nfs-storageclass",
       "storageVolumeMode": "Filesystem"
     }
   ]

4. From inside the virtual machine do a graceful shutdown. Shutdown each updated virtual machine one by one.

5. Power on the virtual machine:

   $ `virtctl start <VM>` to power on the virtual machine.

   • Replace <VM> with the name of your virtual machine.

Procedure

1. Create the openstack namespace:

   $ oc new-project openstack

2. Create the index image and push it to your registry:

   $ podman login registry.redhat.io
   $ podman login your.registry.local
   $ BUNDLE_IMG="registry.redhat.io/rhosp-rhel9/osp-director-operator-bundle@sha256:<bundle digest>"
   $ INDEX_IMG="quay.io/<account>/osp-director-operator-index:x.y.z-a"
   $ opm index add --bundles ${BUNDLE_IMG} --tag ${INDEX_IMG} -u podman --pull-tool podman

   Note

   You can get the latest bundle image from: Certified container images. Search for osp-director-operator-bundle.

3. Retrieve the digest of the index image you created in the previous step:

   $ INDEX_DIGEST="$(skopeo inspect docker://quay.io/<account>/osp-director-operator-index:x.y.z-a | jq '.Digest' -r)"

4. Mirror the relevant images based on the operator index image:

   $ oc adm catalog mirror quay.io/<account>/osp-director-operator-index@${INDEX_DIGEST} your.registry.local --insecure --index-filter-by-os='Linux/x86_64'

5. After mirroring is complete, a manifests directory is generated in your current directory called manifests-osp-director-operator-index-<random_number>. Apply the created ImageContentSourcePolicy to your cluster:

   $ os apply -f manifests-osp-director-operator-index-<random_number>/imageContentSourcePolicy.yaml

   • Replace <random_number> with the randomly generated number.

6. Create a file named osp-director-operator.yaml and include the following YAML content to configure the three resources required to install director Operator:

   apiVersion: operators.coreos.com/v1alpha1
   kind: CatalogSource
   metadata:
     name: osp-director-operator-index
     namespace: openstack
   spec:
     sourceType: grpc
     image: your.registry.local/osp-director-operator-index:1.3.x-y
   ---
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: "osp-director-operator-group"
     namespace: openstack
   spec:
     targetNamespaces:
     - openstack
   ---
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: osp-director-operator-subscription
     namespace: openstack
   spec:
     config:
       env:
       - name: WATCH_NAMESPACE
         value: openstack,openshift-machine-api,openshift-sriov-network-operator
     source: osp-director-operator-index
     sourceNamespace: openstack
     name: osp-director-operator

7. Create the new resources in the openstack namespace:

   $ oc apply -f osp-director-operator.yaml

8. Copy the required overcloud images to the repository:

   $ for i in $(podman search --limit 1000 "registry.redhat.io/rhosp-rhel9/openstack" --format="{{ .Name }}" | awk '{print $1 ":" "17.1.0"}' | awk -F "/" '{print $2 "/" $3}'); do skopeo copy --all docker://registry.redhat.io/$i docker://your.registry.local/$i;done

   Note

   You can refer to Preparing a Satellite server for container images if Red Hat Satellite is used as the local registry.

9. You can now proceed with Installing and preparing director Operator.

Verification

1. Confirm that you have successfully installed director Operator:

   $ oc get operators
   NAME                                     AGE
   osp-director-operator.openstack          5m

Procedure

1. Check the current version of the director Operator in the currentCSV field:

   $ oc get subscription osp-director-operator.openstack -n openstack -o yaml | grep currentCSV

2. Delete the CSV for the director Operator in the target namespace:

   $ oc delete clusterserviceversion <current_CSV> -n openstack

   • Replace <current_CSV> with the currentCSV value from step 1.

3. Delete the subscription:

   $ oc delete subscription  osp-director-operator.openstack -n openstack

4. Install the latest 17.1 director Operator. For information, see Installing director Operator.

Procedure

1. Set openStackRelease to 17.1 on the openstackcontrolplane CR:

   $ oc patch openstackcontrolplane -n openstack overcloud --type=json -p="[{'op': 'replace', 'path': '/spec/openStackRelease', 'value': '17.1'}]"

2. Retrieve the OSPdO ClusterServiceVersion (csv) CR:

   $ oc get csv -n openstack

3. Delete all instances of the OpenStackConfigGenerator CR:

   $ oc delete -n openstack openstackconfiggenerator --all

4. If your deployment includes HCI, the adoption from ceph-ansible to cephadm must be performed using the RHOSP 17.1 on RHEL8 openstackclient image:

   $ oc patch openstackclient -n openstack openstackclient --type=json -p="[{'op': 'replace', 'path': '/spec/imageURL', 'value': 'registry.redhat.io/rhosp-rhel8/openstack-tripleoclient:17.1'}]"

   If your deployment does not include HCI, or the cephadm adoption has already been completed, then switch to the 17.1 OSPdO default openstackclient image by removing the current imageURL from the openstackclient CR:

   $ oc patch openstackclient -n openstack openstackclient --type=json -p="[{'op': 'remove', 'path': '/spec/imageURL'}]"

5. If you have enabled fencing in the overcloud, you must temporarily disable fencing on one of the Controller nodes for the duration of the upgrade:

   $ oc rsh -n openstack openstackclient
   $ ssh controller-0.ctlplane "sudo pcs property set stonith-enabled=false"

Procedure

1. Remove the following services from all roles:

   `OS::TripleO::Services::CinderBackendDellEMCXTREMIOISCSI`
   `OS::TripleO::Services::CinderBackendDellPs`
   `OS::TripleO::Services::CinderBackendVRTSHyperScale`
   `OS::TripleO::Services::Ec2Api`
   `OS::TripleO::Services::Fluentd`
   `OS::TripleO::Services::FluentdAlt`
   `OS::TripleO::Services::Keepalived`
   `OS::TripleO::Services::MistralApi`
   `OS::TripleO::Services::MistralEngine`
   `OS::TripleO::Services::MistralEventEngine`
   `OS::TripleO::Services::MistralExecutor`
   `OS::TripleO::Services::NeutronLbaasv2Agent`
   `OS::TripleO::Services::NeutronLbaasv2Api`
   `OS::TripleO::Services::NeutronML2FujitsuCfab`
   `OS::TripleO::Services::NeutronML2FujitsuFossw`
   `OS::TripleO::Services::NeutronSriovHostConfig`
   `OS::TripleO::Services::NovaConsoleauth`
   `OS::TripleO::Services::Ntp`
   `OS::TripleO::Services::OpenDaylightApi`
   `OS::TripleO::Services::OpenDaylightOvs`
   `OS::TripleO::Services::OpenShift::GlusterFS`
   `OS::TripleO::Services::OpenShift::Infra`
   `OS::TripleO::Services::OpenShift::Master`
   `OS::TripleO::Services::OpenShift::Worker`
   `OS::TripleO::Services::PankoApi`
   `OS::TripleO::Services::Rear`
   `OS::TripleO::Services::SaharaApi`
   `OS::TripleO::Services::SaharaEngine`
   `OS::TripleO::Services::SensuClient`
   `OS::TripleO::Services::SensuClientAlt`
   `OS::TripleO::Services::SkydiveAgent`
   `OS::TripleO::Services::SkydiveAnalyzer`
   `OS::TripleO::Services::Tacker`
   `OS::TripleO::Services::TripleoUI`
   `OS::TripleO::Services::UndercloudMinionMessaging`
   `OS::TripleO::Services::UndercloudUpgradeEphemeralHeat`
   `OS::TripleO::Services::Zaqar`

2. Add the OS::TripleO::Services::GlanceApiInternal service to your Controller role.
3. Update the OS::TripleO::Services::NovaLibvirt service on the Compute roles to OS::TripleO::Services::NovaLibvirtLegacy.
4. If your environment includes Red Hat Ceph Storage, set the DeployedCeph parameter to false to enable director-managed cephadm deployments.

5. If your network configuration templates include the following functions, you must manually convert your NIC templates to Jinja2 Ansible format before you upgrade the overcloud. The following functions are not supported with automatic conversion:

   'get_file'
   'get_resource'
   'digest'
   'repeat'
   'resource_facade'
   'str_replace'
   'str_replace_strict'
   'str_split'
   'map_merge'
   'map_replace'
   'yaql'
   'equals'
   'if'
   'not'
   'and'
   'or'
   'filter'
   'make_url'
   'contains'

   For more information about manually converting your NIC templates, see Manually converting NIC templates to Jinja2 Ansible format in Installing and managing Red Hat OpenStack Platform with director.

Procedure

1. Create an Ansible playbook file named ceph-admin-user-playbook.yaml to create a ceph-admin user on the overcloud nodes.

2. Add the following configuration to the ceph-admin-user-playbook.yaml file:

   - hosts: localhost
     gather_facts: false
     tasks:
       - name: set ssh key path facts
         set_fact:
           private_key: "{{ lookup('env', 'HOME') }}/.ssh/{{ tripleo_admin_user }}-id_rsa"
           public_key: "{{ lookup('env', 'HOME') }}/.ssh/{{ tripleo_admin_user }}-id_rsa.pub"
         run_once: true
       - name: stat private key
         stat:
           path: "{{ private_key }}"
         register: private_key_stat
       - name: create private key if it does not exist
         shell: "ssh-keygen -t rsa -q -N '' -f {{ private_key }}"
         no_log: true
         when:
           - not private_key_stat.stat.exists
       - name: stat public key
         stat:
           path: "{{ public_key }}"
         register: public_key_stat
       - name: create public key if it does not exist
         shell: "ssh-keygen -y -f {{ private_key }} > {{ public_key }}"
         when:
           - not public_key_stat.stat.exists
   
   - hosts: overcloud
     gather_facts: false
     become: true
     pre_tasks:
       - name: Get local private key
         slurp:
           src: "{{ hostvars['localhost']['private_key'] }}"
         register: private_key_get
         delegate_to: localhost
         no_log: true
       - name: Get local public key
         slurp:
           src: "{{ hostvars['localhost']['public_key'] }}"
         register: public_key_get
         delegate_to: localhost
     roles:
       - role: tripleo_create_admin
         tripleo_admin_user: "{{ tripleo_admin_user }}"
         tripleo_admin_pubkey: "{{ public_key_get['content'] | b64decode }}"
         tripleo_admin_prikey: "{{ private_key_get['content'] | b64decode }}"
         no_log: true

3. Copy the playbook to the openstackclient container:

   $ oc cp -n openstack ceph-admin-user-playbook.yml openstackclient:/home/cloud-admin/ceph-admin-user-playbook.yml

4. Run the playbook on the openstackclient container:

   $ oc rsh -n openstack openstackclient
   $ ansible-playbook -i /home/cloud-admin/ctlplane-ansible-inventory -e tripleo_admin_user=ceph-admin -e distribute_private_key=true /home/cloud-admin/ceph-admin-user-playbook.yml

5. Update the Red Hat Ceph Storage container image parameters in the containers-prepare-parameter.yaml file for the version of Red Hat Ceph Storage that your deployment uses:

   ceph_namespace: registry.redhat.io/rhceph
   ceph_image: <ceph_image_file>
   ceph_tag: latest
   ceph_grafana_image: <grafana_image_file>
   ceph_grafana_namespace: registry.redhat.io/rhceph
   ceph_grafana_tag: latest

   • Replace <ceph_image_file> with the name of the image file for the version of Red Hat Ceph Storage that your deployment uses:

     • Red Hat Ceph Storage 5: rhceph-5-rhel8

   • Replace <grafana_image_file> with the name of the image file for the version of Red Hat Ceph Storage that your deployment uses:

     • Red Hat Ceph Storage 5: rhceph-5-dashboard-rhel8
6. If your deployment includes HCI, update the CephAnsibleRepo parameter in compute-hci.yaml to "rhelosp-ceph-5-tools".

7. Create an environment file named upgrade.yaml and add the following configuration to it:

   parameter_defaults:
     UpgradeInitCommand: |
       sudo subscription-manager repos --disable *
       if $( grep -q 9.2 /etc/os-release )
       then
         sudo subscription-manager repos --enable=rhel-9-for-x86_64-baseos-eus-rpms --enable=rhel-9-for-x86_64-appstream-eus-rpms --enable=rhel-9-for-x86_64-highavailability-eus-rpms --enable=openstack-17.1-for-rhel-9-x86_64-rpms --enable=fast-datapath-for-rhel-9-x86_64-rpms
           sudo podman ps | grep -q ceph && subscription-manager repos --enable=rhceph-5-tools-for-rhel-9-x86_64-rpms
           sudo subscription-manager release --set=9.2
       else
         sudo subscription-manager repos --enable=rhel-8-for-x86_64-baseos-tus-rpms --enable=rhel-8-for-x86_64-appstream-tus-rpms --enable=rhel-8-for-x86_64-highavailability-tus-rpms --enable=openstack-17.1-for-rhel-8-x86_64-rpms --enable=fast-datapath-for-rhel-8-x86_64-rpms
           sudo podman ps | grep -q ceph && subscription-manager repos --enable=rhceph-5-tools-for-rhel-8-x86_64-rpms
           sudo subscription-manager release --set=8.4
       fi
       sudo dnf -y install cephadm

8. Create a new OpenStackConfigGenerator CR named ceph-upgrade that includes the updated environment file and tripleo-tarball ConfigMaps.

9. Create a file named openstack-ceph-upgrade.yaml on your workstation to define an OpenStackDeploy CR for the upgrade from Red Hat Ceph Storage 4 to 5:

   apiVersion: osp-director.openstack.org/v1beta1
   kind: OpenStackDeploy
   metadata:
     name: ceph-upgrade
   spec:
     configVersion: <config_version>
     configGenerator: ceph-upgrade
     mode: externalUpgrade
     advancedSettings:
       skipTags:
       - ceph_health
       - opendev-validation
       - ceph_ansible_remote_tmp
       tags:
       - ceph
       - facts

10. Save the openstack-ceph-upgrade.yaml file.

11. Create the OpenStackDeploy resource:

    $ oc create -f openstack-ceph-upgrade.yaml -n openstack

12. Wait for the deployment to finish.

13. Create a file named openstack-ceph-upgrade-packages.yaml on your workstation to define an OpenStackDeploy CR that upgrades the Red Hat Ceph Storage packages:

    apiVersion: osp-director.openstack.org/v1beta1
    kind: OpenStackDeploy
    metadata:
      name: ceph-upgrade-packages
    spec:
      configVersion: <config_version>
      configGenerator: ceph-upgrade
      mode: upgrade
      advancedSettings:
        limit: ceph_osd,ceph_mon,Undercloud
        playbook:
        - upgrade_steps_playbook.yaml
        skipTags:
        - ceph_health
        - opendev-validation
        - ceph_ansible_remote_tmp
        tags:
        - setup_packages

14. Save the openstack-ceph-upgrade-packages.yaml file.

15. Create the OpenStackDeploy resource:

    $ oc create -f openstack-ceph-upgrade-packages.yaml -n openstack

16. Wait for the deployment to finish.

17. Create a file named openstack-ceph-upgrade-to-cephadm.yaml on your workstation to define an OpenStackDeploy CR that runs the cephadm adoption:

    apiVersion: osp-director.openstack.org/v1beta1
    kind: OpenStackDeploy
    metadata:
      name: ceph-upgrade-to-cephadm
    spec:
      configVersion: <config_version>
      configGenerator: ceph-upgrade
      mode: externalUpgrade
      advancedSettings:
        skipTags:
        - ceph_health
        - opendev-validation
        - ceph_ansible_remote_tmp
        tags:
        - cephadm_adopt

18. Save the openstack-ceph-upgrade-to-cephadm.yaml file.

19. Create the OpenStackDeploy resource:

    $ oc create -f openstack-ceph-upgrade-to-cephadm.yaml -n openstack

20. Wait for the deployment to finish.

21. Update the openstackclient image to the RHEL9 container image by removing the current imageURL from the openstackclient CR:

    $ oc patch openstackclient -n openstack openstackclient --type=json -p="[{'op': 'remove', 'path': '/spec/imageURL'}]"

Procedure

1. Open the container preparation file, containers-prepare-parameter.yaml, and check that it obtains the correct image versions.

2. Add the ContainerImagePrepareRhel8 parameter to containers-prepare-parameter.yaml:

   parameter_defaults:
     #default container image configuration for RHEL 9 hosts
     ContainerImagePrepare:
     - push_destination: false
       set: &container_image_prepare_rhel9_contents
         tag: 17.1.2
         name_prefix: openstack-
         namespace: registry.redhat.io/rhosp-rhel9
         ceph_namespace: registry.redhat.io/rhceph
         ceph_image: rhceph-5-rhel8
         ceph_tag: latest
         ceph_alertmanager_image: ose-prometheus-alertmanager
         ceph_alertmanager_namespace: registry.redhat.io/openshift4
         ceph_alertmanager_tag: v4.10
         ceph_grafana_image: rhceph-5-dashboard-rhel8
         ceph_grafana_namespace: registry.redhat.io/rhceph
         ceph_grafana_tag: latest
         ceph_node_exporter_image: ose-prometheus-node-exporter
         ceph_node_exporter_namespace: registry.redhat.io/openshift4
         ceph_node_exporter_tag: v4.10
         ceph_prometheus_image: ose-prometheus
         ceph_prometheus_namespace: registry.redhat.io/openshift4
         ceph_prometheus_tag: v4.10
   
     # RHEL8 hosts pin the collectd and libvirt containers to rhosp-rhel8
     # To apply the following configuration, reference the followingparameter
     # in the role specific parameters below: <Role>ContainerImagePrepare
     ContainerImagePrepareRhel8: &container_image_prepare_rhel8
     - push_destination: false
       set: *container_image_prepare_rhel9_contents
       excludes:
       - collectd
       - nova-libvirt
     - push_destination: false
       set:
         tag: 17.1.2
         name_prefix: openstack-
         namespace: registry.redhat.io/rhosp-rhel8
         ceph_namespace: registry.redhat.io/rhceph
         ceph_image: rhceph-5-rhel8
         ceph_tag: latest
         ceph_alertmanager_image: ose-prometheus-alertmanager
         ceph_alertmanager_namespace: registry.redhat.io/openshift4
         ceph_alertmanager_tag: v4.10
         ceph_grafana_image: rhceph-5-dashboard-rhel8
         ceph_grafana_namespace: registry.redhat.io/rhceph
         ceph_grafana_tag: latest
         ceph_node_exporter_image: ose-prometheus-node-exporter
         ceph_node_exporter_namespace: registry.redhat.io/openshift4
         ceph_node_exporter_tag: v4.10
         ceph_prometheus_image: ose-prometheus
         ceph_prometheus_namespace: registry.redhat.io/openshift4
         ceph_prometheus_tag: v4.10
       includes:
       - collectd
       - nova-libvirt
   # Initially all hosts are RHEL 8 so set the role specific container
   # image prepare parameter to the RHEL 8 configuration
     ControllerContainerImagePrepare: *container_image_prepare_rhel8
     ComputeContainerImagePrepare: *container_image_prepare_rhel8
   ...

3. Create an environment file named upgrade.yaml.

4. Add the following configuration to the upgrade.yaml file:

   parameter_defaults:
     UpgradeInitCommand: |
       sudo subscription-manager repos --disable *
         if $( grep -q  9.2  /etc/os-release )
         then
           sudo subscription-manager repos --enable=rhel-9.2-for-x86_64-baseos-eus-rpms --enable=rhel-9.2-for-x86_64-appstream-eus-rpms --enable=rhel-9.2-for-x86_64-highavailability-eus-rpms --enable=openstack-17.1-for-rhel-9-x86_64-rpms --enable=fast-datapath-for-rhel-9-x86_64-rpms
         else
           sudo subscription-manager repos --enable=rhel-8-for-x86_64-baseos-eus-rpms --enable=rhel-8-for-x86_64-appstream-eus-rpms --enable=rhel-8-for-x86_64-highavailability-eus-rpms --enable=openstack-17.1-for-rhel-8-x86_64-rpms --enable=fast-datapath-for-rhel-8-x86_64-rpms
           fi

5. Create an environment file named disable_compute_service_check.yaml.

6. Add the following configuration to the disable_compute_service_check.yaml file:

   parameter_defaults:
     ExtraConfig:
       nova::workarounds::disable_compute_service_check_for_ffu: true
   
   parameter_merge_strategies:
     ExtraConfig: merge

7. If your deployment includes HCI, update the Red Hat Ceph Storage and HCI parameters from ceph-ansible values in RHOSP 16.2 to cephadm values in RHOSP 17.1. For more information, see Custom environment file for configuring Hyperconverged Infrastructure (HCI) storage in director Operator.

8. Create a file named openstack-configgen-upgrade.yaml on your workstation that defines a new OpenStackConfigGenerator CR named "upgrade":

   apiVersion: osp-director.openstack.org/v1beta1
   kind: OpenStackConfigGenerator
   metadata:
     name: "upgrade"
     namespace: openstack
   spec:
     enableFencing: False
     gitSecret: git-secret
     heatEnvs:
       - ssl/tls-endpoints-public-dns.yaml
       - ssl/enable-tls.yaml
       - nova-hw-machine-type-upgrade.yaml
       - lifecycle/upgrade-prepare.yaml
     heatEnvConfigMap: heat-env-config-upgrade
     tarballConfigMap: tripleo-tarball-config-upgrade

9. Create a file named openstack-upgrade.yaml on your workstation to create an OpenStackDeploy CR for the overcloud upgrade:

   apiVersion: osp-director.openstack.org/v1beta1
   kind: OpenStackDeploy
   metadata:
     name: upgrade
   spec:
     configVersion: <config_version>
     configGenerator: upgrade
     mode: upgrade

10. Save the openstack-upgrade.yaml file.

11. Create the OpenStackDeploy resource:

    $ oc create -f openstack-upgrade.yaml -n openstack

12. Wait for the deployment to finish. The overcloud nodes are now running 17.1 containers on RHEL8.

Procedure

1. Open the container preparation file, containers-prepare-parameter.yaml and check that it obtains the correct image versions.

2. Remove the following role specific overrides from the containers-prepare-paramater.yaml file:

     ControllerContainerImagePrepare: *container_image_prepare_rhel8
     ComputeContainerImagePrepare: *container_image_prepare_rhel8

3. Open the roles_data.yaml file and replace OS::TripleO::Services::NovaLibvirtLegacy with OS::TripleO::Services::NovaLibvirt.

4. Create an environment file named skip_rhel_release.yaml, and add the following configuration:

   parameter_defaults:
     SkipRhelEnforcement: false

5. Create an environment file named system_upgrade.yaml and add the following configuration:

   parameter_defaults:
     NICsPrefixesToUdev: ['en']
     UpgradeLeappDevelSkip: "LEAPP_UNSUPPORTED=1 LEAPP_DEVEL_SKIP_CHECK_OS_RELEASE=1 LEAPP_NO_NETWORK_RENAMING=1 LEAPP_DEVEL_TARGET_RELEASE=9.2"
     UpgradeLeappDebug: false
     UpgradeLeappEnabled: true
     LeappActorsToRemove: ['checkifcfg','persistentnetnamesdisable','checkinstalledkernels','biosdevname']
     LeappRepoInitCommand: |
       sudo subscription-manager repos --disable=*
       subscription-manager repos --enable rhel-8-for-x86_64-baseos-tus-rpms --enable rhel-8-for-x86_64-appstream-tus-rpms --enable openstack-17.1-for-rhel-8-x86_64-rpms
       subscription-manager release --set=8.4
     UpgradeLeappCommandOptions:
   "--enablerepo=rhel-9-for-x86_64-baseos-eus-rpms --enablerepo=rhel-9-for-x86_64-appstream-eus-rpms --enablerepo=rhel-9-for-x86_64-highavailability-eus-rpms --enablerepo=openstack-17.1-for-rhel-9-x86_64-rpms --enablerepo=fast-datapath-for-rhel-9-x86_64-rpms"
     LeappInitCommand: |
       sudo subscription-manager repos --disable=*
       sudo subscription-manager repos
     --enable=rhel-9-for-x86_64-baseos-eus-rpms --enable=rhel-9-for-x86_64-appstream-eus-rpms --enable=rhel-9-for-x86_64-highavailability-eus-rpms --enable=openstack-17.1-for-rhel-9-x86_64-rpms --enable=fast-datapath-for-rhel-9-x86_64-rpms
   
       leapp answer --add --section check_vdo.confirm=True
   
       dnf -y remove irb

   For more information on the recommended Leapp parameters, see Upgrade parameters in the Framework for upgrades (16.2 to 17.1) guide.

6. Create a new OpenStackConfigGenerator CR named system-upgrade that includes the updated heat environment and tripleo tarball ConfigMaps.

7. Create a file named openstack-controller0-upgrade.yaml on your workstation to define an OpenStackDeploy CR for the first controller node:

   apiVersion: osp-director.openstack.org/v1beta1
   kind: OpenStackDeploy
   metadata:
     name: system-upgrade-controller-0
   spec:
     configVersion: <config_version>
     configGenerator: system-upgrade
     mode: upgrade
     advancedSettings:
       limit: Controller[0]
       tags:
       - system_upgrade

8. Save the openstack-controller0-upgrade.yaml file.

9. Create the OpenStackDeploy resource to run the system upgrade on Controller 0:

   $ oc create -f openstack-controller0-upgrade.yaml -n openstack

10. Wait for the deployment to finish.

11. Create a file named openstack-controller1-upgrade.yaml on your workstation to define an OpenStackDeploy CR for the second controller node:

    apiVersion: osp-director.openstack.org/v1beta1
    kind: OpenStackDeploy
    metadata:
      name: system-upgrade-controller-1
    spec:
      configVersion: <config_version>
      configGenerator: system-upgrade
      mode: upgrade
      advancedSettings:
        limit: Controller[1]
        tags:
        - system_upgrade

12. Save the openstack-controller1-upgrade.yaml file.

13. Create the OpenStackDeploy resource to run the system upgrade on Controller 1:

    $ oc create -f openstack-controller1-upgrade.yaml -n openstack

14. Wait for the deployment to finish.

15. Create a file named openstack-controller2-upgrade.yaml on your workstation to define an OpenStackDeploy CR for the third controller node:

    apiVersion: osp-director.openstack.org/v1beta1
    kind: OpenStackDeploy
    metadata:
      name: system-upgrade-controller-2
    spec:
      configVersion: <config_version>
      configGenerator: system-upgrade
      mode: upgrade
      advancedSettings:
        limit: Controller[2]
        tags:
        - system_upgrade

16. Save the openstack-controller2-upgrade.yaml file.

17. Create the OpenStackDeploy resource to run the system upgrade on Controller 1:

    $ oc create -f openstack-controller2-upgrade.yaml -n openstack

18. Wait for the deployment to finish.

19. Create a file named openstack-computes-upgrade.yaml on your workstation to define an OpenStackDeploy CR that upgrades all Compute nodes:

    apiVersion: osp-director.openstack.org/v1beta1
    kind: OpenStackDeploy
    metadata:
      name: system-upgrade-computes
    spec:
      configVersion: <config_version>
      configGenerator: system-upgrade
      mode: upgrade
      advancedSettings:
        limit: Compute
        tags:
        - system_upgrade

20. Save the openstack-computes-upgrade.yaml file.

21. Create the OpenStackDeploy resource to run the system upgrade on the Compute nodes:

    $ oc create -f openstack-computes-upgrade.yaml -n openstack

22. Wait for the deployment to finish.

Procedure

1. Update the baseImageUrl parameter to a RHEL 9.2 guest image in your OpenStackProvisionServer CR and OpenStackBaremetalSet CR.

2. Re-enable fencing on the controllers:

   $ oc rsh -n openstack openstackclient
   $ ssh controller-0.ctlplane "sudo pcs property set stonith-enabled=true"

3. Perform any other post-upgrade actions relevant to your environment. For more information, see Performing post-upgrade actions in the Framework for upgrades (16.2 to 17.1) guide.