Chapter 5. Adopting the data plane

Procedure

1. Create an SSH authentication secret for the data plane nodes:

   $ oc apply -f - <<EOF
   apiVersion: v1
   kind: Secret
   metadata:
       name: dataplane-adoption-secret
   data:
       ssh-privatekey: |
   $(cat <path_to_SSH_key> | base64 | sed 's/^/        /')
   EOF

   • Replace <path_to_SSH_key> with the path to your SSH key.

     For more information about creating data plane secrets, see Creating the data plane secrets in Deploying Red Hat OpenStack Services on OpenShift.

2. Generate an ssh key-pair nova-migration-ssh-key secret:

   $ cd "$(mktemp -d)"
   $ ssh-keygen -f ./id -t ecdsa-sha2-nistp521 -N ''
   $ oc get secret nova-migration-ssh-key || oc create secret generic nova-migration-ssh-key \
     --from-file=ssh-privatekey=id \
     --from-file=ssh-publickey=id.pub \
     --type kubernetes.io/ssh-auth
   $ rm -f id*
   $ cd -

3. If TLS Everywhere is enabled, set LIBVIRT_PASSWORD to match the existing RHOSP deployment password:

   declare -A TRIPLEO_PASSWORDS
   TRIPLEO_PASSWORDS[default]="$HOME/overcloud-passwords.yaml"
   LIBVIRT_PASSWORD=$(cat ${TRIPLEO_PASSWORDS[default]} | grep ' LibvirtTLSPassword:' | awk -F ': ' '{ print $2; }')
   LIBVIRT_PASSWORD_BASE64=$(echo -n "$LIBVIRT_PASSWORD" | base64)

   1. Create libvirt-secret when TLS-e is enabled:

      $ oc apply -f - <<EOF
      apiVersion: v1
      kind: Secret
      metadata:
        name: libvirt-secret
      type: Opaque
      data:
        LibvirtPassword: ${LIBVIRT_PASSWORD_BASE64}
      EOF

4. Create a configuration map to use for all cells to configure a local storage back end for libvirt:

   $ oc apply -f - <<EOF
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: nova-cells-global-config
   data:
     99-nova-compute-cells-workarounds.conf: |
       [workarounds]
       disable_compute_service_check_for_ffu=true
   EOF

   • data provides the configuration files for all the cells.

   • 99-nova-compute-cells-workarounds.conf: | specifies the index of the <*.conf> files. There is a requirement to index the <*.conf> files from 03 to 99, based on precedence. A <99-*.conf> file takes the highest precedence, while indexes below 03 are reserved for internal use.

     Note

     If you adopt a live cloud, you might be required to carry over additional configurations for the default nova data plane services that are stored in the cell1 default nova-extra-config configuration map. Do not delete or overwrite the existing configuration in the cell1 default nova-extra-config configuration map that is assigned to nova. Overwriting the configuration can break the data place services that rely on specific contents of the nova-extra-config configuration map.

5. Configure a Red Hat Ceph Storage back end for libvirt:

   $ oc apply -f - <<EOF
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: nova-cells-global-config
   data:
     99-nova-compute-cells-workarounds.conf: |
       [workarounds]
       disable_compute_service_check_for_ffu=true
     03-ceph-nova.conf: |
       [libvirt]
       images_type=rbd
       images_rbd_pool=vms
       images_rbd_ceph_conf=/etc/ceph/ceph.conf
       images_rbd_glance_store_name=default_backend
       images_rbd_glance_copy_poll_interval=15
       images_rbd_glance_copy_timeout=600
       rbd_user=openstack
       rbd_secret_uuid=$CEPH_FSID
   EOF

   Note

   For Red Hat Ceph Storage environments with multi-cell configurations, you must name configuration maps and Red Hat OpenStack Platform data plane services similar to the following examples: nova-custom-ceph-cellX and nova-compute-extraconfig-cellX.

   Note

   For Distributed Compute Node (DCN) deployments, do not use the single nova-cells-global-config ConfigMap. Create a per-site ConfigMap and per-site OpenStackDataPlaneService for each site in your DCN deployment. Each site’s Compute service nodes require a different Red Hat Ceph Storage configuration and a different Image service endpoint. For more information, see Adopting Compute services with multiple Ceph back ends (DCN).

6. Create the data plane services for Compute service cells to enable pre-upgrade workarounds, and to configure the Compute services for your chosen storage back end:

   for CELL in $(echo $RENAMED_CELLS); do
   oc apply -f - <<EOF
   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneService
   metadata:
    name: nova-$CELL
   spec:
    dataSources:
      - secretRef:
          name: nova-$CELL-compute-config
      - secretRef:
          name: nova-migration-ssh-key
      - configMapRef:
          name: nova-cells-global-config
    playbook: osp.edpm.nova
    caCerts: combined-ca-bundle
    edpmServiceType: nova
    containerImageFields:
    - NovaComputeImage
    - EdpmIscsidImage
   EOF
    done

   • spec.dataSources.secretRef specifies an additional auto-generated nova-cell<X>-metadata-neutron-config secret to enable a local metadata service for cell<X>. You should also set spec.nova.template.cellTemplates.cell<X>.metadataServiceTemplate.enable in the OpenStackControlPlane/openstack CR, as described in Adopting the Compute service. You can configure a single top-level metadata, or define the metadata per cell.
   • nova-$CELL-compute-config specifies the secret that auto-generates for each cell<X>. You must append the nova-cell<X>-compute-config for each custom OpenStackDataPlaneService CR that is related to the Compute service.

   • nova-migration-ssh-key specifies the secret that you must append for each custom OpenStackDataPlaneService CR that is related to the Compute service.

     Note

     When creating your data plane services for Compute service cells, review the following considerations:

     • In this example, the same nova-migration-ssh-key key is shared across cells. However, you should use different keys for different cells.
     • For simple configuration overrides, you do not need a custom data plane service. However, to reconfigure the cell, cell1, the safest option is to create a custom service and a dedicated configuration map for it.
     • The cell, cell1, is already managed with the default OpenStackDataPlaneService CR called nova and its nova-extra-config configuration map. Do not change the default data plane service nova definition. The changes are lost when the RHOSO operator is updated with OLM.
     • When a cell spans multiple node sets, give the custom OpenStackDataPlaneService resources a name that relates to the node set, for example, nova-cell1-nfv and nova-cell1-enterprise. The auto-generated configuration maps are then named nova-cell1-nfv-extra-config and nova-cell1-enterprise-extra-config.
     • Different configurations for nodes in multiple node sets of the same cell are also supported, but are not covered in this guide.

7. If TLS Everywhere is enabled, append the following content to the OpenStackDataPlaneService CR:

     tlsCerts:
       nova:
         contents:
           - dnsnames
           - ips
         networks:
           - ctlplane
         issuer: osp-rootca-issuer-internal
         edpmRoleServiceName: nova
     caCerts: combined-ca-bundle
     edpmServiceType: nova

8. Create a secret for the subscription manager:

   $ oc create secret generic subscription-manager \
   --from-literal rhc_auth='{"login": {"username": "<subscription_manager_username>", "password": "<subscription_manager_password>"}}'

   • Replace <subscription_manager_username> with the applicable username.
   • Replace <subscription_manager_password> with the applicable password.

9. Create a secret for the Red Hat registry:

   $ oc create secret generic redhat-registry \
   --from-literal edpm_container_registry_logins='{"registry.redhat.io": {"<registry_username>": "<registry_password>"}}'

   • Replace <registry_username> with the applicable username.

   • Replace <registry_password> with the applicable password.

     Note

     You do not need to reference the subscription-manager secret in the dataSources field of the OpenStackDataPlaneService CR. The secret is already passed in with a node-specific OpenStackDataPlaneNodeSet CR in the ansibleVarsFrom property in the nodeTemplate field.

10. Create the data plane node set definitions for each cell:

    $ declare -A names
    $ for CELL in $(echo $RENAMED_CELLS); do
      ref="COMPUTES_$(echo ${CELL}|tr [:lower:] [:upper:])"
      eval names=\${!${ref}[@]}
      ref_api="COMPUTES_API_$(echo ${CELL}|tr [:lower:] [:upper:])"
      [ -z "$names" ] && continue
      ind=0
      rm -f computes-$CELL
      for compute in $names; do
        ip="${ref}[$compute]"
        ip_api="${ref_api}[$compute]"
        cat >> computes-$CELL << EOF
        ${compute}:
          hostName: $compute
          ansible:
            ansibleHost: $compute
          networks:
          - defaultRoute: true
            fixedIP: ${!ip}
            name: ctlplane
            subnetName: subnet1
          - name: internalapi
            subnetName: subnet1
            fixedIP: ${!ip_api}
          - name: storage
            subnetName: subnet1
          - name: tenant
            subnetName: subnet1
    EOF
        ind=$(( ind + 1 ))
      done
    
      test -f computes-$CELL || continue
      cat > nodeset-${CELL}.yaml <<EOF
    apiVersion: dataplane.openstack.org/v1beta1
    kind: OpenStackDataPlaneNodeSet
    metadata:
      name: openstack-$CELL
    spec:
      tlsEnabled: false
      networkAttachments:
          - ctlplane
      preProvisioned: true
      services:
        - redhat
        - bootstrap
        - download-cache
        - configure-network
        - validate-network
        - install-os
        - configure-os
        - ssh-known-hosts
        - run-os
        - reboot-os
        - install-certs
        - ovn
        - neutron-metadata
        - libvirt
        - nova-$CELL
        - telemetry
      env:
        - name: ANSIBLE_CALLBACKS_ENABLED
          value: "profile_tasks"
        - name: ANSIBLE_FORCE_COLOR
          value: "True"
        - name: ANSIBLE_VERBOSITY
          value: "3"
      nodeTemplate:
        ansibleSSHPrivateKeySecret: dataplane-adoption-secret
        ansible:
          ansibleUser: root
          ansibleVarsFrom:
          - secretRef:
              name: subscription-manager
          - secretRef:
              name: redhat-registry
          ansibleVars:
            rhc_release: 9.2
            rhc_repositories:
                - {name: "*", state: disabled}
                - {name: "rhel-9-for-x86_64-baseos-eus-rpms", state: enabled}
                - {name: "rhel-9-for-x86_64-appstream-eus-rpms", state: enabled}
                - {name: "rhel-9-for-x86_64-highavailability-eus-rpms", state: enabled}
                - {name: "rhoso-18.0-for-rhel-9-x86_64-rpms", state: enabled}
                - {name: "fast-datapath-for-rhel-9-x86_64-rpms", state: enabled}
                - {name: "rhceph-7-tools-for-rhel-9-x86_64-rpms", state: enabled}
            edpm_bootstrap_release_version_package: []
            # edpm_network_config
            # Default nic config template for a EDPM node
            # These vars are edpm_network_config role vars
            edpm_network_config_template: |
               ---
               {% set mtu_list = [ctlplane_mtu] %}
               {% for network in nodeset_networks %}
               {% set _ = mtu_list.append(lookup(vars, networks_lower[network] ~ _mtu)) %}
               {%- endfor %}
               {% set min_viable_mtu = mtu_list | max %}
               network_config:
               - type: ovs_bridge
                 name: {{ neutron_physical_bridge_name }}
                 mtu: {{ min_viable_mtu }}
                 use_dhcp: false
                 dns_servers: {{ ctlplane_dns_nameservers }}
                 domain: {{ dns_search_domains }}
                 addresses:
                 - ip_netmask: {{ ctlplane_ip }}/{{ ctlplane_cidr }}
                 routes: {{ ctlplane_host_routes }}
                 members:
                 - type: interface
                   name: nic1
                   mtu: {{ min_viable_mtu }}
                   # force the MAC address of the bridge to this interface
                   primary: true
               {% for network in nodeset_networks %}
                 - type: vlan
                   mtu: {{ lookup(vars, networks_lower[network] ~ _mtu) }}
                   vlan_id: {{ lookup(vars, networks_lower[network] ~ _vlan_id) }}
                   addresses:
                   - ip_netmask:
                       {{ lookup(vars, networks_lower[network] ~ _ip) }}/{{ lookup(vars, networks_lower[network] ~ _cidr) }}
                   routes: {{ lookup(vars, networks_lower[network] ~ _host_routes) }}
               {% endfor %}
    
            edpm_network_config_nmstate: false
            # Control resolv.conf management by NetworkManager
            # false = disable NetworkManager resolv.conf update (default)
            # true = enable NetworkManager resolv.conf update
            edpm_bootstrap_network_resolvconf_update: false
            edpm_network_config_hide_sensitive_logs: false
            #
            # These vars are for the network config templates themselves and are
            # considered EDPM network defaults.
            neutron_physical_bridge_name: br-ctlplane
            neutron_public_interface_name: eth0
    
            # edpm_nodes_validation
            edpm_nodes_validation_validate_controllers_icmp: false
            edpm_nodes_validation_validate_gateway_icmp: false
    
            # edpm ovn-controller configuration
            edpm_ovn_bridge_mappings: [<"bridge_mappings">]
            edpm_ovn_bridge: br-int
            edpm_ovn_encap_type: geneve
            ovn_monitor_all: true
            edpm_ovn_remote_probe_interval: 60000
            edpm_ovn_ofctrl_wait_before_clear: 8000
    
            timesync_ntp_servers:
            - hostname: clock.redhat.com
            - hostname: clock2.redhat.com
    
            edpm_bootstrap_command: |
              set -euxo pipefail
              dnf -y upgrade openstack-selinux
              rm -f /run/virtlogd.pid
    
            gather_facts: false
            # edpm firewall, change the allowed CIDR if needed
            edpm_sshd_configure_firewall: true
            edpm_sshd_allowed_ranges: [192.168.122.0/24]
    
            # Do not attempt OVS major upgrades here
            edpm_ovs_packages:
            - openvswitch3.3
            edpm_default_mounts:
              -  path: /dev/hugepages<size>
                 opts: pagesize=<size>
                fstype: hugetlbfs
                group: hugetlbfs
      nodes:
    EOF
      cat computes-$CELL >> nodeset-${CELL}.yaml
    done

    • ${compute}.hostName specifies the FQDN for the node if your deployment has a custom DNS Domain.
    • ${compute}.networks specifies the network composition. The network composition must match the source cloud configuration to avoid data plane connectivity downtime. The ctlplane network must come first. The commands only retain IP addresses for the hosts on the ctlplane and internalapi networks. Repeat this step for other isolated networks, or update the resulting files manually.
    • metadata.name: specifies the node set names for each cell, for example, openstack-cell1, openstack-cell2. Only create node sets for cells that contain Compute nodes.
    • spec.tlsEnabled specifies whether TLS Everywhere is enabled. If it is enabled, change tlsEnabled to true.
    • spec.services specifies the services to be adopted. If you are not adopting telemetry services, omit it from the services list.
    • neutron_physical_bridge_name: br-ctlplane specifies the bridge name. The bridge name and other OVN and Networking service-specific values must match the source cloud configuration to avoid data plane connectivity downtime.
    • edpm_ovn_bridge_mappings: Replace [<"bridge_mappings">] with the value of the bridge mappings in your configuration, for example, ["datacentre:br-ctlplane"].

    • path: /dev/hugepages<size> and opts: pagesize=<size> configures huge pages. Replace <size> with the size of the page. To configure multi-sized huge pages, create more items in the list. Note that the mount points must match the source cloud configuration.

      Note

      Ensure that you use the same ovn-controller settings in the OpenStackDataPlaneNodeSet CR that you used in the Compute service nodes before adoption. This configuration is stored in the external_ids column in the Open_vSwitch table in the Open vSwitch database:

      $ ovs-vsctl list Open .
      ...
      external_ids        : {hostname=standalone.localdomain, ovn-bridge=br-int, ovn-bridge-mappings=<bridge_mappings>, ovn-chassis-mac-mappings="datacentre:1e:0a:bb:e6:7c:ad", ovn-encap-ip="172.19.0.100", ovn-encap-tos="0", ovn-encap-type=geneve, ovn-match-northd-version=False, ovn-monitor-all=True, ovn-ofctrl-wait-before-clear="8000", ovn-openflow-probe-interval="60", ovn-remote="tcp:ovsdbserver-sb.openstack.svc:6642", ovn-remote-probe-interval="60000", rundir="/var/run/openvswitch", system-id="2eec68e6-aa21-4c95-a868-31aeafc11736"}
      ...

11. Deploy the OpenStackDataPlaneNodeSet CRs for each Compute cell:

    for CELL in $(echo $RENAMED_CELLS); do
    test -f nodeset-${CELL}.yaml || continue
    oc apply -f nodeset-${CELL}.yaml
    done

12. If you use a Red Hat Ceph Storage back end for Block Storage service (cinder), prepare the adopted data plane workloads:

    for CELL in $(echo $RENAMED_CELLS); do
    test -f nodeset-${CELL}.yaml || continue
    oc patch osdpns/openstack-$CELL --type=merge --patch "
    spec:
      services:
        - redhat
        - bootstrap
        - download-cache
        - configure-network
        - validate-network
        - install-os
        - configure-os
        - ssh-known-hosts
        - run-os
        - reboot-os
        - ceph-client
        - install-certs
        - ovn
        - neutron-metadata
        - libvirt
        - nova-$CELL
        - telemetry
      nodeTemplate:
        extraMounts:
        - extraVolType: Ceph
          volumes:
          - name: ceph
            secret:
              secretName: ceph-conf-files
          mounts:
          - name: ceph
            mountPath: "/etc/ceph"
            readOnly: true
      "
    done

13. Optional: Enable neutron-sriov-nic-agent in the OpenStackDataPlaneNodeSet CR:

    for CELL in $(echo $RENAMED_CELLS); do
    test -f nodeset-${CELL}.yaml || continue
    oc patch openstackdataplanenodeset openstack-$CELL --type='json' --patch='[
    {
      "op": "add",
      "path": "/spec/services/-",
      "value": "neutron-sriov"
    }, {
      "op": "add",
      "path": "/spec/nodeTemplate/ansible/ansibleVars/edpm_neutron_sriov_agent_SRIOV_NIC_physical_device_mappings",
      "value": "dummy_sriov_net:dummy-dev"
    }, {
      "op": "add",
      "path": "/spec/nodeTemplate/ansible/ansibleVars/edpm_neutron_sriov_agent_SRIOV_NIC_resource_provider_bandwidths",
      "value": "dummy-dev:40000000:40000000"
    }, {
      "op": "add",
      "path": "/spec/nodeTemplate/ansible/ansibleVars/edpm_neutron_sriov_agent_SRIOV_NIC_resource_provider_hypervisors",
      "value": "dummy-dev:standalone.localdomain"
    }]'
    done

14. Optional: Enable neutron-dhcp in the OpenStackDataPlaneNodeSet CR:

    for CELL in $(echo $RENAMED_CELLS); do
    test -f nodeset-${CELL}.yaml || continue
    oc patch openstackdataplanenodeset openstack-$CELL --type='json' --patch='[
    {
      "op": "add",
      "path": "/spec/services/-",
      "value": "neutron-dhcp"
    }]'
    done

    Note

    To use neutron-dhcp with OVN for the Bare Metal Provisioning service (ironic), you must set the disable_ovn_dhcp_for_baremetal_ports configuration option for the Networking service (neutron) to true. You can set this configuration in the NeutronAPI spec:

    ..
    spec:
      serviceUser: neutron
       ...
          customServiceConfig: |
              [DEFAULT]
              dhcp_agent_notification = True
              [ovn]
              disable_ovn_dhcp_for_baremetal_ports = true

15. Run the pre-adoption validation:

    1. Create the validation service:

       $ oc apply -f - <<EOF
       apiVersion: dataplane.openstack.org/v1beta1
       kind: OpenStackDataPlaneService
       metadata:
         name: pre-adoption-validation
       spec:
         playbook: osp.edpm.pre_adoption_validation
       EOF

    2. Create a OpenStackDataPlaneDeployment CR that runs only the validation:

       $ oc apply -f - <<EOF
       apiVersion: dataplane.openstack.org/v1beta1
       kind: OpenStackDataPlaneDeployment
       metadata:
         name: openstack-pre-adoption
       spec:
         nodeSets: $NODESETS
         servicesOverride:
         - pre-adoption-validation
       EOF

       Note

       If you created different migration SSH keys for different OpenStackDataPlaneService CRs, you should also define a separate OpenStackDataPlaneDeployment CR for each node set or node sets that represent a cell.

    3. When the validation is finished, confirm that the status of the Ansible EE pods is Completed:

       $ watch oc get pod -l app=openstackansibleee

       $ oc logs -l app=openstackansibleee -f --max-log-requests 20

    4. Wait for the deployment to reach the Ready status:

       $ oc wait --for condition=Ready openstackdataplanedeployment/openstack-pre-adoption --timeout=10m

       Important

       If any openstack-pre-adoption validations fail, you must reference the Ansible logs to determine which ones were unsuccessful, and then try the following troubleshooting options:

       • If the hostname validation failed, check that the hostname of the data plane node is correctly listed in the OpenStackDataPlaneNodeSet CR.
       • If the kernel argument check failed, ensure that the kernel argument configuration in the edpm_kernel_args and edpm_kernel_hugepages variables in the OpenStackDataPlaneNodeSet CR is the same as the kernel argument configuration that you used in the Red Hat OpenStack Platform (RHOSP) 17.1 node.
       • If the tuned profile check failed, ensure that the edpm_tuned_profile variable in the OpenStackDataPlaneNodeSet CR is configured to use the same profile as the one set on the RHOSP 17.1 node.

16. Remove the remaining director services:

    1. Create an OpenStackDataPlaneService CR to clean up the data plane services you are adopting:

       $ oc apply -f - <<EOF
       apiVersion: dataplane.openstack.org/v1beta1
       kind: OpenStackDataPlaneService
       metadata:
         name: tripleo-cleanup
       spec:
         playbook: osp.edpm.tripleo_cleanup
       EOF

    2. Create the OpenStackDataPlaneDeployment CR to run the clean-up:

       $ oc apply -f - <<EOF
       apiVersion: dataplane.openstack.org/v1beta1
       kind: OpenStackDataPlaneDeployment
       metadata:
         name: tripleo-cleanup
       spec:
         nodeSets: $NODESETS
         servicesOverride:
         - tripleo-cleanup
       EOF

17. When the clean-up is finished, deploy the OpenStackDataPlaneDeployment CR:

    $ oc apply -f - <<EOF
    apiVersion: dataplane.openstack.org/v1beta1
    kind: OpenStackDataPlaneDeployment
    metadata:
      name: openstack
    spec:
      nodeSets: $NODESETS
    EOF

    Note

    If you have other node sets to deploy, such as Networker nodes, you can add them in the nodeSets list in this step, or create separate OpenStackDataPlaneDeployment CRs later. You cannot add new node sets to an OpenStackDataPlaneDeployment CR after deployment.

Verification

1. Confirm that all the Ansible EE pods reach a Completed status:

   $ watch oc get pod -l app=openstackansibleee

   $ oc logs -l app=openstackansibleee -f --max-log-requests 20

2. Wait for the data plane node sets to reach the Ready status:

   for CELL in $(echo $RENAMED_CELLS); do
   oc wait --for condition=Ready osdpns/openstack-$CELL --timeout=30m
   done

3. Verify that the Networking service (neutron) agents are running:

   $ oc exec openstackclient -- openstack network agent list
   +--------------------------------------+------------------------------+------------------------+-------------------+-------+-------+----------------------------+
   | ID                                   | Agent Type                   | Host                   | Availability Zone | Alive | State | Binary                     |
   +--------------------------------------+------------------------------+------------------------+-------------------+-------+-------+----------------------------+
   | 174fc099-5cc9-4348-b8fc-59ed44fcfb0e | DHCP agent                   | standalone.localdomain | nova              | :-)   | UP    | neutron-dhcp-agent         |
   | 10482583-2130-5b0d-958f-3430da21b929 | OVN Metadata agent           | standalone.localdomain |                   | :-)   | UP    | neutron-ovn-metadata-agent |
   | a4f1b584-16f1-4937-b2b0-28102a3f6eaa | OVN Controller agent         | standalone.localdomain |                   | :-)   | UP    | ovn-controller             |
   +--------------------------------------+------------------------------+------------------------+-------------------+-------+-------+----------------------------+

Procedure

1. Define the OVN bridge mappings for each site. Each site requires a unique physnet that maps to the local provider network bridge:

   Expand

   Table 5.1. Example OVN bridge mappings

   Site	OVN bridge mapping
   Central	leaf0:br-ex
   DCN1	leaf1:br-ex
   DCN2	leaf2:br-ex

   Show more

2. Configure OVN for DCN sites. The default OVN controller configuration uses the Kubernetes ClusterIP (ovsdbserver-sb.openstack.svc), which is not routable from remote DCN sites. You must create a DCN-specific configuration that uses direct internalapi IP addresses.

   1. Get the OVN Southbound database internalapi IP addresses:

      $ oc get pod -l service=ovsdbserver-sb -o jsonpath='{range .items[*]}{.metadata.annotations.k8s\.v1\.cni\.cncf\.io/network-status}{"\n"}{end}' | jq -r '.[] | select(.name=="openstack/internalapi") | .ips[0]'

      Example output:

      172.17.0.34
      172.17.0.35
      172.17.0.36

   2. Create a ConfigMap with the OVN SB direct IPs for DCN sites:

      $ oc apply -f - <<EOF
      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: ovncontroller-config-dcn
        namespace: openstack
      data:
        ovsdb-config: |
          ovn-remote: tcp:172.17.0.34:6642,tcp:172.17.0.35:6642,tcp:172.17.0.36:6642
      EOF

      • Replace the IP addresses with the actual internalapi IPs from the previous step.

   3. Create an OpenStackDataPlaneService CR for DCN OVN configuration:

      $ oc apply -f - <<EOF
      apiVersion: dataplane.openstack.org/v1beta1
      kind: OpenStackDataPlaneService
      metadata:
        name: ovn-dcn
        namespace: openstack
      spec:
        addCertMounts: false
        caCerts: combined-ca-bundle
        containerImageFields:
        - OvnControllerImage
        dataSources:
        - configMapRef:
            name: ovncontroller-config-dcn
        edpmServiceType: ovn
        playbook: osp.edpm.ovn
        tlsCerts:
          default:
            contents:
            - dnsnames
            - ips
            issuer: osp-rootca-issuer-ovn
            keyUsages:
            - digital signature
            - key encipherment
            - server auth
            - client auth
            networks:
            - ctlplane
      EOF

      Note

      The ovn-dcn service uses the ovncontroller-config-dcn ConfigMap (through dataSources), which contains the direct internalapi IPs instead of the ClusterIP. DCN node sets must use this service instead of the default ovn service.

3. Create an OpenStackDataPlaneNodeSet CR for the central site Compute nodes:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneNodeSet
   metadata:
     name: openstack-edpm
   spec:
     tlsEnabled: false
     networkAttachments:
       - ctlplane
     preProvisioned: true
     services:
       - redhat
       - bootstrap
       - download-cache
       - configure-network
       - validate-network
       - install-os
       - configure-os
       - ssh-known-hosts
       - run-os
       - reboot-os
       - install-certs
       - ovn
       - neutron-metadata
       - libvirt
       - nova-cell1
       - telemetry
     nodeTemplate:
       ansibleSSHPrivateKeySecret: dataplane-adoption-secret
       ansible:
         ansibleUser: root
         ansibleVars:
           edpm_ovn_bridge_mappings: ["leaf0:br-ex"]
           edpm_ovn_bridge: br-int
           edpm_ovn_encap_type: geneve
           # Network configuration template for central site
           edpm_network_config_template: |
             ---
             network_config:
             - type: ovs_bridge
               name: {{ neutron_physical_bridge_name }}
               use_dhcp: false
               dns_servers: {{ ctlplane_dns_nameservers }}
               addresses:
               - ip_netmask: {{ ctlplane_ip }}/{{ ctlplane_cidr }}
               routes: {{ ctlplane_host_routes }}
               members:
               - type: interface
                 name: nic1
                 primary: true
             {% for network in nodeset_networks %}
               - type: vlan
                 vlan_id: {{ lookup(vars, networks_lower[network] ~ _vlan_id) }}
                 addresses:
                 - ip_netmask:
                     {{ lookup(vars, networks_lower[network] ~ _ip) }}/{{ lookup(vars, networks_lower[network] ~ _cidr) }}
                 routes: {{ lookup(vars, networks_lower[network] ~ _host_routes) }}
             {% endfor %}
     nodes:
       compute-0:
         hostName: compute-0.example.com
         ansible:
           ansibleHost: compute-0.example.com
         networks:
         - defaultRoute: true
           fixedIP: 192.168.122.100
           name: ctlplane
           subnetName: subnet1
         - name: internalapi
           subnetName: subnet1
         - name: storage
           subnetName: subnet1
         - name: tenant
           subnetName: subnet1

   • The OVN bridge mapping uses the central site physnet leaf0.
   • Central site nodes reference subnet1 for all networks.

4. Create an OpenStackDataPlaneNodeSet CR for DCN1 edge site compute nodes. You must add inter-site routes to the network configuration template and use the ovn-dcn service:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneNodeSet
   metadata:
     name: openstack-edpm-dcn1
   spec:
     tlsEnabled: false
     networkAttachments:
       - ctlplane
     preProvisioned: true
     services:
       - redhat
       - bootstrap
       - download-cache
       - configure-network
       - validate-network
       - install-os
       - configure-os
       - ssh-known-hosts
       - run-os
       - reboot-os
       - install-certs
       - ovn-dcn
       - neutron-metadata
       - libvirt
       - nova-cell1
       - telemetry
     nodeTemplate:
       ansibleSSHPrivateKeySecret: dataplane-adoption-secret
       ansible:
         ansibleUser: root
         ansibleVars:
           edpm_ovn_bridge_mappings: ["leaf1:br-ex"]
           edpm_ovn_bridge: br-int
           edpm_ovn_encap_type: geneve
           # Network configuration template for DCN1 site with inter-site routes
           edpm_network_config_template: |
             ---
             network_config:
             - type: ovs_bridge
               name: {{ neutron_physical_bridge_name }}
               use_dhcp: false
               dns_servers: {{ ctlplane_dns_nameservers }}
               addresses:
               - ip_netmask: {{ ctlplane_ip }}/{{ ctlplane_cidr }}
               routes:  

   1

   
                 {{ ctlplane_host_routes }}
                 - ip_netmask: 192.168.122.0/24
                   next_hop: 192.168.133.1
                 - ip_netmask: 192.168.144.0/24
                   next_hop: 192.168.133.1
               members:
               - type: interface
                 name: nic1
                 primary: true
             {% for network in nodeset_networks %}
               - type: vlan
                 vlan_id: {{ lookup(vars, networks_lower[network] ~ _vlan_id) }}
                 addresses:
                 - ip_netmask:
                     {{ lookup(vars, networks_lower[network] ~ _ip) }}/{{ lookup(vars, networks_lower[network] ~ _cidr) }}
                 routes:
                   {{ lookup(vars, networks_lower[network] ~ _host_routes) }}
                   {% if network == internalapi %}
                   - ip_netmask: 172.17.0.0/24
                     next_hop: 172.17.10.1
                   - ip_netmask: 172.17.20.0/24
                     next_hop: 172.17.10.1
                   {% endif %}
                   {% if network == storage %}
                   - ip_netmask: 172.18.0.0/24
                     next_hop: 172.18.10.1
                   - ip_netmask: 172.18.20.0/24
                     next_hop: 172.18.10.1
                   {% endif %}
                   {% if network == tenant %}
                   - ip_netmask: 172.19.0.0/24
                     next_hop: 172.19.10.1
                   - ip_netmask: 172.19.20.0/24
                     next_hop: 172.19.10.1
                   {% endif %}
             {% endfor %}
     nodes:
       dcn1-compute-0:
         hostName: dcn1-compute-0.example.com
         ansible:
           ansibleHost: dcn1-compute-0.example.com
         networks:
         - defaultRoute: true
           fixedIP: 192.168.133.100
           name: ctlplane
           subnetName: ctlplanedcn1
         - name: internalapi
           subnetName: internalapidcn1
         - name: storage
           subnetName: storagedcn1
         - name: tenant
           subnetName: tenantdcn1

   • Replace ovn with ovn-dcn under spec:services. This ensures OVN controller connects to the OVN Southbound database using direct internalapi IPs instead of the unreachable ClusterIP.
   • DCN1 uses the leaf1 physnet, for its OVN bridge mapping under spec:nodeTemplate:ansible:ansibleVars:edpm_ovn_bridge_mappings.
   • Inter-site routes must be added to the network configuration template. These routes enable DCN1 compute nodes to reach the central site (192.168.122.0/24) and other DCN sites (192.168.144.0/24 for DCN2). Similar routes are added for each service network (internalapi, storage, tenant).
   • DCN1 nodes reference site-specific subnet names like ctlplanedcn1 and internalapidcn1. These subnet names must match those defined in the NetConfig CR.

5. Repeat step 3 for all other DCN sites. Adjust site specific parameters:

   • The nodeset name, for example: openstack-edpm-dcn2
   • The OVN bridge mapping, for example: leaf2:br-ex
   • The subnet names, for example: ctlplanedcn2, and internalapidcn2
   • The inter-site routes. The routes from DCN2 should point to the central site subnets and the DCN1 site subnets.
   • The compute node definitions with site-appropriate IP addresses.

6. Deploy all nodesets by creating an OpenStackDataPlaneDeployment CR:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneDeployment
   metadata:
     name: openstack-edpm-deployment
   spec:
     nodeSets:
       - openstack-edpm
       - openstack-edpm-dcn1
       - openstack-edpm-dcn2

   Note

   All nodesets can be deployed in parallel once the control plane adoption is complete.

7. Wait for the deployment to complete:

   $ oc wait --for condition=Ready openstackdataplanedeployment/openstack-edpm-deployment --timeout=40m

Verification

1. Verify that all node sets reach the Ready status:

   $ oc get openstackdataplanenodeset
   NAME                  STATUS   MESSAGE
   openstack-edpm        True     Ready
   openstack-edpm-dcn1   True     Ready
   openstack-edpm-dcn2   True     Ready

2. Verify that Compute services are running across all sites. Ensure that all nova-compute services show State=up for nodes in all availability zones:

   $ oc exec openstackclient -- openstack compute service list

3. Verify inter-site connectivity by checking routes on a DCN Compute node:

   $ ssh dcn1-compute-0 ip route show | grep 172.17.0
   172.17.0.0/24 via 172.17.10.1 dev internalapi

4. Test that DCN Compute nodes can reach the control plane:

   $ ssh dcn1-compute-0 ping -c 3 172.17.0.30

   Replace 172.17.0.30 with an IP address of a control plane service on the internalapi network.

Procedure

1. Set the cell name variable. In a DCN deployment, all node sets belong to a single cell:

   $ DEFAULT_CELL_NAME="cell1"

2. Retrieve the fsid for each Red Hat Ceph Storage cluster and store them in shell variables:

   $ CEPH_FSID_CENTRAL=$(oc get secret ceph-conf-central -o json | jq -r .data."<central.conf>" | base64 -d | awk /fsid/{print $3})
   $ CEPH_FSID_DCN1=$(oc get secret ceph-conf-dcn1 -o json | jq -r .data."<dcn1.conf>" | base64 -d | awk /fsid/{print $3})
   $ CEPH_FSID_DCN2=$(oc get secret ceph-conf-dcn2 -o json | jq -r .data."<dcn2.conf>" | base64 -d | awk /fsid/{print $3})

   where:

   <central.conf>

   Specifies the name of the Red Hat Ceph Storage configuration file for the central site in the ceph-conf-central secret.

   <dcn1.conf>

   Specifies the name of the Red Hat Ceph Storage configuration file for an edge site in the ceph-conf-dcn1 secret.

   <dcn2.conf>

   Specifies the name of the Red Hat Ceph Storage configuration file for an additional edge site in the ceph-conf-dcn2 secret.

3. Create a ConfigMap for each site. Each ConfigMap contains the Red Hat Ceph Storage and Image service configuration specific to that site.

   The following example creates ConfigMap resources for a central site and two edge sites.

   1. Create the ConfigMap for the central site:

      $ oc apply -f - <<EOF
      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: nova-ceph-central
      data:
        99-nova-compute-cells-workarounds.conf: |
          [workarounds]
          disable_compute_service_check_for_ffu=true
        03-ceph-nova.conf: |
          [libvirt]
          images_type=rbd
          images_rbd_pool=vms
          images_rbd_ceph_conf=/etc/ceph/central.conf
          images_rbd_glance_store_name=central
          images_rbd_glance_copy_poll_interval=15
          images_rbd_glance_copy_timeout=600
          rbd_user=openstack
          rbd_secret_uuid=${CEPH_FSID_CENTRAL}
          [glance]
          endpoint_override = http://glance-central-internal.openstack.svc:9292
          valid_interfaces = internal
          [cinder]
          cross_az_attach = False
          catalog_info = volumev3:cinderv3:internalURL
      EOF

      Each ConfigMap contains three configuration sections:

      • [libvirt] points to the local Red Hat Ceph Storage cluster configuration and uses the local fsid as the rbd_secret_uuid.
      • [glance] uses endpoint_override to direct Image service requests to the local Image service API endpoint instead of the endpoint that is registered in the Identity service catalog. The examples use http:// for the Image service endpoints. If your Red Hat OpenStack Platform deployment uses TLS for internal endpoints, use https:// instead, and ensure that you have completed the TLS migration. For more information, see Migrating TLS-e to the RHOSO deployment.
      • [cinder] sets cross_az_attach = False to prevent volumes from being attached to instances in a different availability zone.

   2. Create the ConfigMap for the first edge site:

      $ oc apply -f - <<EOF
      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: nova-ceph-dcn1
      data:
        99-nova-compute-cells-workarounds.conf: |
          [workarounds]
          disable_compute_service_check_for_ffu=true
        03-ceph-nova.conf: |
          [libvirt]
          images_type=rbd
          images_rbd_pool=vms
          images_rbd_ceph_conf=/etc/ceph/dcn1.conf
          images_rbd_glance_store_name=dcn1
          images_rbd_glance_copy_poll_interval=15
          images_rbd_glance_copy_timeout=600
          rbd_user=openstack
          rbd_secret_uuid=${CEPH_FSID_DCN1}
          [glance]
          endpoint_override = http://glance-dcn1-internal.openstack.svc:9292
          valid_interfaces = internal
          [cinder]
          cross_az_attach = False
          catalog_info = volumev3:cinderv3:internalURL
      EOF

   3. Create the ConfigMap for the second edge site:

      $ oc apply -f - <<EOF
      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: nova-ceph-dcn2
      data:
        99-nova-compute-cells-workarounds.conf: |
          [workarounds]
          disable_compute_service_check_for_ffu=true
        03-ceph-nova.conf: |
          [libvirt]
          images_type=rbd
          images_rbd_pool=vms
          images_rbd_ceph_conf=/etc/ceph/dcn2.conf
          images_rbd_glance_store_name=dcn2
          images_rbd_glance_copy_poll_interval=15
          images_rbd_glance_copy_timeout=600
          rbd_user=openstack
          rbd_secret_uuid=${CEPH_FSID_DCN2}
          [glance]
          endpoint_override = http://glance-dcn2-internal.openstack.svc:9292
          valid_interfaces = internal
          [cinder]
          cross_az_attach = False
          catalog_info = volumev3:cinderv3:internalURL
      EOF

      Important

      The endpoint_override in the [glance] section is different for each site. This setting directs the Compute service to contact the local Image service API instead of the central endpoint registered in the Identity service catalog. Without this setting, all Compute nodes contact the central Image service, and image data is transferred across the WAN instead of read from the local Red Hat Ceph Storage cluster.

      • Central Compute nodes use glance-central-internal.openstack.svc
      • DCN1 Compute nodes use glance-dcn1-internal.openstack.svc
      • DCN2 Compute nodes use glance-dcn2-internal.openstack.svc

      These endpoint names correspond to the GlanceAPI instances that are created when you adopt the Image service with DCN back ends.

4. Create a per-site OpenStackDataPlaneService CR for each site. Each service references the site-specific ConfigMap that you created in the previous step:

   $ oc apply -f - <<EOF
   ---
   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneService
   metadata:
     name: nova-custom-ceph-central
   spec:
     dataSources:
       - configMapRef:
           name: nova-ceph-central
       - secretRef:
           name: nova-${DEFAULT_CELL_NAME}-compute-config
       - secretRef:
           name: nova-migration-ssh-key
     playbook: osp.edpm.nova
     caCerts: combined-ca-bundle
     edpmServiceType: nova
     containerImageFields:
     - NovaComputeImage
     - EdpmIscsidImage
   ---
   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneService
   metadata:
     name: nova-custom-ceph-dcn1
   spec:
     dataSources:
       - configMapRef:
           name: nova-ceph-dcn1
       - secretRef:
           name: nova-${DEFAULT_CELL_NAME}-compute-config
       - secretRef:
           name: nova-${DEFAULT_CELL_NAME}-metadata-neutron-config
       - secretRef:
           name: nova-migration-ssh-key
     playbook: osp.edpm.nova
     caCerts: combined-ca-bundle
     edpmServiceType: nova
     containerImageFields:
     - NovaComputeImage
     - EdpmIscsidImage
   ---
   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneService
   metadata:
     name: nova-custom-ceph-dcn2
   spec:
     dataSources:
       - configMapRef:
           name: nova-ceph-dcn2
       - secretRef:
           name: nova-${DEFAULT_CELL_NAME}-compute-config
       - secretRef:
           name: nova-${DEFAULT_CELL_NAME}-metadata-neutron-config
       - secretRef:
           name: nova-migration-ssh-key
     playbook: osp.edpm.nova
     caCerts: combined-ca-bundle
     edpmServiceType: nova
     containerImageFields:
     - NovaComputeImage
     - EdpmIscsidImage
   EOF

   Note

   All OpenStackDataPlaneService CRs reference the same cell secret (nova-cell1-compute-config) because all node sets belong to a single cell. The per-site ConfigMap is what differentiates the Red Hat Ceph Storage and Image service configuration for each site.

5. When you create the OpenStackDataPlaneNodeSet CR for each site, reference the per-site service in the services list instead of nova-$CELL. For example:

   • The central node set uses nova-custom-ceph-central in its services list.
   • The DCN1 node set uses nova-custom-ceph-dcn1 in its services list.

   • The DCN2 node set uses nova-custom-ceph-dcn2 in its services list.

     1. If you have already created the OpenStackDataPlaneNodeSet CRs with the default nova-$CELL service, patch each node set to use the per-site service. The following example patches the central node set:

        $ oc patch osdpns/openstack-${DEFAULT_CELL_NAME} --type=merge --patch "
        spec:
          services:
            - bootstrap
            - download-cache
            - configure-network
            - validate-network
            - install-os
            - configure-os
            - ssh-known-hosts
            - run-os
            - reboot-os
            - install-certs
            - ceph-client
            - ovn
            - neutron-metadata
            - libvirt
            - nova-custom-ceph-central
          nodeTemplate:
            extraMounts:
            - extraVolType: Ceph
              volumes:
              - name: ceph
                secret:
                  secretName: ceph-conf-central
              mounts:
              - name: ceph
                mountPath: "/etc/ceph"
                readOnly: true
        "

     2. Patch each DCN edge node set with the same services list, replacing ovn with ovn-dcn and nova-custom-ceph-central with the per-site service name. You must include the ceph-client service so that the Red Hat Ceph Storage configuration files from the per-site secret are deployed into the Compute service containers on the edge nodes. Without ceph-client, the /etc/ceph/ directory inside the Compute service container is empty and instances fail to launch with a RADOS object not found (error calling conf_read_file) error.

        For example, for the DCN1 node set named dcn1:

        $ oc patch osdpns/dcn1 --type=merge --patch "
        spec:
          services:
            - bootstrap
            - download-cache
            - configure-network
            - validate-network
            - install-os
            - configure-os
            - ssh-known-hosts
            - run-os
            - reboot-os
            - install-certs
            - ceph-client
            - ovn-dcn
            - neutron-metadata
            - libvirt
            - nova-custom-ceph-dcn1
          nodeTemplate:
            extraMounts:
            - extraVolType: Ceph
              volumes:
              - name: ceph
                secret:
                  secretName: ceph-conf-dcn1
              mounts:
              - name: ceph
                mountPath: "/etc/ceph"
                readOnly: true
        "

        Repeat this step for each additional edge site, replacing dcn1 and nova-custom-ceph-dcn1 with the appropriate site name, for example, dcn2 and nova-custom-ceph-dcn2.

Procedure

1. Wait for the Compute service data plane services version to update for all the cells:

   for CELL in $(echo $RENAMED_CELLS); do
   oc exec openstack-$CELL-galera-0 -c galera -- mysql -rs -uroot -p"${PODIFIED_DB_ROOT_PASSWORD[$CELL]}" \
     -e "select a.version from nova_${CELL}.services a join nova_${CELL}.services b where a.version!=b.version and a.binary='nova-compute' and a.deleted=0;"
   done

   Note

   The query returns an empty result when the update is completed. No downtime is expected for virtual machine (VM) workloads.

   Review any errors in the nova Compute agent logs on the data plane, and the nova-conductor journal records on the control plane.

2. Patch the OpenStackControlPlane CR to remove the pre-fast-forward upgrade workarounds from the Compute control plane services:

   $ rm -f celltemplates
   $ for CELL in $(echo $RENAMED_CELLS); do
   $ cat >> celltemplates << EOF
           ${CELL}:
             metadataServiceTemplate:
               customServiceConfig: |
                 [workarounds]
                 disable_compute_service_check_for_ffu=false
             conductorServiceTemplate:
               customServiceConfig: |
                 [workarounds]
                 disable_compute_service_check_for_ffu=false
   EOF
   done
   
   $ cat > oscp-patch.yaml << EOF
   spec:
     nova:
       template:
         apiServiceTemplate:
           customServiceConfig: |
             [workarounds]
             disable_compute_service_check_for_ffu=false
         metadataServiceTemplate:
           customServiceConfig: |
             [workarounds]
             disable_compute_service_check_for_ffu=false
         schedulerServiceTemplate:
           customServiceConfig: |
             [workarounds]
             disable_compute_service_check_for_ffu=false
         cellTemplates:
           cell0:
             conductorServiceTemplate:
               customServiceConfig: |
                 [workarounds]
                 disable_compute_service_check_for_ffu=false
   EOF
   $ cat celltemplates >> oscp-patch.yaml

   • If you are adopting the Compute service with the Bare Metal Provisioning service (ironic), append the following novaComputeTemplates in the cell<X> section of the Compute service CR patch:

             cell<X>:
               novaComputeTemplates:
                 <hostname>:
                   customServiceConfig: |
                     [DEFAULT]
                     host = <hostname>
                     [workarounds]
                     disable_compute_service_check_for_ffu=true
                   computeDriver: ironic.IronicDriver
             ...

     where:

     <hostname>

     Specifies the hostname of the node that is running the ironic Compute driver in the source cloud of cell<X>.

3. Apply the patch file:

   $ oc patch openstackcontrolplane openstack --type=merge --patch-file=oscp-patch.yaml

4. Wait until the Compute control plane services CRs are ready:

   $ oc wait --for condition=Ready --timeout=300s Nova/nova

5. Remove the pre-fast-forward upgrade workarounds from the Compute data plane services:

   $ oc patch cm nova-cells-global-config --type=json -p='[{"op": "replace", "path": "/data/99-nova-compute-cells-workarounds.conf", "value": "[workarounds]\n"}]'
   $ for CELL in $(echo $RENAMED_CELLS); do
   $ oc get Openstackdataplanenodeset openstack-${CELL} || continue
   $ oc apply -f - <<EOF
   ---
   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneDeployment
   metadata:
     name: openstack-nova-compute-ffu-$CELL
   spec:
     nodeSets:
       - openstack-${CELL}
     servicesOverride:
       - nova-${CELL}
   backoffLimit: 3
   EOF
   done

6. Wait for the Compute data plane services to be ready for all the cells:

   $ oc wait --for condition=Ready openstackdataplanedeployments --all --timeout=5m

7. Run Compute database online migrations to complete the upgrade:

   $ oc exec -it nova-cell0-conductor-0 -- nova-manage db online_data_migrations
   $ for CELL in $(echo $RENAMED_CELLS); do
   $ oc exec -it nova-${CELL}-conductor-0 -- nova-manage db online_data_migrations
   done

8. Discover the Compute hosts in the cells:

   $ oc rsh nova-cell0-conductor-0 nova-manage cell_v2 discover_hosts --verbose

9. If you have a test VM that is not a production workload, complete the following verification steps:

   1. Verify if the existing test VM instance is running:

      ${BASH_ALIASES[openstack]} server --os-compute-api-version 2.48 show --diagnostics test 2>&1 || echo FAIL

   2. Verify if the Compute services can stop the existing test VM instance:

      ${BASH_ALIASES[openstack]} server list -c Name -c Status -f value | grep -qF "test ACTIVE" && ${BASH_ALIASES[openstack]} server stop test || echo PASS
      ${BASH_ALIASES[openstack]} server list -c Name -c Status -f value | grep -qF "test SHUTOFF" || echo FAIL
      ${BASH_ALIASES[openstack]} server --os-compute-api-version 2.48 show --diagnostics test 2>&1 || echo PASS

   3. Verify if the Compute services can start the existing test VM instance:

      ${BASH_ALIASES[openstack]} server list -c Name -c Status -f value | grep -qF "test SHUTOFF" && ${BASH_ALIASES[openstack]} server start test || echo PASS
      ${BASH_ALIASES[openstack]} server list -c Name -c Status -f value | grep -qF "test ACTIVE" && \
        ${BASH_ALIASES[openstack]} server --os-compute-api-version 2.48 show --diagnostics test --fit-width -f json | jq -r '.state' | grep running || echo FAIL

1. Check for the nodes where OVN Controller Gateway agent agents are running. The list of agents varies depending on the services you enabled:

   $ oc exec openstackclient -- openstack network agent list
   +--------------------------------------+------------------------------+--------------------------+-------------------+-------+-------+----------------------------+
   | ID                                   | Agent Type                   | Host                     | Availability Zone | Alive | State | Binary                     |
   +--------------------------------------+------------------------------+--------------------------+-------------------+-------+-------+----------------------------+
   | e5075ee0-9dd9-4f0a-a42a-6bbdf1a6111c | OVN Controller Gateway agent | controller-0.localdomain |                   | XXX   | UP    | ovn-controller             |
   | f3112349-054c-403a-b00a-e219238192b8 | OVN Controller agent         | compute-0.localdomain    |                   | XXX   | UP    | ovn-controller             |
   | af9dae2d-1c1c-55a8-a743-f84719f6406d | OVN Metadata agent           | compute-0.localdomain    |                   | XXX   | UP    | neutron-ovn-metadata-agent |
   | 51a11df8-a66e-47a2-aec0-52eb8589626c | OVN Controller Gateway agent | controller-1.localdomain |                   | XXX   | UP    | ovn-controller             |
   | bb817e5e-7832-410a-9e67-934dac8c602f | OVN Controller Gateway agent | controller-2.localdomain |                   | XXX   | UP    | ovn-controller             |
   +--------------------------------------+------------------------------+--------------------------+-------------------+-------+-------+----------------------------+

Procedure

1. Deploy the OpenStackDataPlaneNodeSet CR for your nodes:

   Note

   You can reuse most of the nodeTemplate section from the OpenStackDataPlaneNodeSet CR that is designated for your Compute nodes. You can omit some of the variables because of the limited set of services that are running on the Networker nodes.

   $ oc apply -f - <<EOF
   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneNodeSet
   metadata:
     name: openstack-networker
   spec:
     tlsEnabled: false
     networkAttachments:
         - ctlplane
     preProvisioned: true
     services:
       - redhat
       - bootstrap
       - download-cache
       - configure-network
       - validate-network
       - install-os
       - configure-os
       - ssh-known-hosts
       - run-os
       - install-certs
       - ovn
     env:
       - name: ANSIBLE_CALLBACKS_ENABLED
         value: "profile_tasks"
       - name: ANSIBLE_FORCE_COLOR
         value: "True"
     nodes:
       controller-0:
         hostName: controller-0
         ansible:
           ansibleHost: ${networkers[controller-0.localdomain]}
         networks:
         - defaultRoute: true
           fixedIP: ${networkers[controller-0.localdomain]}
           name: ctlplane
           subnetName: subnet1
         - name: internalapi
           subnetName: subnet1
         - name: storage
           subnetName: subnet1
         - name: tenant
           subnetName: subnet1
       controller-1:
         hostName: controller-1
         ansible:
           ansibleHost: ${networkers[controller-1.localdomain]}
         networks:
         - defaultRoute: true
           fixedIP: ${networkers[controller-1.localdomain]}
           name: ctlplane
           subnetName: subnet1
         - name: internalapi
           subnetName: subnet1
         - name: storage
           subnetName: subnet1
         - name: tenant
           subnetName: subnet1
       controller-2:
         hostName: controller-2
         ansible:
           ansibleHost: ${networkers[controller-2.localdomain]}
         networks:
         - defaultRoute: true
           fixedIP: ${networkers[controller-2.localdomain]}
           name: ctlplane
           subnetName: subnet1
         - name: internalapi
           subnetName: subnet1
         - name: storage
           subnetName: subnet1
         - name: tenant
           subnetName: subnet1
     nodeTemplate:
       ansibleSSHPrivateKeySecret: dataplane-adoption-secret
       ansible:
         ansibleUser: root
         ansibleVarsFrom:
         - secretRef:
             name: subscription-manager
         - secretRef:
             name: redhat-registry
         ansibleVars:
           rhc_release: 9.2
           rhc_repositories:
               - {name: "*", state: disabled}
               - {name: "rhel-9-for-x86_64-baseos-eus-rpms", state: enabled}
               - {name: "rhel-9-for-x86_64-appstream-eus-rpms", state: enabled}
               - {name: "rhel-9-for-x86_64-highavailability-eus-rpms", state: enabled}
               - {name: "rhoso-18.0-for-rhel-9-x86_64-rpms", state: enabled}
               - {name: "fast-datapath-for-rhel-9-x86_64-rpms", state: enabled}
               - {name: "rhceph-7-tools-for-rhel-9-x86_64-rpms", state: enabled}
           edpm_bootstrap_release_version_package: []
           # edpm_network_config
           # Default nic config template for a EDPM node
           # These vars are edpm_network_config role vars
           edpm_network_config_template: |
              ---
              {% set mtu_list = [ctlplane_mtu] %}
              {% for network in nodeset_networks %}
              {% set _ = mtu_list.append(lookup(vars, networks_lower[network] ~ _mtu)) %}
              {%- endfor %}
              {% set min_viable_mtu = mtu_list | max %}
              network_config:
              - type: ovs_bridge
                name: {{ neutron_physical_bridge_name }}
                mtu: {{ min_viable_mtu }}
                use_dhcp: false
                dns_servers: {{ ctlplane_dns_nameservers }}
                domain: {{ dns_search_domains }}
                addresses:
                - ip_netmask: {{ ctlplane_ip }}/{{ ctlplane_cidr }}
                routes: {{ ctlplane_host_routes }}
                members:
                - type: interface
                  name: nic1
                  mtu: {{ min_viable_mtu }}
                  # force the MAC address of the bridge to this interface
                  primary: true
              {% for network in nodeset_networks %}
                - type: vlan
                  mtu: {{ lookup(vars, networks_lower[network] ~ _mtu) }}
                  vlan_id: {{ lookup(vars, networks_lower[network] ~ _vlan_id) }}
                  addresses:
                  - ip_netmask:
                      {{ lookup(vars, networks_lower[network] ~ _ip) }}/{{ lookup(vars, networks_lower[network] ~ _cidr) }}
                  routes: {{ lookup(vars, networks_lower[network] ~ _host_routes) }}
              {% endfor %}
           edpm_network_config_nmstate: false
           edpm_network_config_hide_sensitive_logs: false
           #
           # These vars are for the network config templates themselves and are
           # considered EDPM network defaults.
           neutron_physical_bridge_name: br-ctlplane
           neutron_public_interface_name: eth0
   
           # edpm_nodes_validation
           edpm_nodes_validation_validate_controllers_icmp: false
           edpm_nodes_validation_validate_gateway_icmp: false
   
           # edpm ovn-controller configuration
           edpm_ovn_bridge_mappings: [<"bridge_mappings">]
           edpm_ovn_bridge: br-int
           edpm_ovn_encap_type: geneve
           ovn_monitor_all: true
           edpm_ovn_remote_probe_interval: 60000
           edpm_ovn_ofctrl_wait_before_clear: 8000
   
           # serve as a OVN gateway
           edpm_enable_chassis_gw: true
   
           timesync_ntp_servers:
           - hostname: clock.redhat.com
           - hostname: clock2.redhat.com
   
   
           gather_facts: false
           enable_debug: false
           # edpm firewall, change the allowed CIDR if needed
           edpm_sshd_configure_firewall: true
           edpm_sshd_allowed_ranges: [192.168.122.0/24]
           # SELinux module
           edpm_selinux_mode: enforcing
   
           # Do not attempt OVS major upgrades here
           edpm_ovs_packages:
           - openvswitch3.3
   EOF

   • spec.tlsEnabled specifies whether TLS Everywhere is enabled. If TLS is enabled, change spec:tlsEnabled to true.
   • edpm_ovn_bridge_mappings: Replace [<"bridge_mappings">] with the bridge mapping values that you used in your Red Hat OpenStack Platform 17.1 deployment, for example, ["datacentre:br-ctlplane"].
   • edpm_enable_chassis_gw specifies whether to run ovn-controller in gateway mode.

2. Ensure that you use the same ovn-controller settings in the OpenStackDataPlaneNodeSet CR that you used in the Networker nodes before adoption. This configuration is stored in the external_ids column in the Open_vSwitch table in the Open vSwitch database:

   ovs-vsctl list Open .
   ...
   external_ids        : {hostname=controller-0.localdomain, ovn-bridge=br-int, ovn-bridge-mappings=<bridge_mappings>, ovn-chassis-mac-mappings="datacentre:1e:0a:bb:e6:7c:ad", ovn-cms-options=enable-chassis-as-gw, ovn-encap-ip="172.19.0.100", ovn-encap-tos="0", ovn-encap-type=geneve, ovn-match-northd-version=False, ovn-monitor-all=True, ovn-ofctrl-wait-before-clear="8000", ovn-openflow-probe-interval="60", ovn-remote="tcp:ovsdbserver-sb.openstack.svc:6642", ovn-remote-probe-interval="60000", rundir="/var/run/openvswitch", system-id="2eec68e6-aa21-4c95-a868-31aeafc11736"}
   ...

   • Replace <bridge_mappings> with the value of the bridge mappings in your configuration, for example, "datacentre:br-ctlplane".

3. Optional: Enable neutron-metadata in the OpenStackDataPlaneNodeSet CR:

   $ oc patch openstackdataplanenodeset <networker_CR_name> --type='json' --patch='[
     {
       "op": "add",
       "path": "/spec/services/-",
       "value": "neutron-metadata"
     }]'

   • Replace <networker_CR_name> with the name of the CR that you deployed for your Networker nodes, for example, openstack-networker.

4. Optional: Enable neutron-dhcp in the OpenStackDataPlaneNodeSet CR:

   $ oc patch openstackdataplanenodeset <networker_CR_name> --type='json' --patch='[
     {
       "op": "add",
       "path": "/spec/services/-",
       "value": "neutron-dhcp"
     }]'

5. Run the pre-adoption-validation service for Networker nodes:

   1. Create a OpenStackDataPlaneDeployment CR that runs only the validation:

      $ oc apply -f - <<EOF
      apiVersion: dataplane.openstack.org/v1beta1
      kind: OpenStackDataPlaneDeployment
      metadata:
        name: openstack-pre-adoption-networker
      spec:
        nodeSets:
        - openstack-networker
        servicesOverride:
        - pre-adoption-validation
      EOF

   2. When the validation is finished, confirm that the status of the Ansible EE pods is Completed:

      $ watch oc get pod -l app=openstackansibleee

      $ oc logs -l app=openstackansibleee -f --max-log-requests 20

   3. Wait for the deployment to reach the Ready status:

      $ oc wait --for condition=Ready openstackdataplanedeployment/openstack-pre-adoption-networker --timeout=10m

6. Deploy the OpenStackDataPlaneDeployment CR for Networker nodes:

   $ oc apply -f - <<EOF
   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneDeployment
   metadata:
     name: openstack-networker
   spec:
     nodeSets:
     - openstack-networker
   EOF

   Note

   Alternatively, you can include the Networker node set in the nodeSets list before you deploy the main OpenStackDataPlaneDeployment CR. You cannot add new node sets to the OpenStackDataPlaneDeployment CR after deployment.

7. Clean up any Networking service (neutron) agents that are no longer running.

   Note

   In some cases, agents from the old data plane that are replaced or retired remain in RHOSO. The function these agents provided might be provided by a new agent that is running in RHOSO, or the function might be replaced by other components. For example, DHCP agents might no longer be needed, since OVN DHCP in RHOSO can provide this function.

   1. List the agents:

      $ oc exec openstackclient -- openstack network agent list
      +--------------------------------------+------------------------------+--------------------------+-------------------+-------+-------+----------------------------+
      | ID                                   | Agent Type                   | Host                     | Availability Zone | Alive | State | Binary                     |
      +--------------------------------------+------------------------------+--------------------------+-------------------+-------+-------+----------------------------+
      | e5075ee0-9dd9-4f0a-a42a-6bbdf1a6111c | OVN Controller Gateway agent | controller-0.localdomain |                   | :-)   | UP    | ovn-controller             |
      | 856960f0-5530-46c7-a331-6eadcba362da | DHCP agent                   | controller-1.localdomain | nova              | XXX   | UP    | neutron-dhcp-agent         |
      | 8bd22720-789f-45b8-8d7d-006dee862bf9 | DHCP agent                   | controller-2.localdomain | nova              | XXX   | UP    | neutron-dhcp-agent         |
      | e584e00d-be4c-4e98-a11a-4ecd87d21be7 | DHCP agent                   | controller-0.localdomain | nova              | XXX   | UP    | neutron-dhcp-agent         |
      +--------------------------------------+------------------------------+--------------------------+-------------------+-------+-------+----------------------------+

   2. If any agent in the list shows XXX in the Alive field, verify the Host and Agent Type, if the functions of this agent is no longer required, and the agent has been permanently stopped on the Red Hat OpenStack Platform host. Then, delete the agent:

      $ oc exec openstackclient -- openstack network agent delete <agent_id>

      • Replace <agent_id> with the ID of the agent to delete, for example, 856960f0-5530-46c7-a331-6eadcba362da.

Verification

1. Confirm that all the Ansible EE pods reach a Completed status:

   $ watch oc get pod -l app=openstackansibleee

   $ oc logs -l app=openstackansibleee -f --max-log-requests 20

2. Wait for the data plane node set to reach the Ready status:

   $ oc wait --for condition=Ready osdpns/<networker_CR_name> --timeout=30m

   • Replace <networker_CR_name> with the name of the CR that you deployed for your Networker nodes, for example, openstack-networker.

3. Verify that the Networking service (neutron) agents are running. The list of agents varies depending on the services you enabled:

   $ oc exec openstackclient -- openstack network agent list
   +--------------------------------------+------------------------------+--------------------------+-------------------+-------+-------+----------------------------+
   | ID                                   | Agent Type                   | Host                     | Availability Zone | Alive | State | Binary                     |
   +--------------------------------------+------------------------------+--------------------------+-------------------+-------+-------+----------------------------+
   | e5075ee0-9dd9-4f0a-a42a-6bbdf1a6111c | OVN Controller Gateway agent | controller-0.localdomain |                   | :-)   | UP    | ovn-controller             |
   | f3112349-054c-403a-b00a-e219238192b8 | OVN Controller agent         | compute-0.localdomain    |                   | :-)   | UP    | ovn-controller             |
   | af9dae2d-1c1c-55a8-a743-f84719f6406d | OVN Metadata agent           | compute-0.localdomain    |                   | :-)   | UP    | neutron-ovn-metadata-agent |
   | 51a11df8-a66e-47a2-aec0-52eb8589626c | OVN Controller Gateway agent | controller-1.localdomain |                   | :-)   | UP    | ovn-controller             |
   | bb817e5e-7832-410a-9e67-934dac8c602f | OVN Controller Gateway agent | controller-2.localdomain |                   | :-)   | UP    | ovn-controller             |
   +--------------------------------------+------------------------------+--------------------------+-------------------+-------+-------+----------------------------+

Procedure

1. Create the secret:

   $ oc apply -f fencing-secret.yaml -n openstack

2. Optional: Create the Instance HA configuration map and set the DISABLED parameter to false. For example:

   $ cat << EOF > iha-cm.yaml
   kind: ConfigMap
   metadata:
     name: instanceha-0-config
     namespace: openstack
   apiVersion: v1
   data:
     config.yaml: |
       config:
         EVACUABLE_TAG: "evacuable"
         TAGGED_IMAGES: "true"
         TAGGED_FLAVORS: "true"
         TAGGED_AGGREGATES: "true"
         SMART_EVACUATION: "false"
         DELTA: "30"
         DELAY: "0"
         POLL: "45"
         THRESHOLD: "50"
         WORKERS: "4"
         RESERVED_HOSTS: "false"
         LEAVE_DISABLED: "false"
         CHECK_KDUMP: "false"
         LOGLEVEL: "info"
         DISABLED: "false"
   EOF

   1. Apply the configuration:

      $ oc apply -f iha-cm.yaml -n openstack

      Note

      If you want to restrict which Compute nodes are evacuated, create host aggregates and set them by using the EVACUABLE_TAG parameter. Alternatively, you can set the TAGGED_AGGREGATES parameter to false to enable monitoring and evacuation of all your Compute nodes. For more information about Instance HA service parameters, see Editing the Instance HA service parameters in Configuring high availability for instances.

3. Create an Instance HA resource and reference the fencing secret and configuration map. For example:

   $ cat << EOF > iha.yaml
   apiVersion: instanceha.openstack.org/v1beta1
   kind: InstanceHa
   metadata:
     name: instanceha-0
     namespace: openstack
   spec:
     caBundleSecretName: combined-ca-bundle
     instanceHaConfigMap:
     fencingSecret: fencing-secret
   EOF

   • spec.instanceHaConfigMap defines the name of the YAML file containing the Instance HA configuration map that you created. If you do not create this file, then a YAML file called instanceha-config is created automatically when the Instance HA service is installed, providing the default values of the Instance HA service parameters. You can then edit the values as needed.

4. Deploy the Instance HA resource:

   $ oc apply -f iha.yaml -n openstack

Procedure

1. Ensure that the connectivity between the new control plane and the adopted Compute nodes is functional by creating a new load balancer and checking that its provisioning_status becomes ACTIVE:

   $ alias openstack="oc exec -t openstackclient -- openstack"
   $ openstack loadbalancer create --vip-subnet-id public-subnet --name lb-post-adoption --wait

2. Trigger a failover for all existing load balancers to upgrade the amphorae virtual machines to use the new image and to establish connectivity with the new control plane:

   $ openstack loadbalancer list -f value -c id | \
         xargs -r -n1 -P4 ${BASH_ALIASES[openstack]} loadbalancer failover --wait

3. Delete old flavors that were migrated to the new control plane:

   $ openstack flavor delete octavia_65
   # The following flavors might not exist in OSP 17.1 deployments
   $ openstack flavor show octavia_amphora-mvcpu-ha && \
     openstack flavor delete octavia_amphora-mvcpu-ha
   $ openstack loadbalancer flavor show octavia_amphora-mvcpu-ha && \
     openstack loadbalancer flavor delete octavia_amphora-mvcpu-ha
   $ openstack loadbalancer flavorprofile show octavia_amphora-mvcpu-ha_profile && \
     openstack loadbalancer flavorprofile delete octavia_amphora-mvcpu-ha_profile

   Note

   Some flavors might still be used by load balancers and cannot be deleted.

4. Delete the old management network and its ports:

   $ for net_id in $(openstack network list -f value -c ID --name lb-mgmt-net); do \
       desc=$(openstack network show "$net_id" -f value -c description); \
       [ -z "$desc" ] && WALLABY_LB_MGMT_NET_ID="$net_id" ; \
     done
   $ for id in $(openstack port list --network "$WALLABY_LB_MGMT_NET_ID" -f value -c ID); do \
       openstack port delete "$id" ; \
     done
   $ openstack network delete "$WALLABY_LB_MGMT_NET_ID"

5. Verify that only one lb-mgmt-net and one lb-mgmt-subnet exists:

   $ openstack network list | grep lb-mgmt-net
   | fe470c29-0482-4809-9996-6d636e3feea3 | lb-mgmt-net          | 6a881091-097d-441c-937b-5a23f4f243b7 |
   $ openstack subnet list | grep lb-mgmt-subnet
   | 6a881091-097d-441c-937b-5a23f4f243b7 | lb-mgmt-subnet          | fe470c29-0482-4809-9996-6d636e3feea3 | 172.24.0.0/16   |