Chapter 5. Adopting the data plane

Prerequisites

• Define the following shell variables. The values that are used are examples and refer to a single node standalone director deployment. Replace these example values with values that are correct for your environment:

  EDPM_PRIVATEKEY_PATH="<path to SSH key>"
  declare -A computes
  computes=(
    ["standalone.localdomain"]="192.168.122.100"
    # ...
  )

  • Replace ["standalone.localdomain"]="192.168.122.100" with the name of the Compute node and its IP address.

  • These ssh variables with the ssh commands are used instead of ansible to create instructions that are independent of where they are running. But ansible commands could be used to achieve the same result if you are in the right host, for example to stop a service:

    . stackrc
    ansible -i $(which tripleo-ansible-inventory) Compute -m shell -a "sudo systemctl stop tripleo_virtqemud.service" -b

Procedure

• Run the following script to remove the conflicting repositories and packages (in case of a devsetup that uses Standalone director) from all Compute hosts. That is required to install libvirt packages, when these hosts become adopted as data plane nodes, where modular libvirt daemons are no longer running in podman containers:

  PacemakerResourcesToStop=(
                  "galera-bundle"
                  "haproxy-bundle"
                  "rabbitmq-bundle")
  
  echo "Stopping pacemaker services"
  for i in {1..3}; do
      SSH_CMD=CONTROLLER${i}_SSH
      if [ ! -z "${!SSH_CMD}" ]; then
          echo "Using controller $i to run pacemaker commands"
          for resource in ${PacemakerResourcesToStop[*]}; do
              if ${!SSH_CMD} sudo pcs resource config $resource; then
                  ${!SSH_CMD} sudo pcs resource disable $resource
              fi
          done
          break
      fi
  done

Prerequisites

• Remaining source cloud Stopping infrastructure management and Compute services on Compute hosts.
• Ceph backend for Nova/Libvirt is configured Configuring a Ceph backend.
• Make sure the IPAM is configured

• When neutron-sriov-nic-agent is running on the existing Compute nodes, check the physical device mappings and ensure that they match the values that are defined in the OpenStackDataPlaneNodeSet custom resource (CR). For more information, see Pulling the configuration from a director deployment.
• Define the shell variables necessary to run the script that runs the fast-forward upgrade. Omit setting CEPH_FSID, if the local storage backend is going to be configured by Nova for Libvirt. The storage backend cannot be changed during adoption, and must match the one used on the source cloud:

• Replace ["standalone.localdomain"]="192.168.122.100" with the name of the Compute node and its IP address.

• Or, create a nova-compute-extra-config service (with Ceph backend for libvirt):

  oc apply -f - <<EOF
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: nova-extra-config
    namespace: openstack
  data:
    19-nova-compute-cell1-workarounds.conf: |
      [workarounds]
      disable_compute_service_check_for_ffu=true
    03-ceph-nova.conf: |
      [libvirt]
      images_type=rbd
      images_rbd_pool=vms
      images_rbd_ceph_conf=/etc/ceph/ceph.conf
      images_rbd_glance_store_name=default_backend
      images_rbd_glance_copy_poll_interval=15
      images_rbd_glance_copy_timeout=600
      rbd_user=openstack
      rbd_secret_uuid=$CEPH_FSID
  EOF

  That service removes pre-FFU workarounds and configures Compute services for Ceph storage backend. Provided above resources should contain a cell-specific configurations. For multi-cell, config maps and Red Hat OpenStack Platform data plane services should be named like nova-custom-ceph-cellX and nova-compute-extraconfig-cellX.

  1. Create a secret for the subscription manager and a secret for the Red Hat registry:

     oc apply -f - <<EOF
     apiVersion: v1
     kind: Secret
     metadata:
       name: subscription-manager
     data:
       username: <base64 encoded subscription-manager username>
       password: <base64 encoded subscription-manager password>
     ---
     apiVersion: v1
     kind: Secret
     metadata:
       name: redhat-registry
     data:
       username: <base64 encoded registry username>
       password: <base64 encoded registry password>
     EOF

  2. Deploy the OpenStackDataPlaneNodeSet CR:
  3. If TLS Everywhere is enabled, change spec:tlsEnabled to true

  4. If using a custom DNS Domain, modify the spec:nodes:[NODE NAME]:hostName to use fqdn for the node

     oc apply -f - <<EOF
     apiVersion: dataplane.openstack.org/v1beta1
     kind: OpenStackDataPlaneNodeSet
     metadata:
       name: openstack
     spec:
       tlsEnabled: false
       networkAttachments:
           - ctlplane
       preProvisioned: true
       services:
         - bootstrap
         - download-cache
         - configure-network
         - validate-network
         - install-os
         - configure-os
         - ssh-known-hosts
         - run-os
         - install-certs
         - libvirt
         - nova
         - ovn
         - neutron-metadata
       env:
         - name: ANSIBLE_CALLBACKS_ENABLED
           value: "profile_tasks"
         - name: ANSIBLE_FORCE_COLOR
           value: "True"
       nodes:
         standalone:
           hostName: standalone
           ansible:
             ansibleHost: ${computes[standalone.localdomain]}
           networks:
           - defaultRoute: true
             fixedIP: ${computes[standalone.localdomain]}
             name: ctlplane
             subnetName: subnet1
           - name: internalapi
             subnetName: subnet1
           - name: storage
             subnetName: subnet1
           - name: tenant
             subnetName: subnet1
       nodeTemplate:
         ansibleSSHPrivateKeySecret: dataplane-adoption-secret
         ansible:
           ansibleUser: root
           ansibleVarsFrom:
           - prefix: subscription_manager_
             secretRef:
               name: subscription-manager
           - prefix: registry_
             secretRef:
               name: redhat-registry
           ansibleVars:
             edpm_bootstrap_release_version_package: []
             service_net_map:
               nova_api_network: internalapi
               nova_libvirt_network: internalapi
     
             # edpm_network_config
             # Default nic config template for a EDPM compute node
             # These vars are edpm_network_config role vars
             edpm_network_config_template: |
                ---
                {% set mtu_list = [ctlplane_mtu] %}
                {% for network in nodeset_networks %}
                {{ mtu_list.append(lookup('vars', networks_lower[network] ~ '_mtu')) }}
                {%- endfor %}
                {% set min_viable_mtu = mtu_list | max %}
                network_config:
                - type: ovs_bridge
                  name: {{ neutron_physical_bridge_name }}
                  mtu: {{ min_viable_mtu }}
                  use_dhcp: false
                  dns_servers: {{ ctlplane_dns_nameservers }}
                  domain: {{ dns_search_domains }}
                  addresses:
                  - ip_netmask: {{ ctlplane_ip }}/{{ ctlplane_cidr }}
                  routes: {{ ctlplane_host_routes }}
                  members:
                  - type: interface
                    name: nic1
                    mtu: {{ min_viable_mtu }}
                    # force the MAC address of the bridge to this interface
                    primary: true
                {% for network in nodeset_networks %}
                  - type: vlan
                    mtu: {{ lookup('vars', networks_lower[network] ~ '_mtu') }}
                    vlan_id: {{ lookup('vars', networks_lower[network] ~ '_vlan_id') }}
                    addresses:
                    - ip_netmask:
                        {{ lookup('vars', networks_lower[network] ~ '_ip') }}/{{ lookup('vars', networks_lower[network] ~ '_cidr') }}
                    routes: {{ lookup('vars', networks_lower[network] ~ '_host_routes') }}
                {% endfor %}
     
             edpm_network_config_hide_sensitive_logs: false
             #
             # These vars are for the network config templates themselves and are
             # considered EDPM network defaults.
             neutron_physical_bridge_name: br-ctlplane
             neutron_public_interface_name: eth0
     
             # edpm_nodes_validation
             edpm_nodes_validation_validate_controllers_icmp: false
             edpm_nodes_validation_validate_gateway_icmp: false
     
             # edpm ovn-controller configuration
             edpm_ovn_bridge_mappings: <bridge_mappings>
             edpm_ovn_bridge: br-int
             edpm_ovn_encap_type: geneve
             ovn_match_northd_version: false
             ovn_monitor_all: true
             edpm_ovn_remote_probe_interval: 60000
             edpm_ovn_ofctrl_wait_before_clear: 8000
     
             timesync_ntp_servers:
             - hostname: clock.redhat.com
             - hostname: clock2.redhat.com
     
             edpm_bootstrap_command: |
               subscription-manager register --username {{ subscription_manager_username }} --password {{ subscription_manager_password }}
               subscription-manager release --set=9.2
               subscription-manager repos --disable=*
               subscription-manager repos --enable=rhel-9-for-x86_64-baseos-eus-rpms --enable=rhel-9-for-x86_64-appstream-eus-rpms --enable=rhel-9-for-x86_64-highavailability-eus-rpms --enable=openstack-17.1-for-rhel-9-x86_64-rpms --enable=fast-datapath-for-rhel-9-x86_64-rpms --enable=openstack-dev-preview-for-rhel-9-x86_64-rpms
               # FIXME: perform dnf upgrade for other packages in EDPM ansible
               # here we only ensuring that decontainerized libvirt can start
               dnf -y upgrade openstack-selinux
               rm -f /run/virtlogd.pid
               podman login -u {{ registry_username }} -p {{ registry_password }} registry.redhat.io
     
             gather_facts: false
             enable_debug: false
             # edpm firewall, change the allowed CIDR if needed
             edpm_sshd_configure_firewall: true
             edpm_sshd_allowed_ranges: ['192.168.122.0/24']
             # SELinux module
             edpm_selinux_mode: enforcing
     
             # Do not attempt OVS major upgrades here
             edpm_ovs_packages:
             - openvswitch3.1
     EOF

• Prepare adopted EDPM workloads to use Ceph backend for Block Storage service (cinder), if configured so

  oc patch osdpns/openstack --type=merge --patch "
  spec:
    services:
      - repo-setup
      - download-cache
      - bootstrap
      - configure-network
      - validate-network
      - install-os
      - configure-os
      - run-os
      - install-certs
      - ceph-client
      - libvirt
      - nova
      - ovn
      - neutron-metadata
    nodeTemplate:
      extraMounts:
      - extraVolType: Ceph
        volumes:
        - name: ceph
          secret:
            secretName: ceph-conf-files
        mounts:
        - name: ceph
          mountPath: "/etc/ceph"
          readOnly: true
  "

• Replace <bridge_mappings> with the value of the bridge mappings in your configuration, for example, "datacentre:br-ctlplane".

  1. Ensure that the ovn-controller settings that are configured in the OpenStackDataPlaneNodeSet CR are the same as were set in the Compute nodes before adoption. This configuration is stored in the external_ids` column in the Open_vSwitch table in the Open vSwitch database:

     ovs-vsctl list Open .
     ...
     external_ids        : {hostname=standalone.localdomain, ovn-bridge=br-int, ovn-bridge-mappings=<bridge_mappings>, ovn-chassis-mac-mappings="datacentre:1e:0a:bb:e6:7c:ad", ovn-encap-ip="172.19.0.100", ovn-encap-tos="0", ovn-encap-type=geneve, ovn-match-northd-version=False, ovn-monitor-all=True, ovn-ofctrl-wait-before-clear="8000", ovn-openflow-probe-interval="60", ovn-remote="tcp:ovsdbserver-sb.openstack.svc:6642", ovn-remote-probe-interval="60000", rundir="/var/run/openvswitch", system-id="2eec68e6-aa21-4c95-a868-31aeafc11736"}
     ...