Chapter 5. Configuring Networker nodes

Procedure

1. For unprovisioned nodes, create the SSH key pair for Ansible:

   $ ssh-keygen -f <key_file_name> -N "" -t rsa -b 4096

   • Replace <key_file_name> with the name to use for the key pair.

2. Create the Secret CR for Ansible and apply it to the cluster:

   $ oc create secret generic dataplane-ansible-ssh-private-key-secret \
   --save-config \
   --dry-run=client \
   --from-file=ssh-privatekey=<key_file_name> \
   --from-file=ssh-publickey=<key_file_name>.pub \
   [--from-file=authorized_keys=<key_file_name>.pub] -n openstack \
   -o yaml | oc apply -f -

   • Replace <key_file_name> with the name and location of your SSH key pair file.
   • Optional: Only include the --from-file=authorized_keys option for bare-metal nodes that must be provisioned when creating the data plane.

3. If you are creating Compute nodes, create a secret for migration.

   1. Create the SSH key pair for instance migration:

      $ ssh-keygen -f ./nova-migration-ssh-key -t ecdsa-sha2-nistp521 -N ''

   2. Create the Secret CR for migration and apply it to the cluster:

      $ oc create secret generic nova-migration-ssh-key \
      --save-config \
      --from-file=ssh-privatekey=nova-migration-ssh-key \
      --from-file=ssh-publickey=nova-migration-ssh-key.pub \
      -n openstack \
      -o yaml | oc apply -f -

4. For nodes that have not been registered to the Red Hat Customer Portal, create the Secret CR for subscription-manager credentials to register the nodes:

   $ oc create secret generic subscription-manager \
   --from-literal rhc_auth='{"login": {"username": "<subscription_manager_username>", "password": "<subscription_manager_password>"}}'

   • Replace <subscription_manager_username> with the username you set for subscription-manager.
   • Replace <subscription_manager_password> with the password you set for subscription-manager.

5. Create a Secret CR that contains the Red Hat registry credentials:

   $ oc create secret generic redhat-registry --from-literal edpm_container_registry_logins='{"registry.redhat.io": {"<username>": "<password>"}}'

   • Replace <username> and <password> with your Red Hat registry username and password credentials.

     For information about how to create your registry service account, see the Knowledge Base article Creating Registry Service Accounts.

6. If you are creating Compute nodes, create a secret for libvirt.

   1. Create a file on your workstation named secret_libvirt.yaml to define the libvirt secret:

      apiVersion: v1
      kind: Secret
      metadata:
       name: libvirt-secret
       namespace: openstack
      type: Opaque
      data:
       LibvirtPassword: <base64_password>

      • Replace <base64_password> with a base64-encoded string with maximum length 63 characters. You can use the following command to generate a base64-encoded password:

        $ echo -n <password> | base64

        Tip

        If you do not want to base64-encode the username and password, you can use the stringData field instead of the data field to set the username and password.

   2. Create the Secret CR:

      $ oc apply -f secret_libvirt.yaml -n openstack

7. Verify that the Secret CRs are created:

   $ oc describe secret dataplane-ansible-ssh-private-key-secret
   $ oc describe secret nova-migration-ssh-key
   $ oc describe secret subscription-manager
   $ oc describe secret redhat-registry
   $ oc describe secret libvirt-secret

Procedure

1. Create a file on your workstation named openstack_preprovisioned_networker_node_set.yaml to define the OpenStackDataPlaneNodeSet CR:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneNodeSet
   metadata:
     name: networker-nodes
     namespace: openstack
   spec:
     env:
       - name: ANSIBLE_FORCE_COLOR
         value: "True"

   • name - The OpenStackDataPlaneNodeSet CR name must be unique, contain only lower case alphanumeric characters and - (hyphens) or . (periods), start and end with an alphanumeric character, and have a maximum length of 53 characters. If necessary, replace the example name networker-nodes with a name that more accurately describes your node set.
   • env - Optional: a list of environment variables to pass to the pod.

2. Include the services field to override the default services. Remove the nova, libvirt, and other services that are not required by a Networker node:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneNodeSet
   metadata:
     name: networker-nodes
     namespace: openstack
   spec:
   ...
     services:
      - redhat
      - bootstrap
      - download-cache
      - reboot-os
      - configure-ovs-dpdk
      - configure-network
      - validate-network
      - install-os
      - configure-os
      - ssh-known-hosts
      - run-os
      - install-certs
      - ovn
      - neutron-metadata
      - neutron-dhcp

   • configure-ovs-dpdk - The configure-ovs-dpdk service is required only when DPDK nics are used in the deployment.
   • neutron-metadata - The neutron-metadata service is required only when SR-IOV ports are used in the deployment.
   • neutron-dhcp - You can optionally run the neutron-dhcp service on your Networker nodes. You might not need to use neutron-dhcp with OVN if your deployment uses DHCP relays, or advanced DHCP options that are supported by dnsmasq but not by the OVN DHCP implementation. .

3. Connect the data plane to the control plane network:

   spec:
     ...
     networkAttachments:
       - ctlplane

4. Enable the chassis as gateway:

   spec:
   ...
     nodeTemplate:
       ansible:
       ...
       edpm_enable_chassis_gw: true

5. Specify that the nodes in this set are pre-provisioned:

   spec:
   ...
     nodeTemplate:
       ansible:
       ...
       edpm_enable_chassis_gw: true
       ...
      preProvisioned: true

6. Add the SSH key secret that you created so that Ansible can connect to the data plane nodes:

     nodeTemplate:
       ansibleSSHPrivateKeySecret: <secret-key>

   • Replace <secret-key> with the name of the SSH key Secret CR you created for this node set in <link>[Creating the data plane secrets], for example, dataplane-ansible-ssh-private-key-secret.
7. Create a Persistent Volume Claim (PVC) in the openstack namespace on your Red Hat OpenShift Container Platform (RHOCP) cluster to store logs. Set the volumeMode to Filesystem and accessModes to ReadWriteOnce. Do not request storage for logs from a PersistentVolume (PV) that uses the NFS volume plugin. NFS is incompatible with FIFO and the ansible-runner creates a FIFO file to store logs. For information about PVCs, see Understanding persistent storage in the RHOCP Storage guide and Red Hat OpenShift Container Platform cluster requirements in Planning your deployment.

8. Enable persistent logging for the Networker nodes:

     nodeTemplate:
       ...
       extraMounts:
         - extraVolType: Logs
           volumes:
           - name: ansible-logs
             persistentVolumeClaim:
               claimName: <pvc_name>
           mounts:
           - name: ansible-logs
             mountPath: "/runner/artifacts"

   • Replace <pvc_name> with the name of the PVC storage on your RHOCP cluster.

9. Specify the management network:

     nodeTemplate:
       ...
       managementNetwork: ctlplane

10. Specify the Secret CRs used to source the usernames and passwords to register the operating system of the nodes that are not registered to the Red Hat Customer Portal, and enable repositories for your nodes. The following example demonstrates how to register your nodes to Red Hat Content Delivery Network (CDN). For information about how to register your nodes with Red Hat Satellite 6.13, see Managing Hosts.

      nodeTemplate:
        ...
        ansible:
          ansibleUser: cloud-admin
          ansiblePort: 22
          ansibleVarsFrom:
            - secretRef:
                name: subscription-manager
            - secretRef:
                name: redhat-registry
          ansibleVars:
            rhc_release: 9.4
            rhc_repositories:
                - {name: "*", state: disabled}
                - {name: "rhel-9-for-x86_64-baseos-eus-rpms", state: enabled}
                - {name: "rhel-9-for-x86_64-appstream-eus-rpms", state: enabled}
                - {name: "rhel-9-for-x86_64-highavailability-eus-rpms", state: enabled}
                - {name: "fast-datapath-for-rhel-9-x86_64-rpms", state: enabled}
                - {name: "rhoso-18.0-for-rhel-9-x86_64-rpms", state: enabled}
                - {name: "rhceph-7-tools-for-rhel-9-x86_64-rpms", state: enabled}
            edpm_bootstrap_release_version_package: []

    • ansibleUser - The user associated with the secret you created in <link>[Creating the data plane secrets].

    • ansibleVars - The Ansible variables that customize the set of nodes. For a list of Ansible variables that you can use, see https://openstack-k8s-operators.github.io/edpm-ansible/.

      For a complete list of the Red Hat Customer Portal registration commands, see https://access.redhat.com/solutions/253273. For information about how to log into registry.redhat.io, see https://access.redhat.com/RegistryAuthentication#creating-registry-service-accounts-6.

11. Add the network configuration template to apply to your Networker nodes.

      nodeTemplate:
        ...
        ansible:
          ...
          ansibleVars:
          ...
      nodes:
    
           ...
            neutron_physical_bridge_name: br-ex
            neutron_public_interface_name: eth0
            edpm_network_config_nmstate: true
            edpm_network_config_update: false

    • edpm_network_config_nmstate - Sets the os-net-config provider to nmstate. The default value is true. Change it to false only if a specific limitation of the nmstate provider requires you to use the ifcfg provider. For more information on advantages and limitations of the nmstate provider, see https://docs.redhat.com/en/documentation/red_hat_openstack_services_on_openshift/18.0/html/planning_your_deployment/plan-networks_planning#plan-os-net-config_plan-network in Planning your deployment.

    • edpm_network_config_update - When deploying a node set for the first time, set the edpm_network_config_update variable to false. If you later modify edpm_network_config_template, first set edpm_network_config_update to true. After you complete the update, reset it to false.

      Important

      After an edpm_network_config_template update, you must reset edpm_network_config_update to false. Otherwise, the nodes could lose network access. Whenever edpm_network_config_update is true, the updated network configuration is reapplied every time an OpenStackDataPlaneDeployment CR is created that includes the configure-network service that is a member of the servicesOverride list.

      The following example applies a VLANs network configuration to a set of the data plane Networker nodes with DPDK:

              edpm_network_config_template: |
                ...
                {% set mtu_list = [ctlplane_mtu] %}
                {% for network in nodeset_networks %}
                {{ mtu_list.append(lookup('vars', networks_lower[network] ~ '_mtu')) }}
                {%- endfor %}
                {% set min_viable_mtu = mtu_list | max %}
                network_config:
                - type: ovs_user_bridge
                  name: {{ neutron_physical_bridge_name }}
                  mtu: {{ min_viable_mtu }}
                  use_dhcp: false
                  dns_servers: {{ ctlplane_dns_nameservers }}
                  domain: {{ dns_search_domains }}
                  addresses:
                  - ip_netmask: {{ ctlplane_ip }}/{{ ctlplane_cidr }}
                  routes: {{ ctlplane_host_routes }}
                  members:
                  - type: ovs_dpdk_port
                    name: dpdk0
                    members:
                    - type: interface
                      name: nic1
      
      
                - type: linux_bond
                  name: bond_api
                  use_dhcp: false
                  bonding_options: "mode=active-backup"
                  dns_servers: {{ ctlplane_dns_nameservers }}
                  members:
                  - type: interface
                    name: nic2
                    primary: true
      
      
                - type: vlan
                  vlan_id: {{ lookup('vars', networks_lower['internalapi'] ~ '_vlan_id') }}
                  device: bond_api
                  addresses:
                  - ip_netmask: {{ lookup('vars', networks_lower['internalapi'] ~ '_ip') }}/{{ lookup('vars', networks_lower['internalapi'] ~ '_cidr') }}
      
      
                - type: ovs_user_bridge
                  name: br-link0
                  use_dhcp: false
                  ovs_extra: "set port br-link0 tag={{ lookup('vars', networks_lower['tenant'] ~ '_vlan_id') }}"
                  addresses:
                  - ip_netmask: {{ lookup('vars', networks_lower['tenant'] ~ '_ip') }}/{{ lookup('vars', networks_lower['tenant'] ~ '_cidr')}}
                  members:
                  - type: ovs_dpdk_bond
                    name: dpdkbond0
                    mtu: 9000
                    rx_queue: 1
                    ovs_extra: "set port dpdkbond0 bond_mode=balance-slb"
                    members:
                    - type: ovs_dpdk_port
                      name: dpdk1
                      members:
                      - type: interface
                        name: nic3
                    - type: ovs_dpdk_port
                      name: dpdk2
                      members:
                      - type: interface
                        name: nic4
      
      
                - type: ovs_user_bridge
                  name: br-link1
                  use_dhcp: false
                  members:
                  - type: ovs_dpdk_bond
                    name: dpdkbond1
                    mtu: 9000
                    rx_queue: 1
                    ovs_extra: "set port dpdkbond1 bond_mode=balance-slb"
                    members:
                    - type: ovs_dpdk_port
                      name: dpdk3
                      members:
                      - type: interface
                        name: nic5
                    - type: ovs_dpdk_port
                      name: dpdk4
                      members:
                      - type: interface
                        name: nic6
              neutron_physical_bridge_name: br-ex

      The following example applies a VLANs network configuration to a set of data plane Networker nodes without DPDK:

      edpm_network_config_template: |
                …---
                {% set mtu_list = [ctlplane_mtu] %}
                {% for network in nodeset_networks %}
                {{ mtu_list.append(lookup('vars', networks_lower[network] ~ '_mtu')) }}
                {%- endfor %}
                {% set min_viable_mtu = mtu_list | max %}
                network_config:
                  - type: ovs_bridge
                    name: {{ neutron_physical_bridge_name }}
                    mtu: {{ min_viable_mtu }}
                    use_dhcp: false
                    dns_servers: {{ ctlplane_dns_nameservers }}
                    domain: {{ dns_search_domains }}
                    addresses:
                      - ip_netmask: {{ ctlplane_ip }}/{{ ctlplane_cidr }}
                    routes: {{ ctlplane_host_routes }}
                    members:
                      - type: interface
                        name: nic2
                        mtu: {{ min_viable_mtu }}
                        # force the MAC address of the bridge to this interface
                        primary: true
                {% for network in nodeset_networks %}
                      - type: vlan
                        mtu: {{ lookup('vars', networks_lower[network] ~ '_mtu') }}
                        vlan_id: {{ lookup('vars', networks_lower[network] ~ '_vlan_id') }}
                        addresses:
                          - ip_netmask: >-
                              {{
                                lookup('vars', networks_lower[network] ~ '_ip')
                              }}/{{
                                lookup('vars', networks_lower[network] ~ '_cidr')
                              }}
                        routes: {{ lookup('vars', networks_lower[network] ~ '_host_routes') }}
                {% endfor %}

      For more information about data plane network configuration, see Customizing data plane networks in Configuring network services.

12. Add the common configuration for the set of nodes in this group under the nodeTemplate section. Each node in this OpenStackDataPlaneNodeSet inherits this configuration. For information about the properties you can use to configure common node attributes, see OpenStackDataPlaneNodeSet CR spec properties in the Deploying Red Hat OpenStack Services on OpenShift guide.

13. Define each node in this node set:

    ...
      nodes:
        edpm-networker-0:
          hostName: networker-0
          networks:
          - name: ctlplane
            subnetName: subnet1
            defaultRoute: true
            fixedIP: 192.168.122.100
          - name: internalapi
            subnetName: subnet1
            fixedIP: 172.17.0.100
          - name: storage
            subnetName: subnet1
            fixedIP: 172.18.0.100
          - name: tenant
            subnetName: subnet1
            fixedIP: 172.19.0.100
          ansible:
            ansibleHost: 192.168.122.100
            ansibleUser: cloud-admin
            ansibleVars:
              fqdn_internal_api: edpm-networker-0.example.com
        edpm-networker-1:
          hostName: edpm-networker-1
          networks:
          - name: ctlplane
            subnetName: subnet1
            defaultRoute: true
            fixedIP: 192.168.122.101
          - name: internalapi
            subnetName: subnet1
            fixedIP: 172.17.0.101
          - name: storage
            subnetName: subnet1
            fixedIP: 172.18.0.101
          - name: tenant
            subnetName: subnet1
            fixedIP: 172.19.0.101
          ansible:
            ansibleHost: 192.168.122.101
            ansibleUser: cloud-admin
            ansibleVars:
              fqdn_internal_api: edpm-networker-1.example.com

    • edpm-networker-0 - The node definition reference, for example, edpm-networker-0. Each node in the node set must have a node definition.
    • networks - Defines the IPAM and the DNS records for the node.
    • fixedIP - Specifies a predictable IP address for the network that must be in the allocation range defined for the network in the NetConfig CR.

    • ansibleVars - Node-specific Ansible variables that customize the node.

      Note
      • Nodes defined within the nodes section can configure the same Ansible variables that are configured in the nodeTemplate section. Where an Ansible variable is configured for both a specific node and within the nodeTemplate section, the node-specific values override those from the nodeTemplate section.
      • You do not need to replicate all the nodeTemplate Ansible variables for a node to override the default and set some node-specific values. You only need to configure the Ansible variables you want to override for the node.
      • Many ansibleVars include edpm in the name, which stands for "External Data Plane Management".

      For information about the properties you can use to configure node attributes, see OpenStackDataPlaneNodeSet CR spec properties in the Deploying Red Hat OpenStack Services on OpenShift guide..

14. Save the openstack_preprovisioned_networker_node_set.yaml definition file.

15. Create the data plane resources:

    $ oc create --save-config -f openstack_preprovisioned_networker_node_set.yaml -n openstack

16. Verify that the data plane resources have been created by confirming that the status is SetupReady:

    $ oc wait openstackdataplanenodeset openstack-data-plane --for condition=SetupReady --timeout=10m

    When the status is SetupReady the command returns a condition met message, otherwise it returns a timeout error.

    For information about the data plane conditions and states, see Data plane conditions and states in Deploying Red Hat OpenStack Services on OpenShift.

17. Verify that the Secret resource was created for the node set:

    $ oc get secret | grep openstack-data-plane
    dataplanenodeset-openstack-data-plane Opaque 1 3m50s

18. Verify the services were created:

    $ oc get openstackdataplaneservice -n openstack
    NAME                AGE
    bootstrap           46m
    ceph-client         46m
    ceph-hci-pre        46m
    configure-network   46m
    configure-os        46m
    ...

1. Create a BareMetalHost custom resource (CR) for each bare-metal Networker node.
2. Define an OpenStackDataPlaneNodeSet CR for the Networker nodes.

Procedure

1. Create a file on your workstation named openstack_data_plane_deploy.yaml to define the OpenStackDataPlaneDeployment CR:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneDeployment
   metadata:
     name: data-plane-deploy
     namespace: openstack

   • name - The OpenStackDataPlaneDeployment CR name must be unique, must consist of lower case alphanumeric characters, - (hyphen) or . (period), and must start and end with an alphanumeric character. Update the name in this example to a name that reflects the node sets in the deployment.

2. Add all the OpenStackDataPlaneNodeSet CRs that you want to deploy:

   spec:
     nodeSets:
       - openstack-data-plane
       - <nodeSet_name>
       - ...
       - <nodeSet_name>

   • Replace <nodeSet_name> with the names of the OpenStackDataPlaneNodeSet CRs that you want to include in your data plane deployment.
3. Save the openstack_data_plane_deploy.yaml deployment file.

4. Deploy the data plane:

   $ oc create -f openstack_data_plane_deploy.yaml -n openstack

   You can view the Ansible logs while the deployment executes:

   $ oc get pod -l app=openstackansibleee -w
   $ oc logs -l app=openstackansibleee -f --max-log-requests 10

   If the oc logs command returns an error similar to the following error, increase the --max-log-requests value:

   error: you are attempting to follow 19 log streams, but maximum allowed concurrency is 10, use --max-log-requests to increase the limit

5. Verify that the data plane is deployed:

   $ oc get openstackdataplanedeployment -n openstack
   NAME             	STATUS   MESSAGE
   data-plane-deploy   True     Setup Complete
   
   
   $ oc get openstackdataplanenodeset -n openstack
   NAME             	STATUS   MESSAGE
   openstack-data-plane   True     NodeSet Ready

   For information about the meaning of the returned status, see Data plane conditions and states in Deploying Red Hat OpenStack Services on OpenShift

   If the status indicates that the data plane has not been deployed, then troubleshoot the deployment. For information, see Troubleshooting the data plane creation and deployment in the Deploying Red Hat OpenStack Services on OpenShift guide.