Chapter 3. Preparing networks for Red Hat OpenStack Services on OpenShift

Procedure

1. Create a NodeNetworkConfigurationPolicy (nncp) CR file on your workstation, for example, openstack-nncp.yaml.

2. Retrieve the names of the worker nodes in the RHOCP cluster:

   $ oc get nodes -l node-role.kubernetes.io/worker -o jsonpath="{.items[*].metadata.name}"

   Copy to Clipboard Toggle word wrap

3. Discover the network configuration:

   $ oc get nns/<worker_node> -o yaml | more

   Copy to Clipboard Toggle word wrap
   • Replace <worker_node> with the name of a worker node retrieved in step 2, for example, worker-1. Repeat this step for each worker node.

4. In the nncp CR file, configure the interfaces for each isolated network on each worker node in the RHOCP cluster. For information about the default physical data center networks that must be configured with network isolation, see Default Red Hat OpenStack Services on OpenShift networks.

   In the following example, the nncp CR configures the enp6s0 interface for worker node 1, osp-enp6s0-worker-1, to use VLAN interfaces with IPv4 addresses for network isolation:

   apiVersion: nmstate.io/v1
   kind: NodeNetworkConfigurationPolicy
   metadata:
     name: osp-enp6s0-worker-1
   spec:
     desiredState:
       interfaces:
       - description: internalapi vlan interface
         ipv4:
           address:
           - ip: 172.17.0.10
             prefix-length: 24
           enabled: true
           dhcp: false
         ipv6:
           enabled: false
         name: internalapi
         state: up
         type: vlan
         vlan:
           base-iface: enp6s0
           id: 20
           reorder-headers: true
       - description: storage vlan interface
         ipv4:
           address:
           - ip: 172.18.0.10
             prefix-length: 24
           enabled: true
           dhcp: false
         ipv6:
           enabled: false
         name: storage
         state: up
         type: vlan
         vlan:
           base-iface: enp6s0
           id: 21
           reorder-headers: true
       - description: tenant vlan interface
         ipv4:
           address:
           - ip: 172.19.0.10
             prefix-length: 24
           enabled: true
           dhcp: false
         ipv6:
           enabled: false
         name: tenant
         state: up
         type: vlan
         vlan:
           base-iface: enp6s0
           id: 22
           reorder-headers: true
       - description: Configuring enp6s0
         ipv4:
           address:
           - ip: 192.168.122.10
             prefix-length: 24
           enabled: true
           dhcp: false
         ipv6:
           enabled: false
         mtu: 1500
         name: enp6s0
         state: up
         type: ethernet
       - description: octavia vlan interface
         name: octavia
         state: up
         type: vlan
         vlan:
           base-iface: enp6s0
           id: 24
           reorder-headers: true
       - bridge:
           options:
             stp:
               enabled: false
           port:
           - name: enp6s0.24
         description: Configuring bridge octbr
         mtu: 1500
         name: octbr
         state: up
         type: linux-bridge
       - description: designate vlan interface
         ipv4:
           address:
           - ip: 172.26.0.10
             prefix-length: "24"
           dhcp: false
           enabled: true
         ipv6:
           enabled: false
         mtu: 1500
         name: designate
         state: up
         type: vlan
         vlan:
           base-iface: enp7s0
           id: "25"
           reorder-headers: true
       - description: designate external vlan interface
         ipv4:
           address:
           - ip: 172.34.0.10
             prefix-length: "24"
           dhcp: false
           enabled: true
         ipv6:
           enabled: false
         mtu: 1500
         name: designateext
         state: up
         type: vlan
         vlan:
           base-iface: enp7s0
           id: "26"
           reorder-headers: true
     nodeSelector:
       kubernetes.io/hostname: worker-1
       node-role.kubernetes.io/worker: ""

   Copy to Clipboard Toggle word wrap

5. Create the nncp CR in the cluster:

   $ oc apply -f openstack-nncp.yaml

   Copy to Clipboard Toggle word wrap

6. Verify that the nncp CR is created:

   $ oc get nncp -w
   NAME                        STATUS        REASON
   osp-enp6s0-worker-1   Progressing   ConfigurationProgressing
   osp-enp6s0-worker-1   Progressing   ConfigurationProgressing
   osp-enp6s0-worker-1   Available     SuccessfullyConfigured

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a NetworkAttachmentDefinition (net-attach-def) CR file on your workstation, for example, openstack-net-attach-def.yaml.

2. In the NetworkAttachmentDefinition CR file, configure a NetworkAttachmentDefinition resource for each isolated network to attach a service deployment pod to the network. The following examples create a NetworkAttachmentDefinition resource for the following networks:

   • internalapi, storage, ctlplane, and tenant networks of type macvlan.
   • octavia, the load-balancing management network, of type bridge. This network attachment connects pods that manage load balancer virtual machines (amphorae) and the Open vSwitch pods that are managed by the OVN operator.
   • designate network used internally by the DNS service (designate) to manage the DNS servers.
   • designateext network used to provide external access to the DNS service resolver and the DNS servers.

   apiVersion: k8s.cni.cncf.io/v1
   kind: NetworkAttachmentDefinition
   metadata:
     name: internalapi
     namespace: openstack
   spec:
     config: |
       {
         "cniVersion": "0.3.1",
         "name": "internalapi",
         "type": "macvlan",
         "master": "internalapi",
         "ipam": {
           "type": "whereabouts",
           "range": "172.17.0.0/24",
           "range_start": "172.17.0.30",
           "range_end": "172.17.0.70"
         }
       }
   ---
   apiVersion: k8s.cni.cncf.io/v1
   kind: NetworkAttachmentDefinition
   metadata:
     name: ctlplane
     namespace: openstack
   spec:
     config: |
       {
         "cniVersion": "0.3.1",
         "name": "ctlplane",
         "type": "macvlan",
         "master": "enp6s0",
         "ipam": {
           "type": "whereabouts",
           "range": "192.168.122.0/24",
           "range_start": "192.168.122.30",
           "range_end": "192.168.122.70"
         }
       }
   ---
   apiVersion: k8s.cni.cncf.io/v1
   kind: NetworkAttachmentDefinition
   metadata:
     name: storage
     namespace: openstack
   spec:
     config: |
       {
         "cniVersion": "0.3.1",
         "name": "storage",
         "type": "macvlan",
         "master": "storage",
         "ipam": {
           "type": "whereabouts",
           "range": "172.18.0.0/24",
           "range_start": "172.18.0.30",
           "range_end": "172.18.0.70"
         }
       }
   ---
   apiVersion: k8s.cni.cncf.io/v1
   kind: NetworkAttachmentDefinition
   metadata:
     name: tenant
     namespace: openstack
   spec:
     config: |
       {
         "cniVersion": "0.3.1",
         "name": "tenant",
         "type": "macvlan",
         "master": "tenant",
         "ipam": {
           "type": "whereabouts",
           "range": "172.19.0.0/24",
           "range_start": "172.19.0.30",
           "range_end": "172.19.0.70"
         }
       }
   ---
   apiVersion: k8s.cni.cncf.io/v1
   kind: NetworkAttachmentDefinition
   metadata:
     labels:
       osp/net: octavia
     name: octavia
     namespace: openstack
   spec:
     config: |
       {
         "cniVersion": "0.3.1",
         "name": "octavia",
         "type": "bridge",
         "bridge": "octbr",
         "ipam": {
           "type": "whereabouts",
           "range": "172.23.0.0/24",
           "range_start": "172.23.0.30",
           "range_end": "172.23.0.70",
           "routes": [
              {
                "dst": "172.24.0.0/16",
                "gw" : "172.23.0.150"
              }
            ]
         }
       }
   ---
   apiVersion: k8s.cni.cncf.io/v1
   kind: NetworkAttachmentDefinition
   metadata:
     name: designate
     namespace: openstack
   spec:
     config: |
       {
         "cniVersion": "0.3.1",
         "name": "designate",
         "type": "macvlan",
         "master": "designate",
         "ipam": {
           "type": "whereabouts",
           "range": "172.26.0.0/16",
           "range_start": "172.26.0.30",
           "range_end": "172.26.0.70",
         }
       }
   ---
   apiVersion: k8s.cni.cncf.io/v1
   kind: NetworkAttachmentDefinition
   metadata:
     name: designateext
     namespace: openstack
   spec:
     config: |
       {
         "cniVersion": "0.3.1",
         "name": "designateext",
         "type": "macvlan",
         "master": "designateext",
         "ipam": {
           "type": "whereabouts",
           "range": "172.34.0.0/16",
           "range_start": "172.34.0.30",
           "range_end": "172.34.0.70",
         }
       }

   Copy to Clipboard Toggle word wrap
   • metadata.namespace: The namespace where the services are deployed.
   • "master": The node interface name associated with the network, as defined in the nncp CR.
   • "ipam": The whereabouts CNI IPAM plug-in assigns IPs to the created pods from the range .30 - .70.
   • "range_start" - "range_end": The IP address pool range must not overlap with the MetalLB IPAddressPool range and the NetConfig allocationRange.

3. Create the NetworkAttachmentDefinition CR in the cluster:

   $ oc apply -f openstack-net-attach-def.yaml

   Copy to Clipboard Toggle word wrap

4. Verify that the NetworkAttachmentDefinition CR is created:

   $ oc get net-attach-def -n openstack

   Copy to Clipboard Toggle word wrap

Procedure

1. Create an IPAddressPool CR file on your workstation, for example, openstack-ipaddresspools.yaml.

2. In the IPAddressPool CR file, configure an IPAddressPool resource on the isolated network to specify the IP address ranges over which MetalLB has authority:

   apiVersion: metallb.io/v1beta1
   kind: IPAddressPool
   metadata:
     namespace: metallb-system
     name: ctlplane
   spec:
     addresses:
       - 192.168.122.80-192.168.122.90
     autoAssign: true
     avoidBuggyIPs: false
   ---
   apiVersion: metallb.io/v1beta1
   kind: IPAddressPool
   metadata:
     name: internalapi
     namespace: metallb-system
   spec:
     addresses:
       - 172.17.0.80-172.17.0.90
     autoAssign: true
     avoidBuggyIPs: false
   ---
   apiVersion: metallb.io/v1beta1
   kind: IPAddressPool
   metadata:
     namespace: metallb-system
     name: storage
   spec:
     addresses:
       - 172.18.0.80-172.18.0.90
     autoAssign: true
     avoidBuggyIPs: false
   ---
   apiVersion: metallb.io/v1beta1
   kind: IPAddressPool
   metadata:
     namespace: metallb-system
     name: tenant
   spec:
     addresses:
       - 172.19.0.80-172.19.0.90
     autoAssign: true
     avoidBuggyIPs: false
   ---
   apiVersion: metallb.io/v1beta1
   kind: IPAddressPool
   metadata:
     namespace: metallb-system
     name: designateext
   spec:
     addresses:
       - 172.34.0.80-172.34.0.120
     autoAssign: true
     avoidBuggyIPs: false
   ---

   Copy to Clipboard Toggle word wrap
   • spec.addresses: The IPAddressPool range must not overlap with the whereabouts IPAM range and the NetConfig allocationRange.

   For information about how to configure the other IPAddressPool resource parameters, see Configuring MetalLB address pools in the RHOCP Networking guide.

3. Create the IPAddressPool CR in the cluster:

   $ oc apply -f openstack-ipaddresspools.yaml

   Copy to Clipboard Toggle word wrap

4. Verify that the IPAddressPool CR is created:

   $ oc describe -n metallb-system IPAddressPool

   Copy to Clipboard Toggle word wrap
5. Create a L2Advertisement CR file on your workstation, for example, openstack-l2advertisement.yaml.

6. In the L2Advertisement CR file, configure L2Advertisement CRs to define which node advertises a service to the local network. Create one L2Advertisement resource for each network.

   In the following example, each L2Advertisement CR specifies that the VIPs requested from the network address pools are announced on the interface that is attached to the VLAN:

   apiVersion: metallb.io/v1beta1
   kind: L2Advertisement
   metadata:
     name: ctlplane
     namespace: metallb-system
   spec:
     ipAddressPools:
     - ctlplane
     interfaces:
     - enp6s0
     nodeSelectors:
     - matchLabels:
         node-role.kubernetes.io/worker: ""
   ---
   apiVersion: metallb.io/v1beta1
   kind: L2Advertisement
   metadata:
     name: internalapi
     namespace: metallb-system
   spec:
     ipAddressPools:
     - internalapi
     interfaces:
     - internalapi
     nodeSelectors:
     - matchLabels:
         node-role.kubernetes.io/worker: ""
   ---
   apiVersion: metallb.io/v1beta1
   kind: L2Advertisement
   metadata:
     name: storage
     namespace: metallb-system
   spec:
     ipAddressPools:
     - storage
     interfaces:
     - storage
     nodeSelectors:
     - matchLabels:
         node-role.kubernetes.io/worker: ""
   ---
   apiVersion: metallb.io/v1beta1
   kind: L2Advertisement
   metadata:
     name: tenant
     namespace: metallb-system
   spec:
     ipAddressPools:
     - tenant
     interfaces:
     - tenant
     nodeSelectors:
     - matchLabels:
         node-role.kubernetes.io/worker: ""
   ---
   apiVersion: metallb.io/v1beta1
   kind: L2Advertisement
   metadata:
     name: designateext
     namespace: metallb-system
   spec:
     ipAddressPools:
     - designateext
     interfaces:
     - designateext
     nodeSelectors:
     - matchLabels:
         node-role.kubernetes.io/worker: ""

   Copy to Clipboard Toggle word wrap
   • spec.interfaces: The interface where the VIPs requested from the VLAN address pool are announced.

   For information about how to configure the other L2Advertisement resource parameters, see Configuring MetalLB with a L2 advertisement and label in the RHOCP Networking guide.

7. Create the L2Advertisement CRs in the cluster:

   $ oc apply -f openstack-l2advertisement.yaml

   Copy to Clipboard Toggle word wrap

8. Verify that the L2Advertisement CRs are created:

   $ oc get -n metallb-system L2Advertisement
   NAME          IPADDRESSPOOLS    IPADDRESSPOOL SELECTORS   INTERFACES
   ctlplane      ["ctlplane"]                                ["enp6s0"]
   designateext  ["designateext"]                            ["designateext"]
   internalapi   ["internalapi"]                             ["internalapi"]
   storage       ["storage"]                                 ["storage"]
   tenant        ["tenant"]                                  ["tenant"]

   Copy to Clipboard Toggle word wrap

9. If your cluster has OVNKubernetes as the network back end, then you must enable global forwarding so that MetalLB can work on a secondary network interface.

   1. Check the network back end used by your cluster:

      $ oc get network.operator cluster --output=jsonpath='{.spec.defaultNetwork.type}'

      Copy to Clipboard Toggle word wrap

   2. If the back end is OVNKubernetes, then run the following command to enable global IP forwarding:

      $ oc patch network.operator cluster -p '{"spec":{"defaultNetwork":{"ovnKubernetesConfig":{"gatewayConfig":{"ipForwarding": "Global"}}}}}' --type=merge

      Copy to Clipboard Toggle word wrap