Chapter 4. Preparing networks for Red Hat OpenStack Services on OpenShift with distributed zones

Procedure

1. Create a manifest file named tuned.yaml with content similar to the following:

   apiVersion: tuned.openshift.io/v1
   kind: Tuned
   metadata:
     name: default
     namespace: openshift-cluster-node-tuning-operator
   spec:
     profile:
     - data: |
         [main]
         summary=Optimize systems running OpenShift (provider specific parent profile)
         include=-provider-${f:exec:cat:/var/lib/ocp-tuned/provider},openshift
   
         [sysctl]
         net.ipv4.conf.enp8s0.rp_filter=0
         net.ipv4.conf.enp9s0.rp_filter=0
       name: openshift
     recommend:
     - match:
       - label: kubernetes.io/hostname
         value: worker-0
       - label: kubernetes.io/hostname
         value: worker-1
       - label: kubernetes.io/hostname
         value: worker-2
       - label: node-role.kubernetes.io/master
       operand:
         tunedConfig:
           reapply_sysctl: false
       priority: 15
       profile: openshift-no-reapply-sysctl
   status: {}

2. Save the file and create the Tuned resource:

   $ oc create -f tuned.yaml

Procedure

1. Create a NodeNetworkConfigurationPolicy (nncp) CR file on your workstation, for example, openstack-nncp-bgp.yaml.

2. Retrieve the names of the worker nodes in the RHOCP cluster:

   $ oc get nodes -l node-role.kubernetes.io/worker -o jsonpath="{.items[*].metadata.name}"

3. Discover the network configuration:

   $ oc get nns/<worker_node> -o yaml | more

   • Replace <worker_node> with the name of a worker node retrieved in step 2, for example, worker-1. Repeat this step for each worker node.

4. In the nncp CR file, configure the interfaces for each isolated network on each worker node in the RHOCP cluster.

   In the following example, the nncp CR configures the multiple unconnected bridges that you use to map the Red Hat OpenStack Services on OpenShift (RHOSO) networks. You use BGP interfaces to peer with the network fabric and establish connectivity. The loopback interface is configured with the BGP network source address, 99.99.0.x. You can optionally dedicate a NIC to the ctlplane network.

   apiVersion: nmstate.io/v1
   kind: NodeNetworkConfigurationPolicy
   metadata:
     labels:
       osp/nncm-config-type: standard
     name: worker-0
     namespace: openstack
   spec:
     desiredState:
       dns-resolver:
         config:
           search: []
           server:
           - 192.168.122.1
       interfaces:
       - description: internalapi bridge
         mtu: 1500
         name: internalapi
         state: up
         type: linux-bridge
       - description: storage bridge
         mtu: 1500
         name: storage
         state: up
         type: linux-bridge
       - description: ctlplane bridge
         mtu: 1500
         name: ospbr
         state: up
         type: linux-bridge
       - description: BGP interface 1
         ipv4:
           address:
           - ip: 100.64.0.14
             prefix-length: "30"
           dhcp: false
           enabled: true
         ipv6:
           enabled: false
         mtu: 1500
         name: enp8s0
         state: up
         type: ethernet
       - description: BGP interface 2
         ipv4:
           address:
           - ip: 100.65.0.14
             prefix-length: "30"
           dhcp: false
           enabled: true
         ipv6:
           enabled: false
         mtu: 1500
         name: enp9s0
         state: up
         type: ethernet
       - description: loopback interface
         ipv4:
           address:
           - ip: 99.99.0.3
             prefix-length: "32"
           dhcp: false
           enabled: true
         mtu: 65536
         name: lo
         state: up
       route-rules:
         config: []
       routes:
         config:
         - destination: 99.99.0.0/16
           next-hop-address: 100.64.0.13
           next-hop-interface: enp8s0
           weight: 200
         - destination: 99.99.0.0/16
           next-hop-address: 100.65.0.13
           next-hop-interface: enp9s0
           weight: 200
     nodeSelector:
       kubernetes.io/hostname: worker-0
       node-role.kubernetes.io/worker: ""

5. Create the nncp CR in the cluster:

   $ oc apply -f openstack-nncp-bgp.yaml

6. Verify that the nncp CR is created:

   $ oc get nncp -w

   Sample output

   NAME       STATUS        REASON
   worker-0   Progressing   ConfigurationProgressing
   worker-0   Progressing   ConfigurationProgressing
   worker-0   Available     SuccessfullyConfigured

Procedure

1. Create a NetworkAttachmentDefinition (net-attach-def) CR file on your workstation, for example, openstack-net-attach-def.yaml.

2. In the NetworkAttachmentDefinition CR file, configure a NetworkAttachmentDefinition resource for each isolated network to attach a service deployment pod to the network. The following example creates a NetworkAttachmentDefinition resource that uses a type bridge interface with specific gateway configurations and additional options:

   apiVersion: k8s.cni.cncf.io/v1
   kind: NetworkAttachmentDefinition
   metadata:
     labels:
       osp/net: internalapi
       osp/net-attach-def-type: standard
     name: internalapi
     namespace: openstack
   spec:
     config: |
   	{
     	"cniVersion": "0.3.1",
     	"name": "internalapi",
     	"type": "bridge",
     	"isDefaultGateway": true,
     	"isGateway": true,
     	"forceAddress": false,
     	"ipMasq": true,
     	"hairpinMode": true,
     	"bridge": "internalapi",
     	"ipam": {
         "type": "whereabouts",
         "range": "172.17.0.0/24",
         "range_start": "172.17.0.30",
         "range_end": "172.17.0.70",
         "gateway": "172.17.0.1"
       }
     }

   • metadata.namespace: The namespace where the services are deployed.
   • "name": The node interface name associated with the network, as defined in the nncp CR.
   • "ipMasq": An optional field that, when set to true, enables IP masquerading. If the gateway does not have an IP address, ipMasq has no effect. The default value is false. Set "ipMasq": true when the data plane nodes do not have the necessary routes, or when the data plane nodes do not have connection to the control plane network before Free Range Routing (FRR) is configured.
   • "ipam": The whereabouts CNI IPAM plug-in assigns IPs to the created pods from the range .30 - .70.
   • "range_start" - "range_end": The IP address pool range must not overlap with the MetalLB IPAddressPool range and the NetConfig allocationRange.

3. Create the NetworkAttachmentDefinition CR in the cluster:

   $ oc apply -f openstack-net-attach-def.yaml

4. Verify that the NetworkAttachmentDefinition CR is created:

   $ oc get net-attach-def -n openstack

Procedure

1. Create a BGPPeer CR file on your workstation, for example, bgppeers.yaml.

2. Configure the pairs of BGPPeer CRs for each node to peer with. The following example configures two BGPPeer CRs for the worker-0 node, one for leaf-0 and one for leaf-1:

   apiVersion: metallb.io/v1beta2
   kind: BGPPeer
   metadata:
     name: bgp-peer-node-0-0
     namespace: metallb-system
   spec:
     myASN: 64999
     nodeSelectors:
     - matchExpressions:
   	- key: kubernetes.io/hostname
     	operator: In
     	values:
     	- worker-0
     password: r3dh4t1234
     peerASN: 64999
     peerAddress: 100.64.0.13
   ---
   apiVersion: metallb.io/v1beta2
   kind: BGPPeer
   metadata:
     name: bgp-peer-node-0-1
     namespace: metallb-system
   spec:
     myASN: 64999
     nodeSelectors:
     - matchExpressions:
   	- key: kubernetes.io/hostname
     	operator: In
     	values:
     	- worker-0
     password: r3dh4t1234
     peerASN: 64999
     peerAddress: 100.65.0.13

3. Create the BGPPeer CRs:

   $ oc create -f bgppeers.yaml

4. Create an IPAddressPool CR file on your workstation, for example, ipaddresspools-bgp.yaml.

5. In the IPAddressPool CR file, configure an IPAddressPool resource on each isolated network to specify the IP address ranges over which MetalLB has authority:

   apiVersion: metallb.io/v1beta1
   kind: IPAddressPool
   metadata:
     name: ctlplane
     namespace: metallb-system
   spec:
     addresses:
     - 192.168.125.80-192.168.125.90
   ---
   apiVersion: metallb.io/v1beta1
   kind: IPAddressPool
   metadata:
     name: internalapi
     namespace: metallb-system
   spec:
     addresses:
     - 172.17.0.80-172.17.0.90

   • spec.addresses: The IPAddressPool range must not overlap with the whereabouts IPAM range and the NetConfig allocationRange.

   For information about how to configure the other IPAddressPool resource parameters, see Configuring MetalLB BGP peers in RHOCP Networking overview.

6. Create the IPAddressPool CR in the cluster:

   $ oc apply -f ipaddresspools-bgp.yaml

7. Verify that the IPAddressPool CR is created:

   $ oc describe -n metallb-system IPAddressPool

8. Create a BGPAdvertisement CR file on your workstation, for example, bgpadvert.yaml.

   apiVersion: metallb.io/v1beta1
   kind: BGPAdvertisement
   metadata:
     name: bgpadvertisement
     namespace: metallb-system
   spec:
     ipAddressPools:
     - ctlplane
     - internalapi
     - storage
     peers:
     - bgp-peer-node-0-0
     - bgp-peer-node-0-1
     - bgp-peer-node-1-0
     - bgp-peer-node-1-1
     - bgp-peer-node-2-0
     - bgp-peer-node-2-1
     ...

   • peers: Lists all the BGPPeer CRs you defined for the peer IP addresses that each RHOCP node needs to communicate with.

9. Create the BGPAdvertisement CR in the cluster:

   $ oc apply -f bgpadvert.yaml

10. If your cluster has OVNKubernetes as the network back end, then you must enable global forwarding so that MetalLB can work on a secondary network interface.

    1. Check the network back end used by your cluster:

       $ oc get network.operator cluster --output=jsonpath='{.spec.defaultNetwork.type}'

    2. If the back end is OVNKubernetes, then run the following command to enable global IP forwarding:

       $ oc patch network.operator cluster -p '{"spec":{"defaultNetwork":{"ovnKubernetesConfig":{"gatewayConfig":{"ipForwarding": "Global"}}}}}' --type=merge