Chapter 5. Preparing networks for Red Hat OpenStack Services on OpenShift

Procedure

1. Create a manifest file named tuned.yaml with content similar to the following:

   apiVersion: tuned.openshift.io/v1
   kind: Tuned
   metadata:
     name: default
     namespace: openshift-cluster-node-tuning-operator
   spec:
     profile:
     - data: |
         [main]
         summary=Optimize systems running OpenShift (provider specific parent profile)
         include=-provider-${f:exec:cat:/var/lib/ocp-tuned/provider},openshift
   
         [sysctl]
         net.ipv4.conf.enp8s0.rp_filter=0
         net.ipv4.conf.enp9s0.rp_filter=0
       name: openshift
     recommend:
     - match:
       - label: kubernetes.io/hostname
         value: worker-0
       - label: kubernetes.io/hostname
         value: worker-1
       - label: kubernetes.io/hostname
         value: worker-2
       - label: node-role.kubernetes.io/master
       operand:
         tunedConfig:
           reapply_sysctl: false
       priority: 15
       profile: openshift-no-reapply-sysctl
   status: {}

   Copy to Clipboard Toggle word wrap

2. Save the file and create the Tuned resource:

   $ oc create -f tuned.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a NodeNetworkConfigurationPolicy (nncp) CR file on your workstation, for example, openstack-nncp-bgp.yaml.

2. Retrieve the names of the worker nodes in the RHOCP cluster:

   $ oc get nodes -l node-role.kubernetes.io/worker -o jsonpath="{.items[*].metadata.name}"

   Copy to Clipboard Toggle word wrap

3. Discover the network configuration:

   $ oc get nns/<worker_node> -o yaml | more

   Copy to Clipboard Toggle word wrap
   • Replace <worker_node> with the name of a worker node retrieved in step 2, for example, worker-1. Repeat this step for each worker node.

4. In the nncp CR file, configure the interfaces for each isolated network on each worker node in the RHOCP cluster. For information about the default physical data center networks that must be configured with network isolation, see Default Red Hat OpenStack Services on OpenShift networks for BGP.

   In the following example, the nncp CR configures the multiple unconnected bridges that you use to map the Red Hat OpenStack Services on OpenShift (RHOSO) networks. You use BGP interfaces to peer with the network fabric and establish connectivity. The loopback interface is configured with the BGP network source address, 99.99.0.x. You can optionally dedicate a NIC to the ctlplane network.

   apiVersion: nmstate.io/v1
   kind: NodeNetworkConfigurationPolicy
   metadata:
     labels:
       osp/nncm-config-type: standard
     name: worker-0
     namespace: openstack
   spec:
     desiredState:
       dns-resolver:
         config:
           search: []
           server:
           - 192.168.122.1
       interfaces:
       - description: internalapi bridge
         mtu: 1500
         name: internalapi
         state: up
         type: linux-bridge
       - description: storage bridge
         mtu: 1500
         name: storage
         state: up
         type: linux-bridge
       - description: tenant bridge
         mtu: 1500
         name: tenant
         state: up
         type: linux-bridge
       - description: ctlplane bridge
         mtu: 1500
         name: ospbr
         state: up
         type: linux-bridge
       - description: BGP interface 1
         ipv4:
           address:
           - ip: 100.64.0.14
             prefix-length: "30"
           dhcp: false
           enabled: true
         ipv6:
           enabled: false
         mtu: 1500
         name: enp8s0
         state: up
         type: ethernet
       - description: BGP interface 2
         ipv4:
           address:
           - ip: 100.65.0.14
             prefix-length: "30"
           dhcp: false
           enabled: true
         ipv6:
           enabled: false
         mtu: 1500
         name: enp9s0
         state: up
         type: ethernet
       - description: loopback interface
         ipv4:
           address:
           - ip: 99.99.0.3
             prefix-length: "32"
           dhcp: false
           enabled: true
         mtu: 65536
         name: lo
         state: up
       route-rules:
         config: []
       routes:
         config:
         - destination: 99.99.0.0/16
           next-hop-address: 100.64.0.13
           next-hop-interface: enp8s0
         - destination: 99.99.0.0/16
           next-hop-address: 100.65.0.13
           next-hop-interface: enp9s0
     nodeSelector:
       kubernetes.io/hostname: worker-0
       node-role.kubernetes.io/worker: ""

   Copy to Clipboard Toggle word wrap

5. Create the nncp CR in the cluster:

   $ oc apply -f openstack-nncp-bgp.yaml

   Copy to Clipboard Toggle word wrap

6. Verify that the nncp CR is created:

   $ oc get nncp -w
   NAME                        STATUS        REASON
   osp-enp6s0-worker-1   Progressing   ConfigurationProgressing
   osp-enp6s0-worker-1   Progressing   ConfigurationProgressing
   osp-enp6s0-worker-1   Available     SuccessfullyConfigured

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a NetworkAttachmentDefinition (net-attach-def) CR file on your workstation, for example, openstack-net-attach-def.yaml.

2. In the NetworkAttachmentDefinition CR file, configure a NetworkAttachmentDefinition resource for each isolated network to attach a service deployment pod to the network. The following example creates a NetworkAttachmentDefinition resource that uses a type bridge interface with specific gateway configurations and additional options:

   apiVersion: k8s.cni.cncf.io/v1
   kind: NetworkAttachmentDefinition
   metadata:
     labels:
       osp/net: internalapi
       osp/net-attach-def-type: standard
     name: internalapi
     namespace: openstack 

   1

   
   spec:
     config: |
   	{
     	"cniVersion": "0.3.1",
     	"name": "internalapi", 

   2

   
     	"type": "bridge",
     	"isDefaultGateway": true,
     	"isGateway": true,
     	"forceAddress": false,
     	"ipMasq": true, 

   3

   
     	"hairpinMode": true,
     	"bridge": "internalapi",
     	"ipam": { 

   4

   
         "type": "whereabouts",
         "range": "172.17.0.0/24",
         "range_start": "172.17.0.30", 

   5

   
         "range_end": "172.17.0.70",
         "gateway": "172.17.0.1"
       }
     }

   Copy to Clipboard Toggle word wrap

   1

   The namespace where the services are deployed.

   2

   The node interface name associated with the network, as defined in the nncp CR.

   3

   Optional: Set to true to enable IP masquerading. If the gateway does not have an IP address, ipMasq has no effect. The default value is false.

   4

   The whereabouts CNI IPAM plugin to assign IPs to the created pods from the range .30 - .70.

   5

   The IP address pool range must not overlap with the MetalLB IPAddressPool range and the NetConfig allocationRange.

   Note

   Set "ipMasq": true when the data plane nodes do not have the necessary routes, or when the data plane nodes do not have connection to the control plane network before Free Range Routing (FRR) is configured.

3. Create the NetworkAttachmentDefinition CR in the cluster:

   $ oc apply -f openstack-net-attach-def.yaml

   Copy to Clipboard Toggle word wrap

4. Verify that the NetworkAttachmentDefinition CR is created:

   $ oc get net-attach-def -n openstack

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a BGPPeer CR file on your workstation, for example, bgppeers.yaml.

2. Configure the pairs of BGPPeer CRs for each node to peer with. The following example configures two BGPPeer CRs for the worker-0 node, one for leaf-0 and one for leaf-1:

   apiVersion: metallb.io/v1beta2
   kind: BGPPeer
   metadata:
     name: bgp-peer-node-0-0
     namespace: metallb-system
   spec:
     myASN: 64999
     nodeSelectors:
     - matchExpressions:
   	- key: kubernetes.io/hostname
     	operator: In
     	values:
     	- worker-0
     password: r3dh4t1234
     peerASN: 64999
     peerAddress: 100.64.0.13
   ---
   apiVersion: metallb.io/v1beta2
   kind: BGPPeer
   metadata:
     name: bgp-peer-node-0-1
     namespace: metallb-system
   spec:
     myASN: 64999
     nodeSelectors:
     - matchExpressions:
   	- key: kubernetes.io/hostname
     	operator: In
     	values:
     	- worker-0
     password: r3dh4t1234
     peerASN: 64999
     peerAddress: 100.65.0.13

   Copy to Clipboard Toggle word wrap

3. Create the BGPPeer CRs:

   $ oc create -f bgppeers.yaml

   Copy to Clipboard Toggle word wrap
4. Create an IPAddressPool CR file on your workstation, for example, ipaddresspools-bgp.yaml.

5. In the IPAddressPool CR file, configure an IPAddressPool resource on each isolated network to specify the IP address ranges over which MetalLB has authority:

   apiVersion: metallb.io/v1beta1
   kind: IPAddressPool
   metadata:
     name: ctlplane
     namespace: metallb-system
   spec:
     addresses:
     - 192.168.125.80-192.168.125.90 

   1

   
   ---
   apiVersion: metallb.io/v1beta1
   kind: IPAddressPool
   metadata:
     name: internalapi
     namespace: metallb-system
   spec:
     addresses:
     - 172.17.0.80-172.17.0.90

   Copy to Clipboard Toggle word wrap

   1

   The IPAddressPool range must not overlap with the whereabouts IPAM range and the NetConfig allocationRange.

   For information about how to configure the other IPAddressPool resource parameters, see Configuring MetalLB BGP peers in RHOCP Networking overview.

6. Create the IPAddressPool CR in the cluster:

   $ oc apply -f ipaddresspools-bgp.yaml

   Copy to Clipboard Toggle word wrap

7. Verify that the IPAddressPool CR is created:

   $ oc describe -n metallb-system IPAddressPool

   Copy to Clipboard Toggle word wrap

8. Create a BGPAdvertisement CR file on your workstation, for example, bgpadvert.yaml.

   apiVersion: metallb.io/v1beta1
   kind: BGPAdvertisement
   metadata:
     name: bgpadvertisement
     namespace: metallb-system
   spec:
     ipAddressPools:
     - ctlplane
     - internalapi
     - storage
     - tenant
     peers: 

   1

   
     - bgp-peer-node-0-0
     - bgp-peer-node-0-1
     - bgp-peer-node-1-0
     - bgp-peer-node-1-1
     - bgp-peer-node-2-0
     - bgp-peer-node-2-1
     ...

   Copy to Clipboard Toggle word wrap

   1

   Lists all the BGPPeer CRs you defined for the peer IP addresses that each RHOCP node needs to communicate with.

9. Create the BGPAdvertisement CR in the cluster:

   $ oc apply -f bgpadvert.yaml

   Copy to Clipboard Toggle word wrap

10. If your cluster has OVNKubernetes as the network back end, then you must enable global forwarding so that MetalLB can work on a secondary network interface.

    1. Check the network back end used by your cluster:

       $ oc get network.operator cluster --output=jsonpath='{.spec.defaultNetwork.type}'

       Copy to Clipboard Toggle word wrap

    2. If the back end is OVNKubernetes, then run the following command to enable global IP forwarding:

       $ oc patch network.operator cluster -p '{"spec":{"defaultNetwork":{"ovnKubernetesConfig":{"gatewayConfig":{"ipForwarding": "Global"}}}}}' --type=merge

       Copy to Clipboard Toggle word wrap