Chapter 23. OVN-Kubernetes default CNI network provider

1. Egress firewall is also known as egress network policy in OpenShift SDN. This is not the same as network policy egress.
2. Egress router for OVN-Kubernetes supports only redirect mode.
3. IPv6 is supported only on bare metal clusters.
4. IPv6 single stack does not support Kubernetes NMState.

Procedure

1. To backup the configuration for the cluster network, enter the following command:

   $ oc get Network.config.openshift.io cluster -o yaml > cluster-openshift-sdn.yaml

2. To prepare all the nodes for the migration, set the migration field on the Cluster Network Operator configuration object by entering the following command:

   $ oc patch Network.operator.openshift.io cluster --type='merge' \
     --patch '{ "spec": { "migration": {"networkType": "OVNKubernetes" } } }'

   Note

   This step does not deploy OVN-Kubernetes immediately. Instead, specifying the migration field triggers the Machine Config Operator (MCO) to apply new machine configs to all the nodes in the cluster in preparation for the OVN-Kubernetes deployment.

3. Optional: You can customize the following settings for OVN-Kubernetes to meet your network infrastructure requirements:

   • Maximum transmission unit (MTU). Consider the following before customizing the MTU for this optional step:

     • If you use the default MTU, and you want to keep the default MTU during migration, this step can be ignored.
     • If you used a custom MTU, and you want to keep the custom MTU during migration, you must declare the custom MTU value in this step.

     • This step does not work if you want to change the MTU value during migration. Instead, you must first follow the instructions for "Changing the cluster MTU". You can then keep the custom MTU value by performing this procedure and declaring the custom MTU value in this step.

       Note

       OpenShift-SDN and OVN-Kubernetes have different overlay overhead. MTU values should be selected by following the guidelines found on the "MTU value selection" page.

   • Geneve (Generic Network Virtualization Encapsulation) overlay network port

   To customize either of the previously noted settings, enter and customize the following command. If you do not need to change the default value, omit the key from the patch.

   $ oc patch Network.operator.openshift.io cluster --type=merge \
     --patch '{
       "spec":{
         "defaultNetwork":{
           "ovnKubernetesConfig":{
             "mtu":<mtu>,
             "genevePort":<port>
       }}}}'

   mtu

   The MTU for the Geneve overlay network. This value is normally configured automatically, but if the nodes in your cluster do not all use the same MTU, then you must set this explicitly to 100 less than the smallest node MTU value.

   port

   The UDP port for the Geneve overlay network. If a value is not specified, the default is 6081. The port cannot be the same as the VXLAN port that is used by OpenShift SDN. The default value for the VXLAN port is 4789.

   Example patch command to update mtu field

   $ oc patch Network.operator.openshift.io cluster --type=merge \
     --patch '{
       "spec":{
         "defaultNetwork":{
           "ovnKubernetesConfig":{
             "mtu":1200
       }}}}'

4. As the MCO updates machines in each machine config pool, it reboots each node one by one. You must wait until all the nodes are updated. Check the machine config pool status by entering the following command:

   $ oc get mcp

   A successfully updated node has the following status: UPDATED=true, UPDATING=false, DEGRADED=false.

   Note

   By default, the MCO updates one machine per pool at a time, causing the total time the migration takes to increase with the size of the cluster.

5. Confirm the status of the new machine configuration on the hosts:

   1. To list the machine configuration state and the name of the applied machine configuration, enter the following command:

      $ oc describe node | egrep "hostname|machineconfig"

      Example output

      kubernetes.io/hostname=master-0
      machineconfiguration.openshift.io/currentConfig: rendered-master-c53e221d9d24e1c8bb6ee89dd3d8ad7b
      machineconfiguration.openshift.io/desiredConfig: rendered-master-c53e221d9d24e1c8bb6ee89dd3d8ad7b
      machineconfiguration.openshift.io/reason:
      machineconfiguration.openshift.io/state: Done

      Verify that the following statements are true:

      • The value of machineconfiguration.openshift.io/state field is Done.
      • The value of the machineconfiguration.openshift.io/currentConfig field is equal to the value of the machineconfiguration.openshift.io/desiredConfig field.

   2. To confirm that the machine config is correct, enter the following command:

      $ oc get machineconfig <config_name> -o yaml | grep ExecStart

      where <config_name> is the name of the machine config from the machineconfiguration.openshift.io/currentConfig field.

      The machine config must include the following update to the systemd configuration:

      ExecStart=/usr/local/bin/configure-ovs.sh OVNKubernetes

   3. If a node is stuck in the NotReady state, investigate the machine config daemon pod logs and resolve any errors.

      1. To list the pods, enter the following command:

         $ oc get pod -n openshift-machine-config-operator

         Example output

         NAME                                         READY   STATUS    RESTARTS   AGE
         machine-config-controller-75f756f89d-sjp8b   1/1     Running   0          37m
         machine-config-daemon-5cf4b                  2/2     Running   0          43h
         machine-config-daemon-7wzcd                  2/2     Running   0          43h
         machine-config-daemon-fc946                  2/2     Running   0          43h
         machine-config-daemon-g2v28                  2/2     Running   0          43h
         machine-config-daemon-gcl4f                  2/2     Running   0          43h
         machine-config-daemon-l5tnv                  2/2     Running   0          43h
         machine-config-operator-79d9c55d5-hth92      1/1     Running   0          37m
         machine-config-server-bsc8h                  1/1     Running   0          43h
         machine-config-server-hklrm                  1/1     Running   0          43h
         machine-config-server-k9rtx                  1/1     Running   0          43h

         The names for the config daemon pods are in the following format: machine-config-daemon-<seq>. The <seq> value is a random five character alphanumeric sequence.

      2. Display the pod log for the first machine config daemon pod shown in the previous output by enter the following command:

         $ oc logs <pod> -n openshift-machine-config-operator

         where pod is the name of a machine config daemon pod.

      3. Resolve any errors in the logs shown by the output from the previous command.

6. To start the migration, configure the OVN-Kubernetes cluster network provider by using one of the following commands:

   • To specify the network provider without changing the cluster network IP address block, enter the following command:

     $ oc patch Network.config.openshift.io cluster \
       --type='merge' --patch '{ "spec": { "networkType": "OVNKubernetes" } }'

   • To specify a different cluster network IP address block, enter the following command:

     $ oc patch Network.config.openshift.io cluster \
       --type='merge' --patch '{
         "spec": {
           "clusterNetwork": [
             {
               "cidr": "<cidr>",
               "hostPrefix": <prefix>
             }
           ],
           "networkType": "OVNKubernetes"
         }
       }'

     where cidr is a CIDR block and prefix is the slice of the CIDR block apportioned to each node in your cluster. You cannot use any CIDR block that overlaps with the 100.64.0.0/16 CIDR block because the OVN-Kubernetes network provider uses this block internally.

     Important

     You cannot change the service network address block during the migration.

7. Verify that the Multus daemon set rollout is complete before continuing with subsequent steps:

   $ oc -n openshift-multus rollout status daemonset/multus

   The name of the Multus pods is in the form of multus-<xxxxx> where <xxxxx> is a random sequence of letters. It might take several moments for the pods to restart.

   Example output

   Waiting for daemon set "multus" rollout to finish: 1 out of 6 new pods have been updated...
   ...
   Waiting for daemon set "multus" rollout to finish: 5 of 6 updated pods are available...
   daemon set "multus" successfully rolled out

8. To complete the migration, reboot each node in your cluster. For example, you can use a bash script similar to the following example. The script assumes that you can connect to each host by using ssh and that you have configured sudo to not prompt for a password.

   #!/bin/bash
   
   for ip in $(oc get nodes  -o jsonpath='{.items[*].status.addresses[?(@.type=="InternalIP")].address}')
   do
      echo "reboot node $ip"
      ssh -o StrictHostKeyChecking=no core@$ip sudo shutdown -r -t 3
   done

   If ssh access is not available, you might be able to reboot each node through the management portal for your infrastructure provider.

9. Confirm that the migration succeeded:

   1. To confirm that the CNI cluster network provider is OVN-Kubernetes, enter the following command. The value of status.networkType must be OVNKubernetes.

      $ oc get network.config/cluster -o jsonpath='{.status.networkType}{"\n"}'

   2. To confirm that the cluster nodes are in the Ready state, enter the following command:

      $ oc get nodes

   3. To confirm that your pods are not in an error state, enter the following command:

      $ oc get pods --all-namespaces -o wide --sort-by='{.spec.nodeName}'

      If pods on a node are in an error state, reboot that node.

   4. To confirm that all of the cluster Operators are not in an abnormal state, enter the following command:

      $ oc get co

      The status of every cluster Operator must be the following: AVAILABLE="True", PROGRESSING="False", DEGRADED="False". If a cluster Operator is not available or degraded, check the logs for the cluster Operator for more information.

10. Complete the following steps only if the migration succeeds and your cluster is in a good state:

    1. To remove the migration configuration from the CNO configuration object, enter the following command:

       $ oc patch Network.operator.openshift.io cluster --type='merge' \
         --patch '{ "spec": { "migration": null } }'

    2. To remove custom configuration for the OpenShift SDN network provider, enter the following command:

       $ oc patch Network.operator.openshift.io cluster --type='merge' \
         --patch '{ "spec": { "defaultNetwork": { "openshiftSDNConfig": null } } }'

    3. To remove the OpenShift SDN network provider namespace, enter the following command:

       $ oc delete namespace openshift-sdn

Procedure

1. Stop all of the machine configuration pools managed by the Machine Config Operator (MCO):

   • Stop the master configuration pool:

     $ oc patch MachineConfigPool master --type='merge' --patch \
       '{ "spec": { "paused": true } }'

   • Stop the worker machine configuration pool:

     $ oc patch MachineConfigPool worker --type='merge' --patch \
       '{ "spec":{ "paused" :true } }'

2. To start the migration, set the cluster network provider back to OpenShift SDN by entering the following commands:

   $ oc patch Network.operator.openshift.io cluster --type='merge' \
     --patch '{ "spec": { "migration": { "networkType": "OpenShiftSDN" } } }'
   
   $ oc patch Network.config.openshift.io cluster --type='merge' \
     --patch '{ "spec": { "networkType": "OpenShiftSDN" } }'

3. Optional: You can customize the following settings for OpenShift SDN to meet your network infrastructure requirements:

   • Maximum transmission unit (MTU)
   • VXLAN port

   To customize either or both of the previously noted settings, customize and enter the following command. If you do not need to change the default value, omit the key from the patch.

   $ oc patch Network.operator.openshift.io cluster --type=merge \
     --patch '{
       "spec":{
         "defaultNetwork":{
           "openshiftSDNConfig":{
             "mtu":<mtu>,
             "vxlanPort":<port>
       }}}}'

   mtu

   The MTU for the VXLAN overlay network. This value is normally configured automatically, but if the nodes in your cluster do not all use the same MTU, then you must set this explicitly to 50 less than the smallest node MTU value.

   port

   The UDP port for the VXLAN overlay network. If a value is not specified, the default is 4789. The port cannot be the same as the Geneve port that is used by OVN-Kubernetes. The default value for the Geneve port is 6081.

   Example patch command

   $ oc patch Network.operator.openshift.io cluster --type=merge \
     --patch '{
       "spec":{
         "defaultNetwork":{
           "openshiftSDNConfig":{
             "mtu":1200
       }}}}'

4. Wait until the Multus daemon set rollout completes.

   $ oc -n openshift-multus rollout status daemonset/multus

   The name of the Multus pods is in form of multus-<xxxxx> where <xxxxx> is a random sequence of letters. It might take several moments for the pods to restart.

   Example output

   Waiting for daemon set "multus" rollout to finish: 1 out of 6 new pods have been updated...
   ...
   Waiting for daemon set "multus" rollout to finish: 5 of 6 updated pods are available...
   daemon set "multus" successfully rolled out

5. To complete the rollback, reboot each node in your cluster. For example, you could use a bash script similar to the following. The script assumes that you can connect to each host by using ssh and that you have configured sudo to not prompt for a password.

   #!/bin/bash
   
   for ip in $(oc get nodes  -o jsonpath='{.items[*].status.addresses[?(@.type=="InternalIP")].address}')
   do
      echo "reboot node $ip"
      ssh -o StrictHostKeyChecking=no core@$ip sudo shutdown -r -t 3
   done

   If ssh access is not available, you might be able to reboot each node through the management portal for your infrastructure provider.

6. After the nodes in your cluster have rebooted, start all of the machine configuration pools:

   • Start the master configuration pool:

     $ oc patch MachineConfigPool master --type='merge' --patch \
       '{ "spec": { "paused": false } }'

   • Start the worker configuration pool:

     $ oc patch MachineConfigPool worker --type='merge' --patch \
       '{ "spec": { "paused": false } }'

   As the MCO updates machines in each config pool, it reboots each node.

   By default the MCO updates a single machine per pool at a time, so the time that the migration requires to complete grows with the size of the cluster.

7. Confirm the status of the new machine configuration on the hosts:

   1. To list the machine configuration state and the name of the applied machine configuration, enter the following command:

      $ oc describe node | egrep "hostname|machineconfig"

      Example output

      kubernetes.io/hostname=master-0
      machineconfiguration.openshift.io/currentConfig: rendered-master-c53e221d9d24e1c8bb6ee89dd3d8ad7b
      machineconfiguration.openshift.io/desiredConfig: rendered-master-c53e221d9d24e1c8bb6ee89dd3d8ad7b
      machineconfiguration.openshift.io/reason:
      machineconfiguration.openshift.io/state: Done

      Verify that the following statements are true:

      • The value of machineconfiguration.openshift.io/state field is Done.
      • The value of the machineconfiguration.openshift.io/currentConfig field is equal to the value of the machineconfiguration.openshift.io/desiredConfig field.

   2. To confirm that the machine config is correct, enter the following command:

      $ oc get machineconfig <config_name> -o yaml

      where <config_name> is the name of the machine config from the machineconfiguration.openshift.io/currentConfig field.

8. Confirm that the migration succeeded:

   1. To confirm that the default CNI network provider is OpenShift SDN, enter the following command. The value of status.networkType must be OpenShiftSDN.

      $ oc get network.config/cluster -o jsonpath='{.status.networkType}{"\n"}'

   2. To confirm that the cluster nodes are in the Ready state, enter the following command:

      $ oc get nodes

   3. If a node is stuck in the NotReady state, investigate the machine config daemon pod logs and resolve any errors.

      1. To list the pods, enter the following command:

         $ oc get pod -n openshift-machine-config-operator

         Example output

         NAME                                         READY   STATUS    RESTARTS   AGE
         machine-config-controller-75f756f89d-sjp8b   1/1     Running   0          37m
         machine-config-daemon-5cf4b                  2/2     Running   0          43h
         machine-config-daemon-7wzcd                  2/2     Running   0          43h
         machine-config-daemon-fc946                  2/2     Running   0          43h
         machine-config-daemon-g2v28                  2/2     Running   0          43h
         machine-config-daemon-gcl4f                  2/2     Running   0          43h
         machine-config-daemon-l5tnv                  2/2     Running   0          43h
         machine-config-operator-79d9c55d5-hth92      1/1     Running   0          37m
         machine-config-server-bsc8h                  1/1     Running   0          43h
         machine-config-server-hklrm                  1/1     Running   0          43h
         machine-config-server-k9rtx                  1/1     Running   0          43h

         The names for the config daemon pods are in the following format: machine-config-daemon-<seq>. The <seq> value is a random five character alphanumeric sequence.

      2. To display the pod log for each machine config daemon pod shown in the previous output, enter the following command:

         $ oc logs <pod> -n openshift-machine-config-operator

         where pod is the name of a machine config daemon pod.

      3. Resolve any errors in the logs shown by the output from the previous command.

   4. To confirm that your pods are not in an error state, enter the following command:

      $ oc get pods --all-namespaces -o wide --sort-by='{.spec.nodeName}'

      If pods on a node are in an error state, reboot that node.

9. Complete the following steps only if the migration succeeds and your cluster is in a good state:

   1. To remove the migration configuration from the Cluster Network Operator configuration object, enter the following command:

      $ oc patch Network.operator.openshift.io cluster --type='merge' \
        --patch '{ "spec": { "migration": null } }'

   2. To remove the OVN-Kubernetes configuration, enter the following command:

      $ oc patch Network.operator.openshift.io cluster --type='merge' \
        --patch '{ "spec": { "defaultNetwork": { "ovnKubernetesConfig":null } } }'

   3. To remove the OVN-Kubernetes network provider namespace, enter the following command:

      $ oc delete namespace openshift-ovn-kubernetes

Procedure

1. To specify IPv6 address blocks for the cluster and service networks, create a file containing the following YAML:

   - op: add
     path: /spec/clusterNetwork/-
     value: 1
       cidr: fd01::/48
       hostPrefix: 64
   - op: add
     path: /spec/serviceNetwork/-
     value: fd02::/112 2

   1

   Specify an object with the cidr and hostPrefix fields. The host prefix must be 64 or greater. The IPv6 CIDR prefix must be large enough to accommodate the specified host prefix.

   2

   Specify an IPv6 CIDR with a prefix of 112. Kubernetes uses only the lowest 16 bits. For a prefix of 112, IP addresses are assigned from 112 to 128 bits.

2. To patch the cluster network configuration, enter the following command:

   $ oc patch network.config.openshift.io cluster \
     --type='json' --patch-file <file>.yaml

   where:

   file

   Specifies the name of the file you created in the previous step.

   Example output

   network.config.openshift.io/cluster patched

1. Display the network configuration:

   $ oc describe network

   Example output

   Status:
     Cluster Network:
       Cidr:               10.128.0.0/14
       Host Prefix:        23
       Cidr:               fd01::/48
       Host Prefix:        64
     Cluster Network MTU:  1400
     Network Type:         OVNKubernetes
     Service Network:
       172.30.0.0/16
       fd02::/112

Procedure

1. Edit the networks.config.openshift.io custom resource (CR) by running the following command:

   $ oc edit networks.config.openshift.io

2. Remove the IPv6 specific configuration that you have added to the cidr and hostPrefix fields in the previous procedure.

Verification

1. To find the names of the OVN-Kubernetes control plane pods, enter the following command:

   $ oc get pods -n openshift-ovn-kubernetes | grep ovnkube-master

   Example output

   ovnkube-master-4496s        1/1     Running   0          6h39m
   ovnkube-master-d6cht        1/1     Running   0          6h42m
   ovnkube-master-skblc        1/1     Running   0          6h51m
   ovnkube-master-vf8rf        1/1     Running   0          6h51m
   ovnkube-master-w7hjr        1/1     Running   0          6h51m
   ovnkube-master-zsk7x        1/1     Running   0          6h42m

2. Verify that IPsec is enabled on your cluster:

   $ oc -n openshift-ovn-kubernetes -c nbdb rsh ovnkube-master-<XXXXX> \
     ovn-nbctl --no-leader-only get nb_global . ipsec

   where:

   <XXXXX>

   Specifies the random sequence of letters for a pod from the previous step.

   Example output

   true

Procedure

1. To disable IPsec encryption, enter the following command:

   $ oc patch networks.operator.openshift.io/cluster --type=json \
     -p='[{"op":"remove", "path":"/spec/defaultNetwork/ovnKubernetesConfig/ipsecConfig"}]'

2. Optional: You can increase the size of your cluster MTU by 46 bytes because there is no longer any overhead from the IPsec ESP header in IP packets.

Procedure

1. Create a policy rule:

   1. Create a <policy_name>.yaml file where <policy_name> describes the egress policy rules.
   2. In the file you created, define an egress policy object.

2. Enter the following command to create the policy object. Replace <policy_name> with the name of the policy and <project> with the project that the rule applies to.

   $ oc create -f <policy_name>.yaml -n <project>

   In the following example, a new EgressFirewall object is created in a project named project1:

   $ oc create -f default.yaml -n project1

   Example output

   egressfirewall.k8s.ovn.org/v1 created

3. Optional: Save the <policy_name>.yaml file so that you can make changes later.

Procedure

1. Optional: To view the names of the EgressFirewall objects defined in your cluster, enter the following command:

   $ oc get egressfirewall --all-namespaces

2. To inspect a policy, enter the following command. Replace <policy_name> with the name of the policy to inspect.

   $ oc describe egressfirewall <policy_name>

   Example output

   Name:		default
   Namespace:	project1
   Created:	20 minutes ago
   Labels:		<none>
   Annotations:	<none>
   Rule:		Allow to 1.2.3.0/24
   Rule:		Allow to www.example.com
   Rule:		Deny to 0.0.0.0/0

Procedure

1. Find the name of the EgressFirewall object for the project. Replace <project> with the name of the project.

   $ oc get -n <project> egressfirewall

2. Optional: If you did not save a copy of the EgressFirewall object when you created the egress network firewall, enter the following command to create a copy.

   $ oc get -n <project> egressfirewall <name> -o yaml > <filename>.yaml

   Replace <project> with the name of the project. Replace <name> with the name of the object. Replace <filename> with the name of the file to save the YAML to.

3. After making changes to the policy rules, enter the following command to replace the EgressFirewall object. Replace <filename> with the name of the file containing the updated EgressFirewall object.

   $ oc replace -f <filename>.yaml

Procedure

1. Find the name of the EgressFirewall object for the project. Replace <project> with the name of the project.

   $ oc get -n <project> egressfirewall

2. Enter the following command to delete the EgressFirewall object. Replace <project> with the name of the project and <name> with the name of the object.

   $ oc delete -n <project> egressfirewall <name>

Procedure

1. Create an EgressIP object:

   1. Create a <egressips_name>.yaml file where <egressips_name> is the name of the object.

   2. In the file that you created, define an EgressIP object, as in the following example:

      apiVersion: k8s.ovn.org/v1
      kind: EgressIP
      metadata:
        name: egress-project1
      spec:
        egressIPs:
        - 192.168.127.10
        - 192.168.127.11
        namespaceSelector:
          matchLabels:
            env: qa

2. To create the object, enter the following command.

   $ oc apply -f <egressips_name>.yaml 1

   1

   Replace <egressips_name> with the name of the object.

   Example output

   egressips.k8s.ovn.org/<egressips_name> created

3. Optional: Save the <egressips_name>.yaml file so that you can make changes later.

4. Add labels to the namespace that requires egress IP addresses. To add a label to the namespace of an EgressIP object defined in step 1, run the following command:

   $ oc label ns <namespace> env=qa 1

   1

   Replace <namespace> with the namespace that requires egress IP addresses.

Procedure

1. Create an egress router definition.

2. To ensure that other pods can find the IP address of the egress router pod, create a service that uses the egress router, as in the following example:

   apiVersion: v1
   kind: Service
   metadata:
     name: egress-1
   spec:
     ports:
     - name: web-app
       protocol: TCP
       port: 8080
     type: ClusterIP
     selector:
       app: egress-router-cni <.>

   <.> Specify the label for the egress router. The value shown is added by the Cluster Network Operator and is not configurable.

   After you create the service, your pods can connect to the service. The egress router pod redirects traffic to the corresponding port on the destination IP address. The connections originate from the reserved source IP address.

1. View the network attachment definition that the Operator created for the egress router:

   $ oc get network-attachment-definition egress-router-cni-nad

   The name of the network attachment definition is not configurable.

   Example output

   NAME                    AGE
   egress-router-cni-nad   18m

2. View the deployment for the egress router pod:

   $ oc get deployment egress-router-cni-deployment

   The name of the deployment is not configurable.

   Example output

   NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
   egress-router-cni-deployment   1/1     1            1           18m

3. View the status of the egress router pod:

   $ oc get pods -l app=egress-router-cni

   Example output

   NAME                                            READY   STATUS    RESTARTS   AGE
   egress-router-cni-deployment-575465c75c-qkq6m   1/1     Running   0          18m

4. View the logs and the routing table for the egress router pod.

1. Get the node name for the egress router pod:

   $ POD_NODENAME=$(oc get pod -l app=egress-router-cni -o jsonpath="{.items[0].spec.nodeName}")

2. Enter into a debug session on the target node. This step instantiates a debug pod called <node_name>-debug:

   $ oc debug node/$POD_NODENAME

3. Set /host as the root directory within the debug shell. The debug pod mounts the root file system of the host in /host within the pod. By changing the root directory to /host, you can run binaries from the executable paths of the host:

   # chroot /host

4. From within the chroot environment console, display the egress router logs:

   # cat /tmp/egress-router-log

   Example output

   2021-04-26T12:27:20Z [debug] Called CNI ADD
   2021-04-26T12:27:20Z [debug] Gateway: 192.168.12.1
   2021-04-26T12:27:20Z [debug] IP Source Addresses: [192.168.12.99/24]
   2021-04-26T12:27:20Z [debug] IP Destinations: [80 UDP 10.0.0.99/30 8080 TCP 203.0.113.26/30 80 8443 TCP 203.0.113.27/30 443]
   2021-04-26T12:27:20Z [debug] Created macvlan interface
   2021-04-26T12:27:20Z [debug] Renamed macvlan to "net1"
   2021-04-26T12:27:20Z [debug] Adding route to gateway 192.168.12.1 on macvlan interface
   2021-04-26T12:27:20Z [debug] deleted default route {Ifindex: 3 Dst: <nil> Src: <nil> Gw: 10.128.10.1 Flags: [] Table: 254}
   2021-04-26T12:27:20Z [debug] Added new default route with gateway 192.168.12.1
   2021-04-26T12:27:20Z [debug] Added iptables rule: iptables -t nat PREROUTING -i eth0 -p UDP --dport 80 -j DNAT --to-destination 10.0.0.99
   2021-04-26T12:27:20Z [debug] Added iptables rule: iptables -t nat PREROUTING -i eth0 -p TCP --dport 8080 -j DNAT --to-destination 203.0.113.26:80
   2021-04-26T12:27:20Z [debug] Added iptables rule: iptables -t nat PREROUTING -i eth0 -p TCP --dport 8443 -j DNAT --to-destination 203.0.113.27:443
   2021-04-26T12:27:20Z [debug] Added iptables rule: iptables -t nat -o net1 -j SNAT --to-source 192.168.12.99

   The logging file location and logging level are not configurable when you start the egress router by creating an EgressRouter object as described in this procedure.

5. From within the chroot environment console, get the container ID:

   # crictl ps --name egress-router-cni-pod | awk '{print $1}'

   Example output

   CONTAINER
   bac9fae69ddb6

6. Determine the process ID of the container. In this example, the container ID is bac9fae69ddb6:

   # crictl inspect -o yaml bac9fae69ddb6 | grep 'pid:' | awk '{print $2}'

   Example output

   68857

7. Enter the network namespace of the container:

   # nsenter -n -t 68857

8. Display the routing table:

   # ip route

   In the following example output, the net1 network interface is the default route. Traffic for the cluster network uses the eth0 network interface. Traffic for the 192.168.12.0/24 network uses the net1 network interface and originates from the reserved source IP address 192.168.12.99. The pod routes all other traffic to the gateway at IP address 192.168.12.1. Routing for the service network is not shown.

   Example output

   default via 192.168.12.1 dev net1
   10.128.10.0/23 dev eth0 proto kernel scope link src 10.128.10.18
   192.168.12.0/24 dev net1 proto kernel scope link src 192.168.12.99
   192.168.12.1 dev net1

1. Change your current project to the project that you enabled multicast for. Replace <project> with the project name.

   $ oc project <project>

2. Create a pod to act as a multicast receiver:

   $ cat <<EOF| oc create -f -
   apiVersion: v1
   kind: Pod
   metadata:
     name: mlistener
     labels:
       app: multicast-verify
   spec:
     containers:
       - name: mlistener
         image: registry.access.redhat.com/ubi8
         command: ["/bin/sh", "-c"]
         args:
           ["dnf -y install socat hostname && sleep inf"]
         ports:
           - containerPort: 30102
             name: mlistener
             protocol: UDP
   EOF

3. Create a pod to act as a multicast sender:

   $ cat <<EOF| oc create -f -
   apiVersion: v1
   kind: Pod
   metadata:
     name: msender
     labels:
       app: multicast-verify
   spec:
     containers:
       - name: msender
         image: registry.access.redhat.com/ubi8
         command: ["/bin/sh", "-c"]
         args:
           ["dnf -y install socat && sleep inf"]
   EOF

4. In a new terminal window or tab, start the multicast listener.

   1. Get the IP address for the Pod:

      $ POD_IP=$(oc get pods mlistener -o jsonpath='{.status.podIP}')

   2. Start the multicast listener by entering the following command:

      $ oc exec mlistener -i -t -- \
          socat UDP4-RECVFROM:30102,ip-add-membership=224.1.0.1:$POD_IP,fork EXEC:hostname

5. Start the multicast transmitter.

   1. Get the pod network IP address range:

      $ CIDR=$(oc get Network.config.openshift.io cluster \
          -o jsonpath='{.status.clusterNetwork[0].cidr}')

   2. To send a multicast message, enter the following command:

      $ oc exec msender -i -t -- \
          /bin/bash -c "echo | socat STDIO UDP4-DATAGRAM:224.1.0.1:30102,range=$CIDR,ip-multicast-ttl=64"

      If multicast is working, the previous command returns the following output:

      mlistener

Procedure

1. Create a patch file that specifies the network flows collector type and the IP address and port information of the collectors:

   spec:
     exportNetworkFlows:
       netFlow:
         collectors:
           - 192.168.1.99:2056

2. Configure the CNO with the network flows collectors:

   $ oc patch network.operator cluster --type merge -p "$(cat <file_name>.yaml)"

   Example output

   network.operator.openshift.io/cluster patched

1. View the Operator configuration to confirm that the exportNetworkFlows field is configured:

   $ oc get network.operator cluster -o jsonpath="{.spec.exportNetworkFlows}"

   Example output

   {"netFlow":{"collectors":["192.168.1.99:2056"]}}

2. View the network flows configuration in OVS from each node:

   $ for pod in $(oc get pods -n openshift-ovn-kubernetes -l app=ovnkube-node -o jsonpath='{range@.items[*]}{.metadata.name}{"\n"}{end}');
     do ;
       echo;
       echo $pod;
       oc -n openshift-ovn-kubernetes exec -c ovnkube-node $pod \
         -- bash -c 'for type in ipfix sflow netflow ; do ovs-vsctl find $type ; done';
   done

   Example output

   ovnkube-node-xrn4p
   _uuid               : a4d2aaca-5023-4f3d-9400-7275f92611f9
   active_timeout      : 60
   add_id_to_interface : false
   engine_id           : []
   engine_type         : []
   external_ids        : {}
   targets             : ["192.168.1.99:2056"]
   
   ovnkube-node-z4vq9
   _uuid               : 61d02fdb-9228-4993-8ff5-b27f01a29bd6
   active_timeout      : 60
   add_id_to_interface : false
   engine_id           : []
   engine_type         : []
   external_ids        : {}
   targets             : ["192.168.1.99:2056"]-
   
   ...

Procedure

1. Remove all network flows collectors:

   $ oc patch network.operator cluster --type='json' \
       -p='[{"op":"remove", "path":"/spec/exportNetworkFlows"}]'

   Example output

   network.operator.openshift.io/cluster patched

Procedure

1. Change to the directory that contains the installation program and create the manifests:

   $ ./openshift-install create manifests --dir <installation_directory>

   where:

   <installation_directory>

   Specifies the name of the directory that contains the install-config.yaml file for your cluster.

2. Create a stub manifest file for the advanced network configuration that is named cluster-network-03-config.yml in the <installation_directory>/manifests/ directory:

   $ cat <<EOF > <installation_directory>/manifests/cluster-network-03-config.yml
   apiVersion: operator.openshift.io/v1
   kind: Network
   metadata:
     name: cluster
   spec:
   EOF

   where:

   <installation_directory>

   Specifies the directory name that contains the manifests/ directory for your cluster.

3. Open the cluster-network-03-config.yml file in an editor and configure OVN-Kubernetes with hybrid networking, such as in the following example:

   Specify a hybrid networking configuration

   apiVersion: operator.openshift.io/v1
   kind: Network
   metadata:
     name: cluster
   spec:
     defaultNetwork:
       ovnKubernetesConfig:
         hybridOverlayConfig:
           hybridClusterNetwork: 1
           - cidr: 10.132.0.0/14
             hostPrefix: 23
           hybridOverlayVXLANPort: 9898 2

   1

   Specify the CIDR configuration used for nodes on the additional overlay network. The hybridClusterNetwork CIDR cannot overlap with the clusterNetwork CIDR.

   2

   Specify a custom VXLAN port for the additional overlay network. This is required for running Windows nodes in a cluster installed on vSphere, and must not be configured for any other cloud provider. The custom port can be any open port excluding the default 4789 port. For more information on this requirement, see the Microsoft documentation on Pod-to-pod connectivity between hosts is broken.

   Note

   Windows Server Long-Term Servicing Channel (LTSC): Windows Server 2019 is not supported on clusters with a custom hybridOverlayVXLANPort value because this Windows server version does not support selecting a custom VXLAN port.

4. Save the cluster-network-03-config.yml file and quit the text editor.
5. Optional: Back up the manifests/cluster-network-03-config.yml file. The installation program deletes the manifests/ directory when creating the cluster.