Chapter 26. OVN-Kubernetes network plugin

1. Egress firewall is also known as egress network policy in OpenShift SDN. This is not the same as network policy egress.
2. Egress router for OVN-Kubernetes supports only redirect mode.
3. IPv6 is supported only on bare metal, vSphere, IBM Power®, IBM Z®, and Red Hat OpenStack clusters.
4. IPv6 single stack is not supported on IBM Power®, IBM Z®, and Red Hat OpenStack clusters.

Procedure

1. Run the following command to get all resources, endpoints, and ConfigMaps in the OVN-Kubernetes project:

   $ oc get all,ep,cm -n openshift-ovn-kubernetes

   Example output

   Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
   NAME                                         READY   STATUS    RESTARTS       AGE
   pod/ovnkube-control-plane-65c6f55656-6d55h   2/2     Running   0              114m
   pod/ovnkube-control-plane-65c6f55656-fd7vw   2/2     Running   2 (104m ago)   114m
   pod/ovnkube-node-bcvts                       8/8     Running   0              113m
   pod/ovnkube-node-drgvv                       8/8     Running   0              113m
   pod/ovnkube-node-f2pxt                       8/8     Running   0              113m
   pod/ovnkube-node-frqsb                       8/8     Running   0              105m
   pod/ovnkube-node-lbxkk                       8/8     Running   0              105m
   pod/ovnkube-node-tt7bx                       8/8     Running   1 (102m ago)   105m
   
   NAME                                   TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)             AGE
   service/ovn-kubernetes-control-plane   ClusterIP   None         <none>        9108/TCP            114m
   service/ovn-kubernetes-node            ClusterIP   None         <none>        9103/TCP,9105/TCP   114m
   
   NAME                          DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE
   daemonset.apps/ovnkube-node   6         6         6       6            6           beta.kubernetes.io/os=linux   114m
   
   NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/ovnkube-control-plane   3/3     3            3           114m
   
   NAME                                               DESIRED   CURRENT   READY   AGE
   replicaset.apps/ovnkube-control-plane-65c6f55656   3         3         3       114m
   
   NAME                                     ENDPOINTS                                               AGE
   endpoints/ovn-kubernetes-control-plane   10.0.0.3:9108,10.0.0.4:9108,10.0.0.5:9108               114m
   endpoints/ovn-kubernetes-node            10.0.0.3:9105,10.0.0.4:9105,10.0.0.5:9105 + 9 more...   114m
   
   NAME                                 DATA   AGE
   configmap/control-plane-status       1      113m
   configmap/kube-root-ca.crt           1      114m
   configmap/openshift-service-ca.crt   1      114m
   configmap/ovn-ca                     1      114m
   configmap/ovnkube-config             1      114m
   configmap/signer-ca                  1      114m

   There is one ovnkube-node pod for each node in the cluster. The ovnkube-config config map has the OpenShift Container Platform OVN-Kubernetes configurations.

2. List all of the containers in the ovnkube-node pods by running the following command:

   $ oc get pods ovnkube-node-bcvts -o jsonpath='{.spec.containers[*].name}' -n openshift-ovn-kubernetes

   Expected output

   ovn-controller ovn-acl-logging kube-rbac-proxy-node kube-rbac-proxy-ovn-metrics northd nbdb sbdb ovnkube-controller

   The ovnkube-node pod is made up of several containers. It is responsible for hosting the northbound database (nbdb container), the southbound database (sbdb container), the north daemon (northd container), ovn-controller and the ovnkube-controller container. The ovnkube-controller container watches for API objects like pods, egress IPs, namespaces, services, endpoints, egress firewall, and network policies. It is also responsible for allocating pod IP from the available subnet pool for that node.

3. List all the containers in the ovnkube-control-plane pods by running the following command:

   $ oc get pods ovnkube-control-plane-65c6f55656-6d55h -o jsonpath='{.spec.containers[*].name}' -n openshift-ovn-kubernetes

   Expected output

   kube-rbac-proxy ovnkube-cluster-manager

   The ovnkube-control-plane pod has a container (ovnkube-cluster-manager) that resides on each OpenShift Container Platform node. The ovnkube-cluster-manager container allocates pod subnet, transit switch subnet IP and join switch subnet IP to each node in the cluster. The kube-rbac-proxy container monitors metrics for the ovnkube-cluster-manager container.

1. List pods by running the following command:

   $ oc get po -n openshift-ovn-kubernetes

   Example output

   NAME                                     READY   STATUS    RESTARTS      AGE
   ovnkube-control-plane-8444dff7f9-4lh9k   2/2     Running   0             27m
   ovnkube-control-plane-8444dff7f9-5rjh9   2/2     Running   0             27m
   ovnkube-node-55xs2                       8/8     Running   0             26m
   ovnkube-node-7r84r                       8/8     Running   0             16m
   ovnkube-node-bqq8p                       8/8     Running   0             17m
   ovnkube-node-mkj4f                       8/8     Running   0             26m
   ovnkube-node-mlr8k                       8/8     Running   0             26m
   ovnkube-node-wqn2m                       8/8     Running   0             16m

2. Optional: To list the pods with node information, run the following command:

   $ oc get pods -n openshift-ovn-kubernetes -owide

   Example output

   NAME                                     READY   STATUS    RESTARTS      AGE   IP           NODE                                       NOMINATED NODE   READINESS GATES
   ovnkube-control-plane-8444dff7f9-4lh9k   2/2     Running   0             27m   10.0.0.3     ci-ln-t487nnb-72292-mdcnq-master-1         <none>           <none>
   ovnkube-control-plane-8444dff7f9-5rjh9   2/2     Running   0             27m   10.0.0.4     ci-ln-t487nnb-72292-mdcnq-master-2         <none>           <none>
   ovnkube-node-55xs2                       8/8     Running   0             26m   10.0.0.4     ci-ln-t487nnb-72292-mdcnq-master-2         <none>           <none>
   ovnkube-node-7r84r                       8/8     Running   0             17m   10.0.128.3   ci-ln-t487nnb-72292-mdcnq-worker-b-wbz7z   <none>           <none>
   ovnkube-node-bqq8p                       8/8     Running   0             17m   10.0.128.2   ci-ln-t487nnb-72292-mdcnq-worker-a-lh7ms   <none>           <none>
   ovnkube-node-mkj4f                       8/8     Running   0             27m   10.0.0.5     ci-ln-t487nnb-72292-mdcnq-master-0         <none>           <none>
   ovnkube-node-mlr8k                       8/8     Running   0             27m   10.0.0.3     ci-ln-t487nnb-72292-mdcnq-master-1         <none>           <none>
   ovnkube-node-wqn2m                       8/8     Running   0             17m   10.0.128.4   ci-ln-t487nnb-72292-mdcnq-worker-c-przlm   <none>           <none>

3. Navigate into a pod to look at the northbound database by running the following command:

   $ oc rsh -c nbdb -n openshift-ovn-kubernetes ovnkube-node-55xs2

4. Run the following command to show all the objects in the northbound database:

   $ ovn-nbctl show

   The output is too long to list here. The list includes the NAT rules, logical switches, load balancers and so on.

   You can narrow down and focus on specific components by using some of the following optional commands:

   1. Run the following command to show the list of logical routers:

      $ oc exec -n openshift-ovn-kubernetes -it ovnkube-node-55xs2 \
      -c northd -- ovn-nbctl lr-list

      Example output

      45339f4f-7d0b-41d0-b5f9-9fca9ce40ce6 (GR_ci-ln-t487nnb-72292-mdcnq-master-2)
      96a0a0f0-e7ed-4fec-8393-3195563de1b8 (ovn_cluster_router)

      Note

      From this output you can see there is router on each node plus an ovn_cluster_router.

   2. Run the following command to show the list of logical switches:

      $ oc exec -n openshift-ovn-kubernetes -it ovnkube-node-55xs2 \
      -c nbdb -- ovn-nbctl ls-list

      Example output

      bdd7dc3d-d848-4a74-b293-cc15128ea614 (ci-ln-t487nnb-72292-mdcnq-master-2)
      b349292d-ee03-4914-935f-1940b6cb91e5 (ext_ci-ln-t487nnb-72292-mdcnq-master-2)
      0aac0754-ea32-4e33-b086-35eeabf0a140 (join)
      992509d7-2c3f-4432-88db-c179e43592e5 (transit_switch)

      Note

      From this output you can see there is an ext switch for each node plus switches with the node name itself and a join switch.

   3. Run the following command to show the list of load balancers:

      $ oc exec -n openshift-ovn-kubernetes -it ovnkube-node-55xs2 \
      -c nbdb -- ovn-nbctl lb-list

      Example output

      UUID                                    LB                  PROTO      VIP                     IPs
      7c84c673-ed2a-4436-9a1f-9bc5dd181eea    Service_default/    tcp        172.30.0.1:443          10.0.0.3:6443,169.254.169.2:6443,10.0.0.5:6443
      4d663fd9-ddc8-4271-b333-4c0e279e20bb    Service_default/    tcp        172.30.0.1:443          10.0.0.3:6443,10.0.0.4:6443,10.0.0.5:6443
      292eb07f-b82f-4962-868a-4f541d250bca    Service_openshif    tcp        172.30.105.247:443      10.129.0.12:8443
      034b5a7f-bb6a-45e9-8e6d-573a82dc5ee3    Service_openshif    tcp        172.30.192.38:443       10.0.0.3:10259,10.0.0.4:10259,10.0.0.5:10259
      a68bb53e-be84-48df-bd38-bdd82fcd4026    Service_openshif    tcp        172.30.161.125:8443     10.129.0.32:8443
      6cc21b3d-2c54-4c94-8ff5-d8e017269c2e    Service_openshif    tcp        172.30.3.144:443        10.129.0.22:8443
      37996ffd-7268-4862-a27f-61cd62e09c32    Service_openshif    tcp        172.30.181.107:443      10.129.0.18:8443
      81d4da3c-f811-411f-ae0c-bc6713d0861d    Service_openshif    tcp        172.30.228.23:443       10.129.0.29:8443
      ac5a4f3b-b6ba-4ceb-82d0-d84f2c41306e    Service_openshif    tcp        172.30.14.240:9443      10.129.0.36:9443
      c88979fb-1ef5-414b-90ac-43b579351ac9    Service_openshif    tcp        172.30.231.192:9001     10.128.0.5:9001,10.128.2.5:9001,10.129.0.5:9001,10.129.2.4:9001,10.130.0.3:9001,10.131.0.3:9001
      fcb0a3fb-4a77-4230-a84a-be45dce757e8    Service_openshif    tcp        172.30.189.92:443       10.130.0.17:8440
      67ef3e7b-ceb9-4bf0-8d96-b43bde4c9151    Service_openshif    tcp        172.30.67.218:443       10.129.0.9:8443
      d0032fba-7d5e-424a-af25-4ab9b5d46e81    Service_openshif    tcp        172.30.102.137:2379     10.0.0.3:2379,10.0.0.4:2379,10.0.0.5:2379
                                                                  tcp        172.30.102.137:9979     10.0.0.3:9979,10.0.0.4:9979,10.0.0.5:9979
      7361c537-3eec-4e6c-bc0c-0522d182abd4    Service_openshif    tcp        172.30.198.215:9001     10.0.0.3:9001,10.0.0.4:9001,10.0.0.5:9001,10.0.128.2:9001,10.0.128.3:9001,10.0.128.4:9001
      0296c437-1259-410b-a6fd-81c310ad0af5    Service_openshif    tcp        172.30.198.215:9001     10.0.0.3:9001,169.254.169.2:9001,10.0.0.5:9001,10.0.128.2:9001,10.0.128.3:9001,10.0.128.4:9001
      5d5679f5-45b8-479d-9f7c-08b123c688b8    Service_openshif    tcp        172.30.38.253:17698     10.128.0.52:17698,10.129.0.84:17698,10.130.0.60:17698
      2adcbab4-d1c9-447d-9573-b5dc9f2efbfa    Service_openshif    tcp        172.30.148.52:443       10.0.0.4:9202,10.0.0.5:9202
                                                                  tcp        172.30.148.52:444       10.0.0.4:9203,10.0.0.5:9203
                                                                  tcp        172.30.148.52:445       10.0.0.4:9204,10.0.0.5:9204
                                                                  tcp        172.30.148.52:446       10.0.0.4:9205,10.0.0.5:9205
      2a33a6d7-af1b-4892-87cc-326a380b809b    Service_openshif    tcp        172.30.67.219:9091      10.129.2.16:9091,10.131.0.16:9091
                                                                  tcp        172.30.67.219:9092      10.129.2.16:9092,10.131.0.16:9092
                                                                  tcp        172.30.67.219:9093      10.129.2.16:9093,10.131.0.16:9093
                                                                  tcp        172.30.67.219:9094      10.129.2.16:9094,10.131.0.16:9094
      f56f59d7-231a-4974-99b3-792e2741ec8d    Service_openshif    tcp        172.30.89.212:443       10.128.0.41:8443,10.129.0.68:8443,10.130.0.44:8443
      08c2c6d7-d217-4b96-b5d8-c80c4e258116    Service_openshif    tcp        172.30.102.137:2379     10.0.0.3:2379,169.254.169.2:2379,10.0.0.5:2379
                                                                  tcp        172.30.102.137:9979     10.0.0.3:9979,169.254.169.2:9979,10.0.0.5:9979
      60a69c56-fc6a-4de6-bd88-3f2af5ba5665    Service_openshif    tcp        172.30.10.193:443       10.129.0.25:8443
      ab1ef694-0826-4671-a22c-565fc2d282ec    Service_openshif    tcp        172.30.196.123:443      10.128.0.33:8443,10.129.0.64:8443,10.130.0.37:8443
      b1fb34d3-0944-4770-9ee3-2683e7a630e2    Service_openshif    tcp        172.30.158.93:8443      10.129.0.13:8443
      95811c11-56e2-4877-be1e-c78ccb3a82a9    Service_openshif    tcp        172.30.46.85:9001       10.130.0.16:9001
      4baba1d1-b873-4535-884c-3f6fc07a50fd    Service_openshif    tcp        172.30.28.87:443        10.129.0.26:8443
      6c2e1c90-f0ca-484e-8a8e-40e71442110a    Service_openshif    udp        172.30.0.10:53          10.128.0.13:5353,10.128.2.6:5353,10.129.0.39:5353,10.129.2.6:5353,10.130.0.11:5353,10.131.0.9:5353

      Note

      From this truncated output you can see there are many OVN-Kubernetes load balancers. Load balancers in OVN-Kubernetes are representations of services.

5. Run the following command to display the options available with the command ovn-nbctl:

   $ oc exec -n openshift-ovn-kubernetes -it ovnkube-node-55xs2 \
   -c nbdb ovn-nbctl --help

1. List the pods by running the following command:

   $ oc get po -n openshift-ovn-kubernetes

   Example output

   NAME                                     READY   STATUS    RESTARTS      AGE
   ovnkube-control-plane-8444dff7f9-4lh9k   2/2     Running   0             27m
   ovnkube-control-plane-8444dff7f9-5rjh9   2/2     Running   0             27m
   ovnkube-node-55xs2                       8/8     Running   0             26m
   ovnkube-node-7r84r                       8/8     Running   0             16m
   ovnkube-node-bqq8p                       8/8     Running   0             17m
   ovnkube-node-mkj4f                       8/8     Running   0             26m
   ovnkube-node-mlr8k                       8/8     Running   0             26m
   ovnkube-node-wqn2m                       8/8     Running   0             16m

2. Optional: To list the pods with node information, run the following command:

   $ oc get pods -n openshift-ovn-kubernetes -owide

   Example output

   NAME                                     READY   STATUS    RESTARTS      AGE   IP           NODE                                       NOMINATED NODE   READINESS GATES
   ovnkube-control-plane-8444dff7f9-4lh9k   2/2     Running   0             27m   10.0.0.3     ci-ln-t487nnb-72292-mdcnq-master-1         <none>           <none>
   ovnkube-control-plane-8444dff7f9-5rjh9   2/2     Running   0             27m   10.0.0.4     ci-ln-t487nnb-72292-mdcnq-master-2         <none>           <none>
   ovnkube-node-55xs2                       8/8     Running   0             26m   10.0.0.4     ci-ln-t487nnb-72292-mdcnq-master-2         <none>           <none>
   ovnkube-node-7r84r                       8/8     Running   0             17m   10.0.128.3   ci-ln-t487nnb-72292-mdcnq-worker-b-wbz7z   <none>           <none>
   ovnkube-node-bqq8p                       8/8     Running   0             17m   10.0.128.2   ci-ln-t487nnb-72292-mdcnq-worker-a-lh7ms   <none>           <none>
   ovnkube-node-mkj4f                       8/8     Running   0             27m   10.0.0.5     ci-ln-t487nnb-72292-mdcnq-master-0         <none>           <none>
   ovnkube-node-mlr8k                       8/8     Running   0             27m   10.0.0.3     ci-ln-t487nnb-72292-mdcnq-master-1         <none>           <none>
   ovnkube-node-wqn2m                       8/8     Running   0             17m   10.0.128.4   ci-ln-t487nnb-72292-mdcnq-worker-c-przlm   <none>           <none>

3. Navigate into a pod to look at the southbound database:

   $ oc rsh -c sbdb -n openshift-ovn-kubernetes ovnkube-node-55xs2

4. Run the following command to show all the objects in the southbound database:

   $ ovn-sbctl show

   Example output

   Chassis "5db31703-35e9-413b-8cdf-69e7eecb41f7"
       hostname: ci-ln-9gp362t-72292-v2p94-worker-a-8bmwz
       Encap geneve
           ip: "10.0.128.4"
           options: {csum="true"}
       Port_Binding tstor-ci-ln-9gp362t-72292-v2p94-worker-a-8bmwz
   Chassis "070debed-99b7-4bce-b17d-17e720b7f8bc"
       hostname: ci-ln-9gp362t-72292-v2p94-worker-b-svmp6
       Encap geneve
           ip: "10.0.128.2"
           options: {csum="true"}
       Port_Binding k8s-ci-ln-9gp362t-72292-v2p94-worker-b-svmp6
       Port_Binding rtoe-GR_ci-ln-9gp362t-72292-v2p94-worker-b-svmp6
       Port_Binding openshift-monitoring_alertmanager-main-1
       Port_Binding rtoj-GR_ci-ln-9gp362t-72292-v2p94-worker-b-svmp6
       Port_Binding etor-GR_ci-ln-9gp362t-72292-v2p94-worker-b-svmp6
       Port_Binding cr-rtos-ci-ln-9gp362t-72292-v2p94-worker-b-svmp6
       Port_Binding openshift-e2e-loki_loki-promtail-qcrcz
       Port_Binding jtor-GR_ci-ln-9gp362t-72292-v2p94-worker-b-svmp6
       Port_Binding openshift-multus_network-metrics-daemon-mkd4t
       Port_Binding openshift-ingress-canary_ingress-canary-xtvj4
       Port_Binding openshift-ingress_router-default-6c76cbc498-pvlqk
       Port_Binding openshift-dns_dns-default-zz582
       Port_Binding openshift-monitoring_thanos-querier-57585899f5-lbf4f
       Port_Binding openshift-network-diagnostics_network-check-target-tn228
       Port_Binding openshift-monitoring_prometheus-k8s-0
       Port_Binding openshift-image-registry_image-registry-68899bd877-xqxjj
   Chassis "179ba069-0af1-401c-b044-e5ba90f60fea"
       hostname: ci-ln-9gp362t-72292-v2p94-master-0
       Encap geneve
           ip: "10.0.0.5"
           options: {csum="true"}
       Port_Binding tstor-ci-ln-9gp362t-72292-v2p94-master-0
   Chassis "68c954f2-5a76-47be-9e84-1cb13bd9dab9"
       hostname: ci-ln-9gp362t-72292-v2p94-worker-c-mjf9w
       Encap geneve
           ip: "10.0.128.3"
           options: {csum="true"}
       Port_Binding tstor-ci-ln-9gp362t-72292-v2p94-worker-c-mjf9w
   Chassis "2de65d9e-9abf-4b6e-a51d-a1e038b4d8af"
       hostname: ci-ln-9gp362t-72292-v2p94-master-2
       Encap geneve
           ip: "10.0.0.4"
           options: {csum="true"}
       Port_Binding tstor-ci-ln-9gp362t-72292-v2p94-master-2
   Chassis "1d371cb8-5e21-44fd-9025-c4b162cc4247"
       hostname: ci-ln-9gp362t-72292-v2p94-master-1
       Encap geneve
           ip: "10.0.0.3"
           options: {csum="true"}
       Port_Binding tstor-ci-ln-9gp362t-72292-v2p94-master-1

   This detailed output shows the chassis and the ports that are attached to the chassis which in this case are all of the router ports and anything that runs like host networking. Any pods communicate out to the wider network using source network address translation (SNAT). Their IP address is translated into the IP address of the node that the pod is running on and then sent out into the network.

   In addition to the chassis information the southbound database has all the logic flows and those logic flows are then sent to the ovn-controller running on each of the nodes. The ovn-controller translates the logic flows into open flow rules and ultimately programs OpenvSwitch so that your pods can then follow open flow rules and make it out of the network.

5. Run the following command to display the options available with the command ovn-sbctl:

   $ oc exec -n openshift-ovn-kubernetes -it ovnkube-node-55xs2 \
   -c sbdb ovn-sbctl --help

Procedure

1. Review the details of the ovnkube-node readiness probe by running the following command:

   $ oc get pods -n openshift-ovn-kubernetes -l app=ovnkube-node \
   -o json | jq '.items[0].spec.containers[] | .name,.readinessProbe'

   The readiness probe for the northbound and southbound database containers in the ovnkube-node pod checks for the health of the databases and the ovnkube-controller container.

   The ovnkube-controller container in the ovnkube-node pod has a readiness probe to verify the presence of the OVN-Kubernetes CNI configuration file, the absence of which would indicate that the pod is not running or is not ready to accept requests to configure pods.

2. Show all events including the probe failures, for the namespace by using the following command:

   $ oc get events -n openshift-ovn-kubernetes

3. Show the events for just a specific pod:

   $ oc describe pod ovnkube-node-9lqfk -n openshift-ovn-kubernetes

4. Show the messages and statuses from the cluster network operator:

   $ oc get co/network -o json | jq '.status.conditions[]'

5. Show the ready status of each container in ovnkube-node pods by running the following script:

   $ for p in $(oc get pods --selector app=ovnkube-node -n openshift-ovn-kubernetes \
   -o jsonpath='{range.items[*]}{" "}{.metadata.name}'); do echo === $p ===;  \
   oc get pods -n openshift-ovn-kubernetes $p -o json | jq '.status.containerStatuses[] | .name, .ready'; \
   done

   Note

   The expectation is all container statuses are reporting as true. Failure of a readiness probe sets the status to false.

Procedure (UI)

1. In the Administrator perspective, select Observe Alerting. The three main pages in the Alerting UI in this perspective are the Alerts, Silences, and Alerting Rules pages.
2. View the rules for OVN-Kubernetes alerts by selecting Observe Alerting Alerting Rules.

Procedure

1. View active or firing alerts by running the following commands.

   1. Set the alert manager route environment variable by running the following command:

      $ ALERT_MANAGER=$(oc get route alertmanager-main -n openshift-monitoring \
      -o jsonpath='{@.spec.host}')

   2. Issue a curl request to the alert manager route API by running the following command, replacing $ALERT_MANAGER with the URL of your Alertmanager instance:

      $ curl -s -k -H "Authorization: Bearer $(oc create token prometheus-k8s -n openshift-monitoring)" https://$ALERT_MANAGER/api/v1/alerts | jq '.data[] | "\(.labels.severity) \(.labels.alertname) \(.labels.pod) \(.labels.container) \(.labels.endpoint) \(.labels.instance)"'

2. View alerting rules by running the following command:

   $ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -s 'http://localhost:9090/api/v1/rules' | jq '.data.groups[].rules[] | select(((.name|contains("ovn")) or (.name|contains("OVN")) or (.name|contains("Ovn")) or (.name|contains("North")) or (.name|contains("South"))) and .type=="alerting")'

Procedure

1. View the log for a specific pod:

   $ oc logs -f <pod_name> -c <container_name> -n <namespace>

   where:

   -f

   Optional: Specifies that the output follows what is being written into the logs.

   <pod_name>

   Specifies the name of the pod.

   <container_name>

   Optional: Specifies the name of a container. When a pod has more than one container, you must specify the container name.

   <namespace>

   Specify the namespace the pod is running in.

   For example:

   $ oc logs ovnkube-node-5dx44 -n openshift-ovn-kubernetes

   $ oc logs -f ovnkube-node-5dx44 -c ovnkube-controller -n openshift-ovn-kubernetes

   The contents of log files are printed out.

2. Examine the most recent entries in all the containers in the ovnkube-node pods:

   $ for p in $(oc get pods --selector app=ovnkube-node -n openshift-ovn-kubernetes \
   -o jsonpath='{range.items[*]}{" "}{.metadata.name}'); \
   do echo === $p ===; for container in $(oc get pods -n openshift-ovn-kubernetes $p \
   -o json | jq -r '.status.containerStatuses[] | .name');do echo ---$container---; \
   oc logs -c $container $p -n openshift-ovn-kubernetes --tail=5; done; done

3. View the last 5 lines of every log in every container in an ovnkube-node pod using the following command:

   $ oc logs -l app=ovnkube-node -n openshift-ovn-kubernetes --all-containers --tail 5

Procedure

1. In the OpenShift Container Platform console, navigate to Workloads Pods or navigate to the pod through the resource you want to investigate.
2. Select the openshift-ovn-kubernetes project from the drop-down menu.
3. Click the name of the pod you want to investigate.
4. Click Logs. By default for the ovnkube-master the logs associated with the northd container are displayed.
5. Use the down-down menu to select logs for each container in turn.

Procedure

1. To list the current PodNetworkConnectivityCheck objects, enter the following command:

   $ oc get podnetworkconnectivitychecks -n openshift-network-diagnostics

2. View the most recent success for each connection object by using the following command:

   $ oc get podnetworkconnectivitychecks -n openshift-network-diagnostics \
   -o json | jq '.items[]| .spec.targetEndpoint,.status.successes[0]'

3. View the most recent failures for each connection object by using the following command:

   $ oc get podnetworkconnectivitychecks -n openshift-network-diagnostics \
   -o json | jq '.items[]| .spec.targetEndpoint,.status.failures[0]'

4. View the most recent outages for each connection object by using the following command:

   $ oc get podnetworkconnectivitychecks -n openshift-network-diagnostics \
   -o json | jq '.items[]| .spec.targetEndpoint,.status.outages[0]'

   The connectivity check controller also logs metrics from these checks into Prometheus.

5. View all the metrics by running the following command:

   $ oc exec prometheus-k8s-0 -n openshift-monitoring -- \
   promtool query instant  http://localhost:9090 \
   '{component="openshift-network-diagnostics"}'

6. View the latency between the source pod and the openshift api service for the last 5 minutes:

   $ oc exec prometheus-k8s-0 -n openshift-monitoring -- \
   promtool query instant  http://localhost:9090 \
   '{component="openshift-network-diagnostics"}'

Procedure

1. Create a pod variable by using the following command:

   $  POD=$(oc get pods -n openshift-ovn-kubernetes -l app=ovnkube-control-plane -o name | head -1 | awk -F '/' '{print $NF}')

2. Run the following command on your local host to copy the binary from the ovnkube-control-plane pods:

   $  oc cp -n openshift-ovn-kubernetes $POD:/usr/bin/ovnkube-trace -c ovnkube-cluster-manager ovnkube-trace

   Note

   If you are using Red Hat Enterprise Linux (RHEL) 8 to run the ovnkube-trace tool, you must copy the file /usr/lib/rhel8/ovnkube-trace to your local host.

3. Make ovnkube-trace executable by running the following command:

   $  chmod +x ovnkube-trace

4. Display the options available with ovnkube-trace by running the following command:

   $  ./ovnkube-trace -help

   Expected output

   Usage of ./ovnkube-trace:
     -addr-family string
       	Address family (ip4 or ip6) to be used for tracing (default "ip4")
     -dst string
       	dest: destination pod name
     -dst-ip string
       	destination IP address (meant for tests to external targets)
     -dst-namespace string
       	k8s namespace of dest pod (default "default")
     -dst-port string
       	dst-port: destination port (default "80")
     -kubeconfig string
       	absolute path to the kubeconfig file
     -loglevel string
       	loglevel: klog level (default "0")
     -ovn-config-namespace string
       	namespace used by ovn-config itself
     -service string
       	service: destination service name
     -skip-detrace
       	skip ovn-detrace command
     -src string
       	src: source pod name
     -src-namespace string
       	k8s namespace of source pod (default "default")
     -tcp
       	use tcp transport protocol
     -udp
       	use udp transport protocol

   The command-line arguments supported are familiar Kubernetes constructs, such as namespaces, pods, services so you do not need to find the MAC address, the IP address of the destination nodes, or the ICMP type.

   The log levels are:

   • 0 (minimal output)
   • 2 (more verbose output showing results of trace commands)
   • 5 (debug output)

Procedure

1. Start a web service in the default namespace by entering the following command:

   $ oc run web --namespace=default --image=quay.io/openshifttest/nginx --labels="app=web" --expose --port=80

2. List the pods running in the openshift-dns namespace:

   oc get pods -n openshift-dns

   Example output

   NAME                  READY   STATUS    RESTARTS   AGE
   dns-default-8s42x     2/2     Running   0          5h8m
   dns-default-mdw6r     2/2     Running   0          4h58m
   dns-default-p8t5h     2/2     Running   0          4h58m
   dns-default-rl6nk     2/2     Running   0          5h8m
   dns-default-xbgqx     2/2     Running   0          5h8m
   dns-default-zv8f6     2/2     Running   0          4h58m
   node-resolver-62jjb   1/1     Running   0          5h8m
   node-resolver-8z4cj   1/1     Running   0          4h59m
   node-resolver-bq244   1/1     Running   0          5h8m
   node-resolver-hc58n   1/1     Running   0          4h59m
   node-resolver-lm6z4   1/1     Running   0          5h8m
   node-resolver-zfx5k   1/1     Running   0          5h

3. Run the following ovnkube-trace command to verify DNS resolution is working:

   $ ./ovnkube-trace \
     -src-namespace default \ 1
     -src web \ 2
     -dst-namespace openshift-dns \ 3
     -dst dns-default-p8t5h \ 4
     -udp -dst-port 53 \ 5
     -loglevel 0 6

   1

   Namespace of the source pod

   2

   Source pod name

   3

   Namespace of destination pod

   4

   Destination pod name

   5

   Use the udp transport protocol. Port 53 is the port the DNS service uses.

   6

   Set the log level to 0 (0 is minimal and 5 is debug)

   Example output if the src&dst pod lands on the same node:

   ovn-trace source pod to destination pod indicates success from web to dns-default-p8t5h
   ovn-trace destination pod to source pod indicates success from dns-default-p8t5h to web
   ovs-appctl ofproto/trace source pod to destination pod indicates success from web to dns-default-p8t5h
   ovs-appctl ofproto/trace destination pod to source pod indicates success from dns-default-p8t5h to web
   ovn-detrace source pod to destination pod indicates success from web to dns-default-p8t5h
   ovn-detrace destination pod to source pod indicates success from dns-default-p8t5h to web

   Example output if the src&dst pod lands on a different node:

   ovn-trace source pod to destination pod indicates success from web to dns-default-8s42x
   ovn-trace (remote) source pod to destination pod indicates success from web to dns-default-8s42x
   ovn-trace destination pod to source pod indicates success from dns-default-8s42x to web
   ovn-trace (remote) destination pod to source pod indicates success from dns-default-8s42x to web
   ovs-appctl ofproto/trace source pod to destination pod indicates success from web to dns-default-8s42x
   ovs-appctl ofproto/trace destination pod to source pod indicates success from dns-default-8s42x to web
   ovn-detrace source pod to destination pod indicates success from web to dns-default-8s42x
   ovn-detrace destination pod to source pod indicates success from dns-default-8s42x to web

   The ouput indicates success from the deployed pod to the DNS port and also indicates that it is successful going back in the other direction. So you know bi-directional traffic is supported on UDP port 53 if my web pod wants to do dns resolution from core DNS.

Procedure

1. Create the following YAML that defines a deny-by-default policy to deny ingress from all pods in all namespaces. Save the YAML in the deny-by-default.yaml file:

   kind: NetworkPolicy
   apiVersion: networking.k8s.io/v1
   metadata:
     name: deny-by-default
     namespace: default
   spec:
     podSelector: {}
     ingress: []

2. Apply the policy by entering the following command:

   $ oc apply -f deny-by-default.yaml

   Example output

   networkpolicy.networking.k8s.io/deny-by-default created

3. Start a web service in the default namespace by entering the following command:

   $ oc run web --namespace=default --image=quay.io/openshifttest/nginx --labels="app=web" --expose --port=80

4. Run the following command to create the prod namespace:

   $ oc create namespace prod

5. Run the following command to label the prod namespace:

   $ oc label namespace/prod purpose=production

6. Run the following command to deploy an alpine image in the prod namespace and start a shell:

   $ oc run test-6459 --namespace=prod --rm -i -t --image=alpine -- sh

7. Open another terminal session.

8. In this new terminal session run ovn-trace to verify the failure in communication between the source pod test-6459 running in namespace prod and destination pod running in the default namespace:

   $ ./ovnkube-trace \
    -src-namespace prod \
    -src test-6459 \
    -dst-namespace default \
    -dst web \
    -tcp -dst-port 80 \
    -loglevel 0

   Example output

   ovn-trace source pod to destination pod indicates failure from test-6459 to web

9. Increase the log level to 2 to expose the reason for the failure by running the following command:

   $ ./ovnkube-trace \
    -src-namespace prod \
    -src test-6459 \
    -dst-namespace default \
    -dst web \
    -tcp -dst-port 80 \
    -loglevel 2

   Example output

   ...
   ------------------------------------------------
    3. ls_out_acl_hint (northd.c:7454): !ct.new && ct.est && !ct.rpl && ct_mark.blocked == 0, priority 4, uuid 12efc456
       reg0[8] = 1;
       reg0[10] = 1;
       next;
    5. ls_out_acl_action (northd.c:7835): reg8[30..31] == 0, priority 500, uuid 69372c5d
       reg8[30..31] = 1;
       next(4);
    5. ls_out_acl_action (northd.c:7835): reg8[30..31] == 1, priority 500, uuid 2fa0af89
       reg8[30..31] = 2;
       next(4);
    4. ls_out_acl_eval (northd.c:7691): reg8[30..31] == 2 && reg0[10] == 1 && (outport == @a16982411286042166782_ingressDefaultDeny), priority 2000, uuid 447d0dab
       reg8[17] = 1;
       ct_commit { ct_mark.blocked = 1; }; 1
       next;
   ...

   1

   Ingress traffic is blocked due to the default deny policy being in place.

10. Create a policy that allows traffic from all pods in a particular namespaces with a label purpose=production. Save the YAML in the web-allow-prod.yaml file:

    kind: NetworkPolicy
    apiVersion: networking.k8s.io/v1
    metadata:
      name: web-allow-prod
      namespace: default
    spec:
      podSelector:
        matchLabels:
          app: web
      policyTypes:
      - Ingress
      ingress:
      - from:
        - namespaceSelector:
            matchLabels:
              purpose: production

11. Apply the policy by entering the following command:

    $ oc apply -f web-allow-prod.yaml

12. Run ovnkube-trace to verify that traffic is now allowed by entering the following command:

    $ ./ovnkube-trace \
     -src-namespace prod \
     -src test-6459 \
     -dst-namespace default \
     -dst web \
     -tcp -dst-port 80 \
     -loglevel 0

    Expected output

    ovn-trace source pod to destination pod indicates success from test-6459 to web
    ovn-trace destination pod to source pod indicates success from web to test-6459
    ovs-appctl ofproto/trace source pod to destination pod indicates success from test-6459 to web
    ovs-appctl ofproto/trace destination pod to source pod indicates success from web to test-6459
    ovn-detrace source pod to destination pod indicates success from test-6459 to web
    ovn-detrace destination pod to source pod indicates success from web to test-6459

13. Run the following command in the shell that was opened in step six to connect nginx to the web-server:

     wget -qO- --timeout=2 http://web.default

    Expected output

    <!DOCTYPE html>
    <html>
    <head>
    <title>Welcome to nginx!</title>
    <style>
      body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
      }
    </style>
    </head>
    <body>
    <h1>Welcome to nginx!</h1>
    <p>If you see this page, the nginx web server is successfully installed and
    working. Further configuration is required.</p>
    
    <p>For online documentation and support please refer to
    <a href="http://nginx.org/">nginx.org</a>.<br/>
    Commercial support is available at
    <a href="http://nginx.com/">nginx.com</a>.</p>
    
    <p><em>Thank you for using nginx.</em></p>
    </body>
    </html>

Procedure

1. Stop all of the machine configuration pools managed by the Machine Config Operator (MCO):

   • Stop the master configuration pool:

     $ oc patch MachineConfigPool master --type='merge' --patch \
       '{ "spec": { "paused": true } }'

   • Stop the worker machine configuration pool:

     $ oc patch MachineConfigPool worker --type='merge' --patch \
       '{ "spec":{ "paused": true } }'

2. To prepare for the migration, set the migration field to null by entering the following command:

   $ oc patch Network.operator.openshift.io cluster --type='merge' \
     --patch '{ "spec": { "migration": null } }'

3. To start the migration, set the network plugin back to OpenShift SDN by entering the following commands:

   $ oc patch Network.operator.openshift.io cluster --type='merge' \
     --patch '{ "spec": { "migration": { "networkType": "OpenShiftSDN" } } }'
   
   $ oc patch Network.config.openshift.io cluster --type='merge' \
     --patch '{ "spec": { "networkType": "OpenShiftSDN" } }'

4. Optional: You can disable automatic migration of several OVN-Kubernetes capabilities to the OpenShift SDN equivalents:

   • Egress IPs
   • Egress firewall
   • Multicast

   To disable automatic migration of the configuration for any of the previously noted OpenShift SDN features, specify the following keys:

   $ oc patch Network.operator.openshift.io cluster --type='merge' \
     --patch '{
       "spec": {
         "migration": {
           "networkType": "OpenShiftSDN",
           "features": {
             "egressIP": <bool>,
             "egressFirewall": <bool>,
             "multicast": <bool>
           }
         }
       }
     }'

   where:

   bool: Specifies whether to enable migration of the feature. The default is true.

5. Optional: You can customize the following settings for OpenShift SDN to meet your network infrastructure requirements:

   • Maximum transmission unit (MTU)
   • VXLAN port

   To customize either or both of the previously noted settings, customize and enter the following command. If you do not need to change the default value, omit the key from the patch.

   $ oc patch Network.operator.openshift.io cluster --type=merge \
     --patch '{
       "spec":{
         "defaultNetwork":{
           "openshiftSDNConfig":{
             "mtu":<mtu>,
             "vxlanPort":<port>
       }}}}'

   mtu

   The MTU for the VXLAN overlay network. This value is normally configured automatically, but if the nodes in your cluster do not all use the same MTU, then you must set this explicitly to 50 less than the smallest node MTU value.

   port

   The UDP port for the VXLAN overlay network. If a value is not specified, the default is 4789. The port cannot be the same as the Geneve port that is used by OVN-Kubernetes. The default value for the Geneve port is 6081.

   Example patch command

   $ oc patch Network.operator.openshift.io cluster --type=merge \
     --patch '{
       "spec":{
         "defaultNetwork":{
           "openshiftSDNConfig":{
             "mtu":1200
       }}}}'

6. Reboot each node in your cluster. You can reboot the nodes in your cluster with either of the following approaches:

   • With the oc rsh command, you can use a bash script similar to the following:

     #!/bin/bash
     readarray -t POD_NODES <<< "$(oc get pod -n openshift-machine-config-operator -o wide| grep daemon|awk '{print $1" "$7}')"
     
     for i in "${POD_NODES[@]}"
     do
       read -r POD NODE <<< "$i"
       until oc rsh -n openshift-machine-config-operator "$POD" chroot /rootfs shutdown -r +1
         do
           echo "cannot reboot node $NODE, retry" && sleep 3
         done
     done

   • With the ssh command, you can use a bash script similar to the following. The script assumes that you have configured sudo to not prompt for a password.

     #!/bin/bash
     
     for ip in $(oc get nodes  -o jsonpath='{.items[*].status.addresses[?(@.type=="InternalIP")].address}')
     do
        echo "reboot node $ip"
        ssh -o StrictHostKeyChecking=no core@$ip sudo shutdown -r -t 3
     done

7. Wait until the Multus daemon set rollout completes. Run the following command to see your rollout status:

   $ oc -n openshift-multus rollout status daemonset/multus

   The name of the Multus pods is in the form of multus-<xxxxx> where <xxxxx> is a random sequence of letters. It might take several moments for the pods to restart.

   Example output

   Waiting for daemon set "multus" rollout to finish: 1 out of 6 new pods have been updated...
   ...
   Waiting for daemon set "multus" rollout to finish: 5 of 6 updated pods are available...
   daemon set "multus" successfully rolled out

8. After the nodes in your cluster have rebooted and the multus pods are rolled out, start all of the machine configuration pools by running the following commands::

   • Start the master configuration pool:

     $ oc patch MachineConfigPool master --type='merge' --patch \
       '{ "spec": { "paused": false } }'

   • Start the worker configuration pool:

     $ oc patch MachineConfigPool worker --type='merge' --patch \
       '{ "spec": { "paused": false } }'

   As the MCO updates machines in each config pool, it reboots each node.

   By default the MCO updates a single machine per pool at a time, so the time that the migration requires to complete grows with the size of the cluster.

9. Confirm the status of the new machine configuration on the hosts:

   1. To list the machine configuration state and the name of the applied machine configuration, enter the following command:

      $ oc describe node | egrep "hostname|machineconfig"

      Example output

      kubernetes.io/hostname=master-0
      machineconfiguration.openshift.io/currentConfig: rendered-master-c53e221d9d24e1c8bb6ee89dd3d8ad7b
      machineconfiguration.openshift.io/desiredConfig: rendered-master-c53e221d9d24e1c8bb6ee89dd3d8ad7b
      machineconfiguration.openshift.io/reason:
      machineconfiguration.openshift.io/state: Done

      Verify that the following statements are true:

      • The value of machineconfiguration.openshift.io/state field is Done.
      • The value of the machineconfiguration.openshift.io/currentConfig field is equal to the value of the machineconfiguration.openshift.io/desiredConfig field.

   2. To confirm that the machine config is correct, enter the following command:

      $ oc get machineconfig <config_name> -o yaml

      where <config_name> is the name of the machine config from the machineconfiguration.openshift.io/currentConfig field.

10. Confirm that the migration succeeded:

    1. To confirm that the network plugin is OpenShift SDN, enter the following command. The value of status.networkType must be OpenShiftSDN.

       $ oc get network.config/cluster -o jsonpath='{.status.networkType}{"\n"}'

    2. To confirm that the cluster nodes are in the Ready state, enter the following command:

       $ oc get nodes

    3. If a node is stuck in the NotReady state, investigate the machine config daemon pod logs and resolve any errors.

       1. To list the pods, enter the following command:

          $ oc get pod -n openshift-machine-config-operator

          Example output

          NAME                                         READY   STATUS    RESTARTS   AGE
          machine-config-controller-75f756f89d-sjp8b   1/1     Running   0          37m
          machine-config-daemon-5cf4b                  2/2     Running   0          43h
          machine-config-daemon-7wzcd                  2/2     Running   0          43h
          machine-config-daemon-fc946                  2/2     Running   0          43h
          machine-config-daemon-g2v28                  2/2     Running   0          43h
          machine-config-daemon-gcl4f                  2/2     Running   0          43h
          machine-config-daemon-l5tnv                  2/2     Running   0          43h
          machine-config-operator-79d9c55d5-hth92      1/1     Running   0          37m
          machine-config-server-bsc8h                  1/1     Running   0          43h
          machine-config-server-hklrm                  1/1     Running   0          43h
          machine-config-server-k9rtx                  1/1     Running   0          43h

          The names for the config daemon pods are in the following format: machine-config-daemon-<seq>. The <seq> value is a random five character alphanumeric sequence.

       2. To display the pod log for each machine config daemon pod shown in the previous output, enter the following command:

          $ oc logs <pod> -n openshift-machine-config-operator

          where pod is the name of a machine config daemon pod.

       3. Resolve any errors in the logs shown by the output from the previous command.

    4. To confirm that your pods are not in an error state, enter the following command:

       $ oc get pods --all-namespaces -o wide --sort-by='{.spec.nodeName}'

       If pods on a node are in an error state, reboot that node.

11. Complete the following steps only if the migration succeeds and your cluster is in a good state:

    1. To remove the migration configuration from the Cluster Network Operator configuration object, enter the following command:

       $ oc patch Network.operator.openshift.io cluster --type='merge' \
         --patch '{ "spec": { "migration": null } }'

    2. To remove the OVN-Kubernetes configuration, enter the following command:

       $ oc patch Network.operator.openshift.io cluster --type='merge' \
         --patch '{ "spec": { "defaultNetwork": { "ovnKubernetesConfig":null } } }'

    3. To remove the OVN-Kubernetes network provider namespace, enter the following command:

       $ oc delete namespace openshift-ovn-kubernetes

Procedure

1. To initiate the rollback to OpenShift SDN, enter the following command:

   $ oc patch Network.config.openshift.io cluster --type='merge' --patch '{"metadata":{"annotations":{"network.openshift.io/network-type-migration":""}},"spec":{"networkType":"OpenShiftSDN"}}'

2. To watch the progress of your migration, enter the following command:

   $ watch -n1 'oc get network.config/cluster -o json | jq ".status.conditions[]|\"\\(.type) \\(.status) \\(.reason) \\(.message)\""  -r | column --table --table-columns NAME,STATUS,REASON,MESSAGE --table-columns-limit 4; echo; oc get mcp -o wide; echo; oc get node -o "custom-columns=NAME:metadata.name,STATE:metadata.annotations.machineconfiguration\\.openshift\\.io/state,DESIRED:metadata.annotations.machineconfiguration\\.openshift\\.io/desiredConfig,CURRENT:metadata.annotations.machineconfiguration\\.openshift\\.io/currentConfig,REASON:metadata.annotations.machineconfiguration\\.openshift\\.io/reason"'

   The command prints the following information every second:

   • The conditions on the status of the network.config.openshift.io/cluster object, reporting the progress of the migration.
   • The status of different nodes with respect to the machine-config-operator resource, including whether they are upgrading or have been upgraded, as well as their current and desired configurations.

3. After the rollback procedure has completed, enter the following command to remove the network.openshift.io/network-type-migration= annotation from the network.config custom resource:

   $ oc annotate network.config cluster network.openshift.io/network-type-migration-

Procedure

1. To specify IPv6 address blocks for cluster and service networks, create a YAML configuration patch file that has a similar configuration to the following example:

   - op: add
     path: /spec/clusterNetwork/-
     value: 1
       cidr: fd01::/48
       hostPrefix: 64
   - op: add
     path: /spec/serviceNetwork/-
     value: fd02::/112 2

   1

   Specify an object with the cidr and hostPrefix fields. The host prefix must be 64 or greater. The IPv6 Classless Inter-Domain Routing (CIDR) prefix must be large enough to accommodate the specified host prefix.

   2

   Specify an IPv6 CIDR with a prefix of 112. Kubernetes uses only the lowest 16 bits. For a prefix of 112, IP addresses are assigned from 112 to 128 bits.

2. Patch the cluster network configuration by entering the following command in your CLI:

   $ oc patch network.config.openshift.io cluster \1
     --type='json' --patch-file <file>.yaml

   1

   Where file specifies the name of your created YAML file.

   Example output

   network.config.openshift.io/cluster patched

3. To specify IPv6 VIPs for API and Ingress services for your cluster, create a YAML configuration patch file that has a similar configuration to the following example:

   - op: add
     path: /spec/platformSpec/baremetal/machineNetworks/- 1
     value: fd2e:6f44:5dd8::/64
   - op: add
     path: /spec/platformSpec/baremetal/apiServerInternalIPs/- 2
     value: fd2e:6f44:5dd8::4
   - op: add
     path: /spec/platformSpec/baremetal/ingressIPs/-
     value: fd2e:6f44:5dd8::5

   1

   Ensure that you specify an address block for the machineNetwork network where your machines operate. You must select both API and Ingress IP addresses for the machine network.

   2

   Ensure that you specify each file path according to your platform. The example demonstrates a file path on a bare-metal platform.

4. Patch the cluster’s infrastructure by entering the following command in your CLI:

   $ oc patch infrastructure cluster \1
     --type='json' --patch-file <file>.yaml

   1

   Where file specifies the name of your created YAML file.

   Example output

   infrastructure/cluster patched

Verification

1. Show the cluster network configuration by entering the following command in your CLI:

   $ oc describe network

2. Verify the successful installation of the patch on the network configuration by checking that the cluster network configuration recognizes the IPv6 address blocks that you specified in the YAML file.

   Example output

   # ...
   Status:
     Cluster Network:
       Cidr:               10.128.0.0/14
       Host Prefix:        23
       Cidr:               fd01::/48
       Host Prefix:        64
     Cluster Network MTU:  1400
     Network Type:         OVNKubernetes
     Service Network:
       172.30.0.0/16
       fd02::/112
   # ...

3. Show the cluster infrastructure configuration by entering the following command in your CLI:

   $ oc describe network

4. Verify the successful installation of the patch on the cluster infrastructure by checking that the infrastructure recognizes the IPv6 address blocks that you specified in the YAML file.

   Example output

   # ...
   spec:
   # ...
     platformSpec:
       baremetal:
         apiServerInternalIPs:
         - 192.168.123.5
         - fd2e:6f44:5dd8::4
         ingressIPs:
         - 192.168.123.10
         - fd2e:6f44:5dd8::5
   status:
   # ...
     platformStatus:
       baremetal:
         apiServerInternalIP: 192.168.123.5
         apiServerInternalIPs:
         - 192.168.123.5
         - fd2e:6f44:5dd8::4
         ingressIP: 192.168.123.10
         ingressIPs:
         - 192.168.123.10
         - fd2e:6f44:5dd8::5
   # ...

Procedure

1. Edit the networks.config.openshift.io custom resource (CR) by running the following command:

   $ oc edit networks.config.openshift.io

2. Remove the IPv6 specific configuration that you have added to the cidr and hostPrefix fields in the previous procedure.

Procedure

1. To change the OVN-Kubernetes join subnet, enter the following command:

   $ oc patch network.operator.openshift.io cluster --type='merge' \
     -p='{"spec":{"defaultNetwork":{"ovnKubernetesConfig":
       {"ipv4":{"internalJoinSubnet": "<join_subnet>"},
       "ipv6":{"internalJoinSubnet": "<join_subnet>"}}}}}'

   where:

   <join_subnet>

   Specifies an IP address subnet for internal use by OVN-Kubernetes. The subnet must be larger than the number of nodes in the cluster and it must be large enough to accommodate one IP address per node in the cluster. This subnet cannot overlap with any other subnets used by OpenShift Container Platform or on the host itself. The default value for IPv4 is 100.64.0.0/16 and the default value for IPv6 is fd98::/64.

   Example output

   network.operator.openshift.io/cluster patched

Procedure

1. To change the OVN-Kubernetes transit subnet, enter the following command:

   $ oc patch network.operator.openshift.io cluster --type='merge' \
     -p='{"spec":{"defaultNetwork":{"ovnKubernetesConfig":
       {"ipv4":{"internalTransitSwitchSubnet": "<transit_subnet>"},
       "ipv6":{"internalTransitSwitchSubnet": "<transit_subnet>"}}}}}'

   where:

   <transit_subnet>

   Specifies an IP address subnet for the distributed transit switch that enables east-west traffic. This subnet cannot overlap with any other subnets used by OVN-Kubernetes or on the host itself. The default value for IPv4 is 100.88.0.0/16 and the default value for IPv6 is fd97::/64.

   Example output

   network.operator.openshift.io/cluster patched

Procedure

1. Create a YAML file that contains an AdminPolicyBasedExternalRoute object.

2. To create an admin policy based external route, enter the following command:

   $ oc create -f <file>.yaml

   where:

   <file>

   Specifies the name of the YAML file that you created in the previous step.

   Example output

   adminpolicybasedexternalroute.k8s.ovn.org/default-route-policy created

3. To confirm that the admin policy based external route was created, enter the following command:

   $ oc describe apbexternalroute <name> | tail -n 6

   where:

   <name>

   Specifies the name of the AdminPolicyBasedExternalRoute object.

   Example output

   Status:
     Last Transition Time:  2023-04-24T15:09:01Z
     Messages:
     Configured external gateway IPs: 172.18.0.8
     Status:  Success
   Events:  <none>

Procedure

1. Create an EgressIP object:

   1. Create a <egressips_name>.yaml file where <egressips_name> is the name of the object.

   2. In the file that you created, define an EgressIP object, as in the following example:

      apiVersion: k8s.ovn.org/v1
      kind: EgressIP
      metadata:
        name: egress-project1
      spec:
        egressIPs:
        - 192.168.127.10
        - 192.168.127.11
        namespaceSelector:
          matchLabels:
            env: qa

2. To create the object, enter the following command.

   $ oc apply -f <egressips_name>.yaml 1

   1

   Replace <egressips_name> with the name of the object.

   Example output

   egressips.k8s.ovn.org/<egressips_name> created

3. Optional: Save the <egressips_name>.yaml file so that you can make changes later.

4. Add labels to the namespace that requires egress IP addresses. To add a label to the namespace of an EgressIP object defined in step 1, run the following command:

   $ oc label ns <namespace> env=qa 1

   1

   Replace <namespace> with the namespace that requires egress IP addresses.

Procedure

1. Create an IPAddressPool CR with the desired IP for the service:

   1. Create a file, such as ip-addr-pool.yaml, with content like the following example:

      apiVersion: metallb.io/v1beta1
      kind: IPAddressPool
      metadata:
        name: example-pool
        namespace: metallb-system
      spec:
        addresses:
        - 172.19.0.100/32

   2. Apply the configuration for the IP address pool by running the following command:

      $ oc apply -f ip-addr-pool.yaml

2. Create Service and EgressService CRs:

   1. Create a file, such as service-egress-service.yaml, with content like the following example:

      apiVersion: v1
      kind: Service
      metadata:
        name: example-service
        namespace: example-namespace
        annotations:
          metallb.universe.tf/address-pool: example-pool 1
      spec:
        selector:
          app: example
        ports:
          - name: http
            protocol: TCP
            port: 8080
            targetPort: 8080
        type: LoadBalancer
      ---
      apiVersion: k8s.ovn.org/v1
      kind: EgressService
      metadata:
        name: example-service
        namespace: example-namespace
      spec:
        sourceIPBy: "LoadBalancerIP" 2
        nodeSelector: 3
          matchLabels:
            node-role.kubernetes.io/worker: ""

      1

      The LoadBalancer service uses the IP address assigned by MetalLB from the example-pool IP address pool.

      2

      This example uses the LoadBalancerIP value to assign the ingress IP address of the LoadBalancer service as the source IP address of egress traffic.

      3

      When you specify the LoadBalancerIP value, a single node handles the LoadBalancer service’s traffic. In this example, only nodes with the worker label can be selected to handle the traffic. When a node is selected, OVN-Kubernetes labels the node in the following format egress-service.k8s.ovn.org/<svc-namespace>-<svc-name>: "".

      Note

      If you use the sourceIPBy: "LoadBalancerIP" setting, you must specify the load-balancer node in the BGPAdvertisement custom resource (CR).

   2. Apply the configuration for the service and egress service by running the following command:

      $ oc apply -f service-egress-service.yaml

3. Create a BGPAdvertisement CR to advertise the service:

   1. Create a file, such as service-bgp-advertisement.yaml, with content like the following example:

      apiVersion: metallb.io/v1beta1
      kind: BGPAdvertisement
      metadata:
        name: example-bgp-adv
        namespace: metallb-system
      spec:
        ipAddressPools:
        - example-pool
        nodeSelector:
        - matchLabels:
            egress-service.k8s.ovn.org/example-namespace-example-service: "" 1

      1

      In this example, the EgressService CR configures the source IP address for egress traffic to use the load-balancer service IP address. Therefore, you must specify the load-balancer node for return traffic to use the same return path for the traffic originating from the pod.

Verification

1. Verify that you can access the application endpoint of the pods running behind the MetalLB service by running the following command:

   $ curl <external_ip_address>:<port_number> 1

   1

   Update the external IP address and port number to suit your application endpoint.

2. If you assigned the LoadBalancer service’s ingress IP address as the source IP address for egress traffic, verify this configuration by using tools such as tcpdump to analyze packets received at the external client.

Procedure

1. Create an egress router definition.

2. To ensure that other pods can find the IP address of the egress router pod, create a service that uses the egress router, as in the following example:

   apiVersion: v1
   kind: Service
   metadata:
     name: egress-1
   spec:
     ports:
     - name: web-app
       protocol: TCP
       port: 8080
     type: ClusterIP
     selector:
       app: egress-router-cni 1

   1

   Specify the label for the egress router. The value shown is added by the Cluster Network Operator and is not configurable.

   After you create the service, your pods can connect to the service. The egress router pod redirects traffic to the corresponding port on the destination IP address. The connections originate from the reserved source IP address.

1. View the network attachment definition that the Operator created for the egress router:

   $ oc get network-attachment-definition egress-router-cni-nad

   The name of the network attachment definition is not configurable.

   Example output

   NAME                    AGE
   egress-router-cni-nad   18m

2. View the deployment for the egress router pod:

   $ oc get deployment egress-router-cni-deployment

   The name of the deployment is not configurable.

   Example output

   NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
   egress-router-cni-deployment   1/1     1            1           18m

3. View the status of the egress router pod:

   $ oc get pods -l app=egress-router-cni

   Example output

   NAME                                            READY   STATUS    RESTARTS   AGE
   egress-router-cni-deployment-575465c75c-qkq6m   1/1     Running   0          18m

4. View the logs and the routing table for the egress router pod.

1. Get the node name for the egress router pod:

   $ POD_NODENAME=$(oc get pod -l app=egress-router-cni -o jsonpath="{.items[0].spec.nodeName}")

2. Enter into a debug session on the target node. This step instantiates a debug pod called <node_name>-debug:

   $ oc debug node/$POD_NODENAME

3. Set /host as the root directory within the debug shell. The debug pod mounts the root file system of the host in /host within the pod. By changing the root directory to /host, you can run binaries from the executable paths of the host:

   # chroot /host

4. From within the chroot environment console, display the egress router logs:

   # cat /tmp/egress-router-log

   Example output

   2021-04-26T12:27:20Z [debug] Called CNI ADD
   2021-04-26T12:27:20Z [debug] Gateway: 192.168.12.1
   2021-04-26T12:27:20Z [debug] IP Source Addresses: [192.168.12.99/24]
   2021-04-26T12:27:20Z [debug] IP Destinations: [80 UDP 10.0.0.99/30 8080 TCP 203.0.113.26/30 80 8443 TCP 203.0.113.27/30 443]
   2021-04-26T12:27:20Z [debug] Created macvlan interface
   2021-04-26T12:27:20Z [debug] Renamed macvlan to "net1"
   2021-04-26T12:27:20Z [debug] Adding route to gateway 192.168.12.1 on macvlan interface
   2021-04-26T12:27:20Z [debug] deleted default route {Ifindex: 3 Dst: <nil> Src: <nil> Gw: 10.128.10.1 Flags: [] Table: 254}
   2021-04-26T12:27:20Z [debug] Added new default route with gateway 192.168.12.1
   2021-04-26T12:27:20Z [debug] Added iptables rule: iptables -t nat PREROUTING -i eth0 -p UDP --dport 80 -j DNAT --to-destination 10.0.0.99
   2021-04-26T12:27:20Z [debug] Added iptables rule: iptables -t nat PREROUTING -i eth0 -p TCP --dport 8080 -j DNAT --to-destination 203.0.113.26:80
   2021-04-26T12:27:20Z [debug] Added iptables rule: iptables -t nat PREROUTING -i eth0 -p TCP --dport 8443 -j DNAT --to-destination 203.0.113.27:443
   2021-04-26T12:27:20Z [debug] Added iptables rule: iptables -t nat -o net1 -j SNAT --to-source 192.168.12.99

   The logging file location and logging level are not configurable when you start the egress router by creating an EgressRouter object as described in this procedure.

5. From within the chroot environment console, get the container ID:

   # crictl ps --name egress-router-cni-pod | awk '{print $1}'

   Example output

   CONTAINER
   bac9fae69ddb6

6. Determine the process ID of the container. In this example, the container ID is bac9fae69ddb6:

   # crictl inspect -o yaml bac9fae69ddb6 | grep 'pid:' | awk '{print $2}'

   Example output

   68857

7. Enter the network namespace of the container:

   # nsenter -n -t 68857

8. Display the routing table:

   # ip route

   In the following example output, the net1 network interface is the default route. Traffic for the cluster network uses the eth0 network interface. Traffic for the 192.168.12.0/24 network uses the net1 network interface and originates from the reserved source IP address 192.168.12.99. The pod routes all other traffic to the gateway at IP address 192.168.12.1. Routing for the service network is not shown.

   Example output

   default via 192.168.12.1 dev net1
   10.128.10.0/23 dev eth0 proto kernel scope link src 10.128.10.18
   192.168.12.0/24 dev net1 proto kernel scope link src 192.168.12.99
   192.168.12.1 dev net1

1. Change your current project to the project that you enabled multicast for. Replace <project> with the project name.

   $ oc project <project>

2. Create a pod to act as a multicast receiver:

   $ cat <<EOF| oc create -f -
   apiVersion: v1
   kind: Pod
   metadata:
     name: mlistener
     labels:
       app: multicast-verify
   spec:
     containers:
       - name: mlistener
         image: registry.access.redhat.com/ubi9
         command: ["/bin/sh", "-c"]
         args:
           ["dnf -y install socat hostname && sleep inf"]
         ports:
           - containerPort: 30102
             name: mlistener
             protocol: UDP
   EOF

3. Create a pod to act as a multicast sender:

   $ cat <<EOF| oc create -f -
   apiVersion: v1
   kind: Pod
   metadata:
     name: msender
     labels:
       app: multicast-verify
   spec:
     containers:
       - name: msender
         image: registry.access.redhat.com/ubi9
         command: ["/bin/sh", "-c"]
         args:
           ["dnf -y install socat && sleep inf"]
   EOF

4. In a new terminal window or tab, start the multicast listener.

   1. Get the IP address for the Pod:

      $ POD_IP=$(oc get pods mlistener -o jsonpath='{.status.podIP}')

   2. Start the multicast listener by entering the following command:

      $ oc exec mlistener -i -t -- \
          socat UDP4-RECVFROM:30102,ip-add-membership=224.1.0.1:$POD_IP,fork EXEC:hostname

5. Start the multicast transmitter.

   1. Get the pod network IP address range:

      $ CIDR=$(oc get Network.config.openshift.io cluster \
          -o jsonpath='{.status.clusterNetwork[0].cidr}')

   2. To send a multicast message, enter the following command:

      $ oc exec msender -i -t -- \
          /bin/bash -c "echo | socat STDIO UDP4-DATAGRAM:224.1.0.1:30102,range=$CIDR,ip-multicast-ttl=64"

      If multicast is working, the previous command returns the following output:

      mlistener

Procedure

1. Create a patch file that specifies the network flows collector type and the IP address and port information of the collectors:

   spec:
     exportNetworkFlows:
       netFlow:
         collectors:
           - 192.168.1.99:2056

2. Configure the CNO with the network flows collectors:

   $ oc patch network.operator cluster --type merge -p "$(cat <file_name>.yaml)"

   Example output

   network.operator.openshift.io/cluster patched

1. View the Operator configuration to confirm that the exportNetworkFlows field is configured:

   $ oc get network.operator cluster -o jsonpath="{.spec.exportNetworkFlows}"

   Example output

   {"netFlow":{"collectors":["192.168.1.99:2056"]}}

2. View the network flows configuration in OVS from each node:

   $ for pod in $(oc get pods -n openshift-ovn-kubernetes -l app=ovnkube-node -o jsonpath='{range@.items[*]}{.metadata.name}{"\n"}{end}');
     do ;
       echo;
       echo $pod;
       oc -n openshift-ovn-kubernetes exec -c ovnkube-controller $pod \
         -- bash -c 'for type in ipfix sflow netflow ; do ovs-vsctl find $type ; done';
   done

   Example output

   ovnkube-node-xrn4p
   _uuid               : a4d2aaca-5023-4f3d-9400-7275f92611f9
   active_timeout      : 60
   add_id_to_interface : false
   engine_id           : []
   engine_type         : []
   external_ids        : {}
   targets             : ["192.168.1.99:2056"]
   
   ovnkube-node-z4vq9
   _uuid               : 61d02fdb-9228-4993-8ff5-b27f01a29bd6
   active_timeout      : 60
   add_id_to_interface : false
   engine_id           : []
   engine_type         : []
   external_ids        : {}
   targets             : ["192.168.1.99:2056"]-
   
   ...

Procedure

1. Remove all network flows collectors:

   $ oc patch network.operator cluster --type='json' \
       -p='[{"op":"remove", "path":"/spec/exportNetworkFlows"}]'

   Example output

   network.operator.openshift.io/cluster patched

Procedure

1. To configure the OVN-Kubernetes hybrid network overlay, enter the following command:

   $ oc patch networks.operator.openshift.io cluster --type=merge \
     -p '{
       "spec":{
         "defaultNetwork":{
           "ovnKubernetesConfig":{
             "hybridOverlayConfig":{
               "hybridClusterNetwork":[
                 {
                   "cidr": "<cidr>",
                   "hostPrefix": <prefix>
                 }
               ],
               "hybridOverlayVXLANPort": <overlay_port>
             }
           }
         }
       }
     }'

   where:

   cidr

   Specify the CIDR configuration used for nodes on the additional overlay network. This CIDR cannot overlap with the cluster network CIDR.

   hostPrefix

   Specifies the subnet prefix length to assign to each individual node. For example, if hostPrefix is set to 23, then each node is assigned a /23 subnet out of the given cidr, which allows for 510 (2^(32 - 23) - 2) pod IP addresses. If you are required to provide access to nodes from an external network, configure load balancers and routers to manage the traffic.

   hybridOverlayVXLANPort

   Specify a custom VXLAN port for the additional overlay network. This is required for running Windows nodes in a cluster installed on vSphere, and must not be configured for any other cloud provider. The custom port can be any open port excluding the default 4789 port. For more information on this requirement, see the Microsoft documentation on Pod-to-pod connectivity between hosts is broken.

   Example output

   network.operator.openshift.io/cluster patched

2. To confirm that the configuration is active, enter the following command. It can take several minutes for the update to apply.

   $ oc get network.operator.openshift.io -o jsonpath="{.items[0].spec.defaultNetwork.ovnKubernetesConfig}"