Chapter 19. OVN-Kubernetes network plugin

Procedure

1. Run the following command to get all resources, endpoints, and ConfigMaps in the OVN-Kubernetes project:

   $ oc get all,ep,cm -n openshift-ovn-kubernetes

   Example output

   Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
   NAME                                         READY   STATUS    RESTARTS       AGE
   pod/ovnkube-control-plane-65c6f55656-6d55h   2/2     Running   0              114m
   pod/ovnkube-control-plane-65c6f55656-fd7vw   2/2     Running   2 (104m ago)   114m
   pod/ovnkube-node-bcvts                       8/8     Running   0              113m
   pod/ovnkube-node-drgvv                       8/8     Running   0              113m
   pod/ovnkube-node-f2pxt                       8/8     Running   0              113m
   pod/ovnkube-node-frqsb                       8/8     Running   0              105m
   pod/ovnkube-node-lbxkk                       8/8     Running   0              105m
   pod/ovnkube-node-tt7bx                       8/8     Running   1 (102m ago)   105m
   
   NAME                                   TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)             AGE
   service/ovn-kubernetes-control-plane   ClusterIP   None         <none>        9108/TCP            114m
   service/ovn-kubernetes-node            ClusterIP   None         <none>        9103/TCP,9105/TCP   114m
   
   NAME                          DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE
   daemonset.apps/ovnkube-node   6         6         6       6            6           beta.kubernetes.io/os=linux   114m
   
   NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/ovnkube-control-plane   3/3     3            3           114m
   
   NAME                                               DESIRED   CURRENT   READY   AGE
   replicaset.apps/ovnkube-control-plane-65c6f55656   3         3         3       114m
   
   NAME                                     ENDPOINTS                                               AGE
   endpoints/ovn-kubernetes-control-plane   10.0.0.3:9108,10.0.0.4:9108,10.0.0.5:9108               114m
   endpoints/ovn-kubernetes-node            10.0.0.3:9105,10.0.0.4:9105,10.0.0.5:9105 + 9 more...   114m
   
   NAME                                 DATA   AGE
   configmap/control-plane-status       1      113m
   configmap/kube-root-ca.crt           1      114m
   configmap/openshift-service-ca.crt   1      114m
   configmap/ovn-ca                     1      114m
   configmap/ovnkube-config             1      114m
   configmap/signer-ca                  1      114m

   There is one ovnkube-node pod for each node in the cluster. The ovnkube-config config map has the OpenShift Container Platform OVN-Kubernetes configurations.

2. List all of the containers in the ovnkube-node pods by running the following command:

   $ oc get pods ovnkube-node-bcvts -o jsonpath='{.spec.containers[*].name}' -n openshift-ovn-kubernetes

   Expected output

   ovn-controller ovn-acl-logging kube-rbac-proxy-node kube-rbac-proxy-ovn-metrics northd nbdb sbdb ovnkube-controller

   The ovnkube-node pod is made up of several containers. It is responsible for hosting the northbound database (nbdb container), the southbound database (sbdb container), the north daemon (northd container), ovn-controller and the ovnkube-controller container. The ovnkube-controller container watches for API objects like pods, egress IPs, namespaces, services, endpoints, egress firewall, and network policies. It is also responsible for allocating pod IP from the available subnet pool for that node.

3. List all the containers in the ovnkube-control-plane pods by running the following command:

   $ oc get pods ovnkube-control-plane-65c6f55656-6d55h -o jsonpath='{.spec.containers[*].name}' -n openshift-ovn-kubernetes

   Expected output

   kube-rbac-proxy ovnkube-cluster-manager

   The ovnkube-control-plane pod has a container (ovnkube-cluster-manager) that resides on each OpenShift Container Platform node. The ovnkube-cluster-manager container allocates pod subnet, transit switch subnet IP and join switch subnet IP to each node in the cluster. The kube-rbac-proxy container monitors metrics for the ovnkube-cluster-manager container.

1. List pods by running the following command:

   $ oc get po -n openshift-ovn-kubernetes

   Example output

   NAME                                     READY   STATUS    RESTARTS      AGE
   ovnkube-control-plane-8444dff7f9-4lh9k   2/2     Running   0             27m
   ovnkube-control-plane-8444dff7f9-5rjh9   2/2     Running   0             27m
   ovnkube-node-55xs2                       8/8     Running   0             26m
   ovnkube-node-7r84r                       8/8     Running   0             16m
   ovnkube-node-bqq8p                       8/8     Running   0             17m
   ovnkube-node-mkj4f                       8/8     Running   0             26m
   ovnkube-node-mlr8k                       8/8     Running   0             26m
   ovnkube-node-wqn2m                       8/8     Running   0             16m

2. Optional: To list the pods with node information, run the following command:

   $ oc get pods -n openshift-ovn-kubernetes -owide

   Example output

   NAME                                     READY   STATUS    RESTARTS      AGE   IP           NODE                                       NOMINATED NODE   READINESS GATES
   ovnkube-control-plane-8444dff7f9-4lh9k   2/2     Running   0             27m   10.0.0.3     ci-ln-t487nnb-72292-mdcnq-master-1         <none>           <none>
   ovnkube-control-plane-8444dff7f9-5rjh9   2/2     Running   0             27m   10.0.0.4     ci-ln-t487nnb-72292-mdcnq-master-2         <none>           <none>
   ovnkube-node-55xs2                       8/8     Running   0             26m   10.0.0.4     ci-ln-t487nnb-72292-mdcnq-master-2         <none>           <none>
   ovnkube-node-7r84r                       8/8     Running   0             17m   10.0.128.3   ci-ln-t487nnb-72292-mdcnq-worker-b-wbz7z   <none>           <none>
   ovnkube-node-bqq8p                       8/8     Running   0             17m   10.0.128.2   ci-ln-t487nnb-72292-mdcnq-worker-a-lh7ms   <none>           <none>
   ovnkube-node-mkj4f                       8/8     Running   0             27m   10.0.0.5     ci-ln-t487nnb-72292-mdcnq-master-0         <none>           <none>
   ovnkube-node-mlr8k                       8/8     Running   0             27m   10.0.0.3     ci-ln-t487nnb-72292-mdcnq-master-1         <none>           <none>
   ovnkube-node-wqn2m                       8/8     Running   0             17m   10.0.128.4   ci-ln-t487nnb-72292-mdcnq-worker-c-przlm   <none>           <none>

3. Navigate into a pod to look at the northbound database by running the following command:

   $ oc rsh -c nbdb -n openshift-ovn-kubernetes ovnkube-node-55xs2

4. Run the following command to show all the objects in the northbound database:

   $ ovn-nbctl show

   The output is too long to list here. The list includes the NAT rules, logical switches, load balancers and so on.

   You can narrow down and focus on specific components by using some of the following optional commands:

   1. Run the following command to show the list of logical routers:

      $ oc exec -n openshift-ovn-kubernetes -it ovnkube-node-55xs2 \
      -c northd -- ovn-nbctl lr-list

      Example output

      45339f4f-7d0b-41d0-b5f9-9fca9ce40ce6 (GR_ci-ln-t487nnb-72292-mdcnq-master-2)
      96a0a0f0-e7ed-4fec-8393-3195563de1b8 (ovn_cluster_router)

      Note

      From this output you can see there is router on each node plus an ovn_cluster_router.

   2. Run the following command to show the list of logical switches:

      $ oc exec -n openshift-ovn-kubernetes -it ovnkube-node-55xs2 \
      -c nbdb -- ovn-nbctl ls-list

      Example output

      bdd7dc3d-d848-4a74-b293-cc15128ea614 (ci-ln-t487nnb-72292-mdcnq-master-2)
      b349292d-ee03-4914-935f-1940b6cb91e5 (ext_ci-ln-t487nnb-72292-mdcnq-master-2)
      0aac0754-ea32-4e33-b086-35eeabf0a140 (join)
      992509d7-2c3f-4432-88db-c179e43592e5 (transit_switch)

      Note

      From this output you can see there is an ext switch for each node plus switches with the node name itself and a join switch.

   3. Run the following command to show the list of load balancers:

      $ oc exec -n openshift-ovn-kubernetes -it ovnkube-node-55xs2 \
      -c nbdb -- ovn-nbctl lb-list

      Example output

      UUID                                    LB                  PROTO      VIP                     IPs
      7c84c673-ed2a-4436-9a1f-9bc5dd181eea    Service_default/    tcp        172.30.0.1:443          10.0.0.3:6443,169.254.169.2:6443,10.0.0.5:6443
      4d663fd9-ddc8-4271-b333-4c0e279e20bb    Service_default/    tcp        172.30.0.1:443          10.0.0.3:6443,10.0.0.4:6443,10.0.0.5:6443
      292eb07f-b82f-4962-868a-4f541d250bca    Service_openshif    tcp        172.30.105.247:443      10.129.0.12:8443
      034b5a7f-bb6a-45e9-8e6d-573a82dc5ee3    Service_openshif    tcp        172.30.192.38:443       10.0.0.3:10259,10.0.0.4:10259,10.0.0.5:10259
      a68bb53e-be84-48df-bd38-bdd82fcd4026    Service_openshif    tcp        172.30.161.125:8443     10.129.0.32:8443
      6cc21b3d-2c54-4c94-8ff5-d8e017269c2e    Service_openshif    tcp        172.30.3.144:443        10.129.0.22:8443
      37996ffd-7268-4862-a27f-61cd62e09c32    Service_openshif    tcp        172.30.181.107:443      10.129.0.18:8443
      81d4da3c-f811-411f-ae0c-bc6713d0861d    Service_openshif    tcp        172.30.228.23:443       10.129.0.29:8443
      ac5a4f3b-b6ba-4ceb-82d0-d84f2c41306e    Service_openshif    tcp        172.30.14.240:9443      10.129.0.36:9443
      c88979fb-1ef5-414b-90ac-43b579351ac9    Service_openshif    tcp        172.30.231.192:9001     10.128.0.5:9001,10.128.2.5:9001,10.129.0.5:9001,10.129.2.4:9001,10.130.0.3:9001,10.131.0.3:9001
      fcb0a3fb-4a77-4230-a84a-be45dce757e8    Service_openshif    tcp        172.30.189.92:443       10.130.0.17:8440
      67ef3e7b-ceb9-4bf0-8d96-b43bde4c9151    Service_openshif    tcp        172.30.67.218:443       10.129.0.9:8443
      d0032fba-7d5e-424a-af25-4ab9b5d46e81    Service_openshif    tcp        172.30.102.137:2379     10.0.0.3:2379,10.0.0.4:2379,10.0.0.5:2379
                                                                  tcp        172.30.102.137:9979     10.0.0.3:9979,10.0.0.4:9979,10.0.0.5:9979
      7361c537-3eec-4e6c-bc0c-0522d182abd4    Service_openshif    tcp        172.30.198.215:9001     10.0.0.3:9001,10.0.0.4:9001,10.0.0.5:9001,10.0.128.2:9001,10.0.128.3:9001,10.0.128.4:9001
      0296c437-1259-410b-a6fd-81c310ad0af5    Service_openshif    tcp        172.30.198.215:9001     10.0.0.3:9001,169.254.169.2:9001,10.0.0.5:9001,10.0.128.2:9001,10.0.128.3:9001,10.0.128.4:9001
      5d5679f5-45b8-479d-9f7c-08b123c688b8    Service_openshif    tcp        172.30.38.253:17698     10.128.0.52:17698,10.129.0.84:17698,10.130.0.60:17698
      2adcbab4-d1c9-447d-9573-b5dc9f2efbfa    Service_openshif    tcp        172.30.148.52:443       10.0.0.4:9202,10.0.0.5:9202
                                                                  tcp        172.30.148.52:444       10.0.0.4:9203,10.0.0.5:9203
                                                                  tcp        172.30.148.52:445       10.0.0.4:9204,10.0.0.5:9204
                                                                  tcp        172.30.148.52:446       10.0.0.4:9205,10.0.0.5:9205
      2a33a6d7-af1b-4892-87cc-326a380b809b    Service_openshif    tcp        172.30.67.219:9091      10.129.2.16:9091,10.131.0.16:9091
                                                                  tcp        172.30.67.219:9092      10.129.2.16:9092,10.131.0.16:9092
                                                                  tcp        172.30.67.219:9093      10.129.2.16:9093,10.131.0.16:9093
                                                                  tcp        172.30.67.219:9094      10.129.2.16:9094,10.131.0.16:9094
      f56f59d7-231a-4974-99b3-792e2741ec8d    Service_openshif    tcp        172.30.89.212:443       10.128.0.41:8443,10.129.0.68:8443,10.130.0.44:8443
      08c2c6d7-d217-4b96-b5d8-c80c4e258116    Service_openshif    tcp        172.30.102.137:2379     10.0.0.3:2379,169.254.169.2:2379,10.0.0.5:2379
                                                                  tcp        172.30.102.137:9979     10.0.0.3:9979,169.254.169.2:9979,10.0.0.5:9979
      60a69c56-fc6a-4de6-bd88-3f2af5ba5665    Service_openshif    tcp        172.30.10.193:443       10.129.0.25:8443
      ab1ef694-0826-4671-a22c-565fc2d282ec    Service_openshif    tcp        172.30.196.123:443      10.128.0.33:8443,10.129.0.64:8443,10.130.0.37:8443
      b1fb34d3-0944-4770-9ee3-2683e7a630e2    Service_openshif    tcp        172.30.158.93:8443      10.129.0.13:8443
      95811c11-56e2-4877-be1e-c78ccb3a82a9    Service_openshif    tcp        172.30.46.85:9001       10.130.0.16:9001
      4baba1d1-b873-4535-884c-3f6fc07a50fd    Service_openshif    tcp        172.30.28.87:443        10.129.0.26:8443
      6c2e1c90-f0ca-484e-8a8e-40e71442110a    Service_openshif    udp        172.30.0.10:53          10.128.0.13:5353,10.128.2.6:5353,10.129.0.39:5353,10.129.2.6:5353,10.130.0.11:5353,10.131.0.9:5353

      Note

      From this truncated output you can see there are many OVN-Kubernetes load balancers. Load balancers in OVN-Kubernetes are representations of services.

5. Run the following command to display the options available with the command ovn-nbctl:

   $ oc exec -n openshift-ovn-kubernetes -it ovnkube-node-55xs2 \
   -c nbdb ovn-nbctl --help

1. List the pods by running the following command:

   $ oc get po -n openshift-ovn-kubernetes

   Example output

   NAME                                     READY   STATUS    RESTARTS      AGE
   ovnkube-control-plane-8444dff7f9-4lh9k   2/2     Running   0             27m
   ovnkube-control-plane-8444dff7f9-5rjh9   2/2     Running   0             27m
   ovnkube-node-55xs2                       8/8     Running   0             26m
   ovnkube-node-7r84r                       8/8     Running   0             16m
   ovnkube-node-bqq8p                       8/8     Running   0             17m
   ovnkube-node-mkj4f                       8/8     Running   0             26m
   ovnkube-node-mlr8k                       8/8     Running   0             26m
   ovnkube-node-wqn2m                       8/8     Running   0             16m

2. Optional: To list the pods with node information, run the following command:

   $ oc get pods -n openshift-ovn-kubernetes -owide

   Example output

   NAME                                     READY   STATUS    RESTARTS      AGE   IP           NODE                                       NOMINATED NODE   READINESS GATES
   ovnkube-control-plane-8444dff7f9-4lh9k   2/2     Running   0             27m   10.0.0.3     ci-ln-t487nnb-72292-mdcnq-master-1         <none>           <none>
   ovnkube-control-plane-8444dff7f9-5rjh9   2/2     Running   0             27m   10.0.0.4     ci-ln-t487nnb-72292-mdcnq-master-2         <none>           <none>
   ovnkube-node-55xs2                       8/8     Running   0             26m   10.0.0.4     ci-ln-t487nnb-72292-mdcnq-master-2         <none>           <none>
   ovnkube-node-7r84r                       8/8     Running   0             17m   10.0.128.3   ci-ln-t487nnb-72292-mdcnq-worker-b-wbz7z   <none>           <none>
   ovnkube-node-bqq8p                       8/8     Running   0             17m   10.0.128.2   ci-ln-t487nnb-72292-mdcnq-worker-a-lh7ms   <none>           <none>
   ovnkube-node-mkj4f                       8/8     Running   0             27m   10.0.0.5     ci-ln-t487nnb-72292-mdcnq-master-0         <none>           <none>
   ovnkube-node-mlr8k                       8/8     Running   0             27m   10.0.0.3     ci-ln-t487nnb-72292-mdcnq-master-1         <none>           <none>
   ovnkube-node-wqn2m                       8/8     Running   0             17m   10.0.128.4   ci-ln-t487nnb-72292-mdcnq-worker-c-przlm   <none>           <none>

3. Navigate into a pod to look at the southbound database:

   $ oc rsh -c sbdb -n openshift-ovn-kubernetes ovnkube-node-55xs2

4. Run the following command to show all the objects in the southbound database:

   $ ovn-sbctl show

   Example output

   Chassis "5db31703-35e9-413b-8cdf-69e7eecb41f7"
       hostname: ci-ln-9gp362t-72292-v2p94-worker-a-8bmwz
       Encap geneve
           ip: "10.0.128.4"
           options: {csum="true"}
       Port_Binding tstor-ci-ln-9gp362t-72292-v2p94-worker-a-8bmwz
   Chassis "070debed-99b7-4bce-b17d-17e720b7f8bc"
       hostname: ci-ln-9gp362t-72292-v2p94-worker-b-svmp6
       Encap geneve
           ip: "10.0.128.2"
           options: {csum="true"}
       Port_Binding k8s-ci-ln-9gp362t-72292-v2p94-worker-b-svmp6
       Port_Binding rtoe-GR_ci-ln-9gp362t-72292-v2p94-worker-b-svmp6
       Port_Binding openshift-monitoring_alertmanager-main-1
       Port_Binding rtoj-GR_ci-ln-9gp362t-72292-v2p94-worker-b-svmp6
       Port_Binding etor-GR_ci-ln-9gp362t-72292-v2p94-worker-b-svmp6
       Port_Binding cr-rtos-ci-ln-9gp362t-72292-v2p94-worker-b-svmp6
       Port_Binding openshift-e2e-loki_loki-promtail-qcrcz
       Port_Binding jtor-GR_ci-ln-9gp362t-72292-v2p94-worker-b-svmp6
       Port_Binding openshift-multus_network-metrics-daemon-mkd4t
       Port_Binding openshift-ingress-canary_ingress-canary-xtvj4
       Port_Binding openshift-ingress_router-default-6c76cbc498-pvlqk
       Port_Binding openshift-dns_dns-default-zz582
       Port_Binding openshift-monitoring_thanos-querier-57585899f5-lbf4f
       Port_Binding openshift-network-diagnostics_network-check-target-tn228
       Port_Binding openshift-monitoring_prometheus-k8s-0
       Port_Binding openshift-image-registry_image-registry-68899bd877-xqxjj
   Chassis "179ba069-0af1-401c-b044-e5ba90f60fea"
       hostname: ci-ln-9gp362t-72292-v2p94-master-0
       Encap geneve
           ip: "10.0.0.5"
           options: {csum="true"}
       Port_Binding tstor-ci-ln-9gp362t-72292-v2p94-master-0
   Chassis "68c954f2-5a76-47be-9e84-1cb13bd9dab9"
       hostname: ci-ln-9gp362t-72292-v2p94-worker-c-mjf9w
       Encap geneve
           ip: "10.0.128.3"
           options: {csum="true"}
       Port_Binding tstor-ci-ln-9gp362t-72292-v2p94-worker-c-mjf9w
   Chassis "2de65d9e-9abf-4b6e-a51d-a1e038b4d8af"
       hostname: ci-ln-9gp362t-72292-v2p94-master-2
       Encap geneve
           ip: "10.0.0.4"
           options: {csum="true"}
       Port_Binding tstor-ci-ln-9gp362t-72292-v2p94-master-2
   Chassis "1d371cb8-5e21-44fd-9025-c4b162cc4247"
       hostname: ci-ln-9gp362t-72292-v2p94-master-1
       Encap geneve
           ip: "10.0.0.3"
           options: {csum="true"}
       Port_Binding tstor-ci-ln-9gp362t-72292-v2p94-master-1

   This detailed output shows the chassis and the ports that are attached to the chassis which in this case are all of the router ports and anything that runs like host networking. Any pods communicate out to the wider network using source network address translation (SNAT). Their IP address is translated into the IP address of the node that the pod is running on and then sent out into the network.

   In addition to the chassis information the southbound database has all the logic flows and those logic flows are then sent to the ovn-controller running on each of the nodes. The ovn-controller translates the logic flows into open flow rules and ultimately programs OpenvSwitch so that your pods can then follow open flow rules and make it out of the network.

5. Run the following command to display the options available with the command ovn-sbctl:

   $ oc exec -n openshift-ovn-kubernetes -it ovnkube-node-55xs2 \
   -c sbdb ovn-sbctl --help

Procedure

1. Review the details of the ovnkube-node readiness probe by running the following command:

   $ oc get pods -n openshift-ovn-kubernetes -l app=ovnkube-node \
   -o json | jq '.items[0].spec.containers[] | .name,.readinessProbe'

   The readiness probe for the northbound and southbound database containers in the ovnkube-node pod checks for the health of the databases and the ovnkube-controller container.

   The ovnkube-controller container in the ovnkube-node pod has a readiness probe to verify the presence of the OVN-Kubernetes CNI configuration file, the absence of which would indicate that the pod is not running or is not ready to accept requests to configure pods.

2. Show all events including the probe failures, for the namespace by using the following command:

   $ oc get events -n openshift-ovn-kubernetes

3. Show the events for just a specific pod:

   $ oc describe pod ovnkube-node-9lqfk -n openshift-ovn-kubernetes

4. Show the messages and statuses from the cluster network operator:

   $ oc get co/network -o json | jq '.status.conditions[]'

5. Show the ready status of each container in ovnkube-node pods by running the following script:

   $ for p in $(oc get pods --selector app=ovnkube-node -n openshift-ovn-kubernetes \
   -o jsonpath='{range.items[*]}{" "}{.metadata.name}'); do echo === $p ===;  \
   oc get pods -n openshift-ovn-kubernetes $p -o json | jq '.status.containerStatuses[] | .name, .ready'; \
   done

   Note

   The expectation is all container statuses are reporting as true. Failure of a readiness probe sets the status to false.

Procedure (UI)

1. In the Administrator perspective, select Observe Alerting. The three main pages in the Alerting UI in this perspective are the Alerts, Silences, and Alerting Rules pages.
2. View the rules for OVN-Kubernetes alerts by selecting Observe Alerting Alerting Rules.

Procedure

1. View active or firing alerts by running the following commands.

   1. Set the alert manager route environment variable by running the following command:

      $ ALERT_MANAGER=$(oc get route alertmanager-main -n openshift-monitoring \
      -o jsonpath='{@.spec.host}')

   2. Issue a curl request to the alert manager route API by running the following command, replacing $ALERT_MANAGER with the URL of your Alertmanager instance:

      $ curl -s -k -H "Authorization: Bearer $(oc create token prometheus-k8s -n openshift-monitoring)" https://$ALERT_MANAGER/api/v1/alerts | jq '.data[] | "\(.labels.severity) \(.labels.alertname) \(.labels.pod) \(.labels.container) \(.labels.endpoint) \(.labels.instance)"'

2. View alerting rules by running the following command:

   $ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -s 'http://localhost:9090/api/v1/rules' | jq '.data.groups[].rules[] | select(((.name|contains("ovn")) or (.name|contains("OVN")) or (.name|contains("Ovn")) or (.name|contains("North")) or (.name|contains("South"))) and .type=="alerting")'

Procedure

1. View the log for a specific pod:

   $ oc logs -f <pod_name> -c <container_name> -n <namespace>

   where:

   -f

   Optional: Specifies that the output follows what is being written into the logs.

   <pod_name>

   Specifies the name of the pod.

   <container_name>

   Optional: Specifies the name of a container. When a pod has more than one container, you must specify the container name.

   <namespace>

   Specify the namespace the pod is running in.

   For example:

   $ oc logs ovnkube-node-5dx44 -n openshift-ovn-kubernetes

   $ oc logs -f ovnkube-node-5dx44 -c ovnkube-controller -n openshift-ovn-kubernetes

   The contents of log files are printed out.

2. Examine the most recent entries in all the containers in the ovnkube-node pods:

   $ for p in $(oc get pods --selector app=ovnkube-node -n openshift-ovn-kubernetes \
   -o jsonpath='{range.items[*]}{" "}{.metadata.name}'); \
   do echo === $p ===; for container in $(oc get pods -n openshift-ovn-kubernetes $p \
   -o json | jq -r '.status.containerStatuses[] | .name');do echo ---$container---; \
   oc logs -c $container $p -n openshift-ovn-kubernetes --tail=5; done; done

3. View the last 5 lines of every log in every container in an ovnkube-node pod using the following command:

   $ oc logs -l app=ovnkube-node -n openshift-ovn-kubernetes --all-containers --tail 5

Procedure

1. In the OpenShift Container Platform console, navigate to Workloads Pods or navigate to the pod through the resource you want to investigate.
2. Select the openshift-ovn-kubernetes project from the drop-down menu.
3. Click the name of the pod you want to investigate.
4. Click Logs. By default for the ovnkube-master the logs associated with the northd container are displayed.
5. Use the down-down menu to select logs for each container in turn.

Procedure

1. To list the current PodNetworkConnectivityCheck objects, enter the following command:

   $ oc get podnetworkconnectivitychecks -n openshift-network-diagnostics

2. View the most recent success for each connection object by using the following command:

   $ oc get podnetworkconnectivitychecks -n openshift-network-diagnostics \
   -o json | jq '.items[]| .spec.targetEndpoint,.status.successes[0]'

3. View the most recent failures for each connection object by using the following command:

   $ oc get podnetworkconnectivitychecks -n openshift-network-diagnostics \
   -o json | jq '.items[]| .spec.targetEndpoint,.status.failures[0]'

4. View the most recent outages for each connection object by using the following command:

   $ oc get podnetworkconnectivitychecks -n openshift-network-diagnostics \
   -o json | jq '.items[]| .spec.targetEndpoint,.status.outages[0]'

   The connectivity check controller also logs metrics from these checks into Prometheus.

5. View all the metrics by running the following command:

   $ oc exec prometheus-k8s-0 -n openshift-monitoring -- \
   promtool query instant  http://localhost:9090 \
   '{component="openshift-network-diagnostics"}'

6. View the latency between the source pod and the openshift api service for the last 5 minutes:

   $ oc exec prometheus-k8s-0 -n openshift-monitoring -- \
   promtool query instant  http://localhost:9090 \
   '{component="openshift-network-diagnostics"}'

Procedure

1. Create a pod variable by using the following command:

   $  POD=$(oc get pods -n openshift-ovn-kubernetes -l app=ovnkube-control-plane -o name | head -1 | awk -F '/' '{print $NF}')

2. Run the following command on your local host to copy the binary from the ovnkube-control-plane pods:

   $  oc cp -n openshift-ovn-kubernetes $POD:/usr/bin/ovnkube-trace -c ovnkube-cluster-manager ovnkube-trace

   Note

   If you are using Red Hat Enterprise Linux (RHEL) 8 to run the ovnkube-trace tool, you must copy the file /usr/lib/rhel8/ovnkube-trace to your local host.

3. Make ovnkube-trace executable by running the following command:

   $  chmod +x ovnkube-trace

4. Display the options available with ovnkube-trace by running the following command:

   $  ./ovnkube-trace -help

   Expected output

   Usage of ./ovnkube-trace:
     -addr-family string
       	Address family (ip4 or ip6) to be used for tracing (default "ip4")
     -dst string
       	dest: destination pod name
     -dst-ip string
       	destination IP address (meant for tests to external targets)
     -dst-namespace string
       	k8s namespace of dest pod (default "default")
     -dst-port string
       	dst-port: destination port (default "80")
     -kubeconfig string
       	absolute path to the kubeconfig file
     -loglevel string
       	loglevel: klog level (default "0")
     -ovn-config-namespace string
       	namespace used by ovn-config itself
     -service string
       	service: destination service name
     -skip-detrace
       	skip ovn-detrace command
     -src string
       	src: source pod name
     -src-namespace string
       	k8s namespace of source pod (default "default")
     -tcp
       	use tcp transport protocol
     -udp
       	use udp transport protocol

   The command-line arguments supported are familiar Kubernetes constructs, such as namespaces, pods, services so you do not need to find the MAC address, the IP address of the destination nodes, or the ICMP type.

   The log levels are:

   • 0 (minimal output)
   • 2 (more verbose output showing results of trace commands)
   • 5 (debug output)

Procedure

1. Start a web service in the default namespace by entering the following command:

   $ oc run web --namespace=default --image=quay.io/openshifttest/nginx --labels="app=web" --expose --port=80

2. List the pods running in the openshift-dns namespace:

   oc get pods -n openshift-dns

   Example output

   NAME                  READY   STATUS    RESTARTS   AGE
   dns-default-8s42x     2/2     Running   0          5h8m
   dns-default-mdw6r     2/2     Running   0          4h58m
   dns-default-p8t5h     2/2     Running   0          4h58m
   dns-default-rl6nk     2/2     Running   0          5h8m
   dns-default-xbgqx     2/2     Running   0          5h8m
   dns-default-zv8f6     2/2     Running   0          4h58m
   node-resolver-62jjb   1/1     Running   0          5h8m
   node-resolver-8z4cj   1/1     Running   0          4h59m
   node-resolver-bq244   1/1     Running   0          5h8m
   node-resolver-hc58n   1/1     Running   0          4h59m
   node-resolver-lm6z4   1/1     Running   0          5h8m
   node-resolver-zfx5k   1/1     Running   0          5h

3. Run the following ovnkube-trace command to verify DNS resolution is working:

   $ ./ovnkube-trace \
     -src-namespace default \ 

   1

   
     -src web \ 

   2

   
     -dst-namespace openshift-dns \ 

   3

   
     -dst dns-default-p8t5h \ 

   4

   
     -udp -dst-port 53 \ 

   5

   
     -loglevel 0 

   6

   1

   Namespace of the source pod

   2

   Source pod name

   3

   Namespace of destination pod

   4

   Destination pod name

   5

   Use the udp transport protocol. Port 53 is the port the DNS service uses.

   6

   Set the log level to 0 (0 is minimal and 5 is debug)

   Example output if the src&dst pod lands on the same node

   ovn-trace source pod to destination pod indicates success from web to dns-default-p8t5h
   ovn-trace destination pod to source pod indicates success from dns-default-p8t5h to web
   ovs-appctl ofproto/trace source pod to destination pod indicates success from web to dns-default-p8t5h
   ovs-appctl ofproto/trace destination pod to source pod indicates success from dns-default-p8t5h to web
   ovn-detrace source pod to destination pod indicates success from web to dns-default-p8t5h
   ovn-detrace destination pod to source pod indicates success from dns-default-p8t5h to web

   Example output if the src&dst pod lands on a different node

   ovn-trace source pod to destination pod indicates success from web to dns-default-8s42x
   ovn-trace (remote) source pod to destination pod indicates success from web to dns-default-8s42x
   ovn-trace destination pod to source pod indicates success from dns-default-8s42x to web
   ovn-trace (remote) destination pod to source pod indicates success from dns-default-8s42x to web
   ovs-appctl ofproto/trace source pod to destination pod indicates success from web to dns-default-8s42x
   ovs-appctl ofproto/trace destination pod to source pod indicates success from dns-default-8s42x to web
   ovn-detrace source pod to destination pod indicates success from web to dns-default-8s42x
   ovn-detrace destination pod to source pod indicates success from dns-default-8s42x to web

   The ouput indicates success from the deployed pod to the DNS port and also indicates that it is successful going back in the other direction. So you know bi-directional traffic is supported on UDP port 53 if my web pod wants to do dns resolution from core DNS.

Procedure

1. Create the following YAML that defines a deny-by-default policy to deny ingress from all pods in all namespaces. Save the YAML in the deny-by-default.yaml file:

   kind: NetworkPolicy
   apiVersion: networking.k8s.io/v1
   metadata:
     name: deny-by-default
     namespace: default
   spec:
     podSelector: {}
     ingress: []

2. Apply the policy by entering the following command:

   $ oc apply -f deny-by-default.yaml

   Example output

   networkpolicy.networking.k8s.io/deny-by-default created

3. Start a web service in the default namespace by entering the following command:

   $ oc run web --namespace=default --image=quay.io/openshifttest/nginx --labels="app=web" --expose --port=80

4. Run the following command to create the prod namespace:

   $ oc create namespace prod

5. Run the following command to label the prod namespace:

   $ oc label namespace/prod purpose=production

6. Run the following command to deploy an alpine image in the prod namespace and start a shell:

   $ oc run test-6459 --namespace=prod --rm -i -t --image=alpine -- sh

7. Open another terminal session.

8. In this new terminal session run ovn-trace to verify the failure in communication between the source pod test-6459 running in namespace prod and destination pod running in the default namespace:

   $ ./ovnkube-trace \
    -src-namespace prod \
    -src test-6459 \
    -dst-namespace default \
    -dst web \
    -tcp -dst-port 80 \
    -loglevel 0

   Example output

   ovn-trace source pod to destination pod indicates failure from test-6459 to web

9. Increase the log level to 2 to expose the reason for the failure by running the following command:

   $ ./ovnkube-trace \
    -src-namespace prod \
    -src test-6459 \
    -dst-namespace default \
    -dst web \
    -tcp -dst-port 80 \
    -loglevel 2

   Example output

   ...
   ------------------------------------------------
    3. ls_out_acl_hint (northd.c:7454): !ct.new && ct.est && !ct.rpl && ct_mark.blocked == 0, priority 4, uuid 12efc456
       reg0[8] = 1;
       reg0[10] = 1;
       next;
    5. ls_out_acl_action (northd.c:7835): reg8[30..31] == 0, priority 500, uuid 69372c5d
       reg8[30..31] = 1;
       next(4);
    5. ls_out_acl_action (northd.c:7835): reg8[30..31] == 1, priority 500, uuid 2fa0af89
       reg8[30..31] = 2;
       next(4);
    4. ls_out_acl_eval (northd.c:7691): reg8[30..31] == 2 && reg0[10] == 1 && (outport == @a16982411286042166782_ingressDefaultDeny), priority 2000, uuid 447d0dab
       reg8[17] = 1;
       ct_commit { ct_mark.blocked = 1; }; 

   1

   
       next;
   ...

   1

   Ingress traffic is blocked due to the default deny policy being in place.

10. Create a policy that allows traffic from all pods in a particular namespaces with a label purpose=production. Save the YAML in the web-allow-prod.yaml file:

    kind: NetworkPolicy
    apiVersion: networking.k8s.io/v1
    metadata:
      name: web-allow-prod
      namespace: default
    spec:
      podSelector:
        matchLabels:
          app: web
      policyTypes:
      - Ingress
      ingress:
      - from:
        - namespaceSelector:
            matchLabels:
              purpose: production

11. Apply the policy by entering the following command:

    $ oc apply -f web-allow-prod.yaml

12. Run ovnkube-trace to verify that traffic is now allowed by entering the following command:

    $ ./ovnkube-trace \
     -src-namespace prod \
     -src test-6459 \
     -dst-namespace default \
     -dst web \
     -tcp -dst-port 80 \
     -loglevel 0

    Expected output

    ovn-trace source pod to destination pod indicates success from test-6459 to web
    ovn-trace destination pod to source pod indicates success from web to test-6459
    ovs-appctl ofproto/trace source pod to destination pod indicates success from test-6459 to web
    ovs-appctl ofproto/trace destination pod to source pod indicates success from web to test-6459
    ovn-detrace source pod to destination pod indicates success from test-6459 to web
    ovn-detrace destination pod to source pod indicates success from web to test-6459

13. Run the following command in the shell that was opened in step six to connect nginx to the web-server:

     wget -qO- --timeout=2 http://web.default

    Expected output

    <!DOCTYPE html>
    <html>
    <head>
    <title>Welcome to nginx!</title>
    <style>
      body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
      }
    </style>
    </head>
    <body>
    <h1>Welcome to nginx!</h1>
    <p>If you see this page, the nginx web server is successfully installed and
    working. Further configuration is required.</p>
    
    <p>For online documentation and support please refer to
    <a href="http://nginx.org/">nginx.org</a>.<br/>
    Commercial support is available at
    <a href="http://nginx.com/">nginx.com</a>.</p>
    
    <p><em>Thank you for using nginx.</em></p>
    </body>
    </html>

Procedure

1. Stop all of the machine configuration pools managed by the Machine Config Operator (MCO):

   • Stop the master configuration pool by entering the following command in your CLI:

     $ oc patch MachineConfigPool master --type='merge' --patch \
       '{ "spec": { "paused": true } }'

   • Stop the worker machine configuration pool by entering the following command in your CLI:

     $ oc patch MachineConfigPool worker --type='merge' --patch \
       '{ "spec":{ "paused": true } }'

2. To prepare for the migration, set the migration field to null by entering the following command in your CLI:

   $ oc patch Network.operator.openshift.io cluster --type='merge' \
     --patch '{ "spec": { "migration": null } }'

3. Check that the migration status is empty for the Network.config.openshift.io object by entering the following command in your CLI. Empty command output indicates that the object is not in a migration operation.

   $ oc get Network.config cluster -o jsonpath='{.status.migration}'

4. Apply the patch to the Network.operator.openshift.io object to set the network plugin back to OpenShift SDN by entering the following command in your CLI:

   $ oc patch Network.operator.openshift.io cluster --type='merge' \
     --patch '{ "spec": { "migration": { "networkType": "OpenShiftSDN" } } }'

   Important

   If you applied the patch to the Network.config.openshift.io object before the patch operation finalizes on the Network.operator.openshift.io object, the Cluster Network Operator (CNO) enters into a degradation state and this causes a slight delay until the CNO recovers from the degraded state.

5. Confirm that the migration status of the network plugin for the Network.config.openshift.io cluster object is OpenShiftSDN by entering the following command in your CLI:

   $ oc get Network.config cluster -o jsonpath='{.status.migration.networkType}'

6. Apply the patch to the Network.config.openshift.io object to set the network plugin back to OpenShift SDN by entering the following command in your CLI:

   $ oc patch Network.config.openshift.io cluster --type='merge' \
     --patch '{ "spec": { "networkType": "OpenShiftSDN" } }'

7. Optional: Disable automatic migration of several OVN-Kubernetes capabilities to the OpenShift SDN equivalents:

   • Egress IPs
   • Egress firewall
   • Multicast

   To disable automatic migration of the configuration for any of the previously noted OpenShift SDN features, specify the following keys:

   $ oc patch Network.operator.openshift.io cluster --type='merge' \
     --patch '{
       "spec": {
         "migration": {
           "networkType": "OpenShiftSDN",
           "features": {
             "egressIP": <bool>,
             "egressFirewall": <bool>,
             "multicast": <bool>
           }
         }
       }
     }'

   where:

   bool: Specifies whether to enable migration of the feature. The default is true.

8. Optional: You can customize the following settings for OpenShift SDN to meet your network infrastructure requirements:

   • Maximum transmission unit (MTU)
   • VXLAN port

   To customize either or both of the previously noted settings, customize and enter the following command in your CLI. If you do not need to change the default value, omit the key from the patch.

   $ oc patch Network.operator.openshift.io cluster --type=merge \
     --patch '{
       "spec":{
         "defaultNetwork":{
           "openshiftSDNConfig":{
             "mtu":<mtu>,
             "vxlanPort":<port>
       }}}}'

   mtu

   The MTU for the VXLAN overlay network. This value is normally configured automatically, but if the nodes in your cluster do not all use the same MTU, then you must set this explicitly to 50 less than the smallest node MTU value.

   port

   The UDP port for the VXLAN overlay network. If a value is not specified, the default is 4789. The port cannot be the same as the Geneve port that is used by OVN-Kubernetes. The default value for the Geneve port is 6081.

   Example patch command

   $ oc patch Network.operator.openshift.io cluster --type=merge \
     --patch '{
       "spec":{
         "defaultNetwork":{
           "openshiftSDNConfig":{
             "mtu":1200
       }}}}'

9. Reboot each node in your cluster. You can reboot the nodes in your cluster with either of the following approaches:

   • With the oc rsh command, you can use a bash script similar to the following:

     #!/bin/bash
     readarray -t POD_NODES <<< "$(oc get pod -n openshift-machine-config-operator -o wide| grep daemon|awk '{print $1" "$7}')"
     
     for i in "${POD_NODES[@]}"
     do
       read -r POD NODE <<< "$i"
       until oc rsh -n openshift-machine-config-operator "$POD" chroot /rootfs shutdown -r +1
         do
           echo "cannot reboot node $NODE, retry" && sleep 3
         done
     done

   • With the ssh command, you can use a bash script similar to the following. The script assumes that you have configured sudo to not prompt for a password.

     #!/bin/bash
     
     for ip in $(oc get nodes  -o jsonpath='{.items[*].status.addresses[?(@.type=="InternalIP")].address}')
     do
        echo "reboot node $ip"
        ssh -o StrictHostKeyChecking=no core@$ip sudo shutdown -r -t 3
     done

10. Wait until the Multus daemon set rollout completes. Run the following command to see your rollout status:

    $ oc -n openshift-multus rollout status daemonset/multus

    The name of the Multus pods is in the form of multus-<xxxxx> where <xxxxx> is a random sequence of letters. It might take several moments for the pods to restart.

    Example output

    Waiting for daemon set "multus" rollout to finish: 1 out of 6 new pods have been updated...
    ...
    Waiting for daemon set "multus" rollout to finish: 5 of 6 updated pods are available...
    daemon set "multus" successfully rolled out

11. After the nodes in your cluster have rebooted and the multus pods are rolled out, start all of the machine configuration pools by running the following commands::

    • Start the master configuration pool:

      $ oc patch MachineConfigPool master --type='merge' --patch \
        '{ "spec": { "paused": false } }'

    • Start the worker configuration pool:

      $ oc patch MachineConfigPool worker --type='merge' --patch \
        '{ "spec": { "paused": false } }'

    As the MCO updates machines in each config pool, it reboots each node.

    By default the MCO updates a single machine per pool at a time, so the time that the migration requires to complete grows with the size of the cluster.

12. Confirm the status of the new machine configuration on the hosts:

    1. To list the machine configuration state and the name of the applied machine configuration, enter the following command in your CLI:

       $ oc describe node | egrep "hostname|machineconfig"

       Example output

       kubernetes.io/hostname=master-0
       machineconfiguration.openshift.io/currentConfig: rendered-master-c53e221d9d24e1c8bb6ee89dd3d8ad7b
       machineconfiguration.openshift.io/desiredConfig: rendered-master-c53e221d9d24e1c8bb6ee89dd3d8ad7b
       machineconfiguration.openshift.io/reason:
       machineconfiguration.openshift.io/state: Done

       Verify that the following statements are true:

       • The value of machineconfiguration.openshift.io/state field is Done.
       • The value of the machineconfiguration.openshift.io/currentConfig field is equal to the value of the machineconfiguration.openshift.io/desiredConfig field.

    2. To confirm that the machine config is correct, enter the following command in your CLI:

       $ oc get machineconfig <config_name> -o yaml

       where <config_name> is the name of the machine config from the machineconfiguration.openshift.io/currentConfig field.

13. Confirm that the migration succeeded:

    1. To confirm that the network plugin is OpenShift SDN, enter the following command in your CLI. The value of status.networkType must be OpenShiftSDN.

       $ oc get Network.config/cluster -o jsonpath='{.status.networkType}{"\n"}'

    2. To confirm that the cluster nodes are in the Ready state, enter the following command in your CLI:

       $ oc get nodes

    3. If a node is stuck in the NotReady state, investigate the machine config daemon pod logs and resolve any errors.

       1. To list the pods, enter the following command in your CLI:

          $ oc get pod -n openshift-machine-config-operator

          Example output

          NAME                                         READY   STATUS    RESTARTS   AGE
          machine-config-controller-75f756f89d-sjp8b   1/1     Running   0          37m
          machine-config-daemon-5cf4b                  2/2     Running   0          43h
          machine-config-daemon-7wzcd                  2/2     Running   0          43h
          machine-config-daemon-fc946                  2/2     Running   0          43h
          machine-config-daemon-g2v28                  2/2     Running   0          43h
          machine-config-daemon-gcl4f                  2/2     Running   0          43h
          machine-config-daemon-l5tnv                  2/2     Running   0          43h
          machine-config-operator-79d9c55d5-hth92      1/1     Running   0          37m
          machine-config-server-bsc8h                  1/1     Running   0          43h
          machine-config-server-hklrm                  1/1     Running   0          43h
          machine-config-server-k9rtx                  1/1     Running   0          43h

          The names for the config daemon pods are in the following format: machine-config-daemon-<seq>. The <seq> value is a random five character alphanumeric sequence.

       2. To display the pod log for each machine config daemon pod shown in the previous output, enter the following command in your CLI:

          $ oc logs <pod> -n openshift-machine-config-operator

          where pod is the name of a machine config daemon pod.

       3. Resolve any errors in the logs shown by the output from the previous command.

    4. To confirm that your pods are not in an error state, enter the following command in your CLI:

       $ oc get pods --all-namespaces -o wide --sort-by='{.spec.nodeName}'

       If pods on a node are in an error state, reboot that node.

14. Complete the following steps only if the migration succeeds and your cluster is in a good state:

    1. To remove the migration configuration from the Cluster Network Operator configuration object, enter the following command in your CLI:

       $ oc patch Network.operator.openshift.io cluster --type='merge' \
         --patch '{ "spec": { "migration": null } }'

    2. To remove the OVN-Kubernetes configuration, enter the following command in your CLI:

       $ oc patch Network.operator.openshift.io cluster --type='merge' \
         --patch '{ "spec": { "defaultNetwork": { "ovnKubernetesConfig":null } } }'

    3. To remove the OVN-Kubernetes network provider namespace, enter the following command in your CLI:

       $ oc delete namespace openshift-ovn-kubernetes

Procedure

1. Install the ansible-core package, minimum version 2.15. The following example command shows how to install the ansible-core package on Red Hat Enterprise Linux (RHEL):

   $ sudo dnf install -y ansible-core

2. Create an ansible.cfg file and add information similar to the following example to the file. Ensure that file exists in the same directory as where the ansible-galaxy commands and the playbooks run.

   $ cat << EOF >> ansible.cfg
   [galaxy]
   server_list = automation_hub, validated
   
   [galaxy_server.automation_hub]
   url=https://console.redhat.com/api/automation-hub/content/published/
   auth_url=https://sso.redhat.com/auth/realms/redhat-external/protocol/openid-connect/token
   token=
   
   #[galaxy_server.release_galaxy]
   #url=https://galaxy.ansible.com/
   
   [galaxy_server.validated]
   url=https://console.redhat.com/api/automation-hub/content/validated/
   auth_url=https://sso.redhat.com/auth/realms/redhat-external/protocol/openid-connect/token
   token=
   EOF

3. From the Ansible Automation Platform web console, go to the Connect to Hub page and complete the following steps:

   1. In the Offline token section of the page, click the Load token button.
   2. After the token loads, click the Copy to clipboard icon.
   3. Open the ansible.cfg file and paste the API token in the token= parameter. The API token is required for authenticating against the server URL specified in the ansible.cfg file.

4. Install the network.offline_migration_sdn_to_ovnk Ansible collection by entering the following ansible-galaxy command:

   $ ansible-galaxy collection install network.offline_migration_sdn_to_ovnk

5. Verify that the network.offline_migration_sdn_to_ovnk Ansible collection is installed on your system:

   $ ansible-galaxy collection list | grep network.offline_migration_sdn_to_ovnk

   Example output

   network.offline_migration_sdn_to_ovnk   1.0.2

   The network.offline_migration_sdn_to_ovnk Ansible collection is saved in the default path of ~/.ansible/collections/ansible_collections/network/offline_migration_sdn_to_ovnk/.

6. Configure rollback features in the playbooks/playbook-migration.yml file:

   # ...
       rollback_disable_auto_migration: true
       rollback_egress_ip: false
       rollback_egress_firewall: false
       rollback_multicast: false
       rollback_mtu: 1400
       rollback_vxlanPort: 4790
   # ...

   rollback_disable_auto_migration

   Disables the auto-migration of OVN-Kubernetes plug-in features to the OpenShift SDN CNI plug-in. If you disable auto-migration of features, you must also set the rollback_egress_ip, rollback_egress_firewall, and rollback_multicast parameters to false. If you need to enable auto-migration of features, set the parameter to false.

   rollback_mtu

   Optional parameter that sets a specific maximum transmission unit (MTU) to your cluster network after the migration process.

   rollback_vxlanPort

   Optional parameter that sets a VXLAN (Virtual Extensible LAN) port for use by OpenShift SDN CNI plug-in. The default value for the parameter is 4790.

7. To run the playbooks/playbook-rollback.yml file, enter the following command:

   $ ansible-playbook -v playbooks/playbook-rollback.yml

Procedure

1. To initiate the rollback to OpenShift SDN, enter the following command:

   $ oc patch Network.config.openshift.io cluster --type='merge' --patch '{"metadata":{"annotations":{"network.openshift.io/network-type-migration":""}},"spec":{"networkType":"OpenShiftSDN"}}'

2. To watch the progress of your migration, enter the following command:

   $ watch -n1 'oc get network.config/cluster -o json | jq ".status.conditions[]|\"\\(.type) \\(.status) \\(.reason) \\(.message)\""  -r | column --table --table-columns NAME,STATUS,REASON,MESSAGE --table-columns-limit 4; echo; oc get mcp -o wide; echo; oc get node -o "custom-columns=NAME:metadata.name,STATE:metadata.annotations.machineconfiguration\\.openshift\\.io/state,DESIRED:metadata.annotations.machineconfiguration\\.openshift\\.io/desiredConfig,CURRENT:metadata.annotations.machineconfiguration\\.openshift\\.io/currentConfig,REASON:metadata.annotations.machineconfiguration\\.openshift\\.io/reason"'

   The command prints the following information every second:

   • The conditions on the status of the network.config.openshift.io/cluster object, reporting the progress of the migration.
   • The status of different nodes with respect to the machine-config-operator resource, including whether they are upgrading or have been upgraded, as well as their current and desired configurations.

3. Complete the following steps only if the migration succeeds and your cluster is in a good state:

   1. Remove the network.openshift.io/network-type-migration= annotation from the network.config custom resource by entering the following command:

      $ oc annotate network.config cluster network.openshift.io/network-type-migration-

   2. Remove the OVN-Kubernetes network provider namespace by entering the following command:

      $ oc delete namespace openshift-ovn-kubernetes

Procedure

1. To specify IPv6 address blocks for cluster and service networks, create a YAML configuration patch file that has a similar configuration to the following example:

   - op: add
     path: /spec/clusterNetwork/-
     value: 

   1

   
       cidr: fd01::/48
       hostPrefix: 64
   - op: add
     path: /spec/serviceNetwork/-
     value: fd02::/112 

   2

   1

   Specify an object with the cidr and hostPrefix parameters. The host prefix must be 64 or greater. The IPv6 Classless Inter-Domain Routing (CIDR) prefix must be large enough to accommodate the specified host prefix.

   2

   Specify an IPv6 CIDR with a prefix of 112. Kubernetes uses only the lowest 16 bits. For a prefix of 112, IP addresses are assigned from 112 to 128 bits.

2. Patch the cluster network configuration by entering the following command in your CLI:

   $ oc patch network.config.openshift.io cluster \

   1

   
     --type='json' --patch-file <file>.yaml

   1

   Where file specifies the name of your created YAML file.

   Example output

   network.config.openshift.io/cluster patched

3. On installer-provisioned infrastructure where you added IPv6 VIPs for API and Ingress services, complete the following steps:

   1. Specify IPv6 VIPs for API and Ingress services for your cluster. Create a YAML configuration patch file that has a similar configuration to the following example:

      - op: add
        path: /spec/platformSpec/baremetal/machineNetworks/- 

      1

      
        value: fd2e:6f44:5dd8::/64
      - op: add
        path: /spec/platformSpec/baremetal/apiServerInternalIPs/- 

      2

      
        value: fd2e:6f44:5dd8::4
      - op: add
        path: /spec/platformSpec/baremetal/ingressIPs/-
        value: fd2e:6f44:5dd8::5

      1

      Ensure that you specify an address block for the machineNetwork network where your machines operate. You must select both API and Ingress IP addresses for the machine network.

      2

      Ensure that you specify each file path according to your platform. The example demonstrates a file path on a bare-metal platform.

   2. Patch the infrastructure by entering the following command in your CLI:

      $ oc patch infrastructure cluster \
        --type='json' --patch-file <file>.yaml

      Where:

      <file>

      Specifies the name of your created YAML file.

      Example output

      infrastructure/cluster patched

Verification

1. Show the cluster network configuration by entering the following command in your CLI:

   $ oc describe network

2. Verify the successful installation of the patch on the network configuration by checking that the cluster network configuration recognizes the IPv6 address blocks that you specified in the YAML file.

   Example output

   # ...
   Status:
     Cluster Network:
       Cidr:               10.128.0.0/14
       Host Prefix:        23
       Cidr:               fd01::/48
       Host Prefix:        64
     Cluster Network MTU:  1400
     Network Type:         OVNKubernetes
     Service Network:
       172.30.0.0/16
       fd02::/112
   # ...

3. Complete the following additional tasks for a cluster that runs on installer-provisioned infrastructure:

   1. Show the cluster infrastructure configuration by entering the following command in your CLI:

      $ oc describe infrastructure

   2. Verify the successful installation of the patch on the cluster infrastructure by checking that the infrastructure recognizes the IPv6 address blocks that you specified in the YAML file.

      Example output

      # ...
      spec:
      # ...
        platformSpec:
          baremetal:
            apiServerInternalIPs:
            - 192.168.123.5
            - fd2e:6f44:5dd8::4
            ingressIPs:
            - 192.168.123.10
            - fd2e:6f44:5dd8::5
      status:
      # ...
        platformStatus:
          baremetal:
            apiServerInternalIP: 192.168.123.5
            apiServerInternalIPs:
            - 192.168.123.5
            - fd2e:6f44:5dd8::4
            ingressIP: 192.168.123.10
            ingressIPs:
            - 192.168.123.10
            - fd2e:6f44:5dd8::5
      # ...

Procedure

1. Edit the networks.config.openshift.io custom resource (CR) by running the following command:

   $ oc edit networks.config.openshift.io

2. Remove the IPv4 or IPv6 configuration that you added to the cidr and the hostPrefix parameters from completing the "Converting to a dual-stack cluster network " procedure steps.

Procedure

1. Back up the existing network configuration by running the following command:

   $ oc get network.operator cluster -o yaml > network-config-backup.yaml

2. Set the routingViaHost parameter to true by entering the following command. Egress traffic then gets routed through a specific gateway according to the routes that you configured on the node.

   $ oc patch networks.operator.openshift.io cluster --type=merge -p '{"spec":{"defaultNetwork":{"ovnKubernetesConfig":{"gatewayConfig":{"routingViaHost": true}}}}}'

3. Verify the correct application of the routingViaHost=true configuration by running the following command:

   $ oc get networks.operator.openshift.io cluster -o yaml | grep -A 5 "gatewayConfig"

   Example output

   apiVersion: operator.openshift.io/v1
   kind: Network
   metadata:
     name: cluster
   # ...
   gatewayConfig:
           ipv4: {}
           ipv6: {}
           routingViaHost: true 

   1

   
         genevePort: 6081
         ipsecConfig:
   # ...

   1

   A value of true means that egress traffic gets routed through a specific local gateway on the node that hosts the pod. A value of false for the parameter means that a group of nodes share a single gateway so traffic does not get routed through a single host.

4. Optional: Enable IP forwarding globally by running the following command:

   $ oc patch network.operator cluster --type=merge -p '{"spec":{"defaultNetwork":{"ovnKubernetesConfig":{"gatewayConfig":{"ipForwarding": "Global"}}}}}'

   1. Verify that the ipForwarding spec has been set to Global by running the following command:

      $ oc get networks.operator.openshift.io cluster -o yaml | grep -A 5 "gatewayConfig"

      Example output

      apiVersion: operator.openshift.io/v1
      kind: Network
      metadata:
        name: cluster
      # ...
      gatewayConfig:
              ipForwarding: Global
              ipv4: {}
              ipv6: {}
              routingViaHost: true
            genevePort: 6081
      # ...

Procedure

1. Create a YAML file that contains an AdminPolicyBasedExternalRoute object.

2. To create an admin policy based external route, enter the following command:

   $ oc create -f <file>.yaml

   where:

   <file>

   Specifies the name of the YAML file that you created in the previous step.

   Example output

   adminpolicybasedexternalroute.k8s.ovn.org/default-route-policy created

3. To confirm that the admin policy based external route was created, enter the following command:

   $ oc describe apbexternalroute <name> | tail -n 6

   where:

   <name>

   Specifies the name of the AdminPolicyBasedExternalRoute object.

   Example output

   Status:
     Last Transition Time:  2023-04-24T15:09:01Z
     Messages:
     Configured external gateway IPs: 172.18.0.8
     Status:  Success
   Events:  <none>