Chapter 24. OVN-Kubernetes network plugin

Procedure

1. Run the following command to get all resources, endpoints, and ConfigMaps in the OVN-Kubernetes project:

   $ oc get all,ep,cm -n openshift-ovn-kubernetes

   Example output

   Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
   NAME                                         READY   STATUS    RESTARTS       AGE
   pod/ovnkube-control-plane-65c6f55656-6d55h   2/2     Running   0              114m
   pod/ovnkube-control-plane-65c6f55656-fd7vw   2/2     Running   2 (104m ago)   114m
   pod/ovnkube-node-bcvts                       8/8     Running   0              113m
   pod/ovnkube-node-drgvv                       8/8     Running   0              113m
   pod/ovnkube-node-f2pxt                       8/8     Running   0              113m
   pod/ovnkube-node-frqsb                       8/8     Running   0              105m
   pod/ovnkube-node-lbxkk                       8/8     Running   0              105m
   pod/ovnkube-node-tt7bx                       8/8     Running   1 (102m ago)   105m
   
   NAME                                   TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)             AGE
   service/ovn-kubernetes-control-plane   ClusterIP   None         <none>        9108/TCP            114m
   service/ovn-kubernetes-node            ClusterIP   None         <none>        9103/TCP,9105/TCP   114m
   
   NAME                          DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE
   daemonset.apps/ovnkube-node   6         6         6       6            6           beta.kubernetes.io/os=linux   114m
   
   NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/ovnkube-control-plane   3/3     3            3           114m
   
   NAME                                               DESIRED   CURRENT   READY   AGE
   replicaset.apps/ovnkube-control-plane-65c6f55656   3         3         3       114m
   
   NAME                                     ENDPOINTS                                               AGE
   endpoints/ovn-kubernetes-control-plane   10.0.0.3:9108,10.0.0.4:9108,10.0.0.5:9108               114m
   endpoints/ovn-kubernetes-node            10.0.0.3:9105,10.0.0.4:9105,10.0.0.5:9105 + 9 more...   114m
   
   NAME                                 DATA   AGE
   configmap/control-plane-status       1      113m
   configmap/kube-root-ca.crt           1      114m
   configmap/openshift-service-ca.crt   1      114m
   configmap/ovn-ca                     1      114m
   configmap/ovnkube-config             1      114m
   configmap/signer-ca                  1      114m

   There is one ovnkube-node pod for each node in the cluster. The ovnkube-config config map has the OpenShift Container Platform OVN-Kubernetes configurations.

2. List all of the containers in the ovnkube-node pods by running the following command:

   $ oc get pods ovnkube-node-bcvts -o jsonpath='{.spec.containers[*].name}' -n openshift-ovn-kubernetes

   Expected output

   ovn-controller ovn-acl-logging kube-rbac-proxy-node kube-rbac-proxy-ovn-metrics northd nbdb sbdb ovnkube-controller

   The ovnkube-node pod is made up of several containers. It is responsible for hosting the northbound database (nbdb container), the southbound database (sbdb container), the north daemon (northd container), ovn-controller and the ovnkube-controller container. The ovnkube-controller container watches for API objects like pods, egress IPs, namespaces, services, endpoints, egress firewall, and network policies. It is also responsible for allocating pod IP from the available subnet pool for that node.

3. List all the containers in the ovnkube-control-plane pods by running the following command:

   $ oc get pods ovnkube-control-plane-65c6f55656-6d55h -o jsonpath='{.spec.containers[*].name}' -n openshift-ovn-kubernetes

   Expected output

   kube-rbac-proxy ovnkube-cluster-manager

   The ovnkube-control-plane pod has a container (ovnkube-cluster-manager) that resides on each OpenShift Container Platform node. The ovnkube-cluster-manager container allocates pod subnet, transit switch subnet IP and join switch subnet IP to each node in the cluster. The kube-rbac-proxy container monitors metrics for the ovnkube-cluster-manager container.

1. List pods by running the following command:

   $ oc get po -n openshift-ovn-kubernetes

   Example output

   NAME                                     READY   STATUS    RESTARTS      AGE
   ovnkube-control-plane-8444dff7f9-4lh9k   2/2     Running   0             27m
   ovnkube-control-plane-8444dff7f9-5rjh9   2/2     Running   0             27m
   ovnkube-node-55xs2                       8/8     Running   0             26m
   ovnkube-node-7r84r                       8/8     Running   0             16m
   ovnkube-node-bqq8p                       8/8     Running   0             17m
   ovnkube-node-mkj4f                       8/8     Running   0             26m
   ovnkube-node-mlr8k                       8/8     Running   0             26m
   ovnkube-node-wqn2m                       8/8     Running   0             16m

2. Optional: To list the pods with node information, run the following command:

   $ oc get pods -n openshift-ovn-kubernetes -owide

   Example output

   NAME                                     READY   STATUS    RESTARTS      AGE   IP           NODE                                       NOMINATED NODE   READINESS GATES
   ovnkube-control-plane-8444dff7f9-4lh9k   2/2     Running   0             27m   10.0.0.3     ci-ln-t487nnb-72292-mdcnq-master-1         <none>           <none>
   ovnkube-control-plane-8444dff7f9-5rjh9   2/2     Running   0             27m   10.0.0.4     ci-ln-t487nnb-72292-mdcnq-master-2         <none>           <none>
   ovnkube-node-55xs2                       8/8     Running   0             26m   10.0.0.4     ci-ln-t487nnb-72292-mdcnq-master-2         <none>           <none>
   ovnkube-node-7r84r                       8/8     Running   0             17m   10.0.128.3   ci-ln-t487nnb-72292-mdcnq-worker-b-wbz7z   <none>           <none>
   ovnkube-node-bqq8p                       8/8     Running   0             17m   10.0.128.2   ci-ln-t487nnb-72292-mdcnq-worker-a-lh7ms   <none>           <none>
   ovnkube-node-mkj4f                       8/8     Running   0             27m   10.0.0.5     ci-ln-t487nnb-72292-mdcnq-master-0         <none>           <none>
   ovnkube-node-mlr8k                       8/8     Running   0             27m   10.0.0.3     ci-ln-t487nnb-72292-mdcnq-master-1         <none>           <none>
   ovnkube-node-wqn2m                       8/8     Running   0             17m   10.0.128.4   ci-ln-t487nnb-72292-mdcnq-worker-c-przlm   <none>           <none>

3. Navigate into a pod to look at the northbound database by running the following command:

   $ oc rsh -c nbdb -n openshift-ovn-kubernetes ovnkube-node-55xs2

4. Run the following command to show all the objects in the northbound database:

   $ ovn-nbctl show

   The output is too long to list here. The list includes the NAT rules, logical switches, load balancers and so on.

   You can narrow down and focus on specific components by using some of the following optional commands:

   1. Run the following command to show the list of logical routers:

      $ oc exec -n openshift-ovn-kubernetes -it ovnkube-node-55xs2 \
      -c northd -- ovn-nbctl lr-list

      Example output

      45339f4f-7d0b-41d0-b5f9-9fca9ce40ce6 (GR_ci-ln-t487nnb-72292-mdcnq-master-2)
      96a0a0f0-e7ed-4fec-8393-3195563de1b8 (ovn_cluster_router)

      Note

      From this output you can see there is router on each node plus an ovn_cluster_router.

   2. Run the following command to show the list of logical switches:

      $ oc exec -n openshift-ovn-kubernetes -it ovnkube-node-55xs2 \
      -c nbdb -- ovn-nbctl ls-list

      Example output

      bdd7dc3d-d848-4a74-b293-cc15128ea614 (ci-ln-t487nnb-72292-mdcnq-master-2)
      b349292d-ee03-4914-935f-1940b6cb91e5 (ext_ci-ln-t487nnb-72292-mdcnq-master-2)
      0aac0754-ea32-4e33-b086-35eeabf0a140 (join)
      992509d7-2c3f-4432-88db-c179e43592e5 (transit_switch)

      Note

      From this output you can see there is an ext switch for each node plus switches with the node name itself and a join switch.

   3. Run the following command to show the list of load balancers:

      $ oc exec -n openshift-ovn-kubernetes -it ovnkube-node-55xs2 \
      -c nbdb -- ovn-nbctl lb-list

      Example output

      UUID                                    LB                  PROTO      VIP                     IPs
      7c84c673-ed2a-4436-9a1f-9bc5dd181eea    Service_default/    tcp        172.30.0.1:443          10.0.0.3:6443,169.254.169.2:6443,10.0.0.5:6443
      4d663fd9-ddc8-4271-b333-4c0e279e20bb    Service_default/    tcp        172.30.0.1:443          10.0.0.3:6443,10.0.0.4:6443,10.0.0.5:6443
      292eb07f-b82f-4962-868a-4f541d250bca    Service_openshif    tcp        172.30.105.247:443      10.129.0.12:8443
      034b5a7f-bb6a-45e9-8e6d-573a82dc5ee3    Service_openshif    tcp        172.30.192.38:443       10.0.0.3:10259,10.0.0.4:10259,10.0.0.5:10259
      a68bb53e-be84-48df-bd38-bdd82fcd4026    Service_openshif    tcp        172.30.161.125:8443     10.129.0.32:8443
      6cc21b3d-2c54-4c94-8ff5-d8e017269c2e    Service_openshif    tcp        172.30.3.144:443        10.129.0.22:8443
      37996ffd-7268-4862-a27f-61cd62e09c32    Service_openshif    tcp        172.30.181.107:443      10.129.0.18:8443
      81d4da3c-f811-411f-ae0c-bc6713d0861d    Service_openshif    tcp        172.30.228.23:443       10.129.0.29:8443
      ac5a4f3b-b6ba-4ceb-82d0-d84f2c41306e    Service_openshif    tcp        172.30.14.240:9443      10.129.0.36:9443
      c88979fb-1ef5-414b-90ac-43b579351ac9    Service_openshif    tcp        172.30.231.192:9001     10.128.0.5:9001,10.128.2.5:9001,10.129.0.5:9001,10.129.2.4:9001,10.130.0.3:9001,10.131.0.3:9001
      fcb0a3fb-4a77-4230-a84a-be45dce757e8    Service_openshif    tcp        172.30.189.92:443       10.130.0.17:8440
      67ef3e7b-ceb9-4bf0-8d96-b43bde4c9151    Service_openshif    tcp        172.30.67.218:443       10.129.0.9:8443
      d0032fba-7d5e-424a-af25-4ab9b5d46e81    Service_openshif    tcp        172.30.102.137:2379     10.0.0.3:2379,10.0.0.4:2379,10.0.0.5:2379
                                                                  tcp        172.30.102.137:9979     10.0.0.3:9979,10.0.0.4:9979,10.0.0.5:9979
      7361c537-3eec-4e6c-bc0c-0522d182abd4    Service_openshif    tcp        172.30.198.215:9001     10.0.0.3:9001,10.0.0.4:9001,10.0.0.5:9001,10.0.128.2:9001,10.0.128.3:9001,10.0.128.4:9001
      0296c437-1259-410b-a6fd-81c310ad0af5    Service_openshif    tcp        172.30.198.215:9001     10.0.0.3:9001,169.254.169.2:9001,10.0.0.5:9001,10.0.128.2:9001,10.0.128.3:9001,10.0.128.4:9001
      5d5679f5-45b8-479d-9f7c-08b123c688b8    Service_openshif    tcp        172.30.38.253:17698     10.128.0.52:17698,10.129.0.84:17698,10.130.0.60:17698
      2adcbab4-d1c9-447d-9573-b5dc9f2efbfa    Service_openshif    tcp        172.30.148.52:443       10.0.0.4:9202,10.0.0.5:9202
                                                                  tcp        172.30.148.52:444       10.0.0.4:9203,10.0.0.5:9203
                                                                  tcp        172.30.148.52:445       10.0.0.4:9204,10.0.0.5:9204
                                                                  tcp        172.30.148.52:446       10.0.0.4:9205,10.0.0.5:9205
      2a33a6d7-af1b-4892-87cc-326a380b809b    Service_openshif    tcp        172.30.67.219:9091      10.129.2.16:9091,10.131.0.16:9091
                                                                  tcp        172.30.67.219:9092      10.129.2.16:9092,10.131.0.16:9092
                                                                  tcp        172.30.67.219:9093      10.129.2.16:9093,10.131.0.16:9093
                                                                  tcp        172.30.67.219:9094      10.129.2.16:9094,10.131.0.16:9094
      f56f59d7-231a-4974-99b3-792e2741ec8d    Service_openshif    tcp        172.30.89.212:443       10.128.0.41:8443,10.129.0.68:8443,10.130.0.44:8443
      08c2c6d7-d217-4b96-b5d8-c80c4e258116    Service_openshif    tcp        172.30.102.137:2379     10.0.0.3:2379,169.254.169.2:2379,10.0.0.5:2379
                                                                  tcp        172.30.102.137:9979     10.0.0.3:9979,169.254.169.2:9979,10.0.0.5:9979
      60a69c56-fc6a-4de6-bd88-3f2af5ba5665    Service_openshif    tcp        172.30.10.193:443       10.129.0.25:8443
      ab1ef694-0826-4671-a22c-565fc2d282ec    Service_openshif    tcp        172.30.196.123:443      10.128.0.33:8443,10.129.0.64:8443,10.130.0.37:8443
      b1fb34d3-0944-4770-9ee3-2683e7a630e2    Service_openshif    tcp        172.30.158.93:8443      10.129.0.13:8443
      95811c11-56e2-4877-be1e-c78ccb3a82a9    Service_openshif    tcp        172.30.46.85:9001       10.130.0.16:9001
      4baba1d1-b873-4535-884c-3f6fc07a50fd    Service_openshif    tcp        172.30.28.87:443        10.129.0.26:8443
      6c2e1c90-f0ca-484e-8a8e-40e71442110a    Service_openshif    udp        172.30.0.10:53          10.128.0.13:5353,10.128.2.6:5353,10.129.0.39:5353,10.129.2.6:5353,10.130.0.11:5353,10.131.0.9:5353

      Note

      From this truncated output you can see there are many OVN-Kubernetes load balancers. Load balancers in OVN-Kubernetes are representations of services.

5. Run the following command to display the options available with the command ovn-nbctl:

   $ oc exec -n openshift-ovn-kubernetes -it ovnkube-node-55xs2 \
   -c nbdb ovn-nbctl --help

1. List the pods by running the following command:

   $ oc get po -n openshift-ovn-kubernetes

   Example output

   NAME                                     READY   STATUS    RESTARTS      AGE
   ovnkube-control-plane-8444dff7f9-4lh9k   2/2     Running   0             27m
   ovnkube-control-plane-8444dff7f9-5rjh9   2/2     Running   0             27m
   ovnkube-node-55xs2                       8/8     Running   0             26m
   ovnkube-node-7r84r                       8/8     Running   0             16m
   ovnkube-node-bqq8p                       8/8     Running   0             17m
   ovnkube-node-mkj4f                       8/8     Running   0             26m
   ovnkube-node-mlr8k                       8/8     Running   0             26m
   ovnkube-node-wqn2m                       8/8     Running   0             16m

2. Optional: To list the pods with node information, run the following command:

   $ oc get pods -n openshift-ovn-kubernetes -owide

   Example output

   NAME                                     READY   STATUS    RESTARTS      AGE   IP           NODE                                       NOMINATED NODE   READINESS GATES
   ovnkube-control-plane-8444dff7f9-4lh9k   2/2     Running   0             27m   10.0.0.3     ci-ln-t487nnb-72292-mdcnq-master-1         <none>           <none>
   ovnkube-control-plane-8444dff7f9-5rjh9   2/2     Running   0             27m   10.0.0.4     ci-ln-t487nnb-72292-mdcnq-master-2         <none>           <none>
   ovnkube-node-55xs2                       8/8     Running   0             26m   10.0.0.4     ci-ln-t487nnb-72292-mdcnq-master-2         <none>           <none>
   ovnkube-node-7r84r                       8/8     Running   0             17m   10.0.128.3   ci-ln-t487nnb-72292-mdcnq-worker-b-wbz7z   <none>           <none>
   ovnkube-node-bqq8p                       8/8     Running   0             17m   10.0.128.2   ci-ln-t487nnb-72292-mdcnq-worker-a-lh7ms   <none>           <none>
   ovnkube-node-mkj4f                       8/8     Running   0             27m   10.0.0.5     ci-ln-t487nnb-72292-mdcnq-master-0         <none>           <none>
   ovnkube-node-mlr8k                       8/8     Running   0             27m   10.0.0.3     ci-ln-t487nnb-72292-mdcnq-master-1         <none>           <none>
   ovnkube-node-wqn2m                       8/8     Running   0             17m   10.0.128.4   ci-ln-t487nnb-72292-mdcnq-worker-c-przlm   <none>           <none>

3. Navigate into a pod to look at the southbound database:

   $ oc rsh -c sbdb -n openshift-ovn-kubernetes ovnkube-node-55xs2

4. Run the following command to show all the objects in the southbound database:

   $ ovn-sbctl show

   Example output

   Chassis "5db31703-35e9-413b-8cdf-69e7eecb41f7"
       hostname: ci-ln-9gp362t-72292-v2p94-worker-a-8bmwz
       Encap geneve
           ip: "10.0.128.4"
           options: {csum="true"}
       Port_Binding tstor-ci-ln-9gp362t-72292-v2p94-worker-a-8bmwz
   Chassis "070debed-99b7-4bce-b17d-17e720b7f8bc"
       hostname: ci-ln-9gp362t-72292-v2p94-worker-b-svmp6
       Encap geneve
           ip: "10.0.128.2"
           options: {csum="true"}
       Port_Binding k8s-ci-ln-9gp362t-72292-v2p94-worker-b-svmp6
       Port_Binding rtoe-GR_ci-ln-9gp362t-72292-v2p94-worker-b-svmp6
       Port_Binding openshift-monitoring_alertmanager-main-1
       Port_Binding rtoj-GR_ci-ln-9gp362t-72292-v2p94-worker-b-svmp6
       Port_Binding etor-GR_ci-ln-9gp362t-72292-v2p94-worker-b-svmp6
       Port_Binding cr-rtos-ci-ln-9gp362t-72292-v2p94-worker-b-svmp6
       Port_Binding openshift-e2e-loki_loki-promtail-qcrcz
       Port_Binding jtor-GR_ci-ln-9gp362t-72292-v2p94-worker-b-svmp6
       Port_Binding openshift-multus_network-metrics-daemon-mkd4t
       Port_Binding openshift-ingress-canary_ingress-canary-xtvj4
       Port_Binding openshift-ingress_router-default-6c76cbc498-pvlqk
       Port_Binding openshift-dns_dns-default-zz582
       Port_Binding openshift-monitoring_thanos-querier-57585899f5-lbf4f
       Port_Binding openshift-network-diagnostics_network-check-target-tn228
       Port_Binding openshift-monitoring_prometheus-k8s-0
       Port_Binding openshift-image-registry_image-registry-68899bd877-xqxjj
   Chassis "179ba069-0af1-401c-b044-e5ba90f60fea"
       hostname: ci-ln-9gp362t-72292-v2p94-master-0
       Encap geneve
           ip: "10.0.0.5"
           options: {csum="true"}
       Port_Binding tstor-ci-ln-9gp362t-72292-v2p94-master-0
   Chassis "68c954f2-5a76-47be-9e84-1cb13bd9dab9"
       hostname: ci-ln-9gp362t-72292-v2p94-worker-c-mjf9w
       Encap geneve
           ip: "10.0.128.3"
           options: {csum="true"}
       Port_Binding tstor-ci-ln-9gp362t-72292-v2p94-worker-c-mjf9w
   Chassis "2de65d9e-9abf-4b6e-a51d-a1e038b4d8af"
       hostname: ci-ln-9gp362t-72292-v2p94-master-2
       Encap geneve
           ip: "10.0.0.4"
           options: {csum="true"}
       Port_Binding tstor-ci-ln-9gp362t-72292-v2p94-master-2
   Chassis "1d371cb8-5e21-44fd-9025-c4b162cc4247"
       hostname: ci-ln-9gp362t-72292-v2p94-master-1
       Encap geneve
           ip: "10.0.0.3"
           options: {csum="true"}
       Port_Binding tstor-ci-ln-9gp362t-72292-v2p94-master-1

   This detailed output shows the chassis and the ports that are attached to the chassis which in this case are all of the router ports and anything that runs like host networking. Any pods communicate out to the wider network using source network address translation (SNAT). Their IP address is translated into the IP address of the node that the pod is running on and then sent out into the network.

   In addition to the chassis information the southbound database has all the logic flows and those logic flows are then sent to the ovn-controller running on each of the nodes. The ovn-controller translates the logic flows into open flow rules and ultimately programs OpenvSwitch so that your pods can then follow open flow rules and make it out of the network.

5. Run the following command to display the options available with the command ovn-sbctl:

   $ oc exec -n openshift-ovn-kubernetes -it ovnkube-node-55xs2 \
   -c sbdb ovn-sbctl --help

Procedure

1. Review the details of the ovnkube-node readiness probe by running the following command:

   $ oc get pods -n openshift-ovn-kubernetes -l app=ovnkube-node \
   -o json | jq '.items[0].spec.containers[] | .name,.readinessProbe'

   The readiness probe for the northbound and southbound database containers in the ovnkube-node pod checks for the health of the databases and the ovnkube-controller container.

   The ovnkube-controller container in the ovnkube-node pod has a readiness probe to verify the presence of the OVN-Kubernetes CNI configuration file, the absence of which would indicate that the pod is not running or is not ready to accept requests to configure pods.

2. Show all events including the probe failures, for the namespace by using the following command:

   $ oc get events -n openshift-ovn-kubernetes

3. Show the events for just a specific pod:

   $ oc describe pod ovnkube-node-9lqfk -n openshift-ovn-kubernetes

4. Show the messages and statuses from the cluster network operator:

   $ oc get co/network -o json | jq '.status.conditions[]'

5. Show the ready status of each container in ovnkube-node pods by running the following script:

   $ for p in $(oc get pods --selector app=ovnkube-node -n openshift-ovn-kubernetes \
   -o jsonpath='{range.items[*]}{" "}{.metadata.name}'); do echo === $p ===;  \
   oc get pods -n openshift-ovn-kubernetes $p -o json | jq '.status.containerStatuses[] | .name, .ready'; \
   done

   Note

   The expectation is all container statuses are reporting as true. Failure of a readiness probe sets the status to false.

Procedure (UI)

1. In the Administrator perspective, select Observe Alerting. The three main pages in the Alerting UI in this perspective are the Alerts, Silences, and Alerting Rules pages.
2. View the rules for OVN-Kubernetes alerts by selecting Observe Alerting Alerting Rules.

Procedure

1. View active or firing alerts by running the following commands.

   1. Set the alert manager route environment variable by running the following command:

      $ ALERT_MANAGER=$(oc get route alertmanager-main -n openshift-monitoring \
      -o jsonpath='{@.spec.host}')

   2. Issue a curl request to the alert manager route API by running the following command, replacing $ALERT_MANAGER with the URL of your Alertmanager instance:

      $ curl -s -k -H "Authorization: Bearer $(oc create token prometheus-k8s -n openshift-monitoring)" https://$ALERT_MANAGER/api/v1/alerts | jq '.data[] | "\(.labels.severity) \(.labels.alertname) \(.labels.pod) \(.labels.container) \(.labels.endpoint) \(.labels.instance)"'

2. View alerting rules by running the following command:

   $ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -s 'http://localhost:9090/api/v1/rules' | jq '.data.groups[].rules[] | select(((.name|contains("ovn")) or (.name|contains("OVN")) or (.name|contains("Ovn")) or (.name|contains("North")) or (.name|contains("South"))) and .type=="alerting")'

Procedure

1. View the log for a specific pod:

   $ oc logs -f <pod_name> -c <container_name> -n <namespace>

   where:

   -f

   Optional: Specifies that the output follows what is being written into the logs.

   <pod_name>

   Specifies the name of the pod.

   <container_name>

   Optional: Specifies the name of a container. When a pod has more than one container, you must specify the container name.

   <namespace>

   Specify the namespace the pod is running in.

   For example:

   $ oc logs ovnkube-node-5dx44 -n openshift-ovn-kubernetes

   $ oc logs -f ovnkube-node-5dx44 -c ovnkube-controller -n openshift-ovn-kubernetes

   The contents of log files are printed out.

2. Examine the most recent entries in all the containers in the ovnkube-node pods:

   $ for p in $(oc get pods --selector app=ovnkube-node -n openshift-ovn-kubernetes \
   -o jsonpath='{range.items[*]}{" "}{.metadata.name}'); \
   do echo === $p ===; for container in $(oc get pods -n openshift-ovn-kubernetes $p \
   -o json | jq -r '.status.containerStatuses[] | .name');do echo ---$container---; \
   oc logs -c $container $p -n openshift-ovn-kubernetes --tail=5; done; done

3. View the last 5 lines of every log in every container in an ovnkube-node pod using the following command:

   $ oc logs -l app=ovnkube-node -n openshift-ovn-kubernetes --all-containers --tail 5

Procedure

1. In the OpenShift Container Platform console, navigate to Workloads Pods or navigate to the pod through the resource you want to investigate.
2. Select the openshift-ovn-kubernetes project from the drop-down menu.
3. Click the name of the pod you want to investigate.
4. Click Logs. By default for the ovnkube-master the logs associated with the northd container are displayed.
5. Use the down-down menu to select logs for each container in turn.

Procedure

1. To list the current PodNetworkConnectivityCheck objects, enter the following command:

   $ oc get podnetworkconnectivitychecks -n openshift-network-diagnostics

2. View the most recent success for each connection object by using the following command:

   $ oc get podnetworkconnectivitychecks -n openshift-network-diagnostics \
   -o json | jq '.items[]| .spec.targetEndpoint,.status.successes[0]'

3. View the most recent failures for each connection object by using the following command:

   $ oc get podnetworkconnectivitychecks -n openshift-network-diagnostics \
   -o json | jq '.items[]| .spec.targetEndpoint,.status.failures[0]'

4. View the most recent outages for each connection object by using the following command:

   $ oc get podnetworkconnectivitychecks -n openshift-network-diagnostics \
   -o json | jq '.items[]| .spec.targetEndpoint,.status.outages[0]'

   The connectivity check controller also logs metrics from these checks into Prometheus.

5. View all the metrics by running the following command:

   $ oc exec prometheus-k8s-0 -n openshift-monitoring -- \
   promtool query instant  http://localhost:9090 \
   '{component="openshift-network-diagnostics"}'

6. View the latency between the source pod and the openshift api service for the last 5 minutes:

   $ oc exec prometheus-k8s-0 -n openshift-monitoring -- \
   promtool query instant  http://localhost:9090 \
   '{component="openshift-network-diagnostics"}'

Procedure

1. Create a pod variable by using the following command:

   $  POD=$(oc get pods -n openshift-ovn-kubernetes -l app=ovnkube-control-plane -o name | head -1 | awk -F '/' '{print $NF}')

2. Run the following command on your local host to copy the binary from the ovnkube-control-plane pods:

   $  oc cp -n openshift-ovn-kubernetes $POD:/usr/bin/ovnkube-trace -c ovnkube-cluster-manager ovnkube-trace

   Note

   If you are using Red Hat Enterprise Linux (RHEL) 8 to run the ovnkube-trace tool, you must copy the file /usr/lib/rhel8/ovnkube-trace to your local host.

3. Make ovnkube-trace executable by running the following command:

   $  chmod +x ovnkube-trace

4. Display the options available with ovnkube-trace by running the following command:

   $  ./ovnkube-trace -help

   Expected output

   Usage of ./ovnkube-trace:
     -addr-family string
       	Address family (ip4 or ip6) to be used for tracing (default "ip4")
     -dst string
       	dest: destination pod name
     -dst-ip string
       	destination IP address (meant for tests to external targets)
     -dst-namespace string
       	k8s namespace of dest pod (default "default")
     -dst-port string
       	dst-port: destination port (default "80")
     -kubeconfig string
       	absolute path to the kubeconfig file
     -loglevel string
       	loglevel: klog level (default "0")
     -ovn-config-namespace string
       	namespace used by ovn-config itself
     -service string
       	service: destination service name
     -skip-detrace
       	skip ovn-detrace command
     -src string
       	src: source pod name
     -src-namespace string
       	k8s namespace of source pod (default "default")
     -tcp
       	use tcp transport protocol
     -udp
       	use udp transport protocol

   The command-line arguments supported are familiar Kubernetes constructs, such as namespaces, pods, services so you do not need to find the MAC address, the IP address of the destination nodes, or the ICMP type.

   The log levels are:

   • 0 (minimal output)
   • 2 (more verbose output showing results of trace commands)
   • 5 (debug output)

Procedure

1. Start a web service in the default namespace by entering the following command:

   $ oc run web --namespace=default --image=quay.io/openshifttest/nginx --labels="app=web" --expose --port=80

2. List the pods running in the openshift-dns namespace:

   oc get pods -n openshift-dns

   Example output

   NAME                  READY   STATUS    RESTARTS   AGE
   dns-default-8s42x     2/2     Running   0          5h8m
   dns-default-mdw6r     2/2     Running   0          4h58m
   dns-default-p8t5h     2/2     Running   0          4h58m
   dns-default-rl6nk     2/2     Running   0          5h8m
   dns-default-xbgqx     2/2     Running   0          5h8m
   dns-default-zv8f6     2/2     Running   0          4h58m
   node-resolver-62jjb   1/1     Running   0          5h8m
   node-resolver-8z4cj   1/1     Running   0          4h59m
   node-resolver-bq244   1/1     Running   0          5h8m
   node-resolver-hc58n   1/1     Running   0          4h59m
   node-resolver-lm6z4   1/1     Running   0          5h8m
   node-resolver-zfx5k   1/1     Running   0          5h

3. Run the following ovnkube-trace command to verify DNS resolution is working:

   $ ./ovnkube-trace \
     -src-namespace default \ 

   1

   
     -src web \ 

   2

   
     -dst-namespace openshift-dns \ 

   3

   
     -dst dns-default-p8t5h \ 

   4

   
     -udp -dst-port 53 \ 

   5

   
     -loglevel 0 

   6

   1

   Namespace of the source pod

   2

   Source pod name

   3

   Namespace of destination pod

   4

   Destination pod name

   5

   Use the udp transport protocol. Port 53 is the port the DNS service uses.

   6

   Set the log level to 0 (0 is minimal and 5 is debug)

   Example output if the src&dst pod lands on the same node:

   ovn-trace source pod to destination pod indicates success from web to dns-default-p8t5h
   ovn-trace destination pod to source pod indicates success from dns-default-p8t5h to web
   ovs-appctl ofproto/trace source pod to destination pod indicates success from web to dns-default-p8t5h
   ovs-appctl ofproto/trace destination pod to source pod indicates success from dns-default-p8t5h to web
   ovn-detrace source pod to destination pod indicates success from web to dns-default-p8t5h
   ovn-detrace destination pod to source pod indicates success from dns-default-p8t5h to web

   Example output if the src&dst pod lands on a different node:

   ovn-trace source pod to destination pod indicates success from web to dns-default-8s42x
   ovn-trace (remote) source pod to destination pod indicates success from web to dns-default-8s42x
   ovn-trace destination pod to source pod indicates success from dns-default-8s42x to web
   ovn-trace (remote) destination pod to source pod indicates success from dns-default-8s42x to web
   ovs-appctl ofproto/trace source pod to destination pod indicates success from web to dns-default-8s42x
   ovs-appctl ofproto/trace destination pod to source pod indicates success from dns-default-8s42x to web
   ovn-detrace source pod to destination pod indicates success from web to dns-default-8s42x
   ovn-detrace destination pod to source pod indicates success from dns-default-8s42x to web

   The ouput indicates success from the deployed pod to the DNS port and also indicates that it is successful going back in the other direction. So you know bi-directional traffic is supported on UDP port 53 if my web pod wants to do dns resolution from core DNS.

Procedure

1. Create the following YAML that defines a deny-by-default policy to deny ingress from all pods in all namespaces. Save the YAML in the deny-by-default.yaml file:

   kind: NetworkPolicy
   apiVersion: networking.k8s.io/v1
   metadata:
     name: deny-by-default
     namespace: default
   spec:
     podSelector: {}
     ingress: []

2. Apply the policy by entering the following command:

   $ oc apply -f deny-by-default.yaml

   Example output

   networkpolicy.networking.k8s.io/deny-by-default created

3. Start a web service in the default namespace by entering the following command:

   $ oc run web --namespace=default --image=quay.io/openshifttest/nginx --labels="app=web" --expose --port=80

4. Run the following command to create the prod namespace:

   $ oc create namespace prod

5. Run the following command to label the prod namespace:

   $ oc label namespace/prod purpose=production

6. Run the following command to deploy an alpine image in the prod namespace and start a shell:

   $ oc run test-6459 --namespace=prod --rm -i -t --image=alpine -- sh

7. Open another terminal session.

8. In this new terminal session run ovn-trace to verify the failure in communication between the source pod test-6459 running in namespace prod and destination pod running in the default namespace:

   $ ./ovnkube-trace \
    -src-namespace prod \
    -src test-6459 \
    -dst-namespace default \
    -dst web \
    -tcp -dst-port 80 \
    -loglevel 0

   Example output

   ovn-trace source pod to destination pod indicates failure from test-6459 to web

9. Increase the log level to 2 to expose the reason for the failure by running the following command:

   $ ./ovnkube-trace \
    -src-namespace prod \
    -src test-6459 \
    -dst-namespace default \
    -dst web \
    -tcp -dst-port 80 \
    -loglevel 2

   Example output

   ...
   ------------------------------------------------
    3. ls_out_acl_hint (northd.c:7454): !ct.new && ct.est && !ct.rpl && ct_mark.blocked == 0, priority 4, uuid 12efc456
       reg0[8] = 1;
       reg0[10] = 1;
       next;
    5. ls_out_acl_action (northd.c:7835): reg8[30..31] == 0, priority 500, uuid 69372c5d
       reg8[30..31] = 1;
       next(4);
    5. ls_out_acl_action (northd.c:7835): reg8[30..31] == 1, priority 500, uuid 2fa0af89
       reg8[30..31] = 2;
       next(4);
    4. ls_out_acl_eval (northd.c:7691): reg8[30..31] == 2 && reg0[10] == 1 && (outport == @a16982411286042166782_ingressDefaultDeny), priority 2000, uuid 447d0dab
       reg8[17] = 1;
       ct_commit { ct_mark.blocked = 1; }; 

   1

   
       next;
   ...

   1

   Ingress traffic is blocked due to the default deny policy being in place.

10. Create a policy that allows traffic from all pods in a particular namespaces with a label purpose=production. Save the YAML in the web-allow-prod.yaml file:

    kind: NetworkPolicy
    apiVersion: networking.k8s.io/v1
    metadata:
      name: web-allow-prod
      namespace: default
    spec:
      podSelector:
        matchLabels:
          app: web
      policyTypes:
      - Ingress
      ingress:
      - from:
        - namespaceSelector:
            matchLabels:
              purpose: production

11. Apply the policy by entering the following command:

    $ oc apply -f web-allow-prod.yaml

12. Run ovnkube-trace to verify that traffic is now allowed by entering the following command:

    $ ./ovnkube-trace \
     -src-namespace prod \
     -src test-6459 \
     -dst-namespace default \
     -dst web \
     -tcp -dst-port 80 \
     -loglevel 0

    Expected output

    ovn-trace source pod to destination pod indicates success from test-6459 to web
    ovn-trace destination pod to source pod indicates success from web to test-6459
    ovs-appctl ofproto/trace source pod to destination pod indicates success from test-6459 to web
    ovs-appctl ofproto/trace destination pod to source pod indicates success from web to test-6459
    ovn-detrace source pod to destination pod indicates success from test-6459 to web
    ovn-detrace destination pod to source pod indicates success from web to test-6459

13. Run the following command in the shell that was opened in step six to connect nginx to the web-server:

     wget -qO- --timeout=2 http://web.default

    Expected output

    <!DOCTYPE html>
    <html>
    <head>
    <title>Welcome to nginx!</title>
    <style>
      body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
      }
    </style>
    </head>
    <body>
    <h1>Welcome to nginx!</h1>
    <p>If you see this page, the nginx web server is successfully installed and
    working. Further configuration is required.</p>
    
    <p>For online documentation and support please refer to
    <a href="http://nginx.org/">nginx.org</a>.<br/>
    Commercial support is available at
    <a href="http://nginx.com/">nginx.com</a>.</p>
    
    <p><em>Thank you for using nginx.</em></p>
    </body>
    </html>

Procedure

1. If you did not remove webhooks, set the timeout value for each webhook to 3 seconds by creating a ValidatingWebhookConfiguration custom resource and then specify the timeout value for the timeoutSeconds parameter:

   oc patch ValidatingWebhookConfiguration <webhook_name> --type='json' \ 

   1

   
   -p '[{"op": "replace", "path": "/webhooks/0/timeoutSeconds", "value": 3}]'

   1

   Where <webhook_name> is the name of your webhook.

2. To backup the configuration for the cluster network, enter the following command:

   $ oc get Network.config.openshift.io cluster -o yaml > cluster-openshift-sdn.yaml

3. Verify that the OVN_SDN_MIGRATION_TIMEOUT environment variable is set and is equal to 0s by running the following command:

   #!/bin/bash
   
   if [ -n "$OVN_SDN_MIGRATION_TIMEOUT" ] && [ "$OVN_SDN_MIGRATION_TIMEOUT" = "0s" ]; then
       unset OVN_SDN_MIGRATION_TIMEOUT
   fi
   
   #loops the timeout command of the script to repeatedly check the cluster Operators until all are available.
   
   co_timeout=${OVN_SDN_MIGRATION_TIMEOUT:-1200s}
   timeout "$co_timeout" bash <<EOT
   until
     oc wait co --all --for='condition=AVAILABLE=True' --timeout=10s && \
     oc wait co --all --for='condition=PROGRESSING=False' --timeout=10s && \
     oc wait co --all --for='condition=DEGRADED=False' --timeout=10s;
   do
     sleep 10
     echo "Some ClusterOperators Degraded=False,Progressing=True,or Available=False";
   done
   EOT

4. Remove the configuration from the Cluster Network Operator (CNO) configuration object by running the following command:

   $ oc patch Network.operator.openshift.io cluster --type='merge' \
   --patch '{"spec":{"migration":null}}'

5. . Delete the NodeNetworkConfigurationPolicy (NNCP) custom resource (CR) that defines the primary network interface for the OpenShift SDN network plugin by completing the following steps:

   1. Check that the existing NNCP CR bonded the primary interface to your cluster by entering the following command:

      $ oc get nncp

      Example output

      NAME          STATUS      REASON
      bondmaster0   Available   SuccessfullyConfigured

      Network Manager stores the connection profile for the bonded primary interface in the /etc/NetworkManager/system-connections system path.

   2. Remove the NNCP from your cluster:

      $ oc delete nncp <nncp_manifest_filename>

6. To prepare all the nodes for the migration, set the migration field on the CNO configuration object by running the following command:

   $ oc patch Network.operator.openshift.io cluster --type='merge' \
     --patch '{ "spec": { "migration": { "networkType": "OVNKubernetes" } } }'

   Note

   This step does not deploy OVN-Kubernetes immediately. Instead, specifying the migration field triggers the Machine Config Operator (MCO) to apply new machine configs to all the nodes in the cluster in preparation for the OVN-Kubernetes deployment.

   1. Check that the reboot is finished by running the following command:

      $ oc get mcp

   2. Check that all cluster Operators are available by running the following command:

      $ oc get co

   3. Alternatively: You can disable automatic migration of several OpenShift SDN capabilities to the OVN-Kubernetes equivalents:

      • Egress IPs
      • Egress firewall
      • Multicast

      To disable automatic migration of the configuration for any of the previously noted OpenShift SDN features, specify the following keys:

      $ oc patch Network.operator.openshift.io cluster --type='merge' \
        --patch '{
          "spec": {
            "migration": {
              "networkType": "OVNKubernetes",
              "features": {
                "egressIP": <bool>,
                "egressFirewall": <bool>,
                "multicast": <bool>
              }
            }
          }
        }'

      where:

      bool: Specifies whether to enable migration of the feature. The default is true.

7. Optional: You can customize the following settings for OVN-Kubernetes to meet your network infrastructure requirements:

   • Maximum transmission unit (MTU). Consider the following before customizing the MTU for this optional step:

     • If you use the default MTU, and you want to keep the default MTU during migration, this step can be ignored.
     • If you used a custom MTU, and you want to keep the custom MTU during migration, you must declare the custom MTU value in this step.

     • This step does not work if you want to change the MTU value during migration. Instead, you must first follow the instructions for "Changing the cluster MTU". You can then keep the custom MTU value by performing this procedure and declaring the custom MTU value in this step.

       Note

       OpenShift-SDN and OVN-Kubernetes have different overlay overhead. MTU values should be selected by following the guidelines found on the "MTU value selection" page.

   • Geneve (Generic Network Virtualization Encapsulation) overlay network port
   • OVN-Kubernetes IPv4 internal subnet

   To customize either of the previously noted settings, enter and customize the following command. If you do not need to change the default value, omit the key from the patch.

   $ oc patch Network.operator.openshift.io cluster --type=merge \
     --patch '{
       "spec":{
         "defaultNetwork":{
           "ovnKubernetesConfig":{
             "mtu":<mtu>,
             "genevePort":<port>,
             "v4InternalSubnet":"<ipv4_subnet>"
       }}}}'

   where:

   mtu

   The MTU for the Geneve overlay network. This value is normally configured automatically, but if the nodes in your cluster do not all use the same MTU, then you must set this explicitly to 100 less than the smallest node MTU value.

   port

   The UDP port for the Geneve overlay network. If a value is not specified, the default is 6081. The port cannot be the same as the VXLAN port that is used by OpenShift SDN. The default value for the VXLAN port is 4789.

   ipv4_subnet

   An IPv4 address range for internal use by OVN-Kubernetes. You must ensure that the IP address range does not overlap with any other subnet used by your OpenShift Container Platform installation. The IP address range must be larger than the maximum number of nodes that can be added to the cluster. The default value is 100.64.0.0/16.

   Example patch command to update mtu field

   $ oc patch Network.operator.openshift.io cluster --type=merge \
     --patch '{
       "spec":{
         "defaultNetwork":{
           "ovnKubernetesConfig":{
             "mtu":1200
       }}}}'

8. As the MCO updates machines in each machine config pool, it reboots each node one by one. You must wait until all the nodes are updated. Check the machine config pool status by entering the following command:

   $ oc get mcp

   A successfully updated node has the following status: UPDATED=true, UPDATING=false, DEGRADED=false.

   Note

   By default, the MCO updates one machine per pool at a time, causing the total time the migration takes to increase with the size of the cluster.

9. Confirm the status of the new machine configuration on the hosts:

   1. To list the machine configuration state and the name of the applied machine configuration, enter the following command:

      $ oc describe node | egrep "hostname|machineconfig"

      Example output

      kubernetes.io/hostname=master-0
      machineconfiguration.openshift.io/currentConfig: rendered-master-c53e221d9d24e1c8bb6ee89dd3d8ad7b
      machineconfiguration.openshift.io/desiredConfig: rendered-master-c53e221d9d24e1c8bb6ee89dd3d8ad7b
      machineconfiguration.openshift.io/reason:
      machineconfiguration.openshift.io/state: Done

      Verify that the following statements are true:

      • The value of machineconfiguration.openshift.io/state field is Done.
      • The value of the machineconfiguration.openshift.io/currentConfig field is equal to the value of the machineconfiguration.openshift.io/desiredConfig field.

   2. To confirm that the machine config is correct, enter the following command:

      $ oc get machineconfig <config_name> -o yaml | grep ExecStart

      where <config_name> is the name of the machine config from the machineconfiguration.openshift.io/currentConfig field.

      The machine config must include the following update to the systemd configuration:

      ExecStart=/usr/local/bin/configure-ovs.sh OVNKubernetes

   3. If a node is stuck in the NotReady state, investigate the machine config daemon pod logs and resolve any errors.

      1. To list the pods, enter the following command:

         $ oc get pod -n openshift-machine-config-operator

         Example output

         NAME                                         READY   STATUS    RESTARTS   AGE
         machine-config-controller-75f756f89d-sjp8b   1/1     Running   0          37m
         machine-config-daemon-5cf4b                  2/2     Running   0          43h
         machine-config-daemon-7wzcd                  2/2     Running   0          43h
         machine-config-daemon-fc946                  2/2     Running   0          43h
         machine-config-daemon-g2v28                  2/2     Running   0          43h
         machine-config-daemon-gcl4f                  2/2     Running   0          43h
         machine-config-daemon-l5tnv                  2/2     Running   0          43h
         machine-config-operator-79d9c55d5-hth92      1/1     Running   0          37m
         machine-config-server-bsc8h                  1/1     Running   0          43h
         machine-config-server-hklrm                  1/1     Running   0          43h
         machine-config-server-k9rtx                  1/1     Running   0          43h

         The names for the config daemon pods are in the following format: machine-config-daemon-<seq>. The <seq> value is a random five character alphanumeric sequence.

      2. Display the pod log for the first machine config daemon pod shown in the previous output by enter the following command:

         $ oc logs <pod> -n openshift-machine-config-operator

         where pod is the name of a machine config daemon pod.

      3. Resolve any errors in the logs shown by the output from the previous command.

10. To start the migration, configure the OVN-Kubernetes network plugin by using one of the following commands:

    • To specify the network provider without changing the cluster network IP address block, enter the following command:

      $ oc patch Network.config.openshift.io cluster \
        --type='merge' --patch '{ "spec": { "networkType": "OVNKubernetes" } }'

    • To specify a different cluster network IP address block, enter the following command:

      $ oc patch Network.config.openshift.io cluster \
        --type='merge' --patch '{
          "spec": {
            "clusterNetwork": [
              {
                "cidr": "<cidr>",
                "hostPrefix": <prefix>
              }
            ],
            "networkType": "OVNKubernetes"
          }
        }'

      where cidr is a CIDR block and prefix is the slice of the CIDR block apportioned to each node in your cluster. You cannot use any CIDR block that overlaps with the 100.64.0.0/16 CIDR block because the OVN-Kubernetes network provider uses this block internally.

      Important

      You cannot change the service network address block during the migration.

11. Verify that the Multus daemon set rollout is complete before continuing with subsequent steps:

    $ oc -n openshift-multus rollout status daemonset/multus

    The name of the Multus pods is in the form of multus-<xxxxx> where <xxxxx> is a random sequence of letters. It might take several moments for the pods to restart.

    Example output

    Waiting for daemon set "multus" rollout to finish: 1 out of 6 new pods have been updated...
    ...
    Waiting for daemon set "multus" rollout to finish: 5 of 6 updated pods are available...
    daemon set "multus" successfully rolled out

12. To complete changing the network plugin, reboot each node in your cluster. You can reboot the nodes in your cluster with either of the following approaches:

    Important

    The following scripts reboot all of the nodes in the cluster at the same time. This can cause your cluster to be unstable. Another option is to reboot your nodes manually one at a time. Rebooting nodes one-by-one causes considerable downtime in a cluster with many nodes.

    Cluster Operators will not work correctly before you reboot the nodes.

    • With the oc rsh command, you can use a bash script similar to the following:

      #!/bin/bash
      readarray -t POD_NODES <<< "$(oc get pod -n openshift-machine-config-operator -o wide| grep daemon|awk '{print $1" "$7}')"
      
      for i in "${POD_NODES[@]}"
      do
        read -r POD NODE <<< "$i"
        until oc rsh -n openshift-machine-config-operator "$POD" chroot /rootfs shutdown -r +1
          do
            echo "cannot reboot node $NODE, retry" && sleep 3
          done
      done

    • With the ssh command, you can use a bash script similar to the following. The script assumes that you have configured sudo to not prompt for a password.

      #!/bin/bash
      
      for ip in $(oc get nodes  -o jsonpath='{.items[*].status.addresses[?(@.type=="InternalIP")].address}')
      do
         echo "reboot node $ip"
         ssh -o StrictHostKeyChecking=no core@$ip sudo shutdown -r -t 3
      done

13. Confirm that the migration succeeded:

    1. To confirm that the network plugin is OVN-Kubernetes, enter the following command. The value of status.networkType must be OVNKubernetes.

       $ oc get network.config/cluster -o jsonpath='{.status.networkType}{"\n"}'

    2. To confirm that the cluster nodes are in the Ready state, enter the following command:

       $ oc get nodes

    3. To confirm that your pods are not in an error state, enter the following command:

       $ oc get pods --all-namespaces -o wide --sort-by='{.spec.nodeName}'

       If pods on a node are in an error state, reboot that node.

    4. To confirm that all of the cluster Operators are not in an abnormal state, enter the following command:

       $ oc get co

       The status of every cluster Operator must be the following: AVAILABLE="True", PROGRESSING="False", DEGRADED="False". If a cluster Operator is not available or degraded, check the logs for the cluster Operator for more information.

14. Complete the following steps only if the migration succeeds and your cluster is in a good state:

    1. To remove the migration configuration from the CNO configuration object, enter the following command:

       $ oc patch Network.operator.openshift.io cluster --type='merge' \
         --patch '{ "spec": { "migration": null } }'

    2. To remove custom configuration for the OpenShift SDN network provider, enter the following command:

       $ oc patch Network.operator.openshift.io cluster --type='merge' \
         --patch '{ "spec": { "defaultNetwork": { "openshiftSDNConfig": null } } }'

    3. To remove the OpenShift SDN network provider namespace, enter the following command:

       $ oc delete namespace openshift-sdn

    4. After a successful migration operation, remove the network.openshift.io/network-type-migration- annotation from the network.config custom resource by entering the following command:

       $ oc annotate network.config cluster network.openshift.io/network-type-migration-

Procedure

1. Stop all of the machine configuration pools managed by the Machine Config Operator (MCO):

   • Stop the master configuration pool by entering the following command in your CLI:

     $ oc patch MachineConfigPool master --type='merge' --patch \
       '{ "spec": { "paused": true } }'

   • Stop the worker machine configuration pool by entering the following command in your CLI:

     $ oc patch MachineConfigPool worker --type='merge' --patch \
       '{ "spec":{ "paused": true } }'

2. To prepare for the migration, set the migration field to null by entering the following command in your CLI:

   $ oc patch Network.operator.openshift.io cluster --type='merge' \
     --patch '{ "spec": { "migration": null } }'

3. Check that the migration status is empty for the Network.config.openshift.io object by entering the following command in your CLI. Empty command output indicates that the object is not in a migration operation.

   $ oc get Network.config cluster -o jsonpath='{.status.migration}'

4. Apply the patch to the Network.operator.openshift.io object to set the network plugin back to OpenShift SDN by entering the following command in your CLI:

   $ oc patch Network.operator.openshift.io cluster --type='merge' \
     --patch '{ "spec": { "migration": { "networkType": "OpenShiftSDN" } } }'

   Important

   If you applied the patch to the Network.config.openshift.io object before the patch operation finalizes on the Network.operator.openshift.io object, the Cluster Network Operator (CNO) enters into a degradation state and this causes a slight delay until the CNO recovers from the degraded state.

5. Confirm that the migration status of the network plugin for the Network.config.openshift.io cluster object is OpenShiftSDN by entering the following command in your CLI:

   $ oc get Network.config cluster -o jsonpath='{.status.migration.networkType}'

6. Apply the patch to the Network.config.openshift.io object to set the network plugin back to OpenShift SDN by entering the following command in your CLI:

   $ oc patch Network.config.openshift.io cluster --type='merge' \
     --patch '{ "spec": { "networkType": "OpenShiftSDN" } }'

7. Optional: Disable automatic migration of several OVN-Kubernetes capabilities to the OpenShift SDN equivalents:

   • Egress IPs
   • Egress firewall
   • Multicast

   To disable automatic migration of the configuration for any of the previously noted OpenShift SDN features, specify the following keys:

   $ oc patch Network.operator.openshift.io cluster --type='merge' \
     --patch '{
       "spec": {
         "migration": {
           "networkType": "OpenShiftSDN",
           "features": {
             "egressIP": <bool>,
             "egressFirewall": <bool>,
             "multicast": <bool>
           }
         }
       }
     }'

   where:

   bool: Specifies whether to enable migration of the feature. The default is true.

8. Optional: You can customize the following settings for OpenShift SDN to meet your network infrastructure requirements:

   • Maximum transmission unit (MTU)
   • VXLAN port

   To customize either or both of the previously noted settings, customize and enter the following command in your CLI. If you do not need to change the default value, omit the key from the patch.

   $ oc patch Network.operator.openshift.io cluster --type=merge \
     --patch '{
       "spec":{
         "defaultNetwork":{
           "openshiftSDNConfig":{
             "mtu":<mtu>,
             "vxlanPort":<port>
       }}}}'

   mtu

   The MTU for the VXLAN overlay network. This value is normally configured automatically, but if the nodes in your cluster do not all use the same MTU, then you must set this explicitly to 50 less than the smallest node MTU value.

   port

   The UDP port for the VXLAN overlay network. If a value is not specified, the default is 4789. The port cannot be the same as the Geneve port that is used by OVN-Kubernetes. The default value for the Geneve port is 6081.

   Example patch command

   $ oc patch Network.operator.openshift.io cluster --type=merge \
     --patch '{
       "spec":{
         "defaultNetwork":{
           "openshiftSDNConfig":{
             "mtu":1200
       }}}}'

9. Reboot each node in your cluster. You can reboot the nodes in your cluster with either of the following approaches:

   • With the oc rsh command, you can use a bash script similar to the following:

     #!/bin/bash
     readarray -t POD_NODES <<< "$(oc get pod -n openshift-machine-config-operator -o wide| grep daemon|awk '{print $1" "$7}')"
     
     for i in "${POD_NODES[@]}"
     do
       read -r POD NODE <<< "$i"
       until oc rsh -n openshift-machine-config-operator "$POD" chroot /rootfs shutdown -r +1
         do
           echo "cannot reboot node $NODE, retry" && sleep 3
         done
     done

   • With the ssh command, you can use a bash script similar to the following. The script assumes that you have configured sudo to not prompt for a password.

     #!/bin/bash
     
     for ip in $(oc get nodes  -o jsonpath='{.items[*].status.addresses[?(@.type=="InternalIP")].address}')
     do
        echo "reboot node $ip"
        ssh -o StrictHostKeyChecking=no core@$ip sudo shutdown -r -t 3
     done

10. Wait until the Multus daemon set rollout completes. Run the following command to see your rollout status:

    $ oc -n openshift-multus rollout status daemonset/multus

    The name of the Multus pods is in the form of multus-<xxxxx> where <xxxxx> is a random sequence of letters. It might take several moments for the pods to restart.

    Example output

    Waiting for daemon set "multus" rollout to finish: 1 out of 6 new pods have been updated...
    ...
    Waiting for daemon set "multus" rollout to finish: 5 of 6 updated pods are available...
    daemon set "multus" successfully rolled out

11. After the nodes in your cluster have rebooted and the multus pods are rolled out, start all of the machine configuration pools by running the following commands::

    • Start the master configuration pool:

      $ oc patch MachineConfigPool master --type='merge' --patch \
        '{ "spec": { "paused": false } }'

    • Start the worker configuration pool:

      $ oc patch MachineConfigPool worker --type='merge' --patch \
        '{ "spec": { "paused": false } }'

    As the MCO updates machines in each config pool, it reboots each node.

    By default the MCO updates a single machine per pool at a time, so the time that the migration requires to complete grows with the size of the cluster.

12. Confirm the status of the new machine configuration on the hosts:

    1. To list the machine configuration state and the name of the applied machine configuration, enter the following command in your CLI:

       $ oc describe node | egrep "hostname|machineconfig"

       Example output

       kubernetes.io/hostname=master-0
       machineconfiguration.openshift.io/currentConfig: rendered-master-c53e221d9d24e1c8bb6ee89dd3d8ad7b
       machineconfiguration.openshift.io/desiredConfig: rendered-master-c53e221d9d24e1c8bb6ee89dd3d8ad7b
       machineconfiguration.openshift.io/reason:
       machineconfiguration.openshift.io/state: Done

       Verify that the following statements are true:

       • The value of machineconfiguration.openshift.io/state field is Done.
       • The value of the machineconfiguration.openshift.io/currentConfig field is equal to the value of the machineconfiguration.openshift.io/desiredConfig field.

    2. To confirm that the machine config is correct, enter the following command in your CLI:

       $ oc get machineconfig <config_name> -o yaml

       where <config_name> is the name of the machine config from the machineconfiguration.openshift.io/currentConfig field.

13. Confirm that the migration succeeded:

    1. To confirm that the network plugin is OpenShift SDN, enter the following command in your CLI. The value of status.networkType must be OpenShiftSDN.

       $ oc get Network.config/cluster -o jsonpath='{.status.networkType}{"\n"}'

    2. To confirm that the cluster nodes are in the Ready state, enter the following command in your CLI:

       $ oc get nodes

    3. If a node is stuck in the NotReady state, investigate the machine config daemon pod logs and resolve any errors.

       1. To list the pods, enter the following command in your CLI:

          $ oc get pod -n openshift-machine-config-operator

          Example output

          NAME                                         READY   STATUS    RESTARTS   AGE
          machine-config-controller-75f756f89d-sjp8b   1/1     Running   0          37m
          machine-config-daemon-5cf4b                  2/2     Running   0          43h
          machine-config-daemon-7wzcd                  2/2     Running   0          43h
          machine-config-daemon-fc946                  2/2     Running   0          43h
          machine-config-daemon-g2v28                  2/2     Running   0          43h
          machine-config-daemon-gcl4f                  2/2     Running   0          43h
          machine-config-daemon-l5tnv                  2/2     Running   0          43h
          machine-config-operator-79d9c55d5-hth92      1/1     Running   0          37m
          machine-config-server-bsc8h                  1/1     Running   0          43h
          machine-config-server-hklrm                  1/1     Running   0          43h
          machine-config-server-k9rtx                  1/1     Running   0          43h

          The names for the config daemon pods are in the following format: machine-config-daemon-<seq>. The <seq> value is a random five character alphanumeric sequence.

       2. To display the pod log for each machine config daemon pod shown in the previous output, enter the following command in your CLI:

          $ oc logs <pod> -n openshift-machine-config-operator

          where pod is the name of a machine config daemon pod.

       3. Resolve any errors in the logs shown by the output from the previous command.

    4. To confirm that your pods are not in an error state, enter the following command in your CLI:

       $ oc get pods --all-namespaces -o wide --sort-by='{.spec.nodeName}'

       If pods on a node are in an error state, reboot that node.

14. Complete the following steps only if the migration succeeds and your cluster is in a good state:

    1. To remove the migration configuration from the Cluster Network Operator configuration object, enter the following command in your CLI:

       $ oc patch Network.operator.openshift.io cluster --type='merge' \
         --patch '{ "spec": { "migration": null } }'

    2. To remove the OVN-Kubernetes configuration, enter the following command in your CLI:

       $ oc patch Network.operator.openshift.io cluster --type='merge' \
         --patch '{ "spec": { "defaultNetwork": { "ovnKubernetesConfig":null } } }'

    3. To remove the OVN-Kubernetes network provider namespace, enter the following command in your CLI:

       $ oc delete namespace openshift-ovn-kubernetes

Procedure

1. Install the ansible-core package, minimum version 2.15. The following example command shows how to install the ansible-core package on Red Hat Enterprise Linux (RHEL):

   $ sudo dnf install -y ansible-core

2. Create an ansible.cfg file and add information similar to the following example to the file. Ensure that file exists in the same directory as where the ansible-galaxy commands and the playbooks run.

   $ cat << EOF >> ansible.cfg
   [galaxy]
   server_list = automation_hub, validated
   
   [galaxy_server.automation_hub]
   url=https://console.redhat.com/api/automation-hub/content/published/
   auth_url=https://sso.redhat.com/auth/realms/redhat-external/protocol/openid-connect/token
   token=
   
   #[galaxy_server.release_galaxy]
   #url=https://galaxy.ansible.com/
   
   [galaxy_server.validated]
   url=https://console.redhat.com/api/automation-hub/content/validated/
   auth_url=https://sso.redhat.com/auth/realms/redhat-external/protocol/openid-connect/token
   token=
   EOF

3. From the Ansible Automation Platform web console, go to the Connect to Hub page and complete the following steps:

   1. In the Offline token section of the page, click the Load token button.
   2. After the token loads, click the Copy to clipboard icon.
   3. Open the ansible.cfg file and paste the API token in the token= parameter. The API token is required for authenticating against the server URL specified in the ansible.cfg file.

4. Install the network.offline_migration_sdn_to_ovnk Ansible collection by entering the following ansible-galaxy command:

   $ ansible-galaxy collection install network.offline_migration_sdn_to_ovnk

5. Verify that the network.offline_migration_sdn_to_ovnk Ansible collection is installed on your system:

   $ ansible-galaxy collection list | grep network.offline_migration_sdn_to_ovnk

   Example output

   network.offline_migration_sdn_to_ovnk   1.0.2

   The network.offline_migration_sdn_to_ovnk Ansible collection is saved in the default path of ~/.ansible/collections/ansible_collections/network/offline_migration_sdn_to_ovnk/.

6. Configure rollback features in the playbooks/playbook-migration.yml file:

   # ...
       rollback_disable_auto_migration: true
       rollback_egress_ip: false
       rollback_egress_firewall: false
       rollback_multicast: false
       rollback_mtu: 1400
       rollback_vxlanPort: 4790
   # ...

   rollback_disable_auto_migration

   Disables the auto-migration of OVN-Kubernetes plug-in features to the OpenShift SDN CNI plug-in. If you disable auto-migration of features, you must also set the rollback_egress_ip, rollback_egress_firewall, and rollback_multicast parameters to false. If you need to enable auto-migration of features, set the parameter to false.

   rollback_mtu

   Optional parameter that sets a specific maximum transmission unit (MTU) to your cluster network after the migration process.

   rollback_vxlanPort

   Optional parameter that sets a VXLAN (Virtual Extensible LAN) port for use by OpenShift SDN CNI plug-in. The default value for the parameter is 4790.

7. To run the playbooks/playbook-rollback.yml file, enter the following command:

   $ ansible-playbook -v playbooks/playbook-rollback.yml

Procedure

1. Back up the configuration for the cluster network by running the following command:

   $ oc get Network.config.openshift.io cluster -o yaml > cluster-kuryr.yaml

2. To set the CLUSTERID variable, run the following command:

   $ CLUSTERID=$(oc get infrastructure.config.openshift.io cluster -o=jsonpath='{.status.infrastructureName}')

3. To prepare all the nodes for the migration, set the migration field on the Cluster Network Operator configuration object by running the following command:

   $ oc patch Network.operator.openshift.io cluster --type=merge \
       --patch '{"spec": {"migration": {"networkType": "OVNKubernetes"}}}'

   Note

   This step does not deploy OVN-Kubernetes immediately. Specifying the migration field triggers the Machine Config Operator (MCO) to apply new machine configs to all the nodes in the cluster. This prepares the cluster for the OVN-Kubernetes deployment.

4. Optional: Customize the following settings for OVN-Kubernetes for your network infrastructure requirements:

   • Maximum transmission unit (MTU)
   • Geneve (Generic Network Virtualization Encapsulation) overlay network port
   • OVN-Kubernetes IPv4 internal subnet
   • OVN-Kubernetes IPv6 internal subnet

   To customize these settings, enter and customize the following command:

   $ oc patch Network.operator.openshift.io cluster --type=merge \
     --patch '{
       "spec":{
         "defaultNetwork":{
           "ovnKubernetesConfig":{
             "mtu":<mtu>,
             "genevePort":<port>,
             "v4InternalSubnet":"<ipv4_subnet>",
             "v6InternalSubnet":"<ipv6_subnet>"
       }}}}'

   where:

   mtu

   Specifies the MTU for the Geneve overlay network. This value is normally configured automatically, but if the nodes in your cluster do not all use the same MTU, then you must set this explicitly to 100 less than the smallest node MTU value.

   port

   Specifies the UDP port for the Geneve overlay network. If a value is not specified, the default is 6081. The port cannot be the same as the VXLAN port that is used by Kuryr. The default value for the VXLAN port is 4789.

   ipv4_subnet

   Specifies an IPv4 address range for internal use by OVN-Kubernetes. You must ensure that the IP address range does not overlap with any other subnet used by your OpenShift Container Platform installation. The IP address range must be larger than the maximum number of nodes that can be added to the cluster. The default value is 100.64.0.0/16.

   ipv6_subnet

   Specifies an IPv6 address range for internal use by OVN-Kubernetes. You must ensure that the IP address range does not overlap with any other subnet used by your OpenShift Container Platform installation. The IP address range must be larger than the maximum number of nodes that can be added to the cluster. The default value is fd98::/48.

   If you do not need to change the default value, omit the key from the patch.

   Example patch command to update mtu field

   $ oc patch Network.operator.openshift.io cluster --type=merge \
     --patch '{
       "spec":{
         "defaultNetwork":{
           "ovnKubernetesConfig":{
             "mtu":1200
       }}}}'

5. Check the machine config pool status by entering the following command:

   $ oc get mcp

   While the MCO updates machines in each machine config pool, it reboots each node one by one. You must wait until all the nodes are updated before continuing.

   A successfully updated node has the following status: UPDATED=true, UPDATING=false, DEGRADED=false.

   Note

   By default, the MCO updates one machine per pool at a time. Large clusters take more time to migrate than small clusters.

6. Confirm the status of the new machine configuration on the hosts:

   1. To list the machine configuration state and the name of the applied machine configuration, enter the following command:

      $ oc describe node | egrep "hostname|machineconfig"

      Example output

      kubernetes.io/hostname=master-0
      machineconfiguration.openshift.io/currentConfig: rendered-master-c53e221d9d24e1c8bb6ee89dd3d8ad7b 

      1

      
      machineconfiguration.openshift.io/desiredConfig: rendered-master-c53e221d9d24e1c8bb6ee89dd3d8ad7b 

      2

      
      machineconfiguration.openshift.io/reason:
      machineconfiguration.openshift.io/state: Done

   2. Review the output from the previous step. The following statements must be true:

      • The value of machineconfiguration.openshift.io/state field is Done.
      • The value of the machineconfiguration.openshift.io/currentConfig field is equal to the value of the machineconfiguration.openshift.io/desiredConfig field.

   3. To confirm that the machine config is correct, enter the following command:

      $ oc get machineconfig <config_name> -o yaml | grep ExecStart

      where:

      <config_name>

      Specifies the name of the machine config from the machineconfiguration.openshift.io/currentConfig field.

      The machine config must include the following update to the systemd configuration:

      Example output

      ExecStart=/usr/local/bin/configure-ovs.sh OVNKubernetes

   4. If a node is stuck in the NotReady state, investigate the machine config daemon pod logs and resolve any errors:

      1. To list the pods, enter the following command:

         $ oc get pod -n openshift-machine-config-operator

         Example output

         NAME                                         READY   STATUS    RESTARTS   AGE
         machine-config-controller-75f756f89d-sjp8b   1/1     Running   0          37m
         machine-config-daemon-5cf4b                  2/2     Running   0          43h
         machine-config-daemon-7wzcd                  2/2     Running   0          43h
         machine-config-daemon-fc946                  2/2     Running   0          43h
         machine-config-daemon-g2v28                  2/2     Running   0          43h
         machine-config-daemon-gcl4f                  2/2     Running   0          43h
         machine-config-daemon-l5tnv                  2/2     Running   0          43h
         machine-config-operator-79d9c55d5-hth92      1/1     Running   0          37m
         machine-config-server-bsc8h                  1/1     Running   0          43h
         machine-config-server-hklrm                  1/1     Running   0          43h
         machine-config-server-k9rtx                  1/1     Running   0          43h

         The names for the config daemon pods are in the following format: machine-config-daemon-<seq>. The <seq> value is a random five character alphanumeric sequence.

      2. Display the pod log for the first machine config daemon pod shown in the previous output by enter the following command:

         $ oc logs <pod> -n openshift-machine-config-operator

         where:

         <pod>

         Specifies the name of a machine config daemon pod.

      3. Resolve any errors in the logs shown by the output from the previous command.

7. To start the migration, configure the OVN-Kubernetes network plugin by using one of the following commands:

   • To specify the network provider without changing the cluster network IP address block, enter the following command:

     $ oc patch Network.config.openshift.io cluster --type=merge \
         --patch '{"spec": {"networkType": "OVNKubernetes"}}'

   • To specify a different cluster network IP address block, enter the following command:

     $ oc patch Network.config.openshift.io cluster \
       --type='merge' --patch '{
         "spec": {
           "clusterNetwork": [
             {
               "cidr": "<cidr>",
               "hostPrefix": "<prefix>"
             }
           ]
           "networkType": "OVNKubernetes"
         }
       }'

     where:

     <cidr>

     Specifies a CIDR block.

     <prefix>

     Specifies a slice of the CIDR block that is apportioned to each node in your cluster.

     Important

     You cannot change the service network address block during the migration.

     You cannot use any CIDR block that overlaps with the 100.64.0.0/16 CIDR block because the OVN-Kubernetes network provider uses this block internally.

8. To complete the migration, reboot each node in your cluster. For example, you can use a bash script similar to the following example. The script assumes that you can connect to each host by using ssh and that you have configured sudo to not prompt for a password:

   #!/bin/bash
   
   for ip in $(oc get nodes  -o jsonpath='{.items[*].status.addresses[?(@.type=="InternalIP")].address}')
   do
      echo "reboot node $ip"
      ssh -o StrictHostKeyChecking=no core@$ip sudo shutdown -r -t 3
   done

   Note

   If SSH access is not available, you can use the openstack command:

   $ for name in $(openstack server list --name "${CLUSTERID}*" -f value -c Name); do openstack server reboot "${name}"; done

   Alternatively, you might be able to reboot each node through the management portal for your infrastructure provider. Otherwise, contact the appropriate authority to either gain access to the virtual machines through SSH or the management portal and OpenStack client.

Verification

1. Confirm that the migration succeeded, and then remove the migration resources:

   1. To confirm that the network plugin is OVN-Kubernetes, enter the following command.

      $ oc get network.config/cluster -o jsonpath='{.status.networkType}{"\n"}'

      The value of status.networkType must be OVNKubernetes.

   2. To confirm that the cluster nodes are in the Ready state, enter the following command:

      $ oc get nodes

   3. To confirm that your pods are not in an error state, enter the following command:

      $ oc get pods --all-namespaces -o wide --sort-by='{.spec.nodeName}'

      If pods on a node are in an error state, reboot that node.

   4. To confirm that all of the cluster Operators are not in an abnormal state, enter the following command:

      $ oc get co

      The status of every cluster Operator must be the following: AVAILABLE="True", PROGRESSING="False", DEGRADED="False". If a cluster Operator is not available or degraded, check the logs for the cluster Operator for more information.

      Important

      Do not proceed if any of the previous verification steps indicate errors. You might encounter pods that have a Terminating state due to finalizers that are removed during clean up. They are not an error indication.

2. If the migration completed and your cluster is in a good state, remove the migration configuration from the CNO configuration object by entering the following command:

   $ oc patch Network.operator.openshift.io cluster --type=merge \
       --patch '{"spec": {"migration": null}}'

Procedure

1. Create a clean-up Python virtual environment:

   1. Create a temporary directory for your environment. For example:

      $ python3 -m venv /tmp/venv

      The virtual environment located in /tmp/venv directory is used in all clean up examples.

   2. Enter the virtual environment. For example:

      $ source /tmp/venv/bin/activate

   3. Upgrade the pip command in the virtual environment by running the following command:

      (venv) $ pip install --upgrade pip

   4. Install the required Python packages by running the following command:

      (venv) $ pip install openstacksdk==0.54.0 python-openstackclient==5.5.0 python-octaviaclient==2.3.0 'python-neutronclient<9.0.0'

2. In your terminal, set variables to cluster and Kuryr identifiers by running the following commands:

   1. Set the cluster ID:

      (venv) $ CLUSTERID=$(oc get infrastructure.config.openshift.io cluster -o=jsonpath='{.status.infrastructureName}')

   2. Set the cluster tag:

      (venv) $ CLUSTERTAG="openshiftClusterID=${CLUSTERID}"

   3. Set the router ID:

      (venv) $ ROUTERID=$(oc get kuryrnetwork -A --no-headers -o custom-columns=":status.routerId"|uniq)

3. Create a Bash function that removes finalizers from specified resources by running the following command:

   (venv) $ function REMFIN {
       local resource=$1
       local finalizer=$2
       for res in $(oc get "${resource}" -A --template='{{range $i,$p := .items}}{{ $p.metadata.name }}|{{ $p.metadata.namespace }}{{"\n"}}{{end}}'); do
           name=${res%%|*}
           ns=${res##*|}
           yaml=$(oc get -n "${ns}" "${resource}" "${name}" -o yaml)
           if echo "${yaml}" | grep -q "${finalizer}"; then
               echo "${yaml}" | grep -v  "${finalizer}" | oc replace -n "${ns}" "${resource}" "${name}" -f -
           fi
       done
   }

   The function takes two parameters: the first parameter is name of the resource, and the second parameter is the finalizer to remove. The named resource is removed from the cluster and its definition is replaced with copied data, excluding the specified finalizer.

4. To remove Kuryr finalizers from services, enter the following command:

   (venv) $ REMFIN services kuryr.openstack.org/service-finalizer

5. To remove the Kuryr service-subnet-gateway-ip service, enter the following command:

   (venv) $ if oc get -n openshift-kuryr service service-subnet-gateway-ip &>/dev/null; then
       oc -n openshift-kuryr delete service service-subnet-gateway-ip
   fi

6. To remove all tagged RHOSP load balancers from Octavia, enter the following command:

   (venv) $ for lb in $(openstack loadbalancer list --tags "${CLUSTERTAG}" -f value -c id); do
       openstack loadbalancer delete --cascade "${lb}"
   done

7. To remove Kuryr finalizers from all KuryrLoadBalancer CRs, enter the following command:

   (venv) $ REMFIN kuryrloadbalancers.openstack.org kuryr.openstack.org/kuryrloadbalancer-finalizers

8. To remove the openshift-kuryr namespace, enter the following command:

   (venv) $ oc delete namespace openshift-kuryr

9. To remove the Kuryr service subnet from the router, enter the following command:

   (venv) $ openstack router remove subnet "${ROUTERID}" "${CLUSTERID}-kuryr-service-subnet"

10. To remove the Kuryr service network, enter the following command:

    (venv) $ openstack network delete "${CLUSTERID}-kuryr-service-network"

11. To remove Kuryr finalizers from all pods, enter the following command:

    (venv) $ REMFIN pods kuryr.openstack.org/pod-finalizer

12. To remove Kuryr finalizers from all KuryrPort CRs, enter the following command:

    (venv) $ REMFIN kuryrports.openstack.org kuryr.openstack.org/kuryrport-finalizer

    This command deletes the KuryrPort CRs.

13. To remove Kuryr finalizers from network policies, enter the following command:

    (venv) $ REMFIN networkpolicy kuryr.openstack.org/networkpolicy-finalizer

14. To remove Kuryr finalizers from remaining network policies, enter the following command:

    (venv) $ REMFIN kuryrnetworkpolicies.openstack.org kuryr.openstack.org/networkpolicy-finalizer

15. To remove subports that Kuryr created from trunks, enter the following command:

    (venv) $ mapfile trunks < <(python -c "import openstack; n = openstack.connect().network; print('\n'.join([x.id for x in n.trunks(any_tags='$CLUSTERTAG')]))") && \
    i=0 && \
    for trunk in "${trunks[@]}"; do
        trunk=$(echo "$trunk"|tr -d '\n')
        i=$((i+1))
        echo "Processing trunk $trunk, ${i}/${#trunks[@]}."
        subports=()
        for subport in $(python -c "import openstack; n = openstack.connect().network; print(' '.join([x['port_id'] for x in n.get_trunk('$trunk').sub_ports if '$CLUSTERTAG' in n.get_port(x['port_id']).tags]))"); do
            subports+=("$subport");
        done
        args=()
        for sub in "${subports[@]}" ; do
            args+=("--subport $sub")
        done
        if [ ${#args[@]} -gt 0 ]; then
            openstack network trunk unset ${args[*]} "${trunk}"
        fi
    done

16. To retrieve all networks and subnets from KuryrNetwork CRs and remove ports, router interfaces and the network itself, enter the following command:

    (venv) $ mapfile -t kuryrnetworks < <(oc get kuryrnetwork -A --template='{{range $i,$p := .items}}{{ $p.status.netId }}|{{ $p.status.subnetId }}{{"\n"}}{{end}}') && \
    i=0 && \
    for kn in "${kuryrnetworks[@]}"; do
        i=$((i+1))
        netID=${kn%%|*}
        subnetID=${kn##*|}
        echo "Processing network $netID, ${i}/${#kuryrnetworks[@]}"
        # Remove all ports from the network.
        for port in $(python -c "import openstack; n = openstack.connect().network; print(' '.join([x.id for x in n.ports(network_id='$netID') if x.device_owner != 'network:router_interface']))"); do
            ( openstack port delete "${port}" ) &
    
            # Only allow 20 jobs in parallel.
            if [[ $(jobs -r -p | wc -l) -ge 20 ]]; then
                wait -n
            fi
        done
        wait
    
        # Remove the subnet from the router.
        openstack router remove subnet "${ROUTERID}" "${subnetID}"
    
        # Remove the network.
        openstack network delete "${netID}"
    done

17. To remove the Kuryr security group, enter the following command:

    (venv) $ openstack security group delete "${CLUSTERID}-kuryr-pods-security-group"

18. To remove all tagged subnet pools, enter the following command:

    (venv) $ for subnetpool in $(openstack subnet pool list --tags "${CLUSTERTAG}" -f value -c ID); do
        openstack subnet pool delete "${subnetpool}"
    done

19. To check that all of the networks based on KuryrNetwork CRs were removed, enter the following command:

    (venv) $ networks=$(oc get kuryrnetwork -A --no-headers -o custom-columns=":status.netId") && \
    for existingNet in $(openstack network list --tags "${CLUSTERTAG}" -f value -c ID); do
        if [[ $networks =~ $existingNet ]]; then
            echo "Network still exists: $existingNet"
        fi
    done

    If the command returns any existing networks, intestigate and remove them before you continue.

20. To remove security groups that are related to network policy, enter the following command:

    (venv) $ for sgid in $(openstack security group list -f value -c ID -c Description | grep 'Kuryr-Kubernetes Network Policy' | cut -f 1 -d ' '); do
        openstack security group delete "${sgid}"
    done

21. To remove finalizers from KuryrNetwork CRs, enter the following command:

    (venv) $ REMFIN kuryrnetworks.openstack.org kuryrnetwork.finalizers.kuryr.openstack.org

22. To remove the Kuryr router, enter the following command:

    (venv) $ if python3 -c "import sys; import openstack; n = openstack.connect().network; r = n.get_router('$ROUTERID'); sys.exit(0) if r.description != 'Created By OpenShift Installer' else sys.exit(1)"; then
        openstack router delete "${ROUTERID}"
    fi

Procedure

1. To specify IPv6 address blocks for the cluster and service networks, create a file containing the following YAML:

   - op: add
     path: /spec/clusterNetwork/-
     value: 

   1

   
       cidr: fd01::/48
       hostPrefix: 64
   - op: add
     path: /spec/serviceNetwork/-
     value: fd02::/112 

   2

   1

   Specify an object with the cidr and hostPrefix parameters. The host prefix must be 64 or greater. The IPv6 Classless Inter-Domain Routing (CIDR) prefix must be large enough to accommodate the specified host prefix.

   1 2

   Specify an IPv6 CIDR with a prefix of 112. Kubernetes uses only the lowest 16 bits. For a prefix of 112, IP addresses are assigned from 112 to 128 bits.

2. To patch the cluster network configuration, enter the following command:

   $ oc patch network.config.openshift.io cluster \
     --type='json' --patch-file <file>.yaml

   where:

   file

   Specifies the name of the file you created in the previous step.

   Example output

   network.config.openshift.io/cluster patched

1. Display the network configuration:

   $ oc describe network

   Example output

   Status:
     Cluster Network:
       Cidr:               10.128.0.0/14
       Host Prefix:        23
       Cidr:               fd01::/48
       Host Prefix:        64
     Cluster Network MTU:  1400
     Network Type:         OVNKubernetes
     Service Network:
       172.30.0.0/16
       fd02::/112

Procedure

1. Edit the networks.config.openshift.io custom resource (CR) by running the following command:

   $ oc edit networks.config.openshift.io

2. Remove the IPv4 or IPv6 configuration that you added to the cidr and the hostPrefix parameters from completing the "Converting to a dual-stack cluster network " procedure steps.

Verification

1. To create a namespace with network policies complete the following steps:

   1. Create a namespace for verification:

      $ cat <<EOF| oc create -f -
      kind: Namespace
      apiVersion: v1
      metadata:
        name: verify-audit-logging
        annotations:
          k8s.ovn.org/acl-logging: '{ "deny": "alert", "allow": "alert" }'
      EOF

      Example output

      namespace/verify-audit-logging created

   2. Create network policies for the namespace:

      $ cat <<EOF| oc create -n verify-audit-logging -f -
      apiVersion: networking.k8s.io/v1
      kind: NetworkPolicy
      metadata:
        name: deny-all
      spec:
        podSelector:
          matchLabels:
        policyTypes:
        - Ingress
        - Egress
      ---
      apiVersion: networking.k8s.io/v1
      kind: NetworkPolicy
      metadata:
        name: allow-from-same-namespace
        namespace: verify-audit-logging
      spec:
        podSelector: {}
        policyTypes:
         - Ingress
         - Egress
        ingress:
          - from:
              - podSelector: {}
        egress:
          - to:
             - namespaceSelector:
                matchLabels:
                  kubernetes.io/metadata.name: verify-audit-logging
      EOF

      Example output

      networkpolicy.networking.k8s.io/deny-all created
      networkpolicy.networking.k8s.io/allow-from-same-namespace created

2. Create a pod for source traffic in the default namespace:

   $ cat <<EOF| oc create -n default -f -
   apiVersion: v1
   kind: Pod
   metadata:
     name: client
   spec:
     containers:
       - name: client
         image: registry.access.redhat.com/rhel7/rhel-tools
         command: ["/bin/sh", "-c"]
         args:
           ["sleep inf"]
   EOF

3. Create two pods in the verify-audit-logging namespace:

   $ for name in client server; do
   cat <<EOF| oc create -n verify-audit-logging -f -
   apiVersion: v1
   kind: Pod
   metadata:
     name: ${name}
   spec:
     containers:
       - name: ${name}
         image: registry.access.redhat.com/rhel7/rhel-tools
         command: ["/bin/sh", "-c"]
         args:
           ["sleep inf"]
   EOF
   done

   Example output

   pod/client created
   pod/server created

4. To generate traffic and produce network policy audit log entries, complete the following steps:

   1. Obtain the IP address for pod named server in the verify-audit-logging namespace:

      $ POD_IP=$(oc get pods server -n verify-audit-logging -o jsonpath='{.status.podIP}')

   2. Ping the IP address from the previous command from the pod named client in the default namespace and confirm that all packets are dropped:

      $ oc exec -it client -n default -- /bin/ping -c 2 $POD_IP

      Example output

      PING 10.128.2.55 (10.128.2.55) 56(84) bytes of data.
      
      --- 10.128.2.55 ping statistics ---
      2 packets transmitted, 0 received, 100% packet loss, time 2041ms

   3. Ping the IP address saved in the POD_IP shell environment variable from the pod named client in the verify-audit-logging namespace and confirm that all packets are allowed:

      $ oc exec -it client -n verify-audit-logging -- /bin/ping -c 2 $POD_IP

      Example output

      PING 10.128.0.86 (10.128.0.86) 56(84) bytes of data.
      64 bytes from 10.128.0.86: icmp_seq=1 ttl=64 time=2.21 ms
      64 bytes from 10.128.0.86: icmp_seq=2 ttl=64 time=0.440 ms
      
      --- 10.128.0.86 ping statistics ---
      2 packets transmitted, 2 received, 0% packet loss, time 1001ms
      rtt min/avg/max/mdev = 0.440/1.329/2.219/0.890 ms

5. Display the latest entries in the network policy audit log:

   $ for pod in $(oc get pods -n openshift-ovn-kubernetes -l app=ovnkube-node --no-headers=true | awk '{ print $1 }') ; do
       oc exec -it $pod -n openshift-ovn-kubernetes -- tail -4 /var/log/ovn/acl-audit-log.log
     done

   Example output

   2023-11-02T16:28:54.139Z|00004|acl_log(ovn_pinctrl0)|INFO|name="NP:verify-audit-logging:Ingress", verdict=drop, severity=alert, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:81:02:01,dl_dst=0a:58:0a:81:02:23,nw_src=10.131.0.39,nw_dst=10.129.2.35,nw_tos=0,nw_ecn=0,nw_ttl=62,nw_frag=no,tp_src=58496,tp_dst=8080,tcp_flags=syn
   2023-11-02T16:28:55.187Z|00005|acl_log(ovn_pinctrl0)|INFO|name="NP:verify-audit-logging:Ingress", verdict=drop, severity=alert, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:81:02:01,dl_dst=0a:58:0a:81:02:23,nw_src=10.131.0.39,nw_dst=10.129.2.35,nw_tos=0,nw_ecn=0,nw_ttl=62,nw_frag=no,tp_src=58496,tp_dst=8080,tcp_flags=syn
   2023-11-02T16:28:57.235Z|00006|acl_log(ovn_pinctrl0)|INFO|name="NP:verify-audit-logging:Ingress", verdict=drop, severity=alert, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:81:02:01,dl_dst=0a:58:0a:81:02:23,nw_src=10.131.0.39,nw_dst=10.129.2.35,nw_tos=0,nw_ecn=0,nw_ttl=62,nw_frag=no,tp_src=58496,tp_dst=8080,tcp_flags=syn
   2023-11-02T16:49:57.909Z|00028|acl_log(ovn_pinctrl0)|INFO|name="NP:verify-audit-logging:allow-from-same-namespace:Egress:0", verdict=allow, severity=alert, direction=from-lport: icmp,vlan_tci=0x0000,dl_src=0a:58:0a:81:02:22,dl_dst=0a:58:0a:81:02:23,nw_src=10.129.2.34,nw_dst=10.129.2.35,nw_tos=0,nw_ecn=0,nw_ttl=64,nw_frag=no,icmp_type=8,icmp_code=0
   2023-11-02T16:49:57.909Z|00029|acl_log(ovn_pinctrl0)|INFO|name="NP:verify-audit-logging:allow-from-same-namespace:Ingress:0", verdict=allow, severity=alert, direction=to-lport: icmp,vlan_tci=0x0000,dl_src=0a:58:0a:81:02:22,dl_dst=0a:58:0a:81:02:23,nw_src=10.129.2.34,nw_dst=10.129.2.35,nw_tos=0,nw_ecn=0,nw_ttl=64,nw_frag=no,icmp_type=8,icmp_code=0
   2023-11-02T16:49:58.932Z|00030|acl_log(ovn_pinctrl0)|INFO|name="NP:verify-audit-logging:allow-from-same-namespace:Egress:0", verdict=allow, severity=alert, direction=from-lport: icmp,vlan_tci=0x0000,dl_src=0a:58:0a:81:02:22,dl_dst=0a:58:0a:81:02:23,nw_src=10.129.2.34,nw_dst=10.129.2.35,nw_tos=0,nw_ecn=0,nw_ttl=64,nw_frag=no,icmp_type=8,icmp_code=0
   2023-11-02T16:49:58.932Z|00031|acl_log(ovn_pinctrl0)|INFO|name="NP:verify-audit-logging:allow-from-same-namespace:Ingress:0", verdict=allow, severity=alert, direction=to-lport: icmp,vlan_tci=0x0000,dl_src=0a:58:0a:81:02:22,dl_dst=0a:58:0a:81:02:23,nw_src=10.129.2.34,nw_dst=10.129.2.35,nw_tos=0,nw_ecn=0,nw_ttl=64,nw_frag=no,icmp_type=8,icmp_code=0