Chapter 25. OVN-Kubernetes network plugin

Procedure

1. Run the following command to get all resources, endpoints, and ConfigMaps in the OVN-Kubernetes project:

   $ oc get all,ep,cm -n openshift-ovn-kubernetes

   Example output

   NAME                       READY   STATUS    RESTARTS      AGE
   pod/ovnkube-master-9g7zt   6/6     Running   1 (48m ago)   57m
   pod/ovnkube-master-lqs4v   6/6     Running   0             57m
   pod/ovnkube-master-vxhtq   6/6     Running   0             57m
   pod/ovnkube-node-9k9kc     5/5     Running   0             57m
   pod/ovnkube-node-jg52r     5/5     Running   0             51m
   pod/ovnkube-node-k8wf7     5/5     Running   0             57m
   pod/ovnkube-node-tlwk6     5/5     Running   0             47m
   pod/ovnkube-node-xsvnk     5/5     Running   0             57m
   
   NAME                            TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)             AGE
   service/ovn-kubernetes-master   ClusterIP   None         <none>        9102/TCP            57m
   service/ovn-kubernetes-node     ClusterIP   None         <none>        9103/TCP,9105/TCP   57m
   service/ovnkube-db              ClusterIP   None         <none>        9641/TCP,9642/TCP   57m
   
   NAME                            DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                                                 AGE
   daemonset.apps/ovnkube-master   3         3         3       3            3           beta.kubernetes.io/os=linux,node-role.kubernetes.io/master=   57m
   daemonset.apps/ovnkube-node     5         5         5       5            5           beta.kubernetes.io/os=linux                                   57m
   
   NAME                              ENDPOINTS                                                        AGE
   endpoints/ovn-kubernetes-master   10.0.132.11:9102,10.0.151.18:9102,10.0.192.45:9102               57m
   endpoints/ovn-kubernetes-node     10.0.132.11:9105,10.0.143.72:9105,10.0.151.18:9105 + 7 more...   57m
   endpoints/ovnkube-db              10.0.132.11:9642,10.0.151.18:9642,10.0.192.45:9642 + 3 more...   57m
   
   NAME                                 DATA   AGE
   configmap/control-plane-status       1      55m
   configmap/kube-root-ca.crt           1      57m
   configmap/openshift-service-ca.crt   1      57m
   configmap/ovn-ca                     1      57m
   configmap/ovnkube-config             1      57m
   configmap/signer-ca                  1      57m

   There are three ovnkube-masters that run on the control plane nodes, and two daemon sets used to deploy the ovnkube-master and ovnkube-node pods. There is one ovnkube-node pod for each node in the cluster. The ovnkube-config ConfigMap has the OpenShift Container Platform OVN-Kubernetes configurations started by online-master and ovnkube-node.

2. List all the containers in the ovnkube-master pods by running the following command:

   $ oc get pods ovnkube-master-9g7zt \
   -o jsonpath='{.spec.containers[*].name}' -n openshift-ovn-kubernetes

   Expected output

   northd nbdb kube-rbac-proxy sbdb ovnkube-master ovn-dbchecker

   The ovnkube-master pod is made up of several containers. It is responsible for hosting the northbound database (nbdb container), the southbound database (sbdb container), watching for cluster events for pods, egressIP, namespaces, services, endpoints, egress firewall, and network policy and writing them to the northbound database (ovnkube-master pod), as well as managing pod subnet allocation to nodes.

3. List all the containers in the ovnkube-node pods by running the following command:

   $ oc get pods ovnkube-node-jg52r \
   -o jsonpath='{.spec.containers[*].name}' -n openshift-ovn-kubernetes

   Expected output

   ovn-controller ovn-acl-logging kube-rbac-proxy kube-rbac-proxy-ovn-metrics ovnkube-node

   The ovnkube-node pod has a container (ovn-controller) that resides on each OpenShift Container Platform node. Each node’s ovn-controller connects the OVN northbound to the OVN southbound database to learn about the OVN configuration. The ovn-controller connects southbound to ovs-vswitchd as an OpenFlow controller, for control over network traffic, and to the local ovsdb-server to allow it to monitor and control Open vSwitch configuration.

4. List the currently elected OVN-Kubernetes master leader by running the following command:

   $ oc get lease -n openshift-ovn-kubernetes

   Expected output

   NAME                    HOLDER                               AGE
   ovn-kubernetes-master   ci-ln-gz990pb-72292-rthz2-master-2   50m

Procedure

1. Find the OVN Raft leader for the northbound database.

   Note

   The Raft leader stores the most up to date information.

   1. List the pods by running the following command:

      $ oc get po -n openshift-ovn-kubernetes

      Example output

      NAME                   READY   STATUS    RESTARTS       AGE
      ovnkube-master-7j97q   6/6     Running   2 (148m ago)   149m
      ovnkube-master-gt4ms   6/6     Running   1 (140m ago)   147m
      ovnkube-master-mk6p6   6/6     Running   0              148m
      ovnkube-node-8qvtr     5/5     Running   0              149m
      ovnkube-node-fqdc9     5/5     Running   0              149m
      ovnkube-node-tlfwv     5/5     Running   0              149m
      ovnkube-node-wlwkn     5/5     Running   0              142m

   2. Choose one of the master pods at random and run the following command:

      $ oc exec -n openshift-ovn-kubernetes ovnkube-master-7j97q \
      -- /usr/bin/ovn-appctl -t /var/run/ovn/ovnnb_db.ctl \
      --timeout=3 cluster/status OVN_Northbound

      Example output

      Defaulted container "northd" out of: northd, nbdb, kube-rbac-proxy, sbdb, ovnkube-master, ovn-dbchecker
      1c57
      Name: OVN_Northbound
      Cluster ID: c48a (c48aa5c0-a704-4c77-a066-24fe99d9b338)
      Server ID: 1c57 (1c57b6fc-2849-49b7-8679-fbf18bafe339)
      Address: ssl:10.0.147.219:9643
      Status: cluster member
      Role: follower 

      1

      
      Term: 5
      Leader: 2b4f 

      2

      
      Vote: unknown
      
      Election timer: 10000
      Log: [2, 3018]
      Entries not yet committed: 0
      Entries not yet applied: 0
      Connections: ->0000 ->0000 <-8844 <-2b4f
      Disconnections: 0
      Servers:
          1c57 (1c57 at ssl:10.0.147.219:9643) (self)
          8844 (8844 at ssl:10.0.163.212:9643) last msg 8928047 ms ago
          2b4f (2b4f at ssl:10.0.242.240:9643) last msg 620 ms ago 

      3

      1

      This pod is identified as a follower

      2

      The leader is identified as 2b4f

      3

      The 2b4f is on IP address 10.0.242.240

   3. Find the ovnkube-master pod running on IP Address 10.0.242.240 using the following command:

      $ oc get po -o wide -n openshift-ovn-kubernetes | grep 10.0.242.240 | grep -v ovnkube-node

      Example output

      ovnkube-master-gt4ms   6/6     Running             1 (143m ago)   150m   10.0.242.240   ip-10-0-242-240.ec2.internal   <none>           <none>

      The ovnkube-master-gt4ms pod runs on IP Address 10.0.242.240.

2. Run the following command to show all the objects in the northbound database:

   $ oc exec -n openshift-ovn-kubernetes -it ovnkube-master-gt4ms \
   -c northd -- ovn-nbctl show

   The output is too long to list here. The list includes the NAT rules, logical switches, load balancers and so on.

   Run the following command to display the options available with the command ovn-nbctl:

   $ oc exec -n openshift-ovn-kubernetes -it ovnkube-master-mk6p6 \
   -c northd ovn-nbctl --help

   You can narrow down and focus on specific components by using some of the following commands:

3. Run the following command to show the list of logical routers:

   $ oc exec -n openshift-ovn-kubernetes -it ovnkube-master-gt4ms \
   -c northd -- ovn-nbctl lr-list

   Example output

   f971f1f3-5112-402f-9d1e-48f1d091ff04 (GR_ip-10-0-145-205.ec2.internal)
   69c992d8-a4cf-429e-81a3-5361209ffe44 (GR_ip-10-0-147-219.ec2.internal)
   7d164271-af9e-4283-b84a-48f2a44851cd (GR_ip-10-0-163-212.ec2.internal)
   111052e3-c395-408b-97b2-8dd0a20a29a5 (GR_ip-10-0-165-9.ec2.internal)
   ed50ce33-df5d-48e8-8862-2df6a59169a0 (GR_ip-10-0-209-170.ec2.internal)
   f44e2a96-8d1e-4a4d-abae-ed8728ac6851 (GR_ip-10-0-242-240.ec2.internal)
   ef3d0057-e557-4b1a-b3c6-fcc3463790b0 (ovn_cluster_router)

   Note

   From this output you can see there is router on each node plus an ovn_cluster_router.

4. Run the following command to show the list of logical switches:

   $ oc exec -n openshift-ovn-kubernetes -it ovnkube-master-gt4ms \
   -c northd -- ovn-nbctl ls-list

   Example output

   82808c5c-b3bc-414a-bb59-8fec4b07eb14 (ext_ip-10-0-145-205.ec2.internal)
   3d22444f-0272-4c51-afc6-de9e03db3291 (ext_ip-10-0-147-219.ec2.internal)
   bf73b9df-59ab-4c58-a456-ce8205b34ac5 (ext_ip-10-0-163-212.ec2.internal)
   bee1e8d0-ec87-45eb-b98b-63f9ec213e5e (ext_ip-10-0-165-9.ec2.internal)
   812f08f2-6476-4abf-9a78-635f8516f95e (ext_ip-10-0-209-170.ec2.internal)
   f65e710b-32f9-482b-8eab-8d96a44799c1 (ext_ip-10-0-242-240.ec2.internal)
   84dad700-afb8-4129-86f9-923a1ddeace9 (ip-10-0-145-205.ec2.internal)
   1b7b448b-e36c-4ca3-9f38-4a2cf6814bfd (ip-10-0-147-219.ec2.internal)
   d92d1f56-2606-4f23-8b6a-4396a78951de (ip-10-0-163-212.ec2.internal)
   6864a6b2-de15-4de3-92d8-f95014b6f28f (ip-10-0-165-9.ec2.internal)
   c26bf618-4d7e-4afd-804f-1a2cbc96ec6d (ip-10-0-209-170.ec2.internal)
   ab9a4526-44ed-4f82-ae1c-e20da04947d9 (ip-10-0-242-240.ec2.internal)
   a8588aba-21da-4276-ba0f-9d68e88911f0 (join)

   Note

   From this output you can see there is an ext switch for each node plus switches with the node name itself and a join switch.

5. Run the following command to show the list of load balancers:

   $ oc exec -n openshift-ovn-kubernetes -it ovnkube-master-gt4ms \
   -c northd -- ovn-nbctl lb-list

   Example output

   UUID                                    LB                  PROTO      VIP                     IPs
   f0fb50f9-4968-4b55-908c-616bae4db0a2    Service_default/    tcp        172.30.0.1:443          10.0.147.219:6443,10.0.163.212:6443,169.254.169.2:6443
   0dc42012-4f5b-432e-ae01-2cc4bfe81b00    Service_default/    tcp        172.30.0.1:443          10.0.147.219:6443,169.254.169.2:6443,10.0.242.240:6443
   f7fff5d5-5eff-4a40-98b1-3a4ba8f7f69c    Service_default/    tcp        172.30.0.1:443          169.254.169.2:6443,10.0.163.212:6443,10.0.242.240:6443
   12fe57a0-50a4-4a1b-ac10-5f288badee07    Service_default/    tcp        172.30.0.1:443          10.0.147.219:6443,10.0.163.212:6443,10.0.242.240:6443
   3f137fbf-0b78-4875-ba44-fbf89f254cf7    Service_openshif    tcp        172.30.23.153:443       10.130.0.14:8443
   174199fe-0562-4141-b410-12094db922a7    Service_openshif    tcp        172.30.69.51:50051      10.130.0.84:50051
   5ee2d4bd-c9e2-4d16-a6df-f54cd17c9ac3    Service_openshif    tcp        172.30.143.87:9001      10.0.145.205:9001,10.0.147.219:9001,10.0.163.212:9001,10.0.165.9:9001,10.0.209.170:9001,10.0.242.240:9001
   a056ae3d-83f8-45bc-9c80-ef89bce7b162    Service_openshif    tcp        172.30.164.74:443       10.0.147.219:6443,10.0.163.212:6443,10.0.242.240:6443
   bac51f3d-9a6f-4f5e-ac02-28fd343a332a    Service_openshif    tcp        172.30.0.10:53          10.131.0.6:5353
                                                               tcp        172.30.0.10:9154        10.131.0.6:9154
   48105bbc-51d7-4178-b975-417433f9c20a    Service_openshif    tcp        172.30.26.159:2379      10.0.147.219:2379,169.254.169.2:2379,10.0.242.240:2379
                                                               tcp        172.30.26.159:9979      10.0.147.219:9979,169.254.169.2:9979,10.0.242.240:9979
   7de2b8fc-342a-415f-ac13-1a493f4e39c0    Service_openshif    tcp        172.30.53.219:443       10.128.0.7:8443
                                                               tcp        172.30.53.219:9192      10.128.0.7:9192
   2cef36bc-d720-4afb-8d95-9350eff1d27a    Service_openshif    tcp        172.30.81.66:443        10.128.0.23:8443
   365cb6fb-e15e-45a4-a55b-21868b3cf513    Service_openshif    tcp        172.30.96.51:50051      10.130.0.19:50051
   41691cbb-ec55-4cdb-8431-afce679c5e8d    Service_openshif    tcp        172.30.98.218:9099      169.254.169.2:9099
   82df10ba-8143-400b-977a-8f5f416a4541    Service_openshif    tcp        172.30.26.159:2379      10.0.147.219:2379,10.0.163.212:2379,169.254.169.2:2379
                                                               tcp        172.30.26.159:9979      10.0.147.219:9979,10.0.163.212:9979,169.254.169.2:9979
   debe7f3a-39a8-490e-bc0a-ebbfafdffb16    Service_openshif    tcp        172.30.23.244:443       10.128.0.48:8443,10.129.0.27:8443,10.130.0.45:8443
   8a749239-02d9-4dc2-8737-716528e0da7b    Service_openshif    tcp        172.30.124.255:8443     10.128.0.14:8443
   880c7c78-c790-403d-a3cb-9f06592717a3    Service_openshif    tcp        172.30.0.10:53          10.130.0.20:5353
                                                               tcp        172.30.0.10:9154        10.130.0.20:9154
   d2f39078-6751-4311-a161-815bbaf7f9c7    Service_openshif    tcp        172.30.26.159:2379      169.254.169.2:2379,10.0.163.212:2379,10.0.242.240:2379
                                                               tcp        172.30.26.159:9979      169.254.169.2:9979,10.0.163.212:9979,10.0.242.240:9979
   30948278-602b-455c-934a-28e64c46de12    Service_openshif    tcp        172.30.157.35:9443      10.130.0.43:9443
   2cc7e376-7c02-4a82-89e8-dfa1e23fb003    Service_openshif    tcp        172.30.159.212:17698    10.128.0.48:17698,10.129.0.27:17698,10.130.0.45:17698
   e7d22d35-61c2-40c2-bc30-265cff8ed18d    Service_openshif    tcp        172.30.143.87:9001      10.0.145.205:9001,10.0.147.219:9001,10.0.163.212:9001,10.0.165.9:9001,10.0.209.170:9001,169.254.169.2:9001
   75164e75-e0c5-40fb-9636-bfdbf4223a02    Service_openshif    tcp        172.30.150.68:1936      10.129.4.8:1936,10.131.0.10:1936
                                                               tcp        172.30.150.68:443       10.129.4.8:443,10.131.0.10:443
                                                               tcp        172.30.150.68:80        10.129.4.8:80,10.131.0.10:80
   7bc4ee74-dccf-47e9-9149-b011f09aff39    Service_openshif    tcp        172.30.164.74:443       10.0.147.219:6443,10.0.163.212:6443,169.254.169.2:6443
   0db59e74-1cc6-470c-bf44-57c520e0aa8f    Service_openshif    tcp        10.0.163.212:31460
                                                               tcp        10.0.163.212:32361
   c300e134-018c-49af-9f84-9deb1d0715f8    Service_openshif    tcp        172.30.42.244:50051     10.130.0.47:50051
   5e352773-429b-4881-afb3-a13b7ba8b081    Service_openshif    tcp        172.30.244.66:443       10.129.0.8:8443,10.130.0.8:8443
   54b82d32-1939-4465-a87d-f26321442a7a    Service_openshif    tcp        172.30.12.9:8443        10.128.0.35:8443

   Note

   From this truncated output you can see there are many OVN-Kubernetes load balancers. Load balancers in OVN-Kubernetes are representations of services.

Procedure

1. Find the OVN Raft leader for the southbound database.

   Note

   The Raft leader stores the most up to date information.

   1. List the pods by running the following command:

      $ oc get po -n openshift-ovn-kubernetes

      Example output

      NAME                   READY   STATUS    RESTARTS       AGE
      ovnkube-master-7j97q   6/6     Running   2 (134m ago)   135m
      ovnkube-master-gt4ms   6/6     Running   1 (126m ago)   133m
      ovnkube-master-mk6p6   6/6     Running   0              134m
      ovnkube-node-8qvtr     5/5     Running   0              135m
      ovnkube-node-bqztb     5/5     Running   0              117m
      ovnkube-node-fqdc9     5/5     Running   0              135m
      ovnkube-node-tlfwv     5/5     Running   0              135m
      ovnkube-node-wlwkn     5/5     Running   0              128m

   2. Choose one of the master pods at random and run the following command to find the OVN southbound Raft leader:

      $ oc exec -n openshift-ovn-kubernetes ovnkube-master-7j97q \
      -- /usr/bin/ovn-appctl -t /var/run/ovn/ovnsb_db.ctl \
      --timeout=3 cluster/status OVN_Southbound

      Example output

      Defaulted container "northd" out of: northd, nbdb, kube-rbac-proxy, sbdb, ovnkube-master, ovn-dbchecker
      1930
      Name: OVN_Southbound
      Cluster ID: f772 (f77273c0-7986-42dd-bd3c-a9f18e25701f)
      Server ID: 1930 (1930f4b7-314b-406f-9dcb-b81fe2729ae1)
      Address: ssl:10.0.147.219:9644
      Status: cluster member
      Role: follower 

      1

      
      Term: 3
      Leader: 7081 

      2

      
      Vote: unknown
      
      Election timer: 16000
      Log: [2, 2423]
      Entries not yet committed: 0
      Entries not yet applied: 0
      Connections: ->0000 ->7145 <-7081 <-7145
      Disconnections: 0
      Servers:
          7081 (7081 at ssl:10.0.163.212:9644) last msg 59 ms ago 

      3

      
          1930 (1930 at ssl:10.0.147.219:9644) (self)
          7145 (7145 at ssl:10.0.242.240:9644) last msg 7871735 ms ago

      1

      This pod is identified as a follower

      2

      The leader is identified as 7081

      3

      The 7081 is on IP address 10.0.163.212

   3. Find the ovnkube-master pod running on IP Address 10.0.163.212 using the following command:

      $ oc get po -o wide -n openshift-ovn-kubernetes | grep 10.0.163.212 | grep -v ovnkube-node

      Example output

      ovnkube-master-mk6p6   6/6     Running   0              136m   10.0.163.212   ip-10-0-163-212.ec2.internal   <none>           <none>

      The ovnkube-master-mk6p6 pod runs on IP Address 10.0.163.212.

2. Run the following command to show all the information stored in the southbound database:

   $ oc exec -n openshift-ovn-kubernetes -it ovnkube-master-mk6p6 \
   -c northd -- ovn-sbctl show

   Example output

   Chassis "8ca57b28-9834-45f0-99b0-96486c22e1be"
       hostname: ip-10-0-156-16.ec2.internal
       Encap geneve
           ip: "10.0.156.16"
           options: {csum="true"}
       Port_Binding k8s-ip-10-0-156-16.ec2.internal
       Port_Binding etor-GR_ip-10-0-156-16.ec2.internal
       Port_Binding jtor-GR_ip-10-0-156-16.ec2.internal
       Port_Binding openshift-ingress-canary_ingress-canary-hsblx
       Port_Binding rtoj-GR_ip-10-0-156-16.ec2.internal
       Port_Binding openshift-monitoring_prometheus-adapter-658fc5967-9l46x
       Port_Binding rtoe-GR_ip-10-0-156-16.ec2.internal
       Port_Binding openshift-multus_network-metrics-daemon-77nvz
       Port_Binding openshift-ingress_router-default-64fd8c67c7-df598
       Port_Binding openshift-dns_dns-default-ttpcq
       Port_Binding openshift-monitoring_alertmanager-main-0
       Port_Binding openshift-e2e-loki_loki-promtail-g2pbh
       Port_Binding openshift-network-diagnostics_network-check-target-m6tn4
       Port_Binding openshift-monitoring_thanos-querier-75b5cf8dcb-qf8qj
       Port_Binding cr-rtos-ip-10-0-156-16.ec2.internal
       Port_Binding openshift-image-registry_image-registry-7b7bc44566-mp9b8

   This detailed output shows the chassis and the ports that are attached to the chassis which in this case are all of the router ports and anything that runs like host networking. Any pods communicate out to the wider network using source network address translation (SNAT). Their IP address is translated into the IP address of the node that the pod is running on and then sent out into the network.

   In addition to the chassis information the southbound database has all the logic flows and those logic flows are then sent to the ovn-controller running on each of the nodes. The ovn-controller translates the logic flows into open flow rules and ultimately programs OpenvSwitch so that your pods can then follow open flow rules and make it out of the network.

   Run the following command to display the options available with the command ovn-sbctl:

   $ oc exec -n openshift-ovn-kubernetes -it ovnkube-master-mk6p6 \
   -c northd -- ovn-sbctl --help

Procedure

1. Review the details of the ovnkube-master readiness probe by running the following command:

   $ oc get pods -n openshift-ovn-kubernetes -l app=ovnkube-master \
   -o json | jq '.items[0].spec.containers[] | .name,.readinessProbe'

   The readiness probe for the northbound and southbound database containers in the ovnkube-master pod checks for the health of the Raft cluster hosting the databases.

2. Review the details of the ovnkube-node readiness probe by running the following command:

   $ oc get pods -n openshift-ovn-kubernetes -l app=ovnkube-master \
   -o json | jq '.items[0].spec.containers[] | .name,.readinessProbe'

   The ovnkube-node container in the ovnkube-node pod has a readiness probe to verify the presence of the ovn-kubernetes CNI configuration file, the absence of which would indicate that the pod is not running or is not ready to accept requests to configure pods.

3. Show all events including the probe failures, for the namespace by using the following command:

   $ oc get events -n openshift-ovn-kubernetes

4. Show the events for just this pod:

   $ oc describe pod ovnkube-master-tp2z8 -n openshift-ovn-kubernetes

5. Show the messages and statuses from the cluster network operator:

   $ oc get co/network -o json | jq '.status.conditions[]'

6. Show the ready status of each container in ovnkube-master pods by running the following script:

   $ for p in $(oc get pods --selector app=ovnkube-master -n openshift-ovn-kubernetes \
   -o jsonpath='{range.items[*]}{" "}{.metadata.name}'); do echo === $p ===;  \
   oc get pods -n openshift-ovn-kubernetes $p -o json | jq '.status.containerStatuses[] | .name, .ready'; \
   done

   Note

   The expectation is all container statuses are reporting as true. Failure of a readiness probe sets the status to false.

Procedure (UI)

1. In the Administrator perspective, select Observe Alerting. The three main pages in the Alerting UI in this perspective are the Alerts, Silences, and Alerting Rules pages.
2. View the rules for OVN-Kubernetes alerts by selecting Observe Alerting Alerting Rules.

Procedure

1. View active or firing alerts by running the following commands.

   1. Set the alert manager route environment variable by running the following command:

      $ ALERT_MANAGER=$(oc get route alertmanager-main -n openshift-monitoring \
      -o jsonpath='{@.spec.host}')

   2. Issue a curl request to the alert manager route API with the correct authorization details requesting specific fields by running the following command:

      $ curl -s -k -H "Authorization: Bearer \
      $(oc create token prometheus-k8s -n openshift-monitoring)" \
      https://$ALERT_MANAGER/api/v1/alerts \
      | jq '.data[] | "\(.labels.severity) \(.labels.alertname) \(.labels.pod) \(.labels.container) \(.labels.endpoint) \(.labels.instance)"'

2. View alerting rules by running the following command:

   $ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -s 'http://localhost:9090/api/v1/rules' | jq '.data.groups[].rules[] | select(((.name|contains("ovn")) or (.name|contains("OVN")) or (.name|contains("Ovn")) or (.name|contains("North")) or (.name|contains("South"))) and .type=="alerting")'

Procedure

1. View the log for a specific pod:

   $ oc logs -f <pod_name> -c <container_name> -n <namespace>

   where:

   -f

   Optional: Specifies that the output follows what is being written into the logs.

   <pod_name>

   Specifies the name of the pod.

   <container_name>

   Optional: Specifies the name of a container. When a pod has more than one container, you must specify the container name.

   <namespace>

   Specify the namespace the pod is running in.

   For example:

   $ oc logs ovnkube-master-7h4q7 -n openshift-ovn-kubernetes

   $ oc logs -f ovnkube-master-7h4q7 -n openshift-ovn-kubernetes -c ovn-dbchecker

   The contents of log files are printed out.

2. Examine the most recent entries in all the containers in the ovnkube-master pods:

   $ for p in $(oc get pods --selector app=ovnkube-master -n openshift-ovn-kubernetes \
   -o jsonpath='{range.items[*]}{" "}{.metadata.name}'); \
   do echo === $p ===; for container in $(oc get pods -n openshift-ovn-kubernetes $p \
   -o json | jq -r '.status.containerStatuses[] | .name');do echo ---$container---; \
   oc logs -c $container $p -n openshift-ovn-kubernetes --tail=5; done; done

3. View the last 5 lines of every log in every container in an ovnkube-master pod using the following command:

   $ oc logs -l app=ovnkube-master -n openshift-ovn-kubernetes --all-containers --tail 5

Procedure

1. In the OpenShift Container Platform console, navigate to Workloads Pods or navigate to the pod through the resource you want to investigate.
2. Select the openshift-ovn-kubernetes project from the drop-down menu.
3. Click the name of the pod you want to investigate.
4. Click Logs. By default for the ovnkube-master the logs associated with the northd container are displayed.
5. Use the down-down menu to select logs for each container in turn.

Procedure

1. To list the current PodNetworkConnectivityCheck objects, enter the following command:

   $ oc get podnetworkconnectivitychecks -n openshift-network-diagnostics

2. View the most recent success for each connection object by using the following command:

   $ oc get podnetworkconnectivitychecks -n openshift-network-diagnostics \
   -o json | jq '.items[]| .spec.targetEndpoint,.status.successes[0]'

3. View the most recent failures for each connection object by using the following command:

   $ oc get podnetworkconnectivitychecks -n openshift-network-diagnostics \
   -o json | jq '.items[]| .spec.targetEndpoint,.status.failures[0]'

4. View the most recent outages for each connection object by using the following command:

   $ oc get podnetworkconnectivitychecks -n openshift-network-diagnostics \
   -o json | jq '.items[]| .spec.targetEndpoint,.status.outages[0]'

   The connectivity check controller also logs metrics from these checks into Prometheus.

5. View all the metrics by running the following command:

   $ oc exec prometheus-k8s-0 -n openshift-monitoring -- \
   promtool query instant  http://localhost:9090 \
   '{component="openshift-network-diagnostics"}'

6. View the latency between the source pod and the openshift api service for the last 5 minutes:

   $ oc exec prometheus-k8s-0 -n openshift-monitoring -- \
   promtool query instant  http://localhost:9090 \
   '{component="openshift-network-diagnostics"}'

Procedure

1. Create a pod variable by using the following command:

   $  POD=$(oc get pods -n openshift-ovn-kubernetes -l app=ovnkube-master -o name | head -1 | awk -F '/' '{print $NF}')

2. Run the following command on your local host to copy the binary from the ovnkube-master pods:

   $  oc cp -n openshift-ovn-kubernetes $POD:/usr/bin/ovnkube-trace ovnkube-trace

3. Make ovnkube-trace executable by running the following command:

   $  chmod +x ovnkube-trace

4. Display the options available with ovnkube-trace by running the following command:

   $  ./ovnkube-trace -help

   Expected output

   I0111 15:05:27.973305  204872 ovs.go:90] Maximum command line arguments set to: 191102
   Usage of ./ovnkube-trace:
     -dst string
       	dest: destination pod name
     -dst-ip string
       	destination IP address (meant for tests to external targets)
     -dst-namespace string
       	k8s namespace of dest pod (default "default")
     -dst-port string
       	dst-port: destination port (default "80")
     -kubeconfig string
       	absolute path to the kubeconfig file
     -loglevel string
       	loglevel: klog level (default "0")
     -ovn-config-namespace string
       	namespace used by ovn-config itself
     -service string
       	service: destination service name
     -skip-detrace
       	skip ovn-detrace command
     -src string
       	src: source pod name
     -src-namespace string
       	k8s namespace of source pod (default "default")
     -tcp
       	use tcp transport protocol
     -udp
       	use udp transport protocol

   The command-line arguments supported are familiar Kubernetes constructs, such as namespaces, pods, services so you do not need to find the MAC address, the IP address of the destination nodes, or the ICMP type.

   The log levels are:

   • 0 (minimal output)
   • 2 (more verbose output showing results of trace commands)
   • 5 (debug output)

Procedure

1. Start a web service in the default namespace by entering the following command:

   $ oc run web --namespace=default --image=nginx --labels="app=web" --expose --port=80

2. List the pods running in the openshift-dns namespace:

   oc get pods -n openshift-dns

   Example output

   NAME                  READY   STATUS    RESTARTS   AGE
   dns-default-467qw     2/2     Running   0          49m
   dns-default-6prvx     2/2     Running   0          53m
   dns-default-fkqr8     2/2     Running   0          53m
   dns-default-qv2rg     2/2     Running   0          49m
   dns-default-s29vr     2/2     Running   0          49m
   dns-default-vdsbn     2/2     Running   0          53m
   node-resolver-6thtt   1/1     Running   0          53m
   node-resolver-7ksdn   1/1     Running   0          49m
   node-resolver-8sthh   1/1     Running   0          53m
   node-resolver-c5ksw   1/1     Running   0          50m
   node-resolver-gbvdp   1/1     Running   0          53m
   node-resolver-sxhkd   1/1     Running   0          50m

3. Run the following ovn-kube-trace command to verify DNS resolution is working:

   $ ./ovnkube-trace \
     -src-namespace default \ 

   1

   
     -src web \ 

   2

   
     -dst-namespace openshift-dns \ 

   3

   
     -dst dns-default-467qw \ 

   4

   
     -udp -dst-port 53 \ 

   5

   
     -loglevel 0 

   6

   1

   Namespace of the source pod

   2

   Source pod name

   3

   Namespace of destination pod

   4

   Destination pod name

   5

   Use the udp transport protocol. Port 53 is the port the DNS service uses.

   6

   Set the log level to 1 (0 is minimal and 5 is debug)

   Expected output

   I0116 10:19:35.601303   17900 ovs.go:90] Maximum command line arguments set to: 191102
   ovn-trace source pod to destination pod indicates success from web to dns-default-467qw
   ovn-trace destination pod to source pod indicates success from dns-default-467qw to web
   ovs-appctl ofproto/trace source pod to destination pod indicates success from web to dns-default-467qw
   ovs-appctl ofproto/trace destination pod to source pod indicates success from dns-default-467qw to web
   ovn-detrace source pod to destination pod indicates success from web to dns-default-467qw
   ovn-detrace destination pod to source pod indicates success from dns-default-467qw to web

   The ouput indicates success from the deployed pod to the DNS port and also indicates that it is successful going back in the other direction. So you know bi-directional traffic is supported on UDP port 53 if my web pod wants to do dns resolution from core DNS.

Procedure

1. Create the following YAML that defines a deny-by-default policy to deny ingress from all pods in all namespaces. Save the YAML in the deny-by-default.yaml file:

   kind: NetworkPolicy
   apiVersion: networking.k8s.io/v1
   metadata:
     name: deny-by-default
     namespace: default
   spec:
     podSelector: {}
     ingress: []

2. Apply the policy by entering the following command:

   $ oc apply -f deny-by-default.yaml

   Example output

   networkpolicy.networking.k8s.io/deny-by-default created

3. Start a web service in the default namespace by entering the following command:

   $ oc run web --namespace=default --image=nginx --labels="app=web" --expose --port=80

4. Run the following command to create the prod namespace:

   $ oc create namespace prod

5. Run the following command to label the prod namespace:

   $ oc label namespace/prod purpose=production

6. Run the following command to deploy an alpine image in the prod namespace and start a shell:

   $ oc run test-6459 --namespace=prod --rm -i -t --image=alpine -- sh

7. Open another terminal session.

8. In this new terminal session run ovn-trace to verify the failure in communication between the source pod test-6459 running in namespace prod and destination pod running in the default namespace:

   $ ./ovnkube-trace \
    -src-namespace prod \
    -src test-6459 \
    -dst-namespace default \
    -dst web \
    -tcp -dst-port 80 \
    -loglevel 0

   Expected output

   I0116 14:20:47.380775   50822 ovs.go:90] Maximum command line arguments set to: 191102
   ovn-trace source pod to destination pod indicates failure from test-6459 to web

9. Increase the log level to 2 to expose the reason for the failure by running the following command:

   $ ./ovnkube-trace \
    -src-namespace prod \
    -src test-6459 \
    -dst-namespace default \
    -dst web \
    -tcp -dst-port 80 \
    -loglevel 2

   Expected output

   ct_lb_mark /* default (use --ct to customize) */
   ------------------------------------------------
    3. ls_out_acl_hint (northd.c:6092): !ct.new && ct.est && !ct.rpl && ct_mark.blocked == 0, priority 4, uuid 32d45ad4
       reg0[8] = 1;
       reg0[10] = 1;
       next;
    4. ls_out_acl (northd.c:6435): reg0[10] == 1 && (outport == @a16982411286042166782_ingressDefaultDeny), priority 2000, uuid f730a887 

   1

   
       ct_commit { ct_mark.blocked = 1; };

   1

   Ingress traffic is blocked due to the default deny policy being in place

10. Create a policy that allows traffic from all pods in a particular namespaces with a label purpose=production. Save the YAML in the web-allow-prod.yaml file:

    kind: NetworkPolicy
    apiVersion: networking.k8s.io/v1
    metadata:
      name: web-allow-prod
      namespace: default
    spec:
      podSelector:
        matchLabels:
          app: web
      policyTypes:
      - Ingress
      ingress:
      - from:
        - namespaceSelector:
            matchLabels:
              purpose: production

11. Apply the policy by entering the following command:

    $ oc apply -f web-allow-prod.yaml

12. Run ovnkube-trace to verify that traffic is now allowed by entering the following command:

    $ ./ovnkube-trace \
     -src-namespace prod \
     -src test-6459 \
     -dst-namespace default \
     -dst web \
     -tcp -dst-port 80 \
     -loglevel 0

    Expected output

    I0116 14:25:44.055207   51695 ovs.go:90] Maximum command line arguments set to: 191102
    ovn-trace source pod to destination pod indicates success from test-6459 to web
    ovn-trace destination pod to source pod indicates success from web to test-6459
    ovs-appctl ofproto/trace source pod to destination pod indicates success from test-6459 to web
    ovs-appctl ofproto/trace destination pod to source pod indicates success from web to test-6459
    ovn-detrace source pod to destination pod indicates success from test-6459 to web
    ovn-detrace destination pod to source pod indicates success from web to test-6459

13. In the open shell run the following command:

     wget -qO- --timeout=2 http://web.default

    Expected output

    <!DOCTYPE html>
    <html>
    <head>
    <title>Welcome to nginx!</title>
    <style>
    html { color-scheme: light dark; }
    body { width: 35em; margin: 0 auto;
    font-family: Tahoma, Verdana, Arial, sans-serif; }
    </style>
    </head>
    <body>
    <h1>Welcome to nginx!</h1>
    <p>If you see this page, the nginx web server is successfully installed and
    working. Further configuration is required.</p>
    
    <p>For online documentation and support please refer to
    <a href="http://nginx.org/">nginx.org</a>.<br/>
    Commercial support is available at
    <a href="http://nginx.com/">nginx.com</a>.</p>
    
    <p><em>Thank you for using nginx.</em></p>
    </body>
    </html>

Procedure

1. To backup the configuration for the cluster network, enter the following command:

   $ oc get Network.config.openshift.io cluster -o yaml > cluster-openshift-sdn.yaml

2. Verify that the OVN_SDN_MIGRATION_TIMEOUT environment variable is set and is equal to 0s by running the following command:

   #!/bin/bash
   
   if [ -n "$OVN_SDN_MIGRATION_TIMEOUT" ] && [ "$OVN_SDN_MIGRATION_TIMEOUT" = "0s" ]; then
       unset OVN_SDN_MIGRATION_TIMEOUT
   fi
   
   #loops the timeout command of the script to repeatedly check the cluster Operators until all are available.
   
   co_timeout=${OVN_SDN_MIGRATION_TIMEOUT:-1200s}
   timeout "$co_timeout" bash <<EOT
   until
     oc wait co --all --for='condition=AVAILABLE=True' --timeout=10s && \
     oc wait co --all --for='condition=PROGRESSING=False' --timeout=10s && \
     oc wait co --all --for='condition=DEGRADED=False' --timeout=10s;
   do
     sleep 10
     echo "Some ClusterOperators Degraded=False,Progressing=True,or Available=False";
   done
   EOT

3. Remove the configuration from the Cluster Network Operator (CNO) configuration object by running the following command:

   $ oc patch Network.operator.openshift.io cluster --type='merge' \
   --patch '{"spec":{"migration":null}}'

4. Delete the NodeNetworkConfigurationPolicy (NNCP) custom resource (CR) that defines the primary network interface for the OpenShift SDN network plugin by completing the following steps:

   1. Check that the existing NNCP CR bonded the primary interface to your cluster by entering the following command:

      $ oc get nncp

      Example output

      NAME          STATUS      REASON
      bondmaster0   Available   SuccessfullyConfigured

      Network Manager stores the connection profile for the bonded primary interface in the /etc/NetworkManager/system-connections system path.

   2. Remove the NNCP from your cluster:

      $ oc delete nncp <nncp_manifest_filename>

5. To prepare all the nodes for the migration, set the migration field on the CNO configuration object by running the following command:

   $ oc patch Network.operator.openshift.io cluster --type='merge' \
     --patch '{ "spec": { "migration": { "networkType": "OVNKubernetes" } } }'

   Note

   This step does not deploy OVN-Kubernetes immediately. Instead, specifying the migration field triggers the Machine Config Operator (MCO) to apply new machine configs to all the nodes in the cluster in preparation for the OVN-Kubernetes deployment.

   1. Check that the reboot is finished by running the following command:

      $ oc get mcp

   2. Check that all cluster Operators are available by running the following command:

      $ oc get co

   3. Alternatively: You can disable automatic migration of several OpenShift SDN capabilities to the OVN-Kubernetes equivalents:

      • Egress IPs
      • Egress firewall
      • Multicast

      To disable automatic migration of the configuration for any of the previously noted OpenShift SDN features, specify the following keys:

      $ oc patch Network.operator.openshift.io cluster --type='merge' \
        --patch '{
          "spec": {
            "migration": {
              "networkType": "OVNKubernetes",
              "features": {
                "egressIP": <bool>,
                "egressFirewall": <bool>,
                "multicast": <bool>
              }
            }
          }
        }'

      where:

      bool: Specifies whether to enable migration of the feature. The default is true.

6. Optional: You can customize the following settings for OVN-Kubernetes to meet your network infrastructure requirements:

   • Maximum transmission unit (MTU). Consider the following before customizing the MTU for this optional step:

     • If you use the default MTU, and you want to keep the default MTU during migration, this step can be ignored.
     • If you used a custom MTU, and you want to keep the custom MTU during migration, you must declare the custom MTU value in this step.

     • This step does not work if you want to change the MTU value during migration. Instead, you must first follow the instructions for "Changing the cluster MTU". You can then keep the custom MTU value by performing this procedure and declaring the custom MTU value in this step.

       Note

       OpenShift-SDN and OVN-Kubernetes have different overlay overhead. MTU values should be selected by following the guidelines found on the "MTU value selection" page.

   • Geneve (Generic Network Virtualization Encapsulation) overlay network port
   • OVN-Kubernetes IPv4 internal subnet

   To customize either of the previously noted settings, enter and customize the following command. If you do not need to change the default value, omit the key from the patch.

   $ oc patch Network.operator.openshift.io cluster --type=merge \
     --patch '{
       "spec":{
         "defaultNetwork":{
           "ovnKubernetesConfig":{
             "mtu":<mtu>,
             "genevePort":<port>,
             "v4InternalSubnet":"<ipv4_subnet>"
       }}}}'

   where:

   mtu

   The MTU for the Geneve overlay network. This value is normally configured automatically, but if the nodes in your cluster do not all use the same MTU, then you must set this explicitly to 100 less than the smallest node MTU value.

   port

   The UDP port for the Geneve overlay network. If a value is not specified, the default is 6081. The port cannot be the same as the VXLAN port that is used by OpenShift SDN. The default value for the VXLAN port is 4789.

   ipv4_subnet

   An IPv4 address range for internal use by OVN-Kubernetes. You must ensure that the IP address range does not overlap with any other subnet used by your OpenShift Container Platform installation. The IP address range must be larger than the maximum number of nodes that can be added to the cluster. The default value is 100.64.0.0/16.

   Example patch command to update mtu field

   $ oc patch Network.operator.openshift.io cluster --type=merge \
     --patch '{
       "spec":{
         "defaultNetwork":{
           "ovnKubernetesConfig":{
             "mtu":1200
       }}}}'

7. As the MCO updates machines in each machine config pool, it reboots each node one by one. You must wait until all the nodes are updated. Check the machine config pool status by entering the following command:

   $ oc get mcp

   A successfully updated node has the following status: UPDATED=true, UPDATING=false, DEGRADED=false.

   Note

   By default, the MCO updates one machine per pool at a time, causing the total time the migration takes to increase with the size of the cluster.

8. Confirm the status of the new machine configuration on the hosts:

   1. To list the machine configuration state and the name of the applied machine configuration, enter the following command:

      $ oc describe node | egrep "hostname|machineconfig"

      Example output

      kubernetes.io/hostname=master-0
      machineconfiguration.openshift.io/currentConfig: rendered-master-c53e221d9d24e1c8bb6ee89dd3d8ad7b
      machineconfiguration.openshift.io/desiredConfig: rendered-master-c53e221d9d24e1c8bb6ee89dd3d8ad7b
      machineconfiguration.openshift.io/reason:
      machineconfiguration.openshift.io/state: Done

      Verify that the following statements are true:

      • The value of machineconfiguration.openshift.io/state field is Done.
      • The value of the machineconfiguration.openshift.io/currentConfig field is equal to the value of the machineconfiguration.openshift.io/desiredConfig field.

   2. To confirm that the machine config is correct, enter the following command:

      $ oc get machineconfig <config_name> -o yaml | grep ExecStart

      where <config_name> is the name of the machine config from the machineconfiguration.openshift.io/currentConfig field.

      The machine config must include the following update to the systemd configuration:

      ExecStart=/usr/local/bin/configure-ovs.sh OVNKubernetes

   3. If a node is stuck in the NotReady state, investigate the machine config daemon pod logs and resolve any errors.

      1. To list the pods, enter the following command:

         $ oc get pod -n openshift-machine-config-operator

         Example output

         NAME                                         READY   STATUS    RESTARTS   AGE
         machine-config-controller-75f756f89d-sjp8b   1/1     Running   0          37m
         machine-config-daemon-5cf4b                  2/2     Running   0          43h
         machine-config-daemon-7wzcd                  2/2     Running   0          43h
         machine-config-daemon-fc946                  2/2     Running   0          43h
         machine-config-daemon-g2v28                  2/2     Running   0          43h
         machine-config-daemon-gcl4f                  2/2     Running   0          43h
         machine-config-daemon-l5tnv                  2/2     Running   0          43h
         machine-config-operator-79d9c55d5-hth92      1/1     Running   0          37m
         machine-config-server-bsc8h                  1/1     Running   0          43h
         machine-config-server-hklrm                  1/1     Running   0          43h
         machine-config-server-k9rtx                  1/1     Running   0          43h

         The names for the config daemon pods are in the following format: machine-config-daemon-<seq>. The <seq> value is a random five character alphanumeric sequence.

      2. Display the pod log for the first machine config daemon pod shown in the previous output by enter the following command:

         $ oc logs <pod> -n openshift-machine-config-operator

         where pod is the name of a machine config daemon pod.

      3. Resolve any errors in the logs shown by the output from the previous command.

9. To start the migration, configure the OVN-Kubernetes network plugin by using one of the following commands:

   • To specify the network provider without changing the cluster network IP address block, enter the following command:

     $ oc patch Network.config.openshift.io cluster \
       --type='merge' --patch '{ "spec": { "networkType": "OVNKubernetes" } }'

   • To specify a different cluster network IP address block, enter the following command:

     $ oc patch Network.config.openshift.io cluster \
       --type='merge' --patch '{
         "spec": {
           "clusterNetwork": [
             {
               "cidr": "<cidr>",
               "hostPrefix": <prefix>
             }
           ],
           "networkType": "OVNKubernetes"
         }
       }'

     where cidr is a CIDR block and prefix is the slice of the CIDR block apportioned to each node in your cluster. You cannot use any CIDR block that overlaps with the 100.64.0.0/16 CIDR block because the OVN-Kubernetes network provider uses this block internally.

     Important

     You cannot change the service network address block during the migration.

10. Verify that the Multus daemon set rollout is complete before continuing with subsequent steps:

    $ oc -n openshift-multus rollout status daemonset/multus

    The name of the Multus pods is in the form of multus-<xxxxx> where <xxxxx> is a random sequence of letters. It might take several moments for the pods to restart.

    Example output

    Waiting for daemon set "multus" rollout to finish: 1 out of 6 new pods have been updated...
    ...
    Waiting for daemon set "multus" rollout to finish: 5 of 6 updated pods are available...
    daemon set "multus" successfully rolled out

11. To complete changing the network plugin, reboot each node in your cluster. You can reboot the nodes in your cluster with either of the following approaches:

    Important

    The following scripts reboot all of the nodes in the cluster at the same time. This can cause your cluster to be unstable. Another option is to reboot your nodes manually one at a time. Rebooting nodes one-by-one causes considerable downtime in a cluster with many nodes.

    Cluster Operators will not work correctly before you reboot the nodes.

    • With the oc rsh command, you can use a bash script similar to the following:

      #!/bin/bash
      readarray -t POD_NODES <<< "$(oc get pod -n openshift-machine-config-operator -o wide| grep daemon|awk '{print $1" "$7}')"
      
      for i in "${POD_NODES[@]}"
      do
        read -r POD NODE <<< "$i"
        until oc rsh -n openshift-machine-config-operator "$POD" chroot /rootfs shutdown -r +1
          do
            echo "cannot reboot node $NODE, retry" && sleep 3
          done
      done

    • With the ssh command, you can use a bash script similar to the following. The script assumes that you have configured sudo to not prompt for a password.

      #!/bin/bash
      
      for ip in $(oc get nodes  -o jsonpath='{.items[*].status.addresses[?(@.type=="InternalIP")].address}')
      do
         echo "reboot node $ip"
         ssh -o StrictHostKeyChecking=no core@$ip sudo shutdown -r -t 3
      done

12. Confirm that the migration succeeded:

    1. To confirm that the network plugin is OVN-Kubernetes, enter the following command. The value of status.networkType must be OVNKubernetes.

       $ oc get network.config/cluster -o jsonpath='{.status.networkType}{"\n"}'

    2. To confirm that the cluster nodes are in the Ready state, enter the following command:

       $ oc get nodes

    3. To confirm that your pods are not in an error state, enter the following command:

       $ oc get pods --all-namespaces -o wide --sort-by='{.spec.nodeName}'

       If pods on a node are in an error state, reboot that node.

    4. To confirm that all of the cluster Operators are not in an abnormal state, enter the following command:

       $ oc get co

       The status of every cluster Operator must be the following: AVAILABLE="True", PROGRESSING="False", DEGRADED="False". If a cluster Operator is not available or degraded, check the logs for the cluster Operator for more information.

13. Complete the following steps only if the migration succeeds and your cluster is in a good state:

    1. To remove the migration configuration from the CNO configuration object, enter the following command:

       $ oc patch Network.operator.openshift.io cluster --type='merge' \
         --patch '{ "spec": { "migration": null } }'

    2. To remove custom configuration for the OpenShift SDN network provider, enter the following command:

       $ oc patch Network.operator.openshift.io cluster --type='merge' \
         --patch '{ "spec": { "defaultNetwork": { "openshiftSDNConfig": null } } }'

    3. To remove the OpenShift SDN network provider namespace, enter the following command:

       $ oc delete namespace openshift-sdn

    4. After a successful migration operation, remove the network.openshift.io/network-type-migration- annotation from the network.config custom resource by entering the following command:

       $ oc annotate network.config cluster network.openshift.io/network-type-migration-

Procedure

1. Stop all of the machine configuration pools managed by the Machine Config Operator (MCO):

   • Stop the master configuration pool by entering the following command in your CLI:

     $ oc patch MachineConfigPool master --type='merge' --patch \
       '{ "spec": { "paused": true } }'

   • Stop the worker machine configuration pool by entering the following command in your CLI:

     $ oc patch MachineConfigPool worker --type='merge' --patch \
       '{ "spec":{ "paused": true } }'

2. To prepare for the migration, set the migration field to null by entering the following command in your CLI:

   $ oc patch Network.operator.openshift.io cluster --type='merge' \
     --patch '{ "spec": { "migration": null } }'

3. Check that the migration status is empty for the Network.config.openshift.io object by entering the following command in your CLI. Empty command output indicates that the object is not in a migration operation.

   $ oc get Network.config cluster -o jsonpath='{.status.migration}'

4. Apply the patch to the Network.operator.openshift.io object to set the network plugin back to OpenShift SDN by entering the following command in your CLI:

   $ oc patch Network.operator.openshift.io cluster --type='merge' \
     --patch '{ "spec": { "migration": { "networkType": "OpenShiftSDN" } } }'

   Important

   If you applied the patch to the Network.config.openshift.io object before the patch operation finalizes on the Network.operator.openshift.io object, the Cluster Network Operator (CNO) enters into a degradation state and this causes a slight delay until the CNO recovers from the degraded state.

5. Confirm that the migration status of the network plugin for the Network.config.openshift.io cluster object is OpenShiftSDN by entering the following command in your CLI:

   $ oc get Network.config cluster -o jsonpath='{.status.migration.networkType}'

6. Apply the patch to the Network.config.openshift.io object to set the network plugin back to OpenShift SDN by entering the following command in your CLI:

   $ oc patch Network.config.openshift.io cluster --type='merge' \
     --patch '{ "spec": { "networkType": "OpenShiftSDN" } }'

7. Optional: Disable automatic migration of several OVN-Kubernetes capabilities to the OpenShift SDN equivalents:

   • Egress IPs
   • Egress firewall
   • Multicast

   To disable automatic migration of the configuration for any of the previously noted OpenShift SDN features, specify the following keys:

   $ oc patch Network.operator.openshift.io cluster --type='merge' \
     --patch '{
       "spec": {
         "migration": {
           "networkType": "OpenShiftSDN",
           "features": {
             "egressIP": <bool>,
             "egressFirewall": <bool>,
             "multicast": <bool>
           }
         }
       }
     }'

   where:

   bool: Specifies whether to enable migration of the feature. The default is true.

8. Optional: You can customize the following settings for OpenShift SDN to meet your network infrastructure requirements:

   • Maximum transmission unit (MTU)
   • VXLAN port

   To customize either or both of the previously noted settings, customize and enter the following command in your CLI. If you do not need to change the default value, omit the key from the patch.

   $ oc patch Network.operator.openshift.io cluster --type=merge \
     --patch '{
       "spec":{
         "defaultNetwork":{
           "openshiftSDNConfig":{
             "mtu":<mtu>,
             "vxlanPort":<port>
       }}}}'

   mtu

   The MTU for the VXLAN overlay network. This value is normally configured automatically, but if the nodes in your cluster do not all use the same MTU, then you must set this explicitly to 50 less than the smallest node MTU value.

   port

   The UDP port for the VXLAN overlay network. If a value is not specified, the default is 4789. The port cannot be the same as the Geneve port that is used by OVN-Kubernetes. The default value for the Geneve port is 6081.

   Example patch command

   $ oc patch Network.operator.openshift.io cluster --type=merge \
     --patch '{
       "spec":{
         "defaultNetwork":{
           "openshiftSDNConfig":{
             "mtu":1200
       }}}}'

9. Reboot each node in your cluster. You can reboot the nodes in your cluster with either of the following approaches:

   • With the oc rsh command, you can use a bash script similar to the following:

     #!/bin/bash
     readarray -t POD_NODES <<< "$(oc get pod -n openshift-machine-config-operator -o wide| grep daemon|awk '{print $1" "$7}')"
     
     for i in "${POD_NODES[@]}"
     do
       read -r POD NODE <<< "$i"
       until oc rsh -n openshift-machine-config-operator "$POD" chroot /rootfs shutdown -r +1
         do
           echo "cannot reboot node $NODE, retry" && sleep 3
         done
     done

   • With the ssh command, you can use a bash script similar to the following. The script assumes that you have configured sudo to not prompt for a password.

     #!/bin/bash
     
     for ip in $(oc get nodes  -o jsonpath='{.items[*].status.addresses[?(@.type=="InternalIP")].address}')
     do
        echo "reboot node $ip"
        ssh -o StrictHostKeyChecking=no core@$ip sudo shutdown -r -t 3
     done

10. Wait until the Multus daemon set rollout completes. Run the following command to see your rollout status:

    $ oc -n openshift-multus rollout status daemonset/multus

    The name of the Multus pods is in the form of multus-<xxxxx> where <xxxxx> is a random sequence of letters. It might take several moments for the pods to restart.

    Example output

    Waiting for daemon set "multus" rollout to finish: 1 out of 6 new pods have been updated...
    ...
    Waiting for daemon set "multus" rollout to finish: 5 of 6 updated pods are available...
    daemon set "multus" successfully rolled out

11. After the nodes in your cluster have rebooted and the multus pods are rolled out, start all of the machine configuration pools by running the following commands::

    • Start the master configuration pool:

      $ oc patch MachineConfigPool master --type='merge' --patch \
        '{ "spec": { "paused": false } }'

    • Start the worker configuration pool:

      $ oc patch MachineConfigPool worker --type='merge' --patch \
        '{ "spec": { "paused": false } }'

    As the MCO updates machines in each config pool, it reboots each node.

    By default the MCO updates a single machine per pool at a time, so the time that the migration requires to complete grows with the size of the cluster.

12. Confirm the status of the new machine configuration on the hosts:

    1. To list the machine configuration state and the name of the applied machine configuration, enter the following command in your CLI:

       $ oc describe node | egrep "hostname|machineconfig"

       Example output

       kubernetes.io/hostname=master-0
       machineconfiguration.openshift.io/currentConfig: rendered-master-c53e221d9d24e1c8bb6ee89dd3d8ad7b
       machineconfiguration.openshift.io/desiredConfig: rendered-master-c53e221d9d24e1c8bb6ee89dd3d8ad7b
       machineconfiguration.openshift.io/reason:
       machineconfiguration.openshift.io/state: Done

       Verify that the following statements are true:

       • The value of machineconfiguration.openshift.io/state field is Done.
       • The value of the machineconfiguration.openshift.io/currentConfig field is equal to the value of the machineconfiguration.openshift.io/desiredConfig field.

    2. To confirm that the machine config is correct, enter the following command in your CLI:

       $ oc get machineconfig <config_name> -o yaml

       where <config_name> is the name of the machine config from the machineconfiguration.openshift.io/currentConfig field.

13. Confirm that the migration succeeded:

    1. To confirm that the network plugin is OpenShift SDN, enter the following command in your CLI. The value of status.networkType must be OpenShiftSDN.

       $ oc get Network.config/cluster -o jsonpath='{.status.networkType}{"\n"}'

    2. To confirm that the cluster nodes are in the Ready state, enter the following command in your CLI:

       $ oc get nodes

    3. If a node is stuck in the NotReady state, investigate the machine config daemon pod logs and resolve any errors.

       1. To list the pods, enter the following command in your CLI:

          $ oc get pod -n openshift-machine-config-operator

          Example output

          NAME                                         READY   STATUS    RESTARTS   AGE
          machine-config-controller-75f756f89d-sjp8b   1/1     Running   0          37m
          machine-config-daemon-5cf4b                  2/2     Running   0          43h
          machine-config-daemon-7wzcd                  2/2     Running   0          43h
          machine-config-daemon-fc946                  2/2     Running   0          43h
          machine-config-daemon-g2v28                  2/2     Running   0          43h
          machine-config-daemon-gcl4f                  2/2     Running   0          43h
          machine-config-daemon-l5tnv                  2/2     Running   0          43h
          machine-config-operator-79d9c55d5-hth92      1/1     Running   0          37m
          machine-config-server-bsc8h                  1/1     Running   0          43h
          machine-config-server-hklrm                  1/1     Running   0          43h
          machine-config-server-k9rtx                  1/1     Running   0          43h

          The names for the config daemon pods are in the following format: machine-config-daemon-<seq>. The <seq> value is a random five character alphanumeric sequence.

       2. To display the pod log for each machine config daemon pod shown in the previous output, enter the following command in your CLI:

          $ oc logs <pod> -n openshift-machine-config-operator

          where pod is the name of a machine config daemon pod.

       3. Resolve any errors in the logs shown by the output from the previous command.

    4. To confirm that your pods are not in an error state, enter the following command in your CLI:

       $ oc get pods --all-namespaces -o wide --sort-by='{.spec.nodeName}'

       If pods on a node are in an error state, reboot that node.

14. Complete the following steps only if the migration succeeds and your cluster is in a good state:

    1. To remove the migration configuration from the Cluster Network Operator configuration object, enter the following command in your CLI:

       $ oc patch Network.operator.openshift.io cluster --type='merge' \
         --patch '{ "spec": { "migration": null } }'

    2. To remove the OVN-Kubernetes configuration, enter the following command in your CLI:

       $ oc patch Network.operator.openshift.io cluster --type='merge' \
         --patch '{ "spec": { "defaultNetwork": { "ovnKubernetesConfig":null } } }'

    3. To remove the OVN-Kubernetes network provider namespace, enter the following command in your CLI:

       $ oc delete namespace openshift-ovn-kubernetes

Procedure

1. Install the ansible-core package, minimum version 2.15. The following example command shows how to install the ansible-core package on Red Hat Enterprise Linux (RHEL):

   $ sudo dnf install -y ansible-core

2. Create an ansible.cfg file and add information similar to the following example to the file. Ensure that file exists in the same directory as where the ansible-galaxy commands and the playbooks run.

   $ cat << EOF >> ansible.cfg
   [galaxy]
   server_list = automation_hub, validated
   
   [galaxy_server.automation_hub]
   url=https://console.redhat.com/api/automation-hub/content/published/
   auth_url=https://sso.redhat.com/auth/realms/redhat-external/protocol/openid-connect/token
   token=
   
   #[galaxy_server.release_galaxy]
   #url=https://galaxy.ansible.com/
   
   [galaxy_server.validated]
   url=https://console.redhat.com/api/automation-hub/content/validated/
   auth_url=https://sso.redhat.com/auth/realms/redhat-external/protocol/openid-connect/token
   token=
   EOF

3. From the Ansible Automation Platform web console, go to the Connect to Hub page and complete the following steps:

   1. In the Offline token section of the page, click the Load token button.
   2. After the token loads, click the Copy to clipboard icon.
   3. Open the ansible.cfg file and paste the API token in the token= parameter. The API token is required for authenticating against the server URL specified in the ansible.cfg file.

4. Install the network.offline_migration_sdn_to_ovnk Ansible collection by entering the following ansible-galaxy command:

   $ ansible-galaxy collection install network.offline_migration_sdn_to_ovnk

5. Verify that the network.offline_migration_sdn_to_ovnk Ansible collection is installed on your system:

   $ ansible-galaxy collection list | grep network.offline_migration_sdn_to_ovnk

   Example output

   network.offline_migration_sdn_to_ovnk   1.0.2

   The network.offline_migration_sdn_to_ovnk Ansible collection is saved in the default path of ~/.ansible/collections/ansible_collections/network/offline_migration_sdn_to_ovnk/.

6. Configure rollback features in the playbooks/playbook-migration.yml file:

   # ...
       rollback_disable_auto_migration: true
       rollback_egress_ip: false
       rollback_egress_firewall: false
       rollback_multicast: false
       rollback_mtu: 1400
       rollback_vxlanPort: 4790
   # ...

   rollback_disable_auto_migration

   Disables the auto-migration of OVN-Kubernetes plug-in features to the OpenShift SDN CNI plug-in. If you disable auto-migration of features, you must also set the rollback_egress_ip, rollback_egress_firewall, and rollback_multicast parameters to false. If you need to enable auto-migration of features, set the parameter to false.

   rollback_mtu

   Optional parameter that sets a specific maximum transmission unit (MTU) to your cluster network after the migration process.

   rollback_vxlanPort

   Optional parameter that sets a VXLAN (Virtual Extensible LAN) port for use by OpenShift SDN CNI plug-in. The default value for the parameter is 4790.

7. To run the playbooks/playbook-rollback.yml file, enter the following command:

   $ ansible-playbook -v playbooks/playbook-rollback.yml

Procedure

1. Back up the configuration for the cluster network by running the following command:

   $ oc get Network.config.openshift.io cluster -o yaml > cluster-kuryr.yaml

2. To set the CLUSTERID variable, run the following command:

   $ CLUSTERID=$(oc get infrastructure.config.openshift.io cluster -o=jsonpath='{.status.infrastructureName}')

3. To prepare all the nodes for the migration, set the migration field on the Cluster Network Operator configuration object by running the following command:

   $ oc patch Network.operator.openshift.io cluster --type='merge' \
     --patch '{ "spec": { "migration": { "networkType": "OVNKubernetes" } } }'

   Note

   This step does not deploy OVN-Kubernetes immediately. Specifying the migration field triggers the Machine Config Operator (MCO) to apply new machine configs to all the nodes in the cluster. This prepares the cluster for the OVN-Kubernetes deployment.

4. Optional: Customize the following settings for OVN-Kubernetes for your network infrastructure requirements:

   • Maximum transmission unit (MTU)
   • Geneve (Generic Network Virtualization Encapsulation) overlay network port
   • OVN-Kubernetes IPv4 internal subnet
   • OVN-Kubernetes IPv6 internal subnet

   To customize these settings, enter and customize the following command:

   $ oc patch Network.operator.openshift.io cluster --type=merge \
     --patch '{
       "spec":{
         "defaultNetwork":{
           "ovnKubernetesConfig":{
             "mtu":<mtu>,
             "genevePort":<port>,
             "v4InternalSubnet":"<ipv4_subnet>",
             "v6InternalSubnet":"<ipv6_subnet>"
       }}}}'

   where:

   mtu

   Specifies the MTU for the Geneve overlay network. This value is normally configured automatically, but if the nodes in your cluster do not all use the same MTU, then you must set this explicitly to 100 less than the smallest node MTU value.

   port

   Specifies the UDP port for the Geneve overlay network. If a value is not specified, the default is 6081. The port cannot be the same as the VXLAN port that is used by Kuryr. The default value for the VXLAN port is 4789.

   ipv4_subnet

   Specifies an IPv4 address range for internal use by OVN-Kubernetes. You must ensure that the IP address range does not overlap with any other subnet used by your OpenShift Container Platform installation. The IP address range must be larger than the maximum number of nodes that can be added to the cluster. The default value is 100.64.0.0/16.

   ipv6_subnet

   Specifies an IPv6 address range for internal use by OVN-Kubernetes. You must ensure that the IP address range does not overlap with any other subnet used by your OpenShift Container Platform installation. The IP address range must be larger than the maximum number of nodes that can be added to the cluster. The default value is fd98::/48.

   If you do not need to change the default value, omit the key from the patch.

   Example patch command to update mtu field

   $ oc patch Network.operator.openshift.io cluster --type=merge \
     --patch '{
       "spec":{
         "defaultNetwork":{
           "ovnKubernetesConfig":{
             "mtu":1200
       }}}}'

5. Check the machine config pool status by entering the following command:

   $ oc get mcp

   While the MCO updates machines in each machine config pool, it reboots each node one by one. You must wait until all the nodes are updated before continuing.

   A successfully updated node has the following status: UPDATED=true, UPDATING=false, DEGRADED=false.

   Note

   By default, the MCO updates one machine per pool at a time. Large clusters take more time to migrate than small clusters.

6. Confirm the status of the new machine configuration on the hosts:

   1. To list the machine configuration state and the name of the applied machine configuration, enter the following command:

      $ oc describe node | egrep "hostname|machineconfig"

      Example output

      kubernetes.io/hostname=master-0
      machineconfiguration.openshift.io/currentConfig: rendered-master-c53e221d9d24e1c8bb6ee89dd3d8ad7b 

      1

      
      machineconfiguration.openshift.io/desiredConfig: rendered-master-c53e221d9d24e1c8bb6ee89dd3d8ad7b 

      2

      
      machineconfiguration.openshift.io/reason:
      machineconfiguration.openshift.io/state: Done

   2. Review the output from the previous step. The following statements must be true:

      • The value of machineconfiguration.openshift.io/state field is Done.
      • The value of the machineconfiguration.openshift.io/currentConfig field is equal to the value of the machineconfiguration.openshift.io/desiredConfig field.

   3. To confirm that the machine config is correct, enter the following command:

      $ oc get machineconfig <config_name> -o yaml | grep ExecStart

      where:

      <config_name>

      Specifies the name of the machine config from the machineconfiguration.openshift.io/currentConfig field.

      The machine config must include the following update to the systemd configuration:

      Example output

      ExecStart=/usr/local/bin/configure-ovs.sh OVNKubernetes

   4. If a node is stuck in the NotReady state, investigate the machine config daemon pod logs and resolve any errors:

      1. To list the pods, enter the following command:

         $ oc get pod -n openshift-machine-config-operator

         Example output

         NAME                                         READY   STATUS    RESTARTS   AGE
         machine-config-controller-75f756f89d-sjp8b   1/1     Running   0          37m
         machine-config-daemon-5cf4b                  2/2     Running   0          43h
         machine-config-daemon-7wzcd                  2/2     Running   0          43h
         machine-config-daemon-fc946                  2/2     Running   0          43h
         machine-config-daemon-g2v28                  2/2     Running   0          43h
         machine-config-daemon-gcl4f                  2/2     Running   0          43h
         machine-config-daemon-l5tnv                  2/2     Running   0          43h
         machine-config-operator-79d9c55d5-hth92      1/1     Running   0          37m
         machine-config-server-bsc8h                  1/1     Running   0          43h
         machine-config-server-hklrm                  1/1     Running   0          43h
         machine-config-server-k9rtx                  1/1     Running   0          43h

         The names for the config daemon pods are in the following format: machine-config-daemon-<seq>. The <seq> value is a random five character alphanumeric sequence.

      2. Display the pod log for the first machine config daemon pod shown in the previous output by enter the following command:

         $ oc logs <pod> -n openshift-machine-config-operator

         where:

         <pod>

         Specifies the name of a machine config daemon pod.

      3. Resolve any errors in the logs shown by the output from the previous command.

7. To start the migration, configure the OVN-Kubernetes network plugin by using one of the following commands:

   • To specify the network provider without changing the cluster network IP address block, enter the following command:

     $ oc patch Network.config.openshift.io cluster \
       --type='merge' --patch '{ "spec": { "networkType": "OVNKubernetes" } }'

   • To specify a different cluster network IP address block, enter the following command:

     $ oc patch Network.config.openshift.io cluster \
       --type='merge' --patch '{
         "spec": {
           "clusterNetwork": [
             {
               "cidr": "<cidr>",
               "hostPrefix": "<prefix>"
             }
           ]
           "networkType": "OVNKubernetes"
         }
       }'

     where:

     <cidr>

     Specifies a CIDR block.

     <prefix>

     Specifies a slice of the CIDR block that is apportioned to each node in your cluster.

     Important

     You cannot change the service network address block during the migration.

     You cannot use any CIDR block that overlaps with the 100.64.0.0/16 CIDR block because the OVN-Kubernetes network provider uses this block internally.

8. Verify that the Multus daemon set rollout is complete by entering the following command:

   $ oc -n openshift-multus rollout status daemonset/multus

   The name of the Multus pods is in the form of multus-<xxxxx>, where <xxxxx> is a random sequence of letters. It might take several moments for the pods to restart.

   Example output

   Waiting for daemon set "multus" rollout to finish: 1 out of 6 new pods have been updated...
   ...
   Waiting for daemon set "multus" rollout to finish: 5 of 6 updated pods are available...
   daemon set "multus" successfully rolled out

9. To complete the migration, reboot each node in your cluster. For example, you can use a bash script similar to the following example. The script assumes that you can connect to each host by using ssh and that you have configured sudo to not prompt for a password:

   #!/bin/bash
   
   for ip in $(oc get nodes  -o jsonpath='{.items[*].status.addresses[?(@.type=="InternalIP")].address}')
   do
      echo "reboot node $ip"
      ssh -o StrictHostKeyChecking=no core@$ip sudo shutdown -r -t 3
   done

   Note

   If SSH access is not available, you can use the openstack command:

   $ for name in $(openstack server list --name ${CLUSTERID}\* -f value -c Name); do openstack server reboot $name; done

   Alternatively, you might be able to reboot each node through the management portal for your infrastructure provider. Otherwise, contact the appropriate authority to either gain access to the virtual machines through SSH or the management portal and OpenStack client.

Verification

1. Confirm that the migration succeeded, and then remove the migration resources:

   1. To confirm that the network plugin is OVN-Kubernetes, enter the following command.

      $ oc get network.config/cluster -o jsonpath='{.status.networkType}{"\n"}'

      The value of status.networkType must be OVNKubernetes.

   2. To confirm that the cluster nodes are in the Ready state, enter the following command:

      $ oc get nodes

   3. To confirm that your pods are not in an error state, enter the following command:

      $ oc get pods --all-namespaces -o wide --sort-by='{.spec.nodeName}'

      If pods on a node are in an error state, reboot that node.

   4. To confirm that all of the cluster Operators are not in an abnormal state, enter the following command:

      $ oc get co

      The status of every cluster Operator must be the following: AVAILABLE="True", PROGRESSING="False", DEGRADED="False". If a cluster Operator is not available or degraded, check the logs for the cluster Operator for more information.

      Important

      Do not proceed if any of the previous verification steps indicate errors. You might encounter pods that have a Terminating state due to finalizers that are removed during clean up. They are not an error indication.

2. If the migration completed and your cluster is in a good state, remove the migration configuration from the CNO configuration object by entering the following command:

   $ oc patch Network.operator.openshift.io cluster --type='merge' \
     --patch '{ "spec": { "migration": null } }'

Procedure

1. Create a clean-up Python virtual environment:

   1. Create a temporary directory for your environment. For example:

      $ python3 -m venv /tmp/venv

      The virtual environment located in /tmp/venv directory is used in all clean up examples.

   2. Enter the virtual environment. For example:

      $ source /tmp/venv/bin/activate

   3. Upgrade the pip command in the virtual environment by running the following command:

      (venv) $ pip install pip --upgrade

   4. Install the required Python packages by running the following command:

      (venv) $ pip install openstacksdk==0.54.0 python-openstackclient==5.5.0 python-octaviaclient==2.3.0

2. In your terminal, set variables to cluster and Kuryr identifiers by running the following commands:

   1. Set the cluster ID:

      (venv) $ CLUSTERID=$(oc get infrastructure.config.openshift.io cluster -o=jsonpath='{.status.infrastructureName}')

   2. Set the cluster tag:

      (venv) $ CLUSTERTAG="openshiftClusterID=${CLUSTERID}"

   3. Set the router ID:

      (venv) $ ROUTERID=$(oc get kuryrnetwork -A --no-headers -o custom-columns=":status.routerId"|head -n 1)

3. Create a Bash function that removes finalizers from specified resources by running the following command:

   (venv) $ function REMFIN {
       local resource=$1
       local finalizer=$2
       for res in $(oc get $resource -A --template='{{range $i,$p := .items}}{{ $p.metadata.name }}|{{ $p.metadata.namespace }}{{"\n"}}{{end}}'); do
           name=${res%%|*}
           ns=${res##*|}
           yaml=$(oc get -n $ns $resource $name -o yaml)
           if echo "${yaml}" | grep -q "${finalizer}"; then
               echo "${yaml}" | grep -v  "${finalizer}" | oc replace -n $ns $resource $name -f -
           fi
       done
   }

   The function takes two parameters: the first parameter is name of the resource, and the second parameter is the finalizer to remove. The named resource is removed from the cluster and its definition is replaced with copied data, excluding the specified finalizer.

4. To remove Kuryr finalizers from services, enter the following command:

   (venv) $ REMFIN services kuryr.openstack.org/service-finalizer

5. To remove the Kuryr service-subnet-gateway-ip service, enter the following command:

   (venv) $ if $(oc get -n openshift-kuryr service service-subnet-gateway-ip &>/dev/null); then
       oc -n openshift-kuryr delete service service-subnet-gateway-ip
   fi

6. To remove all tagged RHOSP load balancers from Octavia, enter the following command:

   (venv) $ for lb in $(openstack loadbalancer list --tags $CLUSTERTAG -f value -c id); do
       openstack loadbalancer delete --cascade $lb
   done

7. To remove Kuryr finalizers from all KuryrLoadBalancer CRs, enter the following command:

   (venv) $ REMFIN kuryrloadbalancers.openstack.org kuryr.openstack.org/kuryrloadbalancer-finalizers

8. To remove the openshift-kuryr namespace, enter the following command:

   (venv) $ oc delete namespace openshift-kuryr

9. To remove the Kuryr service subnet from the router, enter the following command:

   (venv) $ openstack router remove subnet $ROUTERID ${CLUSTERID}-kuryr-service-subnet

10. To remove the Kuryr service network, enter the following command:

    (venv) $ openstack network delete ${CLUSTERID}-kuryr-service-network

11. To remove Kuryr finalizers from all pods, enter the following command:

    (venv) $ REMFIN pods kuryr.openstack.org/pod-finalizer

12. To remove Kuryr finalizers from all KuryrPort CRs, enter the following command:

    (venv) $ REMFIN kuryrports.openstack.org kuryr.openstack.org/kuryrport-finalizer

    This command deletes the KuryrPort CRs.

13. To remove Kuryr finalizers from network policies, enter the following command:

    (venv) $ REMFIN networkpolicy kuryr.openstack.org/networkpolicy-finalizer

14. To remove Kuryr finalizers from remaining network policies, enter the following command:

    (venv) $ REMFIN kuryrnetworkpolicies.openstack.org kuryr.openstack.org/networkpolicy-finalizer

15. To remove subports that Kuryr created from trunks, enter the following command:

    (venv) $ read -ra trunks <<< $(python -c "import openstack; n = openstack.connect().network; print(' '.join([x.id for x in n.trunks(any_tags='$CLUSTERTAG')]))") && \
    i=0 && \
    for trunk in "${trunks[@]}"; do
        i=$((i+1))
        echo "Processing trunk $trunk, ${i}/${#trunks[@]}."
        subports=()
        for subport in $(python -c "import openstack; n = openstack.connect().network; print(' '.join([x['port_id'] for x in n.get_trunk('$trunk').sub_ports if '$CLUSTERTAG' in n.get_port(x['port_id']).tags]))"); do
            subports+=("$subport");
        done
        args=()
        for sub in "${subports[@]}" ; do
            args+=("--subport $sub")
        done
        if [ ${#args[@]} -gt 0 ]; then
            openstack network trunk unset ${args[*]} $trunk
        fi
    done

16. To retrieve all networks and subnets from KuryrNetwork CRs and remove ports, router interfaces and the network itself, enter the following command:

    (venv) $ mapfile -t kuryrnetworks < <(oc get kuryrnetwork -A --template='{{range $i,$p := .items}}{{ $p.status.netId }}|{{ $p.status.subnetId }}{{"\n"}}{{end}}') && \
    i=0 && \
    for kn in "${kuryrnetworks[@]}"; do
        i=$((i+1))
        netID=${kn%%|*}
        subnetID=${kn##*|}
        echo "Processing network $netID, ${i}/${#kuryrnetworks[@]}"
        # Remove all ports from the network.
        for port in $(python -c "import openstack; n = openstack.connect().network; print(' '.join([x.id for x in n.ports(network_id='$netID') if x.device_owner != 'network:router_interface']))"); do
            ( openstack port delete $port ) &
    
            # Only allow 20 jobs in parallel.
            if [[ $(jobs -r -p | wc -l) -ge 20 ]]; then
                wait -n
            fi
        done
        wait
    
        # Remove the subnet from the router.
        openstack router remove subnet $ROUTERID $subnetID
    
        # Remove the network.
        openstack network delete $netID
    done

17. To remove the Kuryr security group, enter the following command:

    (venv) $ openstack security group delete ${CLUSTERID}-kuryr-pods-security-group

18. To remove all tagged subnet pools, enter the following command:

    (venv) $ for subnetpool in $(openstack subnet pool list --tags $CLUSTERTAG -f value -c ID); do
        openstack subnet pool delete $subnetpool
    done

19. To check that all of the networks based on KuryrNetwork CRs were removed, enter the following command:

    (venv) $ networks=$(oc get kuryrnetwork -A --no-headers -o custom-columns=":status.netId") && \
    for existingNet in $(openstack network list --tags $CLUSTERTAG -f value -c ID); do
        if [[ $networks =~ $existingNet ]]; then
            echo "Network still exists: $existingNet"
        fi
    done

    If the command returns any existing networks, intestigate and remove them before you continue.

20. To remove security groups that are related to network policy, enter the following command:

    (venv) $ for sgid in $(openstack security group list -f value -c ID -c Description | grep 'Kuryr-Kubernetes Network Policy' | cut -f 1 -d ' '); do
        openstack security group delete $sgid
    done

21. To remove finalizers from KuryrNetwork CRs, enter the following command:

    (venv) $ REMFIN kuryrnetworks.openstack.org kuryrnetwork.finalizers.kuryr.openstack.org

22. To remove the Kuryr router, enter the following command:

    (venv) $ if $(python3 -c "import sys; import openstack; n = openstack.connect().network; r = n.get_router('$ROUTERID'); sys.exit(0) if r.description != 'Created By OpenShift Installer' else sys.exit(1)"); then
        openstack router delete $ROUTERID
    fi

Procedure

1. To specify IPv6 address blocks for the cluster and service networks, create a file containing the following YAML:

   - op: add
     path: /spec/clusterNetwork/-
     value: 

   1

   
       cidr: fd01::/48
       hostPrefix: 64
   - op: add
     path: /spec/serviceNetwork/-
     value: fd02::/112 

   2

   1

   Specify an object with the cidr and hostPrefix parameters. The host prefix must be 64 or greater. The IPv6 Classless Inter-Domain Routing (CIDR) prefix must be large enough to accommodate the specified host prefix.

   1 2

   Specify an IPv6 CIDR with a prefix of 112. Kubernetes uses only the lowest 16 bits. For a prefix of 112, IP addresses are assigned from 112 to 128 bits.

2. To patch the cluster network configuration, enter the following command:

   $ oc patch network.config.openshift.io cluster \
     --type='json' --patch-file <file>.yaml

   where:

   file

   Specifies the name of the file you created in the previous step.

   Example output

   network.config.openshift.io/cluster patched

1. Display the network configuration:

   $ oc describe network

   Example output

   Status:
     Cluster Network:
       Cidr:               10.128.0.0/14
       Host Prefix:        23
       Cidr:               fd01::/48
       Host Prefix:        64
     Cluster Network MTU:  1400
     Network Type:         OVNKubernetes
     Service Network:
       172.30.0.0/16
       fd02::/112

Procedure

1. Edit the networks.config.openshift.io custom resource (CR) by running the following command:

   $ oc edit networks.config.openshift.io

2. Remove the IPv4 or IPv6 configuration that you added to the cidr and the hostPrefix parameters from completing the "Converting to a dual-stack cluster network " procedure steps.

Verification

1. To create a namespace with network policies complete the following steps:

   1. Create a namespace for verification:

      $ cat <<EOF| oc create -f -
      kind: Namespace
      apiVersion: v1
      metadata:
        name: verify-audit-logging
        annotations:
          k8s.ovn.org/acl-logging: '{ "deny": "alert", "allow": "alert" }'
      EOF

      Example output

      namespace/verify-audit-logging created

   2. Create network policies for the namespace:

      $ cat <<EOF| oc create -n verify-audit-logging -f -
      apiVersion: networking.k8s.io/v1
      kind: NetworkPolicy
      metadata:
        name: deny-all
      spec:
        podSelector:
          matchLabels:
        policyTypes:
        - Ingress
        - Egress
      ---
      apiVersion: networking.k8s.io/v1
      kind: NetworkPolicy
      metadata:
        name: allow-from-same-namespace
        namespace: verify-audit-logging
      spec:
        podSelector: {}
        policyTypes:
         - Ingress
         - Egress
        ingress:
          - from:
              - podSelector: {}
        egress:
          - to:
             - namespaceSelector:
                matchLabels:
                  kubernetes.io/metadata.name: verify-audit-logging
      EOF

      Example output

      networkpolicy.networking.k8s.io/deny-all created
      networkpolicy.networking.k8s.io/allow-from-same-namespace created

2. Create a pod for source traffic in the default namespace:

   $ cat <<EOF| oc create -n default -f -
   apiVersion: v1
   kind: Pod
   metadata:
     name: client
   spec:
     containers:
       - name: client
         image: registry.access.redhat.com/rhel7/rhel-tools
         command: ["/bin/sh", "-c"]
         args:
           ["sleep inf"]
   EOF

3. Create two pods in the verify-audit-logging namespace:

   $ for name in client server; do
   cat <<EOF| oc create -n verify-audit-logging -f -
   apiVersion: v1
   kind: Pod
   metadata:
     name: ${name}
   spec:
     containers:
       - name: ${name}
         image: registry.access.redhat.com/rhel7/rhel-tools
         command: ["/bin/sh", "-c"]
         args:
           ["sleep inf"]
   EOF
   done

   Example output

   pod/client created
   pod/server created

4. To generate traffic and produce network policy audit log entries, complete the following steps:

   1. Obtain the IP address for pod named server in the verify-audit-logging namespace:

      $ POD_IP=$(oc get pods server -n verify-audit-logging -o jsonpath='{.status.podIP}')

   2. Ping the IP address from the previous command from the pod named client in the default namespace and confirm that all packets are dropped:

      $ oc exec -it client -n default -- /bin/ping -c 2 $POD_IP

      Example output

      PING 10.128.2.55 (10.128.2.55) 56(84) bytes of data.
      
      --- 10.128.2.55 ping statistics ---
      2 packets transmitted, 0 received, 100% packet loss, time 2041ms

   3. Ping the IP address saved in the POD_IP shell environment variable from the pod named client in the verify-audit-logging namespace and confirm that all packets are allowed:

      $ oc exec -it client -n verify-audit-logging -- /bin/ping -c 2 $POD_IP

      Example output

      PING 10.128.0.86 (10.128.0.86) 56(84) bytes of data.
      64 bytes from 10.128.0.86: icmp_seq=1 ttl=64 time=2.21 ms
      64 bytes from 10.128.0.86: icmp_seq=2 ttl=64 time=0.440 ms
      
      --- 10.128.0.86 ping statistics ---
      2 packets transmitted, 2 received, 0% packet loss, time 1001ms
      rtt min/avg/max/mdev = 0.440/1.329/2.219/0.890 ms

5. Display the latest entries in the network policy audit log:

   $ for pod in $(oc get pods -n openshift-ovn-kubernetes -l app=ovnkube-node --no-headers=true | awk '{ print $1 }') ; do
       oc exec -it $pod -n openshift-ovn-kubernetes -- tail -4 /var/log/ovn/acl-audit-log.log
     done

   Example output

   Defaulting container name to ovn-controller.
   Use 'oc describe pod/ovnkube-node-hdb8v -n openshift-ovn-kubernetes' to see all of the containers in this pod.
   2021-06-13T19:33:11.590Z|00005|acl_log(ovn_pinctrl0)|INFO|name="verify-audit-logging_ingressDefaultDeny", verdict=drop, severity=alert: icmp,vlan_tci=0x0000,dl_src=0a:58:0a:80:02:39,dl_dst=0a:58:0a:80:02:37,nw_src=10.128.2.57,nw_dst=10.128.2.55,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=8,icmp_code=0
   2021-06-13T19:33:12.614Z|00006|acl_log(ovn_pinctrl0)|INFO|name="verify-audit-logging_ingressDefaultDeny", verdict=drop, severity=alert: icmp,vlan_tci=0x0000,dl_src=0a:58:0a:80:02:39,dl_dst=0a:58:0a:80:02:37,nw_src=10.128.2.57,nw_dst=10.128.2.55,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=8,icmp_code=0
   2021-06-13T19:44:10.037Z|00007|acl_log(ovn_pinctrl0)|INFO|name="verify-audit-logging_allow-from-same-namespace_0", verdict=allow, severity=alert: icmp,vlan_tci=0x0000,dl_src=0a:58:0a:80:02:3b,dl_dst=0a:58:0a:80:02:3a,nw_src=10.128.2.59,nw_dst=10.128.2.58,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=8,icmp_code=0
   2021-06-13T19:44:11.037Z|00008|acl_log(ovn_pinctrl0)|INFO|name="verify-audit-logging_allow-from-same-namespace_0", verdict=allow, severity=alert: icmp,vlan_tci=0x0000,dl_src=0a:58:0a:80:02:3b,dl_dst=0a:58:0a:80:02:3a,nw_src=10.128.2.59,nw_dst=10.128.2.58,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=8,icmp_code=0

Verification

1. To find the names of the OVN-Kubernetes control plane pods, enter the following command:

   $ oc get pods -l app=ovnkube-master -n openshift-ovn-kubernetes

   Example output

   NAME                   READY   STATUS    RESTARTS   AGE
   ovnkube-master-fvtnh   6/6     Running   0          122m
   ovnkube-master-hsgmm   6/6     Running   0          122m
   ovnkube-master-qcmdc   6/6     Running   0          122m

2. Verify that IPsec is enabled on your cluster by entering the following command. The command output must state true to indicate that the node has IPsec enabled.

   $ oc -n openshift-ovn-kubernetes rsh ovnkube-master-<pod_number_sequence> \ 

   1

   
     ovn-nbctl --no-leader-only get nb_global . ipsec

   1

   Replace <pod_number_sequence> with the random sequence of letters, fvtnh, for a data plane pod from the previous step.

Procedure

1. To disable IPsec encryption, enter the following command:

   $ oc patch networks.operator.openshift.io/cluster --type=json \
     -p='[{"op":"remove", "path":"/spec/defaultNetwork/ovnKubernetesConfig/ipsecConfig"}]'

2. To find the name of the OVN-Kubernetes data plane pod that exists on the master node in your cluster, enter the following command:

   $ oc get pods -n openshift-ovn-kubernetes -l=app=ovnkube-master

   Example output

   ovnkube-master-5xqbf                      8/8     Running   0              28m
   ...

3. Verify that the master node in your cluster has IPsec disabled by entering the following command. The command output must state false to indicate that the node has IPsec disabled.

   $ oc -n openshift-ovn-kubernetes -c nbdb rsh ovnkube-master-<pod_number_sequence> \

   1

   
     ovn-nbctl --no-leader-only get nb_global . ipsec

   1

   Replace <pod_number_sequence> with the random sequence of letters, such as 5xqbf, for the data plane pod from the previous step.

4. To remove the IPsec ovn-ipsec daemonset pod from the openshift-ovn-kubernetes namespace on the node, enter the following command:

   $ oc delete daemonset ovn-ipsec -n openshift-ovn-kubernetes 

   1

   1

   The ovn-ipsec daemonset configures IPsec connections for east-west traffic on the node.

5. Verify that the ovn-ipsec daemonset pod was removed from the all nodes in your cluster by entering the following command. If the command output does not list the pod, the removal operation is successful.

   $ oc get pods -n openshift-ovn-kubernetes -l=app=ovn-ipsec

   Note

   You might need to re-run the command for deleting the pod because sometimes the initial command attempt might not delete the pod.

6. Optional: You can increase the size of your cluster MTU by 46 bytes because there is no longer any overhead from the IPsec ESP header in IP packets.

Procedure

1. Create a policy rule:

   1. Create a <policy_name>.yaml file where <policy_name> describes the egress policy rules.
   2. In the file you created, define an egress policy object.

2. Enter the following command to create the policy object. Replace <policy_name> with the name of the policy and <project> with the project that the rule applies to.

   $ oc create -f <policy_name>.yaml -n <project>

   In the following example, a new EgressFirewall object is created in a project named project1:

   $ oc create -f default.yaml -n project1

   Example output

   egressfirewall.k8s.ovn.org/v1 created

3. Optional: Save the <policy_name>.yaml file so that you can make changes later.

Procedure

1. Optional: To view the names of the EgressFirewall objects defined in your cluster, enter the following command:

   $ oc get egressfirewall --all-namespaces

2. To inspect a policy, enter the following command. Replace <policy_name> with the name of the policy to inspect.

   $ oc describe egressfirewall <policy_name>

   Example output

   Name:		default
   Namespace:	project1
   Created:	20 minutes ago
   Labels:		<none>
   Annotations:	<none>
   Rule:		Allow to 1.2.3.0/24
   Rule:		Allow to www.example.com
   Rule:		Deny to 0.0.0.0/0

Procedure

1. Find the name of the EgressFirewall object for the project. Replace <project> with the name of the project.

   $ oc get -n <project> egressfirewall

2. Optional: If you did not save a copy of the EgressFirewall object when you created the egress network firewall, enter the following command to create a copy.

   $ oc get -n <project> egressfirewall <name> -o yaml > <filename>.yaml

   Replace <project> with the name of the project. Replace <name> with the name of the object. Replace <filename> with the name of the file to save the YAML to.

3. After making changes to the policy rules, enter the following command to replace the EgressFirewall object. Replace <filename> with the name of the file containing the updated EgressFirewall object.

   $ oc replace -f <filename>.yaml

Procedure

1. Find the name of the EgressFirewall object for the project. Replace <project> with the name of the project.

   $ oc get -n <project> egressfirewall

2. Enter the following command to delete the EgressFirewall object. Replace <project> with the name of the project and <name> with the name of the object.

   $ oc delete -n <project> egressfirewall <name>