Chapter 26. OVN-Kubernetes network plugin

Procedure

1. Run the following command to get all resources, endpoints, and ConfigMaps in the OVN-Kubernetes project:

   $ oc get all,ep,cm -n openshift-ovn-kubernetes

   Copy to Clipboard Toggle word wrap

   Example output

   Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
   NAME                                         READY   STATUS    RESTARTS       AGE
   pod/ovnkube-control-plane-65c6f55656-6d55h   2/2     Running   0              114m
   pod/ovnkube-control-plane-65c6f55656-fd7vw   2/2     Running   2 (104m ago)   114m
   pod/ovnkube-node-bcvts                       8/8     Running   0              113m
   pod/ovnkube-node-drgvv                       8/8     Running   0              113m
   pod/ovnkube-node-f2pxt                       8/8     Running   0              113m
   pod/ovnkube-node-frqsb                       8/8     Running   0              105m
   pod/ovnkube-node-lbxkk                       8/8     Running   0              105m
   pod/ovnkube-node-tt7bx                       8/8     Running   1 (102m ago)   105m
   
   NAME                                   TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)             AGE
   service/ovn-kubernetes-control-plane   ClusterIP   None         <none>        9108/TCP            114m
   service/ovn-kubernetes-node            ClusterIP   None         <none>        9103/TCP,9105/TCP   114m
   
   NAME                          DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE
   daemonset.apps/ovnkube-node   6         6         6       6            6           beta.kubernetes.io/os=linux   114m
   
   NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/ovnkube-control-plane   3/3     3            3           114m
   
   NAME                                               DESIRED   CURRENT   READY   AGE
   replicaset.apps/ovnkube-control-plane-65c6f55656   3         3         3       114m
   
   NAME                                     ENDPOINTS                                               AGE
   endpoints/ovn-kubernetes-control-plane   10.0.0.3:9108,10.0.0.4:9108,10.0.0.5:9108               114m
   endpoints/ovn-kubernetes-node            10.0.0.3:9105,10.0.0.4:9105,10.0.0.5:9105 + 9 more...   114m
   
   NAME                                 DATA   AGE
   configmap/control-plane-status       1      113m
   configmap/kube-root-ca.crt           1      114m
   configmap/openshift-service-ca.crt   1      114m
   configmap/ovn-ca                     1      114m
   configmap/ovnkube-config             1      114m
   configmap/signer-ca                  1      114m

   Copy to Clipboard Toggle word wrap

   There is one ovnkube-node pod for each node in the cluster. The ovnkube-config config map has the OpenShift Container Platform OVN-Kubernetes configurations.

2. List all of the containers in the ovnkube-node pods by running the following command:

   $ oc get pods ovnkube-node-bcvts -o jsonpath='{.spec.containers[*].name}' -n openshift-ovn-kubernetes

   Copy to Clipboard Toggle word wrap

   Expected output

   ovn-controller ovn-acl-logging kube-rbac-proxy-node kube-rbac-proxy-ovn-metrics northd nbdb sbdb ovnkube-controller

   Copy to Clipboard Toggle word wrap

   The ovnkube-node pod is made up of several containers. It is responsible for hosting the northbound database (nbdb container), the southbound database (sbdb container), the north daemon (northd container), ovn-controller and the ovnkube-controller container. The ovnkube-controller container watches for API objects like pods, egress IPs, namespaces, services, endpoints, egress firewall, and network policies. It is also responsible for allocating pod IP from the available subnet pool for that node.

3. List all the containers in the ovnkube-control-plane pods by running the following command:

   $ oc get pods ovnkube-control-plane-65c6f55656-6d55h -o jsonpath='{.spec.containers[*].name}' -n openshift-ovn-kubernetes

   Copy to Clipboard Toggle word wrap

   Expected output

   kube-rbac-proxy ovnkube-cluster-manager

   Copy to Clipboard Toggle word wrap

   The ovnkube-control-plane pod has a container (ovnkube-cluster-manager) that resides on each OpenShift Container Platform node. The ovnkube-cluster-manager container allocates pod subnet, transit switch subnet IP and join switch subnet IP to each node in the cluster. The kube-rbac-proxy container monitors metrics for the ovnkube-cluster-manager container.

1. List pods by running the following command:

   $ oc get po -n openshift-ovn-kubernetes

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                                     READY   STATUS    RESTARTS      AGE
   ovnkube-control-plane-8444dff7f9-4lh9k   2/2     Running   0             27m
   ovnkube-control-plane-8444dff7f9-5rjh9   2/2     Running   0             27m
   ovnkube-node-55xs2                       8/8     Running   0             26m
   ovnkube-node-7r84r                       8/8     Running   0             16m
   ovnkube-node-bqq8p                       8/8     Running   0             17m
   ovnkube-node-mkj4f                       8/8     Running   0             26m
   ovnkube-node-mlr8k                       8/8     Running   0             26m
   ovnkube-node-wqn2m                       8/8     Running   0             16m

   Copy to Clipboard Toggle word wrap

2. Optional: To list the pods with node information, run the following command:

   $ oc get pods -n openshift-ovn-kubernetes -owide

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                                     READY   STATUS    RESTARTS      AGE   IP           NODE                                       NOMINATED NODE   READINESS GATES
   ovnkube-control-plane-8444dff7f9-4lh9k   2/2     Running   0             27m   10.0.0.3     ci-ln-t487nnb-72292-mdcnq-master-1         <none>           <none>
   ovnkube-control-plane-8444dff7f9-5rjh9   2/2     Running   0             27m   10.0.0.4     ci-ln-t487nnb-72292-mdcnq-master-2         <none>           <none>
   ovnkube-node-55xs2                       8/8     Running   0             26m   10.0.0.4     ci-ln-t487nnb-72292-mdcnq-master-2         <none>           <none>
   ovnkube-node-7r84r                       8/8     Running   0             17m   10.0.128.3   ci-ln-t487nnb-72292-mdcnq-worker-b-wbz7z   <none>           <none>
   ovnkube-node-bqq8p                       8/8     Running   0             17m   10.0.128.2   ci-ln-t487nnb-72292-mdcnq-worker-a-lh7ms   <none>           <none>
   ovnkube-node-mkj4f                       8/8     Running   0             27m   10.0.0.5     ci-ln-t487nnb-72292-mdcnq-master-0         <none>           <none>
   ovnkube-node-mlr8k                       8/8     Running   0             27m   10.0.0.3     ci-ln-t487nnb-72292-mdcnq-master-1         <none>           <none>
   ovnkube-node-wqn2m                       8/8     Running   0             17m   10.0.128.4   ci-ln-t487nnb-72292-mdcnq-worker-c-przlm   <none>           <none>

   Copy to Clipboard Toggle word wrap

3. Navigate into a pod to look at the northbound database by running the following command:

   $ oc rsh -c nbdb -n openshift-ovn-kubernetes ovnkube-node-55xs2

   Copy to Clipboard Toggle word wrap

4. Run the following command to show all the objects in the northbound database:

   $ ovn-nbctl show

   Copy to Clipboard Toggle word wrap

   The output is too long to list here. The list includes the NAT rules, logical switches, load balancers and so on.

   You can narrow down and focus on specific components by using some of the following optional commands:

   1. Run the following command to show the list of logical routers:

      $ oc exec -n openshift-ovn-kubernetes -it ovnkube-node-55xs2 \
      -c northd -- ovn-nbctl lr-list

      Copy to Clipboard Toggle word wrap

      Example output

      45339f4f-7d0b-41d0-b5f9-9fca9ce40ce6 (GR_ci-ln-t487nnb-72292-mdcnq-master-2)
      96a0a0f0-e7ed-4fec-8393-3195563de1b8 (ovn_cluster_router)

      Copy to Clipboard Toggle word wrap

      Note

      From this output you can see there is router on each node plus an ovn_cluster_router.

   2. Run the following command to show the list of logical switches:

      $ oc exec -n openshift-ovn-kubernetes -it ovnkube-node-55xs2 \
      -c nbdb -- ovn-nbctl ls-list

      Copy to Clipboard Toggle word wrap

      Example output

      bdd7dc3d-d848-4a74-b293-cc15128ea614 (ci-ln-t487nnb-72292-mdcnq-master-2)
      b349292d-ee03-4914-935f-1940b6cb91e5 (ext_ci-ln-t487nnb-72292-mdcnq-master-2)
      0aac0754-ea32-4e33-b086-35eeabf0a140 (join)
      992509d7-2c3f-4432-88db-c179e43592e5 (transit_switch)

      Copy to Clipboard Toggle word wrap

      Note

      From this output you can see there is an ext switch for each node plus switches with the node name itself and a join switch.

   3. Run the following command to show the list of load balancers:

      $ oc exec -n openshift-ovn-kubernetes -it ovnkube-node-55xs2 \
      -c nbdb -- ovn-nbctl lb-list

      Copy to Clipboard Toggle word wrap

      Example output

      UUID                                    LB                  PROTO      VIP                     IPs
      7c84c673-ed2a-4436-9a1f-9bc5dd181eea    Service_default/    tcp        172.30.0.1:443          10.0.0.3:6443,169.254.169.2:6443,10.0.0.5:6443
      4d663fd9-ddc8-4271-b333-4c0e279e20bb    Service_default/    tcp        172.30.0.1:443          10.0.0.3:6443,10.0.0.4:6443,10.0.0.5:6443
      292eb07f-b82f-4962-868a-4f541d250bca    Service_openshif    tcp        172.30.105.247:443      10.129.0.12:8443
      034b5a7f-bb6a-45e9-8e6d-573a82dc5ee3    Service_openshif    tcp        172.30.192.38:443       10.0.0.3:10259,10.0.0.4:10259,10.0.0.5:10259
      a68bb53e-be84-48df-bd38-bdd82fcd4026    Service_openshif    tcp        172.30.161.125:8443     10.129.0.32:8443
      6cc21b3d-2c54-4c94-8ff5-d8e017269c2e    Service_openshif    tcp        172.30.3.144:443        10.129.0.22:8443
      37996ffd-7268-4862-a27f-61cd62e09c32    Service_openshif    tcp        172.30.181.107:443      10.129.0.18:8443
      81d4da3c-f811-411f-ae0c-bc6713d0861d    Service_openshif    tcp        172.30.228.23:443       10.129.0.29:8443
      ac5a4f3b-b6ba-4ceb-82d0-d84f2c41306e    Service_openshif    tcp        172.30.14.240:9443      10.129.0.36:9443
      c88979fb-1ef5-414b-90ac-43b579351ac9    Service_openshif    tcp        172.30.231.192:9001     10.128.0.5:9001,10.128.2.5:9001,10.129.0.5:9001,10.129.2.4:9001,10.130.0.3:9001,10.131.0.3:9001
      fcb0a3fb-4a77-4230-a84a-be45dce757e8    Service_openshif    tcp        172.30.189.92:443       10.130.0.17:8440
      67ef3e7b-ceb9-4bf0-8d96-b43bde4c9151    Service_openshif    tcp        172.30.67.218:443       10.129.0.9:8443
      d0032fba-7d5e-424a-af25-4ab9b5d46e81    Service_openshif    tcp        172.30.102.137:2379     10.0.0.3:2379,10.0.0.4:2379,10.0.0.5:2379
                                                                  tcp        172.30.102.137:9979     10.0.0.3:9979,10.0.0.4:9979,10.0.0.5:9979
      7361c537-3eec-4e6c-bc0c-0522d182abd4    Service_openshif    tcp        172.30.198.215:9001     10.0.0.3:9001,10.0.0.4:9001,10.0.0.5:9001,10.0.128.2:9001,10.0.128.3:9001,10.0.128.4:9001
      0296c437-1259-410b-a6fd-81c310ad0af5    Service_openshif    tcp        172.30.198.215:9001     10.0.0.3:9001,169.254.169.2:9001,10.0.0.5:9001,10.0.128.2:9001,10.0.128.3:9001,10.0.128.4:9001
      5d5679f5-45b8-479d-9f7c-08b123c688b8    Service_openshif    tcp        172.30.38.253:17698     10.128.0.52:17698,10.129.0.84:17698,10.130.0.60:17698
      2adcbab4-d1c9-447d-9573-b5dc9f2efbfa    Service_openshif    tcp        172.30.148.52:443       10.0.0.4:9202,10.0.0.5:9202
                                                                  tcp        172.30.148.52:444       10.0.0.4:9203,10.0.0.5:9203
                                                                  tcp        172.30.148.52:445       10.0.0.4:9204,10.0.0.5:9204
                                                                  tcp        172.30.148.52:446       10.0.0.4:9205,10.0.0.5:9205
      2a33a6d7-af1b-4892-87cc-326a380b809b    Service_openshif    tcp        172.30.67.219:9091      10.129.2.16:9091,10.131.0.16:9091
                                                                  tcp        172.30.67.219:9092      10.129.2.16:9092,10.131.0.16:9092
                                                                  tcp        172.30.67.219:9093      10.129.2.16:9093,10.131.0.16:9093
                                                                  tcp        172.30.67.219:9094      10.129.2.16:9094,10.131.0.16:9094
      f56f59d7-231a-4974-99b3-792e2741ec8d    Service_openshif    tcp        172.30.89.212:443       10.128.0.41:8443,10.129.0.68:8443,10.130.0.44:8443
      08c2c6d7-d217-4b96-b5d8-c80c4e258116    Service_openshif    tcp        172.30.102.137:2379     10.0.0.3:2379,169.254.169.2:2379,10.0.0.5:2379
                                                                  tcp        172.30.102.137:9979     10.0.0.3:9979,169.254.169.2:9979,10.0.0.5:9979
      60a69c56-fc6a-4de6-bd88-3f2af5ba5665    Service_openshif    tcp        172.30.10.193:443       10.129.0.25:8443
      ab1ef694-0826-4671-a22c-565fc2d282ec    Service_openshif    tcp        172.30.196.123:443      10.128.0.33:8443,10.129.0.64:8443,10.130.0.37:8443
      b1fb34d3-0944-4770-9ee3-2683e7a630e2    Service_openshif    tcp        172.30.158.93:8443      10.129.0.13:8443
      95811c11-56e2-4877-be1e-c78ccb3a82a9    Service_openshif    tcp        172.30.46.85:9001       10.130.0.16:9001
      4baba1d1-b873-4535-884c-3f6fc07a50fd    Service_openshif    tcp        172.30.28.87:443        10.129.0.26:8443
      6c2e1c90-f0ca-484e-8a8e-40e71442110a    Service_openshif    udp        172.30.0.10:53          10.128.0.13:5353,10.128.2.6:5353,10.129.0.39:5353,10.129.2.6:5353,10.130.0.11:5353,10.131.0.9:5353

      Copy to Clipboard Toggle word wrap

      Note

      From this truncated output you can see there are many OVN-Kubernetes load balancers. Load balancers in OVN-Kubernetes are representations of services.

5. Run the following command to display the options available with the command ovn-nbctl:

   $ oc exec -n openshift-ovn-kubernetes -it ovnkube-node-55xs2 \
   -c nbdb ovn-nbctl --help

   Copy to Clipboard Toggle word wrap

1. List the pods by running the following command:

   $ oc get po -n openshift-ovn-kubernetes

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                                     READY   STATUS    RESTARTS      AGE
   ovnkube-control-plane-8444dff7f9-4lh9k   2/2     Running   0             27m
   ovnkube-control-plane-8444dff7f9-5rjh9   2/2     Running   0             27m
   ovnkube-node-55xs2                       8/8     Running   0             26m
   ovnkube-node-7r84r                       8/8     Running   0             16m
   ovnkube-node-bqq8p                       8/8     Running   0             17m
   ovnkube-node-mkj4f                       8/8     Running   0             26m
   ovnkube-node-mlr8k                       8/8     Running   0             26m
   ovnkube-node-wqn2m                       8/8     Running   0             16m

   Copy to Clipboard Toggle word wrap

2. Optional: To list the pods with node information, run the following command:

   $ oc get pods -n openshift-ovn-kubernetes -owide

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                                     READY   STATUS    RESTARTS      AGE   IP           NODE                                       NOMINATED NODE   READINESS GATES
   ovnkube-control-plane-8444dff7f9-4lh9k   2/2     Running   0             27m   10.0.0.3     ci-ln-t487nnb-72292-mdcnq-master-1         <none>           <none>
   ovnkube-control-plane-8444dff7f9-5rjh9   2/2     Running   0             27m   10.0.0.4     ci-ln-t487nnb-72292-mdcnq-master-2         <none>           <none>
   ovnkube-node-55xs2                       8/8     Running   0             26m   10.0.0.4     ci-ln-t487nnb-72292-mdcnq-master-2         <none>           <none>
   ovnkube-node-7r84r                       8/8     Running   0             17m   10.0.128.3   ci-ln-t487nnb-72292-mdcnq-worker-b-wbz7z   <none>           <none>
   ovnkube-node-bqq8p                       8/8     Running   0             17m   10.0.128.2   ci-ln-t487nnb-72292-mdcnq-worker-a-lh7ms   <none>           <none>
   ovnkube-node-mkj4f                       8/8     Running   0             27m   10.0.0.5     ci-ln-t487nnb-72292-mdcnq-master-0         <none>           <none>
   ovnkube-node-mlr8k                       8/8     Running   0             27m   10.0.0.3     ci-ln-t487nnb-72292-mdcnq-master-1         <none>           <none>
   ovnkube-node-wqn2m                       8/8     Running   0             17m   10.0.128.4   ci-ln-t487nnb-72292-mdcnq-worker-c-przlm   <none>           <none>

   Copy to Clipboard Toggle word wrap

3. Navigate into a pod to look at the southbound database:

   $ oc rsh -c sbdb -n openshift-ovn-kubernetes ovnkube-node-55xs2

   Copy to Clipboard Toggle word wrap

4. Run the following command to show all the objects in the southbound database:

   $ ovn-sbctl show

   Copy to Clipboard Toggle word wrap

   Example output

   Chassis "5db31703-35e9-413b-8cdf-69e7eecb41f7"
       hostname: ci-ln-9gp362t-72292-v2p94-worker-a-8bmwz
       Encap geneve
           ip: "10.0.128.4"
           options: {csum="true"}
       Port_Binding tstor-ci-ln-9gp362t-72292-v2p94-worker-a-8bmwz
   Chassis "070debed-99b7-4bce-b17d-17e720b7f8bc"
       hostname: ci-ln-9gp362t-72292-v2p94-worker-b-svmp6
       Encap geneve
           ip: "10.0.128.2"
           options: {csum="true"}
       Port_Binding k8s-ci-ln-9gp362t-72292-v2p94-worker-b-svmp6
       Port_Binding rtoe-GR_ci-ln-9gp362t-72292-v2p94-worker-b-svmp6
       Port_Binding openshift-monitoring_alertmanager-main-1
       Port_Binding rtoj-GR_ci-ln-9gp362t-72292-v2p94-worker-b-svmp6
       Port_Binding etor-GR_ci-ln-9gp362t-72292-v2p94-worker-b-svmp6
       Port_Binding cr-rtos-ci-ln-9gp362t-72292-v2p94-worker-b-svmp6
       Port_Binding openshift-e2e-loki_loki-promtail-qcrcz
       Port_Binding jtor-GR_ci-ln-9gp362t-72292-v2p94-worker-b-svmp6
       Port_Binding openshift-multus_network-metrics-daemon-mkd4t
       Port_Binding openshift-ingress-canary_ingress-canary-xtvj4
       Port_Binding openshift-ingress_router-default-6c76cbc498-pvlqk
       Port_Binding openshift-dns_dns-default-zz582
       Port_Binding openshift-monitoring_thanos-querier-57585899f5-lbf4f
       Port_Binding openshift-network-diagnostics_network-check-target-tn228
       Port_Binding openshift-monitoring_prometheus-k8s-0
       Port_Binding openshift-image-registry_image-registry-68899bd877-xqxjj
   Chassis "179ba069-0af1-401c-b044-e5ba90f60fea"
       hostname: ci-ln-9gp362t-72292-v2p94-master-0
       Encap geneve
           ip: "10.0.0.5"
           options: {csum="true"}
       Port_Binding tstor-ci-ln-9gp362t-72292-v2p94-master-0
   Chassis "68c954f2-5a76-47be-9e84-1cb13bd9dab9"
       hostname: ci-ln-9gp362t-72292-v2p94-worker-c-mjf9w
       Encap geneve
           ip: "10.0.128.3"
           options: {csum="true"}
       Port_Binding tstor-ci-ln-9gp362t-72292-v2p94-worker-c-mjf9w
   Chassis "2de65d9e-9abf-4b6e-a51d-a1e038b4d8af"
       hostname: ci-ln-9gp362t-72292-v2p94-master-2
       Encap geneve
           ip: "10.0.0.4"
           options: {csum="true"}
       Port_Binding tstor-ci-ln-9gp362t-72292-v2p94-master-2
   Chassis "1d371cb8-5e21-44fd-9025-c4b162cc4247"
       hostname: ci-ln-9gp362t-72292-v2p94-master-1
       Encap geneve
           ip: "10.0.0.3"
           options: {csum="true"}
       Port_Binding tstor-ci-ln-9gp362t-72292-v2p94-master-1

   Copy to Clipboard Toggle word wrap

   This detailed output shows the chassis and the ports that are attached to the chassis which in this case are all of the router ports and anything that runs like host networking. Any pods communicate out to the wider network using source network address translation (SNAT). Their IP address is translated into the IP address of the node that the pod is running on and then sent out into the network.

   In addition to the chassis information the southbound database has all the logic flows and those logic flows are then sent to the ovn-controller running on each of the nodes. The ovn-controller translates the logic flows into open flow rules and ultimately programs OpenvSwitch so that your pods can then follow open flow rules and make it out of the network.

5. Run the following command to display the options available with the command ovn-sbctl:

   $ oc exec -n openshift-ovn-kubernetes -it ovnkube-node-55xs2 \
   -c sbdb ovn-sbctl --help

   Copy to Clipboard Toggle word wrap

Procedure

1. Review the details of the ovnkube-node readiness probe by running the following command:

   $ oc get pods -n openshift-ovn-kubernetes -l app=ovnkube-node \
   -o json | jq '.items[0].spec.containers[] | .name,.readinessProbe'

   Copy to Clipboard Toggle word wrap

   The readiness probe for the northbound and southbound database containers in the ovnkube-node pod checks for the health of the databases and the ovnkube-controller container.

   The ovnkube-controller container in the ovnkube-node pod has a readiness probe to verify the presence of the OVN-Kubernetes CNI configuration file, the absence of which would indicate that the pod is not running or is not ready to accept requests to configure pods.

2. Show all events including the probe failures, for the namespace by using the following command:

   $ oc get events -n openshift-ovn-kubernetes

   Copy to Clipboard Toggle word wrap

3. Show the events for just a specific pod:

   $ oc describe pod ovnkube-node-9lqfk -n openshift-ovn-kubernetes

   Copy to Clipboard Toggle word wrap

4. Show the messages and statuses from the cluster network operator:

   $ oc get co/network -o json | jq '.status.conditions[]'

   Copy to Clipboard Toggle word wrap

5. Show the ready status of each container in ovnkube-node pods by running the following script:

   $ for p in $(oc get pods --selector app=ovnkube-node -n openshift-ovn-kubernetes \
   -o jsonpath='{range.items[*]}{" "}{.metadata.name}'); do echo === $p ===;  \
   oc get pods -n openshift-ovn-kubernetes $p -o json | jq '.status.containerStatuses[] | .name, .ready'; \
   done

   Copy to Clipboard Toggle word wrap
   Note

   The expectation is all container statuses are reporting as true. Failure of a readiness probe sets the status to false.

Procedure (UI)

1. In the Administrator perspective, select Observe Alerting. The three main pages in the Alerting UI in this perspective are the Alerts, Silences, and Alerting Rules pages.
2. View the rules for OVN-Kubernetes alerts by selecting Observe Alerting Alerting Rules.

Procedure

1. View active or firing alerts by running the following commands.

   1. Set the alert manager route environment variable by running the following command:

      $ ALERT_MANAGER=$(oc get route alertmanager-main -n openshift-monitoring \
      -o jsonpath='{@.spec.host}')

      Copy to Clipboard Toggle word wrap

   2. Issue a curl request to the alert manager route API by running the following command, replacing $ALERT_MANAGER with the URL of your Alertmanager instance:

      $ curl -s -k -H "Authorization: Bearer $(oc create token prometheus-k8s -n openshift-monitoring)" https://$ALERT_MANAGER/api/v1/alerts | jq '.data[] | "\(.labels.severity) \(.labels.alertname) \(.labels.pod) \(.labels.container) \(.labels.endpoint) \(.labels.instance)"'

      Copy to Clipboard Toggle word wrap

2. View alerting rules by running the following command:

   $ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -s 'http://localhost:9090/api/v1/rules' | jq '.data.groups[].rules[] | select(((.name|contains("ovn")) or (.name|contains("OVN")) or (.name|contains("Ovn")) or (.name|contains("North")) or (.name|contains("South"))) and .type=="alerting")'

   Copy to Clipboard Toggle word wrap

Procedure

1. View the log for a specific pod:

   $ oc logs -f <pod_name> -c <container_name> -n <namespace>

   Copy to Clipboard Toggle word wrap

   where:

   -f

   Optional: Specifies that the output follows what is being written into the logs.

   <pod_name>

   Specifies the name of the pod.

   <container_name>

   Optional: Specifies the name of a container. When a pod has more than one container, you must specify the container name.

   <namespace>

   Specify the namespace the pod is running in.

   For example:

   $ oc logs ovnkube-node-5dx44 -n openshift-ovn-kubernetes

   Copy to Clipboard Toggle word wrap

   $ oc logs -f ovnkube-node-5dx44 -c ovnkube-controller -n openshift-ovn-kubernetes

   Copy to Clipboard Toggle word wrap

   The contents of log files are printed out.

2. Examine the most recent entries in all the containers in the ovnkube-node pods:

   $ for p in $(oc get pods --selector app=ovnkube-node -n openshift-ovn-kubernetes \
   -o jsonpath='{range.items[*]}{" "}{.metadata.name}'); \
   do echo === $p ===; for container in $(oc get pods -n openshift-ovn-kubernetes $p \
   -o json | jq -r '.status.containerStatuses[] | .name');do echo ---$container---; \
   oc logs -c $container $p -n openshift-ovn-kubernetes --tail=5; done; done

   Copy to Clipboard Toggle word wrap

3. View the last 5 lines of every log in every container in an ovnkube-node pod using the following command:

   $ oc logs -l app=ovnkube-node -n openshift-ovn-kubernetes --all-containers --tail 5

   Copy to Clipboard Toggle word wrap

Procedure

1. In the OpenShift Container Platform console, navigate to Workloads Pods or navigate to the pod through the resource you want to investigate.
2. Select the openshift-ovn-kubernetes project from the drop-down menu.
3. Click the name of the pod you want to investigate.
4. Click Logs. By default for the ovnkube-master the logs associated with the northd container are displayed.
5. Use the down-down menu to select logs for each container in turn.

Procedure

1. To list the current PodNetworkConnectivityCheck objects, enter the following command:

   $ oc get podnetworkconnectivitychecks -n openshift-network-diagnostics

   Copy to Clipboard Toggle word wrap

2. View the most recent success for each connection object by using the following command:

   $ oc get podnetworkconnectivitychecks -n openshift-network-diagnostics \
   -o json | jq '.items[]| .spec.targetEndpoint,.status.successes[0]'

   Copy to Clipboard Toggle word wrap

3. View the most recent failures for each connection object by using the following command:

   $ oc get podnetworkconnectivitychecks -n openshift-network-diagnostics \
   -o json | jq '.items[]| .spec.targetEndpoint,.status.failures[0]'

   Copy to Clipboard Toggle word wrap

4. View the most recent outages for each connection object by using the following command:

   $ oc get podnetworkconnectivitychecks -n openshift-network-diagnostics \
   -o json | jq '.items[]| .spec.targetEndpoint,.status.outages[0]'

   Copy to Clipboard Toggle word wrap

   The connectivity check controller also logs metrics from these checks into Prometheus.

5. View all the metrics by running the following command:

   $ oc exec prometheus-k8s-0 -n openshift-monitoring -- \
   promtool query instant  http://localhost:9090 \
   '{component="openshift-network-diagnostics"}'

   Copy to Clipboard Toggle word wrap

6. View the latency between the source pod and the openshift api service for the last 5 minutes:

   $ oc exec prometheus-k8s-0 -n openshift-monitoring -- \
   promtool query instant  http://localhost:9090 \
   '{component="openshift-network-diagnostics"}'

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a pod variable by using the following command:

   $  POD=$(oc get pods -n openshift-ovn-kubernetes -l app=ovnkube-control-plane -o name | head -1 | awk -F '/' '{print $NF}')

   Copy to Clipboard Toggle word wrap

2. Run the following command on your local host to copy the binary from the ovnkube-control-plane pods:

   $  oc cp -n openshift-ovn-kubernetes $POD:/usr/bin/ovnkube-trace -c ovnkube-cluster-manager ovnkube-trace

   Copy to Clipboard Toggle word wrap
   Note

   If you are using Red Hat Enterprise Linux (RHEL) 8 to run the ovnkube-trace tool, you must copy the file /usr/lib/rhel8/ovnkube-trace to your local host.

3. Make ovnkube-trace executable by running the following command:

   $  chmod +x ovnkube-trace

   Copy to Clipboard Toggle word wrap

4. Display the options available with ovnkube-trace by running the following command:

   $  ./ovnkube-trace -help

   Copy to Clipboard Toggle word wrap

   Expected output

   Usage of ./ovnkube-trace:
     -addr-family string
       	Address family (ip4 or ip6) to be used for tracing (default "ip4")
     -dst string
       	dest: destination pod name
     -dst-ip string
       	destination IP address (meant for tests to external targets)
     -dst-namespace string
       	k8s namespace of dest pod (default "default")
     -dst-port string
       	dst-port: destination port (default "80")
     -kubeconfig string
       	absolute path to the kubeconfig file
     -loglevel string
       	loglevel: klog level (default "0")
     -ovn-config-namespace string
       	namespace used by ovn-config itself
     -service string
       	service: destination service name
     -skip-detrace
       	skip ovn-detrace command
     -src string
       	src: source pod name
     -src-namespace string
       	k8s namespace of source pod (default "default")
     -tcp
       	use tcp transport protocol
     -udp
       	use udp transport protocol

   Copy to Clipboard Toggle word wrap

   The command-line arguments supported are familiar Kubernetes constructs, such as namespaces, pods, services so you do not need to find the MAC address, the IP address of the destination nodes, or the ICMP type.

   The log levels are:

   • 0 (minimal output)
   • 2 (more verbose output showing results of trace commands)
   • 5 (debug output)

Procedure

1. Start a web service in the default namespace by entering the following command:

   $ oc run web --namespace=default --image=quay.io/openshifttest/nginx --labels="app=web" --expose --port=80

   Copy to Clipboard Toggle word wrap

2. List the pods running in the openshift-dns namespace:

   oc get pods -n openshift-dns

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                  READY   STATUS    RESTARTS   AGE
   dns-default-8s42x     2/2     Running   0          5h8m
   dns-default-mdw6r     2/2     Running   0          4h58m
   dns-default-p8t5h     2/2     Running   0          4h58m
   dns-default-rl6nk     2/2     Running   0          5h8m
   dns-default-xbgqx     2/2     Running   0          5h8m
   dns-default-zv8f6     2/2     Running   0          4h58m
   node-resolver-62jjb   1/1     Running   0          5h8m
   node-resolver-8z4cj   1/1     Running   0          4h59m
   node-resolver-bq244   1/1     Running   0          5h8m
   node-resolver-hc58n   1/1     Running   0          4h59m
   node-resolver-lm6z4   1/1     Running   0          5h8m
   node-resolver-zfx5k   1/1     Running   0          5h

   Copy to Clipboard Toggle word wrap

3. Run the following ovnkube-trace command to verify DNS resolution is working:

   $ ./ovnkube-trace \
     -src-namespace default \ 

   1

   
     -src web \ 

   2

   
     -dst-namespace openshift-dns \ 

   3

   
     -dst dns-default-p8t5h \ 

   4

   
     -udp -dst-port 53 \ 

   5

   
     -loglevel 0 

   6

   Copy to Clipboard Toggle word wrap

   1

   Namespace of the source pod

   2

   Source pod name

   3

   Namespace of destination pod

   4

   Destination pod name

   5

   Use the udp transport protocol. Port 53 is the port the DNS service uses.

   6

   Set the log level to 0 (0 is minimal and 5 is debug)

   Example output if the src&dst pod lands on the same node:

   ovn-trace source pod to destination pod indicates success from web to dns-default-p8t5h
   ovn-trace destination pod to source pod indicates success from dns-default-p8t5h to web
   ovs-appctl ofproto/trace source pod to destination pod indicates success from web to dns-default-p8t5h
   ovs-appctl ofproto/trace destination pod to source pod indicates success from dns-default-p8t5h to web
   ovn-detrace source pod to destination pod indicates success from web to dns-default-p8t5h
   ovn-detrace destination pod to source pod indicates success from dns-default-p8t5h to web

   Copy to Clipboard Toggle word wrap

   Example output if the src&dst pod lands on a different node:

   ovn-trace source pod to destination pod indicates success from web to dns-default-8s42x
   ovn-trace (remote) source pod to destination pod indicates success from web to dns-default-8s42x
   ovn-trace destination pod to source pod indicates success from dns-default-8s42x to web
   ovn-trace (remote) destination pod to source pod indicates success from dns-default-8s42x to web
   ovs-appctl ofproto/trace source pod to destination pod indicates success from web to dns-default-8s42x
   ovs-appctl ofproto/trace destination pod to source pod indicates success from dns-default-8s42x to web
   ovn-detrace source pod to destination pod indicates success from web to dns-default-8s42x
   ovn-detrace destination pod to source pod indicates success from dns-default-8s42x to web

   Copy to Clipboard Toggle word wrap

   The ouput indicates success from the deployed pod to the DNS port and also indicates that it is successful going back in the other direction. So you know bi-directional traffic is supported on UDP port 53 if my web pod wants to do dns resolution from core DNS.

Procedure

1. Create the following YAML that defines a deny-by-default policy to deny ingress from all pods in all namespaces. Save the YAML in the deny-by-default.yaml file:

   kind: NetworkPolicy
   apiVersion: networking.k8s.io/v1
   metadata:
     name: deny-by-default
     namespace: default
   spec:
     podSelector: {}
     ingress: []

   Copy to Clipboard Toggle word wrap

2. Apply the policy by entering the following command:

   $ oc apply -f deny-by-default.yaml

   Copy to Clipboard Toggle word wrap

   Example output

   networkpolicy.networking.k8s.io/deny-by-default created

   Copy to Clipboard Toggle word wrap

3. Start a web service in the default namespace by entering the following command:

   $ oc run web --namespace=default --image=quay.io/openshifttest/nginx --labels="app=web" --expose --port=80

   Copy to Clipboard Toggle word wrap

4. Run the following command to create the prod namespace:

   $ oc create namespace prod

   Copy to Clipboard Toggle word wrap

5. Run the following command to label the prod namespace:

   $ oc label namespace/prod purpose=production

   Copy to Clipboard Toggle word wrap

6. Run the following command to deploy an alpine image in the prod namespace and start a shell:

   $ oc run test-6459 --namespace=prod --rm -i -t --image=alpine -- sh

   Copy to Clipboard Toggle word wrap
7. Open another terminal session.

8. In this new terminal session run ovn-trace to verify the failure in communication between the source pod test-6459 running in namespace prod and destination pod running in the default namespace:

   $ ./ovnkube-trace \
    -src-namespace prod \
    -src test-6459 \
    -dst-namespace default \
    -dst web \
    -tcp -dst-port 80 \
    -loglevel 0

   Copy to Clipboard Toggle word wrap

   Example output

   ovn-trace source pod to destination pod indicates failure from test-6459 to web

   Copy to Clipboard Toggle word wrap

9. Increase the log level to 2 to expose the reason for the failure by running the following command:

   $ ./ovnkube-trace \
    -src-namespace prod \
    -src test-6459 \
    -dst-namespace default \
    -dst web \
    -tcp -dst-port 80 \
    -loglevel 2

   Copy to Clipboard Toggle word wrap

   Example output

   ...
   ------------------------------------------------
    3. ls_out_acl_hint (northd.c:7454): !ct.new && ct.est && !ct.rpl && ct_mark.blocked == 0, priority 4, uuid 12efc456
       reg0[8] = 1;
       reg0[10] = 1;
       next;
    5. ls_out_acl_action (northd.c:7835): reg8[30..31] == 0, priority 500, uuid 69372c5d
       reg8[30..31] = 1;
       next(4);
    5. ls_out_acl_action (northd.c:7835): reg8[30..31] == 1, priority 500, uuid 2fa0af89
       reg8[30..31] = 2;
       next(4);
    4. ls_out_acl_eval (northd.c:7691): reg8[30..31] == 2 && reg0[10] == 1 && (outport == @a16982411286042166782_ingressDefaultDeny), priority 2000, uuid 447d0dab
       reg8[17] = 1;
       ct_commit { ct_mark.blocked = 1; }; 

   1

   
       next;
   ...

   Copy to Clipboard Toggle word wrap

   1

   Ingress traffic is blocked due to the default deny policy being in place.

10. Create a policy that allows traffic from all pods in a particular namespaces with a label purpose=production. Save the YAML in the web-allow-prod.yaml file:

    kind: NetworkPolicy
    apiVersion: networking.k8s.io/v1
    metadata:
      name: web-allow-prod
      namespace: default
    spec:
      podSelector:
        matchLabels:
          app: web
      policyTypes:
      - Ingress
      ingress:
      - from:
        - namespaceSelector:
            matchLabels:
              purpose: production

    Copy to Clipboard Toggle word wrap

11. Apply the policy by entering the following command:

    $ oc apply -f web-allow-prod.yaml

    Copy to Clipboard Toggle word wrap

12. Run ovnkube-trace to verify that traffic is now allowed by entering the following command:

    $ ./ovnkube-trace \
     -src-namespace prod \
     -src test-6459 \
     -dst-namespace default \
     -dst web \
     -tcp -dst-port 80 \
     -loglevel 0

    Copy to Clipboard Toggle word wrap

    Expected output

    ovn-trace source pod to destination pod indicates success from test-6459 to web
    ovn-trace destination pod to source pod indicates success from web to test-6459
    ovs-appctl ofproto/trace source pod to destination pod indicates success from test-6459 to web
    ovs-appctl ofproto/trace destination pod to source pod indicates success from web to test-6459
    ovn-detrace source pod to destination pod indicates success from test-6459 to web
    ovn-detrace destination pod to source pod indicates success from web to test-6459

    Copy to Clipboard Toggle word wrap

13. Run the following command in the shell that was opened in step six to connect nginx to the web-server:

     wget -qO- --timeout=2 http://web.default

    Copy to Clipboard Toggle word wrap

    Expected output

    <!DOCTYPE html>
    <html>
    <head>
    <title>Welcome to nginx!</title>
    <style>
      body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
      }
    </style>
    </head>
    <body>
    <h1>Welcome to nginx!</h1>
    <p>If you see this page, the nginx web server is successfully installed and
    working. Further configuration is required.</p>
    
    <p>For online documentation and support please refer to
    <a href="http://nginx.org/">nginx.org</a>.<br/>
    Commercial support is available at
    <a href="http://nginx.com/">nginx.com</a>.</p>
    
    <p><em>Thank you for using nginx.</em></p>
    </body>
    </html>

    Copy to Clipboard Toggle word wrap