Chapter 3. Troubleshooting OVN-Kubernetes

Procedure

1. Review the details of the ovnkube-node readiness probe by running the following command:

   $ oc get pods -n openshift-ovn-kubernetes -l app=ovnkube-node \
   -o json | jq '.items[0].spec.containers[] | .name,.readinessProbe'

   Copy to Clipboard Toggle word wrap

   The readiness probe for the northbound and southbound database containers in the ovnkube-node pod checks for the health of the databases and the ovnkube-controller container.

   The ovnkube-controller container in the ovnkube-node pod has a readiness probe to verify the presence of the OVN-Kubernetes CNI configuration file, the absence of which would indicate that the pod is not running or is not ready to accept requests to configure pods.

2. Show all events including the probe failures, for the namespace by using the following command:

   $ oc get events -n openshift-ovn-kubernetes

   Copy to Clipboard Toggle word wrap

3. Show the events for just a specific pod:

   $ oc describe pod ovnkube-node-9lqfk -n openshift-ovn-kubernetes

   Copy to Clipboard Toggle word wrap

4. Show the messages and statuses from the cluster network operator:

   $ oc get co/network -o json | jq '.status.conditions[]'

   Copy to Clipboard Toggle word wrap

5. Show the ready status of each container in ovnkube-node pods by running the following script:

   $ for p in $(oc get pods --selector app=ovnkube-node -n openshift-ovn-kubernetes \
   -o jsonpath='{range.items[*]}{" "}{.metadata.name}'); do echo === $p ===;  \
   oc get pods -n openshift-ovn-kubernetes $p -o json | jq '.status.containerStatuses[] | .name, .ready'; \
   done

   Copy to Clipboard Toggle word wrap
   Note

   The expectation is all container statuses are reporting as true. Failure of a readiness probe sets the status to false.

Procedure (UI)

1. In the Administrator perspective, select Observe Alerting. The three main pages in the Alerting UI in this perspective are the Alerts, Silences, and Alerting Rules pages.
2. View the rules for OVN-Kubernetes alerts by selecting Observe Alerting Alerting Rules.

Procedure

1. View active or firing alerts by running the following commands.

   1. Set the alert manager route environment variable by running the following command:

      $ ALERT_MANAGER=$(oc get route alertmanager-main -n openshift-monitoring \
      -o jsonpath='{@.spec.host}')

      Copy to Clipboard Toggle word wrap

   2. Issue a curl request to the alert manager route API by running the following command, replacing $ALERT_MANAGER with the URL of your Alertmanager instance:

      $ curl -s -k -H "Authorization: Bearer $(oc create token prometheus-k8s -n openshift-monitoring)" https://$ALERT_MANAGER/api/v1/alerts | jq '.data[] | "\(.labels.severity) \(.labels.alertname) \(.labels.pod) \(.labels.container) \(.labels.endpoint) \(.labels.instance)"'

      Copy to Clipboard Toggle word wrap

2. View alerting rules by running the following command:

   $ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -s 'http://localhost:9090/api/v1/rules' | jq '.data.groups[].rules[] | select(((.name|contains("ovn")) or (.name|contains("OVN")) or (.name|contains("Ovn")) or (.name|contains("North")) or (.name|contains("South"))) and .type=="alerting")'

   Copy to Clipboard Toggle word wrap

Procedure

1. View the log for a specific pod:

   $ oc logs -f <pod_name> -c <container_name> -n <namespace>

   Copy to Clipboard Toggle word wrap

   where:

   -f

   Optional: Specifies that the output follows what is being written into the logs.

   <pod_name>

   Specifies the name of the pod.

   <container_name>

   Optional: Specifies the name of a container. When a pod has more than one container, you must specify the container name.

   <namespace>

   Specify the namespace the pod is running in.

   For example:

   $ oc logs ovnkube-node-5dx44 -n openshift-ovn-kubernetes

   Copy to Clipboard Toggle word wrap

   $ oc logs -f ovnkube-node-5dx44 -c ovnkube-controller -n openshift-ovn-kubernetes

   Copy to Clipboard Toggle word wrap

   The contents of log files are printed out.

2. Examine the most recent entries in all the containers in the ovnkube-node pods:

   $ for p in $(oc get pods --selector app=ovnkube-node -n openshift-ovn-kubernetes \
   -o jsonpath='{range.items[*]}{" "}{.metadata.name}'); \
   do echo === $p ===; for container in $(oc get pods -n openshift-ovn-kubernetes $p \
   -o json | jq -r '.status.containerStatuses[] | .name');do echo ---$container---; \
   oc logs -c $container $p -n openshift-ovn-kubernetes --tail=5; done; done

   Copy to Clipboard Toggle word wrap

3. View the last 5 lines of every log in every container in an ovnkube-node pod using the following command:

   $ oc logs -l app=ovnkube-node -n openshift-ovn-kubernetes --all-containers --tail 5

   Copy to Clipboard Toggle word wrap

Procedure

1. In the OpenShift Container Platform console, navigate to Workloads Pods or navigate to the pod through the resource you want to investigate.
2. Select the openshift-ovn-kubernetes project from the drop-down menu.
3. Click the name of the pod you want to investigate.
4. Click Logs. By default for the ovnkube-master the logs associated with the northd container are displayed.
5. Use the down-down menu to select logs for each container in turn.

Procedure

1. To list the current PodNetworkConnectivityCheck objects, enter the following command:

   $ oc get podnetworkconnectivitychecks -n openshift-network-diagnostics

   Copy to Clipboard Toggle word wrap

2. View the most recent success for each connection object by using the following command:

   $ oc get podnetworkconnectivitychecks -n openshift-network-diagnostics \
   -o json | jq '.items[]| .spec.targetEndpoint,.status.successes[0]'

   Copy to Clipboard Toggle word wrap

3. View the most recent failures for each connection object by using the following command:

   $ oc get podnetworkconnectivitychecks -n openshift-network-diagnostics \
   -o json | jq '.items[]| .spec.targetEndpoint,.status.failures[0]'

   Copy to Clipboard Toggle word wrap

4. View the most recent outages for each connection object by using the following command:

   $ oc get podnetworkconnectivitychecks -n openshift-network-diagnostics \
   -o json | jq '.items[]| .spec.targetEndpoint,.status.outages[0]'

   Copy to Clipboard Toggle word wrap

   The connectivity check controller also logs metrics from these checks into Prometheus.

5. View all the metrics by running the following command:

   $ oc exec prometheus-k8s-0 -n openshift-monitoring -- \
   promtool query instant  http://localhost:9090 \
   '{component="openshift-network-diagnostics"}'

   Copy to Clipboard Toggle word wrap

6. View the latency between the source pod and the openshift api service for the last 5 minutes:

   $ oc exec prometheus-k8s-0 -n openshift-monitoring -- \
   promtool query instant  http://localhost:9090 \
   '{component="openshift-network-diagnostics"}'

   Copy to Clipboard Toggle word wrap

Procedure

1. To enable the OVNObservability with OVS sampling feature, enable TechPreviewNoUpgrade feature set in the FeatureGate CR named cluster by entering the following command:

   $ oc patch --type=merge --patch '{"spec": {"featureSet": "TechPreviewNoUpgrade"}}' featuregate/cluster

   Copy to Clipboard Toggle word wrap

   Example output

   featuregate.config.openshift.io/cluster patched

   Copy to Clipboard Toggle word wrap

2. Confirm that the OVNObservability feature is enabled by entering the following command:

   $ oc get featuregate cluster -o yaml

   Copy to Clipboard Toggle word wrap

   Example output

     featureGates:
   # ...
       enabled:
       - name: OVNObservability

   Copy to Clipboard Toggle word wrap

3. Obtain a list of the pods inside of the namespace in which you have created one of the relevant network APIs by entering the following command. Note the NODE name of the pods, as they are used in the following step.

   $ oc get pods -n <namespace> -o wide

   Copy to Clipboard Toggle word wrap

   Example output

   NAME              READY   STATUS    RESTARTS   AGE     IP            NODE                                       NOMINATED NODE   READINESS GATES
   destination-pod   1/1     Running   0          53s     10.131.0.23   ci-ln-1gqp7b2-72292-bb9dv-worker-a-gtmpc   <none>           <none>
   source-pod        1/1     Running   0          56s     10.131.0.22   ci-ln-1gqp7b2-72292-bb9dv-worker-a-gtmpc   <none>           <none>

   Copy to Clipboard Toggle word wrap

4. Obtain a list of OVN-Kubernetes pods and locate the pod that shares the same NODE as the pods from the previous step by entering the following command:

   $ oc get pods -n openshift-ovn-kubernetes -o wide

   Copy to Clipboard Toggle word wrap

   Example output

   NAME
   ...                             READY   STATUS    RESTARTS      AGE   IP           NODE                                       NOMINATED NODE
   ovnkube-node-jzn5b              8/8     Running   1 (34m ago)   37m   10.0.128.2   ci-ln-1gqp7b2-72292-bb9dv-worker-a-gtmpc   <none>
   ...

   Copy to Clipboard Toggle word wrap

5. Open a bash shell inside of the ovnkube-node pod by entering the following command:

   $ oc exec -it <pod_name> -n openshift-ovn-kubernetes -- bash

   Copy to Clipboard Toggle word wrap

6. While inside of the ovnkube-node pod, you can run the ovnkube-observ -add-ovs-collector script to show network events using the OVS collector. For example:

   # /usr/bin/ovnkube-observ -add-ovs-collector

   Copy to Clipboard Toggle word wrap

   Example output

   ...
   2024/12/02 19:41:41.327584 OVN-K message: Allowed by default allow from local node policy, direction ingress
   2024/12/02 19:41:41.327593 src=10.131.0.2, dst=10.131.0.6
   
   2024/12/02 19:41:41.327692 OVN-K message: Allowed by default allow from local node policy, direction ingress
   2024/12/02 19:41:41.327715 src=10.131.0.6, dst=10.131.0.2
   ...

   Copy to Clipboard Toggle word wrap

7. You can filter the content by type, such as source pods, by entering the following command with the -filter-src-ip flag and your pod’s IP address. For example:

   # /usr/bin/ovnkube-observ -add-ovs-collector -filter-src-ip <pod_ip_address>

   Copy to Clipboard Toggle word wrap

   Example output

   ...
   Found group packets, id 14
   2024/12/10 16:27:12.456473 OVN-K message: Allowed by admin network policy allow-egress-group1, direction Egress
   2024/12/10 16:27:12.456570 src=10.131.0.22, dst=10.131.0.23
   
   2024/12/10 16:27:14.484421 OVN-K message: Allowed by admin network policy allow-egress-group1, direction Egress
   2024/12/10 16:27:14.484428 src=10.131.0.22, dst=10.131.0.23
   
   2024/12/10 16:27:12.457222 OVN-K message: Allowed by network policy test:allow-ingress-from-specific-pod, direction Ingress
   2024/12/10 16:27:12.457228 src=10.131.0.22, dst=10.131.0.23
   
   2024/12/10 16:27:12.457288 OVN-K message: Allowed by network policy test:allow-ingress-from-specific-pod, direction Ingress
   2024/12/10 16:27:12.457299 src=10.131.0.22, dst=10.131.0.23
   ...

   Copy to Clipboard Toggle word wrap

   For a full list of flags that can be passed in with /usr/bin/ovnkube-observ, see "OVN-Kubernetes network traffic with OVS sampling flags".