Chapter 21. Requesting CRI-O and Kubelet profiling data by using the Node Observability Operator

1. Install the Node Observability Operator in the OpenShift Container Platform cluster.
2. Create a NodeObservability custom resource to enable the CRI-O profiling on the worker nodes of your choice.
3. Run the profiling query to generate the profiling data.

Procedure

1. Log in to the OpenShift Container Platform CLI by running the following command:

   $ oc login -u kubeadmin https://<HOSTNAME>:6443

2. Switch back to the node-observability-operator namespace by running the following command:

   $ oc project node-observability-operator

3. Create a CR file named nodeobservability.yaml that contains the following text:

       apiVersion: nodeobservability.olm.openshift.io/v1alpha1
       kind: NodeObservability
       metadata:
         name: cluster 1
       spec:
         labels:
           node-role.kubernetes.io/worker: ""
         type: crio-kubelet

   1

   You must specify the name as cluster because there should be only one NodeObservability CR per cluster.

4. Run the NodeObservability CR:

   oc apply -f nodeobservability.yaml

   Example output

   nodeobservability.olm.openshift.io/cluster created

5. Review the status of the NodeObservability CR by running the following command:

   $ oc get nob/cluster -o yaml | yq '.status.conditions'

   Example output

   conditions:
     conditions:
     - lastTransitionTime: "2022-07-05T07:33:54Z"
       message: 'DaemonSet node-observability-ds ready: true NodeObservabilityMachineConfig
         ready: true'
       reason: Ready
       status: "True"
       type: Ready

   NodeObservability CR run is completed when the reason is Ready and the status is True.

Procedure

1. Create a NodeObservabilityRun resource file named nodeobservabilityrun.yaml that contains the following text:

   apiVersion: nodeobservability.olm.openshift.io/v1alpha1
   kind: NodeObservabilityRun
   metadata:
     name: nodeobservabilityrun
   spec:
     nodeObservabilityRef:
       name: cluster

2. Trigger the profiling query by running the NodeObservabilityRun resource:

   $ oc apply -f nodeobservabilityrun.yaml

3. Review the status of the NodeObservabilityRun by running the following command:

   $ oc get nodeobservabilityrun nodeobservabilityrun -o yaml  | yq '.status.conditions'

   Example output

   conditions:
   - lastTransitionTime: "2022-07-07T14:57:34Z"
     message: Ready to start profiling
     reason: Ready
     status: "True"
     type: Ready
   - lastTransitionTime: "2022-07-07T14:58:10Z"
     message: Profiling query done
     reason: Finished
     status: "True"
     type: Finished

   The profiling query is complete once the status is True and type is Finished.

4. Retrieve the profiling data from the container’s /run/node-observability path by running the following bash script:

   for a in $(oc get nodeobservabilityrun nodeobservabilityrun -o yaml | yq .status.agents[].name); do
     echo "agent ${a}"
     mkdir -p "/tmp/${a}"
     for p in $(oc exec "${a}" -c node-observability-agent -- bash -c "ls /run/node-observability/*.pprof"); do
       f="$(basename ${p})"
       echo "copying ${f} to /tmp/${a}/${f}"
       oc exec "${a}" -c node-observability-agent -- cat "${p}" > "/tmp/${a}/${f}"
     done
   done