Chapter 21. Requesting CRI-O and Kubelet profiling data by using the Node Observability Operator

1. Install the Node Observability Operator in the OpenShift Container Platform cluster.
2. Create a NodeObservability custom resource to enable the CRI-O profiling on the worker nodes of your choice.
3. Run the profiling query to generate the profiling data.

Procedure

1. Log in to the OpenShift Container Platform CLI by running the following command:

   $ oc login -u kubeadmin https://<HOSTNAME>:6443

   Copy to Clipboard Toggle word wrap

2. Switch back to the node-observability-operator namespace by running the following command:

   $ oc project node-observability-operator

   Copy to Clipboard Toggle word wrap

3. Create a CR file named nodeobservability.yaml that contains the following text:

       apiVersion: nodeobservability.olm.openshift.io/v1alpha1
       kind: NodeObservability
       metadata:
         name: cluster 

   1

   
       spec:
         labels:
           node-role.kubernetes.io/worker: ""
         type: crio-kubelet

   Copy to Clipboard Toggle word wrap

   1

   You must specify the name as cluster because there should be only one NodeObservability CR per cluster.

4. Run the NodeObservability CR:

   oc apply -f nodeobservability.yaml

   Copy to Clipboard Toggle word wrap

   Example output

   nodeobservability.olm.openshift.io/cluster created

   Copy to Clipboard Toggle word wrap

5. Review the status of the NodeObservability CR by running the following command:

   $ oc get nob/cluster -o yaml | yq '.status.conditions'

   Copy to Clipboard Toggle word wrap

   Example output

   conditions:
     conditions:
     - lastTransitionTime: "2022-07-05T07:33:54Z"
       message: 'DaemonSet node-observability-ds ready: true NodeObservabilityMachineConfig
         ready: true'
       reason: Ready
       status: "True"
       type: Ready

   Copy to Clipboard Toggle word wrap

   NodeObservability CR run is completed when the reason is Ready and the status is True.

Procedure

1. Create a NodeObservabilityRun resource file named nodeobservabilityrun.yaml that contains the following text:

   apiVersion: nodeobservability.olm.openshift.io/v1alpha1
   kind: NodeObservabilityRun
   metadata:
     name: nodeobservabilityrun
   spec:
     nodeObservabilityRef:
       name: cluster

   Copy to Clipboard Toggle word wrap

2. Trigger the profiling query by running the NodeObservabilityRun resource:

   $ oc apply -f nodeobservabilityrun.yaml

   Copy to Clipboard Toggle word wrap

3. Review the status of the NodeObservabilityRun by running the following command:

   $ oc get nodeobservabilityrun nodeobservabilityrun -o yaml  | yq '.status.conditions'

   Copy to Clipboard Toggle word wrap

   Example output

   conditions:
   - lastTransitionTime: "2022-07-07T14:57:34Z"
     message: Ready to start profiling
     reason: Ready
     status: "True"
     type: Ready
   - lastTransitionTime: "2022-07-07T14:58:10Z"
     message: Profiling query done
     reason: Finished
     status: "True"
     type: Finished

   Copy to Clipboard Toggle word wrap

   The profiling query is complete once the status is True and type is Finished.

4. Retrieve the profiling data from the container’s /run/node-observability path by running the following bash script:

   for a in $(oc get nodeobservabilityrun nodeobservabilityrun -o yaml | yq .status.agents[].name); do
     echo "agent ${a}"
     mkdir -p "/tmp/${a}"
     for p in $(oc exec "${a}" -c node-observability-agent -- bash -c "ls /run/node-observability/*.pprof"); do
       f="$(basename ${p})"
       echo "copying ${f} to /tmp/${a}/${f}"
       oc exec "${a}" -c node-observability-agent -- cat "${p}" > "/tmp/${a}/${f}"
     done
   done

   Copy to Clipboard Toggle word wrap