Chapter 21. Requesting CRI-O and Kubelet profiling data by using the Node Observability Operator

1. Install the Node Observability Operator in the OpenShift Container Platform cluster.
2. Create a

   NodeObservability

   custom resource to enable the CRI-O profiling on the worker nodes of your choice.
3. Run the profiling query to generate the profiling data.

Procedure

1. Log in to the OpenShift Container Platform CLI by running the following command:

   $ oc login -u kubeadmin https://<HOSTNAME>:6443

2. Switch back to the

   node-observability-operator

   namespace by running the following command:

   $ oc project node-observability-operator

3. Create a CR file named

   nodeobservability.yaml

   that contains the following text:

       apiVersion: nodeobservability.olm.openshift.io/v1alpha1
       kind: NodeObservability
       metadata:
         name: cluster 

   1

   
       spec:
         labels:
           node-role.kubernetes.io/worker: ""
         type: crio-kubelet

   1

   You must specify the name as cluster because there should be only one NodeObservability CR per cluster.

4. Run the

   NodeObservability

   CR:

   oc apply -f nodeobservability.yaml

   Example output

   nodeobservability.olm.openshift.io/cluster created

5. Review the status of the

   NodeObservability

   CR by running the following command:

   $ oc get nob/cluster -o yaml | yq '.status.conditions'

   Example output

   conditions:
     conditions:
     - lastTransitionTime: "2022-07-05T07:33:54Z"
       message: 'DaemonSet node-observability-ds ready: true NodeObservabilityMachineConfig
         ready: true'
       reason: Ready
       status: "True"
       type: Ready

   NodeObservability

   CR run is completed when the reason is

   Ready

   and the status is

   True

   .

Procedure

1. Create a

   NodeObservabilityRun

   resource file named

   nodeobservabilityrun.yaml

   that contains the following text:

   apiVersion: nodeobservability.olm.openshift.io/v1alpha1
   kind: NodeObservabilityRun
   metadata:
     name: nodeobservabilityrun
   spec:
     nodeObservabilityRef:
       name: cluster

2. Trigger the profiling query by running the

   NodeObservabilityRun

   resource:

   $ oc apply -f nodeobservabilityrun.yaml

3. Review the status of the

   NodeObservabilityRun

   by running the following command:

   $ oc get nodeobservabilityrun nodeobservabilityrun -o yaml  | yq '.status.conditions'

   Example output

   conditions:
   - lastTransitionTime: "2022-07-07T14:57:34Z"
     message: Ready to start profiling
     reason: Ready
     status: "True"
     type: Ready
   - lastTransitionTime: "2022-07-07T14:58:10Z"
     message: Profiling query done
     reason: Finished
     status: "True"
     type: Finished

   The profiling query is complete once the status is

   True

   and type is

   Finished

   .

4. Retrieve the profiling data from the container’s

   /run/node-observability

   path by running the following bash script:

   for a in $(oc get nodeobservabilityrun nodeobservabilityrun -o yaml | yq .status.agents[].name); do
     echo "agent ${a}"
     mkdir -p "/tmp/${a}"
     for p in $(oc exec "${a}" -c node-observability-agent -- bash -c "ls /run/node-observability/*.pprof"); do
       f="$(basename ${p})"
       echo "copying ${f} to /tmp/${a}/${f}"
       oc exec "${a}" -c node-observability-agent -- cat "${p}" > "/tmp/${a}/${f}"
     done
   done