Chapter 18. Requesting CRI-O and Kubelet profiling data by using the Node Observability Operator

1. Install the Node Observability Operator in the OpenShift Container Platform cluster.
2. Create a NodeObservability custom resource to enable the CRI-O profiling on the worker nodes of your choice.
3. Run the profiling query to generate the profiling data.

Procedure

1. Log in to the OpenShift Container Platform CLI by running the following command:

   Copy to Clipboard Toggle word wrap

   $ oc login -u kubeadmin https://<HOSTNAME>:6443

2. Switch back to the node-observability-operator namespace by running the following command:

   Copy to Clipboard Toggle word wrap

   $ oc project node-observability-operator

3. Create a CR file named nodeobservability.yaml that contains the following text:

   Copy to Clipboard Toggle word wrap

       apiVersion: nodeobservability.olm.openshift.io/v1alpha2
       kind: NodeObservability
       metadata:
         name: cluster 

   1

   
       spec:
         nodeSelector:
           kubernetes.io/hostname: <node_hostname> 

   2

   
         type: crio-kubelet

   1

   You must specify the name as cluster because there should be only one NodeObservability CR per cluster.

   2

   Specify the nodes on which the Node Observability agent must be deployed.

4. Run the NodeObservability CR:

   Copy to Clipboard Toggle word wrap

   oc apply -f nodeobservability.yaml

   Example output

   Copy to Clipboard Toggle word wrap

   nodeobservability.olm.openshift.io/cluster created

5. Review the status of the NodeObservability CR by running the following command:

   Copy to Clipboard Toggle word wrap

   $ oc get nob/cluster -o yaml | yq '.status.conditions'

   Example output

   Copy to Clipboard Toggle word wrap

   conditions:
     conditions:
     - lastTransitionTime: "2022-07-05T07:33:54Z"
       message: 'DaemonSet node-observability-ds ready: true NodeObservabilityMachineConfig
         ready: true'
       reason: Ready
       status: "True"
       type: Ready

   NodeObservability CR run is completed when the reason is Ready and the status is True.

Procedure

1. Create a NodeObservabilityRun resource file named nodeobservabilityrun.yaml that contains the following text:

   Copy to Clipboard Toggle word wrap

   apiVersion: nodeobservability.olm.openshift.io/v1alpha2
   kind: NodeObservabilityRun
   metadata:
     name: nodeobservabilityrun
   spec:
     nodeObservabilityRef:
       name: cluster

2. Trigger the profiling query by running the NodeObservabilityRun resource:

   Copy to Clipboard Toggle word wrap

   $ oc apply -f nodeobservabilityrun.yaml

3. Review the status of the NodeObservabilityRun by running the following command:

   Copy to Clipboard Toggle word wrap

   $ oc get nodeobservabilityrun nodeobservabilityrun -o yaml  | yq '.status.conditions'

   Example output

   Copy to Clipboard Toggle word wrap

   conditions:
   - lastTransitionTime: "2022-07-07T14:57:34Z"
     message: Ready to start profiling
     reason: Ready
     status: "True"
     type: Ready
   - lastTransitionTime: "2022-07-07T14:58:10Z"
     message: Profiling query done
     reason: Finished
     status: "True"
     type: Finished

   The profiling query is complete once the status is True and type is Finished.

4. Retrieve the profiling data from the container’s /run/node-observability path by running the following bash script:

   Copy to Clipboard Toggle word wrap

   for a in $(oc get nodeobservabilityrun nodeobservabilityrun -o yaml | yq .status.agents[].name); do
     echo "agent ${a}"
     mkdir -p "/tmp/${a}"
     for p in $(oc exec "${a}" -c node-observability-agent -- bash -c "ls /run/node-observability/*.pprof"); do
       f="$(basename ${p})"
       echo "copying ${f} to /tmp/${a}/${f}"
       oc exec "${a}" -c node-observability-agent -- cat "${p}" > "/tmp/${a}/${f}"
     done
   done