Chapter 22. Using the Node Observability Operator

Procedure

1. Confirm that the Node Observability Operator is available by running the following command:

   Copy to Clipboard Toggle word wrap

   $ oc get packagemanifests -n openshift-marketplace node-observability-operator

   Example output

   Copy to Clipboard Toggle word wrap

   NAME                            CATALOG                AGE
   node-observability-operator     Red Hat Operators      9h

2. Create the node-observability-operator namespace by running the following command:

   Copy to Clipboard Toggle word wrap

   $ oc new-project node-observability-operator

3. Create an OperatorGroup object YAML file:

   Copy to Clipboard Toggle word wrap

   cat <<EOF | oc apply -f -
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: node-observability-operator
     namespace: node-observability-operator
   spec:
     targetNamespaces: []
   EOF

4. Create a Subscription object YAML file to subscribe a namespace to an Operator:

   Copy to Clipboard Toggle word wrap

   cat <<EOF | oc apply -f -
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: node-observability-operator
     namespace: node-observability-operator
   spec:
     channel: alpha
     name: node-observability-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
   EOF

Verification

1. View the install plan name by running the following command:

   Copy to Clipboard Toggle word wrap

   $ oc -n node-observability-operator get sub node-observability-operator -o yaml | yq '.status.installplan.name'

   Example output

   Copy to Clipboard Toggle word wrap

   install-dt54w

2. Verify the install plan status by running the following command:

   Copy to Clipboard Toggle word wrap

   $ oc -n node-observability-operator get ip <install_plan_name> -o yaml | yq '.status.phase'

   <install_plan_name> is the install plan name that you obtained from the output of the previous command.

   Example output

   Copy to Clipboard Toggle word wrap

   COMPLETE

3. Verify that the Node Observability Operator is up and running:

   Copy to Clipboard Toggle word wrap

   $ oc get deploy -n node-observability-operator

   Example output

   Copy to Clipboard Toggle word wrap

   NAME                                            READY   UP-TO-DATE  AVAILABLE   AGE
   node-observability-operator-controller-manager  1/1     1           1           40h

Procedure

1. Log in to the OpenShift Container Platform web console.
2. In the Administrator’s navigation panel, expand Operators OperatorHub.
3. In the All items field, enter Node Observability Operator and select the Node Observability Operator tile.
4. Click Install.

5. On the Install Operator page, configure the following settings:

   1. In the Update channel area, click alpha.
   2. In the Installation mode area, click A specific namespace on the cluster.
   3. From the Installed Namespace list, select node-observability-operator from the list.
   4. In the Update approval area, select Automatic.
   5. Click Install.

Verification

1. In the Administrator’s navigation panel, expand Operators Installed Operators.
2. Verify that the Node Observability Operator is listed in the Operators list.

Procedure

1. Log in to the OpenShift Container Platform CLI by running the following command:

   Copy to Clipboard Toggle word wrap

   $ oc login -u kubeadmin https://<HOSTNAME>:6443

2. Switch back to the node-observability-operator namespace by running the following command:

   Copy to Clipboard Toggle word wrap

   $ oc project node-observability-operator

3. Create a CR file named nodeobservability.yaml that contains the following text:

   Copy to Clipboard Toggle word wrap

       apiVersion: nodeobservability.olm.openshift.io/v1alpha2
       kind: NodeObservability
       metadata:
         name: cluster 

   1

   
       spec:
         nodeSelector:
           kubernetes.io/hostname: <node_hostname> 

   2

   
         type: crio-kubelet

   1

   You must specify the name as cluster because there should be only one NodeObservability CR per cluster.

   2

   Specify the nodes on which the Node Observability agent must be deployed.

4. Run the NodeObservability CR:

   Copy to Clipboard Toggle word wrap

   oc apply -f nodeobservability.yaml

   Example output

   Copy to Clipboard Toggle word wrap

   nodeobservability.olm.openshift.io/cluster created

5. Review the status of the NodeObservability CR by running the following command:

   Copy to Clipboard Toggle word wrap

   $ oc get nob/cluster -o yaml | yq '.status.conditions'

   Example output

   Copy to Clipboard Toggle word wrap

   conditions:
     conditions:
     - lastTransitionTime: "2022-07-05T07:33:54Z"
       message: 'DaemonSet node-observability-ds ready: true NodeObservabilityMachineConfig
         ready: true'
       reason: Ready
       status: "True"
       type: Ready

   NodeObservability CR run is completed when the reason is Ready and the status is True.

Procedure

1. Create a NodeObservabilityRun resource file named nodeobservabilityrun.yaml that contains the following text:

   Copy to Clipboard Toggle word wrap

   apiVersion: nodeobservability.olm.openshift.io/v1alpha2
   kind: NodeObservabilityRun
   metadata:
     name: nodeobservabilityrun
   spec:
     nodeObservabilityRef:
       name: cluster

2. Trigger the profiling query by running the NodeObservabilityRun resource:

   Copy to Clipboard Toggle word wrap

   $ oc apply -f nodeobservabilityrun.yaml

3. Review the status of the NodeObservabilityRun by running the following command:

   Copy to Clipboard Toggle word wrap

   $ oc get nodeobservabilityrun nodeobservabilityrun -o yaml  | yq '.status.conditions'

   Example output

   Copy to Clipboard Toggle word wrap

   conditions:
   - lastTransitionTime: "2022-07-07T14:57:34Z"
     message: Ready to start profiling
     reason: Ready
     status: "True"
     type: Ready
   - lastTransitionTime: "2022-07-07T14:58:10Z"
     message: Profiling query done
     reason: Finished
     status: "True"
     type: Finished

   The profiling query is complete once the status is True and type is Finished.

4. Retrieve the profiling data from the container’s /run/node-observability path by running the following bash script:

   Copy to Clipboard Toggle word wrap

   for a in $(oc get nodeobservabilityrun nodeobservabilityrun -o yaml | yq .status.agents[].name); do
     echo "agent ${a}"
     mkdir -p "/tmp/${a}"
     for p in $(oc exec "${a}" -c node-observability-agent -- bash -c "ls /run/node-observability/*.pprof"); do
       f="$(basename ${p})"
       echo "copying ${f} to /tmp/${a}/${f}"
       oc exec "${a}" -c node-observability-agent -- cat "${p}" > "/tmp/${a}/${f}"
     done
   done

Procedure

1. Log in to the OpenShift Container Platform cluster by running the following command:

   Copy to Clipboard Toggle word wrap

   $ oc login -u kubeadmin https://<host_name>:6443

2. Switch to the node-observability-operator namespace by running the following command:

   Copy to Clipboard Toggle word wrap

   $ oc project node-observability-operator

3. Create a file named nodeobservability.yaml that contains the following content:

   Copy to Clipboard Toggle word wrap

       apiVersion: nodeobservability.olm.openshift.io/v1alpha2
       kind: NodeObservability
       metadata:
         name: cluster 

   1

   
       spec:
         nodeSelector:
           kubernetes.io/hostname: <node_hostname> 

   2

   
         type: scripting 

   3

   1

   You must specify the name as cluster because there should be only one NodeObservability CR per cluster.

   2

   Specify the nodes on which the Node Observability agent must be deployed.

   3

   To deploy the agent in scripting mode, you must set the type to scripting.

4. Create the NodeObservability CR by running the following command:

   Copy to Clipboard Toggle word wrap

   $ oc apply -f nodeobservability.yaml

   Example output

   Copy to Clipboard Toggle word wrap

   nodeobservability.olm.openshift.io/cluster created

5. Review the status of the NodeObservability CR by running the following command:

   Copy to Clipboard Toggle word wrap

   $ oc get nob/cluster -o yaml | yq '.status.conditions'

   Example output

   Copy to Clipboard Toggle word wrap

   conditions:
     conditions:
     - lastTransitionTime: "2022-07-05T07:33:54Z"
       message: 'DaemonSet node-observability-ds ready: true NodeObservabilityScripting
         ready: true'
       reason: Ready
       status: "True"
       type: Ready

   The NodeObservability CR run is completed when the reason is Ready and status is "True".

Procedure

1. Create a file named nodeobservabilityrun-script.yaml that contains the following content:

   Copy to Clipboard Toggle word wrap

   apiVersion: nodeobservability.olm.openshift.io/v1alpha2
   kind: NodeObservabilityRun
   metadata:
     name: nodeobservabilityrun-script
     namespace: node-observability-operator
   spec:
     nodeObservabilityRef:
       name: cluster
       type: scripting

   Important

   You can request only the following scripts:

   • metrics.sh
   • network-metrics.sh (uses monitor.sh)

2. Trigger the scripting by creating the NodeObservabilityRun resource with the following command:

   Copy to Clipboard Toggle word wrap

   $ oc apply -f nodeobservabilityrun-script.yaml

3. Review the status of the NodeObservabilityRun scripting by running the following command:

   Copy to Clipboard Toggle word wrap

   $ oc get nodeobservabilityrun nodeobservabilityrun-script -o yaml  | yq '.status.conditions'

   Example output

   Copy to Clipboard Toggle word wrap

   Status:
     Agents:
       Ip:    10.128.2.252
       Name:  node-observability-agent-n2fpm
       Port:  8443
       Ip:    10.131.0.186
       Name:  node-observability-agent-wcc8p
       Port:  8443
     Conditions:
       Conditions:
         Last Transition Time:  2023-12-19T15:10:51Z
         Message:               Ready to start profiling
         Reason:                Ready
         Status:                True
         Type:                  Ready
         Last Transition Time:  2023-12-19T15:11:01Z
         Message:               Profiling query done
         Reason:                Finished
         Status:                True
         Type:                  Finished
     Finished Timestamp:        2023-12-19T15:11:01Z
     Start Timestamp:           2023-12-19T15:10:51Z

   The scripting is complete once Status is True and Type is Finished.

4. Retrieve the scripting data from the root path of the container by running the following bash script:

   Copy to Clipboard Toggle word wrap

   #!/bin/bash
   
   RUN=$(oc get nodeobservabilityrun --no-headers | awk '{print $1}')
   
   for a in $(oc get nodeobservabilityruns.nodeobservability.olm.openshift.io/${RUN} -o json | jq .status.agents[].name); do
     echo "agent ${a}"
     agent=$(echo ${a} | tr -d "\"\'\`")
     base_dir=$(oc exec "${agent}" -c node-observability-agent -- bash -c "ls -t | grep node-observability-agent" | head -1)
     echo "${base_dir}"
     mkdir -p "/tmp/${agent}"
     for p in $(oc exec "${agent}" -c node-observability-agent -- bash -c "ls ${base_dir}"); do
       f="/${base_dir}/${p}"
       echo "copying ${f} to /tmp/${agent}/${p}"
       oc exec "${agent}" -c node-observability-agent -- cat ${f} > "/tmp/${agent}/${p}"
     done
   done