Chapter 11. Monitoring

Procedure

1. From the Administrator perspective in the Red Hat OpenShift Service on AWS web console, select Observe Metrics.

2. To add one or more queries, do any of the following:

   Option	Description
   Create a custom query.	Add your Prometheus Query Language (PromQL) query to the Expression field. As you type a PromQL expression, autocomplete suggestions appear in a drop-down list. These suggestions include functions, metrics, labels, and time tokens. You can use the keyboard arrows to select one of these suggested items and then press Enter to add the item to your expression. You can also move your mouse pointer over a suggested item to view a brief description of that item.
   Add multiple queries.	Select Add query.
   Duplicate an existing query.	Select the Options menu next to the query, then choose Duplicate query.
   Disable a query from being run.	Select the Options menu next to the query and choose Disable query.

3. To run queries that you created, select Run queries. The metrics from the queries are visualized on the plot. If a query is invalid, the UI shows an error message.

   Note

   Queries that operate on large amounts of data might time out or overload the browser when drawing time series graphs. To avoid this, select Hide graph and calibrate your query using only the metrics table. Then, after finding a feasible query, enable the plot to draw the graphs.

   Note

   By default, the query table shows an expanded view that lists every metric and its current value. You can select ˅ to minimize the expanded view for a query.

4. Optional: The page URL now contains the queries you ran. To use this set of queries again in the future, save this URL.

5. Explore the visualized metrics. Initially, all metrics from all enabled queries are shown on the plot. You can select which metrics are shown by doing any of the following:

   Option	Description
   Hide all metrics from a query.	Click the Options menu for the query and click Hide all series.
   Hide a specific metric.	Go to the query table and click the colored square near the metric name.
   Zoom into the plot and change the time range.	Either: Visually select the time range by clicking and dragging on the plot horizontally. Use the menu in the left upper corner to select the time range.
   Reset the time range.	Select Reset zoom.
   Display outputs for all queries at a specific point in time.	Hold the mouse cursor on the plot at that point. The query outputs will appear in a pop-up box.
   Hide the plot.	Select Hide graph.

Procedure

1. From the Developer perspective in the Red Hat OpenShift Service on AWS web console, select Observe Metrics.
2. Select the project that you want to view metrics for in the Project: list.

3. Select a query from the Select query list, or create a custom PromQL query based on the selected query by selecting Show PromQL. The metrics from the queries are visualized on the plot.

   Note

   In the Developer perspective, you can only run one query at a time.

4. Explore the visualized metrics by doing any of the following:

   Option	Description
   Zoom into the plot and change the time range.	Either: Visually select the time range by clicking and dragging on the plot horizontally. Use the menu in the left upper corner to select the time range.
   Reset the time range.	Select Reset zoom.
   Display outputs for all queries at a specific point in time.	Hold the mouse cursor on the plot at that point. The query outputs appear in a pop-up box.

Procedure

1. Obtain the HTTP service endpoint by specifying the namespace for the service:

   $ oc get service -n <namespace> <node-exporter-service>

2. To list all available metrics for the node-exporter service, query the metrics resource.

   $ curl http://<172.30.226.162:9100>/metrics | grep -vE "^#|^$"

   Example output

   node_arp_entries{device="eth0"} 1
   node_boot_time_seconds 1.643153218e+09
   node_context_switches_total 4.4938158e+07
   node_cooling_device_cur_state{name="0",type="Processor"} 0
   node_cooling_device_max_state{name="0",type="Processor"} 0
   node_cpu_guest_seconds_total{cpu="0",mode="nice"} 0
   node_cpu_guest_seconds_total{cpu="0",mode="user"} 0
   node_cpu_seconds_total{cpu="0",mode="idle"} 1.10586485e+06
   node_cpu_seconds_total{cpu="0",mode="iowait"} 37.61
   node_cpu_seconds_total{cpu="0",mode="irq"} 233.91
   node_cpu_seconds_total{cpu="0",mode="nice"} 551.47
   node_cpu_seconds_total{cpu="0",mode="softirq"} 87.3
   node_cpu_seconds_total{cpu="0",mode="steal"} 86.12
   node_cpu_seconds_total{cpu="0",mode="system"} 464.15
   node_cpu_seconds_total{cpu="0",mode="user"} 1075.2
   node_disk_discard_time_seconds_total{device="vda"} 0
   node_disk_discard_time_seconds_total{device="vdb"} 0
   node_disk_discarded_sectors_total{device="vda"} 0
   node_disk_discarded_sectors_total{device="vdb"} 0
   node_disk_discards_completed_total{device="vda"} 0
   node_disk_discards_completed_total{device="vdb"} 0
   node_disk_discards_merged_total{device="vda"} 0
   node_disk_discards_merged_total{device="vdb"} 0
   node_disk_info{device="vda",major="252",minor="0"} 1
   node_disk_info{device="vdb",major="252",minor="16"} 1
   node_disk_io_now{device="vda"} 0
   node_disk_io_now{device="vdb"} 0
   node_disk_io_time_seconds_total{device="vda"} 174
   node_disk_io_time_seconds_total{device="vdb"} 0.054
   node_disk_io_time_weighted_seconds_total{device="vda"} 259.79200000000003
   node_disk_io_time_weighted_seconds_total{device="vdb"} 0.039
   node_disk_read_bytes_total{device="vda"} 3.71867136e+08
   node_disk_read_bytes_total{device="vdb"} 366592
   node_disk_read_time_seconds_total{device="vda"} 19.128
   node_disk_read_time_seconds_total{device="vdb"} 0.039
   node_disk_reads_completed_total{device="vda"} 5619
   node_disk_reads_completed_total{device="vdb"} 96
   node_disk_reads_merged_total{device="vda"} 5
   node_disk_reads_merged_total{device="vdb"} 0
   node_disk_write_time_seconds_total{device="vda"} 240.66400000000002
   node_disk_write_time_seconds_total{device="vdb"} 0
   node_disk_writes_completed_total{device="vda"} 71584
   node_disk_writes_completed_total{device="vdb"} 0
   node_disk_writes_merged_total{device="vda"} 19761
   node_disk_writes_merged_total{device="vdb"} 0
   node_disk_written_bytes_total{device="vda"} 2.007924224e+09
   node_disk_written_bytes_total{device="vdb"} 0

Procedure

1. Expose the node-exporter service.

   $ oc expose service -n <namespace> <node_exporter_service_name>

2. Obtain the FQDN (Fully Qualified Domain Name) for the route.

   $ oc get route -o=custom-columns=NAME:.metadata.name,DNS:.spec.host

   Example output

   NAME                    DNS
   node-exporter-service   node-exporter-service-dynamation.apps.cluster.example.org

3. Use the curl command to display metrics for the node-exporter service.

   $ curl -s http://node-exporter-service-dynamation.apps.cluster.example.org/metrics

   Example output

   go_gc_duration_seconds{quantile="0"} 1.5382e-05
   go_gc_duration_seconds{quantile="0.25"} 3.1163e-05
   go_gc_duration_seconds{quantile="0.5"} 3.8546e-05
   go_gc_duration_seconds{quantile="0.75"} 4.9139e-05
   go_gc_duration_seconds{quantile="1"} 0.000189423

Procedure

1. Include details of the readiness probe in the VM configuration file.

   Sample readiness probe with an HTTP GET test

   apiVersion: kubevirt.io/v1
   kind: VirtualMachine
   metadata:
     annotations:
     name: fedora-vm
     namespace: example-namespace
   # ...
   spec:
     template:
       spec:
         readinessProbe:
           httpGet: 1
             port: 1500 2
             path: /healthz 3
             httpHeaders:
             - name: Custom-Header
               value: Awesome
           initialDelaySeconds: 120 4
           periodSeconds: 20 5
           timeoutSeconds: 10 6
           failureThreshold: 3 7
           successThreshold: 3 8
   # ...

   1

   The HTTP GET request to perform to connect to the VM.

   2

   The port of the VM that the probe queries. In the above example, the probe queries port 1500.

   3

   The path to access on the HTTP server. In the above example, if the handler for the server’s /healthz path returns a success code, the VM is considered to be healthy. If the handler returns a failure code, the VM is removed from the list of available endpoints.

   4

   The time, in seconds, after the VM starts before the readiness probe is initiated.

   5

   The delay, in seconds, between performing probes. The default delay is 10 seconds. This value must be greater than timeoutSeconds.

   6

   The number of seconds of inactivity after which the probe times out and the VM is assumed to have failed. The default value is 1. This value must be lower than periodSeconds.

   7

   The number of times that the probe is allowed to fail. The default is 3. After the specified number of attempts, the pod is marked Unready.

   8

   The number of times that the probe must report success, after a failure, to be considered successful. The default is 1.

2. Create the VM by running the following command:

   $ oc create -f <file_name>.yaml

Procedure

1. Include details of the TCP readiness probe in the VM configuration file.

   Sample readiness probe with a TCP socket test

   apiVersion: kubevirt.io/v1
   kind: VirtualMachine
   metadata:
     annotations:
     name: fedora-vm
     namespace: example-namespace
   # ...
   spec:
     template:
       spec:
         readinessProbe:
           initialDelaySeconds: 120 1
           periodSeconds: 20 2
           tcpSocket: 3
             port: 1500 4
           timeoutSeconds: 10 5
   # ...

   1

   The time, in seconds, after the VM starts before the readiness probe is initiated.

   2

   The delay, in seconds, between performing probes. The default delay is 10 seconds. This value must be greater than timeoutSeconds.

   3

   The TCP action to perform.

   4

   The port of the VM that the probe queries.

   5

   The number of seconds of inactivity after which the probe times out and the VM is assumed to have failed. The default value is 1. This value must be lower than periodSeconds.

2. Create the VM by running the following command:

   $ oc create -f <file_name>.yaml

Procedure

1. Include details of the HTTP liveness probe in the VM configuration file.

   Sample liveness probe with an HTTP GET test

   apiVersion: kubevirt.io/v1
   kind: VirtualMachine
   metadata:
     annotations:
     name: fedora-vm
     namespace: example-namespace
   # ...
   spec:
     template:
       spec:
         livenessProbe:
           initialDelaySeconds: 120 1
           periodSeconds: 20 2
           httpGet: 3
             port: 1500 4
             path: /healthz 5
             httpHeaders:
             - name: Custom-Header
               value: Awesome
           timeoutSeconds: 10 6
   # ...

   1

   The time, in seconds, after the VM starts before the liveness probe is initiated.

   2

   The delay, in seconds, between performing probes. The default delay is 10 seconds. This value must be greater than timeoutSeconds.

   3

   The HTTP GET request to perform to connect to the VM.

   4

   The port of the VM that the probe queries. In the above example, the probe queries port 1500. The VM installs and runs a minimal HTTP server on port 1500 via cloud-init.

   5

   The path to access on the HTTP server. In the above example, if the handler for the server’s /healthz path returns a success code, the VM is considered to be healthy. If the handler returns a failure code, the VM is deleted and a new VM is created.

   6

   The number of seconds of inactivity after which the probe times out and the VM is assumed to have failed. The default value is 1. This value must be lower than periodSeconds.

2. Create the VM by running the following command:

   $ oc create -f <file_name>.yaml

Procedure

1. Create a YAML file with the following contents:

   apiVersion: kubevirt.io/v1
   kind: VirtualMachine
   metadata:
     labels:
       kubevirt.io/vm: vm2-rhel84-watchdog
     name: <vm-name>
   spec:
     running: false
     template:
       metadata:
         labels:
           kubevirt.io/vm: vm2-rhel84-watchdog
       spec:
         domain:
           devices:
             watchdog:
               name: <watchdog>
               i6300esb:
                 action: "poweroff" 1
   # ...

   1

   Specify poweroff, reset, or shutdown.

   The example above configures the i6300esb watchdog device on a RHEL8 VM with the poweroff action and exposes the device as /dev/watchdog.

   This device can now be used by the watchdog binary.

2. Apply the YAML file to your cluster by running the following command:

   $ oc apply -f <file_name>.yaml

1. Run the following command to verify that the VM is connected to the watchdog device:

   $ lspci | grep watchdog -i

2. Run one of the following commands to confirm the watchdog is active:

   • Trigger a kernel panic:

     # echo c > /proc/sysrq-trigger

   • Stop the watchdog service:

     # pkill -9 watchdog

Procedure

1. Log in to the virtual machine as root user.

2. Install the watchdog package and its dependencies:

   # yum install watchdog

3. Uncomment the following line in the /etc/watchdog.conf file and save the changes:

   #watchdog-device = /dev/watchdog

4. Enable the watchdog service to start on boot:

   # systemctl enable --now watchdog.service