Chapter 13. Monitoring

Procedure

1. Obtain the HTTP service endpoint by specifying the namespace for the service:

   $ oc get service -n <namespace> <node-exporter-service>

   Copy to Clipboard Toggle word wrap

2. To list all available metrics for the node-exporter service, query the metrics resource.

   $ curl http://<172.30.226.162:9100>/metrics | grep -vE "^#|^$"

   Copy to Clipboard Toggle word wrap

   Example output

   node_arp_entries{device="eth0"} 1
   node_boot_time_seconds 1.643153218e+09
   node_context_switches_total 4.4938158e+07
   node_cooling_device_cur_state{name="0",type="Processor"} 0
   node_cooling_device_max_state{name="0",type="Processor"} 0
   node_cpu_guest_seconds_total{cpu="0",mode="nice"} 0
   node_cpu_guest_seconds_total{cpu="0",mode="user"} 0
   node_cpu_seconds_total{cpu="0",mode="idle"} 1.10586485e+06
   node_cpu_seconds_total{cpu="0",mode="iowait"} 37.61
   node_cpu_seconds_total{cpu="0",mode="irq"} 233.91
   node_cpu_seconds_total{cpu="0",mode="nice"} 551.47
   node_cpu_seconds_total{cpu="0",mode="softirq"} 87.3
   node_cpu_seconds_total{cpu="0",mode="steal"} 86.12
   node_cpu_seconds_total{cpu="0",mode="system"} 464.15
   node_cpu_seconds_total{cpu="0",mode="user"} 1075.2
   node_disk_discard_time_seconds_total{device="vda"} 0
   node_disk_discard_time_seconds_total{device="vdb"} 0
   node_disk_discarded_sectors_total{device="vda"} 0
   node_disk_discarded_sectors_total{device="vdb"} 0
   node_disk_discards_completed_total{device="vda"} 0
   node_disk_discards_completed_total{device="vdb"} 0
   node_disk_discards_merged_total{device="vda"} 0
   node_disk_discards_merged_total{device="vdb"} 0
   node_disk_info{device="vda",major="252",minor="0"} 1
   node_disk_info{device="vdb",major="252",minor="16"} 1
   node_disk_io_now{device="vda"} 0
   node_disk_io_now{device="vdb"} 0
   node_disk_io_time_seconds_total{device="vda"} 174
   node_disk_io_time_seconds_total{device="vdb"} 0.054
   node_disk_io_time_weighted_seconds_total{device="vda"} 259.79200000000003
   node_disk_io_time_weighted_seconds_total{device="vdb"} 0.039
   node_disk_read_bytes_total{device="vda"} 3.71867136e+08
   node_disk_read_bytes_total{device="vdb"} 366592
   node_disk_read_time_seconds_total{device="vda"} 19.128
   node_disk_read_time_seconds_total{device="vdb"} 0.039
   node_disk_reads_completed_total{device="vda"} 5619
   node_disk_reads_completed_total{device="vdb"} 96
   node_disk_reads_merged_total{device="vda"} 5
   node_disk_reads_merged_total{device="vdb"} 0
   node_disk_write_time_seconds_total{device="vda"} 240.66400000000002
   node_disk_write_time_seconds_total{device="vdb"} 0
   node_disk_writes_completed_total{device="vda"} 71584
   node_disk_writes_completed_total{device="vdb"} 0
   node_disk_writes_merged_total{device="vda"} 19761
   node_disk_writes_merged_total{device="vdb"} 0
   node_disk_written_bytes_total{device="vda"} 2.007924224e+09
   node_disk_written_bytes_total{device="vdb"} 0

   Copy to Clipboard Toggle word wrap

Procedure

1. Expose the node-exporter service.

   $ oc expose service -n <namespace> <node_exporter_service_name>

   Copy to Clipboard Toggle word wrap

2. Obtain the FQDN (Fully Qualified Domain Name) for the route.

   $ oc get route -o=custom-columns=NAME:.metadata.name,DNS:.spec.host

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                    DNS
   node-exporter-service   node-exporter-service-dynamation.apps.cluster.example.org

   Copy to Clipboard Toggle word wrap

3. Use the curl command to display metrics for the node-exporter service.

   $ curl -s http://node-exporter-service-dynamation.apps.cluster.example.org/metrics

   Copy to Clipboard Toggle word wrap

   Example output

   go_gc_duration_seconds{quantile="0"} 1.5382e-05
   go_gc_duration_seconds{quantile="0.25"} 3.1163e-05
   go_gc_duration_seconds{quantile="0.5"} 3.8546e-05
   go_gc_duration_seconds{quantile="0.75"} 4.9139e-05
   go_gc_duration_seconds{quantile="1"} 0.000189423

   Copy to Clipboard Toggle word wrap

Procedure

1. Include details of the readiness probe in the VM configuration file.

   Sample readiness probe with an HTTP GET test

   apiVersion: kubevirt.io/v1
   kind: VirtualMachine
   metadata:
     annotations:
     name: fedora-vm
     namespace: example-namespace
   # ...
   spec:
     template:
       spec:
         readinessProbe:
           httpGet: 

   1

   
             port: 1500 

   2

   
             path: /healthz 

   3

   
             httpHeaders:
             - name: Custom-Header
               value: Awesome
           initialDelaySeconds: 120 

   4

   
           periodSeconds: 20 

   5

   
           timeoutSeconds: 10 

   6

   
           failureThreshold: 3 

   7

   
           successThreshold: 3 

   8

   
   # ...

   Copy to Clipboard Toggle word wrap

   1

   The HTTP GET request to perform to connect to the VM.

   2

   The port of the VM that the probe queries. In the above example, the probe queries port 1500.

   3

   The path to access on the HTTP server. In the above example, if the handler for the server’s /healthz path returns a success code, the VM is considered to be healthy. If the handler returns a failure code, the VM is removed from the list of available endpoints.

   4

   The time, in seconds, after the VM starts before the readiness probe is initiated.

   5

   The delay, in seconds, between performing probes. The default delay is 10 seconds. This value must be greater than timeoutSeconds.

   6

   The number of seconds of inactivity after which the probe times out and the VM is assumed to have failed. The default value is 1. This value must be lower than periodSeconds.

   7

   The number of times that the probe is allowed to fail. The default is 3. After the specified number of attempts, the pod is marked Unready.

   8

   The number of times that the probe must report success, after a failure, to be considered successful. The default is 1.

2. Create the VM by running the following command:

   $ oc create -f <file_name>.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Include details of the TCP readiness probe in the VM configuration file.

   Sample readiness probe with a TCP socket test

   apiVersion: kubevirt.io/v1
   kind: VirtualMachine
   metadata:
     annotations:
     name: fedora-vm
     namespace: example-namespace
   # ...
   spec:
     template:
       spec:
         readinessProbe:
           initialDelaySeconds: 120 

   1

   
           periodSeconds: 20 

   2

   
           tcpSocket: 

   3

   
             port: 1500 

   4

   
           timeoutSeconds: 10 

   5

   
   # ...

   Copy to Clipboard Toggle word wrap

   1

   The time, in seconds, after the VM starts before the readiness probe is initiated.

   2

   The delay, in seconds, between performing probes. The default delay is 10 seconds. This value must be greater than timeoutSeconds.

   3

   The TCP action to perform.

   4

   The port of the VM that the probe queries.

   5

   The number of seconds of inactivity after which the probe times out and the VM is assumed to have failed. The default value is 1. This value must be lower than periodSeconds.

2. Create the VM by running the following command:

   $ oc create -f <file_name>.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Include details of the HTTP liveness probe in the VM configuration file.

   Sample liveness probe with an HTTP GET test

   apiVersion: kubevirt.io/v1
   kind: VirtualMachine
   metadata:
     annotations:
     name: fedora-vm
     namespace: example-namespace
   # ...
   spec:
     template:
       spec:
         livenessProbe:
           initialDelaySeconds: 120 

   1

   
           periodSeconds: 20 

   2

   
           httpGet: 

   3

   
             port: 1500 

   4

   
             path: /healthz 

   5

   
             httpHeaders:
             - name: Custom-Header
               value: Awesome
           timeoutSeconds: 10 

   6

   
   # ...

   Copy to Clipboard Toggle word wrap

   1

   The time, in seconds, after the VM starts before the liveness probe is initiated.

   2

   The delay, in seconds, between performing probes. The default delay is 10 seconds. This value must be greater than timeoutSeconds.

   3

   The HTTP GET request to perform to connect to the VM.

   4

   The port of the VM that the probe queries. In the above example, the probe queries port 1500. The VM installs and runs a minimal HTTP server on port 1500 via cloud-init.

   5

   The path to access on the HTTP server. In the above example, if the handler for the server’s /healthz path returns a success code, the VM is considered to be healthy. If the handler returns a failure code, the VM is deleted and a new VM is created.

   6

   The number of seconds of inactivity after which the probe times out and the VM is assumed to have failed. The default value is 1. This value must be lower than periodSeconds.

2. Create the VM by running the following command:

   $ oc create -f <file_name>.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a YAML file with the following contents:

   apiVersion: kubevirt.io/v1
   kind: VirtualMachine
   metadata:
     labels:
       kubevirt.io/vm: <vm-label>
     name: <vm-name>
   spec:
     runStrategy: Halted
     template:
       metadata:
         labels:
           kubevirt.io/vm: <vm-label>
       spec:
         domain:
           devices:
             watchdog:
               name: <watchdog>
               <watchdog-device-model>: 

   1

   
                 action: "poweroff" 

   2

   
   # ...

   Copy to Clipboard Toggle word wrap

   1

   The watchdog device model to use. For x86 specify i6300esb. For s390x specify diag288.

   2

   Specify poweroff, reset, or shutdown. The shutdown action requires that the guest virtual machine is responsive to ACPI signals. Therefore, using shutdown is not recommended.

   The example above configures the watchdog device on a VM with the poweroff action and exposes the device as /dev/watchdog.

   This device can now be used by the watchdog binary.

2. Apply the YAML file to your cluster by running the following command:

   $ oc apply -f <file_name>.yaml

   Copy to Clipboard Toggle word wrap

1. Run the following command to verify that the VM is connected to the watchdog device:

   $ lspci | grep watchdog -i

   Copy to Clipboard Toggle word wrap

2. Run one of the following commands to confirm the watchdog is active:

   • Trigger a kernel panic:

     # echo c > /proc/sysrq-trigger

     Copy to Clipboard Toggle word wrap

   • Stop the watchdog service:

     # pkill -9 watchdog

     Copy to Clipboard Toggle word wrap

Procedure

1. Log in to the virtual machine as root user.

2. This step is only required when installing on IBM Z® (s390x). Enable watchdog by running the following command:

   # modprobe diag288_wdt

   Copy to Clipboard Toggle word wrap

3. Verify that the /dev/watchdog file path is present in the VM by running the following command:

   # ls /dev/watchdog

   Copy to Clipboard Toggle word wrap

4. Install the watchdog package and its dependencies:

   # yum install watchdog

   Copy to Clipboard Toggle word wrap

5. Uncomment the following line in the /etc/watchdog.conf file and save the changes:

   #watchdog-device = /dev/watchdog

   Copy to Clipboard Toggle word wrap

6. Enable the watchdog service to start on boot:

   # systemctl enable --now watchdog.service

   Copy to Clipboard Toggle word wrap