Chapter 3. The greenboot health check framework

• Default health check scripts run each time the system starts.
• In addition the to the default health checks, you can write, install, and configure application health check scripts to also run every time the system starts.
• Greenboot can reduce your risk of being locked out of edge devices during updates and prevent a significant interruption of service if an update fails.
• When a failure is detected, the system boots into the last known working configuration using the rpm-ostree rollback capability. This feature is especially useful automation for edge devices where direct serviceability is either limited or non-existent.

• Check if repository URLs are still DNS solvable:

  This script is under /usr/lib/greenboot/check/required.d/01_repository_dns_check.sh and ensures that DNS queries to repository URLs are still available.

• Check if update platforms are still reachable:

  This script is under /usr/lib/greenboot/check/wanted.d/01_update_platform_check.sh and tries to connect and get a 2XX or 3XX HTTP code from the update platforms defined in /etc/ostree/remotes.d.

• Check if the current boot has been triggered by the hardware watchdog:

  This script is under /usr/lib/greenboot/check/required.d/02_watchdog.sh and checks whether the current boot has been watchdog-triggered or not.

  • If the watchdog-triggered reboot occurs within the grace period, the current boot is marked as red. Greenboot does not trigger a rollback to the previous deployment.
  • If the watchdog-triggered reboot occurs after the grace period, the current boot is not marked as red. Greenboot does not trigger a rollback to the previous deployment.
  • A 24-hour grace period is enabled by default. This grace period can be either disabled by modifying GREENBOOT_WATCHDOG_CHECK_ENABLED in /etc/greenboot/greenboot.conf to false, or configured by changing the GREENBOOT_WATCHDOG_GRACE_PERIOD=number_of_hours variable value in /etc/greenboot/greenboot.conf.

Procedure

• To access the overall status of system health checks, run the following command:

  $ sudo grub2-editenv - list | grep ^boot_success

  Example output for a successful system start

  boot_success=1 

  1

  1

  If your command returns boot_success=0, either the greenboot health check is still running, or the update is a failure.

Procedure

• To access the results of a health check, run the following command:

  $ sudo journalctl -o cat -u greenboot-healthcheck.service

  Example output of a failed health check

  ...
  ...
  Running Required Health Check Scripts...
  STARTED
  GRUB boot variables:
  boot_success=0
  boot_indeterminate=0
  boot_counter=2
  ...
  ...
  Waiting 600s for MicroShift service to be active and not failed
  FAILURE
  ...
  ...

Procedure

• To access the results of a prerollback script, run the following command:

  $ sudo journalctl -o cat -u redboot-task-runner.service

  Example output of a prerollback script

  ...
  ...
  Running Red Scripts...
  STARTED
  GRUB boot variables:
  boot_success=0
  boot_indeterminate=0
  boot_counter=0
  The ostree status:
  * rhel c0baa75d9b585f3dd989a9cf05f647eb7ca27ee0dbd4b94fe8c93ed3a4b9e4a5.0
      Version: 9.6
      origin: <unknown origin type>
    rhel 6869c1347b0e0ba1bbf0be750cdf32da5138a1fcbc5a4c6325ab9eb647b64663.0 (rollback)
      Version: 9.6
      origin refspec: edge:rhel/9/x86_64/edge
  System rollback imminent - preparing MicroShift for a clean start
  Stopping MicroShift services
  Removing MicroShift pods
  Killing conmon, pause and OVN processes
  Removing OVN configuration
  Finished greenboot Failure Scripts Runner.
  Cleanup succeeded
  Script '40_microshift_pre_rollback.sh' SUCCESS
  FINISHED
  redboot-task-runner.service: Deactivated successfully.

  Note

  In case of a rollback, the pre-rollback script runs the sudo microshift-cleanup-data --ovn command to prepare the system for a potential software downgrade.

Procedure

• To access the result of update checks, run the following command:

  $ sudo grub2-editenv - list | grep ^boot_success

  Example output for a successful update

  boot_success=1 

  1

  1

  If your command returns boot_success=0, either the greenboot health check is still running, or the update is a failure.