第 3 章 The greenboot health check framework
Greenboot is the generic health check framework for the systemd service on rpm-ostree systems such as Red Hat Enterprise Linux for Edge (RHEL for Edge). This framework is included in MicroShift installations with the microshift-greenboot and greenboot-default-health-checks RPM packages.
Greenboot health checks run at various times to assess system health and automate a rollback on rpm-ostree systems to the last healthy state in cases of software trouble, for example:
- Default health check scripts run each time the system starts.
- In addition the to the default health checks, you can write, install, and configure application health check scripts to also run every time the system starts.
- Greenboot can reduce your risk of being locked out of edge devices during updates and prevent a significant interruption of service if an update fails.
-
When a failure is detected, the system boots into the last known working configuration using the
rpm-ostreerollback capability. This feature is especially useful automation for edge devices where direct serviceability is either limited or non-existent.
A MicroShift application health check script is included in the microshift-greenboot RPM. The greenboot-default-health-checks RPM includes health check scripts verifying that DNS and ostree services are accessible. You can create your own health check scripts for the workloads you are running. You can write one that verifies that an application has started, for example.
3.1. How greenboot uses directories to run scripts
Health check scripts run from four /etc/greenboot directories. These scripts run in alphabetical order. Keep this in mind when you configure the scripts for your workloads.
When the system starts, greenboot runs the scripts in the required.d and wanted.d directories. Depending on the outcome of those scripts, greenboot continues the startup or attempts a rollback as follows:
-
System as expected: When all of the scripts in the
required.ddirectory are successfully run, greenboot runs any scripts present in the/etc/greenboot/green.ddirectory. -
System trouble: If any of the scripts in the
required.ddirectory fail, greenboot runs any prerollback scripts present in thered.ddirectory, then restarts the system.
Greenboot redirects script and health check output to the system log. When you are logged in, a daily message provides the overall system health output.
3.1.1. Greenboot directories details
Returning a nonzero exit code from any script means that script has failed. Greenboot restarts the system a few times to retry the scripts before attempting to roll back to the previous version.
/etc/greenboot/check/required.dcontains the health checks that must not fail.-
If the scripts fail, greenboot retries them three times by default. You can configure the number of retries in the
/etc/greenboot/greenboot.conffile by setting theGREENBOOT_MAX_BOOTSparameter to the desired number of retries. - After all retries fail, greenboot automatically initiates a rollback if one is available. If a rollback is not available, the system log output shows that manual intervention is required.
-
The
40_microshift_running_check.shhealth check script for MicroShift is installed into this directory.
-
If the scripts fail, greenboot retries them three times by default. You can configure the number of retries in the
/etc/greenboot/check/wanted.dcontains health scripts that are allowed to fail without causing the system to be rolled back.- If any of these scripts fail, greenboot logs the failure but does not initiate a rollback.
-
/etc/greenboot/green.dcontains scripts that run after greenboot has declared the start successful. -
/etc/greenboot/red.dcontains scripts that run after greenboot has declared the startup as failed, including the40_microshift_pre_rollback.shprerollback script. This script is executed right before a system rollback. The script performs MicroShift pod and OVN-Kubernetes cleanup to avoid potential conflicts after the system is rolled back to a previous version.
If you customize the values of any environment variable in the /etc/greenboot/greenboot.conf file, these changes can be lost when the greenboot RPM package is updated or downgraded.
-
To retain customizations when building system images with MicroShift, add the
greenboot.conffile to a blueprint. -
To retain customizations when using an RPM installation, apply changes to the
greenboot.conffile after you install MicroShift and greenboot RPMs.
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}