Red Hat Summit이 콘텐츠는 선택한 언어로 제공되지 않습니다.
Chapter 7. Using greenboot for application and workload health checks
You can use greenboot health checks to assess the health of your workloads and applications.
7.1. How workload health checks work
Greenboot health checks are helpful on edge devices where direct serviceability is either limited or non-existent. You can use greenboot health checks to assess the health of your workloads and applications. These additional health checks are useful for software problem detection and automatic system rollbacks.
Workload or application health checks can use the MicroShift basic health check functions already implemented for the MicroShift core services. Creating your own comprehensive scripts for your applications is recommended. For example, you can write one that verifies that a service has started.
You can also use the microshift healthcheck command, which can run checks that the basic functions of the workload are operating as expected.
The following functions related to checking workload health in /usr/share/microshift/functions/greenboot.sh are deprecated and planned for removal in a future release:
-
wait_for -
namespace_images_downloaded -
namespace_deployment_ready -
namespace_daemonset_ready -
namespace_pods_ready -
namespace_pods_not_restarting -
print_failure_logs -
log_failure_cmd -
log_script_exit -
lvmsDriverShouldExist -
csiComponentShouldBeDeploy
7.2. How to use the MicroShift health check command
The microshift healthcheck command checks whether a workload of the provided type exists and verifies its status for the specified timeout duration. The number of ready replicas, that is, pods, must match the expected amount.
To run the microshift healthcheck command successfully, use the following prerequisites:
- Execute commands from a root user account.
- Enable the MicroShift service.
You can add the following actions to the microshift healthcheck command:
-
-v=2to increase verbosity of the output -
--timeout="${WAIT_TIMEOUT_SECS}s"to override default 600s timeout value -
--namespace `<namespace>to specify the namespace of the workloads --deployments `<application-deployment>to check the readiness of a specific deploymentExample command
sudo microshift healthcheck -v=2 --timeout="300s" --namespace busybox --deployments busybox-deployment
$ sudo microshift healthcheck -v=2 --timeout="300s" --namespace busybox --deployments busybox-deploymentCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
The microshift healthcheck command also accepts the following additional parameters to specify other kinds of workloads:
-
--daemonsets -
--statefulsets -
These options take a comma-delimited list of resources, for example,
--daemonsets ovnkube-master,ovnkube-node.
Alternatively, a --custom option can be used with a JSON string, for example:
sudo microshift healthcheck --custom '{"openshift-storage":{"deployments":
["lvms-operator"], "daemonsets": ["vg-manager"]}, "openshift-ovn-kubernetes":
{"daemonsets": ["ovnkube-master", "ovnkube-node"]}}'
$ sudo microshift healthcheck --custom '{"openshift-storage":{"deployments":
["lvms-operator"], "daemonsets": ["vg-manager"]}, "openshift-ovn-kubernetes":
{"daemonsets": ["ovnkube-master", "ovnkube-node"]}}'Example output
7.3. How to create a health check script for your application
You can create workload or application health check scripts in the text editor of your choice. Save the scripts in the /etc/greenboot/check/required.d directory. When a script in the /etc/greenboot/check/required.d directory exits with an error, greenboot triggers a reboot in an attempt to heal the system.
Any script in the /etc/greenboot/check/required.d directory triggers a reboot if it exits with an error.
If your health check logic requires any post-check steps, you can also create additional scripts and save them in the relevant greenboot directories. For example:
-
You can also place shell scripts you want to run after a boot has been declared successful in
/etc/greenboot/green.d. -
You can place shell scripts you want to run after a boot has been declared failed in
/etc/greenboot/red.d. For example, if you have steps to heal the system before restarting, you can create scripts for your use case and place them in the/etc/greenboot/red.ddirectory.
7.3.1. Workload max duration or timeout script example
The following example uses the MicroShift core services health check script as a template.
7.3.1.1. Basic prerequisites for creating a health check script
- The workload must be installed.
- You must have root access.
7.3.1.2. Example and functional requirements
You can start with the following example health check script. Add to it for your use case. In your custom workload health check script, you must define the relevant namespace, deployment, daemonset, and statefulset.
Choose a name prefix for your application that ensures it runs after the 40_microshift_running_check.sh script, which implements the MicroShift health check procedure for its core services.
Example greenboot health check script
Functions related to checking workload health previously included in the /usr/share/microshift/functions/greenboot.sh script file are deprecated. You can write a custom script, or use the microshift healthcheck command with various options instead. See "How workload health check scripts work" for more information.
7.3.2. Testing a workload health check script
The output of the greenboot workload health check script varies with the host system type. Example outputs for Red Hat Enterprise Linux (RHEL) system types are included for reference only.
Prerequisites
- You have root access.
- You installed a workload.
- You created a health check script for the workload.
- The MicroShift service is enabled.
Procedure
To test that greenboot is running a health check script file, reboot the host by running the following command:
sudo reboot
$ sudo rebootCopy to Clipboard Copied! Toggle word wrap Toggle overflow Examine the output of greenboot health checks by running the following command:
sudo journalctl -o cat -u greenboot-healthcheck.service
$ sudo journalctl -o cat -u greenboot-healthcheck.serviceCopy to Clipboard Copied! Toggle word wrap Toggle overflow NoteMicroShift core service health checks run before the workload health checks.
Example output for an image mode for RHEL system
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example partial output for a RHEL for Edge system
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example partial output for an RPM system
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}