第 6 章 Monitoring application health by using health checks
In software systems, components can become unhealthy due to transient issues such as temporary connectivity loss, configuration errors, or problems with external dependencies. OpenShift Container Platform applications have a number of options to detect and handle unhealthy containers.
6.1. Understanding health checks
A health check periodically performs diagnostics on a running container using any combination of the readiness, liveness, and startup health checks.
You can include one or more probes in the specification for the pod that contains the container which you want to perform the health checks.
If you want to add or edit health checks in an existing pod, you must edit the pod DeploymentConfig object or use the Developer perspective in the web console. You cannot use the CLI to add or edit health checks for an existing pod.
- Readiness probe
A readiness probe determines if a container is ready to accept service requests. If the readiness probe fails for a container, the kubelet removes the pod from the list of available service endpoints.
After a failure, the probe continues to examine the pod. If the pod becomes available, the kubelet adds the pod to the list of available service endpoints.
- Liveness health check
A liveness probe determines if a container is still running. If the liveness probe fails due to a condition such as a deadlock, the kubelet kills the container. The pod then responds based on its restart policy.
For example, a liveness probe on a pod with a
restartPolicyofAlwaysorOnFailurekills and restarts the container.- Startup probe
A startup probe indicates whether the application within a container is started. All other probes are disabled until the startup succeeds. If the startup probe does not succeed within a specified time period, the kubelet kills the container, and the container is subject to the pod
restartPolicy.Some applications can require additional start-up time on their first initialization. You can use a startup probe with a liveness or readiness probe to delay that probe long enough to handle lengthy start-up time using the
failureThresholdandperiodSecondsparameters.For example, you can add a startup probe, with a
failureThresholdof 30 failures and aperiodSecondsof 10 seconds (30 * 10s = 300s) for a maximum of 5 minutes, to a liveness probe. After the startup probe succeeds the first time, the liveness probe takes over.
You can configure liveness, readiness, and startup probes with any of the following types of tests:
HTTP
GET: When using an HTTPGETtest, the test determines the healthiness of the container by using a web hook. The test is successful if the HTTP response code is between200and399.You can use an HTTP
GETtest with applications that return HTTP status codes when completely initialized.-
Container Command: When using a container command test, the probe executes a command inside the container. The probe is successful if the test exits with a
0status. - TCP socket: When using a TCP socket test, the probe attempts to open a socket to the container. The container is only considered healthy if the probe can establish a connection. You can use a TCP socket test with applications that do not start listening until initialization is complete.
You can configure several fields to control the behavior of a probe:
-
initialDelaySeconds: The time, in seconds, after the container starts before the probe can be scheduled. The default is 0. -
periodSeconds: The delay, in seconds, between performing probes. The default is10. This value must be greater thantimeoutSeconds. -
timeoutSeconds: The number of seconds of inactivity after which the probe times out and the container is assumed to have failed. The default is1. This value must be lower thanperiodSeconds. -
successThreshold: The number of times that the probe must report success after a failure to reset the container status to successful. The value must be1for a liveness probe. The default is1. failureThreshold: The number of times that the probe is allowed to fail. The default is 3. After the specified attempts:- for a liveness probe, the container is restarted
-
for a readiness probe, the pod is marked
Unready -
for a startup probe, the container is killed and is subject to the pod’s
restartPolicy
Example probes
The following are samples of different probes as they would appear in an object specification.
Sample readiness probe with a container command readiness probe in a pod spec
apiVersion: v1
kind: Pod
metadata:
labels:
test: health-check
name: my-application
...
spec:
containers:
- name: goproxy-app 1
args:
image: k8s.gcr.io/goproxy:0.1 2
readinessProbe: 3
exec: 4
command: 5
- cat
- /tmp/healthy
...
Sample container command startup probe and liveness probe with container command tests in a pod spec
apiVersion: v1
kind: Pod
metadata:
labels:
test: health-check
name: my-application
...
spec:
containers:
- name: goproxy-app 1
args:
image: k8s.gcr.io/goproxy:0.1 2
livenessProbe: 3
httpGet: 4
scheme: HTTPS 5
path: /healthz
port: 8080 6
httpHeaders:
- name: X-Custom-Header
value: Awesome
startupProbe: 7
httpGet: 8
path: /healthz
port: 8080 9
failureThreshold: 30 10
periodSeconds: 10 11
...
- 1
- The container name.
- 2
- Specify the container image to deploy.
- 3
- A liveness probe.
- 4
- An HTTP
GETtest. - 5
- The internet scheme:
HTTPorHTTPS. The default value isHTTP. - 6
- The port on which the container is listening.
- 7
- A startup probe.
- 8
- An HTTP
GETtest. - 9
- The port on which the container is listening.
- 10
- The number of times to try the probe after a failure.
- 11
- The number of seconds to perform the probe.
Sample liveness probe with a container command test that uses a timeout in a pod spec
apiVersion: v1
kind: Pod
metadata:
labels:
test: health-check
name: my-application
...
spec:
containers:
- name: goproxy-app 1
args:
image: k8s.gcr.io/goproxy:0.1 2
livenessProbe: 3
exec: 4
command: 5
- /bin/bash
- '-c'
- timeout 60 /opt/eap/bin/livenessProbe.sh
periodSeconds: 10 6
successThreshold: 1 7
failureThreshold: 3 8
...
- 1
- The container name.
- 2
- Specify the container image to deploy.
- 3
- The liveness probe.
- 4
- The type of probe, here a container command probe.
- 5
- The command line to execute inside the container.
- 6
- How often in seconds to perform the probe.
- 7
- The number of consecutive successes needed to show success after a failure.
- 8
- The number of times to try the probe after a failure.
Sample readiness probe and liveness probe with a TCP socket test in a deployment
kind: Deployment
apiVersion: apps/v1
...
spec:
...
template:
spec:
containers:
- resources: {}
readinessProbe: 1
tcpSocket:
port: 8080
timeoutSeconds: 1
periodSeconds: 10
successThreshold: 1
failureThreshold: 3
terminationMessagePath: /dev/termination-log
name: ruby-ex
livenessProbe: 2
tcpSocket:
port: 8080
initialDelaySeconds: 15
timeoutSeconds: 1
periodSeconds: 10
successThreshold: 1
failureThreshold: 3
...
