Chapter 28. Triggering scripts for cluster events
A Pacemaker cluster is an event-driven system, where an event might be a resource or node failure, a configuration change, or a resource starting or stopping. You can configure Pacemaker cluster alerts to take some external action when a cluster event occurs by means of alert agents, which are external programs that the cluster calls in the same manner as the cluster calls resource agents to handle resource configuration and operation.
The cluster passes information about the event to the agent by means of environment variables. Agents can do anything with this information, such as send an email message or log to a file or update a monitoring system.
-
Pacemaker provides several sample alert agents, which are installed in
/usr/share/pacemaker/alerts
by default. These sample scripts may be copied and used as is, or they may be used as templates to be edited to suit your purposes. Refer to the source code of the sample agents for the full set of attributes they support. - If the sample alert agents do not meet your needs, you can write your own alert agents for a Pacemaker alert to call.
28.1. Installing and configuring sample alert agents
When you use one of the sample alert agents, you should review the script to ensure that it suits your needs. These sample agents are provided as a starting point for custom scripts for specific cluster environments. Note that while Red Hat supports the interfaces that the alert agents scripts use to communicate with Pacemaker, Red Hat does not provide support for the custom agents themselves.
To use one of the sample alert agents, you must install the agent on each node in the cluster. For example, the following command installs the alert_file.sh.sample
script as alert_file.sh
.
install --mode=0755 /usr/share/pacemaker/alerts/alert_file.sh.sample /var/lib/pacemaker/alert_file.sh
# install --mode=0755 /usr/share/pacemaker/alerts/alert_file.sh.sample /var/lib/pacemaker/alert_file.sh
After you have installed the script, you can create an alert that uses the script.
28.1.1. Configuring an alert that uses the alert_file.sh
alert agent
The following procedure installs the alert_file.sh
alert agent and configures an alert to log events to a file. Alert agents run as the user hacluster
, which has a minimal set of permissions.
Procedure
On each node in the cluster, install the
alert_file.sh.sample
script asalert_file.sh
.install --mode=0755 /usr/share/pacemaker/alerts/alert_file.sh.sample /var/lib/pacemaker/alert_file.sh
# install --mode=0755 /usr/share/pacemaker/alerts/alert_file.sh.sample /var/lib/pacemaker/alert_file.sh
Copy to Clipboard Copied! On each node in the cluster, create the log file that will be used to record the events.
touch /var/log/pcmk_alert_file.log
# touch /var/log/pcmk_alert_file.log
Copy to Clipboard Copied! Change the ownership of the file
pcmk_alert_file.log
to userhacluster
.chown hacluster:haclient /var/log/pcmk_alert_file.log
# chown hacluster:haclient /var/log/pcmk_alert_file.log
Copy to Clipboard Copied! Change the permissions for the file
pcmk_alert_file.log
to read and write for the userhacluster
.chmod 600 /var/log/pcmk_alert_file.log
# chmod 600 /var/log/pcmk_alert_file.log
Copy to Clipboard Copied!
On one node in the cluster, create the alert.
pcs alert create id=alert_file description="Log events to a file." path=/var/lib/pacemaker/alert_file.sh
# pcs alert create id=alert_file description="Log events to a file." path=/var/lib/pacemaker/alert_file.sh
Copy to Clipboard Copied! Add the path to the log file as the recipient for the alert.
pcs alert recipient add alert_file id=my-alert_logfile value=/var/log/pcmk_alert_file.log
# pcs alert recipient add alert_file id=my-alert_logfile value=/var/log/pcmk_alert_file.log
Copy to Clipboard Copied!
28.1.2. Configuring an alert that uses the alert_snmp.sh
alert agent
The following procedure installs the alert_snmp.sh.sample
script as alert_snmp.sh
and configures an alert that uses the installed alert_snmp.sh
alert agent to send cluster events as SNMP traps. By default, the script will send all events except successful monitor calls to the SNMP server.
Procedure
On each node in the cluster, install the
alert_snmp.sh.sample
script asalert_snmp.sh
.install --mode=0755 /usr/share/pacemaker/alerts/alert_snmp.sh.sample /var/lib/pacemaker/alert_snmp.sh
# install --mode=0755 /usr/share/pacemaker/alerts/alert_snmp.sh.sample /var/lib/pacemaker/alert_snmp.sh
Copy to Clipboard Copied! On one node in the cluster, configure an alert that uses the
alert_snmp.sh
agent, configuring the timestamp format as a meta option.pcs alert create id=snmp_alert path=/var/lib/pacemaker/alert_snmp.sh meta timestamp-format="%Y-%m-%d,%H:%M:%S.%01N"
# pcs alert create id=snmp_alert path=/var/lib/pacemaker/alert_snmp.sh meta timestamp-format="%Y-%m-%d,%H:%M:%S.%01N"
Copy to Clipboard Copied! Configure a recipient for the alert
pcs alert recipient add snmp_alert value=192.168.1.2
# pcs alert recipient add snmp_alert value=192.168.1.2
Copy to Clipboard Copied! Display the alert configuration.
pcs alert
# pcs alert Alerts: Alert: snmp_alert (path=/var/lib/pacemaker/alert_snmp.sh) Meta options: timestamp-format=%Y-%m-%d,%H:%M:%S.%01N. Recipients: Recipient: snmp_alert-recipient (value=192.168.1.2)
Copy to Clipboard Copied!
28.1.3. Configuring an alert that uses the alert_smtp.sh
alert agent
The following procedure installs the alert_smtp.sh
agent and then configures an alert that uses the installed alert agent to send cluster events as email messages.
Procedure
On each node in the cluster, install the
alert_smtp.sh.sample
script asalert_smtp.sh
.install --mode=0755 /usr/share/pacemaker/alerts/alert_smtp.sh.sample /var/lib/pacemaker/alert_smtp.sh
# install --mode=0755 /usr/share/pacemaker/alerts/alert_smtp.sh.sample /var/lib/pacemaker/alert_smtp.sh
Copy to Clipboard Copied! On one node in the cluster, configure an alert to send cluster events as email messages.
pcs alert create id=smtp_alert path=/var/lib/pacemaker/alert_smtp.sh options email_sender=donotreply@example.com
# pcs alert create id=smtp_alert path=/var/lib/pacemaker/alert_smtp.sh options email_sender=donotreply@example.com
Copy to Clipboard Copied! Configure a recipient for the alert.
pcs alert recipient add smtp_alert value=admin@example.com
# pcs alert recipient add smtp_alert value=admin@example.com
Copy to Clipboard Copied! Display the alert configuration.
pcs alert
# pcs alert Alerts: Alert: smtp_alert (path=/var/lib/pacemaker/alert_smtp.sh) Options: email_sender=donotreply@example.com Recipients: Recipient: smtp_alert-recipient (value=admin@example.com)
Copy to Clipboard Copied!
28.2. Creating a cluster alert
The following command creates a cluster alert. The options that you configure are agent-specific configuration values that are passed to the alert agent script at the path you specify as additional environment variables. If you do not specify a value for id
, one will be generated.
pcs alert create path=path [id=alert-id] [description=description] [options [option=value]...] [meta [meta-option=value]...]
pcs alert create path=path [id=alert-id] [description=description] [options [option=value]...] [meta [meta-option=value]...]
Multiple alert agents may be configured; the cluster will call all of them for each event. Alert agents will be called only on cluster nodes. They will be called for events involving Pacemaker Remote nodes, but they will never be called on those nodes.
The following example creates a simple alert that will call myscript.sh
for each event.
pcs alert create id=my_alert path=/path/to/myscript.sh
# pcs alert create id=my_alert path=/path/to/myscript.sh
For a more complete example procedure to create and modify a full cluster alert, see Creating and modifying a cluster alert.
28.3. Displaying, modifying, and removing cluster alerts
There are a variety of pcs
commands you can use to display, modify, and remove cluster alerts.
Displaying configured cluster alerts
The following command shows all configured alerts along with the values of the configured options.
pcs alert [config]
pcs alert [config]
Modifying a configured cluster alert
The following command updates an existing alert with the specified alert-id value.
pcs alert update alert-id [path=path] [description=description] [options [option=value]...] [meta [meta-option=value]...]
pcs alert update alert-id [path=path] [description=description] [options [option=value]...] [meta [meta-option=value]...]
Removing a cluster alert
The following command removes an alert with the specified alert-id value.
pcs alert remove alert-id
pcs alert remove alert-id
Alternately, you can run the pcs alert delete
command, which is identical to the pcs alert remove
command. Both the pcs alert delete
and the pcs alert remove
commands allow you to specify more than one alert to be deleted.
28.4. Configuring cluster alert recipients
Usually alerts are directed towards a recipient. Thus each alert may be additionally configured with one or more recipients. The cluster will call the agent separately for each recipient.
The recipient may be anything the alert agent can recognize: an IP address, an email address, a file name, or whatever the particular agent supports.
Adding a recipient to a configured cluster alert
The following command adds a new recipient to the specified alert.
pcs alert recipient add alert-id value=recipient-value [id=recipient-id] [description=description] [options [option=value]...] [meta [meta-option=value]...]
pcs alert recipient add alert-id value=recipient-value [id=recipient-id] [description=description] [options [option=value]...] [meta [meta-option=value]...]
The following example command adds the alert recipient my-alert-recipient
with a recipient ID of my-recipient-id
to the alert my-alert
. This will configure the cluster to call the alert script that has been configured for my-alert
for each event, passing the recipient some-address
as an environment variable.
pcs alert recipient add my-alert value=my-alert-recipient id=my-recipient-id options value=some-address
# pcs alert recipient add my-alert value=my-alert-recipient id=my-recipient-id options value=some-address
Updating an existing alert recipient
The following command updates an existing alert recipient.
pcs alert recipient update recipient-id [value=recipient-value] [description=description] [options [option=value]...] [meta [meta-option=value]...]
pcs alert recipient update recipient-id [value=recipient-value] [description=description] [options [option=value]...] [meta [meta-option=value]...]
Removing an alert recipient
The following command removes the specified alert recipient.
pcs alert recipient remove recipient-id
pcs alert recipient remove recipient-id
Alternately, you can run the pcs alert recipient delete
command, which is identical to the pcs alert recipient remove
command. Both the pcs alert recipient remove
and the pcs alert recipient delete
commands allow you to remove more than one alert recipient.
28.5. Configuring cluster alert meta options
As with resource agents, meta options can be configured for alert agents to affect how Pacemaker calls them. The following table describes the alert meta options. Meta options can be configured per alert agent as well as per recipient.
Meta-Attribute | Default | Description |
---|---|---|
|
|
If set to |
| %H:%M:%S.%06N |
Format the cluster uses when sending the event’s timestamp to the agent. This is a string as used with the |
| 30s | If the alert agent does not complete within this amount of time, it will be terminated. |
Procedure
The following example procedure configures cluster alert meta options for both an alert agent and for the alert recipients. The procedure configures an alert that calls the script myscript.sh
and then adds two recipients to the alert. The script gets called twice for each event.
Configure an alert that calls the script
myscript.sh
and that uses a 15-second timeout.pcs alert create id=my-alert path=/path/to/myscript.sh meta timeout=15s
# pcs alert create id=my-alert path=/path/to/myscript.sh meta timeout=15s
Copy to Clipboard Copied! Add an alert recipient that has an ID of
my-alert-recipient1
, passing the call to the recipientsomeuser@example.com
with a timestamp in the format %D %H:%M.pcs alert recipient add my-alert value=someuser@example.com id=my-alert-recipient1 meta timestamp-format="%D %H:%M"
# pcs alert recipient add my-alert value=someuser@example.com id=my-alert-recipient1 meta timestamp-format="%D %H:%M"
Copy to Clipboard Copied! Add an alert recipient that has an ID of
my-alert-recipient2
, passing the call to the recipientotheruser@example.com
with a timestamp in the format %c.pcs alert recipient add my-alert value=otheruser@example.com id=my-alert-recipient2 meta timestamp-format="%c"
# pcs alert recipient add my-alert value=otheruser@example.com id=my-alert-recipient2 meta timestamp-format="%c"
Copy to Clipboard Copied!
28.6. Creating and modifying a cluster alert
The following example procedure creates alerts, adds recipients, modifies an alert, removes an alert, and displays the configured alerts at each step.
While you must install the alert agents themselves on each node in a cluster, you need to run the pcs
commands only once.
Procedure
Create a simple alert. Since no alert ID value is specified, the system creates an alert ID value of
alert
.pcs alert create path=/my/path
# pcs alert create path=/my/path
Copy to Clipboard Copied! Add a recipient of
rec_value
to the alert. Since this command does not specify a recipient ID, the value ofalert-recipient
is used as the recipient ID.pcs alert recipient add alert value=rec_value
# pcs alert recipient add alert value=rec_value
Copy to Clipboard Copied! Add a second recipient of
rec_value2
to the alert. This command specifies a recipient ID ofmy-recipient
for the recipient.pcs alert recipient add alert value=rec_value2 id=my-recipient
# pcs alert recipient add alert value=rec_value2 id=my-recipient
Copy to Clipboard Copied! Display the alert configuration.
pcs alert config
# pcs alert config Alerts: Alert: alert (path=/my/path) Recipients: Recipient: alert-recipient (value=rec_value) Recipient: my-recipient (value=rec_value2)
Copy to Clipboard Copied! Add a second alert, with an alert ID of
my-alert
.pcs alert create id=my-alert path=/path/to/script description=alert_description options option1=value1 opt=val meta timeout=50s timestamp-format="%H%B%S"
# pcs alert create id=my-alert path=/path/to/script description=alert_description options option1=value1 opt=val meta timeout=50s timestamp-format="%H%B%S"
Copy to Clipboard Copied! Add a recipient for that alert for the second alert with a recipient value of
my-other-recipient
. Since no recipient ID is specified, the system provides a recipient id ofmy-alert-recipient
.pcs alert recipient add my-alert value=my-other-recipient
# pcs alert recipient add my-alert value=my-other-recipient
Copy to Clipboard Copied! Display the alert configuration.
pcs alert
# pcs alert Alerts: Alert: alert (path=/my/path) Recipients: Recipient: alert-recipient (value=rec_value) Recipient: my-recipient (value=rec_value2) Alert: my-alert (path=/path/to/script) Description: alert_description Options: opt=val option1=value1 Meta options: timestamp-format=%H%B%S timeout=50s Recipients: Recipient: my-alert-recipient (value=my-other-recipient)
Copy to Clipboard Copied! Modify the alert values for the alert
my-alert
.pcs alert update my-alert options option1=newvalue1 meta timestamp-format="%H%M%S"
# pcs alert update my-alert options option1=newvalue1 meta timestamp-format="%H%M%S"
Copy to Clipboard Copied! Modify the alert values for the recipient
my-alert-recipient
.pcs alert recipient update my-alert-recipient options option1=new meta timeout=60s
# pcs alert recipient update my-alert-recipient options option1=new meta timeout=60s
Copy to Clipboard Copied! Display the alert configuration.
pcs alert
# pcs alert Alerts: Alert: alert (path=/my/path) Recipients: Recipient: alert-recipient (value=rec_value) Recipient: my-recipient (value=rec_value2) Alert: my-alert (path=/path/to/script) Description: alert_description Options: opt=val option1=newvalue1 Meta options: timestamp-format=%H%M%S timeout=50s Recipients: Recipient: my-alert-recipient (value=my-other-recipient) Options: option1=new Meta options: timeout=60s
Copy to Clipboard Copied! Remove the recipient
my-alert-recipient
fromalert
.pcs alert recipient remove my-recipient
# pcs alert recipient remove my-recipient
Copy to Clipboard Copied! Display the alert configuration.
pcs alert
# pcs alert Alerts: Alert: alert (path=/my/path) Recipients: Recipient: alert-recipient (value=rec_value) Alert: my-alert (path=/path/to/script) Description: alert_description Options: opt=val option1=newvalue1 Meta options: timestamp-format="%M%B%S" timeout=50s Recipients: Recipient: my-alert-recipient (value=my-other-recipient) Options: option1=new Meta options: timeout=60s
Copy to Clipboard Copied! Remove the alert
myalert
from the configuration.pcs alert remove myalert
# pcs alert remove myalert
Copy to Clipboard Copied! Display the alert configuration.
pcs alert
# pcs alert Alerts: Alert: alert (path=/my/path) Recipients: Recipient: alert-recipient (value=rec_value)
Copy to Clipboard Copied!
28.7. Writing a cluster alert agent
There are three types of Pacemaker cluster alerts: node alerts, fencing alerts, and resource alerts. The environment variables that are passed to the alert agents can differ, depending on the type of alert. The following table describes the environment variables that are passed to alert agents and specifies when the environment variable is associated with a specific alert type.
Environment Variable | Description |
---|---|
| The type of alert (node, fencing, or resource) |
| The version of Pacemaker sending the alert |
| The configured recipient |
| A sequence number increased whenever an alert is being issued on the local node, which can be used to reference the order in which alerts have been issued by Pacemaker. An alert for an event that happened later in time reliably has a higher sequence number than alerts for earlier events. Be aware that this number has no cluster-wide meaning. |
|
A timestamp created prior to executing the agent, in the format specified by the |
| Name of affected node |
|
Detail about event. For node alerts, this is the node’s current state (member or lost). For fencing alerts, this is a summary of the requested fencing operation, including origin, target, and fencing operation error code, if any. For resource alerts, this is a readable string equivalent of |
| ID of node whose status changed (provided with node alerts only) |
| The requested fencing or resource operation (provided with fencing and resource alerts only) |
| The numerical return code of the fencing or resource operation (provided with fencing and resource alerts only) |
| The name of the affected resource (resource alerts only) |
| The interval of the resource operation (resource alerts only) |
| The expected numerical return code of the operation (resource alerts only) |
| A numerical code used by Pacemaker to represent the operation result (resource alerts only) |
When writing an alert agent, you must take the following concerns into account.
- Alert agents may be called with no recipient (if none is configured), so the agent must be able to handle this situation, even if it only exits in that case. Users may modify the configuration in stages, and add a recipient later.
- If more than one recipient is configured for an alert, the alert agent will be called once per recipient. If an agent is not able to run concurrently, it should be configured with only a single recipient. The agent is free, however, to interpret the recipient as a list.
- When a cluster event occurs, all alerts are fired off at the same time as separate processes. Depending on how many alerts and recipients are configured and on what is done within the alert agents, a significant load burst may occur. The agent could be written to take this into consideration, for example by queueing resource-intensive actions into some other instance, instead of directly executing them.
-
Alert agents are run as the
hacluster
user, which has a minimal set of permissions. If an agent requires additional privileges, it is recommended to configuresudo
to allow the agent to run the necessary commands as another user with the appropriate privileges. -
Take care to validate and sanitize user-configured parameters, such as
CRM_alert_timestamp
(whose content is specified by the user-configuredtimestamp-format
),CRM_alert_recipient
, and all alert options. This is necessary to protect against configuration errors. In addition, if some user can modify the CIB without havinghacluster
-level access to the cluster nodes, this is a potential security concern as well, and you should avoid the possibility of code injection. -
If a cluster contains resources with operations for which the
on-fail
parameter is set tofence
, there will be multiple fence notifications on failure, one for each resource for which this parameter is set plus one additional notification. Both thepacemaker-fenced
andpacemaker-controld
will send notifications. Pacemaker performs only one actual fence operation in this case, however, no matter how many notifications are sent.
The alerts interface is designed to be backward compatible with the external scripts interface used by the ocf:pacemaker:ClusterMon
resource. To preserve this compatibility, the environment variables passed to alert agents are available prepended with CRM_notify_
as well as CRM_alert_
. One break in compatibility is that the ClusterMon
resource ran external scripts as the root user, while alert agents are run as the hacluster
user.