13. Alert Responses
Alert conditions define when an alert is supposed to be triggered. An alert notification defines how that alert notification is communicated. An alert notification can be a way of informing administrators and users of the condition, but it can also be a way of having JBoss ON itself address an issue.
A single alert can have multiple responses, to make managing potential problems easier.
Important
Alert notifications cannot be edited after they are added to an alert definition. To change an alert notification, delete the original notification and create a new one with the desired settings.
13.1. Notifying Administrators and Responding to Alerts
Every alert is recorded and viewable in the JBoss ON GUI. Alerts have an optional configuration, though, of sending an external notification whenever the alert is issued.
Once an incident occurs, there has to be a way to let a systems administrator know what is going on, so they can respond to an issue. This is done by configuring a notification. Alert notifications fall into two categories: a way of communicating the alert and an action to take in response to an alert.
There are three methods, by default, of communicating that an alert has been fired:
- Email, to one or multiple addresses
- SNMP traps
- Messages to JBoss ON users
There are also a few default methods of having JBoss ON respond to an alert:
- Running a resource operation (on the alerting resource or any other resource in inventory)
- Running a resource script (specific type of resource operation)
- JBoss ON CLI scripts
Note
It is also possible to write custom alert methods, which are implemented as server-side plug-ins. Creating custom plug-ins is described in the JBoss Operations Network Plug-ins Writing Guide.
Alert notifications can be clustered. That just means that the same alert can be broadcast through several different methods at the same time. For example, if a public website goes down, then a company may want notifications to be sent to their head web administrator and to have the web server restarted at the same time.
Note
A single alert can initiate multiple notifications. Alert notifications are run in the order they are listed in the alert definition.
13.2. Detailed Discussion: Initiating an Operation
A parallel response to an alert is to launch an operation. Resource operations (which, like metrics, are defined in the resource type agent plug-in) are launched, like a notification, in response to a triggered alert. Alert operations can be run on the resource that issued the alert or on any other resource in the inventory, which allows immediate and automatic responses to alert conditions. For instance, a JBoss server may begin performing badly because its JVM is out of memory. The JVM is the resource which issues its alert, but the response by the agent is to restart the JBoss server.
When a certain alert condition occurs, the JBoss ON agent can respond by initiating an operation on a resource. This is part of the alert definition configuration, but it's worth calling out because it is such a useful tool for managing responses to alerts. Whenever an alert is fired, the agent can perform some kind of action, like restarting a server. This can be done either on the resource which issued the alert or on another resource.
Remote operations can be exceedingly useful (and versatile). For example, a JBoss server may begin performing badly because its JVM is out of memory. The JVM is the resource which issues its alert, but the response by the agent is to restart the JBoss server.
Regular operations are either initiated immediately or run on defined schedules for a specific configured resource. Alert operations are even more flexible than regular operations for two reasons:
- Alert operations are fired responsively to address any alert or event.
- Alert operations can be initiated on any resource in the JBoss ON inventory, not only the resource which sent the alert. That means that an operation can be run for a different application on the same host server or even on an entirely different server.
The operations performed in response to an alert are the same as the operations which can be scheduled to run on a resource. The operations available for an alert depend on the target resource on which the operation will run — not the resource where the alert is set.
Alert operations senders can be used to run scripts on remote resources. For example, if a resource goes down, a diagnostic script can be run on its parent platform or another resource can be brought online and properly configured to take its place.
13.2.1. Using Tokens with Alert Operations
Alert operations can use tokens to either send information or supply information about the event. For example, tokens can be used to supply resource information in a command-line script.
Alert operations can accept tokens to fill in certain values automatically. These tokens have the following form:
<%space.param_name%>
The space gives the JBoss ON configuration area where the value is derived; this will commonly be either
alert or resource. The param_name gives the entry value that is being supplied. For example, to point to the URL of the specific fired alert, the token would be <%alert.url%>, while to pull in the resource name, the token would be <%resource.name%>.
JBoss ON has pre-defined token values that relate to the fired alert, the resource which issued the alert, the resource which is the target of the operation, and the operation that was initiated. These are listed in Table 7, “Available Alert Operation Tokens”. All of these potential token values are Java properties the belong to the operation's parent JBoss ON server.
The alert operations plug-in resolves the token value itself when the alert operation is processed to find the value. The realized value is sent to the script service, which ultimately plugs the value into the command-line argument or script which referenced the token.
| Information about ... | Token | Description |
|---|---|---|
| Fired Alert | alert.willBeDisabled | Will the alert definition be disabled after firing? |
| Fired Alert | alert.id | The id of this particular alert |
| Fired Alert | alert.url | Url to the alert details page |
| Fired Alert | alert.name | Name from the defining alert definition |
| Fired Alert | alert.priority | Priority of this alert |
| Fired Alert | alert.description | Description of this alert |
| Fired Alert | alert.firedAt | Time the alert fired |
| Fired Alert | alert.conditions | A text representation of the conditions that led to this alert |
| Alerting Resource | resource.id | ID of the resource |
| Alerting Resource | resource.platformType | Type of the platform the resource is on |
| Alerting Resource | resource.platformName | Name of the platform the resource is on |
| Alerting Resource | resource.typeName | Resource type name |
| Alerting Resource | resource.name | Name of the resource |
| Alerting Resource | resource.platformId | ID of the platform the resource is on |
| Alerting Resource | resource.parentName | Name of the parent resource |
| Alerting Resource | resource.parentId | ID of the parent resource |
| Alerting Resource | resource.typeId | Resource type id |
| Target Resource | targetResource.parentId | ID of the target's parent resource |
| Target Resource | targetResource.platformName | Name of the platform the target resource is on |
| Target Resource | targetResource.platformId | ID of the platform the target resource is on |
| Target Resource | targetResource.parentName | Name of the target's parent resource |
| Target Resource | targetResource.typeId | Resource type of the target resource id |
| Target Resource | targetResource.platformType | Type of the platform the target resource is on |
| Target Resource | targetResource.name | Name of the target resource |
| Target Resource | targetResource.id | ID of the target resource |
| Target Resource | targetResource.typeName | Resource type name of the target resource |
| Operation | operation.id | ID of the operation fired |
| Operation | operation.name | Name of the operation fired |
13.2.2. Setting Alert Operations
Note
A single alert can initiate multiple operations. All alert operations, as with all alert notifications, are run in the order they are listed in the alert definition.
The Notifications tab has a Resource Operations method.
Figure 29. Senders
There are two parts to the operation. First is selecting which resource the operation will run on. The resource type determines what operations are available.
The default is the resource that the alert is set for; it is also possible to set it on another specific resource or on the results of a search.
Figure 30. Resource Selection
Important
If you select a relative resource and do not enter a specific resource name, then the operation will run on every resource which matches that resource type in the relative path. If no resource matches, then it is logged into the audit trail and the alert process proceeds.
For a relative resource, the resource name is not required. For a specific resource, it is.
The second half is selecting the operation type. The available operations and their configuration parameters depend on the type of resource selected as the target of the operation.
Figure 31. Operation Settings
13.3. Detailed Discussion: Initiating Resource Scripts
Scripts, such as shell and bat scripts, can be imported into the JBoss ON inventory and managed as resources. These scripts can include start scripts, configuration scripts, or diagnostic scripts.
As described in Section 13.2.2, “Setting Alert Operations”, an alert notification can run an operation on a resource (even a different resource than the one which triggered the alert). A specific usage scenario is to run a resource script as an operation.
Note
The script must be uploaded to the resource and added into the JBoss ON inventory before it can be used in an alert operation.
Running a resource script is the same as running an operation. The resource script is selected as the resource for the operation, and then the start or restart operation for the script is set and, optionally, any command-line arguments to pass to the script.
Figure 32. Resource Script Settings
Important
If you select a relative resource and do not enter a specific script name in the name filter field, then the operation will run on every script resource that is in the relative path with the command arguments that are given. If no script matches, then it is logged into the audit trail and the alert process proceeds.
For a relative resource, the resource name is not required. For a specific resource, it is. To limit script execution to a single specific script, select the specific resource option and select the precise script from the list.
13.4. Detailed Discussion: Launching JBoss ON CLI Scripts from an Alert
JBoss ON has its own command-line client that can be used to manage server instances in the same way that the web UI manages servers. Much like running a script resource or launching an operation in response to an alert condition, a server CLI script can be run in response to an alert condition.
Note
Unlike resource scripts, CLI scripts are not treated as resources in the inventory. These are tools available to and used by the JBoss ON server itself (not limited or associated with any given resource).
Server CLI scripts must be uploaded to the server as content within a repository before it can be run.
The CLI script must use the proper API to perform the operation on the server. JBoss ON has several different API sets, depending on the task being performed. To connect to a server and run a script requires the remoting API, which allows commands to be executed on the server remotely. Writing CLI scripts is covered more in Running JBoss ON Command-Line Scripts.
Note
The CLI script can actually reference an alert object for the alert which triggers the script by using a pre-defined alert variable.
The
alert variable implicitly identifies the alert definition and specific alert instance which has been fired. This allows you to create a proxy resource definition in the script that could be applied to any resource which uses that alert script.
var myResource = ProxyFactory.getResource(alert.alertDefinition.resource.id)
The resource ID is pulled from the fired alert, and the script can reference the resource from there.
Example 1. Writing Alert-Relevant CLI Scripts
Server-side scripts are both powerful and versatile. They can touch almost any server functionality and any resource, group, or other object and run a series of commands. This versatility makes CLI scripts very useful to responding to alerts in proactive ways, but it means that the alert should be planned to notify on specific conditions and the script should be designed to respond to that specific set of circumstances.
In other words, make the alert specific and the CLI script relevant.
This script checks the recent monitoring statistics for a web application and restarts the web server database if there are connection problems:
var myResource = ProxyFactory.getResource(alert.alertDefinition.resource.id)
var definitionCriteria = new MeasurementDefinitionCriteria()
definitionCriteria.addFilterDisplayName('Sessions created per Minute')
definitionCriteria.addFilterResourceTypeId(myResource.resourceType.id)
var definitions = MeasumentDefinitionManager.findMeasurementDefinitionsByCriteria(definitionCriteria)
if (definitions.empty) {
throw new java.lang.Exception("Could not get 'Sessions created per Minute' metric on resource "
+ myResource.id)
}
var definition = definitions.get(0)
var startDate = new Date() - 8 * 3600 * 1000 //8 hrs in milliseconds
var endDate = new Date()
var data = MeasurementDataManager.findDataForResource(myResource.id, [ definition.id ], startDate, endDate, 60)
exporter.setTarget('csv', '/the/output/folder/for/my/metrics/' + endDate + '.csv')
exporter.write(data.get(0))
var dataSource = ProxyFactory.getResource(10411)
connectionTest = dataSource.testConnection()
if (connectionTest == null || connectionTest.get('result').booleanValue == false) {
//ok, this means we had problems connecting to the database
//let's suppose there's an executable bash script somewhere on the server that
//the admins use to restart the database
java.lang.Runtime.getRuntime().exec('/somewhere/on/the/server/restart-database.sh')
}
Commands, options, and variables for the JBoss ON CLI are listed in Running JBoss ON Command-Line Scripts.
An example alert script is included with the server files in serverInstallDir
/alert-scripts/.
- Upload the script to a content repository.
Note
Create a separate repository for alert CLI scripts. - Search for the resource, and configure the basic alert definition, as in Section 11.2, “Basic Procedure for Setting Alerts for a Resource”.
- In the Notifications tab for the alert definition, give the notification method a name, and select the CLI Script method from the drop-down menu.
- First, select the JBoss ON user as whom to run the script. The default is as the user who is creating the notification.
- Select the repository which contains the CLI script. If you are uploading a new script, this is the repository to which the script will be added.
- Select the CLI script to use from the drop-down menu, which lists all of the scripts in the specified repository. Alternatively, click the button to browse to a script on the local machine.
- Click to save the notification. The line in the Notifications tab shows the script, the repository, and the user as whom it will run.
13.5. Configuring SNMP for Notifications
Configuring JBoss ON to send SNMP alerts has two parts:
- Configuring the SNMP alert plug-in for the server.
- Configuring the actual alert with an SNMP notification.
13.5.1. JBoss ON SNMP Information
JBoss ON can send SNMP traps to other management stations and systems as part of alerting notifications. The data transmitted contain details about the alert, such as the name of the alert that was triggered and the resource name.
The data to include in the traps, as with other SNMP notifications, are defined in the JBoss ON MIB file, in serverRoot
/etc/RHQ-mib.txt. The default configuration for the MIB is shown in Example 2, “Default Alert Object in JBoss ON MIB”. The base OID for the JBoss ON alert is 1.3.6.1.4.1.18016.2.1 (org.dod.internet.private.enterprise.jboss.rhq.alert).
Example 2. Default Alert Object in JBoss ON MIB
alertGroup OBJECT-GROUP
OBJECTS { alertName,
alertResourceName,
alertPlatformName,
alertCondition,
alertSeverity,
alertUrl }
STATUS current
DESCRIPTION "A collection of objects providing information about an alert"
With the default MIB file, each trap sends the alert definition name, resource name, platform, alert conditions, severity, and a URL to the alert details page.
13.5.2. Configuring the SNMP Alert Plug-in
The SNMP alert sender plug-in is the only alert notification plug-in that requires additional configuration before the notification method can be used. The SNMP plug-in has to be configured with the appropriate SNMP version and SNMP agent information.
- In the top menu, select the Administration tab.
- In the System Configuration menu, select the ServerPlugins item.
- Click the name of the SNMP plug-in in the list.
- In the plug-in details page, expand the Plugin Configuration section.
- All SNMP versions require information about the JBoss ON MIB OID and selected version. Fill in the appropriate values.
- SNMP version 1 and version 3 both require additional configuration. Expand the version-specific configuration section and fill in the information about the SNMP agent.
It may be necessary to unselect the Unset checkbox to allow the fields to be edited.
13.5.3. Configuring the SNMP Alert Notification
Before JBoss ON can send any SNMP notifications, SNMP traps have to be configured for the server.
When configuring the alert, select the SNMP Trap notification type, and fill in the JBoss ON SNMP information.
Figure 33. JBoss ON SNMP Trap Information
- The hostname for the SNMP manager.
- The port number for the SNMP manager. JBoss ON supports UDP, so this must be the UDP port.
- The JBoss ON OID. This is
1.3.6.1.4.1.18016.2.1.
