Release Notes
JBoss Operations Network 3.1
for release information
Copyright © 2012 Red Hat, Inc.
June 12, 2012
Abstract
These release notes contain important information about new features, known issues, and other technical notes available at the time that JBoss Operations Network 3.1 was released.
These release notes contain feature information, changed support, and structural changes in JBoss Operations Network 3.1.
JBoss Operations Network provides centralized control, configuration, and visibility to JBoss applications and related infrastructure. Broad and detailed monitoring, versioned application deployment, auditable configuration management, and controlled software patching through JBoss ON reduce the complexity, improve the reliability, and decrease the cost of large scale JBoss deployments.
This version of JBoss Operations Network contains new features, enhancements, and bug fixes. It is recommended that all JBoss Operations Network users upgrade to JBoss Operations Network 3.1.
1. New Features in JBoss Operations Network 3.1
This version of JBoss ON introduces both new features and feature enhancements that improve JBoss ON's performance for managing resources.
1.1. New: JBoss Resource Plug-ins for JBoss Enterprise Application Platform 6
JBoss Enterprise Application Platform (EAP) 6 emphasizes performance, speed, and deployment flexibility. In particular, there are two key features in EAP 6:
- A new architecture for managed domains, which introduce the idea of centralized domain controllers, local system host controllers, and low-level managed servers, with configuration defined in concentric levels from the domain down to local applications.A classic standalone server still exists that parallels the JBoss EAP 4 and 5 servers, which can be clustered or configured for high availability.
- Classes are modular, so classes can be loaded and unloaded as needed, lowering the EAP footprint and improving system performance.
JBoss ON manages and monitors all JBoss EAP 6-related resources: domain and host controllers, standalone servers, mod_cluster services, Infinispan services, HornetQ servers, and web contexts and applications.
The new architectures possible with JBoss EAP 6 can make simple deployment and management decisions very complex — and that's where a centralized, consistent management framework is invaluable. JBoss ON helps bring order to a diverse and intimidatingly complex EAP 6 deployment.
Working in tandem with EAP 6's native management console and CLI tools, JBoss ON offers a platform-centric perspective on the hierarchy and relationship between elements within a JBoss EAP 6 domain or a cluster. JBoss ON provides a sense of structure on otherwise unrelated EAP 6 resources and makes it easier to manage the entire deployment.
The benefits of monitoring JBoss EAP 6 through JBoss ON include:
- Define and manage application workflows. Test an application on a single server and then promote the application to a server group or deploy it on a new domain, a standalone server, or a cluster through a single provisioning path.
- Manage multiple standalone instances and EAP 6 domains from a single management console.
- Gain complete, chronological visibility into availability, performance metrics, auditable operations, events, and configuration changes.
- Organize EAP 6 resources in a platform-oriented hierarchy that offers a deployment-focused perspective on servers, services, and child applications.For example, a datasource in the EAP 6 console has three separate resources — the datasource definitions and then two separate views for metrics and configuration. In JBoss ON, this is consolidated into a single view for the datasource.
- Automatically detect runtime environment variables and launch script arguments. These start options can be viewed and edited through JBoss ON, which makes them easy to apply to other managed resources.
Note
If you installed the tech preview plug-in for JBoss AS 7.1, that resource must be uninventoried and re-added using the new EAP 6 plug-in. It cannot be migrated.
A full list of changes, current work items, and issues for the JBoss ON EAP 6 plug-in is available in Bugzilla #707223.
Documentation for managing EAP 6 standalone servers, domain controllers, manager servers, and related services is covered in the "How to Manage JBoss Servers." As always, complete reference material for the EAP 6 plug-ins is available in the Complete Resource Reference, which lists all supported configuration properties, connection properties, metrics, and operations.
1.2. Enahanced: More Control over the Start Script Environment
Part of the JBoss EAP resource connection settings define start script values that the JBoss ON agent uses when running a start operation on the EAP resource. In previous versions of the JBoss EAP plug-in, the agent used a bind address and the Java home directory to define the script environment.
In JBoss ON 3.1, both the JBoss EAP/AS 5 plug-in and the new JBoss EAP 6 plug-in perform more detailed discoveries of the start script at the time they discover the EAP resources. This improved discovery scan attempts to identify more information about the start script environment, and then recreate those settings in the settings it uses for the start operation. The discovery scan detects:
- The start script used, including custom start scripts.
- Required environment variables set in the
run.conffile or the parent process (a subset of possible or all environment variables). - Any arguments passed with the start script itself.
- EAP 6 only. What user the script is running as. If the EAP server and JBoss ON agent users are different, JBoss ON assigns a
sudocommand as a prefix command to use with the start script, such assudo -u jboss -g jboss.
The discovery process handles missing information gracefully. If it cannot detect some information or if it is not set, the agent simply ignores that parameter.
Note
On EAP 5, if the Script Arguments field is empty, then the bind address and Java home directory are still used for invoking the start script.
If any script arguments are set, then the bind address are Java home values are ignored.
1.3. Enhanced: More Responsive, Accurate, and Granular Availability Monitoring
Availability — monitoring whether a resource is running (up) or not — is one of the most basic and most critical aspects of infrastructure management.
In previous versions of JBoss ON, availability was polled for every resource, by the agent, on a set interval defined in the agent configuration (5 minutes by default). This rigid schedule led to some inefficiencies: periodic CPU spikes, perceived collection lags from slow reporting times, and gaps of several minutes between availability checks for a resource which could miss resource cycling. Additionally, the agent's own availability was tied to its sending the availability report to the server, so every agent "heartbeat" actually contained a heavyweight report for every resource in the agent's inventory.
JBoss ON 3.1 introduces a new approach to availability scheduling and monitoring that both improves overall JBoss ON performance and provides better accuracy and detail in availability reports.
Full documentation on availability monitoring is located in the availability chapter of "Setting up Monitoring, Alerts, and Operations."
Prioritized Availability Collection and Availability Schedules
One of the biggest changes to availability is allowing availability to be scheduled at different intervals for different resources and resource types, much like a metric schedule. This allows important resources to be prioritized higher than others.
Default availability schedules have been reset in three ways:
- Server availability is collected every minute.
- Service availability is collected every 10 minutes.
- Some dependent child resources have availability checking disabled entirely, and they default to the parent availability state. These resource can have availability checking enabled if desired.
Along with checking availability on schedules for individual resources and types, the availability scan itself has been changed. Previously, scans occurred every five minutes, and the agent checked every resource at the same time. Now, the agent runs a scan every 30 seconds and checks only a subset of resources, whichever ones (according to their collection schedule) are ready to be checked. On the initial scan, the resources are spaced out more or less evenly; after that, the resource is checked at the scan interval (30 seconds) plus its own configured interval. For example, a serve has an interval of one minute, so its availability is checked every 90 seconds — the time to run the agent scan plus the availability schedule.
Having a more frequent availability scan which checks only a subset of resources prevents CPU spikes and improves agent performance, while actually increasing how frequently the availability of critical resources is checked.
More Availability States
A resource can be in several different states: UP, DOWN, UNKNOWN, and DISABLED.
UP, DOWN, and UNKNOWN states have always existed, but now there are slight difference in how UNKNOWN and DOWN are used. UNKNOWN is now used for child resource when the agent goes down — because the children could be running fine. Their status is, legitimately, unknown rather than down.
Disabled is a new, administrative state. This is a signal to the JBoss ON server to ignore whatever availability changes come in. The agent still detects and sends reports, but the server ignores them. This allows resource to be cycled for maintenance or upgrades without sending spurious alert notifications or having to manually disable alert definitions.
The additional availability states for a resource have an impact on how availability is displayed for compatible groups.
Note
Availability states are evaluated "top down." If a resource is down, disabled, or unknown, then all of its children are immediately assumed to be in that state, as well.
| If the Resource States Are .... | ... the Group State Is ... |
|---|---|
| Empty Group (Unknown) | Empty |
| All Red (Down) | Red (Down) |
| Some Down or Unknown | Yellow (Mixed) |
| Some Orange (Disabled) | Orange (Disabled) |
| All Green (Up) | Green (Up) |
New Availability Alert Conditions
Previously, JBoss ON could send an alert on an availability state change, whether a resource went up or down. Alerting based on availability has been enhanced in two ways:
- A new not up option has been added for availability state changes. An alert can now be fired if a resource comes up, goes down, or goes into another (not up) state.
- A new alert condition has been added for availability duration. If a resource changes state and then stays in that state for a given amount of time, an alert is fired.
Changes to the avail Prompt Command
The avail command for the agent command prompt can send an availability report that contains only changed availability states or all current states, regardless of changes. This has always been true.
However, in previous version, running the
avail command checked every single resource in the agent's inventory. Now, it only checks the next scheduled resource by default. Checking the complete inventory requires the --force option.
New Agent Heartbeat
The agent heartbeat (the signal to the server that the agent is running) has been removed from general availability reporting. Instead, the agent sends a lightweight ping to the server.
Additionally, the agent quiet time — the amount of time the server waits between pings before assuming the agent is offline — has been lowered from 15 minutes to 5 minutes. This helps the JBoss ON server recognize more quickly when the agent goes offline and to backfill resource states.
The agent availability used to be tied only to its availability report. Even if the agent shut down, the server still waited the entire quiet time interval before marking the agent as offline. The agent now informs the server when it shuts down gracefully and all the platform child resource are immediately backfilled to UNKNOWN states, rather than waiting the full quiet time.
1.4. New: Export Reports to CSV
The JBoss ON web UI has a main tab, Reports, which defines several queries for different elements of the JBoss ON inventory and resource management, including an inventory breakdown by resource type, platform performance, and lists of configured definitions for alerts and drift.
For additional analysis and manipulation, these reports can now be exported to a comma-separated values (CSV) spreadsheet. Some reports define filters, such as filtering fired alerts by priority or by date. When a filtered report is exported, all of the filters are preserved, and only the displayed information is included in the CSV file.
For information on JBoss ON reporting, see the reporting section of "Initial Setup for the Resource Inventory, Groups, and Users."
1.5. Tech Preview: REST Interface
A new REST interface is being designed for JBoss ON. The REST interface is accessible for the JBoss ON server at:
http://server.example.com:7080/rest/
This REST API is under active development. While some interfaces are probably stable (such as resources and metrics methods), some interfaces are still being designed and could change significantly as development progresses (such as the alerts methods). As with all preview technologies, expect significant changes in subsequent releases of the REST API. Compatibility is not guaranteed, so use caution when developing clients using the tech preview version of the JBoss ON REST API.
The REST API is available for developers at http://docs.redhat.com/docs/en-US/JBoss_Operations_Network/3.1/html/Rest_API/index.html.
1.6. Enhanced: Additional Monitoring Metrics
New monitoring metrics have been added:
- A new trait has been added to datasource and connection factory monitoring metrics for JBoss AS 5 servers which determines whether the connection is available. This tests the connection settings every 15 minutes, by default.
- A new count for the number of open files for a JVM process has been added. Operating systems commonly put limits on how many open files a single process can have, which can cause otherwise performant JVM processes to fail when they hit that limit. Tracking the open files metric can help signal administrators when the JVM is approaching that limit.This metric is only relevant on Linux and Unix systems; it is not available on Windows systems.
- New actual free memory and actual total memory metrics have been added for platforms. These memory counts include cache and buffer settings in the available memory calculation.
- A new maximum thread count metric has been added to the embedded JBossWeb Connector resource for JBoss AS 5 servers.
Metrics for all support resource plug-ins are available in the Complete Resource Reference, which lists all supported configuration properties, connection properties, metrics, and operations.
1.7. New: Access Control Permission to View User Details
In previous versions of JBoss ON, any user with global permissions could view other user accounts (even though creating or editing user accounts was limited to users with Security permissions).
A new global permission has been added that explicitly grants the right to view user details such as the user name, full name, and contact information for a JBoss ON user (while excluding any view of the user's role assignments). Roles without the View User permission are implicitly denied the ability to view any user details.
Only roles with the Security permission, which is roughly equivalent to a superuser, have the ability view role assignments for users.
1.8. Enhanced: Better Discovery and Management for Custom JVM Resources
In previous versions of JBoss ON, the Generic JMX agent plug-in only automatically discovered JMX resources if they were using JMX remoting services and the agent could connect to the agent over the JMX remoting ports. In large deployments, it is not realistic to manage many open JMX remoting ports.
The Generic JMX agent plug-in in JBoss ON 3.1 has been enhanced to discover any JMX server running on the platform, as long as it is configured in one of two ways that the agent detects:
- JMX remoting is enabled, and the remoting port is set through the command line, so that the agent can read the options. This is the method that has always been used to discover JMX servers.
- The JVM process is accessible through the Java attach API and has a resource key system property set in its command line. The key allows the agent to identify the appropriate Java process and provide a unique resource identity.In this case, the agent and the JVM must be running as the same system user, or the agent cannot attach to the process.
Any JVM process explicitly discovered by another resource plug-in, such as a Tomcat server, is still preferentially discovered by the specific resource plug-in. Customer or generic JVM processes are discovered by the Generic JMX plug-in as a top-level[1] JVM server.
If a JVM server should be ignored by the Generic JMX plug-in, then an excludes rule can be defined in the JBoss ON agent configuration that instructs the agent to ignore any matching JVM processes.
Managing JMX servers and generic JVMs is covered in "How to Manage JBoss Servers."
1.9. New: Resource Plug-in to Monitor and Manage JBoss Data Grid (Infinispan)
JBoss ON 3.1 introduces a new resource plug-in to manage JBoss Data Grid 6.0. This plug-in pack is available with the JBoss Data Grid packages in the Customer Portal.
Complete reference material for the JBoss Data Grid plug-in is available in the Resource Reference, which lists all supported configuration properties, connection properties, metrics, and operations.
1.10. Enhanced: Additional Information in SNMP Notifications
The resource ancestry, including the parent and grandparent resource names, are now included in SNMP alert notifications.
1.11. New: Send the Current JBoss ON Server State to the Logs
A new button was added to the System Settings area of the Administration tab. The button exports the current server configuration to the server logs.
2012-05-14 19:44:28,587 INFO [SystemInfoManager] SystemInformation: ******** CAM_LDAP_BIND_PW: [- non null -] AlertDefinitionCount: [2] CAM_LDAP_BASE_DN: [o=JBoss,c=US] AVAILABILITY_PURGE: [31536000000] CAM_JAAS_PROVIDER: [false] BuildNumber: [ca099bc:3a46aff] ServerCount: [27] DATABASE_DRIVER_NAME: [PostgreSQL Native Driver] RESOURCE_GENERIC_PROPERTIES_UPGRADE: [false] ... 8< ...
This is useful for debugging and for improving the server performance, as well as providing a way to verify that the server configuration was updated after synchronizing configuration between JBoss ON servers.
