24.3. Deploying and Managing Storage Nodes

As touched on in Section 16.3, “Potential Impact on Server Performance”, collecting metrics and generating alerts can be resource-intensive. It results in near-constant write operations on the backend database. That creates a natural threshold for the number of metrics that can be collected (30,000 per minute) before encountering performance degradation.

JBoss ON uses two databases to store its information. One is a central relational database (PostgreSQL or Oracle) which stores all configuration about the JBoss ON servers and agents, all resource inventory data, resource configuration, and other data. The other database is a distributed database (a cluster of storage nodes) which stores all numeric monitoring data — in other words, all collected metrics.

The metrics storage node can be installed on its own dedicated machine, which can significantly improve write performance to the database (and, therefore, improve monitoring performance):

These are the same performance considerations as installing the relational database on a separate machine from the JBoss ON server. By using two databases, it is also possible to move write-intensive and resource-intensive metrics storage away from resource-intensive configuration data, such as drift snapshots and bundle configuration.

Additionally, the distributed database can be expanded, with multiple nodes in a cluster. This ability to add additional nodes according to load is a crucial management tool for administrators. Rather than encountering that hardware-driven limit of 30,000 metrics collected per minute, additional nodes can be added to improve performance.

The storage node cluster is created and managed by JBoss ON (which also minimizes the metrics storage node management overhead). A storage node is installed on a system, using the rhqctl install --storage command. The storage node always requires a companion agent.

At least one storage node must be created and managed by JBoss ON (which minimizes the metrics storage node management overhead since no external tools are required).

There are several paths of communication to handle metrics data, all working in parallel.

Node cluster communication requires three elements:

The metrics storage provides data availability and integrity by backing up the data in two ways:

There can be multiple metrics storage databases in the JBoss ON environment. Much like the JBoss ON servers can operate in a cloud, multiple storage nodes communicate with each other to function in a cluster.

Nodes can be added and removed from the cluster by administrators or dynamically using JBoss ON scripts in response to changes in the environment.

With the default cluster configuration, a node is deployed as soon as it is installed and configured in the JBoss ON server. These are actually two separate steps.

Whether to deploy nodes automatically or manually can be determined by the provisioning and monitoring requirements of the environment.

[root@server ~]# serverRoot/jon-server-3.2.GA1/bin/rhqctl install --storage --agent-preference="rhq.agent.server.bind-address=0.0.0.0"

// deploy a storage node 
nodes = StorageNodeManager.findStorageNodesByCriteria(StorageNodeCriteria()); 
node = nodes.get(0); StorageNodeManager.deployStorageNode(node);

// undeploy a storage node 
nodes = StorageNodeManager.findStorageNodesByCriteria(StorageNodeCriteria()); 
node = nodes.get(0); StorageNodeManager.undeployStorageNode(node);