4. Known Issues in JBoss Operations Network 3.1.2
This section contains a subset of the known issues at the time that JBoss Operations Network 3.1.2 was released. When possible, workarounds are provided.
4.1. Upgrade and Installation
When upgrading the server and installed JBoss plug-ins, there may be an error in the logs that one of the plug-in JAR files was corrupted:
2012-08-28 10:31:36,473 ERROR [org.rhq.enterprise.server.core.plugin.PluginDeploymentScanner] Scan failed. Cause: java.lang.Exception:File [/home/jon/jon-server-3.1.0.GA/jbossas/server/default/deploy/rhq.ear/rhq-downloads/rhq-plugins/hornetq-jopr-plugin-2.0.0.Final.jar] is not a valid jarfile - it is either corrupted or file has not been fully written yet.
This appears to happen if there is an EAP instance in the inventory and several different JBoss plug-ins are installed, though it may occur in other situations.
That error prevents the other updated plug-ins deployed in the JBoss ON server's
plugins/ directory from being picked up.
If that error occurs, simply restart the JBoss ON server again. All of the plug-ins will be successfully deployed after restarting another time.
If the installer is not bound to 0.0.0.0 when setting up a server, then it does not set all of the required connection properties. Specifically, the installer does not set the
java.rmi.server.hostname parameter to the real value, and it uses the default of 0.0.0.0. This parameter must be set to the real IP address of the server by manually editing the rhq-server.properties file.
Automatic upgrades of agents on Windows appear to fail. Two conditions must be true for automatic upgrade of agents to work on Windows:
- The agent must be installed and running as a Windows service.
- No Windows Explorer window can be open to any agent subdirectory. Windows sets a lock on directories when they are open, which blocks the upgrade process.
When starting a server while agents are running, the server may log servlet errors in the logs. This is because the remoting (communications) classes are loaded early in the startup sequence, before the server is completely started. This causes some communications interruptions until the server is completely started. These errors can be ignored.
If a JBoss ON server was added to the resource inventory, the old JBoss ON server resource must be deleted from the inventory after upgrade. The old resource entry is unusable after upgrade.
Null pointer exceptions for the
org.apache.catalina.connector.CoyoteAdapter service are returned when the JBoss ON 3.1.2 server is first installed. These errors are harmless and can be ignored. Installation will complete successfully, and both the server and the GUI will start and run properly.
The installer shows non-replaced {} properties.
4.2. Server
On Windows, the server can hang while shutting down. This can take several minutes to stop the server process, or the
java.exe process may need to be killed manually. If the process is stopped manually, then there is the risk of a memory leak since the process did not stop cleanly.
The password mechanism in the server installer doesn't properly hash Unicode characters. The Unicode characters are translated into the HTML entities, and then the HTML characters are hashed and stored as the password.
For example, the string
你好 is treated as the string 你好 in the installer, and the second string is the one that is hashed and stored in the rhq-server.properties file.
After configuring the server, but before logging in, manually update the password in the
rhq-server.properties file:
- Manually generate a hashed password based on the Unicode password:
serverRoot/jon-server-3.1.2.GA/bin/generate-db-password.sh 你好 Encoded password: 68f725778bb36d3b
- Replace the database password value with the new hashed password.
rhq.server.database.password=68f725778bb36d3b
4.3. Agents
The agent command prompt should accept the long name of arguments with an equals sign between the argument and given value. For example:
discovery --resourceType="JBossAS Server"However, the command fails with an equals sign and prints the help output for the agent command.
To work around this, use either the long name or short name of the command argument with a space between the argument and the given value. For example:
discovery --resourceType "JBossAS Server" discovery -r "JBossAS Server"
Attempting to use the
config import command in the agent command prompt fails with an InvalidPreferencesFormatException exception.
java.util.prefs.InvalidPreferencesFormatException: org.xml.sax.SAXParseException; lineNumber: 2; columnNumber: 41; Document root element "preferences", must match DOCTYPE root "null". ...
The JVM can sometimes cause the agent to crash when Augeas is running.
A fatal error has been detected by the Java Runtime Environment ...
The agent uses Sigar libraries by default, but Sigar is not compatible on all operating systems. To avoid using Sigar, the agent can be started with the
--nonative argument. However, the agent process on some systems — this has been observed on Suse 10.x — ignores the --nonative argument and attempts to start and load Sigar. Loading the Sigar libraries on an incompatible operating system causes the agent to fail to start, with this error:
A fatal error has been detected by the Java Runtime Environment...
4.4. Databases
The agent assumes that PostgreSQL uses a default password. If a different password is used, the PostgreSQL resource is shown as unavailable in the UI even if it is running.
4.5. Groups
A dynagroup definition with a short recalculation interval (less than two minutes) can cause duplicate groups to be created.
Changing the recursive setting on a group disassociates the group from any roles it was (previously) associated with.
4.6. Provisioning
Hot deployment fails if the file changes during provisioning.
If a bundle is being deployed in a hot deployment and one of the underlying bundle files, like a WAR or JAR, is changed while the bundle is being uploaded, then provisioning will fail.
The Java utilities library (
java.util.zip) included for the bundling system does not support symbolic links. Therefore, bundle recipes and configuration files cannot use symlinks.
Symlinks may be an issue for an application server (such as EAP or EPP) which was installed initially from an RPM and then compressed to be used in a bundle. The RPMs available from Red Hat Network contain operating system-specific symlinks which would cause failures during provisioning.
If it is necessary to reference shared libraries or other files (which would normally be referenced with a symlink), then include the required libraries in an archive with the bundle and use an Ant task to extract the files when deploying the bundle. For example:
<untar src="abc.tar.gz" compression="gzip" dest="somedirectory"/>
The
<rhq:system-service> Ant tag only works on Red Hat Enterprise Linux systems. Using the tag and then deploying the bundle to other operating systems causes the Ant task to fail to operate and can cause the deployment to fail.
4.7. Content and Repositories
A repository can only be used for one resource, the one it was initially defined for.
On Oracle databases, attempting to delete a package immediately after adding it to a repository fails.
Attempting to delete a repository with a space in the name throws an exception and the deletion fails with SQL errors. A SQLGrammarException is thrown in the UI:
Failed to delete packages: [I@7d7cec2f from repository: 10051 Error: javax.persistence.PersistenceException: org.hibernate.exception.SQLGrammarException: could not execute queryIn the server log, it shows a SQL error and the message ORA-00920:
2012-11-15 14:49:19,381 WARN [org.hibernate.util.JDBCExceptionReporter] SQL Error: 920, SQLState: 42000 2012-11-15 14:49:19,382 ERROR [org.hibernate.util.JDBCExceptionReporter] ORA-00920: invalid relational operator
Deploying WARs from a content repository is not supported. WARs and EARs should be deployed using bundles.
Deploying an exploded archive on JBoss EAP 4 results in deploying a zipped archive, which makes the deployed application unavailable.
4.8. Remote API and the CLI
JDK 7 includes a different version of Rhino, so that native arrays implement the Collection interfaces. This is different than how the version of Rhino in JDK 6 implemented native arrays.
One of the side effects of this is that the
assertEquals() method in the CLI does not properly handle native arrays (meaning, arrays created through a CLI script rather than provided by JBoss ON).
For example, this assert call works using JDK 6:
a = [] b = [] assertEquals(a, b)
However, it fails when using the CLI with JDK 7:
javax.script.ScriptException: sun.org.mozilla.javascript.internal.EvaluatorException: The choice of Java constructor assertEquals matching JavaScript argument types (object,object) is ambiguous; candidate constructors are:
void assertEquals(java.util.Collection,java.util.Collection)
void assertEquals(java.lang.Object[],java.lang.Object[]) (<Unknown source>#1) in <Unknown source> at line number 1
To work around this error, explicitly create a
java.util array for the assertEquals() call.
a = [] b = [] assertEquals(new java.util.ArrayList(a), new java.util.ArrayList(b))
ResourceFactoryManager does not throw any exceptions when attempting to create duplicate or invalid resources in the CLI.
The JBoss ON remote API cannot be used to connect to the JBoss ON server when the client is running inside an application server. For example, the remote API cannot be run from a client inside an EAP instance; it fails with errors like the following:
Caused by: java.lang.IllegalArgumentException: interface org.rhq.enterprise.server.auth.SubjectManagerRemote is not visible from class loader at java.lang.reflect.Proxy.getProxyClass(Proxy.java:353) at java.lang.reflect.Proxy.newProxyInstance(Proxy.java:581) at org.rhq.enterprise.client.RemoteClientProxy.getProcessor(RemoteClientProxy.java:69)
4.9. UI
Problems with Sessions and Multiple Tabs
If there are multiple tabs open to the JBoss ON UI and one of those sessions times out, then the other tabs will register errors. This can also occur if multiple people are logged into JBoss ON as the same user.
Searching for a resource with a backslash (\) in the name returned no results, even if the resource exists.
If a browser has an odd row height set, a narrow window width, or a large font size, then there can be problems rendering results in a list of resources. This can cause a paging exception to be returned, java.lang.IllegalArgumentException: PageList was passed an empty collection but 'totalSize' was 1.
If an operation is performed on a group with a relatively large number of resources (such as 20 or more), then the resource navigation tree on the left can freeze and become unresponsive when the operation is scheduled and the page refreshed.
Bugzilla 736732: Bookmarked UI Pages May Display the Old UI
The page names used for different pages in the JBoss ON UI for version 3.0 are different than the ones used in JBoss ON 2.4. However, because JBoss ON 3.0 retains some of the elements used in the older JSF-based UI, these pages still exist. This means that bookmarks will go to the old style pages.
For example, the Dashboard page in the old JSF UI had the URL http://localhost:7080/Dashboard.do/. In the new UI, the location is http://localhost:7080/coregui/#Dashboards.
All of the new pages have the
coregui/ element in them. Remove any old bookmarks and replace them with the new GWT-based pages in the coregui/ area.
When the results for a dynamic search are long, the list is truncated but no scrollbar is presented, so it is not possible to scroll to the end of the list.
4.10. Operations
The incorrect time unit is shown in the operation detail. For example, for an interval of 60 seconds, the UI displays 60000 seconds. In actuality, it means 60000 milliseconds. This is only a problem with the display; the actual schedule sets a 60 second interval and runs on time.
Attempting to run an operation on a resource without parameters immediately after running an operation with parameters can fail with an error that the second operation did not have the required parameters set, even though there are no required parameters.
4.11. Measurements and Monitoring
A user-defined baseline value for a metric is ignored and overwritten by the server-calculated baseline.
4.12. Alerts
In some rare cases, if an alert notification fails to send, then the alert itself is not written to the alert history.
The default SNMP configuration for JBoss ON claims to use port 162 for SNMP traps. However, it actually sends SNMP trap notifications over port 161. If the port value is manually set to 162, then the notification is properly sent over port 162.
Attempting to upload a CLI script directly to a repository to use for alerting fails because it says the SHA-256 checksum for the script is not an OSGi string. Uploading a CLI script through the alert configuration UI works successfully.
Alerts with compound AND conditions can incorrectly fire when one of the conditions goes from true to false within 30 seconds of the other condition going from false to true.
4.13. Drift
Drift definitions can be created for resources which do not support drift, using the remote API to create a script.
4.14. Inventory and General Resource Issues
If a new resource is imported with an invalid configuration that prevents the resource from being started, all resources within the agent's inventory are marked as obsolete. This causes the agent to purge its entire inventory and attempt to re-sync with the inventory stored on the server. This purge and re-sync occurs for every resource with an invalid configuration. This is time-consuming and CPU-intensive on the local machine.
If a server is imported manually through a platform or parent Inventory tab, it is possible that that resource will also be detected and listed in the discovery queue. Manually importing a resource doesn't remove it from the discovery queue. It is possible, then, for the same resource to be imported twice.
4.15. Platform Resources
The inventory for a Linux platform may discover non-existent network adapters, which are added and then shown as unavailable in the inventory.
4.16. Tomcat Resources
If the type is changed on a Tomcat connector, then the original resource stays in the inventory as an unavailable resource and the new resource is never discovered. For example, if you change a JK connector to an AJP connector, the JK connector remains in the inventory but appears to be inactive, while the AJP connector is never added to the inventory.
Because both connectors connect over the same host and port, the discovery process assumes that the connectors are the same resource, so it never "rediscovers" the new resource. But since the type/connection protocol is different, it also never connects to the old connector.
To work around this, manually remove the old connector resource from the entry. The new, replacement connector is then discovered and imported automatically.
The Store Configuration operation is supposed to persiste the current or given MBean settings to the Tomcat server's
server.xml file, which remain in effect until the Tomcat instance is restarted. However, the Store Configuration operation is not writing any changes to the server.xml file, so the configuration is not changed by the operation.
Manually importing a Tomcat instance through the platform resource (right-clicking the platform and selecting Inventory > Import > Tomcat) fails with authentication errors, even if proper credentials are given for the Tomcat user.
One potential problem could be if the agent and the Tomcat server are running as different system users. The agent and Tomcat server must run as the same system user for the agent to be able to manage the Tomcat server.
The JBoss ON agent does not detect the Tomcat server if it is running as a Windows service.
It is possible to edit the configuration of a Tomcat web application without ever setting a value for the configuration file name and path. The configuration file is a required parameter.
When the
mod_cluster resource is imported for a Tomcat server, the mod_cluster configuration file is not set in the Inventory > Connection Settings area, leading to a connection failure. Even if the file is manually set to the Tomcat server.xml, the configuration file property is not saved and the connection error persists.
When attempting to edit the Tomcat user database by adding users or groups, the UI reports that the operation completed successfully. However, in reality, the operation failed, with this message written to the Tomcat server logs:
org.apache.catalina.users.MemoryUserDatabase save SEVERE: User database has been configured to be read only. Changes cannot be saved
Attempting to run an operation which requires parameters, such as creating a group or a user, fails with the exception java.lang.IllegalStateException: Name [roles] specified for parameter of operation [createGroup] on MBean [Users:type=UserDatabase,database=UserDatabase] is invalid. The MBean operation does not take a parameter by that name.
If a WAR file is configured to be deployed to a non-standard location on a Tomcat application server, an invalid deployment name is added to the inventory. The
unpackWARs option extracts the web app to a second, invalid directory. For example, if it should be extracted to /myapps/helloworld, the web app is extracted both there and to /myapps#helloworld.
The agent cannot discover a Tomcat instance if its Java process is running as a user other than the system user that the agent is running as.
4.17. Apache Resources
Response time metrics are only collected if SNMP is being used with the Apache server. fails with this error:
An Augeas lens is required to configure Apache servers through JBoss ON, but it is not required for monitoring and other operations. If Augeas is disabled (which may be done for performance improvements), the agent reports an error that configuration is disabled. This error message is superfluous and can be ignored.
Editing any option in the directory inside an Apache virtual host fails with this error:
2012-08-07 15:32:10,835 ERROR [ResourceContainer.invoker.nonDaemon-4] (org.rhq.plugins.apache.ApacheDirectoryComponent)- Augeas failed to save configuration
When trying to add a second entry for the Allow Override directive in an Apache directory resource, deselecting the Unset checkbox for the Param field returns a globally uncaught exception.
Apache directives are case-insensitive, but the Apache plug-in treats them as if they were case-sensitive. This makes the plug-in disregard directives with non-default case.
If the
/tmp directory is configured a noexec in the fstab file, the agent throws exceptions because it cannot properly initialize the Augeas lens. In that case, the Configuration tab is unavailable for the Apache resource.
To work around this, make sure that the
/tmp directory does not have noexec set as an option.
# # /etc/fstab # tmpfs /dev/shm tmpfs defaults 0 0 devpts /dev/pts devpts gid=5,mode=620 0 0 sysfs /sys sysfs defaults 0 0 proc /proc proc defaults 0 0
noexec is one security method available to that directory to prevent unauthorized or malicious scripts from running. Disabling noexec for the directory is not ideal, but it is also not a security issue by itself because other filesystem security settings can be used.
4.18. JBoss EAP 6 Resources
For different EAP 6 child resources, the Unset? option does not work to unset the configuration property. The Unset? option should allow either the default value or an empty value to be used for a property. Rather, the previous property values are used.
This only applies to optional attributes; required attributes must have an explicitly-set value.
The different cluster operations for
mod_cluster — enable, disable, stop — do not work on all cluster members. Rather, the operation runs properly on a single web context and then is never executed on the others. The problem comes because the mod_cluster plug-in requires a unique JVM route identifier for each web context, but that identifier is not properly applied.
When a new datasource is created, the Enabled setting is ignored, so that a new datasource is always created in a disabled state. The resource must be manually enabled after it is created.
When creating a datasource through the JBoss ON CLI, it is possible to create a resource with invalid configuration. Although the script reports a failure and throws a null-pointer exception, the datasource is, in fact, created and discovered by JBoss ON.
If an EAP 6 instance is installed by an RPM, then the EAP instance user belongs to a different group (by default) than the JBoss ON user. If the JBoss ON agent is not run as root, then the JBoss ON agent cannot connect to the EAP instance and, therefore, cannot discover the instance.
When content is uploaded to the server group, the content version history is successfully updated. However, when that deployment is then propagated to the managed servers within the server group, the content history is not updated, leaving the previous SHA and version in the history.
If a host controller is not named master, then when the HostControllerComponent resource is initialized by the plug-in container, the agent cannot connect to the host controller. The agent expects a name of master, always, and if the controller has a different name, the agent returns the error (rhq.modules.plugins.jbossas7.BaseServerComponent)- Failed to validate base dir for {JBossAS7}JBossAS7 Host Controller.
A restart operation can fail when using a system init script or a custom start script to restart the EAP server. The BaseServerComponent.waitUntilDown() method only checks to see if the management interface is unavailable before attempting to run the start method. However, the management interface is stopped before the server itself is stopped, so there may still be a run lock or PID file that prevents the start operation from succeeding.
A start operation incorrectly reports that it failed when it in fact succeeded, while a blocked or timed out process returns a success message. This can happen with a custom script or a system init script. The JBoss ON agent incorrectly interprets a null return (blocked or timed out) as success and a zero (0) status (success) as a failure.
The first time that a reload is run on an EAP 6 resource after the resource is imported into the inventory, an exception is thrown in the UI.
The reload operation completes successfully and subsequent reload operations do not throw an exception.
When an EAP 6 server is running in domain mode with the full-ha profile, unsetting the proxy URL to save a null value fails with the error We got a 500 with the following response body back. Setting the proxy URL to a different value (not null) is successful.
When running a group of operations in a specific order on a server group, attempting to open the operations schedule returns a null pointer exception.
When editing a network interface for a JBoss EAP 6 server, the network configuration requires either a specific IP address or a setting to use any IP address for IPv4, for IPv6, or any address. However, the UI does not enforce that an IP address selection is made, which can cause the interface to be unreachable.
Attempting to deploy a child resource such as a WAR application fails with a malformed URL exception if the EAP 6 server is listening on IPv6.
Some path elements defined in the configuration of a domain controller or standalone server are computed by the server itself. These computed paths are all prefixed with
java.*, jboss.*, and user.*.
While the configuration values are viewable, they are not editable. However, the Configuration tab for the server shows both edit and delete options for the path elements. If a user attempts to change or delete the path, the path simply reverts to its previous state.
When a child JVM definition is created for a managed resource, the name of the JVM definition must be the same as the host JVM definition which it is based on. However, the JBoss ON UI creation wizard does not detect or enforce a pre-computed name, so it allows a user to enter any name.
The UI will not let users enter a driver name when manually adding a datasource, so it is not possible to deploy a module-loaded driver.
The agent properly detects an EAP 6 server running over an IPv6 address, but it incorrectly reports it as offline.
4.19. JBoss BRMS Resources
BRMS has some attributes defined for each knowledge base, but these parameters are not listed in the resource monitoring tab. This is because the knowledge base parameters are defined in a dynamic MBean, and JBoss ON does not automatically detect dynamic MBeans.
To work around this, run the Start Internal Beans operation (in JBoss ON or the JConsole), and view the knowledge base parameters in JConsole.
BRMS has some attributes defined for each knowledge session, but these parameters are not listed in the resource monitoring tab.
The list of active rules for a knowledge session, and their state and parameters, are not displayed in the resource monitoring tab.
The list of rule flows for a knowledge session, and their parameters, are not displayed in the resource monitoring tab.
Running an operation which requires parameters on a knowledge session resource throws an IllegalStateException error, stating that the operation does not take a parameter of that name.
To work around this, run the operation in JConsole.
4.20. JBoss Data Grid Resources
None of the cache statistics for EDG resources are displayed in JBoss ON. The default, memcahedCache, and namedCache resources all throw the error java.util.MissingResourceException: Can't find resource for bundle java.util.PropertyResourceBundle, key infinispan.distributed-cache.cache-loader-stores.
4.21. SOA-P Resources
When editing a local Tx datasource in the JBoss ON UI, the UI forces the user to unset and then select the security deployment type radio button, even though the default value is (acceptably) None. Even if the None radio button is selected, the value is reset to some other value and improperly saved in the corresponding
*-ds.xml file.
Attempting to deploy an ESB archive to a SOA-P resource can throw java.lang.reflect.UndeclaredThrowableException if it attempts to use the RemoteDeployer.
Attempting to deploy an ESB archive to a SOA-P resource can throw java.io.FileNotFoundException.
When deploying either an ESB or WAR archive to a SOA-P resource, in some instances the archive is not successfully deployed. The deployment operation can create an empty directory with the archive name and MANIFEST.MF file, but no application.
4.22. JBoss EAP 4 and 5 Resources
Shutdown operations have a maximum wait time, which defines how long the operation can run before it times out. If two EAP 5 instances are shut down at the same time, then the max wait time setting on the second instance overrides the max wait time setting for the first instance which is shut down.
The agent fails to collect metrics for the web application context. When a sessions-relate metric is enabled, such as active sessions, the agent throws a null pointer exception with the message Failed to collect metric 'activeSessions' for {JBossAS5}Web Application Context Resource with key localhost.
A patch for this issue is available from Red Hat Global Support Services.
When editing a datasource in the JBoss ON UI, the UI forces the user to unset and then select the security deployment type radio button, even though the default value is (acceptably) None. Even if the None radio button is selected, the value is reset to some other value and improperly saved in the corresponding
*-ds.xml file.
Changing an EAR or WAR deployment from exploded to archive or vice versa makes it so that the resource is always listed as unavailable.
Changing the name of a JNDI resource results in the renamed JNDI resource being listed as a new resource, while another entry with the original name remains in the inventory in an unavailable state. To resolve this, run a manual discovery scan, which resets the original entry with the new name.
If a datasource, JMS queue, or JMS topic resource is created and then deleted, a new datasource, JMS, topic, or JMS queue resource cannot be created with the same resource name. The XML file in the JBoss server's deployment directory, with that original name, is left behind after the delete operation, and that prevents any other resource from being created with that name.
JBoss ON will not use a custom binding set with a JBoss EAP 5 server if the server's configuration is changed after it is imported. The JBoss ON server persists in using the default binding set.
When creating a new datasource, JBoss ON names the resource according to its JNDI name, not the resource name entered in the creation wizard.
If EAR, WAR, or script resources are deleted outside of JBoss ON, the agent throws an exception during discovery, and the server is not informed that the resource was deleted.
Discovering a JBoss AS instance fails if the configuration directory is a URL.
The discovery process fails to determine the JNP URL if the data directory is not located at ${jboss.server.home.dir}/data.
Shutting down a JBoss AS 5 resource can fail with java.rmi.NoSuchObjectException: no such object in table.
The JBoss AS plug-in is unable to load resource configuration for JMS queues that are deployed using a SAR file.
4.23. JVM Resources
On Windows, the JBoss ON server cannot discover a JVM resource is there is a space in the JAVA_HOME path. To work around this, use the shortname (with the tilde, ~) for the path.
The discovery process attempts to discover an embedded JVM process through its JMX remoting rather than using the EMSConnection of its parent Tomcat or JBoss resource.
4.24. Performance
Attempting to update the members in a compatible group with thousands of members times out and fails with 'Failed to fetch Resource Group.'
Viewing configuration, alerts, or other data for groups with more than 1000 members can fail with the SQL exception maximum number of expressions in a list is 1000.
4.25. Plug-in Writing
If an embedded plug-in extension is defined and the parent resource type has subcategories, those subcategories cannot be changed or it causes a database deadlock.
