4. Server
- Q: When I start the server, I see servlet errors in my logs. What's wrong?
- Q: How do I get debug messages from the JBoss ON server?
- Q: How can I specify command-line options for the server JVM?
- Q: How do I purge my schema of all data?
- Q: How can I debug JDBC access and trace SQL?
- Q: How can I confirm my server's email/SMTP settings are correct?
- Q: My server machine does not have a writable directory called /var/run. How can I get my rhq-server.sh script to successfully write out its pid file?
- Q: When I try to start the server, I get an exception with the cause "Exception creating identity" and the server fails to start. How can I fix this?
- Q: My server logs are showing the message "Have not heard from agent ... Will be backfilled since we suspect it is down." What does that mean?
- Q: What ports do I have to be concerned about when setting up a firewall between servers and agents?
- Q: I installed the server as a Windows service, but it is failing to start with no error messages. How can I start the server as a Windows service?
- Q: How do I fix an ORA-12519, TNS:no appropriate service handler found error when using Oracle XE?
- Q: I am seeing this error in my server logs or stack trace: WARN [QueryTranslatorImpl] firstResult/maxResults specified with collection fetch; applying in memory. What does that mean and what is causing it?
- Q: How do I stop the server from periodically logging messages that say a plug-in is "the same logical plug-in" but has "different content" and "will be considered obsolete"?
- Q: What is the difference between LDAP user authentication and LDAP group authorization in JBoss ON?
- Q: How do I set up LDAP group authorization?
Q:
When I start the server, I see servlet errors in my logs. What's wrong?
A:
As the server starts and if agents are already running, there can be errors related to the
Servlet.service() class recorded in the logs:
22:55:35,319 ERROR [[ServerInvokerServlet]] Servlet.service() for servlet
ServerInvokerServlet threw exception
java.lang.reflect.UndeclaredThrowableException
at $Proxy421.processRequest(Unknown Source)
at
org.jboss.remoting.transport.servlet.web.ServerInvokerServlet.processRequest(ServerInvokerServlet.java:128)
at
org.jboss.remoting.transport.servlet.web.ServerInvokerServlet.doPost(ServerInvokerServlet.java:157)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
... more ...
This error is normal and is related to the sequence that the server loads its classes when it starts. The remoting classes are loaded early in the startup sequence, which means the server begins attempting to contact agents before it is fully started, and this can cause the errors recorded in the logs. These errors should go away once the server is completely started.
The errors can be safely ignored.
Q:
How do I get debug messages from the JBoss ON server?
A:
You can edit the
serverRoot/jon-server-3.1.0.GA1/jbossas/server/default/conf/jboss-log4j.xml configuration file to enable debug messages by uncommenting the org.rhq category. This will set its priority to DEBUG. Debug messages will now be emitted for all JBoss ON subsystems to the log file. If you want debug messages to be emitted only for a smaller subset of the JBoss ON server internals, you can specify which categories you want by uncommenting them, or alternatively, you can add your own categories.
There are several commented-out categories in
log4j.xml with comments that briefly explain what types of debug messages can be expected from a particular category. You can also emit debug messages for third-party subsystems like JBoss/Remoting and Hibernate. Some of these are already commented out in log4j.xml.
After you make your changes to the
log4j.xml file, save the file and then restart the JBoss ON server.
serverRoot/jon-server-3.1.0.GA1/bin/rhq-server.sh|bat stop serverRoot/jon-server-3.1.0.GA1/bin/rhq-server.sh|bat start
Debug messages are in the log file,
serverRoot/jon-server-3.1.0.GA1/logs/rhq-server-log4j.log.
Note
By default, the console window will not display the debug messages. This is because the log4j CONSOLE appender has a threshold at
INFO. If you want your debug messages to also appear on the console, you must change the CONSOLE appender's threshold setting to DEBUG.
In some cases, you will want debug messages from the JBoss ON server launcher scripts. To do this, you need to set the environment variable
RHQ_SERVER_DEBUG to any value. After setting this variable when you start the launcher, scripts will output debug messages.
Q:
How can I specify command-line options for the server JVM?
A:
On Red Hat Enterprise Linux, override the default max heap and permgen sizes, set them via the
RHQ_SERVER_JAVA_OPTS environment variable. For example:
RHQ_SERVER_JAVA_OPTS="-Dapp.name=rhq-server -Xms256M -Xmx1024M -XX:PermSize=128M -XX:MaxPermSize=256M -Djava.net.preferIPv4Stack= true" export RHQ_SERVER_JAVA_OPTS
Set all other JVM options via the
RHQ_SERVER_ADDITIONAL_JAVA_OPTS environment variable. For example:
RHQ_SERVER_ADDITIONAL_JAVA_OPTS= "-Dfoo= true" export RHQ_SERVER_ADDITIONAL_JAVA_OPTS
On Windows, for all other JVM options, add
wrapper.java.additional.n lines to <server-install-dir>\bin\wrapper\rhq-server-wrapper.inc (you may need to create the file). For example:
- wrapper.java.additional.12=-verbosegc:file=gc-log.txt
- wrapper.java.additional.13=-XX:+HeapDumpOnOutOfMemoryError
- wrapper.java.additional.14=-XX:HeapDumpPath=heap-dump.txt
Q:
How do I purge my schema of all data?
A:
There are instances where it's necessary to completely purge the database schema of all data. This is helpful when writing custom plug-ins and a lot of the resource hierarchy information and metadata needs to be replaced. To delete all the data from the database but keep the schema intact, simply re-install the server:
- Save the current JBoss ON server directory.
mv jon-server-3.1.0.GA1/ jon-server-3.1.0.GA1.bak/
- Unzip the latest JBoss ON binaries.
unzip jon-server-3.1.0.GA1.zip
- Start the new server process.
serverRoot/jon-server-3.1.0.GA1/bin/rhq-server.sh start
- Open the JBoss ON GUI and go through the installation setup. When given the choice, select the option to Overwrite existing data. This removes all of the data for the previous installation of the server.
Q:
How can I debug JDBC access and trace SQL?
A:
You can debug JDBC and access and trace SQL using log4jdbc.
Q:
How can I confirm my server's email/SMTP settings are correct?
A:
To check that the server can send emails successfully, log into the GUI as the
rhqadmin user and open the email test page:
http://server.example.com/admin/test/email.jsp
Q:
My server machine does not have a writable directory called /var/run. How can I get my rhq-server.sh script to successfully write out its pid file?
A:
Set the environment variable
RHQ_SERVER_PIDFILE_DIR to the full path of the directory where you want the pid file to be stored. When you run the script, that variable's value will override the default location. If you have a script that is 2.1 or older, directly edit rhq-server.sh and change /var/run to the desired directory.
Q:
When I try to start the server, I get an exception with the cause "Exception creating identity" and the server fails to start. How can I fix this?
A:
The message you are referring to probably looks similar to this:
Caused by: java.lang.RuntimeException: Exception creating identity: my.host.name.com: my.host.name.com | at org.jboss.remoting.ident.Identity.get(Identity.java:211)
This is not JBoss ON-specific. It is caused by a failure with JBoss/Remoting communications. The core issue is typically because your hostname is not resolvable. The issue is normally hidden from you because JBoss/Remoting isn't producing the real error message. This error normally indicates that a machine's hostname is not externally resolvable. In order for JBoss ON to work correctly, all servers and agents must be able to resolve each other's hostnames. Best practice is to maintain a mapping of all servers and agents by using host files (e.g.
/etc/hosts). This will ensure that JBoss ON will continue to work correctly even if DNS fails. However, using host files may not be practical for your environment. If this is the case, please take some time before you begin your JBoss ON installation to verify that each host you plan to run JBoss ON on can correctly resolve every other hostname in your planned environment using a tool such as nslookup.
Note
This applies even if you use IP addresses exclusively for configured values, as the server and agents perform host lookups for certain functions.
Q:
My server logs are showing the message "Have not heard from agent ... Will be backfilled since we suspect it is down." What does that mean?
A:
[org.rhq.enterprise.server.core.AgentManagerBean] Have not heard from agent [agent_name] since [timestamp]. Will be backfilled since we suspect it is down
This means that the agent did not send its availability report in the required amount of time. The default is 2 minutes, but you can configure this on the Administration > System Configuration > Settings page. When the availability report is not sent in the required amount of time, the server assumes the agent is down. At this time it back-fills the availability of all resources managed by that agent to DOWN and the resource availabilities turn red.
This can happen for a number of reasons:
- The agent actually shut down or crashed.
- The machine the agent is running on shut down or crashed.
- The network between the agent and server went down, prohibiting the agent from connecting to the server and sending the availability report.
- The machine the agent is running on is bogged down, thus slowing up the agent and prohibiting the agent from being able to send up reports fast enough.
Q:
What ports do I have to be concerned about when setting up a firewall between servers and agents?
A:
Note
These are the default values. Different values can be configured for JBoss ON servers or agents when they are installed.
The default server ports are 7080 (standard) and 7443 (secure SSL).
The default agent port is 16163 for both standard and secure connections.
The server also has to communicate with its database. The default port depends on the type of database.
Q:
I installed the server as a Windows service, but it is failing to start with no error messages. How can I start the server as a Windows service?
A:
You probably installed the server to run as the local system account and that account probably doesn't have the proper permissions to run the server or machine has been locked down due to security concerns and that local system account cannot access the network or run Java.
To solve this, create a user on your Windows box that can run the server properly. To test the user permissions, log in as the user and execute
rhq-server.bat console to see if it can be run by that user. Then, install the server as a Windows Service with the RHQ_SERVER_RUN_AS_ME environment variable set to true:
rhq-server.bat remove set RHQ_SERVER_RUN_AS_ME=true rhq-server.bat install
Q:
How do I fix an ORA-12519, TNS:no appropriate service handler found error when using Oracle XE?
A:
Although Oracle XE is not supported for production environments, some places use it for test or development environments. To stop the ORA-12519 error, set this setting:
ALTER SYSTEM SET PROCESSES=150 SCOPE=SPFILE;Then restart the Oracle XE database.
Q:
I am seeing this error in my server logs or stack trace: WARN [QueryTranslatorImpl] firstResult/maxResults specified with collection fetch; applying in memory. What does that mean and what is causing it?
A:
This error is issued by the Hibernate service and can be triggered for a number of different reasons. This error can be ignored.
Q:
How do I stop the server from periodically logging messages that say a plug-in is "the same logical plug-in" but has "different content" and "will be considered obsolete"?
A:
This is a known issue, in Bugzilla 676073. To work around it, shutdown the server, remove the plug-in jars from the server's filesystem, and restart the server.
Q:
What is the difference between LDAP user authentication and LDAP group authorization in JBoss ON?
A:
Authentication is the process that is used to verify that an entity attempting to access a resource is the identity that it is claiming to be. Authorization is the process of determining what rights an entity has to access a resource after its identity has been established. Let's say that a user named
jsmith is attempting to log into JBoss ON. Authentication is the process of checking that the jsmith trying to log in is the same as the jsmith user that JBoss ON has in its database; this can be validated by verifying the password. Once jsmith logs in, then JBoss ON determines what resources jsmith can view and whether he can edit those resources' configuration, provision new applications, change server settings, and perform other tasks in JBoss ON.
Typically, JBoss ON uses its own user database to identify and authenticate users. It is possible to enable LDAP authentication by letting JBoss ON check an LDAP server for user information first, and using that base of users for valid JBoss ON users. This is LDAP authentication. This is essentially pass-through authentication. A user attempts to log into JBoss ON. JBoss ON first sends the credentials to the LDAP server to see if the LDAP server has stored that user; if authentication fails at the LDAP server, then JBoss ON checks its own database.
All authorization in JBoss ON is based on roles. Both users and resource groups are added to roles, and then permissions are assigned to those roles. These roles are created and managed in JBoss ON, but it is possible to use group membership in an LDAP group to supply the users in JBoss ON role. Essentially, this takes an existing list of users in LDAP and just says, "use this list for the role members." The LDAP group is added to a JBoss ON role, and then every member in that LDAP group automatically has whatever rights the role has. That is LDAP authorization.
Note
LDAP authentication is recommended for LDAP authorization, but it is not required.
This covered in more detail in "Integrating LDAP Services for Authentication and Authorization".
Q:
How do I set up LDAP group authorization?
A:
LDAP authorization is set up in the Administration tab, under System Settings.
First set up JBoss ON to allow LDAP users to authenticate using LDAP user accounts ("Configuring LDAP User Authentication"). (LDAP authentication isn't required, but it is recommended.) Then, configure JBoss ON to check for LDAP groups on the LDAP server ("Associating LDAP User Groups to Roles in JBoss ON").
There are five elements in the LDAP server configuration that you need to know to configure LDAP group authorization:
- The information to connect to the LDAP server, in the form of an LDAP URL. For example, ldap://server.example.com:1389.
- The username and password to use to connect to the server. This account should have read access to the subtrees being searched.
- The search base. This is the point in the directory tree to begin looking for entries. This should be high enough to include all entries that you want to include and low enough to improve performance and prevent unwanted access. For example, if you have
ou=Web Team,dc=example,dc=comandou=Engineering,dc=example,dc=comand you want to include groups in both subtrees in JBoss ON, then set the search base high up the tree, todc=example,dc=com. If you only want the engineering groups to be used by JBoss ON, then set the search base toou=Engineering,dc=example,dc=com. - The group filter. This creates the search filter to use to search for group entries. This can use the group object class, which is particularly useful if there is a custom attribute for JBoss ON-related entries. This can also point to other elements — like the group name, a locality, or a string in the entry description — that are useful or meaningful to identify JBoss ON-related groups.
- The member attribute. There are different types of group object classes, and most use different attributes to identify group members. For example, the
groupOfUniqueNamesobject classes lists its members with theuniqueMemberattribute.
After LDAP authorization is enabled, then you can associate the roles in JBoss ON to the appropriate groups in the LDAP directory.
