Chapter 17. Configuring subsystem logs

All subsystems keep an install log.

Every time a subsystem is created either through the initial installation or creating additional instances with pkispawn, an installation file with the complete debug output from the installation, including any errors and, if the installation is successful, the URL and PIN to the configuration interface for the instance. The file is created in the /var/log/pki/ directory for the instance with a name in the form pki-subsystem_name-spawn.timestamp.log.

Each line in the install log follows a step in the installation process.

...
2015-07-22 20:43:13 pkispawn    : INFO     ... finalizing 'pki.server.deployment.scriptlets.finalization'
2015-07-22 20:43:13 pkispawn    : INFO     ....... cp -p /etc/sysconfig/pki/tomcat/pki-tomcat/ca/deployment.cfg /var/log/pki/pki-tomcat/ca/archive/spawn_deployment.cfg.20150722204136
2015-07-22 20:43:13 pkispawn    : DEBUG    ........... chmod 660 /var/log/pki/pki-tomcat/ca/archive/spawn_deployment.cfg.20150722204136
2015-07-22 20:43:13 pkispawn    : DEBUG    ........... chown 26445:26445 /var/log/pki/pki-tomcat/ca/archive/spawn_deployment.cfg.20150722204136
2015-07-22 20:43:13 pkispawn    : INFO     ....... generating manifest file called '/etc/sysconfig/pki/tomcat/pki-tomcat/ca/manifest'
2015-07-22 20:43:13 pkispawn    : INFO     ....... cp -p /etc/sysconfig/pki/tomcat/pki-tomcat/ca/manifest /var/log/pki/pki-tomcat/ca/archive/spawn_manifest.20150722204136
2015-07-22 20:43:13 pkispawn    : DEBUG    ........... chmod 660 /var/log/pki/pki-tomcat/ca/archive/spawn_manifest.20150722204136
2015-07-22 20:43:13 pkispawn    : DEBUG    ........... chown 26445:26445 /var/log/pki/pki-tomcat/ca/archive/spawn_manifest.20150722204136
2015-07-22 20:43:13 pkispawn    : INFO     ....... executing 'systemctl enable pki-tomcatd.target'
2015-07-22 20:43:13 pkispawn    : INFO     ....... executing 'systemctl daemon-reload'
2015-07-22 20:43:13 pkispawn    : INFO     ....... executing 'systemctl restart pki-tomcatd@pki-tomcat.service'
2015-07-22 20:43:14 pkispawn    : INFO     END spawning subsystem 'CA' of instance 'pki-tomcat'
2015-07-22 20:43:14 pkispawn    : DEBUG

The CA, KRA, OCSP, TKS, and TPS subsystems use a Tomcat web server instance for their agent and end-entities' interfaces.

Error and access logs are created by the Tomcat web server, which are installed with the Certificate System and provide HTTP services. The error log contains the HTTP error messages the server has encountered. The access log lists access activity through the HTTP interface.

Logs created by Tomcat:

These logs are not available or configurable within the Certificate System; they are only configurable within Apache or Tomcat. See the Apache documentation for information about configuring these logs.

The self-tests log records information obtained during the self-tests run when the server starts or when the self-tests are manually run. The tests can be viewed by opening this log. This log is not configurable through the Console, it can only be configured by changing settings in the CS.cfg file. For instruction on how to configure logs by editing the CS.cfg file, see the Enabling the Publishing Queue section in the Red Hat Certificate System Planning, Installation, and Deployment Guide.

The information about logs in this section does not pertain to this log. See Section 15.8, “Running self-tests” for more information about self-tests.

All major components and protocols of Certificate System log messages to log files. The following table lists services that are logged by default. To view messages logged by a specific service, customize log settings accordingly. For details, see Section 17.3.1, “Viewing logs in the console”.

The different events logged by Certificate System services are determined by the log levels, which makes identifying and filtering events simpler. The different Certificate System log levels are listed in the following table.

Log levels are represented by numbers indicating how detailed the level of logging to be performed by the server should be.

A higher priority level means less detail because only events of high priority are logged.

Log levels can be used to filter log entries based on the severity of an event. The default log level is 10.

Log data can be extensive, especially at lower (more verbose) logging levels. Make sure that the host machine has sufficient disk space for all the log files. It is also important to define the logging level, log rotation, and server-backup policies appropriately so that all the log files are backed up and the host system does not get overloaded; otherwise, information can be lost.

The Java subsystems support buffered logging for all types of logs. The server can be configured for either buffered or unbuffered logging.

If buffered logging is configured, the server creates buffers for the corresponding logs and holds the messages in the buffers for as long as possible. The server flushes out the messages to the log files only when one of the following conditions occurs:

If the server is configured for unbuffered logging, the server flushes out messages as they are generated to the log files. Because the server performs an I/O operation (writing to the log file) each time a message is generated, configuring the server for unbuffered logging decreases performance.

Setting log parameters is described in Section 17.2.2, “Configuring logs in the console”.

The subsystem logs have an optional log setting that allows them to be rotated and start a new log file instead of letting log files grow indefinitely. Log files are rotated when either of the following occur:

When a log file is rotated, the old file is named using the name of the file with an appended time stamp. The appended time stamp is an integer that indicates the date and time the corresponding active log file was rotated. The date and time have the forms YYYYMMDD (year, month, day) and HHMMSS (hour, minute, second).

Log files, especially the audit log file, contain critical information. These files should be periodically archived to some backup medium by copying the entire Java`log` directory to an archive medium.

The Certificate System provides a command-line utility, signtool, that signs log files before archiving them as a means of tamper detection. For details, see Section 17.2.4.5, “Signing log files”.

Signing log files is an alternative to the signed audit logs feature. Signed audit logs create audit logs that are automatically signed with a subsystem signing certificate. See Section 17.2.4.3, “Configuring a signed audit log in the console” for details about signed audit logs.

Rotated log files are not deleted.

For a list of audit events in Certificate System, see Appendix E, Audit events.

Signed audit logs can be enabled by default when an instance is first created by using the pki_audit_group deployment parameter with the pkispawn command. If, however, signed audit logs were not configured when an instance was created, they can be enabled afterwards by reassigning ownership of the audit log directory to the auditor system users group, such as pkiaudit.

Signed audit logs are configured by default when the instance is first created, but it is possible to edit the configuration or change the signing certificates after configuration.

A log is set to a signed audit log by setting the logSigning parameter to enable and providing the nickname of the certificate used to sign the log. A special log signing certificate is created when the subsystems are first configured.

Only a user with auditor privileges can access and view a signed audit log. Auditors can use the AuditVerify tool to verify that signed audit logs have not been tampered with.

The signed audit log is created and enabled when the subsystem is configured, but it needs additional configuration to begin creating and signing audit logs.

After enabling signed audit logging, assign auditor users by creating the user and assigning that entry to the auditor group. Members of the auditor group are the only users who can view and verify the signed audit log. See Section 16.3.2.1, “Creating users” for details about setting up auditors.

Auditors can verify logs by using the AuditVerify tool. See the AuditVerify(1) man page for details about using this tool.

There are events that could cause the audit logging function to fail, so events cannot be written to the log. For example, audit logging can fail when the file system containing the audit log file is full or when the file permissions for the log file are accidentally changed. If audit logging fails, the Certificate System instance shuts down in the following manner.

When this happens, administrators and auditors should work together with the operating system administrator to resolve the disk space or file permission issues. When the IT problem is resolved, the auditor should make sure that the last audit log entries are signed. If not, they should be preserved by manual signing (Section 17.2.4.5, “Signing log files”), archived, and removed to prevent audit verification failures in the future. When this is completed, the administrators can restart the Certificate System.

The Certificate System can digitally sign log files before they are archived or distributed for audit purposes. This feature allows files to be checked for tampering.

This is an alternative to the signed audit logs feature. The signed audit log feature creates audit logs that are automatically signed; this tool manually signs archived logs. See Section 17.2.4.3, “Configuring a signed audit log in the console” for details about signed audit logs.

For signing log files, use a command-line utility called the Signing Tool (signtool). For details about this utility, see http://www.mozilla.org/projects/security/pki/nss/tools/.

The utility uses information in the certificate, key, and security module databases of the subsystem instance.

As a user with auditor privilegesuse the signtool command to sign the log directories:

signtool -d secdb_dir -k cert_nickname -Z output input

In Certificate System, administrators can set filters to configure which audit events will be logged in the audit file based on the event attributes.

The format of the filters is the same as for LDAP filters. However, Certificate System only supports the following filters:

For further details on LDAP filters, see the Using Compound Search Filters in the Red Hat Directory Server Administration Guide.

As a user with auditor privileges, use the the pki subsystem-audit-file-find command to list existing audit log files on the server.

For example, to list the audit log files on the CA hosted on server.example.com:

# pki -h server.example.com -p 8443 -n auditor ca-audit-file-find
-----------------
3 entries matched
-----------------
	File name: ca_audit.20170331225716
	Size: 2883

	File name: ca_audit.20170401001030
	Size: 189

	File name: ca_audit
	Size: 6705
----------------------------
Number of entries returned 3
----------------------------

The command uses the client certificate with the auditor nickname stored in the ~/.dogtag/nssdb/ directory for authenticating to the CA. For further details about the parameters used in the command and alternative authentication methods, see the pki(1) man page.

As a user with auditor privileges, use the pki subsystem-audit-file-retrieve command to download a specific audit log from the server.

For example, to download an audit log file from the CA hosted on server.example.com:

After downloading a log file, you can search for specific log entries, for example, using the grep utility:

# grep "\[AuditEvent=ACCESS_SESSION_ESTABLISH\]" log_file

If audit log signing is enabled, users with auditor privileges can verify the logs:

Since these events are keyed (with rhcs_audit_deletion), use the -k parameter to find events matching that key:

# ausearch -k rhcs_audit_deletion

Since these events are keyed (with rhcs_audit_nssdb), use the -k parameter to find events matching that key:

# ausearch -k rhcs_audit_nssdb

Since these events are keyed (with rhcs_audit_time_change), use the -k parameter to find events matching that key:

# ausearch -k rhcs_audit_time_change

Since these events are a typed message (of type SOFTWARE_UPDATE), use the -m parameter to find events matching that type:

# ausearch -m SOFTWARE_UPDATE

Since these events are keyed (with rhcs_audit_config), use the -k parameter to find events matching that key:

# ausearch -k rhcs_audit_config