Search
SupportConsoleDevelopersStart a trialContact

Chapter 16. Logging and reporting problems

download PDF

The following sections describe the most commonly used log files and debugging tools available in Satellite.

16.1. Increasing logging levels to help with debugging

You can increase the logging level of Satellite components to troubleshoot Satellite. Increasing the logging level to debug provides the most detailed information. The default logging level is info.

16.1.1. Increasing the logging level for the Foreman application

To increase logging for the Foreman application:

  1. Set the logging level to debug: 

    # satellite-installer --foreman-logging-level debug

  2. After you complete debugging, reset the logging level to the default value: 

    # satellite-installer --reset-foreman-logging-level

For more information about Satellite logging settings, use satellite-installer with the --full-help option: 

# satellite-installer --full-help | grep logging

16.1.2. Increasing the logging level for Hammer

You can find the log in ~/.hammer/log/hammer.log.

To increase logging for Hammer:

  • In /etc/hammer/cli_config.yml, set the :log_level: option to debug: 
:log_level: 'debug'

16.1.3. Increasing the logging level for Capsule

You can find the log in /var/log/foreman-proxy/proxy.log.

To increase logging for Capsule:

  1. Set the logging level to debug: 

    # satellite-installer --foreman-proxy-log-level DEBUG

  2. After you complete debugging, reset the logging level to the default value: 

    # satellite-installer --reset-foreman-proxy-log-level
Note

Running satellite-installer will revert the setting back to default.

16.1.4. Increasing the logging level for Candlepin

You can find the logs in /var/log/candlepin/candlepin.log and /var/log/candlepin/error.log.

To increase logging for Candlepin:

  1. Set the logging level to DEBUG: 

    # satellite-installer --katello-candlepin-loggers log4j.logger.org.candlepin:DEBUG

If the candlepin log files are too verbose, you can decrease the default debug level: 

# satellite-installer \
--katello-candlepin-loggers log4j.logger.org.candlepin:DEBUG \
--katello-candlepin-loggers log4j.logger.org.candlepin.resource.ConsumerResource:WARN \
--katello-candlepin-loggers log4j.logger.org.candlepin.resource.HypervisorResource:WARN

  1. After you complete debugging, reset the logging level to the default value: 

    # satellite-installer --reset-katello-candlepin-loggers

16.1.5. Increasing the logging level for Redis

You can find the log for Redis in /var/log/redis/redis.log.

To increase logging for Redis:

  1. In /etc/redis.conf, set the logging level to debug: 

    loglevel debug

  2. Restart the Redis service: 

    # systemctl restart redis
Note

Running satellite-installer will revert the setting to default.

16.1.6. Increasing the logging level for Satellite installer

You can find the log files in /var/log/foreman-installer/.

To increase the logging level of the Satellite installer, add the --verbose-log-level debug option during the installation: 

# satellite-installer --verbose-log-level debug

16.1.7. Increasing the logging level for Pulp

By default, Pulp logs to syslog. You can view the log in /var/log/messages or with journalctl.

To increase logging for Pulp:

  1. In /etc/pulp/settings.py, set the logging level to DEBUG: 

    LOGGING = {"dynaconf_merge": True, "loggers": {'': {'handlers': ['console'], 'level': 'DEBUG'}}}

  2. Restart the Pulp services: 

    # systemctl restart \
pulpcore-api \
pulpcore-content \
pulpcore-resource-manager \
pulpcore-worker@1 \
pulpcore-worker@2 \
redis

16.1.8. Increasing the logging level for Puppet agent

You can find the logs in /var/log/puppetlabs/puppet/.

To increase logging for Puppet agent:

  1. Ensure Puppet is enabled in your Satellite. For more information, see Enabling Puppet Integration with Satellite in Managing configurations using Puppet integration.

  2. Set the logging level to debug: 

    # satellite-installer --puppet-agent-additional-settings log_level:debug

16.1.9. Increasing the logging level for Puppet server

You can find the logs in /var/log/puppetlabs/puppetserver/.

To increase logging for Puppet server:

  1. Ensure Puppet is enabled in your Satellite. For more information, see Enabling Puppet Integration with Satellite in Managing configurations using Puppet integration.

  2. Set the logging level to debug: 

    # satellite-installer --puppet-server-additional-settings log_level:debug

  3. Restart the Puppet server: 

    # satellite-maintain service restart --only puppetserver

16.2. Configuring logging levels with Hammer CLI

You can use Hammer to log debugging information for various Satellite components. The default logging level is INFO.

  • To list the components whose logging level you can change by using hammer: 

    # hammer admin logging --list

  • To set DEBUG level logging for all components: 

    # hammer admin logging --all --level-debug
# satellite-maintain service restart

  • To set PRODUCTION level logging for all components: 

    # hammer admin logging --all --level-production
# satellite-maintain service restart

  • To set DEBUG level logging for a specific component: 

    # hammer admin logging --components My_Component --level-debug
# satellite-maintain service restart

  • To list all available logging options: 

    # hammer admin logging --help

Additional resources

16.3. Configuring logging type and layout

By default, Satellite uses file-based logging. You can use satellite-installer to change the logging type and logging layout.

Procedure for configuring logging with journal

  1. Enter the following satellite-installer command to configure logging to the journald service: 

    # satellite-installer \
--foreman-logging-type journald \
--foreman-proxy-log JOURNAL

  2. Optional: To inspect the log messages, use the journalctl utility. For example:

    • journalctl --unit foreman and journalctl --unit foreman-proxy show messages for the foreman and foreman-proxy units
    • journalctl REQUEST=request_ID shows messages for a specified request

Procedure for configuring file-based logging

  1. Enter the following satellite-installer command to configure file-based logging: 

    # satellite-installer \
--reset-foreman-logging-type \
--reset-foreman-proxy-log

  2. Optional: To inspect the log messages, view these files:

    • /var/log/foreman/production.log
    • /var/log/foreman-proxy.log

Procedure for configuring logging to JSON output

  1. On your Satellite Server, enable logging to JSON output for satellite-installer: 

    # satellite-installer \
--foreman-logging-layout json \
--foreman-logging-type file

  2. Optional: Inspect the log messages using jq: 

    # cat /var/log/foreman/production.log | jq

Additional resources

For more information about Journal, see Viewing logs using the command line in the Red Hat Enterprise Linux 8 Configuring Basic System Settings Guide.

16.4. Selective debugging with individual loggers

You can enable individual loggers for selective debugging.

Procedure

  1. Enable the required individual loggers. For example, to enable sql and ldap loggers, enter the following command: 

    # satellite-installer \
--foreman-loggers ldap:true \
--foreman-loggers sql:true

  2. Optional: Reset all loggers to their default values: 

    # satellite-installer --reset-foreman-loggers

16.5. Overview of individual loggers

The following list provides an overview of selected loggers. You can find the complete list of loggers with their default values in /usr/share/foreman/config/application.rb under Foreman::Logging.add_loggers.

app

Logs web requests and all general application messages.

Default value: true.

audit

Logs additional fact statistics, numbers of added, updated, and removed facts.

Default value: true.

background
Logs information from the background processing component.
blob

Logs contents of rendered templates for auditing purposes.

Important

The blob logger might contain sensitive data.

dynflow
Logs information from the Dynflow process.
ldap

Logs high level LDAP queries and LDAP operations.

Default value: false.

notifications
Logs information from the notifications component.
permissions

Logs queries to user roles, filters, and permissions when loading pages.

Default value: false.

sql

Logs SQL queries made through Rails ActiveRecord.

Default value: false.

telemetry
Logs debugging information from telemetry.
templates
Logs information from the template renderer component.

16.6. Retrieving the status of services

Satellite uses a set of back-end services. When troubleshooting, you can check the status of Satellite services.

Procedure

  • In the Satellite web UI, navigate to Administer > About.

    • On the Smart Proxies tab, view the status of all Capsules.
    • On the Compute Resources tab, view the status of attached compute resource providers.
    • In the Backend System Status table, view the status of all back-end services.

CLI procedure

  • Get information from the database and Satellite services: 

    # hammer ping

  • Check the status of the services running in systemd: 

    # satellite-maintain service status

    Run satellite-maintain service --help for more information.

  • Perform a health check: 

    $ satellite-maintain health check

    Run satellite-maintain health --help for more information.

16.7. Restarting services

Satellite uses a set of back-end services. When troubleshooting, you can restart the services if needed.

Procedure

  • Restart Satellite services: 

    # satellite-maintain service restart

16.8. Utilities for processing log information

You can collect information from log files to troubleshoot Satellite.

sosreport

The sosreport command collects configuration and diagnostic information from a Linux system, such as the running Kernel version, loaded modules, running services, and system and service configuration files. This output is stored in a tar file located at /var/tmp/sosreport-XXX-20171002230919.tar.xz.

For more information, run sosreport --help or see What is a sosreport and how can I create one?.

Important

The sosreport command removes security information such as passwords, tokens, and keys while collecting information. However, the tar files can still contain sensitive information about the Satellite Server. Send the tar files directly to the intended recipient and not to a public target.

foreman-tail

The foreman-tail command displays Satellite logs in real time.

For more information, see the foreman-tail(8) man page.

16.9. Log file directories

The following table provides an overview of selected log directories on Satellite Server:

Log file directoryDescription of log file content

/var/log/candlepin/

Subscription management

/var/log/foreman-installer/

Installer

/var/log/foreman-maintain/

Foreman maintain

/var/log/foreman-proxy/

Foreman proxy

/var/log/foreman/

Foreman

/var/log/httpd/

Apache HTTP server

/var/log/messages/

Various other log messages

/var/log/puppetlabs/puppet/

Puppet

/var/log/tomcat/

Candlepin web service logs

16.10. System journal metadata

The following table lists metadata that the journald service uses in Satellite. You can use this metadata to filter your queries.

AUDIT_ACTION

Audit action performed

Example: Create, update, or delete

AUDIT_TYPE

Audit resource type

Example: Host, Subnet, or ContentView

AUDIT_ID
Audit resource database ID as a number
AUDIT_ATTRIBUTE
Audit resource field or an updated database column
AUDIT_FIELD_OLD
Old audit value of an update action
AUDIT_FIELD_NEW
New audit value of an update action
AUDIT_ID
Record database ID of the audit subject
AUDIT_ATTRIBUTE

Attribute name or column on which an action was performed

Example: Name or description

EXCEPTION_MESSAGE
Exception message when error is logged
EXCEPTION_CLASS
Exception Ruby class when error is logged
EXCEPTION_BACKTRACE
Exception backtrace as a multiline string when error is logged
LOC_ID
Location database ID
LOC_NAME
Location name
LOC_LABEL
Location label
LOGGER

Logger name

To see the current list of loggers enabled by default, enter this command: 

# awk '/add_loggers/,/^$/' /usr/share/foreman/config/application.rb
ORG_ID
Organization database ID
ORG_NAME
Organization name
ORG_LABEL
Organization label
REMOTE_IP
Remote IP address of a client
REQUEST
Request ID generated by the Action Dispatch module
SESSION
Random ID generated per session or a request for a sessionless request
TEMPLATE_NAME
Template name
TEMPLATE_DIGEST
Digest (SHA256) of rendered template contents
TEMPLATE_HOST_NAME
Host name for a rendered template if present
TEMPLATE_HOST_ID
Host database ID for a rendered template if present
USER_LOGIN
User login name
Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

© 2024 Red Hat, Inc.