SupportConsoleDevelopersStart a trialContact

Chapter 3. Installing and configuring the logs service

You can use log messages for troubleshooting and monitoring system events. The log collection agent Rsyslog collects logs on the client side and sends these log records to a remote Elasticsearch storage system separate from your supported Red Hat OpenStack Platform (RHOSP) environment, for example.

3.1. The log system architecture and components

Monitoring tools use a client-server model, with the client deployed onto the Red Hat OpenStack Platform (RHOSP) overcloud nodes. The Rsyslog service provides client-side logging.

Examples of logs in RHOSP include the following:

  • Operating system logs, such as syslog and audit log files.
  • Logs from the infrastructure components, such as RabbitMQ and MariaDB.
  • Logs from the RHOSP services, such as Identity (keystone) and Compute (nova).

The log files record actions, errors, warnings, and other events. In a distributed environment, collecting the various logs in one location can help you with debugging and administration.

Note

RHOSP director does not deploy the server-side components for logging.

3.2. Enabling logging with Elasticsearch

Elasticsearch is a server-side database that you can use to store logs. To enable the logs service for Elasticsearch, you must authenticate the logs service for Elasticsearch.

Note

The Rsyslog service only uses Elasticsearch as a data store for logging.

Prerequisites

  • You have deployed Elasticsearch.
  • You have the username, password, and URL of the server.

Procedure

  1. Create a file in your custom templates directory, such as $HOME/custom_templates/logging-connector.yaml, that you can edit to configure the RsyslogElasticsearchSetting parameters for your environment, such as in the following example: 

    parameter_defaults:
    RsyslogElasticsearchSetting:
        uid: "elastic"
        pwd: "yourownpassword"
        skipverifyhost: "on"
        allowunsignedcerts: "on"
        server: "https://openstack-log-storage.elasticsearch.tld"
        serverport: 443

  2. Add the file paths of the logging-environment-rsyslog.yaml and the logging-connector.yaml environment files to the overcloud deployment command: 

    $ openstack overcloud deploy \
<overcloud_environment_files> \
-e <filepath>/logging-environment-rsyslog.yaml
-e $HOME/custom_templates/logging-connector.yaml
    • Replace <overcloud_environment_files> with the list of environment files in your existing deployment.
    • Replace <filepath> with the file path to your logging-environment-rsyslog.yaml file, for example /usr/share/openstack-tripleo-heat-templates/environments/.

3.3. Configurable logging parameters

This table contains descriptions of logging parameters that you use to configure logging features in Red Hat OpenStack Platform (RHOSP). You can find these parameters in the /usr/share/openstack-tripleo-heat-templates/deployment/logging/rsyslog-container-puppet.yaml file.

Table 3.1. Configurable logging parameters
ParameterDescription

RsyslogElasticsearchSetting

Configuration for rsyslog-elasticsearch plugin.

RsyslogElasticsearchTlsCACert

Contains the content of the CA cert for the CA that issued the Elasticsearch server cert.

RsyslogElasticsearchTlsClientCert

Contains the content of the client cert for doing client cert authorization against Elasticsearch.

RsyslogElasticsearchTlsClientKey

Contains the content of the private key corresponding to the cert RsyslogElasticsearchTlsClientCert.

RsyslogReopenOnTruncate

Prompts rsyslog to reopen the input file when the file size on disk is less than the current offset in memory.

RsyslogMaxMessageSize

Sets the limit for the size of the log message.

3.4. Overriding the default path for a log file

If you modify the default containers to include the path to the service log file, you must also modify the default log file path. Every composable service has a <service_name>LoggingSource parameter. For example, for the nova-compute service, the parameter is NovaComputeLoggingSource.

Procedure

  1. To override the default path for the nova-compute service, add the path to the NovaComputeLoggingSource parameter in your configuration file: 

     NovaComputeLoggingSource:
      tag: openstack.nova.compute
      file: <filepath>/nova-compute.log
    • Replace <filepath> with the file path to your nova-compute.log file
    • Ensure that you define the values of the tag and file parameters for the service. You can use the default values for the other parameters.

  2. You can modify the format for a specific service. The format passes to the Rsyslog configuration. The following example shows the basic syntax: 

    <service_name>LoggingSource:
    tag: <service_name>.tag
    path: <service_name>.path
    format: <service_name>.format

    The following example shows a more complex transformation: 

     ServiceLoggingSource:
    tag: openstack.Service
    path: /var/log/containers/service/service.log
    format: multiline
    format_firstline: '/^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d{3} \d+ \S+ \S+ \[(req-\S+ \S+ \S+ \S+ \S+ \S+|-)\]/'
    format1: '/^(?<Timestamp>\S+ \S+) (?<Pid>\d+) (?<log_level>\S+) (?<python_module>\S+) (\[(req-(?<request_id>\S+) (?<user_id>\S+) (?<tenant_id>\S+) (?<domain_id>\S+) (?<user_domain>\S+) (?<project_domain>\S+)|-)\])? (?<Payload>.*)?$/'

  3. If you enabled centralized logging, you can use the following definition in a custom template to forward additional log files, for example, /var/log/messages: 

    parameter_defaults:
  ExtraConfig:
    tripleo_logging_sources_messages:
      - tag: openstack.host.messages
        file: /var/log/host/messages
        startmsg.regex: "^[a-zA-Z]{3} [ 1-9][0-9] [:0-9]{8}"

3.5. Modifying the format of a log record

You can modify the format of the start of the log record for a specific service. This passes directly to the Rsyslog configuration.

The default format for the Red Hat OpenStack Platform (RHOSP) log record is ('^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}(.[0-9]+ [0-9]+)? (DEBUG|INFO|WARNING|ERROR) ').

Procedure

  • To add a different regular expression for parsing the start of log records, add startmsg.regex to the configuration: 

    NovaComputeLoggingSource:
       tag: openstack.nova.compute
       file: /some/other/path/nova-compute.log
       startmsg.regex: `^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}(.[0-9]+ \\+[0-9]+)? [A-Z]+ \\([a-z]+\\)`

3.6. Verifying the connection between Rsyslog and Elasticsearch

On the client side, you can test and verify the communication between Rsyslog and Elasticsearch.

Procedure

  • Navigate to the Elasticsearch connection log file, which is /var/log/rsyslog/omelasticsearch.log in the Rsyslog container or /var/log/containers/rsyslog/omelasticsearch.log on the host. If this log file does not exist or if the log file exists but does not contain logs, there is no connection problem. If the log file is present and contains logs, Rsyslog has not connected successfully.
Note

To test the connection from the server side, view the Elasticsearch logs for connection issues.

3.7. Tracebacks

If you are troubleshooting an issue, you can use a traceback log to diagnose the issue. In log files, tracebacks usually have several lines of information, all relating to the same issue.

Rsyslog provides a regular expression to define how a log record starts. Each log record usually starts with a timestamp and the first line of the traceback is the only line that contains this information. Rsyslog bundles the indented records with the first line and sends them as one log record.

For that behaviour configuration option startmsg.regex in <Service>LoggingSource is used. The following regular expression is the default value for all <service>LoggingSource parameters in director: 

startmsg.regex='^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}(.[0-9]+ [0-9]+)? (DEBUG|INFO|WARNING|ERROR) '

When this default does not match log records of your added or modified LoggingSource, you must change startmsg.regex accordingly.

3.8. Location of log files for Red Hat OpenStack Platform services

Each Red Hat OpenStack Platform (RHOSP) component has a separate logging directory containing files specific to a running service.

3.8.1. Bare Metal Provisioning (ironic) log files

ServiceService nameLog path

OpenStack Ironic API

openstack-ironic-api.service

/var/log/containers/ironic/ironic-api.log

OpenStack Ironic Conductor

openstack-ironic-conductor.service

/var/log/containers/ironic/ironic-conductor.log

3.8.2. Block Storage (cinder) log files

ServiceService nameLog path

Block Storage API

openstack-cinder-api.service

/var/log/containers/cinder-api.log

Block Storage Backup

openstack-cinder-backup.service

/var/log/containers/cinder/backup.log

Informational messages

The cinder-manage command

/var/log/containers/cinder/cinder-manage.log

Block Storage Scheduler

openstack-cinder-scheduler.service

/var/log/containers/cinder/scheduler.log

Block Storage Volume

openstack-cinder-volume.service

/var/log/containers/cinder/volume.log

3.8.3. Compute (nova) log files

ServiceService nameLog path

OpenStack Compute API service

openstack-nova-api.service

/var/log/containers/nova/nova-api.log

OpenStack Compute certificate server

openstack-nova-cert.service

/var/log/containers/nova/nova-cert.log

OpenStack Compute service

openstack-nova-compute.service

/var/log/containers/nova/nova-compute.log

OpenStack Compute Conductor service

openstack-nova-conductor.service

/var/log/containers/nova/nova-conductor.log

OpenStack Compute VNC console authentication server

openstack-nova-consoleauth.service

/var/log/containers/nova/nova-consoleauth.log

Informational messages

nova-manage command

/var/log/containers/nova/nova-manage.log

OpenStack Compute NoVNC Proxy service

openstack-nova-novncproxy.service

/var/log/containers/nova/nova-novncproxy.log

OpenStack Compute Scheduler service

openstack-nova-scheduler.service

/var/log/containers/nova/nova-scheduler.log

3.8.4. Dashboard (horizon) log files

ServiceService nameLog path

Log of certain user interactions

Dashboard interface

/var/log/containers/horizon/horizon.log

The Apache HTTP server uses additional log files for the Dashboard web interface, which you can access by using a web browser or command-line client, for example, keystone and nova. You can track Dashboard usage and diagnose faults with the following log files:

PurposeLog path

All processed HTTP requests

/var/log/containers/httpd/horizon_access.log

HTTP errors

/var/log/containers/httpd/horizon_error.log

Admin-role API requests

/var/log/containers/httpd/keystone_wsgi_admin_access.log

Admin-role API errors

/var/log/containers/httpd/keystone_wsgi_admin_error.log

Member-role API requests

/var/log/containers/httpd/keystone_wsgi_main_access.log

Member-role API errors

/var/log/containers/httpd/keystone_wsgi_main_error.log

Note

There is also /var/log/containers/httpd/default_error.log, which stores errors reported by other web services that are running on the same host.

3.8.5. Identity Service (keystone) log files

ServiceService nameLog Path

OpenStack Identity Service

openstack-keystone.service

/var/log/containers/keystone/keystone.log

3.8.6. Image Service (glance) log files

ServiceService nameLog path

OpenStack Image Service API server

openstack-glance-api.service

/var/log/containers/glance/api.log

OpenStack Image Service Registry server

openstack-glance-registry.service

/var/log/containers/glance/registry.log

3.8.7. Networking (neutron) log files

ServiceService nameLog path

OpenStack Neutron DHCP Agent

neutron-dhcp-agent.service

/var/log/containers/neutron/dhcp-agent.log

OpenStack Networking Layer 3 Agent

neutron-l3-agent.service

/var/log/containers/neutron/l3-agent.log

Metadata agent service

neutron-metadata-agent.service

/var/log/containers/neutron/metadata-agent.log

Metadata namespace proxy

n/a

/var/log/containers/neutron/neutron-ns-metadata-proxy-UUID.log

Open vSwitch agent

neutron-openvswitch-agent.service

/var/log/containers/neutron/openvswitch-agent.log

OpenStack Networking service

neutron-server.service

/var/log/containers/neutron/server.log

3.8.8. Object Storage (swift) log files

OpenStack Object Storage sends logs to the system logging facility only.

Note

By default, all Object Storage log files go to /var/log/containers/swift/swift.log, using the local0, local1, and local2 syslog facilities.

The Object Storage log messages are either from REST API services or background daemons.

  • The API service messages contain one line per API request. The front-end and back-end services both post messages.
  • The daemon messages contain human-readable information about daemon tasks. The source identity is always at the beginning of the line.

Here is an example of a proxy message: 

Apr 20 15:20:34 rhev-a24c-01 proxy-server: 127.0.0.1 127.0.0.1 20/Apr/2015/19/20/34 GET /v1/AUTH_zaitcev%3Fformat%3Djson%26marker%3Dtestcont HTTP/1.0 200 - python-swiftclient-2.1.0 AUTH_tk737d6... - 2 - txc454fa8ea4844d909820a-0055355182 - 0.0162 - - 1429557634.806570053 1429557634.822791100

Here is an example of daemon messages: 

Apr 27 17:08:15 rhev-a24c-02 object-auditor: Object audit (ZBF). Since Mon Apr 27 21:08:15 2015: Locally: 1 passed, 0 quarantined, 0 errors files/sec: 4.34 , bytes/sec: 0.00, Total time: 0.23, Auditing time: 0.00, Rate: 0.00
Apr 27 17:08:16 rhev-a24c-02 object-auditor: Object audit (ZBF) "forever" mode completed: 0.56s. Total quarantined: 0, Total errors: 0, Total files/sec: 14.31, Total bytes/sec: 0.00, Auditing time: 0.02, Rate: 0.04
Apr 27 17:08:16 rhev-a24c-02 account-replicator: Beginning replication run
Apr 27 17:08:16 rhev-a24c-02 account-replicator: Replication run OVER
Apr 27 17:08:16 rhev-a24c-02 account-replicator: Attempted to replicate 5 dbs in 0.12589 seconds (39.71876/s)
Apr 27 17:08:16 rhev-a24c-02 account-replicator: Removed 0 dbs
Apr 27 17:08:16 rhev-a24c-02 account-replicator: 10 successes, 0 failures

3.8.9. Orchestration (heat) log files

ServiceService nameLog path

OpenStack Heat API Service

openstack-heat-api.service

/var/log/containers/heat/heat-api.log

OpenStack Heat Engine Service

openstack-heat-engine.service

/var/log/containers/heat/heat-engine.log

Orchestration service events

n/a

/var/log/containers/heat/heat-manage.log

3.8.10. Shared Filesystem Service (manila) log files

ServiceService nameLog path

OpenStack Manila API Server

openstack-manila-api.service

/var/log/containers/manila/api.log

OpenStack Manila Scheduler

openstack-manila-scheduler.service

/var/log/containers/manila/scheduler.log//

OpenStack Manila Share Service

openstack-manila-share.service

/var/log/containers/manila/share.log

You can also log information from the Manila Python library in /var/log/containers/manila/manila-manage.log.

3.8.11. Telemetry (ceilometer) log files

ServiceService nameLog path

OpenStack ceilometer notification agent

ceilometer_agent_notification

/var/log/containers/ceilometer/agent-notification.log

OpenStack ceilometer central agent

ceilometer_agent_central

/var/log/containers/ceilometer/central.log

OpenStack ceilometer collection

openstack-ceilometer-collector.service

/var/log/containers/ceilometer/collector.log

OpenStack ceilometer compute agent

ceilometer_agent_compute

/var/log/containers/ceilometer/compute.log

3.8.12. Log files for supporting services

The following services are used by the core RHOSP components and have their own log directories and files.

ServiceService nameLog path

Message broker (RabbitMQ)

rabbitmq-server.service

/var/log/rabbitmq/rabbit@short_hostname.log
/var/log/rabbitmq/rabbit@short_hostname-sasl.log (for Simple Authentication and Security Layer related log messages)

Database server (MariaDB)

mariadb.service

/var/log/mariadb/mariadb.log

Virtual network switch (Open vSwitch)

openvswitch-nonetwork.service

/var/log/openvswitch/ovsdb-server.log
/var/log/openvswitch/ovs-vswitchd.log

3.8.13. aodh (alarming service) log files

ServiceContainer nameLog path

Alarming API

aodh_api

/var/log/containers/httpd/aodh-api/aodh_wsgi_access.log

Alarm evaluator log

aodh_evaluator

/var/log/containers/aodh/aodh-evaluator.log

Alarm listener

aodh_listener

/var/log/containers/aodh/aodh-listener.log

Alarm notification

aodh_notifier

/var/log/containers/aodh/aodh-notifier.log

3.8.14. gnocchi (metric storage) log files

ServiceContainer nameLog path

Gnocchi API

gnocchi_api

/var/log/containers/httpd/gnocchi-api/gnocchi_wsgi_access.log

Gnocchi metricd

gnocchi_metricd

/var/log/containers/gnocchi/gnocchi-metricd.log

Gnocchi statsd

gnocchi_statsd  

/var/log/containers/gnocchi/gnocchi-statsd.log

Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

© 2024 Red Hat, Inc.