Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 2. Installing the Centralized Logging Suite

  1. Locate a bare metal system that meets the following minimum specifications:

    • 8 GB of memory
    • Single-socket Xeon class CPU
    • 500 GB of disk space
  2. Install Red Hat Enterprise Linux 7.

  3. Allow the system to access the Operational Tools packages:

    1. Register the system and subscribe it: 

      # subscription-manager register
# subscription-manager list --consumed
      Copy to Clipboard Toggle word wrap

      If an OpenStack subscription is not attached automatically, see the documentation for manually attaching subscriptions.

    2. Disable initially enabled repositories and enable only the ones appropriate for the Operational Tools: 

      # subscription-manager repos --disable=*
# subscription-manager repos --enable=rhel-7-server-rpms --enable=rhel-7-server-optional-rpms --enable=rhel-7-server-openstack-8-optools-rpms
      Copy to Clipboard Toggle word wrap
      Note

      The base OpenStack repository (rhel-7-server-openstack-8-rpms) must not be enabled on this node. This repository may contain newer versions of certain Operational Tools dependencies which may be incompatible with the Operational Tools packages.

  4. Install the Elasticsearch, Fluentd, and Kibana software by running the following command: 

    # yum install elasticsearch fluentd rubygem-fluent-plugin-elasticsearch kibana httpd
    Copy to Clipboard Toggle word wrap

  5. Enable Cross-origin resource sharing (CORS) in Elasticsearch. To do so, edit /etc/elasticsearch/elasticsearch.yml and add the following lines to the end of the file: 

    http.cors.enabled: true
http.cors.allow-origin: "/.*/"
    Copy to Clipboard Toggle word wrap
    Note

    This will allow the Elasticsearch JavaScript applications to be called from any web page on any web server. To allow CORS from your Kibana server only, use: 

    http.cors.allow-origin: "http://LOGGING_SERVER"
    Copy to Clipboard Toggle word wrap

    Replace LOGGING_SERVER with the IP address or the host name of your Kibana server, depending on whether you are going to access Kibana using the IP address or the host name. However, if the Elasticsearch service is only accessible from trusted hosts, it is safe to use "/.*/".

  6. Start the Elasticsearch instance and enable it at boot: 

    # systemctl start elasticsearch
# systemctl enable elasticsearch
    Copy to Clipboard Toggle word wrap

    To confirm the Elasticsearch instance is working, run the following command: 

    # curl http://localhost:9200/
    Copy to Clipboard Toggle word wrap

    This should give a response similar to the following: 

    {
  "status" : 200,
  "name" : "elasticsearch.example.com",
  "cluster_name" : "elasticsearch",
  "version" : {
    "number" : "1.5.2",
    "build_hash" : "c88f77ffc81301dfa9dfd81ca2232f09588bd512",
    "build_timestamp" : "2015-02-19T13:05:36Z",
    "build_snapshot" : false,
    "lucene_version" : "4.10.3"
  },
  "tagline" : "You Know, for Search"
}
    Copy to Clipboard Toggle word wrap

  7. Configure Fluentd to accept log data and write it to Elasticsearch. Edit /etc/fluentd/fluent.conf and replace its content with the following: 

    # In v1 configuration, type and id are @ prefix parameters.
# @type and @id are recommended. type and id are still available for backward compatibility

<source>
  @type forward
  port 4000
  bind 0.0.0.0
</source>

<match **>
  @type elasticsearch
  host localhost
  port 9200
  logstash_format true
  flush_interval 5
</match>
    Copy to Clipboard Toggle word wrap

  8. Start Fluentd and enable it at boot: 

    # systemctl start fluentd
# systemctl enable fluentd
    Copy to Clipboard Toggle word wrap
    Tip

    Check the journal for Fluentd and ensure it has no errors at start: 

    # journalctl -u fluentd -l -f
    Copy to Clipboard Toggle word wrap

  9. Configure Kibana to point to the Elasticsearch instance. Create /etc/httpd/conf.d/kibana3.conf and place the following content inside: 

    <VirtualHost *:80>

  DocumentRoot /usr/share/kibana
  <Directory /usr/share/kibana>
    Require all granted
    Options -Multiviews
  </Directory>

</VirtualHost>
    Copy to Clipboard Toggle word wrap

  10. If you want to restrict access to Kibana and Elasticsearch to authorized users only, for example, because these services are running on a system in an open network, secure the virtual host using HTTP Basic authentication and move Elasticseach behind a proxy. To do so, follow these steps:

    1. Create (or rewrite) the /etc/httpd/conf.d/kibana3.conf file with the following content: 

      <VirtualHost *:80>

  DocumentRoot /usr/share/kibana
  <Directory /usr/share/kibana>
    Options -Multiviews
    AuthUserFile /etc/httpd/conf/htpasswd-kibana
    AuthName "Restricted Kibana Server"
    AuthType Basic
    Require valid-user
  </Directory>

  # Proxy for Elasticsearch
  <LocationMatch "^/(_nodes|_aliases|.*/_aliases|_search|.*/_search|_mapping|.*/_mapping)$">
    ProxyPassMatch http://127.0.0.1:9200/$1
    ProxyPassReverse http://127.0.0.1:9200/$1
  </LocationMatch>

  # Proxy for kibana-int/{dashboard,temp}
  <LocationMatch "^/(kibana-int/dashboard/|kibana-int/temp)(.*)$">
    ProxyPassMatch http://127.0.0.1:9200/$1$2
    ProxyPassReverse http://127.0.0.1:9200/$1$2
  </LocationMatch>

</Virtualhost>
      Copy to Clipboard Toggle word wrap
      Note

      You can use a different path for the AuthUserFile option as well as any other description for the AuthName option.

    2. Create a pair of user name and password which will be allowed to access Kibana. To do so, run the following command: 

      # htpasswd -c /etc/httpd/conf/htpasswd-kibana user_name
      Copy to Clipboard Toggle word wrap
      Note

      If you are using a different path for the AuthUserFile option, change the command accordingly.

      Replace user_name with a user name of your choice. When prompted, enter the password that will be used with this user name. You will be prompted to re-type the password.

    3. Optionally, create more users with their own passwords by running the following command: 

      # htpasswd /etc/httpd/conf/htpasswd-kibana another_user_name
      Copy to Clipboard Toggle word wrap

    4. Configure Elasticsearch to listen on the localhost interface only. To do so, open the /etc/elasticsearch/elasticsearch.yml file in an editor and append the following option: 

      network.host: 127.0.0.1
      Copy to Clipboard Toggle word wrap

    5. You must also configure Elasticsearch to allow using HTTP Basic authentication data with CORS by appending the following option to /etc/elasticsearch/elasticsearch.yml: 

      http.cors.allow-credentials: true
      Copy to Clipboard Toggle word wrap

    6. Restart Elasticsearch for these changes to take effect: 

      # systemctl restart elasticsearch
      Copy to Clipboard Toggle word wrap

    7. Finally, ensure that the Elasticsearch files are downloaded using the proxy, and that the HTTP Basic authentication data is sent by the browser. To do so, edit the /usr/share/kibana/config.js file. Find the following line in this file: 

      elasticsearch: "http://"+window.location.hostname+":9200",
      Copy to Clipboard Toggle word wrap

      and change it as follows: 

      elasticsearch: {server: "http://"+window.location.hostname, withCredentials: true},
      Copy to Clipboard Toggle word wrap

  11. Enable Kibana (inside Apache) to connect to Elasticsearch, and then start Apache and enable it at boot: 

    # setsebool -P httpd_can_network_connect 1
# systemctl start httpd
# systemctl enable httpd
    Copy to Clipboard Toggle word wrap

  12. Open the firewall on the system to allow connections to Fluentd and httpd: 

    # firewall-cmd --zone=public --add-port=4000/tcp --permanent
# firewall-cmd --zone=public --add-service=http --permanent
# firewall-cmd --reload
    Copy to Clipboard Toggle word wrap

  13. Moreover, if you have not configured HTTP authentication and Elasticsearch proxy, open the firewall to allow direct connections to Elasticsearch: 

    # firewall-cmd --zone=public --add-port=9200/tcp --permanent
# firewall-cmd --reload
    Copy to Clipboard Toggle word wrap
    Important

    If you do not restrict access to Kibana and Elasticseach using HTTP authentication, the information provided by Kibana and Elasticsearch is available to anyone without any authentication. To secure the data, ensure that the system or the open TCP ports (80, 4000, and 9200) are only accessible from trusted hosts.

To collect the logs from all the systems in the OpenStack environment and send them to your centralized logging server, run the following commands on all the OpenStack systems.

  1. Enable the Operational Tools repository: 

    # subscription-manager repos --enable=rhel-7-server-openstack-8-optools-rpms
    Copy to Clipboard Toggle word wrap

  2. Install fluentd and rubygem-fluent-plugin-add: 

    # yum install fluentd rubygem-fluent-plugin-add
    Copy to Clipboard Toggle word wrap

  3. Configure the Fluentd user so it has permissions to read all the OpenStack log files. Do this by running the following command: 

    # for user in {keystone,nova,neutron,cinder,glance}; do usermod -a -G $user fluentd; done
    Copy to Clipboard Toggle word wrap

    Note that you may get an error on some nodes about missing groups. This can be disregarded as not all the nodes run all the services.

  4. Configure Fluentd. Make sure /etc/fluentd/fluent.conf looks like the following; be sure to replace LOGGING_SERVER with the host name or IP address of your centralized logging server configured above: 

    # In v1 configuration, type and id are @ prefix parameters.
# @type and @id are recommended. type and id are still available for backward compatibility

# Nova compute
<source>
  @type tail
  path /var/log/nova/nova-compute.log
  tag nova.compute
  format /(?<time>[^ ]* [^ ]*) (?<pid>[^ ]*) (?<loglevel>[^ ]*) (?<class>[^ ]*) \[(?<context>.*)\] (?<message>.*)/
  time_format %F %T.%L
</source>

<match nova.compute>
  type add
  <pair>
    service nova.compute
    hostname "#{Socket.gethostname}"
  </pair>
</match>

# Nova API
<source>
  @type tail
  path /var/log/nova/nova-api.log
  tag nova.api
  format multiline
  format_firstline /(?<time>[^ ]* [^ ]*) (?<pid>[^ ]*) (?<loglevel>[^ ]*) (?<class>[^ ]*) \[(?<context>.*)\] (?<message>.*)/
  format /(?<time>[^ ]* [^ ]*) (?<pid>[^ ]*) (?<loglevel>[^ ]*) (?<class>[^ ]*) \[(?<context>.*)\] (?<message>.*)/
  time_format %F %T.%L
</source>

<match nova.api>
  type add
  <pair>
    service nova.api
    hostname "#{Socket.gethostname}"
  </pair>
</match>

# Nova Cert
<source>
  @type tail
  path /var/log/nova/nova-cert.log
  tag nova.cert
  format multiline
  format_firstline /(?<time>[^ ]* [^ ]*) (?<pid>[^ ]*) (?<loglevel>[^ ]*) (?<class>[^ ]*) \[(?<context>.*)\] (?<message>.*)/
  format /(?<time>[^ ]* [^ ]*) (?<pid>[^ ]*) (?<loglevel>[^ ]*) (?<class>[^ ]*) \[(?<context>.*)\] (?<message>.*)/
  time_format %F %T.%L
</source>

<match nova.cert>
  type add
  <pair>
    service nova.cert
    hostname "#{Socket.gethostname}"
  </pair>
</match>

# Nova Conductor
<source>
  @type tail
  path /var/log/nova/nova-conductor.log
  tag nova.conductor
  format multiline
  format_firstline /(?<time>[^ ]* [^ ]*) (?<pid>[^ ]*) (?<loglevel>[^ ]*) (?<class>[^ ]*) \[(?<context>.*)\] (?<message>.*)/
  format /(?<time>[^ ]* [^ ]*) (?<pid>[^ ]*) (?<loglevel>[^ ]*) (?<class>[^ ]*) \[(?<context>.*)\] (?<message>.*)/
  time_format %F %T.%L
</source>

<match nova.conductor>
  type add
  <pair>
    service nova.conductor
    hostname "#{Socket.gethostname}"
  </pair>
</match>

# Nova Consoleauth
<source>
  @type tail
  path /var/log/nova/nova-consoleauth.log
  tag nova.consoleauth
  format multiline
  format_firstline /(?<time>[^ ]* [^ ]*) (?<pid>[^ ]*) (?<loglevel>[^ ]*) (?<class>[^ ]*) \[(?<context>.*)\] (?<message>.*)/
  format /(?<time>[^ ]* [^ ]*) (?<pid>[^ ]*) (?<loglevel>[^ ]*) (?<class>[^ ]*) \[(?<context>.*)\] (?<message>.*)/
  time_format %F %T.%L
</source>

<match nova.consoleauth>
  type add
  <pair>
    service nova.consoleauth
    hostname "#{Socket.gethostname}"
  </pair>
</match>

# Nova Scheduler
<source>
  @type tail
  path /var/log/nova/nova-scheduler.log
  tag nova.scheduler
  format multiline
  format_firstline /(?<time>[^ ]* [^ ]*) (?<pid>[^ ]*) (?<loglevel>[^ ]*) (?<class>[^ ]*) \[(?<context>.*)\] (?<message>.*)/
  format /(?<time>[^ ]* [^ ]*) (?<pid>[^ ]*) (?<loglevel>[^ ]*) (?<class>[^ ]*) \[(?<context>.*)\] (?<message>.*)/
  time_format %F %T.%L
</source>

<match nova.scheduler>
  type add
  <pair>
    service nova.scheduler
    hostname "#{Socket.gethostname}"
  </pair>
</match>

# Neutron Openvswitch Agent
<source>
  @type tail
  path /var/log/neutron/openvswitch-agent.log
  tag neutron.openvswitch
  format /(?<time>[^ ]* [^ ]*) (?<pid>[^ ]*) (?<loglevel>[^ ]*) (?<class>[^ ]*) \[(?<context>.*)\] (?<message>.*)/
  time_format %F %T.%L
</source>

<match neutron.openvswitch>
  type add
  <pair>
    service neutron.openvswitch
    hostname "#{Socket.gethostname}"
  </pair>
</match>

# Neutron Server
<source>
  @type tail
  path /var/log/neutron/server.log
  tag neutron.server
  format multiline
  format_firstline /(?<time>[^ ]* [^ ]*) (?<pid>[^ ]*) (?<loglevel>[^ ]*) (?<class>[^ ]*) \[(?<context>.*)\] (?<message>.*)/
  format /(?<time>[^ ]* [^ ]*) (?<pid>[^ ]*) (?<loglevel>[^ ]*) (?<class>[^ ]*) \[(?<context>.*)\] (?<message>.*)/
  time_format %F %T.%L
</source>

<match neutron.server>
  type add
  <pair>
    service neutron.server
    hostname "#{Socket.gethostname}"
  </pair>
</match>

# Neutron DHCP Agent
<source>
  @type tail
  path /var/log/neutron/dhcp-agent.log
  tag neutron.dhcp
  format multiline
  format_firstline /(?<time>[^ ]* [^ ]*) (?<pid>[^ ]*) (?<loglevel>[^ ]*) (?<class>[^ ]*) \[(?<context>.*)\] (?<message>.*)/
  format /(?<time>[^ ]* [^ ]*) (?<pid>[^ ]*) (?<loglevel>[^ ]*) (?<class>[^ ]*) \[(?<context>.*)\] (?<message>.*)/
  time_format %F %T.%L
</source>

<match neutron.dhcp>
  type add
  <pair>
    service neutron.dhcp
    hostname "#{Socket.gethostname}"
  </pair>
</match>

# Neutron L3 Agent
<source>
  @type tail
  path /var/log/neutron/l3-agent.log
  tag neutron.l3
  format multiline
  format_firstline /(?<time>[^ ]* [^ ]*) (?<pid>[^ ]*) (?<loglevel>[^ ]*) (?<class>[^ ]*) \[(?<context>.*)\] (?<message>.*)/
  format /(?<time>[^ ]* [^ ]*) (?<pid>[^ ]*) (?<loglevel>[^ ]*) (?<class>[^ ]*) \[(?<context>.*)\] (?<message>.*)/
  time_format %F %T.%L
</source>

<match neutron.l3>
  type add
  <pair>
    service neutron.l3
    hostname "#{Socket.gethostname}"
  </pair>
</match>

# Neutron Metadata Agent
<source>
  @type tail
  path /var/log/neutron/metadata-agent.log
  tag neutron.metadata
  format multiline
  format_firstline /(?<time>[^ ]* [^ ]*) (?<pid>[^ ]*) (?<loglevel>[^ ]*) (?<class>[^ ]*) \[(?<context>.*)\] (?<message>.*)/
  format /(?<time>[^ ]* [^ ]*) (?<pid>[^ ]*) (?<loglevel>[^ ]*) (?<class>[^ ]*) \[(?<context>.*)\] (?<message>.*)/
  time_format %F %T.%L
</source>

<match neutron.metadata>
  type add
  <pair>
    service neutron.metadata
    hostname "#{Socket.gethostname}"
  </pair>
</match>

# Keystone
<source>
  @type tail
  path /var/log/keystone/keystone.log
  tag keystone
  format multiline
  format_firstline /(?<time>[^ ]* [^ ]*) (?<pid>[^ ]*) (?<loglevel>[^ ]*) (?<class>[^ ]*) \[(?<context>.*)\] (?<message>.*)/
  format /(?<time>[^ ]* [^ ]*) (?<pid>[^ ]*) (?<loglevel>[^ ]*) (?<class>[^ ]*) \[(?<context>.*)\] (?<message>.*)/
  time_format %F %T.%L
</source>

<match keystone>
  type add
  <pair>
    service keystone
    hostname "#{Socket.gethostname}"
  </pair>
</match>

# Glance API
<source>
  @type tail
  path /var/log/glance/api.log
  tag glance.api
  format multiline
  format_firstline /(?<time>[^ ]* [^ ]*) (?<pid>[^ ]*) (?<loglevel>[^ ]*) (?<class>[^ ]*) \[(?<context>.*)\] (?<message>.*)/
  format /(?<time>[^ ]* [^ ]*) (?<pid>[^ ]*) (?<loglevel>[^ ]*) (?<class>[^ ]*) \[(?<context>.*)\] (?<message>.*)/
  time_format %F %T.%L
</source>

<match glance.api>
  type add
  <pair>
    service glance.api
    hostname "#{Socket.gethostname}"
  </pair>
</match>

# Glance Registry
<source>
  @type tail
  path /var/log/glance/registry.log
  tag glance.registry
  format multiline
  format_firstline /(?<time>[^ ]* [^ ]*) (?<pid>[^ ]*) (?<loglevel>[^ ]*) (?<class>[^ ]*) \[(?<context>.*)\] (?<message>.*)/
  format /(?<time>[^ ]* [^ ]*) (?<pid>[^ ]*) (?<loglevel>[^ ]*) (?<class>[^ ]*) \[(?<context>.*)\] (?<message>.*)/
  time_format %F %T.%L
</source>

<match glance.registry>
  type add
  <pair>
    service glance.registry
    hostname "#{Socket.gethostname}"
  </pair>
</match>

# Cinder API
<source>
  @type tail
  path /var/log/cinder/api.log
  tag cinder.api
  format multiline
  format_firstline /(?<time>[^ ]* [^ ]*) (?<pid>[^ ]*) (?<loglevel>[^ ]*) (?<class>[^ ]*) \[(?<context>.*)\] (?<message>.*)/
  format /(?<time>[^ ]* [^ ]*) (?<pid>[^ ]*) (?<loglevel>[^ ]*) (?<class>[^ ]*) \[(?<context>.*)\] (?<message>.*)/
  time_format %F %T.%L
</source>

<match cinder.api>
  type add
  <pair>
    service cinder.api
    hostname "#{Socket.gethostname}"
  </pair>
</match>

# Cinder Scheduler
<source>
  @type tail
  path /var/log/cinder/scheduler.log
  tag cinder.scheduler
  format multiline
  format_firstline /(?<time>[^ ]* [^ ]*) (?<pid>[^ ]*) (?<loglevel>[^ ]*) (?<class>[^ ]*) \[(?<context>.*)\] (?<message>.*)/
  format /(?<time>[^ ]* [^ ]*) (?<pid>[^ ]*) (?<loglevel>[^ ]*) (?<class>[^ ]*) \[(?<context>.*)\] (?<message>.*)/
  time_format %F %T.%L
</source>

<match cinder.scheduler>
  type add
  <pair>
    service cinder.scheduler
    hostname "#{Socket.gethostname}"
  </pair>
</match>

# Cinder Volume
<source>
  @type tail
  path /var/log/cinder/volume.log
  tag cinder.volume
  format multiline
  format_firstline /(?<time>[^ ]* [^ ]*) (?<pid>[^ ]*) (?<loglevel>[^ ]*) (?<class>[^ ]*) \[(?<context>.*)\] (?<message>.*)/
  format /(?<time>[^ ]* [^ ]*) (?<pid>[^ ]*) (?<loglevel>[^ ]*) (?<class>[^ ]*) \[(?<context>.*)\] (?<message>.*)/
  time_format %F %T.%L
</source>

<match cinder.volume>
  type add
  <pair>
    service cinder.volume
    hostname "#{Socket.gethostname}"
  </pair>
</match>

<match greped.**>
  @type forward
  heartbeat_type tcp
  <server>
    name LOGGING_SERVER
    host LOGGING_SERVER
    port 4000
  </server>
</match>
    Copy to Clipboard Toggle word wrap

  5. Now that Fluentd has been configured, start the Fluentd service and enable it at boot: 

    # systemctl start fluentd
# systemctl enable fluentd
    Copy to Clipboard Toggle word wrap

You should now be able to access Kibana running at http://LOGGING_SERVER/index.html#/dashboard/file/logstash.json and see logs start to populate. If you have enabled HTTP Basic authentication in the Kibana configuration, you must enter a valid user name and password to access this page.

Note

By default, the front page of the logging server, http://LOGGING_SERVER/, is a Kibana welcome screen providing technical requirements and additional configuration information. If you want the logs to be available here, replace the default.json file in the Kibana application directory with logstash.json, but first create a backup copy of default.json in case you need this file again in the future: 

# mv /usr/share/kibana/app/dashboards/default.json /usr/share/kibana/app/dashboards/default.json.orig
# cp /usr/share/kibana/app/dashboards/logstash.json /usr/share/kibana/app/dashboards/default.json
Copy to Clipboard Toggle word wrap
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2026 Red HatBack to top