Chapter 3. Install the Client-Side Tools

For Fluentd configuration settings, copy /usr/share/openstack-tripleo-heat-templates/environments/logging-environment.yaml and modify the file to suit your environment. For example:

resource_registry:
  OS::TripleO::Services::Fluentd: ../deployment/deprecated/logging/fluentd-container-puppet.yaml

parameter_defaults:
  LoggingServers:
    - host: log0.example.com
      port: 24224
    - host: log1.example.com
      port: 24224

## (note the use of port 24284 for ssl connections)

resource_registry:
  OS::TripleO::Services::Fluentd: ../deployment/deprecated/logging/fluentd-container-puppet.yaml

parameter_defaults:
  LoggingServers:
    - host: 192.0.2.11
      port: 24284
  LoggingUsesSSL: true
  LoggingSharedKey: secret
  LoggingSSLCertificate: |
    -----BEGIN CERTIFICATE-----
    ...certificate data here...
    -----END CERTIFICATE-----

For the Sensu client configuration settings, copy /usr/share/openstack-tripleo-heat-templates/environments/monitoring-environment.yaml and modify the file to suit your environment. For example:

resource_registry:
  OS::TripleO::Services::SensuClient: ../deployment/deprecated/monitoring/sensu-client-container-puppet.yaml

parameter_defaults:
  MonitoringRabbitHost: 10.10.10.10
  MonitoringRabbitPort: 5672
  MonitoringRabbitUserName: sensu
  MonitoringRabbitPassword: sensu
  MonitoringRabbitUseSSL: true
  MonitoringRabbitVhost: "/sensu"
  SensuClientCustomConfig:
    api:
      warning: 10
      critical: 20

Include the modified YAML files with your openstack overcloud deploy command to install the Sensu client and Fluentd tools on all overcloud nodes. For example:

$ openstack overcloud deploy --templates  -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e network-environment.yaml -e ~/templates/monitoring-environment.yaml -e ~/templates/logging-environment.yaml --control-scale 3 --compute-scale 1 --ntp-server 192.168.122.10

You can filter and transform events sent to Fluentd by setting the LoggingDefaultFilters parameter in your environment file. For example, the record_transformer type can modify incoming events:

parameter_defaults:

  LoggingDefaultFilters:
      - tag_pattern: '**'
        type: record_transformer
        enable_ruby: true
        record:
          openstack: '{"hostname": "${hostname}","tag": "${tag}","region": "regionOne","inputname":"fluent-plugin-in_tail","name":"fluentd openstack","fluentd version":"0.12.26", "pipeline_metadata": {"collector":{"ipaddr4":"#{ begin Socket.ip_address_list.select { |a| a.ipv4? && !a.ipv4_loopback? }.map { |a| a.ip_address } end}","ipaddr6":"#{ begin Socket.ip_address_list.select { |a| a.ipv6? && !a.ipv6_loopback? }.map { |a| a.ip_address } end}"}}}'
        remove_keys: host

      - tag_pattern: 'openstack.**'
        type: record_transformer
        record:
          service: '${tag_parts[1]}'

      - tag_pattern: 'system.messages**'
        type: record_transformer
        enable_ruby: true
        record:
          openstack: '{"info": "${record[''message'']}", "systemd":{"t":{"PID":"${record[''pid'']}"},"u":{"SYSLOG_IDENTIFIER":"${record[''ident'']}"}}}'
        remove_keys: 'ident,message,pid'

As a result, the data received by Kibana has been transformed accordingly:

{
  "_index": "logstash-2017.06.29",
  "_type": "fluentd",
  "_id": "AVz132QmRtyd8nnlv_11",
  "_score": null,
  "_source": {
    "pid": "22691",
    "priority": "INFO",
    "message": "cinder.api.openstack.wsgi [req-04bc2808-f86f-4443-86e6-bfc596969937 - - - - -] OPTIONS http://overcloud-controller-0.lab.local/",
    "openstack": {
      "hostname": "overcloud-controller-0",
      "tag": "openstack.cinder.api",
      "region": "regionOne",
      "inputname": "fluent-plugin-in_tail",
      "name": "fluentd openstack",
      "fluentd version": "0.12.26",
      "pipeline_metadata": {
        "collector": {
          "ipaddr4": "[\"192.168.24.14\", \"192.168.24.8\", \"10.0.0.4\", \"10.0.0.5\", \"172.16.2.8\", \"172.16.2.4\", \"172.16.2.14\", \"172.16.1.7\", \"172.16.1.9\", \"172.16.3.10\", \"172.16.3.11\", \"172.16.0.14\"]",
          "ipaddr6": "[\"fe80::293:33ff:fed8:2228%eth0\", \"fe80::293:33ff:fed8:2228%br-ex\", \"fe80::b86c:79ff:fe8f:9fb8%vlan10\", \"fe80::4c78:6fff:feff:14fc%vlan20\", \"fe80::9ced:dfff:fe8c:2d62%vlan30\", \"fe80::ecde:1bff:fe5d:e362%vlan40\", \"fe80::549c:51ff:feea:dfa8%vlan50\", \"fe80::e093:8fff:fef9:69b6%vxlan_sys_4789\"]"
        }
      }
    },
    "service": "cinder",
    "@timestamp": "2017-06-29T21:59:38+00:00"
  },
  "fields": {
    "@timestamp": [
      1498773578000
    ]
  },
  "sort": [
    1498773578000
  ]
}