Chapter 5. Aggregating Container Logs
5.1. Overview
As an OpenShift administrator, you may want to view the logs from all containers in one user interface. The currently supported method for aggregating container logs in OpenShift Enterprise is using a centralized file system. Additional supported methods are planned for inclusion in future releases.
As packaging improvements are made, these instructions will be simplified.
5.2. Using a Centralized File System
This method reads all container logs and forwards them to a central server for storage on the file system.
5.2.1. Installing fluentd (td-agent) on Nodes
Perform the following steps on each node to install and configure fluentd (td-agent):
Run the following commands:
# export RPM=td-agent-2.2.0-0.x86_64.rpm # curl https://packages.treasuredata.com/2/redhat/7/x86_64/$RPM \ -o /tmp/$RPM # yum localinstall /tmp/$RPM # /opt/td-agent/embedded/bin/gem install fluent-plugin-kubernetes # mkdir -p /etc/td-agent/config.d # chown td-agent:td-agent /etc/td-agent/config.dCreate a directory to house the logs:
# mkdir -p /var/log/td-agent/tmp # chown td-agent:td-agent /var/log/td-agent/tmp
To allow td-agent access to the containers logs, create the /etc/sysconfig/td-agent file and ensure it contains the following:
DAEMON_ARGS= TD_AGENT_ARGS="/usr/sbin/td-agent --log /var/log/td-agent/td-agent.log --use-v1-config"
Add the following line to the /etc/td-agent/td-agent.conf file:
@include config.d/*.conf
Create the /etc/td-agent/config.d/kubernetes.conf file with the following contents:
<source> type tail path /var/lib/docker/containers/*/*-json.log pos_file /var/log/td-agent/tmp/fluentd-docker.pos time_format %Y-%m-%dT%H:%M:%S tag docker.* format json read_from_head true </source> <match docker.var.lib.docker.containers.*.*.log> type kubernetes container_id ${tag_parts[5]} tag docker.${name} </match> <match kubernetes> type copy <store> type forward send_timeout 60s recover_wait 10s heartbeat_interval 1s phi_threshold 16 hard_timeout 60s log_level trace require_ack_response true heartbeat_type tcp <server> name logging_name 1 host host_name 2 port 24224 weight 60 </server> <secondary> type file path /var/log/td-agent/forward-failed </secondary> </store> <store> type file path /var/log/td-agent/containers.log time_slice_format %Y%m%d time_slice_wait 10m time_format %Y%m%dT%H%M%S%z compress gzip utc </store> </match>- Enable fluentd:
# systemctl enable td-agent # systemctl start td-agent
Any errors are logged in the /var/log/td-agent/td-agent.log file.
5.2.2. Optional Method to Verify Working Nodes
You can optionally set up the master to be the aggregator to test and verify that the nodes are working properly.
Install fluentd (td-agent) on the master:
# export RPM=td-agent-2.2.0-0.x86_64.rpm # curl https://packages.treasuredata.com/2/redhat/7/x86_64/$RPM \ -o /tmp/$RPM # yum localinstall /tmp/$RPM # mkdir -p /etc/td-agent/config.d # chown td-agent:td-agent /etc/td-agent/config.d- Ensure port 24224 is open on the master’s firewall to allow the nodes access.
Configure fluentd to aggregate container logs by adding the following line to the /etc/td-agent/td-agent.conf file:
@include config.d/*.conf
Create the /etc/td-agent/config.d/kubernetes.conf file with the following contents:
<match kubernetes.**> type file path /var/log/td-agent/containers.log time_slice_format %Y%m%d time_slice_wait 10m time_format %Y%m%dT%H%M%S%z compress gzip utc </match>Enable fluentd:
# systemctl enable td-agent # systemctl start td-agent
TipAny errors are logged in the /var/log/td-agent/td-agent.log file.
You should now find all the containers' logs available on the master in the /var/log/td-agent/containers.log file.
