Red Hat SummitThis documentation is for a release that is no longer maintained
See documentation for the latest supported version 3 or the latest supported version 4.Ce contenu n'est pas disponible dans la langue sélectionnée.
Chapter 29. Aggregate Logging Sizing Guidelines
29.1. Overview
The Elasticsearch, Fluentd, and Kibana (EFK) stack aggregates logs from nodes and applications running inside your OpenShift Container Platform installation. Once deployed it uses Fluentd to aggregate event logs from all nodes, projects, and pods into Elasticsearch (ES). It also provides a centralized Kibana web UI where users and administrators can create rich visualizations and dashboards with the aggregated data.
Fluentd bulk uploads logs to an index, in JSON format, then Elasticsearch routes your search requests to the appropriate shards.
29.2. Installation
The general procedure for installing an aggregate logging stack in OpenShift Container Platform is described in Aggregating Container Logs. There are some important things to keep in mind while going through the installation guide:
In order for the logging pods to spread evenly across your cluster, an empty node selector should be used.
oadm new-project logging --node-selector=""
$ oadm new-project logging --node-selector=""In conjunction with node labeling, which is done later, this controls pod placement across the logging project. You can now create the logging project.
oc project logging
$ oc project loggingA local openshift-ansible template is used by the deployer.
oc create -f ${OPENSHIFT_ANSIBLE_REPO}/roles/openshift_hosted_templates/files/${VERSION}/enterprise/logging-deployer.yaml
$ oc create -f ${OPENSHIFT_ANSIBLE_REPO}/roles/openshift_hosted_templates/files/${VERSION}/enterprise/logging-deployer.yaml
Elasticsearch (ES) should be deployed with a cluster size of at least three for resiliency to node failures. This is specified by passing the ES_CLUSTER_SIZE parameter to the installer.
oc new-app logging-deployer-template \
--param ES_CLUSTER_SIZE=3 \
--param PUBLIC_MASTER_URL=$PUBLIC_MASTER_URL \
--param KIBANA_HOSTNAME=$KIBANA_URL
$ oc new-app logging-deployer-template \
--param ES_CLUSTER_SIZE=3 \
--param PUBLIC_MASTER_URL=$PUBLIC_MASTER_URL \
--param KIBANA_HOSTNAME=$KIBANA_URLRefer to Deploying the EFK Stack for a full list of parameters.
If you do not have an existing Kibana installation, you can use kibana.example.com as a value to KIBANA_HOSTNAME.
As a last step, ensure Fluentd pod spreading through labeling.
oc label nodes --all logging-infra-fluentd=true
$ oc label nodes --all logging-infra-fluentd=true
This operation requires the cluster-admin default role.
Installation can take some time depending on whether the images were already retrieved from the registry or not, and on the size of your cluster.
Inside the logging namespace, you can check your deployment with oc get all.
You should end up with a similar setup to the below.
By default the amount of RAM allocated to each ES instance is 8GB. ES_INSTANCE_RAM is the parameter used in the openshift-ansibletemplate. Keep in mind that half of this value will be passed to the individual elasticsearch pods java processes heap size.
Learn more about installing EFK.
29.2.1. Large Clusters
At 100 nodes or more, it is recommended to pre-pull the logging images first and to set ImagePullPolicy: IfNotPresent in the logging-deployer.yaml file. After deploying the logging infrastructure pods (Elasticsearch, Kibana and Curator), node labeling should be done in steps of 20 nodes at a time. For example:
Using a simple loop:
while read node; do oc label nodes $node logging-infra-fluentd=true; done < 20_fluentd.lst
$ while read node; do oc label nodes $node logging-infra-fluentd=true; done < 20_fluentd.lstThe below also works:
oc label nodes 10.10.0.{100..119} logging-infra-fluentd=true
$ oc label nodes 10.10.0.{100..119} logging-infra-fluentd=trueLabeling nodes in groups paces the DaemonSets used by OpenShift logging, helping to avoid contention on shared resources such as the image registry.
Check for the occurence of any "CrashLoopBackOff | ImagePullFailed | Error" issues. oc logs <pod>, oc describe pod <pod> and oc get event are helpful diagnostic commands.
29.3. Systemd-journald and rsyslog
Rate-limiting
In Red Hat Enterprise Linux (RHEL) 7 the systemd-journald.socket unit creates /dev/log during the boot process, and then passes input to systemd-journald.service. Every syslog() call goes to the journal.
Rsyslog uses the imjournal module as a default input mode for journal files. Refer to Interaction of rsyslog and journal for detailed information about this topic.
A simple test harness was developed, which uses logger across the cluster nodes to make entries of different sizes at different rates in the system log. During testing simulations under a default Red Hat Enterprise Linux (RHEL) 7 installation with systemd-219-19.el7.x86_64 at certain logging rates (approximately 40 log lines per second), we encountered the default rate limit of rsyslogd. After adjusting these limits, entries stopped being written to journald due to local journal file corruption. This issue is resolved in later versions of systemd.
Scaling up
As you scale up your project, the default logging environment might need some adjustments. After updating to systemd-219-22.el7.x86_64, we added:
$IMUXSockRateLimitInterval 0 $IMJournalRatelimitInterval 0
$IMUXSockRateLimitInterval 0
$IMJournalRatelimitInterval 0to /etc/rsyslog.conf and:
to /etc/systemd/journald.conf.
Now, restart the services.
systemctl restart systemd-journald.service systemctl restart rsyslog.service
$ systemctl restart systemd-journald.service
$ systemctl restart rsyslog.serviceThese settings account for the bursty nature of uploading in bulk.
After removing the rate limit, you may see increased CPU utilization on the system logging daemons as it processes any messages that would have previously been throttled.
Rsyslog is configured (see ratelimit.interval, ratelimit.burst) to rate-limit entries read from the journal at 10,000 messages in 300 seconds. A good rule of thumb is to ensure that the rsyslog rate-limits account for the systemd-journald rate-limits.
29.4. Scaling up EFK Logging
If you do not indicate the desired scale at first deployment, the least disruptive way of adjusting your cluster is by re-running the deployer with the updated ES_CLUSTER_SIZE value and using the MODE=reinstall template parameter. Refer to the Performing Administrative Elasticsearch Operations section for more in-depth information.
oc edit configmap logging-deployer oc new-app logging-deployer-template --param MODE=reinstall
$ oc edit configmap logging-deployer
[change es-cluster-size value to 5]
$ oc new-app logging-deployer-template --param MODE=reinstall29.5. Storage Considerations
An Elasticsearch index is a collection of shards and its corresponding replica shards. This is how ES implements high availability internally, therefore there is little need to use hardware based mirroring RAID variants. RAID 0 can still be used to increase overall disk performance.
Every search request needs to hit a copy of every shard in the index. Each ES instance requires its own individual storage, but an OpenShift Container Platform deployment can only provide volumes shared by all of its pods, which again means that Elasticsearch shouldn’t be implemented with a single node.
A persistent volume should be added to each Elasticsearch deployment configuration so that we have one volume per replica shard. On OpenShift Container Platform this is often achieved through Persistent Volume Claims
- 1 volume per shard
- 1 volume per replica shard
The PVCs must be named based on the es-pvc-prefix setting. Refer to Persistent Elasticsearch Storage for more details.
Below are capacity planning guidelines for OpenShift Container Platform aggregate logging. Example scenario
Assumptions:
- Which application: Apache
- Bytes per line: 256
- Lines per second load on application: 1
-
Raw text data
JSON
Baseline (256 characters per second
| Logging Infra Pods | Storage Throughput |
|---|---|
| 3 es 1 kibana 1 curator 1 fluentd | 6 pods total: 90000 x 1440 = 128,6 MB/day |
| 3 es 1 kibana 1 curator 11 fluentd | 16 pods total: 240000 x 1440 = 345,6 MB/day |
| 3 es 1 kibana 1 curator 20 fluentd | 25 pods total: 375000 x 1440 = 540 MB/day |
Calculating total logging throughput and disk space required for your logging environment requires knowledge of your application. For example, if one of your applications on average logs 10 lines-per-second, each 256 bytes-per-line, calculate per-application throughput and disk space as follows:
(bytes-per-line * (lines-per-second) = 2560 bytes per app per second (2560) * (number-of-pods-per-node,100) = 256,000 bytes per second per node 256k * (number-of-nodes) = total logging throughput per cluster
(bytes-per-line * (lines-per-second) = 2560 bytes per app per second
(2560) * (number-of-pods-per-node,100) = 256,000 bytes per second per node
256k * (number-of-nodes) = total logging throughput per clusterFluentd ships any logs from /var/log/messages and /var/lib/docker/containers/ to Elasticsearch. Learn more.
Local SSD drives are recommended in order to achieve the best performance. In Red Hat Enterprise Linux (RHEL) 7, the deadline IO scheduler is the default for all block devices except SATA disks. For SATA disks, the default IO scheduler is cfq.
Sizing storage for ES is greatly dependent on how you optimize your indices. Therefore, consider how much data you need in advance and that you are aggregating application log data.
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}