Logging
Logging installation, usage, and release notes on Red Hat OpenShift Service on AWS
Abstract
Chapter 1. Release notes
1.1. Logging 5.9
Logging is provided as an installable component, with a distinct release cycle from the core Red Hat OpenShift Service on AWS. The Red Hat OpenShift Container Platform Life Cycle Policy outlines release compatibility.
The stable channel only provides updates to the most recent release of logging. To continue receiving updates for prior releases, you must change your subscription channel to stable-x.y, where x.y
represents the major and minor version of logging you have installed. For example, stable-5.7.
1.1.1. Logging 5.9.3
This release includes OpenShift Logging Bug Fix Release 5.9.3
1.1.1.1. Bug Fixes
-
Before this update, there was a delay in restarting Ingesters when configuring
LokiStack
, because the Loki Operator sets the write-ahead logreplay_memory_ceiling
to zero bytes for the1x.demo
size. With this update, the minimum value used for thereplay_memory_ceiling
has been increased to avoid delays. (LOG-5614) - Before this update, monitoring the Vector collector output buffer state was not possible. With this update, monitoring and alerting the Vector collector output buffer size is possible that improves observability capabilities and helps keep the system running optimally. (LOG-5586)
1.1.1.2. CVEs
1.1.2. Logging 5.9.2
This release includes OpenShift Logging Bug Fix Release 5.9.2
1.1.2.1. Bug Fixes
-
Before this update, changes to the Logging Operator caused an error due to an incorrect configuration in the
ClusterLogForwarder
CR. As a result, upgrades to logging deleted the daemonset collector. With this update, the Logging Operator re-creates collector daemonsets except when aNot authorized to collect
error occurs. (LOG-4910) - Before this update, the rotated infrastructure log files were sent to the application index in some scenarios due to an incorrect configuration in the Vector log collector. With this update, the Vector log collector configuration avoids collecting any rotated infrastructure log files. (LOG-5156)
-
Before this update, the Logging Operator did not monitor changes to the
grafana-dashboard-cluster-logging
config map. With this update, the Logging Operator monitors changes in theConfigMap
objects, ensuring the system stays synchronized and responds effectively to config map modifications. (LOG-5308) - Before this update, an issue in the metrics collection code of the Logging Operator caused it to report stale telemetry metrics. With this update, the Logging Operator does not report stale telemetry metrics. (LOG-5426)
-
Before this change, the Fluentd
out_http
plugin ignored theno_proxy
environment variable. With this update, the Fluentd patches theHTTP#start
method of ruby to honor theno_proxy
environment variable. (LOG-5466)
1.1.2.2. CVEs
1.1.3. Logging 5.9.1
This release includes OpenShift Logging Bug Fix Release 5.9.1
1.1.3.1. Enhancements
- Before this update, the Loki Operator configured Loki to use path-based style access for the Amazon Simple Storage Service (S3), which has been deprecated. With this update, the Loki Operator defaults to virtual-host style without users needing to change their configuration. (LOG-5401)
-
Before this update, the Loki Operator did not validate the Amazon Simple Storage Service (S3) endpoint used in the storage secret. With this update, the validation process ensures the S3 endpoint is a valid S3 URL, and the
LokiStack
status updates to indicate any invalid URLs. (LOG-5395)
1.1.3.2. Bug Fixes
- Before this update, a bug in LogQL parsing left out some line filters from the query. With this update, the parsing now includes all the line filters while keeping the original query unchanged. (LOG-5268)
-
Before this update, a prune filter without a defined
pruneFilterSpec
would cause a segfault. With this update, there is a validation error if a prune filter is without a definedpuneFilterSpec
. (LOG-5322) -
Before this update, a drop filter without a defined
dropTestsSpec
would cause a segfault. With this update, there is a validation error if a prune filter is without a definedpuneFilterSpec
. (LOG-5323) -
Before this update, the Loki Operator did not validate the Amazon Simple Storage Service (S3) endpoint URL format used in the storage secret. With this update, the S3 endpoint URL goes through a validation step that reflects on the status of the
LokiStack
. (LOG-5397) -
Before this update, poorly formatted timestamp fields in audit log records led to
WARN
messages in Red Hat OpenShift Logging Operator logs. With this update, a remap transformation ensures that the timestamp field is properly formatted. (LOG-4672) -
Before this update, the error message thrown while validating a
ClusterLogForwarder
resource name and namespace did not correspond to the correct error. With this update, the system checks if aClusterLogForwarder
resource with the same name exists in the same namespace. If not, it corresponds to the correct error. (LOG-5062) - Before this update, the validation feature for output config required a TLS URL, even for services such as Amazon CloudWatch or Google Cloud Logging where a URL is not needed by design. With this update, the validation logic for services without URLs are improved, and the error message are more informative. (LOG-5307)
- Before this update, defining an infrastructure input type did not exclude logging workloads from the collection. With this update, the collection excludes logging services to avoid feedback loops. (LOG-5309)
1.1.3.3. CVEs
No CVEs.
1.1.4. Logging 5.9.0
This release includes OpenShift Logging Bug Fix Release 5.9.0
1.1.4.1. Removal notice
The Logging 5.9 release does not contain an updated version of the OpenShift Elasticsearch Operator. Instances of OpenShift Elasticsearch Operator from prior logging releases, remain supported until the EOL of the logging release. As an alternative to using the OpenShift Elasticsearch Operator to manage the default log storage, you can use the Loki Operator. For more information on the Logging lifecycle dates, see Platform Agnostic Operators.
1.1.4.2. Deprecation notice
- In Logging 5.9, Fluentd, and Kibana are deprecated and are planned to be removed in Logging 6.0, which is expected to be shipped alongside a future release of Red Hat OpenShift Service on AWS. Red Hat will provide critical and above CVE bug fixes and support for these components during the current release lifecycle, but these components will no longer receive feature enhancements. The Vector-based collector provided by the Red Hat OpenShift Logging Operator and LokiStack provided by the Loki Operator are the preferred Operators for log collection and storage. We encourage all users to adopt the Vector and Loki log stack, as this will be the stack that will be enhanced going forward.
-
In Logging 5.9, the
Fields
option for the Splunk output type was never implemented and is now deprecated. It will be removed in a future release.
1.1.4.3. Enhancements
1.1.4.3.1. Log Collection
-
This enhancement adds the ability to refine the process of log collection by using a workload’s metadata to
drop
orprune
logs based on their content. Additionally, it allows the collection of infrastructure logs, such as journal or container logs, and audit logs, such askube api
orovn
logs, to only collect individual sources. (LOG-2155) - This enhancement introduces a new type of remote log receiver, the syslog receiver. You can configure it to expose a port over a network, allowing external systems to send syslog logs using compatible tools such as rsyslog. (LOG-3527)
-
With this update, the
ClusterLogForwarder
API now supports log forwarding to Azure Monitor Logs, giving users better monitoring abilities. This feature helps users to maintain optimal system performance and streamline the log analysis processes in Azure Monitor, which speeds up issue resolution and improves operational efficiency. (LOG-4605) -
This enhancement improves collector resource utilization by deploying collectors as a deployment with two replicas. This occurs when the only input source defined in the
ClusterLogForwarder
custom resource (CR) is a receiver input instead of using a daemon set on all nodes. Additionally, collectors deployed in this manner do not mount the host file system. To use this enhancement, you need to annotate theClusterLogForwarder
CR with thelogging.openshift.io/dev-preview-enable-collector-as-deployment
annotation. (LOG-4779) - This enhancement introduces the capability for custom tenant configuration across all supported outputs, facilitating the organization of log records in a logical manner. However, it does not permit custom tenant configuration for logging managed storage. (LOG-4843)
-
With this update, the
ClusterLogForwarder
CR that specifies an application input with one or more infrastructure namespaces likedefault
,openshift*
, orkube*
, now requires a service account with thecollect-infrastructure-logs
role. (LOG-4943) -
This enhancement introduces the capability for tuning some output settings, such as compression, retry duration, and maximum payloads, to match the characteristics of the receiver. Additionally, this feature includes a delivery mode to allow administrators to choose between throughput and log durability. For example, the
AtLeastOnce
option configures minimal disk buffering of collected logs so that the collector can deliver those logs after a restart. (LOG-5026) - This enhancement adds three new Prometheus alerts, warning users about the deprecation of Elasticsearch, Fluentd, and Kibana. (LOG-5055)
1.1.4.3.2. Log Storage
- This enhancement in LokiStack improves support for OTEL by using the new V13 object storage format and enabling automatic stream sharding by default. This also prepares the collector for future enhancements and configurations. (LOG-4538)
-
This enhancement introduces support for short-lived token workload identity federation with Azure and AWS log stores for STS enabled Red Hat OpenShift Service on AWS 4.14 and later clusters. Local storage requires the addition of a
CredentialMode: static
annotation underspec.storage.secret
in the LokiStack CR. (LOG-4540) - With this update, the validation of the Azure storage secret is now extended to give early warning for certain error conditions. (LOG-4571)
- With this update, Loki now adds upstream and downstream support for GCP workload identity federation mechanism. This allows authenticated and authorized access to the corresponding object storage services. (LOG-4754)
1.1.4.4. Bug Fixes
-
Before this update, the logging must-gather could not collect any logs on a FIPS-enabled cluster. With this update, a new
oc
client is available incluster-logging-rhel9-operator
, and must-gather works properly on FIPS clusters. (LOG-4403) - Before this update, the LokiStack ruler pods could not format the IPv6 pod IP in HTTP URLs used for cross-pod communication. This issue caused querying rules and alerts through the Prometheus-compatible API to fail. With this update, the LokiStack ruler pods encapsulate the IPv6 pod IP in square brackets, resolving the problem. Now, querying rules and alerts through the Prometheus-compatible API works just like in IPv4 environments. (LOG-4709)
- Before this fix, the YAML content from the logging must-gather was exported in a single line, making it unreadable. With this update, the YAML white spaces are preserved, ensuring that the file is properly formatted. (LOG-4792)
-
Before this update, when the
ClusterLogForwarder
CR was enabled, the Red Hat OpenShift Logging Operator could run into a nil pointer exception whenClusterLogging.Spec.Collection
was nil. With this update, the issue is now resolved in the Red Hat OpenShift Logging Operator. (LOG-5006) -
Before this update, in specific corner cases, replacing the
ClusterLogForwarder
CR status field caused theresourceVersion
to constantly update due to changing timestamps inStatus
conditions. This condition led to an infinite reconciliation loop. With this update, all status conditions synchronize, so that timestamps remain unchanged if conditions stay the same. (LOG-5007) -
Before this update, there was an internal buffering behavior to
drop_newest
to address high memory consumption by the collector resulting in significant log loss. With this update, the behavior reverts to using the collector defaults. (LOG-5123) -
Before this update, the Loki Operator
ServiceMonitor
in theopenshift-operators-redhat
namespace used static token and CA files for authentication, causing errors in the Prometheus Operator in the User Workload Monitoring spec on theServiceMonitor
configuration. With this update, the Loki OperatorServiceMonitor
inopenshift-operators-redhat
namespace now references a service account token secret by aLocalReference
object. This approach allows the User Workload Monitoring spec in the Prometheus Operator to handle the Loki OperatorServiceMonitor
successfully, enabling Prometheus to scrape the Loki Operator metrics. (LOG-5165) -
Before this update, the configuration of the Loki Operator
ServiceMonitor
could match many Kubernetes services, resulting in the Loki Operator metrics being collected multiple times. With this update, the configuration ofServiceMonitor
now only matches the dedicated metrics service. (LOG-5212)
1.1.4.5. Known Issues
None.
1.1.4.6. CVEs
1.2. Logging 5.8
Logging is provided as an installable component, with a distinct release cycle from the core Red Hat OpenShift Service on AWS. The Red Hat OpenShift Container Platform Life Cycle Policy outlines release compatibility.
The stable channel only provides updates to the most recent release of logging. To continue receiving updates for prior releases, you must change your subscription channel to stable-x.y, where x.y
represents the major and minor version of logging you have installed. For example, stable-5.7.
1.2.1. Logging 5.8.4
This release includes OpenShift Logging Bug Fix Release 5.8.4.
1.2.1.1. Bug fixes
- Before this update, the developer console’s logs did not account for the current namespace, resulting in query rejection for users without cluster-wide log access. With this update, all supported OCP versions ensure correct namespace inclusion. (LOG-4905)
-
Before this update, the Cluster Logging Operator deployed
ClusterRoles
supporting LokiStack deployments only when the default log output was LokiStack. With this update, the roles are split into two groups: read and write. The write roles deploys based on the setting of the default log storage, just like all the roles used to do before. The read roles deploys based on whether the logging console plugin is active. (LOG-4987) -
Before this update, multiple
ClusterLogForwarders
defining the same input receiver name had their service endlessly reconciled because of changingownerReferences
on one service. With this update, each receiver input will have its own service named with the convention of<CLF.Name>-<input.Name>
. (LOG-5009) -
Before this update, the
ClusterLogForwarder
did not report errors when forwarding logs to cloudwatch without a secret. With this update, the following error message appears when forwarding logs to cloudwatch without a secret:secret must be provided for cloudwatch output
. (LOG-5021) -
Before this update, the
log_forwarder_input_info
includedapplication
,infrastructure
, andaudit
input metric points. With this update,http
is also added as a metric point. (LOG-5043)
1.2.1.2. CVEs
- CVE-2021-35937
- CVE-2021-35938
- CVE-2021-35939
- CVE-2022-3545
- CVE-2022-24963
- CVE-2022-36402
- CVE-2022-41858
- CVE-2023-2166
- CVE-2023-2176
- CVE-2023-3777
- CVE-2023-3812
- CVE-2023-4015
- CVE-2023-4622
- CVE-2023-4623
- CVE-2023-5178
- CVE-2023-5363
- CVE-2023-5388
- CVE-2023-5633
- CVE-2023-6679
- CVE-2023-7104
- CVE-2023-27043
- CVE-2023-38409
- CVE-2023-40283
- CVE-2023-42753
- CVE-2023-43804
- CVE-2023-45803
- CVE-2023-46813
- CVE-2024-20918
- CVE-2024-20919
- CVE-2024-20921
- CVE-2024-20926
- CVE-2024-20945
- CVE-2024-20952
1.2.2. Logging 5.8.3
This release includes Logging Bug Fix 5.8.3 and Logging Security Fix 5.8.3
1.2.2.1. Bug fixes
- Before this update, when configured to read a custom S3 Certificate Authority the Loki Operator would not automatically update the configuration when the name of the ConfigMap or the contents changed. With this update, the Loki Operator is watching for changes to the ConfigMap and automatically updates the generated configuration. (LOG-4969)
- Before this update, Loki outputs configured without a valid URL caused the collector pods to crash. With this update, outputs are subject to URL validation, resolving the issue. (LOG-4822)
- Before this update the Cluster Logging Operator would generate collector configuration fields for outputs that did not specify a secret to use the service account bearer token. With this update, an output does not require authentication, resolving the issue. (LOG-4962)
-
Before this update, the
tls.insecureSkipVerify
field of an output was not set to a value oftrue
without a secret defined. With this update, a secret is no longer required to set this value. (LOG-4963) - Before this update, output configurations allowed the combination of an insecure (HTTP) URL with TLS authentication. With this update, outputs configured for TLS authentication require a secure (HTTPS) URL. (LOG-4893)
1.2.2.2. CVEs
1.2.3. Logging 5.8.2
This release includes OpenShift Logging Bug Fix Release 5.8.2.
1.2.3.1. Bug fixes
- Before this update, the LokiStack ruler pods would not format the IPv6 pod IP in HTTP URLs used for cross pod communication, causing querying rules and alerts through the Prometheus-compatible API to fail. With this update, the LokiStack ruler pods encapsulate the IPv6 pod IP in square brackets, resolving the issue. (LOG-4890)
- Before this update, the developer console logs did not account for the current namespace, resulting in query rejection for users without cluster-wide log access. With this update, namespace inclusion has been corrected, resolving the issue. (LOG-4947)
- Before this update, the logging view plugin of the Red Hat OpenShift Service on AWS web console did not allow for custom node placement and tolerations. With this update, defining custom node placements and tolerations has been added to the logging view plugin of the Red Hat OpenShift Service on AWS web console. (LOG-4912)
1.2.3.2. CVEs
1.2.4. Logging 5.8.1
This release includes OpenShift Logging Bug Fix Release 5.8.1 and OpenShift Logging Bug Fix Release 5.8.1 Kibana.
1.2.4.1. Enhancements
1.2.4.1.1. Log Collection
- With this update, while configuring Vector as a collector, you can add logic to the Red Hat OpenShift Logging Operator to use a token specified in the secret in place of the token associated with the service account. (LOG-4780)
- With this update, the BoltDB Shipper Loki dashboards are now renamed to Index dashboards. (LOG-4828)
1.2.4.2. Bug fixes
-
Before this update, the
ClusterLogForwarder
created empty indices after enabling the parsing of JSON logs, even when the rollover conditions were not met. With this update, theClusterLogForwarder
skips the rollover when thewrite-index
is empty. (LOG-4452) -
Before this update, the Vector set the
default
log level incorrectly. With this update, the correct log level is set by improving the enhancement of regular expression, orregexp
, for log level detection. (LOG-4480) -
Before this update, during the process of creating index patterns, the default alias was missing from the initial index in each log output. As a result, Kibana users were unable to create index patterns by using OpenShift Elasticsearch Operator. This update adds the missing aliases to OpenShift Elasticsearch Operator, resolving the issue. Kibana users can now create index patterns that include the
{app,infra,audit}-000001
indexes. (LOG-4683) -
Before this update, Fluentd collector pods were in a
CrashLoopBackOff
state due to binding of the Prometheus server on IPv6 clusters. With this update, the collectors work properly on IPv6 clusters. (LOG-4706) -
Before this update, the Red Hat OpenShift Logging Operator would undergo numerous reconciliations whenever there was a change in the
ClusterLogForwarder
. With this update, the Red Hat OpenShift Logging Operator disregards the status changes in the collector daemonsets that triggered the reconciliations. (LOG-4741) -
Before this update, the Vector log collector pods were stuck in the
CrashLoopBackOff
state on IBM Power machines. With this update, the Vector log collector pods start successfully on IBM Power architecture machines. (LOG-4768) -
Before this update, forwarding with a legacy forwarder to an internal LokiStack would produce SSL certificate errors using Fluentd collector pods. With this update, the log collector service account is used by default for authentication, using the associated token and
ca.crt
. (LOG-4791) -
Before this update, forwarding with a legacy forwarder to an internal LokiStack would produce SSL certificate errors using Vector collector pods. With this update, the log collector service account is used by default for authentication and also using the associated token and
ca.crt
. (LOG-4852) - Before this fix, IPv6 addresses would not be parsed correctly after evaluating a host or multiple hosts for placeholders. With this update, IPv6 addresses are correctly parsed. (LOG-4811)
-
Before this update, it was necessary to create a
ClusterRoleBinding
to collect audit permissions for HTTP receiver inputs. With this update, it is not necessary to create theClusterRoleBinding
because the endpoint already depends upon the cluster certificate authority. (LOG-4815) - Before this update, the Loki Operator did not mount a custom CA bundle to the ruler pods. As a result, during the process to evaluate alerting or recording rules, object storage access failed. With this update, the Loki Operator mounts the custom CA bundle to all ruler pods. The ruler pods can download logs from object storage to evaluate alerting or recording rules. (LOG-4836)
-
Before this update, while removing the
inputs.receiver
section in theClusterLogForwarder
, the HTTP input services and its associated secrets were not deleted. With this update, the HTTP input resources are deleted when not needed. (LOG-4612) -
Before this update, the
ClusterLogForwarder
indicated validation errors in the status, but the outputs and the pipeline status did not accurately reflect the specific issues. With this update, the pipeline status displays the validation failure reasons correctly in case of misconfigured outputs, inputs, or filters. (LOG-4821) -
Before this update, changing a
LogQL
query that used controls such as time range or severity changed the label matcher operator defining it like a regular expression. With this update, regular expression operators remain unchanged when updating the query. (LOG-4841)
1.2.4.3. CVEs
- CVE-2007-4559
- CVE-2021-3468
- CVE-2021-3502
- CVE-2021-3826
- CVE-2021-43618
- CVE-2022-3523
- CVE-2022-3565
- CVE-2022-3594
- CVE-2022-4285
- CVE-2022-38457
- CVE-2022-40133
- CVE-2022-40982
- CVE-2022-41862
- CVE-2022-42895
- CVE-2023-0597
- CVE-2023-1073
- CVE-2023-1074
- CVE-2023-1075
- CVE-2023-1076
- CVE-2023-1079
- CVE-2023-1206
- CVE-2023-1249
- CVE-2023-1252
- CVE-2023-1652
- CVE-2023-1855
- CVE-2023-1981
- CVE-2023-1989
- CVE-2023-2731
- CVE-2023-3138
- CVE-2023-3141
- CVE-2023-3161
- CVE-2023-3212
- CVE-2023-3268
- CVE-2023-3316
- CVE-2023-3358
- CVE-2023-3576
- CVE-2023-3609
- CVE-2023-3772
- CVE-2023-3773
- CVE-2023-4016
- CVE-2023-4128
- CVE-2023-4155
- CVE-2023-4194
- CVE-2023-4206
- CVE-2023-4207
- CVE-2023-4208
- CVE-2023-4273
- CVE-2023-4641
- CVE-2023-22745
- CVE-2023-26545
- CVE-2023-26965
- CVE-2023-26966
- CVE-2023-27522
- CVE-2023-29491
- CVE-2023-29499
- CVE-2023-30456
- CVE-2023-31486
- CVE-2023-32324
- CVE-2023-32573
- CVE-2023-32611
- CVE-2023-32665
- CVE-2023-33203
- CVE-2023-33285
- CVE-2023-33951
- CVE-2023-33952
- CVE-2023-34241
- CVE-2023-34410
- CVE-2023-35825
- CVE-2023-36054
- CVE-2023-37369
- CVE-2023-38197
- CVE-2023-38545
- CVE-2023-38546
- CVE-2023-39191
- CVE-2023-39975
- CVE-2023-44487
1.2.5. Logging 5.8.0
This release includes OpenShift Logging Bug Fix Release 5.8.0 and OpenShift Logging Bug Fix Release 5.8.0 Kibana.
1.2.5.1. Deprecation notice
In Logging 5.8, Elasticsearch, Fluentd, and Kibana are deprecated and are planned to be removed in Logging 6.0, which is expected to be shipped alongside a future release of Red Hat OpenShift Service on AWS. Red Hat will provide critical and above CVE bug fixes and support for these components during the current release lifecycle, but these components will no longer receive feature enhancements. The Vector-based collector provided by the Red Hat OpenShift Logging Operator and LokiStack provided by the Loki Operator are the preferred Operators for log collection and storage. We encourage all users to adopt the Vector and Loki log stack, as this will be the stack that will be enhanced going forward.
1.2.5.2. Enhancements
1.2.5.2.1. Log Collection
-
With this update, the LogFileMetricExporter is no longer deployed with the collector by default. You must manually create a
LogFileMetricExporter
custom resource (CR) to generate metrics from the logs produced by running containers. If you do not create theLogFileMetricExporter
CR, you may see a No datapoints found message in the Red Hat OpenShift Service on AWS web console dashboard for Produced Logs. (LOG-3819) With this update, you can deploy multiple, isolated, and RBAC-protected
ClusterLogForwarder
custom resource (CR) instances in any namespace. This allows independent groups to forward desired logs to any destination while isolating their configuration from other collector deployments. (LOG-1343)ImportantIn order to support multi-cluster log forwarding in additional namespaces other than the
openshift-logging
namespace, you must update the Red Hat OpenShift Logging Operator to watch all namespaces. This functionality is supported by default in new Red Hat OpenShift Logging Operator version 5.8 installations.- With this update, you can use the flow control or rate limiting mechanism to limit the volume of log data that can be collected or forwarded by dropping excess log records. The input limits prevent poorly-performing containers from overloading the Logging and the output limits put a ceiling on the rate of logs shipped to a given data store. (LOG-884)
- With this update, you can configure the log collector to look for HTTP connections and receive logs as an HTTP server, also known as a webhook. (LOG-4562)
- With this update, you can configure audit policies to control which Kubernetes and OpenShift API server events are forwarded by the log collector. (LOG-3982)
1.2.5.2.2. Log Storage
- With this update, LokiStack administrators can have more fine-grained control over who can access which logs by granting access to logs on a namespace basis. (LOG-3841)
-
With this update, the Loki Operator introduces
PodDisruptionBudget
configuration on LokiStack deployments to ensure normal operations during Red Hat OpenShift Service on AWS cluster restarts by keeping ingestion and the query path available. (LOG-3839) - With this update, the reliability of existing LokiStack installations are seamlessly improved by applying a set of default Affinity and Anti-Affinity policies. (LOG-3840)
- With this update, you can manage zone-aware data replication as an administrator in LokiStack, in order to enhance reliability in the event of a zone failure. (LOG-3266)
- With this update, a new supported small-scale LokiStack size of 1x.extra-small is introduced for Red Hat OpenShift Service on AWS clusters hosting a few workloads and smaller ingestion volumes (up to 100GB/day). (LOG-4329)
- With this update, the LokiStack administrator has access to an official Loki dashboard to inspect the storage performance and the health of each component. (LOG-4327)
1.2.5.2.3. Log Console
- With this update, you can enable the Logging Console Plugin when Elasticsearch is the default Log Store. (LOG-3856)
- With this update, Red Hat OpenShift Service on AWS application owners can receive notifications for application log-based alerts on the Red Hat OpenShift Service on AWS web console Developer perspective for Red Hat OpenShift Service on AWS version 4.14 and later. (LOG-3548)
1.2.5.3. Known Issues
Currently, Splunk log forwarding might not work after upgrading to version 5.8 of the Red Hat OpenShift Logging Operator. This issue is caused by transitioning from OpenSSL version 1.1.1 to version 3.0.7. In the newer OpenSSL version, there is a default behavior change, where connections to TLS 1.2 endpoints are rejected if they do not expose the RFC 5746 extension.
As a workaround, enable TLS 1.3 support on the TLS terminating load balancer in front of the Splunk HEC (HTTP Event Collector) endpoint. Splunk is a third-party system and this should be configured from the Splunk end.
-
Currently, there is a flaw in handling multiplexed streams in the HTTP/2 protocol, where you can repeatedly make a request for a new multiplex stream and immediately send an
RST_STREAM
frame to cancel it. This created extra work for the server set up and tore down the streams, resulting in a denial of service due to server resource consumption. There is currently no workaround for this issue. (LOG-4609) -
Currently, when using FluentD as the collector, the collector pod cannot start on the Red Hat OpenShift Service on AWS IPv6-enabled cluster. The pod logs produce the
fluentd pod [error]: unexpected error error_class=SocketError error="getaddrinfo: Name or service not known
error. There is currently no workaround for this issue. (LOG-4706) - Currently, the log alert is not available on an IPv6-enabled cluster. There is currently no workaround for this issue. (LOG-4709)
-
Currently,
must-gather
cannot gather any logs on a FIPS-enabled cluster, because the required OpenSSL library is not available in thecluster-logging-rhel9-operator
. There is currently no workaround for this issue. (LOG-4403) -
Currently, when deploying the logging version 5.8 on a FIPS-enabled cluster, the collector pods cannot start and are stuck in
CrashLoopBackOff
status, while using FluentD as a collector. There is currently no workaround for this issue. (LOG-3933)
1.2.5.4. CVEs
1.3. Logging 5.7
Logging is provided as an installable component, with a distinct release cycle from the core Red Hat OpenShift Service on AWS. The Red Hat OpenShift Container Platform Life Cycle Policy outlines release compatibility.
The stable channel only provides updates to the most recent release of logging. To continue receiving updates for prior releases, you must change your subscription channel to stable-x.y, where x.y
represents the major and minor version of logging you have installed. For example, stable-5.7.
1.3.1. Logging 5.7.8
This release includes OpenShift Logging Bug Fix Release 5.7.8.
1.3.1.1. Bug fixes
-
Before this update, there was a potential conflict when the same name was used for the
outputRefs
andinputRefs
parameters in theClusterLogForwarder
custom resource (CR). As a result, the collector pods entered in aCrashLoopBackOff
status. With this update, the output labels contain theOUTPUT_
prefix to ensure a distinction between output labels and pipeline names. (LOG-4383) -
Before this update, while configuring the JSON log parser, if you did not set the
structuredTypeKey
orstructuredTypeName
parameters for the Cluster Logging Operator, no alert would display about an invalid configuration. With this update, the Cluster Logging Operator informs you about the configuration issue. (LOG-4441) -
Before this update, if the
hecToken
key was missing or incorrect in the secret specified for a Splunk output, the validation failed because the Vector forwarded logs to Splunk without a token. With this update, if thehecToken
key is missing or incorrect, the validation fails with theA non-empty hecToken entry is required
error message. (LOG-4580) -
Before this update, selecting a date from the
Custom time range
for logs caused an error in the web console. With this update, you can select a date from the time range model in the web console successfully. (LOG-4684)
1.3.1.2. CVEs
1.3.2. Logging 5.7.7
This release includes OpenShift Logging Bug Fix Release 5.7.7.
1.3.2.1. Bug fixes
- Before this update, FluentD normalized the logs emitted by the EventRouter differently from Vector. With this update, the Vector produces log records in a consistent format. (LOG-4178)
- Before this update, there was an error in the query used for the FluentD Buffer Availability graph in the metrics dashboard created by the Cluster Logging Operator as it showed the minimum buffer usage. With this update, the graph shows the maximum buffer usage and is now renamed to FluentD Buffer Usage. (LOG-4555)
-
Before this update, deploying a LokiStack on IPv6-only or dual-stack Red Hat OpenShift Service on AWS clusters caused the LokiStack memberlist registration to fail. As a result, the distributor pods went into a crash loop. With this update, an administrator can enable IPv6 by setting the
lokistack.spec.hashRing.memberlist.enableIPv6:
value totrue
, which resolves the issue. (LOG-4569) - Before this update, the log collector relied on the default configuration settings for reading the container log lines. As a result, the log collector did not read the rotated files efficiently. With this update, there is an increase in the number of bytes read, which allows the log collector to efficiently process rotated files. (LOG-4575)
- Before this update, the unused metrics in the Event Router caused the container to fail due to excessive memory usage. With this update, there is reduction in the memory usage of the Event Router by removing the unused metrics. (LOG-4686)
1.3.2.2. CVEs
1.3.3. Logging 5.7.6
This release includes OpenShift Logging Bug Fix Release 5.7.6.
1.3.3.1. Bug fixes
- Before this update, the collector relied on the default configuration settings for reading the container log lines. As a result, the collector did not read the rotated files efficiently. With this update, there is an increase in the number of bytes read, which allows the collector to efficiently process rotated files. (LOG-4501)
- Before this update, when users pasted a URL with predefined filters, some filters did not reflect. With this update, the UI reflects all the filters in the URL. (LOG-4459)
- Before this update, forwarding to Loki using custom labels generated an error when switching from Fluentd to Vector. With this update, the Vector configuration sanitizes labels in the same way as Fluentd to ensure the collector starts and correctly processes labels. (LOG-4460)
- Before this update, the Observability Logs console search field did not accept special characters that it should escape. With this update, it is escaping special characters properly in the query. (LOG-4456)
-
Before this update, the following warning message appeared while sending logs to Splunk:
Timestamp was not found.
With this update, the change overrides the name of the log field used to retrieve the Timestamp and sends it to Splunk without warning. (LOG-4413) -
Before this update, the CPU and memory usage of Vector was increasing over time. With this update, the Vector configuration now contains the
expire_metrics_secs=60
setting to limit the lifetime of the metrics and cap the associated CPU usage and memory footprint. (LOG-4171) - Before this update, the LokiStack gateway cached authorized requests very broadly. As a result, this caused wrong authorization results. With this update, LokiStack gateway caches on a more fine-grained basis which resolves this issue. (LOG-4393)
- Before this update, the Fluentd runtime image included builder tools which were unnecessary at runtime. With this update, the builder tools are removed, resolving the issue. (LOG-4467)
1.3.3.2. CVEs
1.3.4. Logging 5.7.4
This release includes OpenShift Logging Bug Fix Release 5.7.4.
1.3.4.1. Bug fixes
-
Before this update, when forwarding logs to CloudWatch, a
namespaceUUID
value was not appended to thelogGroupName
field. With this update, thenamespaceUUID
value is included, so alogGroupName
in CloudWatch appears aslogGroupName: vectorcw.b443fb9e-bd4c-4b6a-b9d3-c0097f9ed286
. (LOG-2701) - Before this update, when forwarding logs over HTTP to an off-cluster destination, the Vector collector was unable to authenticate to the cluster-wide HTTP proxy even though correct credentials were provided in the proxy URL. With this update, the Vector log collector can now authenticate to the cluster-wide HTTP proxy. (LOG-3381)
- Before this update, the Operator would fail if the Fluentd collector was configured with Splunk as an output, due to this configuration being unsupported. With this update, configuration validation rejects unsupported outputs, resolving the issue. (LOG-4237)
-
Before this update, when the Vector collector was updated an
enabled = true
value in the TLS configuration for AWS Cloudwatch logs and the GCP Stackdriver caused a configuration error. With this update,enabled = true
value will be removed for these outputs, resolving the issue. (LOG-4242) -
Before this update, the Vector collector occasionally panicked with the following error message in its log:
thread 'vector-worker' panicked at 'all branches are disabled and there is no else branch', src/kubernetes/reflector.rs:26:9
. With this update, the error has been resolved. (LOG-4275) -
Before this update, an issue in the Loki Operator caused the
alert-manager
configuration for the application tenant to disappear if the Operator was configured with additional options for that tenant. With this update, the generated Loki configuration now contains both the custom and the auto-generated configuration. (LOG-4361) - Before this update, when multiple roles were used to authenticate using STS with AWS Cloudwatch forwarding, a recent update caused the credentials to be non-unique. With this update, multiple combinations of STS roles and static credentials can once again be used to authenticate with AWS Cloudwatch. (LOG-4368)
- Before this update, Loki filtered label values for active streams but did not remove duplicates, making Grafana’s Label Browser unusable. With this update, Loki filters out duplicate label values for active streams, resolving the issue. (LOG-4389)
-
Pipelines with no
name
field specified in theClusterLogForwarder
custom resource (CR) stopped working after upgrading to OpenShift Logging 5.7. With this update, the error has been resolved. (LOG-4120)
1.3.4.2. CVEs
1.3.5. Logging 5.7.3
This release includes OpenShift Logging Bug Fix Release 5.7.3.
1.3.5.1. Bug fixes
- Before this update, when viewing logs within the Red Hat OpenShift Service on AWS web console, cached files caused the data to not refresh. With this update the bootstrap files are not cached, resolving the issue. (LOG-4100)
- Before this update, the Loki Operator reset errors in a way that made identifying configuration problems difficult to troubleshoot. With this update, errors persist until the configuration error is resolved. (LOG-4156)
-
Before this update, the LokiStack ruler did not restart after changes were made to the
RulerConfig
custom resource (CR). With this update, the Loki Operator restarts the ruler pods after theRulerConfig
CR is updated. (LOG-4161) -
Before this update, the vector collector terminated unexpectedly when input match label values contained a
/
character within theClusterLogForwarder
. This update resolves the issue by quoting the match label, enabling the collector to start and collect logs. (LOG-4176) -
Before this update, the Loki Operator terminated unexpectedly when a
LokiStack
CR defined tenant limits, but not global limits. With this update, the Loki Operator can processLokiStack
CRs without global limits, resolving the issue. (LOG-4198) - Before this update, Fluentd did not send logs to an Elasticsearch cluster when the private key provided was passphrase-protected. With this update, Fluentd properly handles passphrase-protected private keys when establishing a connection with Elasticsearch. (LOG-4258)
-
Before this update, clusters with more than 8,000 namespaces caused Elasticsearch to reject queries because the list of namespaces was larger than the
http.max_header_size
setting. With this update, the default value for header size has been increased, resolving the issue. (LOG-4277) -
Before this update, label values containing a
/
character within theClusterLogForwarder
CR would cause the collector to terminate unexpectedly. With this update, slashes are replaced with underscores, resolving the issue. (LOG-4095) -
Before this update, the Cluster Logging Operator terminated unexpectedly when set to an unmanaged state. With this update, a check to ensure that the
ClusterLogging
resource is in the correct Management state before initiating the reconciliation of theClusterLogForwarder
CR, resolving the issue. (LOG-4177) - Before this update, when viewing logs within the Red Hat OpenShift Service on AWS web console, selecting a time range by dragging over the histogram did not work on the aggregated logs view inside the pod detail. With this update, the time range can be selected by dragging on the histogram in this view. (LOG-4108)
- Before this update, when viewing logs within the Red Hat OpenShift Service on AWS web console, queries longer than 30 seconds timed out. With this update, the timeout value can be configured in the configmap/logging-view-plugin. (LOG-3498)
- Before this update, when viewing logs within the Red Hat OpenShift Service on AWS web console, clicking the more data available option loaded more log entries only the first time it was clicked. With this update, more entries are loaded with each click. (OU-188)
- Before this update, when viewing logs within the Red Hat OpenShift Service on AWS web console, clicking the streaming option would only display the streaming logs message without showing the actual logs. With this update, both the message and the log stream are displayed correctly. (OU-166)
1.3.5.2. CVEs
1.3.6. Logging 5.7.2
This release includes OpenShift Logging Bug Fix Release 5.7.2.
1.3.6.1. Bug fixes
-
Before this update, it was not possible to delete the
openshift-logging
namespace directly due to the presence of a pending finalizer. With this update, the finalizer is no longer utilized, enabling direct deletion of the namespace. (LOG-3316) -
Before this update, the
run.sh
script would display an incorrectchunk_limit_size
value if it was changed according to the Red Hat OpenShift Service on AWS documentation. However, when setting thechunk_limit_size
via the environment variable$BUFFER_SIZE_LIMIT
, the script would show the correct value. With this update, therun.sh
script now consistently displays the correctchunk_limit_size
value in both scenarios. (LOG-3330) - Before this update, the Red Hat OpenShift Service on AWS web console’s logging view plugin did not allow for custom node placement or tolerations. This update adds the ability to define node placement and tolerations for the logging view plugin. (LOG-3749)
-
Before this update, the Cluster Logging Operator encountered an Unsupported Media Type exception when trying to send logs to DataDog via the Fluentd HTTP Plugin. With this update, users can seamlessly assign the content type for log forwarding by configuring the HTTP header Content-Type. The value provided is automatically assigned to the
content_type
parameter within the plugin, ensuring successful log transmission. (LOG-3784) -
Before this update, when the
detectMultilineErrors
field was set totrue
in theClusterLogForwarder
custom resource (CR), PHP multi-line errors were recorded as separate log entries, causing the stack trace to be split across multiple messages. With this update, multi-line error detection for PHP is enabled, ensuring that the entire stack trace is included in a single log message. (LOG-3878) -
Before this update,
ClusterLogForwarder
pipelines containing a space in their name caused the Vector collector pods to continuously crash. With this update, all spaces, dashes (-), and dots (.) in pipeline names are replaced with underscores (_). (LOG-3945) -
Before this update, the
log_forwarder_output
metric did not include thehttp
parameter. This update adds the missing parameter to the metric. (LOG-3997) - Before this update, Fluentd did not identify some multi-line JavaScript client exceptions when they ended with a colon. With this update, the Fluentd buffer name is prefixed with an underscore, resolving the issue. (LOG-4019)
- Before this update, when configuring log forwarding to write to a Kafka output topic which matched a key in the payload, logs dropped due to an error. With this update, Fluentd’s buffer name has been prefixed with an underscore, resolving the issue.(LOG-4027)
- Before this update, the LokiStack gateway returned label values for namespaces without applying the access rights of a user. With this update, the LokiStack gateway applies permissions to label value requests, resolving the issue. (LOG-4049)
Before this update, the Cluster Logging Operator API required a certificate to be provided by a secret when the
tls.insecureSkipVerify
option was set totrue
. With this update, the Cluster Logging Operator API no longer requires a certificate to be provided by a secret in such cases. The following configuration has been added to the Operator’s CR:tls.verify_certificate = false tls.verify_hostname = false
(LOG-3445)
-
Before this update, the LokiStack route configuration caused queries running longer than 30 seconds to timeout. With this update, the LokiStack global and per-tenant
queryTimeout
settings affect the route timeout settings, resolving the issue. (LOG-4052) -
Before this update, a prior fix to remove defaulting of the
collection.type
resulted in the Operator no longer honoring the deprecated specs for resource, node selections, and tolerations. This update modifies the Operator behavior to always prefer thecollection.logs
spec over those ofcollection
. This varies from previous behavior that allowed using both the preferred fields and deprecated fields but would ignore the deprecated fields whencollection.type
was populated. (LOG-4185) - Before this update, the Vector log collector did not generate TLS configuration for forwarding logs to multiple Kafka brokers if the broker URLs were not specified in the output. With this update, TLS configuration is generated appropriately for multiple brokers. (LOG-4163)
- Before this update, the option to enable passphrase for log forwarding to Kafka was unavailable. This limitation presented a security risk as it could potentially expose sensitive information. With this update, users now have a seamless option to enable passphrase for log forwarding to Kafka. (LOG-3314)
-
Before this update, Vector log collector did not honor the
tlsSecurityProfile
settings for outgoing TLS connections. After this update, Vector handles TLS connection settings appropriately. (LOG-4011) -
Before this update, not all available output types were included in the
log_forwarder_output_info
metrics. With this update, metrics contain Splunk and Google Cloud Logging data which was missing previously. (LOG-4098) -
Before this update, when
follow_inodes
was set totrue
, the Fluentd collector could crash on file rotation. With this update, thefollow_inodes
setting does not crash the collector. (LOG-4151) - Before this update, the Fluentd collector could incorrectly close files that should be watched because of how those files were tracked. With this update, the tracking parameters have been corrected. (LOG-4149)
Before this update, forwarding logs with the Vector collector and naming a pipeline in the
ClusterLogForwarder
instanceaudit
,application
orinfrastructure
resulted in collector pods staying in theCrashLoopBackOff
state with the following error in the collector log:ERROR vector::cli: Configuration error. error=redefinition of table transforms.audit for key transforms.audit
After this update, pipeline names no longer clash with reserved input names, and pipelines can be named
audit
,application
orinfrastructure
. (LOG-4218)-
Before this update, when forwarding logs to a syslog destination with the Vector collector and setting the
addLogSource
flag totrue
, the following extra empty fields were added to the forwarded messages:namespace_name=
,container_name=
, andpod_name=
. With this update, these fields are no longer added to journal logs. (LOG-4219) -
Before this update, when a
structuredTypeKey
was not found, and astructuredTypeName
was not specified, log messages were still parsed into structured object. With this update, parsing of logs is as expected. (LOG-4220)
1.3.6.2. CVEs
- CVE-2021-26341
- CVE-2021-33655
- CVE-2021-33656
- CVE-2022-1462
- CVE-2022-1679
- CVE-2022-1789
- CVE-2022-2196
- CVE-2022-2663
- CVE-2022-3028
- CVE-2022-3239
- CVE-2022-3522
- CVE-2022-3524
- CVE-2022-3564
- CVE-2022-3566
- CVE-2022-3567
- CVE-2022-3619
- CVE-2022-3623
- CVE-2022-3625
- CVE-2022-3627
- CVE-2022-3628
- CVE-2022-3707
- CVE-2022-3970
- CVE-2022-4129
- CVE-2022-20141
- CVE-2022-25147
- CVE-2022-25265
- CVE-2022-30594
- CVE-2022-36227
- CVE-2022-39188
- CVE-2022-39189
- CVE-2022-41218
- CVE-2022-41674
- CVE-2022-42703
- CVE-2022-42720
- CVE-2022-42721
- CVE-2022-42722
- CVE-2022-43750
- CVE-2022-47929
- CVE-2023-0394
- CVE-2023-0461
- CVE-2023-1195
- CVE-2023-1582
- CVE-2023-2491
- CVE-2023-22490
- CVE-2023-23454
- CVE-2023-23946
- CVE-2023-25652
- CVE-2023-25815
- CVE-2023-27535
- CVE-2023-29007
1.3.7. Logging 5.7.1
This release includes: OpenShift Logging Bug Fix Release 5.7.1.
1.3.7.1. Bug fixes
- Before this update, the presence of numerous noisy messages within the Cluster Logging Operator pod logs caused reduced log readability, and increased difficulty in identifying important system events. With this update, the issue is resolved by significantly reducing the noisy messages within Cluster Logging Operator pod logs. (LOG-3482)
-
Before this update, the API server would reset the value for the
CollectorSpec.Type
field tovector
, even when the custom resource used a different value. This update removes the default for theCollectorSpec.Type
field to restore the previous behavior. (LOG-4086) - Before this update, a time range could not be selected in the Red Hat OpenShift Service on AWS web console by clicking and dragging over the logs histogram. With this update, clicking and dragging can be used to successfully select a time range. (LOG-4501)
- Before this update, clicking on the Show Resources link in the Red Hat OpenShift Service on AWS web console did not produce any effect. With this update, the issue is resolved by fixing the functionality of the "Show Resources" link to toggle the display of resources for each log entry. (LOG-3218)
1.3.7.2. CVEs
1.3.8. Logging 5.7.0
This release includes OpenShift Logging Bug Fix Release 5.7.0.
1.3.8.1. Enhancements
With this update, you can enable logging to detect multi-line exceptions and reassemble them into a single log entry.
To enable logging to detect multi-line exceptions and reassemble them into a single log entry, ensure that the ClusterLogForwarder
Custom Resource (CR) contains a detectMultilineErrors
field, with a value of true
.
1.3.8.2. Known Issues
None.
1.3.8.3. Bug fixes
-
Before this update, the
nodeSelector
attribute for the Gateway component of the LokiStack did not impact node scheduling. With this update, thenodeSelector
attribute works as expected. (LOG-3713)
1.3.8.4. CVEs
Chapter 2. Support
Only the configuration options described in this documentation are supported for logging.
Do not use any other configuration options, as they are unsupported. Configuration paradigms might change across Red Hat OpenShift Service on AWS releases, and such cases can only be handled gracefully if all configuration possibilities are controlled. If you use configurations other than those described in this documentation, your changes will be overwritten, because Operators are designed to reconcile any differences.
If you must perform configurations not described in the Red Hat OpenShift Service on AWS documentation, you must set your Red Hat OpenShift Logging Operator to Unmanaged
. An unmanaged logging instance is not supported and does not receive updates until you return its status to Managed
.
Logging is provided as an installable component, with a distinct release cycle from the core Red Hat OpenShift Service on AWS. The Red Hat OpenShift Container Platform Life Cycle Policy outlines release compatibility.
Loki is a horizontally scalable, highly available, multi-tenant log aggregation system offered as a GA log store for logging for Red Hat OpenShift that can be visualized with the OpenShift Observability UI. The Loki configuration provided by OpenShift Logging is a short-term log store designed to enable users to perform fast troubleshooting with the collected logs. For that purpose, the logging for Red Hat OpenShift configuration of Loki has short-term storage, and is optimized for very recent queries.
For long-term storage or queries over a long time period, users should look to log stores external to their cluster. Loki sizing is only tested and supported for short term storage, for a maximum of 30 days.
Logging for Red Hat OpenShift is an opinionated collector and normalizer of application, infrastructure, and audit logs. It is intended to be used for forwarding logs to various supported systems.
Logging is not:
- A high scale log collection system
- Security Information and Event Monitoring (SIEM) compliant
- Historical or long term log retention or storage
- A guaranteed log sink
- Secure storage - audit logs are not stored by default
2.1. Supported API custom resource definitions
LokiStack development is ongoing. Not all APIs are currently supported.
CustomResourceDefinition (CRD) | ApiVersion | Support state |
---|---|---|
LokiStack | lokistack.loki.grafana.com/v1 | Supported in 5.5 |
RulerConfig | rulerconfig.loki.grafana/v1 | Supported in 5.7 |
AlertingRule | alertingrule.loki.grafana/v1 | Supported in 5.7 |
RecordingRule | recordingrule.loki.grafana/v1 | Supported in 5.7 |
2.2. Unsupported configurations
You must set the Red Hat OpenShift Logging Operator to the Unmanaged
state to modify the following components:
-
The
Elasticsearch
custom resource (CR) - The Kibana deployment
-
The
fluent.conf
file - The Fluentd daemon set
You must set the OpenShift Elasticsearch Operator to the Unmanaged
state to modify the Elasticsearch deployment files.
Explicitly unsupported cases include:
- Configuring default log rotation. You cannot modify the default log rotation configuration.
-
Configuring the collected log location. You cannot change the location of the log collector output file, which by default is
/var/log/fluentd/fluentd.log
. - Throttling log collection. You cannot throttle down the rate at which the logs are read in by the log collector.
- Configuring the logging collector using environment variables. You cannot use environment variables to modify the log collector.
- Configuring how the log collector normalizes logs. You cannot modify default log normalization.
2.3. Support policy for unmanaged Operators
The management state of an Operator determines whether an Operator is actively managing the resources for its related component in the cluster as designed. If an Operator is set to an unmanaged state, it does not respond to changes in configuration nor does it receive updates.
While this can be helpful in non-production clusters or during debugging, Operators in an unmanaged state are unsupported and the cluster administrator assumes full control of the individual component configurations and upgrades.
An Operator can be set to an unmanaged state using the following methods:
Individual Operator configuration
Individual Operators have a
managementState
parameter in their configuration. This can be accessed in different ways, depending on the Operator. For example, the Red Hat OpenShift Logging Operator accomplishes this by modifying a custom resource (CR) that it manages, while the Cluster Samples Operator uses a cluster-wide configuration resource.Changing the
managementState
parameter toUnmanaged
means that the Operator is not actively managing its resources and will take no action related to the related component. Some Operators might not support this management state as it might damage the cluster and require manual recovery.WarningChanging individual Operators to the
Unmanaged
state renders that particular component and functionality unsupported. Reported issues must be reproduced inManaged
state for support to proceed.Cluster Version Operator (CVO) overrides
The
spec.overrides
parameter can be added to the CVO’s configuration to allow administrators to provide a list of overrides to the CVO’s behavior for a component. Setting thespec.overrides[].unmanaged
parameter totrue
for a component blocks cluster upgrades and alerts the administrator after a CVO override has been set:Disabling ownership via cluster version overrides prevents upgrades. Please remove overrides before continuing.
WarningSetting a CVO override puts the entire cluster in an unsupported state. Reported issues must be reproduced after removing any overrides for support to proceed.
2.4. Support exception for the Logging UI Plugin
Until the approaching General Availability (GA) release of the Cluster Observability Operator (COO), which is currently in Technology Preview (TP), Red Hat provides support to customers who are using Logging 6.0 or later with the COO for its Logging UI Plugin on Red Hat OpenShift Service on AWS 4.14 or later. This support exception is temporary as the COO includes several independent features, some of which are still TP features, but the Logging UI Plugin is ready for GA.
2.5. Collecting logging data for Red Hat Support
When opening a support case, it is helpful to provide debugging information about your cluster to Red Hat Support.
You can use the must-gather
tool to collect diagnostic information for project-level resources, cluster-level resources, and each of the logging components.
For prompt support, supply diagnostic information for both Red Hat OpenShift Service on AWS and logging.
Do not use the hack/logging-dump.sh
script. The script is no longer supported and does not collect data.
2.5.1. About the must-gather tool
The oc adm must-gather
CLI command collects the information from your cluster that is most likely needed for debugging issues.
For your logging, must-gather
collects the following information:
- Project-level resources, including pods, configuration maps, service accounts, roles, role bindings, and events at the project level
- Cluster-level resources, including nodes, roles, and role bindings at the cluster level
-
OpenShift Logging resources in the
openshift-logging
andopenshift-operators-redhat
namespaces, including health status for the log collector, the log store, and the log visualizer
When you run oc adm must-gather
, a new pod is created on the cluster. The data is collected on that pod and saved in a new directory that starts with must-gather.local
. This directory is created in the current working directory.
2.5.2. Collecting logging data
You can use the oc adm must-gather
CLI command to collect information about logging.
Procedure
To collect logging information with must-gather
:
-
Navigate to the directory where you want to store the
must-gather
information. Run the
oc adm must-gather
command against the logging image:$ oc adm must-gather --image=$(oc -n openshift-logging get deployment.apps/cluster-logging-operator -o jsonpath='{.spec.template.spec.containers[?(@.name == "cluster-logging-operator")].image}')
The
must-gather
tool creates a new directory that starts withmust-gather.local
within the current directory. For example:must-gather.local.4157245944708210408
.Create a compressed file from the
must-gather
directory that was just created. For example, on a computer that uses a Linux operating system, run the following command:$ tar -cvaf must-gather.tar.gz must-gather.local.4157245944708210408
- Attach the compressed file to your support case on the Red Hat Customer Portal.
Chapter 3. Troubleshooting logging
3.1. Viewing Logging status
You can view the status of the Red Hat OpenShift Logging Operator and other logging components.
3.1.1. Viewing the status of the Red Hat OpenShift Logging Operator
You can view the status of the Red Hat OpenShift Logging Operator.
Prerequisites
- The Red Hat OpenShift Logging Operator and OpenShift Elasticsearch Operator are installed.
Procedure
Change to the
openshift-logging
project by running the following command:$ oc project openshift-logging
Get the
ClusterLogging
instance status by running the following command:$ oc get clusterlogging instance -o yaml
Example output
apiVersion: logging.openshift.io/v1 kind: ClusterLogging # ... status: 1 collection: logs: fluentdStatus: daemonSet: fluentd 2 nodes: collector-2rhqp: ip-10-0-169-13.ec2.internal collector-6fgjh: ip-10-0-165-244.ec2.internal collector-6l2ff: ip-10-0-128-218.ec2.internal collector-54nx5: ip-10-0-139-30.ec2.internal collector-flpnn: ip-10-0-147-228.ec2.internal collector-n2frh: ip-10-0-157-45.ec2.internal pods: failed: [] notReady: [] ready: - collector-2rhqp - collector-54nx5 - collector-6fgjh - collector-6l2ff - collector-flpnn - collector-n2frh logstore: 3 elasticsearchStatus: - ShardAllocationEnabled: all cluster: activePrimaryShards: 5 activeShards: 5 initializingShards: 0 numDataNodes: 1 numNodes: 1 pendingTasks: 0 relocatingShards: 0 status: green unassignedShards: 0 clusterName: elasticsearch nodeConditions: elasticsearch-cdm-mkkdys93-1: nodeCount: 1 pods: client: failed: notReady: ready: - elasticsearch-cdm-mkkdys93-1-7f7c6-mjm7c data: failed: notReady: ready: - elasticsearch-cdm-mkkdys93-1-7f7c6-mjm7c master: failed: notReady: ready: - elasticsearch-cdm-mkkdys93-1-7f7c6-mjm7c visualization: 4 kibanaStatus: - deployment: kibana pods: failed: [] notReady: [] ready: - kibana-7fb4fd4cc9-f2nls replicaSets: - kibana-7fb4fd4cc9 replicas: 1
3.1.1.1. Example condition messages
The following are examples of some condition messages from the Status.Nodes
section of the ClusterLogging
instance.
A status message similar to the following indicates a node has exceeded the configured low watermark and no shard will be allocated to this node:
Example output
nodes: - conditions: - lastTransitionTime: 2019-03-15T15:57:22Z message: Disk storage usage for node is 27.5gb (36.74%). Shards will be not be allocated on this node. reason: Disk Watermark Low status: "True" type: NodeStorage deploymentName: example-elasticsearch-clientdatamaster-0-1 upgradeStatus: {}
A status message similar to the following indicates a node has exceeded the configured high watermark and shards will be relocated to other nodes:
Example output
nodes: - conditions: - lastTransitionTime: 2019-03-15T16:04:45Z message: Disk storage usage for node is 27.5gb (36.74%). Shards will be relocated from this node. reason: Disk Watermark High status: "True" type: NodeStorage deploymentName: cluster-logging-operator upgradeStatus: {}
A status message similar to the following indicates the Elasticsearch node selector in the CR does not match any nodes in the cluster:
Example output
Elasticsearch Status: Shard Allocation Enabled: shard allocation unknown Cluster: Active Primary Shards: 0 Active Shards: 0 Initializing Shards: 0 Num Data Nodes: 0 Num Nodes: 0 Pending Tasks: 0 Relocating Shards: 0 Status: cluster health unknown Unassigned Shards: 0 Cluster Name: elasticsearch Node Conditions: elasticsearch-cdm-mkkdys93-1: Last Transition Time: 2019-06-26T03:37:32Z Message: 0/5 nodes are available: 5 node(s) didn't match node selector. Reason: Unschedulable Status: True Type: Unschedulable elasticsearch-cdm-mkkdys93-2: Node Count: 2 Pods: Client: Failed: Not Ready: elasticsearch-cdm-mkkdys93-1-75dd69dccd-f7f49 elasticsearch-cdm-mkkdys93-2-67c64f5f4c-n58vl Ready: Data: Failed: Not Ready: elasticsearch-cdm-mkkdys93-1-75dd69dccd-f7f49 elasticsearch-cdm-mkkdys93-2-67c64f5f4c-n58vl Ready: Master: Failed: Not Ready: elasticsearch-cdm-mkkdys93-1-75dd69dccd-f7f49 elasticsearch-cdm-mkkdys93-2-67c64f5f4c-n58vl Ready:
A status message similar to the following indicates that the requested PVC could not bind to PV:
Example output
Node Conditions: elasticsearch-cdm-mkkdys93-1: Last Transition Time: 2019-06-26T03:37:32Z Message: pod has unbound immediate PersistentVolumeClaims (repeated 5 times) Reason: Unschedulable Status: True Type: Unschedulable
A status message similar to the following indicates that the Fluentd pods cannot be scheduled because the node selector did not match any nodes:
Example output
Status: Collection: Logs: Fluentd Status: Daemon Set: fluentd Nodes: Pods: Failed: Not Ready: Ready:
3.1.2. Viewing the status of logging components
You can view the status for a number of logging components.
Prerequisites
- The Red Hat OpenShift Logging Operator and OpenShift Elasticsearch Operator are installed.
Procedure
Change to the
openshift-logging
project.$ oc project openshift-logging
View the status of logging environment:
$ oc describe deployment cluster-logging-operator
Example output
Name: cluster-logging-operator .... Conditions: Type Status Reason ---- ------ ------ Available True MinimumReplicasAvailable Progressing True NewReplicaSetAvailable .... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ScalingReplicaSet 62m deployment-controller Scaled up replica set cluster-logging-operator-574b8987df to 1----
View the status of the logging replica set:
Get the name of a replica set:
Example output
$ oc get replicaset
Example output
NAME DESIRED CURRENT READY AGE cluster-logging-operator-574b8987df 1 1 1 159m elasticsearch-cdm-uhr537yu-1-6869694fb 1 1 1 157m elasticsearch-cdm-uhr537yu-2-857b6d676f 1 1 1 156m elasticsearch-cdm-uhr537yu-3-5b6fdd8cfd 1 1 1 155m kibana-5bd5544f87 1 1 1 157m
Get the status of the replica set:
$ oc describe replicaset cluster-logging-operator-574b8987df
Example output
Name: cluster-logging-operator-574b8987df .... Replicas: 1 current / 1 desired Pods Status: 1 Running / 0 Waiting / 0 Succeeded / 0 Failed .... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal SuccessfulCreate 66m replicaset-controller Created pod: cluster-logging-operator-574b8987df-qjhqv----
3.2. Troubleshooting log forwarding
3.2.1. Redeploying Fluentd pods
When you create a ClusterLogForwarder
custom resource (CR), if the Red Hat OpenShift Logging Operator does not redeploy the Fluentd pods automatically, you can delete the Fluentd pods to force them to redeploy.
Prerequisites
-
You have created a
ClusterLogForwarder
custom resource (CR) object.
Procedure
Delete the Fluentd pods to force them to redeploy by running the following command:
$ oc delete pod --selector logging-infra=collector
3.2.2. Troubleshooting Loki rate limit errors
If the Log Forwarder API forwards a large block of messages that exceeds the rate limit to Loki, Loki generates rate limit (429
) errors.
These errors can occur during normal operation. For example, when adding the logging to a cluster that already has some logs, rate limit errors might occur while the logging tries to ingest all of the existing log entries. In this case, if the rate of addition of new logs is less than the total rate limit, the historical data is eventually ingested, and the rate limit errors are resolved without requiring user intervention.
In cases where the rate limit errors continue to occur, you can fix the issue by modifying the LokiStack
custom resource (CR).
The LokiStack
CR is not available on Grafana-hosted Loki. This topic does not apply to Grafana-hosted Loki servers.
Conditions
- The Log Forwarder API is configured to forward logs to Loki.
Your system sends a block of messages that is larger than 2 MB to Loki. For example:
"values":[["1630410392689800468","{\"kind\":\"Event\",\"apiVersion\":\ ....... ...... ...... ...... \"received_at\":\"2021-08-31T11:46:32.800278+00:00\",\"version\":\"1.7.4 1.6.0\"}},\"@timestamp\":\"2021-08-31T11:46:32.799692+00:00\",\"viaq_index_name\":\"audit-write\",\"viaq_msg_id\":\"MzFjYjJkZjItNjY0MC00YWU4LWIwMTEtNGNmM2E5ZmViMGU4\",\"log_type\":\"audit\"}"]]}]}
After you enter
oc logs -n openshift-logging -l component=collector
, the collector logs in your cluster show a line containing one of the following error messages:429 Too Many Requests Ingestion rate limit exceeded
Example Vector error message
2023-08-25T16:08:49.301780Z WARN sink{component_kind="sink" component_id=default_loki_infra component_type=loki component_name=default_loki_infra}: vector::sinks::util::retries: Retrying after error. error=Server responded with an error: 429 Too Many Requests internal_log_rate_limit=true
Example Fluentd error message
2023-08-30 14:52:15 +0000 [warn]: [default_loki_infra] failed to flush the buffer. retry_times=2 next_retry_time=2023-08-30 14:52:19 +0000 chunk="604251225bf5378ed1567231a1c03b8b" error_class=Fluent::Plugin::LokiOutput::LogPostError error="429 Too Many Requests Ingestion rate limit exceeded for user infrastructure (limit: 4194304 bytes/sec) while attempting to ingest '4082' lines totaling '7820025' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased\n"
The error is also visible on the receiving end. For example, in the LokiStack ingester pod:
Example Loki ingester error message
level=warn ts=2023-08-30T14:57:34.155592243Z caller=grpc_logging.go:43 duration=1.434942ms method=/logproto.Pusher/Push err="rpc error: code = Code(429) desc = entry with timestamp 2023-08-30 14:57:32.012778399 +0000 UTC ignored, reason: 'Per stream rate limit exceeded (limit: 3MB/sec) while attempting to ingest for stream
Procedure
Update the
ingestionBurstSize
andingestionRate
fields in theLokiStack
CR:apiVersion: loki.grafana.com/v1 kind: LokiStack metadata: name: logging-loki namespace: openshift-logging spec: limits: global: ingestion: ingestionBurstSize: 16 1 ingestionRate: 8 2 # ...
- 1
- The
ingestionBurstSize
field defines the maximum local rate-limited sample size per distributor replica in MB. This value is a hard limit. Set this value to at least the maximum logs size expected in a single push request. Single requests that are larger than theingestionBurstSize
value are not permitted. - 2
- The
ingestionRate
field is a soft limit on the maximum amount of ingested samples per second in MB. Rate limit errors occur if the rate of logs exceeds the limit, but the collector retries sending the logs. As long as the total average is lower than the limit, the system recovers and errors are resolved without user intervention.
3.3. Troubleshooting logging alerts
You can use the following procedures to troubleshoot logging alerts on your cluster.
3.3.1. Elasticsearch cluster health status is red
At least one primary shard and its replicas are not allocated to a node. Use the following procedure to troubleshoot this alert.
Some commands in this documentation reference an Elasticsearch pod by using a $ES_POD_NAME
shell variable. If you want to copy and paste the commands directly from this documentation, you must set this variable to a value that is valid for your Elasticsearch cluster.
You can list the available Elasticsearch pods by running the following command:
$ oc -n openshift-logging get pods -l component=elasticsearch
Choose one of the pods listed and set the $ES_POD_NAME
variable, by running the following command:
$ export ES_POD_NAME=<elasticsearch_pod_name>
You can now use the $ES_POD_NAME
variable in commands.
Procedure
Check the Elasticsearch cluster health and verify that the cluster
status
is red by running the following command:$ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME -- health
List the nodes that have joined the cluster by running the following command:
$ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \ -- es_util --query=_cat/nodes?v
List the Elasticsearch pods and compare them with the nodes in the command output from the previous step, by running the following command:
$ oc -n openshift-logging get pods -l component=elasticsearch
If some of the Elasticsearch nodes have not joined the cluster, perform the following steps.
Confirm that Elasticsearch has an elected master node by running the following command and observing the output:
$ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \ -- es_util --query=_cat/master?v
Review the pod logs of the elected master node for issues by running the following command and observing the output:
$ oc logs <elasticsearch_master_pod_name> -c elasticsearch -n openshift-logging
Review the logs of nodes that have not joined the cluster for issues by running the following command and observing the output:
$ oc logs <elasticsearch_node_name> -c elasticsearch -n openshift-logging
If all the nodes have joined the cluster, check if the cluster is in the process of recovering by running the following command and observing the output:
$ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \ -- es_util --query=_cat/recovery?active_only=true
If there is no command output, the recovery process might be delayed or stalled by pending tasks.
Check if there are pending tasks by running the following command and observing the output:
$ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \ -- health | grep number_of_pending_tasks
- If there are pending tasks, monitor their status. If their status changes and indicates that the cluster is recovering, continue waiting. The recovery time varies according to the size of the cluster and other factors. Otherwise, if the status of the pending tasks does not change, this indicates that the recovery has stalled.
If it seems like the recovery has stalled, check if the
cluster.routing.allocation.enable
value is set tonone
, by running the following command and observing the output:$ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \ -- es_util --query=_cluster/settings?pretty
If the
cluster.routing.allocation.enable
value is set tonone
, set it toall
, by running the following command:$ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \ -- es_util --query=_cluster/settings?pretty \ -X PUT -d '{"persistent": {"cluster.routing.allocation.enable":"all"}}'
Check if any indices are still red by running the following command and observing the output:
$ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \ -- es_util --query=_cat/indices?v
If any indices are still red, try to clear them by performing the following steps.
Clear the cache by running the following command:
$ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \ -- es_util --query=<elasticsearch_index_name>/_cache/clear?pretty
Increase the max allocation retries by running the following command:
$ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \ -- es_util --query=<elasticsearch_index_name>/_settings?pretty \ -X PUT -d '{"index.allocation.max_retries":10}'
Delete all the scroll items by running the following command:
$ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \ -- es_util --query=_search/scroll/_all -X DELETE
Increase the timeout by running the following command:
$ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \ -- es_util --query=<elasticsearch_index_name>/_settings?pretty \ -X PUT -d '{"index.unassigned.node_left.delayed_timeout":"10m"}'
If the preceding steps do not clear the red indices, delete the indices individually.
Identify the red index name by running the following command:
$ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \ -- es_util --query=_cat/indices?v
Delete the red index by running the following command:
$ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \ -- es_util --query=<elasticsearch_red_index_name> -X DELETE
If there are no red indices and the cluster status is red, check for a continuous heavy processing load on a data node.
Check if the Elasticsearch JVM Heap usage is high by running the following command:
$ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \ -- es_util --query=_nodes/stats?pretty
In the command output, review the
node_name.jvm.mem.heap_used_percent
field to determine the JVM Heap usage.- Check for high CPU utilization. For more information about CPU utilitzation, see the Red Hat OpenShift Service on AWS "Reviewing monitoring dashboards" documentation.
Additional resources
3.3.2. Elasticsearch cluster health status is yellow
Replica shards for at least one primary shard are not allocated to nodes. Increase the node count by adjusting the nodeCount
value in the ClusterLogging
custom resource (CR).
Additional resources
3.3.3. Elasticsearch node disk low watermark reached
Elasticsearch does not allocate shards to nodes that reach the low watermark.
Some commands in this documentation reference an Elasticsearch pod by using a $ES_POD_NAME
shell variable. If you want to copy and paste the commands directly from this documentation, you must set this variable to a value that is valid for your Elasticsearch cluster.
You can list the available Elasticsearch pods by running the following command:
$ oc -n openshift-logging get pods -l component=elasticsearch
Choose one of the pods listed and set the $ES_POD_NAME
variable, by running the following command:
$ export ES_POD_NAME=<elasticsearch_pod_name>
You can now use the $ES_POD_NAME
variable in commands.
Procedure
Identify the node on which Elasticsearch is deployed by running the following command:
$ oc -n openshift-logging get po -o wide
Check if there are unassigned shards by running the following command:
$ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \ -- es_util --query=_cluster/health?pretty | grep unassigned_shards
If there are unassigned shards, check the disk space on each node, by running the following command:
$ for pod in `oc -n openshift-logging get po -l component=elasticsearch -o jsonpath='{.items[*].metadata.name}'`; \ do echo $pod; oc -n openshift-logging exec -c elasticsearch $pod \ -- df -h /elasticsearch/persistent; done
In the command output, check the
Use
column to determine the used disk percentage on that node.Example output
elasticsearch-cdm-kcrsda6l-1-586cc95d4f-h8zq8 Filesystem Size Used Avail Use% Mounted on /dev/nvme1n1 19G 522M 19G 3% /elasticsearch/persistent elasticsearch-cdm-kcrsda6l-2-5b548fc7b-cwwk7 Filesystem Size Used Avail Use% Mounted on /dev/nvme2n1 19G 522M 19G 3% /elasticsearch/persistent elasticsearch-cdm-kcrsda6l-3-5dfc884d99-59tjw Filesystem Size Used Avail Use% Mounted on /dev/nvme3n1 19G 528M 19G 3% /elasticsearch/persistent
If the used disk percentage is above 85%, the node has exceeded the low watermark, and shards can no longer be allocated to this node.
To check the current
redundancyPolicy
, run the following command:$ oc -n openshift-logging get es elasticsearch \ -o jsonpath='{.spec.redundancyPolicy}'
If you are using a
ClusterLogging
resource on your cluster, run the following command:$ oc -n openshift-logging get cl \ -o jsonpath='{.items[*].spec.logStore.elasticsearch.redundancyPolicy}'
If the cluster
redundancyPolicy
value is higher than theSingleRedundancy
value, set it to theSingleRedundancy
value and save this change.If the preceding steps do not fix the issue, delete the old indices.
Check the status of all indices on Elasticsearch by running the following command:
$ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME -- indices
- Identify an old index that can be deleted.
Delete the index by running the following command:
$ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \ -- es_util --query=<elasticsearch_index_name> -X DELETE
3.3.4. Elasticsearch node disk high watermark reached
Elasticsearch attempts to relocate shards away from a node that has reached the high watermark to a node with low disk usage that has not crossed any watermark threshold limits.
To allocate shards to a particular node, you must free up some space on that node. If increasing the disk space is not possible, try adding a new data node to the cluster, or decrease the total cluster redundancy policy.
Some commands in this documentation reference an Elasticsearch pod by using a $ES_POD_NAME
shell variable. If you want to copy and paste the commands directly from this documentation, you must set this variable to a value that is valid for your Elasticsearch cluster.
You can list the available Elasticsearch pods by running the following command:
$ oc -n openshift-logging get pods -l component=elasticsearch
Choose one of the pods listed and set the $ES_POD_NAME
variable, by running the following command:
$ export ES_POD_NAME=<elasticsearch_pod_name>
You can now use the $ES_POD_NAME
variable in commands.
Procedure
Identify the node on which Elasticsearch is deployed by running the following command:
$ oc -n openshift-logging get po -o wide
Check the disk space on each node:
$ for pod in `oc -n openshift-logging get po -l component=elasticsearch -o jsonpath='{.items[*].metadata.name}'`; \ do echo $pod; oc -n openshift-logging exec -c elasticsearch $pod \ -- df -h /elasticsearch/persistent; done
Check if the cluster is rebalancing:
$ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \ -- es_util --query=_cluster/health?pretty | grep relocating_shards
If the command output shows relocating shards, the high watermark has been exceeded. The default value of the high watermark is 90%.
- Increase the disk space on all nodes. If increasing the disk space is not possible, try adding a new data node to the cluster, or decrease the total cluster redundancy policy.
To check the current
redundancyPolicy
, run the following command:$ oc -n openshift-logging get es elasticsearch \ -o jsonpath='{.spec.redundancyPolicy}'
If you are using a
ClusterLogging
resource on your cluster, run the following command:$ oc -n openshift-logging get cl \ -o jsonpath='{.items[*].spec.logStore.elasticsearch.redundancyPolicy}'
If the cluster
redundancyPolicy
value is higher than theSingleRedundancy
value, set it to theSingleRedundancy
value and save this change.If the preceding steps do not fix the issue, delete the old indices.
Check the status of all indices on Elasticsearch by running the following command:
$ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME -- indices
- Identify an old index that can be deleted.
Delete the index by running the following command:
$ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \ -- es_util --query=<elasticsearch_index_name> -X DELETE
3.3.5. Elasticsearch node disk flood watermark reached
Elasticsearch enforces a read-only index block on every index that has both of these conditions:
- One or more shards are allocated to the node.
- One or more disks exceed the flood stage.
Use the following procedure to troubleshoot this alert.
Some commands in this documentation reference an Elasticsearch pod by using a $ES_POD_NAME
shell variable. If you want to copy and paste the commands directly from this documentation, you must set this variable to a value that is valid for your Elasticsearch cluster.
You can list the available Elasticsearch pods by running the following command:
$ oc -n openshift-logging get pods -l component=elasticsearch
Choose one of the pods listed and set the $ES_POD_NAME
variable, by running the following command:
$ export ES_POD_NAME=<elasticsearch_pod_name>
You can now use the $ES_POD_NAME
variable in commands.
Procedure
Get the disk space of the Elasticsearch node:
$ for pod in `oc -n openshift-logging get po -l component=elasticsearch -o jsonpath='{.items[*].metadata.name}'`; \ do echo $pod; oc -n openshift-logging exec -c elasticsearch $pod \ -- df -h /elasticsearch/persistent; done
In the command output, check the
Avail
column to determine the free disk space on that node.Example output
elasticsearch-cdm-kcrsda6l-1-586cc95d4f-h8zq8 Filesystem Size Used Avail Use% Mounted on /dev/nvme1n1 19G 522M 19G 3% /elasticsearch/persistent elasticsearch-cdm-kcrsda6l-2-5b548fc7b-cwwk7 Filesystem Size Used Avail Use% Mounted on /dev/nvme2n1 19G 522M 19G 3% /elasticsearch/persistent elasticsearch-cdm-kcrsda6l-3-5dfc884d99-59tjw Filesystem Size Used Avail Use% Mounted on /dev/nvme3n1 19G 528M 19G 3% /elasticsearch/persistent
- Increase the disk space on all nodes. If increasing the disk space is not possible, try adding a new data node to the cluster, or decrease the total cluster redundancy policy.
To check the current
redundancyPolicy
, run the following command:$ oc -n openshift-logging get es elasticsearch \ -o jsonpath='{.spec.redundancyPolicy}'
If you are using a
ClusterLogging
resource on your cluster, run the following command:$ oc -n openshift-logging get cl \ -o jsonpath='{.items[*].spec.logStore.elasticsearch.redundancyPolicy}'
If the cluster
redundancyPolicy
value is higher than theSingleRedundancy
value, set it to theSingleRedundancy
value and save this change.If the preceding steps do not fix the issue, delete the old indices.
Check the status of all indices on Elasticsearch by running the following command:
$ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME -- indices
- Identify an old index that can be deleted.
Delete the index by running the following command:
$ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \ -- es_util --query=<elasticsearch_index_name> -X DELETE
Continue freeing up and monitoring the disk space. After the used disk space drops below 90%, unblock writing to this node by running the following command:
$ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \ -- es_util --query=_all/_settings?pretty \ -X PUT -d '{"index.blocks.read_only_allow_delete": null}'
3.3.6. Elasticsearch JVM heap usage is high
The Elasticsearch node Java virtual machine (JVM) heap memory used is above 75%. Consider increasing the heap size.
3.3.7. Aggregated logging system CPU is high
System CPU usage on the node is high. Check the CPU of the cluster node. Consider allocating more CPU resources to the node.
3.3.8. Elasticsearch process CPU is high
Elasticsearch process CPU usage on the node is high. Check the CPU of the cluster node. Consider allocating more CPU resources to the node.
3.3.9. Elasticsearch disk space is running low
Elasticsearch is predicted to run out of disk space within the next 6 hours based on current disk usage. Use the following procedure to troubleshoot this alert.
Procedure
Get the disk space of the Elasticsearch node:
$ for pod in `oc -n openshift-logging get po -l component=elasticsearch -o jsonpath='{.items[*].metadata.name}'`; \ do echo $pod; oc -n openshift-logging exec -c elasticsearch $pod \ -- df -h /elasticsearch/persistent; done
In the command output, check the
Avail
column to determine the free disk space on that node.Example output
elasticsearch-cdm-kcrsda6l-1-586cc95d4f-h8zq8 Filesystem Size Used Avail Use% Mounted on /dev/nvme1n1 19G 522M 19G 3% /elasticsearch/persistent elasticsearch-cdm-kcrsda6l-2-5b548fc7b-cwwk7 Filesystem Size Used Avail Use% Mounted on /dev/nvme2n1 19G 522M 19G 3% /elasticsearch/persistent elasticsearch-cdm-kcrsda6l-3-5dfc884d99-59tjw Filesystem Size Used Avail Use% Mounted on /dev/nvme3n1 19G 528M 19G 3% /elasticsearch/persistent
- Increase the disk space on all nodes. If increasing the disk space is not possible, try adding a new data node to the cluster, or decrease the total cluster redundancy policy.
To check the current
redundancyPolicy
, run the following command:$ oc -n openshift-logging get es elasticsearch -o jsonpath='{.spec.redundancyPolicy}'
If you are using a
ClusterLogging
resource on your cluster, run the following command:$ oc -n openshift-logging get cl \ -o jsonpath='{.items[*].spec.logStore.elasticsearch.redundancyPolicy}'
If the cluster
redundancyPolicy
value is higher than theSingleRedundancy
value, set it to theSingleRedundancy
value and save this change.If the preceding steps do not fix the issue, delete the old indices.
Check the status of all indices on Elasticsearch by running the following command:
$ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME -- indices
- Identify an old index that can be deleted.
Delete the index by running the following command:
$ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \ -- es_util --query=<elasticsearch_index_name> -X DELETE
Additional resources
3.3.10. Elasticsearch FileDescriptor usage is high
Based on current usage trends, the predicted number of file descriptors on the node is insufficient. Check the value of max_file_descriptors
for each node as described in the Elasticsearch File Descriptors documentation.
3.4. Viewing the status of the Elasticsearch log store
You can view the status of the OpenShift Elasticsearch Operator and for a number of Elasticsearch components.
3.4.1. Viewing the status of the Elasticsearch log store
You can view the status of the Elasticsearch log store.
Prerequisites
- The Red Hat OpenShift Logging Operator and OpenShift Elasticsearch Operator are installed.
Procedure
Change to the
openshift-logging
project by running the following command:$ oc project openshift-logging
To view the status:
Get the name of the Elasticsearch log store instance by running the following command:
$ oc get Elasticsearch
Example output
NAME AGE elasticsearch 5h9m
Get the Elasticsearch log store status by running the following command:
$ oc get Elasticsearch <Elasticsearch-instance> -o yaml
For example:
$ oc get Elasticsearch elasticsearch -n openshift-logging -o yaml
The output includes information similar to the following:
Example output
status: 1 cluster: 2 activePrimaryShards: 30 activeShards: 60 initializingShards: 0 numDataNodes: 3 numNodes: 3 pendingTasks: 0 relocatingShards: 0 status: green unassignedShards: 0 clusterHealth: "" conditions: [] 3 nodes: 4 - deploymentName: elasticsearch-cdm-zjf34ved-1 upgradeStatus: {} - deploymentName: elasticsearch-cdm-zjf34ved-2 upgradeStatus: {} - deploymentName: elasticsearch-cdm-zjf34ved-3 upgradeStatus: {} pods: 5 client: failed: [] notReady: [] ready: - elasticsearch-cdm-zjf34ved-1-6d7fbf844f-sn422 - elasticsearch-cdm-zjf34ved-2-dfbd988bc-qkzjz - elasticsearch-cdm-zjf34ved-3-c8f566f7c-t7zkt data: failed: [] notReady: [] ready: - elasticsearch-cdm-zjf34ved-1-6d7fbf844f-sn422 - elasticsearch-cdm-zjf34ved-2-dfbd988bc-qkzjz - elasticsearch-cdm-zjf34ved-3-c8f566f7c-t7zkt master: failed: [] notReady: [] ready: - elasticsearch-cdm-zjf34ved-1-6d7fbf844f-sn422 - elasticsearch-cdm-zjf34ved-2-dfbd988bc-qkzjz - elasticsearch-cdm-zjf34ved-3-c8f566f7c-t7zkt shardAllocationEnabled: all
- 1
- In the output, the cluster status fields appear in the
status
stanza. - 2
- The status of the Elasticsearch log store:
- The number of active primary shards.
- The number of active shards.
- The number of shards that are initializing.
- The number of Elasticsearch log store data nodes.
- The total number of Elasticsearch log store nodes.
- The number of pending tasks.
-
The Elasticsearch log store status:
green
,red
,yellow
. - The number of unassigned shards.
- 3
- Any status conditions, if present. The Elasticsearch log store status indicates the reasons from the scheduler if a pod could not be placed. Any events related to the following conditions are shown:
- Container Waiting for both the Elasticsearch log store and proxy containers.
- Container Terminated for both the Elasticsearch log store and proxy containers.
- Pod unschedulable. Also, a condition is shown for a number of issues; see Example condition messages.
- 4
- The Elasticsearch log store nodes in the cluster, with
upgradeStatus
. - 5
- The Elasticsearch log store client, data, and master pods in the cluster, listed under
failed
,notReady
, orready
state.
3.4.1.1. Example condition messages
The following are examples of some condition messages from the Status
section of the Elasticsearch instance.
The following status message indicates that a node has exceeded the configured low watermark, and no shard will be allocated to this node.
status: nodes: - conditions: - lastTransitionTime: 2019-03-15T15:57:22Z message: Disk storage usage for node is 27.5gb (36.74%). Shards will be not be allocated on this node. reason: Disk Watermark Low status: "True" type: NodeStorage deploymentName: example-elasticsearch-cdm-0-1 upgradeStatus: {}
The following status message indicates that a node has exceeded the configured high watermark, and shards will be relocated to other nodes.
status: nodes: - conditions: - lastTransitionTime: 2019-03-15T16:04:45Z message: Disk storage usage for node is 27.5gb (36.74%). Shards will be relocated from this node. reason: Disk Watermark High status: "True" type: NodeStorage deploymentName: example-elasticsearch-cdm-0-1 upgradeStatus: {}
The following status message indicates that the Elasticsearch log store node selector in the custom resource (CR) does not match any nodes in the cluster:
status: nodes: - conditions: - lastTransitionTime: 2019-04-10T02:26:24Z message: '0/8 nodes are available: 8 node(s) didn''t match node selector.' reason: Unschedulable status: "True" type: Unschedulable
The following status message indicates that the Elasticsearch log store CR uses a non-existent persistent volume claim (PVC).
status: nodes: - conditions: - last Transition Time: 2019-04-10T05:55:51Z message: pod has unbound immediate PersistentVolumeClaims (repeated 5 times) reason: Unschedulable status: True type: Unschedulable
The following status message indicates that your Elasticsearch log store cluster does not have enough nodes to support the redundancy policy.
status: clusterHealth: "" conditions: - lastTransitionTime: 2019-04-17T20:01:31Z message: Wrong RedundancyPolicy selected. Choose different RedundancyPolicy or add more nodes with data roles reason: Invalid Settings status: "True" type: InvalidRedundancy
This status message indicates your cluster has too many control plane nodes:
status: clusterHealth: green conditions: - lastTransitionTime: '2019-04-17T20:12:34Z' message: >- Invalid master nodes count. Please ensure there are no more than 3 total nodes with master roles reason: Invalid Settings status: 'True' type: InvalidMasters
The following status message indicates that Elasticsearch storage does not support the change you tried to make.
For example:
status: clusterHealth: green conditions: - lastTransitionTime: "2021-05-07T01:05:13Z" message: Changing the storage structure for a custom resource is not supported reason: StorageStructureChangeIgnored status: 'True' type: StorageStructureChangeIgnored
The reason
and type
fields specify the type of unsupported change:
StorageClassNameChangeIgnored
- Unsupported change to the storage class name.
StorageSizeChangeIgnored
- Unsupported change the storage size.
StorageStructureChangeIgnored
Unsupported change between ephemeral and persistent storage structures.
ImportantIf you try to configure the
ClusterLogging
CR to switch from ephemeral to persistent storage, the OpenShift Elasticsearch Operator creates a persistent volume claim (PVC) but does not create a persistent volume (PV). To clear theStorageStructureChangeIgnored
status, you must revert the change to theClusterLogging
CR and delete the PVC.
3.4.2. Viewing the status of the log store components
You can view the status for a number of the log store components.
- Elasticsearch indices
You can view the status of the Elasticsearch indices.
Get the name of an Elasticsearch pod:
$ oc get pods --selector component=elasticsearch -o name
Example output
pod/elasticsearch-cdm-1godmszn-1-6f8495-vp4lw pod/elasticsearch-cdm-1godmszn-2-5769cf-9ms2n pod/elasticsearch-cdm-1godmszn-3-f66f7d-zqkz7
Get the status of the indices:
$ oc exec elasticsearch-cdm-4vjor49p-2-6d4d7db474-q2w7z -- indices
Example output
Defaulting container name to elasticsearch. Use 'oc describe pod/elasticsearch-cdm-4vjor49p-2-6d4d7db474-q2w7z -n openshift-logging' to see all of the containers in this pod. green open infra-000002 S4QANnf1QP6NgCegfnrnbQ 3 1 119926 0 157 78 green open audit-000001 8_EQx77iQCSTzFOXtxRqFw 3 1 0 0 0 0 green open .security iDjscH7aSUGhIdq0LheLBQ 1 1 5 0 0 0 green open .kibana_-377444158_kubeadmin yBywZ9GfSrKebz5gWBZbjw 3 1 1 0 0 0 green open infra-000001 z6Dpe__ORgiopEpW6Yl44A 3 1 871000 0 874 436 green open app-000001 hIrazQCeSISewG3c2VIvsQ 3 1 2453 0 3 1 green open .kibana_1 JCitcBMSQxKOvIq6iQW6wg 1 1 0 0 0 0 green open .kibana_-1595131456_user1 gIYFIEGRRe-ka0W3okS-mQ 3 1 1 0 0 0
- Log store pods
You can view the status of the pods that host the log store.
Get the name of a pod:
$ oc get pods --selector component=elasticsearch -o name
Example output
pod/elasticsearch-cdm-1godmszn-1-6f8495-vp4lw pod/elasticsearch-cdm-1godmszn-2-5769cf-9ms2n pod/elasticsearch-cdm-1godmszn-3-f66f7d-zqkz7
Get the status of a pod:
$ oc describe pod elasticsearch-cdm-1godmszn-1-6f8495-vp4lw
The output includes the following status information:
Example output
.... Status: Running .... Containers: elasticsearch: Container ID: cri-o://b7d44e0a9ea486e27f47763f5bb4c39dfd2 State: Running Started: Mon, 08 Jun 2020 10:17:56 -0400 Ready: True Restart Count: 0 Readiness: exec [/usr/share/elasticsearch/probe/readiness.sh] delay=10s timeout=30s period=5s #success=1 #failure=3 .... proxy: Container ID: cri-o://3f77032abaddbb1652c116278652908dc01860320b8a4e741d06894b2f8f9aa1 State: Running Started: Mon, 08 Jun 2020 10:18:38 -0400 Ready: True Restart Count: 0 .... Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True .... Events: <none>
- Log storage pod deployment configuration
You can view the status of the log store deployment configuration.
Get the name of a deployment configuration:
$ oc get deployment --selector component=elasticsearch -o name
Example output
deployment.extensions/elasticsearch-cdm-1gon-1 deployment.extensions/elasticsearch-cdm-1gon-2 deployment.extensions/elasticsearch-cdm-1gon-3
Get the deployment configuration status:
$ oc describe deployment elasticsearch-cdm-1gon-1
The output includes the following status information:
Example output
.... Containers: elasticsearch: Image: registry.redhat.io/openshift-logging/elasticsearch6-rhel8 Readiness: exec [/usr/share/elasticsearch/probe/readiness.sh] delay=10s timeout=30s period=5s #success=1 #failure=3 .... Conditions: Type Status Reason ---- ------ ------ Progressing Unknown DeploymentPaused Available True MinimumReplicasAvailable .... Events: <none>
- Log store replica set
You can view the status of the log store replica set.
Get the name of a replica set:
$ oc get replicaSet --selector component=elasticsearch -o name replicaset.extensions/elasticsearch-cdm-1gon-1-6f8495 replicaset.extensions/elasticsearch-cdm-1gon-2-5769cf replicaset.extensions/elasticsearch-cdm-1gon-3-f66f7d
Get the status of the replica set:
$ oc describe replicaSet elasticsearch-cdm-1gon-1-6f8495
The output includes the following status information:
Example output
.... Containers: elasticsearch: Image: registry.redhat.io/openshift-logging/elasticsearch6-rhel8@sha256:4265742c7cdd85359140e2d7d703e4311b6497eec7676957f455d6908e7b1c25 Readiness: exec [/usr/share/elasticsearch/probe/readiness.sh] delay=10s timeout=30s period=5s #success=1 #failure=3 .... Events: <none>
3.4.3. Elasticsearch cluster status
A dashboard in the Observe section of the OpenShift Cluster Manager displays the status of the Elasticsearch cluster.
To get the status of the OpenShift Elasticsearch cluster, visit the dashboard in the Observe section of the OpenShift Cluster Manager at <cluster_url>/monitoring/dashboards/grafana-dashboard-cluster-logging
.
Elasticsearch status fields
eo_elasticsearch_cr_cluster_management_state
Shows whether the Elasticsearch cluster is in a managed or unmanaged state. For example:
eo_elasticsearch_cr_cluster_management_state{state="managed"} 1 eo_elasticsearch_cr_cluster_management_state{state="unmanaged"} 0
eo_elasticsearch_cr_restart_total
Shows the number of times the Elasticsearch nodes have restarted for certificate restarts, rolling restarts, or scheduled restarts. For example:
eo_elasticsearch_cr_restart_total{reason="cert_restart"} 1 eo_elasticsearch_cr_restart_total{reason="rolling_restart"} 1 eo_elasticsearch_cr_restart_total{reason="scheduled_restart"} 3
es_index_namespaces_total
Shows the total number of Elasticsearch index namespaces. For example:
Total number of Namespaces. es_index_namespaces_total 5
es_index_document_count
Shows the number of records for each namespace. For example:
es_index_document_count{namespace="namespace_1"} 25 es_index_document_count{namespace="namespace_2"} 10 es_index_document_count{namespace="namespace_3"} 5
The "Secret Elasticsearch fields are either missing or empty" message
If Elasticsearch is missing the admin-cert
, admin-key
, logging-es.crt
, or logging-es.key
files, the dashboard shows a status message similar to the following example:
message": "Secret \"elasticsearch\" fields are either missing or empty: [admin-cert, admin-key, logging-es.crt, logging-es.key]", "reason": "Missing Required Secrets",
Chapter 4. About Logging
As a cluster administrator, you can deploy logging on an Red Hat OpenShift Service on AWS cluster, and use it to collect and aggregate node system audit logs, application container logs, and infrastructure logs. You can forward logs to your chosen log outputs, including on-cluster, Red Hat managed log storage. You can also visualize your log data in the Red Hat OpenShift Service on AWS web console, or the Kibana web console, depending on your deployed log storage solution.
The Kibana web console is now deprecated is planned to be removed in a future logging release.
Red Hat OpenShift Service on AWS cluster administrators can deploy logging by using Operators. For information, see xref :../../observability/logging/cluster-logging-deploying.adoc#cluster-logging-deploying[Installing logging].
The Operators are responsible for deploying, upgrading, and maintaining logging. After the Operators are installed, you can create a ClusterLogging
custom resource (CR) to schedule logging pods and other resources necessary to support logging. You can also create a ClusterLogForwarder
CR to specify which logs are collected, how they are transformed, and where they are forwarded to.
Because the internal Red Hat OpenShift Service on AWS Elasticsearch log store does not provide secure storage for audit logs, audit logs are not stored in the internal Elasticsearch instance by default. If you want to send the audit logs to the default internal Elasticsearch log store, for example to view the audit logs in Kibana, you must use the Log Forwarding API as described in xref :../../observability/logging/log_storage/logging-config-es-store.adoc#cluster-logging-elasticsearch-audit_logging-config-es-store[Forward audit logs to the log store].
4.1. Logging architecture
The major components of the logging are:
- Collector
The collector is a daemonset that deploys pods to each Red Hat OpenShift Service on AWS node. It collects log data from each node, transforms the data, and forwards it to configured outputs. You can use the Vector collector or the legacy Fluentd collector.
NoteFluentd is deprecated and is planned to be removed in a future release. Red Hat provides bug fixes and support for this feature during the current release lifecycle, but this feature no longer receives enhancements. As an alternative to Fluentd, you can use Vector instead.
- Log store
The log store stores log data for analysis and is the default output for the log forwarder. You can use the default LokiStack log store, the legacy Elasticsearch log store, or forward logs to additional external log stores.
NoteThe Logging 5.9 release does not contain an updated version of the OpenShift Elasticsearch Operator. If you currently use the OpenShift Elasticsearch Operator released with Logging 5.8, it will continue to work with Logging until the EOL of Logging 5.8. As an alternative to using the OpenShift Elasticsearch Operator to manage the default log storage, you can use the Loki Operator. For more information on the Logging lifecycle dates, see Platform Agnostic Operators.
- Visualization
You can use a UI component to view a visual representation of your log data. The UI provides a graphical interface to search, query, and view stored logs. The Red Hat OpenShift Service on AWS web console UI is provided by enabling the Red Hat OpenShift Service on AWS console plugin.
NoteThe Kibana web console is now deprecated is planned to be removed in a future logging release.
Logging collects container logs and node logs. These are categorized into types:
- Application logs
- Container logs generated by user applications running in the cluster, except infrastructure container applications.
- Infrastructure logs
-
Container logs generated by infrastructure namespaces:
openshift*
,kube*
, ordefault
, as well as journald messages from nodes. - Audit logs
-
Logs generated by auditd, the node audit system, which are stored in the /var/log/audit/audit.log file, and logs from the
auditd
,kube-apiserver
,openshift-apiserver
services, as well as theovn
project if enabled.
Additional resources
- xref :../../observability/logging/log_visualization/log-visualization-ocp-console.adoc#log-visualization-ocp-console[Log visualization with the web console]
4.2. About deploying logging
Administrators can deploy the logging by using the Red Hat OpenShift Service on AWS web console or the OpenShift CLI (oc
) to install the logging Operators. The Operators are responsible for deploying, upgrading, and maintaining the logging.
Administrators and application developers can view the logs of the projects for which they have view access.
4.2.1. Logging custom resources
You can configure your logging deployment with custom resource (CR) YAML files implemented by each Operator.
Red Hat OpenShift Logging Operator:
-
ClusterLogging
(CL) - After the Operators are installed, you create aClusterLogging
custom resource (CR) to schedule logging pods and other resources necessary to support the logging. TheClusterLogging
CR deploys the collector and forwarder, which currently are both implemented by a daemonset running on each node. The Red Hat OpenShift Logging Operator watches theClusterLogging
CR and adjusts the logging deployment accordingly. -
ClusterLogForwarder
(CLF) - Generates collector configuration to forward logs per user configuration.
Loki Operator:
-
LokiStack
- Controls the Loki cluster as log store and the web proxy with Red Hat OpenShift Service on AWS authentication integration to enforce multi-tenancy.
OpenShift Elasticsearch Operator:
These CRs are generated and managed by the OpenShift Elasticsearch Operator. Manual changes cannot be made without being overwritten by the Operator.
-
ElasticSearch
- Configure and deploy an Elasticsearch instance as the default log store. -
Kibana
- Configure and deploy Kibana instance to search, query and view logs.
4.2.2. Logging requirements
Hosting your own logging stack requires a large amount of compute resources and storage, which might be dependent on your cloud service quota. The compute resource requirements can start at 48 GB or more, while the storage requirement can be as large as 1600 GB or more. The logging stack runs on your worker nodes, which reduces your available workload resource. With these considerations, hosting your own logging stack increases your cluster operating costs.
For information, see xref :../../observability/logging/log_collection_forwarding/log-forwarding.adoc#about-log-collection_log-forwarding[About log collection and forwarding].
4.2.3. About JSON Red Hat OpenShift Service on AWS Logging
You can use JSON logging to configure the Log Forwarding API to parse JSON strings into a structured object. You can perform the following tasks:
- Parse JSON logs
- Configure JSON log data for Elasticsearch
- Forward JSON logs to the Elasticsearch log store
4.2.4. About collecting and storing Kubernetes events
The Red Hat OpenShift Service on AWS Event Router is a pod that watches Kubernetes events and logs them for collection by Red Hat OpenShift Service on AWS Logging. You must manually deploy the Event Router.
For information, see xref :../../observability/logging/log_collection_forwarding/cluster-logging-eventrouter.adoc#cluster-logging-eventrouter[About collecting and storing Kubernetes events].
4.2.5. About troubleshooting Red Hat OpenShift Service on AWS Logging
You can troubleshoot the logging issues by performing the following tasks:
- Viewing logging status
- Viewing the status of the log store
- Understanding logging alerts
- Collecting logging data for Red Hat Support
- Troubleshooting for critical alerts
4.2.6. About exporting fields
The logging system exports fields. Exported fields are present in the log records and are available for searching from Elasticsearch and Kibana.
For information, see xref :../../observability/logging/cluster-logging-exported-fields.adoc#cluster-logging-exported-fields[About exporting fields].
4.2.7. About event routing
The Event Router is a pod that watches Red Hat OpenShift Service on AWS events so they can be collected by logging. The Event Router collects events from all projects and writes them to STDOUT
. Fluentd collects those events and forwards them into the Red Hat OpenShift Service on AWS Elasticsearch instance. Elasticsearch indexes the events to the infra
index.
You must manually deploy the Event Router.
For information, see xref :../../observability/logging/log_collection_forwarding/cluster-logging-eventrouter.adoc#cluster-logging-eventrouter[Collecting and storing Kubernetes events].
Chapter 5. Installing Logging
Red Hat OpenShift Service on AWS Operators use custom resources (CR) to manage applications and their components. High-level configuration and settings are provided by the user within a CR. The Operator translates high-level directives into low-level actions, based on best practices embedded within the Operator’s logic. A custom resource definition (CRD) defines a CR and lists all the configurations available to users of the Operator. Installing an Operator creates the CRDs, which are then used to generate CRs.
You must install the Red Hat OpenShift Logging Operator after the log store Operator.
You deploy logging by installing the Loki Operator or OpenShift Elasticsearch Operator to manage your log store, followed by the Red Hat OpenShift Logging Operator to manage the components of logging. You can use either the Red Hat OpenShift Service on AWS web console or the Red Hat OpenShift Service on AWS CLI to install or configure logging.
The Logging 5.9 release does not contain an updated version of the OpenShift Elasticsearch Operator. If you currently use the OpenShift Elasticsearch Operator released with Logging 5.8, it will continue to work with Logging until the EOL of Logging 5.8. As an alternative to using the OpenShift Elasticsearch Operator to manage the default log storage, you can use the Loki Operator. For more information on the Logging lifecycle dates, see Platform Agnostic Operators.
You can alternatively apply all example objects.
5.1. Installing Logging with Elasticsearch using the web console
You can use the Red Hat OpenShift Service on AWS web console to install the OpenShift Elasticsearch and Red Hat OpenShift Logging Operators. Elasticsearch is a memory-intensive application. By default, Red Hat OpenShift Service on AWS installs three Elasticsearch nodes with memory requests and limits of 16 GB. This initial set of three Red Hat OpenShift Service on AWS nodes might not have enough memory to run Elasticsearch within your cluster. If you experience memory issues that are related to Elasticsearch, add more Elasticsearch nodes to your cluster rather than increasing the memory on existing nodes.
If you do not want to use the default Elasticsearch log store, you can remove the internal Elasticsearch logStore
and Kibana visualization
components from the ClusterLogging
custom resource (CR). Removing these components is optional but saves resources.
Prerequisites
Ensure that you have the necessary persistent storage for Elasticsearch. Note that each Elasticsearch node requires its own storage volume.
NoteIf you use a local volume for persistent storage, do not use a raw block volume, which is described with
volumeMode: block
in theLocalVolume
object. Elasticsearch cannot use raw block volumes.
Procedure
To install the OpenShift Elasticsearch Operator and Red Hat OpenShift Logging Operator using the Red Hat OpenShift Service on AWS web console:
Install the OpenShift Elasticsearch Operator:
- In the Red Hat OpenShift Service on AWS web console, click Operators → OperatorHub.
- Choose OpenShift Elasticsearch Operator from the list of available Operators, and click Install.
- Ensure that the All namespaces on the cluster is selected under Installation Mode.
Ensure that openshift-operators-redhat is selected under Installed Namespace.
You must specify the
openshift-operators-redhat
namespace. Theopenshift-operators
namespace might contain Community Operators, which are untrusted and could publish a metric with the same name as an Red Hat OpenShift Service on AWS metric, which would cause conflicts.Select Enable Operator recommended cluster monitoring on this namespace.
This option sets the
openshift.io/cluster-monitoring: "true"
label in the Namespace object. You must select this option to ensure that cluster monitoring scrapes theopenshift-operators-redhat
namespace.Select stable-5.y as the Update Channel.
NoteThe stable channel only provides updates to the most recent release of logging. To continue receiving updates for prior releases, you must change your subscription channel to stable-x.y, where
x.y
represents the major and minor version of logging you have installed. For example, stable-5.7.Select an Approval Strategy.
- The Automatic strategy allows Operator Lifecycle Manager (OLM) to automatically update the Operator when a new version is available.
- The Manual strategy requires a user with appropriate credentials to approve the Operator update.
- Click Install.
- Verify that the OpenShift Elasticsearch Operator installed by switching to the Operators → Installed Operators page.
- Ensure that OpenShift Elasticsearch Operator is listed in all projects with a Status of Succeeded.
Install the Red Hat OpenShift Logging Operator:
- In the Red Hat OpenShift Service on AWS web console, click Operators → OperatorHub.
- Choose Red Hat OpenShift Logging from the list of available Operators, and click Install.
- Ensure that the A specific namespace on the cluster is selected under Installation Mode.
- Ensure that Operator recommended namespace is openshift-logging under Installed Namespace.
Select Enable Operator recommended cluster monitoring on this namespace.
This option sets the
openshift.io/cluster-monitoring: "true"
label in the Namespace object. You must select this option to ensure that cluster monitoring scrapes theopenshift-logging
namespace.- Select stable-5.y as the Update Channel.
Select an Approval Strategy.
- The Automatic strategy allows Operator Lifecycle Manager (OLM) to automatically update the Operator when a new version is available.
- The Manual strategy requires a user with appropriate credentials to approve the Operator update.
- Click Install.
- Verify that the Red Hat OpenShift Logging Operator installed by switching to the Operators → Installed Operators page.
Ensure that Red Hat OpenShift Logging is listed in the openshift-logging project with a Status of Succeeded.
If the Operator does not appear as installed, to troubleshoot further:
- Switch to the Operators → Installed Operators page and inspect the Status column for any errors or failures.
-
Switch to the Workloads → Pods page and check the logs in any pods in the
openshift-logging
project that are reporting issues.
Create an OpenShift Logging instance:
- Switch to the Administration → Custom Resource Definitions page.
- On the Custom Resource Definitions page, click ClusterLogging.
- On the Custom Resource Definition details page, select View Instances from the Actions menu.
On the ClusterLoggings page, click Create ClusterLogging.
You might have to refresh the page to load the data.
In the YAML field, replace the code with the following:
NoteThis default OpenShift Logging configuration should support a wide array of environments. Review the topics on tuning and configuring logging components for information on modifications you can make to your OpenShift Logging cluster.
apiVersion: logging.openshift.io/v1 kind: ClusterLogging metadata: name: instance 1 namespace: openshift-logging spec: managementState: Managed 2 logStore: type: elasticsearch 3 retentionPolicy: 4 application: maxAge: 1d infra: maxAge: 7d audit: maxAge: 7d elasticsearch: nodeCount: 3 5 storage: storageClassName: <storage_class_name> 6 size: 200G resources: 7 limits: memory: 16Gi requests: memory: 16Gi proxy: 8 resources: limits: memory: 256Mi requests: memory: 256Mi redundancyPolicy: SingleRedundancy visualization: type: kibana 9 kibana: replicas: 1 collection: type: fluentd 10 fluentd: {}
- 1
- The name must be
instance
. - 2
- The OpenShift Logging management state. In some cases, if you change the OpenShift Logging defaults, you must set this to
Unmanaged
. However, an unmanaged deployment does not receive updates until OpenShift Logging is placed back into a managed state. - 3
- Settings for configuring Elasticsearch. Using the CR, you can configure shard replication policy and persistent storage.
- 4
- Specify the length of time that Elasticsearch should retain each log source. Enter an integer and a time designation: weeks(w), hours(h/H), minutes(m) and seconds(s). For example,
7d
for seven days. Logs older than themaxAge
are deleted. You must specify a retention policy for each log source or the Elasticsearch indices will not be created for that source. - 5
- Specify the number of Elasticsearch nodes. See the note that follows this list.
- 6
- Enter the name of an existing storage class for Elasticsearch storage. For best performance, specify a storage class that allocates block storage. If you do not specify a storage class, OpenShift Logging uses ephemeral storage.
- 7
- Specify the CPU and memory requests for Elasticsearch as needed. If you leave these values blank, the OpenShift Elasticsearch Operator sets default values that should be sufficient for most deployments. The default values are
16Gi
for the memory request and1
for the CPU request. - 8
- Specify the CPU and memory requests for the Elasticsearch proxy as needed. If you leave these values blank, the OpenShift Elasticsearch Operator sets default values that should be sufficient for most deployments. The default values are
256Mi
for the memory request and100m
for the CPU request. - 9
- Settings for configuring Kibana. Using the CR, you can scale Kibana for redundancy and configure the CPU and memory for your Kibana nodes. For more information, see Configuring the log visualizer.
- 10
- Settings for configuring Fluentd. Using the CR, you can configure Fluentd CPU and memory limits. For more information, see "Configuring Fluentd".
NoteThe maximum number of master nodes is three. If you specify a
nodeCount
greater than3
, Red Hat OpenShift Service on AWS creates three Elasticsearch nodes that are Master-eligible nodes, with the master, client, and data roles. The additional Elasticsearch nodes are created as Data-only nodes, using client and data roles. Master nodes perform cluster-wide actions such as creating or deleting an index, shard allocation, and tracking nodes. Data nodes hold the shards and perform data-related operations such as CRUD, search, and aggregations. Data-related operations are I/O-, memory-, and CPU-intensive. It is important to monitor these resources and to add more Data nodes if the current nodes are overloaded.For example, if
nodeCount=4
, the following nodes are created:$ oc get deployment
Example output
cluster-logging-operator-66f77ffccb-ppzbg 1/1 Running 0 7m elasticsearch-cd-tuhduuw-1-f5c885dbf-dlqws 1/1 Running 0 2m4s elasticsearch-cdm-ftuhduuw-1-ffc4b9566-q6bhp 2/2 Running 0 2m40s elasticsearch-cdm-ftuhduuw-2-7b4994dbfc-rd2gc 2/2 Running 0 2m36s elasticsearch-cdm-ftuhduuw-3-84b5ff7ff8-gqnm2 2/2 Running 0 2m4s
-
Click Create. This creates the logging components, the
Elasticsearch
custom resource and components, and the Kibana interface.
Verify the install:
- Switch to the Workloads → Pods page.
Select the openshift-logging project.
You should see several pods for OpenShift Logging, Elasticsearch, your collector, and Kibana similar to the following list:
Example output
cluster-logging-operator-66f77ffccb-ppzbg 1/1 Running 0 7m elasticsearch-cdm-ftuhduuw-1-ffc4b9566-q6bhp 2/2 Running 0 2m40s elasticsearch-cdm-ftuhduuw-2-7b4994dbfc-rd2gc 2/2 Running 0 2m36s elasticsearch-cdm-ftuhduuw-3-84b5ff7ff8-gqnm2 2/2 Running 0 2m4s collector-587vb 1/1 Running 0 2m26s collector-7mpb9 1/1 Running 0 2m30s collector-flm6j 1/1 Running 0 2m33s collector-gn4rn 1/1 Running 0 2m26s collector-nlgb6 1/1 Running 0 2m30s collector-snpkt 1/1 Running 0 2m28s kibana-d6d5668c5-rppqm 2/2 Running 0 2m39s
5.2. Installing Logging with Elasticsearch using the CLI
Elasticsearch is a memory-intensive application. By default, Red Hat OpenShift Service on AWS installs three Elasticsearch nodes with memory requests and limits of 16 GB. This initial set of three Red Hat OpenShift Service on AWS nodes might not have enough memory to run Elasticsearch within your cluster. If you experience memory issues that are related to Elasticsearch, add more Elasticsearch nodes to your cluster rather than increasing the memory on existing nodes.
Prerequisites
Ensure that you have the necessary persistent storage for Elasticsearch. Note that each Elasticsearch node requires its own storage volume.
NoteIf you use a local volume for persistent storage, do not use a raw block volume, which is described with
volumeMode: block
in theLocalVolume
object. Elasticsearch cannot use raw block volumes.
Procedure
Create a
Namespace
object for the OpenShift Elasticsearch Operator:Example
Namespace
objectapiVersion: v1 kind: Namespace metadata: name: openshift-operators-redhat 1 annotations: openshift.io/node-selector: "" labels: openshift.io/cluster-monitoring: "true" 2
- 1
- You must specify the
openshift-operators-redhat
namespace. Theopenshift-operators
namespace might contain Community Operators, which are untrusted and could publish a metric with the same name as an Red Hat OpenShift Service on AWS metric, which would cause conflicts. - 2
- A string value that specifies the label as shown to ensure that cluster monitoring scrapes the
openshift-operators-redhat
namespace.
Apply the
Namespace
object by running the following command:$ oc apply -f <filename>.yaml
Create a
Namespace
object for the Red Hat OpenShift Logging Operator:Example
Namespace
objectapiVersion: v1 kind: Namespace metadata: name: openshift-logging 1 annotations: openshift.io/node-selector: "" labels: openshift.io/cluster-monitoring: "true"
- 1
- You must specify
openshift-logging
as the namespace for logging versions 5.7 and earlier. For logging 5.8 and later, you can use any namespace.
Apply the
Namespace
object by running the following command:$ oc apply -f <filename>.yaml
Create an
OperatorGroup
object for the OpenShift Elasticsearch Operator:Example
OperatorGroup
objectapiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: openshift-operators-redhat namespace: openshift-operators-redhat 1 spec: {}
- 1
- You must specify the
openshift-operators-redhat
namespace.
Apply the
OperatorGroup
object by running the following command:$ oc apply -f <filename>.yaml
Create a
Subscription
object to subscribe a namespace to the OpenShift Elasticsearch Operator:NoteThe stable channel only provides updates to the most recent release of logging. To continue receiving updates for prior releases, you must change your subscription channel to stable-x.y, where
x.y
represents the major and minor version of logging you have installed. For example, stable-5.7.Example
Subscription
objectapiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: elasticsearch-operator namespace: openshift-operators-redhat 1 spec: channel: <channel> 2 installPlanApproval: Automatic 3 source: redhat-operators 4 sourceNamespace: openshift-marketplace name: elasticsearch-operator
- 1
- You must specify the
openshift-operators-redhat
namespace. - 2
- Specify
stable
, orstable-<x.y>
as the channel. - 3
Automatic
allows the Operator Lifecycle Manager (OLM) to automatically update the Operator when a new version is available.Manual
requires a user with appropriate credentials to approve the Operator update.- 4
- Specify
redhat-operators
. If your Red Hat OpenShift Service on AWS cluster is installed on a restricted network, also known as a disconnected cluster, specify the name of theCatalogSource
object you created when you configured the Operator Lifecycle Manager (OLM)
Apply the subscription by running the following command:
$ oc apply -f <filename>.yaml
Verify the Operator installation by running the following command:
$ oc get csv --all-namespaces
Example output
NAMESPACE NAME DISPLAY VERSION REPLACES PHASE default elasticsearch-operator.v5.8.3 OpenShift Elasticsearch Operator 5.8.3 elasticsearch-operator.v5.8.2 Succeeded kube-node-lease elasticsearch-operator.v5.8.3 OpenShift Elasticsearch Operator 5.8.3 elasticsearch-operator.v5.8.2 Succeeded kube-public elasticsearch-operator.v5.8.3 OpenShift Elasticsearch Operator 5.8.3 elasticsearch-operator.v5.8.2 Succeeded kube-system elasticsearch-operator.v5.8.3 OpenShift Elasticsearch Operator 5.8.3 elasticsearch-operator.v5.8.2 Succeeded openshift-apiserver-operator elasticsearch-operator.v5.8.3 OpenShift Elasticsearch Operator 5.8.3 elasticsearch-operator.v5.8.2 Succeeded openshift-apiserver elasticsearch-operator.v5.8.3 OpenShift Elasticsearch Operator 5.8.3 elasticsearch-operator.v5.8.2 Succeeded openshift-authentication-operator elasticsearch-operator.v5.8.3 OpenShift Elasticsearch Operator 5.8.3 elasticsearch-operator.v5.8.2 Succeeded openshift-authentication elasticsearch-operator.v5.8.3 OpenShift Elasticsearch Operator 5.8.3 elasticsearch-operator.v5.8.2 Succeeded openshift-cloud-controller-manager-operator elasticsearch-operator.v5.8.3 OpenShift Elasticsearch Operator 5.8.3 elasticsearch-operator.v5.8.2 Succeeded openshift-cloud-controller-manager elasticsearch-operator.v5.8.3 OpenShift Elasticsearch Operator 5.8.3 elasticsearch-operator.v5.8.2 Succeeded openshift-cloud-credential-operator elasticsearch-operator.v5.8.3 OpenShift Elasticsearch Operator 5.8.3 elasticsearch-operator.v5.8.2 Succeeded
Create an
OperatorGroup
object for the Red Hat OpenShift Logging Operator:Example
OperatorGroup
objectapiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: cluster-logging namespace: openshift-logging 1 spec: targetNamespaces: - openshift-logging 2
Apply the
OperatorGroup
object by running the following command:$ oc apply -f <filename>.yaml
Create a
Subscription
object to subscribe the namespace to the Red Hat OpenShift Logging Operator:Example
Subscription
objectapiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: cluster-logging namespace: openshift-logging 1 spec: channel: stable 2 name: cluster-logging source: redhat-operators 3 sourceNamespace: openshift-marketplace
- 1
- You must specify the
openshift-logging
namespace for logging versions 5.7 and older. For logging 5.8 and later versions, you can use any namespace. - 2
- Specify
stable
orstable-x.y
as the channel. - 3
- Specify
redhat-operators
. If your Red Hat OpenShift Service on AWS cluster is installed on a restricted network, also known as a disconnected cluster, specify the name of theCatalogSource
object you created when you configured the Operator Lifecycle Manager (OLM).
Apply the
subscription
object by running the following command:$ oc apply -f <filename>.yaml
Create a
ClusterLogging
object as a YAML file:Example
ClusterLogging
objectapiVersion: logging.openshift.io/v1 kind: ClusterLogging metadata: name: instance 1 namespace: openshift-logging spec: managementState: Managed 2 logStore: type: elasticsearch 3 retentionPolicy: 4 application: maxAge: 1d infra: maxAge: 7d audit: maxAge: 7d elasticsearch: nodeCount: 3 5 storage: storageClassName: <storage_class_name> 6 size: 200G resources: 7 limits: memory: 16Gi requests: memory: 16Gi proxy: 8 resources: limits: memory: 256Mi requests: memory: 256Mi redundancyPolicy: SingleRedundancy visualization: type: kibana 9 kibana: replicas: 1 collection: type: fluentd 10 fluentd: {}
- 1
- The name must be
instance
. - 2
- The OpenShift Logging management state. In some cases, if you change the OpenShift Logging defaults, you must set this to
Unmanaged
. However, an unmanaged deployment does not receive updates until OpenShift Logging is placed back into a managed state. - 3
- Settings for configuring Elasticsearch. Using the CR, you can configure shard replication policy and persistent storage.
- 4
- Specify the length of time that Elasticsearch should retain each log source. Enter an integer and a time designation: weeks(w), hours(h/H), minutes(m) and seconds(s). For example,
7d
for seven days. Logs older than themaxAge
are deleted. You must specify a retention policy for each log source or the Elasticsearch indices will not be created for that source. - 5
- Specify the number of Elasticsearch nodes.
- 6
- Enter the name of an existing storage class for Elasticsearch storage. For best performance, specify a storage class that allocates block storage. If you do not specify a storage class, OpenShift Logging uses ephemeral storage.
- 7
- Specify the CPU and memory requests for Elasticsearch as needed. If you leave these values blank, the OpenShift Elasticsearch Operator sets default values that should be sufficient for most deployments. The default values are
16Gi
for the memory request and1
for the CPU request. - 8
- Specify the CPU and memory requests for the Elasticsearch proxy as needed. If you leave these values blank, the OpenShift Elasticsearch Operator sets default values that should be sufficient for most deployments. The default values are
256Mi
for the memory request and100m
for the CPU request. - 9
- Settings for configuring Kibana. Using the CR, you can scale Kibana for redundancy and configure the CPU and memory for your Kibana nodes.
- 10
- Settings for configuring Fluentd. Using the CR, you can configure Fluentd CPU and memory limits.
NoteThe maximum number of master nodes is three. If you specify a
nodeCount
greater than3
, Red Hat OpenShift Service on AWS creates three Elasticsearch nodes that are Master-eligible nodes, with the master, client, and data roles. The additional Elasticsearch nodes are created as Data-only nodes, using client and data roles. Master nodes perform cluster-wide actions such as creating or deleting an index, shard allocation, and tracking nodes. Data nodes hold the shards and perform data-related operations such as CRUD, search, and aggregations. Data-related operations are I/O-, memory-, and CPU-intensive. It is important to monitor these resources and to add more Data nodes if the current nodes are overloaded.For example, if
nodeCount=4
, the following nodes are created:$ oc get deployment
Example output
cluster-logging-operator-66f77ffccb-ppzbg 1/1 Running 0 7m elasticsearch-cdm-ftuhduuw-1-ffc4b9566-q6bhp 2/2 Running 0 2m40s elasticsearch-cdm-ftuhduuw-2-7b4994dbfc-rd2gc 2/2 Running 0 2m36s elasticsearch-cdm-ftuhduuw-3-84b5ff7ff8-gqnm2 2/2 Running 0 2m4s
Apply the
ClusterLogging
CR by running the following command:$ oc apply -f <filename>.yaml
Verify the installation by running the following command:
$ oc get pods -n openshift-logging
Example output
NAME READY STATUS RESTARTS AGE cluster-logging-operator-66f77ffccb-ppzbg 1/1 Running 0 7m elasticsearch-cdm-ftuhduuw-1-ffc4b9566-q6bhp 2/2 Running 0 2m40s elasticsearch-cdm-ftuhduuw-2-7b4994dbfc-rd2gc 2/2 Running 0 2m36s elasticsearch-cdm-ftuhduuw-3-84b5ff7ff8-gqnm2 2/2 Running 0 2m4s collector-587vb 1/1 Running 0 2m26s collector-7mpb9 1/1 Running 0 2m30s collector-flm6j 1/1 Running 0 2m33s collector-gn4rn 1/1 Running 0 2m26s collector-nlgb6 1/1 Running 0 2m30s collector-snpkt 1/1 Running 0 2m28s kibana-d6d5668c5-rppqm 2/2 Running 0 2m39s
If there is no retention period defined on the s3 bucket or in the LokiStack custom resource (CR), then the logs are not pruned and they stay in the s3 bucket forever, which might fill up the s3 storage.
5.3. Installing Logging and the Loki Operator using the CLI
To install and configure logging on your Red Hat OpenShift Service on AWS cluster, an Operator such as Loki Operator for log storage must be installed first. This can be done from the Red Hat OpenShift Service on AWS CLI.
Prerequisites
- You have administrator permissions.
-
You installed the OpenShift CLI (
oc
). - You have access to a supported object store. For example: AWS S3, Google Cloud Storage, Azure, Swift, Minio, or OpenShift Data Foundation.
The stable channel only provides updates to the most recent release of logging. To continue receiving updates for prior releases, you must change your subscription channel to stable-x.y, where x.y
represents the major and minor version of logging you have installed. For example, stable-5.7.
Create a
Namespace
object for Loki Operator:Example
Namespace
objectapiVersion: v1 kind: Namespace metadata: name: openshift-operators-redhat 1 annotations: openshift.io/node-selector: "" labels: openshift.io/cluster-monitoring: "true" 2
- 1
- You must specify the
openshift-operators-redhat
namespace. To prevent possible conflicts with metrics, you should configure the Prometheus Cluster Monitoring stack to scrape metrics from theopenshift-operators-redhat
namespace and not theopenshift-operators
namespace. Theopenshift-operators
namespace might contain community Operators, which are untrusted and could publish a metric with the same name as an Red Hat OpenShift Service on AWS metric, which would cause conflicts. - 2
- A string value that specifies the label as shown to ensure that cluster monitoring scrapes the
openshift-operators-redhat
namespace.
Apply the
Namespace
object by running the following command:$ oc apply -f <filename>.yaml
Create a
Subscription
object for Loki Operator:Example
Subscription
objectapiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: loki-operator namespace: openshift-operators-redhat 1 spec: channel: stable 2 name: loki-operator source: redhat-operators 3 sourceNamespace: openshift-marketplace
- 1
- You must specify the
openshift-operators-redhat
namespace. - 2
- Specify
stable
, orstable-5.<y>
as the channel. - 3
- Specify
redhat-operators
. If your Red Hat OpenShift Service on AWS cluster is installed on a restricted network, also known as a disconnected cluster, specify the name of theCatalogSource
object you created when you configured the Operator Lifecycle Manager (OLM).
Apply the
Subscription
object by running the following command:$ oc apply -f <filename>.yaml
Create a
namespace
object for the Red Hat OpenShift Logging Operator:Example
namespace
objectapiVersion: v1 kind: Namespace metadata: name: openshift-logging 1 annotations: openshift.io/node-selector: "" labels: openshift.io/cluster-logging: "true" openshift.io/cluster-monitoring: "true" 2
Apply the
namespace
object by running the following command:$ oc apply -f <filename>.yaml
Create an
OperatorGroup
objectExample
OperatorGroup
objectapiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: cluster-logging namespace: openshift-logging 1 spec: targetNamespaces: - openshift-logging
- 1
- You must specify the
openshift-logging
namespace.
Apply the
OperatorGroup
object by running the following command:$ oc apply -f <filename>.yaml
Create a
Subscription
object:apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: cluster-logging namespace: openshift-logging 1 spec: channel: stable 2 name: cluster-logging source: redhat-operators 3 sourceNamespace: openshift-marketplace
- 1
- You must specify the
openshift-logging
namespace. - 2
- Specify
stable
, orstable-5.<y>
as the channel. - 3
- Specify
redhat-operators
. If your Red Hat OpenShift Service on AWS cluster is installed on a restricted network, also known as a disconnected cluster, specify the name of the CatalogSource object you created when you configured the Operator Lifecycle Manager (OLM).
Apply the
Subscription
object by running the following command:$ oc apply -f <filename>.yaml
Create a
LokiStack
CR:Example
LokiStack
CRapiVersion: loki.grafana.com/v1 kind: LokiStack metadata: name: logging-loki 1 namespace: openshift-logging 2 spec: size: 1x.small 3 storage: schemas: - version: v13 effectiveDate: "<yyyy>-<mm>-<dd>" secret: name: logging-loki-s3 4 type: s3 5 credentialMode: 6 storageClassName: <storage_class_name> 7 tenants: mode: openshift-logging 8
- 1
- Use the name
logging-loki
. - 2
- You must specify the
openshift-logging
namespace. - 3
- Specify the deployment size. In the logging 5.8 and later versions, the supported size options for production instances of Loki are
1x.extra-small
,1x.small
, or1x.medium
. - 4
- Specify the name of your log store secret.
- 5
- Specify the corresponding storage type.
- 6
- Optional field, logging 5.9 and later. Supported user configured values are as follows:
static
is the default authentication mode available for all supported object storage types using credentials stored in a Secret.token
for short-lived tokens retrieved from a credential source. In this mode the static configuration does not contain credentials needed for the object storage. Instead, they are generated during runtime using a service, which allows for shorter-lived credentials and much more granular control. This authentication mode is not supported for all object storage types.token-cco
is the default value when Loki is running on managed STS mode and using CCO on STS/WIF clusters. - 7
- Specify the name of a storage class for temporary storage. For best performance, specify a storage class that allocates block storage. Available storage classes for your cluster can be listed by using the
oc get storageclasses
command. - 8
- LokiStack defaults to running in multi-tenant mode, which cannot be modified. One tenant is provided for each log type: audit, infrastructure, and application logs. This enables access control for individual users and user groups to different log streams.
Apply the
LokiStack CR
object by running the following command:$ oc apply -f <filename>.yaml
Create a
ClusterLogging
CR object:Example ClusterLogging CR object
apiVersion: logging.openshift.io/v1 kind: ClusterLogging metadata: name: instance 1 namespace: openshift-logging 2 spec: collection: type: vector logStore: lokistack: name: logging-loki retentionPolicy: application: maxAge: 7d audit: maxAge: 7d infra: maxAge: 7d type: lokistack visualization: type: ocp-console ocpConsole: logsLimit: 15 managementState: Managed
Apply the
ClusterLogging CR
object by running the following command:$ oc apply -f <filename>.yaml
Verify the installation by running the following command:
$ oc get pods -n openshift-logging
Example output
$ oc get pods -n openshift-logging NAME READY STATUS RESTARTS AGE cluster-logging-operator-fb7f7cf69-8jsbq 1/1 Running 0 98m collector-222js 2/2 Running 0 18m collector-g9ddv 2/2 Running 0 18m collector-hfqq8 2/2 Running 0 18m collector-sphwg 2/2 Running 0 18m collector-vv7zn 2/2 Running 0 18m collector-wk5zz 2/2 Running 0 18m logging-view-plugin-6f76fbb78f-n2n4n 1/1 Running 0 18m lokistack-sample-compactor-0 1/1 Running 0 42m lokistack-sample-distributor-7d7688bcb9-dvcj8 1/1 Running 0 42m lokistack-sample-gateway-5f6c75f879-bl7k9 2/2 Running 0 42m lokistack-sample-gateway-5f6c75f879-xhq98 2/2 Running 0 42m lokistack-sample-index-gateway-0 1/1 Running 0 42m lokistack-sample-ingester-0 1/1 Running 0 42m lokistack-sample-querier-6b7b56bccc-2v9q4 1/1 Running 0 42m lokistack-sample-query-frontend-84fb57c578-gq2f7 1/1 Running 0 42m
5.4. Installing Logging and the Loki Operator using the web console
To install and configure logging on your Red Hat OpenShift Service on AWS cluster, an Operator such as Loki Operator for log storage must be installed first. This can be done from the OperatorHub within the web console.
Prerequisites
- You have access to a supported object store (AWS S3, Google Cloud Storage, Azure, Swift, Minio, OpenShift Data Foundation).
- You have administrator permissions.
- You have access to the Red Hat OpenShift Service on AWS web console.
Procedure
- In the Red Hat OpenShift Service on AWS web console Administrator perspective, go to Operators → OperatorHub.
Type Loki Operator in the Filter by keyword field. Click Loki Operator in the list of available Operators, and then click Install.
ImportantThe Community Loki Operator is not supported by Red Hat.
Select stable or stable-x.y as the Update channel.
NoteThe stable channel only provides updates to the most recent release of logging. To continue receiving updates for prior releases, you must change your subscription channel to stable-x.y, where
x.y
represents the major and minor version of logging you have installed. For example, stable-5.7.The Loki Operator must be deployed to the global operator group namespace
openshift-operators-redhat
, so the Installation mode and Installed Namespace are already selected. If this namespace does not already exist, it is created for you.Select Enable Operator-recommended cluster monitoring on this namespace.
This option sets the
openshift.io/cluster-monitoring: "true"
label in theNamespace
object. You must select this option to ensure that cluster monitoring scrapes theopenshift-operators-redhat
namespace.For Update approval select Automatic, then click Install.
If the approval strategy in the subscription is set to Automatic, the update process initiates as soon as a new Operator version is available in the selected channel. If the approval strategy is set to Manual, you must manually approve pending updates.
Install the Red Hat OpenShift Logging Operator:
- In the Red Hat OpenShift Service on AWS web console, click Operators → OperatorHub.
- Choose Red Hat OpenShift Logging from the list of available Operators, and click Install.
- Ensure that the A specific namespace on the cluster is selected under Installation Mode.
- Ensure that Operator recommended namespace is openshift-logging under Installed Namespace.
Select Enable Operator recommended cluster monitoring on this namespace.
This option sets the
openshift.io/cluster-monitoring: "true"
label in the Namespace object. You must select this option to ensure that cluster monitoring scrapes theopenshift-logging
namespace.- Select stable-5.y as the Update Channel.
Select an Approval Strategy.
- The Automatic strategy allows Operator Lifecycle Manager (OLM) to automatically update the Operator when a new version is available.
- The Manual strategy requires a user with appropriate credentials to approve the Operator update.
- Click Install.
- Go to the Operators → Installed Operators page. Click the All instances tab.
- From the Create new drop-down list, select LokiStack.
Select YAML view, and then use the following template to create a
LokiStack
CR:Example
LokiStack
CRapiVersion: loki.grafana.com/v1 kind: LokiStack metadata: name: logging-loki 1 namespace: openshift-logging 2 spec: size: 1x.small 3 storage: schemas: - version: v13 effectiveDate: "<yyyy>-<mm>-<dd>" secret: name: logging-loki-s3 4 type: s3 5 credentialMode: 6 storageClassName: <storage_class_name> 7 tenants: mode: openshift-logging 8
- 1
- Use the name
logging-loki
. - 2
- You must specify the
openshift-logging
namespace. - 3
- Specify the deployment size. In the logging 5.8 and later versions, the supported size options for production instances of Loki are
1x.extra-small
,1x.small
, or1x.medium
. - 4
- Specify the name of your log store secret.
- 5
- Specify the corresponding storage type.
- 6
- Optional field, logging 5.9 and later. Supported user configured values are as follows: static is the default authentication mode available for all supported object storage types using credentials stored in a Secret. token for short-lived tokens retrieved from a credential source. In this mode the static configuration does not contain credentials needed for the object storage. Instead, they are generated during runtime using a service, which allows for shorter-lived credentials and much more granular control. This authentication mode is not supported for all object storage types. token-cco is the default value when Loki is running on managed STS mode and using CCO on STS/WIF clusters.
- 7
- Specify the name of a storage class for temporary storage. For best performance, specify a storage class that allocates block storage. Available storage classes for your cluster can be listed by using the
oc get storageclasses
command. - 8
- LokiStack defaults to running in multi-tenant mode, which cannot be modified. One tenant is provided for each log type: audit, infrastructure, and application logs. This enables access control for individual users and user groups to different log streams.
ImportantIt is not possible to change the number
1x
for the deployment size.- Click Create.
Create an OpenShift Logging instance:
- Switch to the Administration → Custom Resource Definitions page.
- On the Custom Resource Definitions page, click ClusterLogging.
- On the Custom Resource Definition details page, select View Instances from the Actions menu.
On the ClusterLoggings page, click Create ClusterLogging.
You might have to refresh the page to load the data.
In the YAML field, replace the code with the following:
apiVersion: logging.openshift.io/v1 kind: ClusterLogging metadata: name: instance 1 namespace: openshift-logging 2 spec: collection: type: vector logStore: lokistack: name: logging-loki retentionPolicy: application: maxAge: 7d audit: maxAge: 7d infra: maxAge: 7d type: lokistack visualization: type: ocp-console ocpConsole: logsLimit: 15 managementState: Managed
Verification
- Go to Operators → Installed Operators.
- Make sure the openshift-logging project is selected.
- In the Status column, verify that you see green checkmarks with InstallSucceeded and the text Up to date.
An Operator might display a Failed
status before the installation finishes. If the Operator install completes with an InstallSucceeded
message, refresh the page.
Additional resources
Chapter 6. Updating Logging
There are two types of logging updates: minor release updates (5.y.z) and major release updates (5.y).
6.1. Minor release updates
If you installed the logging Operators using the Automatic update approval option, your Operators receive minor version updates automatically. You do not need to complete any manual update steps.
If you installed the logging Operators using the Manual update approval option, you must manually approve minor version updates. For more information, see Manually approving a pending Operator update.
6.2. Major release updates
For major version updates you must complete some manual steps.
For major release version compatibility and support information, see OpenShift Operator Life Cycles.
6.3. Upgrading the Red Hat OpenShift Logging Operator to watch all namespaces
In logging 5.7 and older versions, the Red Hat OpenShift Logging Operator only watches the openshift-logging
namespace. If you want the Red Hat OpenShift Logging Operator to watch all namespaces on your cluster, you must redeploy the Operator. You can complete the following procedure to redeploy the Operator without deleting your logging components.
Prerequisites
-
You have installed the OpenShift CLI (
oc
). - You have administrator permissions.
Procedure
Delete the subscription by running the following command:
$ oc -n openshift-logging delete subscription <subscription>
Delete the Operator group by running the following command:
$ oc -n openshift-logging delete operatorgroup <operator_group_name>
Delete the cluster service version (CSV) by running the following command:
$ oc delete clusterserviceversion cluster-logging.<version>
- Redeploy the Red Hat OpenShift Logging Operator by following the "Installing Logging" documentation.
Verification
Check that the
targetNamespaces
field in theOperatorGroup
resource is not present or is set to an empty string.To do this, run the following command and inspect the output:
$ oc get operatorgroup <operator_group_name> -o yaml
Example output
apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: openshift-logging-f52cn namespace: openshift-logging spec: upgradeStrategy: Default status: namespaces: - "" # ...
6.4. Updating the Red Hat OpenShift Logging Operator
To update the Red Hat OpenShift Logging Operator to a new major release version, you must modify the update channel for the Operator subscription.
Prerequisites
- You have installed the Red Hat OpenShift Logging Operator.
- You have administrator permissions.
- You have access to the Red Hat OpenShift Service on AWS web console and are viewing the Administrator perspective.
Procedure
- Navigate to Operators → Installed Operators.
- Select the openshift-logging project.
- Click the Red Hat OpenShift Logging Operator.
- Click Subscription. In the Subscription details section, click the Update channel link. This link text might be stable or stable-5.9, depending on your current update channel.
-
In the Change Subscription Update Channel window, select the latest major version update channel, stable-5.9, and click Save. Note the
cluster-logging.v5.9.<z>
version. -
Wait for a few seconds, and then go to Operators → Installed Operators to verify that the Red Hat OpenShift Logging Operator version matches the latest
cluster-logging.v5.9.<z>
version. - On the Operators → Installed Operators page, wait for the Status field to report Succeeded.
-
Check if the
LokiStack
custom resource contains thev13
schema version and add it if it is missing. For correctly adding thev13
schema version, see "Upgrading the LokiStack storage schema".
6.5. Updating the Loki Operator
To update the Loki Operator to a new major release version, you must modify the update channel for the Operator subscription.
Prerequisites
- You have installed the Loki Operator.
- You have administrator permissions.
- You have access to the Red Hat OpenShift Service on AWS web console and are viewing the Administrator perspective.
Procedure
- Navigate to Operators → Installed Operators.
- Select the openshift-operators-redhat project.
- Click the Loki Operator.
- Click Subscription. In the Subscription details section, click the Update channel link. This link text might be stable or stable-5.y, depending on your current update channel.
-
In the Change Subscription Update Channel window, select the latest major version update channel, stable-5.y, and click Save. Note the
loki-operator.v5.y.z
version. -
Wait for a few seconds, then click Operators → Installed Operators. Verify that the Loki Operator version matches the latest
loki-operator.v5.y.z
version. - On the Operators → Installed Operators page, wait for the Status field to report Succeeded.
-
Check if the
LokiStack
custom resource contains thev13
schema version and add it if it is missing. For correctly adding thev13
schema version, see "Upgrading the LokiStack storage schema".
6.6. Upgrading the LokiStack storage schema
If you are using the Red Hat OpenShift Logging Operator with the Loki Operator, the Red Hat OpenShift Logging Operator 5.9 or later supports the v13
schema version in the LokiStack
custom resource. Upgrading to the v13
schema version is recommended because it is the schema version to be supported going forward.
Procedure
Add the
v13
schema version in theLokiStack
custom resource as follows:apiVersion: loki.grafana.com/v1 kind: LokiStack # ... spec: # ... storage: schemas: # ... version: v12 1 - effectiveDate: "<yyyy>-<mm>-<future_dd>" 2 version: v13 # ...
TipTo edit the
LokiStack
custom resource, you can run theoc edit
command:$ oc edit lokistack <name> -n openshift-logging
Verification
-
On or after the specified
effectiveDate
date, check that there is no LokistackSchemaUpgradesRequired alert in the web console in Administrator → Observe → Alerting.
6.7. Updating the OpenShift Elasticsearch Operator
To update the OpenShift Elasticsearch Operator to the current version, you must modify the subscription.
The Logging 5.9 release does not contain an updated version of the OpenShift Elasticsearch Operator. If you currently use the OpenShift Elasticsearch Operator released with Logging 5.8, it will continue to work with Logging until the EOL of Logging 5.8. As an alternative to using the OpenShift Elasticsearch Operator to manage the default log storage, you can use the Loki Operator. For more information on the Logging lifecycle dates, see Platform Agnostic Operators.
Prerequisites
If you are using Elasticsearch as the default log store, and Kibana as the UI, update the OpenShift Elasticsearch Operator before you update the Red Hat OpenShift Logging Operator.
ImportantIf you update the Operators in the wrong order, Kibana does not update and the Kibana custom resource (CR) is not created. To fix this issue, delete the Red Hat OpenShift Logging Operator pod. When the Red Hat OpenShift Logging Operator pod redeploys, it creates the Kibana CR and Kibana becomes available again.
The Logging status is healthy:
-
All pods have a
ready
status. - The Elasticsearch cluster is healthy.
-
All pods have a
- Your Elasticsearch and Kibana data is backed up.
- You have administrator permissions.
-
You have installed the OpenShift CLI (
oc
) for the verification steps.
Procedure
- In the Red Hat Hybrid Cloud Console, click Operators → Installed Operators.
- Select the openshift-operators-redhat project.
- Click OpenShift Elasticsearch Operator.
- Click Subscription → Channel.
-
In the Change Subscription Update Channel window, select stable-5.y and click Save. Note the
elasticsearch-operator.v5.y.z
version. -
Wait for a few seconds, then click Operators → Installed Operators. Verify that the OpenShift Elasticsearch Operator version matches the latest
elasticsearch-operator.v5.y.z
version. - On the Operators → Installed Operators page, wait for the Status field to report Succeeded.
Verification
Verify that all Elasticsearch pods have a Ready status by entering the following command and observing the output:
$ oc get pod -n openshift-logging --selector component=elasticsearch
Example output
NAME READY STATUS RESTARTS AGE elasticsearch-cdm-1pbrl44l-1-55b7546f4c-mshhk 2/2 Running 0 31m elasticsearch-cdm-1pbrl44l-2-5c6d87589f-gx5hk 2/2 Running 0 30m elasticsearch-cdm-1pbrl44l-3-88df5d47-m45jc 2/2 Running 0 29m
Verify that the Elasticsearch cluster status is
green
by entering the following command and observing the output:$ oc exec -n openshift-logging -c elasticsearch elasticsearch-cdm-1pbrl44l-1-55b7546f4c-mshhk -- health
Example output
{ "cluster_name" : "elasticsearch", "status" : "green", }
Verify that the Elasticsearch cron jobs are created by entering the following commands and observing the output:
$ oc project openshift-logging
$ oc get cronjob
Example output
NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE elasticsearch-im-app */15 * * * * False 0 <none> 56s elasticsearch-im-audit */15 * * * * False 0 <none> 56s elasticsearch-im-infra */15 * * * * False 0 <none> 56s
Verify that the log store is updated to the correct version and the indices are
green
by entering the following command and observing the output:$ oc exec -c elasticsearch <any_es_pod_in_the_cluster> -- indices
Verify that the output includes the
app-00000x
,infra-00000x
,audit-00000x
,.security
indices:Example 6.1. Sample output with indices in a green status
Tue Jun 30 14:30:54 UTC 2020 health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open infra-000008 bnBvUFEXTWi92z3zWAzieQ 3 1 222195 0 289 144 green open infra-000004 rtDSzoqsSl6saisSK7Au1Q 3 1 226717 0 297 148 green open infra-000012 RSf_kUwDSR2xEuKRZMPqZQ 3 1 227623 0 295 147 green open .kibana_7 1SJdCqlZTPWlIAaOUd78yg 1 1 4 0 0 0 green open infra-000010 iXwL3bnqTuGEABbUDa6OVw 3 1 248368 0 317 158 green open infra-000009 YN9EsULWSNaxWeeNvOs0RA 3 1 258799 0 337 168 green open infra-000014 YP0U6R7FQ_GVQVQZ6Yh9Ig 3 1 223788 0 292 146 green open infra-000015 JRBbAbEmSMqK5X40df9HbQ 3 1 224371 0 291 145 green open .orphaned.2020.06.30 n_xQC2dWQzConkvQqei3YA 3 1 9 0 0 0 green open infra-000007 llkkAVSzSOmosWTSAJM_hg 3 1 228584 0 296 148 green open infra-000005 d9BoGQdiQASsS3BBFm2iRA 3 1 227987 0 297 148 green open infra-000003 1-goREK1QUKlQPAIVkWVaQ 3 1 226719 0 295 147 green open .security zeT65uOuRTKZMjg_bbUc1g 1 1 5 0 0 0 green open .kibana-377444158_kubeadmin wvMhDwJkR-mRZQO84K0gUQ 3 1 1 0 0 0 green open infra-000006 5H-KBSXGQKiO7hdapDE23g 3 1 226676 0 295 147 green open infra-000001 eH53BQ-bSxSWR5xYZB6lVg 3 1 341800 0 443 220 green open .kibana-6 RVp7TemSSemGJcsSUmuf3A 1 1 4 0 0 0 green open infra-000011 J7XWBauWSTe0jnzX02fU6A 3 1 226100 0 293 146 green open app-000001 axSAFfONQDmKwatkjPXdtw 3 1 103186 0 126 57 green open infra-000016 m9c1iRLtStWSF1GopaRyCg 3 1 13685 0 19 9 green open infra-000002 Hz6WvINtTvKcQzw-ewmbYg 3 1 228994 0 296 148 green open infra-000013 KR9mMFUpQl-jraYtanyIGw 3 1 228166 0 298 148 green open audit-000001 eERqLdLmQOiQDFES1LBATQ 3 1 0 0 0 0
Verify that the log visualizer is updated to the correct version by entering the following command and observing the output:
$ oc get kibana kibana -o json
Verify that the output includes a Kibana pod with the
ready
status:Example 6.2. Sample output with a ready Kibana pod
[ { "clusterCondition": { "kibana-5fdd766ffd-nb2jj": [ { "lastTransitionTime": "2020-06-30T14:11:07Z", "reason": "ContainerCreating", "status": "True", "type": "" }, { "lastTransitionTime": "2020-06-30T14:11:07Z", "reason": "ContainerCreating", "status": "True", "type": "" } ] }, "deployment": "kibana", "pods": { "failed": [], "notReady": [] "ready": [] }, "replicaSets": [ "kibana-5fdd766ffd" ], "replicas": 1 } ]
Chapter 7. Visualizing logs
7.1. About log visualization
You can visualize your log data in the Red Hat OpenShift Service on AWS web console, or the Kibana web console, depending on your deployed log storage solution. The Kibana console can be used with ElasticSearch log stores, and the Red Hat OpenShift Service on AWS web console can be used with the ElasticSearch log store or the LokiStack.
The Kibana web console is now deprecated is planned to be removed in a future logging release.
7.1.1. Configuring the log visualizer
You can configure which log visualizer type your logging uses by modifying the ClusterLogging
custom resource (CR).
Prerequisites
- You have administrator permissions.
-
You have installed the OpenShift CLI (
oc
). - You have installed the Red Hat OpenShift Logging Operator.
-
You have created a
ClusterLogging
CR.
If you want to use the Red Hat OpenShift Service on AWS web console for visualization, you must enable the logging Console Plugin. See the documentation about "Log visualization with the web console".
Procedure
Modify the
ClusterLogging
CRvisualization
spec:ClusterLogging
CR exampleapiVersion: logging.openshift.io/v1 kind: ClusterLogging metadata: # ... spec: # ... visualization: type: <visualizer_type> 1 kibana: 2 resources: {} nodeSelector: {} proxy: {} replicas: {} tolerations: {} ocpConsole: 3 logsLimit: {} timeout: {} # ...
- 1
- The type of visualizer you want to use for your logging. This can be either
kibana
orocp-console
. The Kibana console is only compatible with deployments that use Elasticsearch log storage, while the Red Hat OpenShift Service on AWS console is only compatible with LokiStack deployments. - 2
- Optional configurations for the Kibana console.
- 3
- Optional configurations for the Red Hat OpenShift Service on AWS web console.
Apply the
ClusterLogging
CR by running the following command:$ oc apply -f <filename>.yaml
7.1.2. Viewing logs for a resource
Resource logs are a default feature that provides limited log viewing capability. You can view the logs for various resources, such as builds, deployments, and pods by using the OpenShift CLI (oc
) and the web console.
To enhance your log retrieving and viewing experience, install the logging. The logging aggregates all the logs from your Red Hat OpenShift Service on AWS cluster, such as node system audit logs, application container logs, and infrastructure logs, into a dedicated log store. You can then query, discover, and visualize your log data through the Kibana console or the Red Hat OpenShift Service on AWS web console. Resource logs do not access the logging log store.
7.1.2.1. Viewing resource logs
You can view the log for various resources in the OpenShift CLI (oc
) and web console. Logs read from the tail, or end, of the log.
Prerequisites
-
Access to the OpenShift CLI (
oc
).
Procedure (UI)
In the Red Hat OpenShift Service on AWS console, navigate to Workloads → Pods or navigate to the pod through the resource you want to investigate.
NoteSome resources, such as builds, do not have pods to query directly. In such instances, you can locate the Logs link on the Details page for the resource.
- Select a project from the drop-down menu.
- Click the name of the pod you want to investigate.
- Click Logs.
Procedure (CLI)
View the log for a specific pod:
$ oc logs -f <pod_name> -c <container_name>
where:
-f
- Optional: Specifies that the output follows what is being written into the logs.
<pod_name>
- Specifies the name of the pod.
<container_name>
- Optional: Specifies the name of a container. When a pod has more than one container, you must specify the container name.
For example:
$ oc logs ruby-58cd97df55-mww7r
$ oc logs -f ruby-57f7f4855b-znl92 -c ruby
The contents of log files are printed out.
View the log for a specific resource:
$ oc logs <object_type>/<resource_name> 1
- 1
- Specifies the resource type and name.
For example:
$ oc logs deployment/ruby
The contents of log files are printed out.
7.2. Log visualization with the web console
You can use the Red Hat OpenShift Service on AWS web console to visualize log data by configuring the logging Console Plugin. Options for configuration are available during installation of logging on the web console.
If you have already installed logging and want to configure the plugin, use one of the following procedures.
7.2.1. Enabling the logging Console Plugin after you have installed the Red Hat OpenShift Logging Operator
You can enable the logging Console Plugin as part of the Red Hat OpenShift Logging Operator installation, but you can also enable the plugin if you have already installed the Red Hat OpenShift Logging Operator with the plugin disabled.
Prerequisites
- You have administrator permissions.
- You have installed the Red Hat OpenShift Logging Operator and selected Disabled for the Console plugin.
- You have access to the Red Hat OpenShift Service on AWS web console.
Procedure
- In the Red Hat OpenShift Service on AWS web console Administrator perspective, navigate to Operators → Installed Operators.
- Click Red Hat OpenShift Logging. This takes you to the Operator Details page.
- In the Details page, click Disabled for the Console plugin option.
- In the Console plugin enablement dialog, select Enable.
- Click Save.
- Verify that the Console plugin option now shows Enabled.
- The web console displays a pop-up window when changes have been applied. The window prompts you to reload the web console. Refresh the browser when you see the pop-up window to apply the changes.
7.2.2. Configuring the logging Console Plugin when you have the Elasticsearch log store and LokiStack installed
In logging version 5.8 and later, if the Elasticsearch log store is your default log store but you have also installed the LokiStack, you can enable the logging Console Plugin by using the following procedure.
Prerequisites
- You have administrator permissions.
- You have installed the Red Hat OpenShift Logging Operator, the OpenShift Elasticsearch Operator, and the Loki Operator.
-
You have installed the OpenShift CLI (
oc
). -
You have created a
ClusterLogging
custom resource (CR).
Procedure
Ensure that the logging Console Plugin is enabled by running the following command:
$ oc get consoles.operator.openshift.io cluster -o yaml |grep logging-view-plugin \ || oc patch consoles.operator.openshift.io cluster --type=merge \ --patch '{ "spec": { "plugins": ["logging-view-plugin"]}}'
Add the
.metadata.annotations.logging.openshift.io/ocp-console-migration-target: lokistack-dev
annotation to theClusterLogging
CR, by running the following command:$ oc patch clusterlogging instance --type=merge --patch \ '{ "metadata": { "annotations": { "logging.openshift.io/ocp-console-migration-target": "lokistack-dev" }}}' \ -n openshift-logging
Example output
clusterlogging.logging.openshift.io/instance patched
Verification
Verify that the annotation was added successfully, by running the following command and observing the output:
$ oc get clusterlogging instance \ -o=jsonpath='{.metadata.annotations.logging\.openshift\.io/ocp-console-migration-target}' \ -n openshift-logging
Example output
"lokistack-dev"
The logging Console Plugin pod is now deployed. You can view logging data by navigating to the Red Hat OpenShift Service on AWS web console and viewing the Observe → Logs page.
7.3. Viewing cluster dashboards
The Logging/Elasticsearch Nodes and Openshift Logging dashboards in the OpenShift Cluster Manager contain in-depth details about your Elasticsearch instance and the individual Elasticsearch nodes that you can use to prevent and diagnose problems.
The OpenShift Logging dashboard contains charts that show details about your Elasticsearch instance at a cluster level, including cluster resources, garbage collection, shards in the cluster, and Fluentd statistics.
The Logging/Elasticsearch Nodes dashboard contains charts that show details about your Elasticsearch instance, many at node level, including details on indexing, shards, resources, and so forth.
7.3.1. Accessing the Elasticsearch and OpenShift Logging dashboards
You can view the Logging/Elasticsearch Nodes and OpenShift Logging dashboards in the OpenShift Cluster Manager.
Procedure
To launch the dashboards:
- In the Red Hat OpenShift Service on AWS Red Hat Hybrid Cloud Console, click Observe → Dashboards.
On the Dashboards page, select Logging/Elasticsearch Nodes or OpenShift Logging from the Dashboard menu.
For the Logging/Elasticsearch Nodes dashboard, you can select the Elasticsearch node you want to view and set the data resolution.
The appropriate dashboard is displayed, showing multiple charts of data.
- Optional: Select a different time range to display or refresh rate for the data from the Time Range and Refresh Interval menus.
For information on the dashboard charts, see About the OpenShift Logging dashboard and About the Logging/Elastisearch Nodes dashboard.
7.3.2. About the OpenShift Logging dashboard
The OpenShift Logging dashboard contains charts that show details about your Elasticsearch instance at a cluster-level that you can use to diagnose and anticipate problems.
Metric | Description |
---|---|
Elastic Cluster Status | The current Elasticsearch status:
|
Elastic Nodes | The total number of Elasticsearch nodes in the Elasticsearch instance. |
Elastic Shards | The total number of Elasticsearch shards in the Elasticsearch instance. |
Elastic Documents | The total number of Elasticsearch documents in the Elasticsearch instance. |
Total Index Size on Disk | The total disk space that is being used for the Elasticsearch indices. |
Elastic Pending Tasks | The total number of Elasticsearch changes that have not been completed, such as index creation, index mapping, shard allocation, or shard failure. |
Elastic JVM GC time | The amount of time that the JVM spent executing Elasticsearch garbage collection operations in the cluster. |
Elastic JVM GC Rate | The total number of times that JVM executed garbage activities per second. |
Elastic Query/Fetch Latency Sum |
Fetch latency typically takes less time than query latency. If fetch latency is consistently increasing, it might indicate slow disks, data enrichment, or large requests with too many results. |
Elastic Query Rate | The total queries executed against the Elasticsearch instance per second for each Elasticsearch node. |
CPU | The amount of CPU used by Elasticsearch, Fluentd, and Kibana, shown for each component. |
Elastic JVM Heap Used | The amount of JVM memory used. In a healthy cluster, the graph shows regular drops as memory is freed by JVM garbage collection. |
Elasticsearch Disk Usage | The total disk space used by the Elasticsearch instance for each Elasticsearch node. |
File Descriptors In Use | The total number of file descriptors used by Elasticsearch, Fluentd, and Kibana. |
FluentD emit count | The total number of Fluentd messages per second for the Fluentd default output, and the retry count for the default output. |
FluentD Buffer Usage | The percent of the Fluentd buffer that is being used for chunks. A full buffer might indicate that Fluentd is not able to process the number of logs received. |
Elastic rx bytes | The total number of bytes that Elasticsearch has received from FluentD, the Elasticsearch nodes, and other sources. |
Elastic Index Failure Rate | The total number of times per second that an Elasticsearch index fails. A high rate might indicate an issue with indexing. |
FluentD Output Error Rate | The total number of times per second that FluentD is not able to output logs. |
7.3.3. Charts on the Logging/Elasticsearch nodes dashboard
The Logging/Elasticsearch Nodes dashboard contains charts that show details about your Elasticsearch instance, many at node-level, for further diagnostics.
- Elasticsearch status
- The Logging/Elasticsearch Nodes dashboard contains the following charts about the status of your Elasticsearch instance.
Metric | Description |
---|---|
Cluster status | The cluster health status during the selected time period, using the Elasticsearch green, yellow, and red statuses:
|
Cluster nodes | The total number of Elasticsearch nodes in the cluster. |
Cluster data nodes | The number of Elasticsearch data nodes in the cluster. |
Cluster pending tasks | The number of cluster state changes that are not finished and are waiting in a cluster queue, for example, index creation, index deletion, or shard allocation. A growing trend indicates that the cluster is not able to keep up with changes. |
- Elasticsearch cluster index shard status
- Each Elasticsearch index is a logical group of one or more shards, which are basic units of persisted data. There are two types of index shards: primary shards, and replica shards. When a document is indexed into an index, it is stored in one of its primary shards and copied into every replica of that shard. The number of primary shards is specified when the index is created, and the number cannot change during index lifetime. You can change the number of replica shards at any time.
The index shard can be in several states depending on its lifecycle phase or events occurring in the cluster. When the shard is able to perform search and indexing requests, the shard is active. If the shard cannot perform these requests, the shard is non–active. A shard might be non-active if the shard is initializing, reallocating, unassigned, and so forth.
Index shards consist of a number of smaller internal blocks, called index segments, which are physical representations of the data. An index segment is a relatively small, immutable Lucene index that is created when Lucene commits newly-indexed data. Lucene, a search library used by Elasticsearch, merges index segments into larger segments in the background to keep the total number of segments low. If the process of merging segments is slower than the speed at which new segments are created, it could indicate a problem.
When Lucene performs data operations, such as a search operation, Lucene performs the operation against the index segments in the relevant index. For that purpose, each segment contains specific data structures that are loaded in the memory and mapped. Index mapping can have a significant impact on the memory used by segment data structures.
The Logging/Elasticsearch Nodes dashboard contains the following charts about the Elasticsearch index shards.
Metric | Description |
---|---|
Cluster active shards | The number of active primary shards and the total number of shards, including replicas, in the cluster. If the number of shards grows higher, the cluster performance can start degrading. |
Cluster initializing shards | The number of non-active shards in the cluster. A non-active shard is one that is initializing, being reallocated to a different node, or is unassigned. A cluster typically has non–active shards for short periods. A growing number of non–active shards over longer periods could indicate a problem. |
Cluster relocating shards | The number of shards that Elasticsearch is relocating to a new node. Elasticsearch relocates nodes for multiple reasons, such as high memory use on a node or after a new node is added to the cluster. |
Cluster unassigned shards | The number of unassigned shards. Elasticsearch shards might be unassigned for reasons such as a new index being added or the failure of a node. |
- Elasticsearch node metrics
- Each Elasticsearch node has a finite amount of resources that can be used to process tasks. When all the resources are being used and Elasticsearch attempts to perform a new task, Elasticsearch puts the tasks into a queue until some resources become available.
The Logging/Elasticsearch Nodes dashboard contains the following charts about resource usage for a selected node and the number of tasks waiting in the Elasticsearch queue.
Metric | Description |
---|---|
ThreadPool tasks | The number of waiting tasks in individual queues, shown by task type. A long–term accumulation of tasks in any queue could indicate node resource shortages or some other problem. |
CPU usage | The amount of CPU being used by the selected Elasticsearch node as a percentage of the total CPU allocated to the host container. |
Memory usage | The amount of memory being used by the selected Elasticsearch node. |
Disk usage | The total disk space being used for index data and metadata on the selected Elasticsearch node. |
Documents indexing rate | The rate that documents are indexed on the selected Elasticsearch node. |
Indexing latency | The time taken to index the documents on the selected Elasticsearch node. Indexing latency can be affected by many factors, such as JVM Heap memory and overall load. A growing latency indicates a resource capacity shortage in the instance. |
Search rate | The number of search requests run on the selected Elasticsearch node. |
Search latency | The time taken to complete search requests on the selected Elasticsearch node. Search latency can be affected by many factors. A growing latency indicates a resource capacity shortage in the instance. |
Documents count (with replicas) | The number of Elasticsearch documents stored on the selected Elasticsearch node, including documents stored in both the primary shards and replica shards that are allocated on the node. |
Documents deleting rate | The number of Elasticsearch documents being deleted from any of the index shards that are allocated to the selected Elasticsearch node. |
Documents merging rate | The number of Elasticsearch documents being merged in any of index shards that are allocated to the selected Elasticsearch node. |
- Elasticsearch node fielddata
- Fielddata is an Elasticsearch data structure that holds lists of terms in an index and is kept in the JVM Heap. Because fielddata building is an expensive operation, Elasticsearch caches the fielddata structures. Elasticsearch can evict a fielddata cache when the underlying index segment is deleted or merged, or if there is not enough JVM HEAP memory for all the fielddata caches.
The Logging/Elasticsearch Nodes dashboard contains the following charts about Elasticsearch fielddata.
Metric | Description |
---|---|
Fielddata memory size | The amount of JVM Heap used for the fielddata cache on the selected Elasticsearch node. |
Fielddata evictions | The number of fielddata structures that were deleted from the selected Elasticsearch node. |
- Elasticsearch node query cache
- If the data stored in the index does not change, search query results are cached in a node-level query cache for reuse by Elasticsearch.
The Logging/Elasticsearch Nodes dashboard contains the following charts about the Elasticsearch node query cache.
Metric | Description |
---|---|
Query cache size | The total amount of memory used for the query cache for all the shards allocated to the selected Elasticsearch node. |
Query cache evictions | The number of query cache evictions on the selected Elasticsearch node. |
Query cache hits | The number of query cache hits on the selected Elasticsearch node. |
Query cache misses | The number of query cache misses on the selected Elasticsearch node. |
- Elasticsearch index throttling
- When indexing documents, Elasticsearch stores the documents in index segments, which are physical representations of the data. At the same time, Elasticsearch periodically merges smaller segments into a larger segment as a way to optimize resource use. If the indexing is faster then the ability to merge segments, the merge process does not complete quickly enough, which can lead to issues with searches and performance. To prevent this situation, Elasticsearch throttles indexing, typically by reducing the number of threads allocated to indexing down to a single thread.
The Logging/Elasticsearch Nodes dashboard contains the following charts about Elasticsearch index throttling.
Metric | Description |
---|---|
Indexing throttling | The amount of time that Elasticsearch has been throttling the indexing operations on the selected Elasticsearch node. |
Merging throttling | The amount of time that Elasticsearch has been throttling the segment merge operations on the selected Elasticsearch node. |
- Node JVM Heap statistics
- The Logging/Elasticsearch Nodes dashboard contains the following charts about JVM Heap operations.
Metric | Description |
---|---|
Heap used | The amount of the total allocated JVM Heap space that is used on the selected Elasticsearch node. |
GC count | The number of garbage collection operations that have been run on the selected Elasticsearch node, by old and young garbage collection. |
GC time | The amount of time that the JVM spent running garbage collection operations on the selected Elasticsearch node, by old and young garbage collection. |
7.4. Log visualization with Kibana
If you are using the ElasticSearch log store, you can use the Kibana console to visualize collected log data.
Using Kibana, you can do the following with your data:
- Search and browse the data using the Discover tab.
- Chart and map the data using the Visualize tab.
- Create and view custom dashboards using the Dashboard tab.
Use and configuration of the Kibana interface is beyond the scope of this documentation. For more information about using the interface, see the Kibana documentation.
The audit logs are not stored in the internal Red Hat OpenShift Service on AWS Elasticsearch instance by default. To view the audit logs in Kibana, you must use the Log Forwarding API to configure a pipeline that uses the default
output for audit logs.
7.4.1. Defining Kibana index patterns
An index pattern defines the Elasticsearch indices that you want to visualize. To explore and visualize data in Kibana, you must create an index pattern.
Prerequisites
A user must have the
cluster-admin
role, thecluster-reader
role, or both roles to view the infra and audit indices in Kibana. The defaultkubeadmin
user has proper permissions to view these indices.If you can view the pods and logs in the
default
,kube-
andopenshift-
projects, you should be able to access these indices. You can use the following command to check if the current user has appropriate permissions:$ oc auth can-i get pods --subresource log -n <project>
Example output
yes
NoteThe audit logs are not stored in the internal Red Hat OpenShift Service on AWS Elasticsearch instance by default. To view the audit logs in Kibana, you must use the Log Forwarding API to configure a pipeline that uses the
default
output for audit logs.- Elasticsearch documents must be indexed before you can create index patterns. This is done automatically, but it might take a few minutes in a new or updated cluster.
Procedure
To define index patterns and create visualizations in Kibana:
- In the Red Hat OpenShift Service on AWS console, click the Application Launcher and select Logging.
Create your Kibana index patterns by clicking Management → Index Patterns → Create index pattern:
-
Each user must manually create index patterns when logging into Kibana the first time to see logs for their projects. Users must create an index pattern named
app
and use the@timestamp
time field to view their container logs. -
Each admin user must create index patterns when logged into Kibana the first time for the
app
,infra
, andaudit
indices using the@timestamp
time field.
-
Each user must manually create index patterns when logging into Kibana the first time to see logs for their projects. Users must create an index pattern named
- Create Kibana Visualizations from the new index patterns.
7.4.2. Viewing cluster logs in Kibana
You view cluster logs in the Kibana web console. The methods for viewing and visualizing your data in Kibana that are beyond the scope of this documentation. For more information, refer to the Kibana documentation.
Prerequisites
- The Red Hat OpenShift Logging and Elasticsearch Operators must be installed.
- Kibana index patterns must exist.
A user must have the
cluster-admin
role, thecluster-reader
role, or both roles to view the infra and audit indices in Kibana. The defaultkubeadmin
user has proper permissions to view these indices.If you can view the pods and logs in the
default
,kube-
andopenshift-
projects, you should be able to access these indices. You can use the following command to check if the current user has appropriate permissions:$ oc auth can-i get pods --subresource log -n <project>
Example output
yes
NoteThe audit logs are not stored in the internal Red Hat OpenShift Service on AWS Elasticsearch instance by default. To view the audit logs in Kibana, you must use the Log Forwarding API to configure a pipeline that uses the
default
output for audit logs.
Procedure
To view logs in Kibana:
- In the Red Hat OpenShift Service on AWS console, click the Application Launcher and select Logging.
Log in using the same credentials you use to log in to the Red Hat OpenShift Service on AWS console.
The Kibana interface launches.
- In Kibana, click Discover.
Select the index pattern you created from the drop-down menu in the top-left corner: app, audit, or infra.
The log data displays as time-stamped documents.
- Expand one of the time-stamped documents.
Click the JSON tab to display the log entry for that document.
Example 7.1. Sample infrastructure log entry in Kibana
{ "_index": "infra-000001", "_type": "_doc", "_id": "YmJmYTBlNDkZTRmLTliMGQtMjE3NmFiOGUyOWM3", "_version": 1, "_score": null, "_source": { "docker": { "container_id": "f85fa55bbef7bb783f041066be1e7c267a6b88c4603dfce213e32c1" }, "kubernetes": { "container_name": "registry-server", "namespace_name": "openshift-marketplace", "pod_name": "redhat-marketplace-n64gc", "container_image": "registry.redhat.io/redhat/redhat-marketplace-index:v4.7", "container_image_id": "registry.redhat.io/redhat/redhat-marketplace-index@sha256:65fc0c45aabb95809e376feb065771ecda9e5e59cc8b3024c4545c168f", "pod_id": "8f594ea2-c866-4b5c-a1c8-a50756704b2a", "host": "ip-10-0-182-28.us-east-2.compute.internal", "master_url": "https://kubernetes.default.svc", "namespace_id": "3abab127-7669-4eb3-b9ef-44c04ad68d38", "namespace_labels": { "openshift_io/cluster-monitoring": "true" }, "flat_labels": [ "catalogsource_operators_coreos_com/update=redhat-marketplace" ] }, "message": "time=\"2020-09-23T20:47:03Z\" level=info msg=\"serving registry\" database=/database/index.db port=50051", "level": "unknown", "hostname": "ip-10-0-182-28.internal", "pipeline_metadata": { "collector": { "ipaddr4": "10.0.182.28", "inputname": "fluent-plugin-systemd", "name": "fluentd", "received_at": "2020-09-23T20:47:15.007583+00:00", "version": "1.7.4 1.6.0" } }, "@timestamp": "2020-09-23T20:47:03.422465+00:00", "viaq_msg_id": "YmJmYTBlNDktMDMGQtMjE3NmFiOGUyOWM3", "openshift": { "labels": { "logging": "infra" } } }, "fields": { "@timestamp": [ "2020-09-23T20:47:03.422Z" ], "pipeline_metadata.collector.received_at": [ "2020-09-23T20:47:15.007Z" ] }, "sort": [ 1600894023422 ] }
7.4.3. Configuring Kibana
You can configure using the Kibana console by modifying the ClusterLogging
custom resource (CR).
7.4.3.1. Configuring CPU and memory limits
The logging components allow for adjustments to both the CPU and memory limits.
Procedure
Edit the
ClusterLogging
custom resource (CR) in theopenshift-logging
project:$ oc -n openshift-logging edit ClusterLogging instance
apiVersion: "logging.openshift.io/v1" kind: "ClusterLogging" metadata: name: "instance" namespace: openshift-logging ... spec: managementState: "Managed" logStore: type: "elasticsearch" elasticsearch: nodeCount: 3 resources: 1 limits: memory: 16Gi requests: cpu: 200m memory: 16Gi storage: storageClassName: "gp2" size: "200G" redundancyPolicy: "SingleRedundancy" visualization: type: "kibana" kibana: resources: 2 limits: memory: 1Gi requests: cpu: 500m memory: 1Gi proxy: resources: 3 limits: memory: 100Mi requests: cpu: 100m memory: 100Mi replicas: 2 collection: logs: type: "fluentd" fluentd: resources: 4 limits: memory: 736Mi requests: cpu: 200m memory: 736Mi
- 1
- Specify the CPU and memory limits and requests for the log store as needed. For Elasticsearch, you must adjust both the request value and the limit value.
- 2 3
- Specify the CPU and memory limits and requests for the log visualizer as needed.
- 4
- Specify the CPU and memory limits and requests for the log collector as needed.
7.4.3.2. Scaling redundancy for the log visualizer nodes
You can scale the pod that hosts the log visualizer for redundancy.
Procedure
Edit the
ClusterLogging
custom resource (CR) in theopenshift-logging
project:$ oc edit ClusterLogging instance
$ oc edit ClusterLogging instance apiVersion: "logging.openshift.io/v1" kind: "ClusterLogging" metadata: name: "instance" .... spec: visualization: type: "kibana" kibana: replicas: 1 1
- 1
- Specify the number of Kibana nodes.
Chapter 8. Configuring your Logging deployment
8.1. Configuring CPU and memory limits for logging components
You can configure both the CPU and memory limits for each of the logging components as needed.
8.1.1. Configuring CPU and memory limits
The logging components allow for adjustments to both the CPU and memory limits.
Procedure
Edit the
ClusterLogging
custom resource (CR) in theopenshift-logging
project:$ oc -n openshift-logging edit ClusterLogging instance
apiVersion: "logging.openshift.io/v1" kind: "ClusterLogging" metadata: name: "instance" namespace: openshift-logging ... spec: managementState: "Managed" logStore: type: "elasticsearch" elasticsearch: nodeCount: 3 resources: 1 limits: memory: 16Gi requests: cpu: 200m memory: 16Gi storage: storageClassName: "gp2" size: "200G" redundancyPolicy: "SingleRedundancy" visualization: type: "kibana" kibana: resources: 2 limits: memory: 1Gi requests: cpu: 500m memory: 1Gi proxy: resources: 3 limits: memory: 100Mi requests: cpu: 100m memory: 100Mi replicas: 2 collection: logs: type: "fluentd" fluentd: resources: 4 limits: memory: 736Mi requests: cpu: 200m memory: 736Mi
- 1
- Specify the CPU and memory limits and requests for the log store as needed. For Elasticsearch, you must adjust both the request value and the limit value.
- 2 3
- Specify the CPU and memory limits and requests for the log visualizer as needed.
- 4
- Specify the CPU and memory limits and requests for the log collector as needed.
Chapter 9. Log collection and forwarding
9.1. About log collection and forwarding
The Red Hat OpenShift Logging Operator deploys a collector based on the ClusterLogForwarder
resource specification. There are two collector options supported by this Operator: the legacy Fluentd collector, and the Vector collector.
Fluentd is deprecated and is planned to be removed in a future release. Red Hat provides bug fixes and support for this feature during the current release lifecycle, but this feature no longer receives enhancements. As an alternative to Fluentd, you can use Vector instead.
9.1.1. Log collection
The log collector is a daemon set that deploys pods to each Red Hat OpenShift Service on AWS node to collect container and node logs.
By default, the log collector uses the following sources:
- System and infrastructure logs generated by journald log messages from the operating system, the container runtime, and Red Hat OpenShift Service on AWS.
-
/var/log/containers/*.log
for all container logs.
If you configure the log collector to collect audit logs, it collects them from /var/log/audit/audit.log
.
The log collector collects the logs from these sources and forwards them internally or externally depending on your logging configuration.
9.1.1.1. Log collector types
Vector is a log collector offered as an alternative to Fluentd for the logging.
You can configure which logging collector type your cluster uses by modifying the ClusterLogging
custom resource (CR) collection
spec:
Example ClusterLogging CR that configures Vector as the collector
apiVersion: logging.openshift.io/v1 kind: ClusterLogging metadata: name: instance namespace: openshift-logging spec: collection: logs: type: vector vector: {} # ...
9.1.1.2. Log collection limitations
The container runtimes provide minimal information to identify the source of log messages: project, pod name, and container ID. This information is not sufficient to uniquely identify the source of the logs. If a pod with a given name and project is deleted before the log collector begins processing its logs, information from the API server, such as labels and annotations, might not be available. There might not be a way to distinguish the log messages from a similarly named pod and project or trace the logs to their source. This limitation means that log collection and normalization are considered best effort.
The available container runtimes provide minimal information to identify the source of log messages and do not guarantee unique individual log messages or that these messages can be traced to their source.
9.1.1.3. Log collector features by type
Feature | Fluentd | Vector |
---|---|---|
App container logs | ✓ | ✓ |
App-specific routing | ✓ | ✓ |
App-specific routing by namespace | ✓ | ✓ |
Infra container logs | ✓ | ✓ |
Infra journal logs | ✓ | ✓ |
Kube API audit logs | ✓ | ✓ |
OpenShift API audit logs | ✓ | ✓ |
Open Virtual Network (OVN) audit logs | ✓ | ✓ |
Feature | Fluentd | Vector |
---|---|---|
Elasticsearch certificates | ✓ | ✓ |
Elasticsearch username / password | ✓ | ✓ |
Amazon Cloudwatch keys | ✓ | ✓ |
Amazon Cloudwatch STS | ✓ | ✓ |
Kafka certificates | ✓ | ✓ |
Kafka username / password | ✓ | ✓ |
Kafka SASL | ✓ | ✓ |
Loki bearer token | ✓ | ✓ |
Feature | Fluentd | Vector |
---|---|---|
Viaq data model - app | ✓ | ✓ |
Viaq data model - infra | ✓ | ✓ |
Viaq data model - infra(journal) | ✓ | ✓ |
Viaq data model - Linux audit | ✓ | ✓ |
Viaq data model - kube-apiserver audit | ✓ | ✓ |
Viaq data model - OpenShift API audit | ✓ | ✓ |
Viaq data model - OVN | ✓ | ✓ |
Loglevel Normalization | ✓ | ✓ |
JSON parsing | ✓ | ✓ |
Structured Index | ✓ | ✓ |
Multiline error detection | ✓ | ✓ |
Multicontainer / split indices | ✓ | ✓ |
Flatten labels | ✓ | ✓ |
CLF static labels | ✓ | ✓ |
Feature | Fluentd | Vector |
---|---|---|
Fluentd readlinelimit | ✓ | |
Fluentd buffer | ✓ | |
- chunklimitsize | ✓ | |
- totallimitsize | ✓ | |
- overflowaction | ✓ | |
- flushthreadcount | ✓ | |
- flushmode | ✓ | |
- flushinterval | ✓ | |
- retrywait | ✓ | |
- retrytype | ✓ | |
- retrymaxinterval | ✓ | |
- retrytimeout | ✓ |
Feature | Fluentd | Vector |
---|---|---|
Metrics | ✓ | ✓ |
Dashboard | ✓ | ✓ |
Alerts | ✓ | ✓ |
Feature | Fluentd | Vector |
---|---|---|
Global proxy support | ✓ | ✓ |
x86 support | ✓ | ✓ |
ARM support | ✓ | ✓ |
IBM Power® support | ✓ | ✓ |
IBM Z® support | ✓ | ✓ |
IPv6 support | ✓ | ✓ |
Log event buffering | ✓ | |
Disconnected Cluster | ✓ | ✓ |
9.1.1.4. Collector outputs
The following collector outputs are supported:
Feature | Fluentd | Vector |
---|---|---|
Elasticsearch v6-v8 | ✓ | ✓ |
Fluent forward | ✓ | |
Syslog RFC3164 | ✓ | ✓ (Logging 5.7+) |
Syslog RFC5424 | ✓ | ✓ (Logging 5.7+) |
Kafka | ✓ | ✓ |
Amazon Cloudwatch | ✓ | ✓ |
Amazon Cloudwatch STS | ✓ | ✓ |
Loki | ✓ | ✓ |
HTTP | ✓ | ✓ (Logging 5.7+) |
Google Cloud Logging | ✓ | ✓ |
Splunk | ✓ (Logging 5.6+) |
9.1.2. Log forwarding
Administrators can create ClusterLogForwarder
resources that specify which logs are collected, how they are transformed, and where they are forwarded to.
ClusterLogForwarder
resources can be used up to forward container, infrastructure, and audit logs to specific endpoints within or outside of a cluster. Transport Layer Security (TLS) is supported so that log forwarders can be configured to send logs securely.
Administrators can also authorize RBAC permissions that define which service accounts and users can access and forward which types of logs.
9.1.2.1. Log forwarding implementations
There are two log forwarding implementations available: the legacy implementation, and the multi log forwarder feature.
Only the Vector collector is supported for use with the multi log forwarder feature. The Fluentd collector can only be used with legacy implementations.
9.1.2.1.1. Legacy implementation
In legacy implementations, you can only use one log forwarder in your cluster. The ClusterLogForwarder
resource in this mode must be named instance
, and must be created in the openshift-logging
namespace. The ClusterLogForwarder
resource also requires a corresponding ClusterLogging
resource named instance
in the openshift-logging
namespace.
9.1.2.1.2. Multi log forwarder feature
The multi log forwarder feature is available in logging 5.8 and later, and provides the following functionality:
- Administrators can control which users are allowed to define log collection and which logs they are allowed to collect.
- Users who have the required permissions are able to specify additional log collection configurations.
- Administrators who are migrating from the deprecated Fluentd collector to the Vector collector can deploy a new log forwarder separately from their existing deployment. The existing and new log forwarders can operate simultaneously while workloads are being migrated.
In multi log forwarder implementations, you are not required to create a corresponding ClusterLogging
resource for your ClusterLogForwarder
resource. You can create multiple ClusterLogForwarder
resources using any name, in any namespace, with the following exceptions:
-
You cannot create a
ClusterLogForwarder
resource namedinstance
in theopenshift-logging
namespace, because this is reserved for a log forwarder that supports the legacy workflow using the Fluentd collector. -
You cannot create a
ClusterLogForwarder
resource namedcollector
in theopenshift-logging
namespace, because this is reserved for the collector.
9.1.2.2. Enabling the multi log forwarder feature for a cluster
To use the multi log forwarder feature, you must create a service account and cluster role bindings for that service account. You can then reference the service account in the ClusterLogForwarder
resource to control access permissions.
In order to support multi log forwarding in additional namespaces other than the openshift-logging
namespace, you must update the Red Hat OpenShift Logging Operator to watch all namespaces]. This functionality is supported by default in new Red Hat OpenShift Logging Operator version 5.8 installations.
9.1.2.2.1. Authorizing log collection RBAC permissions
In logging 5.8 and later, the Red Hat OpenShift Logging Operator provides collect-audit-logs
, collect-application-logs
, and collect-infrastructure-logs
cluster roles, which enable the collector to collect audit logs, application logs, and infrastructure logs respectively.
You can authorize RBAC permissions for log collection by binding the required cluster roles to a service account.
Prerequisites
-
The Red Hat OpenShift Logging Operator is installed in the
openshift-logging
namespace. - You have administrator permissions.
Procedure
- Create a service account for the collector. If you want to write logs to storage that requires a token for authentication, you must include a token in the service account.
Bind the appropriate cluster roles to the service account:
Example binding command
$ oc adm policy add-cluster-role-to-user <cluster_role_name> system:serviceaccount:<namespace_name>:<service_account_name>
Additional resources
9.2. Log output types
Outputs define the destination where logs are sent to from a log forwarder. You can configure multiple types of outputs in the ClusterLogForwarder
custom resource (CR) to send logs to servers that support different protocols.
9.2.1. Supported log forwarding outputs
Outputs can be any of the following types:
Output type | Protocol | Tested with | Logging versions | Supported collector type |
---|---|---|---|---|
Elasticsearch v6 | HTTP 1.1 | 6.8.1, 6.8.23 | 5.6+ | Fluentd, Vector |
Elasticsearch v7 | HTTP 1.1 | 7.12.2, 7.17.7, 7.10.1 | 5.6+ | Fluentd, Vector |
Elasticsearch v8 | HTTP 1.1 | 8.4.3, 8.6.1 | 5.6+ | Fluentd [1], Vector |
Fluent Forward | Fluentd forward v1 | Fluentd 1.14.6, Logstash 7.10.1, Fluentd 1.14.5 | 5.4+ | Fluentd |
Google Cloud Logging | REST over HTTPS | Latest | 5.7+ | Vector |
HTTP | HTTP 1.1 | Fluentd 1.14.6, Vector 0.21 | 5.7+ | Fluentd, Vector |
Kafka | Kafka 0.11 | Kafka 2.4.1, 2.7.0, 3.3.1 | 5.4+ | Fluentd, Vector |
Loki | REST over HTTP and HTTPS | 2.3.0, 2.5.0, 2.7, 2.2.1 | 5.4+ | Fluentd, Vector |
Splunk | HEC | 8.2.9, 9.0.0 | 5.7+ | Vector |
Syslog | RFC3164, RFC5424 | Rsyslog 8.37.0-9.el7, rsyslog-8.39.0 | 5.4+ | Fluentd, Vector [2] |
Amazon CloudWatch | REST over HTTPS | Latest | 5.4+ | Fluentd, Vector |
- Fluentd does not support Elasticsearch 8 in the logging version 5.6.2.
- Vector supports Syslog in the logging version 5.7 and higher.
9.2.2. Output type descriptions
default
The on-cluster, Red Hat managed log store. You are not required to configure the default output.
NoteIf you configure a
default
output, you receive an error message, because thedefault
output name is reserved for referencing the on-cluster, Red Hat managed log store.loki
- Loki, a horizontally scalable, highly available, multi-tenant log aggregation system.
kafka
-
A Kafka broker. The
kafka
output can use a TCP or TLS connection. elasticsearch
-
An external Elasticsearch instance. The
elasticsearch
output can use a TLS connection. fluentdForward
An external log aggregation solution that supports Fluentd. This option uses the Fluentd
forward
protocols. ThefluentForward
output can use a TCP or TLS connection and supports shared-key authentication by providing ashared_key
field in a secret. Shared-key authentication can be used with or without TLS.ImportantThe
fluentdForward
output is only supported if you are using the Fluentd collector. It is not supported if you are using the Vector collector. If you are using the Vector collector, you can forward logs to Fluentd by using thehttp
output.syslog
-
An external log aggregation solution that supports the syslog RFC3164 or RFC5424 protocols. The
syslog
output can use a UDP, TCP, or TLS connection. cloudwatch
- Amazon CloudWatch, a monitoring and log storage service hosted by Amazon Web Services (AWS).
cloudlogging
- Google Cloud Logging, a monitoring and log storage service hosted by Google Cloud Platform (GCP).
9.3. Enabling JSON log forwarding
You can configure the Log Forwarding API to parse JSON strings into a structured object.
9.3.1. Parsing JSON logs
You can use a ClusterLogForwarder
object to parse JSON logs into a structured object and forward them to a supported output.
To illustrate how this works, suppose that you have the following structured JSON log entry:
Example structured JSON log entry
{"level":"info","name":"fred","home":"bedrock"}
To enable parsing JSON log, you add parse: json
to a pipeline in the ClusterLogForwarder
CR, as shown in the following example:
Example snippet showing parse: json
pipelines: - inputRefs: [ application ] outputRefs: myFluentd parse: json
When you enable parsing JSON logs by using parse: json
, the CR copies the JSON-structured log entry in a structured
field, as shown in the following example:
Example structured
output containing the structured JSON log entry
{"structured": { "level": "info", "name": "fred", "home": "bedrock" }, "more fields..."}
If the log entry does not contain valid structured JSON, the structured
field is absent.
9.3.2. Configuring JSON log data for Elasticsearch
If your JSON logs follow more than one schema, storing them in a single index might cause type conflicts and cardinality problems. To avoid that, you must configure the ClusterLogForwarder
custom resource (CR) to group each schema into a single output definition. This way, each schema is forwarded to a separate index.
If you forward JSON logs to the default Elasticsearch instance managed by OpenShift Logging, it generates new indices based on your configuration. To avoid performance issues associated with having too many indices, consider keeping the number of possible schemas low by standardizing to common schemas.
Structure types
You can use the following structure types in the ClusterLogForwarder
CR to construct index names for the Elasticsearch log store:
structuredTypeKey
is the name of a message field. The value of that field is used to construct the index name.-
kubernetes.labels.<key>
is the Kubernetes pod label whose value is used to construct the index name. -
openshift.labels.<key>
is thepipeline.label.<key>
element in theClusterLogForwarder
CR whose value is used to construct the index name. -
kubernetes.container_name
uses the container name to construct the index name.
-
-
structuredTypeName
: If thestructuredTypeKey
field is not set or its key is not present, thestructuredTypeName
value is used as the structured type. When you use both thestructuredTypeKey
field and thestructuredTypeName
field together, thestructuredTypeName
value provides a fallback index name if the key in thestructuredTypeKey
field is missing from the JSON log data.
Although you can set the value of structuredTypeKey
to any field shown in the "Log Record Fields" topic, the most useful fields are shown in the preceding list of structure types.
A structuredTypeKey: kubernetes.labels.<key> example
Suppose the following:
- Your cluster is running application pods that produce JSON logs in two different formats, "apache" and "google".
-
The user labels these application pods with
logFormat=apache
andlogFormat=google
. -
You use the following snippet in your
ClusterLogForwarder
CR YAML file.
apiVersion: logging.openshift.io/v1 kind: ClusterLogForwarder metadata: # ... spec: # ... outputDefaults: elasticsearch: structuredTypeKey: kubernetes.labels.logFormat 1 structuredTypeName: nologformat pipelines: - inputRefs: - application outputRefs: - default parse: json 2
In that case, the following structured log record goes to the app-apache-write
index:
{ "structured":{"name":"fred","home":"bedrock"}, "kubernetes":{"labels":{"logFormat": "apache", ...}} }
And the following structured log record goes to the app-google-write
index:
{ "structured":{"name":"wilma","home":"bedrock"}, "kubernetes":{"labels":{"logFormat": "google", ...}} }
A structuredTypeKey: openshift.labels.<key> example
Suppose that you use the following snippet in your ClusterLogForwarder
CR YAML file.
outputDefaults: elasticsearch: structuredTypeKey: openshift.labels.myLabel 1 structuredTypeName: nologformat pipelines: - name: application-logs inputRefs: - application - audit outputRefs: - elasticsearch-secure - default parse: json labels: myLabel: myValue 2
In that case, the following structured log record goes to the app-myValue-write
index:
{ "structured":{"name":"fred","home":"bedrock"}, "openshift":{"labels":{"myLabel": "myValue", ...}} }
Additional considerations
- The Elasticsearch index for structured records is formed by prepending "app-" to the structured type and appending "-write".
- Unstructured records are not sent to the structured index. They are indexed as usual in the application, infrastructure, or audit indices.
-
If there is no non-empty structured type, forward an unstructured record with no
structured
field.
It is important not to overload Elasticsearch with too many indices. Only use distinct structured types for distinct log formats, not for each application or namespace. For example, most Apache applications use the same JSON log format and structured type, such as LogApache
.
9.3.3. Forwarding JSON logs to the Elasticsearch log store
For an Elasticsearch log store, if your JSON log entries follow different schemas, configure the ClusterLogForwarder
custom resource (CR) to group each JSON schema into a single output definition. This way, Elasticsearch uses a separate index for each schema.
Because forwarding different schemas to the same index can cause type conflicts and cardinality problems, you must perform this configuration before you forward data to the Elasticsearch store.
To avoid performance issues associated with having too many indices, consider keeping the number of possible schemas low by standardizing to common schemas.
Procedure
Add the following snippet to your
ClusterLogForwarder
CR YAML file.outputDefaults: elasticsearch: structuredTypeKey: <log record field> structuredTypeName: <name> pipelines: - inputRefs: - application outputRefs: default parse: json
-
Use
structuredTypeKey
field to specify one of the log record fields. Use
structuredTypeName
field to specify a name.ImportantTo parse JSON logs, you must set both the
structuredTypeKey
andstructuredTypeName
fields.-
For
inputRefs
, specify which log types to forward by using that pipeline, such asapplication,
infrastructure
, oraudit
. -
Add the
parse: json
element to pipelines. Create the CR object:
$ oc create -f <filename>.yaml
The Red Hat OpenShift Logging Operator redeploys the collector pods. However, if they do not redeploy, delete the collector pods to force them to redeploy.
$ oc delete pod --selector logging-infra=collector
9.3.4. Forwarding JSON logs from containers in the same pod to separate indices
You can forward structured logs from different containers within the same pod to different indices. To use this feature, you must configure the pipeline with multi-container support and annotate the pods. Logs are written to indices with a prefix of app-
. It is recommended that Elasticsearch be configured with aliases to accommodate this.
JSON formatting of logs varies by application. Because creating too many indices impacts performance, limit your use of this feature to creating indices for logs that have incompatible JSON formats. Use queries to separate logs from different namespaces, or applications with compatible JSON formats.
Prerequisites
- Logging for Red Hat OpenShift: 5.5
Procedure
Create or edit a YAML file that defines the
ClusterLogForwarder
CR object:apiVersion: logging.openshift.io/v1 kind: ClusterLogForwarder metadata: name: instance namespace: openshift-logging spec: outputDefaults: elasticsearch: structuredTypeKey: kubernetes.labels.logFormat 1 structuredTypeName: nologformat enableStructuredContainerLogs: true 2 pipelines: - inputRefs: - application name: application-logs outputRefs: - default parse: json
Create or edit a YAML file that defines the
Pod
CR object:apiVersion: v1 kind: Pod metadata: annotations: containerType.logging.openshift.io/heavy: heavy 1 containerType.logging.openshift.io/low: low spec: containers: - name: heavy 2 image: heavyimage - name: low image: lowimage
This configuration might significantly increase the number of shards on the cluster.
Additional resources
Additional resources
9.4. Configuring log forwarding
In a logging deployment, container and infrastructure logs are forwarded to the internal log store defined in the ClusterLogging
custom resource (CR) by default.
Audit logs are not forwarded to the internal log store by default because this does not provide secure storage. You are responsible for ensuring that the system to which you forward audit logs is compliant with your organizational and governmental regulations, and is properly secured.
If this default configuration meets your needs, you do not need to configure a ClusterLogForwarder
CR. If a ClusterLogForwarder
CR exists, logs are not forwarded to the internal log store unless a pipeline is defined that contains the default
output.
9.4.1. About forwarding logs to third-party systems
To send logs to specific endpoints inside and outside your Red Hat OpenShift Service on AWS cluster, you specify a combination of outputs and pipelines in a ClusterLogForwarder
custom resource (CR). You can also use inputs to forward the application logs associated with a specific project to an endpoint. Authentication is provided by a Kubernetes Secret object.
- pipeline
Defines simple routing from one log type to one or more outputs, or which logs you want to send. The log types are one of the following:
-
application
. Container logs generated by user applications running in the cluster, except infrastructure container applications. -
infrastructure
. Container logs from pods that run in theopenshift*
,kube*
, ordefault
projects and journal logs sourced from node file system. -
audit
. Audit logs generated by the node audit system,auditd
, Kubernetes API server, OpenShift API server, and OVN network.
You can add labels to outbound log messages by using
key:value
pairs in the pipeline. For example, you might add a label to messages that are forwarded to other data centers or label the logs by type. Labels that are added to objects are also forwarded with the log message.-
- input
Forwards the application logs associated with a specific project to a pipeline.
In the pipeline, you define which log types to forward using an
inputRef
parameter and where to forward the logs to using anoutputRef
parameter.- Secret
-
A
key:value map
that contains confidential data such as user credentials.
Note the following:
-
If you do not define a pipeline for a log type, the logs of the undefined types are dropped. For example, if you specify a pipeline for the
application
andaudit
types, but do not specify a pipeline for theinfrastructure
type,infrastructure
logs are dropped. -
You can use multiple types of outputs in the
ClusterLogForwarder
custom resource (CR) to send logs to servers that support different protocols.
The following example forwards the audit logs to a secure external Elasticsearch instance, the infrastructure logs to an insecure external Elasticsearch instance, the application logs to a Kafka broker, and the application logs from the my-apps-logs
project to the internal Elasticsearch instance.
Sample log forwarding outputs and pipelines
apiVersion: "logging.openshift.io/v1" kind: ClusterLogForwarder metadata: name: <log_forwarder_name> 1 namespace: <log_forwarder_namespace> 2 spec: serviceAccountName: <service_account_name> 3 outputs: - name: elasticsearch-secure 4 type: "elasticsearch" url: https://elasticsearch.secure.com:9200 secret: name: elasticsearch - name: elasticsearch-insecure 5 type: "elasticsearch" url: http://elasticsearch.insecure.com:9200 - name: kafka-app 6 type: "kafka" url: tls://kafka.secure.com:9093/app-topic inputs: 7 - name: my-app-logs application: namespaces: - my-project pipelines: - name: audit-logs 8 inputRefs: - audit outputRefs: - elasticsearch-secure - default labels: secure: "true" 9 datacenter: "east" - name: infrastructure-logs 10 inputRefs: - infrastructure outputRefs: - elasticsearch-insecure labels: datacenter: "west" - name: my-app 11 inputRefs: - my-app-logs outputRefs: - default - inputRefs: 12 - application outputRefs: - kafka-app labels: datacenter: "south"
- 1
- In legacy implementations, the CR name must be
instance
. In multi log forwarder implementations, you can use any name. - 2
- In legacy implementations, the CR namespace must be
openshift-logging
. In multi log forwarder implementations, you can use any namespace. - 3
- The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the
openshift-logging
namespace. - 4
- Configuration for an secure Elasticsearch output using a secret with a secure URL.
- A name to describe the output.
-
The type of output:
elasticsearch
. - The secure URL and port of the Elasticsearch instance as a valid absolute URL, including the prefix.
-
The secret required by the endpoint for TLS communication. The secret must exist in the
openshift-logging
project.
- 5
- Configuration for an insecure Elasticsearch output:
- A name to describe the output.
-
The type of output:
elasticsearch
. - The insecure URL and port of the Elasticsearch instance as a valid absolute URL, including the prefix.
- 6
- Configuration for a Kafka output using a client-authenticated TLS communication over a secure URL:
- A name to describe the output.
-
The type of output:
kafka
. - Specify the URL and port of the Kafka broker as a valid absolute URL, including the prefix.
- 7
- Configuration for an input to filter application logs from the
my-project
namespace. - 8
- Configuration for a pipeline to send audit logs to the secure external Elasticsearch instance:
- A name to describe the pipeline.
-
The
inputRefs
is the log type, in this exampleaudit
. -
The
outputRefs
is the name of the output to use, in this exampleelasticsearch-secure
to forward to the secure Elasticsearch instance anddefault
to forward to the internal Elasticsearch instance. - Optional: Labels to add to the logs.
- 9
- Optional: String. One or more labels to add to the logs. Quote values like "true" so they are recognized as string values, not as a boolean.
- 10
- Configuration for a pipeline to send infrastructure logs to the insecure external Elasticsearch instance.
- 11
- Configuration for a pipeline to send logs from the
my-project
project to the internal Elasticsearch instance.- A name to describe the pipeline.
-
The
inputRefs
is a specific input:my-app-logs
. -
The
outputRefs
isdefault
. - Optional: String. One or more labels to add to the logs.
- 12
- Configuration for a pipeline to send logs to the Kafka broker, with no pipeline name:
-
The
inputRefs
is the log type, in this exampleapplication
. -
The
outputRefs
is the name of the output to use. - Optional: String. One or more labels to add to the logs.
-
The
Fluentd log handling when the external log aggregator is unavailable
If your external logging aggregator becomes unavailable and cannot receive logs, Fluentd continues to collect logs and stores them in a buffer. When the log aggregator becomes available, log forwarding resumes, including the buffered logs. If the buffer fills completely, Fluentd stops collecting logs. Red Hat OpenShift Service on AWS rotates the logs and deletes them. You cannot adjust the buffer size or add a persistent volume claim (PVC) to the Fluentd daemon set or pods.
Supported Authorization Keys
Common key types are provided here. Some output types support additional specialized keys, documented with the output-specific configuration field. All secret keys are optional. Enable the security features you want by setting the relevant keys. You are responsible for creating and maintaining any additional configurations that external destinations might require, such as keys and secrets, service accounts, port openings, or global proxy configuration. Open Shift Logging will not attempt to verify a mismatch between authorization combinations.
- Transport Layer Security (TLS)
Using a TLS URL (
http://...
orssl://...
) without a secret enables basic TLS server-side authentication. Additional TLS features are enabled by including a secret and setting the following optional fields:-
passphrase
: (string) Passphrase to decode an encoded TLS private key. Requirestls.key
. -
ca-bundle.crt
: (string) File name of a customer CA for server authentication.
-
- Username and Password
-
username
: (string) Authentication user name. Requirespassword
. -
password
: (string) Authentication password. Requiresusername
.
-
- Simple Authentication Security Layer (SASL)
-
sasl.enable
(boolean) Explicitly enable or disable SASL. If missing, SASL is automatically enabled when any of the othersasl.
keys are set. -
sasl.mechanisms
: (array) List of allowed SASL mechanism names. If missing or empty, the system defaults are used. -
sasl.allow-insecure
: (boolean) Allow mechanisms that send clear-text passwords. Defaults to false.
-
9.4.1.1. Creating a Secret
You can create a secret in the directory that contains your certificate and key files by using the following command:
$ oc create secret generic -n <namespace> <secret_name> \ --from-file=ca-bundle.crt=<your_bundle_file> \ --from-literal=username=<your_username> \ --from-literal=password=<your_password>
Generic or opaque secrets are recommended for best results.
9.4.2. Creating a log forwarder
To create a log forwarder, you must create a ClusterLogForwarder
CR that specifies the log input types that the service account can collect. You can also specify which outputs the logs can be forwarded to. If you are using the multi log forwarder feature, you must also reference the service account in the ClusterLogForwarder
CR.
If you are using the multi log forwarder feature on your cluster, you can create ClusterLogForwarder
custom resources (CRs) in any namespace, using any name. If you are using a legacy implementation, the ClusterLogForwarder
CR must be named instance
, and must be created in the openshift-logging
namespace.
You need administrator permissions for the namespace where you create the ClusterLogForwarder
CR.
ClusterLogForwarder resource example
apiVersion: logging.openshift.io/v1 kind: ClusterLogForwarder metadata: name: <log_forwarder_name> 1 namespace: <log_forwarder_namespace> 2 spec: serviceAccountName: <service_account_name> 3 pipelines: - inputRefs: - <log_type> 4 outputRefs: - <output_name> 5 outputs: - name: <output_name> 6 type: <output_type> 7 url: <log_output_url> 8 # ...
- 1
- In legacy implementations, the CR name must be
instance
. In multi log forwarder implementations, you can use any name. - 2
- In legacy implementations, the CR namespace must be
openshift-logging
. In multi log forwarder implementations, you can use any namespace. - 3
- The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the
openshift-logging
namespace. - 4
- The log types that are collected. The value for this field can be
audit
for audit logs,application
for application logs,infrastructure
for infrastructure logs, or a named input that has been defined for your application. - 5 7
- The type of output that you want to forward logs to. The value of this field can be
default
,loki
,kafka
,elasticsearch
,fluentdForward
,syslog
, orcloudwatch
.NoteThe
default
output type is not supported in mutli log forwarder implementations. - 6
- A name for the output that you want to forward logs to.
- 8
- The URL of the output that you want to forward logs to.
9.4.3. Tuning log payloads and delivery
In logging 5.9 and newer versions, the tuning
spec in the ClusterLogForwarder
custom resource (CR) provides a means of configuring your deployment to prioritize either throughput or durability of logs.
For example, if you need to reduce the possibility of log loss when the collector restarts, or you require collected log messages to survive a collector restart to support regulatory mandates, you can tune your deployment to prioritize log durability. If you use outputs that have hard limitations on the size of batches they can receive, you may want to tune your deployment to prioritize log throughput.
To use this feature, your logging deployment must be configured to use the Vector collector. The tuning
spec in the ClusterLogForwarder
CR is not supported when using the Fluentd collector.
The following example shows the ClusterLogForwarder
CR options that you can modify to tune log forwarder outputs:
Example ClusterLogForwarder
CR tuning options
apiVersion: logging.openshift.io/v1 kind: ClusterLogForwarder metadata: # ... spec: tuning: delivery: AtLeastOnce 1 compression: none 2 maxWrite: <integer> 3 minRetryDuration: 1s 4 maxRetryDuration: 1s 5 # ...
- 1
- Specify the delivery mode for log forwarding.
-
AtLeastOnce
delivery means that if the log forwarder crashes or is restarted, any logs that were read before the crash but not sent to their destination are re-sent. It is possible that some logs are duplicated after a crash. -
AtMostOnce
delivery means that the log forwarder makes no effort to recover logs lost during a crash. This mode gives better throughput, but may result in greater log loss.
-
- 2
- Specifying a
compression
configuration causes data to be compressed before it is sent over the network. Note that not all output types support compression, and if the specified compression type is not supported by the output, this results in an error. The possible values for this configuration arenone
for no compression,gzip
,snappy
,zlib
, orzstd
.lz4
compression is also available if you are using a Kafka output. See the table "Supported compression types for tuning outputs" for more information. - 3
- Specifies a limit for the maximum payload of a single send operation to the output.
- 4
- Specifies a minimum duration to wait between attempts before retrying delivery after a failure. This value is a string, and can be specified as milliseconds (
ms
), seconds (s
), or minutes (m
). - 5
- Specifies a maximum duration to wait between attempts before retrying delivery after a failure. This value is a string, and can be specified as milliseconds (
ms
), seconds (s
), or minutes (m
).
Compression algorithm | Splunk | Amazon Cloudwatch | Elasticsearch 8 | LokiStack | Apache Kafka | HTTP | Syslog | Google Cloud | Microsoft Azure Monitoring |
---|---|---|---|---|---|---|---|---|---|
| X | X | X | X | X | ||||
| X | X | X | X | |||||
| X | X | X | ||||||
| X | X | X | ||||||
| X |
9.4.4. Enabling multi-line exception detection
Enables multi-line error detection of container logs.
Enabling this feature could have performance implications and may require additional computing resources or alternate logging solutions.
Log parsers often incorrectly identify separate lines of the same exception as separate exceptions. This leads to extra log entries and an incomplete or inaccurate view of the traced information.
Example java exception
java.lang.NullPointerException: Cannot invoke "String.toString()" because "<param1>" is null at testjava.Main.handle(Main.java:47) at testjava.Main.printMe(Main.java:19) at testjava.Main.main(Main.java:10)
-
To enable logging to detect multi-line exceptions and reassemble them into a single log entry, ensure that the
ClusterLogForwarder
Custom Resource (CR) contains adetectMultilineErrors
field, with a value oftrue
.
Example ClusterLogForwarder CR
apiVersion: logging.openshift.io/v1 kind: ClusterLogForwarder metadata: name: instance namespace: openshift-logging spec: pipelines: - name: my-app-logs inputRefs: - application outputRefs: - default detectMultilineErrors: true
9.4.4.1. Details
When log messages appear as a consecutive sequence forming an exception stack trace, they are combined into a single, unified log record. The first log message’s content is replaced with the concatenated content of all the message fields in the sequence.
Language | Fluentd | Vector |
---|---|---|
Java | ✓ | ✓ |
JS | ✓ | ✓ |
Ruby | ✓ | ✓ |
Python | ✓ | ✓ |
Golang | ✓ | ✓ |
PHP | ✓ | ✓ |
Dart | ✓ | ✓ |
9.4.4.2. Troubleshooting
When enabled, the collector configuration will include a new section with type: detect_exceptions
Example vector configuration section
[transforms.detect_exceptions_app-logs] type = "detect_exceptions" inputs = ["application"] languages = ["All"] group_by = ["kubernetes.namespace_name","kubernetes.pod_name","kubernetes.container_name"] expire_after_ms = 2000 multiline_flush_interval_ms = 1000
Example fluentd config section
<label @MULTILINE_APP_LOGS> <match kubernetes.**> @type detect_exceptions remove_tag_prefix 'kubernetes' message message force_line_breaks true multiline_flush_interval .2 </match> </label>
9.4.5. Forwarding logs to Splunk
You can forward logs to the Splunk HTTP Event Collector (HEC) in addition to, or instead of, the internal default Red Hat OpenShift Service on AWS log store.
Using this feature with Fluentd is not supported.
Prerequisites
- Red Hat OpenShift Logging Operator 5.6 or later
-
A
ClusterLogging
instance withvector
specified as the collector - Base64 encoded Splunk HEC token
Procedure
Create a secret using your Base64 encoded Splunk HEC token.
$ oc -n openshift-logging create secret generic vector-splunk-secret --from-literal hecToken=<HEC_Token>
Create or edit the
ClusterLogForwarder
Custom Resource (CR) using the template below:apiVersion: logging.openshift.io/v1 kind: ClusterLogForwarder metadata: name: <log_forwarder_name> 1 namespace: <log_forwarder_namespace> 2 spec: serviceAccountName: <service_account_name> 3 outputs: - name: splunk-receiver 4 secret: name: vector-splunk-secret 5 type: splunk 6 url: <http://your.splunk.hec.url:8088> 7 pipelines: 8 - inputRefs: - application - infrastructure name: 9 outputRefs: - splunk-receiver 10
- 1
- In legacy implementations, the CR name must be
instance
. In multi log forwarder implementations, you can use any name. - 2
- In legacy implementations, the CR namespace must be
openshift-logging
. In multi log forwarder implementations, you can use any namespace. - 3
- The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the
openshift-logging
namespace. - 4
- Specify a name for the output.
- 5
- Specify the name of the secret that contains your HEC token.
- 6
- Specify the output type as
splunk
. - 7
- Specify the URL (including port) of your Splunk HEC.
- 8
- Specify which log types to forward by using the pipeline:
application
,infrastructure
, oraudit
. - 9
- Optional: Specify a name for the pipeline.
- 10
- Specify the name of the output to use when forwarding logs with this pipeline.
9.4.6. Forwarding logs over HTTP
Forwarding logs over HTTP is supported for both the Fluentd and Vector log collectors. To enable, specify http
as the output type in the ClusterLogForwarder
custom resource (CR).
Procedure
Create or edit the
ClusterLogForwarder
CR using the template below:Example ClusterLogForwarder CR
apiVersion: logging.openshift.io/v1 kind: ClusterLogForwarder metadata: name: <log_forwarder_name> 1 namespace: <log_forwarder_namespace> 2 spec: serviceAccountName: <service_account_name> 3 outputs: - name: httpout-app type: http url: 4 http: headers: 5 h1: v1 h2: v2 method: POST secret: name: 6 tls: insecureSkipVerify: 7 pipelines: - name: inputRefs: - application outputRefs: - httpout-app 8
- 1
- In legacy implementations, the CR name must be
instance
. In multi log forwarder implementations, you can use any name. - 2
- In legacy implementations, the CR namespace must be
openshift-logging
. In multi log forwarder implementations, you can use any namespace. - 3
- The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the
openshift-logging
namespace. - 4
- Destination address for logs.
- 5
- Additional headers to send with the log record.
- 6
- Secret name for destination credentials.
- 7
- Values are either
true
orfalse
. - 8
- This value should be the same as the output name.
9.4.7. Forwarding to Azure Monitor Logs
With logging 5.9 and later, you can forward logs to Azure Monitor Logs in addition to, or instead of, the default log store. This functionality is provided by the Vector Azure Monitor Logs sink.
Prerequisites
-
You are familiar with how to administer and create a
ClusterLogging
custom resource (CR) instance. -
You are familiar with how to administer and create a
ClusterLogForwarder
CR instance. -
You understand the
ClusterLogForwarder
CR specifications. - You have basic familiarity with Azure services.
- You have an Azure account configured for Azure Portal or Azure CLI access.
- You have obtained your Azure Monitor Logs primary or the secondary security key.
- You have determined which log types to forward.
To enable log forwarding to Azure Monitor Logs via the HTTP Data Collector API:
Create a secret with your shared key:
apiVersion: v1
kind: Secret
metadata:
name: my-secret
namespace: openshift-logging
type: Opaque
data:
shared_key: <your_shared_key> 1
- 1
- Must contain a primary or secondary key for the Log Analytics workspace making the request.
To obtain a shared key, you can use this command in Azure CLI:
Get-AzOperationalInsightsWorkspaceSharedKey -ResourceGroupName "<resource_name>" -Name "<workspace_name>”
Create or edit your ClusterLogForwarder
CR using the template matching your log selection.
Forward all logs
apiVersion: "logging.openshift.io/v1" kind: "ClusterLogForwarder" metadata: name: instance namespace: openshift-logging spec: outputs: - name: azure-monitor type: azureMonitor azureMonitor: customerId: my-customer-id 1 logType: my_log_type 2 secret: name: my-secret pipelines: - name: app-pipeline inputRefs: - application outputRefs: - azure-monitor
- 1
- Unique identifier for the Log Analytics workspace. Required field.
- 2
- Azure record type of the data being submitted. May only contain letters, numbers, and underscores (_), and may not exceed 100 characters.
Forward application and infrastructure logs
apiVersion: "logging.openshift.io/v1"
kind: "ClusterLogForwarder"
metadata:
name: instance
namespace: openshift-logging
spec:
outputs:
- name: azure-monitor-app
type: azureMonitor
azureMonitor:
customerId: my-customer-id
logType: application_log 1
secret:
name: my-secret
- name: azure-monitor-infra
type: azureMonitor
azureMonitor:
customerId: my-customer-id
logType: infra_log #
secret:
name: my-secret
pipelines:
- name: app-pipeline
inputRefs:
- application
outputRefs:
- azure-monitor-app
- name: infra-pipeline
inputRefs:
- infrastructure
outputRefs:
- azure-monitor-infra
- 1
- Azure record type of the data being submitted. May only contain letters, numbers, and underscores (_), and may not exceed 100 characters.
Advanced configuration options
apiVersion: "logging.openshift.io/v1" kind: "ClusterLogForwarder" metadata: name: instance namespace: openshift-logging spec: outputs: - name: azure-monitor type: azureMonitor azureMonitor: customerId: my-customer-id logType: my_log_type azureResourceId: "/subscriptions/111111111" 1 host: "ods.opinsights.azure.com" 2 secret: name: my-secret pipelines: - name: app-pipeline inputRefs: - application outputRefs: - azure-monitor
9.4.8. Forwarding application logs from specific projects
You can forward a copy of the application logs from specific projects to an external log aggregator, in addition to, or instead of, using the internal log store. You must also configure the external log aggregator to receive log data from Red Hat OpenShift Service on AWS.
To configure forwarding application logs from a project, you must create a ClusterLogForwarder
custom resource (CR) with at least one input from a project, optional outputs for other log aggregators, and pipelines that use those inputs and outputs.
Prerequisites
- You must have a logging server that is configured to receive the logging data using the specified protocol or format.
Procedure
Create or edit a YAML file that defines the
ClusterLogForwarder
CR:Example
ClusterLogForwarder
CRapiVersion: logging.openshift.io/v1 kind: ClusterLogForwarder metadata: name: instance 1 namespace: openshift-logging 2 spec: outputs: - name: fluentd-server-secure 3 type: fluentdForward 4 url: 'tls://fluentdserver.security.example.com:24224' 5 secret: 6 name: fluentd-secret - name: fluentd-server-insecure type: fluentdForward url: 'tcp://fluentdserver.home.example.com:24224' inputs: 7 - name: my-app-logs application: namespaces: - my-project 8 pipelines: - name: forward-to-fluentd-insecure 9 inputRefs: 10 - my-app-logs outputRefs: 11 - fluentd-server-insecure labels: project: "my-project" 12 - name: forward-to-fluentd-secure 13 inputRefs: - application 14 - audit - infrastructure outputRefs: - fluentd-server-secure - default labels: clusterId: "C1234"
- 1
- The name of the
ClusterLogForwarder
CR must beinstance
. - 2
- The namespace for the
ClusterLogForwarder
CR must beopenshift-logging
. - 3
- The name of the output.
- 4
- The output type:
elasticsearch
,fluentdForward
,syslog
, orkafka
. - 5
- The URL and port of the external log aggregator as a valid absolute URL. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP address.
- 6
- If using a
tls
prefix, you must specify the name of the secret required by the endpoint for TLS communication. The secret must exist in theopenshift-logging
project and have tls.crt, tls.key, and ca-bundle.crt keys that each point to the certificates they represent. - 7
- The configuration for an input to filter application logs from the specified projects.
- 8
- If no namespace is specified, logs are collected from all namespaces.
- 9
- The pipeline configuration directs logs from a named input to a named output. In this example, a pipeline named
forward-to-fluentd-insecure
forwards logs from an input namedmy-app-logs
to an output namedfluentd-server-insecure
. - 10
- A list of inputs.
- 11
- The name of the output to use.
- 12
- Optional: String. One or more labels to add to the logs.
- 13
- Configuration for a pipeline to send logs to other log aggregators.
- Optional: Specify a name for the pipeline.
-
Specify which log types to forward by using the pipeline:
application,
infrastructure
, oraudit
. - Specify the name of the output to use when forwarding logs with this pipeline.
-
Optional: Specify the
default
output to forward logs to the default log store. - Optional: String. One or more labels to add to the logs.
- 14
- Note that application logs from all namespaces are collected when using this configuration.
Apply the
ClusterLogForwarder
CR by running the following command:$ oc apply -f <filename>.yaml
9.4.9. Forwarding application logs from specific pods
As a cluster administrator, you can use Kubernetes pod labels to gather log data from specific pods and forward it to a log collector.
Suppose that you have an application composed of pods running alongside other pods in various namespaces. If those pods have labels that identify the application, you can gather and output their log data to a specific log collector.
To specify the pod labels, you use one or more matchLabels
key-value pairs. If you specify multiple key-value pairs, the pods must match all of them to be selected.
Procedure
Create or edit a YAML file that defines the
ClusterLogForwarder
CR object. In the file, specify the pod labels using simple equality-based selectors underinputs[].name.application.selector.matchLabels
, as shown in the following example.Example
ClusterLogForwarder
CR YAML fileapiVersion: logging.openshift.io/v1 kind: ClusterLogForwarder metadata: name: <log_forwarder_name> 1 namespace: <log_forwarder_namespace> 2 spec: pipelines: - inputRefs: [ myAppLogData ] 3 outputRefs: [ default ] 4 inputs: 5 - name: myAppLogData application: selector: matchLabels: 6 environment: production app: nginx namespaces: 7 - app1 - app2 outputs: 8 - <output_name> ...
- 1
- In legacy implementations, the CR name must be
instance
. In multi log forwarder implementations, you can use any name. - 2
- In legacy implementations, the CR namespace must be
openshift-logging
. In multi log forwarder implementations, you can use any namespace. - 3
- Specify one or more comma-separated values from
inputs[].name
. - 4
- Specify one or more comma-separated values from
outputs[]
. - 5
- Define a unique
inputs[].name
for each application that has a unique set of pod labels. - 6
- Specify the key-value pairs of pod labels whose log data you want to gather. You must specify both a key and value, not just a key. To be selected, the pods must match all the key-value pairs.
- 7
- Optional: Specify one or more namespaces.
- 8
- Specify one or more outputs to forward your log data to.
-
Optional: To restrict the gathering of log data to specific namespaces, use
inputs[].name.application.namespaces
, as shown in the preceding example. Optional: You can send log data from additional applications that have different pod labels to the same pipeline.
-
For each unique combination of pod labels, create an additional
inputs[].name
section similar to the one shown. -
Update the
selectors
to match the pod labels of this application. Add the new
inputs[].name
value toinputRefs
. For example:- inputRefs: [ myAppLogData, myOtherAppLogData ]
-
For each unique combination of pod labels, create an additional
Create the CR object:
$ oc create -f <file-name>.yaml
Additional resources
-
For more information on
matchLabels
in Kubernetes, see Resources that support set-based requirements.
9.4.10. Overview of API audit filter
OpenShift API servers generate audit events for each API call, detailing the request, response, and the identity of the requester, leading to large volumes of data. The API Audit filter uses rules to enable the exclusion of non-essential events and the reduction of event size, facilitating a more manageable audit trail. Rules are checked in order, checking stops at the first match. How much data is included in an event is determined by the value of the level
field:
-
None
: The event is dropped. -
Metadata
: Audit metadata is included, request and response bodies are removed. -
Request
: Audit metadata and the request body are included, the response body is removed. -
RequestResponse
: All data is included: metadata, request body and response body. The response body can be very large. For example,oc get pods -A
generates a response body containing the YAML description of every pod in the cluster.
You can use this feature only if the Vector collector is set up in your logging deployment.
In logging 5.8 and later, the ClusterLogForwarder
custom resource (CR) uses the same format as the standard Kubernetes audit policy, while providing the following additional functions:
- Wildcards
-
Names of users, groups, namespaces, and resources can have a leading or trailing
*
asterisk character. For example, namespaceopenshift-\*
matchesopenshift-apiserver
oropenshift-authentication
. Resource\*/status
matchesPod/status
orDeployment/status
. - Default Rules
Events that do not match any rule in the policy are filtered as follows:
-
Read-only system events such as
get
,list
,watch
are dropped. - Service account write events that occur within the same namespace as the service account are dropped.
- All other events are forwarded, subject to any configured rate limits.
-
Read-only system events such as
To disable these defaults, either end your rules list with a rule that has only a level
field or add an empty rule.
- Omit Response Codes
-
A list of integer status codes to omit. You can drop events based on the HTTP status code in the response by using the
OmitResponseCodes
field, a list of HTTP status code for which no events are created. The default value is[404, 409, 422, 429]
. If the value is an empty list,[]
, then no status codes are omitted.
The ClusterLogForwarder
CR audit policy acts in addition to the Red Hat OpenShift Service on AWS audit policy. The ClusterLogForwarder
CR audit filter changes what the log collector forwards, and provides the ability to filter by verb, user, group, namespace, or resource. You can create multiple filters to send different summaries of the same audit stream to different places. For example, you can send a detailed stream to the local cluster log store, and a less detailed stream to a remote site.
The example provided is intended to illustrate the range of rules possible in an audit policy and is not a recommended configuration.
Example audit policy
apiVersion: logging.openshift.io/v1 kind: ClusterLogForwarder metadata: name: instance namespace: openshift-logging spec: pipelines: - name: my-pipeline inputRefs: audit 1 filterRefs: my-policy 2 outputRefs: default filters: - name: my-policy type: kubeAPIAudit kubeAPIAudit: # Don't generate audit events for all requests in RequestReceived stage. omitStages: - "RequestReceived" rules: # Log pod changes at RequestResponse level - level: RequestResponse resources: - group: "" resources: ["pods"] # Log "pods/log", "pods/status" at Metadata level - level: Metadata resources: - group: "" resources: ["pods/log", "pods/status"] # Don't log requests to a configmap called "controller-leader" - level: None resources: - group: "" resources: ["configmaps"] resourceNames: ["controller-leader"] # Don't log watch requests by the "system:kube-proxy" on endpoints or services - level: None users: ["system:kube-proxy"] verbs: ["watch"] resources: - group: "" # core API group resources: ["endpoints", "services"] # Don't log authenticated requests to certain non-resource URL paths. - level: None userGroups: ["system:authenticated"] nonResourceURLs: - "/api*" # Wildcard matching. - "/version" # Log the request body of configmap changes in kube-system. - level: Request resources: - group: "" # core API group resources: ["configmaps"] # This rule only applies to resources in the "kube-system" namespace. # The empty string "" can be used to select non-namespaced resources. namespaces: ["kube-system"] # Log configmap and secret changes in all other namespaces at the Metadata level. - level: Metadata resources: - group: "" # core API group resources: ["secrets", "configmaps"] # Log all other resources in core and extensions at the Request level. - level: Request resources: - group: "" # core API group - group: "extensions" # Version of group should NOT be included. # A catch-all rule to log all other requests at the Metadata level. - level: Metadata
Additional resources
9.4.11. Forwarding logs to an external Loki logging system
You can forward logs to an external Loki logging system in addition to, or instead of, the default log store.
To configure log forwarding to Loki, you must create a ClusterLogForwarder
custom resource (CR) with an output to Loki, and a pipeline that uses the output. The output to Loki can use the HTTP (insecure) or HTTPS (secure HTTP) connection.
Prerequisites
-
You must have a Loki logging system running at the URL you specify with the
url
field in the CR.
Procedure
Create or edit a YAML file that defines the
ClusterLogForwarder
CR object:apiVersion: logging.openshift.io/v1 kind: ClusterLogForwarder metadata: name: <log_forwarder_name> 1 namespace: <log_forwarder_namespace> 2 spec: serviceAccountName: <service_account_name> 3 outputs: - name: loki-insecure 4 type: "loki" 5 url: http://loki.insecure.com:3100 6 loki: tenantKey: kubernetes.namespace_name labelKeys: - kubernetes.labels.foo - name: loki-secure 7 type: "loki" url: https://loki.secure.com:3100 secret: name: loki-secret 8 loki: tenantKey: kubernetes.namespace_name 9 labelKeys: - kubernetes.labels.foo 10 pipelines: - name: application-logs 11 inputRefs: 12 - application - audit outputRefs: 13 - loki-secure
- 1
- In legacy implementations, the CR name must be
instance
. In multi log forwarder implementations, you can use any name. - 2
- In legacy implementations, the CR namespace must be
openshift-logging
. In multi log forwarder implementations, you can use any namespace. - 3
- The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the
openshift-logging
namespace. - 4
- Specify a name for the output.
- 5
- Specify the type as
"loki"
. - 6
- Specify the URL and port of the Loki system as a valid absolute URL. You can use the
http
(insecure) orhttps
(secure HTTP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP Address. Loki’s default port for HTTP(S) communication is 3100. - 7
- For a secure connection, you can specify an
https
orhttp
URL that you authenticate by specifying asecret
. - 8
- For an
https
prefix, specify the name of the secret required by the endpoint for TLS communication. The secret must contain aca-bundle.crt
key that points to the certificates it represents. Otherwise, forhttp
andhttps
prefixes, you can specify a secret that contains a username and password. In legacy implementations, the secret must exist in theopenshift-logging
project. For more information, see the following "Example: Setting a secret that contains a username and password." - 9
- Optional: Specify a metadata key field to generate values for the
TenantID
field in Loki. For example, settingtenantKey: kubernetes.namespace_name
uses the names of the Kubernetes namespaces as values for tenant IDs in Loki. To see which other log record fields you can specify, see the "Log Record Fields" link in the following "Additional resources" section. - 10
- Optional: Specify a list of metadata field keys to replace the default Loki labels. Loki label names must match the regular expression
[a-zA-Z_:][a-zA-Z0-9_:]*
. Illegal characters in metadata keys are replaced with_
to form the label name. For example, thekubernetes.labels.foo
metadata key becomes Loki labelkubernetes_labels_foo
. If you do not setlabelKeys
, the default value is:[log_type, kubernetes.namespace_name, kubernetes.pod_name, kubernetes_host]
. Keep the set of labels small because Loki limits the size and number of labels allowed. See Configuring Loki, limits_config. You can still query based on any log record field using query filters. - 11
- Optional: Specify a name for the pipeline.
- 12
- Specify which log types to forward by using the pipeline:
application,
infrastructure
, oraudit
. - 13
- Specify the name of the output to use when forwarding logs with this pipeline.
NoteBecause Loki requires log streams to be correctly ordered by timestamp,
labelKeys
always includes thekubernetes_host
label set, even if you do not specify it. This inclusion ensures that each stream originates from a single host, which prevents timestamps from becoming disordered due to clock differences on different hosts.Apply the
ClusterLogForwarder
CR object by running the following command:$ oc apply -f <filename>.yaml
Additional resources
9.4.12. Forwarding logs to an external Elasticsearch instance
You can forward logs to an external Elasticsearch instance in addition to, or instead of, the internal log store. You are responsible for configuring the external log aggregator to receive log data from Red Hat OpenShift Service on AWS.
To configure log forwarding to an external Elasticsearch instance, you must create a ClusterLogForwarder
custom resource (CR) with an output to that instance, and a pipeline that uses the output. The external Elasticsearch output can use the HTTP (insecure) or HTTPS (secure HTTP) connection.
To forward logs to both an external and the internal Elasticsearch instance, create outputs and pipelines to the external instance and a pipeline that uses the default
output to forward logs to the internal instance.
If you only want to forward logs to an internal Elasticsearch instance, you do not need to create a ClusterLogForwarder
CR.
Prerequisites
- You must have a logging server that is configured to receive the logging data using the specified protocol or format.
Procedure
Create or edit a YAML file that defines the
ClusterLogForwarder
CR:Example
ClusterLogForwarder
CRapiVersion: logging.openshift.io/v1 kind: ClusterLogForwarder metadata: name: <log_forwarder_name> 1 namespace: <log_forwarder_namespace> 2 spec: serviceAccountName: <service_account_name> 3 outputs: - name: elasticsearch-example 4 type: elasticsearch 5 elasticsearch: version: 8 6 url: http://elasticsearch.example.com:9200 7 secret: name: es-secret 8 pipelines: - name: application-logs 9 inputRefs: 10 - application - audit outputRefs: - elasticsearch-example 11 - default 12 labels: myLabel: "myValue" 13 # ...
- 1
- In legacy implementations, the CR name must be
instance
. In multi log forwarder implementations, you can use any name. - 2
- In legacy implementations, the CR namespace must be
openshift-logging
. In multi log forwarder implementations, you can use any namespace. - 3
- The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the
openshift-logging
namespace. - 4
- Specify a name for the output.
- 5
- Specify the
elasticsearch
type. - 6
- Specify the Elasticsearch version. This can be
6
,7
, or8
. - 7
- Specify the URL and port of the external Elasticsearch instance as a valid absolute URL. You can use the
http
(insecure) orhttps
(secure HTTP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP Address. - 8
- For an
https
prefix, specify the name of the secret required by the endpoint for TLS communication. The secret must contain aca-bundle.crt
key that points to the certificate it represents. Otherwise, forhttp
andhttps
prefixes, you can specify a secret that contains a username and password. In legacy implementations, the secret must exist in theopenshift-logging
project. For more information, see the following "Example: Setting a secret that contains a username and password." - 9
- Optional: Specify a name for the pipeline.
- 10
- Specify which log types to forward by using the pipeline:
application,
infrastructure
, oraudit
. - 11
- Specify the name of the output to use when forwarding logs with this pipeline.
- 12
- Optional: Specify the
default
output to send the logs to the internal Elasticsearch instance. - 13
- Optional: String. One or more labels to add to the logs.
Apply the
ClusterLogForwarder
CR:$ oc apply -f <filename>.yaml
Example: Setting a secret that contains a username and password
You can use a secret that contains a username and password to authenticate a secure connection to an external Elasticsearch instance.
For example, if you cannot use mutual TLS (mTLS) keys because a third party operates the Elasticsearch instance, you can use HTTP or HTTPS and set a secret that contains the username and password.
Create a
Secret
YAML file similar to the following example. Use base64-encoded values for theusername
andpassword
fields. The secret type is opaque by default.apiVersion: v1 kind: Secret metadata: name: openshift-test-secret data: username: <username> password: <password> # ...
Create the secret:
$ oc create secret -n openshift-logging openshift-test-secret.yaml
Specify the name of the secret in the
ClusterLogForwarder
CR:kind: ClusterLogForwarder metadata: name: instance namespace: openshift-logging spec: outputs: - name: elasticsearch type: "elasticsearch" url: https://elasticsearch.secure.com:9200 secret: name: openshift-test-secret # ...
NoteIn the value of the
url
field, the prefix can behttp
orhttps
.Apply the CR object:
$ oc apply -f <filename>.yaml
9.4.13. Forwarding logs using the Fluentd forward protocol
You can use the Fluentd forward protocol to send a copy of your logs to an external log aggregator that is configured to accept the protocol instead of, or in addition to, the default Elasticsearch log store. You are responsible for configuring the external log aggregator to receive the logs from Red Hat OpenShift Service on AWS.
To configure log forwarding using the forward protocol, you must create a ClusterLogForwarder
custom resource (CR) with one or more outputs to the Fluentd servers, and pipelines that use those outputs. The Fluentd output can use a TCP (insecure) or TLS (secure TCP) connection.
Prerequisites
- You must have a logging server that is configured to receive the logging data using the specified protocol or format.
Procedure
Create or edit a YAML file that defines the
ClusterLogForwarder
CR object:apiVersion: logging.openshift.io/v1 kind: ClusterLogForwarder metadata: name: instance 1 namespace: openshift-logging 2 spec: outputs: - name: fluentd-server-secure 3 type: fluentdForward 4 url: 'tls://fluentdserver.security.example.com:24224' 5 secret: 6 name: fluentd-secret - name: fluentd-server-insecure type: fluentdForward url: 'tcp://fluentdserver.home.example.com:24224' pipelines: - name: forward-to-fluentd-secure 7 inputRefs: 8 - application - audit outputRefs: - fluentd-server-secure 9 - default 10 labels: clusterId: "C1234" 11 - name: forward-to-fluentd-insecure 12 inputRefs: - infrastructure outputRefs: - fluentd-server-insecure labels: clusterId: "C1234"
- 1
- The name of the
ClusterLogForwarder
CR must beinstance
. - 2
- The namespace for the
ClusterLogForwarder
CR must beopenshift-logging
. - 3
- Specify a name for the output.
- 4
- Specify the
fluentdForward
type. - 5
- Specify the URL and port of the external Fluentd instance as a valid absolute URL. You can use the
tcp
(insecure) ortls
(secure TCP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP address. - 6
- If you are using a
tls
prefix, you must specify the name of the secret required by the endpoint for TLS communication. The secret must exist in theopenshift-logging
project and must contain aca-bundle.crt
key that points to the certificate it represents. - 7
- Optional: Specify a name for the pipeline.
- 8
- Specify which log types to forward by using the pipeline:
application,
infrastructure
, oraudit
. - 9
- Specify the name of the output to use when forwarding logs with this pipeline.
- 10
- Optional: Specify the
default
output to forward logs to the internal Elasticsearch instance. - 11
- Optional: String. One or more labels to add to the logs.
- 12
- Optional: Configure multiple outputs to forward logs to other external log aggregators of any supported type:
- A name to describe the pipeline.
-
The
inputRefs
is the log type to forward by using the pipeline:application,
infrastructure
, oraudit
. -
The
outputRefs
is the name of the output to use. - Optional: String. One or more labels to add to the logs.
Create the CR object:
$ oc create -f <file-name>.yaml
9.4.13.1. Enabling nanosecond precision for Logstash to ingest data from fluentd
For Logstash to ingest log data from fluentd, you must enable nanosecond precision in the Logstash configuration file.
Procedure
-
In the Logstash configuration file, set
nanosecond_precision
totrue
.
Example Logstash configuration file
input { tcp { codec => fluent { nanosecond_precision => true } port => 24114 } } filter { } output { stdout { codec => rubydebug } }
9.4.14. Forwarding logs using the syslog protocol
You can use the syslog RFC3164 or RFC5424 protocol to send a copy of your logs to an external log aggregator that is configured to accept the protocol instead of, or in addition to, the default Elasticsearch log store. You are responsible for configuring the external log aggregator, such as a syslog server, to receive the logs from Red Hat OpenShift Service on AWS.
To configure log forwarding using the syslog protocol, you must create a ClusterLogForwarder
custom resource (CR) with one or more outputs to the syslog servers, and pipelines that use those outputs. The syslog output can use a UDP, TCP, or TLS connection.
Prerequisites
- You must have a logging server that is configured to receive the logging data using the specified protocol or format.
Procedure
Create or edit a YAML file that defines the
ClusterLogForwarder
CR object:apiVersion: logging.openshift.io/v1 kind: ClusterLogForwarder metadata: name: <log_forwarder_name> 1 namespace: <log_forwarder_namespace> 2 spec: serviceAccountName: <service_account_name> 3 outputs: - name: rsyslog-east 4 type: syslog 5 syslog: 6 facility: local0 rfc: RFC3164 payloadKey: message severity: informational url: 'tls://rsyslogserver.east.example.com:514' 7 secret: 8 name: syslog-secret - name: rsyslog-west type: syslog syslog: appName: myapp facility: user msgID: mymsg procID: myproc rfc: RFC5424 severity: debug url: 'tcp://rsyslogserver.west.example.com:514' pipelines: - name: syslog-east 9 inputRefs: 10 - audit - application outputRefs: 11 - rsyslog-east - default 12 labels: secure: "true" 13 syslog: "east" - name: syslog-west 14 inputRefs: - infrastructure outputRefs: - rsyslog-west - default labels: syslog: "west"
- 1
- In legacy implementations, the CR name must be
instance
. In multi log forwarder implementations, you can use any name. - 2
- In legacy implementations, the CR namespace must be
openshift-logging
. In multi log forwarder implementations, you can use any namespace. - 3
- The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the
openshift-logging
namespace. - 4
- Specify a name for the output.
- 5
- Specify the
syslog
type. - 6
- Optional: Specify the syslog parameters, listed below.
- 7
- Specify the URL and port of the external syslog instance. You can use the
udp
(insecure),tcp
(insecure) ortls
(secure TCP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP address. - 8
- If using a
tls
prefix, you must specify the name of the secret required by the endpoint for TLS communication. The secret must contain aca-bundle.crt
key that points to the certificate it represents. In legacy implementations, the secret must exist in theopenshift-logging
project. - 9
- Optional: Specify a name for the pipeline.
- 10
- Specify which log types to forward by using the pipeline:
application,
infrastructure
, oraudit
. - 11
- Specify the name of the output to use when forwarding logs with this pipeline.
- 12
- Optional: Specify the
default
output to forward logs to the internal Elasticsearch instance. - 13
- Optional: String. One or more labels to add to the logs. Quote values like "true" so they are recognized as string values, not as a boolean.
- 14
- Optional: Configure multiple outputs to forward logs to other external log aggregators of any supported type:
- A name to describe the pipeline.
-
The
inputRefs
is the log type to forward by using the pipeline:application,
infrastructure
, oraudit
. -
The
outputRefs
is the name of the output to use. - Optional: String. One or more labels to add to the logs.
Create the CR object:
$ oc create -f <filename>.yaml
9.4.14.1. Adding log source information to message output
You can add namespace_name
, pod_name
, and container_name
elements to the message
field of the record by adding the AddLogSource
field to your ClusterLogForwarder
custom resource (CR).
spec: outputs: - name: syslogout syslog: addLogSource: true facility: user payloadKey: message rfc: RFC3164 severity: debug tag: mytag type: syslog url: tls://syslog-receiver.openshift-logging.svc:24224 pipelines: - inputRefs: - application name: test-app outputRefs: - syslogout
This configuration is compatible with both RFC3164 and RFC5424.
Example syslog message output without AddLogSource
<15>1 2020-11-15T17:06:14+00:00 fluentd-9hkb4 mytag - - - {"msgcontent"=>"Message Contents", "timestamp"=>"2020-11-15 17:06:09", "tag_key"=>"rec_tag", "index"=>56}
Example syslog message output with AddLogSource
<15>1 2020-11-16T10:49:37+00:00 crc-j55b9-master-0 mytag - - - namespace_name=clo-test-6327,pod_name=log-generator-ff9746c49-qxm7l,container_name=log-generator,message={"msgcontent":"My life is my message", "timestamp":"2020-11-16 10:49:36", "tag_key":"rec_tag", "index":76}
9.4.14.2. Syslog parameters
You can configure the following for the syslog
outputs. For more information, see the syslog RFC3164 or RFC5424 RFC.
facility: The syslog facility. The value can be a decimal integer or a case-insensitive keyword:
-
0
orkern
for kernel messages -
1
oruser
for user-level messages, the default. -
2
ormail
for the mail system -
3
ordaemon
for system daemons -
4
orauth
for security/authentication messages -
5
orsyslog
for messages generated internally by syslogd -
6
orlpr
for the line printer subsystem -
7
ornews
for the network news subsystem -
8
oruucp
for the UUCP subsystem -
9
orcron
for the clock daemon -
10
orauthpriv
for security authentication messages -
11
orftp
for the FTP daemon -
12
orntp
for the NTP subsystem -
13
orsecurity
for the syslog audit log -
14
orconsole
for the syslog alert log -
15
orsolaris-cron
for the scheduling daemon -
16
–23
orlocal0
–local7
for locally used facilities
-
Optional:
payloadKey
: The record field to use as payload for the syslog message.NoteConfiguring the
payloadKey
parameter prevents other parameters from being forwarded to the syslog.- rfc: The RFC to be used for sending logs using syslog. The default is RFC5424.
severity: The syslog severity to set on outgoing syslog records. The value can be a decimal integer or a case-insensitive keyword:
-
0
orEmergency
for messages indicating the system is unusable -
1
orAlert
for messages indicating action must be taken immediately -
2
orCritical
for messages indicating critical conditions -
3
orError
for messages indicating error conditions -
4
orWarning
for messages indicating warning conditions -
5
orNotice
for messages indicating normal but significant conditions -
6
orInformational
for messages indicating informational messages -
7
orDebug
for messages indicating debug-level messages, the default
-
- tag: Tag specifies a record field to use as a tag on the syslog message.
- trimPrefix: Remove the specified prefix from the tag.
9.4.14.3. Additional RFC5424 syslog parameters
The following parameters apply to RFC5424:
-
appName: The APP-NAME is a free-text string that identifies the application that sent the log. Must be specified for
RFC5424
. -
msgID: The MSGID is a free-text string that identifies the type of message. Must be specified for
RFC5424
. -
procID: The PROCID is a free-text string. A change in the value indicates a discontinuity in syslog reporting. Must be specified for
RFC5424
.
9.4.15. Forwarding logs to a Kafka broker
You can forward logs to an external Kafka broker in addition to, or instead of, the default log store.
To configure log forwarding to an external Kafka instance, you must create a ClusterLogForwarder
custom resource (CR) with an output to that instance, and a pipeline that uses the output. You can include a specific Kafka topic in the output or use the default. The Kafka output can use a TCP (insecure) or TLS (secure TCP) connection.
Procedure
Create or edit a YAML file that defines the
ClusterLogForwarder
CR object:apiVersion: logging.openshift.io/v1 kind: ClusterLogForwarder metadata: name: <log_forwarder_name> 1 namespace: <log_forwarder_namespace> 2 spec: serviceAccountName: <service_account_name> 3 outputs: - name: app-logs 4 type: kafka 5 url: tls://kafka.example.devlab.com:9093/app-topic 6 secret: name: kafka-secret 7 - name: infra-logs type: kafka url: tcp://kafka.devlab2.example.com:9093/infra-topic 8 - name: audit-logs type: kafka url: tls://kafka.qelab.example.com:9093/audit-topic secret: name: kafka-secret-qe pipelines: - name: app-topic 9 inputRefs: 10 - application outputRefs: 11 - app-logs labels: logType: "application" 12 - name: infra-topic 13 inputRefs: - infrastructure outputRefs: - infra-logs labels: logType: "infra" - name: audit-topic inputRefs: - audit outputRefs: - audit-logs labels: logType: "audit"
- 1
- In legacy implementations, the CR name must be
instance
. In multi log forwarder implementations, you can use any name. - 2
- In legacy implementations, the CR namespace must be
openshift-logging
. In multi log forwarder implementations, you can use any namespace. - 3
- The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the
openshift-logging
namespace. - 4
- Specify a name for the output.
- 5
- Specify the
kafka
type. - 6
- Specify the URL and port of the Kafka broker as a valid absolute URL, optionally with a specific topic. You can use the
tcp
(insecure) ortls
(secure TCP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP address. - 7
- If you are using a
tls
prefix, you must specify the name of the secret required by the endpoint for TLS communication. The secret must contain aca-bundle.crt
key that points to the certificate it represents. In legacy implementations, the secret must exist in theopenshift-logging
project. - 8
- Optional: To send an insecure output, use a
tcp
prefix in front of the URL. Also omit thesecret
key and itsname
from this output. - 9
- Optional: Specify a name for the pipeline.
- 10
- Specify which log types to forward by using the pipeline:
application,
infrastructure
, oraudit
. - 11
- Specify the name of the output to use when forwarding logs with this pipeline.
- 12
- Optional: String. One or more labels to add to the logs.
- 13
- Optional: Configure multiple outputs to forward logs to other external log aggregators of any supported type:
- A name to describe the pipeline.
-
The
inputRefs
is the log type to forward by using the pipeline:application,
infrastructure
, oraudit
. -
The
outputRefs
is the name of the output to use. - Optional: String. One or more labels to add to the logs.
Optional: To forward a single output to multiple Kafka brokers, specify an array of Kafka brokers as shown in the following example:
# ... spec: outputs: - name: app-logs type: kafka secret: name: kafka-secret-dev kafka: 1 brokers: 2 - tls://kafka-broker1.example.com:9093/ - tls://kafka-broker2.example.com:9093/ topic: app-topic 3 # ...
Apply the
ClusterLogForwarder
CR by running the following command:$ oc apply -f <filename>.yaml
9.4.16. Forwarding logs to Amazon CloudWatch
You can forward logs to Amazon CloudWatch, a monitoring and log storage service hosted by Amazon Web Services (AWS). You can forward logs to CloudWatch in addition to, or instead of, the default log store.
To configure log forwarding to CloudWatch, you must create a ClusterLogForwarder
custom resource (CR) with an output for CloudWatch, and a pipeline that uses the output.
Procedure
Create a
Secret
YAML file that uses theaws_access_key_id
andaws_secret_access_key
fields to specify your base64-encoded AWS credentials. For example:apiVersion: v1 kind: Secret metadata: name: cw-secret namespace: openshift-logging data: aws_access_key_id: QUtJQUlPU0ZPRE5ON0VYQU1QTEUK aws_secret_access_key: d0phbHJYVXRuRkVNSS9LN01ERU5HL2JQeFJmaUNZRVhBTVBMRUtFWQo=
Create the secret. For example:
$ oc apply -f cw-secret.yaml
Create or edit a YAML file that defines the
ClusterLogForwarder
CR object. In the file, specify the name of the secret. For example:apiVersion: logging.openshift.io/v1 kind: ClusterLogForwarder metadata: name: <log_forwarder_name> 1 namespace: <log_forwarder_namespace> 2 spec: serviceAccountName: <service_account_name> 3 outputs: - name: cw 4 type: cloudwatch 5 cloudwatch: groupBy: logType 6 groupPrefix: <group prefix> 7 region: us-east-2 8 secret: name: cw-secret 9 pipelines: - name: infra-logs 10 inputRefs: 11 - infrastructure - audit - application outputRefs: - cw 12
- 1
- In legacy implementations, the CR name must be
instance
. In multi log forwarder implementations, you can use any name. - 2
- In legacy implementations, the CR namespace must be
openshift-logging
. In multi log forwarder implementations, you can use any namespace. - 3
- The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the
openshift-logging
namespace. - 4
- Specify a name for the output.
- 5
- Specify the
cloudwatch
type. - 6
- Optional: Specify how to group the logs:
-
logType
creates log groups for each log type. -
namespaceName
creates a log group for each application name space. It also creates separate log groups for infrastructure and audit logs. -
namespaceUUID
creates a new log groups for each application namespace UUID. It also creates separate log groups for infrastructure and audit logs.
-
- 7
- Optional: Specify a string to replace the default
infrastructureName
prefix in the names of the log groups. - 8
- Specify the AWS region.
- 9
- Specify the name of the secret that contains your AWS credentials.
- 10
- Optional: Specify a name for the pipeline.
- 11
- Specify which log types to forward by using the pipeline:
application,
infrastructure
, oraudit
. - 12
- Specify the name of the output to use when forwarding logs with this pipeline.
Create the CR object:
$ oc create -f <file-name>.yaml
Example: Using ClusterLogForwarder with Amazon CloudWatch
Here, you see an example ClusterLogForwarder
custom resource (CR) and the log data that it outputs to Amazon CloudWatch.
Suppose that you are running a ROSA cluster named mycluster
. The following command returns the cluster’s infrastructureName
, which you will use to compose aws
commands later on:
$ oc get Infrastructure/cluster -ojson | jq .status.infrastructureName "mycluster-7977k"
To generate log data for this example, you run a busybox
pod in a namespace called app
. The busybox
pod writes a message to stdout every three seconds:
$ oc run busybox --image=busybox -- sh -c 'while true; do echo "My life is my message"; sleep 3; done' $ oc logs -f busybox My life is my message My life is my message My life is my message ...
You can look up the UUID of the app
namespace where the busybox
pod runs:
$ oc get ns/app -ojson | jq .metadata.uid "794e1e1a-b9f5-4958-a190-e76a9b53d7bf"
In your ClusterLogForwarder
custom resource (CR), you configure the infrastructure
, audit
, and application
log types as inputs to the all-logs
pipeline. You also connect this pipeline to cw
output, which forwards the logs to a CloudWatch instance in the us-east-2
region:
apiVersion: "logging.openshift.io/v1" kind: ClusterLogForwarder metadata: name: instance namespace: openshift-logging spec: outputs: - name: cw type: cloudwatch cloudwatch: groupBy: logType region: us-east-2 secret: name: cw-secret pipelines: - name: all-logs inputRefs: - infrastructure - audit - application outputRefs: - cw
Each region in CloudWatch contains three levels of objects:
log group
log stream
- log event
With groupBy: logType
in the ClusterLogForwarding
CR, the three log types in the inputRefs
produce three log groups in Amazon Cloudwatch:
$ aws --output json logs describe-log-groups | jq .logGroups[].logGroupName "mycluster-7977k.application" "mycluster-7977k.audit" "mycluster-7977k.infrastructure"
Each of the log groups contains log streams:
$ aws --output json logs describe-log-streams --log-group-name mycluster-7977k.application | jq .logStreams[].logStreamName "kubernetes.var.log.containers.busybox_app_busybox-da085893053e20beddd6747acdbaf98e77c37718f85a7f6a4facf09ca195ad76.log"
$ aws --output json logs describe-log-streams --log-group-name mycluster-7977k.audit | jq .logStreams[].logStreamName "ip-10-0-131-228.us-east-2.compute.internal.k8s-audit.log" "ip-10-0-131-228.us-east-2.compute.internal.linux-audit.log" "ip-10-0-131-228.us-east-2.compute.internal.openshift-audit.log" ...
$ aws --output json logs describe-log-streams --log-group-name mycluster-7977k.infrastructure | jq .logStreams[].logStreamName "ip-10-0-131-228.us-east-2.compute.internal.kubernetes.var.log.containers.apiserver-69f9fd9b58-zqzw5_openshift-oauth-apiserver_oauth-apiserver-453c5c4ee026fe20a6139ba6b1cdd1bed25989c905bf5ac5ca211b7cbb5c3d7b.log" "ip-10-0-131-228.us-east-2.compute.internal.kubernetes.var.log.containers.apiserver-797774f7c5-lftrx_openshift-apiserver_openshift-apiserver-ce51532df7d4e4d5f21c4f4be05f6575b93196336be0027067fd7d93d70f66a4.log" "ip-10-0-131-228.us-east-2.compute.internal.kubernetes.var.log.containers.apiserver-797774f7c5-lftrx_openshift-apiserver_openshift-apiserver-check-endpoints-82a9096b5931b5c3b1d6dc4b66113252da4a6472c9fff48623baee761911a9ef.log" ...
Each log stream contains log events. To see a log event from the busybox
Pod, you specify its log stream from the application
log group:
$ aws logs get-log-events --log-group-name mycluster-7977k.application --log-stream-name kubernetes.var.log.containers.busybox_app_busybox-da085893053e20beddd6747acdbaf98e77c37718f85a7f6a4facf09ca195ad76.log { "events": [ { "timestamp": 1629422704178, "message": "{\"docker\":{\"container_id\":\"da085893053e20beddd6747acdbaf98e77c37718f85a7f6a4facf09ca195ad76\"},\"kubernetes\":{\"container_name\":\"busybox\",\"namespace_name\":\"app\",\"pod_name\":\"busybox\",\"container_image\":\"docker.io/library/busybox:latest\",\"container_image_id\":\"docker.io/library/busybox@sha256:0f354ec1728d9ff32edcd7d1b8bbdfc798277ad36120dc3dc683be44524c8b60\",\"pod_id\":\"870be234-90a3-4258-b73f-4f4d6e2777c7\",\"host\":\"ip-10-0-216-3.us-east-2.compute.internal\",\"labels\":{\"run\":\"busybox\"},\"master_url\":\"https://kubernetes.default.svc\",\"namespace_id\":\"794e1e1a-b9f5-4958-a190-e76a9b53d7bf\",\"namespace_labels\":{\"kubernetes_io/metadata_name\":\"app\"}},\"message\":\"My life is my message\",\"level\":\"unknown\",\"hostname\":\"ip-10-0-216-3.us-east-2.compute.internal\",\"pipeline_metadata\":{\"collector\":{\"ipaddr4\":\"10.0.216.3\",\"inputname\":\"fluent-plugin-systemd\",\"name\":\"fluentd\",\"received_at\":\"2021-08-20T01:25:08.085760+00:00\",\"version\":\"1.7.4 1.6.0\"}},\"@timestamp\":\"2021-08-20T01:25:04.178986+00:00\",\"viaq_index_name\":\"app-write\",\"viaq_msg_id\":\"NWRjZmUyMWQtZjgzNC00MjI4LTk3MjMtNTk3NmY3ZjU4NDk1\",\"log_type\":\"application\",\"time\":\"2021-08-20T01:25:04+00:00\"}", "ingestionTime": 1629422744016 }, ...
Example: Customizing the prefix in log group names
In the log group names, you can replace the default infrastructureName
prefix, mycluster-7977k
, with an arbitrary string like demo-group-prefix
. To make this change, you update the groupPrefix
field in the ClusterLogForwarding
CR:
cloudwatch: groupBy: logType groupPrefix: demo-group-prefix region: us-east-2
The value of groupPrefix
replaces the default infrastructureName
prefix:
$ aws --output json logs describe-log-groups | jq .logGroups[].logGroupName "demo-group-prefix.application" "demo-group-prefix.audit" "demo-group-prefix.infrastructure"
Example: Naming log groups after application namespace names
For each application namespace in your cluster, you can create a log group in CloudWatch whose name is based on the name of the application namespace.
If you delete an application namespace object and create a new one that has the same name, CloudWatch continues using the same log group as before.
If you consider successive application namespace objects that have the same name as equivalent to each other, use the approach described in this example. Otherwise, if you need to distinguish the resulting log groups from each other, see the following "Naming log groups for application namespace UUIDs" section instead.
To create application log groups whose names are based on the names of the application namespaces, you set the value of the groupBy
field to namespaceName
in the ClusterLogForwarder
CR:
cloudwatch: groupBy: namespaceName region: us-east-2
Setting groupBy
to namespaceName
affects the application log group only. It does not affect the audit
and infrastructure
log groups.
In Amazon Cloudwatch, the namespace name appears at the end of each log group name. Because there is a single application namespace, "app", the following output shows a new mycluster-7977k.app
log group instead of mycluster-7977k.application
:
$ aws --output json logs describe-log-groups | jq .logGroups[].logGroupName "mycluster-7977k.app" "mycluster-7977k.audit" "mycluster-7977k.infrastructure"
If the cluster in this example had contained multiple application namespaces, the output would show multiple log groups, one for each namespace.
The groupBy
field affects the application log group only. It does not affect the audit
and infrastructure
log groups.
Example: Naming log groups after application namespace UUIDs
For each application namespace in your cluster, you can create a log group in CloudWatch whose name is based on the UUID of the application namespace.
If you delete an application namespace object and create a new one, CloudWatch creates a new log group.
If you consider successive application namespace objects with the same name as different from each other, use the approach described in this example. Otherwise, see the preceding "Example: Naming log groups for application namespace names" section instead.
To name log groups after application namespace UUIDs, you set the value of the groupBy
field to namespaceUUID
in the ClusterLogForwarder
CR:
cloudwatch: groupBy: namespaceUUID region: us-east-2
In Amazon Cloudwatch, the namespace UUID appears at the end of each log group name. Because there is a single application namespace, "app", the following output shows a new mycluster-7977k.794e1e1a-b9f5-4958-a190-e76a9b53d7bf
log group instead of mycluster-7977k.application
:
$ aws --output json logs describe-log-groups | jq .logGroups[].logGroupName "mycluster-7977k.794e1e1a-b9f5-4958-a190-e76a9b53d7bf" // uid of the "app" namespace "mycluster-7977k.audit" "mycluster-7977k.infrastructure"
The groupBy
field affects the application log group only. It does not affect the audit
and infrastructure
log groups.
9.4.17. Creating a secret for AWS CloudWatch with an existing AWS role
If you have an existing role for AWS, you can create a secret for AWS with STS using the oc create secret --from-literal
command.
Procedure
In the CLI, enter the following to generate a secret for AWS:
$ oc create secret generic cw-sts-secret -n openshift-logging --from-literal=role_arn=arn:aws:iam::123456789012:role/my-role_with-permissions
Example Secret
apiVersion: v1 kind: Secret metadata: namespace: openshift-logging name: my-secret-name stringData: role_arn: arn:aws:iam::123456789012:role/my-role_with-permissions
9.4.18. Forwarding logs to Amazon CloudWatch from STS enabled clusters
For clusters with AWS Security Token Service (STS) enabled, you must create the AWS IAM roles and policies that enable log forwarding, and a ClusterLogForwarder
custom resource (CR) with an output for CloudWatch.
Prerequisites
- Logging for Red Hat OpenShift: 5.5 and later
Procedure
Prepare the AWS account:
Create an IAM policy JSON file with the following content:
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "logs:CreateLogGroup", "logs:CreateLogStream", "logs:DescribeLogGroups", "logs:DescribeLogStreams", "logs:PutLogEvents", "logs:PutRetentionPolicy" ], "Resource": "arn:aws:logs:*:*:*" } ] }
Create an IAM trust JSON file with the following content:
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::<your_aws_account_id>:oidc-provider/<openshift_oidc_provider>" 1 }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "<openshift_oidc_provider>:sub": "system:serviceaccount:openshift-logging:logcollector" 2 } } } ] }
- 1
- Specify your AWS account ID and the OpenShift OIDC provider endpoint. Obtain the endpoint by running the following command:
$ rosa describe cluster \ -c $(oc get clusterversion -o jsonpath='{.items[].spec.clusterID}{"\n"}') \ -o yaml | awk '/oidc_endpoint_url/ {print $2}' | cut -d '/' -f 3,4
- 2
- Specify the OpenShift OIDC endpoint again.
Create the IAM role:
$ aws iam create-role --role-name “<your_rosa_cluster_name>-RosaCloudWatch” \ --assume-role-policy-document file://<your_trust_file_name>.json \ --query Role.Arn \ --output text
Save the output. You will use it in the next steps.
Create the IAM policy:
$ aws iam create-policy \ --policy-name "RosaCloudWatch" \ --policy-document file:///<your_policy_file_name>.json \ --query Policy.Arn \ --output text
Save the output. You will use it in the next steps.
Attach the IAM policy to the IAM role:
$ aws iam attach-role-policy \ --role-name “<your_rosa_cluster_name>-RosaCloudWatch” \ --policy-arn <policy_ARN> 1
- 1
- Replace
policy_ARN
with the output you saved while creating the policy.
Create a
Secret
YAML file for the Red Hat OpenShift Logging Operator:apiVersion: v1 kind: Secret metadata: name: cloudwatch-credentials namespace: openshift-logging stringData: credentials: |- [default] sts_regional_endpoints = regional role_arn: <role_ARN> 1 web_identity_token_file = /var/run/secrets/openshift/serviceaccount/token
- 1
- Replace
role_ARN
with the output you saved while creating the role.
Create the secret:
$ oc apply -f cloudwatch-credentials.yaml
Create or edit a
ClusterLogForwarder
custom resource:apiVersion: logging.openshift.io/v1 kind: ClusterLogForwarder metadata: name: <log_forwarder_name> 1 namespace: <log_forwarder_namespace> 2 spec: serviceAccountName: <service_account_name> 3 outputs: - name: cw 4 type: cloudwatch 5 cloudwatch: groupBy: logType 6 groupPrefix: <group prefix> 7 region: us-east-2 8 secret: name: <your_secret_name> 9 pipelines: - name: to-cloudwatch 10 inputRefs: 11 - infrastructure - audit - application outputRefs: - cw 12
- 1
- In legacy implementations, the CR name must be
instance
. In multi log forwarder implementations, you can use any name. - 2
- In legacy implementations, the CR namespace must be
openshift-logging
. In multi log forwarder implementations, you can use any namespace. - 3
- The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the
openshift-logging
namespace. - 4
- Specify a name for the output.
- 5
- Specify the
cloudwatch
type. - 6
- Optional: Specify how to group the logs:
-
logType
creates log groups for each log type -
namespaceName
creates a log group for each application name space. Infrastructure and audit logs are unaffected, remaining grouped bylogType
. -
namespaceUUID
creates a new log groups for each application namespace UUID. It also creates separate log groups for infrastructure and audit logs.
-
- 7
- Optional: Specify a string to replace the default
infrastructureName
prefix in the names of the log groups. - 8
- Specify the AWS region.
- 9
- Specify the name of the secret you created previously.
- 10
- Optional: Specify a name for the pipeline.
- 11
- Specify which log types to forward by using the pipeline:
application,
infrastructure
, oraudit
. - 12
- Specify the name of the output to use when forwarding logs with this pipeline.
Additional resources
9.5. Configuring the logging collector
Logging for Red Hat OpenShift collects operations and application logs from your cluster and enriches the data with Kubernetes pod and project metadata. All supported modifications to the log collector can be performed though the spec.collection
stanza in the ClusterLogging
custom resource (CR).
9.5.1. Configuring the log collector
You can configure which log collector type your logging uses by modifying the ClusterLogging
custom resource (CR).
Fluentd is deprecated and is planned to be removed in a future release. Red Hat provides bug fixes and support for this feature during the current release lifecycle, but this feature no longer receives enhancements. As an alternative to Fluentd, you can use Vector instead.
Prerequisites
- You have administrator permissions.
-
You have installed the OpenShift CLI (
oc
). - You have installed the Red Hat OpenShift Logging Operator.
-
You have created a
ClusterLogging
CR.
Procedure
Modify the
ClusterLogging
CRcollection
spec:ClusterLogging
CR exampleapiVersion: logging.openshift.io/v1 kind: ClusterLogging metadata: # ... spec: # ... collection: type: <log_collector_type> 1 resources: {} tolerations: {} # ...
- 1
- The log collector type you want to use for the logging. This can be
vector
orfluentd
.
Apply the
ClusterLogging
CR by running the following command:$ oc apply -f <filename>.yaml
9.5.2. Creating a LogFileMetricExporter resource
In logging version 5.8 and newer versions, the LogFileMetricExporter is no longer deployed with the collector by default. You must manually create a LogFileMetricExporter
custom resource (CR) to generate metrics from the logs produced by running containers.
If you do not create the LogFileMetricExporter
CR, you may see a No datapoints found message in the Red Hat OpenShift Service on AWS web console dashboard for Produced Logs.
Prerequisites
- You have administrator permissions.
- You have installed the Red Hat OpenShift Logging Operator.
-
You have installed the OpenShift CLI (
oc
).
Procedure
Create a
LogFileMetricExporter
CR as a YAML file:Example
LogFileMetricExporter
CRapiVersion: logging.openshift.io/v1alpha1 kind: LogFileMetricExporter metadata: name: instance namespace: openshift-logging spec: nodeSelector: {} 1 resources: 2 limits: cpu: 500m memory: 256Mi requests: cpu: 200m memory: 128Mi tolerations: [] 3 # ...
Apply the
LogFileMetricExporter
CR by running the following command:$ oc apply -f <filename>.yaml
Verification
A logfilesmetricexporter
pod runs concurrently with a collector
pod on each node.
Verify that the
logfilesmetricexporter
pods are running in the namespace where you have created theLogFileMetricExporter
CR, by running the following command and observing the output:$ oc get pods -l app.kubernetes.io/component=logfilesmetricexporter -n openshift-logging
Example output
NAME READY STATUS RESTARTS AGE logfilesmetricexporter-9qbjj 1/1 Running 0 2m46s logfilesmetricexporter-cbc4v 1/1 Running 0 2m46s
9.5.3. Configure log collector CPU and memory limits
The log collector allows for adjustments to both the CPU and memory limits.
Procedure
Edit the
ClusterLogging
custom resource (CR) in theopenshift-logging
project:$ oc -n openshift-logging edit ClusterLogging instance
apiVersion: logging.openshift.io/v1 kind: ClusterLogging metadata: name: instance namespace: openshift-logging spec: collection: type: fluentd resources: limits: 1 memory: 736Mi requests: cpu: 100m memory: 736Mi # ...
- 1
- Specify the CPU and memory limits and requests as needed. The values shown are the default values.
9.5.4. Configuring input receivers
The Red Hat OpenShift Logging Operator deploys a service for each configured input receiver so that clients can write to the collector. This service exposes the port specified for the input receiver. The service name is generated based on the following:
-
For multi log forwarder
ClusterLogForwarder
CR deployments, the service name is in the format<ClusterLogForwarder_CR_name>-<input_name>
. For example,example-http-receiver
. -
For legacy
ClusterLogForwarder
CR deployments, meaning those namedinstance
and located in theopenshift-logging
namespace, the service name is in the formatcollector-<input_name>
. For example,collector-http-receiver
.
9.5.4.1. Configuring the collector to receive audit logs as an HTTP server
You can configure your log collector to listen for HTTP connections and receive audit logs as an HTTP server by specifying http
as a receiver input in the ClusterLogForwarder
custom resource (CR). This enables you to use a common log store for audit logs that are collected from both inside and outside of your Red Hat OpenShift Service on AWS cluster.
Prerequisites
- You have administrator permissions.
-
You have installed the OpenShift CLI (
oc
). - You have installed the Red Hat OpenShift Logging Operator.
-
You have created a
ClusterLogForwarder
CR.
Procedure
Modify the
ClusterLogForwarder
CR to add configuration for thehttp
receiver input:Example
ClusterLogForwarder
CR if you are using a multi log forwarder deploymentapiVersion: logging.openshift.io/v1beta1 kind: ClusterLogForwarder metadata: # ... spec: serviceAccountName: <service_account_name> inputs: - name: http-receiver 1 receiver: type: http 2 http: format: kubeAPIAudit 3 port: 8443 4 pipelines: 5 - name: http-pipeline inputRefs: - http-receiver # ...
- 1
- Specify a name for your input receiver.
- 2
- Specify the input receiver type as
http
. - 3
- Currently, only the
kube-apiserver
webhook format is supported forhttp
input receivers. - 4
- Optional: Specify the port that the input receiver listens on. This must be a value between
1024
and65535
. The default value is8443
if this is not specified. - 5
- Configure a pipeline for your input receiver.
Example
ClusterLogForwarder
CR if you are using a legacy deploymentapiVersion: logging.openshift.io/v1 kind: ClusterLogForwarder metadata: name: instance namespace: openshift-logging spec: inputs: - name: http-receiver 1 receiver: type: http 2 http: format: kubeAPIAudit 3 port: 8443 4 pipelines: 5 - inputRefs: - http-receiver name: http-pipeline # ...
- 1
- Specify a name for your input receiver.
- 2
- Specify the input receiver type as
http
. - 3
- Currently, only the
kube-apiserver
webhook format is supported forhttp
input receivers. - 4
- Optional: Specify the port that the input receiver listens on. This must be a value between
1024
and65535
. The default value is8443
if this is not specified. - 5
- Configure a pipeline for your input receiver.
Apply the changes to the
ClusterLogForwarder
CR by running the following command:$ oc apply -f <filename>.yaml
Additional resources
9.5.5. Advanced configuration for the Fluentd log forwarder
Fluentd is deprecated and is planned to be removed in a future release. Red Hat provides bug fixes and support for this feature during the current release lifecycle, but this feature no longer receives enhancements. As an alternative to Fluentd, you can use Vector instead.
Logging includes multiple Fluentd parameters that you can use for tuning the performance of the Fluentd log forwarder. With these parameters, you can change the following Fluentd behaviors:
- Chunk and chunk buffer sizes
- Chunk flushing behavior
- Chunk forwarding retry behavior
Fluentd collects log data in a single blob called a chunk. When Fluentd creates a chunk, the chunk is considered to be in the stage, where the chunk gets filled with data. When the chunk is full, Fluentd moves the chunk to the queue, where chunks are held before being flushed, or written out to their destination. Fluentd can fail to flush a chunk for a number of reasons, such as network issues or capacity issues at the destination. If a chunk cannot be flushed, Fluentd retries flushing as configured.
By default in Red Hat OpenShift Service on AWS, Fluentd uses the exponential backoff method to retry flushing, where Fluentd doubles the time it waits between attempts to retry flushing again, which helps reduce connection requests to the destination. You can disable exponential backoff and use the periodic retry method instead, which retries flushing the chunks at a specified interval.
These parameters can help you determine the trade-offs between latency and throughput.
- To optimize Fluentd for throughput, you could use these parameters to reduce network packet count by configuring larger buffers and queues, delaying flushes, and setting longer times between retries. Be aware that larger buffers require more space on the node file system.
- To optimize for low latency, you could use the parameters to send data as soon as possible, avoid the build-up of batches, have shorter queues and buffers, and use more frequent flush and retries.
You can configure the chunking and flushing behavior using the following parameters in the ClusterLogging
custom resource (CR). The parameters are then automatically added to the Fluentd config map for use by Fluentd.
These parameters are:
- Not relevant to most users. The default settings should give good general performance.
- Only for advanced users with detailed knowledge of Fluentd configuration and performance.
- Only for performance tuning. They have no effect on functional aspects of logging.
Parameter | Description | Default |
---|---|---|
| The maximum size of each chunk. Fluentd stops writing data to a chunk when it reaches this size. Then, Fluentd sends the chunk to the queue and opens a new chunk. |
|
| The maximum size of the buffer, which is the total size of the stage and the queue. If the buffer size exceeds this value, Fluentd stops adding data to chunks and fails with an error. All data not in chunks is lost. | Approximately 15% of the node disk distributed across all outputs. |
|
The interval between chunk flushes. You can use |
|
| The method to perform flushes:
|
|
| The number of threads that perform chunk flushing. Increasing the number of threads improves the flush throughput, which hides network latency. |
|
| The chunking behavior when the queue is full:
|
|
|
The maximum time in seconds for the |
|
| The retry method when flushing fails:
|
|
| The maximum time interval to attempt retries before the record is discarded. |
|
| The time in seconds before the next chunk flush. |
|
For more information on the Fluentd chunk lifecycle, see Buffer Plugins in the Fluentd documentation.
Procedure
Edit the
ClusterLogging
custom resource (CR) in theopenshift-logging
project:$ oc edit ClusterLogging instance
Add or modify any of the following parameters:
apiVersion: logging.openshift.io/v1 kind: ClusterLogging metadata: name: instance namespace: openshift-logging spec: collection: fluentd: buffer: chunkLimitSize: 8m 1 flushInterval: 5s 2 flushMode: interval 3 flushThreadCount: 3 4 overflowAction: throw_exception 5 retryMaxInterval: "300s" 6 retryType: periodic 7 retryWait: 1s 8 totalLimitSize: 32m 9 # ...
- 1
- Specify the maximum size of each chunk before it is queued for flushing.
- 2
- Specify the interval between chunk flushes.
- 3
- Specify the method to perform chunk flushes:
lazy
,interval
, orimmediate
. - 4
- Specify the number of threads to use for chunk flushes.
- 5
- Specify the chunking behavior when the queue is full:
throw_exception
,block
, ordrop_oldest_chunk
. - 6
- Specify the maximum interval in seconds for the
exponential_backoff
chunk flushing method. - 7
- Specify the retry type when chunk flushing fails:
exponential_backoff
orperiodic
. - 8
- Specify the time in seconds before the next chunk flush.
- 9
- Specify the maximum size of the chunk buffer.
Verify that the Fluentd pods are redeployed:
$ oc get pods -l component=collector -n openshift-logging
Check that the new values are in the
fluentd
config map:$ oc extract configmap/collector-config --confirm
Example fluentd.conf
<buffer> @type file path '/var/lib/fluentd/default' flush_mode interval flush_interval 5s flush_thread_count 3 retry_type periodic retry_wait 1s retry_max_interval 300s retry_timeout 60m queued_chunks_limit_size "#{ENV['BUFFER_QUEUE_LIMIT'] || '32'}" total_limit_size "#{ENV['TOTAL_LIMIT_SIZE_PER_BUFFER'] || '8589934592'}" chunk_limit_size 8m overflow_action throw_exception disable_chunk_backup true </buffer>
9.6. Collecting and storing Kubernetes events
The Red Hat OpenShift Service on AWS Event Router is a pod that watches Kubernetes events and logs them for collection by the logging. You must manually deploy the Event Router.
The Event Router collects events from all projects and writes them to STDOUT
. The collector then forwards those events to the store defined in the ClusterLogForwarder
custom resource (CR).
The Event Router adds additional load to Fluentd and can impact the number of other log messages that can be processed.
9.6.1. Deploying and configuring the Event Router
Use the following steps to deploy the Event Router into your cluster. You should always deploy the Event Router to the openshift-logging
project to ensure it collects events from across the cluster.
The Event Router image is not a part of the Red Hat OpenShift Logging Operator and must be downloaded separately.
The following Template
object creates the service account, cluster role, and cluster role binding required for the Event Router. The template also configures and deploys the Event Router pod. You can either use this template without making changes or edit the template to change the deployment object CPU and memory requests.
Prerequisites
- You need proper permissions to create service accounts and update cluster role bindings. For example, you can run the following template with a user that has the cluster-admin role.
- The Red Hat OpenShift Logging Operator must be installed.
Procedure
Create a template for the Event Router:
apiVersion: template.openshift.io/v1 kind: Template metadata: name: eventrouter-template annotations: description: "A pod forwarding kubernetes events to OpenShift Logging stack." tags: "events,EFK,logging,cluster-logging" objects: - kind: ServiceAccount 1 apiVersion: v1 metadata: name: eventrouter namespace: ${NAMESPACE} - kind: ClusterRole 2 apiVersion: rbac.authorization.k8s.io/v1 metadata: name: event-reader rules: - apiGroups: [""] resources: ["events"] verbs: ["get", "watch", "list"] - kind: ClusterRoleBinding 3 apiVersion: rbac.authorization.k8s.io/v1 metadata: name: event-reader-binding subjects: - kind: ServiceAccount name: eventrouter namespace: ${NAMESPACE} roleRef: kind: ClusterRole name: event-reader - kind: ConfigMap 4 apiVersion: v1 metadata: name: eventrouter namespace: ${NAMESPACE} data: config.json: |- { "sink": "stdout" } - kind: Deployment 5 apiVersion: apps/v1 metadata: name: eventrouter namespace: ${NAMESPACE} labels: component: "eventrouter" logging-infra: "eventrouter" provider: "openshift" spec: selector: matchLabels: component: "eventrouter" logging-infra: "eventrouter" provider: "openshift" replicas: 1 template: metadata: labels: component: "eventrouter" logging-infra: "eventrouter" provider: "openshift" name: eventrouter spec: serviceAccount: eventrouter containers: - name: kube-eventrouter image: ${IMAGE} imagePullPolicy: IfNotPresent resources: requests: cpu: ${CPU} memory: ${MEMORY} volumeMounts: - name: config-volume mountPath: /etc/eventrouter securityContext: allowPrivilegeEscalation: false capabilities: drop: ["ALL"] securityContext: runAsNonRoot: true seccompProfile: type: RuntimeDefault volumes: - name: config-volume configMap: name: eventrouter parameters: - name: IMAGE 6 displayName: Image value: "registry.redhat.io/openshift-logging/eventrouter-rhel9:v0.4" - name: CPU 7 displayName: CPU value: "100m" - name: MEMORY 8 displayName: Memory value: "128Mi" - name: NAMESPACE displayName: Namespace value: "openshift-logging" 9
- 1
- Creates a Service Account in the
openshift-logging
project for the Event Router. - 2
- Creates a ClusterRole to monitor for events in the cluster.
- 3
- Creates a ClusterRoleBinding to bind the ClusterRole to the service account.
- 4
- Creates a config map in the
openshift-logging
project to generate the requiredconfig.json
file. - 5
- Creates a deployment in the
openshift-logging
project to generate and configure the Event Router pod. - 6
- Specifies the image, identified by a tag such as
v0.4
. - 7
- Specifies the minimum amount of CPU to allocate to the Event Router pod. Defaults to
100m
. - 8
- Specifies the minimum amount of memory to allocate to the Event Router pod. Defaults to
128Mi
. - 9
- Specifies the
openshift-logging
project to install objects in.
Use the following command to process and apply the template:
$ oc process -f <templatefile> | oc apply -n openshift-logging -f -
For example:
$ oc process -f eventrouter.yaml | oc apply -n openshift-logging -f -
Example output
serviceaccount/eventrouter created clusterrole.rbac.authorization.k8s.io/event-reader created clusterrolebinding.rbac.authorization.k8s.io/event-reader-binding created configmap/eventrouter created deployment.apps/eventrouter created
Validate that the Event Router installed in the
openshift-logging
project:View the new Event Router pod:
$ oc get pods --selector component=eventrouter -o name -n openshift-logging
Example output
pod/cluster-logging-eventrouter-d649f97c8-qvv8r
View the events collected by the Event Router:
$ oc logs <cluster_logging_eventrouter_pod> -n openshift-logging
For example:
$ oc logs cluster-logging-eventrouter-d649f97c8-qvv8r -n openshift-logging
Example output
{"verb":"ADDED","event":{"metadata":{"name":"openshift-service-catalog-controller-manager-remover.1632d931e88fcd8f","namespace":"openshift-service-catalog-removed","selfLink":"/api/v1/namespaces/openshift-service-catalog-removed/events/openshift-service-catalog-controller-manager-remover.1632d931e88fcd8f","uid":"787d7b26-3d2f-4017-b0b0-420db4ae62c0","resourceVersion":"21399","creationTimestamp":"2020-09-08T15:40:26Z"},"involvedObject":{"kind":"Job","namespace":"openshift-service-catalog-removed","name":"openshift-service-catalog-controller-manager-remover","uid":"fac9f479-4ad5-4a57-8adc-cb25d3d9cf8f","apiVersion":"batch/v1","resourceVersion":"21280"},"reason":"Completed","message":"Job completed","source":{"component":"job-controller"},"firstTimestamp":"2020-09-08T15:40:26Z","lastTimestamp":"2020-09-08T15:40:26Z","count":1,"type":"Normal"}}
You can also use Kibana to view events by creating an index pattern using the Elasticsearch
infra
index.
Chapter 10. Log storage
10.1. About log storage
You can use an internal Loki or Elasticsearch log store on your cluster for storing logs, or you can use a ClusterLogForwarder
custom resource (CR) to forward logs to an external store.
10.1.1. Log storage types
Loki is a horizontally scalable, highly available, multi-tenant log aggregation system offered as a GA log store for logging for Red Hat OpenShift that can be visualized with the OpenShift Observability UI. The Loki configuration provided by OpenShift Logging is a short-term log store designed to enable users to perform fast troubleshooting with the collected logs. For that purpose, the logging for Red Hat OpenShift configuration of Loki has short-term storage, and is optimized for very recent queries. For long-term storage or queries over a long time period, users should look to log stores external to their cluster.
Elasticsearch indexes incoming log records completely during ingestion. Loki indexes only a few fixed labels during ingestion and defers more complex parsing until after the logs have been stored. This means Loki can collect logs more quickly.
10.1.1.1. About the Elasticsearch log store
The logging Elasticsearch instance is optimized and tested for short term storage, approximately seven days. If you want to retain your logs over a longer term, it is recommended you move the data to a third-party storage system.
Elasticsearch organizes the log data from Fluentd into datastores, or indices, then subdivides each index into multiple pieces called shards, which it spreads across a set of Elasticsearch nodes in an Elasticsearch cluster. You can configure Elasticsearch to make copies of the shards, called replicas, which Elasticsearch also spreads across the Elasticsearch nodes. The ClusterLogging
custom resource (CR) allows you to specify how the shards are replicated to provide data redundancy and resilience to failure. You can also specify how long the different types of logs are retained using a retention policy in the ClusterLogging
CR.
The number of primary shards for the index templates is equal to the number of Elasticsearch data nodes.
The Red Hat OpenShift Logging Operator and companion OpenShift Elasticsearch Operator ensure that each Elasticsearch node is deployed using a unique deployment that includes its own storage volume. You can use a ClusterLogging
custom resource (CR) to increase the number of Elasticsearch nodes, as needed. See the Elasticsearch documentation for considerations involved in configuring storage.
A highly-available Elasticsearch environment requires at least three Elasticsearch nodes, each on a different host.
Role-based access control (RBAC) applied on the Elasticsearch indices enables the controlled access of the logs to the developers. Administrators can access all logs and developers can access only the logs in their projects.
10.1.2. Querying log stores
You can query Loki by using the LogQL log query language.
10.1.3. Additional resources
10.2. Installing log storage
You can use the OpenShift CLI (oc
) or the Red Hat OpenShift Service on AWS web console to deploy a log store on your Red Hat OpenShift Service on AWS cluster.
The Logging 5.9 release does not contain an updated version of the OpenShift Elasticsearch Operator. If you currently use the OpenShift Elasticsearch Operator released with Logging 5.8, it will continue to work with Logging until the EOL of Logging 5.8. As an alternative to using the OpenShift Elasticsearch Operator to manage the default log storage, you can use the Loki Operator. For more information on the Logging lifecycle dates, see Platform Agnostic Operators.
10.2.1. Deploying a Loki log store
You can use the Loki Operator to deploy an internal Loki log store on your Red Hat OpenShift Service on AWS cluster. After install the Loki Operator, you must configure Loki object storage by creating a secret, and create a LokiStack
custom resource (CR).
10.2.1.1. Loki deployment sizing
Sizing for Loki follows the format of 1x.<size>
where the value 1x
is number of instances and <size>
specifies performance capabilities.
It is not possible to change the number 1x
for the deployment size.
1x.demo | 1x.extra-small | 1x.small | 1x.medium | |
---|---|---|---|---|
Data transfer | Demo use only | 100GB/day | 500GB/day | 2TB/day |
Queries per second (QPS) | Demo use only | 1-25 QPS at 200ms | 25-50 QPS at 200ms | 25-75 QPS at 200ms |
Replication factor | None | 2 | 2 | 2 |
Total CPU requests | None | 14 vCPUs | 34 vCPUs | 54 vCPUs |
Total CPU requests if using the ruler | None | 16 vCPUs | 42 vCPUs | 70 vCPUs |
Total memory requests | None | 31Gi | 67Gi | 139Gi |
Total memory requests if using the ruler | None | 35Gi | 83Gi | 171Gi |
Total disk requests | 40Gi | 430Gi | 430Gi | 590Gi |
Total disk requests if using the ruler | 80Gi | 750Gi | 750Gi | 910Gi |
10.2.1.2. Installing Logging and the Loki Operator using the web console
To install and configure logging on your Red Hat OpenShift Service on AWS cluster, an Operator such as Loki Operator for log storage must be installed first. This can be done from the OperatorHub within the web console.
Prerequisites
- You have access to a supported object store (AWS S3, Google Cloud Storage, Azure, Swift, Minio, OpenShift Data Foundation).
- You have administrator permissions.
- You have access to the Red Hat OpenShift Service on AWS web console.
Procedure
- In the Red Hat OpenShift Service on AWS web console Administrator perspective, go to Operators → OperatorHub.
Type Loki Operator in the Filter by keyword field. Click Loki Operator in the list of available Operators, and then click Install.
ImportantThe Community Loki Operator is not supported by Red Hat.
Select stable or stable-x.y as the Update channel.
NoteThe stable channel only provides updates to the most recent release of logging. To continue receiving updates for prior releases, you must change your subscription channel to stable-x.y, where
x.y
represents the major and minor version of logging you have installed. For example, stable-5.7.The Loki Operator must be deployed to the global operator group namespace
openshift-operators-redhat
, so the Installation mode and Installed Namespace are already selected. If this namespace does not already exist, it is created for you.Select Enable Operator-recommended cluster monitoring on this namespace.
This option sets the
openshift.io/cluster-monitoring: "true"
label in theNamespace
object. You must select this option to ensure that cluster monitoring scrapes theopenshift-operators-redhat
namespace.For Update approval select Automatic, then click Install.
If the approval strategy in the subscription is set to Automatic, the update process initiates as soon as a new Operator version is available in the selected channel. If the approval strategy is set to Manual, you must manually approve pending updates.
Install the Red Hat OpenShift Logging Operator:
- In the Red Hat OpenShift Service on AWS web console, click Operators → OperatorHub.
- Choose Red Hat OpenShift Logging from the list of available Operators, and click Install.
- Ensure that the A specific namespace on the cluster is selected under Installation Mode.
- Ensure that Operator recommended namespace is openshift-logging under Installed Namespace.
Select Enable Operator recommended cluster monitoring on this namespace.
This option sets the
openshift.io/cluster-monitoring: "true"
label in the Namespace object. You must select this option to ensure that cluster monitoring scrapes theopenshift-logging
namespace.- Select stable-5.y as the Update Channel.
Select an Approval Strategy.
- The Automatic strategy allows Operator Lifecycle Manager (OLM) to automatically update the Operator when a new version is available.
- The Manual strategy requires a user with appropriate credentials to approve the Operator update.
- Click Install.
- Go to the Operators → Installed Operators page. Click the All instances tab.
- From the Create new drop-down list, select LokiStack.
Select YAML view, and then use the following template to create a
LokiStack
CR:Example
LokiStack
CRapiVersion: loki.grafana.com/v1 kind: LokiStack metadata: name: logging-loki 1 namespace: openshift-logging 2 spec: size: 1x.small 3 storage: schemas: - version: v13 effectiveDate: "<yyyy>-<mm>-<dd>" secret: name: logging-loki-s3 4 type: s3 5 credentialMode: 6 storageClassName: <storage_class_name> 7 tenants: mode: openshift-logging 8
- 1
- Use the name
logging-loki
. - 2
- You must specify the
openshift-logging
namespace. - 3
- Specify the deployment size. In the logging 5.8 and later versions, the supported size options for production instances of Loki are
1x.extra-small
,1x.small
, or1x.medium
. - 4
- Specify the name of your log store secret.
- 5
- Specify the corresponding storage type.
- 6
- Optional field, logging 5.9 and later. Supported user configured values are as follows: static is the default authentication mode available for all supported object storage types using credentials stored in a Secret. token for short-lived tokens retrieved from a credential source. In this mode the static configuration does not contain credentials needed for the object storage. Instead, they are generated during runtime using a service, which allows for shorter-lived credentials and much more granular control. This authentication mode is not supported for all object storage types. token-cco is the default value when Loki is running on managed STS mode and using CCO on STS/WIF clusters.
- 7
- Specify the name of a storage class for temporary storage. For best performance, specify a storage class that allocates block storage. Available storage classes for your cluster can be listed by using the
oc get storageclasses
command. - 8
- LokiStack defaults to running in multi-tenant mode, which cannot be modified. One tenant is provided for each log type: audit, infrastructure, and application logs. This enables access control for individual users and user groups to different log streams.
ImportantIt is not possible to change the number
1x
for the deployment size.- Click Create.
Create an OpenShift Logging instance:
- Switch to the Administration → Custom Resource Definitions page.
- On the Custom Resource Definitions page, click ClusterLogging.
- On the Custom Resource Definition details page, select View Instances from the Actions menu.
On the ClusterLoggings page, click Create ClusterLogging.
You might have to refresh the page to load the data.
In the YAML field, replace the code with the following:
apiVersion: logging.openshift.io/v1 kind: ClusterLogging metadata: name: instance 1 namespace: openshift-logging 2 spec: collection: type: vector logStore: lokistack: name: logging-loki retentionPolicy: application: maxAge: 7d audit: maxAge: 7d infra: maxAge: 7d type: lokistack visualization: type: ocp-console ocpConsole: logsLimit: 15 managementState: Managed
Verification
- Go to Operators → Installed Operators.
- Make sure the openshift-logging project is selected.
- In the Status column, verify that you see green checkmarks with InstallSucceeded and the text Up to date.
An Operator might display a Failed
status before the installation finishes. If the Operator install completes with an InstallSucceeded
message, refresh the page.
10.2.1.3. Creating a secret for Loki object storage by using the web console
To configure Loki object storage, you must create a secret. You can create a secret by using the Red Hat OpenShift Service on AWS web console.
Prerequisites
- You have administrator permissions.
- You have access to the Red Hat OpenShift Service on AWS web console.
- You installed the Loki Operator.
Procedure
- Go to Workloads → Secrets in the Administrator perspective of the Red Hat OpenShift Service on AWS web console.
- From the Create drop-down list, select From YAML.
Create a secret that uses the
access_key_id
andaccess_key_secret
fields to specify your credentials and thebucketnames
,endpoint
, andregion
fields to define the object storage location. AWS is used in the following example:Example
Secret
objectapiVersion: v1 kind: Secret metadata: name: logging-loki-s3 namespace: openshift-logging stringData: access_key_id: AKIAIOSFODNN7EXAMPLE access_key_secret: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY bucketnames: s3-bucket-name endpoint: https://s3.eu-central-1.amazonaws.com region: eu-central-1
Additional resources
10.2.1.4. Workload identity federation
Workload identity federation enables authentication to cloud-based log stores using short-lived tokens.
Prerequisites
- Red Hat OpenShift Service on AWS 4.14 and later
- Logging 5.9 and later
Procedure
-
If you use the Red Hat OpenShift Service on AWS web console to install the Loki Operator, clusters that use short-lived tokens are automatically detected. You are prompted to create roles and supply the data required for the Loki Operator to create a
CredentialsRequest
object, which populates a secret. -
If you use the OpenShift CLI (
oc
) to install the Loki Operator, you must manually create a subscription object using the appropriate template for your storage provider, as shown in the following examples. This authentication strategy is only supported for the storage providers indicated.
Azure sample subscription
apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: loki-operator namespace: openshift-operators-redhat spec: channel: "stable-5.9" installPlanApproval: Manual name: loki-operator source: redhat-operators sourceNamespace: openshift-marketplace config: env: - name: CLIENTID value: <your_client_id> - name: TENANTID value: <your_tenant_id> - name: SUBSCRIPTIONID value: <your_subscription_id> - name: REGION value: <your_region>
AWS sample subscription
apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: loki-operator namespace: openshift-operators-redhat spec: channel: "stable-5.9" installPlanApproval: Manual name: loki-operator source: redhat-operators sourceNamespace: openshift-marketplace config: env: - name: ROLEARN value: <role_ARN>
10.2.1.5. Creating a LokiStack custom resource by using the web console
You can create a LokiStack
custom resource (CR) by using the Red Hat OpenShift Service on AWS web console.
Prerequisites
- You have administrator permissions.
- You have access to the Red Hat OpenShift Service on AWS web console.
- You installed the Loki Operator.
Procedure
- Go to the Operators → Installed Operators page. Click the All instances tab.
- From the Create new drop-down list, select LokiStack.
Select YAML view, and then use the following template to create a
LokiStack
CR:apiVersion: loki.grafana.com/v1 kind: LokiStack metadata: name: logging-loki 1 namespace: openshift-logging spec: size: 1x.small 2 storage: schemas: - effectiveDate: '2023-10-15' version: v13 secret: name: logging-loki-s3 3 type: s3 4 credentialMode: 5 storageClassName: <storage_class_name> 6 tenants: mode: openshift-logging
- 1
- Use the name
logging-loki
. - 2
- Specify the deployment size. In the logging 5.8 and later versions, the supported size options for production instances of Loki are
1x.extra-small
,1x.small
, or1x.medium
. - 3
- Specify the secret used for your log storage.
- 4
- Specify the corresponding storage type.
- 5
- Optional field, logging 5.9 and later. Supported user configured values are as follows:
static
is the default authentication mode available for all supported object storage types using credentials stored in a Secret.token
for short-lived tokens retrieved from a credential source. In this mode the static configuration does not contain credentials needed for the object storage. Instead, they are generated during runtime using a service, which allows for shorter-lived credentials and much more granular control. This authentication mode is not supported for all object storage types.token-cco
is the default value when Loki is running on managed STS mode and using CCO on STS/WIF clusters. - 6
- Enter the name of a storage class for temporary storage. For best performance, specify a storage class that allocates block storage. Available storage classes for your cluster can be listed by using the
oc get storageclasses
command.
10.2.1.6. Installing Logging and the Loki Operator using the CLI
To install and configure logging on your Red Hat OpenShift Service on AWS cluster, an Operator such as Loki Operator for log storage must be installed first. This can be done from the Red Hat OpenShift Service on AWS CLI.
Prerequisites
- You have administrator permissions.
-
You installed the OpenShift CLI (
oc
). - You have access to a supported object store. For example: AWS S3, Google Cloud Storage, Azure, Swift, Minio, or OpenShift Data Foundation.
The stable channel only provides updates to the most recent release of logging. To continue receiving updates for prior releases, you must change your subscription channel to stable-x.y, where x.y
represents the major and minor version of logging you have installed. For example, stable-5.7.
Create a
Namespace
object for Loki Operator:Example
Namespace
objectapiVersion: v1 kind: Namespace metadata: name: openshift-operators-redhat 1 annotations: openshift.io/node-selector: "" labels: openshift.io/cluster-monitoring: "true" 2
- 1
- You must specify the
openshift-operators-redhat
namespace. To prevent possible conflicts with metrics, you should configure the Prometheus Cluster Monitoring stack to scrape metrics from theopenshift-operators-redhat
namespace and not theopenshift-operators
namespace. Theopenshift-operators
namespace might contain community Operators, which are untrusted and could publish a metric with the same name as an Red Hat OpenShift Service on AWS metric, which would cause conflicts. - 2
- A string value that specifies the label as shown to ensure that cluster monitoring scrapes the
openshift-operators-redhat
namespace.
Apply the
Namespace
object by running the following command:$ oc apply -f <filename>.yaml
Create a
Subscription
object for Loki Operator:Example
Subscription
objectapiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: loki-operator namespace: openshift-operators-redhat 1 spec: channel: stable 2 name: loki-operator source: redhat-operators 3 sourceNamespace: openshift-marketplace
- 1
- You must specify the
openshift-operators-redhat
namespace. - 2
- Specify
stable
, orstable-5.<y>
as the channel. - 3
- Specify
redhat-operators
. If your Red Hat OpenShift Service on AWS cluster is installed on a restricted network, also known as a disconnected cluster, specify the name of theCatalogSource
object you created when you configured the Operator Lifecycle Manager (OLM).
Apply the
Subscription
object by running the following command:$ oc apply -f <filename>.yaml
Create a
namespace
object for the Red Hat OpenShift Logging Operator:Example
namespace
objectapiVersion: v1 kind: Namespace metadata: name: openshift-logging 1 annotations: openshift.io/node-selector: "" labels: openshift.io/cluster-logging: "true" openshift.io/cluster-monitoring: "true" 2
Apply the
namespace
object by running the following command:$ oc apply -f <filename>.yaml
Create an
OperatorGroup
objectExample
OperatorGroup
objectapiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: cluster-logging namespace: openshift-logging 1 spec: targetNamespaces: - openshift-logging
- 1
- You must specify the
openshift-logging
namespace.
Apply the
OperatorGroup
object by running the following command:$ oc apply -f <filename>.yaml
Create a
Subscription
object:apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: cluster-logging namespace: openshift-logging 1 spec: channel: stable 2 name: cluster-logging source: redhat-operators 3 sourceNamespace: openshift-marketplace
- 1
- You must specify the
openshift-logging
namespace. - 2
- Specify
stable
, orstable-5.<y>
as the channel. - 3
- Specify
redhat-operators
. If your Red Hat OpenShift Service on AWS cluster is installed on a restricted network, also known as a disconnected cluster, specify the name of the CatalogSource object you created when you configured the Operator Lifecycle Manager (OLM).
Apply the
Subscription
object by running the following command:$ oc apply -f <filename>.yaml
Create a
LokiStack
CR:Example
LokiStack
CRapiVersion: loki.grafana.com/v1 kind: LokiStack metadata: name: logging-loki 1 namespace: openshift-logging 2 spec: size: 1x.small 3 storage: schemas: - version: v13 effectiveDate: "<yyyy>-<mm>-<dd>" secret: name: logging-loki-s3 4 type: s3 5 credentialMode: 6 storageClassName: <storage_class_name> 7 tenants: mode: openshift-logging 8
- 1
- Use the name
logging-loki
. - 2
- You must specify the
openshift-logging
namespace. - 3
- Specify the deployment size. In the logging 5.8 and later versions, the supported size options for production instances of Loki are
1x.extra-small
,1x.small
, or1x.medium
. - 4
- Specify the name of your log store secret.
- 5
- Specify the corresponding storage type.
- 6
- Optional field, logging 5.9 and later. Supported user configured values are as follows:
static
is the default authentication mode available for all supported object storage types using credentials stored in a Secret.token
for short-lived tokens retrieved from a credential source. In this mode the static configuration does not contain credentials needed for the object storage. Instead, they are generated during runtime using a service, which allows for shorter-lived credentials and much more granular control. This authentication mode is not supported for all object storage types.token-cco
is the default value when Loki is running on managed STS mode and using CCO on STS/WIF clusters. - 7
- Specify the name of a storage class for temporary storage. For best performance, specify a storage class that allocates block storage. Available storage classes for your cluster can be listed by using the
oc get storageclasses
command. - 8
- LokiStack defaults to running in multi-tenant mode, which cannot be modified. One tenant is provided for each log type: audit, infrastructure, and application logs. This enables access control for individual users and user groups to different log streams.
Apply the
LokiStack CR
object by running the following command:$ oc apply -f <filename>.yaml
Create a
ClusterLogging
CR object:Example ClusterLogging CR object
apiVersion: logging.openshift.io/v1 kind: ClusterLogging metadata: name: instance 1 namespace: openshift-logging 2 spec: collection: type: vector logStore: lokistack: name: logging-loki retentionPolicy: application: maxAge: 7d audit: maxAge: 7d infra: maxAge: 7d type: lokistack visualization: type: ocp-console ocpConsole: logsLimit: 15 managementState: Managed
Apply the
ClusterLogging CR
object by running the following command:$ oc apply -f <filename>.yaml
Verify the installation by running the following command:
$ oc get pods -n openshift-logging
Example output
$ oc get pods -n openshift-logging NAME READY STATUS RESTARTS AGE cluster-logging-operator-fb7f7cf69-8jsbq 1/1 Running 0 98m collector-222js 2/2 Running 0 18m collector-g9ddv 2/2 Running 0 18m collector-hfqq8 2/2 Running 0 18m collector-sphwg 2/2 Running 0 1