Dieser Inhalt ist in der von Ihnen ausgewählten Sprache nicht verfügbar.
Chapter 1. Release notes
1.1. Logging 5.9
Logging is provided as an installable component, with a distinct release cycle from the core Red Hat OpenShift Service on AWS. The Red Hat OpenShift Container Platform Life Cycle Policy outlines release compatibility.
The stable channel only provides updates to the most recent release of logging. To continue receiving updates for prior releases, you must change your subscription channel to stable-x.y, where x.y
represents the major and minor version of logging you have installed. For example, stable-5.7.
1.1.1. Logging 5.9.3
This release includes OpenShift Logging Bug Fix Release 5.9.3
1.1.1.1. Bug Fixes
-
Before this update, there was a delay in restarting Ingesters when configuring
LokiStack
, because the Loki Operator sets the write-ahead logreplay_memory_ceiling
to zero bytes for the1x.demo
size. With this update, the minimum value used for thereplay_memory_ceiling
has been increased to avoid delays. (LOG-5614) - Before this update, monitoring the Vector collector output buffer state was not possible. With this update, monitoring and alerting the Vector collector output buffer size is possible that improves observability capabilities and helps keep the system running optimally. (LOG-5586)
1.1.1.2. CVEs
1.1.2. Logging 5.9.2
This release includes OpenShift Logging Bug Fix Release 5.9.2
1.1.2.1. Bug Fixes
-
Before this update, changes to the Logging Operator caused an error due to an incorrect configuration in the
ClusterLogForwarder
CR. As a result, upgrades to logging deleted the daemonset collector. With this update, the Logging Operator re-creates collector daemonsets except when aNot authorized to collect
error occurs. (LOG-4910) - Before this update, the rotated infrastructure log files were sent to the application index in some scenarios due to an incorrect configuration in the Vector log collector. With this update, the Vector log collector configuration avoids collecting any rotated infrastructure log files. (LOG-5156)
-
Before this update, the Logging Operator did not monitor changes to the
grafana-dashboard-cluster-logging
config map. With this update, the Logging Operator monitors changes in theConfigMap
objects, ensuring the system stays synchronized and responds effectively to config map modifications. (LOG-5308) - Before this update, an issue in the metrics collection code of the Logging Operator caused it to report stale telemetry metrics. With this update, the Logging Operator does not report stale telemetry metrics. (LOG-5426)
-
Before this change, the Fluentd
out_http
plugin ignored theno_proxy
environment variable. With this update, the Fluentd patches theHTTP#start
method of ruby to honor theno_proxy
environment variable. (LOG-5466)
1.1.2.2. CVEs
1.1.3. Logging 5.9.1
This release includes OpenShift Logging Bug Fix Release 5.9.1
1.1.3.1. Enhancements
- Before this update, the Loki Operator configured Loki to use path-based style access for the Amazon Simple Storage Service (S3), which has been deprecated. With this update, the Loki Operator defaults to virtual-host style without users needing to change their configuration. (LOG-5401)
-
Before this update, the Loki Operator did not validate the Amazon Simple Storage Service (S3) endpoint used in the storage secret. With this update, the validation process ensures the S3 endpoint is a valid S3 URL, and the
LokiStack
status updates to indicate any invalid URLs. (LOG-5395)
1.1.3.2. Bug Fixes
- Before this update, a bug in LogQL parsing left out some line filters from the query. With this update, the parsing now includes all the line filters while keeping the original query unchanged. (LOG-5268)
-
Before this update, a prune filter without a defined
pruneFilterSpec
would cause a segfault. With this update, there is a validation error if a prune filter is without a definedpuneFilterSpec
. (LOG-5322) -
Before this update, a drop filter without a defined
dropTestsSpec
would cause a segfault. With this update, there is a validation error if a prune filter is without a definedpuneFilterSpec
. (LOG-5323) -
Before this update, the Loki Operator did not validate the Amazon Simple Storage Service (S3) endpoint URL format used in the storage secret. With this update, the S3 endpoint URL goes through a validation step that reflects on the status of the
LokiStack
. (LOG-5397) -
Before this update, poorly formatted timestamp fields in audit log records led to
WARN
messages in Red Hat OpenShift Logging Operator logs. With this update, a remap transformation ensures that the timestamp field is properly formatted. (LOG-4672) -
Before this update, the error message thrown while validating a
ClusterLogForwarder
resource name and namespace did not correspond to the correct error. With this update, the system checks if aClusterLogForwarder
resource with the same name exists in the same namespace. If not, it corresponds to the correct error. (LOG-5062) - Before this update, the validation feature for output config required a TLS URL, even for services such as Amazon CloudWatch or Google Cloud Logging where a URL is not needed by design. With this update, the validation logic for services without URLs are improved, and the error message are more informative. (LOG-5307)
- Before this update, defining an infrastructure input type did not exclude logging workloads from the collection. With this update, the collection excludes logging services to avoid feedback loops. (LOG-5309)
1.1.3.3. CVEs
No CVEs.
1.1.4. Logging 5.9.0
This release includes OpenShift Logging Bug Fix Release 5.9.0
1.1.4.1. Removal notice
The Logging 5.9 release does not contain an updated version of the OpenShift Elasticsearch Operator. Instances of OpenShift Elasticsearch Operator from prior logging releases, remain supported until the EOL of the logging release. As an alternative to using the OpenShift Elasticsearch Operator to manage the default log storage, you can use the Loki Operator. For more information on the Logging lifecycle dates, see Platform Agnostic Operators.
1.1.4.2. Deprecation notice
- In Logging 5.9, Fluentd, and Kibana are deprecated and are planned to be removed in Logging 6.0, which is expected to be shipped alongside a future release of Red Hat OpenShift Service on AWS. Red Hat will provide critical and above CVE bug fixes and support for these components during the current release lifecycle, but these components will no longer receive feature enhancements. The Vector-based collector provided by the Red Hat OpenShift Logging Operator and LokiStack provided by the Loki Operator are the preferred Operators for log collection and storage. We encourage all users to adopt the Vector and Loki log stack, as this will be the stack that will be enhanced going forward.
-
In Logging 5.9, the
Fields
option for the Splunk output type was never implemented and is now deprecated. It will be removed in a future release.
1.1.4.3. Enhancements
1.1.4.3.1. Log Collection
-
This enhancement adds the ability to refine the process of log collection by using a workload’s metadata to
drop
orprune
logs based on their content. Additionally, it allows the collection of infrastructure logs, such as journal or container logs, and audit logs, such askube api
orovn
logs, to only collect individual sources. (LOG-2155) - This enhancement introduces a new type of remote log receiver, the syslog receiver. You can configure it to expose a port over a network, allowing external systems to send syslog logs using compatible tools such as rsyslog. (LOG-3527)
-
With this update, the
ClusterLogForwarder
API now supports log forwarding to Azure Monitor Logs, giving users better monitoring abilities. This feature helps users to maintain optimal system performance and streamline the log analysis processes in Azure Monitor, which speeds up issue resolution and improves operational efficiency. (LOG-4605) -
This enhancement improves collector resource utilization by deploying collectors as a deployment with two replicas. This occurs when the only input source defined in the
ClusterLogForwarder
custom resource (CR) is a receiver input instead of using a daemon set on all nodes. Additionally, collectors deployed in this manner do not mount the host file system. To use this enhancement, you need to annotate theClusterLogForwarder
CR with thelogging.openshift.io/dev-preview-enable-collector-as-deployment
annotation. (LOG-4779) - This enhancement introduces the capability for custom tenant configuration across all supported outputs, facilitating the organization of log records in a logical manner. However, it does not permit custom tenant configuration for logging managed storage. (LOG-4843)
-
With this update, the
ClusterLogForwarder
CR that specifies an application input with one or more infrastructure namespaces likedefault
,openshift*
, orkube*
, now requires a service account with thecollect-infrastructure-logs
role. (LOG-4943) -
This enhancement introduces the capability for tuning some output settings, such as compression, retry duration, and maximum payloads, to match the characteristics of the receiver. Additionally, this feature includes a delivery mode to allow administrators to choose between throughput and log durability. For example, the
AtLeastOnce
option configures minimal disk buffering of collected logs so that the collector can deliver those logs after a restart. (LOG-5026) - This enhancement adds three new Prometheus alerts, warning users about the deprecation of Elasticsearch, Fluentd, and Kibana. (LOG-5055)
1.1.4.3.2. Log Storage
- This enhancement in LokiStack improves support for OTEL by using the new V13 object storage format and enabling automatic stream sharding by default. This also prepares the collector for future enhancements and configurations. (LOG-4538)
-
This enhancement introduces support for short-lived token workload identity federation with Azure and AWS log stores for STS enabled Red Hat OpenShift Service on AWS 4.14 and later clusters. Local storage requires the addition of a
CredentialMode: static
annotation underspec.storage.secret
in the LokiStack CR. (LOG-4540) - With this update, the validation of the Azure storage secret is now extended to give early warning for certain error conditions. (LOG-4571)
- With this update, Loki now adds upstream and downstream support for GCP workload identity federation mechanism. This allows authenticated and authorized access to the corresponding object storage services. (LOG-4754)
1.1.4.4. Bug Fixes
-
Before this update, the logging must-gather could not collect any logs on a FIPS-enabled cluster. With this update, a new
oc
client is available incluster-logging-rhel9-operator
, and must-gather works properly on FIPS clusters. (LOG-4403) - Before this update, the LokiStack ruler pods could not format the IPv6 pod IP in HTTP URLs used for cross-pod communication. This issue caused querying rules and alerts through the Prometheus-compatible API to fail. With this update, the LokiStack ruler pods encapsulate the IPv6 pod IP in square brackets, resolving the problem. Now, querying rules and alerts through the Prometheus-compatible API works just like in IPv4 environments. (LOG-4709)
- Before this fix, the YAML content from the logging must-gather was exported in a single line, making it unreadable. With this update, the YAML white spaces are preserved, ensuring that the file is properly formatted. (LOG-4792)
-
Before this update, when the
ClusterLogForwarder
CR was enabled, the Red Hat OpenShift Logging Operator could run into a nil pointer exception whenClusterLogging.Spec.Collection
was nil. With this update, the issue is now resolved in the Red Hat OpenShift Logging Operator. (LOG-5006) -
Before this update, in specific corner cases, replacing the
ClusterLogForwarder
CR status field caused theresourceVersion
to constantly update due to changing timestamps inStatus
conditions. This condition led to an infinite reconciliation loop. With this update, all status conditions synchronize, so that timestamps remain unchanged if conditions stay the same. (LOG-5007) -
Before this update, there was an internal buffering behavior to
drop_newest
to address high memory consumption by the collector resulting in significant log loss. With this update, the behavior reverts to using the collector defaults. (LOG-5123) -
Before this update, the Loki Operator
ServiceMonitor
in theopenshift-operators-redhat
namespace used static token and CA files for authentication, causing errors in the Prometheus Operator in the User Workload Monitoring spec on theServiceMonitor
configuration. With this update, the Loki OperatorServiceMonitor
inopenshift-operators-redhat
namespace now references a service account token secret by aLocalReference
object. This approach allows the User Workload Monitoring spec in the Prometheus Operator to handle the Loki OperatorServiceMonitor
successfully, enabling Prometheus to scrape the Loki Operator metrics. (LOG-5165) -
Before this update, the configuration of the Loki Operator
ServiceMonitor
could match many Kubernetes services, resulting in the Loki Operator metrics being collected multiple times. With this update, the configuration ofServiceMonitor
now only matches the dedicated metrics service. (LOG-5212)
1.1.4.5. Known Issues
None.
1.1.4.6. CVEs
1.2. Logging 5.8
Logging is provided as an installable component, with a distinct release cycle from the core Red Hat OpenShift Service on AWS. The Red Hat OpenShift Container Platform Life Cycle Policy outlines release compatibility.
The stable channel only provides updates to the most recent release of logging. To continue receiving updates for prior releases, you must change your subscription channel to stable-x.y, where x.y
represents the major and minor version of logging you have installed. For example, stable-5.7.
1.2.1. Logging 5.8.4
This release includes OpenShift Logging Bug Fix Release 5.8.4.
1.2.1.1. Bug fixes
- Before this update, the developer console’s logs did not account for the current namespace, resulting in query rejection for users without cluster-wide log access. With this update, all supported OCP versions ensure correct namespace inclusion. (LOG-4905)
-
Before this update, the Cluster Logging Operator deployed
ClusterRoles
supporting LokiStack deployments only when the default log output was LokiStack. With this update, the roles are split into two groups: read and write. The write roles deploys based on the setting of the default log storage, just like all the roles used to do before. The read roles deploys based on whether the logging console plugin is active. (LOG-4987) -
Before this update, multiple
ClusterLogForwarders
defining the same input receiver name had their service endlessly reconciled because of changingownerReferences
on one service. With this update, each receiver input will have its own service named with the convention of<CLF.Name>-<input.Name>
. (LOG-5009) -
Before this update, the
ClusterLogForwarder
did not report errors when forwarding logs to cloudwatch without a secret. With this update, the following error message appears when forwarding logs to cloudwatch without a secret:secret must be provided for cloudwatch output
. (LOG-5021) -
Before this update, the
log_forwarder_input_info
includedapplication
,infrastructure
, andaudit
input metric points. With this update,http
is also added as a metric point. (LOG-5043)
1.2.1.2. CVEs
- CVE-2021-35937
- CVE-2021-35938
- CVE-2021-35939
- CVE-2022-3545
- CVE-2022-24963
- CVE-2022-36402
- CVE-2022-41858
- CVE-2023-2166
- CVE-2023-2176
- CVE-2023-3777
- CVE-2023-3812
- CVE-2023-4015
- CVE-2023-4622
- CVE-2023-4623
- CVE-2023-5178
- CVE-2023-5363
- CVE-2023-5388
- CVE-2023-5633
- CVE-2023-6679
- CVE-2023-7104
- CVE-2023-27043
- CVE-2023-38409
- CVE-2023-40283
- CVE-2023-42753
- CVE-2023-43804
- CVE-2023-45803
- CVE-2023-46813
- CVE-2024-20918
- CVE-2024-20919
- CVE-2024-20921
- CVE-2024-20926
- CVE-2024-20945
- CVE-2024-20952
1.2.2. Logging 5.8.3
This release includes Logging Bug Fix 5.8.3 and Logging Security Fix 5.8.3
1.2.2.1. Bug fixes
- Before this update, when configured to read a custom S3 Certificate Authority the Loki Operator would not automatically update the configuration when the name of the ConfigMap or the contents changed. With this update, the Loki Operator is watching for changes to the ConfigMap and automatically updates the generated configuration. (LOG-4969)
- Before this update, Loki outputs configured without a valid URL caused the collector pods to crash. With this update, outputs are subject to URL validation, resolving the issue. (LOG-4822)
- Before this update the Cluster Logging Operator would generate collector configuration fields for outputs that did not specify a secret to use the service account bearer token. With this update, an output does not require authentication, resolving the issue. (LOG-4962)
-
Before this update, the
tls.insecureSkipVerify
field of an output was not set to a value oftrue
without a secret defined. With this update, a secret is no longer required to set this value. (LOG-4963) - Before this update, output configurations allowed the combination of an insecure (HTTP) URL with TLS authentication. With this update, outputs configured for TLS authentication require a secure (HTTPS) URL. (LOG-4893)
1.2.2.2. CVEs
1.2.3. Logging 5.8.2
This release includes OpenShift Logging Bug Fix Release 5.8.2.
1.2.3.1. Bug fixes
- Before this update, the LokiStack ruler pods would not format the IPv6 pod IP in HTTP URLs used for cross pod communication, causing querying rules and alerts through the Prometheus-compatible API to fail. With this update, the LokiStack ruler pods encapsulate the IPv6 pod IP in square brackets, resolving the issue. (LOG-4890)
- Before this update, the developer console logs did not account for the current namespace, resulting in query rejection for users without cluster-wide log access. With this update, namespace inclusion has been corrected, resolving the issue. (LOG-4947)
- Before this update, the logging view plugin of the Red Hat OpenShift Service on AWS web console did not allow for custom node placement and tolerations. With this update, defining custom node placements and tolerations has been added to the logging view plugin of the Red Hat OpenShift Service on AWS web console. (LOG-4912)
1.2.3.2. CVEs
1.2.4. Logging 5.8.1
This release includes OpenShift Logging Bug Fix Release 5.8.1 and OpenShift Logging Bug Fix Release 5.8.1 Kibana.
1.2.4.1. Enhancements
1.2.4.1.1. Log Collection
- With this update, while configuring Vector as a collector, you can add logic to the Red Hat OpenShift Logging Operator to use a token specified in the secret in place of the token associated with the service account. (LOG-4780)
- With this update, the BoltDB Shipper Loki dashboards are now renamed to Index dashboards. (LOG-4828)
1.2.4.2. Bug fixes
-
Before this update, the
ClusterLogForwarder
created empty indices after enabling the parsing of JSON logs, even when the rollover conditions were not met. With this update, theClusterLogForwarder
skips the rollover when thewrite-index
is empty. (LOG-4452) -
Before this update, the Vector set the
default
log level incorrectly. With this update, the correct log level is set by improving the enhancement of regular expression, orregexp
, for log level detection. (LOG-4480) -
Before this update, during the process of creating index patterns, the default alias was missing from the initial index in each log output. As a result, Kibana users were unable to create index patterns by using OpenShift Elasticsearch Operator. This update adds the missing aliases to OpenShift Elasticsearch Operator, resolving the issue. Kibana users can now create index patterns that include the
{app,infra,audit}-000001
indexes. (LOG-4683) -
Before this update, Fluentd collector pods were in a
CrashLoopBackOff
state due to binding of the Prometheus server on IPv6 clusters. With this update, the collectors work properly on IPv6 clusters. (LOG-4706) -
Before this update, the Red Hat OpenShift Logging Operator would undergo numerous reconciliations whenever there was a change in the
ClusterLogForwarder
. With this update, the Red Hat OpenShift Logging Operator disregards the status changes in the collector daemonsets that triggered the reconciliations. (LOG-4741) -
Before this update, the Vector log collector pods were stuck in the
CrashLoopBackOff
state on IBM Power machines. With this update, the Vector log collector pods start successfully on IBM Power architecture machines. (LOG-4768) -
Before this update, forwarding with a legacy forwarder to an internal LokiStack would produce SSL certificate errors using Fluentd collector pods. With this update, the log collector service account is used by default for authentication, using the associated token and
ca.crt
. (LOG-4791) -
Before this update, forwarding with a legacy forwarder to an internal LokiStack would produce SSL certificate errors using Vector collector pods. With this update, the log collector service account is used by default for authentication and also using the associated token and
ca.crt
. (LOG-4852) - Before this fix, IPv6 addresses would not be parsed correctly after evaluating a host or multiple hosts for placeholders. With this update, IPv6 addresses are correctly parsed. (LOG-4811)
-
Before this update, it was necessary to create a
ClusterRoleBinding
to collect audit permissions for HTTP receiver inputs. With this update, it is not necessary to create theClusterRoleBinding
because the endpoint already depends upon the cluster certificate authority. (LOG-4815) - Before this update, the Loki Operator did not mount a custom CA bundle to the ruler pods. As a result, during the process to evaluate alerting or recording rules, object storage access failed. With this update, the Loki Operator mounts the custom CA bundle to all ruler pods. The ruler pods can download logs from object storage to evaluate alerting or recording rules. (LOG-4836)
-
Before this update, while removing the
inputs.receiver
section in theClusterLogForwarder
, the HTTP input services and its associated secrets were not deleted. With this update, the HTTP input resources are deleted when not needed. (LOG-4612) -
Before this update, the
ClusterLogForwarder
indicated validation errors in the status, but the outputs and the pipeline status did not accurately reflect the specific issues. With this update, the pipeline status displays the validation failure reasons correctly in case of misconfigured outputs, inputs, or filters. (LOG-4821) -
Before this update, changing a
LogQL
query that used controls such as time range or severity changed the label matcher operator defining it like a regular expression. With this update, regular expression operators remain unchanged when updating the query. (LOG-4841)
1.2.4.3. CVEs
- CVE-2007-4559
- CVE-2021-3468
- CVE-2021-3502
- CVE-2021-3826
- CVE-2021-43618
- CVE-2022-3523
- CVE-2022-3565
- CVE-2022-3594
- CVE-2022-4285
- CVE-2022-38457
- CVE-2022-40133
- CVE-2022-40982
- CVE-2022-41862
- CVE-2022-42895
- CVE-2023-0597
- CVE-2023-1073
- CVE-2023-1074
- CVE-2023-1075
- CVE-2023-1076
- CVE-2023-1079
- CVE-2023-1206
- CVE-2023-1249
- CVE-2023-1252
- CVE-2023-1652
- CVE-2023-1855
- CVE-2023-1981
- CVE-2023-1989
- CVE-2023-2731
- CVE-2023-3138
- CVE-2023-3141
- CVE-2023-3161
- CVE-2023-3212
- CVE-2023-3268
- CVE-2023-3316
- CVE-2023-3358
- CVE-2023-3576
- CVE-2023-3609
- CVE-2023-3772
- CVE-2023-3773
- CVE-2023-4016
- CVE-2023-4128
- CVE-2023-4155
- CVE-2023-4194
- CVE-2023-4206
- CVE-2023-4207
- CVE-2023-4208
- CVE-2023-4273
- CVE-2023-4641
- CVE-2023-22745
- CVE-2023-26545
- CVE-2023-26965
- CVE-2023-26966
- CVE-2023-27522
- CVE-2023-29491
- CVE-2023-29499
- CVE-2023-30456
- CVE-2023-31486
- CVE-2023-32324
- CVE-2023-32573
- CVE-2023-32611
- CVE-2023-32665
- CVE-2023-33203
- CVE-2023-33285
- CVE-2023-33951
- CVE-2023-33952
- CVE-2023-34241
- CVE-2023-34410
- CVE-2023-35825
- CVE-2023-36054
- CVE-2023-37369
- CVE-2023-38197
- CVE-2023-38545
- CVE-2023-38546
- CVE-2023-39191
- CVE-2023-39975
- CVE-2023-44487
1.2.5. Logging 5.8.0
This release includes OpenShift Logging Bug Fix Release 5.8.0 and OpenShift Logging Bug Fix Release 5.8.0 Kibana.
1.2.5.1. Deprecation notice
In Logging 5.8, Elasticsearch, Fluentd, and Kibana are deprecated and are planned to be removed in Logging 6.0, which is expected to be shipped alongside a future release of Red Hat OpenShift Service on AWS. Red Hat will provide critical and above CVE bug fixes and support for these components during the current release lifecycle, but these components will no longer receive feature enhancements. The Vector-based collector provided by the Red Hat OpenShift Logging Operator and LokiStack provided by the Loki Operator are the preferred Operators for log collection and storage. We encourage all users to adopt the Vector and Loki log stack, as this will be the stack that will be enhanced going forward.
1.2.5.2. Enhancements
1.2.5.2.1. Log Collection
-
With this update, the LogFileMetricExporter is no longer deployed with the collector by default. You must manually create a
LogFileMetricExporter
custom resource (CR) to generate metrics from the logs produced by running containers. If you do not create theLogFileMetricExporter
CR, you may see a No datapoints found message in the Red Hat OpenShift Service on AWS web console dashboard for Produced Logs. (LOG-3819) With this update, you can deploy multiple, isolated, and RBAC-protected
ClusterLogForwarder
custom resource (CR) instances in any namespace. This allows independent groups to forward desired logs to any destination while isolating their configuration from other collector deployments. (LOG-1343)ImportantIn order to support multi-cluster log forwarding in additional namespaces other than the
openshift-logging
namespace, you must update the Red Hat OpenShift Logging Operator to watch all namespaces. This functionality is supported by default in new Red Hat OpenShift Logging Operator version 5.8 installations.- With this update, you can use the flow control or rate limiting mechanism to limit the volume of log data that can be collected or forwarded by dropping excess log records. The input limits prevent poorly-performing containers from overloading the Logging and the output limits put a ceiling on the rate of logs shipped to a given data store. (LOG-884)
- With this update, you can configure the log collector to look for HTTP connections and receive logs as an HTTP server, also known as a webhook. (LOG-4562)
- With this update, you can configure audit polices to control which Kubernetes and OpenShift API server events are forwarded by the log collector. (LOG-3982)
1.2.5.2.2. Log Storage
- With this update, LokiStack administrators can have more fine-grained control over who can access which logs by granting access to logs on a namespace basis. (LOG-3841)
-
With this update, the Loki Operator introduces
PodDisruptionBudget
configuration on LokiStack deployments to ensure normal operations during Red Hat OpenShift Service on AWS cluster restarts by keeping ingestion and the query path available. (LOG-3839) - With this update, the reliability of existing LokiStack installations are seamlessly improved by applying a set of default Affinity and Anti-Affinity policies. (LOG-3840)
- With this update, you can manage zone-aware data replication as an administrator in LokiStack, in order to enhance reliability in the event of a zone failure. (LOG-3266)
- With this update, a new supported small-scale LokiStack size of 1x.extra-small is introduced for Red Hat OpenShift Service on AWS clusters hosting a few workloads and smaller ingestion volumes (up to 100GB/day). (LOG-4329)
- With this update, the LokiStack administrator has access to an official Loki dashboard to inspect the storage performance and the health of each component. (LOG-4327)
1.2.5.2.3. Log Console
- With this update, you can enable the Logging Console Plugin when Elasticsearch is the default Log Store. (LOG-3856)
- With this update, Red Hat OpenShift Service on AWS application owners can receive notifications for application log-based alerts on the Red Hat OpenShift Service on AWS web console Developer perspective for Red Hat OpenShift Service on AWS version 4.14 and later. (LOG-3548)
1.2.5.3. Known Issues
Currently, Splunk log forwarding might not work after upgrading to version 5.8 of the Red Hat OpenShift Logging Operator. This issue is caused by transitioning from OpenSSL version 1.1.1 to version 3.0.7. In the newer OpenSSL version, there is a default behavior change, where connections to TLS 1.2 endpoints are rejected if they do not expose the RFC 5746 extension.
As a workaround, enable TLS 1.3 support on the TLS terminating load balancer in front of the Splunk HEC (HTTP Event Collector) endpoint. Splunk is a third-party system and this should be configured from the Splunk end.
-
Currently, there is a flaw in handling multiplexed streams in the HTTP/2 protocol, where you can repeatedly make a request for a new multiplex stream and immediately send an
RST_STREAM
frame to cancel it. This created extra work for the server set up and tore down the streams, resulting in a denial of service due to server resource consumption. There is currently no workaround for this issue. (LOG-4609) -
Currently, when using FluentD as the collector, the collector pod cannot start on the Red Hat OpenShift Service on AWS IPv6-enabled cluster. The pod logs produce the
fluentd pod [error]: unexpected error error_class=SocketError error="getaddrinfo: Name or service not known
error. There is currently no workaround for this issue. (LOG-4706) - Currently, the log alert is not available on an IPv6-enabled cluster. There is currently no workaround for this issue. (LOG-4709)
-
Currently,
must-gather
cannot gather any logs on a FIPS-enabled cluster, because the required OpenSSL library is not available in thecluster-logging-rhel9-operator
. There is currently no workaround for this issue. (LOG-4403) -
Currently, when deploying the logging version 5.8 on a FIPS-enabled cluster, the collector pods cannot start and are stuck in
CrashLoopBackOff
status, while using FluentD as a collector. There is currently no workaround for this issue. (LOG-3933)
1.2.5.4. CVEs
1.3. Logging 5.7
Logging is provided as an installable component, with a distinct release cycle from the core Red Hat OpenShift Service on AWS. The Red Hat OpenShift Container Platform Life Cycle Policy outlines release compatibility.
The stable channel only provides updates to the most recent release of logging. To continue receiving updates for prior releases, you must change your subscription channel to stable-x.y, where x.y
represents the major and minor version of logging you have installed. For example, stable-5.7.
1.3.1. Logging 5.7.8
This release includes OpenShift Logging Bug Fix Release 5.7.8.
1.3.1.1. Bug fixes
-
Before this update, there was a potential conflict when the same name was used for the
outputRefs
andinputRefs
parameters in theClusterLogForwarder
custom resource (CR). As a result, the collector pods entered in aCrashLoopBackOff
status. With this update, the output labels contain theOUTPUT_
prefix to ensure a distinction between output labels and pipeline names. (LOG-4383) -
Before this update, while configuring the JSON log parser, if you did not set the
structuredTypeKey
orstructuredTypeName
parameters for the Cluster Logging Operator, no alert would display about an invalid configuration. With this update, the Cluster Logging Operator informs you about the configuration issue. (LOG-4441) -
Before this update, if the
hecToken
key was missing or incorrect in the secret specified for a Splunk output, the validation failed because the Vector forwarded logs to Splunk without a token. With this update, if thehecToken
key is missing or incorrect, the validation fails with theA non-empty hecToken entry is required
error message. (LOG-4580) -
Before this update, selecting a date from the
Custom time range
for logs caused an error in the web console. With this update, you can select a date from the time range model in the web console successfully. (LOG-4684)
1.3.1.2. CVEs
1.3.2. Logging 5.7.7
This release includes OpenShift Logging Bug Fix Release 5.7.7.
1.3.2.1. Bug fixes
- Before this update, FluentD normalized the logs emitted by the EventRouter differently from Vector. With this update, the Vector produces log records in a consistent format. (LOG-4178)
- Before this update, there was an error in the query used for the FluentD Buffer Availability graph in the metrics dashboard created by the Cluster Logging Operator as it showed the minimum buffer usage. With this update, the graph shows the maximum buffer usage and is now renamed to FluentD Buffer Usage. (LOG-4555)
-
Before this update, deploying a LokiStack on IPv6-only or dual-stack Red Hat OpenShift Service on AWS clusters caused the LokiStack memberlist registration to fail. As a result, the distributor pods went into a crash loop. With this update, an administrator can enable IPv6 by setting the
lokistack.spec.hashRing.memberlist.enableIPv6:
value totrue
, which resolves the issue. (LOG-4569) - Before this update, the log collector relied on the default configuration settings for reading the container log lines. As a result, the log collector did not read the rotated files efficiently. With this update, there is an increase in the number of bytes read, which allows the log collector to efficiently process rotated files. (LOG-4575)
- Before this update, the unused metrics in the Event Router caused the container to fail due to excessive memory usage. With this update, there is reduction in the memory usage of the Event Router by removing the unused metrics. (LOG-4686)
1.3.2.2. CVEs
1.3.3. Logging 5.7.6
This release includes OpenShift Logging Bug Fix Release 5.7.6.
1.3.3.1. Bug fixes
- Before this update, the collector relied on the default configuration settings for reading the container log lines. As a result, the collector did not read the rotated files efficiently. With this update, there is an increase in the number of bytes read, which allows the collector to efficiently process rotated files. (LOG-4501)
- Before this update, when users pasted a URL with predefined filters, some filters did not reflect. With this update, the UI reflects all the filters in the URL. (LOG-4459)
- Before this update, forwarding to Loki using custom labels generated an error when switching from Fluentd to Vector. With this update, the Vector configuration sanitizes labels in the same way as Fluentd to ensure the collector starts and correctly processes labels. (LOG-4460)
- Before this update, the Observability Logs console search field did not accept special characters that it should escape. With this update, it is escaping special characters properly in the query. (LOG-4456)
-
Before this update, the following warning message appeared while sending logs to Splunk:
Timestamp was not found.
With this update, the change overrides the name of the log field used to retrieve the Timestamp and sends it to Splunk without warning. (LOG-4413) -
Before this update, the CPU and memory usage of Vector was increasing over time. With this update, the Vector configuration now contains the
expire_metrics_secs=60
setting to limit the lifetime of the metrics and cap the associated CPU usage and memory footprint. (LOG-4171) - Before this update, the LokiStack gateway cached authorized requests very broadly. As a result, this caused wrong authorization results. With this update, LokiStack gateway caches on a more fine-grained basis which resolves this issue. (LOG-4393)
- Before this update, the Fluentd runtime image included builder tools which were unnecessary at runtime. With this update, the builder tools are removed, resolving the issue. (LOG-4467)
1.3.3.2. CVEs
1.3.4. Logging 5.7.4
This release includes OpenShift Logging Bug Fix Release 5.7.4.
1.3.4.1. Bug fixes
-
Before this update, when forwarding logs to CloudWatch, a
namespaceUUID
value was not appended to thelogGroupName
field. With this update, thenamespaceUUID
value is included, so alogGroupName
in CloudWatch appears aslogGroupName: vectorcw.b443fb9e-bd4c-4b6a-b9d3-c0097f9ed286
. (LOG-2701) - Before this update, when forwarding logs over HTTP to an off-cluster destination, the Vector collector was unable to authenticate to the cluster-wide HTTP proxy even though correct credentials were provided in the proxy URL. With this update, the Vector log collector can now authenticate to the cluster-wide HTTP proxy. (LOG-3381)
- Before this update, the Operator would fail if the Fluentd collector was configured with Splunk as an output, due to this configuration being unsupported. With this update, configuration validation rejects unsupported outputs, resolving the issue. (LOG-4237)
-
Before this update, when the Vector collector was updated an
enabled = true
value in the TLS configuration for AWS Cloudwatch logs and the GCP Stackdriver caused a configuration error. With this update,enabled = true
value will be removed for these outputs, resolving the issue. (LOG-4242) -
Before this update, the Vector collector occasionally panicked with the following error message in its log:
thread 'vector-worker' panicked at 'all branches are disabled and there is no else branch', src/kubernetes/reflector.rs:26:9
. With this update, the error has been resolved. (LOG-4275) -
Before this update, an issue in the Loki Operator caused the
alert-manager
configuration for the application tenant to disappear if the Operator was configured with additional options for that tenant. With this update, the generated Loki configuration now contains both the custom and the auto-generated configuration. (LOG-4361) - Before this update, when multiple roles were used to authenticate using STS with AWS Cloudwatch forwarding, a recent update caused the credentials to be non-unique. With this update, multiple combinations of STS roles and static credentials can once again be used to authenticate with AWS Cloudwatch. (LOG-4368)
- Before this update, Loki filtered label values for active streams but did not remove duplicates, making Grafana’s Label Browser unusable. With this update, Loki filters out duplicate label values for active streams, resolving the issue. (LOG-4389)
-
Pipelines with no
name
field specified in theClusterLogForwarder
custom resource (CR) stopped working after upgrading to OpenShift Logging 5.7. With this update, the error has been resolved. (LOG-4120)
1.3.4.2. CVEs
1.3.5. Logging 5.7.3
This release includes OpenShift Logging Bug Fix Release 5.7.3.
1.3.5.1. Bug fixes
- Before this update, when viewing logs within the Red Hat OpenShift Service on AWS web console, cached files caused the data to not refresh. With this update the bootstrap files are not cached, resolving the issue. (LOG-4100)
- Before this update, the Loki Operator reset errors in a way that made identifying configuration problems difficult to troubleshoot. With this update, errors persist until the configuration error is resolved. (LOG-4156)
-
Before this update, the LokiStack ruler did not restart after changes were made to the
RulerConfig
custom resource (CR). With this update, the Loki Operator restarts the ruler pods after theRulerConfig
CR is updated. (LOG-4161) -
Before this update, the vector collector terminated unexpectedly when input match label values contained a
/
character within theClusterLogForwarder
. This update resolves the issue by quoting the match label, enabling the collector to start and collect logs. (LOG-4176) -
Before this update, the Loki Operator terminated unexpectedly when a
LokiStack
CR defined tenant limits, but not global limits. With this update, the Loki Operator can processLokiStack
CRs without global limits, resolving the issue. (LOG-4198) - Before this update, Fluentd did not send logs to an Elasticsearch cluster when the private key provided was passphrase-protected. With this update, Fluentd properly handles passphrase-protected private keys when establishing a connection with Elasticsearch. (LOG-4258)
-
Before this update, clusters with more than 8,000 namespaces caused Elasticsearch to reject queries because the list of namespaces was larger than the
http.max_header_size
setting. With this update, the default value for header size has been increased, resolving the issue. (LOG-4277) -
Before this update, label values containing a
/
character within theClusterLogForwarder
CR would cause the collector to terminate unexpectedly. With this update, slashes are replaced with underscores, resolving the issue. (LOG-4095) -
Before this update, the Cluster Logging Operator terminated unexpectedly when set to an unmanaged state. With this update, a check to ensure that the
ClusterLogging
resource is in the correct Management state before initiating the reconciliation of theClusterLogForwarder
CR, resolving the issue. (LOG-4177) - Before this update, when viewing logs within the Red Hat OpenShift Service on AWS web console, selecting a time range by dragging over the histogram did not work on the aggregated logs view inside the pod detail. With this update, the time range can be selected by dragging on the histogram in this view. (LOG-4108)
- Before this update, when viewing logs within the Red Hat OpenShift Service on AWS web console, queries longer than 30 seconds timed out. With this update, the timeout value can be configured in the configmap/logging-view-plugin. (LOG-3498)
- Before this update, when viewing logs within the Red Hat OpenShift Service on AWS web console, clicking the more data available option loaded more log entries only the first time it was clicked. With this update, more entries are loaded with each click. (OU-188)
- Before this update, when viewing logs within the Red Hat OpenShift Service on AWS web console, clicking the streaming option would only display the streaming logs message without showing the actual logs. With this update, both the message and the log stream are displayed correctly. (OU-166)
1.3.5.2. CVEs
1.3.6. Logging 5.7.2
This release includes OpenShift Logging Bug Fix Release 5.7.2.
1.3.6.1. Bug fixes
-
Before this update, it was not possible to delete the
openshift-logging
namespace directly due to the presence of a pending finalizer. With this update, the finalizer is no longer utilized, enabling direct deletion of the namespace. (LOG-3316) -
Before this update, the
run.sh
script would display an incorrectchunk_limit_size
value if it was changed according to the Red Hat OpenShift Service on AWS documentation. However, when setting thechunk_limit_size
via the environment variable$BUFFER_SIZE_LIMIT
, the script would show the correct value. With this update, therun.sh
script now consistently displays the correctchunk_limit_size
value in both scenarios. (LOG-3330) - Before this update, the Red Hat OpenShift Service on AWS web console’s logging view plugin did not allow for custom node placement or tolerations. This update adds the ability to define node placement and tolerations for the logging view plugin. (LOG-3749)
-
Before this update, the Cluster Logging Operator encountered an Unsupported Media Type exception when trying to send logs to DataDog via the Fluentd HTTP Plugin. With this update, users can seamlessly assign the content type for log forwarding by configuring the HTTP header Content-Type. The value provided is automatically assigned to the
content_type
parameter within the plugin, ensuring successful log transmission. (LOG-3784) -
Before this update, when the
detectMultilineErrors
field was set totrue
in theClusterLogForwarder
custom resource (CR), PHP multi-line errors were recorded as separate log entries, causing the stack trace to be split across multiple messages. With this update, multi-line error detection for PHP is enabled, ensuring that the entire stack trace is included in a single log message. (LOG-3878) -
Before this update,
ClusterLogForwarder
pipelines containing a space in their name caused the Vector collector pods to continuously crash. With this update, all spaces, dashes (-), and dots (.) in pipeline names are replaced with underscores (_). (LOG-3945) -
Before this update, the
log_forwarder_output
metric did not include thehttp
parameter. This update adds the missing parameter to the metric. (LOG-3997) - Before this update, Fluentd did not identify some multi-line JavaScript client exceptions when they ended with a colon. With this update, the Fluentd buffer name is prefixed with an underscore, resolving the issue. (LOG-4019)
- Before this update, when configuring log forwarding to write to a Kafka output topic which matched a key in the payload, logs dropped due to an error. With this update, Fluentd’s buffer name has been prefixed with an underscore, resolving the issue.(LOG-4027)
- Before this update, the LokiStack gateway returned label values for namespaces without applying the access rights of a user. With this update, the LokiStack gateway applies permissions to label value requests, resolving the issue. (LOG-4049)
Before this update, the Cluster Logging Operator API required a certificate to be provided by a secret when the
tls.insecureSkipVerify
option was set totrue
. With this update, the Cluster Logging Operator API no longer requires a certificate to be provided by a secret in such cases. The following configuration has been added to the Operator’s CR:tls.verify_certificate = false tls.verify_hostname = false
(LOG-3445)
-
Before this update, the LokiStack route configuration caused queries running longer than 30 seconds to timeout. With this update, the LokiStack global and per-tenant
queryTimeout
settings affect the route timeout settings, resolving the issue. (LOG-4052) -
Before this update, a prior fix to remove defaulting of the
collection.type
resulted in the Operator no longer honoring the deprecated specs for resource, node selections, and tolerations. This update modifies the Operator behavior to always prefer thecollection.logs
spec over those ofcollection
. This varies from previous behavior that allowed using both the preferred fields and deprecated fields but would ignore the deprecated fields whencollection.type
was populated. (LOG-4185) - Before this update, the Vector log collector did not generate TLS configuration for forwarding logs to multiple Kafka brokers if the broker URLs were not specified in the output. With this update, TLS configuration is generated appropriately for multiple brokers. (LOG-4163)
- Before this update, the option to enable passphrase for log forwarding to Kafka was unavailable. This limitation presented a security risk as it could potentially expose sensitive information. With this update, users now have a seamless option to enable passphrase for log forwarding to Kafka. (LOG-3314)
-
Before this update, Vector log collector did not honor the
tlsSecurityProfile
settings for outgoing TLS connections. After this update, Vector handles TLS connection settings appropriately. (LOG-4011) -
Before this update, not all available output types were included in the
log_forwarder_output_info
metrics. With this update, metrics contain Splunk and Google Cloud Logging data which was missing previously. (LOG-4098) -
Before this update, when
follow_inodes
was set totrue
, the Fluentd collector could crash on file rotation. With this update, thefollow_inodes
setting does not crash the collector. (LOG-4151) - Before this update, the Fluentd collector could incorrectly close files that should be watched because of how those files were tracked. With this update, the tracking parameters have been corrected. (LOG-4149)
Before this update, forwarding logs with the Vector collector and naming a pipeline in the
ClusterLogForwarder
instanceaudit
,application
orinfrastructure
resulted in collector pods staying in theCrashLoopBackOff
state with the following error in the collector log:ERROR vector::cli: Configuration error. error=redefinition of table transforms.audit for key transforms.audit
After this update, pipeline names no longer clash with reserved input names, and pipelines can be named
audit
,application
orinfrastructure
. (LOG-4218)-
Before this update, when forwarding logs to a syslog destination with the Vector collector and setting the
addLogSource
flag totrue
, the following extra empty fields were added to the forwarded messages:namespace_name=
,container_name=
, andpod_name=
. With this update, these fields are no longer added to journal logs. (LOG-4219) -
Before this update, when a
structuredTypeKey
was not found, and astructuredTypeName
was not specified, log messages were still parsed into structured object. With this update, parsing of logs is as expected. (LOG-4220)
1.3.6.2. CVEs
- CVE-2021-26341
- CVE-2021-33655
- CVE-2021-33656
- CVE-2022-1462
- CVE-2022-1679
- CVE-2022-1789
- CVE-2022-2196
- CVE-2022-2663
- CVE-2022-3028
- CVE-2022-3239
- CVE-2022-3522
- CVE-2022-3524
- CVE-2022-3564
- CVE-2022-3566
- CVE-2022-3567
- CVE-2022-3619
- CVE-2022-3623
- CVE-2022-3625
- CVE-2022-3627
- CVE-2022-3628
- CVE-2022-3707
- CVE-2022-3970
- CVE-2022-4129
- CVE-2022-20141
- CVE-2022-25147
- CVE-2022-25265
- CVE-2022-30594
- CVE-2022-36227
- CVE-2022-39188
- CVE-2022-39189
- CVE-2022-41218
- CVE-2022-41674
- CVE-2022-42703
- CVE-2022-42720
- CVE-2022-42721
- CVE-2022-42722
- CVE-2022-43750
- CVE-2022-47929
- CVE-2023-0394
- CVE-2023-0461
- CVE-2023-1195
- CVE-2023-1582
- CVE-2023-2491
- CVE-2023-22490
- CVE-2023-23454
- CVE-2023-23946
- CVE-2023-25652
- CVE-2023-25815
- CVE-2023-27535
- CVE-2023-29007
1.3.7. Logging 5.7.1
This release includes: OpenShift Logging Bug Fix Release 5.7.1.
1.3.7.1. Bug fixes
- Before this update, the presence of numerous noisy messages within the Cluster Logging Operator pod logs caused reduced log readability, and increased difficulty in identifying important system events. With this update, the issue is resolved by significantly reducing the noisy messages within Cluster Logging Operator pod logs. (LOG-3482)
-
Before this update, the API server would reset the value for the
CollectorSpec.Type
field tovector
, even when the custom resource used a different value. This update removes the default for theCollectorSpec.Type
field to restore the previous behavior. (LOG-4086) - Before this update, a time range could not be selected in the Red Hat OpenShift Service on AWS web console by clicking and dragging over the logs histogram. With this update, clicking and dragging can be used to successfully select a time range. (LOG-4501)
- Before this update, clicking on the Show Resources link in the Red Hat OpenShift Service on AWS web console did not produce any effect. With this update, the issue is resolved by fixing the functionality of the "Show Resources" link to toggle the display of resources for each log entry. (LOG-3218)
1.3.7.2. CVEs
1.3.8. Logging 5.7.0
This release includes OpenShift Logging Bug Fix Release 5.7.0.
1.3.8.1. Enhancements
With this update, you can enable logging to detect multi-line exceptions and reassemble them into a single log entry.
To enable logging to detect multi-line exceptions and reassemble them into a single log entry, ensure that the ClusterLogForwarder
Custom Resource (CR) contains a detectMultilineErrors
field, with a value of true
.
1.3.8.2. Known Issues
None.
1.3.8.3. Bug fixes
-
Before this update, the
nodeSelector
attribute for the Gateway component of the LokiStack did not impact node scheduling. With this update, thenodeSelector
attribute works as expected. (LOG-3713)