Chapter 11. Forwarding logs to external third-party logging systems

By default, the logging subsystem sends container and infrastructure logs to the default internal log store defined in the ClusterLogging custom resource. However, it does not send audit logs to the internal store because it does not provide secure storage. If this default configuration meets your needs, you do not need to configure the Cluster Log Forwarder.

To send logs to other log aggregators, you use the OpenShift Container Platform Cluster Log Forwarder. This API enables you to send container, infrastructure, and audit logs to specific endpoints within or outside your cluster. In addition, you can send different types of logs to various systems so that various individuals can access each type. You can also enable Transport Layer Security (TLS) support to send logs securely, as required by your organization.

Note

To send audit logs to the default internal Elasticsearch log store, use the Cluster Log Forwarder as described in Forward audit logs to the log store.

When you forward logs externally, the logging subsystem creates or modifies a Fluentd config map to send logs using your desired protocols. You are responsible for configuring the protocol on the external log aggregator.

11.1. About forwarding logs to third-party systems

To send logs to specific endpoints inside and outside your OpenShift Container Platform cluster, you specify a combination of outputs and pipelines in a ClusterLogForwarder custom resource (CR). You can also use inputs to forward the application logs associated with a specific project to an endpoint. Authentication is provided by a Kubernetes Secret object.

output

The destination for log data that you define, or where you want the logs sent. An output can be one of the following types:

elasticsearch. An external Elasticsearch instance. The elasticsearch output can use a TLS connection.
fluentdForward. An external log aggregation solution that supports Fluentd. This option uses the Fluentd forward protocols. The fluentForward output can use a TCP or TLS connection and supports shared-key authentication by providing a shared_key field in a secret. Shared-key authentication can be used with or without TLS.
syslog. An external log aggregation solution that supports the syslog RFC3164 or RFC5424 protocols. The syslog output can use a UDP, TCP, or TLS connection.
cloudwatch. Amazon CloudWatch, a monitoring and log storage service hosted by Amazon Web Services (AWS).
loki. Loki, a horizontally scalable, highly available, multi-tenant log aggregation system.
kafka. A Kafka broker. The kafka output can use a TCP or TLS connection.
default. The internal OpenShift Container Platform Elasticsearch instance. You are not required to configure the default output. If you do configure a default output, you receive an error message because the default output is reserved for the Red Hat OpenShift Logging Operator.

pipeline

Defines simple routing from one log type to one or more outputs, or which logs you want to send. The log types are one of the following:

application. Container logs generated by user applications running in the cluster, except infrastructure container applications.
infrastructure. Container logs from pods that run in the openshift*, kube*, or default projects and journal logs sourced from node file system.
audit. Audit logs generated by the node audit system, auditd, Kubernetes API server, OpenShift API server, and OVN network.

You can add labels to outbound log messages by using key:value pairs in the pipeline. For example, you might add a label to messages that are forwarded to other data centers or label the logs by type. Labels that are added to objects are also forwarded with the log message.

input

Forwards the application logs associated with a specific project to a pipeline.

In the pipeline, you define which log types to forward using an inputRef parameter and where to forward the logs to using an outputRef parameter.

Secret

A key:value map that contains confidential data such as user credentials.

Note the following:

If a ClusterLogForwarder CR object exists, logs are not forwarded to the default Elasticsearch instance, unless there is a pipeline with the default output.
By default, the logging subsystem sends container and infrastructure logs to the default internal Elasticsearch log store defined in the ClusterLogging custom resource. However, it does not send audit logs to the internal store because it does not provide secure storage. If this default configuration meets your needs, do not configure the Log Forwarding API.
If you do not define a pipeline for a log type, the logs of the undefined types are dropped. For example, if you specify a pipeline for the application and audit types, but do not specify a pipeline for the infrastructure type, infrastructure logs are dropped.
You can use multiple types of outputs in the ClusterLogForwarder custom resource (CR) to send logs to servers that support different protocols.
The internal OpenShift Container Platform Elasticsearch instance does not provide secure storage for audit logs. We recommend you ensure that the system to which you forward audit logs is compliant with your organizational and governmental regulations and is properly secured. The logging subsystem does not comply with those regulations.

The following example forwards the audit logs to a secure external Elasticsearch instance, the infrastructure logs to an insecure external Elasticsearch instance, the application logs to a Kafka broker, and the application logs from the my-apps-logs project to the internal Elasticsearch instance.

Sample log forwarding outputs and pipelines

apiVersion: "logging.openshift.io/v1"
kind: ClusterLogForwarder
metadata:
  name: instance 1
  namespace: openshift-logging 2
spec:
  outputs:
   - name: elasticsearch-secure 3
     type: "elasticsearch"
     url: https://elasticsearch.secure.com:9200
     secret:
        name: elasticsearch
   - name: elasticsearch-insecure 4
     type: "elasticsearch"
     url: http://elasticsearch.insecure.com:9200
   - name: kafka-app 5
     type: "kafka"
     url: tls://kafka.secure.com:9093/app-topic
  inputs: 6
   - name: my-app-logs
     application:
        namespaces:
        - my-project
  pipelines:
   - name: audit-logs 7
     inputRefs:
      - audit
     outputRefs:
      - elasticsearch-secure
      - default
     parse: json 8
     labels:
       secure: "true" 9
       datacenter: "east"
   - name: infrastructure-logs 10
     inputRefs:
      - infrastructure
     outputRefs:
      - elasticsearch-insecure
     labels:
       datacenter: "west"
   - name: my-app 11
     inputRefs:
      - my-app-logs
     outputRefs:
      - default
   - inputRefs: 12
      - application
     outputRefs:
      - kafka-app
     labels:
       datacenter: "south"

1

The name of the ClusterLogForwarder CR must be instance.

2

The namespace for the ClusterLogForwarder CR must be openshift-logging.

3

Configuration for an secure Elasticsearch output using a secret with a secure URL.

A name to describe the output.
The type of output: elasticsearch.
The secure URL and port of the Elasticsearch instance as a valid absolute URL, including the prefix.
The secret required by the endpoint for TLS communication. The secret must exist in the openshift-logging project.

4

Configuration for an insecure Elasticsearch output:

A name to describe the output.
The type of output: elasticsearch.
The insecure URL and port of the Elasticsearch instance as a valid absolute URL, including the prefix.

5

Configuration for a Kafka output using a client-authenticated TLS communication over a secure URL

A name to describe the output.
The type of output: kafka.
Specify the URL and port of the Kafka broker as a valid absolute URL, including the prefix.

6

Configuration for an input to filter application logs from the my-project namespace.

7

Configuration for a pipeline to send audit logs to the secure external Elasticsearch instance:

A name to describe the pipeline.
The inputRefs is the log type, in this example audit.
The outputRefs is the name of the output to use, in this example elasticsearch-secure to forward to the secure Elasticsearch instance and default to forward to the internal Elasticsearch instance.
Optional: Labels to add to the logs.

8

Optional: Specify whether to forward structured JSON log entries as JSON objects in the structured field. The log entry must contain valid structured JSON; otherwise, OpenShift Logging removes the structured field and instead sends the log entry to the default index, app-00000x.

9

Optional: String. One or more labels to add to the logs. Quote values like "true" so they are recognized as string values, not as a boolean.

10

Configuration for a pipeline to send infrastructure logs to the insecure external Elasticsearch instance.

11

Configuration for a pipeline to send logs from the my-project project to the internal Elasticsearch instance.

A name to describe the pipeline.
The inputRefs is a specific input: my-app-logs.
The outputRefs is default.
Optional: String. One or more labels to add to the logs.

12

Configuration for a pipeline to send logs to the Kafka broker, with no pipeline name:

The inputRefs is the log type, in this example application.
The outputRefs is the name of the output to use.
Optional: String. One or more labels to add to the logs.

Fluentd log handling when the external log aggregator is unavailable

If your external logging aggregator becomes unavailable and cannot receive logs, Fluentd continues to collect logs and stores them in a buffer. When the log aggregator becomes available, log forwarding resumes, including the buffered logs. If the buffer fills completely, Fluentd stops collecting logs. OpenShift Container Platform rotates the logs and deletes them. You cannot adjust the buffer size or add a persistent volume claim (PVC) to the Fluentd daemon set or pods.

Supported Authorization Keys

Common key types are provided here. Some output types support additional specialized keys, documented with the output-specific configuration field. All secret keys are optional. Enable the security features you want by setting the relevant keys. You are responsible for creating and maintaining any additional configurations that external destinations might require, such as keys and secrets, service accounts, port openings, or global proxy configuration. Open Shift Logging will not attempt to verify a mismatch between authorization combinations.

Transport Layer Security (TLS)

Using a TLS URL ('http://…' or 'ssl://…') without a Secret enables basic TLS server-side authentication. Additional TLS features are enabled by including a Secret and setting the following optional fields:

tls.crt: (string) File name containing a client certificate. Enables mutual authentication. Requires tls.key.
tls.key: (string) File name containing the private key to unlock the client certificate. Requires tls.crt.
passphrase: (string) Passphrase to decode an encoded TLS private key. Requires tls.key.
ca-bundle.crt: (string) File name of a customer CA for server authentication.

Username and Password

username: (string) Authentication user name. Requires password.
password: (string) Authentication password. Requires username.

Simple Authentication Security Layer (SASL)

sasl.enable (boolean) Explicitly enable or disable SASL. If missing, SASL is automatically enabled when any of the other sasl. keys are set.
sasl.mechanisms: (array) List of allowed SASL mechanism names. If missing or empty, the system defaults are used.
sasl.allow-insecure: (boolean) Allow mechanisms that send clear-text passwords. Defaults to false.

11.1.1. Creating a Secret

You can create a secret in the directory that contains your certificate and key files by using the following command:

$ oc create secret generic -n openshift-logging <my-secret> \
 --from-file=tls.key=<your_key_file>
 --from-file=tls.crt=<your_crt_file>
 --from-file=ca-bundle.crt=<your_bundle_file>
 --from-literal=username=<your_username>
 --from-literal=password=<your_password>

Note

Generic or opaque secrets are recommended for best results.

11.2. Forwarding JSON logs from containers in the same pod to separate indices

You can forward structured logs from different containers within the same pod to different indices. To use this feature, you must configure the pipeline with multi-container support and annotate the pods. Logs are written to indices with a prefix of app-. It is recommended that Elasticsearch be configured with aliases to accommodate this.

Important

JSON formatting of logs varies by application. Because creating too many indices impacts performance, limit your use of this feature to creating indices for logs that have incompatible JSON formats. Use queries to separate logs from different namespaces, or applications with compatible JSON formats.

Prerequisites

Logging subsystem for Red Hat OpenShift: 5.5

Procedure

Create or edit a YAML file that defines the ClusterLogForwarder CR object:

apiVersion: "logging.openshift.io/v1"
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  outputDefaults:
    elasticsearch:
      enableStructuredContainerLogs: true 1
  pipelines:
  - inputRefs:
    - application
    name: application-logs
    outputRefs:
    - default
    parse: json

1: Enables multi-container outputs.

Create or edit a YAML file that defines the Pod CR object:

    apiVersion: v1
    kind: Pod
    metadata:
      annotations:
        containerType.logging.openshift.io/heavy: heavy 1
        containerType.logging.openshift.io/low: low
    spec:
      containers:
      - name: heavy 2
        image: heavyimage
      - name: low
        image: lowimage

1: Format: containerType.logging.openshift.io/<container-name>: <index>
2: Annotation names must match container names

Warning

This configuration might significantly increase the number of shards on the cluster.

Additional Resources

Kubernetes Annotations

11.3. Supported log data output types in OpenShift Logging 5.1

Red Hat OpenShift Logging 5.1 provides the following output types and protocols for sending log data to target log collectors.