Chapter 9. Log collection and forwarding

1. Fluentd does not support Elasticsearch 8 in the logging version 5.6.2.
2. Vector supports Syslog in the logging version 5.7 and higher.

Procedure

1. Add the following snippet to your ClusterLogForwarder CR YAML file.

   outputDefaults:
    elasticsearch:
       structuredTypeKey: <log record field>
       structuredTypeName: <name>
   pipelines:
   - inputRefs:
     - application
     outputRefs: default
     parse: json

2. Use structuredTypeKey field to specify one of the log record fields.

3. Use structuredTypeName field to specify a name.

   Important

   To parse JSON logs, you must set both the structuredTypeKey and structuredTypeName fields.

4. For inputRefs, specify which log types to forward by using that pipeline, such as application, infrastructure, or audit.
5. Add the parse: json element to pipelines.

6. Create the CR object:

   $ oc create -f <filename>.yaml

   The Red Hat OpenShift Logging Operator redeploys the collector pods. However, if they do not redeploy, delete the collector pods to force them to redeploy.

   $ oc delete pod --selector logging-infra=collector

Procedure

1. Create or edit a YAML file that defines the ClusterLogForwarder CR object:

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: instance
     namespace: openshift-logging
   spec:
     outputDefaults:
       elasticsearch:
         structuredTypeKey: kubernetes.labels.logFormat 1
         structuredTypeName: nologformat
         enableStructuredContainerLogs: true 2
     pipelines:
     - inputRefs:
       - application
       name: application-logs
       outputRefs:
       - default
       parse: json

   1

   Uses the value of the key-value pair that is formed by the Kubernetes logFormat label.

   2

   Enables multi-container outputs.

2. Create or edit a YAML file that defines the Pod CR object:

   apiVersion: v1
   kind: Pod
   metadata:
     annotations:
       containerType.logging.openshift.io/heavy: heavy 1
       containerType.logging.openshift.io/low: low
   spec:
     containers:
     - name: heavy 2
       image: heavyimage
     - name: low
       image: lowimage

   1

   Format: containerType.logging.openshift.io/<container-name>: <index>

   2

   Annotation names must match container names

Procedure

1. Create a secret using your Base64 encoded Splunk HEC token.

   $ oc -n openshift-logging create secret generic vector-splunk-secret --from-literal hecToken=<HEC_Token>

2. Create or edit the ClusterLogForwarder Custom Resource (CR) using the template below:

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: <log_forwarder_name> 1
     namespace: <log_forwarder_namespace> 2
   spec:
     serviceAccountName: <service_account_name> 3
     outputs:
       - name: splunk-receiver 4
         secret:
           name: vector-splunk-secret 5
         type: splunk 6
         url: <http://your.splunk.hec.url:8088> 7
     pipelines: 8
       - inputRefs:
           - application
           - infrastructure
         name: 9
         outputRefs:
           - splunk-receiver 10

   1

   In legacy implementations, the CR name must be instance. In multi log forwarder implementations, you can use any name.

   2

   In legacy implementations, the CR namespace must be openshift-logging. In multi log forwarder implementations, you can use any namespace.

   3

   The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the openshift-logging namespace.

   4

   Specify a name for the output.

   5

   Specify the name of the secret that contains your HEC token.

   6

   Specify the output type as splunk.

   7

   Specify the URL (including port) of your Splunk HEC.

   8

   Specify which log types to forward by using the pipeline: application, infrastructure, or audit.

   9

   Optional: Specify a name for the pipeline.

   10

   Specify the name of the output to use when forwarding logs with this pipeline.

Procedure

1. Create or edit a YAML file that defines the ClusterLogForwarder CR:

   Example ClusterLogForwarder CR

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: instance 1
     namespace: openshift-logging 2
   spec:
     outputs:
      - name: fluentd-server-secure 3
        type: fluentdForward 4
        url: 'tls://fluentdserver.security.example.com:24224' 5
        secret: 6
           name: fluentd-secret
      - name: fluentd-server-insecure
        type: fluentdForward
        url: 'tcp://fluentdserver.home.example.com:24224'
     inputs: 7
      - name: my-app-logs
        application:
           namespaces:
           - my-project 8
     pipelines:
      - name: forward-to-fluentd-insecure 9
        inputRefs: 10
        - my-app-logs
        outputRefs: 11
        - fluentd-server-insecure
        labels:
          project: "my-project" 12
      - name: forward-to-fluentd-secure 13
        inputRefs:
        - application 14
        - audit
        - infrastructure
        outputRefs:
        - fluentd-server-secure
        - default
        labels:
          clusterId: "C1234"

   1

   The name of the ClusterLogForwarder CR must be instance.

   2

   The namespace for the ClusterLogForwarder CR must be openshift-logging.

   3

   The name of the output.

   4

   The output type: elasticsearch, fluentdForward, syslog, or kafka.

   5

   The URL and port of the external log aggregator as a valid absolute URL. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP address.

   6

   If using a tls prefix, you must specify the name of the secret required by the endpoint for TLS communication. The secret must exist in the openshift-logging project and have tls.crt, tls.key, and ca-bundle.crt keys that each point to the certificates they represent.

   7

   The configuration for an input to filter application logs from the specified projects.

   8

   If no namespace is specified, logs are collected from all namespaces.

   9

   The pipeline configuration directs logs from a named input to a named output. In this example, a pipeline named forward-to-fluentd-insecure forwards logs from an input named my-app-logs to an output named fluentd-server-insecure.

   10

   A list of inputs.

   11

   The name of the output to use.

   12

   Optional: String. One or more labels to add to the logs.

   13

   Configuration for a pipeline to send logs to other log aggregators.

   • Optional: Specify a name for the pipeline.
   • Specify which log types to forward by using the pipeline: application, infrastructure, or audit.
   • Specify the name of the output to use when forwarding logs with this pipeline.
   • Optional: Specify the default output to forward logs to the default log store.
   • Optional: String. One or more labels to add to the logs.

   14

   Note that application logs from all namespaces are collected when using this configuration.

2. Apply the ClusterLogForwarder CR by running the following command:

   $ oc apply -f <filename>.yaml

Procedure

1. Create or edit a YAML file that defines the ClusterLogForwarder CR object. In the file, specify the pod labels using simple equality-based selectors under inputs[].name.application.selector.matchLabels, as shown in the following example.

   Example ClusterLogForwarder CR YAML file

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: <log_forwarder_name> 1
     namespace: <log_forwarder_namespace> 2
   spec:
     pipelines:
       - inputRefs: [ myAppLogData ] 3
         outputRefs: [ default ] 4
     inputs: 5
       - name: myAppLogData
         application:
           selector:
             matchLabels: 6
               environment: production
               app: nginx
           namespaces: 7
           - app1
           - app2
     outputs: 8
       - <output_name>
       ...

   1

   In legacy implementations, the CR name must be instance. In multi log forwarder implementations, you can use any name.

   2

   In legacy implementations, the CR namespace must be openshift-logging. In multi log forwarder implementations, you can use any namespace.

   3

   Specify one or more comma-separated values from inputs[].name.

   4

   Specify one or more comma-separated values from outputs[].

   5

   Define a unique inputs[].name for each application that has a unique set of pod labels.

   6

   Specify the key-value pairs of pod labels whose log data you want to gather. You must specify both a key and value, not just a key. To be selected, the pods must match all the key-value pairs.

   7

   Optional: Specify one or more namespaces.

   8

   Specify one or more outputs to forward your log data to.

2. Optional: To restrict the gathering of log data to specific namespaces, use inputs[].name.application.namespaces, as shown in the preceding example.

3. Optional: You can send log data from additional applications that have different pod labels to the same pipeline.

   1. For each unique combination of pod labels, create an additional inputs[].name section similar to the one shown.
   2. Update the selectors to match the pod labels of this application.

   3. Add the new inputs[].name value to inputRefs. For example:

      - inputRefs: [ myAppLogData, myOtherAppLogData ]

4. Create the CR object:

   $ oc create -f <file-name>.yaml

Procedure

1. Create or edit a YAML file that defines the ClusterLogForwarder CR object:

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: <log_forwarder_name> 1
     namespace: <log_forwarder_namespace> 2
   spec:
     serviceAccountName: <service_account_name> 3
     outputs:
     - name: loki-insecure 4
       type: "loki" 5
       url: http://loki.insecure.com:3100 6
       loki:
         tenantKey: kubernetes.namespace_name
         labelKeys:
         - kubernetes.labels.foo
     - name: loki-secure 7
       type: "loki"
       url: https://loki.secure.com:3100
       secret:
         name: loki-secret 8
       loki:
         tenantKey: kubernetes.namespace_name 9
         labelKeys:
         - kubernetes.labels.foo 10
     pipelines:
     - name: application-logs 11
       inputRefs:  12
       - application
       - audit
       outputRefs: 13
       - loki-secure

   1

   In legacy implementations, the CR name must be instance. In multi log forwarder implementations, you can use any name.

   2

   In legacy implementations, the CR namespace must be openshift-logging. In multi log forwarder implementations, you can use any namespace.

   3

   The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the openshift-logging namespace.

   4

   Specify a name for the output.

   5

   Specify the type as "loki".

   6

   Specify the URL and port of the Loki system as a valid absolute URL. You can use the http (insecure) or https (secure HTTP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP Address. Loki’s default port for HTTP(S) communication is 3100.

   7

   For a secure connection, you can specify an https or http URL that you authenticate by specifying a secret.

   8

   For an https prefix, specify the name of the secret required by the endpoint for TLS communication. The secret must contain a ca-bundle.crt key that points to the certificates it represents. Otherwise, for http and https prefixes, you can specify a secret that contains a username and password. In legacy implementations, the secret must exist in the openshift-logging project. For more information, see the following "Example: Setting a secret that contains a username and password."

   9

   Optional: Specify a metadata key field to generate values for the TenantID field in Loki. For example, setting tenantKey: kubernetes.namespace_name uses the names of the Kubernetes namespaces as values for tenant IDs in Loki. To see which other log record fields you can specify, see the "Log Record Fields" link in the following "Additional resources" section.

   10

   Optional: Specify a list of metadata field keys to replace the default Loki labels. Loki label names must match the regular expression [a-zA-Z_:][a-zA-Z0-9_:]*. Illegal characters in metadata keys are replaced with _ to form the label name. For example, the kubernetes.labels.foo metadata key becomes Loki label kubernetes_labels_foo. If you do not set labelKeys, the default value is: [log_type, kubernetes.namespace_name, kubernetes.pod_name, kubernetes_host]. Keep the set of labels small because Loki limits the size and number of labels allowed. See Configuring Loki, limits_config. You can still query based on any log record field using query filters.

   11

   Optional: Specify a name for the pipeline.

   12

   Specify which log types to forward by using the pipeline: application, infrastructure, or audit.

   13

   Specify the name of the output to use when forwarding logs with this pipeline.

   Note

   Because Loki requires log streams to be correctly ordered by timestamp, labelKeys always includes the kubernetes_host label set, even if you do not specify it. This inclusion ensures that each stream originates from a single host, which prevents timestamps from becoming disordered due to clock differences on different hosts.

2. Apply the ClusterLogForwarder CR object by running the following command:

   $ oc apply -f <filename>.yaml

Procedure

1. Create or edit a YAML file that defines the ClusterLogForwarder CR:

   Example ClusterLogForwarder CR

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: <log_forwarder_name> 1
     namespace: <log_forwarder_namespace> 2
   spec:
     serviceAccountName: <service_account_name> 3
     outputs:
      - name: elasticsearch-example 4
        type: elasticsearch 5
        elasticsearch:
          version: 8 6
        url: http://elasticsearch.example.com:9200 7
        secret:
          name: es-secret 8
     pipelines:
      - name: application-logs 9
        inputRefs: 10
        - application
        - audit
        outputRefs:
        - elasticsearch-example 11
        - default 12
        labels:
          myLabel: "myValue" 13
   # ...

   1

   In legacy implementations, the CR name must be instance. In multi log forwarder implementations, you can use any name.

   2

   In legacy implementations, the CR namespace must be openshift-logging. In multi log forwarder implementations, you can use any namespace.

   3

   The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the openshift-logging namespace.

   4

   Specify a name for the output.

   5

   Specify the elasticsearch type.

   6

   Specify the Elasticsearch version. This can be 6, 7, or 8.

   7

   Specify the URL and port of the external Elasticsearch instance as a valid absolute URL. You can use the http (insecure) or https (secure HTTP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP Address.

   8

   For an https prefix, specify the name of the secret required by the endpoint for TLS communication. The secret must contain a ca-bundle.crt key that points to the certificate it represents. Otherwise, for http and https prefixes, you can specify a secret that contains a username and password. In legacy implementations, the secret must exist in the openshift-logging project. For more information, see the following "Example: Setting a secret that contains a username and password."

   9

   Optional: Specify a name for the pipeline.

   10

   Specify which log types to forward by using the pipeline: application, infrastructure, or audit.

   11

   Specify the name of the output to use when forwarding logs with this pipeline.

   12

   Optional: Specify the default output to send the logs to the internal Elasticsearch instance.

   13

   Optional: String. One or more labels to add to the logs.

2. Apply the ClusterLogForwarder CR:

   $ oc apply -f <filename>.yaml

1. Create a Secret YAML file similar to the following example. Use base64-encoded values for the username and password fields. The secret type is opaque by default.

   apiVersion: v1
   kind: Secret
   metadata:
     name: openshift-test-secret
   data:
     username: <username>
     password: <password>
   # ...

2. Create the secret:

   $ oc create secret -n openshift-logging openshift-test-secret.yaml

3. Specify the name of the secret in the ClusterLogForwarder CR:

   kind: ClusterLogForwarder
   metadata:
     name: instance
     namespace: openshift-logging
   spec:
     outputs:
      - name: elasticsearch
        type: "elasticsearch"
        url: https://elasticsearch.secure.com:9200
        secret:
           name: openshift-test-secret
   # ...

   Note

   In the value of the url field, the prefix can be http or https.

4. Apply the CR object:

   $ oc apply -f <filename>.yaml

Procedure

1. Create or edit a YAML file that defines the ClusterLogForwarder CR object:

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: instance 1
     namespace: openshift-logging 2
   spec:
     outputs:
      - name: fluentd-server-secure 3
        type: fluentdForward 4
        url: 'tls://fluentdserver.security.example.com:24224' 5
        secret: 6
           name: fluentd-secret
      - name: fluentd-server-insecure
        type: fluentdForward
        url: 'tcp://fluentdserver.home.example.com:24224'
     pipelines:
      - name: forward-to-fluentd-secure 7
        inputRefs:  8
        - application
        - audit
        outputRefs:
        - fluentd-server-secure 9
        - default 10
        labels:
          clusterId: "C1234" 11
      - name: forward-to-fluentd-insecure 12
        inputRefs:
        - infrastructure
        outputRefs:
        - fluentd-server-insecure
        labels:
          clusterId: "C1234"

   1

   The name of the ClusterLogForwarder CR must be instance.

   2

   The namespace for the ClusterLogForwarder CR must be openshift-logging.

   3

   Specify a name for the output.

   4

   Specify the fluentdForward type.

   5

   Specify the URL and port of the external Fluentd instance as a valid absolute URL. You can use the tcp (insecure) or tls (secure TCP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP address.

   6

   If you are using a tls prefix, you must specify the name of the secret required by the endpoint for TLS communication. The secret must exist in the openshift-logging project and must contain a ca-bundle.crt key that points to the certificate it represents.

   7

   Optional: Specify a name for the pipeline.

   8

   Specify which log types to forward by using the pipeline: application, infrastructure, or audit.

   9

   Specify the name of the output to use when forwarding logs with this pipeline.

   10

   Optional: Specify the default output to forward logs to the internal Elasticsearch instance.

   11

   Optional: String. One or more labels to add to the logs.

   12

   Optional: Configure multiple outputs to forward logs to other external log aggregators of any supported type:

   • A name to describe the pipeline.
   • The inputRefs is the log type to forward by using the pipeline: application, infrastructure, or audit.
   • The outputRefs is the name of the output to use.
   • Optional: String. One or more labels to add to the logs.

2. Create the CR object:

   $ oc create -f <file-name>.yaml

Procedure

1. Create or edit a YAML file that defines the ClusterLogForwarder CR object:

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: <log_forwarder_name> 1
     namespace: <log_forwarder_namespace> 2
   spec:
     serviceAccountName: <service_account_name> 3
     outputs:
      - name: rsyslog-east 4
        type: syslog 5
        syslog: 6
          facility: local0
          rfc: RFC3164
          payloadKey: message
          severity: informational
        url: 'tls://rsyslogserver.east.example.com:514' 7
        secret: 8
           name: syslog-secret
      - name: rsyslog-west
        type: syslog
        syslog:
         appName: myapp
         facility: user
         msgID: mymsg
         procID: myproc
         rfc: RFC5424
         severity: debug
        url: 'tcp://rsyslogserver.west.example.com:514'
     pipelines:
      - name: syslog-east 9
        inputRefs: 10
        - audit
        - application
        outputRefs: 11
        - rsyslog-east
        - default 12
        labels:
          secure: "true" 13
          syslog: "east"
      - name: syslog-west 14
        inputRefs:
        - infrastructure
        outputRefs:
        - rsyslog-west
        - default
        labels:
          syslog: "west"

   1

   In legacy implementations, the CR name must be instance. In multi log forwarder implementations, you can use any name.

   2

   In legacy implementations, the CR namespace must be openshift-logging. In multi log forwarder implementations, you can use any namespace.

   3

   The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the openshift-logging namespace.

   4

   Specify a name for the output.

   5

   Specify the syslog type.

   6

   Optional: Specify the syslog parameters, listed below.

   7

   Specify the URL and port of the external syslog instance. You can use the udp (insecure), tcp (insecure) or tls (secure TCP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP address.

   8

   If using a tls prefix, you must specify the name of the secret required by the endpoint for TLS communication. The secret must contain a ca-bundle.crt key that points to the certificate it represents. In legacy implementations, the secret must exist in the openshift-logging project.

   9

   Optional: Specify a name for the pipeline.

   10

   Specify which log types to forward by using the pipeline: application, infrastructure, or audit.

   11

   Specify the name of the output to use when forwarding logs with this pipeline.

   12

   Optional: Specify the default output to forward logs to the internal Elasticsearch instance.

   13

   Optional: String. One or more labels to add to the logs. Quote values like "true" so they are recognized as string values, not as a boolean.

   14

   Optional: Configure multiple outputs to forward logs to other external log aggregators of any supported type:

   • A name to describe the pipeline.
   • The inputRefs is the log type to forward by using the pipeline: application, infrastructure, or audit.
   • The outputRefs is the name of the output to use.
   • Optional: String. One or more labels to add to the logs.

2. Create the CR object:

   $ oc create -f <filename>.yaml

Procedure

1. Create or edit a YAML file that defines the ClusterLogForwarder CR object:

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: <log_forwarder_name> 1
     namespace: <log_forwarder_namespace> 2
   spec:
     serviceAccountName: <service_account_name> 3
     outputs:
      - name: app-logs 4
        type: kafka 5
        url: tls://kafka.example.devlab.com:9093/app-topic 6
        secret:
          name: kafka-secret 7
      - name: infra-logs
        type: kafka
        url: tcp://kafka.devlab2.example.com:9093/infra-topic 8
      - name: audit-logs
        type: kafka
        url: tls://kafka.qelab.example.com:9093/audit-topic
        secret:
           name: kafka-secret-qe
     pipelines:
      - name: app-topic 9
        inputRefs: 10
        - application
        outputRefs: 11
        - app-logs
        labels:
          logType: "application" 12
      - name: infra-topic 13
        inputRefs:
        - infrastructure
        outputRefs:
        - infra-logs
        labels:
          logType: "infra"
      - name: audit-topic
        inputRefs:
        - audit
        outputRefs:
        - audit-logs
        labels:
          logType: "audit"

   1

   In legacy implementations, the CR name must be instance. In multi log forwarder implementations, you can use any name.

   2

   In legacy implementations, the CR namespace must be openshift-logging. In multi log forwarder implementations, you can use any namespace.

   3

   The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the openshift-logging namespace.

   4

   Specify a name for the output.

   5

   Specify the kafka type.

   6

   Specify the URL and port of the Kafka broker as a valid absolute URL, optionally with a specific topic. You can use the tcp (insecure) or tls (secure TCP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP address.

   7

   If you are using a tls prefix, you must specify the name of the secret required by the endpoint for TLS communication. The secret must contain a ca-bundle.crt key that points to the certificate it represents. In legacy implementations, the secret must exist in the openshift-logging project.

   8

   Optional: To send an insecure output, use a tcp prefix in front of the URL. Also omit the secret key and its name from this output.

   9

   Optional: Specify a name for the pipeline.

   10

   Specify which log types to forward by using the pipeline: application, infrastructure, or audit.

   11

   Specify the name of the output to use when forwarding logs with this pipeline.

   12

   Optional: String. One or more labels to add to the logs.

   13

   Optional: Configure multiple outputs to forward logs to other external log aggregators of any supported type:

   • A name to describe the pipeline.
   • The inputRefs is the log type to forward by using the pipeline: application, infrastructure, or audit.
   • The outputRefs is the name of the output to use.
   • Optional: String. One or more labels to add to the logs.

2. Optional: To forward a single output to multiple Kafka brokers, specify an array of Kafka brokers as shown in the following example:

   # ...
   spec:
     outputs:
     - name: app-logs
       type: kafka
       secret:
         name: kafka-secret-dev
       kafka:  1
         brokers: 2
           - tls://kafka-broker1.example.com:9093/
           - tls://kafka-broker2.example.com:9093/
         topic: app-topic 3
   # ...

   1

   Specify a kafka key that has a brokers and topic key.

   2

   Use the brokers key to specify an array of one or more brokers.

   3

   Use the topic key to specify the target topic that receives the logs.

3. Apply the ClusterLogForwarder CR by running the following command:

   $ oc apply -f <filename>.yaml

Procedure

1. Create a Secret YAML file that uses the aws_access_key_id and aws_secret_access_key fields to specify your base64-encoded AWS credentials. For example:

   apiVersion: v1
   kind: Secret
   metadata:
     name: cw-secret
     namespace: openshift-logging
   data:
     aws_access_key_id: QUtJQUlPU0ZPRE5ON0VYQU1QTEUK
     aws_secret_access_key: d0phbHJYVXRuRkVNSS9LN01ERU5HL2JQeFJmaUNZRVhBTVBMRUtFWQo=

2. Create the secret. For example:

   $ oc apply -f cw-secret.yaml

3. Create or edit a YAML file that defines the ClusterLogForwarder CR object. In the file, specify the name of the secret. For example:

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: <log_forwarder_name> 1
     namespace: <log_forwarder_namespace> 2
   spec:
     serviceAccountName: <service_account_name> 3
     outputs:
      - name: cw 4
        type: cloudwatch 5
        cloudwatch:
          groupBy: logType 6
          groupPrefix: <group prefix> 7
          region: us-east-2 8
        secret:
           name: cw-secret 9
     pipelines:
       - name: infra-logs 10
         inputRefs: 11
           - infrastructure
           - audit
           - application
         outputRefs:
           - cw 12

   1

   In legacy implementations, the CR name must be instance. In multi log forwarder implementations, you can use any name.

   2

   In legacy implementations, the CR namespace must be openshift-logging. In multi log forwarder implementations, you can use any namespace.

   3

   The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the openshift-logging namespace.

   4

   Specify a name for the output.

   5

   Specify the cloudwatch type.

   6

   Optional: Specify how to group the logs:

   • logType creates log groups for each log type.
   • namespaceName creates a log group for each application name space. It also creates separate log groups for infrastructure and audit logs.
   • namespaceUUID creates a new log groups for each application namespace UUID. It also creates separate log groups for infrastructure and audit logs.

   7

   Optional: Specify a string to replace the default infrastructureName prefix in the names of the log groups.

   8

   Specify the AWS region.

   9

   Specify the name of the secret that contains your AWS credentials.

   10

   Optional: Specify a name for the pipeline.

   11

   Specify which log types to forward by using the pipeline: application, infrastructure, or audit.

   12

   Specify the name of the output to use when forwarding logs with this pipeline.

4. Create the CR object:

   $ oc create -f <file-name>.yaml

Procedure

1. Prepare the AWS account:

   1. Create an IAM policy JSON file with the following content:

      {
      "Version": "2012-10-17",
      "Statement": [
          {
          "Effect": "Allow",
          "Action": [
            "logs:CreateLogGroup",
            "logs:CreateLogStream",
            "logs:DescribeLogGroups",
            "logs:DescribeLogStreams",
            "logs:PutLogEvents",
            "logs:PutRetentionPolicy"
          ],
          "Resource": "arn:aws:logs:*:*:*"
          }
        ]
      }

   2. Create an IAM trust JSON file with the following content:

      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Principal": {
              "Federated": "arn:aws:iam::<your_aws_account_id>:oidc-provider/<openshift_oidc_provider>" 1
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
              "StringEquals": {
                "<openshift_oidc_provider>:sub": "system:serviceaccount:openshift-logging:logcollector" 2
              }
            }
          }
        ]
      }

      1

      Specify your AWS account ID and the OpenShift OIDC provider endpoint. Obtain the endpoint by running the following command:

      $ rosa describe cluster \
        -c $(oc get clusterversion -o jsonpath='{.items[].spec.clusterID}{"\n"}') \
        -o yaml | awk '/oidc_endpoint_url/ {print $2}' | cut -d '/' -f 3,4

      2

      Specify the OpenShift OIDC endpoint again.

   3. Create the IAM role:

      $ aws iam create-role
        --role-name “<your_rosa_cluster_name>-RosaCloudWatch” \
        --assume-role-policy-document file://<your_trust_file_name>.json \
        --query Role.Arn \
        --output text

      Save the output. You will use it in the next steps.

   4. Create the IAM policy:

      $ aws iam create-policy \
      --policy-name "RosaCloudWatch" \
      --policy-document file:///<your_policy_file_name>.json \
      --query Policy.Arn \
      --output text

      Save the output. You will use it in the next steps.

   5. Attach the IAM policy to the IAM role:

      $ aws iam attach-role-policy \
       --role-name “<your_rosa_cluster_name>-RosaCloudWatch” \
       --policy-arn <policy_ARN> 1

      1

      Replace policy_ARN with the output you saved while creating the policy.

2. Create a Secret YAML file for the Red Hat OpenShift Logging Operator:

   apiVersion: v1
   kind: Secret
   metadata:
     name: cloudwatch-credentials
     namespace: openshift-logging
   stringData:
     credentials: |-
       [default]
       sts_regional_endpoints = regional
       role_arn: <role_ARN>  1
       web_identity_token_file = /var/run/secrets/openshift/serviceaccount/token

   1

   Replace role_ARN with the output you saved while creating the role.

3. Create the secret:

   $ oc apply -f cloudwatch-credentials.yaml

4. Create or edit a ClusterLogForwarder custom resource:

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: <log_forwarder_name> 1
     namespace: <log_forwarder_namespace> 2
   spec:
     serviceAccountName: <service_account_name> 3
     outputs:
      - name: cw 4
        type: cloudwatch 5
        cloudwatch:
          groupBy: logType 6
          groupPrefix: <group prefix> 7
          region: us-east-2 8
        secret:
           name: <your_secret_name> 9
     pipelines:
       - name: to-cloudwatch 10
         inputRefs: 11
           - infrastructure
           - audit
           - application
         outputRefs:
           - cw 12

   1

   In legacy implementations, the CR name must be instance. In multi log forwarder implementations, you can use any name.

   2

   In legacy implementations, the CR namespace must be openshift-logging. In multi log forwarder implementations, you can use any namespace.

   3

   The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the openshift-logging namespace.

   4

   Specify a name for the output.

   5

   Specify the cloudwatch type.

   6

   Optional: Specify how to group the logs:

   • logType creates log groups for each log type
   • namespaceName creates a log group for each application name space. Infrastructure and audit logs are unaffected, remaining grouped by logType.
   • namespaceUUID creates a new log groups for each application namespace UUID. It also creates separate log groups for infrastructure and audit logs.

   7

   Optional: Specify a string to replace the default infrastructureName prefix in the names of the log groups.

   8

   Specify the AWS region.

   9

   Specify the name of the secret you created previously.

   10

   Optional: Specify a name for the pipeline.

   11

   Specify which log types to forward by using the pipeline: application, infrastructure, or audit.

   12

   Specify the name of the output to use when forwarding logs with this pipeline.

Procedure

1. Modify the ClusterLogging CR collection spec:

   ClusterLogging CR example

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogging
   metadata:
   # ...
   spec:
   # ...
     collection:
       type: <log_collector_type> 1
       resources: {}
       tolerations: {}
   # ...

   1

   The log collector type you want to use for the logging. This can be vector or fluentd.

2. Apply the ClusterLogging CR by running the following command:

   $ oc apply -f <filename>.yaml

Procedure

1. Create a LogFileMetricExporter CR as a YAML file:

   Example LogFileMetricExporter CR

   apiVersion: logging.openshift.io/v1alpha1
   kind: LogFileMetricExporter
   metadata:
     name: instance
     namespace: openshift-logging
   spec:
     nodeSelector: {} 1
     resources: 2
       limits:
         cpu: 500m
         memory: 256Mi
       requests:
         cpu: 200m
         memory: 128Mi
     tolerations: [] 3
   # ...

   1

   Optional: The nodeSelector stanza defines which nodes the pods are scheduled on.

   2

   The resources stanza defines resource requirements for the LogFileMetricExporter CR.

   3

   Optional: The tolerations stanza defines the tolerations that the pods accept.

2. Apply the LogFileMetricExporter CR by running the following command:

   $ oc apply -f <filename>.yaml

Procedure

1. Edit the ClusterLogging custom resource (CR) in the openshift-logging project:

   $ oc edit ClusterLogging instance

2. Add or modify any of the following parameters:

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogging
   metadata:
     name: instance
     namespace: openshift-logging
   spec:
     collection:
       fluentd:
         buffer:
           chunkLimitSize: 8m 1
           flushInterval: 5s 2
           flushMode: interval 3
           flushThreadCount: 3 4
           overflowAction: throw_exception 5
           retryMaxInterval: "300s" 6
           retryType: periodic 7
           retryWait: 1s 8
           totalLimitSize: 32m 9
   # ...

   1

   Specify the maximum size of each chunk before it is queued for flushing.

   2

   Specify the interval between chunk flushes.

   3

   Specify the method to perform chunk flushes: lazy, interval, or immediate.

   4

   Specify the number of threads to use for chunk flushes.

   5

   Specify the chunking behavior when the queue is full: throw_exception, block, or drop_oldest_chunk.

   6

   Specify the maximum interval in seconds for the exponential_backoff chunk flushing method.

   7

   Specify the retry type when chunk flushing fails: exponential_backoff or periodic.

   8

   Specify the time in seconds before the next chunk flush.

   9

   Specify the maximum size of the chunk buffer.

3. Verify that the Fluentd pods are redeployed:

   $ oc get pods -l component=collector -n openshift-logging

4. Check that the new values are in the fluentd config map:

   $ oc extract configmap/collector-config --confirm

   Example fluentd.conf

   <buffer>
     @type file
     path '/var/lib/fluentd/default'
     flush_mode interval
     flush_interval 5s
     flush_thread_count 3
     retry_type periodic
     retry_wait 1s
     retry_max_interval 300s
     retry_timeout 60m
     queued_chunks_limit_size "#{ENV['BUFFER_QUEUE_LIMIT'] || '32'}"
     total_limit_size "#{ENV['TOTAL_LIMIT_SIZE_PER_BUFFER'] || '8589934592'}"
     chunk_limit_size 8m
     overflow_action throw_exception
     disable_chunk_backup true
   </buffer>

Procedure

1. Create a template for the Event Router:

   apiVersion: template.openshift.io/v1
   kind: Template
   metadata:
     name: eventrouter-template
     annotations:
       description: "A pod forwarding kubernetes events to OpenShift Logging stack."
       tags: "events,EFK,logging,cluster-logging"
   objects:
     - kind: ServiceAccount 1
       apiVersion: v1
       metadata:
         name: eventrouter
         namespace: ${NAMESPACE}
     - kind: ClusterRole 2
       apiVersion: rbac.authorization.k8s.io/v1
       metadata:
         name: event-reader
       rules:
       - apiGroups: [""]
         resources: ["events"]
         verbs: ["get", "watch", "list"]
     - kind: ClusterRoleBinding 3
       apiVersion: rbac.authorization.k8s.io/v1
       metadata:
         name: event-reader-binding
       subjects:
       - kind: ServiceAccount
         name: eventrouter
         namespace: ${NAMESPACE}
       roleRef:
         kind: ClusterRole
         name: event-reader
     - kind: ConfigMap 4
       apiVersion: v1
       metadata:
         name: eventrouter
         namespace: ${NAMESPACE}
       data:
         config.json: |-
           {
             "sink": "stdout"
           }
     - kind: Deployment 5
       apiVersion: apps/v1
       metadata:
         name: eventrouter
         namespace: ${NAMESPACE}
         labels:
           component: "eventrouter"
           logging-infra: "eventrouter"
           provider: "openshift"
       spec:
         selector:
           matchLabels:
             component: "eventrouter"
             logging-infra: "eventrouter"
             provider: "openshift"
         replicas: 1
         template:
           metadata:
             labels:
               component: "eventrouter"
               logging-infra: "eventrouter"
               provider: "openshift"
             name: eventrouter
           spec:
             serviceAccount: eventrouter
             containers:
               - name: kube-eventrouter
                 image: ${IMAGE}
                 imagePullPolicy: IfNotPresent
                 resources:
                   requests:
                     cpu: ${CPU}
                     memory: ${MEMORY}
                 volumeMounts:
                 - name: config-volume
                   mountPath: /etc/eventrouter
                 securityContext:
                   allowPrivilegeEscalation: false
                   capabilities:
                     drop: ["ALL"]
             securityContext:
               runAsNonRoot: true
               seccompProfile:
                 type: RuntimeDefault
             volumes:
             - name: config-volume
               configMap:
                 name: eventrouter
   parameters:
     - name: IMAGE 6
       displayName: Image
       value: "registry.redhat.io/openshift-logging/eventrouter-rhel9:v0.4"
     - name: CPU 7
       displayName: CPU
       value: "100m"
     - name: MEMORY 8
       displayName: Memory
       value: "128Mi"
     - name: NAMESPACE
       displayName: Namespace
       value: "openshift-logging" 9

   1

   Creates a Service Account in the openshift-logging project for the Event Router.

   2

   Creates a ClusterRole to monitor for events in the cluster.

   3

   Creates a ClusterRoleBinding to bind the ClusterRole to the service account.

   4

   Creates a config map in the openshift-logging project to generate the required config.json file.

   5

   Creates a deployment in the openshift-logging project to generate and configure the Event Router pod.

   6

   Specifies the image, identified by a tag such as v0.4.

   7

   Specifies the minimum amount of CPU to allocate to the Event Router pod. Defaults to 100m.

   8

   Specifies the minimum amount of memory to allocate to the Event Router pod. Defaults to 128Mi.

   9

   Specifies the openshift-logging project to install objects in.

2. Use the following command to process and apply the template:

   $ oc process -f <templatefile> | oc apply -n openshift-logging -f -

   For example:

   $ oc process -f eventrouter.yaml | oc apply -n openshift-logging -f -

   Example output

   serviceaccount/eventrouter created
   clusterrole.rbac.authorization.k8s.io/event-reader created
   clusterrolebinding.rbac.authorization.k8s.io/event-reader-binding created
   configmap/eventrouter created
   deployment.apps/eventrouter created

3. Validate that the Event Router installed in the openshift-logging project:

   1. View the new Event Router pod:

      $ oc get pods --selector  component=eventrouter -o name -n openshift-logging

      Example output

      pod/cluster-logging-eventrouter-d649f97c8-qvv8r

   2. View the events collected by the Event Router:

      $ oc logs <cluster_logging_eventrouter_pod> -n openshift-logging

      For example:

      $ oc logs cluster-logging-eventrouter-d649f97c8-qvv8r -n openshift-logging

      Example output

      {"verb":"ADDED","event":{"metadata":{"name":"openshift-service-catalog-controller-manager-remover.1632d931e88fcd8f","namespace":"openshift-service-catalog-removed","selfLink":"/api/v1/namespaces/openshift-service-catalog-removed/events/openshift-service-catalog-controller-manager-remover.1632d931e88fcd8f","uid":"787d7b26-3d2f-4017-b0b0-420db4ae62c0","resourceVersion":"21399","creationTimestamp":"2020-09-08T15:40:26Z"},"involvedObject":{"kind":"Job","namespace":"openshift-service-catalog-removed","name":"openshift-service-catalog-controller-manager-remover","uid":"fac9f479-4ad5-4a57-8adc-cb25d3d9cf8f","apiVersion":"batch/v1","resourceVersion":"21280"},"reason":"Completed","message":"Job completed","source":{"component":"job-controller"},"firstTimestamp":"2020-09-08T15:40:26Z","lastTimestamp":"2020-09-08T15:40:26Z","count":1,"type":"Normal"}}

      You can also use Kibana to view events by creating an index pattern using the Elasticsearch infra index.