Chapter 9. Log collection and forwarding

1. Fluentd does not support Elasticsearch 8 in the logging version 5.6.2.
2. Vector supports Syslog in the logging version 5.7 and higher.

Procedure

1. Add the following snippet to your

   ClusterLogForwarder

   CR YAML file.

   outputDefaults:
    elasticsearch:
       structuredTypeKey: <log record field>
       structuredTypeName: <name>
   pipelines:
   - inputRefs:
     - application
     outputRefs: default
     parse: json

2. Use

   structuredTypeKey

   field to specify one of the log record fields.

3. Use

   structuredTypeName

   field to specify a name.
   Important

   To parse JSON logs, you must set both the

   structuredTypeKey

   and

   structuredTypeName

   fields.

4. For

   inputRefs

   , specify which log types to forward by using that pipeline, such as

   application,

   infrastructure

   , or

   audit

   .
5. Add the

   parse: json

   element to pipelines.

6. Create the CR object:

   $ oc create -f <filename>.yaml

   The Red Hat OpenShift Logging Operator redeploys the collector pods. However, if they do not redeploy, delete the collector pods to force them to redeploy.

   $ oc delete pod --selector logging-infra=collector

Procedure

1. Create or edit a YAML file that defines the

   ClusterLogForwarder

   CR object:

   apiVersion: "logging.openshift.io/v1"
   kind: ClusterLogForwarder
   metadata:
     name: instance
     namespace: openshift-logging
   spec:
     outputDefaults:
       elasticsearch:
         structuredTypeKey: kubernetes.labels.logFormat 

   1

   
         structuredTypeName: nologformat
         enableStructuredContainerLogs: true 

   2

   
     pipelines:
     - inputRefs:
       - application
       name: application-logs
       outputRefs:
       - default
       parse: json

   1

   Uses the value of the key-value pair that is formed by the Kubernetes logFormat label.

   2

   Enables multi-container outputs.

2. Create or edit a YAML file that defines the

   Pod

   CR object:

       apiVersion: v1
       kind: Pod
       metadata:
         annotations:
           containerType.logging.openshift.io/heavy: heavy 

   1

   
           containerType.logging.openshift.io/low: low
       spec:
         containers:
         - name: heavy 

   2

   
           image: heavyimage
         - name: low
           image: lowimage

   1

   Format: containerType.logging.openshift.io/<container-name>: <index>

   2

   Annotation names must match container names

Procedure

1. Create a secret using your Google service account key.

   $ oc -n openshift-logging create secret generic gcp-secret --from-file google-application-credentials.json=<your_service_account_key_file.json>

2. Create a

   ClusterLogForwarder

   Custom Resource YAML using the template below:

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogForwarder"
   metadata:
     name: "instance"
     namespace: "openshift-logging"
   spec:
     outputs:
       - name: gcp-1
         type: googleCloudLogging
         secret:
           name: gcp-secret
         googleCloudLogging:
           projectId : "openshift-gce-devel" 

   1

   
           logId : "app-gcp" 

   2

   
     pipelines:
       - name: test-app
         inputRefs: 

   3

   
           - application
         outputRefs:
           - gcp-1

   1

   Set either a projectId, folderId, organizationId, or billingAccountId field and its corresponding value, depending on where you want to store your logs in the GCP resource hierarchy.

   2

   Set the value to add to the logName field of the Log Entry.

   3

   Specify which log types to forward by using the pipeline: application, infrastructure, or audit.

Procedure

1. Create a secret using your Base64 encoded Splunk HEC token.

   $ oc -n openshift-logging create secret generic vector-splunk-secret --from-literal hecToken=<HEC_Token>

2. Create or edit the

   ClusterLogForwarder

   Custom Resource (CR) using the template below:

     apiVersion: "logging.openshift.io/v1"
     kind: "ClusterLogForwarder"
     metadata:
       name: "instance" 

   1

   
       namespace: "openshift-logging" 

   2

   
     spec:
       outputs:
         - name: splunk-receiver 

   3

   
           secret:
             name: vector-splunk-secret 

   4

   
           type: splunk 

   5

   
           url: <http://your.splunk.hec.url:8088> 

   6

   
       pipelines: 

   7

   
         - inputRefs:
             - application
             - infrastructure
           name: 

   8

   
           outputRefs:
             - splunk-receiver 

   9

   1

   The name of the ClusterLogForwarder CR must be instance.

   2

   The namespace for the ClusterLogForwarder CR must be openshift-logging.

   3

   Specify a name for the output.

   4

   Specify the name of the secret that contains your HEC token.

   5

   Specify the output type as splunk.

   6

   Specify the URL (including port) of your Splunk HEC.

   7

   Specify which log types to forward by using the pipeline: application, infrastructure, or audit.

   8

   Optional: Specify a name for the pipeline.

   9

   Specify the name of the output to use when forwarding logs with this pipeline.

Procedure

1. Create or edit a YAML file that defines the

   ClusterLogForwarder

   CR:

   Example ClusterLogForwarder CR

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: instance 

   1

   
     namespace: openshift-logging 

   2

   
   spec:
     outputs:
      - name: fluentd-server-secure 

   3

   
        type: fluentdForward 

   4

   
        url: 'tls://fluentdserver.security.example.com:24224' 

   5

   
        secret: 

   6

   
           name: fluentd-secret
      - name: fluentd-server-insecure
        type: fluentdForward
        url: 'tcp://fluentdserver.home.example.com:24224'
     inputs: 

   7

   
      - name: my-app-logs
        application:
           namespaces:
           - my-project 

   8

   
     pipelines:
      - name: forward-to-fluentd-insecure 

   9

   
        inputRefs: 

   10

   
        - my-app-logs
        outputRefs: 

   11

   
        - fluentd-server-insecure
        labels:
          project: "my-project" 

   12

   
      - name: forward-to-fluentd-secure 

   13

   
        inputRefs:
        - application 

   14

   
        - audit
        - infrastructure
        outputRefs:
        - fluentd-server-secure
        - default
        labels:
          clusterId: "C1234"

   1

   The name of the ClusterLogForwarder CR must be instance.

   2

   The namespace for the ClusterLogForwarder CR must be openshift-logging.

   3

   The name of the output.

   4

   The output type: elasticsearch, fluentdForward, syslog, or kafka.

   5

   The URL and port of the external log aggregator as a valid absolute URL. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP address.

   6

   If using a tls prefix, you must specify the name of the secret required by the endpoint for TLS communication. The secret must exist in the openshift-logging project and have tls.crt, tls.key, and ca-bundle.crt keys that each point to the certificates they represent.

   7

   The configuration for an input to filter application logs from the specified projects.

   8

   If no namespace is specified, logs are collected from all namespaces.

   9

   The pipeline configuration directs logs from a named input to a named output. In this example, a pipeline named forward-to-fluentd-insecure forwards logs from an input named my-app-logs to an output named fluentd-server-insecure.

   10

   A list of inputs.

   11

   The name of the output to use.

   12

   Optional: String. One or more labels to add to the logs.

   13

   Configuration for a pipeline to send logs to other log aggregators.

   • Optional: Specify a name for the pipeline.
   • Specify which log types to forward by using the pipeline:

     application,

     infrastructure

     , or

     audit

     .
   • Specify the name of the output to use when forwarding logs with this pipeline.
   • Optional: Specify the

     default

     output to forward logs to the default log store.
   • Optional: String. One or more labels to add to the logs.

   14

   Note that application logs from all namespaces are collected when using this configuration.

2. Apply the

   ClusterLogForwarder

   CR by running the following command:

   $ oc apply -f <filename>.yaml

Procedure

1. Create or edit a YAML file that defines the

   ClusterLogForwarder

   CR object. In the file, specify the pod labels using simple equality-based selectors under

   inputs[].name.application.selector.matchLabels

   , as shown in the following example.

   Example ClusterLogForwarder CR YAML file

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: instance 

   1

   
     namespace: openshift-logging 

   2

   
   spec:
     pipelines:
       - inputRefs: [ myAppLogData ] 

   3

   
         outputRefs: [ default ] 

   4

   
     inputs: 

   5

   
       - name: myAppLogData
         application:
           selector:
             matchLabels: 

   6

   
               environment: production
               app: nginx
           namespaces: 

   7

   
           - app1
           - app2
     outputs: 

   8

   
       - default
       ...

   1

   The name of the ClusterLogForwarder CR must be instance.

   2

   The namespace for the ClusterLogForwarder CR must be openshift-logging.

   3

   Specify one or more comma-separated values from inputs[].name.

   4

   Specify one or more comma-separated values from outputs[].

   5

   Define a unique inputs[].name for each application that has a unique set of pod labels.

   6

   Specify the key-value pairs of pod labels whose log data you want to gather. You must specify both a key and value, not just a key. To be selected, the pods must match all the key-value pairs.

   7

   Optional: Specify one or more namespaces.

   8

   Specify one or more outputs to forward your log data to. The optional default output shown here sends log data to the internal Elasticsearch instance.

2. Optional: To restrict the gathering of log data to specific namespaces, use

   inputs[].name.application.namespaces

   , as shown in the preceding example.

3. Optional: You can send log data from additional applications that have different pod labels to the same pipeline.

   1. For each unique combination of pod labels, create an additional

      inputs[].name

      section similar to the one shown.
   2. Update the

      selectors

      to match the pod labels of this application.

   3. Add the new

      inputs[].name

      value to

      inputRefs

      . For example:

      - inputRefs: [ myAppLogData, myOtherAppLogData ]

4. Create the CR object:

   $ oc create -f <file-name>.yaml

Procedure

1. Create or edit a YAML file that defines the

   ClusterLogForwarder

   CR object:

   apiVersion: "logging.openshift.io/v1"
   kind: ClusterLogForwarder
   metadata:
     name: instance 

   1

   
     namespace: openshift-logging 

   2

   
   spec:
     outputs:
     - name: loki-insecure 

   3

   
       type: "loki" 

   4

   
       url: http://loki.insecure.com:3100 

   5

   
       loki:
         tenantKey: kubernetes.namespace_name
         labelKeys:
         - kubernetes.labels.foo
     - name: loki-secure 

   6

   
       type: "loki"
       url: https://loki.secure.com:3100
       secret:
         name: loki-secret 

   7

   
       loki:
         tenantKey: kubernetes.namespace_name 

   8

   
         labelKeys:
         - kubernetes.labels.foo 

   9

   
     pipelines:
     - name: application-logs 

   10

   
       inputRefs:  

   11

   
       - application
       - audit
       outputRefs: 

   12

   
       - loki-secure

   1

   The name of the ClusterLogForwarder CR must be instance.

   2

   The namespace for the ClusterLogForwarder CR must be openshift-logging.

   3

   Specify a name for the output.

   4

   Specify the type as "loki".

   5

   Specify the URL and port of the Loki system as a valid absolute URL. You can use the http (insecure) or https (secure HTTP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP Address. Loki’s default port for HTTP(S) communication is 3100.

   6

   For a secure connection, you can specify an https or http URL that you authenticate by specifying a secret.

   7

   For an https prefix, specify the name of the secret required by the endpoint for TLS communication. The secret must exist in the openshift-logging project and must contain a ca-bundle.crt key that points to the certificate it represents. Otherwise, for http and https prefixes, you can specify a secret that contains a username and password. For more information, see the following "Example: Setting secret that contains a username and password."

   8

   Optional: Specify a meta-data key field to generate values for the TenantID field in Loki. For example, setting tenantKey: kubernetes.namespace_name uses the names of the Kubernetes namespaces as values for tenant IDs in Loki. To see which other log record fields you can specify, see the "Log Record Fields" link in the following "Additional resources" section.

   9

   Optional: Specify a list of meta-data field keys to replace the default Loki labels. Loki label names must match the regular expression [a-zA-Z_:][a-zA-Z0-9_:]*. Illegal characters in meta-data keys are replaced with _ to form the label name. For example, the kubernetes.labels.foo meta-data key becomes Loki label kubernetes_labels_foo. If you do not set labelKeys, the default value is: [log_type, kubernetes.namespace_name, kubernetes.pod_name, kubernetes_host]. Keep the set of labels small because Loki limits the size and number of labels allowed. See Configuring Loki, limits_config. You can still query based on any log record field using query filters.

   10

   Optional: Specify a name for the pipeline.

   11

   Specify which log types to forward by using the pipeline: application, infrastructure, or audit.

   12

   Specify the name of the output to use when forwarding logs with this pipeline.

   Note

   Because Loki requires log streams to be correctly ordered by timestamp,

   labelKeys

   always includes the

   kubernetes_host

   label set, even if you do not specify it. This inclusion ensures that each stream originates from a single host, which prevents timestamps from becoming disordered due to clock differences on different hosts.

2. Apply the

   ClusterLogForwarder

   CR object by running the following command:

   $ oc apply -f <filename>.yaml

Procedure

1. Create or edit a YAML file that defines the

   ClusterLogForwarder

   CR:

   Example ClusterLogForwarder CR

   apiVersion: "logging.openshift.io/v1"
   kind: ClusterLogForwarder
   metadata:
     name: instance 

   1

   
     namespace: openshift-logging 

   2

   
   spec:
     outputs:
      - name: elasticsearch-example 

   3

   
        type: elasticsearch 

   4

   
        elasticsearch:
          version: 8 

   5

   
        url: http://elasticsearch.example.com:9200 

   6

   
        secret:
          name: es-secret 

   7

   
     pipelines:
      - name: application-logs 

   8

   
        inputRefs: 

   9

   
        - application
        - audit
        outputRefs:
        - elasticsearch-example 

   10

   
        - default 

   11

   
        labels:
          myLabel: "myValue" 

   12

   
   # ...

   1

   The name of the ClusterLogForwarder CR must be instance.

   2

   The namespace for the ClusterLogForwarder CR must be openshift-logging.

   3

   Specify a name for the output.

   4

   Specify the elasticsearch type.

   5

   Specify the Elasticsearch version. This can be 6, 7, or 8.

   6

   Specify the URL and port of the external Elasticsearch instance as a valid absolute URL. You can use the http (insecure) or https (secure HTTP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP Address.

   7

   For an https prefix, specify the name of the secret required by the endpoint for TLS communication. The secret must contain a ca-bundle.crt key that points to the certificate it represents. Otherwise, for http and https prefixes, you can specify a secret that contains a username and password. The secret must exist in the openshift-logging project. For more information, see the following "Example: Setting a secret that contains a username and password."

   8

   Optional: Specify a name for the pipeline.

   9

   Specify which log types to forward by using the pipeline: application, infrastructure, or audit.

   10

   Specify the name of the output to use when forwarding logs with this pipeline.

   11

   Optional: Specify the default output to send the logs to the internal Elasticsearch instance.

   12

   Optional: String. One or more labels to add to the logs.

2. Apply the

   ClusterLogForwarder

   CR:

   $ oc apply -f <filename>.yaml

1. Create a

   Secret

   YAML file similar to the following example. Use base64-encoded values for the

   username

   and

   password

   fields. The secret type is opaque by default.

   apiVersion: v1
   kind: Secret
   metadata:
     name: openshift-test-secret
   data:
     username: <username>
     password: <password>
   # ...

2. Create the secret:

   $ oc create secret -n openshift-logging openshift-test-secret.yaml

3. Specify the name of the secret in the

   ClusterLogForwarder

   CR:

   kind: ClusterLogForwarder
   metadata:
     name: instance
     namespace: openshift-logging
   spec:
     outputs:
      - name: elasticsearch
        type: "elasticsearch"
        url: https://elasticsearch.secure.com:9200
        secret:
           name: openshift-test-secret
   # ...

   Note

   In the value of the

   url

   field, the prefix can be

   http

   or

   https

   .

4. Apply the CR object:

   $ oc apply -f <filename>.yaml

Procedure

1. Create or edit a YAML file that defines the

   ClusterLogForwarder

   CR object:

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: instance 

   1

   
     namespace: openshift-logging 

   2

   
   spec:
     outputs:
      - name: fluentd-server-secure 

   3

   
        type: fluentdForward 

   4

   
        url: 'tls://fluentdserver.security.example.com:24224' 

   5

   
        secret: 

   6

   
           name: fluentd-secret
      - name: fluentd-server-insecure
        type: fluentdForward
        url: 'tcp://fluentdserver.home.example.com:24224'
     pipelines:
      - name: forward-to-fluentd-secure 

   7

   
        inputRefs:  

   8

   
        - application
        - audit
        outputRefs:
        - fluentd-server-secure 

   9

   
        - default 

   10

   
        labels:
          clusterId: "C1234" 

   11

   
      - name: forward-to-fluentd-insecure 

   12

   
        inputRefs:
        - infrastructure
        outputRefs:
        - fluentd-server-insecure
        labels:
          clusterId: "C1234"

   1

   The name of the ClusterLogForwarder CR must be instance.

   2

   The namespace for the ClusterLogForwarder CR must be openshift-logging.

   3

   Specify a name for the output.

   4

   Specify the fluentdForward type.

   5

   Specify the URL and port of the external Fluentd instance as a valid absolute URL. You can use the tcp (insecure) or tls (secure TCP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP address.

   6

   If you are using a tls prefix, you must specify the name of the secret required by the endpoint for TLS communication. The secret must exist in the openshift-logging project and must contain a ca-bundle.crt key that points to the certificate it represents.

   7

   Optional: Specify a name for the pipeline.

   8

   Specify which log types to forward by using the pipeline: application, infrastructure, or audit.

   9

   Specify the name of the output to use when forwarding logs with this pipeline.

   10

   Optional: Specify the default output to forward logs to the internal Elasticsearch instance.

   11

   Optional: String. One or more labels to add to the logs.

   12

   Optional: Configure multiple outputs to forward logs to other external log aggregators of any supported type:

   • A name to describe the pipeline.
   • The

     inputRefs

     is the log type to forward by using the pipeline:

     application,

     infrastructure

     , or

     audit

     .
   • The

     outputRefs

     is the name of the output to use.
   • Optional: String. One or more labels to add to the logs.

2. Create the CR object:

   $ oc create -f <file-name>.yaml

Procedure

1. Create or edit a YAML file that defines the

   ClusterLogForwarder

   CR object:

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: instance 

   1

   
     namespace: openshift-logging 

   2

   
   spec:
     outputs:
      - name: rsyslog-east 

   3

   
        type: syslog 

   4

   
        syslog: 

   5

   
          facility: local0
          rfc: RFC3164
          payloadKey: message
          severity: informational
        url: 'tls://rsyslogserver.east.example.com:514' 

   6

   
        secret: 

   7

   
           name: syslog-secret
      - name: rsyslog-west
        type: syslog
        syslog:
         appName: myapp
         facility: user
         msgID: mymsg
         procID: myproc
         rfc: RFC5424
         severity: debug
        url: 'tcp://rsyslogserver.west.example.com:514'
     pipelines:
      - name: syslog-east 

   8

   
        inputRefs: 

   9

   
        - audit
        - application
        outputRefs: 

   10

   
        - rsyslog-east
        - default 

   11

   
        labels:
          secure: "true" 

   12

   
          syslog: "east"
      - name: syslog-west 

   13

   
        inputRefs:
        - infrastructure
        outputRefs:
        - rsyslog-west
        - default
        labels:
          syslog: "west"

   1

   The name of the ClusterLogForwarder CR must be instance.

   2

   The namespace for the ClusterLogForwarder CR must be openshift-logging.

   3

   Specify a name for the output.

   4

   Specify the syslog type.

   5

   Optional: Specify the syslog parameters, listed below.

   6

   Specify the URL and port of the external syslog instance. You can use the udp (insecure), tcp (insecure) or tls (secure TCP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP address.

   7

   If using a tls prefix, you must specify the name of the secret required by the endpoint for TLS communication. The secret must exist in the openshift-logging project and must contain a ca-bundle.crt key that points to the certificate it represents.

   8

   Optional: Specify a name for the pipeline.

   9

   Specify which log types to forward by using the pipeline: application, infrastructure, or audit.

   10

   Specify the name of the output to use when forwarding logs with this pipeline.

   11

   Optional: Specify the default output to forward logs to the internal Elasticsearch instance.

   12

   Optional: String. One or more labels to add to the logs. Quote values like "true" so they are recognized as string values, not as a boolean.

   13

   Optional: Configure multiple outputs to forward logs to other external log aggregators of any supported type:

   • A name to describe the pipeline.
   • The

     inputRefs

     is the log type to forward by using the pipeline:

     application,

     infrastructure

     , or

     audit

     .
   • The

     outputRefs

     is the name of the output to use.
   • Optional: String. One or more labels to add to the logs.

2. Create the CR object:

   $ oc create -f <filename>.yaml

Procedure

1. Create or edit a YAML file that defines the

   ClusterLogForwarder

   CR object:

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: instance 

   1

   
     namespace: openshift-logging 

   2

   
   spec:
     outputs:
      - name: app-logs 

   3

   
        type: kafka 

   4

   
        url: tls://kafka.example.devlab.com:9093/app-topic 

   5

   
        secret:
          name: kafka-secret 

   6

   
      - name: infra-logs
        type: kafka
        url: tcp://kafka.devlab2.example.com:9093/infra-topic 

   7

   
      - name: audit-logs
        type: kafka
        url: tls://kafka.qelab.example.com:9093/audit-topic
        secret:
           name: kafka-secret-qe
     pipelines:
      - name: app-topic 

   8

   
        inputRefs: 

   9

   
        - application
        outputRefs: 

   10

   
        - app-logs
        labels:
          logType: "application" 

   11

   
      - name: infra-topic 

   12

   
        inputRefs:
        - infrastructure
        outputRefs:
        - infra-logs
        labels:
          logType: "infra"
      - name: audit-topic
        inputRefs:
        - audit
        outputRefs:
        - audit-logs
        - default 

   13

   
        labels:
          logType: "audit"

   1

   The name of the ClusterLogForwarder CR must be instance.

   2

   The namespace for the ClusterLogForwarder CR must be openshift-logging.

   3

   Specify a name for the output.

   4

   Specify the kafka type.

   5

   Specify the URL and port of the Kafka broker as a valid absolute URL, optionally with a specific topic. You can use the tcp (insecure) or tls (secure TCP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP address.

   6

   If using a tls prefix, you must specify the name of the secret required by the endpoint for TLS communication. The secret must exist in the openshift-logging project and must contain a ca-bundle.crt key that points to the certificate it represents.

   7

   Optional: To send an insecure output, use a tcp prefix in front of the URL. Also omit the secret key and its name from this output.

   8

   Optional: Specify a name for the pipeline.

   9

   Specify which log types to forward by using the pipeline: application, infrastructure, or audit.

   10

   Specify the name of the output to use when forwarding logs with this pipeline.

   11

   Optional: String. One or more labels to add to the logs.

   12

   Optional: Configure multiple outputs to forward logs to other external log aggregators of any supported type:

   • A name to describe the pipeline.
   • The

     inputRefs

     is the log type to forward by using the pipeline:

     application,

     infrastructure

     , or

     audit

     .
   • The

     outputRefs

     is the name of the output to use.
   • Optional: String. One or more labels to add to the logs.

   13

   Optional: Specify default to forward logs to the internal Elasticsearch instance.

2. Optional: To forward a single output to multiple Kafka brokers, specify an array of Kafka brokers as shown in the following example:

   # ...
   spec:
     outputs:
     - name: app-logs
       type: kafka
       secret:
         name: kafka-secret-dev
       kafka:  

   1

   
         brokers: 

   2

   
           - tls://kafka-broker1.example.com:9093/
           - tls://kafka-broker2.example.com:9093/
         topic: app-topic 

   3

   
   # ...

   1

   Specify a kafka key that has a brokers and topic key.

   2

   Use the brokers key to specify an array of one or more brokers.

   3

   Use the topic key to specify the target topic that receives the logs.

3. Apply the

   ClusterLogForwarder

   CR by running the following command:

   $ oc apply -f <filename>.yaml

Procedure

1. Create a

   Secret

   YAML file that uses the

   aws_access_key_id

   and

   aws_secret_access_key

   fields to specify your base64-encoded AWS credentials. For example:

   apiVersion: v1
   kind: Secret
   metadata:
     name: cw-secret
     namespace: openshift-logging
   data:
     aws_access_key_id: QUtJQUlPU0ZPRE5ON0VYQU1QTEUK
     aws_secret_access_key: d0phbHJYVXRuRkVNSS9LN01ERU5HL2JQeFJmaUNZRVhBTVBMRUtFWQo=

2. Create the secret. For example:

   $ oc apply -f cw-secret.yaml

3. Create or edit a YAML file that defines the

   ClusterLogForwarder

   CR object. In the file, specify the name of the secret. For example:

   apiVersion: "logging.openshift.io/v1"
   kind: ClusterLogForwarder
   metadata:
     name: instance 

   1

   
     namespace: openshift-logging 

   2

   
   spec:
     outputs:
      - name: cw 

   3

   
        type: cloudwatch 

   4

   
        cloudwatch:
          groupBy: logType 

   5

   
          groupPrefix: <group prefix> 

   6

   
          region: us-east-2 

   7

   
        secret:
           name: cw-secret 

   8

   
     pipelines:
       - name: infra-logs 

   9

   
         inputRefs: 

   10

   
           - infrastructure
           - audit
           - application
         outputRefs:
           - cw 

   11

   1

   The name of the ClusterLogForwarder CR must be instance.

   2

   The namespace for the ClusterLogForwarder CR must be openshift-logging.

   3

   Specify a name for the output.

   4

   Specify the cloudwatch type.

   5

   Optional: Specify how to group the logs:

   • logType

     creates log groups for each log type

   • namespaceName

     creates a log group for each application name space. It also creates separate log groups for infrastructure and audit logs.

   • namespaceUUID

     creates a new log groups for each application namespace UUID. It also creates separate log groups for infrastructure and audit logs.

   6

   Optional: Specify a string to replace the default infrastructureName prefix in the names of the log groups.

   7

   Specify the AWS region.

   8

   Specify the name of the secret that contains your AWS credentials.

   9

   Optional: Specify a name for the pipeline.

   10

   Specify which log types to forward by using the pipeline: application, infrastructure, or audit.

   11

   Specify the name of the output to use when forwarding logs with this pipeline.

4. Create the CR object:

   $ oc create -f <file-name>.yaml

Procedure

1. Create a

   CredentialsRequest

   custom resource YAML by using the template below:

   CloudWatch credentials request template

   apiVersion: cloudcredential.openshift.io/v1
   kind: CredentialsRequest
   metadata:
     name: <your_role_name>-credrequest
     namespace: openshift-cloud-credential-operator
   spec:
     providerSpec:
       apiVersion: cloudcredential.openshift.io/v1
       kind: AWSProviderSpec
       statementEntries:
         - action:
             - logs:PutLogEvents
             - logs:CreateLogGroup
             - logs:PutRetentionPolicy
             - logs:CreateLogStream
             - logs:DescribeLogGroups
             - logs:DescribeLogStreams
           effect: Allow
           resource: arn:aws:logs:*:*:*
     secretRef:
       name: <your_role_name>
       namespace: openshift-logging
     serviceAccountNames:
       - logcollector

2. Use the

   ccoctl

   command to create a role for AWS using your

   CredentialsRequest

   CR. With the

   CredentialsRequest

   object, this

   ccoctl

   command creates an IAM role with a trust policy that is tied to the specified OIDC identity provider, and a permissions policy that grants permissions to perform operations on CloudWatch resources. This command also creates a YAML configuration file in

   /<path_to_ccoctl_output_dir>/manifests/openshift-logging-<your_role_name>-credentials.yaml

   . This secret file contains the

   role_arn

   key/value used during authentication with the AWS IAM identity provider.

   $ ccoctl aws create-iam-roles \
   --name=<name> \
   --region=<aws_region> \
   --credentials-requests-dir=<path_to_directory_with_list_of_credentials_requests>/credrequests \
   --identity-provider-arn=arn:aws:iam::<aws_account_id>:oidc-provider/<name>-oidc.s3.<aws_region>.amazonaws.com 

   1

   1

   <name> is the name used to tag your cloud resources and should match the name used during your STS cluster install

3. Apply the secret created:

   $ oc apply -f output/manifests/openshift-logging-<your_role_name>-credentials.yaml

4. Create or edit a

   ClusterLogForwarder

   custom resource:

   apiVersion: "logging.openshift.io/v1"
   kind: ClusterLogForwarder
   metadata:
     name: instance 

   1

   
     namespace: openshift-logging 

   2

   
   spec:
     outputs:
      - name: cw 

   3

   
        type: cloudwatch 

   4

   
        cloudwatch:
          groupBy: logType 

   5

   
          groupPrefix: <group prefix> 

   6

   
          region: us-east-2 

   7

   
        secret:
           name: <your_role_name> 

   8

   
     pipelines:
       - name: to-cloudwatch 

   9

   
         inputRefs: 

   10

   
           - infrastructure
           - audit
           - application
         outputRefs:
           - cw 

   11

   1

   The name of the ClusterLogForwarder CR must be instance.

   2

   The namespace for the ClusterLogForwarder CR must be openshift-logging.

   3

   Specify a name for the output.

   4

   Specify the cloudwatch type.

   5

   Optional: Specify how to group the logs:

   • logType

     creates log groups for each log type

   • namespaceName

     creates a log group for each application name space. Infrastructure and audit logs are unaffected, remaining grouped by

     logType

     .

   • namespaceUUID

     creates a new log groups for each application namespace UUID. It also creates separate log groups for infrastructure and audit logs.

   6

   Optional: Specify a string to replace the default infrastructureName prefix in the names of the log groups.

   7

   Specify the AWS region.

   8

   Specify the name of the secret that contains your AWS credentials.

   9

   Optional: Specify a name for the pipeline.

   10

   Specify which log types to forward by using the pipeline: application, infrastructure, or audit.

   11

   Specify the name of the output to use when forwarding logs with this pipeline.

Procedure

1. Modify the

   ClusterLogging

   CR

   collection

   spec:

   ClusterLogging CR example

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogging
   metadata:
   # ...
   spec:
   # ...
     collection:
       type: <log_collector_type> 

   1

   
       resources: {}
       tolerations: {}
   # ...

   1

   The log collector type you want to use for the logging. This can be vector or fluentd.

2. Apply the

   ClusterLogging

   CR by running the following command:

   $ oc apply -f <filename>.yaml

Procedure

1. Edit the

   ClusterLogging

   custom resource (CR) in the

   openshift-logging

   project:

   $ oc edit ClusterLogging instance

2. Add or modify any of the following parameters:

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogging
   metadata:
     name: instance
     namespace: openshift-logging
   spec:
     collection:
       fluentd:
         buffer:
           chunkLimitSize: 8m 

   1

   
           flushInterval: 5s 

   2

   
           flushMode: interval 

   3

   
           flushThreadCount: 3 

   4

   
           overflowAction: throw_exception 

   5

   
           retryMaxInterval: "300s" 

   6

   
           retryType: periodic 

   7

   
           retryWait: 1s 

   8

   
           totalLimitSize: 32m 

   9

   
   # ...

   1

   Specify the maximum size of each chunk before it is queued for flushing.

   2

   Specify the interval between chunk flushes.

   3

   Specify the method to perform chunk flushes: lazy, interval, or immediate.

   4

   Specify the number of threads to use for chunk flushes.

   5

   Specify the chunking behavior when the queue is full: throw_exception, block, or drop_oldest_chunk.

   6

   Specify the maximum interval in seconds for the exponential_backoff chunk flushing method.

   7

   Specify the retry type when chunk flushing fails: exponential_backoff or periodic.

   8

   Specify the time in seconds before the next chunk flush.

   9

   Specify the maximum size of the chunk buffer.

3. Verify that the Fluentd pods are redeployed:

   $ oc get pods -l component=collector -n openshift-logging

4. Check that the new values are in the

   fluentd

   config map:

   $ oc extract configmap/collector --confirm

   Example fluentd.conf

   <buffer>
     @type file
     path '/var/lib/fluentd/default'
     flush_mode interval
     flush_interval 5s
     flush_thread_count 3
     retry_type periodic
     retry_wait 1s
     retry_max_interval 300s
     retry_timeout 60m
     queued_chunks_limit_size "#{ENV['BUFFER_QUEUE_LIMIT'] || '32'}"
     total_limit_size "#{ENV['TOTAL_LIMIT_SIZE_PER_BUFFER'] || '8589934592'}"
     chunk_limit_size 8m
     overflow_action throw_exception
     disable_chunk_backup true
   </buffer>

Procedure

1. Create a template for the Event Router:

   apiVersion: template.openshift.io/v1
   kind: Template
   metadata:
     name: eventrouter-template
     annotations:
       description: "A pod forwarding kubernetes events to OpenShift Logging stack."
       tags: "events,EFK,logging,cluster-logging"
   objects:
     - kind: ServiceAccount 

   1

   
       apiVersion: v1
       metadata:
         name: eventrouter
         namespace: ${NAMESPACE}
     - kind: ClusterRole 

   2

   
       apiVersion: rbac.authorization.k8s.io/v1
       metadata:
         name: event-reader
       rules:
       - apiGroups: [""]
         resources: ["events"]
         verbs: ["get", "watch", "list"]
     - kind: ClusterRoleBinding 

   3

   
       apiVersion: rbac.authorization.k8s.io/v1
       metadata:
         name: event-reader-binding
       subjects:
       - kind: ServiceAccount
         name: eventrouter
         namespace: ${NAMESPACE}
       roleRef:
         kind: ClusterRole
         name: event-reader
     - kind: ConfigMap 

   4

   
       apiVersion: v1
       metadata:
         name: eventrouter
         namespace: ${NAMESPACE}
       data:
         config.json: |-
           {
             "sink": "stdout"
           }
     - kind: Deployment 

   5

   
       apiVersion: apps/v1
       metadata:
         name: eventrouter
         namespace: ${NAMESPACE}
         labels:
           component: "eventrouter"
           logging-infra: "eventrouter"
           provider: "openshift"
       spec:
         selector:
           matchLabels:
             component: "eventrouter"
             logging-infra: "eventrouter"
             provider: "openshift"
         replicas: 1
         template:
           metadata:
             labels:
               component: "eventrouter"
               logging-infra: "eventrouter"
               provider: "openshift"
             name: eventrouter
           spec:
             serviceAccount: eventrouter
             containers:
               - name: kube-eventrouter
                 image: ${IMAGE}
                 imagePullPolicy: IfNotPresent
                 resources:
                   requests:
                     cpu: ${CPU}
                     memory: ${MEMORY}
                 volumeMounts:
                 - name: config-volume
                   mountPath: /etc/eventrouter
                 securityContext:
                   allowPrivilegeEscalation: false
                   capabilities:
                     drop: ["ALL"]
             securityContext:
               runAsNonRoot: true
               seccompProfile:
                 type: RuntimeDefault
             volumes:
             - name: config-volume
               configMap:
                 name: eventrouter
   parameters:
     - name: IMAGE 

   6

   
       displayName: Image
       value: "registry.redhat.io/openshift-logging/eventrouter-rhel8:v0.4"
     - name: CPU 

   7

   
       displayName: CPU
       value: "100m"
     - name: MEMORY 

   8

   
       displayName: Memory
       value: "128Mi"
     - name: NAMESPACE
       displayName: Namespace
       value: "openshift-logging" 

   9

   1

   Creates a Service Account in the openshift-logging project for the Event Router.

   2

   Creates a ClusterRole to monitor for events in the cluster.

   3

   Creates a ClusterRoleBinding to bind the ClusterRole to the service account.

   4

   Creates a config map in the openshift-logging project to generate the required config.json file.

   5

   Creates a deployment in the openshift-logging project to generate and configure the Event Router pod.

   6

   Specifies the image, identified by a tag such as v0.4.

   7

   Specifies the minimum amount of CPU to allocate to the Event Router pod. Defaults to 100m.

   8

   Specifies the minimum amount of memory to allocate to the Event Router pod. Defaults to 128Mi.

   9

   Specifies the openshift-logging project to install objects in.

2. Use the following command to process and apply the template:

   $ oc process -f <templatefile> | oc apply -n openshift-logging -f -

   For example:

   $ oc process -f eventrouter.yaml | oc apply -n openshift-logging -f -

   Example output

   serviceaccount/eventrouter created
   clusterrole.rbac.authorization.k8s.io/event-reader created
   clusterrolebinding.rbac.authorization.k8s.io/event-reader-binding created
   configmap/eventrouter created
   deployment.apps/eventrouter created

3. Validate that the Event Router installed in the

   openshift-logging

   project:

   1. View the new Event Router pod:

      $ oc get pods --selector  component=eventrouter -o name -n openshift-logging

      Example output

      pod/cluster-logging-eventrouter-d649f97c8-qvv8r

   2. View the events collected by the Event Router:

      $ oc logs <cluster_logging_eventrouter_pod> -n openshift-logging

      For example:

      $ oc logs cluster-logging-eventrouter-d649f97c8-qvv8r -n openshift-logging

      Example output

      {"verb":"ADDED","event":{"metadata":{"name":"openshift-service-catalog-controller-manager-remover.1632d931e88fcd8f","namespace":"openshift-service-catalog-removed","selfLink":"/api/v1/namespaces/openshift-service-catalog-removed/events/openshift-service-catalog-controller-manager-remover.1632d931e88fcd8f","uid":"787d7b26-3d2f-4017-b0b0-420db4ae62c0","resourceVersion":"21399","creationTimestamp":"2020-09-08T15:40:26Z"},"involvedObject":{"kind":"Job","namespace":"openshift-service-catalog-removed","name":"openshift-service-catalog-controller-manager-remover","uid":"fac9f479-4ad5-4a57-8adc-cb25d3d9cf8f","apiVersion":"batch/v1","resourceVersion":"21280"},"reason":"Completed","message":"Job completed","source":{"component":"job-controller"},"firstTimestamp":"2020-09-08T15:40:26Z","lastTimestamp":"2020-09-08T15:40:26Z","count":1,"type":"Normal"}}

      You can also use Kibana to view events by creating an index pattern using the Elasticsearch

      infra

      index.