Chapter 10. Log collection and forwarding

Procedure

1. Add the following snippet to your

   ClusterLogForwarder

   CR YAML file.

   outputDefaults:
    elasticsearch:
       structuredTypeKey: <log record field>
       structuredTypeName: <name>
   pipelines:
   - inputRefs:
     - application
     outputRefs: default
     parse: json

2. Use

   structuredTypeKey

   field to specify one of the log record fields.

3. Use

   structuredTypeName

   field to specify a name.
   Important

   To parse JSON logs, you must set both the

   structuredTypeKey

   and

   structuredTypeName

   fields.

4. For

   inputRefs

   , specify which log types to forward by using that pipeline, such as

   application,

   infrastructure

   , or

   audit

   .
5. Add the

   parse: json

   element to pipelines.

6. Create the CR object:

   $ oc create -f <filename>.yaml

   The Red Hat OpenShift Logging Operator redeploys the collector pods. However, if they do not redeploy, delete the collector pods to force them to redeploy.

   $ oc delete pod --selector logging-infra=collector

Procedure

1. Create or edit a YAML file that defines the

   ClusterLogForwarder

   CR object:

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: instance
     namespace: openshift-logging
   spec:
     outputDefaults:
       elasticsearch:
         structuredTypeKey: kubernetes.labels.logFormat 

   1

   
         structuredTypeName: nologformat
         enableStructuredContainerLogs: true 

   2

   
     pipelines:
     - inputRefs:
       - application
       name: application-logs
       outputRefs:
       - default
       parse: json

   1

   Uses the value of the key-value pair that is formed by the Kubernetes logFormat label.

   2

   Enables multi-container outputs.

2. Create or edit a YAML file that defines the

   Pod

   CR object:

   apiVersion: v1
   kind: Pod
   metadata:
     annotations:
       containerType.logging.openshift.io/heavy: heavy 

   1

   
       containerType.logging.openshift.io/low: low
   spec:
     containers:
     - name: heavy 

   2

   
       image: heavyimage
     - name: low
       image: lowimage

   1

   Format: containerType.logging.openshift.io/<container-name>: <index>

   2

   Annotation names must match container names

Procedure

1. Create a secret using your Google service account key.

   $ oc -n openshift-logging create secret generic gcp-secret --from-file google-application-credentials.json=<your_service_account_key_file.json>

2. Create a

   ClusterLogForwarder

   Custom Resource YAML using the template below:

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: <log_forwarder_name> 

   1

   
     namespace: <log_forwarder_namespace> 

   2

   
   spec:
     serviceAccountName: <service_account_name> 

   3

   
     outputs:
       - name: gcp-1
         type: googleCloudLogging
         secret:
           name: gcp-secret
         googleCloudLogging:
           projectId : "openshift-gce-devel" 

   4

   
           logId : "app-gcp" 

   5

   
     pipelines:
       - name: test-app
         inputRefs: 

   6

   
           - application
         outputRefs:
           - gcp-1

   1

   In legacy implementations, the CR name must be instance. In multi log forwarder implementations, you can use any name.

   2

   In legacy implementations, the CR namespace must be openshift-logging. In multi log forwarder implementations, you can use any namespace.

   3

   The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the openshift-logging namespace.

   4

   Set a projectId, folderId, organizationId, or billingAccountId field and its corresponding value, depending on where you want to store your logs in the Google Cloud resource hierarchy.

   5

   Set the value to add to the logName field of the Log Entry.

   6

   Specify which log types to forward by using the pipeline: application, infrastructure, or audit.

Procedure

1. Create a secret using your Base64 encoded Splunk HEC token.

   $ oc -n openshift-logging create secret generic vector-splunk-secret --from-literal hecToken=<HEC_Token>

2. Create or edit the

   ClusterLogForwarder

   Custom Resource (CR) using the template below:

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: <log_forwarder_name> 

   1

   
     namespace: <log_forwarder_namespace> 

   2

   
   spec:
     serviceAccountName: <service_account_name> 

   3

   
     outputs:
       - name: splunk-receiver 

   4

   
         secret:
           name: vector-splunk-secret 

   5

   
         type: splunk 

   6

   
         url: <http://your.splunk.hec.url:8088> 

   7

   
     pipelines: 

   8

   
       - inputRefs:
           - application
           - infrastructure
         name: 

   9

   
         outputRefs:
           - splunk-receiver 

   10

   1

   In legacy implementations, the CR name must be instance. In multi log forwarder implementations, you can use any name.

   2

   In legacy implementations, the CR namespace must be openshift-logging. In multi log forwarder implementations, you can use any namespace.

   3

   The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the openshift-logging namespace.

   4

   Specify a name for the output.

   5

   Specify the name of the secret that contains your HEC token.

   6

   Specify the output type as splunk.

   7

   Specify the URL (including port) of your Splunk HEC.

   8

   Specify which log types to forward by using the pipeline: application, infrastructure, or audit.

   9

   Optional: Specify a name for the pipeline.

   10

   Specify the name of the output to use when forwarding logs with this pipeline.

Procedure

1. Create or edit a YAML file that defines the

   ClusterLogForwarder

   CR:

   Example ClusterLogForwarder CR

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: instance 

   1

   
     namespace: openshift-logging 

   2

   
   spec:
     outputs:
      - name: fluentd-server-secure 

   3

   
        type: fluentdForward 

   4

   
        url: 'tls://fluentdserver.security.example.com:24224' 

   5

   
        secret: 

   6

   
           name: fluentd-secret
      - name: fluentd-server-insecure
        type: fluentdForward
        url: 'tcp://fluentdserver.home.example.com:24224'
     inputs: 

   7

   
      - name: my-app-logs
        application:
           namespaces:
           - my-project 

   8

   
     pipelines:
      - name: forward-to-fluentd-insecure 

   9

   
        inputRefs: 

   10

   
        - my-app-logs
        outputRefs: 

   11

   
        - fluentd-server-insecure
        labels:
          project: "my-project" 

   12

   
      - name: forward-to-fluentd-secure 

   13

   
        inputRefs:
        - application 

   14

   
        - audit
        - infrastructure
        outputRefs:
        - fluentd-server-secure
        - default
        labels:
          clusterId: "C1234"

   1

   The name of the ClusterLogForwarder CR must be instance.

   2

   The namespace for the ClusterLogForwarder CR must be openshift-logging.

   3

   The name of the output.

   4

   The output type: elasticsearch, fluentdForward, syslog, or kafka.

   5

   The URL and port of the external log aggregator as a valid absolute URL. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP address.

   6

   If using a tls prefix, you must specify the name of the secret required by the endpoint for TLS communication. The secret must exist in the openshift-logging project and have tls.crt, tls.key, and ca-bundle.crt keys that each point to the certificates they represent.

   7

   The configuration for an input to filter application logs from the specified projects.

   8

   If no namespace is specified, logs are collected from all namespaces.

   9

   The pipeline configuration directs logs from a named input to a named output. In this example, a pipeline named forward-to-fluentd-insecure forwards logs from an input named my-app-logs to an output named fluentd-server-insecure.

   10

   A list of inputs.

   11

   The name of the output to use.

   12

   Optional: String. One or more labels to add to the logs.

   13

   Configuration for a pipeline to send logs to other log aggregators.

   • Optional: Specify a name for the pipeline.
   • Specify which log types to forward by using the pipeline:

     application,

     infrastructure

     , or

     audit

     .
   • Specify the name of the output to use when forwarding logs with this pipeline.
   • Optional: Specify the

     default

     output to forward logs to the default log store.
   • Optional: String. One or more labels to add to the logs.

   14

   Note that application logs from all namespaces are collected when using this configuration.

2. Apply the

   ClusterLogForwarder

   CR by running the following command:

   $ oc apply -f <filename>.yaml

Procedure

1. Create or edit a YAML file that defines the

   ClusterLogForwarder

   CR object. In the file, specify the pod labels using simple equality-based selectors under

   inputs[].name.application.selector.matchLabels

   , as shown in the following example.

   Example ClusterLogForwarder CR YAML file

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: <log_forwarder_name> 

   1

   
     namespace: <log_forwarder_namespace> 

   2

   
   spec:
     pipelines:
       - inputRefs: [ myAppLogData ] 

   3

   
         outputRefs: [ default ] 

   4

   
     inputs: 

   5

   
       - name: myAppLogData
         application:
           selector:
             matchLabels: 

   6

   
               environment: production
               app: nginx
           namespaces: 

   7

   
           - app1
           - app2
     outputs: 

   8

   
       - <output_name>
       ...

   1

   In legacy implementations, the CR name must be instance. In multi log forwarder implementations, you can use any name.

   2

   In legacy implementations, the CR namespace must be openshift-logging. In multi log forwarder implementations, you can use any namespace.

   3

   Specify one or more comma-separated values from inputs[].name.

   4

   Specify one or more comma-separated values from outputs[].

   5

   Define a unique inputs[].name for each application that has a unique set of pod labels.

   6

   Specify the key-value pairs of pod labels whose log data you want to gather. You must specify both a key and value, not just a key. To be selected, the pods must match all the key-value pairs.

   7

   Optional: Specify one or more namespaces.

   8

   Specify one or more outputs to forward your log data to.

2. Optional: To restrict the gathering of log data to specific namespaces, use

   inputs[].name.application.namespaces

   , as shown in the preceding example.

3. Optional: You can send log data from additional applications that have different pod labels to the same pipeline.

   1. For each unique combination of pod labels, create an additional

      inputs[].name

      section similar to the one shown.
   2. Update the

      selectors

      to match the pod labels of this application.

   3. Add the new

      inputs[].name

      value to

      inputRefs

      . For example:

      - inputRefs: [ myAppLogData, myOtherAppLogData ]

4. Create the CR object:

   $ oc create -f <file-name>.yaml

Procedure

1. Create or edit a YAML file that defines the

   ClusterLogForwarder

   CR object:

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: <log_forwarder_name> 

   1

   
     namespace: <log_forwarder_namespace> 

   2

   
   spec:
     serviceAccountName: <service_account_name> 

   3

   
     outputs:
     - name: loki-insecure 

   4

   
       type: "loki" 

   5

   
       url: http://loki.insecure.com:3100 

   6

   
       loki:
         tenantKey: kubernetes.namespace_name
         labelKeys:
         - kubernetes.labels.foo
     - name: loki-secure 

   7

   
       type: "loki"
       url: https://loki.secure.com:3100
       secret:
         name: loki-secret 

   8

   
       loki:
         tenantKey: kubernetes.namespace_name 

   9

   
         labelKeys:
         - kubernetes.labels.foo 

   10

   
     pipelines:
     - name: application-logs 

   11

   
       inputRefs:  

   12

   
       - application
       - audit
       outputRefs: 

   13

   
       - loki-secure

   1

   In legacy implementations, the CR name must be instance. In multi log forwarder implementations, you can use any name.

   2

   In legacy implementations, the CR namespace must be openshift-logging. In multi log forwarder implementations, you can use any namespace.

   3

   The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the openshift-logging namespace.

   4

   Specify a name for the output.

   5

   Specify the type as "loki".

   6

   Specify the URL and port of the Loki system as a valid absolute URL. You can use the http (insecure) or https (secure HTTP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP Address. Loki’s default port for HTTP(S) communication is 3100.

   7

   For a secure connection, you can specify an https or http URL that you authenticate by specifying a secret.

   8

   For an https prefix, specify the name of the secret required by the endpoint for TLS communication. The secret must contain a ca-bundle.crt key that points to the certificates it represents. Otherwise, for http and https prefixes, you can specify a secret that contains a username and password. In legacy implementations, the secret must exist in the openshift-logging project. For more information, see the following "Example: Setting a secret that contains a username and password."

   9

   Optional: Specify a metadata key field to generate values for the TenantID field in Loki. For example, setting tenantKey: kubernetes.namespace_name uses the names of the Kubernetes namespaces as values for tenant IDs in Loki. To see which other log record fields you can specify, see the "Log Record Fields" link in the following "Additional resources" section.

   10

   Optional: Specify a list of metadata field keys to replace the default Loki labels. Loki label names must match the regular expression [a-zA-Z_:][a-zA-Z0-9_:]*. Illegal characters in metadata keys are replaced with _ to form the label name. For example, the kubernetes.labels.foo metadata key becomes Loki label kubernetes_labels_foo. If you do not set labelKeys, the default value is: [log_type, kubernetes.namespace_name, kubernetes.pod_name, kubernetes_host]. Keep the set of labels small because Loki limits the size and number of labels allowed. See Configuring Loki, limits_config. You can still query based on any log record field using query filters.

   11

   Optional: Specify a name for the pipeline.

   12

   Specify which log types to forward by using the pipeline: application, infrastructure, or audit.

   13

   Specify the name of the output to use when forwarding logs with this pipeline.

   Note

   Because Loki requires log streams to be correctly ordered by timestamp,

   labelKeys

   always includes the

   kubernetes_host

   label set, even if you do not specify it. This inclusion ensures that each stream originates from a single host, which prevents timestamps from becoming disordered due to clock differences on different hosts.

2. Apply the

   ClusterLogForwarder

   CR object by running the following command:

   $ oc apply -f <filename>.yaml

Procedure

1. Create or edit a YAML file that defines the

   ClusterLogForwarder

   CR:

   Example ClusterLogForwarder CR

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: <log_forwarder_name> 

   1

   
     namespace: <log_forwarder_namespace> 

   2

   
   spec:
     serviceAccountName: <service_account_name> 

   3

   
     outputs:
      - name: elasticsearch-example 

   4

   
        type: elasticsearch 

   5

   
        elasticsearch:
          version: 8 

   6

   
        url: http://elasticsearch.example.com:9200 

   7

   
        secret:
          name: es-secret 

   8

   
     pipelines:
      - name: application-logs 

   9

   
        inputRefs: 

   10

   
        - application
        - audit
        outputRefs:
        - elasticsearch-example 

   11

   
        - default 

   12

   
        labels:
          myLabel: "myValue" 

   13

   
   # ...

   1

   In legacy implementations, the CR name must be instance. In multi log forwarder implementations, you can use any name.

   2

   In legacy implementations, the CR namespace must be openshift-logging. In multi log forwarder implementations, you can use any namespace.

   3

   The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the openshift-logging namespace.

   4

   Specify a name for the output.

   5

   Specify the elasticsearch type.

   6

   Specify the Elasticsearch version. This can be 6, 7, or 8.

   7

   Specify the URL and port of the external Elasticsearch instance as a valid absolute URL. You can use the http (insecure) or https (secure HTTP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP Address.

   8

   For an https prefix, specify the name of the secret required by the endpoint for TLS communication. The secret must contain a ca-bundle.crt key that points to the certificate it represents. Otherwise, for http and https prefixes, you can specify a secret that contains a username and password. In legacy implementations, the secret must exist in the openshift-logging project. For more information, see the following "Example: Setting a secret that contains a username and password."

   9

   Optional: Specify a name for the pipeline.

   10

   Specify which log types to forward by using the pipeline: application, infrastructure, or audit.

   11

   Specify the name of the output to use when forwarding logs with this pipeline.

   12

   Optional: Specify the default output to send the logs to the internal Elasticsearch instance.

   13

   Optional: String. One or more labels to add to the logs.

2. Apply the

   ClusterLogForwarder

   CR:

   $ oc apply -f <filename>.yaml

1. Create a

   Secret

   YAML file similar to the following example. Use base64-encoded values for the

   username

   and

   password

   fields. The secret type is opaque by default.

   apiVersion: v1
   kind: Secret
   metadata:
     name: openshift-test-secret
   data:
     username: <username>
     password: <password>
   # ...

2. Create the secret:

   $ oc create secret -n openshift-logging openshift-test-secret.yaml

3. Specify the name of the secret in the

   ClusterLogForwarder

   CR:

   kind: ClusterLogForwarder
   metadata:
     name: instance
     namespace: openshift-logging
   spec:
     outputs:
      - name: elasticsearch
        type: "elasticsearch"
        url: https://elasticsearch.secure.com:9200
        secret:
           name: openshift-test-secret
   # ...

   Note

   In the value of the

   url

   field, the prefix can be

   http

   or

   https

   .

4. Apply the CR object:

   $ oc apply -f <filename>.yaml

Procedure

1. Create or edit a YAML file that defines the

   ClusterLogForwarder

   CR object:

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: instance 

   1

   
     namespace: openshift-logging 

   2

   
   spec:
     outputs:
      - name: fluentd-server-secure 

   3

   
        type: fluentdForward 

   4

   
        url: 'tls://fluentdserver.security.example.com:24224' 

   5

   
        secret: 

   6

   
           name: fluentd-secret
      - name: fluentd-server-insecure
        type: fluentdForward
        url: 'tcp://fluentdserver.home.example.com:24224'
     pipelines:
      - name: forward-to-fluentd-secure 

   7

   
        inputRefs:  

   8

   
        - application
        - audit
        outputRefs:
        - fluentd-server-secure 

   9

   
        - default 

   10

   
        labels:
          clusterId: "C1234" 

   11

   
      - name: forward-to-fluentd-insecure 

   12

   
        inputRefs:
        - infrastructure
        outputRefs:
        - fluentd-server-insecure
        labels:
          clusterId: "C1234"

   1

   The name of the ClusterLogForwarder CR must be instance.

   2

   The namespace for the ClusterLogForwarder CR must be openshift-logging.

   3

   Specify a name for the output.

   4

   Specify the fluentdForward type.

   5

   Specify the URL and port of the external Fluentd instance as a valid absolute URL. You can use the tcp (insecure) or tls (secure TCP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP address.

   6

   If you are using a tls prefix, you must specify the name of the secret required by the endpoint for TLS communication. The secret must exist in the openshift-logging project and must contain a ca-bundle.crt key that points to the certificate it represents.

   7

   Optional: Specify a name for the pipeline.

   8

   Specify which log types to forward by using the pipeline: application, infrastructure, or audit.

   9

   Specify the name of the output to use when forwarding logs with this pipeline.

   10

   Optional: Specify the default output to forward logs to the internal Elasticsearch instance.

   11

   Optional: String. One or more labels to add to the logs.

   12

   Optional: Configure multiple outputs to forward logs to other external log aggregators of any supported type:

   • A name to describe the pipeline.
   • The

     inputRefs

     is the log type to forward by using the pipeline:

     application,

     infrastructure

     , or

     audit

     .
   • The

     outputRefs

     is the name of the output to use.
   • Optional: String. One or more labels to add to the logs.

2. Create the CR object:

   $ oc create -f <file-name>.yaml

Procedure

1. Create or edit a YAML file that defines the

   ClusterLogForwarder

   CR object:

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: <log_forwarder_name> 

   1

   
     namespace: <log_forwarder_namespace> 

   2

   
   spec:
     serviceAccountName: <service_account_name> 

   3

   
     outputs:
      - name: rsyslog-east 

   4

   
        type: syslog 

   5

   
        syslog: 

   6

   
          facility: local0
          rfc: RFC3164
          payloadKey: message
          severity: informational
        url: 'tls://rsyslogserver.east.example.com:514' 

   7

   
        secret: 

   8

   
           name: syslog-secret
      - name: rsyslog-west
        type: syslog
        syslog:
         appName: myapp
         facility: user
         msgID: mymsg
         procID: myproc
         rfc: RFC5424
         severity: debug
        url: 'tcp://rsyslogserver.west.example.com:514'
     pipelines:
      - name: syslog-east 

   9

   
        inputRefs: 

   10

   
        - audit
        - application
        outputRefs: 

   11

   
        - rsyslog-east
        - default 

   12

   
        labels:
          secure: "true" 

   13

   
          syslog: "east"
      - name: syslog-west 

   14

   
        inputRefs:
        - infrastructure
        outputRefs:
        - rsyslog-west
        - default
        labels:
          syslog: "west"

   1

   In legacy implementations, the CR name must be instance. In multi log forwarder implementations, you can use any name.

   2

   In legacy implementations, the CR namespace must be openshift-logging. In multi log forwarder implementations, you can use any namespace.

   3

   The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the openshift-logging namespace.

   4

   Specify a name for the output.

   5

   Specify the syslog type.

   6

   Optional: Specify the syslog parameters, listed below.

   7

   Specify the URL and port of the external syslog instance. You can use the udp (insecure), tcp (insecure) or tls (secure TCP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP address.

   8

   If using a tls prefix, you must specify the name of the secret required by the endpoint for TLS communication. The secret must contain a ca-bundle.crt key that points to the certificate it represents. In legacy implementations, the secret must exist in the openshift-logging project.

   9

   Optional: Specify a name for the pipeline.

   10

   Specify which log types to forward by using the pipeline: application, infrastructure, or audit.

   11

   Specify the name of the output to use when forwarding logs with this pipeline.

   12

   Optional: Specify the default output to forward logs to the internal Elasticsearch instance.

   13

   Optional: String. One or more labels to add to the logs. Quote values like "true" so they are recognized as string values, not as a boolean.

   14

   Optional: Configure multiple outputs to forward logs to other external log aggregators of any supported type:

   • A name to describe the pipeline.
   • The

     inputRefs

     is the log type to forward by using the pipeline:

     application,

     infrastructure

     , or

     audit

     .
   • The

     outputRefs

     is the name of the output to use.
   • Optional: String. One or more labels to add to the logs.

2. Create the CR object:

   $ oc create -f <filename>.yaml

Procedure

1. Create or edit a YAML file that defines the

   ClusterLogForwarder

   CR object:

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: <log_forwarder_name> 

   1

   
     namespace: <log_forwarder_namespace> 

   2

   
   spec:
     serviceAccountName: <service_account_name> 

   3

   
     outputs:
      - name: app-logs 

   4

   
        type: kafka 

   5

   
        url: tls://kafka.example.devlab.com:9093/app-topic 

   6

   
        secret:
          name: kafka-secret 

   7

   
      - name: infra-logs
        type: kafka
        url: tcp://kafka.devlab2.example.com:9093/infra-topic 

   8

   
      - name: audit-logs
        type: kafka
        url: tls://kafka.qelab.example.com:9093/audit-topic
        secret:
           name: kafka-secret-qe
     pipelines:
      - name: app-topic 

   9

   
        inputRefs: 

   10

   
        - application
        outputRefs: 

   11

   
        - app-logs
        labels:
          logType: "application" 

   12

   
      - name: infra-topic 

   13

   
        inputRefs:
        - infrastructure
        outputRefs:
        - infra-logs
        labels:
          logType: "infra"
      - name: audit-topic
        inputRefs:
        - audit
        outputRefs:
        - audit-logs
        labels:
          logType: "audit"

   1

   In legacy implementations, the CR name must be instance. In multi log forwarder implementations, you can use any name.

   2

   In legacy implementations, the CR namespace must be openshift-logging. In multi log forwarder implementations, you can use any namespace.

   3

   The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the openshift-logging namespace.

   4

   Specify a name for the output.

   5

   Specify the kafka type.

   6

   Specify the URL and port of the Kafka broker as a valid absolute URL, optionally with a specific topic. You can use the tcp (insecure) or tls (secure TCP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP address.

   7

   If you are using a tls prefix, you must specify the name of the secret required by the endpoint for TLS communication. The secret must contain a ca-bundle.crt key that points to the certificate it represents. In legacy implementations, the secret must exist in the openshift-logging project.

   8

   Optional: To send an insecure output, use a tcp prefix in front of the URL. Also omit the secret key and its name from this output.

   9

   Optional: Specify a name for the pipeline.

   10

   Specify which log types to forward by using the pipeline: application, infrastructure, or audit.

   11

   Specify the name of the output to use when forwarding logs with this pipeline.

   12

   Optional: String. One or more labels to add to the logs.

   13

   Optional: Configure multiple outputs to forward logs to other external log aggregators of any supported type:

   • A name to describe the pipeline.
   • The

     inputRefs

     is the log type to forward by using the pipeline:

     application,

     infrastructure

     , or

     audit

     .
   • The

     outputRefs

     is the name of the output to use.
   • Optional: String. One or more labels to add to the logs.

2. Optional: To forward a single output to multiple Kafka brokers, specify an array of Kafka brokers as shown in the following example:

   # ...
   spec:
     outputs:
     - name: app-logs
       type: kafka
       secret:
         name: kafka-secret-dev
       kafka:  

   1

   
         brokers: 

   2

   
           - tls://kafka-broker1.example.com:9093/
           - tls://kafka-broker2.example.com:9093/
         topic: app-topic 

   3

   
   # ...

   1

   Specify a kafka key that has a brokers and topic key.

   2

   Use the brokers key to specify an array of one or more brokers.

   3

   Use the topic key to specify the target topic that receives the logs.

3. Apply the

   ClusterLogForwarder

   CR by running the following command:

   $ oc apply -f <filename>.yaml

Procedure

1. Create a

   Secret

   YAML file that uses the

   aws_access_key_id

   and

   aws_secret_access_key

   fields to specify your base64-encoded AWS credentials. For example:

   apiVersion: v1
   kind: Secret
   metadata:
     name: cw-secret
     namespace: openshift-logging
   data:
     aws_access_key_id: QUtJQUlPU0ZPRE5ON0VYQU1QTEUK
     aws_secret_access_key: d0phbHJYVXRuRkVNSS9LN01ERU5HL2JQeFJmaUNZRVhBTVBMRUtFWQo=

2. Create the secret. For example:

   $ oc apply -f cw-secret.yaml

3. Create or edit a YAML file that defines the

   ClusterLogForwarder

   CR object. In the file, specify the name of the secret. For example:

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: <log_forwarder_name> 

   1

   
     namespace: <log_forwarder_namespace> 

   2

   
   spec:
     serviceAccountName: <service_account_name> 

   3

   
     outputs:
      - name: cw 

   4

   
        type: cloudwatch 

   5

   
        cloudwatch:
          groupBy: logType 

   6

   
          groupPrefix: <group prefix> 

   7

   
          region: us-east-2 

   8

   
        secret:
           name: cw-secret 

   9

   
     pipelines:
       - name: infra-logs 

   10

   
         inputRefs: 

   11

   
           - infrastructure
           - audit
           - application
         outputRefs:
           - cw 

   12

   1

   In legacy implementations, the CR name must be instance. In multi log forwarder implementations, you can use any name.

   2

   In legacy implementations, the CR namespace must be openshift-logging. In multi log forwarder implementations, you can use any namespace.

   3

   The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the openshift-logging namespace.

   4

   Specify a name for the output.

   5

   Specify the cloudwatch type.

   6

   Optional: Specify how to group the logs:

   • logType

     creates log groups for each log type.

   • namespaceName

     creates a log group for each application name space. It also creates separate log groups for infrastructure and audit logs.

   • namespaceUUID

     creates a new log groups for each application namespace UUID. It also creates separate log groups for infrastructure and audit logs.

   7

   Optional: Specify a string to replace the default infrastructureName prefix in the names of the log groups.

   8

   Specify the AWS region.

   9

   Specify the name of the secret that contains your AWS credentials.

   10

   Optional: Specify a name for the pipeline.

   11

   Specify which log types to forward by using the pipeline: application, infrastructure, or audit.

   12

   Specify the name of the output to use when forwarding logs with this pipeline.

4. Create the CR object:

   $ oc create -f <file-name>.yaml

Procedure

1. Create a

   CredentialsRequest

   custom resource YAML by using the template below:

   CloudWatch credentials request template

   apiVersion: cloudcredential.openshift.io/v1
   kind: CredentialsRequest
   metadata:
     name: <your_role_name>-credrequest
     namespace: openshift-cloud-credential-operator
   spec:
     providerSpec:
       apiVersion: cloudcredential.openshift.io/v1
       kind: AWSProviderSpec
       statementEntries:
         - action:
             - logs:PutLogEvents
             - logs:CreateLogGroup
             - logs:PutRetentionPolicy
             - logs:CreateLogStream
             - logs:DescribeLogGroups
             - logs:DescribeLogStreams
           effect: Allow
           resource: arn:aws:logs:*:*:*
     secretRef:
       name: <your_role_name>
       namespace: openshift-logging
     serviceAccountNames:
       - logcollector

2. Use the

   ccoctl

   command to create a role for AWS using your

   CredentialsRequest

   CR. With the

   CredentialsRequest

   object, this

   ccoctl

   command creates an IAM role with a trust policy that is tied to the specified OIDC identity provider, and a permissions policy that grants permissions to perform operations on CloudWatch resources. This command also creates a YAML configuration file in

   /<path_to_ccoctl_output_dir>/manifests/openshift-logging-<your_role_name>-credentials.yaml

   . This secret file contains the

   role_arn

   key/value used during authentication with the AWS IAM identity provider.

   $ ccoctl aws create-iam-roles \
   --name=<name> \
   --region=<aws_region> \
   --credentials-requests-dir=<path_to_directory_with_list_of_credentials_requests>/credrequests \
   --identity-provider-arn=arn:aws:iam::<aws_account_id>:oidc-provider/<name>-oidc.s3.<aws_region>.amazonaws.com 

   1

   1

   <name> is the name used to tag your cloud resources and should match the name used during your STS cluster install

3. Apply the secret created:

   $ oc apply -f output/manifests/openshift-logging-<your_role_name>-credentials.yaml

4. Create or edit a

   ClusterLogForwarder

   custom resource:

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: <log_forwarder_name> 

   1

   
     namespace: <log_forwarder_namespace> 

   2

   
   spec:
     serviceAccountName: clf-collector 

   3

   
     outputs:
      - name: cw 

   4

   
        type: cloudwatch 

   5

   
        cloudwatch:
          groupBy: logType 

   6

   
          groupPrefix: <group prefix> 

   7

   
          region: us-east-2 

   8

   
        secret:
           name: <your_secret_name> 

   9

   
     pipelines:
       - name: to-cloudwatch 

   10

   
         inputRefs: 

   11

   
           - infrastructure
           - audit
           - application
         outputRefs:
           - cw 

   12

   1

   In legacy implementations, the CR name must be instance. In multi log forwarder implementations, you can use any name.

   2

   In legacy implementations, the CR namespace must be openshift-logging. In multi log forwarder implementations, you can use any namespace.

   3

   Specify the clf-collector service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the openshift-logging namespace.

   4

   Specify a name for the output.

   5

   Specify the cloudwatch type.

   6

   Optional: Specify how to group the logs:

   • logType

     creates log groups for each log type.

   • namespaceName

     creates a log group for each application name space. Infrastructure and audit logs are unaffected, remaining grouped by

     logType

     .

   • namespaceUUID

     creates a new log groups for each application namespace UUID. It also creates separate log groups for infrastructure and audit logs.

   7

   Optional: Specify a string to replace the default infrastructureName prefix in the names of the log groups.

   8

   Specify the AWS region.

   9

   Specify the name of the secret that contains your AWS credentials.

   10

   Optional: Specify a name for the pipeline.

   11

   Specify which log types to forward by using the pipeline: application, infrastructure, or audit.

   12

   Specify the name of the output to use when forwarding logs with this pipeline.