Chapter 1. Configuring log forwarding

Procedure

1. Create a service account for the collector. If you want to write logs to storage that requires a token for authentication, you must include a token in the service account.

2. Bind the appropriate cluster roles to the service account:

   Example binding command

   $ oc adm policy add-cluster-role-to-user <cluster_role_name> system:serviceaccount:<namespace_name>:<service_account_name>

   Copy to Clipboard Toggle word wrap

Procedure

1. Create or edit a YAML file that defines the ClusterLogForwarder CR object:

   apiVersion: observability.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: collector
   spec:
     managementState: Managed
     outputs:
     - name: rsyslog-east 

   1

   
       syslog:
         appName: <app_name> 

   2

   
         enrichment: KubernetesMinimal
         facility: <facility_value> 

   3

   
         msgId: <message_ID> 

   4

   
         payloadKey: <record_field> 

   5

   
         procId: <process_ID> 

   6

   
         rfc: <RFC3164_or_RFC5424> 

   7

   
         severity: informational 

   8

   
         tuning:
           deliveryMode: <AtLeastOnce_or_AtMostOnce> 

   9

   
         url: <url> 

   10

   
       tls: 

   11

   
         ca:
           key: ca-bundle.crt
           secretName: syslog-secret
       type: syslog
     pipelines:
     - inputRefs: 

   12

   
       - application
       name: syslog-east 

   13

   
       outputRefs:
       - rsyslog-east
     serviceAccount: 

   14

   
       name: logcollector

   Copy to Clipboard Toggle word wrap

   1

   Specify a name for the output.

   2

   Optional: Specify the value for the APP-NAME part of the syslog message header. The value must conform with The Syslog Protocol. The value can be a combination of static and dynamic values consisting of field paths followed by ||, and then followed by another field path or a static value. The maximum length of the final values is truncated to 48 characters. You must encase a dynamic value curly brackets and the value must be followed with a static fallback value separated with ||. Static values can only contain alphanumeric characters along with dashes, underscores, dots and forward slashes. Example value: <value1>-{.<value2>||"none"}.

   3

   Optional: Specify the value for Facility part of the syslog-msg header.

   4

   Optional: Specify the value for MSGID part of the syslog-msg header. The value can be a combination of static and dynamic values consisting of field paths followed by ||, and then followed by another field path or a static value. The maximum length of the final values is truncated to 32 characters. You must encase a dynamic value curly brackets and the value must be followed with a static fallback value separated with ||. Static values can only contain alphanumeric characters along with dashes, underscores, dots and forward slashes. Example value: <value1>-{.<value2>||"none"}.

   5

   Optional: Specify the record field to use as the payload. The payloadKey value must be a single field path encased in single curly brackets {}. Example: {.<value>}.

   6

   Optional: Specify the value for the PROCID part of the syslog message header. The value must conform with The Syslog Protocol. The value can be a combination of static and dynamic values consisting of field paths followed by ||, and then followed by another field path or a static value. The maximum length of the final values is truncated to 48 characters. You must encase a dynamic value curly brackets and the value must be followed with a static fallback value separated with ||. Static values can only contain alphanumeric characters along with dashes, underscores, dots and forward slashes. Example value: <value1>-{.<value2>||"none"}.

   7

   Optional: Set the RFC that the generated messages conform to. The value can be RFC3164 or RFC5424.

   8

   Optional: Set the severity level for the message. For more information, see The Syslog Protocol.

   9

   Optional: Set the delivery mode for log forwarding. The value can be either AtLeastOnce, or AtMostOnce.

   10

   Specify the absolute URL with a scheme. Valid schemes are: tcp, tls, and udp. For example: tls://syslog-receiver.example.com:6514.

   11

   Specify the settings for controlling options of the transport layer security (TLS) client connections.

   12

   Specify which log types to forward by using the pipeline: application, infrastructure, or audit.

   13

   Specify a name for the pipeline.

   14

   The name of your service account.

2. Create the CR object:

   $ oc create -f <filename>.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Create an IAM policy that grants permissions to write logs to CloudWatch.

   1. Create a file, for example cw-iam-role-policy.json, with the following content:

      {
          "Version": "2012-10-17",
          "Statement": [
              {
                  "Effect": "Allow",
                  "Action": [
                      "logs:PutLogEvents",
                      "logs:CreateLogGroup",
                      "logs:PutRetentionPolicy",
                      "logs:CreateLogStream",
                      "logs:DescribeLogGroups",
                      "logs:DescribeLogStreams"
                  ],
                  "Resource": "arn:aws:logs:*:*:*"
              }
          ]
      }

      Copy to Clipboard Toggle word wrap

   2. Create the IAM policy based on the previous policy definition by running the following command:

      aws iam create-policy \
          --policy-name cluster-logging-allow \
          --policy-document file://cw-iam-role-policy.json

      Copy to Clipboard Toggle word wrap

      Note the Arn value of the created policy.

2. Create a trust policy to allow the logging service account to assume an IAM role:

   1. Create a file, for example cw-trust-policy.json, with the following content:

      {
      "Version": "2012-10-17",
      "Statement": [
          {
              "Effect": "Allow",
              "Principal": {
                  "Federated": "arn:aws:iam::123456789012:oidc-provider/<OPENSHIFT_OIDC_PROVIDER_URL>" 

      1

      
              },
              "Action": "sts:AssumeRoleWithWebIdentity",
              "Condition": {
                  "StringEquals": {
                      "<OPENSHIFT_OIDC_PROVIDER_URL>:sub": "system:serviceaccount:openshift-logging:logcollector" 

      2

      
                  }
              }
          }
      ]
      }

      Copy to Clipboard Toggle word wrap

      1

      Replace <OPENSHIFT_OIDC_PROVIDER_URL> with the URL of your Red Hat OpenShift Logging OIDC URL.

      2

      The namespace and service account must match the namespace and service account that the log forwarder uses.

3. Create an IAM role based on the previously defined trust policy by running the following command:

   $ aws iam create-role --role-name openshift-logger --assume-role-policy-document file://cw-trust-policy.json

   Copy to Clipboard Toggle word wrap

   Note the Arn value of the created role.

4. Attach the policy to the role by running the following command:

   $ aws iam put-role-policy \
         --role-name openshift-logger --policy-name cluster-logging-allow \
         --policy-document file://cw-role-policy.json

   Copy to Clipboard Toggle word wrap

Procedure

1. Extract the existing ClusterLogForwarder configuration and save it as a local file.

   $ oc get clusterlogforwarder <name> -n <namespace> -o yaml > <filename>.yaml

   Copy to Clipboard Toggle word wrap

   Where:

   • <name> is the name of the ClusterLogForwarder instance you want to configure.
   • <namespace> is the namespace where you created the ClusterLogForwarder instance, for example openshift-logging.
   • <filename> is the name of the local file where you save the configuration.

2. Add a configuration to drop unwanted log records to the filters spec in the ClusterLogForwarder CR.

   Example ClusterLogForwarder CR

   apiVersion: observability.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: instance
     namespace: openshift-logging
   spec:
     # ...
     filters:
     - name: drop-filter
       type: drop 

   1

   
       drop: 

   2

   
       - test: 

   3

   
         - field: .kubernetes.labels."app.version-1.2/beta" 

   4

   
           matches: .+ 

   5

   
         - field: .kubernetes.pod_name
           notMatches: "my-pod" 

   6

   
     pipelines:
     - name: my-pipeline 

   7

   
       filterRefs:
       - drop-filter
     # ...

   Copy to Clipboard Toggle word wrap

   1

   Specify the type of filter. The drop filter drops log records that match the filter configuration.

   2

   Specify configuration options for the drop filter.

   3

   Specify conditions for tests to evaluate whether the filter drops a log record.

   4

   Specify dot-delimited paths to fields in log records.

   • Each path segment can contain alphanumeric characters and underscores, a-z, A-Z, 0-9, _, for example, .kubernetes.namespace_name.
   • If segments contain different characters, the segment must be in quotes, for example, .kubernetes.labels."app.version-1.2/beta".
   • You can include several field paths in a single test configuration, but they must all evaluate to true for the test to pass and the drop filter to apply.

   5

   Specify a regular expression. If log records match this regular expression, they are dropped.

   6

   Specify a regular expression. If log records do not match this regular expression, they are dropped.

   7

   Specify the pipeline that uses the drop filter.

   Note

   You can set either the matches or notMatches condition for a single field path, but not both.

   Example configuration that keeps only high-priority log records

   # ...
   filters:
   - name: important
     type: drop
     drop:
     - test:
       - field: .message
         notMatches: "(?i)critical|error"
       - field: .level
         matches: "info|warning"
   # ...

   Copy to Clipboard Toggle word wrap

   Example configuration with several tests

   # ...
   filters:
   - name: important
     type: drop
     drop:
     - test: 

   1

   
       - field: .kubernetes.namespace_name
         matches: "openshift.*"
     - test: 

   2

   
       - field: .log_type
         matches: "application"
       - field: .kubernetes.pod_name
         notMatches: "my-pod"
   # ...

   Copy to Clipboard Toggle word wrap

   1

   The filter drops logs that contain a namespace that starts with openshift.

   2

   The filter drops application logs that do not have my-pod in the pod name.

3. Apply the ClusterLogForwarder CR by running the following command:

   $ oc apply -f <filename>.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Add a configuration for a filter to the input spec in the ClusterLogForwarder CR.

   The following example shows how to configure the ClusterLogForwarder CR to include logs based on label expressions or matched label key/values:

   Example ClusterLogForwarder CR

   apiVersion: observability.openshift.io/v1
   kind: ClusterLogForwarder
   # ...
   spec:
     serviceAccount:
       name: <service_account_name>
     inputs:
       - name: mylogs
         application:
           selector:
             matchExpressions:
             - key: env 

   1

   
               operator: In 

   2

   
               values: ["prod", "qa"] 

   3

   
             - key: zone
               operator: NotIn
               values: ["east", "west"]
             matchLabels: 

   4

   
               app: one
               name: app1
         type: application
   # ...

   Copy to Clipboard Toggle word wrap

   1

   Specifies the label key to match.

   2

   Specifies the operator. Valid values include: In, NotIn, Exists, and DoesNotExist.

   3

   Specifies an array of string values. If the operator value is either Exists or DoesNotExist, the value array must be empty.

   4

   Specifies an exact key or value mapping.

2. Apply the ClusterLogForwarder CR by running the following command:

   $ oc apply -f <filename>.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Extract the existing ClusterLogForwarder configuration and save it as a local file.

   $ oc get clusterlogforwarder <name> -n <namespace> -o yaml > <filename>.yaml

   Copy to Clipboard Toggle word wrap

   Where:

   • <name> is the name of the ClusterLogForwarder instance you want to configure.
   • <namespace> is the namespace where you created the ClusterLogForwarder instance, for example openshift-logging.
   • <filename> is the name of the local file where you save the configuration.

2. Add a configuration to prune log records to the filters spec in the ClusterLogForwarder CR.

   Important

   If you specify both in and notIn parameters, the notIn array takes precedence over in during pruning. After records are pruned by using the notIn array, they are then pruned by using the in array.

   Example ClusterLogForwarder CR

   apiVersion: observability.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: instance
     namespace: openshift-logging
   spec:
     serviceAccount:
       name: my-account
     filters:
     - name: prune-filter
       type: prune 

   1

   
       prune: 

   2

   
         in: [.kubernetes.annotations, .kubernetes.namespace_id] 

   3

   
         notIn: [.kubernetes,.log_type,.message,."@timestamp",.log_source] 

   4

   
     pipelines:
     - name: my-pipeline 

   5

   
       filterRefs: ["prune-filter"]
     # ...

   Copy to Clipboard Toggle word wrap

   1

   Specify the type of filter. The prune filter prunes log records by configured fields.

   2

   Specify configuration options for the prune filter.

   • The in and notIn fields are arrays of dot-delimited paths to fields in log records.
   • Each path segment can contain alpha-numeric characters and underscores, a-z, A-Z, 0-9, _, for example, .kubernetes.namespace_name.
   • If segments contain different characters, the segment must be in quotes, for example, .kubernetes.labels."app.version-1.2/beta".

   3

   Optional: Specify fields to remove from the log record. The log collector keeps all other fields.

   4

   Optional: Specify fields to keep in the log record. The log collector removes all other fields.

   5

   Specify the pipeline that the prune filter is applied to.

   Important

   • The filters cannot remove the .log_type, .log_source, .message fields from the log records. You must include them in the notIn field.
   • If you use the googleCloudLogging output, you must include .hostname in the notIn field.

3. Apply the ClusterLogForwarder CR by running the following command:

   $ oc apply -f <filename>.yaml

   Copy to Clipboard Toggle word wrap