Chapter 1. Configuring log forwarding

Procedure

1. Create a service account for the collector. If you want to write logs to storage that requires a token for authentication, you must include a token in the service account.

2. Bind the appropriate cluster roles to the service account:

   Example binding command

   $ oc adm policy add-cluster-role-to-user <cluster_role_name> system:serviceaccount:<namespace_name>:<service_account_name>

   Copy to Clipboard Toggle word wrap

Procedure

1. Extract the existing ClusterLogForwarder configuration and save it as a local file.

   $ oc get clusterlogforwarder <name> -n <namespace> -o yaml > <filename>.yaml

   Copy to Clipboard Toggle word wrap

   Where:

   • <name> is the name of the ClusterLogForwarder instance you want to configure.
   • <namespace> is the namespace where you created the ClusterLogForwarder instance, for example openshift-logging.
   • <filename> is the name of the local file where you save the configuration.

2. Add a configuration to drop unwanted log records to the filters spec in the ClusterLogForwarder CR.

   Example ClusterLogForwarder CR

   apiVersion: observability.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: instance
     namespace: openshift-logging
   spec:
     # ...
     filters:
     - name: drop-filter
       type: drop 

   1

   
       drop: 

   2

   
       - test: 

   3

   
         - field: .kubernetes.labels."app.version-1.2/beta" 

   4

   
           matches: .+ 

   5

   
         - field: .kubernetes.pod_name
           notMatches: "my-pod" 

   6

   
     pipelines:
     - name: my-pipeline 

   7

   
       filterRefs:
       - drop-filter
     # ...

   Copy to Clipboard Toggle word wrap

   1

   Specify the type of filter. The drop filter drops log records that match the filter configuration.

   2

   Specify configuration options for the drop filter.

   3

   Specify conditions for tests to evaluate whether the filter drops a log record.

   4

   Specify dot-delimited paths to fields in log records.

   • Each path segment can contain alphanumeric characters and underscores, a-z, A-Z, 0-9, _, for example, .kubernetes.namespace_name.
   • If segments contain different characters, the segment must be in quotes, for example, .kubernetes.labels."app.version-1.2/beta".
   • You can include several field paths in a single test configuration, but they must all evaluate to true for the test to pass and the drop filter to apply.

   5

   Specify a regular expression. If log records match this regular expression, they are dropped.

   6

   Specify a regular expression. If log records do not match this regular expression, they are dropped.

   7

   Specify the pipeline that uses the drop filter.

   Note

   You can set either the matches or notMatches condition for a single field path, but not both.

   Example configuration that keeps only high-priority log records

   # ...
   filters:
   - name: important
     type: drop
     drop:
     - test:
       - field: .message
         notMatches: "(?i)critical|error"
       - field: .level
         matches: "info|warning"
   # ...

   Copy to Clipboard Toggle word wrap

   Example configuration with several tests

   # ...
   filters:
   - name: important
     type: drop
     drop:
     - test: 

   1

   
       - field: .kubernetes.namespace_name
         matches: "openshift.*"
     - test: 

   2

   
       - field: .log_type
         matches: "application"
       - field: .kubernetes.pod_name
         notMatches: "my-pod"
   # ...

   Copy to Clipboard Toggle word wrap

   1

   The filter drops logs that contain a namespace that starts with openshift.

   2

   The filter drops application logs that do not have my-pod in the pod name.

3. Apply the ClusterLogForwarder CR by running the following command:

   $ oc apply -f <filename>.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Add a configuration for a filter to the input spec in the ClusterLogForwarder CR.

   The following example shows how to configure the ClusterLogForwarder CR to include logs based on label expressions or matched label key/values:

   Example ClusterLogForwarder CR

   apiVersion: observability.openshift.io/v1
   kind: ClusterLogForwarder
   # ...
   spec:
     serviceAccount:
       name: <service_account_name>
     inputs:
       - name: mylogs
         application:
           selector:
             matchExpressions:
             - key: env 

   1

   
               operator: In 

   2

   
               values: ["prod", "qa"] 

   3

   
             - key: zone
               operator: NotIn
               values: ["east", "west"]
             matchLabels: 

   4

   
               app: one
               name: app1
         type: application
   # ...

   Copy to Clipboard Toggle word wrap

   1

   Specifies the label key to match.

   2

   Specifies the operator. Valid values include: In, NotIn, Exists, and DoesNotExist.

   3

   Specifies an array of string values. If the operator value is either Exists or DoesNotExist, the value array must be empty.

   4

   Specifies an exact key or value mapping.

2. Apply the ClusterLogForwarder CR by running the following command:

   $ oc apply -f <filename>.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Extract the existing ClusterLogForwarder configuration and save it as a local file.

   $ oc get clusterlogforwarder <name> -n <namespace> -o yaml > <filename>.yaml

   Copy to Clipboard Toggle word wrap

   Where:

   • <name> is the name of the ClusterLogForwarder instance you want to configure.
   • <namespace> is the namespace where you created the ClusterLogForwarder instance, for example openshift-logging.
   • <filename> is the name of the local file where you save the configuration.

2. Add a configuration to prune log records to the filters spec in the ClusterLogForwarder CR.

   Important

   If you specify both in and notIn parameters, the notIn array takes precedence over in during pruning. After records are pruned by using the notIn array, they are then pruned by using the in array.

   Example ClusterLogForwarder CR

   apiVersion: observability.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: instance
     namespace: openshift-logging
   spec:
     serviceAccount:
       name: my-account
     filters:
     - name: prune-filter
       type: prune 

   1

   
       prune: 

   2

   
         in: [.kubernetes.annotations, .kubernetes.namespace_id] 

   3

   
         notIn: [.kubernetes,.log_type,.message,."@timestamp",.log_source] 

   4

   
     pipelines:
     - name: my-pipeline 

   5

   
       filterRefs: ["prune-filter"]
     # ...

   Copy to Clipboard Toggle word wrap

   1

   Specify the type of filter. The prune filter prunes log records by configured fields.

   2

   Specify configuration options for the prune filter.

   • The in and notIn fields are arrays of dot-delimited paths to fields in log records.
   • Each path segment can contain alpha-numeric characters and underscores, a-z, A-Z, 0-9, _, for example, .kubernetes.namespace_name.
   • If segments contain different characters, the segment must be in quotes, for example, .kubernetes.labels."app.version-1.2/beta".

   3

   Optional: Specify fields to remove from the log record. The log collector keeps all other fields.

   4

   Optional: Specify fields to keep in the log record. The log collector removes all other fields.

   5

   Specify the pipeline that the prune filter is applied to.

   Important

   • The filters cannot remove the .log_type, .log_source, .message fields from the log records. You must include them in the notIn field.
   • If you use the googleCloudLogging output, you must include .hostname in the notIn field.

3. Apply the ClusterLogForwarder CR by running the following command:

   $ oc apply -f <filename>.yaml

   Copy to Clipboard Toggle word wrap