Logging

Procedure

1. Create or edit a YAML file that defines the ClusterLogForwarder CR object:

   apiVersion: observability.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: collector
   spec:
     managementState: Managed
     outputs:
     - name: rsyslog-east 

   1

   
       syslog:
         appName: <app_name> 

   2

   
         enrichment: KubernetesMinimal
         facility: <facility_value> 

   3

   
         msgId: <message_ID> 

   4

   
         payloadKey: <record_field> 

   5

   
         procId: <process_ID> 

   6

   
         rfc: <RFC3164_or_RFC5424> 

   7

   
         severity: informational 

   8

   
         tuning:
           deliveryMode: <AtLeastOnce_or_AtMostOnce> 

   9

   
         url: <url> 

   10

   
       tls: 

   11

   
         ca:
           key: ca-bundle.crt
           secretName: syslog-secret
       type: syslog
     pipelines:
     - inputRefs: 

   12

   
       - application
       name: syslog-east 

   13

   
       outputRefs:
       - rsyslog-east
     serviceAccount: 

   14

   
       name: logcollector

   Copy to Clipboard Toggle word wrap

   1

   Specify a name for the output.

   2

   Optional: Specify the value for the APP-NAME part of the syslog message header. The value must conform with The Syslog Protocol. The value can be a combination of static and dynamic values consisting of field paths followed by ||, and then followed by another field path or a static value. The maximum length of the final values is truncated to 48 characters. You must encase a dynamic value curly brackets and the value must be followed with a static fallback value separated with ||. Static values can only contain alphanumeric characters along with dashes, underscores, dots and forward slashes. Example value: <value1>-{.<value2>||"none"}.

   3

   Optional: Specify the value for Facility part of the syslog-msg header.

   4

   Optional: Specify the value for MSGID part of the syslog-msg header. The value can be a combination of static and dynamic values consisting of field paths followed by ||, and then followed by another field path or a static value. The maximum length of the final values is truncated to 32 characters. You must encase a dynamic value curly brackets and the value must be followed with a static fallback value separated with ||. Static values can only contain alphanumeric characters along with dashes, underscores, dots and forward slashes. Example value: <value1>-{.<value2>||"none"}.

   5

   Optional: Specify the record field to use as the payload. The payloadKey value must be a single field path encased in single curly brackets {}. Example: {.<value>}.

   6

   Optional: Specify the value for the PROCID part of the syslog message header. The value must conform with The Syslog Protocol. The value can be a combination of static and dynamic values consisting of field paths followed by ||, and then followed by another field path or a static value. The maximum length of the final values is truncated to 48 characters. You must encase a dynamic value curly brackets and the value must be followed with a static fallback value separated with ||. Static values can only contain alphanumeric characters along with dashes, underscores, dots and forward slashes. Example value: <value1>-{.<value2>||"none"}.

   7

   Optional: Set the RFC that the generated messages conform to. The value can be RFC3164 or RFC5424.

   8

   Optional: Set the severity level for the message. For more information, see The Syslog Protocol.

   9

   Optional: Set the delivery mode for log forwarding. The value can be either AtLeastOnce, or AtMostOnce.

   10

   Specify the absolute URL with a scheme. Valid schemes are: tcp, tls, and udp. For example: tls://syslog-receiver.example.com:6514.

   11

   Specify the settings for controlling options of the transport layer security (TLS) client connections.

   12

   Specify which log types to forward by using the pipeline: application, infrastructure, or audit.

   13

   Specify a name for the pipeline.

   14

   The name of your service account.

2. Create the CR object:

   $ oc create -f <filename>.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Add a configuration for a filter to the filters spec in the ClusterLogForwarder CR.

   The following example shows how to configure the ClusterLogForwarder CR to drop log records based on regular expressions:

   Example ClusterLogForwarder CR

   apiVersion: observability.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
   # ...
   spec:
     serviceAccount:
       name: <service_account_name>
     filters:
     - name: <filter_name>
       type: drop 

   1

   
       drop: 

   2

   
       - test: 

   3

   
         - field: .kubernetes.labels."foo-bar/baz" 

   4

   
           matches: .+ 

   5

   
         - field: .kubernetes.pod_name
           notMatches: "my-pod" 

   6

   
     pipelines:
     - name: <pipeline_name> 

   7

   
       filterRefs: ["<filter_name>"]
   # ...

   Copy to Clipboard Toggle word wrap

   1

   Specifies the type of filter. The drop filter drops log records that match the filter configuration.

   2

   Specifies configuration options for applying the drop filter.

   3

   Specifies the configuration for tests that are used to evaluate whether a log record is dropped.

   • If all the conditions specified for a test are true, the test passes and the log record is dropped.
   • When multiple tests are specified for the drop filter configuration, if any of the tests pass, the record is dropped.
   • If there is an error evaluating a condition, for example, the field is missing from the log record being evaluated, that condition evaluates to false.

   4

   Specifies a dot-delimited field path, which is a path to a field in the log record. The path can contain alpha-numeric characters and underscores (a-zA-Z0-9_), for example, .kubernetes.namespace_name. If segments contain characters outside of this range, the segment must be in quotes, for example, .kubernetes.labels."foo.bar-bar/baz". You can include multiple field paths in a single test configuration, but they must all evaluate to true for the test to pass and the drop filter to be applied.

   5

   Specifies a regular expression. If log records match this regular expression, they are dropped. You can set either the matches or notMatches condition for a single field path, but not both.

   6

   Specifies a regular expression. If log records do not match this regular expression, they are dropped. You can set either the matches or notMatches condition for a single field path, but not both.

   7

   Specifies the pipeline that the drop filter is applied to.

2. Apply the ClusterLogForwarder CR by running the following command:

   $ oc apply -f <filename>.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Add a configuration for a filter to the input spec in the ClusterLogForwarder CR.

   The following example shows how to configure the ClusterLogForwarder CR to include logs based on label expressions or matched label key/values:

   Example ClusterLogForwarder CR

   apiVersion: observability.openshift.io/v1
   kind: ClusterLogForwarder
   # ...
   spec:
     serviceAccount:
       name: <service_account_name>
     inputs:
       - name: mylogs
         application:
           selector:
             matchExpressions:
             - key: env 

   1

   
               operator: In 

   2

   
               values: ["prod", "qa"] 

   3

   
             - key: zone
               operator: NotIn
               values: ["east", "west"]
             matchLabels: 

   4

   
               app: one
               name: app1
         type: application
   # ...

   Copy to Clipboard Toggle word wrap

   1

   Specifies the label key to match.

   2

   Specifies the operator. Valid values include: In, NotIn, Exists, and DoesNotExist.

   3

   Specifies an array of string values. If the operator value is either Exists or DoesNotExist, the value array must be empty.

   4

   Specifies an exact key or value mapping.

2. Apply the ClusterLogForwarder CR by running the following command:

   $ oc apply -f <filename>.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Add a configuration for a filter to the prune spec in the ClusterLogForwarder CR.

   The following example shows how to configure the ClusterLogForwarder CR to prune log records based on field paths:

   Important

   If both are specified, records are pruned based on the notIn array first, which takes precedence over the in array. After records have been pruned by using the notIn array, they are then pruned by using the in array.

   Example ClusterLogForwarder CR

   apiVersion: observability.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
   # ...
   spec:
     serviceAccount:
       name: <service_account_name>
     filters:
     - name: <filter_name>
       type: prune 

   1

   
       prune: 

   2

   
         in: [.kubernetes.annotations, .kubernetes.namespace_id] 

   3

   
         notIn: [.kubernetes,.log_type,.message,."@timestamp"] 

   4

   
     pipelines:
     - name: <pipeline_name> 

   5

   
       filterRefs: ["<filter_name>"]
   # ...

   Copy to Clipboard Toggle word wrap

   1

   Specify the type of filter. The prune filter prunes log records by configured fields.

   2

   Specify configuration options for applying the prune filter. The in and notIn fields are specified as arrays of dot-delimited field paths, which are paths to fields in log records. These paths can contain alpha-numeric characters and underscores (a-zA-Z0-9_), for example, .kubernetes.namespace_name. If segments contain characters outside of this range, the segment must be in quotes, for example, .kubernetes.labels."foo.bar-bar/baz".

   3

   Optional: Any fields that are specified in this array are removed from the log record.

   4

   Optional: Any fields that are not specified in this array are removed from the log record.

   5

   Specify the pipeline that the prune filter is applied to.

   Note

   The filters exempts the log_type, .log_source, and .message fields.

2. Apply the ClusterLogForwarder CR by running the following command:

   $ oc apply -f <filename>.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Add a configuration to define the audit and infrastructure sources in the ClusterLogForwarder CR.

   The following example shows how to configure the ClusterLogForwarder CR to define audit and infrastructure sources:

   Example ClusterLogForwarder CR

   apiVersion: observability.openshift.io/v1
   kind: ClusterLogForwarder
   # ...
   spec:
     serviceAccount:
       name: <service_account_name>
     inputs:
       - name: mylogs1
         type: infrastructure
         infrastructure:
           sources: 

   1

   
             - node
       - name: mylogs2
         type: audit
         audit:
           sources: 

   2

   
             - kubeAPI
             - openshiftAPI
             - ovn
   # ...

   Copy to Clipboard Toggle word wrap

   1

   Specifies the list of infrastructure sources to collect. The valid sources include:

   • node: Journal log from the node
   • container: Logs from the workloads deployed in the namespaces

   2

   Specifies the list of audit sources to collect. The valid sources include:

   • kubeAPI: Logs from the Kubernetes API servers
   • openshiftAPI: Logs from the OpenShift API servers
   • auditd: Logs from a node auditd service
   • ovn: Logs from an open virtual network service

2. Apply the ClusterLogForwarder CR by running the following command:

   $ oc apply -f <filename>.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Add a configuration to include or exclude the namespace and container names in the ClusterLogForwarder CR.

   The following example shows how to configure the ClusterLogForwarder CR to include or exclude namespaces and container names:

   Example ClusterLogForwarder CR

   apiVersion: observability.openshift.io/v1
   kind: ClusterLogForwarder
   # ...
   spec:
     serviceAccount:
       name: <service_account_name>
     inputs:
       - name: mylogs
         application:
           includes:
             - namespace: "my-project" 

   1

   
               container: "my-container" 

   2

   
           excludes:
             - container: "other-container*" 

   3

   
               namespace: "other-namespace" 

   4

   
         type: application
   # ...

   Copy to Clipboard Toggle word wrap

   1

   Specifies that the logs are only collected from these namespaces.

   2

   Specifies that the logs are only collected from these containers.

   3

   Specifies the pattern of namespaces to ignore when collecting the logs.

   4

   Specifies the set of containers to ignore when collecting the logs.

   Note

   The excludes field takes precedence over the includes field.

2. Apply the ClusterLogForwarder CR by running the following command:

   $ oc apply -f <filename>.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a LogFileMetricExporter CR as a YAML file:

   Example LogFileMetricExporter CR

   apiVersion: logging.openshift.io/v1alpha1
   kind: LogFileMetricExporter
   metadata:
     name: instance
     namespace: openshift-logging
   spec:
     nodeSelector: {} 

   1

   
     resources: 

   2

   
       limits:
         cpu: 500m
         memory: 256Mi
       requests:
         cpu: 200m
         memory: 128Mi
     tolerations: [] 

   3

   
   # ...

   Copy to Clipboard Toggle word wrap

   1

   Optional: The nodeSelector stanza defines which pods are scheduled on which nodes.

   2

   The resources stanza defines resource requirements for the LogFileMetricExporter CR.

   3

   Optional: The tolerations stanza defines the tolerations that the pods accept.

2. Apply the LogFileMetricExporter CR by running the following command:

   $ oc apply -f <filename>.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Create an AlertingRule custom resource (CR):

   Example infrastructure AlertingRule CR

     apiVersion: loki.grafana.com/v1
     kind: AlertingRule
     metadata:
       name: loki-operator-alerts
       namespace: openshift-operators-redhat 

   1

   
       labels: 

   2

   
         openshift.io/<label_name>: "true"
     spec:
       tenantID: "infrastructure" 

   3

   
       groups:
         - name: LokiOperatorHighReconciliationError
           rules:
             - alert: HighPercentageError
               expr: | 

   4

   
                 sum(rate({kubernetes_namespace_name="openshift-operators-redhat", kubernetes_pod_name=~"loki-operator-controller-manager.*"} |= "error" [1m])) by (job)
                   /
                 sum(rate({kubernetes_namespace_name="openshift-operators-redhat", kubernetes_pod_name=~"loki-operator-controller-manager.*"}[1m])) by (job)
                   > 0.01
               for: 10s
               labels:
                 severity: critical 

   5

   
               annotations:
                 summary: High Loki Operator Reconciliation Errors 

   6

   
                 description: High Loki Operator Reconciliation Errors 

   7

   Copy to Clipboard Toggle word wrap

   1

   The namespace where this AlertingRule CR is created must have a label matching the LokiStack spec.rules.namespaceSelector definition.

   2

   The labels block must match the LokiStack spec.rules.selector definition.

   3

   AlertingRule CRs for infrastructure tenants are only supported in the openshift-*, kube-\*, or default namespaces.

   4

   The value for kubernetes_namespace_name: must match the value for metadata.namespace.

   5

   The value of this mandatory field must be critical, warning, or info.

   6

   This field is mandatory.

   7

   This field is mandatory.

   Example application AlertingRule CR

     apiVersion: loki.grafana.com/v1
     kind: AlertingRule
     metadata:
       name: app-user-workload
       namespace: app-ns 

   1

   
       labels: 

   2

   
         openshift.io/<label_name>: "true"
     spec:
       tenantID: "application"
       groups:
         - name: AppUserWorkloadHighError
           rules:
             - alert:
               expr: | 

   3

   
                 sum(rate({kubernetes_namespace_name="app-ns", kubernetes_pod_name=~"podName.*"} |= "error" [1m])) by (job)
               for: 10s
               labels:
                 severity: critical 

   4

   
               annotations:
                 summary:  

   5

   
                 description:  

   6

   Copy to Clipboard Toggle word wrap

   1

   The namespace where this AlertingRule CR is created must have a label matching the LokiStack spec.rules.namespaceSelector definition.

   2

   The labels block must match the LokiStack spec.rules.selector definition.

   3

   Value for kubernetes_namespace_name: must match the value for metadata.namespace.

   4

   The value of this mandatory field must be critical, warning, or info.

   5

   The value of this mandatory field is a summary of the rule.

   6

   The value of this mandatory field is a detailed description of the rule.

2. Apply the AlertingRule CR:

   $ oc apply -f <filename>.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a LokiStack CR:

   • Enable stream-based retention globally as shown in the following example:

     Example global stream-based retention for AWS

     apiVersion: loki.grafana.com/v1
     kind: LokiStack
     metadata:
       name: logging-loki
       namespace: openshift-logging
     spec:
       limits:
        global: 

     1

     
           retention: 

     2

     
             days: 20
             streams:
             - days: 4
               priority: 1
               selector: '{kubernetes_namespace_name=~"test.+"}' 

     3

     
             - days: 1
               priority: 1
               selector: '{log_type="infrastructure"}'
       managementState: Managed
       replicationFactor: 1
       size: 1x.small
       storage:
         schemas:
         - effectiveDate: "2020-10-11"
           version: v13
         secret:
           name: logging-loki-s3
           type: aws
       storageClassName: gp3-csi
       tenants:
         mode: openshift-logging

     Copy to Clipboard Toggle word wrap

     1

     Sets retention policy for all log streams. Note: This field does not impact the retention period for stored logs in object storage.

     2

     Retention is enabled in the cluster when this block is added to the CR.

     3

     Contains the LogQL query used to define the log stream.spec: limits:

   • Enable stream-based retention per-tenant basis as shown in the following example:

     Example per-tenant stream-based retention for AWS

     apiVersion: loki.grafana.com/v1
     kind: LokiStack
     metadata:
       name: logging-loki
       namespace: openshift-logging
     spec:
       limits:
         global:
           retention:
             days: 20
         tenants: 

     1

     
           application:
             retention:
               days: 1
               streams:
                 - days: 4
                   selector: '{kubernetes_namespace_name=~"test.+"}' 

     2

     
           infrastructure:
             retention:
               days: 5
               streams:
                 - days: 1
                   selector: '{kubernetes_namespace_name=~"openshift-cluster.+"}'
       managementState: Managed
       replicationFactor: 1
       size: 1x.small
       storage:
         schemas:
         - effectiveDate: "2020-10-11"
           version: v13
         secret:
           name: logging-loki-s3
           type: aws
       storageClassName: gp3-csi
       tenants:
         mode: openshift-logging

     Copy to Clipboard Toggle word wrap

     1

     Sets retention policy by tenant. Valid tenant types are application, audit, and infrastructure.

     2

     Contains the LogQL query used to define the log stream.

2. Apply the LokiStack CR:

   $ oc apply -f <filename>.yaml

   Copy to Clipboard Toggle word wrap
   Note

   This is not for managing the retention for stored logs. Global retention periods for stored logs to a supported maximum of 30 days is configured with your object storage.