Chapter 2. Logging 6.0

Procedure

1. Install the

   Red Hat OpenShift Logging Operator

   ,

   Loki Operator

   , and

   Cluster Observability Operator (COO)

   from OperatorHub.

2. Create a secret to access an existing object storage bucket:

   Example command for AWS

   $ oc create secret generic logging-loki-s3 \
     --from-literal=bucketnames="<bucket_name>" \
     --from-literal=endpoint="<aws_bucket_endpoint>" \
     --from-literal=access_key_id="<aws_access_key_id>" \
     --from-literal=access_key_secret="<aws_access_key_secret>" \
     --from-literal=region="<aws_region_of_your_bucket>" \
     -n openshift-logging

3. Create a

   LokiStack

   custom resource (CR) in the

   openshift-logging

   namespace:

   apiVersion: loki.grafana.com/v1
   kind: LokiStack
   metadata:
     name: logging-loki
     namespace: openshift-logging
   spec:
     managementState: Managed
     size: 1x.extra-small
     storage:
       schemas:
       - effectiveDate: '2022-06-01'
         version: v13
       secret:
         name: logging-loki-s3
         type: s3
     storageClassName: gp3-csi
     tenants:
       mode: openshift-logging

4. Create a service account for the collector:

   $ oc create sa collector -n openshift-logging

5. Bind the

   ClusterRole

   to the service account:

   $ oc adm policy add-cluster-role-to-user logging-collector-logs-writer -z collector -n openshift-logging

6. Create a

   UIPlugin

   to enable the Log section in the Observe tab:

   apiVersion: observability.openshift.io/v1alpha1
   kind: UIPlugin
   metadata:
     name: logging
   spec:
     type: Logging
     logging:
       lokiStack:
         name: logging-loki

7. Add additional roles to the collector service account:

   $ oc adm policy add-cluster-role-to-user collect-application-logs -z collector -n openshift-logging

   $ oc adm policy add-cluster-role-to-user collect-audit-logs -z collector -n openshift-logging

   $ oc adm policy add-cluster-role-to-user collect-infrastructure-logs -z collector -n openshift-logging

8. Create a

   ClusterLogForwarder

   CR to configure log forwarding:

   apiVersion: observability.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: collector
     namespace: openshift-logging
   spec:
     serviceAccount:
       name: collector
     outputs:
     - name: default-lokistack
       type: lokiStack
       lokiStack:
         target:
           name: logging-loki
           namespace: openshift-logging
         authentication:
           token:
             from: serviceAccount
       tls:
         ca:
           key: service-ca.crt
           configMapName: openshift-service-ca.crt
     pipelines:
     - name: default-logstore
       inputRefs:
       - application
       - infrastructure
       outputRefs:
       - default-lokistack

1. Temporarily set

   ClusterLogging

   resource to the

   Unmanaged

   state by running the following command:

   $ oc -n openshift-logging patch clusterlogging/instance -p '{"spec":{"managementState": "Unmanaged"}}' --type=merge

2. Remove the

   ownerReferences

   parameter from the

   Elasticsearch

   resource by running the following command:

   The following command ensures that

   ClusterLogging

   no longer owns the

   Elasticsearch

   resource. Updates to the

   ClusterLogging

   resource’s

   logStore

   field will no longer affect the

   Elasticsearch

   resource.

   $ oc -n openshift-logging patch elasticsearch/elasticsearch -p '{"metadata":{"ownerReferences": []}}' --type=merge

3. Remove the

   ownerReferences

   parameter from the

   Kibana

   resource.

   The following command ensures that Cluster Logging no longer owns the

   Kibana

   resource. Updates to the

   ClusterLogging

   resource’s

   visualization

   field will no longer affect the

   Kibana

   resource.

   $ oc -n openshift-logging patch kibana/kibana -p '{"metadata":{"ownerReferences": []}}' --type=merge

4. Set the

   ClusterLogging

   resource to the

   Managed

   state by running the following command:

   $ oc -n openshift-logging patch clusterlogging/instance -p '{"spec":{"managementState": "Managed"}}' --type=merge

Procedure

1. Add a configuration to define the

   audit

   and

   infrastructure

   sources in the

   ClusterLogForwarder

   CR.

   The following example shows how to configure the

   ClusterLogForwarder

   CR to define

   audit

   and

   infrastructure

   sources:

   Example ClusterLogForwarder CR

   apiVersion: observability.openshift.io/v1
   kind: ClusterLogForwarder
   # ...
   spec:
     serviceAccount:
       name: <service_account_name>
     inputs:
       - name: mylogs1
         type: infrastructure
         infrastructure:
           sources: 

   1

   
             - node
       - name: mylogs2
         type: audit
         audit:
           sources: 

   2

   
             - kubeAPI
             - openshiftAPI
             - ovn
   # ...

   1

   Specifies the list of infrastructure sources to collect. The valid sources include:

   • node

     : Journal log from the node

   • container

     : Logs from the workloads deployed in the namespaces

   2

   Specifies the list of audit sources to collect. The valid sources include:

   • kubeAPI

     : Logs from the Kubernetes API servers

   • openshiftAPI

     : Logs from the OpenShift API servers

   • auditd

     : Logs from a node auditd service

   • ovn

     : Logs from an open virtual network service

2. Apply the

   ClusterLogForwarder

   CR by running the following command:

   $ oc apply -f <filename>.yaml

Procedure

1. Add a configuration to include or exclude the namespace and container names in the

   ClusterLogForwarder

   CR.

   The following example shows how to configure the

   ClusterLogForwarder

   CR to include or exclude namespaces and container names:

   Example ClusterLogForwarder CR

   apiVersion: observability.openshift.io/v1
   kind: ClusterLogForwarder
   # ...
   spec:
     serviceAccount:
       name: <service_account_name>
     inputs:
       - name: mylogs
         application:
           includes:
             - namespace: "my-project" 

   1

   
               container: "my-container" 

   2

   
           excludes:
             - container: "other-container*" 

   3

   
               namespace: "other-namespace" 

   4

   
         type: application
   # ...

   1

   Specifies that the logs are only collected from these namespaces.

   2

   Specifies that the logs are only collected from these containers.

   3

   Specifies the pattern of namespaces to ignore when collecting the logs.

   4

   Specifies the set of containers to ignore when collecting the logs.

   Note

   The

   excludes

   field takes precedence over the

   includes

   field.

2. Apply the

   ClusterLogForwarder

   CR by running the following command:

   $ oc apply -f <filename>.yaml

Procedure

1. Create an

   AlertingRule

   custom resource (CR):

   Example infrastructure AlertingRule CR

     apiVersion: loki.grafana.com/v1
     kind: AlertingRule
     metadata:
       name: loki-operator-alerts
       namespace: openshift-operators-redhat 

   1

   
       labels: 

   2

   
         openshift.io/<label_name>: "true"
     spec:
       tenantID: "infrastructure" 

   3

   
       groups:
         - name: LokiOperatorHighReconciliationError
           rules:
             - alert: HighPercentageError
               expr: | 

   4

   
                 sum(rate({kubernetes_namespace_name="openshift-operators-redhat", kubernetes_pod_name=~"loki-operator-controller-manager.*"} |= "error" [1m])) by (job)
                   /
                 sum(rate({kubernetes_namespace_name="openshift-operators-redhat", kubernetes_pod_name=~"loki-operator-controller-manager.*"}[1m])) by (job)
                   > 0.01
               for: 10s
               labels:
                 severity: critical 

   5

   
               annotations:
                 summary: High Loki Operator Reconciliation Errors 

   6

   
                 description: High Loki Operator Reconciliation Errors 

   7

   1

   The namespace where this AlertingRule CR is created must have a label matching the LokiStack spec.rules.namespaceSelector definition.

   2

   The labels block must match the LokiStack spec.rules.selector definition.

   3

   AlertingRule CRs for infrastructure tenants are only supported in the openshift-*, kube-\*, or default namespaces.

   4

   The value for kubernetes_namespace_name: must match the value for metadata.namespace.

   5

   The value of this mandatory field must be critical, warning, or info.

   6

   This field is mandatory.

   7

   This field is mandatory.

   Example application AlertingRule CR

     apiVersion: loki.grafana.com/v1
     kind: AlertingRule
     metadata:
       name: app-user-workload
       namespace: app-ns 

   1

   
       labels: 

   2

   
         openshift.io/<label_name>: "true"
     spec:
       tenantID: "application"
       groups:
         - name: AppUserWorkloadHighError
           rules:
             - alert:
               expr: | 

   3

   
                 sum(rate({kubernetes_namespace_name="app-ns", kubernetes_pod_name=~"podName.*"} |= "error" [1m])) by (job)
               for: 10s
               labels:
                 severity: critical 

   4

   
               annotations:
                 summary:  

   5

   
                 description:  

   6

   1

   The namespace where this AlertingRule CR is created must have a label matching the LokiStack spec.rules.namespaceSelector definition.

   2

   The labels block must match the LokiStack spec.rules.selector definition.

   3

   Value for kubernetes_namespace_name: must match the value for metadata.namespace.

   4

   The value of this mandatory field must be critical, warning, or info.

   5

   The value of this mandatory field is a summary of the rule.

   6

   The value of this mandatory field is a detailed description of the rule.

2. Apply the

   AlertingRule

   CR:

   $ oc apply -f <filename>.yaml

Procedure

1. Create a

   LokiStack

   CR:

   • Enable stream-based retention globally as shown in the following example:

     Example global stream-based retention for AWS

     apiVersion: loki.grafana.com/v1
     kind: LokiStack
     metadata:
       name: logging-loki
       namespace: openshift-logging
     spec:
       limits:
        global: 

     1

     
           retention: 

     2

     
             days: 20
             streams:
             - days: 4
               priority: 1
               selector: '{kubernetes_namespace_name=~"test.+"}' 

     3

     
             - days: 1
               priority: 1
               selector: '{log_type="infrastructure"}'
       managementState: Managed
       replicationFactor: 1
       size: 1x.small
       storage:
         schemas:
         - effectiveDate: "2020-10-11"
           version: v13
         secret:
           name: logging-loki-s3
           type: aws
       storageClassName: gp3-csi
       tenants:
         mode: openshift-logging

     1

     Sets retention policy for all log streams. Note: This field does not impact the retention period for stored logs in object storage.

     2

     Retention is enabled in the cluster when this block is added to the CR.

     3

     Contains the LogQL query used to define the log stream.spec: limits:

   • Enable stream-based retention per-tenant basis as shown in the following example:

     Example per-tenant stream-based retention for AWS

     apiVersion: loki.grafana.com/v1
     kind: LokiStack
     metadata:
       name: logging-loki
       namespace: openshift-logging
     spec:
       limits:
         global:
           retention:
             days: 20
         tenants: 

     1

     
           application:
             retention:
               days: 1
               streams:
                 - days: 4
                   selector: '{kubernetes_namespace_name=~"test.+"}' 

     2

     
           infrastructure:
             retention:
               days: 5
               streams:
                 - days: 1
                   selector: '{kubernetes_namespace_name=~"openshift-cluster.+"}'
       managementState: Managed
       replicationFactor: 1
       size: 1x.small
       storage:
         schemas:
         - effectiveDate: "2020-10-11"
           version: v13
         secret:
           name: logging-loki-s3
           type: aws
       storageClassName: gp3-csi
       tenants:
         mode: openshift-logging

     1

     Sets retention policy by tenant. Valid tenant types are application, audit, and infrastructure.

     2

     Contains the LogQL query used to define the log stream.

2. Apply the

   LokiStack

   CR:

   $ oc apply -f <filename>.yaml

   Note

   This is not for managing the retention for stored logs. Global retention periods for stored logs to a supported maximum of 30 days is configured with your object storage.

Procedure

1. List the pods in

   Pending

   status by running the following command:

   $ oc get pods --field-selector status.phase==Pending -n openshift-logging

   Example oc get pods output

   NAME                           READY   STATUS    RESTARTS   AGE 

   1

   
   logging-loki-index-gateway-1   0/1     Pending   0          17m
   logging-loki-ingester-1        0/1     Pending   0          16m
   logging-loki-ruler-1           0/1     Pending   0          16m

   1

   These pods are in Pending status because their corresponding PVCs are in the failed zone.

2. List the PVCs in

   Pending

   status by running the following command:

   $ oc get pvc -o=json -n openshift-logging | jq '.items[] | select(.status.phase == "Pending") | .metadata.name' -r

   Example oc get pvc output

   storage-logging-loki-index-gateway-1
   storage-logging-loki-ingester-1
   wal-logging-loki-ingester-1
   storage-logging-loki-ruler-1
   wal-logging-loki-ruler-1

3. Delete the PVC(s) for a pod by running the following command:

   $ oc delete pvc <pvc_name> -n openshift-logging

4. Delete the pod(s) by running the following command:

   $ oc delete pod <pod_name> -n openshift-logging

   Once these objects have been successfully deleted, they should automatically be rescheduled in an available zone.