Chapter 5. Tuning the log store

Procedure

1. To enable stream-based retention, create a LokiStack CR and save it as a YAML file. In the following example, it is called lokistack.yaml.

   Example global stream-based retention for S3

   apiVersion: loki.grafana.com/v1
   kind: LokiStack
   metadata:
     name: logging-loki
     namespace: openshift-logging
   spec:
     limits:
      global: 

   1

   
         retention: 

   2

   
           days: 20
           streams:
           - days: 4
             priority: 1
             selector: '{kubernetes_namespace_name=~"test.+"}' 

   3

   
           - days: 1
             priority: 1
             selector: '{log_type="infrastructure"}'
     managementState: Managed
     replicationFactor: 1
     size: 1x.small
     storage:
       schemas:
       - effectiveDate: "2020-10-11"
         version: v13
       secret:
         name: logging-loki-s3
         type: s3
     storageClassName: gp3-csi
     tenants:
       mode: openshift-logging

   1

   Set the retention policy for all log streams. This policy does not impact the retention period for stored logs in object storage.

   2

   Enable retention in the cluster by adding the retention block to the CR.

   3

   Specify the LogQL query to match log streams to the retention rule.

   Example per-tenant stream-based retention for S3

   apiVersion: loki.grafana.com/v1
   kind: LokiStack
   metadata:
     name: logging-loki
     namespace: openshift-logging
   spec:
     limits:
       global:
         retention:
           days: 20
       tenants: 

   1

   
         application:
           retention:
             days: 1
             streams:
               - days: 4
                 selector: '{kubernetes_namespace_name=~"test.+"}' 

   2

   
         infrastructure:
           retention:
             days: 5
             streams:
               - days: 1
                 selector: '{kubernetes_namespace_name=~"openshift-cluster.+"}'
     managementState: Managed
     replicationFactor: 1
     size: 1x.small
     storage:
       schemas:
       - effectiveDate: "2020-10-11"
         version: v13
       secret:
         name: logging-loki-s3
         type: s3
     storageClassName: gp3-csi
     tenants:
       mode: openshift-logging

   1

   Set the retention policy per-tenant. Valid tenant types are application, audit, and infrastructure.

   2

   Specify the LogQL query to match log streams to the retention rule.

2. Apply the LokiStack CR:

   $ oc apply -f lokistack.yaml

Procedure

1. List the pods in Pending status by running the following command:

   $ oc get pods --field-selector status.phase==Pending -n openshift-logging

   Example oc get pods output

   NAME                           READY   STATUS    RESTARTS   AGE 

   1

   
   logging-loki-index-gateway-1   0/1     Pending   0          17m
   logging-loki-ingester-1        0/1     Pending   0          16m
   logging-loki-ruler-1           0/1     Pending   0          16m

   1

   These pods are in Pending status because their corresponding PVCs are in the failed zone.

2. List the PVCs in Pending status by running the following command:

   $ oc get pvc -o=json -n openshift-logging | jq '.items[] | select(.status.phase == "Pending") | .metadata.name' -r

   Example oc get pvc output

   storage-logging-loki-index-gateway-1
   storage-logging-loki-ingester-1
   wal-logging-loki-ingester-1
   storage-logging-loki-ruler-1
   wal-logging-loki-ruler-1

3. Delete the PVC(s) for a pod by running the following command:

   $ oc delete pvc <pvc_name> -n openshift-logging

4. Delete the pod(s) by running the following command:

   $ oc delete pod <pod_name> -n openshift-logging

   Once these objects have been successfully deleted, they should automatically be rescheduled in an available zone.

Procedure

1. Update the LokiStack CR:

   apiVersion: loki.grafana.com/v1
   kind: LokiStack
   metadata:
     name: logging-loki
     namespace: openshift-logging
   spec:
     size: 1x.small
     storage:
       schemas:
       - version: v13
         effectiveDate: "<yyyy>-<mm>-<dd>"
       secret:
         name: logging-loki-s3
         type: s3
     storageClassName: <storage_class_name>
     tenants:
       mode: openshift-logging
     networkPolicies:
       ruleSet: RestrictIngressEgress

   You can set one of the following values for the spec.networkPolicies.ruleSet field:

   None

   Loki Operator will not deploy any network policy.

   RestrictIngressEgress

   Loki Operator will deploy a set of network policies that restrict the communications to and from the Loki components.

   If you do not define a spec.networkPolicies.ruleSet value, the platform and operator default values are inherited and full network access is allowed.

2. Apply the LokiStack CR object by running the following command:

   $ oc apply -f <filename>.yaml

Procedure

1. Create a network policy:

   apiVersion: networking.k8s.io/v1
   kind: NetworkPolicy
   metadata:
     name: <name>
   spec:
     podSelector:
       matchLabels:
         app.kubernetes.io/name: lokistack
         app.kubernetes.io/instance: <instance_name>
         app.kubernetes.io/component: <loki_component>
     policyTypes:
     - Egress
     egress:
     - to:
       - namespaceSelector:
           matchLabels:
             kubernetes.io/metadata.name: <namespace_name>
       ports:
       - protocol: TCP
         port: <port_number>

   Replace <component_name> with the component you want to integrate with.

2. Apply the network policy:

   $ oc apply -f <file_name>.yaml

Procedure

1. Create an AlertingRule custom resource (CR):

   Example infrastructure AlertingRule CR

     apiVersion: loki.grafana.com/v1
     kind: AlertingRule
     metadata:
       name: loki-operator-alerts
       namespace: openshift-operators-redhat 

   1

   
       labels: 

   2

   
         openshift.io/<label_name>: "true"
     spec:
       tenantID: "infrastructure" 

   3

   
       groups:
         - name: LokiOperatorHighReconciliationError
           rules:
             - alert: HighPercentageError
               expr: | 

   4

   
                 sum(rate({kubernetes_namespace_name="openshift-operators-redhat", kubernetes_pod_name=~"loki-operator-controller-manager.*"} |= "error" [1m])) by (job)
                   /
                 sum(rate({kubernetes_namespace_name="openshift-operators-redhat", kubernetes_pod_name=~"loki-operator-controller-manager.*"}[1m])) by (job)
                   > 0.01
               for: 10s
               labels:
                 severity: critical 

   5

   
               annotations:
                 summary: High Loki Operator Reconciliation Errors 

   6

   
                 description: High Loki Operator Reconciliation Errors 

   7

   1

   The namespace where this AlertingRule CR is created must have a label matching the LokiStack spec.rules.namespaceSelector definition.

   2

   The labels block must match the LokiStack spec.rules.selector definition.

   3

   AlertingRule CRs for infrastructure tenants are only supported in the openshift-*, kube-\*, or default namespaces.

   4

   The value for kubernetes_namespace_name: must match the value for metadata.namespace.

   5

   The value of this mandatory field must be critical, warning, or info.

   6

   This field is mandatory.

   7

   This field is mandatory.

   Example application AlertingRule CR

     apiVersion: loki.grafana.com/v1
     kind: AlertingRule
     metadata:
       name: app-user-workload
       namespace: app-ns 

   1

   
       labels: 

   2

   
         openshift.io/<label_name>: "true"
     spec:
       tenantID: "application"
       groups:
         - name: AppUserWorkloadHighError
           rules:
             - alert:
               expr: | 

   3

   
                 sum(rate({kubernetes_namespace_name="app-ns", kubernetes_pod_name=~"podName.*"} |= "error" [1m])) by (job)
               for: 10s
               labels:
                 severity: critical 

   4

   
               annotations:
                 summary:  

   5

   
                 description:  

   6

   1

   The namespace where this AlertingRule CR is created must have a label matching the LokiStack spec.rules.namespaceSelector definition.

   2

   The labels block must match the LokiStack spec.rules.selector definition.

   3

   Value for kubernetes_namespace_name: must match the value for metadata.namespace.

   4

   The value of this mandatory field must be critical, warning, or info.

   5

   The value of this mandatory field is a summary of the rule.

   6

   The value of this mandatory field is a detailed description of the rule.

2. Apply the AlertingRule CR:

   $ oc apply -f <filename>.yaml