Chapter 2. Configuring the log store

Procedure

1. Copy the service account credentials received from GCP into a file called key.json.

2. Create an object storage secret with the name logging-loki-gcs by running the following command:

   $ oc create secret generic logging-loki-gcs \
     --from-literal=bucketname="<bucket_name>" \
     --from-file=key.json="<path/to/key.json>"

   Copy to Clipboard Toggle word wrap

Procedure

1. Create an ObjectBucketClaim custom resource in the openshift-logging namespace:

   apiVersion: objectbucket.io/v1alpha1
   kind: ObjectBucketClaim
   metadata:
     name: loki-bucket-odf
     namespace: openshift-logging
   spec:
     generateBucketName: loki-bucket-odf
     storageClassName: openshift-storage.noobaa.io

   Copy to Clipboard Toggle word wrap

2. Get bucket properties from the associated ConfigMap object by running the following command:

   BUCKET_HOST=$(oc get -n openshift-logging configmap loki-bucket-odf -o jsonpath='{.data.BUCKET_HOST}')
   BUCKET_NAME=$(oc get -n openshift-logging configmap loki-bucket-odf -o jsonpath='{.data.BUCKET_NAME}')
   BUCKET_PORT=$(oc get -n openshift-logging configmap loki-bucket-odf -o jsonpath='{.data.BUCKET_PORT}')

   Copy to Clipboard Toggle word wrap

3. Get bucket access key from the associated secret by running the following command:

   ACCESS_KEY_ID=$(oc get -n openshift-logging secret loki-bucket-odf -o jsonpath='{.data.AWS_ACCESS_KEY_ID}' | base64 -d)
   SECRET_ACCESS_KEY=$(oc get -n openshift-logging secret loki-bucket-odf -o jsonpath='{.data.AWS_SECRET_ACCESS_KEY}' | base64 -d)

   Copy to Clipboard Toggle word wrap

4. Create an object storage secret with the name logging-loki-odf by running the following command:

   $ oc create -n openshift-logging secret generic logging-loki-odf \ 

   1

   
     --from-literal=access_key_id="<access_key_id>" \
     --from-literal=access_key_secret="<secret_access_key>" \
     --from-literal=bucketnames="<bucket_name>" \
     --from-literal=endpoint="https://<bucket_host>:<bucket_port>"
     --from-literal=forcepathstyle="true" 

   2

   Copy to Clipboard Toggle word wrap

   1

   logging-loki-odf is the name of the secret.

   2

   AWS endpoints (those ending in .amazonaws.com) use a virtual-hosted style by default, which is equivalent to setting the forcepathstyle attribute to false. Conversely, non-AWS endpoints use a path style, equivalent to setting forcepathstyle attribute to true. If you need to use a virtual-hosted style with non-AWS S3 services, you must explicitly set forcepathstyle to false.

Procedure

1. Enter the following command to create a new group:

   $ oc adm groups new cluster-admin

   Copy to Clipboard Toggle word wrap

2. Enter the following command to add the desired user to the cluster-admin group:

   $ oc adm groups add-users cluster-admin <username>

   Copy to Clipboard Toggle word wrap

3. Enter the following command to add cluster-admin user role to the group:

   $ oc adm policy add-cluster-role-to-group cluster-admin cluster-admin

   Copy to Clipboard Toggle word wrap

Procedure

1. To enable stream-based retention, create a LokiStack CR and save it as a YAML file. In the following example, it is called lokistack.yaml.

   Example global stream-based retention for S3

   apiVersion: loki.grafana.com/v1
   kind: LokiStack
   metadata:
     name: logging-loki
     namespace: openshift-logging
   spec:
     limits:
      global: 

   1

   
         retention: 

   2

   
           days: 20
           streams:
           - days: 4
             priority: 1
             selector: '{kubernetes_namespace_name=~"test.+"}' 

   3

   
           - days: 1
             priority: 1
             selector: '{log_type="infrastructure"}'
     managementState: Managed
     replicationFactor: 1
     size: 1x.small
     storage:
       schemas:
       - effectiveDate: "2020-10-11"
         version: v13
       secret:
         name: logging-loki-s3
         type: s3
     storageClassName: gp3-csi
     tenants:
       mode: openshift-logging

   Copy to Clipboard Toggle word wrap

   1

   Set the retention policy for all log streams. This policy does not impact the retention period for stored logs in object storage.

   2

   Enable retention in the cluster by adding the retention block to the CR.

   3

   Specify the LogQL query to match log streams to the retention rule.

   Example per-tenant stream-based retention for S3

   apiVersion: loki.grafana.com/v1
   kind: LokiStack
   metadata:
     name: logging-loki
     namespace: openshift-logging
   spec:
     limits:
       global:
         retention:
           days: 20
       tenants: 

   1

   
         application:
           retention:
             days: 1
             streams:
               - days: 4
                 selector: '{kubernetes_namespace_name=~"test.+"}' 

   2

   
         infrastructure:
           retention:
             days: 5
             streams:
               - days: 1
                 selector: '{kubernetes_namespace_name=~"openshift-cluster.+"}'
     managementState: Managed
     replicationFactor: 1
     size: 1x.small
     storage:
       schemas:
       - effectiveDate: "2020-10-11"
         version: v13
       secret:
         name: logging-loki-s3
         type: s3
     storageClassName: gp3-csi
     tenants:
       mode: openshift-logging

   Copy to Clipboard Toggle word wrap

   1

   Set the retention policy per-tenant. Valid tenant types are application, audit, and infrastructure.

   2

   Specify the LogQL query to match log streams to the retention rule.

2. Apply the LokiStack CR:

   $ oc apply -f lokistack.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. List the pods in Pending status by running the following command:

   $ oc get pods --field-selector status.phase==Pending -n openshift-logging

   Copy to Clipboard Toggle word wrap

   Example oc get pods output

   NAME                           READY   STATUS    RESTARTS   AGE 

   1

   
   logging-loki-index-gateway-1   0/1     Pending   0          17m
   logging-loki-ingester-1        0/1     Pending   0          16m
   logging-loki-ruler-1           0/1     Pending   0          16m

   Copy to Clipboard Toggle word wrap

   1

   These pods are in Pending status because their corresponding PVCs are in the failed zone.

2. List the PVCs in Pending status by running the following command:

   $ oc get pvc -o=json -n openshift-logging | jq '.items[] | select(.status.phase == "Pending") | .metadata.name' -r

   Copy to Clipboard Toggle word wrap

   Example oc get pvc output

   storage-logging-loki-index-gateway-1
   storage-logging-loki-ingester-1
   wal-logging-loki-ingester-1
   storage-logging-loki-ruler-1
   wal-logging-loki-ruler-1

   Copy to Clipboard Toggle word wrap

3. Delete the PVC(s) for a pod by running the following command:

   $ oc delete pvc <pvc_name> -n openshift-logging

   Copy to Clipboard Toggle word wrap

4. Delete the pod(s) by running the following command:

   $ oc delete pod <pod_name> -n openshift-logging

   Copy to Clipboard Toggle word wrap

   Once these objects have been successfully deleted, they should automatically be rescheduled in an available zone.

Procedure

1. Create an AlertingRule custom resource (CR):

   Example infrastructure AlertingRule CR

     apiVersion: loki.grafana.com/v1
     kind: AlertingRule
     metadata:
       name: loki-operator-alerts
       namespace: openshift-operators-redhat 

   1

   
       labels: 

   2

   
         openshift.io/<label_name>: "true"
     spec:
       tenantID: "infrastructure" 

   3

   
       groups:
         - name: LokiOperatorHighReconciliationError
           rules:
             - alert: HighPercentageError
               expr: | 

   4

   
                 sum(rate({kubernetes_namespace_name="openshift-operators-redhat", kubernetes_pod_name=~"loki-operator-controller-manager.*"} |= "error" [1m])) by (job)
                   /
                 sum(rate({kubernetes_namespace_name="openshift-operators-redhat", kubernetes_pod_name=~"loki-operator-controller-manager.*"}[1m])) by (job)
                   > 0.01
               for: 10s
               labels:
                 severity: critical 

   5

   
               annotations:
                 summary: High Loki Operator Reconciliation Errors 

   6

   
                 description: High Loki Operator Reconciliation Errors 

   7

   Copy to Clipboard Toggle word wrap

   1

   The namespace where this AlertingRule CR is created must have a label matching the LokiStack spec.rules.namespaceSelector definition.

   2

   The labels block must match the LokiStack spec.rules.selector definition.

   3

   AlertingRule CRs for infrastructure tenants are only supported in the openshift-*, kube-\*, or default namespaces.

   4

   The value for kubernetes_namespace_name: must match the value for metadata.namespace.

   5

   The value of this mandatory field must be critical, warning, or info.

   6

   This field is mandatory.

   7

   This field is mandatory.

   Example application AlertingRule CR

     apiVersion: loki.grafana.com/v1
     kind: AlertingRule
     metadata:
       name: app-user-workload
       namespace: app-ns 

   1

   
       labels: 

   2

   
         openshift.io/<label_name>: "true"
     spec:
       tenantID: "application"
       groups:
         - name: AppUserWorkloadHighError
           rules:
             - alert:
               expr: | 

   3

   
                 sum(rate({kubernetes_namespace_name="app-ns", kubernetes_pod_name=~"podName.*"} |= "error" [1m])) by (job)
               for: 10s
               labels:
                 severity: critical 

   4

   
               annotations:
                 summary:  

   5

   
                 description:  

   6

   Copy to Clipboard Toggle word wrap

   1

   The namespace where this AlertingRule CR is created must have a label matching the LokiStack spec.rules.namespaceSelector definition.

   2

   The labels block must match the LokiStack spec.rules.selector definition.

   3

   Value for kubernetes_namespace_name: must match the value for metadata.namespace.

   4

   The value of this mandatory field must be critical, warning, or info.

   5

   The value of this mandatory field is a summary of the rule.

   6

   The value of this mandatory field is a detailed description of the rule.

2. Apply the AlertingRule CR:

   $ oc apply -f <filename>.yaml

   Copy to Clipboard Toggle word wrap