Chapter 10. Log storage

Procedure

1. Modify the ClusterLogging CR logStore spec:

   ClusterLogging CR example

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogging
   metadata:
   # ...
   spec:
   # ...
     logStore:
       type: <log_store_type> 

   1

   
       elasticsearch: 

   2

   
         nodeCount: <integer>
         resources: {}
         storage: {}
         redundancyPolicy: <redundancy_type> 

   3

   
       lokistack: 

   4

   
         name: {}
   # ...

   Copy to Clipboard Toggle word wrap

   1

   Specify the log store type. This can be either lokistack or elasticsearch.

   2

   Optional configuration options for the Elasticsearch log store.

   3

   Specify the redundancy type. This value can be ZeroRedundancy, SingleRedundancy, MultipleRedundancy, or FullRedundancy.

   4

   Optional configuration options for LokiStack.

   Example ClusterLogging CR to specify LokiStack as the log store

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogging
   metadata:
     name: instance
     namespace: openshift-logging
   spec:
     managementState: Managed
     logStore:
       type: lokistack
       lokistack:
         name: logging-loki
   # ...

   Copy to Clipboard Toggle word wrap

2. Apply the ClusterLogging CR by running the following command:

   $ oc apply -f <filename>.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Enter the following command to create a new group:

   $ oc adm groups new cluster-admin

   Copy to Clipboard Toggle word wrap

2. Enter the following command to add the desired user to the cluster-admin group:

   $ oc adm groups add-users cluster-admin <username>

   Copy to Clipboard Toggle word wrap

3. Enter the following command to add cluster-admin user role to the group:

   $ oc adm policy add-cluster-role-to-group cluster-admin cluster-admin

   Copy to Clipboard Toggle word wrap

1. To enable stream-based retention, create a LokiStack custom resource (CR):

   Example global stream-based retention

   apiVersion: loki.grafana.com/v1
   kind: LokiStack
   metadata:
     name: logging-loki
     namespace: openshift-logging
   spec:
     limits:
       global: 

   1

   
         retention: 

   2

   
           days: 20
           streams:
           - days: 4
             priority: 1
             selector: '{kubernetes_namespace_name=~"test.+"}' 

   3

   
           - days: 1
             priority: 1
             selector: '{log_type="infrastructure"}'
     managementState: Managed
     replicationFactor: 1
     size: 1x.small
     storage:
       schemas:
       - effectiveDate: "2020-10-11"
         version: v11
       secret:
         name: logging-loki-s3
         type: aws
     storageClassName: standard
     tenants:
       mode: openshift-logging

   Copy to Clipboard Toggle word wrap

   1

   Sets retention policy for all log streams. Note: This field does not impact the retention period for stored logs in object storage.

   2

   Retention is enabled in the cluster when this block is added to the CR.

   3

   Contains the LogQL query used to define the log stream.

   Example per-tenant stream-based retention

   apiVersion: loki.grafana.com/v1
   kind: LokiStack
   metadata:
     name: logging-loki
     namespace: openshift-logging
   spec:
     limits:
       global:
         retention:
           days: 20
       tenants: 

   1

   
         application:
           retention:
             days: 1
             streams:
               - days: 4
                 selector: '{kubernetes_namespace_name=~"test.+"}' 

   2

   
         infrastructure:
           retention:
             days: 5
             streams:
               - days: 1
                 selector: '{kubernetes_namespace_name=~"openshift-cluster.+"}'
     managementState: Managed
     replicationFactor: 1
     size: 1x.small
     storage:
       schemas:
       - effectiveDate: "2020-10-11"
         version: v11
       secret:
         name: logging-loki-s3
         type: aws
     storageClassName: standard
     tenants:
       mode: openshift-logging

   Copy to Clipboard Toggle word wrap

   1

   Sets retention policy by tenant. Valid tenant types are application, audit, and infrastructure.

   2

   Contains the LogQL query used to define the log stream.

2. Apply the LokiStack CR:

   $ oc apply -f <filename>.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Modify the ClusterLogging CR logStore spec:

   ClusterLogging CR example

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogging
   metadata:
   # ...
   spec:
   # ...
     logStore:
       type: <log_store_type> 

   1

   
       elasticsearch: 

   2

   
         nodeCount: <integer>
         resources: {}
         storage: {}
         redundancyPolicy: <redundancy_type> 

   3

   
       lokistack: 

   4

   
         name: {}
   # ...

   Copy to Clipboard Toggle word wrap

   1

   Specify the log store type. This can be either lokistack or elasticsearch.

   2

   Optional configuration options for the Elasticsearch log store.

   3

   Specify the redundancy type. This value can be ZeroRedundancy, SingleRedundancy, MultipleRedundancy, or FullRedundancy.

   4

   Optional configuration options for LokiStack.

   Example ClusterLogging CR to specify LokiStack as the log store

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogging
   metadata:
     name: instance
     namespace: openshift-logging
   spec:
     managementState: Managed
     logStore:
       type: lokistack
       lokistack:
         name: logging-loki
   # ...

   Copy to Clipboard Toggle word wrap

2. Apply the ClusterLogging CR by running the following command:

   $ oc apply -f <filename>.yaml

   Copy to Clipboard Toggle word wrap

1. Create or edit a YAML file that defines the ClusterLogForwarder CR object:

   • Create a CR to send all log types to the internal Elasticsearch instance. You can use the following example without making any changes:

     apiVersion: logging.openshift.io/v1
     kind: ClusterLogForwarder
     metadata:
       name: instance
       namespace: openshift-logging
     spec:
       pipelines: 

     1

     
       - name: all-to-default
         inputRefs:
         - infrastructure
         - application
         - audit
         outputRefs:
         - default

     Copy to Clipboard Toggle word wrap

     1

     A pipeline defines the type of logs to forward using the specified output. The default output forwards logs to the internal Elasticsearch instance.

     Note

     You must specify all three types of logs in the pipeline: application, infrastructure, and audit. If you do not specify a log type, those logs are not stored and will be lost.

   • If you have an existing ClusterLogForwarder CR, add a pipeline to the default output for the audit logs. You do not need to define the default output. For example:

     apiVersion: "logging.openshift.io/v1"
     kind: ClusterLogForwarder
     metadata:
       name: instance
       namespace: openshift-logging
     spec:
       outputs:
        - name: elasticsearch-insecure
          type: "elasticsearch"
          url: http://elasticsearch-insecure.messaging.svc.cluster.local
          insecure: true
        - name: elasticsearch-secure
          type: "elasticsearch"
          url: https://elasticsearch-secure.messaging.svc.cluster.local
          secret:
            name: es-audit
        - name: secureforward-offcluster
          type: "fluentdForward"
          url: https://secureforward.offcluster.com:24224
          secret:
            name: secureforward
       pipelines:
        - name: container-logs
          inputRefs:
          - application
          outputRefs:
          - secureforward-offcluster
        - name: infra-logs
          inputRefs:
          - infrastructure
          outputRefs:
          - elasticsearch-insecure
        - name: audit-logs
          inputRefs:
          - audit
          outputRefs:
          - elasticsearch-secure
          - default 

     1

     Copy to Clipboard Toggle word wrap

     1

     This pipeline sends the audit logs to the internal Elasticsearch instance in addition to an external instance.

1. Edit the ClusterLogging CR to add or modify the retentionPolicy parameter:

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   ...
   spec:
     managementState: "Managed"
     logStore:
       type: "elasticsearch"
       retentionPolicy: 

   1

   
         application:
           maxAge: 1d
         infra:
           maxAge: 7d
         audit:
           maxAge: 7d
       elasticsearch:
         nodeCount: 3
   ...

   Copy to Clipboard Toggle word wrap

   1

   Specify the time that Elasticsearch should retain each log source. Enter an integer and a time designation: weeks(w), hours(h/H), minutes(m) and seconds(s). For example, 1d for one day. Logs older than the maxAge are deleted. By default, logs are retained for seven days.

2. You can verify the settings in the Elasticsearch custom resource (CR).

   For example, the Red Hat OpenShift Logging Operator updated the following Elasticsearch CR to configure a retention policy that includes settings to roll over active indices for the infrastructure logs every eight hours and the rolled-over indices are deleted seven days after rollover. OpenShift Container Platform checks every 15 minutes to determine if the indices need to be rolled over.

   apiVersion: "logging.openshift.io/v1"
   kind: "Elasticsearch"
   metadata:
     name: "elasticsearch"
   spec:
   ...
     indexManagement:
       policies: 

   1

   
         - name: infra-policy
           phases:
             delete:
               minAge: 7d 

   2

   
             hot:
               actions:
                 rollover:
                   maxAge: 8h 

   3

   
           pollInterval: 15m 

   4

   
   ...

   Copy to Clipboard Toggle word wrap

   1

   For each log source, the retention policy indicates when to delete and roll over logs for that source.

   2

   When OpenShift Container Platform deletes the rolled-over indices. This setting is the maxAge you set in the ClusterLogging CR.

   3

   The index age for OpenShift Container Platform to consider when rolling over the indices. This value is determined from the maxAge you set in the ClusterLogging CR.

   4

   When OpenShift Container Platform checks if the indices should be rolled over. This setting is the default and cannot be changed.

   Note

   Modifying the Elasticsearch CR is not supported. All changes to the retention policies must be made in the ClusterLogging CR.

   The OpenShift Elasticsearch Operator deploys a cron job to roll over indices for each mapping using the defined policy, scheduled using the pollInterval.

   $ oc get cronjob

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                     SCHEDULE       SUSPEND   ACTIVE   LAST SCHEDULE   AGE
   elasticsearch-im-app     */15 * * * *   False     0        <none>          4s
   elasticsearch-im-audit   */15 * * * *   False     0        <none>          4s
   elasticsearch-im-infra   */15 * * * *   False     0        <none>          4s

   Copy to Clipboard Toggle word wrap

Procedure

1. Edit the ClusterLogging custom resource (CR) in the openshift-logging project:

   $ oc edit ClusterLogging instance

   Copy to Clipboard Toggle word wrap

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   ....
   spec:
       logStore:
         type: "elasticsearch"
         elasticsearch:

   1

   
           resources:
             limits: 

   2

   
               memory: "32Gi"
             requests: 

   3

   
               cpu: "1"
               memory: "16Gi"
           proxy: 

   4

   
             resources:
               limits:
                 memory: 100Mi
               requests:
                 memory: 100Mi

   Copy to Clipboard Toggle word wrap

   1

   Specify the CPU and memory requests for Elasticsearch as needed. If you leave these values blank, the OpenShift Elasticsearch Operator sets default values that should be sufficient for most deployments. The default values are 16Gi for the memory request and 1 for the CPU request.

   2

   The maximum amount of resources a pod can use.

   3

   The minimum resources required to schedule a pod.

   4

   Specify the CPU and memory requests for the Elasticsearch proxy as needed. If you leave these values blank, the OpenShift Elasticsearch Operator sets default values that are sufficient for most deployments. The default values are 256Mi for the memory request and 100m for the CPU request.

Procedure

1. Edit the ClusterLogging custom resource (CR) in the openshift-logging project:

   $ oc edit clusterlogging instance

   Copy to Clipboard Toggle word wrap

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   
   ....
   
   spec:
     logStore:
       type: "elasticsearch"
       elasticsearch:
         redundancyPolicy: "SingleRedundancy" 

   1

   Copy to Clipboard Toggle word wrap

   1

   Specify a redundancy policy for the shards. The change is applied upon saving the changes.

   • FullRedundancy. Elasticsearch fully replicates the primary shards for each index to every data node. This provides the highest safety, but at the cost of the highest amount of disk required and the poorest performance.
   • MultipleRedundancy. Elasticsearch fully replicates the primary shards for each index to half of the data nodes. This provides a good tradeoff between safety and performance.
   • SingleRedundancy. Elasticsearch makes one copy of the primary shards for each index. Logs are always available and recoverable as long as at least two data nodes exist. Better performance than MultipleRedundancy, when using 5 or more nodes. You cannot apply this policy on deployments of single Elasticsearch node.
   • ZeroRedundancy. Elasticsearch does not make copies of the primary shards. Logs might be unavailable or lost in the event a node is down or fails. Use this mode when you are more concerned with performance than safety, or have implemented your own disk/PVC backup/restore strategy.

Procedure

1. Edit the ClusterLogging CR to specify that each data node in the cluster is bound to a Persistent Volume Claim.

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   # ...
   spec:
     logStore:
       type: "elasticsearch"
       elasticsearch:
         nodeCount: 3
         storage:
           storageClassName: "gp2"
           size: "200G"

   Copy to Clipboard Toggle word wrap

Procedure

1. Edit the ClusterLogging CR to specify emptyDir:

    spec:
       logStore:
         type: "elasticsearch"
         elasticsearch:
           nodeCount: 3
           storage: {}

   Copy to Clipboard Toggle word wrap

1. Change to the openshift-logging project:

   $ oc project openshift-logging

   Copy to Clipboard Toggle word wrap

2. Get the names of the Elasticsearch pods:

   $ oc get pods -l component=elasticsearch

   Copy to Clipboard Toggle word wrap

3. Scale down the collector pods so they stop sending new logs to Elasticsearch:

   $ oc -n openshift-logging patch daemonset/collector -p '{"spec":{"template":{"spec":{"nodeSelector":{"logging-infra-collector": "false"}}}}}'

   Copy to Clipboard Toggle word wrap

4. Perform a shard synced flush using the OpenShift Container Platform es_util tool to ensure there are no pending operations waiting to be written to disk prior to shutting down:

   $ oc exec <any_es_pod_in_the_cluster> -c elasticsearch -- es_util --query="_flush/synced" -XPOST

   Copy to Clipboard Toggle word wrap

   For example:

   $ oc exec -c elasticsearch-cdm-5ceex6ts-1-dcd6c4c7c-jpw6  -c elasticsearch -- es_util --query="_flush/synced" -XPOST

   Copy to Clipboard Toggle word wrap

   Example output

   {"_shards":{"total":4,"successful":4,"failed":0},".security":{"total":2,"successful":2,"failed":0},".kibana_1":{"total":2,"successful":2,"failed":0}}

   Copy to Clipboard Toggle word wrap

5. Prevent shard balancing when purposely bringing down nodes using the OpenShift Container Platform es_util tool:

   $ oc exec <any_es_pod_in_the_cluster> -c elasticsearch -- es_util --query="_cluster/settings" -XPUT -d '{ "persistent": { "cluster.routing.allocation.enable" : "primaries" } }'

   Copy to Clipboard Toggle word wrap

   For example:

   $ oc exec elasticsearch-cdm-5ceex6ts-1-dcd6c4c7c-jpw6 -c elasticsearch -- es_util --query="_cluster/settings" -XPUT -d '{ "persistent": { "cluster.routing.allocation.enable" : "primaries" } }'

   Copy to Clipboard Toggle word wrap

   Example output

   {"acknowledged":true,"persistent":{"cluster":{"routing":{"allocation":{"enable":"primaries"}}}},"transient":

   Copy to Clipboard Toggle word wrap

6. After the command is complete, for each deployment you have for an ES cluster:

   1. By default, the OpenShift Container Platform Elasticsearch cluster blocks rollouts to their nodes. Use the following command to allow rollouts and allow the pod to pick up the changes:

      $ oc rollout resume deployment/<deployment-name>

      Copy to Clipboard Toggle word wrap

      For example:

      $ oc rollout resume deployment/elasticsearch-cdm-0-1

      Copy to Clipboard Toggle word wrap

      Example output

      deployment.extensions/elasticsearch-cdm-0-1 resumed

      Copy to Clipboard Toggle word wrap

      A new pod is deployed. After the pod has a ready container, you can move on to the next deployment.

      $ oc get pods -l component=elasticsearch-

      Copy to Clipboard Toggle word wrap

      Example output

      NAME                                            READY   STATUS    RESTARTS   AGE
      elasticsearch-cdm-5ceex6ts-1-dcd6c4c7c-jpw6k    2/2     Running   0          22h
      elasticsearch-cdm-5ceex6ts-2-f799564cb-l9mj7    2/2     Running   0          22h
      elasticsearch-cdm-5ceex6ts-3-585968dc68-k7kjr   2/2     Running   0          22h

      Copy to Clipboard Toggle word wrap

   2. After the deployments are complete, reset the pod to disallow rollouts:

      $ oc rollout pause deployment/<deployment-name>

      Copy to Clipboard Toggle word wrap

      For example:

      $ oc rollout pause deployment/elasticsearch-cdm-0-1

      Copy to Clipboard Toggle word wrap

      Example output

      deployment.extensions/elasticsearch-cdm-0-1 paused

      Copy to Clipboard Toggle word wrap

   3. Check that the Elasticsearch cluster is in a green or yellow state:

      $ oc exec <any_es_pod_in_the_cluster> -c elasticsearch -- es_util --query=_cluster/health?pretty=true

      Copy to Clipboard Toggle word wrap
      Note

      If you performed a rollout on the Elasticsearch pod you used in the previous commands, the pod no longer exists and you need a new pod name here.

      For example:

      $ oc exec elasticsearch-cdm-5ceex6ts-1-dcd6c4c7c-jpw6 -c elasticsearch -- es_util --query=_cluster/health?pretty=true

      Copy to Clipboard Toggle word wrap

      {
        "cluster_name" : "elasticsearch",
        "status" : "yellow", 

      1

      
        "timed_out" : false,
        "number_of_nodes" : 3,
        "number_of_data_nodes" : 3,
        "active_primary_shards" : 8,
        "active_shards" : 16,
        "relocating_shards" : 0,
        "initializing_shards" : 0,
        "unassigned_shards" : 1,
        "delayed_unassigned_shards" : 0,
        "number_of_pending_tasks" : 0,
        "number_of_in_flight_fetch" : 0,
        "task_max_waiting_in_queue_millis" : 0,
        "active_shards_percent_as_number" : 100.0
      }

      Copy to Clipboard Toggle word wrap

      1

      Make sure this parameter value is green or yellow before proceeding.

7. If you changed the Elasticsearch configuration map, repeat these steps for each Elasticsearch pod.

8. After all the deployments for the cluster have been rolled out, re-enable shard balancing:

   $ oc exec <any_es_pod_in_the_cluster> -c elasticsearch -- es_util --query="_cluster/settings" -XPUT -d '{ "persistent": { "cluster.routing.allocation.enable" : "all" } }'

   Copy to Clipboard Toggle word wrap

   For example:

   $ oc exec elasticsearch-cdm-5ceex6ts-1-dcd6c4c7c-jpw6 -c elasticsearch -- es_util --query="_cluster/settings" -XPUT -d '{ "persistent": { "cluster.routing.allocation.enable" : "all" } }'

   Copy to Clipboard Toggle word wrap

   Example output

   {
     "acknowledged" : true,
     "persistent" : { },
     "transient" : {
       "cluster" : {
         "routing" : {
           "allocation" : {
             "enable" : "all"
           }
         }
       }
     }
   }

   Copy to Clipboard Toggle word wrap

9. Scale up the collector pods so they send new logs to Elasticsearch.

   $ oc -n openshift-logging patch daemonset/collector -p '{"spec":{"template":{"spec":{"nodeSelector":{"logging-infra-collector": "true"}}}}}'

   Copy to Clipboard Toggle word wrap

1. Change to the openshift-logging project:

   $ oc project openshift-logging

   Copy to Clipboard Toggle word wrap

2. Extract the CA certificate from the log store and write to the admin-ca file:

   $ oc extract secret/elasticsearch --to=. --keys=admin-ca

   Copy to Clipboard Toggle word wrap

   Example output

   admin-ca

   Copy to Clipboard Toggle word wrap

3. Create the route for the log store service as a YAML file:

   1. Create a YAML file with the following:

      apiVersion: route.openshift.io/v1
      kind: Route
      metadata:
        name: elasticsearch
        namespace: openshift-logging
      spec:
        host:
        to:
          kind: Service
          name: elasticsearch
        tls:
          termination: reencrypt
          destinationCACertificate: | 

      1

      Copy to Clipboard Toggle word wrap

      1

      Add the log store CA certifcate or use the command in the next step. You do not have to set the spec.tls.key, spec.tls.certificate, and spec.tls.caCertificate parameters required by some reencrypt routes.

   2. Run the following command to add the log store CA certificate to the route YAML you created in the previous step:

      $ cat ./admin-ca | sed -e "s/^/      /" >> <file-name>.yaml

      Copy to Clipboard Toggle word wrap

   3. Create the route:

      $ oc create -f <file-name>.yaml

      Copy to Clipboard Toggle word wrap

      Example output

      route.route.openshift.io/elasticsearch created

      Copy to Clipboard Toggle word wrap

4. Check that the Elasticsearch service is exposed:

   1. Get the token of this service account to be used in the request:

      $ token=$(oc whoami -t)

      Copy to Clipboard Toggle word wrap

   2. Set the elasticsearch route you created as an environment variable.

      $ routeES=`oc get route elasticsearch -o jsonpath={.spec.host}`

      Copy to Clipboard Toggle word wrap

   3. To verify the route was successfully created, run the following command that accesses Elasticsearch through the exposed route:

      curl -tlsv1.2 --insecure -H "Authorization: Bearer ${token}" "https://${routeES}"

      Copy to Clipboard Toggle word wrap

      The response appears similar to the following:

      Example output

      {
        "name" : "elasticsearch-cdm-i40ktba0-1",
        "cluster_name" : "elasticsearch",
        "cluster_uuid" : "0eY-tJzcR3KOdpgeMJo-MQ",
        "version" : {
        "number" : "6.8.1",
        "build_flavor" : "oss",
        "build_type" : "zip",
        "build_hash" : "Unknown",
        "build_date" : "Unknown",
        "build_snapshot" : true,
        "lucene_version" : "7.7.0",
        "minimum_wire_compatibility_version" : "5.6.0",
        "minimum_index_compatibility_version" : "5.0.0"
      },
        "<tagline>" : "<for search>"
      }

      Copy to Clipboard Toggle word wrap

Procedure

1. Edit the ClusterLogging custom resource (CR) in the openshift-logging project:

   $ oc edit ClusterLogging instance

   Copy to Clipboard Toggle word wrap
2. If they are present, remove the logStore and visualization stanzas from the ClusterLogging CR.

3. Preserve the collection stanza of the ClusterLogging CR. The result should look similar to the following example:

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
     namespace: "openshift-logging"
   spec:
     managementState: "Managed"
     collection:
       type: "fluentd"
       fluentd: {}

   Copy to Clipboard Toggle word wrap

4. Verify that the collector pods are redeployed:

   $ oc get pods -l component=collector -n openshift-logging

   Copy to Clipboard Toggle word wrap