3.3. Configuring the log store

1. If the Log Forwarding API is not enabled:

   1. Edit the ClusterLogging custom resource (CR) in the openshift-logging project:

      $ oc edit ClusterLogging instance

   2. Add the clusterlogging.openshift.io/logforwardingtechpreview annotation and set to enabled:

      apiVersion: "logging.openshift.io/v1"
      kind: "ClusterLogging"
      metadata:
        annotations:
          clusterlogging.openshift.io/logforwardingtechpreview: enabled 1
        name: "instance"
        namespace: "openshift-logging"
      spec:
      
      ...
      
        collection: 2
          logs:
            type: "fluentd"
            fluentd: {}

      1

      Enables and disables the Log Forwarding API. Set to enabled to use log forwarding.

      2

      The spec.collection block must be defined to use Fluentd in the ClusterLogging CR.

2. Create a LogForwarding CR YAML file or edit your existing CR:

   • Create a CR to send all log types to the internal Elasticsearch instance. You can use the following example without making any changes:

     apiVersion: logging.openshift.io/v1alpha1
     kind: LogForwarding
     metadata:
       name: instance
       namespace: openshift-logging
     spec:
       disableDefaultForwarding: true
       outputs:
         - name: clo-es
           type: elasticsearch
           endpoint: 'elasticsearch.openshift-logging.svc:9200' 1
           secret:
             name: fluentd
       pipelines:
         - name: audit-pipeline 2
           inputSource: logs.audit
           outputRefs:
             - clo-es
         - name: app-pipeline 3
           inputSource: logs.app
           outputRefs:
             - clo-es
         - name: infra-pipeline 4
           inputSource: logs.infra
           outputRefs:
             - clo-es

     1

     The endpoint parameter points to the internal Elasticsearch instance.

     2

     This parameter sends the audit logs to the specified endpoint.

     3

     This parameter sends the application logs to the specified endpoint.

     4

     This parameter sends the infrastructure logs to the specified endpoint.

     注意

     You must configure a pipeline and output for all three types of logs: application, infrastructure, and audit. If you do not specify a pipeline and output for a log type, those logs are not stored and will be lost.

   • If you have an existing LogForwarding CR, add an output for the internal Elasticsearch instance and a pipeline to that output for the audit logs. For example:

     apiVersion: "logging.openshift.io/v1alpha1"
     kind: "LogForwarding"
     metadata:
       name: instance
       namespace: openshift-logging
     spec:
       disableDefaultForwarding: true
       outputs:
        - name: elasticsearch 1
          type: "elasticsearch"
          endpoint: elasticsearch.openshift-logging.svc:9200
          secret:
             name: fluentd
        - name: elasticsearch-insecure
          type: "elasticsearch"
          endpoint: elasticsearch-insecure.messaging.svc.cluster.local
          insecure: true
        - name: secureforward-offcluster
          type: "forward"
          endpoint: https://secureforward.offcluster.com:24224
          secret:
             name: secureforward
       pipelines:
        - name: container-logs
          inputSource: logs.app
          outputRefs:
          - secureforward-offcluster
        - name: infra-logs
          inputSource: logs.infra
          outputRefs:
          - elasticsearch-insecure
        - name: audit-logs 2
          inputSource: logs.audit
          outputRefs:
          - elasticsearch

     1

     An output for the internal Elasticsearch instance.

     2

     A pipeline for sending the audit logs to the internal Elasticsearch instance.

1. Edit the ClusterLogging CR to add or modify the retentionPolicy parameter:

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   ...
   spec:
     managementState: "Managed"
     logStore:
       type: "elasticsearch"
       retentionPolicy: 1
         application:
           maxAge: 1d
         infra:
           maxAge: 7d
         audit:
           maxAge: 7d
       elasticsearch:
         nodeCount: 3
   ...

   1

   Specify the time that Elasticsearch should retain each log source. Enter an integer and a time designation: weeks(w), hours(h/H), minutes(m) and seconds(s). For example, 1d for one day. Logs older than the maxAge are deleted. By default, logs are retained for seven days.

2. You can verify the settings in the Elasticsearch custom resource (CR).

   For example, the Cluster Logging Operator updated the following Elasticsearch CR to configure a retention policy that includes settings to roll over active indices for the infrastructure logs every eight hours and the rolled-ver indices are deleted seven days after rollover. OpenShift Container Platform checks every 15 minutes to determine if the indices need to be rolled over.

   apiVersion: "logging.openshift.io/v1"
   kind: "Elasticsearch"
   metadata:
     name: "elasticsearch"
   spec:
   ...
     indexManagement:
       policies: 1
         - name: infra-policy
           phases:
             delete:
               minAge: 7d 2
             hot:
               actions:
                 rollover:
                   maxAge: 8h 3
           pollInterval: 15m 4
   ...

   1

   For each log source, the retention policy indicates when to delete and rollover logs for that source.

   2

   When OpenShift Container Platform deletes the rolled-over indices. This setting is the maxAge you set in the ClusterLogging CR.

   3

   The index age for OpenShift Container Platform to consider when rolling over the indices. This value is determined from the maxAge you set in the ClusterLogging CR.

   4

   When OpenShift Container Platform checks if the indices should be rolled over. This setting is the default and cannot be changed.

   注意

   Modifying the Elasticsearch CR is not supported. All changes to the retention policies must be made in the ClusterLogging CR.

   The Elasticsearch Operator deploys a cron job to roll over indices for each mapping using the defined policy, scheduled using the pollInterval.

   $ oc get cronjobs

   Example output

   NAME                           SCHEDULE       SUSPEND   ACTIVE   LAST SCHEDULE   AGE
   elasticsearch-delete-app       */15 * * * *   False     0        <none>          27s
   elasticsearch-delete-audit     */15 * * * *   False     0        <none>          27s
   elasticsearch-delete-infra     */15 * * * *   False     0        <none>          27s
   elasticsearch-rollover-app     */15 * * * *   False     0        <none>          27s
   elasticsearch-rollover-audit   */15 * * * *   False     0        <none>          27s
   elasticsearch-rollover-infra   */15 * * * *   False     0        <none>          27s

Procedure

1. Edit the ClusterLogging custom resource (CR) in the openshift-logging project:

   $ oc edit ClusterLogging instance

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   ....
   spec:
       logStore:
         type: "elasticsearch"
         elasticsearch:
           resources: 1
             limits:
               memory: 16Gi
             requests:
               cpu: "1"
               memory: "64Gi"
           proxy: 2
             resources:
               limits:
                 memory: 100Mi
               requests:
                 memory: 100Mi

   1

   Specify the CPU and memory requests for Elasticsearch as needed. If you leave these values blank, the Elasticsearch Operator sets default values that should be sufficient for most deployments. The default values are 16Gi for the memory request and 1 for the CPU request.

   2

   Specify the CPU and memory requests for the Elasticsearch proxy as needed. If you leave these values blank, the Elasticsearch Operator sets default values that should be sufficient for most deployments. The default values are 256Mi for the memory request and 100m for the CPU request.

Procedure

1. Edit the ClusterLogging custom resource (CR) in the openshift-logging project:

   $ oc edit clusterlogging instance

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   
   ....
   
   spec:
     logStore:
       type: "elasticsearch"
       elasticsearch:
         redundancyPolicy: "SingleRedundancy" 1

   1

   Specify a redundancy policy for the shards. The change is applied upon saving the changes.

   • FullRedundancy. Elasticsearch fully replicates the primary shards for each index to every data node. This provides the highest safety, but at the cost of the highest amount of disk required and the poorest performance.
   • MultipleRedundancy. Elasticsearch fully replicates the primary shards for each index to half of the data nodes. This provides a good tradeoff between safety and performance.
   • SingleRedundancy. Elasticsearch makes one copy of the primary shards for each index. Logs are always available and recoverable as long as at least two data nodes exist. Better performance than MultipleRedundancy, when using 5 or more nodes. You cannot apply this policy on deployments of single Elasticsearch node.
   • ZeroRedundancy. Elasticsearch does not make copies of the primary shards. Logs might be unavailable or lost in the event a node is down or fails. Use this mode when you are more concerned with performance than safety, or have implemented your own disk/PVC backup/restore strategy.

Procedure

1. Edit the ClusterLogging CR to specify that each data node in the cluster is bound to a Persistent Volume Claim.

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   
   ....
   
    spec:
       logStore:
         type: "elasticsearch"
         elasticsearch:
           nodeCount: 3
           storage:
             storageClassName: "gp2"
             size: "200G"

Procedure

1. Edit the ClusterLogging CR to specify emptyDir:

    spec:
       logStore:
         type: "elasticsearch"
         elasticsearch:
           nodeCount: 3
           storage: {}

1. Change to the openshift-logging project:

   $ oc project openshift-logging

2. Get the names of the Elasticsearch pods:

   $ oc get pods | grep elasticsearch-

3. Perform a shard synced flush using the OpenShift Container Platform es_util tool to ensure there are no pending operations waiting to be written to disk prior to shutting down:

   $ oc exec <any_es_pod_in_the_cluster> -c elasticsearch -- es_util --query="_flush/synced" -XPOST

   For example:

   $ oc exec -c elasticsearch-cdm-5ceex6ts-1-dcd6c4c7c-jpw6  -c elasticsearch -- es_util --query="_flush/synced" -XPOST

   {"_shards":{"total":4,"successful":4,"failed":0},".security":{"total":2,"successful":2,"failed":0},".kibana_1":{"total":2,"successful":2,"failed":0}}

4. Prevent shard balancing when purposely bringing down nodes using the OpenShift Container Platform es_util tool:

   $ oc exec <any_es_pod_in_the_cluster> -c elasticsearch -- es_util --query="_cluster/settings" -XPUT -d '{ "persistent": { "cluster.routing.allocation.enable" : "primaries" } }'

   For example:

   $ oc exec elasticsearch-cdm-5ceex6ts-1-dcd6c4c7c-jpw6 -c elasticsearch -- es_util --query="_cluster/settings" -XPUT -d '{ "persistent": { "cluster.routing.allocation.enable" : "primaries" } }'

   Example output

   {"acknowledged":true,"persistent":{"cluster":{"routing":{"allocation":{"enable":"primaries"}}}},"transient":

5. After the command is complete, for each deployment you have for an ES cluster:

   1. By default, the OpenShift Container Platform Elasticsearch cluster blocks rollouts to their nodes. Use the following command to allow rollouts and allow the pod to pick up the changes:

      $ oc rollout resume deployment/<deployment-name>

      For example:

      $ oc rollout resume deployment/elasticsearch-cdm-0-1

      deployment.extensions/elasticsearch-cdm-0-1 resumed

      A new pod is deployed. After the pod has a ready container, you can move on to the next deployment.

      $ oc get pods | grep elasticsearch-

      Example output

      NAME                                            READY   STATUS    RESTARTS   AGE
      elasticsearch-cdm-5ceex6ts-1-dcd6c4c7c-jpw6k    2/2     Running   0          22h
      elasticsearch-cdm-5ceex6ts-2-f799564cb-l9mj7    2/2     Running   0          22h
      elasticsearch-cdm-5ceex6ts-3-585968dc68-k7kjr   2/2     Running   0          22h

   2. After the deployments are complete, reset the pod to disallow rollouts:

      $ oc rollout pause deployment/<deployment-name>

      For example:

      $ oc rollout pause deployment/elasticsearch-cdm-0-1

      deployment.extensions/elasticsearch-cdm-0-1 paused

   3. Check that the Elasticsearch cluster is in a green or yellow state:

      $ oc exec <any_es_pod_in_the_cluster> -c elasticsearch -- es_util --query=_cluster/health?pretty=true

      注意

      If you performed a rollout on the Elasticsearch pod you used in the previous commands, the pod no longer exists and you need a new pod name here.

      For example:

      $ oc exec elasticsearch-cdm-5ceex6ts-1-dcd6c4c7c-jpw6 -c elasticsearch -- es_util --query=_cluster/health?pretty=true

      {
        "cluster_name" : "elasticsearch",
        "status" : "yellow", 1
        "timed_out" : false,
        "number_of_nodes" : 3,
        "number_of_data_nodes" : 3,
        "active_primary_shards" : 8,
        "active_shards" : 16,
        "relocating_shards" : 0,
        "initializing_shards" : 0,
        "unassigned_shards" : 1,
        "delayed_unassigned_shards" : 0,
        "number_of_pending_tasks" : 0,
        "number_of_in_flight_fetch" : 0,
        "task_max_waiting_in_queue_millis" : 0,
        "active_shards_percent_as_number" : 100.0
      }

      1

      Make sure this parameter value is green or yellow before proceeding.

6. If you changed the Elasticsearch configuration map, repeat these steps for each Elasticsearch pod.

7. After all the deployments for the cluster have been rolled out, re-enable shard balancing:

   $ oc exec <any_es_pod_in_the_cluster> -c elasticsearch -- es_util --query="_cluster/settings" -XPUT -d '{ "persistent": { "cluster.routing.allocation.enable" : "all" } }'

   For example:

   $ oc exec elasticsearch-cdm-5ceex6ts-1-dcd6c4c7c-jpw6 -c elasticsearch -- es_util --query="_cluster/settings" -XPUT -d '{ "persistent": { "cluster.routing.allocation.enable" : "all" } }'

   Example output

   {
     "acknowledged" : true,
     "persistent" : { },
     "transient" : {
       "cluster" : {
         "routing" : {
           "allocation" : {
             "enable" : "all"
           }
         }
       }
     }
   }

1. Change to the openshift-logging project:

   $ oc project openshift-logging

2. Extract the CA certificate from the log store and write to the admin-ca file:

   $ oc extract secret/elasticsearch --to=. --keys=admin-ca

   Example output

   admin-ca

3. Create the route for the log store service as a YAML file:

   1. Create a YAML file with the following:

      apiVersion: route.openshift.io/v1
      kind: Route
      metadata:
        name: elasticsearch
        namespace: openshift-logging
      spec:
        host:
        to:
          kind: Service
          name: elasticsearch
        tls:
          termination: reencrypt
          destinationCACertificate: | 1

      1

      Add the log store CA certifcate or use the command in the next step. You do not have to set the spec.tls.key, spec.tls.certificate, and spec.tls.caCertificate parameters required by some reencrypt routes.

   2. Run the following command to add the log store CA certificate to the route YAML you created:

      $ cat ./admin-ca | sed -e "s/^/      /" >> <file-name>.yaml

   3. Create the route:

      $ oc create -f <file-name>.yaml

      Example output

      route.route.openshift.io/elasticsearch created

4. Check that the Elasticsearch service is exposed:

   1. Get the token of this service account to be used in the request:

      $ token=$(oc whoami -t)

   2. Set the elasticsearch route you created as an environment variable.

      $ routeES=`oc get route elasticsearch -o jsonpath={.spec.host}`

   3. To verify the route was successfully created, run the following command that accesses Elasticsearch through the exposed route:

      curl -tlsv1.2 --insecure -H "Authorization: Bearer ${token}" "https://${routeES}"

      The response appears similar to the following:

      Example output

      {
        "name" : "elasticsearch-cdm-i40ktba0-1",
        "cluster_name" : "elasticsearch",
        "cluster_uuid" : "0eY-tJzcR3KOdpgeMJo-MQ",
        "version" : {
        "number" : "6.8.1",
        "build_flavor" : "oss",
        "build_type" : "zip",
        "build_hash" : "Unknown",
        "build_date" : "Unknown",
        "build_snapshot" : true,
        "lucene_version" : "7.7.0",
        "minimum_wire_compatibility_version" : "5.6.0",
        "minimum_index_compatibility_version" : "5.0.0"
      },
        "<tagline>" : "<for search>"
      }