Chapter 4. Configuring your Logging deployment

Procedure

1. Edit the

   ClusterLogging

   custom resource (CR) in the

   openshift-logging

   project:

   $ oc -n openshift-logging edit ClusterLogging instance

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
     namespace: openshift-logging
   
   ...
   
   spec:
     collection:
       logs:
         fluentd:
           resources:
             limits: 

   1

   
               memory: 736Mi
             requests:
               cpu: 100m
               memory: 736Mi

   1

   Specify the CPU and memory limits and requests as needed. The values shown are the default values.

Procedure

1. Edit the

   ClusterLogging

   custom resource (CR) in the

   openshift-logging

   project:

   $ oc edit ClusterLogging instance

2. Add or modify any of the following parameters:

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogging
   metadata:
     name: instance
     namespace: openshift-logging
   spec:
     forwarder:
       fluentd:
         buffer:
           chunkLimitSize: 8m 

   1

   
           flushInterval: 5s 

   2

   
           flushMode: interval 

   3

   
           flushThreadCount: 3 

   4

   
           overflowAction: throw_exception 

   5

   
           retryMaxInterval: "300s" 

   6

   
           retryType: periodic 

   7

   
           retryWait: 1s 

   8

   
           totalLimitSize: 32m 

   9

   
   ...

   1

   Specify the maximum size of each chunk before it is queued for flushing.

   2

   Specify the interval between chunk flushes.

   3

   Specify the method to perform chunk flushes: lazy, interval, or immediate.

   4

   Specify the number of threads to use for chunk flushes.

   5

   Specify the chunking behavior when the queue is full: throw_exception, block, or drop_oldest_chunk.

   6

   Specify the maximum interval in seconds for the exponential_backoff chunk flushing method.

   7

   Specify the retry type when chunk flushing fails: exponential_backoff or periodic.

   8

   Specify the time in seconds before the next chunk flush.

   9

   Specify the maximum size of the chunk buffer.

3. Verify that the Fluentd pods are redeployed:

   $ oc get pods -n openshift-logging

4. Check that the new values are in the

   fluentd

   config map:

   $ oc extract configmap/fluentd --confirm

   Example fluentd.conf

   <buffer>
    @type file
    path '/var/lib/fluentd/default'
    flush_mode interval
    flush_interval 5s
    flush_thread_count 3
    retry_type periodic
    retry_wait 1s
    retry_max_interval 300s
    retry_timeout 60m
    queued_chunks_limit_size "#{ENV['BUFFER_QUEUE_LIMIT'] || '32'}"
    total_limit_size 32m
    chunk_limit_size 8m
    overflow_action throw_exception
   </buffer>

Procedure

1. Edit the

   ClusterLogging

   custom resource (CR) in the

   openshift-logging

   project:

   $ oc edit ClusterLogging instance

2. If they are present, remove the

   logStore

   and

   visualization

   stanzas from the

   ClusterLogging

   CR.

3. Preserve the

   collection

   stanza of the

   ClusterLogging

   CR. The result should look similar to the following example:

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
     namespace: "openshift-logging"
   spec:
     managementState: "Managed"
     collection:
       logs:
         type: "fluentd"
         fluentd: {}

4. Verify that the Fluentd pods are redeployed:

   $ oc get pods -n openshift-logging

1. Create or edit a YAML file that defines the

   ClusterLogForwarder

   CR object:

   • Create a CR to send all log types to the internal Elasticsearch instance. You can use the following example without making any changes:

     apiVersion: logging.openshift.io/v1
     kind: ClusterLogForwarder
     metadata:
       name: instance
       namespace: openshift-logging
     spec:
       pipelines: 

     1

     
       - name: all-to-default
         inputRefs:
         - infrastructure
         - application
         - audit
         outputRefs:
         - default

     1

     A pipeline defines the type of logs to forward using the specified output. The default output forwards logs to the internal Elasticsearch instance.

     Note

     You must specify all three types of logs in the pipeline: application, infrastructure, and audit. If you do not specify a log type, those logs are not stored and will be lost.

   • If you have an existing

     ClusterLogForwarder

     CR, add a pipeline to the default output for the audit logs. You do not need to define the default output. For example:

     apiVersion: "logging.openshift.io/v1"
     kind: ClusterLogForwarder
     metadata:
       name: instance
       namespace: openshift-logging
     spec:
       outputs:
        - name: elasticsearch-insecure
          type: "elasticsearch"
          url: http://elasticsearch-insecure.messaging.svc.cluster.local
          insecure: true
        - name: elasticsearch-secure
          type: "elasticsearch"
          url: https://elasticsearch-secure.messaging.svc.cluster.local
          secret:
            name: es-audit
        - name: secureforward-offcluster
          type: "fluentdForward"
          url: https://secureforward.offcluster.com:24224
          secret:
            name: secureforward
       pipelines:
        - name: container-logs
          inputRefs:
          - application
          outputRefs:
          - secureforward-offcluster
        - name: infra-logs
          inputRefs:
          - infrastructure
          outputRefs:
          - elasticsearch-insecure
        - name: audit-logs
          inputRefs:
          - audit
          outputRefs:
          - elasticsearch-secure
          - default 

     1

     1

     This pipeline sends the audit logs to the internal Elasticsearch instance in addition to an external instance.

1. Edit the

   ClusterLogging

   CR to add or modify the

   retentionPolicy

   parameter:

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   ...
   spec:
     managementState: "Managed"
     logStore:
       type: "elasticsearch"
       retentionPolicy: 

   1

   
         application:
           maxAge: 1d
         infra:
           maxAge: 7d
         audit:
           maxAge: 7d
       elasticsearch:
         nodeCount: 3
   ...

   1

   Specify the time that Elasticsearch should retain each log source. Enter an integer and a time designation: weeks(w), hours(h/H), minutes(m) and seconds(s). For example, 1d for one day. Logs older than the maxAge are deleted. By default, logs are retained for seven days.

2. You can verify the settings in the

   Elasticsearch

   custom resource (CR).

   For example, the Red Hat OpenShift Logging Operator updated the following

   Elasticsearch

   CR to configure a retention policy that includes settings to roll over active indices for the infrastructure logs every eight hours and the rolled-over indices are deleted seven days after rollover. OpenShift Container Platform checks every 15 minutes to determine if the indices need to be rolled over.

   apiVersion: "logging.openshift.io/v1"
   kind: "Elasticsearch"
   metadata:
     name: "elasticsearch"
   spec:
   ...
     indexManagement:
       policies: 

   1

   
         - name: infra-policy
           phases:
             delete:
               minAge: 7d 

   2

   
             hot:
               actions:
                 rollover:
                   maxAge: 8h 

   3

   
           pollInterval: 15m 

   4

   
   ...

   1

   For each log source, the retention policy indicates when to delete and roll over logs for that source.

   2

   When OpenShift Container Platform deletes the rolled-over indices. This setting is the maxAge you set in the ClusterLogging CR.

   3

   The index age for OpenShift Container Platform to consider when rolling over the indices. This value is determined from the maxAge you set in the ClusterLogging CR.

   4

   When OpenShift Container Platform checks if the indices should be rolled over. This setting is the default and cannot be changed.

   Note

   Modifying the

   Elasticsearch

   CR is not supported. All changes to the retention policies must be made in the

   ClusterLogging

   CR.

   The OpenShift Elasticsearch Operator deploys a cron job to roll over indices for each mapping using the defined policy, scheduled using the

   pollInterval

   .

   $ oc get cronjob

   Example output

   NAME                     SCHEDULE       SUSPEND   ACTIVE   LAST SCHEDULE   AGE
   elasticsearch-im-app     */15 * * * *   False     0        <none>          4s
   elasticsearch-im-audit   */15 * * * *   False     0        <none>          4s
   elasticsearch-im-infra   */15 * * * *   False     0        <none>          4s

Procedure

1. Edit the

   ClusterLogging

   custom resource (CR) in the

   openshift-logging

   project:

   $ oc edit ClusterLogging instance

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   ....
   spec:
       logStore:
         type: "elasticsearch"
         elasticsearch:

   1

   
           resources:
             limits: 

   2

   
               memory: "32Gi"
             requests: 

   3

   
               cpu: "1"
               memory: "16Gi"
           proxy: 

   4

   
             resources:
               limits:
                 memory: 100Mi
               requests:
                 memory: 100Mi

   1

   Specify the CPU and memory requests for Elasticsearch as needed. If you leave these values blank, the OpenShift Elasticsearch Operator sets default values that should be sufficient for most deployments. The default values are 16Gi for the memory request and 1 for the CPU request.

   2

   The maximum amount of resources a pod can use.

   3

   The minimum resources required to schedule a pod.

   4

   Specify the CPU and memory requests for the Elasticsearch proxy as needed. If you leave these values blank, the OpenShift Elasticsearch Operator sets default values that are sufficient for most deployments. The default values are 256Mi for the memory request and 100m for the CPU request.

Procedure

1. Edit the

   ClusterLogging

   custom resource (CR) in the

   openshift-logging

   project:

   $ oc edit clusterlogging instance

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   
   ....
   
   spec:
     logStore:
       type: "elasticsearch"
       elasticsearch:
         redundancyPolicy: "SingleRedundancy" 

   1

   1

   Specify a redundancy policy for the shards. The change is applied upon saving the changes.

   • FullRedundancy. Elasticsearch fully replicates the primary shards for each index to every data node. This provides the highest safety, but at the cost of the highest amount of disk required and the poorest performance.
   • MultipleRedundancy. Elasticsearch fully replicates the primary shards for each index to half of the data nodes. This provides a good tradeoff between safety and performance.
   • SingleRedundancy. Elasticsearch makes one copy of the primary shards for each index. Logs are always available and recoverable as long as at least two data nodes exist. Better performance than MultipleRedundancy, when using 5 or more nodes. You cannot apply this policy on deployments of single Elasticsearch node.
   • ZeroRedundancy. Elasticsearch does not make copies of the primary shards. Logs might be unavailable or lost in the event a node is down or fails. Use this mode when you are more concerned with performance than safety, or have implemented your own disk/PVC backup/restore strategy.

Procedure

1. Edit the

   ClusterLogging

   CR to specify that each data node in the cluster is bound to a Persistent Volume Claim.

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   # ...
   spec:
     logStore:
       type: "elasticsearch"
       elasticsearch:
         nodeCount: 3
         storage:
           storageClassName: "gp2"
           size: "200G"

Procedure

1. Edit the

   ClusterLogging

   CR to specify emptyDir:

    spec:
       logStore:
         type: "elasticsearch"
         elasticsearch:
           nodeCount: 3
           storage: {}

1. Change to the

   openshift-logging

   project:

   $ oc project openshift-logging

2. Get the names of the Elasticsearch pods:

   $ oc get pods | grep elasticsearch-

3. Scale down the Fluentd pods so they stop sending new logs to Elasticsearch:

   $ oc -n openshift-logging patch daemonset/logging-fluentd -p '{"spec":{"template":{"spec":{"nodeSelector":{"logging-infra-fluentd": "false"}}}}}'

4. Perform a shard synced flush using the OpenShift Container Platform es_util tool to ensure there are no pending operations waiting to be written to disk prior to shutting down:

   $ oc exec <any_es_pod_in_the_cluster> -c elasticsearch -- es_util --query="_flush/synced" -XPOST

   For example:

   $ oc exec -c elasticsearch-cdm-5ceex6ts-1-dcd6c4c7c-jpw6  -c elasticsearch -- es_util --query="_flush/synced" -XPOST

   Example output

   {"_shards":{"total":4,"successful":4,"failed":0},".security":{"total":2,"successful":2,"failed":0},".kibana_1":{"total":2,"successful":2,"failed":0}}

5. Prevent shard balancing when purposely bringing down nodes using the OpenShift Container Platform es_util tool:

   $ oc exec <any_es_pod_in_the_cluster> -c elasticsearch -- es_util --query="_cluster/settings" -XPUT -d '{ "persistent": { "cluster.routing.allocation.enable" : "primaries" } }'

   For example:

   $ oc exec elasticsearch-cdm-5ceex6ts-1-dcd6c4c7c-jpw6 -c elasticsearch -- es_util --query="_cluster/settings" -XPUT -d '{ "persistent": { "cluster.routing.allocation.enable" : "primaries" } }'

   Example output

   {"acknowledged":true,"persistent":{"cluster":{"routing":{"allocation":{"enable":"primaries"}}}},"transient":

6. After the command is complete, for each deployment you have for an ES cluster:

   1. By default, the OpenShift Container Platform Elasticsearch cluster blocks rollouts to their nodes. Use the following command to allow rollouts and allow the pod to pick up the changes:

      $ oc rollout resume deployment/<deployment-name>

      For example:

      $ oc rollout resume deployment/elasticsearch-cdm-0-1

      Example output

      deployment.extensions/elasticsearch-cdm-0-1 resumed

      A new pod is deployed. After the pod has a ready container, you can move on to the next deployment.

      $ oc get pods | grep elasticsearch-

      Example output

      NAME                                            READY   STATUS    RESTARTS   AGE
      elasticsearch-cdm-5ceex6ts-1-dcd6c4c7c-jpw6k    2/2     Running   0          22h
      elasticsearch-cdm-5ceex6ts-2-f799564cb-l9mj7    2/2     Running   0          22h
      elasticsearch-cdm-5ceex6ts-3-585968dc68-k7kjr   2/2     Running   0          22h

   2. After the deployments are complete, reset the pod to disallow rollouts:

      $ oc rollout pause deployment/<deployment-name>

      For example:

      $ oc rollout pause deployment/elasticsearch-cdm-0-1

      Example output

      deployment.extensions/elasticsearch-cdm-0-1 paused

   3. Check that the Elasticsearch cluster is in a

      green

      or

      yellow

      state:

      $ oc exec <any_es_pod_in_the_cluster> -c elasticsearch -- es_util --query=_cluster/health?pretty=true

      Note

      If you performed a rollout on the Elasticsearch pod you used in the previous commands, the pod no longer exists and you need a new pod name here.

      For example:

      $ oc exec elasticsearch-cdm-5ceex6ts-1-dcd6c4c7c-jpw6 -c elasticsearch -- es_util --query=_cluster/health?pretty=true

      {
        "cluster_name" : "elasticsearch",
        "status" : "yellow", 

      1

      
        "timed_out" : false,
        "number_of_nodes" : 3,
        "number_of_data_nodes" : 3,
        "active_primary_shards" : 8,
        "active_shards" : 16,
        "relocating_shards" : 0,
        "initializing_shards" : 0,
        "unassigned_shards" : 1,
        "delayed_unassigned_shards" : 0,
        "number_of_pending_tasks" : 0,
        "number_of_in_flight_fetch" : 0,
        "task_max_waiting_in_queue_millis" : 0,
        "active_shards_percent_as_number" : 100.0
      }

      1

      Make sure this parameter value is green or yellow before proceeding.

7. If you changed the Elasticsearch configuration map, repeat these steps for each Elasticsearch pod.

8. After all the deployments for the cluster have been rolled out, re-enable shard balancing:

   $ oc exec <any_es_pod_in_the_cluster> -c elasticsearch -- es_util --query="_cluster/settings" -XPUT -d '{ "persistent": { "cluster.routing.allocation.enable" : "all" } }'

   For example:

   $ oc exec elasticsearch-cdm-5ceex6ts-1-dcd6c4c7c-jpw6 -c elasticsearch -- es_util --query="_cluster/settings" -XPUT -d '{ "persistent": { "cluster.routing.allocation.enable" : "all" } }'

   Example output

   {
     "acknowledged" : true,
     "persistent" : { },
     "transient" : {
       "cluster" : {
         "routing" : {
           "allocation" : {
             "enable" : "all"
           }
         }
       }
     }
   }

9. Scale up the Fluentd pods so they send new logs to Elasticsearch.

   $ oc -n openshift-logging patch daemonset/logging-fluentd -p '{"spec":{"template":{"spec":{"nodeSelector":{"logging-infra-fluentd": "true"}}}}}'

1. Change to the

   openshift-logging

   project:

   $ oc project openshift-logging

2. Extract the CA certificate from the log store and write to the admin-ca file:

   $ oc extract secret/elasticsearch --to=. --keys=admin-ca

   Example output

   admin-ca

3. Create the route for the log store service as a YAML file:

   1. Create a YAML file with the following:

      apiVersion: route.openshift.io/v1
      kind: Route
      metadata:
        name: elasticsearch
        namespace: openshift-logging
      spec:
        host:
        to:
          kind: Service
          name: elasticsearch
        tls:
          termination: reencrypt
          destinationCACertificate: | 

      1

      1

      Add the log store CA certifcate or use the command in the next step. You do not have to set the spec.tls.key, spec.tls.certificate, and spec.tls.caCertificate parameters required by some reencrypt routes.

   2. Run the following command to add the log store CA certificate to the route YAML you created in the previous step:

      $ cat ./admin-ca | sed -e "s/^/      /" >> <file-name>.yaml

   3. Create the route:

      $ oc create -f <file-name>.yaml

      Example output

      route.route.openshift.io/elasticsearch created

4. Check that the Elasticsearch service is exposed:

   1. Get the token of this service account to be used in the request:

      $ token=$(oc whoami -t)

   2. Set the elasticsearch route you created as an environment variable.

      $ routeES=`oc get route elasticsearch -o jsonpath={.spec.host}`

   3. To verify the route was successfully created, run the following command that accesses Elasticsearch through the exposed route:

      curl -tlsv1.2 --insecure -H "Authorization: Bearer ${token}" "https://${routeES}"

      The response appears similar to the following:

      Example output

      {
        "name" : "elasticsearch-cdm-i40ktba0-1",
        "cluster_name" : "elasticsearch",
        "cluster_uuid" : "0eY-tJzcR3KOdpgeMJo-MQ",
        "version" : {
        "number" : "6.8.1",
        "build_flavor" : "oss",
        "build_type" : "zip",
        "build_hash" : "Unknown",
        "build_date" : "Unknown",
        "build_snapshot" : true,
        "lucene_version" : "7.7.0",
        "minimum_wire_compatibility_version" : "5.6.0",
        "minimum_index_compatibility_version" : "5.0.0"
      },
        "<tagline>" : "<for search>"
      }

Procedure

1. Edit the

   ClusterLogging

   custom resource (CR) in the

   openshift-logging

   project:

   $ oc -n openshift-logging edit ClusterLogging instance

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
     namespace: openshift-logging
   
   ...
   
   spec:
     managementState: "Managed"
     logStore:
       type: "elasticsearch"
       elasticsearch:
         nodeCount: 3
         resources: 

   1

   
           limits:
             memory: 16Gi
           requests:
             cpu: 200m
             memory: 16Gi
         storage:
           storageClassName: "gp2"
           size: "200G"
         redundancyPolicy: "SingleRedundancy"
     visualization:
       type: "kibana"
       kibana:
         resources: 

   2

   
           limits:
             memory: 1Gi
           requests:
             cpu: 500m
             memory: 1Gi
         proxy:
           resources: 

   3

   
             limits:
               memory: 100Mi
             requests:
               cpu: 100m
               memory: 100Mi
         replicas: 2
     collection:
       logs:
         type: "fluentd"
         fluentd:
           resources: 

   4

   
             limits:
               memory: 736Mi
             requests:
               cpu: 200m
               memory: 736Mi

   1

   Specify the CPU and memory limits and requests for the log store as needed. For Elasticsearch, you must adjust both the request value and the limit value.

   2 3

   Specify the CPU and memory limits and requests for the log visualizer as needed.

   4

   Specify the CPU and memory limits and requests for the log collector as needed.

Procedure

1. Edit the

   ClusterLogging

   custom resource (CR) in the

   openshift-logging

   project:

   $ oc edit ClusterLogging instance

   $ oc edit ClusterLogging instance
   
   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   
   ....
   
   spec:
       visualization:
         type: "kibana"
         kibana:
           replicas: 1 

   1

   1

   Specify the number of Kibana nodes.

Procedure

1. Edit the

   ClusterLogging

   custom resource (CR) in the

   openshift-logging

   project:

   $ oc -n openshift-logging edit ClusterLogging instance

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
     namespace: openshift-logging
   
   ...
   
   spec:
     managementState: "Managed"
     logStore:
       type: "elasticsearch"
       elasticsearch:
         nodeCount: 3
         resources: 

   1

   
           limits:
             memory: 16Gi
           requests:
             cpu: 200m
             memory: 16Gi
         storage:
           storageClassName: "gp2"
           size: "200G"
         redundancyPolicy: "SingleRedundancy"
     visualization:
       type: "kibana"
       kibana:
         resources: 

   2

   
           limits:
             memory: 1Gi
           requests:
             cpu: 500m
             memory: 1Gi
         proxy:
           resources: 

   3

   
             limits:
               memory: 100Mi
             requests:
               cpu: 100m
               memory: 100Mi
         replicas: 2
     collection:
       logs:
         type: "fluentd"
         fluentd:
           resources: 

   4

   
             limits:
               memory: 736Mi
             requests:
               cpu: 200m
               memory: 736Mi

   1

   Specify the CPU and memory limits and requests for the log store as needed. For Elasticsearch, you must adjust both the request value and the limit value.

   2 3

   Specify the CPU and memory limits and requests for the log visualizer as needed.

   4

   Specify the CPU and memory limits and requests for the log collector as needed.

Procedure

1. Use the following command to add a taint to a node where you want to schedule the OpenShift Logging pods:

   $ oc adm taint nodes <node-name> <key>=<value>:<effect>

   For example:

   $ oc adm taint nodes node1 elasticsearch=node:NoExecute

   This example places a taint on

   node1

   that has key

   elasticsearch

   , value

   node

   , and taint effect

   NoExecute

   . Nodes with the

   NoExecute

   effect schedule only pods that match the taint and remove existing pods that do not match.

2. Edit the

   logstore

   section of the

   ClusterLogging

   CR to configure a toleration for the Elasticsearch pods:

     logStore:
       type: "elasticsearch"
       elasticsearch:
         nodeCount: 1
         tolerations:
         - key: "elasticsearch"  

   1

   
           operator: "Exists"  

   2

   
           effect: "NoExecute"  

   3

   
           tolerationSeconds: 6000  

   4

   1

   Specify the key that you added to the node.

   2

   Specify the Exists operator to require a taint with the key elasticsearch to be present on the Node.

   3

   Specify the NoExecute effect.

   4

   Optionally, specify the tolerationSeconds parameter to set how long a pod can remain bound to a node before being evicted.

Procedure

1. Use the following command to add a taint to a node where you want to schedule the log visualizer pod:

   $ oc adm taint nodes <node-name> <key>=<value>:<effect>

   For example:

   $ oc adm taint nodes node1 kibana=node:NoExecute

   This example places a taint on

   node1

   that has key

   kibana

   , value

   node

   , and taint effect

   NoExecute

   . You must use the

   NoExecute

   taint effect.

   NoExecute

   schedules only pods that match the taint and remove existing pods that do not match.

2. Edit the

   visualization

   section of the

   ClusterLogging

   CR to configure a toleration for the Kibana pod:

     visualization:
       type: "kibana"
       kibana:
         tolerations:
         - key: "kibana"  

   1

   
           operator: "Exists"  

   2

   
           effect: "NoExecute"  

   3

   
           tolerationSeconds: 6000 

   4

   1

   Specify the key that you added to the node.

   2

   Specify the Exists operator to require the key/value/effect parameters to match.

   3

   Specify the NoExecute effect.

   4

   Optionally, specify the tolerationSeconds parameter to set how long a pod can remain bound to a node before being evicted.

Procedure

1. Use the following command to add a taint to a node where you want logging collector pods to schedule logging collector pods:

   $ oc adm taint nodes <node-name> <key>=<value>:<effect>

   For example:

   $ oc adm taint nodes node1 collector=node:NoExecute

   This example places a taint on

   node1

   that has key

   collector

   , value

   node

   , and taint effect

   NoExecute

   . You must use the

   NoExecute

   taint effect.

   NoExecute

   schedules only pods that match the taint and removes existing pods that do not match.

2. Edit the

   collection

   stanza of the

   ClusterLogging

   custom resource (CR) to configure a toleration for the logging collector pods:

     collection:
       logs:
         type: "fluentd"
         fluentd:
           tolerations:
           - key: "collector"  

   1

   
             operator: "Exists"  

   2

   
             effect: "NoExecute"  

   3

   
             tolerationSeconds: 6000  

   4

   1

   Specify the key that you added to the node.

   2

   Specify the Exists operator to require the key/value/effect parameters to match.

   3

   Specify the NoExecute effect.

   4

   Optionally, specify the tolerationSeconds parameter to set how long a pod can remain bound to a node before being evicted.

Procedure

1. Edit the

   ClusterLogging

   custom resource (CR) in the

   openshift-logging

   project:

   $ oc edit ClusterLogging instance

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogging
   
   ...
   
   spec:
     collection:
       logs:
         fluentd:
           resources: null
         type: fluentd
     logStore:
       elasticsearch:
         nodeCount: 3
         nodeSelector: 

   1

   
           node-role.kubernetes.io/infra: ''
         tolerations:
         - effect: NoSchedule
           key: node-role.kubernetes.io/infra
           value: reserved
         - effect: NoExecute
           key: node-role.kubernetes.io/infra
           value: reserved
         redundancyPolicy: SingleRedundancy
         resources:
           limits:
             cpu: 500m
             memory: 16Gi
           requests:
             cpu: 500m
             memory: 16Gi
         storage: {}
       type: elasticsearch
     managementState: Managed
     visualization:
       kibana:
         nodeSelector: 

   2

   
           node-role.kubernetes.io/infra: ''
         tolerations:
         - effect: NoSchedule
           key: node-role.kubernetes.io/infra
           value: reserved
         - effect: NoExecute
           key: node-role.kubernetes.io/infra
           value: reserved
         proxy:
           resources: null
         replicas: 1
         resources: null
       type: kibana
   
   ...

   1 2

   Add a nodeSelector parameter with the appropriate value to the component you want to move. You can use a nodeSelector in the format shown or use <key>: <value> pairs, based on the value specified for the node. If you added a taint to the infrasructure node, also add a matching toleration.

Procedure

1. Create a Butane config file,

   40-worker-custom-journald.bu

   , that includes an

   /etc/systemd/journald.conf

   file with the required settings.
   Note

   See "Creating machine configs with Butane" for information about Butane.

   variant: openshift
   version: 4.8.0
   metadata:
     name: 40-worker-custom-journald
     labels:
       machineconfiguration.openshift.io/role: "worker"
   storage:
     files:
     - path: /etc/systemd/journald.conf
       mode: 0644 

   1

   
       overwrite: true
       contents:
         inline: |
           Compress=yes 

   2

   
           ForwardToConsole=no 

   3

   
           ForwardToSyslog=no
           MaxRetentionSec=1month 

   4

   
           RateLimitBurst=10000 

   5

   
           RateLimitIntervalSec=30s
           Storage=persistent 

   6

   
           SyncIntervalSec=1s 

   7

   
           SystemMaxUse=8G 

   8

   
           SystemKeepFree=20% 

   9

   
           SystemMaxFileSize=10M 

   10

   1

   Set the permissions for the journal.conf file. It is recommended to set 0644 permissions.

   2

   Specify whether you want logs compressed before they are written to the file system. Specify yes to compress the message or no to not compress. The default is yes.

   3

   Configure whether to forward log messages. Defaults to no for each. Specify:

   • ForwardToConsole

     to forward logs to the system console.

   • ForwardToKsmg

     to forward logs to the kernel log buffer.

   • ForwardToSyslog

     to forward to a syslog daemon.

   • ForwardToWall

     to forward messages as wall messages to all logged-in users.

   4

   Specify the maximum time to store journal entries. Enter a number to specify seconds. Or include a unit: "year", "month", "week", "day", "h" or "m". Enter 0 to disable. The default is 1month.

   5

   Configure rate limiting. If more logs are received than what is specified in RateLimitBurst during the time interval defined by RateLimitIntervalSec, all further messages within the interval are dropped until the interval is over. It is recommended to set RateLimitIntervalSec=30s and RateLimitBurst=10000, which are the defaults.

   6

   Specify how logs are stored. The default is persistent:

   • volatile

     to store logs in memory in

     /var/log/journal/

     .

   • persistent

     to store logs to disk in

     /var/log/journal/

     . systemd creates the directory if it does not exist.

   • auto

     to store logs in

     /var/log/journal/

     if the directory exists. If it does not exist, systemd temporarily stores logs in

     /run/systemd/journal

     .

   • none

     to not store logs. systemd drops all logs.

   7

   Specify the timeout before synchronizing journal files to disk for ERR, WARNING, NOTICE, INFO, and DEBUG logs. systemd immediately syncs after receiving a CRIT, ALERT, or EMERG log. The default is 1s.

   8

   Specify the maximum size the journal can use. The default is 8G.

   9

   Specify how much disk space systemd must leave free. The default is 20%.

   10

   Specify the maximum size for individual journal files stored persistently in /var/log/journal. The default is 10M.

   Note

   If you are removing the rate limit, you might see increased CPU utilization on the system logging daemons as it processes any messages that would have previously been throttled.

   For more information on systemd settings, see https://www.freedesktop.org/software/systemd/man/journald.conf.html. The default settings listed on that page might not apply to OpenShift Container Platform.

2. Use Butane to generate a

   MachineConfig

   object file,

   40-worker-custom-journald.yaml

   , containing the configuration to be delivered to the nodes:

   $ butane 40-worker-custom-journald.bu -o 40-worker-custom-journald.yaml

3. Apply the machine config. For example:

   $ oc apply -f 40-worker-custom-journald.yaml

   The controller detects the new

   MachineConfig

   object and generates a new

   rendered-worker-<hash>

   version.

4. Monitor the status of the rollout of the new rendered configuration to each node:

   $ oc describe machineconfigpool/worker

   Example output

   Name:         worker
   Namespace:
   Labels:       machineconfiguration.openshift.io/mco-built-in=
   Annotations:  <none>
   API Version:  machineconfiguration.openshift.io/v1
   Kind:         MachineConfigPool
   
   ...
   
   Conditions:
     Message:
     Reason:                All nodes are updating to rendered-worker-913514517bcea7c93bd446f4830bc64e