Chapter 8. Configuring your cluster logging deployment

Procedure

1. Edit the Cluster Logging Custom Resource (CR) in the openshift-logging project:

   $ oc edit ClusterLogging instance

   $ oc edit ClusterLogging instance
   
   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   
   ....
   
   spec:
     managementState: "Managed" 1

   1

   Specify the management state as Managed or Unmanaged.

Procedure

1. Edit the Cluster Logging Custom Resource (CR) in the openshift-logging project:

   $ oc edit ClusterLogging instance

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   ....
   spec:
       logStore:
         type: "elasticsearch"
         elasticsearch:
           resources: 1
             limits:
               memory: "16Gi"
             requests:
               cpu: "1"
               memory: "16Gi"

   1

   Specify the CPU and memory limits as needed. If you leave these values blank, the Elasticsearch Operator sets default values that should be sufficient for most deployments.

   If you adjust the amount of Elasticsearch CPU and memory, you must change both the request value and the limit value.

   For example:

         resources:
           limits:
             cpu: "8"
             memory: "32Gi"
           requests:
             cpu: "8"
             memory: "32Gi"

   Kubernetes generally adheres the node CPU configuration and DOES not allow Elasticsearch to use the specified limits. Setting the same value for the requests and limits ensures that Elasticseach can use the CPU and memory you want, assuming the node has the CPU and memory available.

Procedure

1. Edit the Cluster Logging Custom Resource (CR) in the openshift-logging project:

   oc edit clusterlogging instance

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   
   ....
   
   spec:
     logStore:
       type: "elasticsearch"
       elasticsearch:
         redundancyPolicy: "SingleRedundancy" 1

   1

   Specify a redundancy policy for the shards. The change is applied upon saving the changes.

   • FullRedundancy. Elasticsearch fully replicates the primary shards for each index to every data node. This provides the highest safety, but at the cost of the highest amount of disk required and the poorest performance.
   • MultipleRedundancy. Elasticsearch fully replicates the primary shards for each index to half of the data nodes. This provides a good tradeoff between safety and performance.
   • SingleRedundancy. Elasticsearch makes one copy of the primary shards for each index. Logs are always available and recoverable as long as at least two data nodes exist. Better performance than MultipleRedundancy, when using 5 or more nodes. You cannot apply this policy on deployments of single Elasticsearch node.
   • ZeroRedundancy. Elasticsearch does not make copies of the primary shards. Logs might be unavailable or lost in the event a node is down or fails. Use this mode when you are more concerned with performance than safety, or have implemented your own disk/PVC backup/restore strategy.

Procedure

1. Edit the Cluster Logging CR to specify that each data node in the cluster is bound to a Persistent Volume Claim.

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   
   ....
   
    spec:
       logStore:
         type: "elasticsearch"
         elasticsearch:
           nodeCount: 3
           storage:
             storageClassName: "gp2"
             size: "200G"

Procedure

1. Edit the Cluster Logging CR to specify emptyDir:

    spec:
       logStore:
         type: "elasticsearch"
         elasticsearch:
           nodeCount: 3
           storage: {}

1. Change to the openshift-logging project:

   $ oc project openshift-logging

2. Extract the CA certificate from Elasticsearch and write to the admin-ca file:

   $ oc extract secret/elasticsearch --to=. --keys=admin-ca
   
   admin-ca

3. Create the route for the Elasticsearch service as a YAML file:

   1. Create a YAML file with the following:

      apiVersion: route.openshift.io/v1
      kind: Route
      metadata:
        name: elasticsearch
        namespace: openshift-logging
      spec:
        host:
        to:
          kind: Service
          name: elasticsearch
        tls:
          termination: reencrypt
          destinationCACertificate: | 1

      1

      Add the Elasticsearch CA certifcate or use the command in the next step. You do not have to set the spec.tls.key, spec.tls.certificate, and spec.tls.caCertificate parameters required by some reencrypt routes.

   2. Run the following command to add the Elasticsearch CA certificate to the route YAML you created:

      cat ./admin-ca | sed -e "s/^/      /" >> <file-name>.yaml

   3. Create the route:

      $ oc create -f <file-name>.yaml
      
      route.route.openshift.io/elasticsearch created

4. Check that the Elasticsearch service is exposed:

   1. Get the token of this ServiceAccount to be used in the request:

      $ token=$(oc whoami -t)

   2. Set the elasticsearch route you created as an environment variable.

      $ routeES=`oc get route elasticsearch -o jsonpath={.spec.host}`

   3. To verify the route was successfully created, run the following command that accesses Elasticsearch through the exposed route:

      curl -tlsv1.2 --insecure -H "Authorization: Bearer ${token}" "https://${routeES}/.operations.*/_search?size=1" | jq

      The response appears similar to the following:

        % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                       Dload  Upload   Total   Spent    Left  Speed
      100   944  100   944    0     0     62      0  0:00:15  0:00:15 --:--:--   204
      {
        "took": 441,
        "timed_out": false,
        "_shards": {
          "total": 3,
          "successful": 3,
          "skipped": 0,
          "failed": 0
        },
        "hits": {
          "total": 89157,
          "max_score": 1,
          "hits": [
            {
              "_index": ".operations.2019.03.15",
              "_type": "com.example.viaq.common",
              "_id": "ODdiNWIyYzAtMjg5Ni0TAtNWE3MDY1MjMzNTc3",
              "_score": 1,
              "_source": {
                "_SOURCE_MONOTONIC_TIMESTAMP": "673396",
                "systemd": {
                  "t": {
                    "BOOT_ID": "246c34ee9cdeecb41a608e94",
                    "MACHINE_ID": "e904a0bb5efd3e36badee0c",
                    "TRANSPORT": "kernel"
                  },
                  "u": {
                    "SYSLOG_FACILITY": "0",
                    "SYSLOG_IDENTIFIER": "kernel"
                  }
                },
                "level": "info",
                "message": "acpiphp: Slot [30] registered",
                "hostname": "localhost.localdomain",
                "pipeline_metadata": {
                  "collector": {
                    "ipaddr4": "10.128.2.12",
                    "ipaddr6": "fe80::xx:xxxx:fe4c:5b09",
                    "inputname": "fluent-plugin-systemd",
                    "name": "fluentd",
                    "received_at": "2019-03-15T20:25:06.273017+00:00",
                    "version": "1.3.2 1.6.0"
                  }
                },
                "@timestamp": "2019-03-15T20:00:13.808226+00:00",
                "viaq_msg_id": "ODdiNWIyYzAtMYTAtNWE3MDY1MjMzNTc3"
              }
            }
          ]
        }
      }

Procedure

1. Edit the Cluster Logging Custom Resource (CR) in the openshift-logging project:

   $ oc edit ClusterLogging instance

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   
   ....
   
   spec:
       visualization:
         type: "kibana"
         kibana:
           replicas:
         resources:  1
           limits:
             memory: 1Gi
           requests:
             cpu: 500m
             memory: 1Gi
         proxy:  2
           resources:
             limits:
               memory: 100Mi
             requests:
               cpu: 100m
               memory: 100Mi

   1

   Specify the CPU and memory limits to allocate for each node.

   2

   Specify the CPU and memory limits to allocate to the Kibana proxy.

1. Edit the Cluster Logging Custom Resource (CR) in the openshift-logging project:

   $ oc edit ClusterLogging instance

   $ oc edit ClusterLogging instance
   
   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   
   ....
   
   spec:
       visualization:
         type: "kibana"
         kibana:
           replicas: 1 1

   1

   Specify the number of Kibana nodes.

Procedure

1. Use the following command to add a taint to a node where you want to schedule the Kibana Pod:

   $ oc adm taint nodes <node-name> <key>=<value>:<effect>

   For example:

   $ oc adm taint nodes node1 kibana=node:NoExecute

   This example places a taint on node1 that has key kibana, value node, and taint effect NoExecute. You must use the NoExecute taint effect. NoExecute schedules only Pods that match the taint and remove existing Pods that do not match.

2. Edit the visualization section of the Cluster Logging Custom Resource (CR) to configure a toleration for the Kibana Pod:

     visualization:
       type: "kibana"
       kibana:
         tolerations:
         - key: "kibana"  1
           operator: "Exists"  2
           effect: "NoExecute"  3
           tolerationSeconds: 6000 4

   1

   Specify the key that you added to the node.

   2

   Specify the Exists operator to require the key/value/effect parameters to match.

   3

   Specify the NoExecute effect.

   4

   Optionally, specify the tolerationSeconds parameter to set how long a Pod can remain bound to a node before being evicted.

1. If necessary, get the Kibana route, which is created by default upon installation of the Cluster Logging Operator:

   $ oc get routes -n openshift-logging
   
   NAMESPACE                  NAME                       HOST/PORT                                                            PATH     SERVICES                   PORT    TERMINATION          WILDCARD
   openshift-logging          kibana                     kibana-openshift-logging.apps.openshift.com                                   kibana                     <all>   reencrypt/Redirect   None

2. Get the name of your Elasticsearch pods.

   $ oc get pods -l component=elasticsearch
   
   NAME                                            READY   STATUS    RESTARTS   AGE
   elasticsearch-cdm-5ceex6ts-1-dcd6c4c7c-jpw6k    2/2     Running   0          22h
   elasticsearch-cdm-5ceex6ts-2-f799564cb-l9mj7    2/2     Running   0          22h
   elasticsearch-cdm-5ceex6ts-3-585968dc68-k7kjr   2/2     Running   0          22h

3. Create the necessary per-user configuration that this procedure requires:

   1. Log in to the Kibana dashboard as the user you want to add the dashboards to.

      https://kibana-openshift-logging.apps.openshift.com 1

      1

      Where the URL is Kibana route.

   2. If the Authorize Access page appears, select all permissions and click Allow selected permissions.
   3. Log out of the Kibana dashboard.

4. Run the following command from the project where the pod is located using the name of any of your Elastiscearch pods:

   $ oc exec <es-pod> -- es_load_kibana_ui_objects <user-name>

   For example:

   $ oc exec elasticsearch-cdm-5ceex6ts-1-dcd6c4c7c-jpw6k -- es_load_kibana_ui_objects <user-name>

1. Edit the Cluster Logging Custom Resource in the openshift-logging project:

   $ oc edit clusterlogging instance

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   
   ...
   
     curation:
       curator:
         schedule: 30 3 * * * 1
       type: curator

   1

   Specify the schedule for Curator in cron format.

   Note

   The time zone is set based on the host node where the Curator pod runs.

1. Edit the OpenShift Container Platform custom Curator configuration file:

   $ oc edit configmap/curator

2. Set the following parameters as needed:

   config.yaml: |
     project_name:
       action
         unit:value

   The available parameters are:

   Table 8.1. Project options

   Variable Name	Description
   project_name	The actual name of a project, such as myapp-devel. For OpenShift Container Platform operations logs, use the name .operations as the project name.
   action	The action to take, currently only delete is allowed.
   unit	The period to use for deletion, days, weeks, or months.
   value	The number of units.

   Table 8.2. Filter options

   Variable Name	Description
   .defaults	Use .defaults as the project_name to set the defaults for projects that are not specified.
   .regex	The list of regular expressions that match project names.
   pattern	The valid and properly escaped regular expression pattern enclosed by single quotation marks.

1. Enable debug log of Curator:

   $ oc set env cronjob/curator CURATOR_LOG_LEVEL=DEBUG CURATOR_SCRIPT_LOG_LEVEL=DEBUG

   Specify the log level:

   • CRITICAL. Curator displays only critical messages.
   • ERROR. Curator displays only error and critical messages.
   • WARNING. Curator displays only error, warning, and critical messages.
   • INFO. Curator displays only informational, error, warning, and critical messages.

   • DEBUG. Curator displays only debug messages, in addition to all of the above.

     The default value is INFO.

     Note

     Cluster logging uses the OpenShift Container Platform custom environment variable CURATOR_SCRIPT_LOG_LEVEL in OpenShift Container Platform wrapper scripts (run.sh and convert.py). The environment variable takes the same values as CURATOR_LOG_LEVEL for script debugging, as needed.

2. Trigger next curator iteration:

   $ oc create job --from=cronjob/curator <job_name>

3. Use the following commands to control the CronJob:

   • Suspend a CronJob:

     $ oc patch cronjob curator -p '{"spec":{"suspend":true}}'

   • Resume a CronJob:

     $ oc patch cronjob curator -p '{"spec":{"suspend":false}}'

   • Change a CronJob schedule:

     $ oc patch cronjob curator -p '{"spec":{"schedule":"0 0 * * *"}}' 1

     1

     The schedule option accepts schedules in cron format.

1. Edit the Curator ConfigMap:

   oc edit cm/curator -n openshift-logging

2. Make the following changes to the action file:

   actions:
   1:
         action: delete_indices 1
         description: >-
           Delete .operations indices older than 30 days.
           Ignore the error if the filter does not
           result in an actionable list of indices (ignore_empty_list).
           See https://www.elastic.co/guide/en/elasticsearch/client/curator/5.2/ex_delete_indices.html
         options:
           # Swallow curator.exception.NoIndices exception
           ignore_empty_list: True
           # In seconds, default is 300
           timeout_override: ${CURATOR_TIMEOUT}
           # Don't swallow any other exceptions
           continue_if_exception: False
           # Optionally disable action, useful for debugging
           disable_action: False
         # All filters are bound by logical AND
         filters:            2
         - filtertype: pattern
           kind: regex
           value: '^\.operations\..*$'
           exclude: False    3
         - filtertype: age
           # Parse timestamp from index name
           source: name
           direction: older
           timestring: '%Y.%m.%d'
           unit: days
           unit_count: 30
           exclude: False

   1

   Specify delete_indices to delete the specified index.

   2

   Use the filers parameters to specify the index to be deleted. See the Elastic Search curator documentation for information on these parameters.

   3

   Specify false to allow the index to be deleted.

Procedure

1. Edit the Cluster Logging Custom Resource (CR) in the openshift-logging project:

   $ oc edit ClusterLogging instance

   $ oc edit ClusterLogging instance
   
   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   
   ....
   
   spec:
     collection:
       logs:
         fluentd:
           resources:
             limits: 1
               cpu: 250m
               memory: 1Gi
             requests:
               cpu: 250m
               memory: 1Gi

   1

   Specify the CPU and memory limits and requests as needed. The values shown are the default values.

1. Edit the LOGGING_FILE_PATH parameter in the fluentd daemonset. You can specify a particular file or console:

   spec:
     template:
       spec:
         containers:
             env:
               - name: LOGGING_FILE_PATH
                 value: console 1

   1

   Specify the log output method:

   • use console to use the Fluentd default location. Retrieve the logs with the oc logs [-f] <pod_name> command.

   • use <path-to-log/fluentd.log> to send the log output to the specified file. Retrieve the logs with the oc exec <pod_name> — logs command. This is the default setting.

     Or, use the CLI:

     $ oc -n openshift-logging set env daemonset/fluentd LOGGING_FILE_PATH=/logs/fluentd.log

Procedure

1. To configure Fluentd to restrict specific projects, edit the throttle configuration in the Fluentd ConfigMap after deployment:

   $ oc edit configmap/fluentd

   The format of the throttle-config.yaml key is a YAML file that contains project names and the desired rate at which logs are read in on each node. The default is 1000 lines at a time per node. For example:

   throttle-config.yaml: |
     - opensift-logging:
         read_lines_limit: 10
     - .operations:
         read_lines_limit: 100

Procedure

1. Use the following command to add a taint to a node where you want to schedule the cluster logging Pods:

   $ oc adm taint nodes <node-name> <key>=<value>:<effect>

   For example:

   $ oc adm taint nodes node1 elasticsearch=node:NoExecute

   This example places a taint on node1 that has key elasticsearch, value node, and taint effect NoExecute. Nodes with the NoExecute effect schedule only Pods that match the taint and remove existing Pods that do not match.

2. Edit the logstore section of the Cluster Logging Custom Resource (CR) to configure a toleration for the Elasticsearch Pods:

     logStore:
       type: "elasticsearch"
       elasticsearch:
         nodeCount: 1
         tolerations:
         - key: "elasticsearch"  1
           operator: "Exists"  2
           effect: "NoExecute"  3
           tolerationSeconds: 6000  4

   1

   Specify the key that you added to the node.

   2

   Specify the Exists operator to require a taint with the key elasticsearch to be present on the Node.

   3

   Specify the NoExecute effect.

   4

   Optionally, specify the tolerationSeconds parameter to set how long a Pod can remain bound to a node before being evicted.

Procedure

1. Use the following command to add a taint to a node where you want to schedule the Kibana Pod:

   $ oc adm taint nodes <node-name> <key>=<value>:<effect>

   For example:

   $ oc adm taint nodes node1 kibana=node:NoExecute

   This example places a taint on node1 that has key kibana, value node, and taint effect NoExecute. You must use the NoExecute taint effect. NoExecute schedules only Pods that match the taint and remove existing Pods that do not match.

2. Edit the visualization section of the Cluster Logging Custom Resource (CR) to configure a toleration for the Kibana Pod:

     visualization:
       type: "kibana"
       kibana:
         tolerations:
         - key: "kibana"  1
           operator: "Exists"  2
           effect: "NoExecute"  3
           tolerationSeconds: 6000 4

   1

   Specify the key that you added to the node.

   2

   Specify the Exists operator to require the key/value/effect parameters to match.

   3

   Specify the NoExecute effect.

   4

   Optionally, specify the tolerationSeconds parameter to set how long a Pod can remain bound to a node before being evicted.

Procedure

1. Use the following command to add a taint to a node where you want to schedule the Curator Pod:

   $ oc adm taint nodes <node-name> <key>=<value>:<effect>

   For example:

   $ oc adm taint nodes node1 curator=node:NoExecute

   This example places a taint on node1 that has key curator, value node, and taint effect NoExecute. You must use the NoExecute taint effect. NoExecute schedules only Pods that match the taint and remove existing Pods that do not match.

2. Edit the curation section of the Cluster Logging Custom Resource (CR) to configure a toleration for the Curator Pod:

     curation:
       type: "curator"
       curator:
         tolerations:
         - key: "curator"  1
           operator: "Exists"  2
           effect: "NoExecute"  3
           tolerationSeconds: 6000  4

   1

   Specify the key that you added to the node.

   2

   Specify the Exists operator to require the key/value/effect parameters to match.

   3

   Specify the NoExecute effect.

   4

   Optionally, specify the tolerationSeconds parameter to set how long a Pod can remain bound to a node before being evicted.

Procedure

1. Use the following command to add a taint to a node where you want logging collector Pods to schedule logging collector Pods:

   $ oc adm taint nodes <node-name> <key>=<value>:<effect>

   For example:

   $ oc adm taint nodes node1 collector=node:NoExecute

   This example places a taint on node1 that has key collector, value node, and taint effect NoExecute. You must use the NoExecute taint effect. NoExecute schedules only Pods that match the taint and removes existing Pods that do not match.

2. Edit the collection section of the Cluster Logging Custom Resource (CR) to configure a toleration for the logging collector Pods:

     collection:
       logs:
         type: "fluentd"
         rsyslog:
           tolerations:
           - key: "collector"  1
             operator: "Exists"  2
             effect: "NoExecute"  3
             tolerationSeconds: 6000  4

   1

   Specify the key that you added to the node.

   2

   Specify the Exists operator to require the key/value/effect parameters to match.

   3

   Specify the NoExecute effect.

   4

   Optionally, specify the tolerationSeconds parameter to set how long a Pod can remain bound to a node before being evicted.

1. Edit the fluentd DaemonSet in the openshift-logging project:

   $ oc edit ds/fluentd
   
   spec:
     template:
       spec:
         containers:
             env:
             - name: ES_HOST
               value: elasticsearch
             - name: ES_PORT
               value: '9200'
             - name: ES_CLIENT_CERT
               value: /etc/fluent/keys/app-cert
             - name: ES_CLIENT_KEY
               value: /etc/fluent/keys/app-key
             - name: ES_CA
               value: /etc/fluent/keys/app-ca
             - name: OPS_HOST
               value: elasticsearch
             - name: OPS_PORT
               value: '9200'
             - name: OPS_CLIENT_CERT
               value: /etc/fluent/keys/infra-cert
             - name: OPS_CLIENT_KEY
               value: /etc/fluent/keys/infra-key
             - name: OPS_CA
               value: /etc/fluent/keys/infra-ca

2. Set ES_HOST and OPS_HOST to the same destination, while ensuring that ES_PORT and OPS_PORT also have the same value for an external Elasticsearch instance to contain both application and operations logs.
3. Configure your externally-hosted Elasticsearch instance for TLS. Only externally-hosted Elasticsearch instances that use Mutual TLS are allowed.

Procedure

1. Set environment variables in the fluentd daemonset in the openshift-logging project:

   spec:
     template:
       spec:
         containers:
           - name: fluentd
             image: 'registry.redhat.io/openshift4/ose-logging-fluentd:v4.2'
             env:
               - name: REMOTE_SYSLOG_HOST 1
                 value: host1
               - name: REMOTE_SYSLOG_HOST_BACKUP
                 value: host2
               - name: REMOTE_SYSLOG_PORT_BACKUP
                 value: 5555

   1

   The desired remote syslog host. Required for each host.

   This will build two destinations. The syslog server on host1 will be receiving messages on the default port of 514, while host2 will be receiving the same messages on port 5555.

2. Alternatively, you can configure your own custom the fluentd daemonset in the openshift-logging project.

   Fluentd Environment Variables

   Parameter	Description
   USE_REMOTE_SYSLOG	Defaults to false. Set to true to enable use of the fluent-plugin-remote-syslog gem
   REMOTE_SYSLOG_HOST	(Required) Hostname or IP address of the remote syslog server.
   REMOTE_SYSLOG_PORT	Port number to connect on. Defaults to 514.
   REMOTE_SYSLOG_SEVERITY	Set the syslog severity level. Defaults to debug.
   REMOTE_SYSLOG_FACILITY	Set the syslog facility. Defaults to local0.
   REMOTE_SYSLOG_USE_RECORD	Defaults to false. Set to true to use the record’s severity and facility fields to set on the syslog message.
   REMOTE_SYSLOG_REMOVE_TAG_PREFIX	Removes the prefix from the tag, defaults to '' (empty).
   REMOTE_SYSLOG_TAG_KEY	If specified, uses this field as the key to look on the record, to set the tag on the syslog message.
   REMOTE_SYSLOG_PAYLOAD_KEY	If specified, uses this field as the key to look on the record, to set the payload on the syslog message.
   REMOTE_SYSLOG_TYPE	Set the transport layer protocol type. Defaults to syslog_buffered, which sets the TCP protocol. To switch to UDP, set this to syslog.

   Warning

   This implementation is insecure, and should only be used in environments where you can guarantee no snooping on the connection.

1. Edit the secure-forward.conf section of the Fluentd configuration map:

   $ oc edit configmap/fluentd -n openshift-logging

2. Enter the name, host, and port for your external Fluentd server:

   #   <server>
   #     host server.fqdn.example.com  # or IP
   #     port 24284
   #   </server>
   #   <server>
   #     host 203.0.113.8 # ip address to connect
   #     name server.fqdn.example.com # The name of the server. Used for logging and certificate verification in TLS transport (when host is address).
   #   </server>

   For example:

     <server>
       name externalserver1 1
       host 192.168.1.1 2
       port 24224 3
     </server>
     <server> 4
       name externalserver1
       host 192.168.1.2
       port 24224
     </server>
   </store>

   1

   Optionally, enter a name for this external aggregator.

   2

   Specify the host name or IP of the external aggregator.

   3

   Specify the port of the external aggregator.

   4

   Optionally, add additional external aggregator.

3. Add the path to your CA certificate and private key to the secure-forward.conf section:

   #   <security>
   #     self_hostname ${hostname} # ${hostname} is a placeholder. 1
   #     shared_key <shared_key_between_forwarder_and_forwardee> 2
   #   </security>
   
   #   tls_cert_path /path/for/certificate/ca_cert.pem 3

   1

   Specify the default value of the auto-generated certificate common name (CN).

   2

   Specify a shared key for authentication.

   3

   Specify the path to your CA certificate.

   For example:

      <security>
        self_hostname client.fqdn.local
        shared_key cluster_logging_key
      </security>
   
      tls_cert_path /etc/fluent/keys/ca.crt

   To use mTLS, see the Fluentd documentation for information about client certificate and key parameters and other settings.

4. Add certificates to be used in secure-forward.conf to the existing secret that is mounted on the Fluentd pods. The your_ca_cert and your_private_key values must match what is specified in secure-forward.conf in configmap/fluentd:

   $ oc patch secrets/fluentd --type=json \
     --patch "[{'op':'add','path':'/data/your_ca_cert','value':'$(base64 -w0 /path/to/your_ca_cert.pem)'}]"
   $ oc patch secrets/fluentd --type=json \
     --patch "[{'op':'add','path':'/data/your_private_key','value':'$(base64 -w0 /path/to/your_private_key.pem)'}]"

   Note

   Replace your_private_key with a generic name. This is a link to the JSON path, not a path on your host system.

   For example:

   $ oc patch secrets/fluentd --type=json \
     --patch "[{'op':'add','path':'/data/ca.crt','value':'$(base64 -w0 /etc/fluent/keys/ca.crt)'}]"
   $ oc patch secrets/fluentd --type=json \
     --patch "[{'op':'add','path':'/data/ext-agg','value':'$(base64 -w0 /etc/fluent/keys/ext-agg.pem)'}]"

5. Configure the secure-forward.conf file on the external aggregator to accept messages securely from Fluentd.

   When configuring the external aggregator, it must be able to accept messages securely from Fluentd.

Procedure

1. Create a journald.conf file with the required settings:

   Compress=no 1
   ForwardToConsole=yes 2
   ForwardToSyslog=no 3
   MaxRetentionSec=30 4
   RateLimitBurst=10000 5
   RateLimitInterval=1s 6
   Storage=volatile 7
   SyncIntervalSec=1s 8
   SystemMaxUse=8g 9
   SystemKeepFree=20% 10
   SystemMaxFileSize10M 11

   1

   Specify whether you want logs compressed before they are written to the file system. Specify yes to compress the message or no to not compress. The default is yes.

   2 3

   Configure whether to forward log messages. Defaults to no for each. Specify:

   • ForwardToConsole to forward logs to the system console.
   • ForwardToKsmg to forward logs to the kernel log buffer.
   • ForwardToSyslog to forward to a syslog daemon.
   • ForwardToWall to forward messages as wall messages to all logged-in users.

   4

   Specify the maximum time to store journal entries. Enter a number to specify seconds. Or include a unit: "year", "month", "week", "day", "h" or "m". Enter 0 to disable. The default is 1month.

   5 6

   Configure rate limiting. If, during the time interval defined by RateLimitIntervalSec, more logs than specified in RateLimitBurst are received, all further messages within the interval are dropped until the interval is over. It is recommended to set RateLimitInterval=1s and RateLimitBurst=10000, which are the defaults.

   7

   Specify how logs are stored. The default is persistent:

   • volatile to store logs in memory in /var/log/journal/.
   • persistent to store logs to disk in /var/log/journal/. systemd creates the directory if it does not exist.
   • auto to store logs in in /var/log/journal/ if the directory exists. If it does not exist, systemd temporarily stores logs in /run/systemd/journal.
   • none to not store logs. systemd drops all logs.

   8

   Specify the timeout before synchronizing journal files to disk for ERR, WARNING, NOTICE, INFO, and DEBUG logs. systemd immediately syncs after receiving a CRIT, ALERT, or EMERG log. The default is 1s.

   9

   Specify the maximum size the journal can use. The default is 8g.

   10

   Specify how much disk space systemd must leave free. The default is 20%.

   11

   Specify the maximum size for individual journal files stored persistently in /var/log/journal. The default is 10M.

   Note

   If you are removing the rate limit, you might see increased CPU utilization on the system logging daemons as it processes any messages that would have previously been throttled.

   For more information on systemd settings, see https://www.freedesktop.org/software/systemd/man/journald.conf.html. The default settings listed on that page might not apply to OpenShift Container Platform.

2. Convert the journal.conf file to base64:

   $ export jrnl_cnf=$( cat /journald.conf | base64 -w0 )

3. Create a new MachineConfig for master or worker and add the journal.conf parameters:

   For example:

   ...
   
   config:
     storage:
       files:
       - contents:
           source: data:text/plain;charset=utf-8;base64,${jrnl_cnf}
           verification: {}
         filesystem: root
         mode: 0644 1
         path: /etc/systemd/journald.conf 2
     systemd: {}

   1

   Set the permissions for the journal.conf file. It is recommended to set 0644 permissions.

   2

   Specify the path to the base64-encoded journal.conf file.

4. Create the MachineConfig:

   $ oc apply -f <filename>.yaml

   The controller detects the new MachineConfig and generates a new rendered-worker-<hash> version.

5. Monitor the status of the rollout of the new rendered configuration to each node:

   $ oc describe machineconfigpool/worker
   
   
   Name:         worker
   Namespace:
   Labels:       machineconfiguration.openshift.io/mco-built-in=
   Annotations:  <none>
   API Version:  machineconfiguration.openshift.io/v1
   Kind:         MachineConfigPool
   
   ...
   
   Conditions:
     Message:
     Reason:                All nodes are updating to rendered-worker-913514517bcea7c93bd446f4830bc64e