Logging

Procedure

1. Edit the Cluster Logging Custom Resource (CR) in the openshift-logging project:

   $ oc edit ClusterLogging instance

   Copy to Clipboard Toggle word wrap

   $ oc edit ClusterLogging instance
   
   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   
   ....
   
   spec:
     managementState: "Managed" 

   1

   Copy to Clipboard Toggle word wrap

   1

   Specify the management state as Managed or Unmanaged.

Procedure

1. Edit the Cluster Logging Custom Resource (CR) in the openshift-logging project:

   $ oc edit ClusterLogging instance

   Copy to Clipboard Toggle word wrap

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   ....
   spec:
       logStore:
         type: "elasticsearch"
         elasticsearch:
           resources: 

   1

   
             limits:
               memory: 16Gi
             requests:
               cpu: 500m
               memory: 16Gi

   Copy to Clipboard Toggle word wrap

   1

   Specify the CPU and memory limits as needed. If you leave these values blank, the Elasticsearch Operator sets default values that should be sufficient for most deployments.

   If you adjust the amount of Elasticsearch CPU and memory, you must change both the request value and the limit value.

   For example:

         resources:
           limits:
             cpu: "8"
             memory: "32Gi"
           requests:
             cpu: "8"
             memory: "32Gi"

   Copy to Clipboard Toggle word wrap

   Kubernetes generally adheres the node CPU configuration and DOES not allow Elasticsearch to use the specified limits. Setting the same value for the requests and limits ensures that Elasticseach can use the CPU and memory you want, assuming the node has the CPU and memory available.

Procedure

1. Edit the Cluster Logging Custom Resource (CR) in the openshift-logging project:

   oc edit clusterlogging instance

   Copy to Clipboard Toggle word wrap

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   
   ....
   
   spec:
     logStore:
       type: "elasticsearch"
       elasticsearch:
         redundancyPolicy: "SingleRedundancy" 

   1

   Copy to Clipboard Toggle word wrap

   1

   Specify a redundancy policy for the shards. The change is applied upon saving the changes.

   • FullRedundancy. Elasticsearch fully replicates the primary shards for each index to every data node. This provides the highest safety, but at the cost of the highest amount of disk required and the poorest performance.
   • MultipleRedundancy. Elasticsearch fully replicates the primary shards for each index to half of the data nodes. This provides a good tradeoff between safety and performance.
   • SingleRedundancy. Elasticsearch makes one copy of the primary shards for each index. Logs are always available and recoverable as long as at least two data nodes exist. Better performance than MultipleRedundancy, when using 5 or more nodes. You cannot apply this policy on deployments of single Elasticsearch node.
   • ZeroRedundancy. Elasticsearch does not make copies of the primary shards. Logs might be unavailable or lost in the event a node is down or fails. Use this mode when you are more concerned with performance than safety, or have implemented your own disk/PVC backup/restore strategy.

Procedure

1. Edit the Cluster Logging CR to specify that each data node in the cluster is bound to a Persistent Volume Claim.

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   
   ....
   
    spec:
       logStore:
         type: "elasticsearch"
         elasticsearch:
           nodeCount: 3
           storage:
             storageClassName: "gp2"
             size: "200G"

   Copy to Clipboard Toggle word wrap

Procedure

1. Edit the Cluster Logging CR to specify emptyDir:

    spec:
       logStore:
         type: "elasticsearch"
         elasticsearch:
           nodeCount: 3
           storage: {}

   Copy to Clipboard Toggle word wrap

1. Change to the openshift-logging project:

   $ oc project openshift-logging

   Copy to Clipboard Toggle word wrap

2. Extract the CA certificate from Elasticsearch and write to the admin-ca file:

   $ oc extract secret/elasticsearch --to=. --keys=admin-ca
   
   admin-ca

   Copy to Clipboard Toggle word wrap

3. Create the route for the Elasticsearch service as a YAML file:

   1. Create a YAML file with the following:

      apiVersion: route.openshift.io/v1
      kind: Route
      metadata:
        name: elasticsearch
        namespace: openshift-logging
      spec:
        host:
        to:
          kind: Service
          name: elasticsearch
        tls:
          termination: reencrypt
          destinationCACertificate: | 

      1

      Copy to Clipboard Toggle word wrap

      1

      Add the Elasticsearch CA certifcate or use the command in the next step. You do not have to set the spec.tls.key, spec.tls.certificate, and spec.tls.caCertificate parameters required by some reencrypt routes.

   2. Run the following command to add the Elasticsearch CA certificate to the route YAML you created:

      cat ./admin-ca | sed -e "s/^/      /" >> <file-name>.yaml

      Copy to Clipboard Toggle word wrap

   3. Create the route:

      $ oc create -f <file-name>.yaml
      
      route.route.openshift.io/elasticsearch created

      Copy to Clipboard Toggle word wrap

4. Check that the Elasticsearch service is exposed:

   1. Get the token of this ServiceAccount to be used in the request:

      $ token=$(oc whoami -t)

      Copy to Clipboard Toggle word wrap

   2. Set the elasticsearch route you created as an environment variable.

      $ routeES=`oc get route elasticsearch -o jsonpath={.spec.host}`

      Copy to Clipboard Toggle word wrap

   3. To verify the route was successfully created, run the following command that accesses Elasticsearch through the exposed route:

      curl -tlsv1.2 --insecure -H "Authorization: Bearer ${token}" "https://${routeES}/.operations.*/_search?size=1" | jq

      Copy to Clipboard Toggle word wrap

      The response appears similar to the following:

        % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                       Dload  Upload   Total   Spent    Left  Speed
      100   944  100   944    0     0     62      0  0:00:15  0:00:15 --:--:--   204
      {
        "took": 441,
        "timed_out": false,
        "_shards": {
          "total": 3,
          "successful": 3,
          "skipped": 0,
          "failed": 0
        },
        "hits": {
          "total": 89157,
          "max_score": 1,
          "hits": [
            {
              "_index": ".operations.2019.03.15",
              "_type": "com.example.viaq.common",
              "_id": "ODdiNWIyYzAtMjg5Ni0TAtNWE3MDY1MjMzNTc3",
              "_score": 1,
              "_source": {
                "_SOURCE_MONOTONIC_TIMESTAMP": "673396",
                "systemd": {
                  "t": {
                    "BOOT_ID": "246c34ee9cdeecb41a608e94",
                    "MACHINE_ID": "e904a0bb5efd3e36badee0c",
                    "TRANSPORT": "kernel"
                  },
                  "u": {
                    "SYSLOG_FACILITY": "0",
                    "SYSLOG_IDENTIFIER": "kernel"
                  }
                },
                "level": "info",
                "message": "acpiphp: Slot [30] registered",
                "hostname": "localhost.localdomain",
                "pipeline_metadata": {
                  "collector": {
                    "ipaddr4": "10.128.2.12",
                    "ipaddr6": "fe80::xx:xxxx:fe4c:5b09",
                    "inputname": "fluent-plugin-systemd",
                    "name": "fluentd",
                    "received_at": "2019-03-15T20:25:06.273017+00:00",
                    "version": "1.3.2 1.6.0"
                  }
                },
                "@timestamp": "2019-03-15T20:00:13.808226+00:00",
                "viaq_msg_id": "ODdiNWIyYzAtMYTAtNWE3MDY1MjMzNTc3"
              }
            }
          ]
        }
      }

      Copy to Clipboard Toggle word wrap

Procedure

1. Edit the Cluster Logging Custom Resource (CR) in the openshift-logging project:

   $ oc edit ClusterLogging instance

   Copy to Clipboard Toggle word wrap

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   
   ....
   
   spec:
       visualization:
         type: "kibana"
         kibana:
           replicas:
         resources:  

   1

   
           limits:
             memory: 1Gi
           requests:
             cpu: 500m
             memory: 1Gi
         proxy:  

   2

   
           resources:
             limits:
               memory: 100Mi
             requests:
               cpu: 100m
               memory: 100Mi

   Copy to Clipboard Toggle word wrap

   1

   Specify the CPU and memory limits to allocate for each node.

   2

   Specify the CPU and memory limits to allocate to the Kibana proxy.

1. Edit the Cluster Logging Custom Resource (CR) in the openshift-logging project:

   $ oc edit ClusterLogging instance

   Copy to Clipboard Toggle word wrap

   $ oc edit ClusterLogging instance
   
   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   
   ....
   
   spec:
       visualization:
         type: "kibana"
         kibana:
           replicas: 1 

   1

   Copy to Clipboard Toggle word wrap

   1

   Specify the number of Kibana nodes.

Procedure

1. Use the following command to add a taint to a node where you want to schedule the Kibana Pod:

   $ oc adm taint nodes <node-name> <key>=<value>:<effect>

   Copy to Clipboard Toggle word wrap

   For example:

   $ oc adm taint nodes node1 kibana=node:NoExecute

   Copy to Clipboard Toggle word wrap

   This example places a taint on node1 that has key kibana, value node, and taint effect NoExecute. You must use the NoExecute taint effect. NoExecute schedules only Pods that match the taint and remove existing Pods that do not match.

2. Edit the visualization section of the Cluster Logging Custom Resource (CR) to configure a toleration for the Kibana Pod:

     visualization:
       type: "kibana"
       kibana:
         tolerations:
         - key: "kibana"  

   1

   
           operator: "Exists"  

   2

   
           effect: "NoExecute"  

   3

   
           tolerationSeconds: 6000 

   4

   Copy to Clipboard Toggle word wrap

   1

   Specify the key that you added to the node.

   2

   Specify the Exists operator to require the key/value/effect parameters to match.

   3

   Specify the NoExecute effect.

   4

   Optionally, specify the tolerationSeconds parameter to set how long a Pod can remain bound to a node before being evicted.

1. If necessary, get the Kibana route, which is created by default upon installation of the Cluster Logging Operator:

   $ oc get routes -n openshift-logging
   
   NAMESPACE                  NAME                       HOST/PORT                                                            PATH     SERVICES                   PORT    TERMINATION          WILDCARD
   openshift-logging          kibana                     kibana-openshift-logging.apps.openshift.com                                   kibana                     <all>   reencrypt/Redirect   None

   Copy to Clipboard Toggle word wrap

2. Get the name of your Elasticsearch pods.

   $ oc get pods -l component=elasticsearch
   
   NAME                                            READY   STATUS    RESTARTS   AGE
   elasticsearch-cdm-5ceex6ts-1-dcd6c4c7c-jpw6k    2/2     Running   0          22h
   elasticsearch-cdm-5ceex6ts-2-f799564cb-l9mj7    2/2     Running   0          22h
   elasticsearch-cdm-5ceex6ts-3-585968dc68-k7kjr   2/2     Running   0          22h

   Copy to Clipboard Toggle word wrap

3. Create the necessary per-user configuration that this procedure requires:

   1. Log in to the Kibana dashboard as the user you want to add the dashboards to.

      https://kibana-openshift-logging.apps.openshift.com 

      1

      Copy to Clipboard Toggle word wrap

      1

      Where the URL is Kibana route.

   2. If the Authorize Access page appears, select all permissions and click Allow selected permissions.
   3. Log out of the Kibana dashboard.

4. Run the following command from the project where the pod is located using the name of any of your Elastiscearch pods:

   $ oc exec <es-pod> -- es_load_kibana_ui_objects <user-name>

   Copy to Clipboard Toggle word wrap

   For example:

   $ oc exec elasticsearch-cdm-5ceex6ts-1-dcd6c4c7c-jpw6k -- es_load_kibana_ui_objects <user-name>

   Copy to Clipboard Toggle word wrap

1. Edit the Cluster Logging Custom Resource in the openshift-logging project:

   $ oc edit clusterlogging instance

   Copy to Clipboard Toggle word wrap

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   
   ...
   
     curation:
       curator:
         schedule: 30 3 * * * 

   1

   
       type: curator

   Copy to Clipboard Toggle word wrap

   1

   Specify the schedule for Curator in cron format.

   Note

   The time zone is set based on the host node where the Curator pod runs.

1. Edit the OpenShift Container Platform custom Curator configuration file:

   $ oc edit configmap/curator

   Copy to Clipboard Toggle word wrap

2. Set the following parameters as needed:

   config.yaml: |
     project_name:
       action
         unit:value

   Copy to Clipboard Toggle word wrap

   The available parameters are:

   Expand

   Table 7.1. Project options

   Variable Name	Description
   project_name	The actual name of a project, such as myapp-devel. For OpenShift Container Platform operations logs, use the name .operations as the project name.
   action	The action to take, currently only delete is allowed.
   unit	The period to use for deletion, days, weeks, or months.
   value	The number of units.

   Show more
   Expand

   Table 7.2. Filter options

   Variable Name	Description
   .defaults	Use .defaults as the project_name to set the defaults for projects that are not specified.
   .regex	The list of regular expressions that match project names.
   pattern	The valid and properly escaped regular expression pattern enclosed by single quotation marks.

   Show more

1. Enable debug log of Curator:

   $ oc set env cronjob/curator CURATOR_LOG_LEVEL=DEBUG CURATOR_SCRIPT_LOG_LEVEL=DEBUG

   Copy to Clipboard Toggle word wrap

   Specify the log level:

   • CRITICAL. Curator displays only critical messages.
   • ERROR. Curator displays only error and critical messages.
   • WARNING. Curator displays only error, warning, and critical messages.
   • INFO. Curator displays only informational, error, warning, and critical messages.

   • DEBUG. Curator displays only debug messages, in addition to all of the above.

     The default value is INFO.

     Note

     Cluster logging uses the OpenShift Container Platform custom environment variable CURATOR_SCRIPT_LOG_LEVEL in OpenShift Container Platform wrapper scripts (run.sh and convert.py). The environment variable takes the same values as CURATOR_LOG_LEVEL for script debugging, as needed.

2. Trigger next curator iteration:

   $ oc create job --from=cronjob/curator <job_name>

   Copy to Clipboard Toggle word wrap

3. Use the following commands to control the CronJob:

   • Suspend a CronJob:

     $ oc patch cronjob curator -p '{"spec":{"suspend":true}}'

     Copy to Clipboard Toggle word wrap

   • Resume a CronJob:

     $ oc patch cronjob curator -p '{"spec":{"suspend":false}}'

     Copy to Clipboard Toggle word wrap

   • Change a CronJob schedule:

     $ oc patch cronjob curator -p '{"spec":{"schedule":"0 0 * * *"}}' 

     1

     Copy to Clipboard Toggle word wrap

     1

     The schedule option accepts schedules in cron format.

1. Edit the Curator ConfigMap:

   oc edit cm/curator -n openshift-logging

   Copy to Clipboard Toggle word wrap

2. Make the following changes to the action file:

   actions:
   1:
         action: delete_indices 

   1

   
         description: >-
           Delete .operations indices older than 30 days.
           Ignore the error if the filter does not
           result in an actionable list of indices (ignore_empty_list).
           See https://www.elastic.co/guide/en/elasticsearch/client/curator/5.2/ex_delete_indices.html
         options:
           # Swallow curator.exception.NoIndices exception
           ignore_empty_list: True
           # In seconds, default is 300
           timeout_override: ${CURATOR_TIMEOUT}
           # Don't swallow any other exceptions
           continue_if_exception: False
           # Optionally disable action, useful for debugging
           disable_action: False
         # All filters are bound by logical AND
         filters:            

   2

   
         - filtertype: pattern
           kind: regex
           value: '^\.operations\..*$'
           exclude: False    

   3

   
         - filtertype: age
           # Parse timestamp from index name
           source: name
           direction: older
           timestring: '%Y.%m.%d'
           unit: days
           unit_count: 30
           exclude: False

   Copy to Clipboard Toggle word wrap

   1

   Specify delete_indices to delete the specified index.

   2

   Use the filers parameters to specify the index to be deleted. See the Elastic Search curator documentation for information on these parameters.

   3

   Specify false to allow the index to be deleted.

Procedure

1. Edit the Cluster Logging Custom Resource (CR) in the openshift-logging project:

   $ oc edit ClusterLogging instance

   Copy to Clipboard Toggle word wrap

   $ oc edit ClusterLogging instance
   
   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
   
   ....
   
   spec:
     collection:
       logs:
         fluentd:
           resources:
             limits: 

   1

   
               memory: 736Mi
             requests:
               cpu: 100m
               memory: 736Mi

   Copy to Clipboard Toggle word wrap

   1

   Specify the CPU and memory limits and requests as needed. The values shown are the default values.

1. Edit either of the following parameters in the fluentd Daemonset.

   spec:
     template:
       spec:
         containers:
             env:
             - name: FILE_BUFFER_LIMIT 

   1

   
               value: "256"
             - name: BUFFER_SIZE_LIMIT 

   2

   
               value: 8Mi

   Copy to Clipboard Toggle word wrap

   1

   Specify the Fluentd file buffer size per output.

   2

   Specify the maximum size of each Fluentd buffer chunk.

Procedure

1. Create a template for the Event Router:

   kind: Template
   apiVersion: v1
   metadata:
     name: eventrouter-template
     annotations:
       description: "A pod forwarding kubernetes events to cluster logging stack."
       tags: "events,EFK,logging,cluster-logging"
   objects:
     - kind: ServiceAccount 

   1

   
       apiVersion: v1
       metadata:
         name: eventrouter
         namespace: ${NAMESPACE}
     - kind: ClusterRole 

   2

   
       apiVersion: v1
       metadata:
         name: event-reader
       rules:
       - apiGroups: [""]
         resources: ["events"]
         verbs: ["get", "watch", "list"]
     - kind: ClusterRoleBinding  

   3

   
       apiVersion: v1
       metadata:
         name: event-reader-binding
       subjects:
       - kind: ServiceAccount
         name: eventrouter
         namespace: ${NAMESPACE}
       roleRef:
         kind: ClusterRole
         name: event-reader
     - kind: ConfigMap 

   4

   
       apiVersion: v1
       metadata:
         name: eventrouter
         namespace: ${NAMESPACE}
       data:
         config.json: |-
           {
             "sink": "stdout"
           }
     - kind: Deployment 

   5

   
       apiVersion: apps/v1
       metadata:
         name: eventrouter
         namespace: ${NAMESPACE}
         labels:
           component: eventrouter
           logging-infra: eventrouter
           provider: openshift
       spec:
         selector:
           matchLabels:
             component: eventrouter
             logging-infra: eventrouter
             provider: openshift
         replicas: 1
         template:
           metadata:
             labels:
               component: eventrouter
               logging-infra: eventrouter
               provider: openshift
             name: eventrouter
           spec:
             serviceAccount: eventrouter
             containers:
               - name: kube-eventrouter
                 image: ${IMAGE}
                 imagePullPolicy: IfNotPresent
                 resources:
                   requests:
                     cpu: ${CPU}
                     memory: ${MEMORY}
                 volumeMounts:
                 - name: config-volume
                   mountPath: /etc/eventrouter
             volumes:
               - name: config-volume
                 configMap:
                   name: eventrouter
   parameters:
     - name: IMAGE
       displayName: Image
       value: "registry.redhat.io/openshift4/ose-logging-eventrouter:latest"
     - name: CPU  

   6

   
       displayName: CPU
       value: "100m"
     - name: MEMORY 

   7

   
       displayName: Memory
       value: "128Mi"
     - name: NAMESPACE
       displayName: Namespace
       value: "openshift-logging" 

   8

   Copy to Clipboard Toggle word wrap

   1

   Creates a Service Account in the openshift-logging project for the Event Router.

   2

   Creates a ClusterRole to monitor for events in the cluster.

   3

   Creates a ClusterRoleBinding to bind the ClusterRole to the ServiceAccount.

   4

   Creates a ConfigMap in the openshift-logging project to generate the required config.json file.

   5

   Creates a Deployment in the openshift-logging project to generate and configure the Event Router Pod.

   6

   Specifies the minimum amount of memory to allocate to the Event Router Pod. Defaults to 128Mi.

   7

   Specifies the minimum amount of CPU to allocate to the Event Router Pod. Defaults to 100m.

   8

   Specifies the openshift-logging project to install objects in.

2. Use the following command to process and apply the template:

   $ oc process -f <templatefile> | oc apply -f -

   Copy to Clipboard Toggle word wrap

   For example:

   $ oc process -f eventrouter.yaml | oc apply -f -

   Copy to Clipboard Toggle word wrap

   Example output

   serviceaccount/logging-eventrouter created
   clusterrole.authorization.openshift.io/event-reader created
   clusterrolebinding.authorization.openshift.io/event-reader-binding created
   configmap/logging-eventrouter created
   deployment.apps/logging-eventrouter created

   Copy to Clipboard Toggle word wrap

3. Validate that the Event Router installed in the openshift-logging project:

   1. View the new Event Router Pod:

      $ oc get pods --selector  component=eventrouter -o name -n openshift-logging

      Copy to Clipboard Toggle word wrap

      Example output

      pod/cluster-logging-eventrouter-d649f97c8-qvv8r

      Copy to Clipboard Toggle word wrap

   2. View the events collected by the Event Router:

      $ oc logs <cluster_logging_eventrouter_pod> -n openshift-logging

      Copy to Clipboard Toggle word wrap

      For example:

      $ oc logs cluster-logging-eventrouter-d649f97c8-qvv8r -n openshift-logging

      Copy to Clipboard Toggle word wrap

      Example output

      {"verb":"ADDED","event":{"metadata":{"name":"openshift-service-catalog-controller-manager-remover.1632d931e88fcd8f","namespace":"openshift-service-catalog-removed","selfLink":"/api/v1/namespaces/openshift-service-catalog-removed/events/openshift-service-catalog-controller-manager-remover.1632d931e88fcd8f","uid":"787d7b26-3d2f-4017-b0b0-420db4ae62c0","resourceVersion":"21399","creationTimestamp":"2020-09-08T15:40:26Z"},"involvedObject":{"kind":"Job","namespace":"openshift-service-catalog-removed","name":"openshift-service-catalog-controller-manager-remover","uid":"fac9f479-4ad5-4a57-8adc-cb25d3d9cf8f","apiVersion":"batch/v1","resourceVersion":"21280"},"reason":"Completed","message":"Job completed","source":{"component":"job-controller"},"firstTimestamp":"2020-09-08T15:40:26Z","lastTimestamp":"2020-09-08T15:40:26Z","count":1,"type":"Normal"}}

      Copy to Clipboard Toggle word wrap

      You can also use Kibana to view events by creating an index pattern using the Elasticsearch infra index.