Logging

1. Install the OpenShift Elasticsearch Operator:

   1. In the OpenShift Container Platform web console, click Operators → OperatorHub.
   2. Choose OpenShift Elasticsearch Operator from the list of available Operators, and click Install.
   3. Ensure that the All namespaces on the cluster is selected under Installation Mode.

   4. Ensure that openshift-operators-redhat is selected under Installed Namespace.

      You must specify the openshift-operators-redhat namespace. The openshift-operators namespace might contain Community Operators, which are untrusted and could publish a metric with the same name as an OpenShift Container Platform metric, which would cause conflicts.

   5. Select Enable operator recommended cluster monitoring on this namespace.

      This option sets the openshift.io/cluster-monitoring: "true" label in the Namespace object. You must select this option to ensure that cluster monitoring scrapes the openshift-operators-redhat namespace.

   6. Select stable-5.x as the Update Channel.

   7. Select an Approval Strategy.

      • The Automatic strategy allows Operator Lifecycle Manager (OLM) to automatically update the Operator when a new version is available.
      • The Manual strategy requires a user with appropriate credentials to approve the Operator update.
   8. Click Install.
   9. Verify that the OpenShift Elasticsearch Operator installed by switching to the Operators → Installed Operators page.
   10. Ensure that OpenShift Elasticsearch Operator is listed in all projects with a Status of Succeeded.

2. Install the Red Hat OpenShift Logging Operator:

   1. In the OpenShift Container Platform web console, click Operators → OperatorHub.
   2. Choose Red Hat OpenShift Logging from the list of available Operators, and click Install.
   3. Ensure that the A specific namespace on the cluster is selected under Installation Mode.
   4. Ensure that Operator recommended namespace is openshift-logging under Installed Namespace.

   5. Select Enable operator recommended cluster monitoring on this namespace.

      This option sets the openshift.io/cluster-monitoring: "true" label in the Namespace object. You must select this option to ensure that cluster monitoring scrapes the openshift-logging namespace.

   6. Select stable-5.x as the Update Channel.

   7. Select an Approval Strategy.

      • The Automatic strategy allows Operator Lifecycle Manager (OLM) to automatically update the Operator when a new version is available.
      • The Manual strategy requires a user with appropriate credentials to approve the Operator update.
   8. Click Install.
   9. Verify that the Red Hat OpenShift Logging Operator installed by switching to the Operators → Installed Operators page.

   10. Ensure that Red Hat OpenShift Logging is listed in the openshift-logging project with a Status of Succeeded.

       If the Operator does not appear as installed, to troubleshoot further:

       • Switch to the Operators → Installed Operators page and inspect the Status column for any errors or failures.
       • Switch to the Workloads → Pods page and check the logs in any pods in the openshift-logging project that are reporting issues.

3. Create an OpenShift Logging instance:

   1. Switch to the Administration → Custom Resource Definitions page.
   2. On the Custom Resource Definitions page, click ClusterLogging.
   3. On the Custom Resource Definition details page, select View Instances from the Actions menu.

   4. On the ClusterLoggings page, click Create ClusterLogging.

      You might have to refresh the page to load the data.

   5. In the YAML field, replace the code with the following:

      Note

      This default OpenShift Logging configuration should support a wide array of environments. Review the topics on tuning and configuring OpenShift Logging components for information on modifications you can make to your OpenShift Logging cluster.

      apiVersion: "logging.openshift.io/v1"
      kind: "ClusterLogging"
      metadata:
        name: "instance" 1
        namespace: "openshift-logging"
      spec:
        managementState: "Managed"  2
        logStore:
          type: "elasticsearch"  3
          retentionPolicy: 4
            application:
              maxAge: 1d
            infra:
              maxAge: 7d
            audit:
              maxAge: 7d
          elasticsearch:
            nodeCount: 3 5
            storage:
              storageClassName: "<storage_class_name>" 6
              size: 200G
            resources: 7
                limits:
                  memory: "16Gi"
                requests:
                  memory: "16Gi"
            proxy: 8
              resources:
                limits:
                  memory: 256Mi
                requests:
                  memory: 256Mi
            redundancyPolicy: "SingleRedundancy"
        visualization:
          type: "kibana"  9
          kibana:
            replicas: 1
        collection:
          logs:
            type: "fluentd"  10
            fluentd: {}

      1

      The name must be instance.

      2

      The OpenShift Logging management state. In some cases, if you change the OpenShift Logging defaults, you must set this to Unmanaged. However, an unmanaged deployment does not receive updates until OpenShift Logging is placed back into a managed state.

      3

      Settings for configuring Elasticsearch. Using the CR, you can configure shard replication policy and persistent storage.

      4

      Specify the length of time that Elasticsearch should retain each log source. Enter an integer and a time designation: weeks(w), hours(h/H), minutes(m) and seconds(s). For example, 7d for seven days. Logs older than the maxAge are deleted. You must specify a retention policy for each log source or the Elasticsearch indices will not be created for that source.

      5

      Specify the number of Elasticsearch nodes. See the note that follows this list.

      6

      Enter the name of an existing storage class for Elasticsearch storage. For best performance, specify a storage class that allocates block storage. If you do not specify a storage class, OpenShift Logging uses ephemeral storage.

      7

      Specify the CPU and memory requests for Elasticsearch as needed. If you leave these values blank, the OpenShift Elasticsearch Operator sets default values that should be sufficient for most deployments. The default values are 16Gi for the memory request and 1 for the CPU request.

      8

      Specify the CPU and memory requests for the Elasticsearch proxy as needed. If you leave these values blank, the OpenShift Elasticsearch Operator sets default values that should be sufficient for most deployments. The default values are 256Mi for the memory request and 100m for the CPU request.

      9

      Settings for configuring Kibana. Using the CR, you can scale Kibana for redundancy and configure the CPU and memory for your Kibana nodes. For more information, see Configuring the log visualizer.

      10

      Settings for configuring Fluentd. Using the CR, you can configure Fluentd CPU and memory limits. For more information, see Configuring Fluentd.

      Note

      The maximum number of Elasticsearch control plane nodes (also known as the master nodes) is three. If you specify a nodeCount greater than 3, OpenShift Container Platform creates three Elasticsearch nodes that are Master-eligible nodes, with the master, client, and data roles. The additional Elasticsearch nodes are created as Data-only nodes, using client and data roles. Control plane nodes perform cluster-wide actions such as creating or deleting an index, shard allocation, and tracking nodes. Data nodes hold the shards and perform data-related operations such as CRUD, search, and aggregations. Data-related operations are I/O-, memory-, and CPU-intensive. It is important to monitor these resources and to add more Data nodes if the current nodes are overloaded.

      For example, if nodeCount=4, the following nodes are created:

      $ oc get deployment

      Example output

      cluster-logging-operator       1/1     1            1           18h
      elasticsearch-cd-x6kdekli-1    0/1     1            0           6m54s
      elasticsearch-cdm-x6kdekli-1   1/1     1            1           18h
      elasticsearch-cdm-x6kdekli-2   0/1     1            0           6m49s
      elasticsearch-cdm-x6kdekli-3   0/1     1            0           6m44s

      The number of primary shards for the index templates is equal to the number of Elasticsearch data nodes.

   6. Click Create. This creates the OpenShift Logging components, the Elasticsearch custom resource and components, and the Kibana interface.

4. Verify the install:

   1. Switch to the Workloads → Pods page.

   2. Select the openshift-logging project.

      You should see several pods for OpenShift Logging, Elasticsearch, Fluentd, and Kibana similar to the following list:

      • cluster-logging-operator-cb795f8dc-xkckc
      • elasticsearch-cdm-b3nqzchd-1-5c6797-67kfz
      • elasticsearch-cdm-b3nqzchd-2-6657f4-wtprv
      • elasticsearch-cdm-b3nqzchd-3-588c65-clg7g
      • fluentd-2c7dg
      • fluentd-9z7kk
      • fluentd-br7r2
      • fluentd-fn2sb
      • fluentd-pb2f8
      • fluentd-zqgqx
      • kibana-7fb4fd4cc9-bvt4p

1. Create a namespace for the OpenShift Elasticsearch Operator.

   1. Create a namespace object YAML file (for example, eo-namespace.yaml) for the OpenShift Elasticsearch Operator:

      apiVersion: v1
      kind: Namespace
      metadata:
        name: openshift-operators-redhat 1
        annotations:
          openshift.io/node-selector: ""
        labels:
          openshift.io/cluster-monitoring: "true" 2

      1

      You must specify the openshift-operators-redhat namespace. To prevent possible conflicts with metrics, you should configure the Prometheus Cluster Monitoring stack to scrape metrics from the openshift-operators-redhat namespace and not the openshift-operators namespace. The openshift-operators namespace might contain community Operators, which are untrusted and could publish a metric with the same name as an OpenShift Container Platform metric, which would cause conflicts.

      2

      String. You must specify this label as shown to ensure that cluster monitoring scrapes the openshift-operators-redhat namespace.

   2. Create the namespace:

      $ oc create -f <file-name>.yaml

      For example:

      $ oc create -f eo-namespace.yaml

2. Create a namespace for the Red Hat OpenShift Logging Operator:

   1. Create a namespace object YAML file (for example, olo-namespace.yaml) for the Red Hat OpenShift Logging Operator:

      apiVersion: v1
      kind: Namespace
      metadata:
        name: openshift-logging
        annotations:
          openshift.io/node-selector: ""
        labels:
          openshift.io/cluster-monitoring: "true"

   2. Create the namespace:

      $ oc create -f <file-name>.yaml

      For example:

      $ oc create -f olo-namespace.yaml

3. Install the OpenShift Elasticsearch Operator by creating the following objects:

   1. Create an Operator Group object YAML file (for example, eo-og.yaml) for the OpenShift Elasticsearch Operator:

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: openshift-operators-redhat
        namespace: openshift-operators-redhat 1
      spec: {}

      1

      You must specify the openshift-operators-redhat namespace.

   2. Create an Operator Group object:

      $ oc create -f <file-name>.yaml

      For example:

      $ oc create -f eo-og.yaml

   3. Create a Subscription object YAML file (for example, eo-sub.yaml) to subscribe a namespace to the OpenShift Elasticsearch Operator.

      Example Subscription

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: "elasticsearch-operator"
        namespace: "openshift-operators-redhat" 1
      spec:
        channel: "stable-5.1" 2
        installPlanApproval: "Automatic"
        source: "redhat-operators" 3
        sourceNamespace: "openshift-marketplace"
        name: "elasticsearch-operator"

      1

      You must specify the openshift-operators-redhat namespace.

      2

      Specify 5.0, stable, or stable-5.<x> as the channel. See the following note.

      3

      Specify redhat-operators. If your OpenShift Container Platform cluster is installed on a restricted network, also known as a disconnected cluster, specify the name of the CatalogSource object created when you configured the Operator Lifecycle Manager (OLM).

      Note

      Specifying stable installs the current version of the latest stable release. Using stable with installPlanApproval: "Automatic", will automatically upgrade your operators to the latest stable major and minor release.

      Specifying stable-5.<x> installs the current minor version of a specific major release. Using stable-5.<x> with installPlanApproval: "Automatic", will automatically upgrade your operators to the latest stable minor release within the major release you specify with x.

   4. Create the Subscription object:

      $ oc create -f <file-name>.yaml

      For example:

      $ oc create -f eo-sub.yaml

      The OpenShift Elasticsearch Operator is installed to the openshift-operators-redhat namespace and copied to each project in the cluster.

   5. Verify the Operator installation:

      $ oc get csv --all-namespaces

      Example output

      NAMESPACE                                               NAME                                            DISPLAY                  VERSION               REPLACES   PHASE
      default                                                 elasticsearch-operator.5.1.0-202007012112.p0    OpenShift Elasticsearch Operator   5.1.0-202007012112.p0               Succeeded
      kube-node-lease                                         elasticsearch-operator.5.1.0-202007012112.p0    OpenShift Elasticsearch Operator   5.1.0-202007012112.p0               Succeeded
      kube-public                                             elasticsearch-operator.5.1.0-202007012112.p0    OpenShift Elasticsearch Operator   5.1.0-202007012112.p0               Succeeded
      kube-system                                             elasticsearch-operator.5.1.0-202007012112.p0    OpenShift Elasticsearch Operator   5.1.0-202007012112.p0               Succeeded
      openshift-apiserver-operator                            elasticsearch-operator.5.1.0-202007012112.p0    OpenShift Elasticsearch Operator   5.1.0-202007012112.p0               Succeeded
      openshift-apiserver                                     elasticsearch-operator.5.1.0-202007012112.p0    OpenShift Elasticsearch Operator   5.1.0-202007012112.p0               Succeeded
      openshift-authentication-operator                       elasticsearch-operator.5.1.0-202007012112.p0    OpenShift Elasticsearch Operator   5.1.0-202007012112.p0               Succeeded
      openshift-authentication                                elasticsearch-operator.5.1.0-202007012112.p0    OpenShift Elasticsearch Operator   5.1.0-202007012112.p0               Succeeded
      ...

      There should be an OpenShift Elasticsearch Operator in each namespace. The version number might be different than shown.

4. Install the Red Hat OpenShift Logging Operator by creating the following objects:

   1. Create an Operator Group object YAML file (for example, olo-og.yaml) for the Red Hat OpenShift Logging Operator:

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: cluster-logging
        namespace: openshift-logging 1
      spec:
        targetNamespaces:
        - openshift-logging 2

      1 2

      You must specify the openshift-logging namespace.

   2. Create the OperatorGroup object:

      $ oc create -f <file-name>.yaml

      For example:

      $ oc create -f olo-og.yaml

   3. Create a Subscription object YAML file (for example, olo-sub.yaml) to subscribe a namespace to the Red Hat OpenShift Logging Operator.

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: cluster-logging
        namespace: openshift-logging 1
      spec:
        channel: "stable" 2
        name: cluster-logging
        source: redhat-operators 3
        sourceNamespace: openshift-marketplace

      1

      You must specify the openshift-logging namespace.

      2

      Specify 5.0, stable, or stable-5.<x> as the channel.

      3

      Specify redhat-operators. If your OpenShift Container Platform cluster is installed on a restricted network, also known as a disconnected cluster, specify the name of the CatalogSource object you created when you configured the Operator Lifecycle Manager (OLM).

      $ oc create -f <file-name>.yaml

      For example:

      $ oc create -f olo-sub.yaml

      The Red Hat OpenShift Logging Operator is installed to the openshift-logging namespace.

   4. Verify the Operator installation.

      There should be a Red Hat OpenShift Logging Operator in the openshift-logging namespace. The Version number might be different than shown.

      $ oc get csv -n openshift-logging

      Example output

      NAMESPACE                                               NAME                                         DISPLAY                  VERSION               REPLACES   PHASE
      ...
      openshift-logging                                       clusterlogging.5.1.0-202007012112.p0         OpenShift Logging          5.1.0-202007012112.p0              Succeeded
      ...

5. Create an OpenShift Logging instance:

   1. Create an instance object YAML file (for example, olo-instance.yaml) for the Red Hat OpenShift Logging Operator:

      Note

      This default OpenShift Logging configuration should support a wide array of environments. Review the topics on tuning and configuring OpenShift Logging components for information on modifications you can make to your OpenShift Logging cluster.

      apiVersion: "logging.openshift.io/v1"
      kind: "ClusterLogging"
      metadata:
        name: "instance" 1
        namespace: "openshift-logging"
      spec:
        managementState: "Managed"  2
        logStore:
          type: "elasticsearch"  3
          retentionPolicy: 4
            application:
              maxAge: 1d
            infra:
              maxAge: 7d
            audit:
              maxAge: 7d
          elasticsearch:
            nodeCount: 3 5
            storage:
              storageClassName: "<storage-class-name>" 6
              size: 200G
            resources: 7
              limits:
                memory: "16Gi"
              requests:
                memory: "16Gi"
            proxy: 8
              resources:
                limits:
                  memory: 256Mi
                requests:
                   memory: 256Mi
            redundancyPolicy: "SingleRedundancy"
        visualization:
          type: "kibana"  9
          kibana:
            replicas: 1
        collection:
          logs:
            type: "fluentd"  10
            fluentd: {}

      1

      The name must be instance.

      2

      The OpenShift Logging management state. In some cases, if you change the OpenShift Logging defaults, you must set this to Unmanaged. However, an unmanaged deployment does not receive updates until OpenShift Logging is placed back into a managed state. Placing a deployment back into a managed state might revert any modifications you made.

      3

      Settings for configuring Elasticsearch. Using the custom resource (CR), you can configure shard replication policy and persistent storage.

      4

      Specify the length of time that Elasticsearch should retain each log source. Enter an integer and a time designation: weeks(w), hours(h/H), minutes(m) and seconds(s). For example, 7d for seven days. Logs older than the maxAge are deleted. You must specify a retention policy for each log source or the Elasticsearch indices will not be created for that source.

      5

      Specify the number of Elasticsearch nodes. See the note that follows this list.

      6

      Enter the name of an existing storage class for Elasticsearch storage. For best performance, specify a storage class that allocates block storage. If you do not specify a storage class, OpenShift Container Platform deploys OpenShift Logging with ephemeral storage only.

      7

      Specify the CPU and memory requests for Elasticsearch as needed. If you leave these values blank, the OpenShift Elasticsearch Operator sets default values that are sufficient for most deployments. The default values are 16Gi for the memory request and 1 for the CPU request.

      8

      Specify the CPU and memory requests for the Elasticsearch proxy as needed. If you leave these values blank, the OpenShift Elasticsearch Operator sets default values that should be sufficient for most deployments. The default values are 256Mi for the memory request and 100m for the CPU request.

      9

      Settings for configuring Kibana. Using the CR, you can scale Kibana for redundancy and configure the CPU and memory for your Kibana pods. For more information, see Configuring the log visualizer.

      10

      Settings for configuring Fluentd. Using the CR, you can configure Fluentd CPU and memory limits. For more information, see Configuring Fluentd.

      Note

      The maximum number of Elasticsearch control plane nodes is three. If you specify a nodeCount greater than 3, OpenShift Container Platform creates three Elasticsearch nodes that are Master-eligible nodes, with the master, client, and data roles. The additional Elasticsearch nodes are created as Data-only nodes, using client and data roles. Control plane nodes perform cluster-wide actions such as creating or deleting an index, shard allocation, and tracking nodes. Data nodes hold the shards and perform data-related operations such as CRUD, search, and aggregations. Data-related operations are I/O-, memory-, and CPU-intensive. It is important to monitor these resources and to add more Data nodes if the current nodes are overloaded.

      For example, if nodeCount=4, the following nodes are created:

      $ oc get deployment

      Example output

      cluster-logging-operator       1/1     1            1           18h
      elasticsearch-cd-x6kdekli-1    1/1     1            0           6m54s
      elasticsearch-cdm-x6kdekli-1   1/1     1            1           18h
      elasticsearch-cdm-x6kdekli-2   1/1     1            0           6m49s
      elasticsearch-cdm-x6kdekli-3   1/1     1            0           6m44s

      The number of primary shards for the index templates is equal to the number of Elasticsearch data nodes.

   2. Create the instance:

      $ oc create -f <file-name>.yaml

      For example:

      $ oc create -f olo-instance.yaml

      This creates the OpenShift Logging components, the Elasticsearch custom resource and components, and the Kibana interface.

6. Verify the installation by listing the pods in the openshift-logging project.

   You should see several pods for OpenShift Logging, Elasticsearch, Fluentd, and Kibana similar to the following list:

   $ oc get pods -n openshift-logging

   Example output

   NAME                                            READY   STATUS    RESTARTS   AGE
   cluster-logging-operator-66f77ffccb-ppzbg       1/1     Running   0          7m
   elasticsearch-cdm-ftuhduuw-1-ffc4b9566-q6bhp    2/2     Running   0          2m40s
   elasticsearch-cdm-ftuhduuw-2-7b4994dbfc-rd2gc   2/2     Running   0          2m36s
   elasticsearch-cdm-ftuhduuw-3-84b5ff7ff8-gqnm2   2/2     Running   0          2m4s
   fluentd-587vb                                   1/1     Running   0          2m26s
   fluentd-7mpb9                                   1/1     Running   0          2m30s
   fluentd-flm6j                                   1/1     Running   0          2m33s
   fluentd-gn4rn                                   1/1     Running   0          2m26s
   fluentd-nlgb6                                   1/1     Running   0          2m30s
   fluentd-snpkt                                   1/1     Running   0          2m28s
   kibana-d6d5668c5-rppqm                          2/2     Running   0          2m39s

Procedure (UI)

1. In the OpenShift Container Platform console, navigate to Workloads → Pods or navigate to the pod through the resource you want to investigate.

   Note

   Some resources, such as builds, do not have pods to query directly. In such instances, you can locate the Logs link on the Details page for the resource.

2. Select a project from the drop-down menu.
3. Click the name of the pod you want to investigate.
4. Click Logs.

1. In the OpenShift Container Platform console, click the Application Launcher and select Logging.

2. Create your Kibana index patterns by clicking Management → Index Patterns → Create index pattern:

   • Each user must manually create index patterns when logging into Kibana the first time to see logs for their projects. Users must create an index pattern named app and use the @timestamp time field to view their container logs.
   • Each admin user must create index patterns when logged into Kibana the first time for the app, infra, and audit indices using the @timestamp time field.
3. Create Kibana Visualizations from the new index patterns.

1. In the OpenShift Container Platform console, click the Application Launcher and select Logging.

2. Log in using the same credentials you use to log in to the OpenShift Container Platform console.

   The Kibana interface launches.

3. In Kibana, click Discover.

4. Select the index pattern you created from the drop-down menu in the top-left corner: app, audit, or infra.

   The log data displays as time-stamped documents.

5. Expand one of the time-stamped documents.

6. Click the JSON tab to display the log entry for that document.

   Example 6.1. Sample infrastructure log entry in Kibana

   {
     "_index": "infra-000001",
     "_type": "_doc",
     "_id": "YmJmYTBlNDkZTRmLTliMGQtMjE3NmFiOGUyOWM3",
     "_version": 1,
     "_score": null,
     "_source": {
       "docker": {
         "container_id": "f85fa55bbef7bb783f041066be1e7c267a6b88c4603dfce213e32c1"
       },
       "kubernetes": {
         "container_name": "registry-server",
         "namespace_name": "openshift-marketplace",
         "pod_name": "redhat-marketplace-n64gc",
         "container_image": "registry.redhat.io/redhat/redhat-marketplace-index:v4.7",
         "container_image_id": "registry.redhat.io/redhat/redhat-marketplace-index@sha256:65fc0c45aabb95809e376feb065771ecda9e5e59cc8b3024c4545c168f",
         "pod_id": "8f594ea2-c866-4b5c-a1c8-a50756704b2a",
         "host": "ip-10-0-182-28.us-east-2.compute.internal",
         "master_url": "https://kubernetes.default.svc",
         "namespace_id": "3abab127-7669-4eb3-b9ef-44c04ad68d38",
         "namespace_labels": {
           "openshift_io/cluster-monitoring": "true"
         },
         "flat_labels": [
           "catalogsource_operators_coreos_com/update=redhat-marketplace"
         ]
       },
       "message": "time=\"2020-09-23T20:47:03Z\" level=info msg=\"serving registry\" database=/database/index.db port=50051",
       "level": "unknown",
       "hostname": "ip-10-0-182-28.internal",
       "pipeline_metadata": {
         "collector": {
           "ipaddr4": "10.0.182.28",
           "inputname": "fluent-plugin-systemd",
           "name": "fluentd",
           "received_at": "2020-09-23T20:47:15.007583+00:00",
           "version": "1.7.4 1.6.0"
         }
       },
       "@timestamp": "2020-09-23T20:47:03.422465+00:00",
       "viaq_msg_id": "YmJmYTBlNDktMDMGQtMjE3NmFiOGUyOWM3",
       "openshift": {
         "labels": {
           "logging": "infra"
         }
       }
     },
     "fields": {
       "@timestamp": [
         "2020-09-23T20:47:03.422Z"
       ],
       "pipeline_metadata.collector.received_at": [
         "2020-09-23T20:47:15.007Z"
       ]
     },
     "sort": [
       1600894023422
     ]
   }

Procedure

1. Create a ClusterLogForwarder CR YAML file similar to the following:

   apiVersion: "logging.openshift.io/v1"
   kind: ClusterLogForwarder
   metadata:
     name: instance 1
     namespace: openshift-logging 2
   spec:
     outputs:
      - name: elasticsearch-insecure 3
        type: "elasticsearch" 4
        url: http://elasticsearch.insecure.com:9200 5
      - name: elasticsearch-secure
        type: "elasticsearch"
        url: https://elasticsearch.secure.com:9200
        secret:
           name: es-secret 6
     pipelines:
      - name: application-logs 7
        inputRefs: 8
        - application
        - audit
        outputRefs:
        - elasticsearch-secure 9
        - default 10
        parse: json 11
        labels:
          myLabel: "myValue" 12
      - name: infrastructure-audit-logs 13
        inputRefs:
        - infrastructure
        outputRefs:
        - elasticsearch-insecure
        labels:
          logs: "audit-infra"

   1

   The name of the ClusterLogForwarder CR must be instance.

   2

   The namespace for the ClusterLogForwarder CR must be openshift-logging.

   3

   Specify a name for the output.

   4

   Specify the elasticsearch type.

   5

   Specify the URL and port of the external Elasticsearch instance as a valid absolute URL. You can use the http (insecure) or https (secure HTTP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP Address.

   6

   If using an https prefix, you must specify the name of the secret required by the endpoint for TLS communication. The secret must exist in the openshift-logging project and must have keys of: tls.crt, tls.key, and ca-bundle.crt that point to the respective certificates that they represent.

   7

   Optional: Specify a name for the pipeline.

   8

   Specify which log types should be forwarded using that pipeline: application, infrastructure, or audit.

   9

   Specify the output to use with that pipeline for forwarding the logs.

   10

   Optional: Specify the default output to send the logs to the internal Elasticsearch instance.

   11

   Optional: Forward structured JSON log entries as JSON objects in the structured field. The log entry must contain valid structured JSON; otherwise, OpenShift Logging removes the structured field and instead sends the log entry to the default index, app-00000x.

   12

   Optional: String. One or more labels to add to the logs.

   13

   Optional: Configure multiple outputs to forward logs to other external log aggregators of any supported type:

   • Optional. A name to describe the pipeline.
   • The inputRefs is the log type to forward using that pipeline: application, infrastructure, or audit.
   • The outputRefs is the name of the output to use.
   • Optional: String. One or more labels to add to the logs.

2. Create the CR object:

   $ oc create -f <file-name>.yaml

Procedure

1. Create a ClusterLogForwarder CR YAML file similar to the following:

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: instance 1
     namespace: openshift-logging 2
   spec:
     outputs:
      - name: fluentd-server-secure 3
        type: fluentdForward 4
        url: 'tls://fluentdserver.security.example.com:24224' 5
        secret: 6
           name: fluentd-secret
      - name: fluentd-server-insecure
        type: fluentdForward
        url: 'tcp://fluentdserver.home.example.com:24224'
     pipelines:
      - name: forward-to-fluentd-secure 7
        inputRefs:  8
        - application
        - audit
        outputRefs:
        - fluentd-server-secure 9
        - default 10
        parse: json 11
        labels:
          clusterId: "C1234" 12
      - name: forward-to-fluentd-insecure 13
        inputRefs:
        - infrastructure
        outputRefs:
        - fluentd-server-insecure
        labels:
          clusterId: "C1234"

   1

   The name of the ClusterLogForwarder CR must be instance.

   2

   The namespace for the ClusterLogForwarder CR must be openshift-logging.

   3

   Specify a name for the output.

   4

   Specify the fluentdForward type.

   5

   Specify the URL and port of the external Fluentd instance as a valid absolute URL. You can use the tcp (insecure) or tls (secure TCP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP address.

   6

   If using a tls prefix, you must specify the name of the secret required by the endpoint for TLS communication. The secret must exist in the openshift-logging project and must have keys of: tls.crt, tls.key, and ca-bundle.crt that point to the respective certificates that they represent.

   7

   Optional. Specify a name for the pipeline.

   8

   Specify which log types should be forwarded using that pipeline: application, infrastructure, or audit.

   9

   Specify the output to use with that pipeline for forwarding the logs.

   10

   Optional. Specify the default output to forward logs to the internal Elasticsearch instance.

   11

   Optional: Forward structured JSON log entries as JSON objects in the structured field. The log entry must contain valid structured JSON; otherwise, OpenShift Logging removes the structured field and instead sends the log entry to the default index, app-00000x.

   12

   Optional: String. One or more labels to add to the logs.

   13

   Optional: Configure multiple outputs to forward logs to other external log aggregators of any supported type:

   • Optional. A name to describe the pipeline.
   • The inputRefs is the log type to forward using that pipeline: application, infrastructure, or audit.
   • The outputRefs is the name of the output to use.
   • Optional: String. One or more labels to add to the logs.

2. Create the CR object:

   $ oc create -f <file-name>.yaml

Procedure

1. Create a ClusterLogForwarder CR YAML file similar to the following:

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: instance 1
     namespace: openshift-logging 2
   spec:
     outputs:
      - name: rsyslog-east 3
        type: syslog 4
        syslog: 5
          facility: local0
          rfc: RFC3164
          payloadKey: message
          severity: informational
        url: 'tls://rsyslogserver.east.example.com:514' 6
        secret: 7
           name: syslog-secret
      - name: rsyslog-west
        type: syslog
        syslog:
         appName: myapp
         facility: user
         msgID: mymsg
         procID: myproc
         rfc: RFC5424
         severity: debug
        url: 'udp://rsyslogserver.west.example.com:514'
     pipelines:
      - name: syslog-east 8
        inputRefs: 9
        - audit
        - application
        outputRefs: 10
        - rsyslog-east
        - default 11
        parse: json 12
        labels:
          secure: "true" 13
          syslog: "east"
      - name: syslog-west 14
        inputRefs:
        - infrastructure
        outputRefs:
        - rsyslog-west
        - default
        labels:
          syslog: "west"

   1

   The name of the ClusterLogForwarder CR must be instance.

   2

   The namespace for the ClusterLogForwarder CR must be openshift-logging.

   3

   Specify a name for the output.

   4

   Specify the syslog type.

   5

   Optional. Specify the syslog parameters, listed below.

   6

   Specify the URL and port of the external syslog instance. You can use the udp (insecure), tcp (insecure) or tls (secure TCP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP address.

   7

   If using a tls prefix, you must specify the name of the secret required by the endpoint for TLS communication. The secret must exist in the openshift-logging project and must have keys of: tls.crt, tls.key, and ca-bundle.crt that point to the respective certificates that they represent.

   8

   Optional: Specify a name for the pipeline.

   9

   Specify which log types should be forwarded using that pipeline: application, infrastructure, or audit.

   10

   Specify the output to use with that pipeline for forwarding the logs.

   11

   Optional: Specify the default output to forward logs to the internal Elasticsearch instance.

   12

   Optional: Forward structured JSON log entries as JSON objects in the structured field. The log entry must contain valid structured JSON; otherwise, OpenShift Logging removes the structured field and instead sends the log entry to the default index, app-00000x.

   13

   Optional: String. One or more labels to add to the logs. Quote values like "true" so they are recognized as string values, not as a boolean.

   14

   Optional: Configure multiple outputs to forward logs to other external log aggregators of any supported type:

   • Optional. A name to describe the pipeline.
   • The inputRefs is the log type to forward using that pipeline: application, infrastructure, or audit.
   • The outputRefs is the name of the output to use.
   • Optional: String. One or more labels to add to the logs.

2. Create the CR object:

   $ oc create -f <file-name>.yaml

Procedure

1. Create a ClusterLogForwarder CR YAML file similar to the following:

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: instance 1
     namespace: openshift-logging 2
   spec:
     outputs:
      - name: app-logs 3
        type: kafka 4
        url: tls://kafka.example.devlab.com:9093/app-topic 5
        secret:
          name: kafka-secret 6
      - name: infra-logs
        type: kafka
        url: tcp://kafka.devlab2.example.com:9093/infra-topic 7
      - name: audit-logs
        type: kafka
        url: tls://kafka.qelab.example.com:9093/audit-topic
        secret:
           name: kafka-secret-qe
     pipelines:
      - name: app-topic 8
        inputRefs: 9
        - application
        outputRefs: 10
        - app-logs
        parse: json 11
        labels:
          logType: "application" 12
      - name: infra-topic 13
        inputRefs:
        - infrastructure
        outputRefs:
        - infra-logs
        labels:
          logType: "infra"
      - name: audit-topic
        inputRefs:
        - audit
        outputRefs:
        - audit-logs
        - default 14
        labels:
          logType: "audit"

   1

   The name of the ClusterLogForwarder CR must be instance.

   2

   The namespace for the ClusterLogForwarder CR must be openshift-logging.

   3

   Specify a name for the output.

   4

   Specify the kafka type.

   5

   Specify the URL and port of the Kafka broker as a valid absolute URL, optionally with a specific topic. You can use the tcp (insecure) or tls (secure TCP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP address.

   6

   If using a tls prefix, you must specify the name of the secret required by the endpoint for TLS communication. The secret must exist in the openshift-logging project and must have keys of: tls.crt, tls.key, and ca-bundle.crt that point to the respective certificates that they represent.

   7

   Optional: To send an insecure output, use a tcp prefix in front of the URL. Also omit the secret key and its name from this output.

   8

   Optional: Specify a name for the pipeline.

   9

   Specify which log types should be forwarded using that pipeline: application, infrastructure, or audit.

   10

   Specify the output to use with that pipeline for forwarding the logs.

   11

   Optional: Forward structured JSON log entries as JSON objects in the structured field. The log entry must contain valid structured JSON; otherwise, OpenShift Logging removes the structured field and instead sends the log entry to the default index, app-00000x.

   12

   Optional: String. One or more labels to add to the logs.

   13

   Optional: Configure multiple outputs to forward logs to other external log aggregators of any supported type:

   • Optional. A name to describe the pipeline.
   • The inputRefs is the log type to forward using that pipeline: application, infrastructure, or audit.
   • The outputRefs is the name of the output to use.
   • Optional: String. One or more labels to add to the logs.

   14

   Optional: Specify default to forward logs to the internal Elasticsearch instance.

2. Optional: To forward a single output to multiple Kafka brokers, specify an array of Kafka brokers as shown in this example:

   ...
   spec:
     outputs:
     - name: app-logs
       type: kafka
       secret:
         name: kafka-secret-dev
       kafka:  1
         brokers: 2
           - tls://kafka-broker1.example.com:9093/
           - tls://kafka-broker2.example.com:9093/
         topic: app-topic 3
   ...

   1

   Specify a kafka key that has a brokers and topic key.

   2

   Use the brokers key to specify an array of one or more brokers.

   3

   Use the topic key to specify the target topic that will receive the logs.

3. Create the CR object:

   $ oc create -f <file-name>.yaml

Procedure

1. Create a ClusterLogForwarder CR YAML file similar to the following:

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: instance 1
     namespace: openshift-logging 2
   spec:
     outputs:
      - name: fluentd-server-secure 3
        type: fluentdForward 4
        url: 'tls://fluentdserver.security.example.com:24224' 5
        secret: 6
           name: fluentd-secret
      - name: fluentd-server-insecure
        type: fluentdForward
        url: 'tcp://fluentdserver.home.example.com:24224'
     inputs: 7
      - name: my-app-logs
        application:
           namespaces:
           - my-project
     pipelines:
      - name: forward-to-fluentd-insecure 8
        inputRefs: 9
        - my-app-logs
        outputRefs: 10
        - fluentd-server-insecure
        parse: json 11
        labels:
          project: "my-project" 12
      - name: forward-to-fluentd-secure 13
        inputRefs:
        - application
        - audit
        - infrastructure
        outputRefs:
        - fluentd-server-secure
        - default
        labels:
          clusterId: "C1234"

   1

   The name of the ClusterLogForwarder CR must be instance.

   2

   The namespace for the ClusterLogForwarder CR must be openshift-logging.

   3

   Specify a name for the output.

   4

   Specify the output type: elasticsearch, fluentdForward, syslog, or kafka.

   5

   Specify the URL and port of the external log aggregator as a valid absolute URL. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP address.

   6

   If using a tls prefix, you must specify the name of the secret required by the endpoint for TLS communication. The secret must exist in the openshift-logging project and have tls.crt, tls.key, and ca-bundle.crt keys that each point to the certificates they represent.

   7

   Configuration for an input to filter application logs from the specified projects.

   8

   Configuration for a pipeline to use the input to send project application logs to an external Fluentd instance.

   9

   The my-app-logs input.

   10

   The name of the output to use.

   11

   Optional: Forward structured JSON log entries as JSON objects in the structured field. The log entry must contain valid structured JSON; otherwise, OpenShift Logging removes the structured field and instead sends the log entry to the default index, app-00000x.

   12

   Optional: String. One or more labels to add to the logs.

   13

   Configuration for a pipeline to send logs to other log aggregators.

   • Optional: Specify a name for the pipeline.
   • Specify which log types should be forwarded using that pipeline: application, infrastructure, or audit.
   • Specify the output to use with that pipeline for forwarding the logs.
   • Optional: Specify the default output to forward logs to the internal Elasticsearch instance.
   • Optional: String. One or more labels to add to the logs.

2. Create the CR object:

   $ oc create -f <file-name>.yaml

Procedure

1. Create a ClusterLogForwarder custom resource (CR) YAML file.

2. In the YAML file, specify the pod labels using simple equality-based selectors under inputs[].name.application.selector.matchLabels, as shown in the following example.

   Example ClusterLogForwarder CR YAML file

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: instance 1
     namespace: openshift-logging 2
   spec:
     pipelines:
       - inputRefs: [ myAppLogData ] 3
         outputRefs: [ default ] 4
         parse: json 5
     inputs: 6
       - name: myAppLogData
         application:
           selector:
             matchLabels: 7
               environment: production
               app: nginx
           namespaces: 8
           - app1
           - app2
     outputs: 9
       - default
       ...

   1

   The name of the ClusterLogForwarder CR must be instance.

   2

   The namespace for the ClusterLogForwarder CR must be openshift-logging.

   3

   Specify one or more comma-separated values from inputs[].name.

   4

   Specify one or more comma-separated values from outputs[].

   5

   Optional: Forward structured JSON log entries as JSON objects in the structured field. The log entry must contain valid structured JSON; otherwise, OpenShift Logging removes the structured field and instead sends the log entry to the default index, app-00000x.

   6

   Define a unique inputs[].name for each application that has a unique set of pod labels.

   7

   Specify the key-value pairs of pod labels whose log data you want to gather. You must specify both a key and value, not just a key. To be selected, the pods must match all the key-value pairs.

   8

   Optional: Specify one or more namespaces.

   9

   Specify one or more outputs to forward your log data to. The optional default output shown here sends log data to the internal Elasticsearch instance.

3. Optional: To restrict the gathering of log data to specific namespaces, use inputs[].name.application.namespaces, as shown in the preceding example.

4. Optional: You can send log data from additional applications that have different pod labels to the same pipeline.

   1. For each unique combination of pod labels, create an additional inputs[].name section similar to the one shown.
   2. Update the selectors to match the pod labels of this application.

   3. Add the new inputs[].name value to inputRefs. For example:

      - inputRefs: [ myAppLogData, myOtherAppLogData ]

5. Create the CR object:

   $ oc create -f <file-name>.yaml

1. Create a configuration file named secure-forward and specify parameters similar to the following within the <store> stanza:

   <store>
     @type forward
     <security>
       self_hostname ${hostname}
       shared_key <key> 1
     </security>
     transport tls 2
     tls_verify_hostname <value> 3
     tls_cert_path <path_to_file> 4
     <buffer> 5
       @type file
       path '/var/lib/fluentd/secureforwardlegacy'
       queued_chunks_limit_size "#{ENV['BUFFER_QUEUE_LIMIT'] || '1024' }"
       chunk_limit_size "#{ENV['BUFFER_SIZE_LIMIT'] || '1m' }"
       flush_interval "#{ENV['FORWARD_FLUSH_INTERVAL'] || '5s'}"
       flush_at_shutdown "#{ENV['FLUSH_AT_SHUTDOWN'] || 'false'}"
       flush_thread_count "#{ENV['FLUSH_THREAD_COUNT'] || 2}"
       retry_max_interval "#{ENV['FORWARD_RETRY_WAIT'] || '300'}"
       retry_forever true
     </buffer>
     <server>
       name 6
       host 7
       hostlabel 8
       port 9
     </server>
     <server> 10
       name
       host
     </server>

   1

   Enter the shared key between nodes.

   2

   Specify tls to enable TLS validation.

   3

   Set to true to verify the server cert hostname. Set to false to ignore server cert hostname.

   4

   Specify the path to the private CA certificate file as /etc/ocp-forward/ca_cert.pem.

   5

   Specify the Fluentd buffer parameters as needed.

   6

   Optionally, enter a name for this server.

   7

   Specify the hostname or IP of the server.

   8

   Specify the host label of the server.

   9

   Specify the port of the server.

   10

   Optionally, add additional servers. If you specify two or more servers, forward uses these server nodes in a round-robin order.

   To use Mutual TLS (mTLS) authentication, see the Fluentd documentation for information about client certificate, key parameters, and other settings.

2. Create a config map named secure-forward in the openshift-logging project from the configuration file:

   $ oc create configmap secure-forward --from-file=secure-forward.conf -n openshift-logging

1. Create a configuration file named syslog.conf and specify parameters similar to the following within the <store> stanza:

   <store>
   @type <type> 1
   remote_syslog <syslog-server> 2
   port 514 3
   hostname ${hostname}
   remove_tag_prefix <prefix> 4
   facility <value>
   severity <value>
   use_record <value>
   payload_key message
   rfc 3164 5
   </store>

   1

   Specify the protocol to use, either: syslog or syslog_buffered.

   2

   Specify the FQDN or IP address of the syslog server.

   3

   Specify the port of the syslog server.

   4

   Optional: Specify the appropriate syslog parameters, for example:

   • Parameter to remove the specified tag field from the syslog prefix.
   • Parameter to set the specified field as the syslog key.
   • Parameter to specify the syslog log facility or source.
   • Parameter to specify the syslog log severity.
   • Parameter to use the severity and facility from the record if available. If true, the container_name, namespace_name, and pod_name are included in the output content.
   • Parameter to specify the key to set the payload of the syslog message. Defaults to message.

   5

   With the legacy syslog method, you must specify 3164 for the rfc value.

2. Create a config map named syslog in the openshift-logging project from the configuration file:

   $ oc create configmap syslog --from-file=syslog.conf -n openshift-logging

Procedure

1. Add the following snippet to your ClusterLogForwarder CR YAML file.

   outputDefaults:
   - elasticsearch:
       structuredTypeKey: <log record field>
       structuredTypeName: <name>
   pipelines:
   - inputRefs:
     - application
     outputRefs: default
     parse: json

2. Optional: Use structuredTypeKey to specify one of the log record fields, as described in the preceding topic, Configuring JSON log data for Elasticsearch. Otherwise, remove this line.

3. Optional: Use structuredTypeName to specify a <name>, as described in the preceding topic, Configuring JSON log data for Elasticsearch. Otherwise, remove this line.

   Important

   To parse JSON logs, you must set either structuredTypeKey or structuredTypeName, or both structuredTypeKey and structuredTypeName.

4. For inputRefs, specify which log types should be forwarded using that pipeline, such as application, infrastructure, or audit.
5. Add the parse: json element to pipelines.

6. Create the CR object:

   $ oc create -f <file-name>.yaml

   The Red Hat OpenShift Logging Operator redeploys the Fluentd pods. However, if they do not redeploy, delete the Fluentd pods to force them to redeploy.

   $ oc delete pod --selector logging-infra=fluentd

Procedure

1. Create a template for the Event Router:

   kind: Template
   apiVersion: template.openshift.io/v1
   metadata:
     name: eventrouter-template
     annotations:
       description: "A pod forwarding kubernetes events to OpenShift Logging stack."
       tags: "events,EFK,logging,cluster-logging"
   objects:
     - kind: ServiceAccount 1
       apiVersion: v1
       metadata:
         name: eventrouter
         namespace: ${NAMESPACE}
     - kind: ClusterRole 2
       apiVersion: rbac.authorization.k8s.io/v1
       metadata:
         name: event-reader
       rules:
       - apiGroups: [""]
         resources: ["events"]
         verbs: ["get", "watch", "list"]
     - kind: ClusterRoleBinding  3
       apiVersion: rbac.authorization.k8s.io/v1
       metadata:
         name: event-reader-binding
       subjects:
       - kind: ServiceAccount
         name: eventrouter
         namespace: ${NAMESPACE}
       roleRef:
         kind: ClusterRole
         name: event-reader
     - kind: ConfigMap 4
       apiVersion: v1
       metadata:
         name: eventrouter
         namespace: ${NAMESPACE}
       data:
         config.json: |-
           {
             "sink": "stdout"
           }
     - kind: Deployment 5
       apiVersion: apps/v1
       metadata:
         name: eventrouter
         namespace: ${NAMESPACE}
         labels:
           component: "eventrouter"
           logging-infra: "eventrouter"
           provider: "openshift"
       spec:
         selector:
           matchLabels:
             component: "eventrouter"
             logging-infra: "eventrouter"
             provider: "openshift"
         replicas: 1
         template:
           metadata:
             labels:
               component: "eventrouter"
               logging-infra: "eventrouter"
               provider: "openshift"
             name: eventrouter
           spec:
             serviceAccount: eventrouter
             containers:
               - name: kube-eventrouter
                 image: ${IMAGE}
                 imagePullPolicy: IfNotPresent
                 resources:
                   requests:
                     cpu: ${CPU}
                     memory: ${MEMORY}
                 volumeMounts:
                 - name: config-volume
                   mountPath: /etc/eventrouter
             volumes:
               - name: config-volume
                 configMap:
                   name: eventrouter
   parameters:
     - name: IMAGE 6
       displayName: Image
       value: "registry.redhat.io/openshift-logging/eventrouter-rhel8:v0.3"
     - name: CPU  7
       displayName: CPU
       value: "100m"
     - name: MEMORY 8
       displayName: Memory
       value: "128Mi"
     - name: NAMESPACE
       displayName: Namespace
       value: "openshift-logging" 9

   1

   Creates a Service Account in the openshift-logging project for the Event Router.

   2

   Creates a ClusterRole to monitor for events in the cluster.

   3

   Creates a ClusterRoleBinding to bind the ClusterRole to the service account.

   4

   Creates a config map in the openshift-logging project to generate the required config.json file.

   5

   Creates a deployment in the openshift-logging project to generate and configure the Event Router pod.

   6

   Specifies the image, identified by a tag such as v0.3.

   7

   Specifies the minimum amount of memory to allocate to the Event Router pod. Defaults to 128Mi.

   8

   Specifies the minimum amount of CPU to allocate to the Event Router pod. Defaults to 100m.

   9

   Specifies the openshift-logging project to install objects in.

2. Use the following command to process and apply the template:

   $ oc process -f <templatefile> | oc apply -n openshift-logging -f -

   For example:

   $ oc process -f eventrouter.yaml | oc apply -n openshift-logging -f -

   Example output

   serviceaccount/eventrouter created
   clusterrole.authorization.openshift.io/event-reader created
   clusterrolebinding.authorization.openshift.io/event-reader-binding created
   configmap/eventrouter created
   deployment.apps/eventrouter created

3. Validate that the Event Router installed in the openshift-logging project:

   1. View the new Event Router pod:

      $ oc get pods --selector  component=eventrouter -o name -n openshift-logging

      Example output

      pod/cluster-logging-eventrouter-d649f97c8-qvv8r

   2. View the events collected by the Event Router:

      $ oc logs <cluster_logging_eventrouter_pod> -n openshift-logging

      For example:

      $ oc logs cluster-logging-eventrouter-d649f97c8-qvv8r -n openshift-logging

      Example output

      {"verb":"ADDED","event":{"metadata":{"name":"openshift-service-catalog-controller-manager-remover.1632d931e88fcd8f","namespace":"openshift-service-catalog-removed","selfLink":"/api/v1/namespaces/openshift-service-catalog-removed/events/openshift-service-catalog-controller-manager-remover.1632d931e88fcd8f","uid":"787d7b26-3d2f-4017-b0b0-420db4ae62c0","resourceVersion":"21399","creationTimestamp":"2020-09-08T15:40:26Z"},"involvedObject":{"kind":"Job","namespace":"openshift-service-catalog-removed","name":"openshift-service-catalog-controller-manager-remover","uid":"fac9f479-4ad5-4a57-8adc-cb25d3d9cf8f","apiVersion":"batch/v1","resourceVersion":"21280"},"reason":"Completed","message":"Job completed","source":{"component":"job-controller"},"firstTimestamp":"2020-09-08T15:40:26Z","lastTimestamp":"2020-09-08T15:40:26Z","count":1,"type":"Normal"}}

      You can also use Kibana to view events by creating an index pattern using the Elasticsearch infra index.

Procedure

1. Update the OpenShift Elasticsearch Operator:

   1. From the web console, click Operators → Installed Operators.
   2. Select the openshift-operators-redhat project.
   3. Click the OpenShift Elasticsearch Operator.
   4. Click Subscription → Channel.
   5. In the Change Subscription Update Channel window, select 5.0 or stable-5.1 and click Save.

   6. Wait for a few seconds, then click Operators → Installed Operators.

      Verify that the OpenShift Elasticsearch Operator version is 5.x.x.

      Wait for the Status field to report Succeeded.

2. Update the Cluster Logging Operator:

   1. From the web console, click Operators → Installed Operators.
   2. Select the openshift-logging project.
   3. Click the Cluster Logging Operator.
   4. Click Subscription → Channel.
   5. In the Change Subscription Update Channel window, select 5.0 or stable-5.1 and click Save.

   6. Wait for a few seconds, then click Operators → Installed Operators.

      Verify that the Red Hat OpenShift Logging Operator version is 5.0.x or 5.1.x.

      Wait for the Status field to report Succeeded.

3. Check the logging components:

   1. Ensure that all Elasticsearch pods are in the Ready status:

      $ oc get pod -n openshift-logging --selector component=elasticsearch

      Example output

      NAME                                            READY   STATUS    RESTARTS   AGE
      elasticsearch-cdm-1pbrl44l-1-55b7546f4c-mshhk   2/2     Running   0          31m
      elasticsearch-cdm-1pbrl44l-2-5c6d87589f-gx5hk   2/2     Running   0          30m
      elasticsearch-cdm-1pbrl44l-3-88df5d47-m45jc     2/2     Running   0          29m

   2. Ensure that the Elasticsearch cluster is healthy:

      $ oc exec -n openshift-logging -c elasticsearch elasticsearch-cdm-1pbrl44l-1-55b7546f4c-mshhk -- health

      {
        "cluster_name" : "elasticsearch",
        "status" : "green",
      }

   3. Ensure that the Elasticsearch cron jobs are created:

      $ oc project openshift-logging

      $ oc get cronjob

      NAME                     SCHEDULE       SUSPEND   ACTIVE   LAST SCHEDULE   AGE
      elasticsearch-im-app     */15 * * * *   False     0        <none>          56s
      elasticsearch-im-audit   */15 * * * *   False     0        <none>          56s
      elasticsearch-im-infra   */15 * * * *   False     0        <none>          56s

   4. Verify that the log store is updated to 5.0 or 5.1 and the indices are green:

      $ oc exec -c elasticsearch <any_es_pod_in_the_cluster> -- indices

      Verify that the output includes the app-00000x, infra-00000x, audit-00000x, .security indices.

      Example 10.1. Sample output with indices in a green status

      Tue Jun 30 14:30:54 UTC 2020
      health status index                                                                 uuid                   pri rep docs.count docs.deleted store.size pri.store.size
      green  open   infra-000008                                                          bnBvUFEXTWi92z3zWAzieQ   3 1       222195            0        289            144
      green  open   infra-000004                                                          rtDSzoqsSl6saisSK7Au1Q   3 1       226717            0        297            148
      green  open   infra-000012                                                          RSf_kUwDSR2xEuKRZMPqZQ   3 1       227623            0        295            147
      green  open   .kibana_7                                                             1SJdCqlZTPWlIAaOUd78yg   1 1            4            0          0              0
      green  open   infra-000010                                                          iXwL3bnqTuGEABbUDa6OVw   3 1       248368            0        317            158
      green  open   infra-000009                                                          YN9EsULWSNaxWeeNvOs0RA   3 1       258799            0        337            168
      green  open   infra-000014                                                          YP0U6R7FQ_GVQVQZ6Yh9Ig   3 1       223788            0        292            146
      green  open   infra-000015                                                          JRBbAbEmSMqK5X40df9HbQ   3 1       224371            0        291            145
      green  open   .orphaned.2020.06.30                                                  n_xQC2dWQzConkvQqei3YA   3 1            9            0          0              0
      green  open   infra-000007                                                          llkkAVSzSOmosWTSAJM_hg   3 1       228584            0        296            148
      green  open   infra-000005                                                          d9BoGQdiQASsS3BBFm2iRA   3 1       227987            0        297            148
      green  open   infra-000003                                                          1-goREK1QUKlQPAIVkWVaQ   3 1       226719            0        295            147
      green  open   .security                                                             zeT65uOuRTKZMjg_bbUc1g   1 1            5            0          0              0
      green  open   .kibana-377444158_kubeadmin                                           wvMhDwJkR-mRZQO84K0gUQ   3 1            1            0          0              0
      green  open   infra-000006                                                          5H-KBSXGQKiO7hdapDE23g   3 1       226676            0        295            147
      green  open   infra-000001                                                          eH53BQ-bSxSWR5xYZB6lVg   3 1       341800            0        443            220
      green  open   .kibana-6                                                             RVp7TemSSemGJcsSUmuf3A   1 1            4            0          0              0
      green  open   infra-000011                                                          J7XWBauWSTe0jnzX02fU6A   3 1       226100            0        293            146
      green  open   app-000001                                                            axSAFfONQDmKwatkjPXdtw   3 1       103186            0        126             57
      green  open   infra-000016                                                          m9c1iRLtStWSF1GopaRyCg   3 1        13685            0         19              9
      green  open   infra-000002                                                          Hz6WvINtTvKcQzw-ewmbYg   3 1       228994            0        296            148
      green  open   infra-000013                                                          KR9mMFUpQl-jraYtanyIGw   3 1       228166            0        298            148
      green  open   audit-000001                                                          eERqLdLmQOiQDFES1LBATQ   3 1            0            0          0              0

   5. Verify that the log collector is updated to 5.0 or 5.1:

      $ oc get ds fluentd -o json | grep fluentd-init

      Verify that the output includes a fluentd-init container:

      "containerName": "fluentd-init"

   6. Verify that the log visualizer is updated to 5.0 or 5.1 using the Kibana CRD:

      $ oc get kibana kibana -o json

      Verify that the output includes a Kibana pod with the ready status:

      Example 10.2. Sample output with a ready Kibana pod

      [
      {
      "clusterCondition": {
      "kibana-5fdd766ffd-nb2jj": [
      {
      "lastTransitionTime": "2020-06-30T14:11:07Z",
      "reason": "ContainerCreating",
      "status": "True",
      "type": ""
      },
      {
      "lastTransitionTime": "2020-06-30T14:11:07Z",
      "reason": "ContainerCreating",
      "status": "True",
      "type": ""
      }
      ]
      },
      "deployment": "kibana",
      "pods": {
      "failed": [],
      "notReady": []
      "ready": []
      },
      "replicaSets": [
      "kibana-5fdd766ffd"
      ],
      "replicas": 1
      }
      ]

Procedure

1. Update the OpenShift Elasticsearch Operator:

   1. From the web console, click Operators → Installed Operators.
   2. Select the openshift-operators-redhat project.
   3. Click the OpenShift Elasticsearch Operator.
   4. Click Subscription → Channel.
   5. In the Change Subscription Update Channel window, select stable-5.x and click Save.

   6. Wait for a few seconds, then click Operators → Installed Operators.

      Verify that the OpenShift Elasticsearch Operator version is 5.x.x.

      Wait for the Status field to report Succeeded.

2. Update the Red Hat OpenShift Logging Operator:

   1. From the web console, click Operators → Installed Operators.
   2. Select the openshift-logging project.
   3. Click the Red Hat OpenShift Logging Operator.
   4. Click Subscription → Channel.
   5. In the Change Subscription Update Channel window, select stable-5.x and click Save.

   6. Wait for a few seconds, then click Operators → Installed Operators.

      Verify that the Red Hat OpenShift Logging Operator version is 5.x.x.

      Wait for the Status field to report Succeeded.

3. Check the logging components:

   1. Ensure that all Elasticsearch pods are in the Ready status:

      $ oc get pod -n openshift-logging --selector component=elasticsearch

      Example output

      NAME                                            READY   STATUS    RESTARTS   AGE
      elasticsearch-cdm-1pbrl44l-1-55b7546f4c-mshhk   2/2     Running   0          31m
      elasticsearch-cdm-1pbrl44l-2-5c6d87589f-gx5hk   2/2     Running   0          30m
      elasticsearch-cdm-1pbrl44l-3-88df5d47-m45jc     2/2     Running   0          29m

   2. Ensure that the Elasticsearch cluster is healthy:

      $ oc exec -n openshift-logging -c elasticsearch elasticsearch-cdm-1pbrl44l-1-55b7546f4c-mshhk -- health

      {
        "cluster_name" : "elasticsearch",
        "status" : "green",
      }

   3. Ensure that the Elasticsearch cron jobs are created:

      $ oc project openshift-logging

      $ oc get cronjob

      NAME                     SCHEDULE       SUSPEND   ACTIVE   LAST SCHEDULE   AGE
      elasticsearch-im-app     */15 * * * *   False     0        <none>          56s
      elasticsearch-im-audit   */15 * * * *   False     0        <none>          56s
      elasticsearch-im-infra   */15 * * * *   False     0        <none>          56s

   4. Verify that the log store is updated to 5.x and the indices are green:

      $ oc exec -c elasticsearch <any_es_pod_in_the_cluster> -- indices

      Verify that the output includes the app-00000x, infra-00000x, audit-00000x, .security indices.

      Example 10.3. Sample output with indices in a green status

      Tue Jun 30 14:30:54 UTC 2020
      health status index                                                                 uuid                   pri rep docs.count docs.deleted store.size pri.store.size
      green  open   infra-000008                                                          bnBvUFEXTWi92z3zWAzieQ   3 1       222195            0        289            144
      green  open   infra-000004                                                          rtDSzoqsSl6saisSK7Au1Q   3 1       226717            0        297            148
      green  open   infra-000012                                                          RSf_kUwDSR2xEuKRZMPqZQ   3 1       227623            0        295            147
      green  open   .kibana_7                                                             1SJdCqlZTPWlIAaOUd78yg   1 1            4            0          0              0
      green  open   infra-000010                                                          iXwL3bnqTuGEABbUDa6OVw   3 1       248368            0        317            158
      green  open   infra-000009                                                          YN9EsULWSNaxWeeNvOs0RA   3 1       258799            0        337            168
      green  open   infra-000014                                                          YP0U6R7FQ_GVQVQZ6Yh9Ig   3 1       223788            0        292            146
      green  open   infra-000015                                                          JRBbAbEmSMqK5X40df9HbQ   3 1       224371            0        291            145
      green  open   .orphaned.2020.06.30                                                  n_xQC2dWQzConkvQqei3YA   3 1            9            0          0              0
      green  open   infra-000007                                                          llkkAVSzSOmosWTSAJM_hg   3 1       228584            0        296            148
      green  open   infra-000005                                                          d9BoGQdiQASsS3BBFm2iRA   3 1       227987            0        297            148
      green  open   infra-000003                                                          1-goREK1QUKlQPAIVkWVaQ   3 1       226719            0        295            147
      green  open   .security                                                             zeT65uOuRTKZMjg_bbUc1g   1 1            5            0          0              0
      green  open   .kibana-377444158_kubeadmin                                           wvMhDwJkR-mRZQO84K0gUQ   3 1            1            0          0              0
      green  open   infra-000006                                                          5H-KBSXGQKiO7hdapDE23g   3 1       226676            0        295            147
      green  open   infra-000001                                                          eH53BQ-bSxSWR5xYZB6lVg   3 1       341800            0        443            220
      green  open   .kibana-6                                                             RVp7TemSSemGJcsSUmuf3A   1 1            4            0          0              0
      green  open   infra-000011                                                          J7XWBauWSTe0jnzX02fU6A   3 1       226100            0        293            146
      green  open   app-000001                                                            axSAFfONQDmKwatkjPXdtw   3 1       103186            0        126             57
      green  open   infra-000016                                                          m9c1iRLtStWSF1GopaRyCg   3 1        13685            0         19              9
      green  open   infra-000002                                                          Hz6WvINtTvKcQzw-ewmbYg   3 1       228994            0        296            148
      green  open   infra-000013                                                          KR9mMFUpQl-jraYtanyIGw   3 1       228166            0        298            148
      green  open   audit-000001                                                          eERqLdLmQOiQDFES1LBATQ   3 1            0            0          0              0

   5. Verify that the log collector is updated to 5.x:

      $ oc get ds fluentd -o json | grep fluentd-init

      Verify that the output includes a fluentd-init container:

      "containerName": "fluentd-init"

   6. Verify that the log visualizer is updated to 5.x using the Kibana CRD:

      $ oc get kibana kibana -o json

      Verify that the output includes a Kibana pod with the ready status:

      Example 10.4. Sample output with a ready Kibana pod

      [
      {
      "clusterCondition": {
      "kibana-5fdd766ffd-nb2jj": [
      {
      "lastTransitionTime": "2020-06-30T14:11:07Z",
      "reason": "ContainerCreating",
      "status": "True",
      "type": ""
      },
      {
      "lastTransitionTime": "2020-06-30T14:11:07Z",
      "reason": "ContainerCreating",
      "status": "True",
      "type": ""
      }
      ]
      },
      "deployment": "kibana",
      "pods": {
      "failed": [],
      "notReady": []
      "ready": []
      },
      "replicaSets": [
      "kibana-5fdd766ffd"
      ],
      "replicas": 1
      }
      ]

1. In the OpenShift Container Platform web console, click Monitoring → Dashboards.

2. On the Dashboards page, select Logging/Elasticsearch Nodes or OpenShift Logging from the Dashboard menu.

   For the Logging/Elasticsearch Nodes dashboard, you can select the Elasticsearch node you want to view and set the data resolution.

   The appropriate dashboard is displayed, showing multiple charts of data.

3. Optionally, select a different time range to display or refresh rate for the data from the Time Range and Refresh Interval menus.