Logging

1. Create a Namespace for the Elasticsearch Operator.

   1. Create a Namespace object YAML file (for example, eo-namespace.yaml) for the Elasticsearch Operator:

      apiVersion: v1
      kind: Namespace
      metadata:
        name: openshift-operators-redhat 1
        annotations:
          openshift.io/node-selector: ""
        labels:
          openshift.io/cluster-monitoring: "true" 2

      1

      You must specify the openshift-operators-redhat Namespace. To prevent possible conflicts with metrics, you should configure the Prometheus Cluster Monitoring stack to scrape metrics from the openshift-operators-redhat Namespace and not the openshift-operators Namespace. The openshift-operators Namespace might contain Community Operators, which are untrusted and could publish a metric with the same name as an OpenShift Container Platform metric, which would cause conflicts.

      2

      You must specify this label as shown to ensure that cluster monitoring scrapes the openshift-operators-redhat Namespace.

   2. Create the Namespace:

      $ oc create -f <file-name>.yaml

      For example:

      $ oc create -f eo-namespace.yaml

2. Install the Elasticsearch Operator by creating the following objects:

   1. Create an Operator Group object YAML file (for example, eo-og.yaml) for the Elasticsearch operator:

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: openshift-operators-redhat
        namespace: openshift-operators-redhat 1
      spec: {}

      1

      You must specify the openshift-operators-redhat Namespace.

   2. Create an Operator Group object:

      $ oc create -f <file-name>.yaml

      For example:

      $ oc create -f eo-og.yaml

   3. Create a Subscription object YAML file (for example, eo-sub.yaml) to subscribe a Namespace to an Operator.

      Example Subscription

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: "elasticsearch-operator"
        namespace: "openshift-operators-redhat" 1
      spec:
        channel: "4.2" 2
        installPlanApproval: "Automatic"
        source: "redhat-operators" 3
        sourceNamespace: "openshift-marketplace"
        name: "elasticsearch-operator"

      1

      You must specify the openshift-operators-redhat Namespace.

      2

      Specify 4.2 as the channel.

      3

      Specify redhat-operators. If your OpenShift Container Platform cluster is installed on a restricted network, also known as a disconnected cluster, specify the name of the CatalogSource object created when you configured the Operator Lifecycle Manager (OLM).

   4. Create the Subscription object:

      $ oc create -f <file-name>.yaml

      For example:

      $ oc create -f eo-sub.yaml

   5. Change to the openshift-operators-redhat project:

      $ oc project openshift-operators-redhat
      
      Now using project "openshift-operators-redhat"

   6. Create a Role-based Access Control (RBAC) object file (for example, eo-rbac.yaml) to grant Prometheus permission to access the openshift-operators-redhat Namespace:

      apiVersion: rbac.authorization.k8s.io/v1
      kind: Role
      metadata:
        name: prometheus-k8s
        namespace: openshift-operators-redhat
      rules:
      - apiGroups:
        - ""
        resources:
        - services
        - endpoints
        - pods
        verbs:
        - get
        - list
        - watch
      ---
      apiVersion: rbac.authorization.k8s.io/v1
      kind: RoleBinding
      metadata:
        name: prometheus-k8s
        namespace: openshift-operators-redhat
      roleRef:
        apiGroup: rbac.authorization.k8s.io
        kind: Role
        name: prometheus-k8s
      subjects:
      - kind: ServiceAccount
        name: prometheus-k8s
        namespace: openshift-operators-redhat

   7. Create the RBAC object:

      $ oc create -f <file-name>.yaml

      For example:

      $ oc create -f eo-rbac.yaml

      The Elasticsearch Operator is installed to the openshift-operators-redhat Namespace and copied to each project in the cluster.

3. Verify the Operator installation:

   oc get csv --all-namespaces
   
   NAMESPACE                                               NAME                                         DISPLAY                  VERSION               REPLACES   PHASE
   default                                                 elasticsearch-operator.4.2.1-202002032140    Elasticsearch Operator   4.2.1-202002032140               Succeeded
   kube-node-lease                                         elasticsearch-operator.4.2.1-202002032140    Elasticsearch Operator   4.2.1-202002032140               Succeeded
   kube-public                                             elasticsearch-operator.4.2.1-202002032140    Elasticsearch Operator   4.2.1-202002032140               Succeeded
   kube-system                                             elasticsearch-operator.4.2.1-202002032140    Elasticsearch Operator   4.2.1-202002032140               Succeeded
   openshift-apiserver-operator                            elasticsearch-operator.4.2.1-202002032140    Elasticsearch Operator   4.2.1-202002032140               Succeeded
   openshift-apiserver                                     elasticsearch-operator.4.2.1-202002032140    Elasticsearch Operator   4.2.1-202002032140               Succeeded
   openshift-authentication-operator                       elasticsearch-operator.4.2.1-202002032140    Elasticsearch Operator   4.2.1-202002032140               Succeeded
   openshift-authentication                                elasticsearch-operator.4.2.1-202002032140    Elasticsearch Operator   4.2.1-202002032140               Succeeded
   ...

   There should be an Elasticsearch Operator in each Namespace. The version number might be different than shown.

1. Create a Namespace for the Cluster Logging Operator. You must use the CLI to create the Namespace.

   1. Create a Namespace object YAML file (for example, clo-namespace.yaml) for the Cluster Logging Operator:

      apiVersion: v1
      kind: Namespace
      metadata:
        name: openshift-logging 1
        annotations:
          openshift.io/node-selector: "" 2
        labels:
          openshift.io/cluster-logging: "true"
          openshift.io/cluster-monitoring: "true"

      1 2

      Specify these values as shown.

   2. Create the Namespace:

      $ oc create -f <file-name>.yaml

      For example:

      $ oc create -f clo-namespace.yaml

2. Install the Cluster Logging Operator:

   1. In the OpenShift Container Platform web console, click Operators → OperatorHub.
   2. Choose Cluster Logging from the list of available Operators, and click Install.
   3. On the Create Operator Subscription page, under A specific Namespace on the cluster select openshift-logging. Then, click Subscribe.

3. Verify that the Cluster Logging Operator installed:

   1. Switch to the Operators → Installed Operators page.

   2. Ensure that Cluster Logging is listed in the openshift-logging project with a Status of InstallSucceeded.

      Note

      During installation an Operator might display a Failed status. If the Operator then installs with an InstallSucceeded message, you can safely ignore the Failed message.

      If the Operator does not appear as installed, to troubleshoot further:

      • Switch to the Operators → Installed Operators page and inspect the Status column for any errors or failures.
      • Switch to the Workloads → Pods page and check the logs in any Pods in the openshift-logging and openshift-operators-redhat projects that are reporting issues.

4. Create a cluster logging instance:

   1. Switch to the Administration → Custom Resource Definitions page.
   2. On the Custom Resource Definitions page, click ClusterLogging.
   3. On the Custom Resource Definition Overview page, select View Instances from the Actions menu.

   4. On the Cluster Loggings page, click Create Cluster Logging.

      You might have to refresh the page to load the data.

   5. In the YAML field, replace the code with the following:

      Note

      This default cluster logging configuration should support a wide array of environments. Review the topics on tuning and configuring the cluster logging components for information on modifications you can make to your cluster logging cluster.

      apiVersion: "logging.openshift.io/v1"
      kind: "ClusterLogging"
      metadata:
        name: "instance" 1
        namespace: "openshift-logging"
      spec:
        managementState: "Managed"  2
        logStore:
          type: "elasticsearch"  3
          elasticsearch:
            nodeCount: 3 4
            storage:
              storageClassName: gp2 5
              size: 200G
            redundancyPolicy: "SingleRedundancy"
        visualization:
          type: "kibana"  6
          kibana:
            replicas: 1
        curation:
          type: "curator"  7
          curator:
            schedule: "30 3 * * *"
        collection:
          logs:
            type: "fluentd"  8
            fluentd: {}

      1

      The name must be instance.

      2

      The cluster logging management state. In most cases, if you change the cluster logging defaults, you must set this to Unmanaged. However, an unmanaged deployment does not receive updates until the cluster logging is placed back into a managed state. For more information, see Changing cluster logging management state.

      3

      Settings for configuring Elasticsearch. Using the CR, you can configure shard replication policy and persistent storage. For more information, see Configuring Elasticsearch.

      4

      Specify the number of Elasticsearch nodes. See the note that follows this list.

      5

      Specify that each Elasticsearch node in the cluster is bound to a Persistent Volume Claim.

      6

      Settings for configuring Kibana. Using the CR, you can scale Kibana for redundancy and configure the CPU and memory for your Kibana nodes. For more information, see Configuring Kibana.

      7

      Settings for configuring Curator. Using the CR, you can set the Curator schedule. For more information, see Configuring Curator.

      8

      Settings for configuring Fluentd. Using the CR, you can configure Fluentd CPU and memory limits. For more information, see Configuring Fluentd.

      Note

      The maximum number of Elasticsearch master nodes is three. If you specify a nodeCount greater than 3, OpenShift Container Platform creates three Elasticsearch nodes that are Master-eligible nodes, with the master, client, and data roles. The additional Elasticsearch nodes are created as Data-only nodes, using client and data roles. Master nodes perform cluster-wide actions such as creating or deleting an index, shard allocation, and tracking nodes. Data nodes hold the shards and perform data-related operations such as CRUD, search, and aggregations. Data-related operations are I/O-, memory-, and CPU-intensive. It is important to monitor these resources and to add more Data nodes if the current nodes are overloaded.

      For example, if nodeCount=4, the following nodes are created:

      $ oc get deployment
      
      cluster-logging-operator       1/1     1            1           18h
      elasticsearch-cd-x6kdekli-1    0/1     1            0           6m54s
      elasticsearch-cdm-x6kdekli-1   1/1     1            1           18h
      elasticsearch-cdm-x6kdekli-2   0/1     1            0           6m49s
      elasticsearch-cdm-x6kdekli-3   0/1     1            0           6m44s

      The number of primary shards for the index templates is equal to the number of Elasticsearch data nodes.

   6. Click Create. This creates the Cluster Logging Custom Resource and Elasticsearch Custom Resource, which you can edit to make changes to your cluster logging cluster.

5. Verify the install:

   1. Switch to the Workloads → Pods page.

   2. Select the openshift-logging project.

      You should see several Pods for cluster logging, Elasticsearch, Fluentd, and Kibana similar to the following list:

      • cluster-logging-operator-cb795f8dc-xkckc
      • elasticsearch-cdm-b3nqzchd-1-5c6797-67kfz
      • elasticsearch-cdm-b3nqzchd-2-6657f4-wtprv
      • elasticsearch-cdm-b3nqzchd-3-588c65-clg7g
      • fluentd-2c7dg
      • fluentd-9z7kk
      • fluentd-br7r2
      • fluentd-fn2sb
      • fluentd-pb2f8
      • fluentd-zqgqx
      • kibana-7fb4fd4cc9-bvt4p

1. Create a Namespace for the Cluster Logging Operator:

   1. Create a Namespace object YAML file (for example, clo-namespace.yaml) for the Cluster Logging Operator:

      apiVersion: v1
      kind: Namespace
      metadata:
        name: openshift-logging
        annotations:
          openshift.io/node-selector: ""
        labels:
          openshift.io/cluster-logging: "true"
          openshift.io/cluster-monitoring: "true"

   2. Create the Namespace:

      $ oc create -f <file-name>.yaml

      For example:

      $ oc create -f clo-namespace.yaml

2. Install the Cluster Logging Operator by creating the following objects:

   1. Create an OperatorGroup object YAML file (for example, clo-og.yaml) for the Cluster Logging Operator:

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: cluster-logging
        namespace: openshift-logging 1
      spec:
        targetNamespaces:
        - openshift-logging 2

      1 2

      You must specify the openshift-logging namespace.

   2. Create the OperatorGroup object:

      $ oc create -f <file-name>.yaml

      For example:

      $ oc create -f clo-og.yaml

   3. Create a Subscription object YAML file (for example, clo-sub.yaml) to subscribe a Namespace to an Operator.

      Example Subscription

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: cluster-logging
        namespace: openshift-logging 1
      spec:
        channel: "4.2" 2
        name: cluster-logging
        source: redhat-operators 3
        sourceNamespace: openshift-marketplace

      1

      You must specify the openshift-logging Namespace.

      2

      Specify 4.2 as the channel.

      3

      Specify redhat-operators. If your OpenShift Container Platform cluster is installed on a restricted network, also known as a disconnected cluster, specify the name of the CatalogSource object you created when you configured the Operator Lifecycle Manager (OLM).

   4. Create the Subscription object:

      $ oc create -f <file-name>.yaml

      For example:

      $ oc create -f clo-sub.yaml

      The Cluster Logging Operator is installed to the openshift-logging Namespace.

3. Verify the Operator installation.

   There should be a Cluster Logging Operator in the openshift-logging Namespace. The Version number might be different than shown.

   oc get csv --all-namespaces
   
   NAMESPACE                                               NAME                                         DISPLAY                  VERSION               REPLACES   PHASE
   ...
   openshift-logging                                       clusterlogging.4.2.1-202002032140            Cluster Logging          4.2.1-202002032140               Succeeded
   ...

4. Create a Cluster Logging instance:

   1. Create an instance object YAML file (for example, clo-instance.yaml) for the Cluster Logging Operator:

      Note

      This default Cluster Logging configuration should support a wide array of environments. Review the topics on tuning and configuring the Cluster Logging components for information on modifications you can make to your Cluster Logging cluster.

      apiVersion: "logging.openshift.io/v1"
      kind: "ClusterLogging"
      metadata:
        name: "instance" 1
        namespace: "openshift-logging"
      spec:
        managementState: "Managed"  2
        logStore:
          type: "elasticsearch"  3
          elasticsearch:
            nodeCount: 3 4
            storage:
              storageClassName: gp2 5
              size: 200G
            redundancyPolicy: "SingleRedundancy"
        visualization:
          type: "kibana"  6
          kibana:
            replicas: 1
        curation:
          type: "curator"  7
          curator:
            schedule: "30 3 * * *"
        collection:
          logs:
            type: "fluentd"  8
            fluentd: {}

      1

      The name must be instance.

      2

      The Cluster Logging management state. In most cases, if you change the Cluster Logging defaults, you must set this to Unmanaged. However, an unmanaged deployment does not receive updates until Cluster Logging is placed back into the Managed state. For more information, see Changing cluster logging management state.

      3

      Settings for configuring Elasticsearch. Using the Custom Resource (CR), you can configure shard replication policy and persistent storage. For more information, see Configuring Elasticsearch.

      4

      Specify the number of Elasticsearch nodes. See the note that follows this list.

      5

      Specify that each Elasticsearch node in the cluster is bound to a Persistent Volume Claim.

      6

      Settings for configuring Kibana. Using the CR, you can scale Kibana for redundancy and configure the CPU and memory for your Kibana nodes. For more information, see Configuring Kibana.

      7

      Settings for configuring Curator. Using the CR, you can set the Curator schedule. For more information, see Configuring Curator.

      8

      Settings for configuring Fluentd. Using the CR, you can configure Fluentd CPU and memory limits. For more information, see Configuring Fluentd.

      Note

      The maximum number of Elasticsearch master nodes is three. If you specify a nodeCount greater than 3, OpenShift Container Platform creates three Elasticsearch nodes that are Master-eligible nodes, with the master, client, and data roles. The additional Elasticsearch nodes are created as Data-only nodes, using client and data roles. Master nodes perform cluster-wide actions such as creating or deleting an index, shard allocation, and tracking nodes. Data nodes hold the shards and perform data-related operations such as CRUD, search, and aggregations. Data-related operations are I/O-, memory-, and CPU-intensive. It is important to monitor these resources and to add more Data nodes if the current nodes are overloaded.

      For example, if nodeCount=4, the following nodes are created:

      $ oc get deployment
      
      cluster-logging-operator       1/1     1            1           18h
      elasticsearch-cd-x6kdekli-1    1/1     1            0           6m54s
      elasticsearch-cdm-x6kdekli-1   1/1     1            1           18h
      elasticsearch-cdm-x6kdekli-2   1/1     1            0           6m49s
      elasticsearch-cdm-x6kdekli-3   1/1     1            0           6m44s

      The number of primary shards for the index templates is equal to the number of Elasticsearch data nodes.

   2. Create the instance:

      $ oc create -f <file-name>.yaml

      For example:

      $ oc create -f clo-instance.yaml

5. Verify the install by listing the Pods in the openshift-logging project.

   You should see several Pods for Cluster Logging, Elasticsearch, Fluentd, and Kibana similar to the following list:

   oc get pods -n openshift-logging
   
   NAME                                            READY   STATUS    RESTARTS   AGE
   cluster-logging-operator-66f77ffccb-ppzbg       1/1     Running   0          7m
   elasticsearch-cdm-ftuhduuw-1-ffc4b9566-q6bhp    2/2     Running   0          2m40s
   elasticsearch-cdm-ftuhduuw-2-7b4994dbfc-rd2gc   2/2     Running   0          2m36s
   elasticsearch-cdm-ftuhduuw-3-84b5ff7ff8-gqnm2   2/2     Running   0          2m4s
   fluentd-587vb                                   1/1     Running   0          2m26s
   fluentd-7mpb9                                   1/1     Running   0          2m30s
   fluentd-flm6j                                   1/1     Running   0          2m33s
   fluentd-gn4rn                                   1/1     Running   0          2m26s
   fluentd-nlgb6                                   1/1     Running   0          2m30s
   fluentd-snpkt                                   1/1     Running   0          2m28s
   kibana-d6d5668c5-rppqm                          2/2     Running   0          2m39s

Procedure

1. Edit the CatalogSourceConfig (CSC) and Subscription objects to point to the new catalog Namespace and Cource:

   1. From the CLI, get the name of the Elasticsearch CSC.

      $ oc get csc --all-namespaces
      NAMESPACE               NAME                                 STATUS      MESSAGE                                       AGE
      openshift-marketplace   certified-operators                  Succeeded   The object has been successfully reconciled   42m
      openshift-marketplace   community-operators                  Succeeded   The object has been successfully reconciled   42m
      openshift-marketplace   elasticsearch                        Succeeded   The object has been successfully reconciled   27m
      openshift-marketplace   installed-redhat-default             Succeeded   The object has been successfully reconciled   26m
      openshift-marketplace   installed-redhat-openshift-logging   Succeeded   The object has been successfully reconciled   18m
      openshift-marketplace   redhat-operators                     Succeeded   The object has been successfully reconciled   42m

   2. Edit the file as follows:

      $ oc edit csc elasticsearch -n openshift-marketplace
      
      apiVersion: operators.coreos.com/v1
      kind: CatalogSourceConfig
      metadata:
        creationTimestamp: "2020-02-18T15:09:00Z"
        finalizers:
        - finalizer.catalogsourceconfigs.operators.coreos.com
        generation: 3
        name: elasticsearch
        namespace: openshift-marketplace
        resourceVersion: "17694"
        selfLink: /apis/operators.coreos.com/v1/namespaces/openshift-marketplace/catalogsourceconfigs/elasticsearch
        uid: 97c0cd55-5260-11ea-873c-02939b2f528f
      spec:
        csDisplayName: Custom
        csPublisher: Custom
        packages: elasticsearch-operator
        targetNamespace: openshift-operators-redhat
        source: redhat-operators 1

      1

      Change the current value to redhat-operators.

   3. Get the name of the Elasticsearch Subscription object:

      $ oc get sub
      
      NAME                  PACKAGE                  SOURCE          CHANNEL
      elasticsearch-pj7pf   elasticsearch-operator   elasticsearch   preview

   4. Edit the file as follows:

      $ oc edit sub elasticsearch-pj7pf
      
      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        creationTimestamp: "2020-02-17T17:51:18Z"
        generateName: elasticsearch-
        generation: 2
        name: elasticsearch-p5k7n
        namespace: openshift-operators-redhat
        resourceVersion: "38098"
        selfLink: /apis/operators.coreos.com/v1alpha1/namespaces/openshift-operators-redhat/subscriptions/elasticsearch-p5k7n
        uid: 19f6df33-51ae-11ea-82b9-027dfdb65ec2
      spec:
        channel: "4.2"
        installPlanApproval: Automatic
        name: elasticsearch-operator
        source: redhat-operators 1
        sourceNamespace: openshift-marketplace 2
      ....

      1

      Change the current value to redhat-operators.

      2

      Change the current value to openshift-marketplace.

2. Upgrade the Elasticsearch Operator:

   1. From the web console, click Operator Management.
   2. Change the project to all projects.
   3. Click the Elasticsearch Operator, which has the same name as the Elasticsearch subscription.
   4. Click Subscription → Channel.
   5. In the Change Subscription Update Channel window, select 4.2 and click Save.

   6. Wait for a few seconds, then click Operators → Installed Operators.

      The Elasticsearch Operator is shown as 4.2. For example:

      Elasticsearch Operator
      4.2.0-201909201915 provided
      by Red Hat, Inc

3. Upgrade the Cluster Logging Operator:

   1. From the web console, click Operator Management.
   2. Change the project to all projects.
   3. Click the Cluster Logging Operator.
   4. Click Subscription → Channel.
   5. In the Change Subscription Update Channel window, select 4.2 and click Save.

   6. Wait for a few seconds, then click Operators → Installed Operators.

      The Cluster Logging Operator is shown as 4.2. For example:

      Cluster Logging
      4.2.0-201909201915 provided
      by Red Hat, Inc

4. Check the logging components:

   1. Ensure that the Elasticsearch Pods are using a 4.2 image:

      $ oc get pod -o yaml -n openshift-logging --selector component=elasticsearch |grep 'image:'
      
      image: registry.redhat.io/openshift4/ose-logging-elasticsearch5:v4.2.0-201909201915
      image: registry.redhat.io/openshift4/ose-oauth-proxy:v4.2.0-201909201915
      image: registry.redhat.io/openshift4/ose-logging-elasticsearch5:v4.2.0-201909201915
      image: registry.redhat.io/openshift4/ose-oauth-proxy:v4.2.0-201909201915
      image: registry.redhat.io/openshift4/ose-logging-elasticsearch5:v4.2.0-201909201915
      image: registry.redhat.io/openshift4/ose-oauth-proxy:v4.2.0-201909201915
      image: registry.redhat.io/openshift4/ose-logging-elasticsearch5:v4.2.0-201909201915
      image: registry.redhat.io/openshift4/ose-oauth-proxy:v4.2.0-201909201915
      image: registry.redhat.io/openshift4/ose-logging-elasticsearch5:v4.2.0-201909201915
      image: registry.redhat.io/openshift4/ose-oauth-proxy:v4.2.0-201909201915
      image: registry.redhat.io/openshift4/ose-logging-elasticsearch5:v4.2.0-201909201915
      image: registry.redhat.io/openshift4/ose-oauth-proxy:v4.2.0-201909201915

   2. Ensure that all Elasticsearch Pods are in the Ready status:

      $ oc get pod -n openshift-logging --selector component=elasticsearch
      
      NAME                                            READY   STATUS    RESTARTS   AGE
      elasticsearch-cdm-1pbrl44l-1-55b7546f4c-mshhk   2/2     Running   0          31m
      elasticsearch-cdm-1pbrl44l-2-5c6d87589f-gx5hk   2/2     Running   0          30m
      elasticsearch-cdm-1pbrl44l-3-88df5d47-m45jc     2/2     Running   0          29m

   3. Ensure that the Elasticsearch cluster is healthy:

      oc exec -n openshift-logging -c elasticsearch elasticsearch-cdm-1pbrl44l-1-55b7546f4c-mshhk -- es_cluster_health
      
      {
        "cluster_name" : "elasticsearch",
        "status" : "green",
      
      ....

   4. Ensure that the logging collector Pods are using a 4.2 image:

      $ oc get pod -n openshift-logging --selector logging-infra=fluentd -o yaml |grep 'image:'
      
      image: registry.redhat.io/openshift4/ose-logging-fluentd:v4.2.0-201909201915
      image: registry.redhat.io/openshift4/ose-logging-fluentd:v4.2.0-201909201915
      image: registry.redhat.io/openshift4/ose-logging-fluentd:v4.2.0-201909201915
      image: registry.redhat.io/openshift4/ose-logging-fluentd:v4.2.0-201909201915
      image: registry.redhat.io/openshift4/ose-logging-fluentd:v4.2.0-201909201915
      image: registry.redhat.io/openshift4/ose-logging-fluentd:v4.2.0-201909201915
      image: registry.redhat.io/openshift4/ose-logging-fluentd:v4.2.0-201909201915
      image: registry.redhat.io/openshift4/ose-logging-fluentd:v4.2.0-201909201915
      image: registry.redhat.io/openshift4/ose-logging-fluentd:v4.2.0-201909201915
      image: registry.redhat.io/openshift4/ose-logging-fluentd:v4.2.0-201909201915
      image: registry.redhat.io/openshift4/ose-logging-fluentd:v4.2.0-201909201915
      image: registry.redhat.io/openshift4/ose-logging-fluentd:v4.2.0-201909201915

   5. Ensure that the Kibana Pods are using a 4.2 image:

      $ oc get pod -n openshift-logging --selector logging-infra=kibana -o yaml |grep 'image:'
      
      image: registry.redhat.io/openshift4/ose-logging-kibana5:v4.2.0-201909210748
      image: registry.redhat.io/openshift4/ose-oauth-proxy:v4.2.0-201909201915
      image: registry.redhat.io/openshift4/ose-logging-kibana5:v4.2.0-201909210748
      image: registry.redhat.io/openshift4/ose-oauth-proxy:v4.2.0-201909201915

   6. Ensure that the Curator CronJob is using a 4.2 image:

      $ $ oc get CronJob curator -n openshift-logging -o yaml |grep 'image:'
      
      image: registry.redhat.io/openshift4/ose-logging-curator5:v4.2.0-201909201915

Procedure

1. Create a template for the Event Router:

   kind: Template
   apiVersion: v1
   metadata:
     name: eventrouter-template
     annotations:
       description: "A pod forwarding kubernetes events to cluster logging stack."
       tags: "events,EFK,logging,cluster-logging"
   objects:
     - kind: ServiceAccount 1
       apiVersion: v1
       metadata:
         name: eventrouter
         namespace: ${NAMESPACE}
     - kind: ClusterRole 2
       apiVersion: v1
       metadata:
         name: event-reader
       rules:   3
       - apiGroups: [""]
         resources: ["events"]
         verbs: ["get", "watch", "list"]
     - kind: ClusterRoleBinding  4
       apiVersion: v1
       metadata:
         name: event-reader-binding
       subjects:
       - kind: ServiceAccount
         name: eventrouter
         namespace: ${NAMESPACE}
       roleRef:
         kind: ClusterRole
         name: event-reader
     - kind: ConfigMap
       apiVersion: v1
       metadata:
         name: eventrouter
         namespace: ${NAMESPACE}
       data:
         config.json: |-
           {
             "sink": "stdout"
           }
     - kind: Deployment
       apiVersion: apps/v1
       metadata:
         name: eventrouter
         namespace: ${NAMESPACE}
         labels:
           component: eventrouter
           logging-infra: eventrouter
           provider: openshift
       spec:
         selector:
           matchLabels:
             component: eventrouter
             logging-infra: eventrouter
             provider: openshift
         replicas: 1
         template:
           metadata:
             labels:
               component: eventrouter
               logging-infra: eventrouter
               provider: openshift
             name: eventrouter
           spec:
             serviceAccount: eventrouter
             containers:
               - name: kube-eventrouter
                 image: ${IMAGE}
                 imagePullPolicy: IfNotPresent
                 resources:
                   limits:
                     memory: ${MEMORY}
                   requests:
                     cpu: ${CPU}
                     memory: ${MEMORY}
                 volumeMounts:
                 - name: config-volume
                   mountPath: /etc/eventrouter
             volumes:
               - name: config-volume
                 configMap:
                   name: eventrouter
   parameters:
     - name: IMAGE 5
       displayName: Image
       value: "registry.redhat.io/openshift4/ose-logging-eventrouter:latest"
     - name: MEMORY 6
       displayName: Memory
       value: "128Mi"
     - name: CPU  7
       displayName: CPU
       value: "100m"
     - name: NAMESPACE  8
       displayName: Namespace
       value: "openshift-logging"

   1

   Creates a Service Account for the Event Router.

   2

   Creates a cluster role to monitor for events in the cluster.

   3

   Allows the get, watch, and list permissions for the events resource.

   4

   Creates a ClusterRoleBinding to bind the ClusterRole to the ServiceAccount.

   5

   Specify the image version for the Event Router.

   6

   Specify the memory limit for the Event Router pods. Defaults to '128Mi'.

   7

   Specify the minimum amount of CPU to allocate to the Event Router. Defaults to '100m'.

   8

   Specify the namespace where eventrouter is deployed. Defaults to openshift-logging. The value must be the same as specified for the ServiceAccount and ClusterRoleBinding. The project indicates where in Kibana you can locate events:

   • If the event router pod is deployed in a default project, such as kube-* and openshift-*, you can find the events under the .operation index.
   • If the event router pod is deployed in other projects, you can find the event under the index using the project namespace.

2. Use the following command to process and apply the template:

   $ oc process -f <templatefile> | oc apply -f -

   For example:

   $ oc process -f eventrouter.yaml | oc apply -f -
   
   serviceaccount/logging-eventrouter created
   clusterrole.authorization.openshift.io/event-reader created
   clusterrolebinding.authorization.openshift.io/event-reader-binding created
   configmap/logging-eventrouter created
   deployment.apps/logging-eventrouter created

3. Validate that the Event Router installed:

   $ oc get pods --selector  component=eventrouter -o name
   
   pod/logging-eventrouter-d649f97c8-qvv8r

   $ oc logs logging-eventrouter-d649f97c8-qvv8r
   
   {"verb":"ADDED","event":{"metadata":{"name":"elasticsearch-operator.v0.0.1.158f402e25397146","namespace":"openshift-operators","selfLink":"/api/v1/namespaces/openshift-operators/events/elasticsearch-operator.v0.0.1.158f402e25397146","uid":"37b7ff11-4f1a-11e9-a7ad-0271b2ca69f0","resourceVersion":"523264","creationTimestamp":"2019-03-25T16:22:43Z"},"involvedObject":{"kind":"ClusterServiceVersion","namespace":"openshift-operators","name":"elasticsearch-operator.v0.0.1","uid":"27b2ca6d-4f1a-11e9-8fba-0ea949ad61f6","apiVersion":"operators.coreos.com/v1alpha1","resourceVersion":"523096"},"reason":"InstallSucceeded","message":"waiting for install components to report healthy","source":{"component":"operator-lifecycle-manager"},"firstTimestamp":"2019-03-25T16:22:43Z","lastTimestamp":"2019-03-25T16:22:43Z","count":1,"type":"Normal"}}

1. Determine if the log location is a file or CONSOLE (stdout).

   $ oc -n openshift-logging set env daemonset/fluentd --list | grep LOGGING_FILE_PATH

2. Depending on the log location, execute the logging command:

   • If LOGGING_FILE_PATH points to a file, the default, use the logs utility, from the project, where the pod is located, to print out the contents of Fluentd log files:

     $ oc exec <any-fluentd-pod> -- logs 1

     1

     Specify the name of a log collector pod. Note the space before logs.

     For example:

     $ oc exec fluentd-ht42r -n openshift-logging -- logs

   • If you are using LOGGING_FILE_PATH=console, the log collector writes logs to stdout/stderr`. You can retrieve the logs with the oc logs [-f] <pod_name> command, where the -f is optional.

     $ oc logs -f <any-fluentd-pod> -n openshift-logging 1

     1

     Specify the name of a log collector pod. Use the -f option to follow what is being written into the logs.

     For example

     $ oc logs -f fluentd-ht42r -n openshift-logging

     The contents of log files are printed out.

     By default, Fluentd reads logs from the tail, or end, of the log.

1. In the OpenShift Container Platform console, navigate to Workloads → Pods.
2. Select the openshift-logging project from the drop-down menu.
3. Click one of the logging collector pods with the fluentd prefix.
4. Click Logs.

1. In the OpenShift Container Platform console, click Monitoring → Logging.

2. Log in using the same credentials you use to log in to the OpenShift Container Platform console.

   The Kibana interface launches. You can now:

   • Search and browse your data using the Discover page.
   • Chart and map your data using the Visualize page.

   • Create and view custom dashboards using the Dashboard page.

     Use and configuration of the Kibana interface is beyond the scope of this documentation. For more information, on using the interface, see the Kibana documentation.

Procedure

1. Change to the openshift-logging project.

   $ oc project openshift-logging

2. To view the Elasticsearch cluster status:

   1. Get the name of the Elasticsearch instance:

      $ oc get Elasticsearch
      
      NAME            AGE
      elasticsearch   5h9m

   2. Get the Elasticsearch status:

      $ oc get Elasticsearch <Elasticsearch-instance> -o yaml

      For example:

      $ oc get Elasticsearch elasticsearch -n openshift-logging -o yaml

      The output includes information similar to the following:

      status: 1
        cluster: 2
          activePrimaryShards: 30
          activeShards: 60
          initializingShards: 0
          numDataNodes: 3
          numNodes: 3
          pendingTasks: 0
          relocatingShards: 0
          status: green
          unassignedShards: 0
        clusterHealth: ""
        conditions: [] 3
        nodes: 4
        - deploymentName: elasticsearch-cdm-zjf34ved-1
          upgradeStatus: {}
        - deploymentName: elasticsearch-cdm-zjf34ved-2
          upgradeStatus: {}
        - deploymentName: elasticsearch-cdm-zjf34ved-3
          upgradeStatus: {}
        pods: 5
          client:
            failed: []
            notReady: []
            ready:
            - elasticsearch-cdm-zjf34ved-1-6d7fbf844f-sn422
            - elasticsearch-cdm-zjf34ved-2-dfbd988bc-qkzjz
            - elasticsearch-cdm-zjf34ved-3-c8f566f7c-t7zkt
          data:
            failed: []
            notReady: []
            ready:
            - elasticsearch-cdm-zjf34ved-1-6d7fbf844f-sn422
            - elasticsearch-cdm-zjf34ved-2-dfbd988bc-qkzjz
            - elasticsearch-cdm-zjf34ved-3-c8f566f7c-t7zkt
          master:
            failed: []
            notReady: []
            ready:
            - elasticsearch-cdm-zjf34ved-1-6d7fbf844f-sn422
            - elasticsearch-cdm-zjf34ved-2-dfbd988bc-qkzjz
            - elasticsearch-cdm-zjf34ved-3-c8f566f7c-t7zkt
        shardAllocationEnabled: all

      1

      In the output, the cluster status fields appear in the status stanza.

      2

      The status of the Elasticsearch cluster:

      • The number of active primary shards.
      • The number of active shards.
      • The number of shards that are initializing.
      • The number of Elasticsearch data nodes.
      • The total number of Elasticsearch nodes.
      • The number of pending tasks.
      • The Elasticsearch status: green, red, yellow.
      • The number of unassigned shards.

      3

      Any status conditions, if present. The Elasticsearch cluster status indicates the reasons from the scheduler if a pod could not be placed. Any events related to the following conditions are shown:

      • Container Waiting for both the Elasticsearch and proxy containers.
      • Container Terminated for both the Elasticsearch and proxy containers.
      • Pod unschedulable. Also, a condition is shown for a number of issues, see Example condition messages.

      4

      The Elasticsearch nodes in the cluster, with upgradeStatus.

      5

      The Elasticsearch client, data, and master pods in the cluster, listed under 'failed`, notReady or ready state.

Procedure

1. Change to the openshift-logging project.

   $ oc project openshift-logging

2. To view the cluster logging status:

   1. Get the cluster logging status:

      $ oc get clusterlogging instance -o yaml

      The output includes information similar to the following:

      apiVersion: logging.openshift.io/v1
      kind: ClusterLogging
      
      ....
      
      status:  1
        collection:
          logs:
            fluentdStatus:
              daemonSet: fluentd  2
              nodes:
                fluentd-2rhqp: ip-10-0-169-13.ec2.internal
                fluentd-6fgjh: ip-10-0-165-244.ec2.internal
                fluentd-6l2ff: ip-10-0-128-218.ec2.internal
                fluentd-54nx5: ip-10-0-139-30.ec2.internal
                fluentd-flpnn: ip-10-0-147-228.ec2.internal
                fluentd-n2frh: ip-10-0-157-45.ec2.internal
              pods:
                failed: []
                notReady: []
                ready:
                - fluentd-2rhqp
                - fluentd-54nx5
                - fluentd-6fgjh
                - fluentd-6l2ff
                - fluentd-flpnn
                - fluentd-n2frh
        curation:  3
          curatorStatus:
          - cronJobs: curator
            schedules: 30 3 * * *
            suspended: false
        logstore: 4
          elasticsearchStatus:
          - ShardAllocationEnabled:  all
            cluster:
              activePrimaryShards:    5
              activeShards:           5
              initializingShards:     0
              numDataNodes:           1
              numNodes:               1
              pendingTasks:           0
              relocatingShards:       0
              status:                 green
              unassignedShards:       0
            clusterName:             elasticsearch
            nodeConditions:
              elasticsearch-cdm-mkkdys93-1:
            nodeCount:  1
            pods:
              client:
                failed:
                notReady:
                ready:
                - elasticsearch-cdm-mkkdys93-1-7f7c6-mjm7c
              data:
                failed:
                notReady:
                ready:
                - elasticsearch-cdm-mkkdys93-1-7f7c6-mjm7c
              master:
                failed:
                notReady:
                ready:
                - elasticsearch-cdm-mkkdys93-1-7f7c6-mjm7c
      visualization:  5
          kibanaStatus:
          - deployment: kibana
            pods:
              failed: []
              notReady: []
              ready:
              - kibana-7fb4fd4cc9-f2nls
            replicaSets:
            - kibana-7fb4fd4cc9
            replicas: 1

      1

      In the output, the cluster status fields appear in the status stanza.

      2

      Information on the Fluentd pods.

      3

      Information on the Curator pods.

      4

      Information on the Elasticsearch pods, including Elasticsearch cluster health, green, yellow, or red.

      5

      Information on the Kibana pods.

Procedure

1. Change to the openshift-logging project.

   $ oc project openshift-logging

2. View the status of the cluster logging deployment:

   $ oc describe deployment cluster-logging-operator

   The output includes the following status information:

   Name:                   cluster-logging-operator
   
   ....
   
   Conditions:
     Type           Status  Reason
     ----           ------  ------
     Available      True    MinimumReplicasAvailable
     Progressing    True    NewReplicaSetAvailable
   
   ....
   
   Events:
     Type    Reason             Age   From                   Message
     ----    ------             ----  ----                   -------
     Normal  ScalingReplicaSet  62m   deployment-controller  Scaled up replica set cluster-logging-operator-574b8987df to 1----

3. View the status of the cluster logging ReplicaSet:

   1. Get the name of a ReplicaSet:

      $ oc get replicaset
      NAME                                      DESIRED   CURRENT   READY   AGE
      cluster-logging-operator-574b8987df       1         1         1       159m
      elasticsearch-cdm-uhr537yu-1-6869694fb    1         1         1       157m
      elasticsearch-cdm-uhr537yu-2-857b6d676f   1         1         1       156m
      elasticsearch-cdm-uhr537yu-3-5b6fdd8cfd   1         1         1       155m
      kibana-5bd5544f87                         1         1         1       157m

   2. Get the status of the ReplicaSet:

      $ oc describe replicaset cluster-logging-operator-574b8987df

      The output includes the following status information:

      Name:           cluster-logging-operator-574b8987df
      
      ....
      
      Replicas:       1 current / 1 desired
      Pods Status:    1 Running / 0 Waiting / 0 Succeeded / 0 Failed
      
      ....
      
      Events:
        Type    Reason            Age   From                   Message
        ----    ------            ----  ----                   -------
        Normal  SuccessfulCreate  66m   replicaset-controller  Created pod: cluster-logging-operator-574b8987df-qjhqv----

Procedure

1. Edit the Cluster Logging Custom Resource in the openshift-logging project:

   $ oc edit ClusterLogging instance

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogging
   
   ....
   
   spec:
     collection:
       logs:
         fluentd:
           resources: null
         type: fluentd
     curation:
       curator:
         nodeSelector: 1
             node-role.kubernetes.io/infra: ''
         resources: null
         schedule: 30 3 * * *
       type: curator
     logStore:
       elasticsearch:
         nodeCount: 3
         nodeSelector: 2
             node-role.kubernetes.io/infra: ''
         redundancyPolicy: SingleRedundancy
         resources:
           limits:
             cpu: 500m
             memory: 16Gi
           requests:
             cpu: 500m
             memory: 16Gi
         storage: {}
       type: elasticsearch
     managementState: Managed
     visualization:
       kibana:
         nodeSelector: 3
             node-role.kubernetes.io/infra: '' 4
         proxy:
           resources: null
         replicas: 1
         resources: null
       type: kibana
   
   ....

1. Change to the openshift-logging project:

   $ oc project openshift-logging

2. Use the following command to extract the CA certificate from Elasticsearch and write to the admin-ca file:

   $ oc extract secret/elasticsearch --to=. --keys=admin-ca
   
   admin-ca

3. Perform a shard synced flush to ensure there are no pending operations waiting to be written to disk prior to shutting down:

   $ oc exec <any_es_pod_in_the_cluster> -c elasticsearch -- curl -s --cacert /etc/elasticsearch/secret/admin-ca --cert /etc/elasticsearch/secret/admin-cert --key /etc/elasticsearch/secret/admin-key -XPOST 'https://localhost:9200/_flush/synced'

   For example:

   oc exec -c elasticsearch-cdm-5ceex6ts-1-dcd6c4c7c-jpw6 -- curl -s --cacert /etc/elasticsearch/secret/admin-ca --cert /etc/elasticsearch/secret/admin-cert --key /etc/elasticsearch/secret/admin-key -XPOST 'https://localhost:9200/_flush/synced'

4. Prevent shard balancing when purposely bringing down nodes using the OpenShift Container Platform es_util tool:

   $ oc exec <any_es_pod_in_the_cluster> -c elasticsearch -- es_util --query=_cluster/settings -XPUT 'https://localhost:9200/_cluster/settings' -d '{ "transient": { "cluster.routing.allocation.enable" : "none" } }'

   For example:

   $ oc exec elasticsearch-cdm-5ceex6ts-1-dcd6c4c7c-jpw6 -c elasticsearch -- es_util --query=_cluster/settings?pretty=true -XPUT 'https://localhost:9200/_cluster/settings' -d '{ "transient": { "cluster.routing.allocation.enable" : "none" } }'
   
   {
     "acknowledged" : true,
     "persistent" : { },
     "transient" : {
       "cluster" : {
         "routing" : {
           "allocation" : {
             "enable" : "none"
           }
         }
       }
     }

5. Once complete, for each deployment you have for an ES cluster:

   1. By default, the OpenShift Container Platform Elasticsearch cluster blocks rollouts to their nodes. Use the following command to allow rollouts and allow the pod to pick up the changes:

      $ oc rollout resume deployment/<deployment-name>

      For example:

      $ oc rollout resume deployment/elasticsearch-cdm-0-1
      deployment.extensions/elasticsearch-cdm-0-1 resumed

      A new pod is deployed. Once the pod has a ready container, you can move on to the next deployment.

      $ oc get pods | grep elasticsearch-*
      
      NAME                                            READY   STATUS    RESTARTS   AGE
      elasticsearch-cdm-5ceex6ts-1-dcd6c4c7c-jpw6k    2/2     Running   0          22h
      elasticsearch-cdm-5ceex6ts-2-f799564cb-l9mj7    2/2     Running   0          22h
      elasticsearch-cdm-5ceex6ts-3-585968dc68-k7kjr   2/2     Running   0          22h

   2. Once complete, reset the pod to disallow rollouts:

      $ oc rollout pause deployment/<deployment-name>

      For example:

      $ oc rollout pause deployment/elasticsearch-cdm-0-1
      
      deployment.extensions/elasticsearch-cdm-0-1 paused

   3. Check that the Elasticsearch cluster is in green state:

      $ oc exec <any_es_pod_in_the_cluster> -c elasticsearch -- es_util --query=_cluster/health?pretty=true

      Note

      If you performed a rollout on the Elasticsearch pod you used in the previous commands, the pod no longer exists and you need a new pod name here.

      For example:

      $ oc exec elasticsearch-cdm-5ceex6ts-1-dcd6c4c7c-jpw6 -c elasticsearch -- es_util --query=_cluster/health?pretty=true
      
      {
        "cluster_name" : "elasticsearch",
        "status" : "green", 1
        "timed_out" : false,
        "number_of_nodes" : 3,
        "number_of_data_nodes" : 3,
        "active_primary_shards" : 8,
        "active_shards" : 16,
        "relocating_shards" : 0,
        "initializing_shards" : 0,
        "unassigned_shards" : 1,
        "delayed_unassigned_shards" : 0,
        "number_of_pending_tasks" : 0,
        "number_of_in_flight_fetch" : 0,
        "task_max_waiting_in_queue_millis" : 0,
        "active_shards_percent_as_number" : 100.0
      }

      1

      Make sure this parameter is green before proceeding.

6. If you changed the Elasticsearch configuration map, repeat these steps for each Elasticsearch pod.

7. Once all the deployments for the cluster have been rolled out, re-enable shard balancing:

   $ oc exec <any_es_pod_in_the_cluster> -c elasticsearch -- es_util --query=_cluster/settings -XPUT 'https://localhost:9200/_cluster/settings' -d '{ "transient": { "cluster.routing.allocation.enable" : "none" } }'

   For example:

   $ oc exec elasticsearch-cdm-5ceex6ts-1-dcd6c4c7c-jpw6 -c elasticsearch -- es_util --query=_cluster/settings?pretty=true -XPUT 'https://localhost:9200/_cluster/settings' -d '{ "transient": { "cluster.routing.allocation.enable" : "all" } }'
   
   {
     "acknowledged" : true,
     "persistent" : { },
     "transient" : {
       "cluster" : {
         "routing" : {
           "allocation" : {
             "enable" : "all"
           }
         }
       }
     }
   }

1. Run the following command to delete the current OAuthClient:

   $ oc delete oauthclient/kibana-proxy

1. Run the following command to delete the current OAuthClient:

   $ oc delete oauthclient/kibana-proxy

1. Use the following command to remove everything generated during the deployment.

   $ oc delete clusterlogging instance -n openshift-logging

2. Use the following command to remove the Persistent Volume Claims that remain after the Operator instances are deleted:

   $ oc delete pvc --all -n openshift-logging