Logging

1. Navigate to the directory where you want to store the must-gather information.

2. Run the oc adm must-gather command against the logging image:

   $ oc adm must-gather --image=$(oc -n openshift-logging get deployment.apps/cluster-logging-operator -o jsonpath='{.spec.template.spec.containers[?(@.name == "cluster-logging-operator")].image}')

   The must-gather tool creates a new directory that starts with must-gather.local within the current directory. For example: must-gather.local.4157245944708210408.

3. Create a compressed file from the must-gather directory that was just created. For example, on a computer that uses a Linux operating system, run the following command:

   $ tar -cvaf must-gather.tar.gz must-gather.local.4157245944708210408

4. Attach the compressed file to your support case on the Red Hat Customer Portal.

Procedure

1. Change to the openshift-logging project by running the following command:

   $ oc project openshift-logging

2. Get the ClusterLogging instance status by running the following command:

   $ oc get clusterlogging instance -o yaml

   Example output

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogging
   # ...
   status:  1
     collection:
       logs:
         fluentdStatus:
           daemonSet: fluentd  2
           nodes:
             collector-2rhqp: ip-10-0-169-13.ec2.internal
             collector-6fgjh: ip-10-0-165-244.ec2.internal
             collector-6l2ff: ip-10-0-128-218.ec2.internal
             collector-54nx5: ip-10-0-139-30.ec2.internal
             collector-flpnn: ip-10-0-147-228.ec2.internal
             collector-n2frh: ip-10-0-157-45.ec2.internal
           pods:
             failed: []
             notReady: []
             ready:
             - collector-2rhqp
             - collector-54nx5
             - collector-6fgjh
             - collector-6l2ff
             - collector-flpnn
             - collector-n2frh
     logstore: 3
       elasticsearchStatus:
       - ShardAllocationEnabled:  all
         cluster:
           activePrimaryShards:    5
           activeShards:           5
           initializingShards:     0
           numDataNodes:           1
           numNodes:               1
           pendingTasks:           0
           relocatingShards:       0
           status:                 green
           unassignedShards:       0
         clusterName:             elasticsearch
         nodeConditions:
           elasticsearch-cdm-mkkdys93-1:
         nodeCount:  1
         pods:
           client:
             failed:
             notReady:
             ready:
             - elasticsearch-cdm-mkkdys93-1-7f7c6-mjm7c
           data:
             failed:
             notReady:
             ready:
             - elasticsearch-cdm-mkkdys93-1-7f7c6-mjm7c
           master:
             failed:
             notReady:
             ready:
             - elasticsearch-cdm-mkkdys93-1-7f7c6-mjm7c
   visualization:  4
       kibanaStatus:
       - deployment: kibana
         pods:
           failed: []
           notReady: []
           ready:
           - kibana-7fb4fd4cc9-f2nls
         replicaSets:
         - kibana-7fb4fd4cc9
         replicas: 1

   1

   In the output, the cluster status fields appear in the status stanza.

   2

   Information on the Fluentd pods.

   3

   Information on the Elasticsearch pods, including Elasticsearch cluster health, green, yellow, or red.

   4

   Information on the Kibana pods.

Procedure

1. Change to the openshift-logging project.

   $ oc project openshift-logging

2. View the status of logging environment:

   $ oc describe deployment cluster-logging-operator

   Example output

   Name:                   cluster-logging-operator
   
   ....
   
   Conditions:
     Type           Status  Reason
     ----           ------  ------
     Available      True    MinimumReplicasAvailable
     Progressing    True    NewReplicaSetAvailable
   
   ....
   
   Events:
     Type    Reason             Age   From                   Message
     ----    ------             ----  ----                   -------
     Normal  ScalingReplicaSet  62m   deployment-controller  Scaled up replica set cluster-logging-operator-574b8987df to 1----

3. View the status of the logging replica set:

   1. Get the name of a replica set:

      Example output

      $ oc get replicaset

      Example output

      NAME                                      DESIRED   CURRENT   READY   AGE
      cluster-logging-operator-574b8987df       1         1         1       159m
      elasticsearch-cdm-uhr537yu-1-6869694fb    1         1         1       157m
      elasticsearch-cdm-uhr537yu-2-857b6d676f   1         1         1       156m
      elasticsearch-cdm-uhr537yu-3-5b6fdd8cfd   1         1         1       155m
      kibana-5bd5544f87                         1         1         1       157m

   2. Get the status of the replica set:

      $ oc describe replicaset cluster-logging-operator-574b8987df

      Example output

      Name:           cluster-logging-operator-574b8987df
      
      ....
      
      Replicas:       1 current / 1 desired
      Pods Status:    1 Running / 0 Waiting / 0 Succeeded / 0 Failed
      
      ....
      
      Events:
        Type    Reason            Age   From                   Message
        ----    ------            ----  ----                   -------
        Normal  SuccessfulCreate  66m   replicaset-controller  Created pod: cluster-logging-operator-574b8987df-qjhqv----

Procedure

1. Check the Elasticsearch cluster health and verify that the cluster status is red by running the following command:

   $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME -- health

2. List the nodes that have joined the cluster by running the following command:

   $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \
     -- es_util --query=_cat/nodes?v

3. List the Elasticsearch pods and compare them with the nodes in the command output from the previous step, by running the following command:

   $ oc -n openshift-logging get pods -l component=elasticsearch

4. If some of the Elasticsearch nodes have not joined the cluster, perform the following steps.

   1. Confirm that Elasticsearch has an elected master node by running the following command and observing the output:

      $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \
        -- es_util --query=_cat/master?v

   2. Review the pod logs of the elected master node for issues by running the following command and observing the output:

      $ oc logs <elasticsearch_master_pod_name> -c elasticsearch -n openshift-logging

   3. Review the logs of nodes that have not joined the cluster for issues by running the following command and observing the output:

      $ oc logs <elasticsearch_node_name> -c elasticsearch -n openshift-logging

5. If all the nodes have joined the cluster, check if the cluster is in the process of recovering by running the following command and observing the output:

   $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \
     -- es_util --query=_cat/recovery?active_only=true

   If there is no command output, the recovery process might be delayed or stalled by pending tasks.

6. Check if there are pending tasks by running the following command and observing the output:

   $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \
     -- health | grep number_of_pending_tasks

7. If there are pending tasks, monitor their status. If their status changes and indicates that the cluster is recovering, continue waiting. The recovery time varies according to the size of the cluster and other factors. Otherwise, if the status of the pending tasks does not change, this indicates that the recovery has stalled.

8. If it seems like the recovery has stalled, check if the cluster.routing.allocation.enable value is set to none, by running the following command and observing the output:

   $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \
     -- es_util --query=_cluster/settings?pretty

9. If the cluster.routing.allocation.enable value is set to none, set it to all, by running the following command:

   $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \
     -- es_util --query=_cluster/settings?pretty \
     -X PUT -d '{"persistent": {"cluster.routing.allocation.enable":"all"}}'

10. Check if any indices are still red by running the following command and observing the output:

    $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \
      -- es_util --query=_cat/indices?v

11. If any indices are still red, try to clear them by performing the following steps.

    1. Clear the cache by running the following command:

       $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \
         -- es_util --query=<elasticsearch_index_name>/_cache/clear?pretty

    2. Increase the max allocation retries by running the following command:

       $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \
         -- es_util --query=<elasticsearch_index_name>/_settings?pretty \
         -X PUT -d '{"index.allocation.max_retries":10}'

    3. Delete all the scroll items by running the following command:

       $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \
         -- es_util --query=_search/scroll/_all -X DELETE

    4. Increase the timeout by running the following command:

       $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \
         -- es_util --query=<elasticsearch_index_name>/_settings?pretty \
         -X PUT -d '{"index.unassigned.node_left.delayed_timeout":"10m"}'

12. If the preceding steps do not clear the red indices, delete the indices individually.

    1. Identify the red index name by running the following command:

       $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \
         -- es_util --query=_cat/indices?v

    2. Delete the red index by running the following command:

       $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \
         -- es_util --query=<elasticsearch_red_index_name> -X DELETE

13. If there are no red indices and the cluster status is red, check for a continuous heavy processing load on a data node.

    1. Check if the Elasticsearch JVM Heap usage is high by running the following command:

       $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \
         -- es_util --query=_nodes/stats?pretty

       In the command output, review the node_name.jvm.mem.heap_used_percent field to determine the JVM Heap usage.

    2. Check for high CPU utilization. For more information about CPU utilitzation, see the OpenShift Dedicated "Reviewing monitoring dashboards" documentation.

Procedure

1. Identify the node on which Elasticsearch is deployed by running the following command:

   $ oc -n openshift-logging get po -o wide

2. Check if there are unassigned shards by running the following command:

   $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \
     -- es_util --query=_cluster/health?pretty | grep unassigned_shards

3. If there are unassigned shards, check the disk space on each node, by running the following command:

   $ for pod in `oc -n openshift-logging get po -l component=elasticsearch -o jsonpath='{.items[*].metadata.name}'`; \
     do echo $pod; oc -n openshift-logging exec -c elasticsearch $pod \
     -- df -h /elasticsearch/persistent; done

4. In the command output, check the Use column to determine the used disk percentage on that node.

   Example output

   elasticsearch-cdm-kcrsda6l-1-586cc95d4f-h8zq8
   Filesystem      Size  Used Avail Use% Mounted on
   /dev/nvme1n1     19G  522M   19G   3% /elasticsearch/persistent
   elasticsearch-cdm-kcrsda6l-2-5b548fc7b-cwwk7
   Filesystem      Size  Used Avail Use% Mounted on
   /dev/nvme2n1     19G  522M   19G   3% /elasticsearch/persistent
   elasticsearch-cdm-kcrsda6l-3-5dfc884d99-59tjw
   Filesystem      Size  Used Avail Use% Mounted on
   /dev/nvme3n1     19G  528M   19G   3% /elasticsearch/persistent

   If the used disk percentage is above 85%, the node has exceeded the low watermark, and shards can no longer be allocated to this node.

5. To check the current redundancyPolicy, run the following command:

   $ oc -n openshift-logging get es elasticsearch \
     -o jsonpath='{.spec.redundancyPolicy}'

   If you are using a ClusterLogging resource on your cluster, run the following command:

   $ oc -n openshift-logging get cl \
     -o jsonpath='{.items[*].spec.logStore.elasticsearch.redundancyPolicy}'

   If the cluster redundancyPolicy value is higher than the SingleRedundancy value, set it to the SingleRedundancy value and save this change.

6. If the preceding steps do not fix the issue, delete the old indices.

   1. Check the status of all indices on Elasticsearch by running the following command:

      $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME -- indices

   2. Identify an old index that can be deleted.

   3. Delete the index by running the following command:

      $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \
        -- es_util --query=<elasticsearch_index_name> -X DELETE

Procedure

1. Identify the node on which Elasticsearch is deployed by running the following command:

   $ oc -n openshift-logging get po -o wide

2. Check the disk space on each node:

   $ for pod in `oc -n openshift-logging get po -l component=elasticsearch -o jsonpath='{.items[*].metadata.name}'`; \
     do echo $pod; oc -n openshift-logging exec -c elasticsearch $pod \
     -- df -h /elasticsearch/persistent; done

3. Check if the cluster is rebalancing:

   $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \
     -- es_util --query=_cluster/health?pretty | grep relocating_shards

   If the command output shows relocating shards, the high watermark has been exceeded. The default value of the high watermark is 90%.

4. Increase the disk space on all nodes. If increasing the disk space is not possible, try adding a new data node to the cluster, or decrease the total cluster redundancy policy.

5. To check the current redundancyPolicy, run the following command:

   $ oc -n openshift-logging get es elasticsearch \
     -o jsonpath='{.spec.redundancyPolicy}'

   If you are using a ClusterLogging resource on your cluster, run the following command:

   $ oc -n openshift-logging get cl \
     -o jsonpath='{.items[*].spec.logStore.elasticsearch.redundancyPolicy}'

   If the cluster redundancyPolicy value is higher than the SingleRedundancy value, set it to the SingleRedundancy value and save this change.

6. If the preceding steps do not fix the issue, delete the old indices.

   1. Check the status of all indices on Elasticsearch by running the following command:

      $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME -- indices

   2. Identify an old index that can be deleted.

   3. Delete the index by running the following command:

      $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \
        -- es_util --query=<elasticsearch_index_name> -X DELETE

Procedure

1. Get the disk space of the Elasticsearch node:

   $ for pod in `oc -n openshift-logging get po -l component=elasticsearch -o jsonpath='{.items[*].metadata.name}'`; \
     do echo $pod; oc -n openshift-logging exec -c elasticsearch $pod \
     -- df -h /elasticsearch/persistent; done

2. In the command output, check the Avail column to determine the free disk space on that node.

   Example output

   elasticsearch-cdm-kcrsda6l-1-586cc95d4f-h8zq8
   Filesystem      Size  Used Avail Use% Mounted on
   /dev/nvme1n1     19G  522M   19G   3% /elasticsearch/persistent
   elasticsearch-cdm-kcrsda6l-2-5b548fc7b-cwwk7
   Filesystem      Size  Used Avail Use% Mounted on
   /dev/nvme2n1     19G  522M   19G   3% /elasticsearch/persistent
   elasticsearch-cdm-kcrsda6l-3-5dfc884d99-59tjw
   Filesystem      Size  Used Avail Use% Mounted on
   /dev/nvme3n1     19G  528M   19G   3% /elasticsearch/persistent

3. Increase the disk space on all nodes. If increasing the disk space is not possible, try adding a new data node to the cluster, or decrease the total cluster redundancy policy.

4. To check the current redundancyPolicy, run the following command:

   $ oc -n openshift-logging get es elasticsearch \
     -o jsonpath='{.spec.redundancyPolicy}'

   If you are using a ClusterLogging resource on your cluster, run the following command:

   $ oc -n openshift-logging get cl \
     -o jsonpath='{.items[*].spec.logStore.elasticsearch.redundancyPolicy}'

   If the cluster redundancyPolicy value is higher than the SingleRedundancy value, set it to the SingleRedundancy value and save this change.

5. If the preceding steps do not fix the issue, delete the old indices.

   1. Check the status of all indices on Elasticsearch by running the following command:

      $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME -- indices

   2. Identify an old index that can be deleted.

   3. Delete the index by running the following command:

      $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \
        -- es_util --query=<elasticsearch_index_name> -X DELETE

6. Continue freeing up and monitoring the disk space. After the used disk space drops below 90%, unblock writing to this node by running the following command:

   $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \
     -- es_util --query=_all/_settings?pretty \
     -X PUT -d '{"index.blocks.read_only_allow_delete": null}'

Procedure

1. Get the disk space of the Elasticsearch node:

   $ for pod in `oc -n openshift-logging get po -l component=elasticsearch -o jsonpath='{.items[*].metadata.name}'`; \
     do echo $pod; oc -n openshift-logging exec -c elasticsearch $pod \
     -- df -h /elasticsearch/persistent; done

2. In the command output, check the Avail column to determine the free disk space on that node.

   Example output

   elasticsearch-cdm-kcrsda6l-1-586cc95d4f-h8zq8
   Filesystem      Size  Used Avail Use% Mounted on
   /dev/nvme1n1     19G  522M   19G   3% /elasticsearch/persistent
   elasticsearch-cdm-kcrsda6l-2-5b548fc7b-cwwk7
   Filesystem      Size  Used Avail Use% Mounted on
   /dev/nvme2n1     19G  522M   19G   3% /elasticsearch/persistent
   elasticsearch-cdm-kcrsda6l-3-5dfc884d99-59tjw
   Filesystem      Size  Used Avail Use% Mounted on
   /dev/nvme3n1     19G  528M   19G   3% /elasticsearch/persistent

3. Increase the disk space on all nodes. If increasing the disk space is not possible, try adding a new data node to the cluster, or decrease the total cluster redundancy policy.

4. To check the current redundancyPolicy, run the following command:

   $ oc -n openshift-logging get es elasticsearch -o jsonpath='{.spec.redundancyPolicy}'

   If you are using a ClusterLogging resource on your cluster, run the following command:

   $ oc -n openshift-logging get cl \
     -o jsonpath='{.items[*].spec.logStore.elasticsearch.redundancyPolicy}'

   If the cluster redundancyPolicy value is higher than the SingleRedundancy value, set it to the SingleRedundancy value and save this change.

5. If the preceding steps do not fix the issue, delete the old indices.

   1. Check the status of all indices on Elasticsearch by running the following command:

      $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME -- indices

   2. Identify an old index that can be deleted.

   3. Delete the index by running the following command:

      $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \
        -- es_util --query=<elasticsearch_index_name> -X DELETE

Procedure

1. Change to the openshift-logging project by running the following command:

   $ oc project openshift-logging

2. To view the status:

   1. Get the name of the Elasticsearch log store instance by running the following command:

      $ oc get Elasticsearch

      Example output

      NAME            AGE
      elasticsearch   5h9m

   2. Get the Elasticsearch log store status by running the following command:

      $ oc get Elasticsearch <Elasticsearch-instance> -o yaml

      For example:

      $ oc get Elasticsearch elasticsearch -n openshift-logging -o yaml

      The output includes information similar to the following:

      Example output

      status: 1
        cluster: 2
          activePrimaryShards: 30
          activeShards: 60
          initializingShards: 0
          numDataNodes: 3
          numNodes: 3
          pendingTasks: 0
          relocatingShards: 0
          status: green
          unassignedShards: 0
        clusterHealth: ""
        conditions: [] 3
        nodes: 4
        - deploymentName: elasticsearch-cdm-zjf34ved-1
          upgradeStatus: {}
        - deploymentName: elasticsearch-cdm-zjf34ved-2
          upgradeStatus: {}
        - deploymentName: elasticsearch-cdm-zjf34ved-3
          upgradeStatus: {}
        pods: 5
          client:
            failed: []
            notReady: []
            ready:
            - elasticsearch-cdm-zjf34ved-1-6d7fbf844f-sn422
            - elasticsearch-cdm-zjf34ved-2-dfbd988bc-qkzjz
            - elasticsearch-cdm-zjf34ved-3-c8f566f7c-t7zkt
          data:
            failed: []
            notReady: []
            ready:
            - elasticsearch-cdm-zjf34ved-1-6d7fbf844f-sn422
            - elasticsearch-cdm-zjf34ved-2-dfbd988bc-qkzjz
            - elasticsearch-cdm-zjf34ved-3-c8f566f7c-t7zkt
          master:
            failed: []
            notReady: []
            ready:
            - elasticsearch-cdm-zjf34ved-1-6d7fbf844f-sn422
            - elasticsearch-cdm-zjf34ved-2-dfbd988bc-qkzjz
            - elasticsearch-cdm-zjf34ved-3-c8f566f7c-t7zkt
        shardAllocationEnabled: all

      1

      In the output, the cluster status fields appear in the status stanza.

      2

      The status of the Elasticsearch log store:

      • The number of active primary shards.
      • The number of active shards.
      • The number of shards that are initializing.
      • The number of Elasticsearch log store data nodes.
      • The total number of Elasticsearch log store nodes.
      • The number of pending tasks.
      • The Elasticsearch log store status: green, red, yellow.
      • The number of unassigned shards.

      3

      Any status conditions, if present. The Elasticsearch log store status indicates the reasons from the scheduler if a pod could not be placed. Any events related to the following conditions are shown:

      • Container Waiting for both the Elasticsearch log store and proxy containers.
      • Container Terminated for both the Elasticsearch log store and proxy containers.
      • Pod unschedulable. Also, a condition is shown for a number of issues; see Example condition messages.

      4

      The Elasticsearch log store nodes in the cluster, with upgradeStatus.

      5

      The Elasticsearch log store client, data, and master pods in the cluster, listed under failed, notReady, or ready state.

Procedure

1. Modify the ClusterLogging CR visualization spec:

   ClusterLogging CR example

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogging
   metadata:
   # ...
   spec:
   # ...
     visualization:
       type: <visualizer_type> 1
       kibana: 2
         resources: {}
         nodeSelector: {}
         proxy: {}
         replicas: {}
         tolerations: {}
       ocpConsole: 3
         logsLimit: {}
         timeout: {}
   # ...

   1

   The type of visualizer you want to use for your logging. This can be either kibana or ocp-console. The Kibana console is only compatible with deployments that use Elasticsearch log storage, while the OpenShift Dedicated console is only compatible with LokiStack deployments.

   2

   Optional configurations for the Kibana console.

   3

   Optional configurations for the OpenShift Dedicated web console.

2. Apply the ClusterLogging CR by running the following command:

   $ oc apply -f <filename>.yaml

Procedure

1. In the OpenShift Dedicated web console Administrator perspective, navigate to Operators → Installed Operators.
2. Click Red Hat OpenShift Logging. This takes you to the Operator Details page.
3. In the Details page, click Disabled for the Console plugin option.
4. In the Console plugin enablement dialog, select Enable.
5. Click Save.
6. Verify that the Console plugin option now shows Enabled.
7. The web console displays a pop-up window when changes have been applied. The window prompts you to reload the web console. Refresh the browser when you see the pop-up window to apply the changes.

Procedure

1. Ensure that the logging Console Plugin is enabled by running the following command:

   $ oc get consoles.operator.openshift.io cluster -o yaml |grep logging-view-plugin  \
   || oc patch consoles.operator.openshift.io cluster  --type=merge \
   --patch '{ "spec": { "plugins": ["logging-view-plugin"]}}'

2. Add the .metadata.annotations.logging.openshift.io/ocp-console-migration-target: lokistack-dev annotation to the ClusterLogging CR, by running the following command:

   $ oc patch clusterlogging instance --type=merge --patch \
   '{ "metadata": { "annotations": { "logging.openshift.io/ocp-console-migration-target": "lokistack-dev" }}}' \
   -n openshift-logging

   Example output

   clusterlogging.logging.openshift.io/instance patched

1. In the OpenShift Dedicated Red Hat Hybrid Cloud Console, click Observe → Dashboards.

2. On the Dashboards page, select Logging/Elasticsearch Nodes or OpenShift Logging from the Dashboard menu.

   For the Logging/Elasticsearch Nodes dashboard, you can select the Elasticsearch node you want to view and set the data resolution.

   The appropriate dashboard is displayed, showing multiple charts of data.

3. Optional: Select a different time range to display or refresh rate for the data from the Time Range and Refresh Interval menus.

1. In the OpenShift Dedicated console, click the Application Launcher and select Logging.

2. Create your Kibana index patterns by clicking Management → Index Patterns → Create index pattern:

   • Each user must manually create index patterns when logging into Kibana the first time to see logs for their projects. Users must create an index pattern named app and use the @timestamp time field to view their container logs.
   • Each admin user must create index patterns when logged into Kibana the first time for the app, infra, and audit indices using the @timestamp time field.
3. Create Kibana Visualizations from the new index patterns.

1. In the OpenShift Dedicated console, click the Application Launcher and select Logging.

2. Log in using the same credentials you use to log in to the OpenShift Dedicated console.

   The Kibana interface launches.

3. In Kibana, click Discover.

4. Select the index pattern you created from the drop-down menu in the top-left corner: app, audit, or infra.

   The log data displays as time-stamped documents.

5. Expand one of the time-stamped documents.

6. Click the JSON tab to display the log entry for that document.

   Example 7.1. Sample infrastructure log entry in Kibana

   {
     "_index": "infra-000001",
     "_type": "_doc",
     "_id": "YmJmYTBlNDkZTRmLTliMGQtMjE3NmFiOGUyOWM3",
     "_version": 1,
     "_score": null,
     "_source": {
       "docker": {
         "container_id": "f85fa55bbef7bb783f041066be1e7c267a6b88c4603dfce213e32c1"
       },
       "kubernetes": {
         "container_name": "registry-server",
         "namespace_name": "openshift-marketplace",
         "pod_name": "redhat-marketplace-n64gc",
         "container_image": "registry.redhat.io/redhat/redhat-marketplace-index:v4.7",
         "container_image_id": "registry.redhat.io/redhat/redhat-marketplace-index@sha256:65fc0c45aabb95809e376feb065771ecda9e5e59cc8b3024c4545c168f",
         "pod_id": "8f594ea2-c866-4b5c-a1c8-a50756704b2a",
         "host": "ip-10-0-182-28.us-east-2.compute.internal",
         "master_url": "https://kubernetes.default.svc",
         "namespace_id": "3abab127-7669-4eb3-b9ef-44c04ad68d38",
         "namespace_labels": {
           "openshift_io/cluster-monitoring": "true"
         },
         "flat_labels": [
           "catalogsource_operators_coreos_com/update=redhat-marketplace"
         ]
       },
       "message": "time=\"2020-09-23T20:47:03Z\" level=info msg=\"serving registry\" database=/database/index.db port=50051",
       "level": "unknown",
       "hostname": "ip-10-0-182-28.internal",
       "pipeline_metadata": {
         "collector": {
           "ipaddr4": "10.0.182.28",
           "inputname": "fluent-plugin-systemd",
           "name": "fluentd",
           "received_at": "2020-09-23T20:47:15.007583+00:00",
           "version": "1.7.4 1.6.0"
         }
       },
       "@timestamp": "2020-09-23T20:47:03.422465+00:00",
       "viaq_msg_id": "YmJmYTBlNDktMDMGQtMjE3NmFiOGUyOWM3",
       "openshift": {
         "labels": {
           "logging": "infra"
         }
       }
     },
     "fields": {
       "@timestamp": [
         "2020-09-23T20:47:03.422Z"
       ],
       "pipeline_metadata.collector.received_at": [
         "2020-09-23T20:47:15.007Z"
       ]
     },
     "sort": [
       1600894023422
     ]
   }

Procedure

1. Edit the ClusterLogging custom resource (CR) in the openshift-logging project:

   $ oc -n openshift-logging edit ClusterLogging instance

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
     namespace: openshift-logging
   
   ...
   
   spec:
     managementState: "Managed"
     logStore:
       type: "elasticsearch"
       elasticsearch:
         nodeCount: 3
         resources: 1
           limits:
             memory: 16Gi
           requests:
             cpu: 200m
             memory: 16Gi
         storage:
           storageClassName: "gp2"
           size: "200G"
         redundancyPolicy: "SingleRedundancy"
     visualization:
       type: "kibana"
       kibana:
         resources: 2
           limits:
             memory: 1Gi
           requests:
             cpu: 500m
             memory: 1Gi
         proxy:
           resources: 3
             limits:
               memory: 100Mi
             requests:
               cpu: 100m
               memory: 100Mi
         replicas: 2
   collection:
       resources: 4
         limits:
           memory: 736Mi
         requests:
           cpu: 200m
           memory: 736Mi
       type: fluentd

   1

   Specify the CPU and memory limits and requests for the log store as needed. For Elasticsearch, you must adjust both the request value and the limit value.

   2 3

   Specify the CPU and memory limits and requests for the log visualizer as needed.

   4

   Specify the CPU and memory limits and requests for the log collector as needed.

1. Fluentd does not support Elasticsearch 8 in the logging version 5.6.2.
2. Vector supports Syslog in the logging version 5.7 and higher.

Procedure

1. Add the following snippet to your ClusterLogForwarder CR YAML file.

   outputDefaults:
    elasticsearch:
       structuredTypeKey: <log record field>
       structuredTypeName: <name>
   pipelines:
   - inputRefs:
     - application
     outputRefs: default
     parse: json

2. Use structuredTypeKey field to specify one of the log record fields.

3. Use structuredTypeName field to specify a name.

   Important

   To parse JSON logs, you must set both the structuredTypeKey and structuredTypeName fields.

4. For inputRefs, specify which log types to forward by using that pipeline, such as application, infrastructure, or audit.
5. Add the parse: json element to pipelines.

6. Create the CR object:

   $ oc create -f <filename>.yaml

   The Red Hat OpenShift Logging Operator redeploys the collector pods. However, if they do not redeploy, delete the collector pods to force them to redeploy.

   $ oc delete pod --selector logging-infra=collector

Procedure

1. Create or edit a YAML file that defines the ClusterLogForwarder CR object:

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: instance
     namespace: openshift-logging
   spec:
     outputDefaults:
       elasticsearch:
         structuredTypeKey: kubernetes.labels.logFormat 1
         structuredTypeName: nologformat
         enableStructuredContainerLogs: true 2
     pipelines:
     - inputRefs:
       - application
       name: application-logs
       outputRefs:
       - default
       parse: json

   1

   Uses the value of the key-value pair that is formed by the Kubernetes logFormat label.

   2

   Enables multi-container outputs.

2. Create or edit a YAML file that defines the Pod CR object:

   apiVersion: v1
   kind: Pod
   metadata:
     annotations:
       containerType.logging.openshift.io/heavy: heavy 1
       containerType.logging.openshift.io/low: low
   spec:
     containers:
     - name: heavy 2
       image: heavyimage
     - name: low
       image: lowimage

   1

   Format: containerType.logging.openshift.io/<container-name>: <index>

   2

   Annotation names must match container names

Procedure

1. Create a secret using your Google service account key.

   $ oc -n openshift-logging create secret generic gcp-secret --from-file google-application-credentials.json=<your_service_account_key_file.json>

2. Create a ClusterLogForwarder Custom Resource YAML using the template below:

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: <log_forwarder_name> 1
     namespace: <log_forwarder_namespace> 2
   spec:
     serviceAccountName: <service_account_name> 3
     outputs:
       - name: gcp-1
         type: googleCloudLogging
         secret:
           name: gcp-secret
         googleCloudLogging:
           projectId : "openshift-gce-devel" 4
           logId : "app-gcp" 5
     pipelines:
       - name: test-app
         inputRefs: 6
           - application
         outputRefs:
           - gcp-1

   1

   In legacy implementations, the CR name must be instance. In multi log forwarder implementations, you can use any name.

   2

   In legacy implementations, the CR namespace must be openshift-logging. In multi log forwarder implementations, you can use any namespace.

   3

   The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the openshift-logging namespace.

   4

   Set a projectId, folderId, organizationId, or billingAccountId field and its corresponding value, depending on where you want to store your logs in the GCP resource hierarchy.

   5

   Set the value to add to the logName field of the Log Entry.

   6

   Specify which log types to forward by using the pipeline: application, infrastructure, or audit.

Procedure

1. Create a secret using your Base64 encoded Splunk HEC token.

   $ oc -n openshift-logging create secret generic vector-splunk-secret --from-literal hecToken=<HEC_Token>

2. Create or edit the ClusterLogForwarder Custom Resource (CR) using the template below:

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: <log_forwarder_name> 1
     namespace: <log_forwarder_namespace> 2
   spec:
     serviceAccountName: <service_account_name> 3
     outputs:
       - name: splunk-receiver 4
         secret:
           name: vector-splunk-secret 5
         type: splunk 6
         url: <http://your.splunk.hec.url:8088> 7
     pipelines: 8
       - inputRefs:
           - application
           - infrastructure
         name: 9
         outputRefs:
           - splunk-receiver 10

   1

   In legacy implementations, the CR name must be instance. In multi log forwarder implementations, you can use any name.

   2

   In legacy implementations, the CR namespace must be openshift-logging. In multi log forwarder implementations, you can use any namespace.

   3

   The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the openshift-logging namespace.

   4

   Specify a name for the output.

   5

   Specify the name of the secret that contains your HEC token.

   6

   Specify the output type as splunk.

   7

   Specify the URL (including port) of your Splunk HEC.

   8

   Specify which log types to forward by using the pipeline: application, infrastructure, or audit.

   9

   Optional: Specify a name for the pipeline.

   10

   Specify the name of the output to use when forwarding logs with this pipeline.

Procedure

1. Create or edit a YAML file that defines the ClusterLogForwarder CR:

   Example ClusterLogForwarder CR

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: instance 1
     namespace: openshift-logging 2
   spec:
     outputs:
      - name: fluentd-server-secure 3
        type: fluentdForward 4
        url: 'tls://fluentdserver.security.example.com:24224' 5
        secret: 6
           name: fluentd-secret
      - name: fluentd-server-insecure
        type: fluentdForward
        url: 'tcp://fluentdserver.home.example.com:24224'
     inputs: 7
      - name: my-app-logs
        application:
           namespaces:
           - my-project 8
     pipelines:
      - name: forward-to-fluentd-insecure 9
        inputRefs: 10
        - my-app-logs
        outputRefs: 11
        - fluentd-server-insecure
        labels:
          project: "my-project" 12
      - name: forward-to-fluentd-secure 13
        inputRefs:
        - application 14
        - audit
        - infrastructure
        outputRefs:
        - fluentd-server-secure
        - default
        labels:
          clusterId: "C1234"

   1

   The name of the ClusterLogForwarder CR must be instance.

   2

   The namespace for the ClusterLogForwarder CR must be openshift-logging.

   3

   The name of the output.

   4

   The output type: elasticsearch, fluentdForward, syslog, or kafka.

   5

   The URL and port of the external log aggregator as a valid absolute URL. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP address.

   6

   If using a tls prefix, you must specify the name of the secret required by the endpoint for TLS communication. The secret must exist in the openshift-logging project and have tls.crt, tls.key, and ca-bundle.crt keys that each point to the certificates they represent.

   7

   The configuration for an input to filter application logs from the specified projects.

   8

   If no namespace is specified, logs are collected from all namespaces.

   9

   The pipeline configuration directs logs from a named input to a named output. In this example, a pipeline named forward-to-fluentd-insecure forwards logs from an input named my-app-logs to an output named fluentd-server-insecure.

   10

   A list of inputs.

   11

   The name of the output to use.

   12

   Optional: String. One or more labels to add to the logs.

   13

   Configuration for a pipeline to send logs to other log aggregators.

   • Optional: Specify a name for the pipeline.
   • Specify which log types to forward by using the pipeline: application, infrastructure, or audit.
   • Specify the name of the output to use when forwarding logs with this pipeline.
   • Optional: Specify the default output to forward logs to the default log store.
   • Optional: String. One or more labels to add to the logs.

   14

   Note that application logs from all namespaces are collected when using this configuration.

2. Apply the ClusterLogForwarder CR by running the following command:

   $ oc apply -f <filename>.yaml

Procedure

1. Create or edit a YAML file that defines the ClusterLogForwarder CR object. In the file, specify the pod labels using simple equality-based selectors under inputs[].name.application.selector.matchLabels, as shown in the following example.

   Example ClusterLogForwarder CR YAML file

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: <log_forwarder_name> 1
     namespace: <log_forwarder_namespace> 2
   spec:
     pipelines:
       - inputRefs: [ myAppLogData ] 3
         outputRefs: [ default ] 4
     inputs: 5
       - name: myAppLogData
         application:
           selector:
             matchLabels: 6
               environment: production
               app: nginx
           namespaces: 7
           - app1
           - app2
     outputs: 8
       - <output_name>
       ...

   1

   In legacy implementations, the CR name must be instance. In multi log forwarder implementations, you can use any name.

   2

   In legacy implementations, the CR namespace must be openshift-logging. In multi log forwarder implementations, you can use any namespace.

   3

   Specify one or more comma-separated values from inputs[].name.

   4

   Specify one or more comma-separated values from outputs[].

   5

   Define a unique inputs[].name for each application that has a unique set of pod labels.

   6

   Specify the key-value pairs of pod labels whose log data you want to gather. You must specify both a key and value, not just a key. To be selected, the pods must match all the key-value pairs.

   7

   Optional: Specify one or more namespaces.

   8

   Specify one or more outputs to forward your log data to.

2. Optional: To restrict the gathering of log data to specific namespaces, use inputs[].name.application.namespaces, as shown in the preceding example.

3. Optional: You can send log data from additional applications that have different pod labels to the same pipeline.

   1. For each unique combination of pod labels, create an additional inputs[].name section similar to the one shown.
   2. Update the selectors to match the pod labels of this application.

   3. Add the new inputs[].name value to inputRefs. For example:

      - inputRefs: [ myAppLogData, myOtherAppLogData ]

4. Create the CR object:

   $ oc create -f <file-name>.yaml

Procedure

1. Create or edit a YAML file that defines the ClusterLogForwarder CR object:

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: <log_forwarder_name> 1
     namespace: <log_forwarder_namespace> 2
   spec:
     serviceAccountName: <service_account_name> 3
     outputs:
     - name: loki-insecure 4
       type: "loki" 5
       url: http://loki.insecure.com:3100 6
       loki:
         tenantKey: kubernetes.namespace_name
         labelKeys:
         - kubernetes.labels.foo
     - name: loki-secure 7
       type: "loki"
       url: https://loki.secure.com:3100
       secret:
         name: loki-secret 8
       loki:
         tenantKey: kubernetes.namespace_name 9
         labelKeys:
         - kubernetes.labels.foo 10
     pipelines:
     - name: application-logs 11
       inputRefs:  12
       - application
       - audit
       outputRefs: 13
       - loki-secure

   1

   In legacy implementations, the CR name must be instance. In multi log forwarder implementations, you can use any name.

   2

   In legacy implementations, the CR namespace must be openshift-logging. In multi log forwarder implementations, you can use any namespace.

   3

   The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the openshift-logging namespace.

   4

   Specify a name for the output.

   5

   Specify the type as "loki".

   6

   Specify the URL and port of the Loki system as a valid absolute URL. You can use the http (insecure) or https (secure HTTP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP Address. Loki’s default port for HTTP(S) communication is 3100.

   7

   For a secure connection, you can specify an https or http URL that you authenticate by specifying a secret.

   8

   For an https prefix, specify the name of the secret required by the endpoint for TLS communication. The secret must contain a ca-bundle.crt key that points to the certificates it represents. Otherwise, for http and https prefixes, you can specify a secret that contains a username and password. In legacy implementations, the secret must exist in the openshift-logging project. For more information, see the following "Example: Setting a secret that contains a username and password."

   9

   Optional: Specify a metadata key field to generate values for the TenantID field in Loki. For example, setting tenantKey: kubernetes.namespace_name uses the names of the Kubernetes namespaces as values for tenant IDs in Loki. To see which other log record fields you can specify, see the "Log Record Fields" link in the following "Additional resources" section.

   10

   Optional: Specify a list of metadata field keys to replace the default Loki labels. Loki label names must match the regular expression [a-zA-Z_:][a-zA-Z0-9_:]*. Illegal characters in metadata keys are replaced with _ to form the label name. For example, the kubernetes.labels.foo metadata key becomes Loki label kubernetes_labels_foo. If you do not set labelKeys, the default value is: [log_type, kubernetes.namespace_name, kubernetes.pod_name, kubernetes_host]. Keep the set of labels small because Loki limits the size and number of labels allowed. See Configuring Loki, limits_config. You can still query based on any log record field using query filters.

   11

   Optional: Specify a name for the pipeline.

   12

   Specify which log types to forward by using the pipeline: application, infrastructure, or audit.

   13

   Specify the name of the output to use when forwarding logs with this pipeline.

   Note

   Because Loki requires log streams to be correctly ordered by timestamp, labelKeys always includes the kubernetes_host label set, even if you do not specify it. This inclusion ensures that each stream originates from a single host, which prevents timestamps from becoming disordered due to clock differences on different hosts.

2. Apply the ClusterLogForwarder CR object by running the following command:

   $ oc apply -f <filename>.yaml

Procedure

1. Create or edit a YAML file that defines the ClusterLogForwarder CR:

   Example ClusterLogForwarder CR

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: <log_forwarder_name> 1
     namespace: <log_forwarder_namespace> 2
   spec:
     serviceAccountName: <service_account_name> 3
     outputs:
      - name: elasticsearch-example 4
        type: elasticsearch 5
        elasticsearch:
          version: 8 6
        url: http://elasticsearch.example.com:9200 7
        secret:
          name: es-secret 8
     pipelines:
      - name: application-logs 9
        inputRefs: 10
        - application
        - audit
        outputRefs:
        - elasticsearch-example 11
        - default 12
        labels:
          myLabel: "myValue" 13
   # ...

   1

   In legacy implementations, the CR name must be instance. In multi log forwarder implementations, you can use any name.

   2

   In legacy implementations, the CR namespace must be openshift-logging. In multi log forwarder implementations, you can use any namespace.

   3

   The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the openshift-logging namespace.

   4

   Specify a name for the output.

   5

   Specify the elasticsearch type.

   6

   Specify the Elasticsearch version. This can be 6, 7, or 8.

   7

   Specify the URL and port of the external Elasticsearch instance as a valid absolute URL. You can use the http (insecure) or https (secure HTTP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP Address.

   8

   For an https prefix, specify the name of the secret required by the endpoint for TLS communication. The secret must contain a ca-bundle.crt key that points to the certificate it represents. Otherwise, for http and https prefixes, you can specify a secret that contains a username and password. In legacy implementations, the secret must exist in the openshift-logging project. For more information, see the following "Example: Setting a secret that contains a username and password."

   9

   Optional: Specify a name for the pipeline.

   10

   Specify which log types to forward by using the pipeline: application, infrastructure, or audit.

   11

   Specify the name of the output to use when forwarding logs with this pipeline.

   12

   Optional: Specify the default output to send the logs to the internal Elasticsearch instance.

   13

   Optional: String. One or more labels to add to the logs.

2. Apply the ClusterLogForwarder CR:

   $ oc apply -f <filename>.yaml

1. Create a Secret YAML file similar to the following example. Use base64-encoded values for the username and password fields. The secret type is opaque by default.

   apiVersion: v1
   kind: Secret
   metadata:
     name: openshift-test-secret
   data:
     username: <username>
     password: <password>
   # ...

2. Create the secret:

   $ oc create secret -n openshift-logging openshift-test-secret.yaml

3. Specify the name of the secret in the ClusterLogForwarder CR:

   kind: ClusterLogForwarder
   metadata:
     name: instance
     namespace: openshift-logging
   spec:
     outputs:
      - name: elasticsearch
        type: "elasticsearch"
        url: https://elasticsearch.secure.com:9200
        secret:
           name: openshift-test-secret
   # ...

   Note

   In the value of the url field, the prefix can be http or https.

4. Apply the CR object:

   $ oc apply -f <filename>.yaml

Procedure

1. Create or edit a YAML file that defines the ClusterLogForwarder CR object:

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: instance 1
     namespace: openshift-logging 2
   spec:
     outputs:
      - name: fluentd-server-secure 3
        type: fluentdForward 4
        url: 'tls://fluentdserver.security.example.com:24224' 5
        secret: 6
           name: fluentd-secret
      - name: fluentd-server-insecure
        type: fluentdForward
        url: 'tcp://fluentdserver.home.example.com:24224'
     pipelines:
      - name: forward-to-fluentd-secure 7
        inputRefs:  8
        - application
        - audit
        outputRefs:
        - fluentd-server-secure 9
        - default 10
        labels:
          clusterId: "C1234" 11
      - name: forward-to-fluentd-insecure 12
        inputRefs:
        - infrastructure
        outputRefs:
        - fluentd-server-insecure
        labels:
          clusterId: "C1234"

   1

   The name of the ClusterLogForwarder CR must be instance.

   2

   The namespace for the ClusterLogForwarder CR must be openshift-logging.

   3

   Specify a name for the output.

   4

   Specify the fluentdForward type.

   5

   Specify the URL and port of the external Fluentd instance as a valid absolute URL. You can use the tcp (insecure) or tls (secure TCP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP address.

   6

   If you are using a tls prefix, you must specify the name of the secret required by the endpoint for TLS communication. The secret must exist in the openshift-logging project and must contain a ca-bundle.crt key that points to the certificate it represents.

   7

   Optional: Specify a name for the pipeline.

   8

   Specify which log types to forward by using the pipeline: application, infrastructure, or audit.

   9

   Specify the name of the output to use when forwarding logs with this pipeline.

   10

   Optional: Specify the default output to forward logs to the internal Elasticsearch instance.

   11

   Optional: String. One or more labels to add to the logs.

   12

   Optional: Configure multiple outputs to forward logs to other external log aggregators of any supported type:

   • A name to describe the pipeline.
   • The inputRefs is the log type to forward by using the pipeline: application, infrastructure, or audit.
   • The outputRefs is the name of the output to use.
   • Optional: String. One or more labels to add to the logs.

2. Create the CR object:

   $ oc create -f <file-name>.yaml

Procedure

1. Create or edit a YAML file that defines the ClusterLogForwarder CR object:

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: <log_forwarder_name> 1
     namespace: <log_forwarder_namespace> 2
   spec:
     serviceAccountName: <service_account_name> 3
     outputs:
      - name: rsyslog-east 4
        type: syslog 5
        syslog: 6
          facility: local0
          rfc: RFC3164
          payloadKey: message
          severity: informational
        url: 'tls://rsyslogserver.east.example.com:514' 7
        secret: 8
           name: syslog-secret
      - name: rsyslog-west
        type: syslog
        syslog:
         appName: myapp
         facility: user
         msgID: mymsg
         procID: myproc
         rfc: RFC5424
         severity: debug
        url: 'tcp://rsyslogserver.west.example.com:514'
     pipelines:
      - name: syslog-east 9
        inputRefs: 10
        - audit
        - application
        outputRefs: 11
        - rsyslog-east
        - default 12
        labels:
          secure: "true" 13
          syslog: "east"
      - name: syslog-west 14
        inputRefs:
        - infrastructure
        outputRefs:
        - rsyslog-west
        - default
        labels:
          syslog: "west"

   1

   In legacy implementations, the CR name must be instance. In multi log forwarder implementations, you can use any name.

   2

   In legacy implementations, the CR namespace must be openshift-logging. In multi log forwarder implementations, you can use any namespace.

   3

   The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the openshift-logging namespace.

   4

   Specify a name for the output.

   5

   Specify the syslog type.

   6

   Optional: Specify the syslog parameters, listed below.

   7

   Specify the URL and port of the external syslog instance. You can use the udp (insecure), tcp (insecure) or tls (secure TCP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP address.

   8

   If using a tls prefix, you must specify the name of the secret required by the endpoint for TLS communication. The secret must contain a ca-bundle.crt key that points to the certificate it represents. In legacy implementations, the secret must exist in the openshift-logging project.

   9

   Optional: Specify a name for the pipeline.

   10

   Specify which log types to forward by using the pipeline: application, infrastructure, or audit.

   11

   Specify the name of the output to use when forwarding logs with this pipeline.

   12

   Optional: Specify the default output to forward logs to the internal Elasticsearch instance.

   13

   Optional: String. One or more labels to add to the logs. Quote values like "true" so they are recognized as string values, not as a boolean.

   14

   Optional: Configure multiple outputs to forward logs to other external log aggregators of any supported type:

   • A name to describe the pipeline.
   • The inputRefs is the log type to forward by using the pipeline: application, infrastructure, or audit.
   • The outputRefs is the name of the output to use.
   • Optional: String. One or more labels to add to the logs.

2. Create the CR object:

   $ oc create -f <filename>.yaml