OpenShift Container Platform

In the output, the cluster status fields appear in the status stanza.

The status of the Elasticsearch log store:

The number of active primary shards.
The number of active shards.
The number of shards that are initializing.
The number of Elasticsearch log store data nodes.
The total number of Elasticsearch log store nodes.
The number of pending tasks.
The Elasticsearch log store status: green, red, yellow.
The number of unassigned shards.

Any status conditions, if present. The Elasticsearch log store status indicates the reasons from the scheduler if a pod could not be placed. Any events related to the following conditions are shown:

Container Waiting for both the Elasticsearch log store and proxy containers.
Container Terminated for both the Elasticsearch log store and proxy containers.
Pod unschedulable. Also, a condition is shown for a number of issues; see Example condition messages.

The Elasticsearch log store nodes in the cluster, with upgradeStatus.

The Elasticsearch log store client, data, and master pods in the cluster, listed under failed, notReady, or ready state.

3.4.1.1. Example condition messages
Copia collegamento

The following are examples of some condition messages from the Status section of the Elasticsearch instance.

The following status message indicates that a node has exceeded the configured low watermark, and no shard will be allocated to this node.

status:
  nodes:
  - conditions:
    - lastTransitionTime: 2019-03-15T15:57:22Z
      message: Disk storage usage for node is 27.5gb (36.74%). Shards will be not
        be allocated on this node.
      reason: Disk Watermark Low
      status: "True"
      type: NodeStorage
    deploymentName: example-elasticsearch-cdm-0-1
    upgradeStatus: {}

status:
  nodes:
  - conditions:
    - lastTransitionTime: 2019-03-15T15:57:22Z
      message: Disk storage usage for node is 27.5gb (36.74%). Shards will be not
        be allocated on this node.
      reason: Disk Watermark Low
      status: "True"
      type: NodeStorage
    deploymentName: example-elasticsearch-cdm-0-1
    upgradeStatus: {}

Copy to Clipboard

Toggle word wrap

The following status message indicates that a node has exceeded the configured high watermark, and shards will be relocated to other nodes.

status:
  nodes:
  - conditions:
    - lastTransitionTime: 2019-03-15T16:04:45Z
      message: Disk storage usage for node is 27.5gb (36.74%). Shards will be relocated
        from this node.
      reason: Disk Watermark High
      status: "True"
      type: NodeStorage
    deploymentName: example-elasticsearch-cdm-0-1
    upgradeStatus: {}

status:
  nodes:
  - conditions:
    - lastTransitionTime: 2019-03-15T16:04:45Z
      message: Disk storage usage for node is 27.5gb (36.74%). Shards will be relocated
        from this node.
      reason: Disk Watermark High
      status: "True"
      type: NodeStorage
    deploymentName: example-elasticsearch-cdm-0-1
    upgradeStatus: {}

Copy to Clipboard

Toggle word wrap

The following status message indicates that the Elasticsearch log store node selector in the custom resource (CR) does not match any nodes in the cluster:

status:
    nodes:
    - conditions:
      - lastTransitionTime: 2019-04-10T02:26:24Z
        message: '0/8 nodes are available: 8 node(s) didn''t match node selector.'
        reason: Unschedulable
        status: "True"
        type: Unschedulable

status:
    nodes:
    - conditions:
      - lastTransitionTime: 2019-04-10T02:26:24Z
        message: '0/8 nodes are available: 8 node(s) didn''t match node selector.'
        reason: Unschedulable
        status: "True"
        type: Unschedulable

Copy to Clipboard

Toggle word wrap

The following status message indicates that the Elasticsearch log store CR uses a non-existent persistent volume claim (PVC).

status:
   nodes:
   - conditions:
     - last Transition Time:  2019-04-10T05:55:51Z
       message:               pod has unbound immediate PersistentVolumeClaims (repeated 5 times)
       reason:                Unschedulable
       status:                True
       type:                  Unschedulable

status:
   nodes:
   - conditions:
     - last Transition Time:  2019-04-10T05:55:51Z
       message:               pod has unbound immediate PersistentVolumeClaims (repeated 5 times)
       reason:                Unschedulable
       status:                True
       type:                  Unschedulable

Copy to Clipboard

Toggle word wrap

The following status message indicates that your Elasticsearch log store cluster does not have enough nodes to support the redundancy policy.

status:
  clusterHealth: ""
  conditions:
  - lastTransitionTime: 2019-04-17T20:01:31Z
    message: Wrong RedundancyPolicy selected. Choose different RedundancyPolicy or
      add more nodes with data roles
    reason: Invalid Settings
    status: "True"
    type: InvalidRedundancy

status:
  clusterHealth: ""
  conditions:
  - lastTransitionTime: 2019-04-17T20:01:31Z
    message: Wrong RedundancyPolicy selected. Choose different RedundancyPolicy or
      add more nodes with data roles
    reason: Invalid Settings
    status: "True"
    type: InvalidRedundancy

Copy to Clipboard

Toggle word wrap

This status message indicates your cluster has too many control plane nodes:

status:
  clusterHealth: green
  conditions:
    - lastTransitionTime: '2019-04-17T20:12:34Z'
      message: >-
        Invalid master nodes count. Please ensure there are no more than 3 total
        nodes with master roles
      reason: Invalid Settings
      status: 'True'
      type: InvalidMasters

status:
  clusterHealth: green
  conditions:
    - lastTransitionTime: '2019-04-17T20:12:34Z'
      message: >-
        Invalid master nodes count. Please ensure there are no more than 3 total
        nodes with master roles
      reason: Invalid Settings
      status: 'True'
      type: InvalidMasters

Copy to Clipboard

Toggle word wrap

The following status message indicates that Elasticsearch storage does not support the change you tried to make.

For example:

status:
  clusterHealth: green
  conditions:
    - lastTransitionTime: "2021-05-07T01:05:13Z"
      message: Changing the storage structure for a custom resource is not supported
      reason: StorageStructureChangeIgnored
      status: 'True'
      type: StorageStructureChangeIgnored

status:
  clusterHealth: green
  conditions:
    - lastTransitionTime: "2021-05-07T01:05:13Z"
      message: Changing the storage structure for a custom resource is not supported
      reason: StorageStructureChangeIgnored
      status: 'True'
      type: StorageStructureChangeIgnored

Copy to Clipboard

Toggle word wrap

The reason and type fields specify the type of unsupported change:

StorageClassNameChangeIgnored: Unsupported change to the storage class name.
StorageSizeChangeIgnored: Unsupported change the storage size.
StorageStructureChangeIgnored: Unsupported change between ephemeral and persistent storage structures.
Important
If you try to configure the ClusterLogging CR to switch from ephemeral to persistent storage, the OpenShift Elasticsearch Operator creates a persistent volume claim (PVC) but does not create a persistent volume (PV). To clear the StorageStructureChangeIgnored status, you must revert the change to the ClusterLogging CR and delete the PVC.

3.4.2. Viewing the status of the log store components
Copia collegamento

You can view the status for a number of the log store components.

Elasticsearch indices

You can view the status of the Elasticsearch indices.

Get the name of an Elasticsearch pod:

oc get pods --selector component=elasticsearch -o name

$ oc get pods --selector component=elasticsearch -o name

Copy to Clipboard

Toggle word wrap

Example output

pod/elasticsearch-cdm-1godmszn-1-6f8495-vp4lw
pod/elasticsearch-cdm-1godmszn-2-5769cf-9ms2n
pod/elasticsearch-cdm-1godmszn-3-f66f7d-zqkz7

pod/elasticsearch-cdm-1godmszn-1-6f8495-vp4lw
pod/elasticsearch-cdm-1godmszn-2-5769cf-9ms2n
pod/elasticsearch-cdm-1godmszn-3-f66f7d-zqkz7

Copy to Clipboard

Toggle word wrap

Get the status of the indices:

oc exec elasticsearch-cdm-4vjor49p-2-6d4d7db474-q2w7z -- indices

$ oc exec elasticsearch-cdm-4vjor49p-2-6d4d7db474-q2w7z -- indices

Copy to Clipboard

Toggle word wrap

Example output

Defaulting container name to elasticsearch.
Use 'oc describe pod/elasticsearch-cdm-4vjor49p-2-6d4d7db474-q2w7z -n openshift-logging' to see all of the containers in this pod.

green  open   infra-000002                                                     S4QANnf1QP6NgCegfnrnbQ   3   1     119926            0        157             78
green  open   audit-000001                                                     8_EQx77iQCSTzFOXtxRqFw   3   1          0            0          0              0
green  open   .security                                                        iDjscH7aSUGhIdq0LheLBQ   1   1          5            0          0              0
green  open   .kibana_-377444158_kubeadmin                                     yBywZ9GfSrKebz5gWBZbjw   3   1          1            0          0              0
green  open   infra-000001                                                     z6Dpe__ORgiopEpW6Yl44A   3   1     871000            0        874            436
green  open   app-000001                                                       hIrazQCeSISewG3c2VIvsQ   3   1       2453            0          3              1
green  open   .kibana_1                                                        JCitcBMSQxKOvIq6iQW6wg   1   1          0            0          0              0
green  open   .kibana_-1595131456_user1                                        gIYFIEGRRe-ka0W3okS-mQ   3   1          1            0          0              0

Defaulting container name to elasticsearch.
Use 'oc describe pod/elasticsearch-cdm-4vjor49p-2-6d4d7db474-q2w7z -n openshift-logging' to see all of the containers in this pod.

green  open   infra-000002                                                     S4QANnf1QP6NgCegfnrnbQ   3   1     119926            0        157             78
green  open   audit-000001                                                     8_EQx77iQCSTzFOXtxRqFw   3   1          0            0          0              0
green  open   .security                                                        iDjscH7aSUGhIdq0LheLBQ   1   1          5            0          0              0
green  open   .kibana_-377444158_kubeadmin                                     yBywZ9GfSrKebz5gWBZbjw   3   1          1            0          0              0
green  open   infra-000001                                                     z6Dpe__ORgiopEpW6Yl44A   3   1     871000            0        874            436
green  open   app-000001                                                       hIrazQCeSISewG3c2VIvsQ   3   1       2453            0          3              1
green  open   .kibana_1                                                        JCitcBMSQxKOvIq6iQW6wg   1   1          0            0          0              0
green  open   .kibana_-1595131456_user1                                        gIYFIEGRRe-ka0W3okS-mQ   3   1          1            0          0              0

Copy to Clipboard

Toggle word wrap

Log store pods

You can view the status of the pods that host the log store.

Get the name of a pod:

oc get pods --selector component=elasticsearch -o name

$ oc get pods --selector component=elasticsearch -o name

Copy to Clipboard

Toggle word wrap

Example output

pod/elasticsearch-cdm-1godmszn-1-6f8495-vp4lw
pod/elasticsearch-cdm-1godmszn-2-5769cf-9ms2n
pod/elasticsearch-cdm-1godmszn-3-f66f7d-zqkz7

pod/elasticsearch-cdm-1godmszn-1-6f8495-vp4lw
pod/elasticsearch-cdm-1godmszn-2-5769cf-9ms2n
pod/elasticsearch-cdm-1godmszn-3-f66f7d-zqkz7

Copy to Clipboard

Toggle word wrap

Get the status of a pod:

oc describe pod elasticsearch-cdm-1godmszn-1-6f8495-vp4lw

$ oc describe pod elasticsearch-cdm-1godmszn-1-6f8495-vp4lw

Copy to Clipboard

Toggle word wrap

The output includes the following status information:

Example output

....
Status:             Running

....

Containers:
  elasticsearch:
    Container ID:   cri-o://b7d44e0a9ea486e27f47763f5bb4c39dfd2
    State:          Running
      Started:      Mon, 08 Jun 2020 10:17:56 -0400
    Ready:          True
    Restart Count:  0
    Readiness:  exec [/usr/share/elasticsearch/probe/readiness.sh] delay=10s timeout=30s period=5s #success=1 #failure=3

....

  proxy:
    Container ID:  cri-o://3f77032abaddbb1652c116278652908dc01860320b8a4e741d06894b2f8f9aa1
    State:          Running
      Started:      Mon, 08 Jun 2020 10:18:38 -0400
    Ready:          True
    Restart Count:  0

....

Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True

....

Events:          <none>

....
Status:             Running

....

Containers:
  elasticsearch:
    Container ID:   cri-o://b7d44e0a9ea486e27f47763f5bb4c39dfd2
    State:          Running
      Started:      Mon, 08 Jun 2020 10:17:56 -0400
    Ready:          True
    Restart Count:  0
    Readiness:  exec [/usr/share/elasticsearch/probe/readiness.sh] delay=10s timeout=30s period=5s #success=1 #failure=3

....

  proxy:
    Container ID:  cri-o://3f77032abaddbb1652c116278652908dc01860320b8a4e741d06894b2f8f9aa1
    State:          Running
      Started:      Mon, 08 Jun 2020 10:18:38 -0400
    Ready:          True
    Restart Count:  0

....

Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True

....

Events:          <none>

Copy to Clipboard

Toggle word wrap

Log storage pod deployment configuration

You can view the status of the log store deployment configuration.

Get the name of a deployment configuration:

oc get deployment --selector component=elasticsearch -o name

$ oc get deployment --selector component=elasticsearch -o name

Copy to Clipboard

Toggle word wrap

Example output

deployment.extensions/elasticsearch-cdm-1gon-1
deployment.extensions/elasticsearch-cdm-1gon-2
deployment.extensions/elasticsearch-cdm-1gon-3

deployment.extensions/elasticsearch-cdm-1gon-1
deployment.extensions/elasticsearch-cdm-1gon-2
deployment.extensions/elasticsearch-cdm-1gon-3

Copy to Clipboard

Toggle word wrap

Get the deployment configuration status:

oc describe deployment elasticsearch-cdm-1gon-1

$ oc describe deployment elasticsearch-cdm-1gon-1

Copy to Clipboard

Toggle word wrap

The output includes the following status information:

Example output

....
  Containers:
   elasticsearch:
    Image:      registry.redhat.io/openshift-logging/elasticsearch6-rhel8
    Readiness:  exec [/usr/share/elasticsearch/probe/readiness.sh] delay=10s timeout=30s period=5s #success=1 #failure=3

....

Conditions:
  Type           Status   Reason
  ----           ------   ------
  Progressing    Unknown  DeploymentPaused
  Available      True     MinimumReplicasAvailable

....

Events:          <none>

....
  Containers:
   elasticsearch:
    Image:      registry.redhat.io/openshift-logging/elasticsearch6-rhel8
    Readiness:  exec [/usr/share/elasticsearch/probe/readiness.sh] delay=10s timeout=30s period=5s #success=1 #failure=3

....

Conditions:
  Type           Status   Reason
  ----           ------   ------
  Progressing    Unknown  DeploymentPaused
  Available      True     MinimumReplicasAvailable

....

Events:          <none>

Copy to Clipboard

Toggle word wrap

Log store replica set

You can view the status of the log store replica set.

Get the name of a replica set:

oc get replicaSet --selector component=elasticsearch -o name

$ oc get replicaSet --selector component=elasticsearch -o name

replicaset.extensions/elasticsearch-cdm-1gon-1-6f8495
replicaset.extensions/elasticsearch-cdm-1gon-2-5769cf
replicaset.extensions/elasticsearch-cdm-1gon-3-f66f7d

Copy to Clipboard

Toggle word wrap

Get the status of the replica set:

oc describe replicaSet elasticsearch-cdm-1gon-1-6f8495

$ oc describe replicaSet elasticsearch-cdm-1gon-1-6f8495

Copy to Clipboard

Toggle word wrap

The output includes the following status information:

Example output

....
  Containers:
   elasticsearch:
    Image:      registry.redhat.io/openshift-logging/elasticsearch6-rhel8@sha256:4265742c7cdd85359140e2d7d703e4311b6497eec7676957f455d6908e7b1c25
    Readiness:  exec [/usr/share/elasticsearch/probe/readiness.sh] delay=10s timeout=30s period=5s #success=1 #failure=3

....

Events:          <none>

....
  Containers:
   elasticsearch:
    Image:      registry.redhat.io/openshift-logging/elasticsearch6-rhel8@sha256:4265742c7cdd85359140e2d7d703e4311b6497eec7676957f455d6908e7b1c25
    Readiness:  exec [/usr/share/elasticsearch/probe/readiness.sh] delay=10s timeout=30s period=5s #success=1 #failure=3

....

Events:          <none>

Copy to Clipboard

Toggle word wrap

3.4.3. Elasticsearch cluster status
Copia collegamento

A dashboard in the Observe section of the OpenShift Container Platform web console displays the status of the Elasticsearch cluster.

To get the status of the OpenShift Elasticsearch cluster, visit the dashboard in the Observe section of the OpenShift Container Platform web console at <cluster_url>/monitoring/dashboards/grafana-dashboard-cluster-logging.

Elasticsearch status fields

eo_elasticsearch_cr_cluster_management_state

Shows whether the Elasticsearch cluster is in a managed or unmanaged state. For example:

eo_elasticsearch_cr_cluster_management_state{state="managed"} 1
eo_elasticsearch_cr_cluster_management_state{state="unmanaged"} 0

eo_elasticsearch_cr_cluster_management_state{state="managed"} 1
eo_elasticsearch_cr_cluster_management_state{state="unmanaged"} 0

Copy to Clipboard

Toggle word wrap

eo_elasticsearch_cr_restart_total

Shows the number of times the Elasticsearch nodes have restarted for certificate restarts, rolling restarts, or scheduled restarts. For example:

eo_elasticsearch_cr_restart_total{reason="cert_restart"} 1
eo_elasticsearch_cr_restart_total{reason="rolling_restart"} 1
eo_elasticsearch_cr_restart_total{reason="scheduled_restart"} 3

eo_elasticsearch_cr_restart_total{reason="cert_restart"} 1
eo_elasticsearch_cr_restart_total{reason="rolling_restart"} 1
eo_elasticsearch_cr_restart_total{reason="scheduled_restart"} 3

Copy to Clipboard

Toggle word wrap

es_index_namespaces_total

Shows the total number of Elasticsearch index namespaces. For example:

Total number of Namespaces.
es_index_namespaces_total 5

Total number of Namespaces.
es_index_namespaces_total 5

Copy to Clipboard

Toggle word wrap

es_index_document_count

Shows the number of records for each namespace. For example:

es_index_document_count{namespace="namespace_1"} 25
es_index_document_count{namespace="namespace_2"} 10
es_index_document_count{namespace="namespace_3"} 5

es_index_document_count{namespace="namespace_1"} 25
es_index_document_count{namespace="namespace_2"} 10
es_index_document_count{namespace="namespace_3"} 5

Copy to Clipboard

Toggle word wrap

The "Secret Elasticsearch fields are either missing or empty" message

If Elasticsearch is missing the admin-cert, admin-key, logging-es.crt, or logging-es.key files, the dashboard shows a status message similar to the following example:

message": "Secret \"elasticsearch\" fields are either missing or empty: [admin-cert, admin-key, logging-es.crt, logging-es.key]",
"reason": "Missing Required Secrets",

message": "Secret \"elasticsearch\" fields are either missing or empty: [admin-cert, admin-key, logging-es.crt, logging-es.key]",
"reason": "Missing Required Secrets",

Copy to Clipboard

Toggle word wrap

Chapter 4. About Logging
Copia collegamento

As a cluster administrator, you can deploy logging on an OpenShift Container Platform cluster, and use it to collect and aggregate node system audit logs, application container logs, and infrastructure logs. You can forward logs to your chosen log outputs, including on-cluster, Red Hat managed log storage. You can also visualize your log data in the OpenShift Container Platform web console, or the Kibana web console, depending on your deployed log storage solution.

Note

The Kibana web console is now deprecated is planned to be removed in a future logging release.

OpenShift Container Platform cluster administrators can deploy logging by using Operators. For information, see Installing logging.

The Operators are responsible for deploying, upgrading, and maintaining logging. After the Operators are installed, you can create a ClusterLogging custom resource (CR) to schedule logging pods and other resources necessary to support logging. You can also create a ClusterLogForwarder CR to specify which logs are collected, how they are transformed, and where they are forwarded to.

Note

Because the internal OpenShift Container Platform Elasticsearch log store does not provide secure storage for audit logs, audit logs are not stored in the internal Elasticsearch instance by default. If you want to send the audit logs to the default internal Elasticsearch log store, for example to view the audit logs in Kibana, you must use the Log Forwarding API as described in Forward audit logs to the log store.

4.1. Logging architecture
Copia collegamento

The major components of the logging are:

Collector: The collector is a daemonset that deploys pods to each OpenShift Container Platform node. It collects log data from each node, transforms the data, and forwards it to configured outputs. You can use the Vector collector or the legacy Fluentd collector.
Note
Fluentd is deprecated and is planned to be removed in a future release. Red Hat provides bug fixes and support for this feature during the current release lifecycle, but this feature no longer receives enhancements. As an alternative to Fluentd, you can use Vector instead.
Log store: The log store stores log data for analysis and is the default output for the log forwarder. You can use the default LokiStack log store, the legacy Elasticsearch log store, or forward logs to additional external log stores.
Note
The Logging 5.9 release does not contain an updated version of the OpenShift Elasticsearch Operator. If you currently use the OpenShift Elasticsearch Operator released with Logging 5.8, it will continue to work with Logging until the EOL of Logging 5.8. As an alternative to using the OpenShift Elasticsearch Operator to manage the default log storage, you can use the Loki Operator. For more information on the Logging lifecycle dates, see Platform Agnostic Operators.
Visualization: You can use a UI component to view a visual representation of your log data. The UI provides a graphical interface to search, query, and view stored logs. The OpenShift Container Platform web console UI is provided by enabling the OpenShift Container Platform console plugin.
Note
The Kibana web console is now deprecated is planned to be removed in a future logging release.

Logging collects container logs and node logs. These are categorized into types:

Application logs: Container logs generated by user applications running in the cluster, except infrastructure container applications.
Infrastructure logs: Container logs generated by infrastructure namespaces: openshift*, kube*, or default, as well as journald messages from nodes.
Audit logs: Logs generated by auditd, the node audit system, which are stored in the /var/log/audit/audit.log file, and logs from the auditd, kube-apiserver, openshift-apiserver services, as well as the ovn project if enabled.

4.2. About deploying logging
Copia collegamento

Administrators can deploy the logging by using the OpenShift Container Platform web console or the OpenShift CLI (oc) to install the logging Operators. The Operators are responsible for deploying, upgrading, and maintaining the logging.

Administrators and application developers can view the logs of the projects for which they have view access.

4.2.1. Logging custom resources
Copia collegamento

You can configure your logging deployment with custom resource (CR) YAML files implemented by each Operator.

Red Hat OpenShift Logging Operator:

ClusterLogging (CL) - After the Operators are installed, you create a ClusterLogging custom resource (CR) to schedule logging pods and other resources necessary to support the logging. The ClusterLogging CR deploys the collector and forwarder, which currently are both implemented by a daemonset running on each node. The Red Hat OpenShift Logging Operator watches the ClusterLogging CR and adjusts the logging deployment accordingly.
ClusterLogForwarder (CLF) - Generates collector configuration to forward logs per user configuration.

Loki Operator:

LokiStack - Controls the Loki cluster as log store and the web proxy with OpenShift Container Platform authentication integration to enforce multi-tenancy.

OpenShift Elasticsearch Operator:

Note

These CRs are generated and managed by the OpenShift Elasticsearch Operator. Manual changes cannot be made without being overwritten by the Operator.

ElasticSearch - Configure and deploy an Elasticsearch instance as the default log store.
Kibana - Configure and deploy Kibana instance to search, query and view logs.

4.2.2. About JSON OpenShift Container Platform Logging
Copia collegamento

You can use JSON logging to configure the Log Forwarding API to parse JSON strings into a structured object. You can perform the following tasks:

Parse JSON logs
Configure JSON log data for Elasticsearch
Forward JSON logs to the Elasticsearch log store

4.2.3. About collecting and storing Kubernetes events
Copia collegamento

The OpenShift Container Platform Event Router is a pod that watches Kubernetes events and logs them for collection by OpenShift Container Platform Logging. You must manually deploy the Event Router.

For information, see About collecting and storing Kubernetes events.

4.2.4. About troubleshooting OpenShift Container Platform Logging
Copia collegamento

You can troubleshoot the logging issues by performing the following tasks:

Viewing logging status
Viewing the status of the log store
Understanding logging alerts
Collecting logging data for Red Hat Support
Troubleshooting for critical alerts

4.2.5. About exporting fields
Copia collegamento

The logging system exports fields. Exported fields are present in the log records and are available for searching from Elasticsearch and Kibana.

For information, see About exporting fields.

4.2.6. About event routing
Copia collegamento

The Event Router is a pod that watches OpenShift Container Platform events so they can be collected by logging. The Event Router collects events from all projects and writes them to STDOUT. Fluentd collects those events and forwards them into the OpenShift Container Platform Elasticsearch instance. Elasticsearch indexes the events to the infra index.

You must manually deploy the Event Router.

For information, see Collecting and storing Kubernetes events.

Chapter 5. Installing Logging
Copia collegamento

You can deploy logging by installing the Red Hat OpenShift Logging Operator. The Red Hat OpenShift Logging Operator creates and manages the components of the logging stack.

Note

Important

For new installations, use Vector and LokiStack. Elasticsearch and Fluentd are deprecated and are planned to be removed in a future release.

5.1. Installing the Red Hat OpenShift Logging Operator by using the web console
Copia collegamento

You can install the Red Hat OpenShift Logging Operator by using the OpenShift Container Platform web console.

Prerequisites

You have administrator permissions.
You have access to the OpenShift Container Platform web console.

Procedure

In the OpenShift Container Platform web console, click Operators → OperatorHub.
Type OpenShift Logging in the Filter by keyword box.
Choose Red Hat OpenShift Logging from the list of available Operators, and click Install.
Ensure that A specific namespace on the cluster is selected under Installation mode.
Ensure that Operator recommended namespace is openshift-logging under Installed Namespace.
Select Enable operator recommended cluster monitoring on this namespace.
This option sets the openshift.io/cluster-monitoring: "true" label in the Namespace object. You must select this option to ensure that cluster monitoring scrapes the openshift-logging namespace.
Select stable-5.y as the Update channel.
Note
The stable channel only provides updates to the most recent release of logging. To continue receiving updates for prior releases, you must change your subscription channel to stable-x.y, where x.y represents the major and minor version of logging you have installed. For example, stable-5.7.
Select an Update approval.
- The Automatic strategy allows Operator Lifecycle Manager (OLM) to automatically update the Operator when a new version is available.
- The Manual strategy requires a user with appropriate credentials to approve the Operator update.
Select Enable or Disable for the Console plugin.
Click Install.

Verification

Verify that the Red Hat OpenShift Logging Operator is installed by switching to the Operators → Installed Operators page.
In the Status column, verify that you see green checkmarks with InstallSucceeded and the text Up to date.

Important

An Operator might display a Failed status before the installation finishes. If the Operator install completes with an InstallSucceeded message, refresh the page.

If the Operator does not show as installed, choose one of the following troubleshooting options:

Go to the Operators → Installed Operators page, and inspect the Status column for any errors or failures.
Go to the Workloads → Pods page, and check the logs in any pods in the openshift-logging project that are reporting issues.

5.2. Creating a ClusterLogging object by using the web console
Copia collegamento

After you have installed the logging Operators, you must create a ClusterLogging custom resource to configure log storage, visualization, and the log collector for your cluster.

Prerequisites

You have installed the Red Hat OpenShift Logging Operator.
You have access to the OpenShift Container Platform web console Administrator perspective.

Procedure

Navigate to the Custom Resource Definitions page.
On the Custom Resource Definitions page, click ClusterLogging.
On the Custom Resource Definition details page, select View Instances from the Actions menu.
On the ClusterLoggings page, click Create ClusterLogging.
In the collection section, select a Collector Implementation.
Note
Fluentd is deprecated and is planned to be removed in a future release. Red Hat provides bug fixes and support for this feature during the current release lifecycle, but this feature no longer receives enhancements. As an alternative to Fluentd, you can use Vector instead.
In the logStore section, select a type.
Note
The Logging 5.9 release does not contain an updated version of the OpenShift Elasticsearch Operator. If you currently use the OpenShift Elasticsearch Operator released with Logging 5.8, it will continue to work with Logging until the EOL of Logging 5.8. As an alternative to using the OpenShift Elasticsearch Operator to manage the default log storage, you can use the Loki Operator. For more information on the Logging lifecycle dates, see Platform Agnostic Operators.
Click Create.

5.3. Installing the Red Hat OpenShift Logging Operator by using the CLI
Copia collegamento

You can use the OpenShift CLI (oc) to install the Red Hat OpenShift Logging Operator.

Prerequisites

You have administrator permissions.
You have installed the OpenShift CLI (oc).

Procedure

Create a Namespace object as a YAML file:

Example Namespace object

apiVersion: v1
kind: Namespace
metadata:
  name: <name> 
  annotations:
    openshift.io/node-selector: ""
  labels:
    openshift.io/cluster-monitoring: "true"

apiVersion: v1
kind: Namespace
metadata:
  name: <name>


  annotations:
    openshift.io/node-selector: ""
  labels:
    openshift.io/cluster-monitoring: "true"

Copy to Clipboard

Toggle word wrap

1: You must specify openshift-logging as the name of the namespace for logging versions 5.7 and earlier versions. For logging 5.8 and later versions, you can use any name.

Apply the Namespace object by running the following command:
```
oc apply -f <filename>.yaml
```
```
$ oc apply -f <filename>.yaml
```
Copy to Clipboard Toggle word wrap
Create an OperatorGroup object as a YAML file:
Example OperatorGroup object
```
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: cluster-logging
  namespace: openshift-logging 
spec:
  targetNamespaces: [ ] 
```
```
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: cluster-logging
  namespace: openshift-logging 
```
1
```
spec:
  targetNamespaces: [ ] 
```
2
Copy to Clipboard Toggle word wrap
1
Ensure that the namespace field is set to openshift-logging.
2
For logging versions 5.7 and earlier, set targetNamespaces to openshift-logging. For logging versions 5.8 and later, the default installation mode is all namespaces on the cluster, or specify a list of namespaces separated by a comma.
Apply the OperatorGroup object by running the following command:
```
oc apply -f <filename>.yaml
```
```
$ oc apply -f <filename>.yaml
```
Copy to Clipboard Toggle word wrap
Create a Subscription object to subscribe the namespace to the Red Hat OpenShift Logging Operator:
Example Subscription object
```
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: cluster-logging
  namespace: openshift-logging 
spec:
  channel: stable 
  name: cluster-logging
  source: redhat-operators 
  sourceNamespace: openshift-marketplace
```
```
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: cluster-logging
  namespace: openshift-logging 
```
1
```
spec:
  channel: stable 
```
2
```
  name: cluster-logging
  source: redhat-operators 
```
3
```
  sourceNamespace: openshift-marketplace
```
Copy to Clipboard Toggle word wrap
1
You must specify the openshift-logging namespace for logging versions 5.7 and older. For logging 5.8 and later versions, you can use any namespace.
2
Specify stable or stable-x.y as the channel.
3
Specify redhat-operators. If your OpenShift Container Platform cluster is installed on a restricted network, also known as a disconnected cluster, specify the name of the CatalogSource object you created when you configured the Operator Lifecycle Manager (OLM).
Apply the subscription by running the following command:
```
oc apply -f <filename>.yaml
```
```
$ oc apply -f <filename>.yaml
```
Copy to Clipboard Toggle word wrap
The Red Hat OpenShift Logging Operator is installed to the openshift-logging namespace.

Verification

Run the following command:
```
oc get csv -n <namespace>
```
```
$ oc get csv -n <namespace>
```
Copy to Clipboard Toggle word wrap

Observe the output and confirm that the Red Hat OpenShift Logging Operator exists in the namespace:

Example output

NAMESPACE                                               NAME                                         DISPLAY                  VERSION               REPLACES   PHASE
...
openshift-logging                                       clusterlogging.5.8.0-202007012112.p0         OpenShift Logging          5.8.0-202007012112.p0              Succeeded
...

NAMESPACE                                               NAME                                         DISPLAY                  VERSION               REPLACES   PHASE
...
openshift-logging                                       clusterlogging.5.8.0-202007012112.p0         OpenShift Logging          5.8.0-202007012112.p0              Succeeded
...

Copy to Clipboard

Toggle word wrap

5.4. Creating a ClusterLogging object by using the CLI
Copia collegamento

This default logging configuration supports a wide array of environments. Review the topics on tuning and configuring components for information about modifications you can make.

Prerequisites

You have installed the Red Hat OpenShift Logging Operator.
You have installed the OpenShift Elasticsearch Operator for your log store.
You have installed the OpenShift CLI (oc).

Procedure

Create a ClusterLogging object as a YAML file:
Example ClusterLogging object
```
apiVersion: logging.openshift.io/v1
kind: ClusterLogging
metadata:
  name: instance 
  namespace: openshift-logging
spec:
  managementState: Managed 
  logStore:
    type: elasticsearch 
    retentionPolicy: 
      application:
        maxAge: 1d
      infra:
        maxAge: 7d
      audit:
        maxAge: 7d
    elasticsearch:
      nodeCount: 3 
      storage:
        storageClassName: <storage_class_name> 
        size: 200G
      resources: 
          limits:
            memory: 16Gi
          requests:
            memory: 16Gi
      proxy: 
        resources:
          limits:
            memory: 256Mi
          requests:
            memory: 256Mi
      redundancyPolicy: SingleRedundancy
  visualization:
    type: kibana 
    kibana:
      replicas: 1
  collection:
    type: fluentd 
    fluentd: {}
```
```
apiVersion: logging.openshift.io/v1
kind: ClusterLogging
metadata:
  name: instance 
```
1
```
  namespace: openshift-logging
spec:
  managementState: Managed 
```
2
```
  logStore:
    type: elasticsearch 
```
3
```
    retentionPolicy: 
```
4
```
      application:
        maxAge: 1d
      infra:
        maxAge: 7d
      audit:
        maxAge: 7d
    elasticsearch:
      nodeCount: 3 
```
5
```
      storage:
        storageClassName: <storage_class_name> 
```
6
```
        size: 200G
      resources: 
```
7
```
          limits:
            memory: 16Gi
          requests:
            memory: 16Gi
      proxy: 
```
8
```
        resources:
          limits:
            memory: 256Mi
          requests:
            memory: 256Mi
      redundancyPolicy: SingleRedundancy
  visualization:
    type: kibana 
```
9
```
    kibana:
      replicas: 1
  collection:
    type: fluentd 
```
10
```
    fluentd: {}
```
Copy to Clipboard Toggle word wrap
1
The name must be instance.
2
The OpenShift Logging management state. In some cases, if you change the OpenShift Logging defaults, you must set this to Unmanaged. However, an unmanaged deployment does not receive updates until OpenShift Logging is placed back into a managed state.
3
Settings for configuring Elasticsearch. Using the CR, you can configure shard replication policy and persistent storage.
4
Specify the length of time that Elasticsearch should retain each log source. Enter an integer and a time designation: weeks(w), hours(h/H), minutes(m) and seconds(s). For example, 7d for seven days. Logs older than the maxAge are deleted. You must specify a retention policy for each log source or the Elasticsearch indices will not be created for that source.
5
Specify the number of Elasticsearch nodes. See the note that follows this list.
6
Enter the name of an existing storage class for Elasticsearch storage. For best performance, specify a storage class that allocates block storage. If you do not specify a storage class, OpenShift Logging uses ephemeral storage.
7
Specify the CPU and memory requests for Elasticsearch as needed. If you leave these values blank, the OpenShift Elasticsearch Operator sets default values that should be sufficient for most deployments. The default values are 16Gi for the memory request and 1 for the CPU request.
8
Specify the CPU and memory requests for the Elasticsearch proxy as needed. If you leave these values blank, the OpenShift Elasticsearch Operator sets default values that should be sufficient for most deployments. The default values are 256Mi for the memory request and 100m for the CPU request.
9
Settings for configuring Kibana. Using the CR, you can scale Kibana for redundancy and configure the CPU and memory for your Kibana nodes. For more information, see Configuring the log visualizer.
10
Settings for configuring Fluentd. Using the CR, you can configure Fluentd CPU and memory limits. For more information, see "Configuring Fluentd".
Note
The maximum number of Elasticsearch control plane nodes is three. If you specify a nodeCount greater than 3, OpenShift Container Platform creates three Elasticsearch nodes that are Master-eligible nodes, with the master, client, and data roles. The additional Elasticsearch nodes are created as data-only nodes, using client and data roles. Control plane nodes perform cluster-wide actions such as creating or deleting an index, shard allocation, and tracking nodes. Data nodes hold the shards and perform data-related operations such as CRUD, search, and aggregations. Data-related operations are I/O-, memory-, and CPU-intensive. It is important to monitor these resources and to add more Data nodes if the current nodes are overloaded.
For example, if nodeCount=4, the following nodes are created:
$ oc get deployment
Copy to Clipboard Toggle word wrap
Example output
NAME READY UP-TO-DATE AVAILABLE AGE cluster-logging-operator 1/1 1 1 18h elasticsearch-cd-x6kdekli-1 1/1 1 1 6m54s elasticsearch-cdm-x6kdekli-1 1/1 1 1 18h elasticsearch-cdm-x6kdekli-2 1/1 1 1 6m49s elasticsearch-cdm-x6kdekli-3 1/1 1 1 6m44s

Copy to Clipboard Toggle word wrap
The number of primary shards for the index templates is equal to the number of Elasticsearch data nodes.

Verification

You can verify the installation by listing the pods in the openshift-logging project.

List the pods by running the following command:

oc get pods -n openshift-logging

$ oc get pods -n openshift-logging

Copy to Clipboard

Toggle word wrap

Observe the pods for components of the logging, similar to the following list:

Example output

NAME                                            READY   STATUS    RESTARTS   AGE
cluster-logging-operator-66f77ffccb-ppzbg       1/1     Running   0          7m
elasticsearch-cdm-ftuhduuw-1-ffc4b9566-q6bhp    2/2     Running   0          2m40s
elasticsearch-cdm-ftuhduuw-2-7b4994dbfc-rd2gc   2/2     Running   0          2m36s
elasticsearch-cdm-ftuhduuw-3-84b5ff7ff8-gqnm2   2/2     Running   0          2m4s
collector-587vb                                   1/1     Running   0          2m26s
collector-7mpb9                                   1/1     Running   0          2m30s
collector-flm6j                                   1/1     Running   0          2m33s
collector-gn4rn                                   1/1     Running   0          2m26s
collector-nlgb6                                   1/1     Running   0          2m30s
collector-snpkt                                   1/1     Running   0          2m28s
kibana-d6d5668c5-rppqm                          2/2     Running   0          2m39s

NAME                                            READY   STATUS    RESTARTS   AGE
cluster-logging-operator-66f77ffccb-ppzbg       1/1     Running   0          7m
elasticsearch-cdm-ftuhduuw-1-ffc4b9566-q6bhp    2/2     Running   0          2m40s
elasticsearch-cdm-ftuhduuw-2-7b4994dbfc-rd2gc   2/2     Running   0          2m36s
elasticsearch-cdm-ftuhduuw-3-84b5ff7ff8-gqnm2   2/2     Running   0          2m4s
collector-587vb                                   1/1     Running   0          2m26s
collector-7mpb9                                   1/1     Running   0          2m30s
collector-flm6j                                   1/1     Running   0          2m33s
collector-gn4rn                                   1/1     Running   0          2m26s
collector-nlgb6                                   1/1     Running   0          2m30s
collector-snpkt                                   1/1     Running   0          2m28s
kibana-d6d5668c5-rppqm                          2/2     Running   0          2m39s

Copy to Clipboard

Toggle word wrap

5.5. Postinstallation tasks
Copia collegamento

After you have installed the Red Hat OpenShift Logging Operator, you can configure your deployment by creating and modifying a ClusterLogging custom resource (CR).

Tip

If you are not using the Elasticsearch log store, you can remove the internal Elasticsearch logStore and Kibana visualization components from the ClusterLogging custom resource (CR). Removing these components is optional but saves resources. See Removing unused components if you do not use the Elasticsearch log store.

5.5.1. About the ClusterLogging custom resource
Copia collegamento

To make changes to your logging environment, create and modify the ClusterLogging custom resource (CR).

Sample ClusterLogging custom resource (CR)

apiVersion: logging.openshift.io/v1
kind: ClusterLogging
metadata:
  name: instance 
  namespace: openshift-logging 
spec:
  managementState: Managed 
# ...

apiVersion: logging.openshift.io/v1
kind: ClusterLogging
metadata:
  name: instance


  namespace: openshift-logging


spec:
  managementState: Managed


# ...

Copy to Clipboard

Toggle word wrap

1: The CR name must be instance.
2: The CR must be installed to the openshift-logging namespace.
3: The Red Hat OpenShift Logging Operator management state. When the state is set to unmanaged, the Operator is in an unsupported state and does not receive updates.

5.5.2. Configuring log storage
Copia collegamento

You can configure which log storage type your logging uses by modifying the ClusterLogging custom resource (CR).

Prerequisites

You have administrator permissions.
You have installed the OpenShift CLI (oc).
You have installed the Red Hat OpenShift Logging Operator and an internal log store that is either the LokiStack or Elasticsearch.
You have created a ClusterLogging CR.

Note

The Logging 5.9 release does not contain an updated version of the OpenShift Elasticsearch Operator. If you currently use the OpenShift Elasticsearch Operator released with Logging 5.8, it will continue to work with Logging until the EOL of Logging 5.8. As an alternative to using the OpenShift Elasticsearch Operator to manage the default log storage, you can use the Loki Operator. For more information on the Logging lifecycle dates, see Platform Agnostic Operators.

Procedure

Modify the ClusterLogging CR logStore spec:

ClusterLogging CR example

apiVersion: logging.openshift.io/v1
kind: ClusterLogging
metadata:
# ...
spec:
# ...
  logStore:
    type: <log_store_type> 
    elasticsearch: 
      nodeCount: <integer>
      resources: {}
      storage: {}
      redundancyPolicy: <redundancy_type> 
    lokistack: 
      name: {}
# ...

apiVersion: logging.openshift.io/v1
kind: ClusterLogging
metadata:
# ...
spec:
# ...
  logStore:
    type: <log_store_type>


    elasticsearch:


      nodeCount: <integer>
      resources: {}
      storage: {}
      redundancyPolicy: <redundancy_type>


    lokistack:


      name: {}
# ...

Copy to Clipboard

Toggle word wrap

1: Specify the log store type. This can be either lokistack or elasticsearch.
2: Optional configuration options for the Elasticsearch log store.
3: Specify the redundancy type. This value can be ZeroRedundancy, SingleRedundancy, MultipleRedundancy, or FullRedundancy.
4: Optional configuration options for LokiStack.

Example ClusterLogging CR to specify LokiStack as the log store

apiVersion: logging.openshift.io/v1
kind: ClusterLogging
metadata:
  name: instance
  namespace: openshift-logging
spec:
  managementState: Managed
  logStore:
    type: lokistack
    lokistack:
      name: logging-loki
# ...

apiVersion: logging.openshift.io/v1
kind: ClusterLogging
metadata:
  name: instance
  namespace: openshift-logging
spec:
  managementState: Managed
  logStore:
    type: lokistack
    lokistack:
      name: logging-loki
# ...

Copy to Clipboard

Toggle word wrap

Apply the ClusterLogging CR by running the following command:
```
oc apply -f <filename>.yaml
```
```
$ oc apply -f <filename>.yaml
```
Copy to Clipboard Toggle word wrap

5.5.3. Configuring the log collector
Copia collegamento

You can configure which log collector type your logging uses by modifying the ClusterLogging custom resource (CR).

Note

Fluentd is deprecated and is planned to be removed in a future release. Red Hat provides bug fixes and support for this feature during the current release lifecycle, but this feature no longer receives enhancements. As an alternative to Fluentd, you can use Vector instead.

Prerequisites

You have administrator permissions.
You have installed the OpenShift CLI (oc).
You have installed the Red Hat OpenShift Logging Operator.
You have created a ClusterLogging CR.

Procedure

Modify the ClusterLogging CR collection spec:

ClusterLogging CR example

apiVersion: logging.openshift.io/v1
kind: ClusterLogging
metadata:
# ...
spec:
# ...
  collection:
    type: <log_collector_type> 
    resources: {}
    tolerations: {}
# ...

apiVersion: logging.openshift.io/v1
kind: ClusterLogging
metadata:
# ...
spec:
# ...
  collection:
    type: <log_collector_type>


    resources: {}
    tolerations: {}
# ...

Copy to Clipboard

Toggle word wrap

1: The log collector type you want to use for the logging. This can be vector or fluentd.

Apply the ClusterLogging CR by running the following command:
```
oc apply -f <filename>.yaml
```
```
$ oc apply -f <filename>.yaml
```
Copy to Clipboard Toggle word wrap

5.5.4. Configuring the log visualizer
Copia collegamento

You can configure which log visualizer type your logging uses by modifying the ClusterLogging custom resource (CR).

Prerequisites

You have administrator permissions.
You have installed the OpenShift CLI (oc).
You have installed the Red Hat OpenShift Logging Operator.
You have created a ClusterLogging CR.

Important

If you want to use the OpenShift Container Platform web console for visualization, you must enable the logging Console Plugin. See the documentation about "Log visualization with the web console".

Procedure

Modify the ClusterLogging CR visualization spec:

ClusterLogging CR example

apiVersion: logging.openshift.io/v1
kind: ClusterLogging
metadata:
# ...
spec:
# ...
  visualization:
    type: <visualizer_type> 
    kibana: 
      resources: {}
      nodeSelector: {}
      proxy: {}
      replicas: {}
      tolerations: {}
    ocpConsole: 
      logsLimit: {}
      timeout: {}
# ...

apiVersion: logging.openshift.io/v1
kind: ClusterLogging
metadata:
# ...
spec:
# ...
  visualization:
    type: <visualizer_type>


    kibana:


      resources: {}
      nodeSelector: {}
      proxy: {}
      replicas: {}
      tolerations: {}
    ocpConsole:


      logsLimit: {}
      timeout: {}
# ...

Copy to Clipboard

Toggle word wrap

1: The type of visualizer you want to use for your logging. This can be either kibana or ocp-console. The Kibana console is only compatible with deployments that use Elasticsearch log storage, while the OpenShift Container Platform console is only compatible with LokiStack deployments.
2: Optional configurations for the Kibana console.
3: Optional configurations for the OpenShift Container Platform web console.

Apply the ClusterLogging CR by running the following command:
```
oc apply -f <filename>.yaml
```
```
$ oc apply -f <filename>.yaml
```
Copy to Clipboard Toggle word wrap

5.5.5. Allowing traffic between projects when network isolation is enabled
Copia collegamento

Your cluster network plugin might enforce network isolation. If so, you must allow network traffic between the projects that contain the operators deployed by OpenShift Logging.

Network isolation blocks network traffic between pods or services that are in different projects. The logging installs the OpenShift Elasticsearch Operator in the openshift-operators-redhat project and the Red Hat OpenShift Logging Operator in the openshift-logging project. Therefore, you must allow traffic between these two projects.

OpenShift Container Platform offers two supported choices for the network plugin, OpenShift SDN and OVN-Kubernetes. These two providers implement various network isolation policies.

OpenShift SDN has three modes:

network policy: This is the default mode. If no policy is defined, it allows all traffic. However, if a user defines a policy, they typically start by denying all traffic and then adding exceptions. This process might break applications that are running in different projects. Therefore, explicitly configure the policy to allow traffic to egress from one logging-related project to the other.
multitenant: This mode enforces network isolation. You must join the two logging-related projects to allow traffic between them.
subnet: This mode allows all traffic. It does not enforce network isolation. No action is needed.

OVN-Kubernetes always uses a network policy. Therefore, as with OpenShift SDN, you must configure the policy to allow traffic to egress from one logging-related project to the other.

Procedure

If you are using OpenShift SDN in multitenant mode, join the two projects. For example:

oc adm pod-network join-projects --to=openshift-operators-redhat openshift-logging

$ oc adm pod-network join-projects --to=openshift-operators-redhat openshift-logging

Copy to Clipboard

Toggle word wrap

Otherwise, for OpenShift SDN in network policy mode and OVN-Kubernetes, perform the following actions:

Set a label on the openshift-operators-redhat namespace. For example:

oc label namespace openshift-operators-redhat project=openshift-operators-redhat

$ oc label namespace openshift-operators-redhat project=openshift-operators-redhat

Copy to Clipboard

Toggle word wrap

Create a network policy object in the openshift-logging namespace that allows ingress from the openshift-operators-redhat, openshift-monitoring and openshift-ingress projects to the openshift-logging project. For example:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-from-openshift-monitoring-ingress-operators-redhat
spec:
  ingress:
  - from:
    - podSelector: {}
  - from:
    - namespaceSelector:
        matchLabels:
          project: "openshift-operators-redhat"
  - from:
    - namespaceSelector:
        matchLabels:
          name: "openshift-monitoring"
  - from:
    - namespaceSelector:
        matchLabels:
          network.openshift.io/policy-group: ingress
  podSelector: {}
  policyTypes:
  - Ingress

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-from-openshift-monitoring-ingress-operators-redhat
spec:
  ingress:
  - from:
    - podSelector: {}
  - from:
    - namespaceSelector:
        matchLabels:
          project: "openshift-operators-redhat"
  - from:
    - namespaceSelector:
        matchLabels:
          name: "openshift-monitoring"
  - from:
    - namespaceSelector:
        matchLabels:
          network.openshift.io/policy-group: ingress
  podSelector: {}
  policyTypes:
  - Ingress

Copy to Clipboard

Toggle word wrap

Chapter 6. Updating Logging
Copia collegamento

There are two types of logging updates: minor release updates (5.y.z) and major release updates (5.y).

6.1. Minor release updates
Copia collegamento

If you installed the logging Operators using the Automatic update approval option, your Operators receive minor version updates automatically. You do not need to complete any manual update steps.

If you installed the logging Operators using the Manual update approval option, you must manually approve minor version updates.

For more information, see Manually approving a pending Operator update.

6.2. Major release updates
Copia collegamento

For major version updates you must complete some manual steps.

For major release version compatibility and support information, see OpenShift Operator Life Cycles.

6.3. Upgrading the Red Hat OpenShift Logging Operator to watch all namespaces
Copia collegamento

In logging 5.7 and older versions, the Red Hat OpenShift Logging Operator only watches the openshift-logging namespace. If you want the Red Hat OpenShift Logging Operator to watch all namespaces on your cluster, you must redeploy the Operator. You can complete the following procedure to redeploy the Operator without deleting your logging components.

Prerequisites

You have installed the OpenShift CLI (oc).
You have administrator permissions.

Procedure

Delete the subscription by running the following command:

oc -n openshift-logging delete subscription <subscription>

$ oc -n openshift-logging delete subscription <subscription>

Copy to Clipboard

Toggle word wrap

Delete the Operator group by running the following command:

oc -n openshift-logging delete operatorgroup <operator_group_name>

$ oc -n openshift-logging delete operatorgroup <operator_group_name>

Copy to Clipboard

Toggle word wrap

Delete the cluster service version (CSV) by running the following command:
```
oc delete clusterserviceversion cluster-logging.<version>
```
```
$ oc delete clusterserviceversion cluster-logging.<version>
```
Copy to Clipboard Toggle word wrap
Redeploy the Red Hat OpenShift Logging Operator by following the "Installing Logging" documentation.

Verification

Check that the targetNamespaces field in the OperatorGroup resource is not present or is set to an empty string.

To do this, run the following command and inspect the output:

oc get operatorgroup <operator_group_name> -o yaml

$ oc get operatorgroup <operator_group_name> -o yaml

Copy to Clipboard

Toggle word wrap

Example output

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: openshift-logging-f52cn
  namespace: openshift-logging
spec:
  upgradeStrategy: Default
status:
  namespaces:
  - ""
# ...

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: openshift-logging-f52cn
  namespace: openshift-logging
spec:
  upgradeStrategy: Default
status:
  namespaces:
  - ""
# ...

Copy to Clipboard

Toggle word wrap

6.4. Updating the Red Hat OpenShift Logging Operator
Copia collegamento

To update the Red Hat OpenShift Logging Operator to a new major release version, you must modify the update channel for the Operator subscription.

Prerequisites

You have installed the Red Hat OpenShift Logging Operator.
You have administrator permissions.
You have access to the OpenShift Container Platform web console and are viewing the Administrator perspective.

Procedure

Navigate to Operators → Installed Operators.
Select the openshift-logging project.
Click the Red Hat OpenShift Logging Operator.
Click Subscription. In the Subscription details section, click the Update channel link. This link text might be stable or stable-5.y, depending on your current update channel.
In the Change Subscription Update Channel window, select the latest major version update channel, stable-5.y, and click Save. Note the cluster-logging.v5.y.z version.

Verification

Wait for a few seconds, then click Operators → Installed Operators. Verify that the Red Hat OpenShift Logging Operator version matches the latest cluster-logging.v5.y.z version.
On the Operators → Installed Operators page, wait for the Status field to report Succeeded.

6.5. Updating the Loki Operator
Copia collegamento

To update the Loki Operator to a new major release version, you must modify the update channel for the Operator subscription.

Prerequisites

You have installed the Loki Operator.
You have administrator permissions.
You have access to the OpenShift Container Platform web console and are viewing the Administrator perspective.

Procedure

Navigate to Operators → Installed Operators.
Select the openshift-operators-redhat project.
Click the Loki Operator.
Click Subscription. In the Subscription details section, click the Update channel link. This link text might be stable or stable-5.y, depending on your current update channel.
In the Change Subscription Update Channel window, select the latest major version update channel, stable-5.y, and click Save. Note the loki-operator.v5.y.z version.

Verification

Wait for a few seconds, then click Operators → Installed Operators. Verify that the Loki Operator version matches the latest loki-operator.v5.y.z version.
On the Operators → Installed Operators page, wait for the Status field to report Succeeded.

6.6. Updating the OpenShift Elasticsearch Operator
Copia collegamento

To update the OpenShift Elasticsearch Operator to the current version, you must modify the subscription.

Note

Prerequisites

If you are using Elasticsearch as the default log store, and Kibana as the UI, update the OpenShift Elasticsearch Operator before you update the Red Hat OpenShift Logging Operator.
Important
If you update the Operators in the wrong order, Kibana does not update and the Kibana custom resource (CR) is not created. To fix this issue, delete the Red Hat OpenShift Logging Operator pod. When the Red Hat OpenShift Logging Operator pod redeploys, it creates the Kibana CR and Kibana becomes available again.
The Logging status is healthy:
- All pods have a ready status.
- The Elasticsearch cluster is healthy.
Your Elasticsearch and Kibana data is backed up.
You have administrator permissions.
You have installed the OpenShift CLI (oc) for the verification steps.

Procedure

In the OpenShift Container Platform web console, click Operators → Installed Operators.
Select the openshift-operators-redhat project.
Click OpenShift Elasticsearch Operator.
Click Subscription → Channel.
In the Change Subscription Update Channel window, select stable-5.y and click Save. Note the elasticsearch-operator.v5.y.z version.
Wait for a few seconds, then click Operators → Installed Operators. Verify that the OpenShift Elasticsearch Operator version matches the latest elasticsearch-operator.v5.y.z version.
On the Operators → Installed Operators page, wait for the Status field to report Succeeded.

Verification

Verify that all Elasticsearch pods have a Ready status by entering the following command and observing the output:

oc get pod -n openshift-logging --selector component=elasticsearch

$ oc get pod -n openshift-logging --selector component=elasticsearch

Copy to Clipboard

Toggle word wrap

Example output

NAME                                            READY   STATUS    RESTARTS   AGE
elasticsearch-cdm-1pbrl44l-1-55b7546f4c-mshhk   2/2     Running   0          31m
elasticsearch-cdm-1pbrl44l-2-5c6d87589f-gx5hk   2/2     Running   0          30m
elasticsearch-cdm-1pbrl44l-3-88df5d47-m45jc     2/2     Running   0          29m

NAME                                            READY   STATUS    RESTARTS   AGE
elasticsearch-cdm-1pbrl44l-1-55b7546f4c-mshhk   2/2     Running   0          31m
elasticsearch-cdm-1pbrl44l-2-5c6d87589f-gx5hk   2/2     Running   0          30m
elasticsearch-cdm-1pbrl44l-3-88df5d47-m45jc     2/2     Running   0          29m

Copy to Clipboard

Toggle word wrap

Verify that the Elasticsearch cluster status is green by entering the following command and observing the output:

oc exec -n openshift-logging -c elasticsearch elasticsearch-cdm-1pbrl44l-1-55b7546f4c-mshhk -- health

$ oc exec -n openshift-logging -c elasticsearch elasticsearch-cdm-1pbrl44l-1-55b7546f4c-mshhk -- health

Copy to Clipboard

Toggle word wrap

Example output

{
  "cluster_name" : "elasticsearch",
  "status" : "green",
}

{
  "cluster_name" : "elasticsearch",
  "status" : "green",
}

Copy to Clipboard

Toggle word wrap

Verify that the Elasticsearch cron jobs are created by entering the following commands and observing the output:

oc project openshift-logging

$ oc project openshift-logging

Copy to Clipboard

Toggle word wrap

oc get cronjob

$ oc get cronjob

Copy to Clipboard

Toggle word wrap

Example output

NAME                     SCHEDULE       SUSPEND   ACTIVE   LAST SCHEDULE   AGE
elasticsearch-im-app     */15 * * * *   False     0        <none>          56s
elasticsearch-im-audit   */15 * * * *   False     0        <none>          56s
elasticsearch-im-infra   */15 * * * *   False     0        <none>          56s

NAME                     SCHEDULE       SUSPEND   ACTIVE   LAST SCHEDULE   AGE
elasticsearch-im-app     */15 * * * *   False     0        <none>          56s
elasticsearch-im-audit   */15 * * * *   False     0        <none>          56s
elasticsearch-im-infra   */15 * * * *   False     0        <none>          56s

Copy to Clipboard

Toggle word wrap

Verify that the log store is updated to the correct version and the indices are green by entering the following command and observing the output:

oc exec -c elasticsearch <any_es_pod_in_the_cluster> -- indices

$ oc exec -c elasticsearch <any_es_pod_in_the_cluster> -- indices

Copy to Clipboard

Toggle word wrap

Verify that the output includes the app-00000x, infra-00000x, audit-00000x, .security indices:

Example 6.1. Sample output with indices in a green status

Tue Jun 30 14:30:54 UTC 2020
health status index                                                                 uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   infra-000008                                                          bnBvUFEXTWi92z3zWAzieQ   3 1       222195            0        289            144
green  open   infra-000004                                                          rtDSzoqsSl6saisSK7Au1Q   3 1       226717            0        297            148
green  open   infra-000012                                                          RSf_kUwDSR2xEuKRZMPqZQ   3 1       227623            0        295            147
green  open   .kibana_7                                                             1SJdCqlZTPWlIAaOUd78yg   1 1            4            0          0              0
green  open   infra-000010                                                          iXwL3bnqTuGEABbUDa6OVw   3 1       248368            0        317            158
green  open   infra-000009                                                          YN9EsULWSNaxWeeNvOs0RA   3 1       258799            0        337            168
green  open   infra-000014                                                          YP0U6R7FQ_GVQVQZ6Yh9Ig   3 1       223788            0        292            146
green  open   infra-000015                                                          JRBbAbEmSMqK5X40df9HbQ   3 1       224371            0        291            145
green  open   .orphaned.2020.06.30                                                  n_xQC2dWQzConkvQqei3YA   3 1            9            0          0              0
green  open   infra-000007                                                          llkkAVSzSOmosWTSAJM_hg   3 1       228584            0        296            148
green  open   infra-000005                                                          d9BoGQdiQASsS3BBFm2iRA   3 1       227987            0        297            148
green  open   infra-000003                                                          1-goREK1QUKlQPAIVkWVaQ   3 1       226719            0        295            147
green  open   .security                                                             zeT65uOuRTKZMjg_bbUc1g   1 1            5            0          0              0
green  open   .kibana-377444158_kubeadmin                                           wvMhDwJkR-mRZQO84K0gUQ   3 1            1            0          0              0
green  open   infra-000006                                                          5H-KBSXGQKiO7hdapDE23g   3 1       226676            0        295            147
green  open   infra-000001                                                          eH53BQ-bSxSWR5xYZB6lVg   3 1       341800            0        443            220
green  open   .kibana-6                                                             RVp7TemSSemGJcsSUmuf3A   1 1            4            0          0              0
green  open   infra-000011                                                          J7XWBauWSTe0jnzX02fU6A   3 1       226100            0        293            146
green  open   app-000001                                                            axSAFfONQDmKwatkjPXdtw   3 1       103186            0        126             57
green  open   infra-000016                                                          m9c1iRLtStWSF1GopaRyCg   3 1        13685            0         19              9
green  open   infra-000002                                                          Hz6WvINtTvKcQzw-ewmbYg   3 1       228994            0        296            148
green  open   infra-000013                                                          KR9mMFUpQl-jraYtanyIGw   3 1       228166            0        298            148
green  open   audit-000001                                                          eERqLdLmQOiQDFES1LBATQ   3 1            0            0          0              0

Tue Jun 30 14:30:54 UTC 2020
health status index                                                                 uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   infra-000008                                                          bnBvUFEXTWi92z3zWAzieQ   3 1       222195            0        289            144
green  open   infra-000004                                                          rtDSzoqsSl6saisSK7Au1Q   3 1       226717            0        297            148
green  open   infra-000012                                                          RSf_kUwDSR2xEuKRZMPqZQ   3 1       227623            0        295            147
green  open   .kibana_7                                                             1SJdCqlZTPWlIAaOUd78yg   1 1            4            0          0              0
green  open   infra-000010                                                          iXwL3bnqTuGEABbUDa6OVw   3 1       248368            0        317            158
green  open   infra-000009                                                          YN9EsULWSNaxWeeNvOs0RA   3 1       258799            0        337            168
green  open   infra-000014                                                          YP0U6R7FQ_GVQVQZ6Yh9Ig   3 1       223788            0        292            146
green  open   infra-000015                                                          JRBbAbEmSMqK5X40df9HbQ   3 1       224371            0        291            145
green  open   .orphaned.2020.06.30                                                  n_xQC2dWQzConkvQqei3YA   3 1            9            0          0              0
green  open   infra-000007                                                          llkkAVSzSOmosWTSAJM_hg   3 1       228584            0        296            148
green  open   infra-000005                                                          d9BoGQdiQASsS3BBFm2iRA   3 1       227987            0        297            148
green  open   infra-000003                                                          1-goREK1QUKlQPAIVkWVaQ   3 1       226719            0        295            147
green  open   .security                                                             zeT65uOuRTKZMjg_bbUc1g   1 1            5            0          0              0
green  open   .kibana-377444158_kubeadmin                                           wvMhDwJkR-mRZQO84K0gUQ   3 1            1            0          0              0
green  open   infra-000006                                                          5H-KBSXGQKiO7hdapDE23g   3 1       226676            0        295            147
green  open   infra-000001                                                          eH53BQ-bSxSWR5xYZB6lVg   3 1       341800            0        443            220
green  open   .kibana-6                                                             RVp7TemSSemGJcsSUmuf3A   1 1            4            0          0              0
green  open   infra-000011                                                          J7XWBauWSTe0jnzX02fU6A   3 1       226100            0        293            146
green  open   app-000001                                                            axSAFfONQDmKwatkjPXdtw   3 1       103186            0        126             57
green  open   infra-000016                                                          m9c1iRLtStWSF1GopaRyCg   3 1        13685            0         19              9
green  open   infra-000002                                                          Hz6WvINtTvKcQzw-ewmbYg   3 1       228994            0        296            148
green  open   infra-000013                                                          KR9mMFUpQl-jraYtanyIGw   3 1       228166            0        298            148
green  open   audit-000001                                                          eERqLdLmQOiQDFES1LBATQ   3 1            0            0          0              0

Copy to Clipboard

Toggle word wrap

Verify that the log visualizer is updated to the correct version by entering the following command and observing the output:

oc get kibana kibana -o json

$ oc get kibana kibana -o json

Copy to Clipboard

Toggle word wrap

Verify that the output includes a Kibana pod with the ready status:

Example 6.2. Sample output with a ready Kibana pod

[
{
"clusterCondition": {
"kibana-5fdd766ffd-nb2jj": [
{
"lastTransitionTime": "2020-06-30T14:11:07Z",
"reason": "ContainerCreating",
"status": "True",
"type": ""
},
{
"lastTransitionTime": "2020-06-30T14:11:07Z",
"reason": "ContainerCreating",
"status": "True",
"type": ""
}
]
},
"deployment": "kibana",
"pods": {
"failed": [],
"notReady": []
"ready": []
},
"replicaSets": [
"kibana-5fdd766ffd"
],
"replicas": 1
}
]

[
{
"clusterCondition": {
"kibana-5fdd766ffd-nb2jj": [
{
"lastTransitionTime": "2020-06-30T14:11:07Z",
"reason": "ContainerCreating",
"status": "True",
"type": ""
},
{
"lastTransitionTime": "2020-06-30T14:11:07Z",
"reason": "ContainerCreating",
"status": "True",
"type": ""
}
]
},
"deployment": "kibana",
"pods": {
"failed": [],
"notReady": []
"ready": []
},
"replicaSets": [
"kibana-5fdd766ffd"
],
"replicas": 1
}
]

Copy to Clipboard

Toggle word wrap

Chapter 7. Visualizing logs
Copia collegamento

7.1. About log visualization
Copia collegamento

You can visualize your log data in the OpenShift Container Platform web console, or the Kibana web console, depending on your deployed log storage solution. The Kibana console can be used with ElasticSearch log stores, and the OpenShift Container Platform web console can be used with the ElasticSearch log store or the LokiStack.

Note

The Kibana web console is now deprecated is planned to be removed in a future logging release.

7.1.1. Configuring the log visualizer
Copia collegamento

You can configure which log visualizer type your logging uses by modifying the ClusterLogging custom resource (CR).

Prerequisites

You have administrator permissions.
You have installed the OpenShift CLI (oc).
You have installed the Red Hat OpenShift Logging Operator.
You have created a ClusterLogging CR.

Important

If you want to use the OpenShift Container Platform web console for visualization, you must enable the logging Console Plugin. See the documentation about "Log visualization with the web console".

Procedure

Modify the ClusterLogging CR visualization spec:

ClusterLogging CR example

apiVersion: logging.openshift.io/v1
kind: ClusterLogging
metadata:
# ...
spec:
# ...
  visualization:
    type: <visualizer_type> 
    kibana: 
      resources: {}
      nodeSelector: {}
      proxy: {}
      replicas: {}
      tolerations: {}
    ocpConsole: 
      logsLimit: {}
      timeout: {}
# ...

apiVersion: logging.openshift.io/v1
kind: ClusterLogging
metadata:
# ...
spec:
# ...
  visualization:
    type: <visualizer_type>


    kibana:


      resources: {}
      nodeSelector: {}
      proxy: {}
      replicas: {}
      tolerations: {}
    ocpConsole:


      logsLimit: {}
      timeout: {}
# ...

Copy to Clipboard

Toggle word wrap

1: The type of visualizer you want to use for your logging. This can be either kibana or ocp-console. The Kibana console is only compatible with deployments that use Elasticsearch log storage, while the OpenShift Container Platform console is only compatible with LokiStack deployments.
2: Optional configurations for the Kibana console.
3: Optional configurations for the OpenShift Container Platform web console.

Apply the ClusterLogging CR by running the following command:
```
oc apply -f <filename>.yaml
```
```
$ oc apply -f <filename>.yaml
```
Copy to Clipboard Toggle word wrap

7.1.2. Viewing logs for a resource
Copia collegamento

Resource logs are a default feature that provides limited log viewing capability. You can view the logs for various resources, such as builds, deployments, and pods by using the OpenShift CLI (oc) and the web console.

Tip

To enhance your log retrieving and viewing experience, install the logging. The logging aggregates all the logs from your OpenShift Container Platform cluster, such as node system audit logs, application container logs, and infrastructure logs, into a dedicated log store. You can then query, discover, and visualize your log data through the Kibana console or the OpenShift Container Platform web console. Resource logs do not access the logging log store.

7.1.2.1. Viewing resource logs
Copia collegamento

You can view the log for various resources in the OpenShift CLI (oc) and web console. Logs read from the tail, or end, of the log.

Prerequisites

Access to the OpenShift CLI (oc).

Procedure (UI)

In the OpenShift Container Platform console, navigate to Workloads → Pods or navigate to the pod through the resource you want to investigate.
Note
Some resources, such as builds, do not have pods to query directly. In such instances, you can locate the Logs link on the Details page for the resource.
Select a project from the drop-down menu.
Click the name of the pod you want to investigate.
Click Logs.

Procedure (CLI)

View the log for a specific pod:
```
oc logs -f <pod_name> -c <container_name>
```
```
$ oc logs -f <pod_name> -c <container_name>
```
Copy to Clipboard Toggle word wrap
where:
-f
Optional: Specifies that the output follows what is being written into the logs.
<pod_name>
Specifies the name of the pod.
<container_name>
Optional: Specifies the name of a container. When a pod has more than one container, you must specify the container name.
For example:
```
oc logs ruby-58cd97df55-mww7r
```
```
$ oc logs ruby-58cd97df55-mww7r
```
Copy to Clipboard Toggle word wrap
```
oc logs -f ruby-57f7f4855b-znl92 -c ruby
```
```
$ oc logs -f ruby-57f7f4855b-znl92 -c ruby
```
Copy to Clipboard Toggle word wrap
The contents of log files are printed out.
View the log for a specific resource:
```
oc logs <object_type>/<resource_name>
```
```
$ oc logs <object_type>/<resource_name> 
```
1
Copy to Clipboard Toggle word wrap
1
Specifies the resource type and name.
For example:
```
oc logs deployment/ruby
```
```
$ oc logs deployment/ruby
```
Copy to Clipboard Toggle word wrap
The contents of log files are printed out.

7.2. Log visualization with the web console
Copia collegamento

You can use the OpenShift Container Platform web console to visualize log data by configuring the logging Console Plugin.

For information about configuring the plugin during the logging installation, see Installing the logging using the web console.

If you have already installed the logging and want to configure the plugin, use one of the following procedures.

7.2.1. Enabling the logging Console Plugin after you have installed the Red Hat OpenShift Logging Operator
Copia collegamento

You can enable the logging Console Plugin as part of the Red Hat OpenShift Logging Operator installation, but you can also enable the plugin if you have already installed the Red Hat OpenShift Logging Operator with the plugin disabled.

Prerequisites

You have administrator permissions.
You have installed the Red Hat OpenShift Logging Operator and selected Disabled for the Console plugin.
You have access to the OpenShift Container Platform web console.

Procedure

In the OpenShift Container Platform web console Administrator perspective, navigate to Operators → Installed Operators.
Click Red Hat OpenShift Logging. This takes you to the Operator Details page.
In the Details page, click Disabled for the Console plugin option.
In the Console plugin enablement dialog, select Enable.
Click Save.
Verify that the Console plugin option now shows Enabled.
The web console displays a pop-up window when changes have been applied. The window prompts you to reload the web console. Refresh the browser when you see the pop-up window to apply the changes.

7.2.2. Configuring the logging Console Plugin when you have the Elasticsearch log store and LokiStack installed
Copia collegamento

In logging version 5.8 and later, if the Elasticsearch log store is your default log store but you have also installed the LokiStack, you can enable the logging Console Plugin by using the following procedure.

Prerequisites

You have administrator permissions.
You have installed the Red Hat OpenShift Logging Operator, the OpenShift Elasticsearch Operator, and the Loki Operator.
You have installed the OpenShift CLI (oc).
You have created a ClusterLogging custom resource (CR).

Procedure

Ensure that the logging Console Plugin is enabled by running the following command:

oc get consoles.operator.openshift.io cluster -o yaml |grep logging-view-plugin  \
|| oc patch consoles.operator.openshift.io cluster  --type=merge \
--patch '{ "spec": { "plugins": ["logging-view-plugin"]}}'

$ oc get consoles.operator.openshift.io cluster -o yaml |grep logging-view-plugin  \
|| oc patch consoles.operator.openshift.io cluster  --type=merge \
--patch '{ "spec": { "plugins": ["logging-view-plugin"]}}'

Copy to Clipboard

Toggle word wrap

Add the .metadata.annotations.logging.openshift.io/ocp-console-migration-target: lokistack-dev annotation to the ClusterLogging CR, by running the following command:

oc patch clusterlogging instance --type=merge --patch \
'{ "metadata": { "annotations": { "logging.openshift.io/ocp-console-migration-target": "lokistack-dev" }}}' \
-n openshift-logging

$ oc patch clusterlogging instance --type=merge --patch \
'{ "metadata": { "annotations": { "logging.openshift.io/ocp-console-migration-target": "lokistack-dev" }}}' \
-n openshift-logging

Copy to Clipboard

Toggle word wrap

Example output

clusterlogging.logging.openshift.io/instance patched

clusterlogging.logging.openshift.io/instance patched

Copy to Clipboard

Toggle word wrap

Verification

Verify that the annotation was added successfully, by running the following command and observing the output:

oc get clusterlogging instance \
-o=jsonpath='{.metadata.annotations.logging\.openshift\.io/ocp-console-migration-target}' \
-n openshift-logging

$ oc get clusterlogging instance \
-o=jsonpath='{.metadata.annotations.logging\.openshift\.io/ocp-console-migration-target}' \
-n openshift-logging

Copy to Clipboard

Toggle word wrap

Example output

"lokistack-dev"

"lokistack-dev"

Copy to Clipboard

Toggle word wrap

The logging Console Plugin pod is now deployed. You can view logging data by navigating to the OpenShift Container Platform web console and viewing the Observe → Logs page.

7.3. Viewing cluster dashboards
Copia collegamento

The Logging/Elasticsearch Nodes and Openshift Logging dashboards in the OpenShift Container Platform web console contain in-depth details about your Elasticsearch instance and the individual Elasticsearch nodes that you can use to prevent and diagnose problems.

The OpenShift Logging dashboard contains charts that show details about your Elasticsearch instance at a cluster level, including cluster resources, garbage collection, shards in the cluster, and Fluentd statistics.

The Logging/Elasticsearch Nodes dashboard contains charts that show details about your Elasticsearch instance, many at node level, including details on indexing, shards, resources, and so forth.

7.3.1. Accessing the Elasticsearch and OpenShift Logging dashboards
Copia collegamento

You can view the Logging/Elasticsearch Nodes and OpenShift Logging dashboards in the OpenShift Container Platform web console.

Procedure

To launch the dashboards:

In the OpenShift Container Platform web console, click Observe → Dashboards.
On the Dashboards page, select Logging/Elasticsearch Nodes or OpenShift Logging from the Dashboard menu.
For the Logging/Elasticsearch Nodes dashboard, you can select the Elasticsearch node you want to view and set the data resolution.
The appropriate dashboard is displayed, showing multiple charts of data.
Optional: Select a different time range to display or refresh rate for the data from the Time Range and Refresh Interval menus.

For information on the dashboard charts, see About the OpenShift Logging dashboard and About the Logging/Elastisearch Nodes dashboard.

7.3.2. About the OpenShift Logging dashboard
Copia collegamento

The OpenShift Logging dashboard contains charts that show details about your Elasticsearch instance at a cluster-level that you can use to diagnose and anticipate problems.

Expand

Table 7.1. OpenShift Logging charts
Metric	Description
Elastic Cluster Status	The current Elasticsearch status: ONLINE - Indicates that the Elasticsearch instance is online. OFFLINE - Indicates that the Elasticsearch instance is offline.
Elastic Nodes	The total number of Elasticsearch nodes in the Elasticsearch instance.
Elastic Shards	The total number of Elasticsearch shards in the Elasticsearch instance.
Elastic Documents	The total number of Elasticsearch documents in the Elasticsearch instance.
Total Index Size on Disk	The total disk space that is being used for the Elasticsearch indices.
Elastic Pending Tasks	The total number of Elasticsearch changes that have not been completed, such as index creation, index mapping, shard allocation, or shard failure.
Elastic JVM GC time	The amount of time that the JVM spent executing Elasticsearch garbage collection operations in the cluster.
Elastic JVM GC Rate	The total number of times that JVM executed garbage activities per second.
Elastic Query/Fetch Latency Sum	Query latency: The average time each Elasticsearch search query takes to execute. Fetch latency: The average time each Elasticsearch search query spends fetching data. Fetch latency typically takes less time than query latency. If fetch latency is consistently increasing, it might indicate slow disks, data enrichment, or large requests with too many results.
Elastic Query Rate	The total queries executed against the Elasticsearch instance per second for each Elasticsearch node.
CPU	The amount of CPU used by Elasticsearch, Fluentd, and Kibana, shown for each component.
Elastic JVM Heap Used	The amount of JVM memory used. In a healthy cluster, the graph shows regular drops as memory is freed by JVM garbage collection.
Elasticsearch Disk Usage	The total disk space used by the Elasticsearch instance for each Elasticsearch node.
File Descriptors In Use	The total number of file descriptors used by Elasticsearch, Fluentd, and Kibana.
FluentD emit count	The total number of Fluentd messages per second for the Fluentd default output, and the retry count for the default output.
FluentD Buffer Usage	The percent of the Fluentd buffer that is being used for chunks. A full buffer might indicate that Fluentd is not able to process the number of logs received.
Elastic rx bytes	The total number of bytes that Elasticsearch has received from FluentD, the Elasticsearch nodes, and other sources.
Elastic Index Failure Rate	The total number of times per second that an Elasticsearch index fails. A high rate might indicate an issue with indexing.
FluentD Output Error Rate	The total number of times per second that FluentD is not able to output logs.

7.3.3. Charts on the Logging/Elasticsearch nodes dashboard
Copia collegamento

The Logging/Elasticsearch Nodes dashboard contains charts that show details about your Elasticsearch instance, many at node-level, for further diagnostics.

Elasticsearch status: The Logging/Elasticsearch Nodes dashboard contains the following charts about the status of your Elasticsearch instance.

Expand

Table 7.2. Elasticsearch status fields
Metric	Description
Cluster status	The cluster health status during the selected time period, using the Elasticsearch green, yellow, and red statuses: 0 - Indicates that the Elasticsearch instance is in green status, which means that all shards are allocated. 1 - Indicates that the Elasticsearch instance is in yellow status, which means that replica shards for at least one shard are not allocated. 2 - Indicates that the Elasticsearch instance is in red status, which means that at least one primary shard and its replicas are not allocated.
Cluster nodes	The total number of Elasticsearch nodes in the cluster.
Cluster data nodes	The number of Elasticsearch data nodes in the cluster.
Cluster pending tasks	The number of cluster state changes that are not finished and are waiting in a cluster queue, for example, index creation, index deletion, or shard allocation. A growing trend indicates that the cluster is not able to keep up with changes.

Elasticsearch cluster index shard status: Each Elasticsearch index is a logical group of one or more shards, which are basic units of persisted data. There are two types of index shards: primary shards, and replica shards. When a document is indexed into an index, it is stored in one of its primary shards and copied into every replica of that shard. The number of primary shards is specified when the index is created, and the number cannot change during index lifetime. You can change the number of replica shards at any time.

The index shard can be in several states depending on its lifecycle phase or events occurring in the cluster. When the shard is able to perform search and indexing requests, the shard is active. If the shard cannot perform these requests, the shard is non–active. A shard might be non-active if the shard is initializing, reallocating, unassigned, and so forth.

Index shards consist of a number of smaller internal blocks, called index segments, which are physical representations of the data. An index segment is a relatively small, immutable Lucene index that is created when Lucene commits newly-indexed data. Lucene, a search library used by Elasticsearch, merges index segments into larger segments in the background to keep the total number of segments low. If the process of merging segments is slower than the speed at which new segments are created, it could indicate a problem.

When Lucene performs data operations, such as a search operation, Lucene performs the operation against the index segments in the relevant index. For that purpose, each segment contains specific data structures that are loaded in the memory and mapped. Index mapping can have a significant impact on the memory used by segment data structures.

The Logging/Elasticsearch Nodes dashboard contains the following charts about the Elasticsearch index shards.

Expand

Table 7.3. Elasticsearch cluster shard status charts
Metric	Description
Cluster active shards	The number of active primary shards and the total number of shards, including replicas, in the cluster. If the number of shards grows higher, the cluster performance can start degrading.
Cluster initializing shards	The number of non-active shards in the cluster. A non-active shard is one that is initializing, being reallocated to a different node, or is unassigned. A cluster typically has non–active shards for short periods. A growing number of non–active shards over longer periods could indicate a problem.
Cluster relocating shards	The number of shards that Elasticsearch is relocating to a new node. Elasticsearch relocates nodes for multiple reasons, such as high memory use on a node or after a new node is added to the cluster.
Cluster unassigned shards	The number of unassigned shards. Elasticsearch shards might be unassigned for reasons such as a new index being added or the failure of a node.

Elasticsearch node metrics: Each Elasticsearch node has a finite amount of resources that can be used to process tasks. When all the resources are being used and Elasticsearch attempts to perform a new task, Elasticsearch puts the tasks into a queue until some resources become available.

The Logging/Elasticsearch Nodes dashboard contains the following charts about resource usage for a selected node and the number of tasks waiting in the Elasticsearch queue.

Expand

Table 7.4. Elasticsearch node metric charts
Metric	Description
ThreadPool tasks	The number of waiting tasks in individual queues, shown by task type. A long–term accumulation of tasks in any queue could indicate node resource shortages or some other problem.
CPU usage	The amount of CPU being used by the selected Elasticsearch node as a percentage of the total CPU allocated to the host container.
Memory usage	The amount of memory being used by the selected Elasticsearch node.
Disk usage	The total disk space being used for index data and metadata on the selected Elasticsearch node.
Documents indexing rate	The rate that documents are indexed on the selected Elasticsearch node.
Indexing latency	The time taken to index the documents on the selected Elasticsearch node. Indexing latency can be affected by many factors, such as JVM Heap memory and overall load. A growing latency indicates a resource capacity shortage in the instance.
Search rate	The number of search requests run on the selected Elasticsearch node.
Search latency	The time taken to complete search requests on the selected Elasticsearch node. Search latency can be affected by many factors. A growing latency indicates a resource capacity shortage in the instance.
Documents count (with replicas)	The number of Elasticsearch documents stored on the selected Elasticsearch node, including documents stored in both the primary shards and replica shards that are allocated on the node.
Documents deleting rate	The number of Elasticsearch documents being deleted from any of the index shards that are allocated to the selected Elasticsearch node.
Documents merging rate	The number of Elasticsearch documents being merged in any of index shards that are allocated to the selected Elasticsearch node.

Elasticsearch node fielddata: Fielddata is an Elasticsearch data structure that holds lists of terms in an index and is kept in the JVM Heap. Because fielddata building is an expensive operation, Elasticsearch caches the fielddata structures. Elasticsearch can evict a fielddata cache when the underlying index segment is deleted or merged, or if there is not enough JVM HEAP memory for all the fielddata caches.

The Logging/Elasticsearch Nodes dashboard contains the following charts about Elasticsearch fielddata.

Expand

Table 7.5. Elasticsearch node fielddata charts
Metric	Description
Fielddata memory size	The amount of JVM Heap used for the fielddata cache on the selected Elasticsearch node.
Fielddata evictions	The number of fielddata structures that were deleted from the selected Elasticsearch node.

Elasticsearch node query cache: If the data stored in the index does not change, search query results are cached in a node-level query cache for reuse by Elasticsearch.

The Logging/Elasticsearch Nodes dashboard contains the following charts about the Elasticsearch node query cache.

Expand

Table 7.6. Elasticsearch node query charts
Metric	Description
Query cache size	The total amount of memory used for the query cache for all the shards allocated to the selected Elasticsearch node.
Query cache evictions	The number of query cache evictions on the selected Elasticsearch node.
Query cache hits	The number of query cache hits on the selected Elasticsearch node.
Query cache misses	The number of query cache misses on the selected Elasticsearch node.

Elasticsearch index throttling: When indexing documents, Elasticsearch stores the documents in index segments, which are physical representations of the data. At the same time, Elasticsearch periodically merges smaller segments into a larger segment as a way to optimize resource use. If the indexing is faster then the ability to merge segments, the merge process does not complete quickly enough, which can lead to issues with searches and performance. To prevent this situation, Elasticsearch throttles indexing, typically by reducing the number of threads allocated to indexing down to a single thread.

The Logging/Elasticsearch Nodes dashboard contains the following charts about Elasticsearch index throttling.

Expand

Table 7.7. Index throttling charts
Metric	Description
Indexing throttling	The amount of time that Elasticsearch has been throttling the indexing operations on the selected Elasticsearch node.
Merging throttling	The amount of time that Elasticsearch has been throttling the segment merge operations on the selected Elasticsearch node.

Node JVM Heap statistics: The Logging/Elasticsearch Nodes dashboard contains the following charts about JVM Heap operations.

Expand

Table 7.8. JVM Heap statistic charts
Metric	Description
Heap used	The amount of the total allocated JVM Heap space that is used on the selected Elasticsearch node.
GC count	The number of garbage collection operations that have been run on the selected Elasticsearch node, by old and young garbage collection.
GC time	The amount of time that the JVM spent running garbage collection operations on the selected Elasticsearch node, by old and young garbage collection.

7.4. Log visualization with Kibana
Copia collegamento

If you are using the ElasticSearch log store, you can use the Kibana console to visualize collected log data.

Using Kibana, you can do the following with your data:

Search and browse the data using the Discover tab.
Chart and map the data using the Visualize tab.
Create and view custom dashboards using the Dashboard tab.

Use and configuration of the Kibana interface is beyond the scope of this documentation. For more information about using the interface, see the Kibana documentation.

Note

The audit logs are not stored in the internal OpenShift Container Platform Elasticsearch instance by default. To view the audit logs in Kibana, you must use the Log Forwarding API to configure a pipeline that uses the default output for audit logs.

7.4.1. Defining Kibana index patterns
Copia collegamento

An index pattern defines the Elasticsearch indices that you want to visualize. To explore and visualize data in Kibana, you must create an index pattern.

Prerequisites

A user must have the cluster-admin role, the cluster-reader role, or both roles to view the infra and audit indices in Kibana. The default kubeadmin user has proper permissions to view these indices.
If you can view the pods and logs in the default, kube- and openshift- projects, you should be able to access these indices. You can use the following command to check if the current user has appropriate permissions:
```
oc auth can-i get pods --subresource log -n <project>
```
```
$ oc auth can-i get pods --subresource log -n <project>
```
Copy to Clipboard Toggle word wrap
Example output
```
yes
```
```
yes
```
Copy to Clipboard Toggle word wrap
Note
The audit logs are not stored in the internal OpenShift Container Platform Elasticsearch instance by default. To view the audit logs in Kibana, you must use the Log Forwarding API to configure a pipeline that uses the default output for audit logs.
Elasticsearch documents must be indexed before you can create index patterns. This is done automatically, but it might take a few minutes in a new or updated cluster.

Procedure

To define index patterns and create visualizations in Kibana:

In the OpenShift Container Platform console, click the Application Launcher and select Logging.
Create your Kibana index patterns by clicking Management → Index Patterns → Create index pattern:
- Each user must manually create index patterns when logging into Kibana the first time to see logs for their projects. Users must create an index pattern named app and use the @timestamp time field to view their container logs.
- Each admin user must create index patterns when logged into Kibana the first time for the app, infra, and audit indices using the @timestamp time field.
Create Kibana Visualizations from the new index patterns.

7.4.2. Viewing cluster logs in Kibana
Copia collegamento

You view cluster logs in the Kibana web console. The methods for viewing and visualizing your data in Kibana that are beyond the scope of this documentation. For more information, refer to the Kibana documentation.

Prerequisites

The Red Hat OpenShift Logging and Elasticsearch Operators must be installed.
Kibana index patterns must exist.
A user must have the cluster-admin role, the cluster-reader role, or both roles to view the infra and audit indices in Kibana. The default kubeadmin user has proper permissions to view these indices.
If you can view the pods and logs in the default, kube- and openshift- projects, you should be able to access these indices. You can use the following command to check if the current user has appropriate permissions:
```
oc auth can-i get pods --subresource log -n <project>
```
```
$ oc auth can-i get pods --subresource log -n <project>
```
Copy to Clipboard Toggle word wrap
Example output
```
yes
```
```
yes
```
Copy to Clipboard Toggle word wrap
Note
The audit logs are not stored in the internal OpenShift Container Platform Elasticsearch instance by default. To view the audit logs in Kibana, you must use the Log Forwarding API to configure a pipeline that uses the default output for audit logs.

Procedure

To view logs in Kibana:

In the OpenShift Container Platform console, click the Application Launcher and select Logging.
Log in using the same credentials you use to log in to the OpenShift Container Platform console.
The Kibana interface launches.
In Kibana, click Discover.
Select the index pattern you created from the drop-down menu in the top-left corner: app, audit, or infra.
The log data displays as time-stamped documents.
Expand one of the time-stamped documents.

Click the JSON tab to display the log entry for that document.

Example 7.1. Sample infrastructure log entry in Kibana

{
  "_index": "infra-000001",
  "_type": "_doc",
  "_id": "YmJmYTBlNDkZTRmLTliMGQtMjE3NmFiOGUyOWM3",
  "_version": 1,
  "_score": null,
  "_source": {
    "docker": {
      "container_id": "f85fa55bbef7bb783f041066be1e7c267a6b88c4603dfce213e32c1"
    },
    "kubernetes": {
      "container_name": "registry-server",
      "namespace_name": "openshift-marketplace",
      "pod_name": "redhat-marketplace-n64gc",
      "container_image": "registry.redhat.io/redhat/redhat-marketplace-index:v4.7",
      "container_image_id": "registry.redhat.io/redhat/redhat-marketplace-index@sha256:65fc0c45aabb95809e376feb065771ecda9e5e59cc8b3024c4545c168f",
      "pod_id": "8f594ea2-c866-4b5c-a1c8-a50756704b2a",
      "host": "ip-10-0-182-28.us-east-2.compute.internal",
      "master_url": "https://kubernetes.default.svc",
      "namespace_id": "3abab127-7669-4eb3-b9ef-44c04ad68d38",
      "namespace_labels": {
        "openshift_io/cluster-monitoring": "true"
      },
      "flat_labels": [
        "catalogsource_operators_coreos_com/update=redhat-marketplace"
      ]
    },
    "message": "time=\"2020-09-23T20:47:03Z\" level=info msg=\"serving registry\" database=/database/index.db port=50051",
    "level": "unknown",
    "hostname": "ip-10-0-182-28.internal",
    "pipeline_metadata": {
      "collector": {
        "ipaddr4": "10.0.182.28",
        "inputname": "fluent-plugin-systemd",
        "name": "fluentd",
        "received_at": "2020-09-23T20:47:15.007583+00:00",
        "version": "1.7.4 1.6.0"
      }
    },
    "@timestamp": "2020-09-23T20:47:03.422465+00:00",
    "viaq_msg_id": "YmJmYTBlNDktMDMGQtMjE3NmFiOGUyOWM3",
    "openshift": {
      "labels": {
        "logging": "infra"
      }
    }
  },
  "fields": {
    "@timestamp": [
      "2020-09-23T20:47:03.422Z"
    ],
    "pipeline_metadata.collector.received_at": [
      "2020-09-23T20:47:15.007Z"
    ]
  },
  "sort": [
    1600894023422
  ]
}

{
  "_index": "infra-000001",
  "_type": "_doc",
  "_id": "YmJmYTBlNDkZTRmLTliMGQtMjE3NmFiOGUyOWM3",
  "_version": 1,
  "_score": null,
  "_source": {
    "docker": {
      "container_id": "f85fa55bbef7bb783f041066be1e7c267a6b88c4603dfce213e32c1"
    },
    "kubernetes": {
      "container_name": "registry-server",
      "namespace_name": "openshift-marketplace",
      "pod_name": "redhat-marketplace-n64gc",
      "container_image": "registry.redhat.io/redhat/redhat-marketplace-index:v4.7",
      "container_image_id": "registry.redhat.io/redhat/redhat-marketplace-index@sha256:65fc0c45aabb95809e376feb065771ecda9e5e59cc8b3024c4545c168f",
      "pod_id": "8f594ea2-c866-4b5c-a1c8-a50756704b2a",
      "host": "ip-10-0-182-28.us-east-2.compute.internal",
      "master_url": "https://kubernetes.default.svc",
      "namespace_id": "3abab127-7669-4eb3-b9ef-44c04ad68d38",
      "namespace_labels": {
        "openshift_io/cluster-monitoring": "true"
      },
      "flat_labels": [
        "catalogsource_operators_coreos_com/update=redhat-marketplace"
      ]
    },
    "message": "time=\"2020-09-23T20:47:03Z\" level=info msg=\"serving registry\" database=/database/index.db port=50051",
    "level": "unknown",
    "hostname": "ip-10-0-182-28.internal",
    "pipeline_metadata": {
      "collector": {
        "ipaddr4": "10.0.182.28",
        "inputname": "fluent-plugin-systemd",
        "name": "fluentd",
        "received_at": "2020-09-23T20:47:15.007583+00:00",
        "version": "1.7.4 1.6.0"
      }
    },
    "@timestamp": "2020-09-23T20:47:03.422465+00:00",
    "viaq_msg_id": "YmJmYTBlNDktMDMGQtMjE3NmFiOGUyOWM3",
    "openshift": {
      "labels": {
        "logging": "infra"
      }
    }
  },
  "fields": {
    "@timestamp": [
      "2020-09-23T20:47:03.422Z"
    ],
    "pipeline_metadata.collector.received_at": [
      "2020-09-23T20:47:15.007Z"
    ]
  },
  "sort": [
    1600894023422
  ]
}

Copy to Clipboard

Toggle word wrap

7.4.3. Configuring Kibana
Copia collegamento

You can configure using the Kibana console by modifying the ClusterLogging custom resource (CR).

7.4.3.1. Configuring CPU and memory limits
Copia collegamento

The logging components allow for adjustments to both the CPU and memory limits.

Procedure

Edit the ClusterLogging custom resource (CR) in the openshift-logging project:

oc -n openshift-logging edit ClusterLogging instance

$ oc -n openshift-logging edit ClusterLogging instance

Copy to Clipboard

Toggle word wrap

apiVersion: "logging.openshift.io/v1"
kind: "ClusterLogging"
metadata:
  name: "instance"
  namespace: openshift-logging

...

spec:
  managementState: "Managed"
  logStore:
    type: "elasticsearch"
    elasticsearch:
      nodeCount: 3
      resources: 
        limits:
          memory: 16Gi
        requests:
          cpu: 200m
          memory: 16Gi
      storage:
        storageClassName: "gp2"
        size: "200G"
      redundancyPolicy: "SingleRedundancy"
  visualization:
    type: "kibana"
    kibana:
      resources: 
        limits:
          memory: 1Gi
        requests:
          cpu: 500m
          memory: 1Gi
      proxy:
        resources: 
          limits:
            memory: 100Mi
          requests:
            cpu: 100m
            memory: 100Mi
      replicas: 2
  collection:
    resources: 
      limits:
        memory: 736Mi
      requests:
        cpu: 200m
        memory: 736Mi
    type: fluentd

apiVersion: "logging.openshift.io/v1"
kind: "ClusterLogging"
metadata:
  name: "instance"
  namespace: openshift-logging

...

spec:
  managementState: "Managed"
  logStore:
    type: "elasticsearch"
    elasticsearch:
      nodeCount: 3
      resources:


        limits:
          memory: 16Gi
        requests:
          cpu: 200m
          memory: 16Gi
      storage:
        storageClassName: "gp2"
        size: "200G"
      redundancyPolicy: "SingleRedundancy"
  visualization:
    type: "kibana"
    kibana:
      resources:


        limits:
          memory: 1Gi
        requests:
          cpu: 500m
          memory: 1Gi
      proxy:
        resources:


          limits:
            memory: 100Mi
          requests:
            cpu: 100m
            memory: 100Mi
      replicas: 2
  collection:
    resources:


      limits:
        memory: 736Mi
      requests:
        cpu: 200m
        memory: 736Mi
    type: fluentd

Copy to Clipboard

Toggle word wrap

1: Specify the CPU and memory limits and requests for the log store as needed. For Elasticsearch, you must adjust both the request value and the limit value.
2 3: Specify the CPU and memory limits and requests for the log visualizer as needed.
4: Specify the CPU and memory limits and requests for the log collector as needed.

7.4.3.2. Scaling redundancy for the log visualizer nodes
Copia collegamento

You can scale the pod that hosts the log visualizer for redundancy.

Procedure

Edit the ClusterLogging custom resource (CR) in the openshift-logging project:

oc -n openshift-logging edit ClusterLogging instance

$ oc -n openshift-logging edit ClusterLogging instance

Copy to Clipboard

Toggle word wrap

$ oc edit ClusterLogging instance

apiVersion: "logging.openshift.io/v1"
kind: "ClusterLogging"
metadata:
  name: "instance"
  namespace: openshift-logging
....

spec:
  visualization:
    type: "kibana"
    kibana:
      replicas: 1

$ oc edit ClusterLogging instance

apiVersion: "logging.openshift.io/v1"
kind: "ClusterLogging"
metadata:
  name: "instance"
  namespace: openshift-logging
....

spec:
  visualization:
    type: "kibana"
    kibana:
      replicas: 1

Copy to Clipboard

Toggle word wrap

1: Specify the number of Kibana nodes.

Chapter 8. Configuring your Logging deployment
Copia collegamento

8.1. Configuring CPU and memory limits for logging components
Copia collegamento

You can configure both the CPU and memory limits for each of the logging components as needed.

8.1.1. Configuring CPU and memory limits
Copia collegamento

The logging components allow for adjustments to both the CPU and memory limits.

Procedure

Edit the ClusterLogging custom resource (CR) in the openshift-logging project:

oc -n openshift-logging edit ClusterLogging instance

$ oc -n openshift-logging edit ClusterLogging instance

Copy to Clipboard

Toggle word wrap

apiVersion: "logging.openshift.io/v1"
kind: "ClusterLogging"
metadata:
  name: "instance"
  namespace: openshift-logging

...

spec:
  managementState: "Managed"
  logStore:
    type: "elasticsearch"
    elasticsearch:
      nodeCount: 3
      resources: 
        limits:
          memory: 16Gi
        requests:
          cpu: 200m
          memory: 16Gi
      storage:
        storageClassName: "gp2"
        size: "200G"
      redundancyPolicy: "SingleRedundancy"
  visualization:
    type: "kibana"
    kibana:
      resources: 
        limits:
          memory: 1Gi
        requests:
          cpu: 500m
          memory: 1Gi
      proxy:
        resources: 
          limits:
            memory: 100Mi
          requests:
            cpu: 100m
            memory: 100Mi
      replicas: 2
  collection:
    resources: 
      limits:
        memory: 736Mi
      requests:
        cpu: 200m
        memory: 736Mi
    type: fluentd

apiVersion: "logging.openshift.io/v1"
kind: "ClusterLogging"
metadata:
  name: "instance"
  namespace: openshift-logging

...

spec:
  managementState: "Managed"
  logStore:
    type: "elasticsearch"
    elasticsearch:
      nodeCount: 3
      resources:


        limits:
          memory: 16Gi
        requests:
          cpu: 200m
          memory: 16Gi
      storage:
        storageClassName: "gp2"
        size: "200G"
      redundancyPolicy: "SingleRedundancy"
  visualization:
    type: "kibana"
    kibana:
      resources:


        limits:
          memory: 1Gi
        requests:
          cpu: 500m
          memory: 1Gi
      proxy:
        resources:


          limits:
            memory: 100Mi
          requests:
            cpu: 100m
            memory: 100Mi
      replicas: 2
  collection:
    resources:


      limits:
        memory: 736Mi
      requests:
        cpu: 200m
        memory: 736Mi
    type: fluentd

Copy to Clipboard

Toggle word wrap

1: Specify the CPU and memory limits and requests for the log store as needed. For Elasticsearch, you must adjust both the request value and the limit value.
2 3: Specify the CPU and memory limits and requests for the log visualizer as needed.
4: Specify the CPU and memory limits and requests for the log collector as needed.

8.2. Configuring systemd-journald and Fluentd
Copia collegamento

Because Fluentd reads from the journal, and the journal default settings are very low, journal entries can be lost because the journal cannot keep up with the logging rate from system services.

We recommend setting RateLimitIntervalSec=30s and RateLimitBurst=10000 (or even higher if necessary) to prevent the journal from losing entries.

8.2.1. Configuring systemd-journald for OpenShift Logging
Copia collegamento

As you scale up your project, the default logging environment might need some adjustments.

For example, if you are missing logs, you might have to increase the rate limits for journald. You can adjust the number of messages to retain for a specified period of time to ensure that OpenShift Logging does not use excessive resources without dropping logs.

You can also determine if you want the logs compressed, how long to retain logs, how or if the logs are stored, and other settings.

Procedure

Create a Butane config file, 40-worker-custom-journald.bu, that includes an /etc/systemd/journald.conf file with the required settings.
Note
The Butane version you specify in the config file should match the OpenShift Container Platform version and always ends in 0. For example, 4.12.0. See "Creating machine configs with Butane" for information about Butane.
```
variant: openshift
version: 4.12.0
metadata:
  name: 40-worker-custom-journald
  labels:
    machineconfiguration.openshift.io/role: "worker"
storage:
  files:
  - path: /etc/systemd/journald.conf
    mode: 0644 
    overwrite: true
    contents:
      inline: |
        Compress=yes 
        ForwardToConsole=no 
        ForwardToSyslog=no
        MaxRetentionSec=1month 
        RateLimitBurst=10000 
        RateLimitIntervalSec=30s
        Storage=persistent 
        SyncIntervalSec=1s 
        SystemMaxUse=8G 
        SystemKeepFree=20% 
        SystemMaxFileSize=10M 
```
```
variant: openshift
version: 4.12.0
metadata:
  name: 40-worker-custom-journald
  labels:
    machineconfiguration.openshift.io/role: "worker"
storage:
  files:
  - path: /etc/systemd/journald.conf
    mode: 0644 
```
1
```
    overwrite: true
    contents:
      inline: |
        Compress=yes 
```
2
```
        ForwardToConsole=no 
```
3
```
        ForwardToSyslog=no
        MaxRetentionSec=1month 
```
4
```
        RateLimitBurst=10000 
```
5
```
        RateLimitIntervalSec=30s
        Storage=persistent 
```
6
```
        SyncIntervalSec=1s 
```
7
```
        SystemMaxUse=8G 
```
8
```
        SystemKeepFree=20% 
```
9
```
        SystemMaxFileSize=10M 
```
10
Copy to Clipboard Toggle word wrap
1
Set the permissions for the journald.conf file. It is recommended to set 0644 permissions.
2
Specify whether you want logs compressed before they are written to the file system. Specify yes to compress the message or no to not compress. The default is yes.
3
Configure whether to forward log messages. Defaults to no for each. Specify:
ForwardToConsole to forward logs to the system console.
ForwardToKMsg to forward logs to the kernel log buffer.
ForwardToSyslog to forward to a syslog daemon.
ForwardToWall to forward messages as wall messages to all logged-in users.
4
Specify the maximum time to store journal entries. Enter a number to specify seconds. Or include a unit: "year", "month", "week", "day", "h" or "m". Enter 0 to disable. The default is 1month.
5
Configure rate limiting. If more logs are received than what is specified in RateLimitBurst during the time interval defined by RateLimitIntervalSec, all further messages within the interval are dropped until the interval is over. It is recommended to set RateLimitIntervalSec=30s and RateLimitBurst=10000, which are the defaults.
6
Specify how logs are stored. The default is persistent:
volatile to store logs in memory in /run/log/journal/. These logs are lost after rebooting.
persistent to store logs to disk in /var/log/journal/. systemd creates the directory if it does not exist.
auto to store logs in /var/log/journal/ if the directory exists. If it does not exist, systemd temporarily stores logs in /run/systemd/journal.
none to not store logs. systemd drops all logs.
7
Specify the timeout before synchronizing journal files to disk for ERR, WARNING, NOTICE, INFO, and DEBUG logs. systemd immediately syncs after receiving a CRIT, ALERT, or EMERG log. The default is 1s.
8
Specify the maximum size the journal can use. The default is 8G.
9
Specify how much disk space systemd must leave free. The default is 20%.
10
Specify the maximum size for individual journal files stored persistently in /var/log/journal. The default is 10M.
Note
If you are removing the rate limit, you might see increased CPU utilization on the system logging daemons as it processes any messages that would have previously been throttled.
For more information on systemd settings, see https://www.freedesktop.org/software/systemd/man/journald.conf.html. The default settings listed on that page might not apply to OpenShift Container Platform.
Use Butane to generate a MachineConfig object file, 40-worker-custom-journald.yaml, containing the configuration to be delivered to the nodes:
```
butane 40-worker-custom-journald.bu -o 40-worker-custom-journald.yaml
```
```
$ butane 40-worker-custom-journald.bu -o 40-worker-custom-journald.yaml
```
Copy to Clipboard Toggle word wrap
Apply the machine config. For example:
```
oc apply -f 40-worker-custom-journald.yaml
```
```
$ oc apply -f 40-worker-custom-journald.yaml
```
Copy to Clipboard Toggle word wrap
The controller detects the new MachineConfig object and generates a new rendered-worker-<hash> version.

Monitor the status of the rollout of the new rendered configuration to each node:

oc describe machineconfigpool/worker

$ oc describe machineconfigpool/worker

Copy to Clipboard

Toggle word wrap

Example output

Name:         worker
Namespace:
Labels:       machineconfiguration.openshift.io/mco-built-in=
Annotations:  <none>
API Version:  machineconfiguration.openshift.io/v1
Kind:         MachineConfigPool

...

Conditions:
  Message:
  Reason:                All nodes are updating to rendered-worker-913514517bcea7c93bd446f4830bc64e

Name:         worker
Namespace:
Labels:       machineconfiguration.openshift.io/mco-built-in=
Annotations:  <none>
API Version:  machineconfiguration.openshift.io/v1
Kind:         MachineConfigPool

...

Conditions:
  Message:
  Reason:                All nodes are updating to rendered-worker-913514517bcea7c93bd446f4830bc64e

Copy to Clipboard

Toggle word wrap

Chapter 9. Log collection and forwarding
Copia collegamento

9.1. About log collection and forwarding
Copia collegamento

The Red Hat OpenShift Logging Operator deploys a collector based on the ClusterLogForwarder resource specification. There are two collector options supported by this Operator: the legacy Fluentd collector, and the Vector collector.

Note

9.1.1. Log collection
Copia collegamento

The log collector is a daemon set that deploys pods to each OpenShift Container Platform node to collect container and node logs.

By default, the log collector uses the following sources:

System and infrastructure logs generated by journald log messages from the operating system, the container runtime, and OpenShift Container Platform.
/var/log/containers/*.log for all container logs.

If you configure the log collector to collect audit logs, it collects them from /var/log/audit/audit.log.

The log collector collects the logs from these sources and forwards them internally or externally depending on your logging configuration.

9.1.1.1. Log collector types
Copia collegamento

Vector is a log collector offered as an alternative to Fluentd for the logging.

You can configure which logging collector type your cluster uses by modifying the ClusterLogging custom resource (CR) collection spec:

Example ClusterLogging CR that configures Vector as the collector

apiVersion: logging.openshift.io/v1
kind: ClusterLogging
metadata:
  name: instance
  namespace: openshift-logging
spec:
  collection:
    logs:
      type: vector
      vector: {}
# ...

apiVersion: logging.openshift.io/v1
kind: ClusterLogging
metadata:
  name: instance
  namespace: openshift-logging
spec:
  collection:
    logs:
      type: vector
      vector: {}
# ...

Copy to Clipboard

Toggle word wrap

9.1.1.2. Log collection limitations
Copia collegamento

The container runtimes provide minimal information to identify the source of log messages: project, pod name, and container ID. This information is not sufficient to uniquely identify the source of the logs. If a pod with a given name and project is deleted before the log collector begins processing its logs, information from the API server, such as labels and annotations, might not be available. There might not be a way to distinguish the log messages from a similarly named pod and project or trace the logs to their source. This limitation means that log collection and normalization are considered best effort.

Important

The available container runtimes provide minimal information to identify the source of log messages and do not guarantee unique individual log messages or that these messages can be traced to their source.

9.1.1.3. Log collector features by type
Copia collegamento

Expand

Table 9.1. Log Sources
Feature	Fluentd	Vector
App container logs	✓	✓
App-specific routing	✓	✓
App-specific routing by namespace	✓	✓
Infra container logs	✓	✓
Infra journal logs	✓	✓
Kube API audit logs	✓	✓
OpenShift API audit logs	✓	✓
Open Virtual Network (OVN) audit logs	✓	✓

Expand

Table 9.2. Authorization and Authentication
Feature	Fluentd	Vector
Elasticsearch certificates	✓	✓
Elasticsearch username / password	✓	✓
Cloudwatch keys	✓	✓
Cloudwatch STS	✓	✓
Kafka certificates	✓	✓
Kafka username / password	✓	✓
Kafka SASL	✓	✓
Loki bearer token	✓	✓

Expand

Table 9.3. Normalizations and Transformations
Feature	Fluentd	Vector
Viaq data model - app	✓	✓
Viaq data model - infra	✓	✓
Viaq data model - infra(journal)	✓	✓
Viaq data model - Linux audit	✓	✓
Viaq data model - kube-apiserver audit	✓	✓
Viaq data model - OpenShift API audit	✓	✓
Viaq data model - OVN	✓	✓
Loglevel Normalization	✓	✓
JSON parsing	✓	✓
Structured Index	✓	✓
Multiline error detection	✓	✓
Multicontainer / split indices	✓	✓
Flatten labels	✓	✓
CLF static labels	✓	✓

Expand

Table 9.4. Tuning
Feature	Fluentd	Vector
Fluentd readlinelimit	✓
Fluentd buffer	✓
- chunklimitsize	✓
- totallimitsize	✓
- overflowaction	✓
- flushthreadcount	✓
- flushmode	✓
- flushinterval	✓
- retrywait	✓
- retrytype	✓
- retrymaxinterval	✓
- retrytimeout	✓

Expand

Table 9.5. Visibility
Feature	Fluentd	Vector
Metrics	✓	✓
Dashboard	✓	✓
Alerts	✓	✓

Expand

Table 9.6. Miscellaneous
Feature	Fluentd	Vector
Global proxy support	✓	✓
x86 support	✓	✓
ARM support	✓	✓
IBM Power support	✓	✓
IBM Z support	✓	✓
IPv6 support	✓	✓
Log event buffering	✓
Disconnected Cluster	✓	✓

9.1.1.4. Collector outputs
Copia collegamento

The following collector outputs are supported:

Expand

Table 9.7. Supported outputs
Feature	Fluentd	Vector
Elasticsearch v6-v8	✓	✓
Fluent forward	✓
Syslog RFC3164	✓	✓ (Logging 5.7+)
Syslog RFC5424	✓	✓ (Logging 5.7+)
Kafka	✓	✓
Cloudwatch	✓	✓
Cloudwatch STS	✓	✓
Loki	✓	✓
HTTP	✓	✓ (Logging 5.7+)
Google Cloud Logging	✓	✓
Splunk		✓ (Logging 5.6+)

9.1.2. Log forwarding
Copia collegamento

Administrators can create ClusterLogForwarder resources that specify which logs are collected, how they are transformed, and where they are forwarded to.

ClusterLogForwarder resources can be used up to forward container, infrastructure, and audit logs to specific endpoints within or outside of a cluster. Transport Layer Security (TLS) is supported so that log forwarders can be configured to send logs securely.

Administrators can also authorize RBAC permissions that define which service accounts and users can access and forward which types of logs.

9.1.2.1. Log forwarding implementations
Copia collegamento

There are two log forwarding implementations available: the legacy implementation, and the multi log forwarder feature.

Important

Only the Vector collector is supported for use with the multi log forwarder feature. The Fluentd collector can only be used with legacy implementations.

9.1.2.1.1. Legacy implementation
Copia collegamento

In legacy implementations, you can only use one log forwarder in your cluster. The ClusterLogForwarder resource in this mode must be named instance, and must be created in the openshift-logging namespace. The ClusterLogForwarder resource also requires a corresponding ClusterLogging resource named instance in the openshift-logging namespace.

9.1.2.1.2. Multi log forwarder feature
Copia collegamento

The multi log forwarder feature is available in logging 5.8 and later, and provides the following functionality:

Administrators can control which users are allowed to define log collection and which logs they are allowed to collect.
Users who have the required permissions are able to specify additional log collection configurations.
Administrators who are migrating from the deprecated Fluentd collector to the Vector collector can deploy a new log forwarder separately from their existing deployment. The existing and new log forwarders can operate simultaneously while workloads are being migrated.

In multi log forwarder implementations, you are not required to create a corresponding ClusterLogging resource for your ClusterLogForwarder resource. You can create multiple ClusterLogForwarder resources using any name, in any namespace, with the following exceptions:

You cannot create a ClusterLogForwarder resource named instance in the openshift-logging namespace, because this is reserved for a log forwarder that supports the legacy workflow using the Fluentd collector.
You cannot create a ClusterLogForwarder resource named collector in the openshift-logging namespace, because this is reserved for the collector.

9.1.2.2. Enabling the multi log forwarder feature for a cluster
Copia collegamento

To use the multi log forwarder feature, you must create a service account and cluster role bindings for that service account. You can then reference the service account in the ClusterLogForwarder resource to control access permissions.

Important

In order to support multi log forwarding in additional namespaces other than the openshift-logging namespace, you must update the Red Hat OpenShift Logging Operator to watch all namespaces. This functionality is supported by default in new Red Hat OpenShift Logging Operator version 5.8 installations.

9.1.2.2.1. Authorizing log collection RBAC permissions
Copia collegamento

In logging 5.8 and later, the Red Hat OpenShift Logging Operator provides collect-audit-logs, collect-application-logs, and collect-infrastructure-logs cluster roles, which enable the collector to collect audit logs, application logs, and infrastructure logs respectively.

You can authorize RBAC permissions for log collection by binding the required cluster roles to a service account.

Prerequisites

The Red Hat OpenShift Logging Operator is installed in the openshift-logging namespace.
You have administrator permissions.

Procedure

Create a service account for the collector. If you want to write logs to storage that requires a token for authentication, you must include a token in the service account.

Bind the appropriate cluster roles to the service account:

Example binding command

oc adm policy add-cluster-role-to-user <cluster_role_name> system:serviceaccount:<namespace_name>:<service_account_name>

$ oc adm policy add-cluster-role-to-user <cluster_role_name> system:serviceaccount:<namespace_name>:<service_account_name>

Copy to Clipboard

Toggle word wrap

9.2. Log output types
Copia collegamento

Outputs define the destination where logs are sent to from a log forwarder. You can configure multiple types of outputs in the ClusterLogForwarder custom resource (CR) to send logs to servers that support different protocols.

9.2.1. Supported log forwarding outputs
Copia collegamento

Outputs can be any of the following types:

Expand

Table 9.8. Supported log output types
Output type	Protocol	Tested with	Logging versions	Supported collector type
Elasticsearch v6	HTTP 1.1	6.8.1, 6.8.23	5.6+	Fluentd, Vector
Elasticsearch v7	HTTP 1.1	7.12.2, 7.17.7, 7.10.1	5.6+	Fluentd, Vector
Elasticsearch v8	HTTP 1.1	8.4.3, 8.6.1	5.6+	Fluentd ^[1], Vector
Fluent Forward	Fluentd forward v1	Fluentd 1.14.6, Logstash 7.10.1, Fluentd 1.14.5	5.4+	Fluentd
Google Cloud Logging	REST over HTTPS	Latest	5.7+	Vector
HTTP	HTTP 1.1	Fluentd 1.14.6, Vector 0.21	5.7+	Fluentd, Vector
Kafka	Kafka 0.11	Kafka 2.4.1, 2.7.0, 3.3.1	5.4+	Fluentd, Vector
Loki	REST over HTTP and HTTPS	2.3.0, 2.5.0, 2.7, 2.2.1	5.4+	Fluentd, Vector
Splunk	HEC	8.2.9, 9.0.0	5.7+	Vector
Syslog	RFC3164, RFC5424	Rsyslog 8.37.0-9.el7, rsyslog-8.39.0	5.4+	Fluentd, Vector ^[2]
Amazon CloudWatch	REST over HTTPS	Latest	5.4+	Fluentd, Vector

Fluentd does not support Elasticsearch 8 in the logging version 5.6.2.
Vector supports Syslog in the logging version 5.7 and higher.

9.2.2. Output type descriptions
Copia collegamento

default: The on-cluster, Red Hat managed log store. You are not required to configure the default output.
Note
If you configure a default output, you receive an error message, because the default output name is reserved for referencing the on-cluster, Red Hat managed log store.
loki: Loki, a horizontally scalable, highly available, multi-tenant log aggregation system.
kafka: A Kafka broker. The kafka output can use a TCP or TLS connection.
elasticsearch: An external Elasticsearch instance. The elasticsearch output can use a TLS connection.
fluentdForward: An external log aggregation solution that supports Fluentd. This option uses the Fluentd forward protocols. The fluentForward output can use a TCP or TLS connection and supports shared-key authentication by providing a shared_key field in a secret. Shared-key authentication can be used with or without TLS.
Important
The fluentdForward output is only supported if you are using the Fluentd collector. It is not supported if you are using the Vector collector. If you are using the Vector collector, you can forward logs to Fluentd by using the http output.
syslog: An external log aggregation solution that supports the syslog RFC3164 or RFC5424 protocols. The syslog output can use a UDP, TCP, or TLS connection.
cloudwatch: Amazon CloudWatch, a monitoring and log storage service hosted by Amazon Web Services (AWS).

9.3. Enabling JSON log forwarding
Copia collegamento

You can configure the Log Forwarding API to parse JSON strings into a structured object.

9.3.1. Parsing JSON logs
Copia collegamento

You can use a ClusterLogForwarder object to parse JSON logs into a structured object and forward them to a supported output.

To illustrate how this works, suppose that you have the following structured JSON log entry:

Example structured JSON log entry

{"level":"info","name":"fred","home":"bedrock"}

{"level":"info","name":"fred","home":"bedrock"}

Copy to Clipboard

Toggle word wrap

To enable parsing JSON log, you add parse: json to a pipeline in the ClusterLogForwarder CR, as shown in the following example:

Example snippet showing parse: json

pipelines:
- inputRefs: [ application ]
  outputRefs: myFluentd
  parse: json

pipelines:
- inputRefs: [ application ]
  outputRefs: myFluentd
  parse: json

Copy to Clipboard

Toggle word wrap

When you enable parsing JSON logs by using parse: json, the CR copies the JSON-structured log entry in a structured field, as shown in the following example:

Example structured output containing the structured JSON log entry

{"structured": { "level": "info", "name": "fred", "home": "bedrock" },
 "more fields..."}

{"structured": { "level": "info", "name": "fred", "home": "bedrock" },
 "more fields..."}

Copy to Clipboard

Toggle word wrap

Important

If the log entry does not contain valid structured JSON, the structured field is absent.

9.3.2. Configuring JSON log data for Elasticsearch
Copia collegamento

If your JSON logs follow more than one schema, storing them in a single index might cause type conflicts and cardinality problems. To avoid that, you must configure the ClusterLogForwarder custom resource (CR) to group each schema into a single output definition. This way, each schema is forwarded to a separate index.

Important

If you forward JSON logs to the default Elasticsearch instance managed by OpenShift Logging, it generates new indices based on your configuration. To avoid performance issues associated with having too many indices, consider keeping the number of possible schemas low by standardizing to common schemas.

Structure types

You can use the following structure types in the ClusterLogForwarder CR to construct index names for the Elasticsearch log store:

structuredTypeKey is the name of a message field. The value of that field is used to construct the index name.
- kubernetes.labels.<key> is the Kubernetes pod label whose value is used to construct the index name.
- openshift.labels.<key> is the pipeline.label.<key> element in the ClusterLogForwarder CR whose value is used to construct the index name.
- kubernetes.container_name uses the container name to construct the index name.
structuredTypeName: If the structuredTypeKey field is not set or its key is not present, the structuredTypeName value is used as the structured type. When you use both the structuredTypeKey field and the structuredTypeName field together, the structuredTypeName value provides a fallback index name if the key in the structuredTypeKey field is missing from the JSON log data.

Note

Although you can set the value of structuredTypeKey to any field shown in the "Log Record Fields" topic, the most useful fields are shown in the preceding list of structure types.

A structuredTypeKey: kubernetes.labels.<key> example

Suppose the following:

Your cluster is running application pods that produce JSON logs in two different formats, "apache" and "google".
The user labels these application pods with logFormat=apache and logFormat=google.
You use the following snippet in your ClusterLogForwarder CR YAML file.

apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
# ...
spec:
# ...
  outputDefaults:
    elasticsearch:
      structuredTypeKey: kubernetes.labels.logFormat 
      structuredTypeName: nologformat
  pipelines:
  - inputRefs:
    - application
    outputRefs:
    - default
    parse: json

apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
# ...
spec:
# ...
  outputDefaults:
    elasticsearch:
      structuredTypeKey: kubernetes.labels.logFormat


      structuredTypeName: nologformat
  pipelines:
  - inputRefs:
    - application
    outputRefs:
    - default
    parse: json

Copy to Clipboard

Toggle word wrap

1: Uses the value of the key-value pair that is formed by the Kubernetes logFormat label.
2: Enables parsing JSON logs.

In that case, the following structured log record goes to the app-apache-write index:

{
  "structured":{"name":"fred","home":"bedrock"},
  "kubernetes":{"labels":{"logFormat": "apache", ...}}
}

{
  "structured":{"name":"fred","home":"bedrock"},
  "kubernetes":{"labels":{"logFormat": "apache", ...}}
}

Copy to Clipboard

Toggle word wrap

And the following structured log record goes to the app-google-write index:

{
  "structured":{"name":"wilma","home":"bedrock"},
  "kubernetes":{"labels":{"logFormat": "google", ...}}
}

{
  "structured":{"name":"wilma","home":"bedrock"},
  "kubernetes":{"labels":{"logFormat": "google", ...}}
}

Copy to Clipboard

Toggle word wrap

A structuredTypeKey: openshift.labels.<key> example

Suppose that you use the following snippet in your ClusterLogForwarder CR YAML file.

outputDefaults:
 elasticsearch:
    structuredTypeKey: openshift.labels.myLabel 
    structuredTypeName: nologformat
pipelines:
 - name: application-logs
   inputRefs:
   - application
   - audit
   outputRefs:
   - elasticsearch-secure
   - default
   parse: json
   labels:
     myLabel: myValue

outputDefaults:
 elasticsearch:
    structuredTypeKey: openshift.labels.myLabel


    structuredTypeName: nologformat
pipelines:
 - name: application-logs
   inputRefs:
   - application
   - audit
   outputRefs:
   - elasticsearch-secure
   - default
   parse: json
   labels:
     myLabel: myValue

Copy to Clipboard

Toggle word wrap

1: Uses the value of the key-value pair that is formed by the OpenShift myLabel label.
2: The myLabel element gives its string value, myValue, to the structured log record.

In that case, the following structured log record goes to the app-myValue-write index:

{
  "structured":{"name":"fred","home":"bedrock"},
  "openshift":{"labels":{"myLabel": "myValue", ...}}
}

{
  "structured":{"name":"fred","home":"bedrock"},
  "openshift":{"labels":{"myLabel": "myValue", ...}}
}

Copy to Clipboard

Toggle word wrap

Additional considerations

The Elasticsearch index for structured records is formed by prepending "app-" to the structured type and appending "-write".
Unstructured records are not sent to the structured index. They are indexed as usual in the application, infrastructure, or audit indices.
If there is no non-empty structured type, forward an unstructured record with no structured field.

It is important not to overload Elasticsearch with too many indices. Only use distinct structured types for distinct log formats, not for each application or namespace. For example, most Apache applications use the same JSON log format and structured type, such as LogApache.

9.3.3. Forwarding JSON logs to the Elasticsearch log store
Copia collegamento

For an Elasticsearch log store, if your JSON log entries follow different schemas, configure the ClusterLogForwarder custom resource (CR) to group each JSON schema into a single output definition. This way, Elasticsearch uses a separate index for each schema.

Important

Because forwarding different schemas to the same index can cause type conflicts and cardinality problems, you must perform this configuration before you forward data to the Elasticsearch store.

To avoid performance issues associated with having too many indices, consider keeping the number of possible schemas low by standardizing to common schemas.

Procedure

Add the following snippet to your ClusterLogForwarder CR YAML file.

outputDefaults:
 elasticsearch:
    structuredTypeKey: <log record field>
    structuredTypeName: <name>
pipelines:
- inputRefs:
  - application
  outputRefs: default
  parse: json

outputDefaults:
 elasticsearch:
    structuredTypeKey: <log record field>
    structuredTypeName: <name>
pipelines:
- inputRefs:
  - application
  outputRefs: default
  parse: json

Copy to Clipboard

Toggle word wrap

Use structuredTypeKey field to specify one of the log record fields.
Use structuredTypeName field to specify a name.
Important
To parse JSON logs, you must set both the structuredTypeKey and structuredTypeName fields.
For inputRefs, specify which log types to forward by using that pipeline, such as application, infrastructure, or audit.
Add the parse: json element to pipelines.
Create the CR object:
```
oc create -f <filename>.yaml
```
```
$ oc create -f <filename>.yaml
```
Copy to Clipboard Toggle word wrap
The Red Hat OpenShift Logging Operator redeploys the collector pods. However, if they do not redeploy, delete the collector pods to force them to redeploy.
```
oc delete pod --selector logging-infra=collector
```
```
$ oc delete pod --selector logging-infra=collector
```
Copy to Clipboard Toggle word wrap

9.3.4. Forwarding JSON logs from containers in the same pod to separate indices
Copia collegamento

You can forward structured logs from different containers within the same pod to different indices. To use this feature, you must configure the pipeline with multi-container support and annotate the pods. Logs are written to indices with a prefix of app-. It is recommended that Elasticsearch be configured with aliases to accommodate this.

Important

Prerequisites

Logging for Red Hat OpenShift: 5.5

Procedure

Create or edit a YAML file that defines the ClusterLogForwarder CR object:

apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  outputDefaults:
    elasticsearch:
      structuredTypeKey: kubernetes.labels.logFormat 
      structuredTypeName: nologformat
      enableStructuredContainerLogs: true 
  pipelines:
  - inputRefs:
    - application
    name: application-logs
    outputRefs:
    - default
    parse: json

apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  outputDefaults:
    elasticsearch:
      structuredTypeKey: kubernetes.labels.logFormat


      structuredTypeName: nologformat
      enableStructuredContainerLogs: true


  pipelines:
  - inputRefs:
    - application
    name: application-logs
    outputRefs:
    - default
    parse: json

Copy to Clipboard

Toggle word wrap

1: Uses the value of the key-value pair that is formed by the Kubernetes logFormat label.
2: Enables multi-container outputs.

Create or edit a YAML file that defines the Pod CR object:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    containerType.logging.openshift.io/heavy: heavy 
    containerType.logging.openshift.io/low: low
spec:
  containers:
  - name: heavy 
    image: heavyimage
  - name: low
    image: lowimage

apiVersion: v1
kind: Pod
metadata:
  annotations:
    containerType.logging.openshift.io/heavy: heavy


    containerType.logging.openshift.io/low: low
spec:
  containers:
  - name: heavy


    image: heavyimage
  - name: low
    image: lowimage

Copy to Clipboard

Toggle word wrap

1: Format: containerType.logging.openshift.io/<container-name>: <index>
2: Annotation names must match container names

Warning

This configuration might significantly increase the number of shards on the cluster.

Additional resources

Kubernetes Annotations

9.4. Configuring log forwarding
Copia collegamento

In a logging deployment, container and infrastructure logs are forwarded to the internal log store defined in the ClusterLogging custom resource (CR) by default.

Audit logs are not forwarded to the internal log store by default because this does not provide secure storage. You are responsible for ensuring that the system to which you forward audit logs is compliant with your organizational and governmental regulations, and is properly secured.

If this default configuration meets your needs, you do not need to configure a ClusterLogForwarder CR. If a ClusterLogForwarder CR exists, logs are not forwarded to the internal log store unless a pipeline is defined that contains the default output.

9.4.1. About forwarding logs to third-party systems
Copia collegamento

To send logs to specific endpoints inside and outside your OpenShift Container Platform cluster, you specify a combination of outputs and pipelines in a ClusterLogForwarder custom resource (CR). You can also use inputs to forward the application logs associated with a specific project to an endpoint. Authentication is provided by a Kubernetes Secret object.

pipeline

Defines simple routing from one log type to one or more outputs, or which logs you want to send. The log types are one of the following:

application. Container logs generated by user applications running in the cluster, except infrastructure container applications.
infrastructure. Container logs from pods that run in the openshift*, kube*, or default projects and journal logs sourced from node file system.
audit. Audit logs generated by the node audit system, auditd, Kubernetes API server, OpenShift API server, and OVN network.

You can add labels to outbound log messages by using key:value pairs in the pipeline. For example, you might add a label to messages that are forwarded to other data centers or label the logs by type. Labels that are added to objects are also forwarded with the log message.

input

Forwards the application logs associated with a specific project to a pipeline.

In the pipeline, you define which log types to forward using an inputRef parameter and where to forward the logs to using an outputRef parameter.

Secret

A key:value map that contains confidential data such as user credentials.

Note the following:

If you do not define a pipeline for a log type, the logs of the undefined types are dropped. For example, if you specify a pipeline for the application and audit types, but do not specify a pipeline for the infrastructure type, infrastructure logs are dropped.
You can use multiple types of outputs in the ClusterLogForwarder custom resource (CR) to send logs to servers that support different protocols.

The following example forwards the audit logs to a secure external Elasticsearch instance, the infrastructure logs to an insecure external Elasticsearch instance, the application logs to a Kafka broker, and the application logs from the my-apps-logs project to the internal Elasticsearch instance.

Sample log forwarding outputs and pipelines

apiVersion: "logging.openshift.io/v1"
kind: ClusterLogForwarder
metadata:
  name: <log_forwarder_name> 
  namespace: <log_forwarder_namespace> 
spec:
  serviceAccountName: <service_account_name> 
  outputs:
   - name: elasticsearch-secure 
     type: "elasticsearch"
     url: https://elasticsearch.secure.com:9200
     secret:
        name: elasticsearch
   - name: elasticsearch-insecure 
     type: "elasticsearch"
     url: http://elasticsearch.insecure.com:9200
   - name: kafka-app 
     type: "kafka"
     url: tls://kafka.secure.com:9093/app-topic
  inputs: 
   - name: my-app-logs
     application:
        namespaces:
        - my-project
  pipelines:
   - name: audit-logs 
     inputRefs:
      - audit
     outputRefs:
      - elasticsearch-secure
      - default
     labels:
       secure: "true" 
       datacenter: "east"
   - name: infrastructure-logs 
     inputRefs:
      - infrastructure
     outputRefs:
      - elasticsearch-insecure
     labels:
       datacenter: "west"
   - name: my-app 
     inputRefs:
      - my-app-logs
     outputRefs:
      - default
   - inputRefs: 
      - application
     outputRefs:
      - kafka-app
     labels:
       datacenter: "south"

apiVersion: "logging.openshift.io/v1"
kind: ClusterLogForwarder
metadata:
  name: <log_forwarder_name>


  namespace: <log_forwarder_namespace>


spec:
  serviceAccountName: <service_account_name>


  outputs:
   - name: elasticsearch-secure


     type: "elasticsearch"
     url: https://elasticsearch.secure.com:9200
     secret:
        name: elasticsearch
   - name: elasticsearch-insecure


     type: "elasticsearch"
     url: http://elasticsearch.insecure.com:9200
   - name: kafka-app


     type: "kafka"
     url: tls://kafka.secure.com:9093/app-topic
  inputs:


   - name: my-app-logs
     application:
        namespaces:
        - my-project
  pipelines:
   - name: audit-logs


     inputRefs:
      - audit
     outputRefs:
      - elasticsearch-secure
      - default
     labels:
       secure: "true"


       datacenter: "east"
   - name: infrastructure-logs


     inputRefs:
      - infrastructure
     outputRefs:
      - elasticsearch-insecure
     labels:
       datacenter: "west"
   - name: my-app


     inputRefs:
      - my-app-logs
     outputRefs:
      - default
   - inputRefs:


      - application
     outputRefs:
      - kafka-app
     labels:
       datacenter: "south"

Copy to Clipboard

Toggle word wrap

In legacy implementations, the CR name must be instance. In multi log forwarder implementations, you can use any name.

In legacy implementations, the CR namespace must be openshift-logging. In multi log forwarder implementations, you can use any namespace.

The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the openshift-logging namespace.

Configuration for an secure Elasticsearch output using a secret with a secure URL.

A name to describe the output.
The type of output: elasticsearch.
The secure URL and port of the Elasticsearch instance as a valid absolute URL, including the prefix.
The secret required by the endpoint for TLS communication. The secret must exist in the openshift-logging project.

Configuration for an insecure Elasticsearch output:

A name to describe the output.
The type of output: elasticsearch.
The insecure URL and port of the Elasticsearch instance as a valid absolute URL, including the prefix.

Configuration for a Kafka output using a client-authenticated TLS communication over a secure URL:

A name to describe the output.
The type of output: kafka.
Specify the URL and port of the Kafka broker as a valid absolute URL, including the prefix.

Configuration for an input to filter application logs from the my-project namespace.

Configuration for a pipeline to send audit logs to the secure external Elasticsearch instance:

A name to describe the pipeline.
The inputRefs is the log type, in this example audit.
The outputRefs is the name of the output to use, in this example elasticsearch-secure to forward to the secure Elasticsearch instance and default to forward to the internal Elasticsearch instance.
Optional: Labels to add to the logs.

Optional: String. One or more labels to add to the logs. Quote values like "true" so they are recognized as string values, not as a boolean.

Configuration for a pipeline to send infrastructure logs to the insecure external Elasticsearch instance.

Configuration for a pipeline to send logs from the my-project project to the internal Elasticsearch instance.

A name to describe the pipeline.
The inputRefs is a specific input: my-app-logs.
The outputRefs is default.
Optional: String. One or more labels to add to the logs.

Configuration for a pipeline to send logs to the Kafka broker, with no pipeline name:

The inputRefs is the log type, in this example application.
The outputRefs is the name of the output to use.
Optional: String. One or more labels to add to the logs.

Fluentd log handling when the external log aggregator is unavailable

If your external logging aggregator becomes unavailable and cannot receive logs, Fluentd continues to collect logs and stores them in a buffer. When the log aggregator becomes available, log forwarding resumes, including the buffered logs. If the buffer fills completely, Fluentd stops collecting logs. OpenShift Container Platform rotates the logs and deletes them. You cannot adjust the buffer size or add a persistent volume claim (PVC) to the Fluentd daemon set or pods.

Supported Authorization Keys

Common key types are provided here. Some output types support additional specialized keys, documented with the output-specific configuration field. All secret keys are optional. Enable the security features you want by setting the relevant keys. You are responsible for creating and maintaining any additional configurations that external destinations might require, such as keys and secrets, service accounts, port openings, or global proxy configuration. Open Shift Logging will not attempt to verify a mismatch between authorization combinations.

Transport Layer Security (TLS)

Using a TLS URL (http://... or ssl://...) without a secret enables basic TLS server-side authentication. Additional TLS features are enabled by including a secret and setting the following optional fields:

passphrase: (string) Passphrase to decode an encoded TLS private key. Requires tls.key.
ca-bundle.crt: (string) File name of a customer CA for server authentication.

Username and Password

username: (string) Authentication user name. Requires password.
password: (string) Authentication password. Requires username.

Simple Authentication Security Layer (SASL)

sasl.enable (boolean) Explicitly enable or disable SASL. If missing, SASL is automatically enabled when any of the other sasl. keys are set.
sasl.mechanisms: (array) List of allowed SASL mechanism names. If missing or empty, the system defaults are used.
sasl.allow-insecure: (boolean) Allow mechanisms that send clear-text passwords. Defaults to false.

9.4.1.1. Creating a Secret
Copia collegamento

You can create a secret in the directory that contains your certificate and key files by using the following command:

oc create secret generic -n <namespace> <secret_name> \
  --from-file=ca-bundle.crt=<your_bundle_file> \
  --from-literal=username=<your_username> \
  --from-literal=password=<your_password>

$ oc create secret generic -n <namespace> <secret_name> \
  --from-file=ca-bundle.crt=<your_bundle_file> \
  --from-literal=username=<your_username> \
  --from-literal=password=<your_password>

Copy to Clipboard

Toggle word wrap

Note

Generic or opaque secrets are recommended for best results.

9.4.2. Creating a log forwarder
Copia collegamento

To create a log forwarder, you must create a ClusterLogForwarder CR that specifies the log input types that the service account can collect. You can also specify which outputs the logs can be forwarded to. If you are using the multi log forwarder feature, you must also reference the service account in the ClusterLogForwarder CR.

If you are using the multi log forwarder feature on your cluster, you can create ClusterLogForwarder custom resources (CRs) in any namespace, using any name. If you are using a legacy implementation, the ClusterLogForwarder CR must be named instance, and must be created in the openshift-logging namespace.

Important

You need administrator permissions for the namespace where you create the ClusterLogForwarder CR.

ClusterLogForwarder resource example

apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: <log_forwarder_name> 
  namespace: <log_forwarder_namespace> 
spec:
  serviceAccountName: <service_account_name> 
  pipelines:
   - inputRefs:
     - <log_type> 
     outputRefs:
     - <output_name> 
  outputs:
  - name: <output_name> 
    type: <output_type> 
    url: <log_output_url> 
# ...

apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: <log_forwarder_name>


  namespace: <log_forwarder_namespace>


spec:
  serviceAccountName: <service_account_name>


  pipelines:
   - inputRefs:
     - <log_type>


     outputRefs:
     - <output_name>


  outputs:
  - name: <output_name>


    type: <output_type>


    url: <log_output_url>


# ...

Copy to Clipboard

Toggle word wrap

1: In legacy implementations, the CR name must be instance. In multi log forwarder implementations, you can use any name.
2: In legacy implementations, the CR namespace must be openshift-logging. In multi log forwarder implementations, you can use any namespace.
3: The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the openshift-logging namespace.
4: The log types that are collected. The value for this field can be audit for audit logs, application for application logs, infrastructure for infrastructure logs, or a named input that has been defined for your application.
5 7: The type of output that you want to forward logs to. The value of this field can be default, loki, kafka, elasticsearch, fluentdForward, syslog, or cloudwatch.
Note
The default output type is not supported in mutli log forwarder implementations.
6: A name for the output that you want to forward logs to.
8: The URL of the output that you want to forward logs to.

9.4.3. Enabling multi-line exception detection
Copia collegamento

Enables multi-line error detection of container logs.

Warning

Enabling this feature could have performance implications and may require additional computing resources or alternate logging solutions.

Log parsers often incorrectly identify separate lines of the same exception as separate exceptions. This leads to extra log entries and an incomplete or inaccurate view of the traced information.

Example java exception

java.lang.NullPointerException: Cannot invoke "String.toString()" because "<param1>" is null
    at testjava.Main.handle(Main.java:47)
    at testjava.Main.printMe(Main.java:19)
    at testjava.Main.main(Main.java:10)

java.lang.NullPointerException: Cannot invoke "String.toString()" because "<param1>" is null
    at testjava.Main.handle(Main.java:47)
    at testjava.Main.printMe(Main.java:19)
    at testjava.Main.main(Main.java:10)

Copy to Clipboard

Toggle word wrap

To enable logging to detect multi-line exceptions and reassemble them into a single log entry, ensure that the ClusterLogForwarder Custom Resource (CR) contains a detectMultilineErrors field, with a value of true.

Example ClusterLogForwarder CR

apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  pipelines:
    - name: my-app-logs
      inputRefs:
        - application
      outputRefs:
        - default
      detectMultilineErrors: true

apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  pipelines:
    - name: my-app-logs
      inputRefs:
        - application
      outputRefs:
        - default
      detectMultilineErrors: true

Copy to Clipboard

Toggle word wrap

9.4.3.1. Details
Copia collegamento

When log messages appear as a consecutive sequence forming an exception stack trace, they are combined into a single, unified log record. The first log message’s content is replaced with the concatenated content of all the message fields in the sequence.

Expand

Table 9.9. Supported languages per collector:
Language	Fluentd	Vector
Java	✓	✓
JS	✓	✓
Ruby	✓	✓
Python	✓	✓
Golang	✓	✓
PHP	✓	✓
Dart	✓	✓

9.4.3.2. Troubleshooting
Copia collegamento

When enabled, the collector configuration will include a new section with type: detect_exceptions

Example vector configuration section

[transforms.detect_exceptions_app-logs]
 type = "detect_exceptions"
 inputs = ["application"]
 languages = ["All"]
 group_by = ["kubernetes.namespace_name","kubernetes.pod_name","kubernetes.container_name"]
 expire_after_ms = 2000
 multiline_flush_interval_ms = 1000

[transforms.detect_exceptions_app-logs]
 type = "detect_exceptions"
 inputs = ["application"]
 languages = ["All"]
 group_by = ["kubernetes.namespace_name","kubernetes.pod_name","kubernetes.container_name"]
 expire_after_ms = 2000
 multiline_flush_interval_ms = 1000

Copy to Clipboard

Toggle word wrap

Example fluentd config section

<label @MULTILINE_APP_LOGS>
  <match kubernetes.**>
    @type detect_exceptions
    remove_tag_prefix 'kubernetes'
    message message
    force_line_breaks true
    multiline_flush_interval .2
  </match>
</label>

<label @MULTILINE_APP_LOGS>
  <match kubernetes.**>
    @type detect_exceptions
    remove_tag_prefix 'kubernetes'
    message message
    force_line_breaks true
    multiline_flush_interval .2
  </match>
</label>

Copy to Clipboard

Toggle word wrap

9.4.4. Forwarding logs to Google Cloud Platform (GCP)
Copia collegamento

You can forward logs to Google Cloud Logging in addition to, or instead of, the internal default OpenShift Container Platform log store.

Note

Using this feature with Fluentd is not supported.

Prerequisites

Red Hat OpenShift Logging Operator 5.5.1 and later

Procedure

Create a secret using your Google service account key.

oc -n openshift-logging create secret generic gcp-secret --from-file google-application-credentials.json=<your_service_account_key_file.json>

$ oc -n openshift-logging create secret generic gcp-secret --from-file google-application-credentials.json=<your_service_account_key_file.json>

Copy to Clipboard

Toggle word wrap

Create a ClusterLogForwarder Custom Resource YAML using the template below:

apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: <log_forwarder_name> 
  namespace: <log_forwarder_namespace> 
spec:
  serviceAccountName: <service_account_name> 
  outputs:
    - name: gcp-1
      type: googleCloudLogging
      secret:
        name: gcp-secret
      googleCloudLogging:
        projectId : "openshift-gce-devel" 
        logId : "app-gcp" 
  pipelines:
    - name: test-app
      inputRefs: 
        - application
      outputRefs:
        - gcp-1

apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: <log_forwarder_name>


  namespace: <log_forwarder_namespace>


spec:
  serviceAccountName: <service_account_name>


  outputs:
    - name: gcp-1
      type: googleCloudLogging
      secret:
        name: gcp-secret
      googleCloudLogging:
        projectId : "openshift-gce-devel"


        logId : "app-gcp"


  pipelines:
    - name: test-app
      inputRefs:


        - application
      outputRefs:
        - gcp-1

Copy to Clipboard

Toggle word wrap

1: In legacy implementations, the CR name must be instance. In multi log forwarder implementations, you can use any name.
2: In legacy implementations, the CR namespace must be openshift-logging. In multi log forwarder implementations, you can use any namespace.
3: The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the openshift-logging namespace.
4: Set a projectId, folderId, organizationId, or billingAccountId field and its corresponding value, depending on where you want to store your logs in the GCP resource hierarchy.
5: Set the value to add to the logName field of the Log Entry.
6: Specify which log types to forward by using the pipeline: application, infrastructure, or audit.

9.4.5. Forwarding logs to Splunk
Copia collegamento

You can forward logs to the Splunk HTTP Event Collector (HEC) in addition to, or instead of, the internal default OpenShift Container Platform log store.

Note

Using this feature with Fluentd is not supported.

Prerequisites

Red Hat OpenShift Logging Operator 5.6 or later
A ClusterLogging instance with vector specified as the collector
Base64 encoded Splunk HEC token

Procedure

Create a secret using your Base64 encoded Splunk HEC token.

oc -n openshift-logging create secret generic vector-splunk-secret --from-literal hecToken=<HEC_Token>

$ oc -n openshift-logging create secret generic vector-splunk-secret --from-literal hecToken=<HEC_Token>

Copy to Clipboard

Toggle word wrap

Create or edit the ClusterLogForwarder Custom Resource (CR) using the template below:

apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: <log_forwarder_name> 
  namespace: <log_forwarder_namespace> 
spec:
  serviceAccountName: <service_account_name> 
  outputs:
    - name: splunk-receiver 
      secret:
        name: vector-splunk-secret 
      type: splunk 
      url: <http://your.splunk.hec.url:8088> 
  pipelines: 
    - inputRefs:
        - application
        - infrastructure
      name: 
      outputRefs:
        - splunk-receiver

apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: <log_forwarder_name>


  namespace: <log_forwarder_namespace>


spec:
  serviceAccountName: <service_account_name>


  outputs:
    - name: splunk-receiver


      secret:
        name: vector-splunk-secret


      type: splunk


      url: <http://your.splunk.hec.url:8088>


  pipelines:


    - inputRefs:
        - application
        - infrastructure
      name:


      outputRefs:
        - splunk-receiver

Copy to Clipboard

Toggle word wrap

1: In legacy implementations, the CR name must be instance. In multi log forwarder implementations, you can use any name.
2: In legacy implementations, the CR namespace must be openshift-logging. In multi log forwarder implementations, you can use any namespace.
3: The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the openshift-logging namespace.
4: Specify a name for the output.
5: Specify the name of the secret that contains your HEC token.
6: Specify the output type as splunk.
7: Specify the URL (including port) of your Splunk HEC.
8: Specify which log types to forward by using the pipeline: application, infrastructure, or audit.
9: Optional: Specify a name for the pipeline.
10: Specify the name of the output to use when forwarding logs with this pipeline.

9.4.6. Forwarding logs over HTTP
Copia collegamento

Forwarding logs over HTTP is supported for both the Fluentd and Vector log collectors. To enable, specify http as the output type in the ClusterLogForwarder custom resource (CR).

Procedure

Create or edit the ClusterLogForwarder CR using the template below:

Example ClusterLogForwarder CR

apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: <log_forwarder_name> 
  namespace: <log_forwarder_namespace> 
spec:
  serviceAccountName: <service_account_name> 
  outputs:
    - name: httpout-app
      type: http
      url: 
      http:
        headers: 
          h1: v1
          h2: v2
        method: POST
      secret:
        name: 
      tls:
        insecureSkipVerify: 
  pipelines:
    - name:
      inputRefs:
        - application
      outputRefs:
        - httpout-app

apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: <log_forwarder_name>


  namespace: <log_forwarder_namespace>


spec:
  serviceAccountName: <service_account_name>


  outputs:
    - name: httpout-app
      type: http
      url:


      http:
        headers:


          h1: v1
          h2: v2
        method: POST
      secret:
        name:


      tls:
        insecureSkipVerify:


  pipelines:
    - name:
      inputRefs:
        - application
      outputRefs:
        - httpout-app

Copy to Clipboard

Toggle word wrap

1: In legacy implementations, the CR name must be instance. In multi log forwarder implementations, you can use any name.
2: In legacy implementations, the CR namespace must be openshift-logging. In multi log forwarder implementations, you can use any namespace.
3: The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the openshift-logging namespace.
4: Destination address for logs.
5: Additional headers to send with the log record.
6: Secret name for destination credentials.
7: Values are either true or false.
8: This value should be the same as the output name.

9.4.7. Forwarding application logs from specific projects
Copia collegamento

You can forward a copy of the application logs from specific projects to an external log aggregator, in addition to, or instead of, using the internal log store. You must also configure the external log aggregator to receive log data from OpenShift Container Platform.

To configure forwarding application logs from a project, you must create a ClusterLogForwarder custom resource (CR) with at least one input from a project, optional outputs for other log aggregators, and pipelines that use those inputs and outputs.

Prerequisites

You must have a logging server that is configured to receive the logging data using the specified protocol or format.

Procedure

Create or edit a YAML file that defines the ClusterLogForwarder CR:

Example ClusterLogForwarder CR

apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: instance 
  namespace: openshift-logging 
spec:
  outputs:
   - name: fluentd-server-secure 
     type: fluentdForward 
     url: 'tls://fluentdserver.security.example.com:24224' 
     secret: 
        name: fluentd-secret
   - name: fluentd-server-insecure
     type: fluentdForward
     url: 'tcp://fluentdserver.home.example.com:24224'
  inputs: 
   - name: my-app-logs
     application:
        namespaces:
        - my-project 
  pipelines:
   - name: forward-to-fluentd-insecure 
     inputRefs: 
     - my-app-logs
     outputRefs: 
     - fluentd-server-insecure
     labels:
       project: "my-project" 
   - name: forward-to-fluentd-secure 
     inputRefs:
     - application 
     - audit
     - infrastructure
     outputRefs:
     - fluentd-server-secure
     - default
     labels:
       clusterId: "C1234"

apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: instance


  namespace: openshift-logging


spec:
  outputs:
   - name: fluentd-server-secure


     type: fluentdForward


     url: 'tls://fluentdserver.security.example.com:24224'


     secret:


        name: fluentd-secret
   - name: fluentd-server-insecure
     type: fluentdForward
     url: 'tcp://fluentdserver.home.example.com:24224'
  inputs:


   - name: my-app-logs
     application:
        namespaces:
        - my-project


  pipelines:
   - name: forward-to-fluentd-insecure


     inputRefs:


     - my-app-logs
     outputRefs:


     - fluentd-server-insecure
     labels:
       project: "my-project"


   - name: forward-to-fluentd-secure


     inputRefs:
     - application


     - audit
     - infrastructure
     outputRefs:
     - fluentd-server-secure
     - default
     labels:
       clusterId: "C1234"

Copy to Clipboard

Toggle word wrap

The name of the ClusterLogForwarder CR must be instance.

The namespace for the ClusterLogForwarder CR must be openshift-logging.

The name of the output.

The output type: elasticsearch, fluentdForward, syslog, or kafka.

The URL and port of the external log aggregator as a valid absolute URL. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP address.

If using a tls prefix, you must specify the name of the secret required by the endpoint for TLS communication. The secret must exist in the openshift-logging project and have tls.crt, tls.key, and ca-bundle.crt keys that each point to the certificates they represent.

The configuration for an input to filter application logs from the specified projects.

If no namespace is specified, logs are collected from all namespaces.

The pipeline configuration directs logs from a named input to a named output. In this example, a pipeline named forward-to-fluentd-insecure forwards logs from an input named my-app-logs to an output named fluentd-server-insecure.

A list of inputs.

The name of the output to use.

Optional: String. One or more labels to add to the logs.

13

Configuration for a pipeline to send logs to other log aggregators.

Optional: Specify a name for the pipeline.
Specify which log types to forward by using the pipeline: application, infrastructure, or audit.
Specify the name of the output to use when forwarding logs with this pipeline.
Optional: Specify the default output to forward logs to the default log store.
Optional: String. One or more labels to add to the logs.

14

Note that application logs from all namespaces are collected when using this configuration.

Apply the ClusterLogForwarder CR by running the following command:
```
oc apply -f <filename>.yaml
```
```
$ oc apply -f <filename>.yaml
```
Copy to Clipboard Toggle word wrap

9.4.8. Forwarding application logs from specific pods
Copia collegamento

As a cluster administrator, you can use Kubernetes pod labels to gather log data from specific pods and forward it to a log collector.

Suppose that you have an application composed of pods running alongside other pods in various namespaces. If those pods have labels that identify the application, you can gather and output their log data to a specific log collector.

To specify the pod labels, you use one or more matchLabels key-value pairs. If you specify multiple key-value pairs, the pods must match all of them to be selected.

Procedure

Create or edit a YAML file that defines the ClusterLogForwarder CR object. In the file, specify the pod labels using simple equality-based selectors under inputs[].name.application.selector.matchLabels, as shown in the following example.

Example ClusterLogForwarder CR YAML file

apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: <log_forwarder_name> 
  namespace: <log_forwarder_namespace> 
spec:
  pipelines:
    - inputRefs: [ myAppLogData ] 
      outputRefs: [ default ] 
  inputs: 
    - name: myAppLogData
      application:
        selector:
          matchLabels: 
            environment: production
            app: nginx
        namespaces: 
        - app1
        - app2
  outputs: 
    - <output_name>
    ...

apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: <log_forwarder_name>


  namespace: <log_forwarder_namespace>


spec:
  pipelines:
    - inputRefs: [ myAppLogData ]


      outputRefs: [ default ]


  inputs:


    - name: myAppLogData
      application:
        selector:
          matchLabels:


            environment: production
            app: nginx
        namespaces:


        - app1
        - app2
  outputs:


    - <output_name>
    ...

Copy to Clipboard

Toggle word wrap

1: In legacy implementations, the CR name must be instance. In multi log forwarder implementations, you can use any name.
2: In legacy implementations, the CR namespace must be openshift-logging. In multi log forwarder implementations, you can use any namespace.
3: Specify one or more comma-separated values from inputs[].name.
4: Specify one or more comma-separated values from outputs[].
5: Define a unique inputs[].name for each application that has a unique set of pod labels.
6: Specify the key-value pairs of pod labels whose log data you want to gather. You must specify both a key and value, not just a key. To be selected, the pods must match all the key-value pairs.
7: Optional: Specify one or more namespaces.
8: Specify one or more outputs to forward your log data to.

Optional: To restrict the gathering of log data to specific namespaces, use inputs[].name.application.namespaces, as shown in the preceding example.
Optional: You can send log data from additional applications that have different pod labels to the same pipeline.
1. For each unique combination of pod labels, create an additional inputs[].name section similar to the one shown.
2. Update the selectors to match the pod labels of this application.
3. Add the new inputs[].name value to inputRefs. For example:
  - inputRefs: [ myAppLogData, myOtherAppLogData ]
  Copy to Clipboard Toggle word wrap
Create the CR object:
```
oc create -f <file-name>.yaml
```
```
$ oc create -f <file-name>.yaml
```
Copy to Clipboard Toggle word wrap

9.4.9. Overview of API audit filter
Copia collegamento

OpenShift API servers generate audit events for each API call, detailing the request, response, and the identity of the requester, leading to large volumes of data. The API Audit filter uses rules to enable the exclusion of non-essential events and the reduction of event size, facilitating a more manageable audit trail. Rules are checked in order, checking stops at the first match. How much data is included in an event is determined by the value of the level field:

None: The event is dropped.
Metadata: Audit metadata is included, request and response bodies are removed.
Request: Audit metadata and the request body are included, the response body is removed.
RequestResponse: All data is included: metadata, request body and response body. The response body can be very large. For example, oc get pods -A generates a response body containing the YAML description of every pod in the cluster.

Note

You can use this feature only if the Vector collector is set up in your logging deployment.

In logging 5.8 and later, the ClusterLogForwarder custom resource (CR) uses the same format as the standard Kubernetes audit policy, while providing the following additional functions:

Wildcards

Names of users, groups, namespaces, and resources can have a leading or trailing * asterisk character. For example, namespace openshift-\* matches openshift-apiserver or openshift-authentication. Resource \*/status matches Pod/status or Deployment/status.

Default Rules

Events that do not match any rule in the policy are filtered as follows:

Read-only system events such as get, list, watch are dropped.
Service account write events that occur within the same namespace as the service account are dropped.
All other events are forwarded, subject to any configured rate limits.

To disable these defaults, either end your rules list with a rule that has only a level field or add an empty rule.

Omit Response Codes: A list of integer status codes to omit. You can drop events based on the HTTP status code in the response by using the OmitResponseCodes field, a list of HTTP status code for which no events are created. The default value is [404, 409, 422, 429]. If the value is an empty list, [], then no status codes are omitted.

The ClusterLogForwarder CR audit policy acts in addition to the OpenShift Container Platform audit policy. The ClusterLogForwarder CR audit filter changes what the log collector forwards, and provides the ability to filter by verb, user, group, namespace, or resource. You can create multiple filters to send different summaries of the same audit stream to different places. For example, you can send a detailed stream to the local cluster log store, and a less detailed stream to a remote site.

Note

The example provided is intended to illustrate the range of rules possible in an audit policy and is not a recommended configuration.

Example audit policy

apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  pipelines:
    - name: my-pipeline
      inputRefs: audit 
      filterRefs: my-policy 
      outputRefs: default
  filters:
    - name: my-policy
      type: kubeAPIAudit
      kubeAPIAudit:
        # Don't generate audit events for all requests in RequestReceived stage.
        omitStages:
          - "RequestReceived"

        rules:
          # Log pod changes at RequestResponse level
          - level: RequestResponse
            resources:
            - group: ""
              resources: ["pods"]

          # Log "pods/log", "pods/status" at Metadata level
          - level: Metadata
            resources:
            - group: ""
              resources: ["pods/log", "pods/status"]

          # Don't log requests to a configmap called "controller-leader"
          - level: None
            resources:
            - group: ""
              resources: ["configmaps"]
              resourceNames: ["controller-leader"]

          # Don't log watch requests by the "system:kube-proxy" on endpoints or services
          - level: None
            users: ["system:kube-proxy"]
            verbs: ["watch"]
            resources:
            - group: "" # core API group
              resources: ["endpoints", "services"]

          # Don't log authenticated requests to certain non-resource URL paths.
          - level: None
            userGroups: ["system:authenticated"]
            nonResourceURLs:
            - "/api*" # Wildcard matching.
            - "/version"

          # Log the request body of configmap changes in kube-system.
          - level: Request
            resources:
            - group: "" # core API group
              resources: ["configmaps"]
            # This rule only applies to resources in the "kube-system" namespace.
            # The empty string "" can be used to select non-namespaced resources.
            namespaces: ["kube-system"]

          # Log configmap and secret changes in all other namespaces at the Metadata level.
          - level: Metadata
            resources:
            - group: "" # core API group
              resources: ["secrets", "configmaps"]

          # Log all other resources in core and extensions at the Request level.
          - level: Request
            resources:
            - group: "" # core API group
            - group: "extensions" # Version of group should NOT be included.

          # A catch-all rule to log all other requests at the Metadata level.
          - level: Metadata

apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  pipelines:
    - name: my-pipeline
      inputRefs: audit


      filterRefs: my-policy


      outputRefs: default
  filters:
    - name: my-policy
      type: kubeAPIAudit
      kubeAPIAudit:
        # Don't generate audit events for all requests in RequestReceived stage.
        omitStages:
          - "RequestReceived"

        rules:
          # Log pod changes at RequestResponse level
          - level: RequestResponse
            resources:
            - group: ""
              resources: ["pods"]

          # Log "pods/log", "pods/status" at Metadata level
          - level: Metadata
            resources:
            - group: ""
              resources: ["pods/log", "pods/status"]

          # Don't log requests to a configmap called "controller-leader"
          - level: None
            resources:
            - group: ""
              resources: ["configmaps"]
              resourceNames: ["controller-leader"]

          # Don't log watch requests by the "system:kube-proxy" on endpoints or services
          - level: None
            users: ["system:kube-proxy"]
            verbs: ["watch"]
            resources:
            - group: "" # core API group
              resources: ["endpoints", "services"]

          # Don't log authenticated requests to certain non-resource URL paths.
          - level: None
            userGroups: ["system:authenticated"]
            nonResourceURLs:
            - "/api*" # Wildcard matching.
            - "/version"

          # Log the request body of configmap changes in kube-system.
          - level: Request
            resources:
            - group: "" # core API group
              resources: ["configmaps"]
            # This rule only applies to resources in the "kube-system" namespace.
            # The empty string "" can be used to select non-namespaced resources.
            namespaces: ["kube-system"]

          # Log configmap and secret changes in all other namespaces at the Metadata level.
          - level: Metadata
            resources:
            - group: "" # core API group
              resources: ["secrets", "configmaps"]

          # Log all other resources in core and extensions at the Request level.
          - level: Request
            resources:
            - group: "" # core API group
            - group: "extensions" # Version of group should NOT be included.

          # A catch-all rule to log all other requests at the Metadata level.
          - level: Metadata

Copy to Clipboard

Toggle word wrap

1: The log types that are collected. The value for this field can be audit for audit logs, application for application logs, infrastructure for infrastructure logs, or a named input that has been defined for your application.
2: The name of your audit policy.

9.4.10. Forwarding logs to an external Loki logging system
Copia collegamento

You can forward logs to an external Loki logging system in addition to, or instead of, the default log store.

To configure log forwarding to Loki, you must create a ClusterLogForwarder custom resource (CR) with an output to Loki, and a pipeline that uses the output. The output to Loki can use the HTTP (insecure) or HTTPS (secure HTTP) connection.

Prerequisites

You must have a Loki logging system running at the URL you specify with the url field in the CR.

Procedure

Create or edit a YAML file that defines the ClusterLogForwarder CR object:
```
apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: <log_forwarder_name> 
  namespace: <log_forwarder_namespace> 
spec:
  serviceAccountName: <service_account_name> 
  outputs:
  - name: loki-insecure 
    type: "loki" 
    url: http://loki.insecure.com:3100 
    loki:
      tenantKey: kubernetes.namespace_name
      labelKeys:
      - kubernetes.labels.foo
  - name: loki-secure 
    type: "loki"
    url: https://loki.secure.com:3100
    secret:
      name: loki-secret 
    loki:
      tenantKey: kubernetes.namespace_name 
      labelKeys:
      - kubernetes.labels.foo 
  pipelines:
  - name: application-logs 
    inputRefs:  
    - application
    - audit
    outputRefs: 
    - loki-secure
```
```
apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: <log_forwarder_name> 
```
1
```
  namespace: <log_forwarder_namespace> 
```
2
```
spec:
  serviceAccountName: <service_account_name> 
```
3
```
  outputs:
  - name: loki-insecure 
```
4
```
    type: "loki" 
```
5
```
    url: http://loki.insecure.com:3100 
```
6
```
    loki:
      tenantKey: kubernetes.namespace_name
      labelKeys:
      - kubernetes.labels.foo
  - name: loki-secure 
```
7
```
    type: "loki"
    url: https://loki.secure.com:3100
    secret:
      name: loki-secret 
```
8
```
    loki:
      tenantKey: kubernetes.namespace_name 
```
9
```
      labelKeys:
      - kubernetes.labels.foo 
```
10
```
  pipelines:
  - name: application-logs 
```
11
```
    inputRefs:  
```
12
```
    - application
    - audit
    outputRefs: 
```
13
```
    - loki-secure
```
Copy to Clipboard Toggle word wrap
1
In legacy implementations, the CR name must be instance. In multi log forwarder implementations, you can use any name.
2
In legacy implementations, the CR namespace must be openshift-logging. In multi log forwarder implementations, you can use any namespace.
3
The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the openshift-logging namespace.
4
Specify a name for the output.
5
Specify the type as "loki".
6
Specify the URL and port of the Loki system as a valid absolute URL. You can use the http (insecure) or https (secure HTTP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP Address. Loki’s default port for HTTP(S) communication is 3100.
7
For a secure connection, you can specify an https or http URL that you authenticate by specifying a secret.
8
For an https prefix, specify the name of the secret required by the endpoint for TLS communication. The secret must contain a ca-bundle.crt key that points to the certificates it represents. Otherwise, for http and https prefixes, you can specify a secret that contains a username and password. In legacy implementations, the secret must exist in the openshift-logging project. For more information, see the following "Example: Setting a secret that contains a username and password."
9
Optional: Specify a metadata key field to generate values for the TenantID field in Loki. For example, setting tenantKey: kubernetes.namespace_name uses the names of the Kubernetes namespaces as values for tenant IDs in Loki. To see which other log record fields you can specify, see the "Log Record Fields" link in the following "Additional resources" section.
10
Optional: Specify a list of metadata field keys to replace the default Loki labels. Loki label names must match the regular expression [a-zA-Z_:][a-zA-Z0-9_:]*. Illegal characters in metadata keys are replaced with _ to form the label name. For example, the kubernetes.labels.foo metadata key becomes Loki label kubernetes_labels_foo. If you do not set labelKeys, the default value is: [log_type, kubernetes.namespace_name, kubernetes.pod_name, kubernetes_host]. Keep the set of labels small because Loki limits the size and number of labels allowed. See Configuring Loki, limits_config. You can still query based on any log record field using query filters.
11
Optional: Specify a name for the pipeline.
12
Specify which log types to forward by using the pipeline: application, infrastructure, or audit.
13
Specify the name of the output to use when forwarding logs with this pipeline.
Note
Because Loki requires log streams to be correctly ordered by timestamp, labelKeys always includes the kubernetes_host label set, even if you do not specify it. This inclusion ensures that each stream originates from a single host, which prevents timestamps from becoming disordered due to clock differences on different hosts.
Apply the ClusterLogForwarder CR object by running the following command:
```
oc apply -f <filename>.yaml
```
```
$ oc apply -f <filename>.yaml
```
Copy to Clipboard Toggle word wrap

9.4.11. Forwarding logs to an external Elasticsearch instance
Copia collegamento

You can forward logs to an external Elasticsearch instance in addition to, or instead of, the internal log store. You are responsible for configuring the external log aggregator to receive log data from OpenShift Container Platform.

To configure log forwarding to an external Elasticsearch instance, you must create a ClusterLogForwarder custom resource (CR) with an output to that instance, and a pipeline that uses the output. The external Elasticsearch output can use the HTTP (insecure) or HTTPS (secure HTTP) connection.

To forward logs to both an external and the internal Elasticsearch instance, create outputs and pipelines to the external instance and a pipeline that uses the default output to forward logs to the internal instance.

Note

If you only want to forward logs to an internal Elasticsearch instance, you do not need to create a ClusterLogForwarder CR.

Prerequisites

You must have a logging server that is configured to receive the logging data using the specified protocol or format.

Procedure

Create or edit a YAML file that defines the ClusterLogForwarder CR:
Example ClusterLogForwarder CR
```
apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: <log_forwarder_name> 
  namespace: <log_forwarder_namespace> 
spec:
  serviceAccountName: <service_account_name> 
  outputs:
   - name: elasticsearch-example 
     type: elasticsearch 
     elasticsearch:
       version: 8 
     url: http://elasticsearch.example.com:9200 
     secret:
       name: es-secret 
  pipelines:
   - name: application-logs 
     inputRefs: 
     - application
     - audit
     outputRefs:
     - elasticsearch-example 
     - default 
     labels:
       myLabel: "myValue" 
# ...
```
```
apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: <log_forwarder_name> 
```
1
```
  namespace: <log_forwarder_namespace> 
```
2
```
spec:
  serviceAccountName: <service_account_name> 
```
3
```
  outputs:
   - name: elasticsearch-example 
```
4
```
     type: elasticsearch 
```
5
```
     elasticsearch:
       version: 8 
```
6
```
     url: http://elasticsearch.example.com:9200 
```
7
```
     secret:
       name: es-secret 
```
8
```
  pipelines:
   - name: application-logs 
```
9
```
     inputRefs: 
```
10
```
     - application
     - audit
     outputRefs:
     - elasticsearch-example 
```
11
```
     - default 
```
12
```
     labels:
       myLabel: "myValue" 
```
13
```
# ...
```
Copy to Clipboard Toggle word wrap
1
In legacy implementations, the CR name must be instance. In multi log forwarder implementations, you can use any name.
2
In legacy implementations, the CR namespace must be openshift-logging. In multi log forwarder implementations, you can use any namespace.
3
The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the openshift-logging namespace.
4
Specify a name for the output.
5
Specify the elasticsearch type.
6
Specify the Elasticsearch version. This can be 6, 7, or 8.
7
Specify the URL and port of the external Elasticsearch instance as a valid absolute URL. You can use the http (insecure) or https (secure HTTP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP Address.
8
For an https prefix, specify the name of the secret required by the endpoint for TLS communication. The secret must contain a ca-bundle.crt key that points to the certificate it represents. Otherwise, for http and https prefixes, you can specify a secret that contains a username and password. In legacy implementations, the secret must exist in the openshift-logging project. For more information, see the following "Example: Setting a secret that contains a username and password."
9
Optional: Specify a name for the pipeline.
10
Specify which log types to forward by using the pipeline: application, infrastructure, or audit.
11
Specify the name of the output to use when forwarding logs with this pipeline.
12
Optional: Specify the default output to send the logs to the internal Elasticsearch instance.
13
Optional: String. One or more labels to add to the logs.
Apply the ClusterLogForwarder CR:
```
oc apply -f <filename>.yaml
```
```
$ oc apply -f <filename>.yaml
```
Copy to Clipboard Toggle word wrap

Example: Setting a secret that contains a username and password

You can use a secret that contains a username and password to authenticate a secure connection to an external Elasticsearch instance.

For example, if you cannot use mutual TLS (mTLS) keys because a third party operates the Elasticsearch instance, you can use HTTP or HTTPS and set a secret that contains the username and password.

Create a Secret YAML file similar to the following example. Use base64-encoded values for the username and password fields. The secret type is opaque by default.

apiVersion: v1
kind: Secret
metadata:
  name: openshift-test-secret
data:
  username: <username>
  password: <password>
# ...

apiVersion: v1
kind: Secret
metadata:
  name: openshift-test-secret
data:
  username: <username>
  password: <password>
# ...

Copy to Clipboard

Toggle word wrap

Create the secret:

oc create secret -n openshift-logging openshift-test-secret.yaml

$ oc create secret -n openshift-logging openshift-test-secret.yaml

Copy to Clipboard

Toggle word wrap

Specify the name of the secret in the ClusterLogForwarder CR:

kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  outputs:
   - name: elasticsearch
     type: "elasticsearch"
     url: https://elasticsearch.secure.com:9200
     secret:
        name: openshift-test-secret
# ...

kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  outputs:
   - name: elasticsearch
     type: "elasticsearch"
     url: https://elasticsearch.secure.com:9200
     secret:
        name: openshift-test-secret
# ...

Copy to Clipboard

Toggle word wrap

Note

In the value of the url field, the prefix can be http or https.

Apply the CR object:
```
oc apply -f <filename>.yaml
```
```
$ oc apply -f <filename>.yaml
```
Copy to Clipboard Toggle word wrap

9.4.12. Forwarding logs using the Fluentd forward protocol
Copia collegamento

You can use the Fluentd forward protocol to send a copy of your logs to an external log aggregator that is configured to accept the protocol instead of, or in addition to, the default Elasticsearch log store. You are responsible for configuring the external log aggregator to receive the logs from OpenShift Container Platform.

To configure log forwarding using the forward protocol, you must create a ClusterLogForwarder custom resource (CR) with one or more outputs to the Fluentd servers, and pipelines that use those outputs. The Fluentd output can use a TCP (insecure) or TLS (secure TCP) connection.

Prerequisites

You must have a logging server that is configured to receive the logging data using the specified protocol or format.

Procedure

Create or edit a YAML file that defines the ClusterLogForwarder CR object:

apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: instance 
  namespace: openshift-logging 
spec:
  outputs:
   - name: fluentd-server-secure 
     type: fluentdForward 
     url: 'tls://fluentdserver.security.example.com:24224' 
     secret: 
        name: fluentd-secret
   - name: fluentd-server-insecure
     type: fluentdForward
     url: 'tcp://fluentdserver.home.example.com:24224'
  pipelines:
   - name: forward-to-fluentd-secure 
     inputRefs:  
     - application
     - audit
     outputRefs:
     - fluentd-server-secure 
     - default 
     labels:
       clusterId: "C1234" 
   - name: forward-to-fluentd-insecure 
     inputRefs:
     - infrastructure
     outputRefs:
     - fluentd-server-insecure
     labels:
       clusterId: "C1234"

apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: instance


  namespace: openshift-logging


spec:
  outputs:
   - name: fluentd-server-secure


     type: fluentdForward


     url: 'tls://fluentdserver.security.example.com:24224'


     secret:


        name: fluentd-secret
   - name: fluentd-server-insecure
     type: fluentdForward
     url: 'tcp://fluentdserver.home.example.com:24224'
  pipelines:
   - name: forward-to-fluentd-secure


     inputRefs:


     - application
     - audit
     outputRefs:
     - fluentd-server-secure


     - default


     labels:
       clusterId: "C1234"


   - name: forward-to-fluentd-insecure


     inputRefs:
     - infrastructure
     outputRefs:
     - fluentd-server-insecure
     labels:
       clusterId: "C1234"

Copy to Clipboard

Toggle word wrap

The name of the ClusterLogForwarder CR must be instance.

The namespace for the ClusterLogForwarder CR must be openshift-logging.

Specify a name for the output.

Specify the fluentdForward type.

Specify the URL and port of the external Fluentd instance as a valid absolute URL. You can use the tcp (insecure) or tls (secure TCP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP address.

If you are using a tls prefix, you must specify the name of the secret required by the endpoint for TLS communication. The secret must exist in the openshift-logging project and must contain a ca-bundle.crt key that points to the certificate it represents.

Optional: Specify a name for the pipeline.

Specify which log types to forward by using the pipeline: application, infrastructure, or audit.

Specify the name of the output to use when forwarding logs with this pipeline.

Optional: Specify the default output to forward logs to the internal Elasticsearch instance.

Optional: String. One or more labels to add to the logs.

Optional: Configure multiple outputs to forward logs to other external log aggregators of any supported type:

A name to describe the pipeline.
The inputRefs is the log type to forward by using the pipeline: application, infrastructure, or audit.
The outputRefs is the name of the output to use.
Optional: String. One or more labels to add to the logs.

Create the CR object:
```
oc create -f <file-name>.yaml
```
```
$ oc create -f <file-name>.yaml
```
Copy to Clipboard Toggle word wrap

9.4.12.1. Enabling nanosecond precision for Logstash to ingest data from fluentd
Copia collegamento

For Logstash to ingest log data from fluentd, you must enable nanosecond precision in the Logstash configuration file.

Procedure

In the Logstash configuration file, set nanosecond_precision to true.

Example Logstash configuration file

input { tcp { codec => fluent { nanosecond_precision => true } port => 24114 } }
filter { }
output { stdout { codec => rubydebug } }

input { tcp { codec => fluent { nanosecond_precision => true } port => 24114 } }
filter { }
output { stdout { codec => rubydebug } }

Copy to Clipboard

Toggle word wrap

9.4.13. Forwarding logs using the syslog protocol
Copia collegamento

You can use the syslog RFC3164 or RFC5424 protocol to send a copy of your logs to an external log aggregator that is configured to accept the protocol instead of, or in addition to, the default Elasticsearch log store. You are responsible for configuring the external log aggregator, such as a syslog server, to receive the logs from OpenShift Container Platform.

To configure log forwarding using the syslog protocol, you must create a ClusterLogForwarder custom resource (CR) with one or more outputs to the syslog servers, and pipelines that use those outputs. The syslog output can use a UDP, TCP, or TLS connection.

Prerequisites

You must have a logging server that is configured to receive the logging data using the specified protocol or format.

Procedure

Create or edit a YAML file that defines the ClusterLogForwarder CR object:

apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: <log_forwarder_name> 
  namespace: <log_forwarder_namespace> 
spec:
  serviceAccountName: <service_account_name> 
  outputs:
   - name: rsyslog-east 
     type: syslog 
     syslog: 
       facility: local0
       rfc: RFC3164
       payloadKey: message
       severity: informational
     url: 'tls://rsyslogserver.east.example.com:514' 
     secret: 
        name: syslog-secret
   - name: rsyslog-west
     type: syslog
     syslog:
      appName: myapp
      facility: user
      msgID: mymsg
      procID: myproc
      rfc: RFC5424
      severity: debug
     url: 'tcp://rsyslogserver.west.example.com:514'
  pipelines:
   - name: syslog-east 
     inputRefs: 
     - audit
     - application
     outputRefs: 
     - rsyslog-east
     - default 
     labels:
       secure: "true" 
       syslog: "east"
   - name: syslog-west 
     inputRefs:
     - infrastructure
     outputRefs:
     - rsyslog-west
     - default
     labels:
       syslog: "west"

apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: <log_forwarder_name>


  namespace: <log_forwarder_namespace>


spec:
  serviceAccountName: <service_account_name>


  outputs:
   - name: rsyslog-east


     type: syslog


     syslog:


       facility: local0
       rfc: RFC3164
       payloadKey: message
       severity: informational
     url: 'tls://rsyslogserver.east.example.com:514'


     secret:


        name: syslog-secret
   - name: rsyslog-west
     type: syslog
     syslog:
      appName: myapp
      facility: user
      msgID: mymsg
      procID: myproc
      rfc: RFC5424
      severity: debug
     url: 'tcp://rsyslogserver.west.example.com:514'
  pipelines:
   - name: syslog-east


     inputRefs:


     - audit
     - application
     outputRefs:


     - rsyslog-east
     - default


     labels:
       secure: "true"


       syslog: "east"
   - name: syslog-west


     inputRefs:
     - infrastructure
     outputRefs:
     - rsyslog-west
     - default
     labels:
       syslog: "west"

Copy to Clipboard

Toggle word wrap

In legacy implementations, the CR name must be instance. In multi log forwarder implementations, you can use any name.

In legacy implementations, the CR namespace must be openshift-logging. In multi log forwarder implementations, you can use any namespace.

The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the openshift-logging namespace.

Specify a name for the output.

Specify the syslog type.

Optional: Specify the syslog parameters, listed below.

Specify the URL and port of the external syslog instance. You can use the udp (insecure), tcp (insecure) or tls (secure TCP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP address.

If using a tls prefix, you must specify the name of the secret required by the endpoint for TLS communication. The secret must contain a ca-bundle.crt key that points to the certificate it represents. In legacy implementations, the secret must exist in the openshift-logging project.

Optional: Specify a name for the pipeline.

Specify which log types to forward by using the pipeline: application, infrastructure, or audit.

Specify the name of the output to use when forwarding logs with this pipeline.

Optional: Specify the default output to forward logs to the internal Elasticsearch instance.

13

Optional: String. One or more labels to add to the logs. Quote values like "true" so they are recognized as string values, not as a boolean.

14

Optional: Configure multiple outputs to forward logs to other external log aggregators of any supported type:

A name to describe the pipeline.
The inputRefs is the log type to forward by using the pipeline: application, infrastructure, or audit.
The outputRefs is the name of the output to use.
Optional: String. One or more labels to add to the logs.

Create the CR object:
```
oc create -f <filename>.yaml
```
```
$ oc create -f <filename>.yaml
```
Copy to Clipboard Toggle word wrap

9.4.13.1. Adding log source information to message output
Copia collegamento

You can add namespace_name, pod_name, and container_name elements to the message field of the record by adding the AddLogSource field to your ClusterLogForwarder custom resource (CR).

  spec:
    outputs:
    - name: syslogout
      syslog:
        addLogSource: true
        facility: user
        payloadKey: message
        rfc: RFC3164
        severity: debug
        tag: mytag
      type: syslog
      url: tls://syslog-receiver.openshift-logging.svc:24224
    pipelines:
    - inputRefs:
      - application
      name: test-app
      outputRefs:
      - syslogout

  spec:
    outputs:
    - name: syslogout
      syslog:
        addLogSource: true
        facility: user
        payloadKey: message
        rfc: RFC3164
        severity: debug
        tag: mytag
      type: syslog
      url: tls://syslog-receiver.openshift-logging.svc:24224
    pipelines:
    - inputRefs:
      - application
      name: test-app
      outputRefs:
      - syslogout

Copy to Clipboard

Toggle word wrap

Note

This configuration is compatible with both RFC3164 and RFC5424.

Example syslog message output without AddLogSource

<15>1 2020-11-15T17:06:14+00:00 fluentd-9hkb4 mytag - - -  {"msgcontent"=>"Message Contents", "timestamp"=>"2020-11-15 17:06:09", "tag_key"=>"rec_tag", "index"=>56}

<15>1 2020-11-15T17:06:14+00:00 fluentd-9hkb4 mytag - - -  {"msgcontent"=>"Message Contents", "timestamp"=>"2020-11-15 17:06:09", "tag_key"=>"rec_tag", "index"=>56}

Copy to Clipboard

Toggle word wrap

Example syslog message output with AddLogSource

<15>1 2020-11-16T10:49:37+00:00 crc-j55b9-master-0 mytag - - -  namespace_name=clo-test-6327,pod_name=log-generator-ff9746c49-qxm7l,container_name=log-generator,message={"msgcontent":"My life is my message", "timestamp":"2020-11-16 10:49:36", "tag_key":"rec_tag", "index":76}

<15>1 2020-11-16T10:49:37+00:00 crc-j55b9-master-0 mytag - - -  namespace_name=clo-test-6327,pod_name=log-generator-ff9746c49-qxm7l,container_name=log-generator,message={"msgcontent":"My life is my message", "timestamp":"2020-11-16 10:49:36", "tag_key":"rec_tag", "index":76}

Copy to Clipboard

Toggle word wrap

9.4.13.2. Syslog parameters
Copia collegamento

You can configure the following for the syslog outputs. For more information, see the syslog RFC3164 or RFC5424 RFC.

facility: The syslog facility. The value can be a decimal integer or a case-insensitive keyword:
- 0 or kern for kernel messages
- 1 or user for user-level messages, the default.
- 2 or mail for the mail system
- 3 or daemon for system daemons
- 4 or auth for security/authentication messages
- 5 or syslog for messages generated internally by syslogd
- 6 or lpr for the line printer subsystem
- 7 or news for the network news subsystem
- 8 or uucp for the UUCP subsystem
- 9 or cron for the clock daemon
- 10 or authpriv for security authentication messages
- 11 or ftp for the FTP daemon
- 12 or ntp for the NTP subsystem
- 13 or security for the syslog audit log
- 14 or console for the syslog alert log
- 15 or solaris-cron for the scheduling daemon
- 16–23 or local0 – local7 for locally used facilities
Optional: payloadKey: The record field to use as payload for the syslog message.
Note
Configuring the payloadKey parameter prevents other parameters from being forwarded to the syslog.
rfc: The RFC to be used for sending logs using syslog. The default is RFC5424.
severity: The syslog severity to set on outgoing syslog records. The value can be a decimal integer or a case-insensitive keyword:
- 0 or Emergency for messages indicating the system is unusable
- 1 or Alert for messages indicating action must be taken immediately
- 2 or Critical for messages indicating critical conditions
- 3 or Error for messages indicating error conditions
- 4 or Warning for messages indicating warning conditions
- 5 or Notice for messages indicating normal but significant conditions
- 6 or Informational for messages indicating informational messages
- 7 or Debug for messages indicating debug-level messages, the default
tag: Tag specifies a record field to use as a tag on the syslog message.
trimPrefix: Remove the specified prefix from the tag.

9.4.13.3. Additional RFC5424 syslog parameters
Copia collegamento

The following parameters apply to RFC5424:

appName: The APP-NAME is a free-text string that identifies the application that sent the log. Must be specified for RFC5424.
msgID: The MSGID is a free-text string that identifies the type of message. Must be specified for RFC5424.
procID: The PROCID is a free-text string. A change in the value indicates a discontinuity in syslog reporting. Must be specified for RFC5424.

9.4.14. Forwarding logs to a Kafka broker
Copia collegamento

You can forward logs to an external Kafka broker in addition to, or instead of, the default log store.

To configure log forwarding to an external Kafka instance, you must create a ClusterLogForwarder custom resource (CR) with an output to that instance, and a pipeline that uses the output. You can include a specific Kafka topic in the output or use the default. The Kafka output can use a TCP (insecure) or TLS (secure TCP) connection.

Procedure

Create or edit a YAML file that defines the ClusterLogForwarder CR object:

apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: <log_forwarder_name> 
  namespace: <log_forwarder_namespace> 
spec:
  serviceAccountName: <service_account_name> 
  outputs:
   - name: app-logs 
     type: kafka 
     url: tls://kafka.example.devlab.com:9093/app-topic 
     secret:
       name: kafka-secret 
   - name: infra-logs
     type: kafka
     url: tcp://kafka.devlab2.example.com:9093/infra-topic 
   - name: audit-logs
     type: kafka
     url: tls://kafka.qelab.example.com:9093/audit-topic
     secret:
        name: kafka-secret-qe
  pipelines:
   - name: app-topic 
     inputRefs: 
     - application
     outputRefs: 
     - app-logs
     labels:
       logType: "application" 
   - name: infra-topic 
     inputRefs:
     - infrastructure
     outputRefs:
     - infra-logs
     labels:
       logType: "infra"
   - name: audit-topic
     inputRefs:
     - audit
     outputRefs:
     - audit-logs
     labels:
       logType: "audit"

apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: <log_forwarder_name>


  namespace: <log_forwarder_namespace>


spec:
  serviceAccountName: <service_account_name>


  outputs:
   - name: app-logs


     type: kafka


     url: tls://kafka.example.devlab.com:9093/app-topic


     secret:
       name: kafka-secret


   - name: infra-logs
     type: kafka
     url: tcp://kafka.devlab2.example.com:9093/infra-topic


   - name: audit-logs
     type: kafka
     url: tls://kafka.qelab.example.com:9093/audit-topic
     secret:
        name: kafka-secret-qe
  pipelines:
   - name: app-topic


     inputRefs:


     - application
     outputRefs:


     - app-logs
     labels:
       logType: "application"


   - name: infra-topic


     inputRefs:
     - infrastructure
     outputRefs:
     - infra-logs
     labels:
       logType: "infra"
   - name: audit-topic
     inputRefs:
     - audit
     outputRefs:
     - audit-logs
     labels:
       logType: "audit"

Copy to Clipboard

Toggle word wrap

In legacy implementations, the CR name must be instance. In multi log forwarder implementations, you can use any name.

In legacy implementations, the CR namespace must be openshift-logging. In multi log forwarder implementations, you can use any namespace.

The name of your service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the openshift-logging namespace.

Specify a name for the output.

Specify the kafka type.

Specify the URL and port of the Kafka broker as a valid absolute URL, optionally with a specific topic. You can use the tcp (insecure) or tls (secure TCP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP address.

If you are using a tls prefix, you must specify the name of the secret required by the endpoint for TLS communication. The secret must contain a ca-bundle.crt key that points to the certificate it represents. In legacy implementations, the secret must exist in the openshift-logging project.

Optional: To send an insecure output, use a tcp prefix in front of the URL. Also omit the secret key and its name from this output.

Optional: Specify a name for the pipeline.

Specify which log types to forward by using the pipeline: application, infrastructure, or audit.

Specify the name of the output to use when forwarding logs with this pipeline.