Configuring logging

Procedure

1. Modify the ClusterLogForwarder CR to add configuration for the http receiver input:

   Example ClusterLogForwarder CR

   apiVersion: observability.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: <clusterlogforwarder_name> 

   1

   
     namespace: <namespace>
   # ...
   spec:
     serviceAccount:
       name: <service_account_name>
     inputs:
     - name: http-receiver 

   2

   
       type: receiver
       receiver:
         type: http 

   3

   
         port: 8443 

   4

   
         http:
           format: kubeAPIAudit 

   5

   
     outputs:
     - name: <output_name>
       type: http
       http:
         url: <url>
     pipelines: 

   6

   
       - name: http-pipeline
         inputRefs:
           - http-receiver
         outputRefs:
           - <output_name>
   # ...

   1

   Specify a name for the ClusterLogForwarder CR.

   2

   Specify a name for your input receiver.

   3

   Specify the input receiver type as http.

   4

   Optional: Specify the port that the input receiver listens on. This must be a value between 1024 and 65535. The default value is 8443 if this is not specified.

   5

   Currently, only the kube-apiserver webhook format is supported for http input receivers.

   6

   Configure a pipeline for your input receiver.

2. Apply the changes to the ClusterLogForwarder CR by running the following command:

   $ oc apply -f <filename>.yaml

3. Verify that the collector is listening on the service that has a name in the <clusterlogforwarder_resource_name>-<input_name> format by running the following command:

   $ oc get svc

   Example output

   NAME                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)            AGE
   collector                 ClusterIP   172.30.85.239    <none>        24231/TCP          3m6s
   collector-http-receiver   ClusterIP   172.30.205.160   <none>        8443/TCP           3m6s

   In the example, the service name is collector-http-receiver.

Verification

1. Extract the certificate authority (CA) certificate file by running the following command:

   $ oc extract cm/openshift-service-ca.crt -n <namespace>

   Note

   If the CA in the cluster where the collectors are running changes, you must extract the CA certificate file again.

2. As an example, use the curl command to send logs by running the following command:

   $ curl --cacert <openshift_service_ca.crt> https://collector-http-receiver.<namespace>.svc:8443 -XPOST -d '{"<prefix>":"<message>"}'

   Replace <openshift_service_ca.crt> with the extracted CA certificate file.

Procedure

1. Grant the collect-infrastructure-logs cluster role to the service account by running the following command:

   Example binding command

   $ oc adm policy add-cluster-role-to-user collect-infrastructure-logs -z logcollector

2. Modify the ClusterLogForwarder CR to add configuration for the syslog receiver input:

   Example ClusterLogForwarder CR

   apiVersion: observability.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: <clusterlogforwarder_name> 

   1

   
     namespace: <namespace>
   # ...
   spec:
     serviceAccount:
       name: <service_account_name> 

   2

   
     inputs:
       - name: syslog-receiver 

   3

   
         type: receiver
         receiver:
           type: syslog 

   4

   
           port: 10514 

   5

   
     outputs:
     - name: <output_name>
       lokiStack:
         authentication:
           token:
             from: serviceAccount
         target:
           name: logging-loki
           namespace: openshift-logging
       tls: 

   6

   
         ca:
           key: service-ca.crt
           configMapName: openshift-service-ca.crt
       type: lokiStack
   # ...
     pipelines: 

   7

   
       - name: syslog-pipeline
         inputRefs:
           - syslog-receiver
         outputRefs:
           - <output_name>
   # ...

   1 2

   Use the service account that you granted the collect-infrastructure-logs permission in the previous step.

   3

   Specify a name for your input receiver.

   4

   Specify the input receiver type as syslog.

   5

   Optional: Specify the port that the input receiver listens on. This must be a value between 1024 and 65535.

   6

   If TLS configuration is not set, the default certificates will be used. For more information, run the command oc explain clusterlogforwarders.spec.inputs.receiver.tls.

   7

   Configure a pipeline for your input receiver.

3. Apply the changes to the ClusterLogForwarder CR by running the following command:

   $ oc apply -f <filename>.yaml

4. Verify that the collector is listening on the service that has a name in the <clusterlogforwarder_resource_name>-<input_name> format by running the following command:

   $ oc get svc

   Example output

   NAME                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)            AGE
   collector                   ClusterIP   172.30.85.239    <none>        24231/TCP          33m
   collector-syslog-receiver   ClusterIP   172.30.216.142   <none>        10514/TCP          2m20s

   In this example output, the service name is collector-syslog-receiver.

Verification

1. Extract the certificate authority (CA) certificate file by running the following command:

   $ oc extract cm/openshift-service-ca.crt -n <namespace>

   Note

   If the CA in the cluster where the collectors are running changes, you must extract the CA certificate file again.

2. As an example, use the curl command to send logs by running the following command:

   $ curl --cacert <openshift_service_ca.crt> collector-syslog-receiver.<namespace>.svc:10514  “test message”

   Replace <openshift_service_ca.crt> with the extracted CA certificate file.

Procedure

1. Update the ClusterLogForwarder CR:

   apiVersion: observability.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: my-forwarder
     namespace: openshift-logging
   spec:
     collector:
       networkPolicy:
         ruleSet: <RestrictIngressEgress_or_AllowAllIngressEgress>
     # ...

   Where the value for the ruleSet field can be either RestrictIngressEgress or AllowAllIngressEgress.

2. Apply the configuration by running the following command:

   $ oc apply -f <filename>.yaml

Procedure

1. Update the LogFileMetricExporter CR:

   apiVersion: logging.openshift.io/v1alpha1
   kind: LogFileMetricExporter
   metadata:
     name: my-lfme
     namespace: openshift-logging
   spec:
     networkPolicy:
       ruleSet: <AllowAllIngressEgress_or_AllowIngressMetrics>
   #...

   Where the value for the ruleSet field can be either AllowAllIngressEgress or AllowIngressMetrics.

2. Apply the configuration by running the following command:

   $ oc apply -f <filename>.yaml

Procedure

1. Create an AdminNetworkPolicy rule:

   apiVersion: policy.networking.k8s.io/v1alpha1
   kind: AdminNetworkPolicy
   metadata:
     name: allow-logging-collector-delegation
   spec:
     priority: 50 # Adjust based on your cluster's ANP priority scheme. Lower number means higher priority
     subject:
       pods: # Target the pods in openshift-logging namespace
         namespaceSelector:
           matchLabels:
             kubernetes.io/metadata.name: openshift-logging
         podSelector:
           matchLabels:
             app.kubernetes.io/name: <name>
             app.kubernetes.io/instance: <instance>
             app.kubernetes.io/managed-by: cluster-logging-operator
             app.kubernetes.io/part-of: cluster-logging
             app.kubernetes.io/component: <component>
     ingress:
       - name: "delegate-to-collector-ingress"
         action: "Pass"
         from:
           # Select all pods in all namespaces as the source.
           - pods:
               namespaceSelector: {}
               podSelector: {}
     egress:
       - name: "delegate-to-collector-egress"
         action: "Pass"
         to:
           # Select all pods in all namespaces for internal traffic.
           - pods:
               namespaceSelector: {}
               podSelector: {}
           # Select all external destinations for egress traffic.
           - networks:
             - "0.0.0.0/0"
             - "::/0"

   To apply this network policy to all the resources in a namespace, remove app.kubernetes.io/name, app.kubernetes.io/instance, and app.kubernetes.io/component fields. To apply the admin network policy to specific resources, define the fields as follows:

   • Replace <name> with the name of the collector or the LogFileMetricExporter resource.
   • Replace <instance> with the instance name of the collector or the LogFileMetricExporter resource.
   • Replace <component> with the value collector or logfilesmetricexporter.

2. Apply the AdminNetworkPolicy rule by running the following command:

   $ oc apply -f <filename>.yaml

Procedure

1. Verify that the ClusterLogForwarder status conditions all show True by running the following command:

   $ oc get clusterlogforwarder instance -n openshift-logging \
     -o jsonpath='{.status.conditions}' | python3 -m json.tool

   Confirm the following conditions:

   • observability.openshift.io/Authorized is True and the message lists the log types that the collector can collect.
   • observability.openshift.io/Valid is True.
   • Ready is True.

   If Authorized shows False, the collector service account lacks a required ClusterRoleBinding. See "Granting collector permissions for log collection."

2. Verify that a collector pod is running by running the following command:

   $ oc get pods -n openshift-logging -l app.kubernetes.io/component=collector

   Example output

   NAME               READY   STATUS    RESTARTS   AGE
   instance-jlhjw     1/1     Running   0          5m

3. In the OpenShift Container Platform web console, go to Observe > Logs.
4. Select a log type from the drop-down menu, such as infrastructure, and verify that log entries appear.

5. Optional: Check the collector logs for errors by running the following command:

   $ oc logs -n openshift-logging -l app.kubernetes.io/component=collector --tail=20

Procedure

1. Add a configuration to include or exclude the namespace and container names in the ClusterLogForwarder CR.

   apiVersion: observability.openshift.io/v1
   kind: ClusterLogForwarder
   # ...
   spec:
     serviceAccount:
       name: <service_account_name>
     inputs:
       - name: mylogs
         application:
           includes:
             - namespace: "my-project"
               container: "my-container"
           excludes:
             - container: "other-container*"
               namespace: "other-namespace"
         type: application
   # ...

   includes.namespace

   Specifies that the logs are only collected from these namespaces.

   includes.container

   Specifies that the logs are only collected from these containers.

   excludes.namespace

   Specifies the pattern of namespaces to ignore when collecting the logs.

   excludes.container

   Specifies the set of containers to ignore when collecting the logs.

   Note

   The excludes field takes precedence over the includes field.

2. Apply the ClusterLogForwarder CR by running the following command:

   $ oc apply -f <filename>.yaml

Procedure

1. Add a configuration for a filter to the input spec in the ClusterLogForwarder CR.

   apiVersion: observability.openshift.io/v1
   kind: ClusterLogForwarder
   # ...
   spec:
     serviceAccount:
       name: <service_account_name>
     inputs:
       - name: mylogs
         application:
           selector:
             matchExpressions:
             - key: env
               operator: In
               values: ["prod", "qa"]
             - key: zone
               operator: NotIn
               values: ["east", "west"]
             matchLabels:
               app: one
               name: app1
         type: application
   # ...

   matchExpressions.key

   Specifies the label key to match.

   matchExpressions.operator

   Specifies the operator. Valid values include: In, NotIn, Exists, and DoesNotExist.

   matchExpressions.values

   Specifies an array of string values. If the operator value is either Exists or DoesNotExist, the value array must be empty.

   matchLabels

   Specifies an exact key and value mapping.

2. Apply the ClusterLogForwarder CR by running the following command:

   $ oc apply -f <filename>.yaml

Procedure

1. Add a configuration to define the audit and infrastructure sources in the ClusterLogForwarder CR.

   apiVersion: observability.openshift.io/v1
   kind: ClusterLogForwarder
   # ...
   spec:
     serviceAccount:
       name: <service_account_name>
     inputs:
       - name: mylogs1
         type: infrastructure
         infrastructure:
           sources:
             - node
       - name: mylogs2
         type: audit
         audit:
           sources:
             - kubeAPI
             - openshiftAPI
             - ovn
   # ...

   infrastructure.sources

   Specifies the list of infrastructure sources to collect. The valid sources include:

   • node: Journal log from the node
   • container: Logs from the workloads deployed in the namespaces

   audit.sources

   Specifies the list of audit sources to collect. The valid sources include:

   • kubeAPI: Logs from the Kubernetes API servers
   • openshiftAPI: Logs from the OpenShift API servers
   • auditd: Logs from a node auditd service
   • ovn: Logs from an open virtual network service

2. Apply the ClusterLogForwarder CR by running the following command:

   $ oc apply -f <filename>.yaml

Procedure

1. Grant the collector service account permission to collect audit logs:

   $ oc adm policy add-cluster-role-to-user collect-audit-logs \
     system:serviceaccount:<namespace>:<service_account_name>

   Replace <namespace> with the namespace where your ClusterLogForwarder CR is deployed and <service_account_name> with the name you specified in the ClusterLogForwarder CR spec.serviceAccount.name field.

2. Add audit to the inputRefs list in your ClusterLogForwarder pipeline by editing the CR:

   $ oc edit clusterlogforwarder <clusterlogforwarder_name> -n <namespace>

   Replace <clusterlogforwarder_name> with the name of your ClusterLogForwarder CR and <namespace> with the namespace where it is deployed.

   Note

   In Logging versions before 5.9, the ClusterLogForwarder CR was a singleton named instance in the openshift-logging namespace. Starting in Logging 5.9, you can create multiple ClusterLogForwarder CRs with different names in different namespaces.

   Example configuration

   spec:
     # ... other configuration
     outputs:
     - name: default-loki
       # ... output configuration
     pipelines:
     - name: my-pipeline
       inputRefs:
       - application
       - infrastructure
       - audit
       outputRefs:
       - default-loki

3. Verify that the ClusterLogForwarder still shows Ready by running the following command:

   $ oc get clusterlogforwarder <clusterlogforwarder_name> -n <namespace> \
     -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'

   Replace <clusterlogforwarder_name> with the name of your ClusterLogForwarder CR and <namespace> with the namespace where it is deployed.

   The output should show True.

4. Verify that the Authorized condition lists audit by running the following command:

   $ oc get clusterlogforwarder <clusterlogforwarder_name> -n <namespace> \
     -o jsonpath='{.status.conditions[?(@.type=~"Authorized")].message}'

   Example output

   permitted to collect log types: [application audit infrastructure]

Procedure

1. To enable logging to detect multi-line exceptions and reassemble them into a single log entry, ensure that the ClusterLogForwarder Custom Resource (CR) has a detectMultilineErrors field under the .spec.filters.

   Example ClusterLogForwarder CR

   apiVersion: "observability.openshift.io/v1"
   kind: ClusterLogForwarder
   metadata:
     name: <log_forwarder_name>
     namespace: <log_forwarder_namespace>
   spec:
     serviceAccount:
       name: <service_account_name>
     filters:
     - name: <name>
       type: detectMultilineException
     pipelines:
       - inputRefs:
           - <input_name>
         name: <pipeline_name>
         filterRefs:
           - <filter_name>
         outputRefs:
           - <output_name>

Procedure

1. Configure rate limits on inputs, outputs, or both.

   apiVersion: observability.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: instance
     namespace: openshift-logging
   spec:
     serviceAccount:
       name: collector
     inputs:
     - name: limited-app-logs
       type: application
       application:
         tuning:
           rateLimitPerContainer:
             maxRecordsPerSecond: 100
     outputs:
     - name: limited-splunk
       type: splunk
       splunk:
         url: https://splunk.example.com:8088
         authentication:
           token:
             key: token
             secretName: splunk-secret
       rateLimit:
         maxRecordsPerSecond: 1000
     pipelines:
     - name: app-to-splunk
       inputRefs:
       - limited-app-logs
       outputRefs:
       - limited-splunk

   Input rate limit (inputs[].application.tuning.rateLimitPerContainer)

   Limits the number of log records per second from each container that matches this input. This is a per-container limit. For example, if you set maxRecordsPerSecond: 100 and 10 containers match the input selector, the total input rate could be up to 1000 records per second (100 per container × 10 containers).

   Use case: Site reliability engineers can use input rate limits to prevent noisy applications from overwhelming the log collector and consuming excessive cluster resources.

   Output rate limit (outputs[].rateLimit)

   Limits the total number of log records per second that the collector forwards to this output from a single collector pod. This is an aggregate limit across all pipelines that reference this output. Generally, one collector pod runs per cluster node.

   Use case: Developers can use output rate limits to stay within the ingestion quotas imposed by third-party log aggregation services, preventing overage charges or throttling.

   Important

   When rate limits are exceeded, the collector drops log records. The collector increments the observability_log_forwarder_output_rate_limited_total metric for each dropped record. Monitor this metric to determine if your rate limits are too restrictive.

   Rate limit only on output

   spec:
     outputs:
     - name: rate-limited-kafka
       type: kafka
       kafka:
         brokers:
         - kafka.example.com:9092
         topic: logs
       rateLimit:
         maxRecordsPerSecond: 5000
     pipelines:
     - name: all-logs-to-kafka
       inputRefs:
       - application
       - infrastructure
       outputRefs:
       - rate-limited-kafka

   Rate limit only on input

   spec:
     inputs:
     - name: high-volume-namespace
       type: application
       application:
         includes:
         - namespace: "production"
         tuning:
           rateLimitPerContainer:
             maxRecordsPerSecond: 500
     pipelines:
     - name: production-to-loki
       inputRefs:
       - high-volume-namespace
       outputRefs:
       - default-lokistack

   Different rate limits for different outputs

   spec:
     outputs:
     - name: expensive-splunk
       type: splunk
       splunk:
         url: https://splunk.example.com:8088
         authentication:
           token:
             key: token
             secretName: splunk-secret
       rateLimit:
         maxRecordsPerSecond: 500
     - name: cheap-s3
       type: s3
       s3:
         url: https://s3.us-east-1.amazonaws.com
         bucket: log-archive
       rateLimit:
         maxRecordsPerSecond: 10000
     pipelines:
     - name: critical-to-splunk
       inputRefs:
       - audit
       outputRefs:
       - expensive-splunk
     - name: bulk-to-s3
       inputRefs:
       - application
       - infrastructure
       outputRefs:
       - cheap-s3

2. Apply the updated ClusterLogForwarder CR by running the following command:

   $ oc apply -f <filename>.yaml

Procedure

1. Set the log level by running the following command:

   $ oc annotate clusterlogforwarder instance -n openshift-logging \
     observability.openshift.io/log-level=debug --overwrite

   The supported log levels are trace, debug, info, warn, error, and off. The default level is info.

2. Verify that the collector pod restarted with the new log level by running the following command:

   $ oc logs -n openshift-logging -l app.kubernetes.io/component=collector --tail=5

   At the debug level, entries begin with DEBUG.

   Note

   Use debug or trace only for troubleshooting. These levels increase log volume significantly and can affect collector performance. Reset the level to info after troubleshooting.

Procedure

1. Patch the ClusterLogForwarder CR by running the following command:

   $ oc patch clusterlogforwarder instance -n openshift-logging \
     --type=merge \
     -p '{"spec":{"collector":{"resources":{"requests":{"cpu":"100m","memory":"256Mi"},"limits":{"cpu":"500m","memory":"512Mi"}}}}}'

2. Verify that the collector pod restarted with the updated resources by running the following command:

   $ oc get pods -n openshift-logging -l app.kubernetes.io/component=collector

   The pod age should show a recent restart.

   Note

   If you do not set resource limits explicitly, the collector applies default limits: 500m CPU and 64Mi memory for requests, and 6000m CPU and 2048Mi memory for limits. Setting explicit limits prevents the collector from consuming more resources than expected on nodes with high log volume.

Procedure

1. Create a LogFileMetricExporter CR as a YAML file:

   apiVersion: logging.openshift.io/v1alpha1
   kind: LogFileMetricExporter
   metadata:
     name: instance
     namespace: openshift-logging
   spec:
     nodeSelector: {}
     resources:
       limits:
         cpu: 500m
         memory: 256Mi
       requests:
         cpu: 200m
         memory: 128Mi
     tolerations: []

   spec.nodeSelector

   Optional: Define the nodes that the exporter pods run on.

   spec.resources

   Optional: Define CPU and memory resource limits and requests for the exporter.

   spec.tolerations

   Optional: Define tolerations that the exporter pods accept.

   Note

   The LogFileMetricExporter is a singleton. The Operator only honors openshift-logging/instance for the namespace and name.

2. Apply the LogFileMetricExporter CR by running the following command:

   $ oc apply -f <filename>.yaml

1. Check the ClusterLogForwarder status by running the following command:

   $ oc get clusterlogforwarder instance -n openshift-logging \
     -o jsonpath='{.status.conditions}' | python3 -m json.tool

2. If Authorized shows False, grant the missing ClusterRoleBinding. See "Granting collector permissions for log collection."
3. If Valid shows False, review the message for details about the validation error.

Procedure

1. Create a secret using your Google service account key.

   $ oc -n openshift-logging create secret generic gcp-secret --from-file google-application-credentials.json=<your_service_account_key_file.json>

2. Create a ClusterLogForwarder Custom Resource YAML using the template below:

   apiVersion: observability.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: <log_forwarder_name>
     namespace: openshift-logging
   spec:
     serviceAccount:
       name: <service_account_name>
     outputs:
       - name: gcp-1
         type: googleCloudLogging
         googleCloudLogging:
           authentication:
             credentials:
               secretName: gcp-secret
               key: google-application-credentials.json
           id:
             type : project
             value: openshift-gce-devel
           logId : app-gcp
     pipelines:
       - name: test-app
         inputRefs:
           - application
         outputRefs:
           - gcp-1

   serviceAccount.name

   Specify the name of your service account.

   value

   Set a project, folder, organization, or billingAccount field and its corresponding value, depending on where you want to store your logs in the GCP resource hierarchy.

   logId

   Set the value to add to the logName field of the log entry. The value can be a combination of static and dynamic values consisting of field paths followed by ||, followed by another field path or a static value. You must encase a dynamic value in single curly brackets {} and follow it with a static fallback value separated with ||. Static values can only contain alphanumeric characters along with dashes, underscores, dots and forward slashes.

   inputRefs

   Specify the names of inputs defined in the input.name field for this pipeline. You can also use the built-in values application, infrastructure, audit.

Procedure

1. Create or edit a ClusterLogForwarder custom resource (CR) to enable forwarding using OTLP.

   apiVersion: observability.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     annotations:
       observability.openshift.io/tech-preview-otlp-output: "enabled" 

   1

   
     name: clf-otlp
   spec:
     serviceAccount:
       name: <service_account_name>
     outputs:
     - name: otlp
       type: otlp
       otlp:
         tuning:
           compression: gzip
           deliveryMode: AtLeastOnce
           maxRetryDuration: 20
           maxWrite: 10M
           minRetryDuration: 5
         url: <otlp_url>
     pipelines:
     - inputRefs:
       - application
       - infrastructure
       - audit
       name: otlp-logs
       outputRefs:
       - otlp

   url

   This URL must be absolute and is a placeholder for the OTLP endpoint where logs are sent.

   tuning.compression

   Compression algorithm to use. Valid values are gzip, snappy, or none.

   tuning.deliveryMode

   Delivery mode for log forwarding. Valid values are AtLeastOnce or AtMostOnce.

   tuning.maxRetryDuration

   Maximum time in seconds to retry failed deliveries.

   tuning.maxWrite

   Maximum size of a single write request.

   tuning.minRetryDuration

   Minimum time in seconds between retries.

   Note

   The OTLP output uses the OpenTelemetry data model, which is different from the ViaQ data model that is used by other output types. For more information, see OpenTelemetry Semantic Conventions.

2. Apply the ClusterLogForwarder custom resource:

   $ oc apply -f <filename>.yaml

Verification

1. Verify that the ClusterLogForwarder status shows Ready:

   $ oc get clusterlogforwarder clf-otlp -n openshift-logging \
     -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'

   The output should show True.

2. Verify that OTLP is functioning correctly by checking the LokiStack metrics dashboard:

   1. In the OpenShift Container Platform web console, navigate to Observe → Dashboards.
   2. From the Dashboard dropdown, select OpenShift Logging / LokiStack / Writes.
   3. Check the Distributor - structured metadata panel for incoming OTLP log records.

Procedure

1. Create or edit a YAML file that defines the ClusterLogForwarder CR object:

   apiVersion: observability.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: <log_forwarder_name>
     namespace: <log_forwarder_namespace>
   spec:
     serviceAccount:
       name: <service_account_name>
     outputs:
     - name: loki-output
       type: loki
       loki:
         authentication:
           username:
             key: username
             secretName: to-loki-secret
           password:
             key: password
             secretName: to-loki-secret
           token:
             from: secret
             secret:
               key: ca-bundle.crt
               name: to-loki-secret
         labelKeys:
           - <label_keys>
         tenantKey: '{.kubernetes.namespace_name||"application"}'
         url: https://loki.secure.com:3100
     pipelines:
     - name:  my-pipeline
       inputRefs:
       - application
       - audit
       outputRefs:
       - loki-output

   serviceAccount.name

   Specify the name of your service account.

   outputs.name

   Specify a name for the output.

   type

   Specify the type as loki.

   authentication

   Specify the authentication information.

   labelKeys

   Specify the log record keys to map to Loki stream labels. If you do not set the labelKeys field, the ClusterLogForwarder CR uses these default keys: log_type, kubernetes.container_name, kubernetes.namespace_name, kubernetes.pod_name.

   tenantKey

   Specify the tenant for the logs. The value can be a combination of static and dynamic values consisting of field paths separated by ||. Encase dynamic values inside single curly brackets {}. Follow dynamic values with a static fallback value.

   url

   Specify the URL for Loki.

   inputRefs

   Specify the names of inputs defined in the input.name field for this pipeline. You can also use the built-in values application, infrastructure, audit.

   outputRefs

   Specify the names of outputs defined in the outputs.name field for this pipeline.

   Note

   Because Loki requires log streams to be correctly ordered by timestamp, labelKeys always includes the kubernetes_host label set, even if you do not specify it. This inclusion ensures that each stream originates from a single host, which prevents timestamps from becoming disordered due to clock differences on different hosts.

2. Apply the ClusterLogForwarder CR object by running the following command:

   $ oc apply -f <filename>.yaml

Procedure

1. Create or edit a YAML file that defines the ClusterLogForwarder CR object:

   apiVersion: observability.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: <log_forwarder_name>
     namespace: <log_forwarder_namespace>
   spec:
     serviceAccount:
       name: <service_account_name>
     outputs:
      - name: kafka-output
        type: kafka
        kafka:
          authentication:
            sasl:
              username:
                key: <key>
                secretName: kafka-secret
              password:
               key: <key>
               secretName:  kafka-secret
              mechanism: <sasl_mechanism>
          url: tls://kafka.example.devlab.com:9093/app-topic
          brokers:
            - tls://kafka-broker1.example.com:9093
            - tls://kafka-broker2.example.com:9093
          topic:
     pipelines:
       - name: app-topic
         inputRefs:
           - application
         outputRefs:
           - kafka-output

   serviceAccount.name

   Specify the name of your service account.

   outputs.name

   Specify a name for the output.

   type

   Specify the kafka type.

   secretName

   If you use a tls prefix in the URL, you must specify the name of the secret required by the endpoint for TLS communication.

   mechanism

   Specify the Simple Authentication and Security Layer (SASL) mechanism to use. For example, SCRAM-SHA-256,SCRAM-SHA-512, or PLAIN. The default value is PLAIN.

   url

   Optional: Specify the URL and port of the Kafka broker as a valid absolute URL, optionally with a specific topic. You can use the tcp (insecure) or tls (secure TCP) protocol. If you enable the cluster-wide proxy by using the CIDR annotation, the output must be a server name or FQDN, and not an IP address. You must specify either the URL or Kafka brokers.

   brokers

   Optional: Specify a list of broker endpoints of a Kafka cluster.

   topic

   Specify the target topic. By default, the value for the field is topic. The topic name can be a combination of static and dynamic values consisting of field paths separated by "||". Encase dynamic values in single curly brackets "{}". Follow dynamic values with a static fallback value. The topic specified here overrides the topic defined in the url field.

2. Apply the ClusterLogForwarder CR by running the following command:

   $ oc apply -f <filename>.yaml

Procedure

1. Create a secret with your shared key:

   apiVersion: v1
   kind: Secret
   metadata:
     name: my-secret
     namespace: openshift-logging
   type: Opaque
   data:
     shared_key: <your_shared_key>

   shared_key

   Must contain a primary or secondary key for the Log Analytics workspace making the request.

2. Obtain a shared key by running the following command in the Azure CLI:

   Get-AzOperationalInsightsWorkspaceSharedKey -ResourceGroupName "<resource_name>" -Name "<workspace_name>”

3. Create or edit your ClusterLogForwarder CR using the template matching your log selection.

   Forward all logs

   apiVersion: observability.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: <log_forwarder_name>
     namespace: openshift-logging
   spec:
     serviceAccount:
       name: <service_account_name>
     outputs:
     - name: azure-monitor
       type: azureMonitor
       azureMonitor:
         customerId: my-customer-id
         logType: my_log_type
         authentication:
           sharedKey:
             secretName: my-secret
             key: shared_key
     pipelines:
       - name: app-pipeline
         inputRefs:
         - application
         outputRefs:
         - azure-monitor

   serviceAccount.name

   The name of your service account.

   customerId

   Unique identifier for the Log Analytics workspace. Required field.

   logType

   Record type of the submitted data. Can only contain letters, numbers, and underscores (_), and cannot exceed 100 characters. For more information, see Azure record type in the Microsoft Azure documentation.

Procedure

1. Create or edit a YAML file that defines the ClusterLogForwarder CR:

   Example ClusterLogForwarder CR

   apiVersion: observability.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: <log_forwarder_name>
     namespace: <log_forwarder_namespace>
   spec:
     serviceAccount:
       name: <service_account_name>
     outputs:
     - name: <output_name>
       type: <output_type>
     inputs:
     - name: my-app-logs
       type: application
       application:
         includes:
         - namespace: my-project
     filters:
     - name: my-project-labels
       type: openshiftLabels
       openshiftLabels:
         project: my-project
     - name: cluster-labels
       type: openshiftLabels
       openshiftLabels:
         clusterId: C1234
     pipelines:
     - name: <pipeline_name>
       inputRefs:
       - my-app-logs
       outputRefs:
       - <output_name>
       filterRefs:
       - my-project-labels
       - cluster-labels

   inputs.name

   Specify the name for the input.

   type

   Specify the type as application to collect logs from applications.

   includes

   Specify the set of namespaces and containers to include when collecting logs.

   openshiftLabels

   Specify the labels to apply to log records passing through this pipeline. These labels appear in the openshift.labels map in the log record.

   pipelines.name

   Specify a name for the pipeline.

2. Apply the ClusterLogForwarder CR by running the following command:

   $ oc apply -f <filename>.yaml

Procedure

1. Create or edit a YAML file that defines the ClusterLogForwarder CR object. In the file, specify the pod labels using simple equality-based selectors under inputs[].name.application.selector.matchLabels, as shown in the following example.

   apiVersion: observability.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: <log_forwarder_name>
     namespace: <log_forwarder_namespace>
   spec:
     serviceAccount:
       name: <service_account_name>
     outputs:
       - <output_name>
       # ...
     inputs:
     - name: exampleAppLogData
       type: application
       application:
         includes:
         - namespace: app1
         - namespace: app2
         selector:
           matchLabels:
             environment: production
             app: nginx
     pipelines:
     - inputRefs:
       - exampleAppLogData
       outputRefs:
       # ...

   serviceAccount.name

   Specify the service account name.

   inputs.name

   Specify a name for the input.

   type

   Specify the type as application to collect logs from applications.

   includes

   Specify the set of namespaces to include when collecting logs.

   matchLabels

   Specify the key-value pairs of pod labels whose log data you want to gather. You must specify both a key and value, not just a key. The collector selects only pods that match all the key-value pairs.

2. Optional: You can send log data from additional applications that have different pod labels to the same pipeline.

   1. For each unique combination of pod labels, create an additional inputs[].name section similar to the one shown.
   2. Update the selectors to match the pod labels of this application.

   3. Add the new inputs[].name value to inputRefs. For example:

      - inputRefs: [ myAppLogData, myOtherAppLogData ]

3. Create the CR object:

   $ oc create -f <file_name>.yaml

Procedure

1. Create a Secret YAML file that uses the aws_access_key_id and aws_secret_access_key fields to specify your base64-encoded AWS credentials. For example:

   apiVersion: v1
   kind: Secret
   metadata:
     name: aws-secret
     namespace: <namespace_name>
   data:
     aws_access_key_id: QUtJQUlPU0ZPRE5ON0VYQU1QTEUK
     aws_secret_access_key: d0phbHJYVXRuRkVNSS9LN01ERU5HL2JQeFJmaUNZRVhBTVBMRUtFWQo=

   namespace

   Specify the namespace where your ClusterLogForwarder CR is deployed. The secret must be in the same namespace as the ClusterLogForwarder CR.

2. Create the secret. For example:

   $ oc apply -f aws-secret.yaml

3. Create or edit a YAML file that defines the ClusterLogForwarder CR object. In the file, specify the name of the secret. For example:

   apiVersion: observability.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: <log_forwarder_name>
     namespace: <log_forwarder_namespace>
   spec:
     serviceAccount:
       name: <service_account_name>
     outputs:
      - name: cw-output
        type: cloudwatch
        cloudwatch:
          authentication:
            type: awsAccessKey
            awsAccessKey:
              keyId:
                key: <key>
                secretName: cw-secret
              keySecret:
                key: <key>
                secretName: cw-secret
          groupName: <prefix-{.dynamic_value||"none"}>
          region: us-east-2
     pipelines:
       - name: infra-logs
         inputRefs:
           - infrastructure
           - audit
           - application
         outputRefs:
           - cw-output

   serviceAccount.name

   Specify the name of your service account.

   outputs.name

   Specify a name for the output.

   type

   Specify the cloudwatch type.

   secretName

   Specify the name of the secret that has your AWS credentials.

   groupName

   Specify the strategy for grouping log streams. The value can be a combination of static and dynamic values consisting of field paths separated by "||". Encase dynamic values in single curly brackets "{}". Follow dynamic values with a static fallback value.

   region

   Specify the AWS region.

   pipelines.name

   Specify a name for the pipeline.

   inputRefs

   Specify the names of inputs defined in the input.name field for this pipeline. You can also use the built-in values application, infrastructure, audit.

   output

   Specify the names of outputs defined in the outputs.name field for this pipeline.

4. Apply the CR object:

   $ oc apply -f <file_name>.yaml

Procedure

1. Create a Secret YAML file that uses the aws_access_key_id and aws_secret_access_key fields to specify your base64-encoded AWS credentials. For example:

   apiVersion: v1
   kind: Secret
   metadata:
     name: aws-secret
     namespace: <namespace_name>
   data:
     aws_access_key_id: QUtJQUlPU0ZPRE5ON0VYQU1QTEUK
     aws_secret_access_key: d0phbHJYVXRuRkVNSS9LN01ERU5HL2JQeFJmaUNZRVhBTVBMRUtFWQo=

   namespace

   Specify the namespace where your ClusterLogForwarder CR is deployed. The secret must be in the same namespace as the ClusterLogForwarder CR.

2. Create the secret. For example:

   $ oc apply -f aws-secret.yaml

3. Create or edit a YAML file that defines the ClusterLogForwarder CR object and specify the secret name. For example:

   apiVersion: observability.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: <log_forwarder_name>
   spec:
     serviceAccount:
       name: <service_account_name>
     outputs:
     - name: s3bucket
       type: s3
       s3:
         authentication:
           type: awsAccessKey
           awsAccessKey:
             keyId:
               key: aws_access_key_id
               secretName: aws-secret
             keySecret:
               key: aws_secret_access_key
               secretName: aws-secret
         bucket: <bucket_name>
         keyPrefix: 's3{.log_type||"missing"}'
         region: us-east-1
         url: 'https://s3.us-east-1.amazonaws.com'
     pipelines:
     - name: pipe1
       inputRefs:
       - application
       outputRefs:
       - s3bucket

   serviceAccount.name

   Specify the name of your service account.

   outputs.name

   Specify a name for the output.

   type

   Specify the s3 type.

   secretName

   Specify the name of the secret that has your AWS credentials.

   bucket

   Specify the Amazon S3 bucket name for log storage.

   keyPrefix

   Specify the S3 key prefix for log objects. The value can be a combination of static and dynamic values consisting of field paths separated by "||". Encase dynamic values in single curly brackets "{}". You must follow dynamic values with a static fallback value.

   url

   Optional: Specify the S3 endpoint URL. If you do not specify this, the system uses the default AWS S3 endpoint https://s3.<region>.amazonaws.com based on the region. Use this field for S3-compatible storage endpoints or custom endpoints.

4. Apply the CR object:

   $ oc apply -f <file_name>.yaml