Installing logging

Procedure

1. Create a Namespace for the Loki Operator:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: openshift-operators-redhat
     labels:
       openshift.io/cluster-monitoring: "true"

   where:

   • name: openshift-operators-redhat: Use this namespace for Red Hat Operators. Avoid using openshift-operators, which might contain untrusted community Operators and cause metric conflicts.
   • openshift.io/cluster-monitoring: "true": Ensures that cluster monitoring scrapes metrics from this namespace.

2. Apply the Namespace:

   $ oc apply -f <filename>.yaml

3. Create an OperatorGroup:

   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: loki-operator
     namespace: openshift-operators-redhat
   spec:
     upgradeStrategy: Default

   where:

   • namespace: openshift-operators-redhat: Specifies the namespace for the Operator.

4. Apply the OperatorGroup:

   $ oc apply -f <filename>.yaml

5. Create a Subscription for the Loki Operator:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: loki-operator
     namespace: openshift-operators-redhat
   spec:
     channel: stable-6.<y>
     installPlanApproval: Automatic
     name: loki-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace

   where:

   • channel: stable-6.<y>: Specifies the update channel.
   • installPlanApproval: Automatic: Automatically installs updates when available. Set to Manual to approve updates manually.
   • source: redhat-operators: Uses the Red Hat-provided catalog. For disconnected clusters, specify your custom CatalogSource.

6. Apply the Subscription:

   $ oc apply -f <filename>.yaml

7. Create a Namespace for the LokiStack:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: openshift-logging
     labels:
       openshift.io/cluster-monitoring: "true"

   where:

   • name: openshift-logging: Dedicated namespace for logging workloads.
   • openshift.io/cluster-monitoring: "true": Enables metric collection.

8. Apply the Namespace:

   $ oc apply -f <filename>.yaml

9. Create a secret for object storage credentials:

   apiVersion: v1
   kind: Secret
   metadata:
     name: logging-loki-s3
     namespace: openshift-logging
   stringData:
     access_key_id: <access_key_id>
     access_key_secret: <secret_access_key>
     bucketnames: <s3_bucket_name>
     endpoint: https://s3.eu-central-1.amazonaws.com
     region: eu-central-1

   where:

   • name: logging-loki-s3: Must match the secret name referenced in the LokiStack.
   Important

   If there is no retention period defined on the s3 bucket or in the LokiStack custom resource (CR), then the logs are not pruned and they stay in the s3 bucket forever, which might fill up the s3 storage.

10. Apply the secret:

    $ oc apply -f <filename>.yaml

11. Create a LokiStack custom resource (CR):

    apiVersion: loki.grafana.com/v1
    kind: LokiStack
    metadata:
      name: logging-loki
      namespace: openshift-logging
    spec:
      managementState: Managed
      limits:
        global:
          retention:
            days: 20
      size: 1x.small
      storage:
        schemas:
        - version: v13
          effectiveDate: "<yyyy>-<mm>-<dd>"
        secret:
          name: logging-loki-s3
          type: s3
      storageClassName: <storage_class_name>
      tenants:
        mode: openshift-logging

    where:

    • size: 1x.small: Select a deployment size. Supported production sizes include 1x.extra-small, 1x.small, and 1x.medium.
    • effectiveDate: Set a date two months before the current date.
    • storageClassName: Specify a storage class for temporary storage. Use block storage for better performance.
    • mode: openshift-logging: Enables multitenant logging for application, infrastructure, and audit logs.

12. Apply the LokiStack CR:

    $ oc apply -f <filename>.yaml

Verification

1. Verify that the pods are running:

   $ oc get pods -n openshift-logging

   You get an output similar to the following example:

   NAME                                               READY   STATUS    RESTARTS   AGE
   logging-loki-compactor-0                           1/1     Running   0          42m
   logging-loki-distributor-xxxx                      1/1     Running   0          42m
   ...

Procedure

1. Create an OperatorGroup object:

   The following example displays a OperatorGroup object:

   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: cluster-logging
     namespace: openshift-logging
   spec:
     upgradeStrategy: Default

   where:

   • namespace: openshift-logging: You must specify openshift-logging as the namespace.

2. Apply the OperatorGroup object by running the following command:

   $ oc apply -f <filename>.yaml

3. Create a Subscription object for Red Hat OpenShift Logging Operator:

   The following example displays a Subscription object:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: cluster-logging
     namespace: openshift-logging
   spec:
     channel: stable-6.<y>
     installPlanApproval: Automatic
     name: cluster-logging
     source: redhat-operators
     sourceNamespace: openshift-marketplace

   where:

   • namespace: openshift-logging: You must specify openshift-logging as the namespace.
   • channel: stable-6.<y>: Specify stable-6.<y> as the channel.
   • installPlanApproval: Automatic: If the approval strategy in the subscription is set to Automatic, the update process initiates as soon as a new operator version is available in the selected channel. If the approval strategy is set to Manual, you must manually approve pending updates.
   • source: redhat-operators: Specify redhat-operators as the value. If your OpenShift Container Platform cluster is installed on a restricted network, also known as a disconnected cluster, specify the name of the CatalogSource object that you created when you configured Operator Lifecycle Manager (OLM).

4. Apply the Subscription object by running the following command:

   $ oc apply -f <filename>.yaml

5. Create a service account to be used by the log collector:

   $ oc create sa logging-collector -n openshift-logging

6. Assign the necessary permissions to the service account for the collector to be able to collect and forward logs. In this example, the collector is provided permissions to collect logs from both infrastructure and application logs.

   $ oc adm policy add-cluster-role-to-user logging-collector-logs-writer -z logging-collector -n openshift-logging

   $ oc adm policy add-cluster-role-to-user collect-application-logs -z logging-collector -n openshift-logging

   $ oc adm policy add-cluster-role-to-user collect-infrastructure-logs -z logging-collector -n openshift-logging

7. Create a ClusterLogForwarder CR:

   The following example displays a ClusterLogForwarder CR:

   apiVersion: observability.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: instance
     namespace: openshift-logging
   spec:
     serviceAccount:
       name: logging-collector
     outputs:
     - name: lokistack-out
       type: lokiStack
       lokiStack:
         target:
           name: logging-loki
           namespace: openshift-logging
         authentication:
           token:
             from: serviceAccount
       tls:
         ca:
           key: service-ca.crt
           configMapName: openshift-service-ca.crt
     pipelines:
     - name: infra-app-logs
       inputRefs:
       - application
       - infrastructure
       outputRefs:
       - lokistack-out

   where:

   • namespace: openshift-logging: You must specify the openshift-logging namespace.
   • name: logging-collector: Specify the name of the service account created before.
   • type: lokiStack: Select the lokiStack output type to send logs to the LokiStack instance.
   • outputs.lokiStack.target: Point the ClusterLogForwarder to the LokiStack instance created earlier.
   • pipelines.inputRefs: Select the log output types you want to send to the LokiStack instance.

8. Apply the ClusterLogForwarder CR object by running the following command:

   $ oc apply -f <filename>.yaml

Verification

1. Verify the installation by running the following command:

   $ oc get pods -n openshift-logging

   You get an output similar to the following example:

   $ oc get pods -n openshift-logging
   NAME                                               READY   STATUS    RESTARTS   AGE
   cluster-logging-operator-fb7f7cf69-8jsbq           1/1     Running   0          98m
   instance-222js                                     2/2     Running   0          18m
   instance-g9ddv                                     2/2     Running   0          18m
   instance-hfqq8                                     2/2     Running   0          18m
   instance-sphwg                                     2/2     Running   0          18m
   instance-vv7zn                                     2/2     Running   0          18m
   instance-wk5zz                                     2/2     Running   0          18m
   logging-loki-compactor-0                           1/1     Running   0          42m
   logging-loki-distributor-7d7688bcb9-dvcj8          1/1     Running   0          42m
   logging-loki-gateway-5f6c75f879-bl7k9              2/2     Running   0          42m
   logging-loki-gateway-5f6c75f879-xhq98              2/2     Running   0          42m
   logging-loki-index-gateway-0                       1/1     Running   0          42m
   logging-loki-ingester-0                            1/1     Running   0          42m
   logging-loki-querier-6b7b56bccc-2v9q4              1/1     Running   0          42m
   logging-loki-query-frontend-84fb57c578-gq2f7       1/1     Running   0          42m

Procedure

1. Install the Cluster Observability Operator.

2. Create a UIPlugin custom resource (CR):

   The following example displays a UIPlugin CR:

   apiVersion: observability.openshift.io/v1alpha1
   kind: UIPlugin
   metadata:
     name: logging
   spec:
     type: Logging
     logging:
       lokiStack:
         name: logging-loki
       logsLimit: 50
       timeout: 30s
       schema: otel

   where:

   • name: logging: Set name to logging.
   • type: Logging: Set type to Logging.
   • name: logging-loki: The name value must match the name of your LokiStack instance. If you did not install LokiStack in the openshift-logging namespace, set the LokiStack namespace under the lokiStack configuration.

   • schema: otel: schema is one of otel, viaq, or select. The default is viaq if no value is specified. When you choose select, you can select the mode in the UI when you run a query.

     Note

     These are the known issues for the logging UI plugin. For more information, see OU-587.

     • The schema feature is only supported in Red Hat OpenShift Logging 4.15 and later. In earlier versions of Red Hat OpenShift Logging, the logging UI plugin will only use the viaq attribute, ignoring any other values that might be set.
     • Non-administrator users cannot query logs using the otel attribute with logging for Red Hat OpenShift versions 5.8 to 6.2. This issue will be fixed in a future logging release. (LOG-6589)
     • In logging for Red Hat OpenShift version 5.9, the severity_text Otel attribute is not set.

3. Apply the UIPlugin CR object by running the following command:

   $ oc apply -f <filename>.yaml

Verification

1. Access the Red Hat OpenShift Logging web console, and refresh the page if a pop-up message instructs you to do so.
2. Navigate to the Observe → Logs panel, where you can run LogQL queries. You can also query logs for individual pods from the Aggregated Logs tab of a specific pod.

Procedure

1. In the OpenShift Container Platform web console:

   1. Click Operators > OperatorHub, if your version of OpenShift Container Platform is 4.19 or earlier.
   2. Click Ecosystem > Software Catalog, if your version of OpenShift Container Platform is 4.20 or later.

2. Type Loki Operator in the Filter by keyword field. Click Loki Operator in the list of available Operators, and then click Install.

   Important

   The Community Loki Operator is not supported by Red Hat.

3. Select stable-x.y as the Update channel.

   The Loki Operator must be deployed to the global Operator group namespace openshift-operators-redhat, so the Installation mode and Installed Namespace are already selected. If this namespace does not already exist, it will be created for you.

4. Select Enable Operator-recommended cluster monitoring on this namespace.

   This option sets the openshift.io/cluster-monitoring: "true" label in the Namespace object. You must select this option to ensure that cluster monitoring scrapes the openshift-operators-redhat namespace.

5. For Update approval select Automatic, then click Install.

   If the approval strategy in the subscription is set to Automatic, the update process initiates as soon as a new Operator version is available in the selected channel. If the approval strategy is set to Manual, you must manually approve pending updates.

   Note

   An Operator might display a Failed status before the installation completes. If the Operator install completes with an InstallSucceeded message, refresh the page.

6. While the Operator installs, create the namespace to which the log store will be deployed.

   1. Click + in the top right of the screen to access the Import YAML page.

   2. Add the YAML definition for the openshift-logging namespace:

      The following example displays a namespace object:

      apiVersion: v1
      kind: Namespace
      metadata:
        name: openshift-logging
        labels:
          openshift.io/cluster-monitoring: "true"

      where:

      • name: openshift-logging: The openshift-logging namespace is dedicated for all logging workloads.
      • openshift.io/cluster-monitoring: "true": A string value that specifies the label, as shown, to ensure that cluster monitoring scrapes the openshift-logging namespace.
   3. Click Create.

7. Create a secret with the credentials to access the object storage.

   1. Click + in the top right of the screen to access the Import YAML page.

   2. Add the YAML definition for the secret. For example, create a secret to access Amazon Web Services (AWS) s3:

      The following example displays a Secret object:

      apiVersion: v1
      kind: Secret
      metadata:
        name: logging-loki-s3
        namespace: openshift-logging
      stringData:
        access_key_id: <access_key_id>
        access_key_secret: <secret_access_key>
        bucketnames: <s3_bucket_name>
        endpoint: https://s3.eu-central-1.amazonaws.com
        region: eu-central-1

      where:

      • name: logging-loki-s3: Note down the name used for the secret logging-loki-s3 to use it later when creating the LokiStack resource.

      • namespace: openshift-logging: Set the namespace to openshift-logging as that will be the namespace used to deploy LokiStack.

        Important

        If there is no retention period defined on the s3 bucket or in the LokiStack custom resource (CR), then the logs are not pruned and they stay in the s3 bucket forever, which might fill up the s3 storage.

   3. Click Create.
8. Navigate to the Installed Operators page. Select the Loki Operator under the Provided APIs find the LokiStack resource and click Create Instance.

9. Select YAML view, and then use the following template to create a LokiStack CR:

   The following example displays a LokiStack CR:

   apiVersion: loki.grafana.com/v1
   kind: LokiStack
   metadata:
     name: logging-loki
     namespace: openshift-logging
   spec:
     managementState: Managed
     limits:
       global:
         retention:
           days: 20 # Set the value as per requirement
     size: 1x.small
     storage:
       schemas:
       - version: v13
         effectiveDate: "<yyyy>-<mm>-<dd>"
       secret:
         name: logging-loki-s3
         type: s3
     storageClassName: <storage_class_name>
     tenants:
       mode: openshift-logging

   where:

   • name: logging-loki: Use the name logging-loki.
   • namespace: openshift-logging: You must specify openshift-logging as the namespace.
   • global: Define global limits that apply to the LokiStack instance. For information about setting stream-based retention. This field does not impact the retention period for stored logs in object storage.
   • retention: Retention is enabled in the cluster when this block is added to the CR.
   • size: 1x.small: Specify the deployment size. Supported size options for production instances of Loki are 1x.extra-small, 1x.small, or 1x.medium. Additionally, 1x.pico is supported starting with logging 6.1.
   • effectiveDate: "<yyyy>-<mm>-<dd>: Set the date two months ago.
   • name: logging-loki-s3: Specify the name of your log store secret.
   • type: s3: Specify the corresponding storage type.
   • storageClassName: <storage_class_name>: Specify the name of a storage class for temporary storage. For best performance, specify a storage class that allocates block storage. You can list the available storage classes for your cluster by using the oc get storageclasses command.
   • mode: openshift-logging: The openshift-logging mode is the default tenancy mode where a tenant is created for log types, such as audit, infrastructure, and application. This enables access control for individual users and user groups to different log streams.
10. Click Create.

Verification

1. In the LokiStack tab verify that you see your LokiStack instance.
2. In the Status column, verify that you see the message Condition: Ready with a green checkmark.

Procedure

1. In the OpenShift Container Platform web console Administrator perspective, go to Operators → OperatorHub.
2. Type Red Hat OpenShift Logging Operator in the Filter by keyword field. Click Red Hat OpenShift Logging Operator in the list of available Operators, and then click Install.

3. Select stable-x.y as the Update channel. The latest version is already selected in the Version field.

   The Red Hat OpenShift Logging Operator must be deployed to the logging namespace openshift-logging, so the Installation mode and Installed Namespace are already selected. If this namespace does not already exist, it will be created for you.

4. Select Enable Operator-recommended cluster monitoring on this namespace.

   This option sets the openshift.io/cluster-monitoring: "true" label in the Namespace object. You must select this option to ensure that cluster monitoring scrapes the openshift-logging namespace.

5. For Update approval select Automatic, then click Install.

   If the approval strategy in the subscription is set to Automatic, the update process initiates as soon as a new Operator version is available in the selected channel. If the approval strategy is set to Manual, you must manually approve pending updates.

   Note

   An Operator might display a Failed status before the installation completes. If the operator installation completes with an InstallSucceeded message, refresh the page.

6. While the Operator installs, create the service account that will be used by the log collector to collect the logs.

   1. Click the + in the top right of the screen to access the Import YAML page.

   2. Enter the YAML definition for the service account.

      The following example displays a ServiceAccount object:

      apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: logging-collector
        namespace: openshift-logging

      where:

      • name: logging-collector: Note down the name used for the service account logging-collector to use it later when creating the ClusterLogForwarder resource.
      • namespace: openshift-logging: Set the namespace to openshift-logging because that is the namespace for deploying the ClusterLogForwarder resource.
   3. Click the Create button.

7. Create the ClusterRoleBinding objects to grant the necessary permissions to the log collector for accessing the logs that you want to collect and to write the log store, for example infrastructure and application logs.

   1. Click the + in the top right of the screen to access the Import YAML page.

   2. Enter the YAML definition for the ClusterRoleBinding resources.

      The following example displays a ClusterRoleBinding resources:

      apiVersion: rbac.authorization.k8s.io/v1
      kind: ClusterRoleBinding
      metadata:
        name: logging-collector:write-logs
      roleRef:
        apiGroup: rbac.authorization.k8s.io
        kind: ClusterRole
        name: logging-collector-logs-writer
      subjects:
      - kind: ServiceAccount
        name: logging-collector
        namespace: openshift-logging
      ---
      apiVersion: rbac.authorization.k8s.io/v1
      kind: ClusterRoleBinding
      metadata:
        name: logging-collector:collect-application
      roleRef:
        apiGroup: rbac.authorization.k8s.io
        kind: ClusterRole
        name: collect-application-logs
      subjects:
      - kind: ServiceAccount
        name: logging-collector
        namespace: openshift-logging
      ---
      apiVersion: rbac.authorization.k8s.io/v1
      kind: ClusterRoleBinding
      metadata:
        name: logging-collector:collect-infrastructure
      roleRef:
        apiGroup: rbac.authorization.k8s.io
        kind: ClusterRole
        name: collect-infrastructure-logs
      subjects:
      - kind: ServiceAccount
        name: logging-collector
        namespace: openshift-logging

      where:

      • name: logging-collector-logs-writer: The cluster role to allow the log collector to write logs to LokiStack.
      • name: collect-application-logs: The cluster role to allow the log collector to collect logs from applications.
      • name: collect-infrastructure-logs: The cluster role to allow the log collector to collect logs from infrastructure.
   3. Click the Create button.
8. Go to the Operators → Installed Operators page. Select the operator and click the All instances tab.
9. After granting the necessary permissions to the service account, navigate to the Installed Operators page. Select the Red Hat OpenShift Logging Operator under the Provided APIs, find the ClusterLogForwarder resource and click Create Instance.

10. Select YAML view, and then use the following template to create a ClusterLogForwarder CR:

    The following example displays a ClusterLogForwarder CR:

    apiVersion: observability.openshift.io/v1
    kind: ClusterLogForwarder
    metadata:
      name: instance
      namespace: openshift-logging
    spec:
      serviceAccount:
        name: logging-collector
      outputs:
      - name: lokistack-out
        type: lokiStack
        lokiStack:
          target:
            name: logging-loki
            namespace: openshift-logging
          authentication:
            token:
              from: serviceAccount
        tls:
          ca:
            key: service-ca.crt
            configMapName: openshift-service-ca.crt
      pipelines:
      - name: infra-app-logs
        inputRefs:
        - application
        - infrastructure
        outputRefs:
        - lokistack-out

    where:

    • namespace: openshift-logging: You must specify openshift-logging as the namespace.
    • name: logging-collector: Specify the name of the service account created earlier.
    • type: lokiStack: Select the lokiStack output type to send logs to the LokiStack instance.
    • target: Point the ClusterLogForwarder to the LokiStack instance created earlier.
    • inputRefs: Select the log output types you want to send to the LokiStack instance.
11. Click Create.

Verification

1. In the ClusterLogForwarder tab verify that you see your ClusterLogForwarder instance.

2. In the Status column, verify that you see the following messages:

   • Condition: observability.openshift.io/Authorized
   • observability.openshift.io/Valid, Ready

Procedure

1. Install the Cluster Observability Operator.
2. Navigate to the Installed Operators page. Under Provided APIs, select ClusterObservabilityOperator. Find the UIPlugin resource and click Create Instance.

3. Select the YAML view, and then use the following template to create a UIPlugin custom resource (CR):

   The following example displays a UIPlugin CR:

1. Click Create.

Verification

1. Refresh the page when a pop-up message instructs you to do so.
2. Navigate to the Observe → Logs panel, where you can run LogQL queries. You can also query logs for individual pods from the Aggregated Logs tab of a specific pod.