Chapter 1. Upgrading to logging 6

Procedure

1. Temporarily set the state of the ClusterLogging resource as Unmanaged.

   $ oc -n openshift-logging patch clusterlogging/instance -p '{"spec":{"managementState": "Unmanaged"}}' --type=merge

   Copy to Clipboard Toggle word wrap

2. Remove ClusterLogging ownerReferences from the Elasticsearch resource.

   The following command ensures that the ClusterLogging resource no longer owns the Elasticsearch resource. Updates to the ClusterLogging resource’s logStore field will no longer affect the Elasticsearch resource.

   $ oc -n openshift-logging patch elasticsearch/elasticsearch -p '{"metadata":{"ownerReferences": []}}' --type=merge

   Copy to Clipboard Toggle word wrap

3. Remove ClusterLogging ownerReferences from the Kibana resource.

   The following command ensures that ClusterLogging no longer owns the Kibana resource. Updates to the ClusterLogging resource’s visualization field will no longer affect the Kibana resource.

   $ oc -n openshift-logging patch kibana/kibana -p '{"metadata":{"ownerReferences": []}}' --type=merge

   Copy to Clipboard Toggle word wrap

4. Update cluster logging to use LokiStack as the log store:

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
     namespace: openshift-logging
   spec:
     managementState: "Managed"
     logStore:
       type: "lokistack"
       lokistack:
         name: logging-loki
     collection:
       type: "vector"

   Copy to Clipboard Toggle word wrap

Procedure

1. Install the Cluster Observability Operator. For more information, see Installing the Cluster Observability Operator.
2. Navigate to the Installed Operators page. Under Provided APIs, select ClusterObservabilityOperator. Find the UIPlugin resource and click Create Instance.

3. Select the YAML view, and then use the following template to create a UIPlugin custom resource (CR):

   Example UIPlugin CR

   apiVersion: observability.openshift.io/v1alpha1
   kind: UIPlugin
   metadata:
     name: logging  

   1

   
   spec:
     type: Logging  

   2

   
     logging:
       lokiStack:
         name: logging-loki  

   3

   
       logsLimit: 50
       timeout: 30s
       schema: otel 

   4

   Copy to Clipboard Toggle word wrap

   1

   Set name to logging.

   2

   Set type to Logging.

   3

   The name value must match the name of your LokiStack instance. If you did not install LokiStack in the openshift-logging namespace, set the LokiStack namespace under the lokiStack configuration.

   4

   schema is one of otel, viaq, or select. The default is viaq if no value is specified. When you choose select, you can select the mode in the UI when you run a query.

   Note

   These are the known issues for the logging UI plugin - for more information, see OU-587.

   • The schema feature is only supported in OpenShift Container Platform 4.15 and later. In earlier versions of OpenShift Container Platform, the logging UI plugin will only use the viaq attribute, ignoring any other values that might be set.
   • Non-administrator users cannot query logs using the otel attribute with logging for Red Hat OpenShift versions 5.8 to 6.2. This issue will be fixed in a future logging release. (LOG-6589)
   • In logging for Red Hat OpenShift version 5.9, the severity_text Otel attribute is not set.
4. Click Create.

Verification

1. Refresh the page when a pop-up message instructs you to do so.
2. Navigate to the Observe Logs panel, where you can run LogQL queries. You can also query logs for individual pods from the Aggregated Logs tab of a specific pod.

Procedure

1. Navigate to Operators Installed Operators.
2. Select the openshift-operators-redhat project.
3. Click the Loki Operator.
4. Click Subscription. In the Subscription details section, click the Update channel link. This link text might be stable or stable-5.y, depending on your current update channel.

5. In the Change Subscription Update Channel window, select the update channel, stable-6.y, and click Save. Note the loki-operator.v6.y.z version.

   Important

   Only update to an N+2 version, where N is your current version. For example, if you are upgrading from Logging 5.8, select stable-6.0 as the update channel. Updating to a version that is more than two versions newer is not supported.

6. Wait for a few seconds, then click Operators Installed Operators. Verify that the Loki Operator version matches the latest loki-operator.v6.y.z version.
7. On the Operators Installed Operators page, wait for the Status field to report Succeeded.
8. Check if the LokiStack custom resource contains the v13 schema version and add it if it is missing. For correctly adding the v13 schema version, see "Upgrading the LokiStack storage schema".

Procedure

1. Create a service account by running the following command:

   Note

   If your previous log forwarder is deployed in the namespace openshift-logging and named instance, the earlier versions of the operator created a logcollector service account. This service account gets removed when you delete cluster logging. Therefore, you need to create a new service account. Any other service account will be preserved. and can be used in Logging 6.x.

   $ oc create sa logging-collector -n openshift-logging

   Copy to Clipboard Toggle word wrap

2. Provide the required RBAC permissions to the service account.

   1. Bind the ClusterRole role to the service account to be able to write the logs to the Red Hat LokiStack

      $ oc adm policy add-cluster-role-to-user logging-collector-logs-writer -z logging-collector -n openshift-logging

      Copy to Clipboard Toggle word wrap

   2. Assign permission to collect and forward application logs by running the following command:

      $ oc adm policy add-cluster-role-to-user collect-application-logs -z logging-collector -n openshift-logging

      Copy to Clipboard Toggle word wrap

   3. Assign permission to collect and forward audit logs by running the following command:

      $ oc adm policy add-cluster-role-to-user collect-audit-logs -z logging-collector -n openshift-logging

      Copy to Clipboard Toggle word wrap

   4. Assign permission to collect and forward infrastructure logs by running the following command:

       $ oc adm policy add-cluster-role-to-user collect-infrastructure-log -z logging-collector -n openshift-logging

      Copy to Clipboard Toggle word wrap

3. Move Vector checkpoints to the new path.

   The Vector checkpoints in Logging v5 are located at the path /var/lib/vector/input*/checkpoints.json. Move these checkpoints to the path /var/lib/vector/<namespace>/<clusterlogforwarder cr name>/*. The following example uses openshift-logging as the namespace and collector as the ClusterForwarder custom resource name.

   $ ns="openshift-logging"
   $ cr="collector"
   $ for node in $(oc get nodes -o name); do oc debug $node -- chroot /host /bin/bash -c "mkdir -p /var/lib/vector/$ns/$cr" ; done
   $ for node in $(oc get nodes -o name); do oc debug $node -- chroot /host /bin/bash -c "chmod -R 755 /var/lib/vector/$ns" ; done
   $ for node in $(oc get nodes -o name); do echo "### $node ###"; oc debug $node -- chroot /host /bin/bash -c "cp -Ra /var/lib/vector/input* /var/lib/vector/$ns/$cr/"; done

   Copy to Clipboard Toggle word wrap

4. Update the Red Hat OpenShift Logging Operator by using the OpenShift Container Platform web console.

   1. Navigate to Operators Installed Operators.
   2. Select the openshift-logging project.
   3. Click the Red Hat OpenShift Logging Operator.
   4. Click Subscription. In the Subscription details section, click the Update channel link.

   5. In the Change Subscription Update Channel window, select the update channel, stable-6.x, and click Save. Note the cluster-logging.v6.y.z version.

      Important

      Only update to an N+2 version, where N is your current version. For example, if you are upgrading from Logging 5.8, select stable-6.0 as the update channel. Updating to a version that is more than two versions newer is not supported.

   6. Wait for a few seconds, and then go to Operators Installed Operators to verify that the Red Hat OpenShift Logging Operator version matches the latest cluster-logging.v6.y.z version.

   7. On the Operators Installed Operators page, wait for the Status field to report Succeeded.

      Your existing Logging v5 resources will continue to run, but are no longer managed by your operator. These unmanaged resources can be removed once your new resources are ready to be created.

Verification

1. Verify that no collector pods are running by running the following command:

   $ oc get pods -l component=collector -n <namespace>

   Copy to Clipboard Toggle word wrap

2. Verify that no clusterLogForwarder.logging.openshift.io custom resource (CR) exists by running the following command:

   $ oc get clusterlogforwarders.logging.openshift.io -A

   Copy to Clipboard Toggle word wrap

Procedure

1. Go to the Administration Custom Resource Definitions page, and click Elasticsearch.
2. On the Custom Resource Definition Details page, click Instances.
3. Click the Options menu next to the instance, and then click Delete Elasticsearch.
4. Go to the Administration Custom Resource Definitions page.
5. Click the Options menu next to Elasticsearch, and select Delete Custom Resource Definition.
6. Go to the Operators Installed Operators page.
7. Click the Options menu next to the OpenShift Elasticsearch Operator, and then click Uninstall Operator.

8. Optional: Delete the openshift-operators-redhat project.

   Important

   Do not delete the openshift-operators-redhat project if other global Operators are installed in this namespace.

   1. Go to the Home Projects page.
   2. Click the Options menu next to the openshift-operators-redhat project, and then click Delete Project.
   3. Confirm the deletion by typing openshift-operators-redhat in the dialog box, and then click Delete.