1.6. Upgrading log collection and forwarding
Log collection and forwarding configurations are now specified under the new API, part of the observability.openshift.io API group.
Vector is the only supported collector implementation.
To upgrade the Red Hat OpenShift Logging Operator, follow these steps:
- Update the log collection and forwarding configurations by going through the changes listed in "Changes to Cluster logging and forwarding in Logging 6".
- Update the Red Hat OpenShift Logging Operator.
1.6.1. Changes to cluster logging and forwarding in Logging 6 リンクのコピーリンクがクリップボードにコピーされました!
Log collection and forwarding configurations are now specified under the new API, part of the observability.openshift.io API group. The following sections highlight the differences from the old API resources.
Vector is the only supported collector implementation.
1.6.1.1. Management, resource allocation, and workload scheduling リンクのコピーリンクがクリップボードにコピーされました!
Configuration for management state, resource requests and limits, tolerations, and node selection is now part of the new ClusterLogForwarder API.
Logging 5.x configuration
apiVersion: "logging.openshift.io/v1"
kind: "ClusterLogging"
spec:
managementState: "Managed"
collection:
resources:
limits: {}
requests: {}
nodeSelector: {}
tolerations: {}
Logging 6 configuration
apiVersion: "observability.openshift.io/v1"
kind: ClusterLogForwarder
spec:
managementState: Managed
collector:
resources:
limits: {}
requests: {}
nodeSelector: {}
tolerations: {}
1.6.1.2. Input specifications リンクのコピーリンクがクリップボードにコピーされました!
The input specification is an optional part of the ClusterLogForwarder specification. Administrators can continue to use the predefined values application, infrastructure, and audit to collect these sources.
Namespace and container inclusions and exclusions have been consolidated into a single field.
5.x application input with namespace and container includes and excludes
apiVersion: "logging.openshift.io/v1"
kind: ClusterLogForwarder
spec:
inputs:
- name: application-logs
type: application
application:
namespaces:
- foo
- bar
includes:
- namespace: my-important
container: main
excludes:
- container: too-verbose
6.x application input with namespace and container includes and excludes
apiVersion: "observability.openshift.io/v1"
kind: ClusterLogForwarder
spec:
inputs:
- name: application-logs
type: application
application:
includes:
- namespace: foo
- namespace: bar
- namespace: my-important
container: main
excludes:
- container: too-verbose
"application", "infrastructure", and "audit" are reserved words and cannot be used as names when defining an input.
Changes to input receivers include:
- Explicit configuration of the type at the receiver level.
- Port settings moved to the receiver level.
5.x input receivers
apiVersion: "logging.openshift.io/v1"
kind: ClusterLogForwarder
spec:
inputs:
- name: an-http
receiver:
http:
port: 8443
format: kubeAPIAudit
- name: a-syslog
receiver:
type: syslog
syslog:
port: 9442
6.x input receivers
apiVersion: "observability.openshift.io/v1"
kind: ClusterLogForwarder
spec:
inputs:
- name: an-http
type: receiver
receiver:
type: http
port: 8443
http:
format: kubeAPIAudit
- name: a-syslog
type: receiver
receiver:
type: syslog
port: 9442
1.6.1.3. Output specifications リンクのコピーリンクがクリップボードにコピーされました!
High-level changes to output specifications include:
- URL settings moved to each output type specification.
- Tuning parameters moved to each output type specification.
- Separation of TLS configuration from authentication.
- Explicit configuration of keys and secret/config map for TLS and authentication.
1.6.1.4. Secrets and TLS configuration リンクのコピーリンクがクリップボードにコピーされました!
Secrets and TLS configurations are now separated into authentication and TLS configuration for each output. They must be explicitly defined in the specification rather than relying on administrators to define secrets with recognized keys. Upgrading TLS and authorization configurations requires administrators to understand previously recognized keys to continue using existing secrets. The examples in this section illustrate how to configure ClusterLogForwarder secrets to forward to existing Red Hat managed log storage solutions.
Logging 6.x output configuration using service account token and config map
...
spec:
outputs:
- lokiStack
authentication:
token:
from: serviceAccount
target:
name: logging-loki
namespace: openshift-logging
name: my-lokistack
tls:
ca:
configMapName: openshift-service-ca.crt
key: service-ca.crt
type: lokiStack
...
Logging 6.x output authentication and TLS configuration using secrets
...
spec:
outputs:
- name: my-output
type: http
http:
url: https://my-secure-output:8080
authentication:
password:
key: pass
secretName: my-secret
username:
key: user
secretName: my-secret
tls:
ca:
key: ca-bundle.crt
secretName: collector
certificate:
key: tls.crt
secretName: collector
key:
key: tls.key
secretName: collector
...
1.6.1.5. Filters and pipeline configuration リンクのコピーリンクがクリップボードにコピーされました!
All attributes of pipelines in previous releases have been converted to filters in this release. Individual filters are defined in the filters spec and referenced by a pipeline.
5.x filters
...
spec:
pipelines:
- name: app-logs
detectMultilineErrors: true
parse: json
labels:
<key>: <value>
...
6.x filters and pipelines spec
...
spec:
filters:
- name: my-multiline
type: detectMultilineException
- name: my-parse
type: parse
- name: my-labels
type: openshiftLabels
openshiftLabels:
<key>: <label>
pipelines:
- name: app-logs
filterRefs:
- my-multiline
- my-parse
- my-labels
...
Drop, Prune, and KubeAPIAudit filters remain unchanged.
1.6.1.6. Validation and status リンクのコピーリンクがクリップボードにコピーされました!
Most validations are now enforced when a resource is created or updated which provides immediate feedback. This is a departure from previous releases where all validation occurred post creation requiring inspection of the resource status location. Some validation still occurs post resource creation for cases where is not possible to do so at creation or update time.
Instances of the ClusterLogForwarder.observability.openshift.io resource must satisfy the following conditions before the operator deploys the log collector:
- Resource status conditions: Authorized, Valid, Ready
- Spec validations: Filters, Inputs, Outputs, Pipelines
All must evaluate to the status value of True.
1.6.2. Updating the Red Hat OpenShift Logging Operator リンクのコピーリンクがクリップボードにコピーされました!
The Red Hat OpenShift Logging Operator does not provide an automated upgrade from Logging 5.x to Logging 6.x because of the different combinations in which Logging can be configured. You must install all the different operators for managing logging separately.
You can update Red Hat OpenShift Logging Operator by either changing the subscription channel in the OpenShift Container Platform web console, or by uninstalling it. The following procedure demonstrates updating Red Hat OpenShift Logging Operator by changing the subscription channel in the OpenShift Container Platform web console.
When you migrate, all the logs that have not been compressed will be reprocessed by Vector. The reprocessing might lead to the following issues:
- Duplicated logs during migration.
- Too many requests in the Log storage receiving the logs, or requests reaching the rate limit.
- Impact on the disk and performance because of reading and processing of all old logs in the collector.
- Impact on the Kube API.
- A peak in memory and CPU use by Vector until all the old logs are processed. The logs can be several GB per node.
Prerequisites
-
You have updated the log collection and forwarding configurations to the
observability.openshift.ioAPI. - You have administrator permissions.
- You have access to the OpenShift Container Platform web console and are viewing the Administrator perspective.
Procedure
Create a service account by running the following command:
注記If your previous log forwarder is deployed in the namespace
openshift-loggingand namedinstance, the earlier versions of the operator created alogcollectorservice account. This service account gets removed when you delete cluster logging. Therefore, you need to create a new service account. Any other service account will be preserved. and can be used in Logging 6.x.$ oc create sa logging-collector -n openshift-loggingProvide the required RBAC permissions to the service account.
Bind the
ClusterRolerole to the service account to be able to write the logs to the Red Hat LokiStack$ oc adm policy add-cluster-role-to-user logging-collector-logs-writer -z logging-collector -n openshift-loggingAssign permission to collect and forward application logs by running the following command:
$ oc adm policy add-cluster-role-to-user collect-application-logs -z logging-collector -n openshift-loggingAssign permission to collect and forward audit logs by running the following command:
$ oc adm policy add-cluster-role-to-user collect-audit-logs -z logging-collector -n openshift-loggingAssign permission to collect and forward infrastructure logs by running the following command:
$ oc adm policy add-cluster-role-to-user collect-infrastructure-log -z logging-collector -n openshift-logging
Move Vector checkpoints to the new path.
The Vector checkpoints in Logging v5 are located at the path
/var/lib/vector/input*/checkpoints.json. Move these checkpoints to the path/var/lib/vector/<namespace>/<clusterlogforwarder cr name>/*. The following example usesopenshift-loggingas the namespace andcollectoras the ClusterForwarder custom resource name.$ ns="openshift-logging" $ cr="collector" $ for node in $(oc get nodes -o name); do oc debug $node -- chroot /host /bin/bash -c "mkdir -p /var/lib/vector/$ns/$cr" ; done $ for node in $(oc get nodes -o name); do oc debug $node -- chroot /host /bin/bash -c "chmod -R 755 /var/lib/vector/$ns" ; done $ for node in $(oc get nodes -o name); do echo "### $node ###"; oc debug $node -- chroot /host /bin/bash -c "cp -Ra /var/lib/vector/input* /var/lib/vector/$ns/$cr/"; doneUpdate the Red Hat OpenShift Logging Operator by using the OpenShift Container Platform web console.
-
Navigate to Operators
Installed Operators. - Select the openshift-logging project.
- Click the Red Hat OpenShift Logging Operator.
- Click Subscription. In the Subscription details section, click the Update channel link.
In the Change Subscription Update Channel window, select the update channel, stable-6.x, and click Save. Note the
cluster-logging.v6.y.zversion.重要Only update to an N+2 version, where N is your current version. For example, if you are upgrading from Logging 5.8, select
stable-6.0as the update channel. Updating to a version that is more than two versions newer is not supported.-
Wait for a few seconds, and then go to Operators
Installed Operators to verify that the Red Hat OpenShift Logging Operator version matches the latest cluster-logging.v6.y.zversion. On the Operators
Installed Operators page, wait for the Status field to report Succeeded. Your existing Logging v5 resources will continue to run, but are no longer managed by your operator. These unmanaged resources can be removed once your new resources are ready to be created.
-
Navigate to Operators