SupportConsoleDevelopersStart a trialContact

Chapter 4. Troubleshooting a cluster update

4.1. Gathering data about your cluster update

When reaching out to Red Hat support for issues with an update, it is important to provide data for the support team to use for troubleshooting your failed cluster update.

4.1.1. Gathering log data for a support case

To gather data from your cluster, including log data, use the oc adm must-gather command. See Gathering data about your cluster.

4.1.2. Gathering cluster update status using oc adm upgrade status (Technology Preview)

When updating your cluster, it is useful to understand how your update is progressing. While the oc adm upgrade command returns limited information about the status of your update, this release introduces the oc adm upgrade status command as a Technology Preview feature. This command decouples status information from the oc adm upgrade command and provides specific information regarding a cluster update, including the status of the control plane and worker node updates.

The oc adm upgrade status command is read-only and will never alter any state in your cluster.

Important

The oc adm upgrade status command is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

The oc adm upgrade status command can be used for clusters from version 4.12 up to the latest supported release.

Important

While your cluster does not need to be a Technology Preview-enabled cluster, you must enable the OC_ENABLE_CMD_UPGRADE_STATUS Technology Preview environment variable, otherwise the OpenShift CLI (oc) will not recognize the command and you will not be able to use the feature.

Procedure

  1. Set the OC_ENABLE_CMD_UPGRADE_STATUS environmental variable to true by running the following command: 

    $ export OC_ENABLE_CMD_UPGRADE_STATUS=true

  2. Run the oc adm upgrade status command: 

    $ oc adm upgrade status

    Example 4.1. Example output for an update progressing successfully

    = Control Plane =
Assessment:      Progressing
Target Version:  4.14.1 (from 4.14.0)
Completion:      97%
Duration:        54m
Operator Status: 32 Healthy, 1 Unavailable

Control Plane Nodes
NAME                                        ASSESSMENT    PHASE      VERSION   EST    MESSAGE
ip-10-0-53-40.us-east-2.compute.internal    Progressing   Draining   4.14.0    +10m
ip-10-0-30-217.us-east-2.compute.internal   Outdated      Pending    4.14.0    ?
ip-10-0-92-180.us-east-2.compute.internal   Outdated      Pending    4.14.0    ?

= Worker Upgrade =

= Worker Pool =
Worker Pool:     worker
Assessment:      Progressing
Completion:      0%
Worker Status:   3 Total, 2 Available, 1 Progressing, 3 Outdated, 1 Draining, 0 Excluded, 0 Degraded

Worker Pool Nodes
NAME                                        ASSESSMENT    PHASE      VERSION   EST    MESSAGE
ip-10-0-4-159.us-east-2.compute.internal    Progressing   Draining   4.14.0    +10m
ip-10-0-20-162.us-east-2.compute.internal   Outdated      Pending    4.14.0    ?
ip-10-0-99-40.us-east-2.compute.internal    Outdated      Pending    4.14.0    ?

= Worker Pool =
Worker Pool:     infra
Assessment:      Progressing
Completion:      0%
Worker Status:   1 Total, 0 Available, 1 Progressing, 1 Outdated, 1 Draining, 0 Excluded, 0 Degraded

Worker Pool Node
NAME                                             ASSESSMENT    PHASE      VERSION   EST    MESSAGE
ip-10-0-4-159-infra.us-east-2.compute.internal   Progressing   Draining   4.14.0    +10m

= Update Health =
SINCE   LEVEL   IMPACT   MESSAGE
14m4s   Info    None     Update is proceeding well

    With this information, you can make informed decisions on how to proceed with your update.

4.1.3. Gathering ClusterVersion history

The Cluster Version Operator (CVO) records updates made to a cluster, known as the ClusterVersion history. The entries can reveal correlation between changes in cluster behavior with potential triggers, although correlation does not imply causation.

Note

The initial, minor, and z-stream version updates are stored by the ClusterVersion history. However, the ClusterVersion history has a size limit. If the limit is reached, the oldest z-stream updates in previous minor versions are pruned to accommodate the limit.

You can view the ClusterVersion history by using the OpenShift Container Platform web console or by using the OpenShift CLI (oc).

4.1.3.1. Gathering ClusterVersion history in the OpenShift Container Platform web console

You can view the ClusterVersion history in the OpenShift Container Platform web console.

Prerequisites

  • You have access to the cluster as a user with the cluster-admin role.
  • You have access to the OpenShift Container Platform web console.

Procedure

  • From the web console, click Administration Cluster Settings and review the contents of the Details tab.

4.1.3.2. Gathering ClusterVersion history using the OpenShift CLI (oc)

You can view the ClusterVersion history using the OpenShift CLI (oc).

Prerequisites

  • You have access to the cluster as a user with the cluster-admin role.
  • You have installed the OpenShift CLI (oc).

Procedure

  1. View the cluster update history by entering the following command: 

    $ oc describe clusterversions/version

    Example output

      Desired:
    Channels:
      candidate-4.13
      candidate-4.14
      fast-4.13
      fast-4.14
      stable-4.13
    Image:    quay.io/openshift-release-dev/ocp-release@sha256:a148b19231e4634196717c3597001b7d0af91bf3a887c03c444f59d9582864f4
    URL:      https://access.redhat.com/errata/RHSA-2023:6130
    Version:  4.13.19
  History:
    Completion Time:    2023-11-07T20:26:04Z
    Image:              quay.io/openshift-release-dev/ocp-release@sha256:a148b19231e4634196717c3597001b7d0af91bf3a887c03c444f59d9582864f4
    Started Time:       2023-11-07T19:11:36Z
    State:              Completed
    Verified:           true
    Version:            4.13.19
    Completion Time:    2023-10-04T18:53:29Z
    Image:              quay.io/openshift-release-dev/ocp-release@sha256:eac141144d2ecd6cf27d24efe9209358ba516da22becc5f0abc199d25a9cfcec
    Started Time:       2023-10-04T17:26:31Z
    State:              Completed
    Verified:           true
    Version:            4.13.13
    Completion Time:    2023-09-26T14:21:43Z
    Image:              quay.io/openshift-release-dev/ocp-release@sha256:371328736411972e9640a9b24a07be0af16880863e1c1ab8b013f9984b4ef727
    Started Time:       2023-09-26T14:02:33Z
    State:              Completed
    Verified:           false
    Version:            4.13.12
  Observed Generation:  4
  Version Hash:         CMLl3sLq-EA=
Events:                 <none>

Additional resources

Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

© 2024 Red Hat, Inc.