Updating clusters

1. The target version is stored in the spec.desiredUpdate.version field of the ClusterVersion resource, which may be managed through the web console or the CLI.
2. The Cluster Version Operator (CVO) detects that the desiredUpdate in the ClusterVersion resource differs from the current cluster version. Using graph data from the OpenShift Update Service, the CVO resolves the desired cluster version to a pull spec for the release image.
3. The CVO validates the integrity and authenticity of the release image. Red Hat publishes cryptographically-signed statements about published release images at predefined locations by using image SHA digests as unique and immutable release image identifiers. The CVO utilizes a list of built-in public keys to validate the presence and signatures of the statement matching the checked release image.
4. The CVO creates a job named version-$version-$hash in the openshift-cluster-version namespace. This job uses containers that are executing the release image, so the cluster downloads the image through the container runtime. The job then extracts the manifests and metadata from the release image to a shared volume that is accessible to the CVO.
5. The CVO validates the extracted manifests and metadata.
6. The CVO checks some preconditions to ensure that no problematic condition is detected in the cluster. Certain conditions can prevent updates from proceeding. These conditions are either determined by the CVO itself, or reported by individual cluster Operators that detect some details about the cluster that the Operator considers problematic for the update.
7. The CVO records the accepted release in status.desired and creates a status.history entry about the new update.
8. The CVO begins reconciling the manifests from the release image. Cluster Operators are updated in separate stages called Runlevels, and the CVO ensures that all Operators in a Runlevel finish updating before it proceeds to the next level.
9. Manifests for the CVO itself are applied early in the process. When the CVO deployment is applied, the current CVO pod stops, and a CVO pod that uses the new version starts. The new CVO proceeds to reconcile the remaining manifests.
10. The update proceeds until the entire control plane is updated to the new version. Individual cluster Operators might perform update tasks on their domain of the cluster, and while they do so, they report their state through the Progressing=True condition.
11. The Machine Config Operator (MCO) manifests are applied towards the end of the process. The updated MCO then begins updating the system configuration and operating system of every node. Each node might be drained, updated, and rebooted before it starts to accept workloads again.

1. Cordon and drain the node

   Note

   When a node is cordoned, workloads cannot be scheduled to it.

2. Update the system configuration and operating system (OS) of the node
3. Reboot the node
4. Uncordon the node

1. The MCO cordons nodes 1, 2, and 3, and begins to drain them.
2. Node 2 finishes draining, reboots, and becomes available again. The MCO cordons node 4 and begins draining it.
3. Node 1 finishes draining, reboots, and becomes available again. The MCO cordons node 5 and begins draining it.
4. Node 3 finishes draining, reboots, and becomes available again.
5. Node 5 finishes draining, reboots, and becomes available again.
6. Node 4 finishes draining, reboots, and becomes available again.

1. Cordon and drain all the nodes
2. Update the operating system (OS)
3. Reboot the nodes
4. Uncordon all nodes and schedule workloads on the node

Procedure

1. Using the Administrator perspective on the web console, update any Operator Lifecycle Manager (OLM) Operators to the versions that are compatible with your intended updated version. You can find more information on how to perform this action in "Updating installed Operators"; see "Additional resources".

2. Verify that all machine config pools display a status of Up to date and that no machine config pool displays a status of UPDATING.

   To view the status of all machine config pools, click Compute → MachineConfigPools and review the contents of the Update status column.

   Note

   If your machine config pools have an Updating status, please wait for this status to change to Up to date. This process could take several minutes.

3. Set your channel to eus-<4.y+2>.

   To set your channel, click Administration → Cluster Settings → Channel. You can edit your channel by clicking on the current hyperlinked channel.

4. Pause all worker machine pools except for the master pool. You can perform this action on the MachineConfigPools tab under the Compute page. Select the vertical ellipses next to the machine config pool you’d like to pause and click Pause updates.
5. Update to version <4.y+1> and complete up to the Save step. You can find more information on how to perform these actions in "Updating a cluster by using the web console"; see "Additional resources".
6. Ensure that the <4.y+1> updates are complete by viewing the Last completed version of your cluster. You can find this information on the Cluster Settings page under the Details tab.
7. If necessary, update your OLM Operators by using the Administrator perspective on the web console. You can find more information on how to perform these actions in "Updating installed Operators"; see "Additional resources".
8. Update to version <4.y+2> and complete up to the Save step. You can find more information on how to perform these actions in "Updating a cluster by using the web console"; see "Additional resources".
9. Ensure that the <4.y+2> update is complete by viewing the Last completed version of your cluster. You can find this information on the Cluster Settings page under the Details tab.

10. Unpause all previously paused machine config pools. You can perform this action on the MachineConfigPools tab under the Compute page. Select the vertical ellipses next to the machine config pool you’d like to unpause and click Unpause updates.

    Important

    If pools are not unpaused, the cluster is not permitted to upgrade to any future minor versions, and maintenance tasks such as certificate rotation are inhibited. This puts the cluster at risk for future degradation.

11. Verify that your previously paused pools are updated and that your cluster has completed the update to version <4.y+2>.

    You can verify that your pools have updated on the MachineConfigPools tab under the Compute page by confirming that the Update status has a value of Up to date.

    You can verify that your cluster has completed the update by viewing the Last completed version of your cluster. You can find this information on the Cluster Settings page under the Details tab.

Procedure

1. Using the Administrator perspective on the web console, update any Operator Lifecycle Manager (OLM) Operators to the versions that are compatible with your intended updated version. You can find more information on how to perform this action in "Updating installed Operators"; see "Additional resources".

2. Verify that all machine config pools display a status of UPDATED and that no machine config pool displays a status of UPDATING. To view the status of all machine config pools, run the following command:

   $ oc get mcp

   Example output

   NAME     CONFIG                                         	UPDATED   UPDATING
   master   rendered-master-ecbb9582781c1091e1c9f19d50cf836c       True  	  False
   worker   rendered-worker-00a3f0c68ae94e747193156b491553d5       True  	  False

3. Your current version is <4.y>, and your intended version to update is <4.y+2>. Change to the eus-<4.y+2> channel by running the following command:

   $ oc adm upgrade channel eus-<4.y+2>

   Note

   If you receive an error message indicating that eus-<4.y+2> is not one of the available channels, this indicates that Red Hat is still rolling out EUS version updates. This rollout process generally takes 45-90 days starting at the GA date.

4. Pause all worker machine pools except for the master pool by running the following command:

   $ oc patch mcp/worker --type merge --patch '{"spec":{"paused":true}}'

   Note

   You cannot pause the master pool.

5. Update to the latest version by running the following command:

   $ oc adm upgrade --to-latest

   Example output

   Updating to latest version <4.y+1.z>

6. Review the cluster version to ensure that the updates are complete by running the following command:

   $ oc adm upgrade

   Example output

   Cluster version is <4.y+1.z>
   ...

7. Update to version <4.y+2> by running the following command:

   $ oc adm upgrade --to-latest

8. Retrieve the cluster version to ensure that the <4.y+2> updates are complete by running the following command:

   $ oc adm upgrade

   Example output

   Cluster version is <4.y+2.z>
   ...

9. To update your worker nodes to <4.y+2>, unpause all previously paused machine config pools by running the following command:

   $ oc patch mcp/worker --type merge --patch '{"spec":{"paused":false}}'

   Important

   If pools are not unpaused, the cluster is not permitted to update to any future minor versions, and maintenance tasks such as certificate rotation are inhibited. This puts the cluster at risk for future degradation.

10. Verify that your previously paused pools are updated and that the update to version <4.y+2> is complete by running the following command:

    $ oc get mcp

    Example output

    NAME 	   CONFIG                                            UPDATED     UPDATING
    master   rendered-master-52da4d2760807cb2b96a3402179a9a4c    True  	 False
    worker   rendered-worker-4756f60eccae96fb9dcb4c392c69d497    True 	 False

1. Ensure that all of your Operators previously installed through OLM are updated to their latest version in their latest channel. Updating the Operators ensures that they have a valid update path when the default OperatorHub catalogs switch from the current minor version to the next during a cluster update. For information on how to update your Operators, see "Preparing for an Operator update" in "Additional resources".
2. Confirm the cluster version compatibility between the current and intended Operator versions. You can verify which versions your OLM Operators are compatible with by using the Red Hat OpenShift Container Platform Operator Update Information Checker.

Example workflow

1. Pause the worker machine pools.
2. Upgrade OpenShift <4.y> → OpenShift <4.y+1>.
3. Upgrade ODF <4.y> → ODF <4.y+1>.
4. Upgrade OpenShift <4.y+1> → OpenShift <4.y+2>.
5. Upgrade to ODF <4.y+2>.
6. Unpause the worker machine pools.

Procedure

1. Log in to the OpenShift Container Platform web console as a user with the cluster-admin role.
2. Navigate to Administration → Cluster Settings.
3. On the Cluster Settings page, select the Configuration tab.
4. Under Configuration resource, select CloudCredential.
5. On the CloudCredential details page, select the YAML tab.

6. In the YAML block, check the value of spec.credentialsMode. The following values are possible, though not all are supported on all platforms:

   • '': The CCO is operating in the default mode. In this configuration, the CCO operates in mint or passthrough mode, depending on the credentials provided during installation.
   • Mint: The CCO is operating in mint mode.
   • Passthrough: The CCO is operating in passthrough mode.
   • Manual: The CCO is operating in manual mode.
   Important

   To determine the specific configuration of an AWS or GCP cluster that has a spec.credentialsMode of '', Mint, or Manual, you must investigate further.

   AWS and GCP clusters support using mint mode with the root secret deleted. If the cluster is specifically configured to use mint mode or uses mint mode by default, you must determine if the root secret is present on the cluster before updating.

   An AWS or GCP cluster that uses manual mode might be configured to create and manage cloud credentials from outside of the cluster using the AWS Security Token Service (STS) or GCP Workload Identity. You can determine whether your cluster uses this strategy by examining the cluster Authentication object.

7. AWS or GCP clusters that use mint mode only: To determine whether the cluster is operating without the root secret, navigate to Workloads → Secrets and look for the root secret for your cloud provider.

   Note

   Ensure that the Project dropdown is set to All Projects.

   Platform	Secret name
   AWS	aws-creds
   GCP	gcp-credentials

   • If you see one of these values, your cluster is using mint or passthrough mode with the root secret present.
   • If you do not see these values, your cluster is using the CCO in mint mode with the root secret removed.

8. AWS or GCP clusters that use manual mode only: To determine whether the cluster is configured to create and manage cloud credentials from outside of the cluster, you must check the cluster Authentication object YAML values.

   1. Navigate to Administration → Cluster Settings.
   2. On the Cluster Settings page, select the Configuration tab.
   3. Under Configuration resource, select Authentication.
   4. On the Authentication details page, select the YAML tab.

   5. In the YAML block, check the value of the .spec.serviceAccountIssuer parameter.

      • A value that contains a URL that is associated with your cloud provider indicates that the CCO is using manual mode with AWS STS or GCP Workload Identity to create and manage cloud credentials from outside of the cluster. These clusters are configured using the ccoctl utility.
      • An empty value ('') indicates that the cluster is using the CCO in manual mode but was not configured using the ccoctl utility.

Procedure

1. Log in to oc on the cluster as a user with the cluster-admin role.

2. To determine the mode that the CCO is configured to use, enter the following command:

   $ oc get cloudcredentials cluster \
     -o=jsonpath={.spec.credentialsMode}

   The following output values are possible, though not all are supported on all platforms:

   • '': The CCO is operating in the default mode. In this configuration, the CCO operates in mint or passthrough mode, depending on the credentials provided during installation.
   • Mint: The CCO is operating in mint mode.
   • Passthrough: The CCO is operating in passthrough mode.
   • Manual: The CCO is operating in manual mode.
   Important

   To determine the specific configuration of an AWS or GCP cluster that has a spec.credentialsMode of '', Mint, or Manual, you must investigate further.

   AWS and GCP clusters support using mint mode with the root secret deleted. If the cluster is specifically configured to use mint mode or uses mint mode by default, you must determine if the root secret is present on the cluster before updating.

   An AWS or GCP cluster that uses manual mode might be configured to create and manage cloud credentials from outside of the cluster using the AWS Security Token Service (STS) or GCP Workload Identity. You can determine whether your cluster uses this strategy by examining the cluster Authentication object.

3. AWS or GCP clusters that use mint mode only: To determine whether the cluster is operating without the root secret, run the following command:

   $ oc get secret <secret_name> \
     -n=kube-system

   where <secret_name> is aws-creds for AWS or gcp-credentials for GCP.

   If the root secret is present, the output of this command returns information about the secret. An error indicates that the root secret is not present on the cluster.

4. AWS or GCP clusters that use manual mode only: To determine whether the cluster is configured to create and manage cloud credentials from outside of the cluster, run the following command:

   $ oc get authentication cluster \
     -o jsonpath \
     --template='{ .spec.serviceAccountIssuer }'

   This command displays the value of the .spec.serviceAccountIssuer parameter in the cluster Authentication object.

   • An output of a URL that is associated with your cloud provider indicates that the CCO is using manual mode with AWS STS or GCP Workload Identity to create and manage cloud credentials from outside of the cluster. These clusters are configured using the ccoctl utility.
   • An empty output indicates that the cluster is using the CCO in manual mode but was not configured using the ccoctl utility.

Procedure

1. After you design the hook, create a YAML file that defines the Ansible tasks for it. This file must be a set of tasks and cannot be a playbook, as shown in the following example:

   ---
   # Trivial example forcing an operator to acknowledge the start of an upgrade
   # file=/home/user/openshift-ansible/hooks/pre_compute.yml
   
   - name: note the start of a compute machine update
     debug:
         msg: "Compute machine upgrade of {{ inventory_hostname }} is about to start"
   
   - name: require the user agree to start an upgrade
     pause:
         prompt: "Press Enter to start the compute machine update"

2. Modify the hosts Ansible inventory file to specify the hook files. The hook files are specified as parameter values in the [all:vars] section, as shown:

   Example hook definitions in an inventory file

   [all:vars]
   openshift_node_pre_upgrade_hook=/home/user/openshift-ansible/hooks/pre_node.yml
   openshift_node_post_upgrade_hook=/home/user/openshift-ansible/hooks/post_node.yml

   To avoid ambiguity in the paths to the hook, use absolute paths instead of a relative paths in their definitions.

Procedure

1. Use the Red Hat OpenShift Container Platform Upgrade Graph visualizer and update planner to plan an update from one version to another. The OpenShift Upgrade Graph provides channel graphs and a way to confirm that there is an update path between your current and intended cluster versions.

2. Set the required environment variables:

   1. Export the release version:

      $ export OCP_RELEASE=<release_version>

      For <release_version>, specify the tag that corresponds to the version of OpenShift Container Platform to which you want to update, such as 4.5.4.

   2. Export the local registry name and host port:

      $ LOCAL_REGISTRY='<local_registry_host_name>:<local_registry_host_port>'

      For <local_registry_host_name>, specify the registry domain name for your mirror repository, and for <local_registry_host_port>, specify the port that it serves content on.

   3. Export the local repository name:

      $ LOCAL_REPOSITORY='<local_repository_name>'

      For <local_repository_name>, specify the name of the repository to create in your registry, such as ocp4/openshift4.

   4. If you are using the OpenShift Update Service, export an additional local repository name to contain the release images:

      $ LOCAL_RELEASE_IMAGES_REPOSITORY='<local_release_images_repository_name>'

      For <local_release_images_repository_name>, specify the name of the repository to create in your registry, such as ocp4/openshift4-release-images.

   5. Export the name of the repository to mirror:

      $ PRODUCT_REPO='openshift-release-dev'

      For a production release, you must specify openshift-release-dev.

   6. Export the path to your registry pull secret:

      $ LOCAL_SECRET_JSON='<path_to_pull_secret>'

      For <path_to_pull_secret>, specify the absolute path to and file name of the pull secret for your mirror registry that you created.

      Note

      If your cluster uses an ImageContentSourcePolicy object to configure repository mirroring, you can use only global pull secrets for mirrored registries. You cannot add a pull secret to a project.

   7. Export the release mirror:

      $ RELEASE_NAME="ocp-release"

      For a production release, you must specify ocp-release.

   8. Export the type of architecture for your server, such as x86_64.:

      $ ARCHITECTURE=<server_architecture>

   9. Export the path to the directory to host the mirrored images:

      $ REMOVABLE_MEDIA_PATH=<path> 1

      1

      Specify the full path, including the initial forward slash (/) character.

3. Review the images and configuration manifests to mirror:

   $ oc adm release mirror -a ${LOCAL_SECRET_JSON} --to-dir=${REMOVABLE_MEDIA_PATH}/mirror quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-${ARCHITECTURE} --dry-run

4. Mirror the version images to the mirror registry.

   • If your mirror host does not have internet access, take the following actions:

     1. Connect the removable media to a system that is connected to the internet.

     2. Mirror the images and configuration manifests to a directory on the removable media:

        $ oc adm release mirror -a ${LOCAL_SECRET_JSON} --to-dir=${REMOVABLE_MEDIA_PATH}/mirror quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-${ARCHITECTURE}

        Note

        This command also generates and saves the mirrored release image signature config map onto the removable media.

     3. Take the media to the disconnected environment and upload the images to the local container registry.

        $ oc image mirror  -a ${LOCAL_SECRET_JSON} --from-dir=${REMOVABLE_MEDIA_PATH}/mirror "file://openshift/release:${OCP_RELEASE}*" ${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} 1

        1

        For REMOVABLE_MEDIA_PATH, you must use the same path that you specified when you mirrored the images.

     4. Use oc command-line interface (CLI) to log in to the cluster that you are upgrading.

     5. Apply the mirrored release image signature config map to the connected cluster:

        $ oc apply -f ${REMOVABLE_MEDIA_PATH}/mirror/config/<image_signature_file> 1

        1

        For <image_signature_file>, specify the path and name of the file, for example, signature-sha256-81154f5c03294534.yaml.

     6. If you are using the OpenShift Update Service, mirror the release image to a separate repository:

        $ oc image mirror -a ${LOCAL_SECRET_JSON} ${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE} ${LOCAL_REGISTRY}/${LOCAL_RELEASE_IMAGES_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE}

   • If the local container registry and the cluster are connected to the mirror host, take the following actions:

     1. Directly push the release images to the local registry and apply the config map to the cluster by using following command:

        $ oc adm release mirror -a ${LOCAL_SECRET_JSON} --from=quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-${ARCHITECTURE} \
          --to=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} --apply-release-image-signature

        Note

        If you include the --apply-release-image-signature option, do not create the config map for image signature verification.

     2. If you are using the OpenShift Update Service, mirror the release image to a separate repository:

        $ oc image mirror -a ${LOCAL_SECRET_JSON} ${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE} ${LOCAL_REGISTRY}/${LOCAL_RELEASE_IMAGES_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE}

Procedure

1. Optional: To append a new pull secret to the existing pull secret, complete the following steps:

   1. Enter the following command to download the pull secret:

      $ oc get secret/pull-secret -n openshift-config --template='{{index .data ".dockerconfigjson" | base64decode}}' ><pull_secret_location> 1

      1

      Provide the path to the pull secret file.

   2. Enter the following command to add the new pull secret:

      $ oc registry login --registry="<registry>" \ 1
      --auth-basic="<username>:<password>" \ 2
      --to=<pull_secret_location> 3

      1

      Provide the new registry. You can include multiple repositories within the same registry, for example: --registry="<registry/my-namespace/my-repository>".

      2

      Provide the credentials of the new registry.

      3

      Provide the path to the pull secret file.

      Alternatively, you can perform a manual update to the pull secret file.

2. Enter the following command to update the global pull secret for your cluster:

   $ oc set data secret/pull-secret -n openshift-config --from-file=.dockerconfigjson=<pull_secret_location> 1

   1

   Provide the path to the new pull secret file.

   This update is rolled out to all nodes, which can take some time depending on the size of your cluster.

   Note

   As of OpenShift Container Platform 4.7.4, changes to the global pull secret no longer trigger a node drain or reboot.

Procedure

1. Create a Dockerfile, for example, ./Dockerfile, containing the following:

   FROM registry.access.redhat.com/ubi8/ubi:8.1
   
   RUN curl -L -o cincinnati-graph-data.tar.gz https://api.openshift.com/api/upgrades_info/graph-data
   
   RUN mkdir -p /var/lib/cincinnati-graph-data && tar xvzf cincinnati-graph-data.tar.gz -C /var/lib/cincinnati-graph-data/ --no-overwrite-dir --no-same-owner
   
   CMD ["/bin/bash", "-c" ,"exec cp -rp /var/lib/cincinnati-graph-data/* /var/lib/cincinnati/graph-data"]

2. Use the docker file created in the above step to build a graph data container image, for example, registry.example.com/openshift/graph-data:latest:

   $ podman build -f ./Dockerfile -t registry.example.com/openshift/graph-data:latest

3. Push the graph data container image created in the previous step to a repository that is accessible to the OpenShift Update Service, for example, registry.example.com/openshift/graph-data:latest:

   $ podman push registry.example.com/openshift/graph-data:latest

   Note

   To push a graph data image to a local registry in a disconnected environment, copy the graph data container image created in the previous step to a repository that is accessible to the OpenShift Update Service. Run oc image mirror --help for available options.

Procedure

1. To list all the available MachineHealthCheck resources that you want to pause, run the following command:

   $ oc get machinehealthcheck -n openshift-machine-api

2. To pause the machine health checks, add the cluster.x-k8s.io/paused="" annotation to the MachineHealthCheck resource. Run the following command:

   $ oc -n openshift-machine-api annotate mhc <mhc-name> cluster.x-k8s.io/paused=""

   The annotated MachineHealthCheck resource resembles the following YAML file:

   apiVersion: machine.openshift.io/v1beta1
   kind: MachineHealthCheck
   metadata:
     name: example
     namespace: openshift-machine-api
     annotations:
       cluster.x-k8s.io/paused: ""
   spec:
     selector:
       matchLabels:
         role: worker
     unhealthyConditions:
     - type:    "Ready"
       status:  "Unknown"
       timeout: "300s"
     - type:    "Ready"
       status:  "False"
       timeout: "300s"
     maxUnhealthy: "40%"
   status:
     currentHealthy: 5
     expectedMachines: 5

   Important

   Resume the machine health checks after updating the cluster. To resume the check, remove the pause annotation from the MachineHealthCheck resource by running the following command:

   $ oc -n openshift-machine-api annotate mhc <mhc-name> cluster.x-k8s.io/paused-

Procedure

1. Run the following command on a device that is connected to the internet:

   $ oc adm release info -o 'jsonpath={.digest}{"\n"}' quay.io/openshift-release-dev/ocp-release:${OCP_RELEASE_VERSION}-${ARCHITECTURE}

   For {OCP_RELEASE_VERSION}, specify the version of OpenShift Container Platform to which you want to update, such as 4.10.16.

   For {ARCHITECTURE}, specify the architecture of the cluster, such as x86_64, aarch64, s390x, or ppc64le.

   Example output

   sha256:a8bfba3b6dddd1a2fbbead7dac65fe4fb8335089e4e7cae327f3bad334add31d

2. Copy the sha256 digest for use when updating your cluster.

Procedure

1. Configure mirrored repositories, by either:

   • Setting up a mirrored repository with Red Hat Quay, as described in Red Hat Quay Repository Mirroring. Using Red Hat Quay allows you to copy images from one repository to another and also automatically sync those repositories repeatedly over time.

   • Using a tool such as skopeo to copy images manually from the source directory to the mirrored repository.

     For example, after installing the skopeo RPM package on a Red Hat Enterprise Linux (RHEL) 7 or RHEL 8 system, use the skopeo command as shown in this example:

     $ skopeo copy \
     docker://registry.access.redhat.com/ubi8/ubi-minimal@sha256:5cfbaf45ca96806917830c183e9f37df2e913b187adb32e89fd83fa455ebaa6 \
     docker://example.io/example/ubi-minimal

     In this example, you have a container image registry that is named example.io with an image repository named example to which you want to copy the ubi8/ubi-minimal image from registry.access.redhat.com. After you create the registry, you can configure your OpenShift Container Platform cluster to redirect requests made of the source repository to the mirrored repository.

2. Log in to your OpenShift Container Platform cluster.

3. Create an ImageContentSourcePolicy file (for example, registryrepomirror.yaml), replacing the source and mirrors with your own registry and repository pairs and images:

   apiVersion: operator.openshift.io/v1alpha1
   kind: ImageContentSourcePolicy
   metadata:
     name: ubi8repo
   spec:
     repositoryDigestMirrors:
     - mirrors:
       - example.io/example/ubi-minimal 1
       - example.com/example/ubi-minimal 2
       source: registry.access.redhat.com/ubi8/ubi-minimal 3
     - mirrors:
       - mirror.example.com/redhat
       source: registry.redhat.io/openshift4 4
     - mirrors:
       - mirror.example.com
       source: registry.redhat.io 5
     - mirrors:
       - mirror.example.net/image
       source: registry.example.com/example/myimage 6
     - mirrors:
       - mirror.example.net
       source: registry.example.com/example 7
     - mirrors:
       - mirror.example.net/registry-example-com
       source: registry.example.com 8

   1

   Indicates the name of the image registry and repository.

   2

   Indicates multiple mirror repositories for each target repository. If one mirror is down, the target repository can use another mirror.

   3

   Indicates the registry and repository containing the content that is mirrored.

   4

   You can configure a namespace inside a registry to use any image in that namespace. If you use a registry domain as a source, the ImageContentSourcePolicy resource is applied to all repositories from the registry.

   5

   If you configure the registry name, the ImageContentSourcePolicy resource is applied to all repositories from a source registry to a mirror registry.

   6

   Pulls the image mirror.example.net/image@sha256:…​.

   7

   Pulls the image myimage in the source registry namespace from the mirror mirror.example.net/myimage@sha256:…​.

   8

   Pulls the image registry.example.com/example/myimage from the mirror registry mirror.example.net/registry-example-com/example/myimage@sha256:…​. The ImageContentSourcePolicy resource is applied to all repositories from a source registry to a mirror registry mirror.example.net/registry-example-com.

4. Create the new ImageContentSourcePolicy object:

   $ oc create -f registryrepomirror.yaml

   After the ImageContentSourcePolicy object is created, the new settings are deployed to each node and the cluster starts using the mirrored repository for requests to the source repository.

5. To check that the mirrored configuration settings, are applied, do the following on one of the nodes.

   1. List your nodes:

      $ oc get node

      Example output

      NAME                           STATUS                     ROLES    AGE  VERSION
      ip-10-0-137-44.ec2.internal    Ready                      worker   7m   v1.24.0
      ip-10-0-138-148.ec2.internal   Ready                      master   11m  v1.24.0
      ip-10-0-139-122.ec2.internal   Ready                      master   11m  v1.24.0
      ip-10-0-147-35.ec2.internal    Ready                      worker   7m   v1.24.0
      ip-10-0-153-12.ec2.internal    Ready                      worker   7m   v1.24.0
      ip-10-0-154-10.ec2.internal    Ready                      master   11m  v1.24.0

      The Imagecontentsourcepolicy resource does not restart the nodes.

   2. Start the debugging process to access the node:

      $ oc debug node/ip-10-0-147-35.ec2.internal

      Example output

      Starting pod/ip-10-0-147-35ec2internal-debug ...
      To use host binaries, run `chroot /host`

   3. Change your root directory to /host:

      sh-4.2# chroot /host

   4. Check the /etc/containers/registries.conf file to make sure the changes were made:

      sh-4.2# cat /etc/containers/registries.conf

      Example output

      unqualified-search-registries = ["registry.access.redhat.com", "docker.io"]
      short-name-mode = ""
      
      [[registry]]
        prefix = ""
        location = "registry.access.redhat.com/ubi8/ubi-minimal"
        mirror-by-digest-only = true
      
        [[registry.mirror]]
          location = "example.io/example/ubi-minimal"
      
        [[registry.mirror]]
          location = "example.com/example/ubi-minimal"
      
      [[registry]]
        prefix = ""
        location = "registry.example.com"
        mirror-by-digest-only = true
      
        [[registry.mirror]]
          location = "mirror.example.net/registry-example-com"
      
      [[registry]]
        prefix = ""
        location = "registry.example.com/example"
        mirror-by-digest-only = true
      
        [[registry.mirror]]
          location = "mirror.example.net"
      
      [[registry]]
        prefix = ""
        location = "registry.example.com/example/myimage"
        mirror-by-digest-only = true
      
        [[registry.mirror]]
          location = "mirror.example.net/image"
      
      [[registry]]
        prefix = ""
        location = "registry.redhat.io"
        mirror-by-digest-only = true
      
        [[registry.mirror]]
          location = "mirror.example.com"
      
      [[registry]]
        prefix = ""
        location = "registry.redhat.io/openshift4"
        mirror-by-digest-only = true
      
        [[registry.mirror]]
          location = "mirror.example.com/redhat"

   5. Pull an image digest to the node from the source and check if it is resolved by the mirror. ImageContentSourcePolicy objects support image digests only, not image tags.

      sh-4.2# podman pull --log-level=debug registry.access.redhat.com/ubi8/ubi-minimal@sha256:5cfbaf45ca96806917830c183e9f37df2e913b187adb32e89fd83fa455ebaa6

Procedure

1. Run the following command, specifying values for <local_registry>, <pull_spec>, and <pull_secret_file>:

   $ oc adm catalog mirror <local_registry>/<pull_spec> <local_registry> -a <pull_secret_file> --icsp-scope=registry

   where:

   <local_registry>

   is the local registry you have configured for your disconnected cluster, for example, local.registry:5000.

   <pull_spec>

   is the pull specification as configured in your disconnected registry, for example, redhat/redhat-operator-index:v4.11

   <pull_secret_file>

   is the registry.redhat.io pull secret in .json file format. You can download the pull secret from the Red Hat OpenShift Cluster Manager.

   The oc adm catalog mirror command creates a /redhat-operator-index-manifests directory and generates imageContentSourcePolicy.yaml, catalogSource.yaml, and mapping.txt files.

2. Apply the new ImageContentSourcePolicy resource to the cluster:

   $ oc apply -f imageContentSourcePolicy.yaml

Procedure

1. List the control plane nodes in your cluster.

   $ oc get nodes -l node-role.kubernetes.io/master

   Example output

   NAME                    STATUS   ROLES    AGE   VERSION
   control-plane-node-0    Ready    master   75m   v1.24.0
   control-plane-node-1    Ready    master   75m   v1.24.0
   control-plane-node-2    Ready    master   75m   v1.24.0

   Note the names of your control plane nodes.

2. Mark the control plane node as unschedulable.

   $ oc adm cordon <control_plane_node>

3. Shut down the virtual machine (VM) associated with the control plane node. Do this in the vSphere client by right-clicking the VM and selecting Power → Shut Down Guest OS. Do not shut down the VM using Power Off because it might not shut down safely.
4. Update the VM in the vSphere client. Follow Upgrade the Compatibility of a Virtual Machine Manually in the VMware documentation for more information.
5. Power on the VM associated with the control plane node. Do this in the vSphere client by right-clicking the VM and selecting Power On.

6. Wait for the node to report as Ready:

   $ oc wait --for=condition=Ready node/<control_plane_node>

7. Mark the control plane node as schedulable again:

   $ oc adm uncordon <control_plane_node>

8. Repeat this procedure for each control plane node in your cluster.

Procedure

1. List the compute nodes in your cluster.

   $ oc get nodes -l node-role.kubernetes.io/worker

   Example output

   NAME              STATUS   ROLES    AGE   VERSION
   compute-node-0    Ready    worker   30m   v1.24.0
   compute-node-1    Ready    worker   30m   v1.24.0
   compute-node-2    Ready    worker   30m   v1.24.0

   Note the names of your compute nodes.

2. Mark the compute node as unschedulable:

   $ oc adm cordon <compute_node>

3. Evacuate the pods from the compute node. There are several ways to do this. For example, you can evacuate all or selected pods on a node:

   $ oc adm drain <compute_node> [--pod-selector=<pod_selector>]

   See the "Understanding how to evacuate pods on nodes" section for other options to evacuate pods from a node.

4. Shut down the virtual machine (VM) associated with the compute node. Do this in the vSphere client by right-clicking the VM and selecting Power → Shut Down Guest OS. Do not shut down the VM using Power Off because it might not shut down safely.
5. Update the VM in the vSphere client. Follow Upgrade the Compatibility of a Virtual Machine Manually in the VMware documentation for more information.
6. Power on the VM associated with the compute node. Do this in the vSphere client by right-clicking the VM and selecting Power On.

7. Wait for the node to report as Ready:

   $ oc wait --for=condition=Ready node/<compute_node>

8. Mark the compute node as schedulable again:

   $ oc adm uncordon <compute_node>

9. Repeat this procedure for each compute node in your cluster.

Procedure

1. If the RHCOS template is configured as a vSphere template follow Convert a Template to a Virtual Machine in the VMware documentation prior to the next step.

   Note

   Once converted from a template, do not power on the virtual machine.

2. Update the VM in the vSphere client. Follow Upgrade the Compatibility of a Virtual Machine Manually in the VMware documentation for more information.
3. Convert the VM in the vSphere client from a VM to template. Follow Convert a Virtual Machine to a Template in the vSphere Client in the VMware documentation for more information.