Updating clusters

Procedure

1. From the web console, click Administration → Cluster Settings and review the contents of the Details tab.

2. For production clusters, ensure that the Channel is set to the correct channel for the version that you want to update to, such as stable-4.7.

   Important

   For production clusters, you must subscribe to a stable-* or fast-* channel.

   • If the Update status is not Updates available, you cannot update your cluster.
   • Select channel indicates the cluster version that your cluster is running or is updating to.

3. Select a version to update to, and click Save.

   The Input channel Update status changes to Update to <product-version> in progress, and you can review the progress of the cluster update by watching the progress bars for the Operators and nodes.

   Note

   If you are upgrading your cluster to the next minor version, like version 4.y to 4.(y+1), it is recommended to confirm your nodes are updated before deploying workloads that rely on a new feature. Any pools with worker nodes that are not yet updated are displayed on the Cluster Settings page.

4. After the update completes and the Cluster Version Operator refreshes the available updates, check if more updates are available in your current channel.

   • If updates are available, continue to perform updates in the current channel until you can no longer update.
   • If no updates are available, change the Channel to the stable-* or fast-* channel for the next minor version, and update to the version that you want in that channel.

   You might need to perform several intermediate updates until you reach the version that you want.

Procedure

1. Navigate to Administration → Cluster Settings, click version.

2. Click the YAML tab and then edit the upstream parameter value:

   Example output

     ...
     spec:
       clusterID: db93436d-7b05-42cc-b856-43e11ad2d31a
       upstream: '<update-server-url>' 1
     ...

   1

   The <update-server-url> variable specifies the URL for the update server.

   The default upstream is https://api.openshift.com/api/upgrades_info/v1/graph.

3. Click Save.

Procedure

1. Ensure that your cluster is available:

   $ oc get clusterversion

   Example output

   NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
   version   4.6.9     True        False         158m    Cluster version is 4.6.9

2. Review the current update channel information and confirm that your channel is set to stable-4.7:

   $ oc get clusterversion -o json|jq ".items[0].spec"

   Example output

   {
     "channel": "stable-4.7",
     "clusterID": "990f7ab8-109b-4c95-8480-2bd1deec55ff"
   }

   Important

   For production clusters, you must subscribe to a stable-* or fast-* channel.

3. View the available updates and note the version number of the update that you want to apply:

   $ oc adm upgrade

   Example output

   Cluster version is 4.1.0
   
   Updates:
   
   VERSION IMAGE
   4.1.2   quay.io/openshift-release-dev/ocp-release@sha256:9c5f0df8b192a0d7b46cd5f6a4da2289c155fd5302dec7954f8f06c878160b8b

4. Apply an update:

   • To update to the latest version:

     $ oc adm upgrade --to-latest=true 1

   • To update to a specific version:

     $ oc adm upgrade --to=<version> 1

     1 1

     <version> is the update version that you obtained from the output of the previous command.

5. Review the status of the Cluster Version Operator:

   $ oc get clusterversion -o json|jq ".items[0].spec"

   Example output

   {
     "channel": "stable-4.7",
     "clusterID": "990f7ab8-109b-4c95-8480-2bd1deec55ff",
     "desiredUpdate": {
       "force": false,
       "image": "quay.io/openshift-release-dev/ocp-release@sha256:9c5f0df8b192a0d7b46cd5f6a4da2289c155fd5302dec7954f8f06c878160b8b",
       "version": "4.7.0" 1
     }
   }

   1

   If the version number in the desiredUpdate stanza matches the value that you specified, the update is in progress.

6. Review the cluster version status history to monitor the status of the update. It might take some time for all the objects to finish updating.

   $ oc get clusterversion -o json|jq ".items[0].status.history"

   Example output

   [
     {
       "completionTime": null,
       "image": "quay.io/openshift-release-dev/ocp-release@sha256:b8fa13e09d869089fc5957c32b02b7d3792a0b6f36693432acc0409615ab23b7",
       "startedTime": "2021-01-28T20:30:50Z",
       "state": "Partial",
       "verified": true,
       "version": "4.7.0"
     },
     {
       "completionTime": "2021-01-28T20:30:50Z",
       "image": "quay.io/openshift-release-dev/ocp-release@sha256:b8fa13e09d869089fc5957c32b02b7d3792a0b6f36693432acc0409615ab23b7",
       "startedTime": "2021-01-28T17:38:10Z",
       "state": "Completed",
       "verified": false,
       "version": "4.7.0"
     }
   ]

   The history contains a list of the most recent versions applied to the cluster. This value is updated when the CVO applies an update. The list is ordered by date, where the newest update is first in the list. Updates in the history have state Completed if the rollout completed and Partial if the update failed or did not complete.

7. After the update completes, you can confirm that the cluster version has updated to the new version:

   $ oc get clusterversion

   Example output

   NAME      VERSION     AVAILABLE   PROGRESSING   SINCE     STATUS
   version   4.7.0       True        False         2m        Cluster version is 4.7.0

8. If you are upgrading your cluster to the next minor version, like version 4.y to 4.(y+1), it is recommended to confirm your nodes are upgraded before deploying workloads that rely on a new feature:

   $ oc get nodes

   Example output

   NAME                           STATUS   ROLES    AGE   VERSION
   ip-10-0-168-251.ec2.internal   Ready    master   82m   v1.20.0
   ip-10-0-170-223.ec2.internal   Ready    master   82m   v1.20.0
   ip-10-0-179-95.ec2.internal    Ready    worker   70m   v1.20.0
   ip-10-0-182-134.ec2.internal   Ready    worker   70m   v1.20.0
   ip-10-0-211-16.ec2.internal    Ready    master   82m   v1.20.0
   ip-10-0-250-100.ec2.internal   Ready    worker   69m   v1.20.0

1. Create MCPs based on the worker pool. The number of nodes in each MCP depends on a few factors, such as your maintenance window duration for each MCP, and the amount of reserve capacity, meaning extra worker nodes, available in your cluster.

   Note

   You can change the maxUnavailable setting in an MCP to specify the percentage or the number of machines that can be updating at any given time. The default is 1.

2. Add a node selector to the custom MCPs. For each node that you do not want to update simultaneously with the rest of the cluster, add a matching label to the nodes. This label associates the node to the MCP.

   Note

   Do not remove the default worker label from the nodes. The nodes must have a role label to function properly in the cluster.

3. Pause the MCPs you do not want to update as part of the update process.

   Note

   Pausing the MCP also pauses the kube-apiserver-to-kubelet-signer automatic CA certificates rotation. New CA certificates are generated at 292 days from the installation date and old certificates are removed 365 days from the installation date. See the Understand CA cert auto renewal in Red Hat OpenShift 4 to find out how much time you have before the next automatic CA certificate rotation. Make sure the pools are unpaused when the CA cert rotation happens. If the MCPs are paused, the cert rotation does not happen, which causes the cluster to become degraded and causes failure in multiple oc commands, including but not limited to oc debug, oc logs, oc exec, and oc attach.

4. Perform the cluster update. The update process updates the MCPs that are not paused, including the control plane nodes (also known as the master nodes).
5. Test the applications on the updated nodes to ensure they are working as expected.
6. Unpause the remaining MCPs one-by-one and test the applications on those nodes until all worker nodes are updated. Unpausing an MCP starts the update process for the nodes associated with that MCP. You can check the progress of the update from the web console by clicking Administration → Cluster settings. Or, use the oc get machineconfigpools CLI command.
7. Optionally, remove the custom label from updated nodes and delete the custom MCPs.

1. Create an MCP from a worker node.

   1. List the worker nodes in your cluster.

      $ oc get -l 'node-role.kubernetes.io/master!=' -o 'jsonpath={range .items[*]}{.metadata.name}{"\n"}{end}' nodes

      Example output

      ci-ln-pwnll6b-f76d1-s8t9n-worker-a-s75z4
      ci-ln-pwnll6b-f76d1-s8t9n-worker-b-dglj2
      ci-ln-pwnll6b-f76d1-s8t9n-worker-c-lldbm

   2. For the nodes you want to delay, add a custom label to the node:

      $ oc label node <node name> node-role.kubernetes.io/<custom-label>=

      For example:

      $ oc label node ci-ln-0qv1yp2-f76d1-kl2tq-worker-a-j2ssz node-role.kubernetes.io/workerpool-canary=

      Example output

      node/ci-ln-gtrwm8t-f76d1-spbl7-worker-a-xk76k labeled

   3. Create the new MCP:

      apiVersion: machineconfiguration.openshift.io/v1
      kind: MachineConfigPool
      metadata:
        name: workerpool-canary 1
      spec:
        machineConfigSelector:
          matchExpressions: 2
            - {
               key: machineconfiguration.openshift.io/role,
               operator: In,
               values: [worker,workerpool-canary]
              }
        nodeSelector:
          matchLabels:
            node-role.kubernetes.io/workerpool-canary: "" 3

      1

      Specify a name for the MCP.

      2

      Specify the worker and custom MCP name.

      3

      Specify the custom label you added to the nodes that you want in this pool.

      $ oc create -f <file_name>

      Example output

      machineconfigpool.machineconfiguration.openshift.io/workerpool-canary created

   4. View the list of MCPs in the cluster and their current state:

      $ oc get machineconfigpool

      Example output

      NAME              CONFIG                                                        UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
      master            rendered-master-b0bb90c4921860f2a5d8a2f8137c1867              True      False      False      3              3                   3                     0                      97m
      workerpool-canary rendered-workerpool-canary-87ba3dec1ad78cb6aecebf7fbb476a36   True      False      False      1              1                   1                     0                      2m42s
      worker            rendered-worker-87ba3dec1ad78cb6aecebf7fbb476a36              True      False      False      2              2                   2                     0                      97m

      The new machine config pool, workerpool-canary, is created and the number of nodes to which you added the custom label are shown in the machine counts. The worker MCP machine counts are reduced by the same number. It can take several minutes to update the machine counts. In this example, one node was moved from the worker MCP to the workerpool-canary MCP.

1. Patch the MCP that you want paused:

   $ oc patch mcp/<mcp_name> --patch '{"spec":{"paused":true}}' --type=merge

   For example:

   $  oc patch mcp/workerpool-canary --patch '{"spec":{"paused":true}}' --type=merge

   Example output

   machineconfigpool.machineconfiguration.openshift.io/workerpool-canary patched

1. Patch the MCP that you want to unpause:

   $ oc patch mcp/<mcp_name> --patch '{"spec":{"paused":false}}' --type=merge

   For example:

   $  oc patch mcp/workerpool-canary --patch '{"spec":{"paused":false}}' --type=merge

   Example output

   machineconfigpool.machineconfiguration.openshift.io/workerpool-canary patched

   You can check the progress of the update by using the oc get machineconfigpools command.

2. Test your applications on the updated nodes to ensure that they are working as expected.
3. Unpause any other paused MCPs one-by-one and verify that your applications work.

1. Remove the custom label from the node.

   $ oc label node <node_name> node-role.kubernetes.io/<custom-label>-

   For example:

   $ oc label node ci-ln-0qv1yp2-f76d1-kl2tq-worker-a-j2ssz node-role.kubernetes.io/workerpool-canary-

   Example output

   node/ci-ln-0qv1yp2-f76d1-kl2tq-worker-a-j2ssz labeled

   The MCO moves the nodes back to the original MCP and reconciles the node to the MCP configuration.

2. View the list of MCPs in the cluster and their current state:

   $oc get mcp

   NAME                CONFIG                                                   UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
   master              rendered-master-1203f157d053fd987c7cbd91e3fbc0ed         True      False      False      3              3                   3                     0                      61m
   workerpool-canary   rendered-mcp-noupdate-5ad4791166c468f3a35cd16e734c9028   True      False      False      0              0                   0                     0                      21m
   worker              rendered-worker-5ad4791166c468f3a35cd16e734c9028         True      False      False      3              3                   3                     0                      61m

   The node is removed from the custom MCP and moved back to the original MCP. It can take several minutes to update the machine counts. In this example, one node was moved from the removed workerpool-canary MCP to the `worker`MCP.

3. Optional: Delete the custom MCP:

   $ oc delete mcp <mcp_name>

Procedure

1. From the web console, click Administration → Cluster Settings and review the contents of the Details tab.

2. For production clusters, ensure that the Channel is set to the correct channel for the version that you want to update to, such as stable-4.7.

   Important

   For production clusters, you must subscribe to a stable-* or fast-* channel.

   • If the Update status is not Updates available, you cannot update your cluster.
   • Select channel indicates the cluster version that your cluster is running or is updating to.

3. Select a version to update to, and click Save.

   The Input channel Update status changes to Update to <product-version> in progress, and you can review the progress of the cluster update by watching the progress bars for the Operators and nodes.

   Note

   If you are upgrading your cluster to the next minor version, like version 4.y to 4.(y+1), it is recommended to confirm your nodes are updated before deploying workloads that rely on a new feature. Any pools with worker nodes that are not yet updated are displayed on the Cluster Settings page.

4. After the update completes and the Cluster Version Operator refreshes the available updates, check if more updates are available in your current channel.

   • If updates are available, continue to perform updates in the current channel until you can no longer update.
   • If no updates are available, change the Channel to the stable-* or fast-* channel for the next minor version, and update to the version that you want in that channel.

   You might need to perform several intermediate updates until you reach the version that you want.

   Note

   When you update a cluster that contains Red Hat Enterprise Linux (RHEL) worker machines, those workers temporarily become unavailable during the update process. You must run the update playbook against each RHEL machine as it enters the NotReady state for the cluster to finish updating.

Procedure

1. Stop and disable firewalld on the host:

   # systemctl disable --now firewalld.service

   Note

   By default, the base OS RHEL with "Minimal" installation option enables firewalld serivce. Having the firewalld service enabled on your host prevents you from accessing OpenShift Container Platform logs on the worker. Do not enable firewalld later if you wish to continue accessing OpenShift Container Platform logs on the worker.

2. Enable the repositories that are required for OpenShift Container Platform 4.7:

   1. On the machine that you run the Ansible playbooks, update the required repositories:

      # subscription-manager repos --disable=rhel-7-server-ose-4.6-rpms \
                                   --enable=rhel-7-server-ansible-2.9-rpms \
                                   --enable=rhel-7-server-ose-4.7-rpms

   2. On the machine that you run the Ansible playbooks, update the required packages, including openshift-ansible:

      # yum update openshift-ansible openshift-clients

   3. On each RHEL compute node, update the required repositories:

      # subscription-manager repos --disable=rhel-7-server-ose-4.6-rpms \
                                   --enable=rhel-7-server-ose-4.7-rpms  \
                                   --enable=rhel-7-fast-datapath-rpms   \
                                   --enable=rhel-7-server-optional-rpms

3. Update a RHEL worker machine:

   1. Review the current node status to determine which RHEL worker to update:

      # oc get node

      Example output

      NAME                        STATUS                        ROLES    AGE    VERSION
      mycluster-control-plane-0   Ready                         master   145m   v1.20.0
      mycluster-control-plane-1   Ready                         master   145m   v1.20.0
      mycluster-control-plane-2   Ready                         master   145m   v1.20.0
      mycluster-rhel7-0           NotReady,SchedulingDisabled   worker   98m    v1.14.6+97c81d00e
      mycluster-rhel7-1           Ready                         worker   98m    v1.14.6+97c81d00e
      mycluster-rhel7-2           Ready                         worker   98m    v1.14.6+97c81d00e
      mycluster-rhel7-3           Ready                         worker   98m    v1.14.6+97c81d00e

      Note which machine has the NotReady,SchedulingDisabled status.

   2. Review your Ansible inventory file at /<path>/inventory/hosts and update its contents so that only the machine with the NotReady,SchedulingDisabled status is listed in the [workers] section, as shown in the following example:

      [all:vars]
      ansible_user=root
      #ansible_become=True
      
      openshift_kubeconfig_path="~/.kube/config"
      
      [workers]
      mycluster-rhel7-0.example.com

   3. Change to the openshift-ansible directory:

      $ cd /usr/share/ansible/openshift-ansible

   4. Run the upgrade playbook:

      $ ansible-playbook -i /<path>/inventory/hosts playbooks/upgrade.yml 1

      1

      For <path>, specify the path to the Ansible inventory file that you created.

      Note

      The upgrade playbook only upgrades the OpenShift Container Platform packages. It does not update the operating system packages.

4. Follow the process in the previous step to update each RHEL worker machine in your cluster.

5. After you update all of the workers, confirm that all of your cluster nodes have updated to the new version:

   # oc get node

   Example output

   NAME                        STATUS                        ROLES    AGE    VERSION
   mycluster-control-plane-0   Ready                         master   145m   v1.20.0
   mycluster-control-plane-1   Ready                         master   145m   v1.20.0
   mycluster-control-plane-2   Ready                         master   145m   v1.20.0
   mycluster-rhel7-0           NotReady,SchedulingDisabled   worker   98m    v1.20.0
   mycluster-rhel7-1           Ready                         worker   98m    v1.20.0
   mycluster-rhel7-2           Ready                         worker   98m    v1.20.0
   mycluster-rhel7-3           Ready                         worker   98m    v1.20.0

6. Optional: Update the operating system packages that were not updated by the upgrade playbook. To update packages that are not on 4.7, use the following command:

   # yum update

   Note

   You do not need to exclude RPM packages if you are using the same RPM repository that you used when you installed 4.7.

1. Download your registry.redhat.io pull secret from the Red Hat OpenShift Cluster Manager and save it to a .json file.

2. Generate the base64-encoded user name and password or token for your mirror registry:

   $ echo -n '<user_name>:<password>' | base64 -w0 1
   BGVtbYk3ZHAtqXs=

   1

   For <user_name> and <password>, specify the user name and password that you configured for your registry.

3. Make a copy of your pull secret in JSON format:

   $ cat ./pull-secret.text | jq .  > <path>/<pull_secret_file_in_json>1

   1

   Specify the path to the folder to store the pull secret in and a name for the JSON file that you create.

4. Save the file either as ~/.docker/config.json or $XDG_RUNTIME_DIR/containers/auth.json.

   The contents of the file resemble the following example:

   {
     "auths": {
       "cloud.openshift.com": {
         "auth": "b3BlbnNo...",
         "email": "you@example.com"
       },
       "quay.io": {
         "auth": "b3BlbnNo...",
         "email": "you@example.com"
       },
       "registry.connect.redhat.com": {
         "auth": "NTE3Njg5Nj...",
         "email": "you@example.com"
       },
       "registry.redhat.io": {
         "auth": "NTE3Njg5Nj...",
         "email": "you@example.com"
       }
     }
   }

5. Edit the new file and add a section that describes your registry to it:

     "auths": {
       "<mirror_registry>": { 1
         "auth": "<credentials>", 2
         "email": "you@example.com"
     },

   1

   For <mirror_registry>, specify the registry domain name, and optionally the port, that your mirror registry uses to serve content. For example, registry.example.com or registry.example.com:8443

   2

   For <credentials>, specify the base64-encoded user name and password for the mirror registry.

   The file resembles the following example:

   {
     "auths": {
       "registry.example.com": {
         "auth": "BGVtbYk3ZHAtqXs=",
         "email": "you@example.com"
       },
       "cloud.openshift.com": {
         "auth": "b3BlbnNo...",
         "email": "you@example.com"
       },
       "quay.io": {
         "auth": "b3BlbnNo...",
         "email": "you@example.com"
       },
       "registry.connect.redhat.com": {
         "auth": "NTE3Njg5Nj...",
         "email": "you@example.com"
       },
       "registry.redhat.io": {
         "auth": "NTE3Njg5Nj...",
         "email": "you@example.com"
       }
     }
   }

Procedure

1. Use the Red Hat OpenShift Container Platform Upgrade Graph visualizer and update planner to plan an update from one version to another. The OpenShift Upgrade Graph provides channel graphs and a way to confirm that there is an update path between your current and intended cluster versions.

2. Set the required environment variables:

   1. Export the release version:

      $ export OCP_RELEASE=<release_version>

      For <release_version>, specify the tag that corresponds to the version of OpenShift Container Platform to which you want to update, such as 4.5.4.

   2. Export the local registry name and host port:

      $ LOCAL_REGISTRY='<local_registry_host_name>:<local_registry_host_port>'

      For <local_registry_host_name>, specify the registry domain name for your mirror repository, and for <local_registry_host_port>, specify the port that it serves content on.

   3. Export the local repository name:

      $ LOCAL_REPOSITORY='<local_repository_name>'

      For <local_repository_name>, specify the name of the repository to create in your registry, such as ocp4/openshift4.

   4. Export the name of the repository to mirror:

      $ PRODUCT_REPO='openshift-release-dev'

      For a production release, you must specify openshift-release-dev.

   5. Export the path to your registry pull secret:

      $ LOCAL_SECRET_JSON='<path_to_pull_secret>'

      For <path_to_pull_secret>, specify the absolute path to and file name of the pull secret for your mirror registry that you created.

      Note

      If your cluster uses an ImageContentSourcePolicy object to configure repository mirroring, you can use only global pull secrets for mirrored registries. You cannot add a pull secret to a project.

   6. Export the release mirror:

      $ RELEASE_NAME="ocp-release"

      For a production release, you must specify ocp-release.

   7. Export the type of architecture for your server, such as x86_64.:

      $ ARCHITECTURE=<server_architecture>

   8. Export the path to the directory to host the mirrored images:

      $ REMOVABLE_MEDIA_PATH=<path> 1

      1

      Specify the full path, including the initial forward slash (/) character.

3. Review the images and configuration manifests to mirror:

   $ oc adm release mirror -a ${LOCAL_SECRET_JSON} --to-dir=${REMOVABLE_MEDIA_PATH}/mirror quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-${ARCHITECTURE} --dry-run

4. Mirror the version images to the mirror registry.

   • If your mirror host does not have internet access, take the following actions:

     1. Connect the removable media to a system that is connected to the internet.

     2. Mirror the images and configuration manifests to a directory on the removable media:

        $ oc adm release mirror -a ${LOCAL_SECRET_JSON} --to-dir=${REMOVABLE_MEDIA_PATH}/mirror quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-${ARCHITECTURE}

     3. Take the media to the restricted network environment and upload the images to the local container registry.

        $ oc image mirror  -a ${LOCAL_SECRET_JSON} --from-dir=${REMOVABLE_MEDIA_PATH}/mirror "file://openshift/release:${OCP_RELEASE}*" ${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} 1

        1

        For REMOVABLE_MEDIA_PATH, you must use the same path that you specified when you mirrored the images.

     4. Use oc command-line interface (CLI) to log in to the cluster that you are upgrading.

     5. Apply the mirrored release image signature config map to the connected cluster:

        $ oc apply -f ${REMOVABLE_MEDIA_PATH}/mirror/config/<image_signature_file> 1

        1

        For <image_signature_file>, specify the path and name of the file, for example, signature-sha256-81154f5c03294534.yaml.

   • If the local container registry and the cluster are connected to the mirror host, directly push the release images to the local registry and apply the config map to the cluster by using following command:

     $ oc adm release mirror -a ${LOCAL_SECRET_JSON} --from=quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-${ARCHITECTURE} \
       --to=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} --apply-release-image-signature

     Note

     If you include the --apply-release-image-signature option, do not create the config map for image signature verification.

Procedure

1. Configure mirrored repositories, by either:

   • Setting up a mirrored repository with Red Hat Quay, as described in Red Hat Quay Repository Mirroring. Using Red Hat Quay allows you to copy images from one repository to another and also automatically sync those repositories repeatedly over time.

   • Using a tool such as skopeo to copy images manually from the source directory to the mirrored repository.

     For example, after installing the skopeo RPM package on a Red Hat Enterprise Linux (RHEL) 7 or RHEL 8 system, use the skopeo command as shown in this example:

     $ skopeo copy \
     docker://registry.access.redhat.com/ubi8/ubi-minimal@sha256:5cfbaf45ca96806917830c183e9f37df2e913b187adb32e89fd83fa455ebaa6 \
     docker://example.io/example/ubi-minimal

     In this example, you have a container image registry that is named example.io with an image repository named example to which you want to copy the ubi8/ubi-minimal image from registry.access.redhat.com. After you create the registry, you can configure your OpenShift Container Platform cluster to redirect requests made of the source repository to the mirrored repository.

2. Log in to your OpenShift Container Platform cluster.

3. Create an ImageContentSourcePolicy file (for example, registryrepomirror.yaml), replacing the source and mirrors with your own registry and repository pairs and images:

   apiVersion: operator.openshift.io/v1alpha1
   kind: ImageContentSourcePolicy
   metadata:
     name: ubi8repo
   spec:
     repositoryDigestMirrors:
     - mirrors:
       - example.io/example/ubi-minimal 1
       source: registry.access.redhat.com/ubi8/ubi-minimal 2
     - mirrors:
       - example.com/example/ubi-minimal
       source: registry.access.redhat.com/ubi8/ubi-minimal
     - mirrors:
       - mirror.example.com/redhat
       source: registry.redhat.io/openshift4 3

   1

   Indicates the name of the image registry and repository.

   2

   Indicates the registry and repository containing the content that is mirrored.

   3

   You can configure a namespace inside a registry to use any image in that namespace. If you use a registry domain as a source, the ImageContentSourcePolicy resource is applied to all repositories from the registry.

4. Create the new ImageContentSourcePolicy object:

   $ oc create -f registryrepomirror.yaml

   After the ImageContentSourcePolicy object is created, the new settings are deployed to each node and the cluster starts using the mirrored repository for requests to the source repository.

5. To check that the mirrored configuration settings, are applied, do the following on one of the nodes.

   1. List your nodes:

      $ oc get node

      Example output

      NAME                           STATUS                     ROLES    AGE  VERSION
      ip-10-0-137-44.ec2.internal    Ready                      worker   7m   v1.20.0
      ip-10-0-138-148.ec2.internal   Ready                      master   11m  v1.20.0
      ip-10-0-139-122.ec2.internal   Ready                      master   11m  v1.20.0
      ip-10-0-147-35.ec2.internal    Ready,SchedulingDisabled   worker   7m   v1.20.0
      ip-10-0-153-12.ec2.internal    Ready                      worker   7m   v1.20.0
      ip-10-0-154-10.ec2.internal    Ready                      master   11m  v1.20.0

      You can see that scheduling on each worker node is disabled as the change is being applied.

   2. Start the debugging process to access the node:

      $ oc debug node/ip-10-0-147-35.ec2.internal

      Example output

      Starting pod/ip-10-0-147-35ec2internal-debug ...
      To use host binaries, run `chroot /host`

   3. Access the node’s files:

      sh-4.2# chroot /host

   4. Check the /etc/containers/registries.conf file to make sure the changes were made:

      sh-4.2# cat /etc/containers/registries.conf

      Example output

      unqualified-search-registries = ["registry.access.redhat.com", "docker.io"]
      [[registry]]
        location = "registry.access.redhat.com/ubi8/"
        insecure = false
        blocked = false
        mirror-by-digest-only = true
        prefix = ""
      
        [[registry.mirror]]
          location = "example.io/example/ubi8-minimal"
          insecure = false
      
        [[registry.mirror]]
          location = "example.com/example/ubi8-minimal"
          insecure = false

   5. Pull an image digest to the node from the source and check if it is resolved by the mirror. ImageContentSourcePolicy objects support image digests only, not image tags.

      sh-4.2# podman pull --log-level=debug registry.access.redhat.com/ubi8/ubi-minimal@sha256:5cfbaf45ca96806917830c183e9f37df2e913b187adb32e89fd83fa455ebaa6

Procedure

1. Run the following command, specifying values for <local_registry>, <pull_spec>, and <pull_secret_file>:

   $ oc adm catalog mirror <local_registry>/<pull_spec> <local_registry> -a <pull_secret_file> --icsp-scope=registry

   where:

   <local_registry>

   is the local registry you have configured for your disconnected cluster, for example, local.registry:5000.

   <pull_spec>

   is the pull specification as configured in your disconnected registry, for example, redhat/redhat-operator-index:v4.7

   <pull_secret_file>

   is the registry.redhat.io pull secret in .json file format. You can download the pull secret from the Red Hat OpenShift Cluster Manager.

   The oc adm catalog mirror command creates a /redhat-operator-index-manifests directory and generates imageContentSourcePolicy.yaml, catalogSource.yaml, and mapping.txt files.

2. Apply the new ImageContentSourcePolicy resource to the cluster:

   $ oc apply -f imageContentSourcePolicy.yaml