Chapter 8. Updating a restricted network cluster

1. Download your registry.redhat.io pull secret from the Red Hat OpenShift Cluster Manager and save it to a .json file.

2. Generate the base64-encoded user name and password or token for your mirror registry:

   $ echo -n '<user_name>:<password>' | base64 -w0 1
   BGVtbYk3ZHAtqXs=

   1

   For <user_name> and <password>, specify the user name and password that you configured for your registry.

3. Make a copy of your pull secret in JSON format:

   $ cat ./pull-secret.text | jq .  > <path>/<pull_secret_file_in_json>1

   1

   Specify the path to the folder to store the pull secret in and a name for the JSON file that you create.

   The contents of the file resemble the following example:

   {
     "auths": {
       "cloud.openshift.com": {
         "auth": "b3BlbnNo...",
         "email": "you@example.com"
       },
       "quay.io": {
         "auth": "b3BlbnNo...",
         "email": "you@example.com"
       },
       "registry.connect.redhat.com": {
         "auth": "NTE3Njg5Nj...",
         "email": "you@example.com"
       },
       "registry.redhat.io": {
         "auth": "NTE3Njg5Nj...",
         "email": "you@example.com"
       }
     }
   }

4. Edit the new file and add a section that describes your registry to it:

     "auths": {
       "<mirror_registry>": { 1
         "auth": "<credentials>", 2
         "email": "you@example.com"
     },

   1

   For <mirror_registry>, specify the registry domain name, and optionally the port, that your mirror registry uses to serve content. For example, registry.example.com or registry.example.com:8443

   2

   For <credentials>, specify the base64-encoded user name and password for the mirror registry.

   The file resembles the following example:

   {
     "auths": {
       "registry.example.com": {
         "auth": "BGVtbYk3ZHAtqXs=",
         "email": "you@example.com"
       },
       "cloud.openshift.com": {
         "auth": "b3BlbnNo...",
         "email": "you@example.com"
       },
       "quay.io": {
         "auth": "b3BlbnNo...",
         "email": "you@example.com"
       },
       "registry.connect.redhat.com": {
         "auth": "NTE3Njg5Nj...",
         "email": "you@example.com"
       },
       "registry.redhat.io": {
         "auth": "NTE3Njg5Nj...",
         "email": "you@example.com"
       }
     }
   }

Procedure

1. Use the Red Hat OpenShift Container Platform Upgrade Graph visualizer and update planner to plan an update from one version to another. The OpenShift Upgrade Graph provides channel graphs and a way to confirm that there is an update path between your current and intended cluster versions.

2. Set the required environment variables:

   1. Export the release version:

      $ export OCP_RELEASE=<release_version>

      For <release_version>, specify the tag that corresponds to the version of OpenShift Container Platform to which you want to update, such as 4.5.4.

   2. Export the local registry name and host port:

      $ LOCAL_REGISTRY='<local_registry_host_name>:<local_registry_host_port>'

      For <local_registry_host_name>, specify the registry domain name for your mirror repository, and for <local_registry_host_port>, specify the port that it serves content on.

   3. Export the local repository name:

      $ LOCAL_REPOSITORY='<local_repository_name>'

      For <local_repository_name>, specify the name of the repository to create in your registry, such as ocp4/openshift4.

   4. Export the name of the repository to mirror:

      $ PRODUCT_REPO='openshift-release-dev'

      For a production release, you must specify openshift-release-dev.

   5. Export the path to your registry pull secret:

      $ LOCAL_SECRET_JSON='<path_to_pull_secret>'

      For <path_to_pull_secret>, specify the absolute path to and file name of the pull secret for your mirror registry that you created.

      Note

      If your cluster uses an ImageContentSourcePolicy object to configure repository mirroring, you can use only global pull secrets for mirrored registries. You cannot add a pull secret to a project.

   6. Export the release mirror:

      $ RELEASE_NAME="ocp-release"

      For a production release, you must specify ocp-release.

   7. Export the type of architecture for your server, such as x86_64.:

      $ ARCHITECTURE=<server_architecture>

   8. Export the path to the directory to host the mirrored images:

      $ REMOVABLE_MEDIA_PATH=<path> 1

      1

      Specify the full path, including the initial forward slash (/) character.

3. Review the images and configuration manifests to mirror:

   $ oc adm release mirror -a ${LOCAL_SECRET_JSON} --to-dir=${REMOVABLE_MEDIA_PATH}/mirror quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-${ARCHITECTURE} --dry-run

4. Mirror the version images to the mirror registry.

   • If your mirror host does not have internet access, take the following actions:

     1. Connect the removable media to a system that is connected to the internet.

     2. Mirror the images and configuration manifests to a directory on the removable media:

        $ oc adm release mirror -a ${LOCAL_SECRET_JSON} --to-dir=${REMOVABLE_MEDIA_PATH}/mirror quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-${ARCHITECTURE}

     3. Take the media to the restricted network environment and upload the images to the local container registry.

        $ oc image mirror  -a ${LOCAL_SECRET_JSON} --from-dir=${REMOVABLE_MEDIA_PATH}/mirror "file://openshift/release:${OCP_RELEASE}*" ${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} 1

        1

        For REMOVABLE_MEDIA_PATH, you must use the same path that you specified when you mirrored the images.

     4. Use oc command-line interface (CLI) to log in to the cluster that you are upgrading.

     5. Apply the mirrored release image signature config map to the connected cluster:

        $ oc apply -f ${REMOVABLE_MEDIA_PATH}/mirror/config/<image_signature_file> 1

        1

        For <image_signature_file>, specify the path and name of the file, for example, signature-sha256-81154f5c03294534.yaml.

   • If the local container registry and the cluster are connected to the mirror host, directly push the release images to the local registry and apply the config map to the cluster by using following command:

     $ oc adm release mirror -a ${LOCAL_SECRET_JSON} --from=quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-${ARCHITECTURE} \
       --to=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} --apply-release-image-signature

     Note

     If you include the --apply-release-image-signature option, do not create the config map for image signature verification.

Procedure

1. Configure mirrored repositories, by either:

   • Setting up a mirrored repository with Red Hat Quay, as described in Red Hat Quay Repository Mirroring. Using Red Hat Quay allows you to copy images from one repository to another and also automatically sync those repositories repeatedly over time.

   • Using a tool such as skopeo to copy images manually from the source directory to the mirrored repository.

     For example, after installing the skopeo RPM package on a Red Hat Enterprise Linux (RHEL) 7 or RHEL 8 system, use the skopeo command as shown in this example:

     $ skopeo copy \
     docker://registry.access.redhat.com/ubi8/ubi-minimal@sha256:5cfbaf45ca96806917830c183e9f37df2e913b187adb32e89fd83fa455ebaa6 \
     docker://example.io/example/ubi-minimal

     In this example, you have a container image registry that is named example.io with an image repository named example to which you want to copy the ubi8/ubi-minimal image from registry.access.redhat.com. After you create the registry, you can configure your OpenShift Container Platform cluster to redirect requests made of the source repository to the mirrored repository.

2. Log in to your OpenShift Container Platform cluster.

3. Create an ImageContentSourcePolicy file (for example, registryrepomirror.yaml), replacing the source and mirrors with your own registry and repository pairs and images:

   apiVersion: operator.openshift.io/v1alpha1
   kind: ImageContentSourcePolicy
   metadata:
     name: ubi8repo
   spec:
     repositoryDigestMirrors:
     - mirrors:
       - example.io/example/ubi-minimal 1
       source: registry.access.redhat.com/ubi8/ubi-minimal 2
     - mirrors:
       - example.com/example/ubi-minimal
       source: registry.access.redhat.com/ubi8/ubi-minimal
     - mirrors:
       - mirror.example.com/redhat
       source: registry.redhat.io/openshift4 3

   1

   Indicates the name of the image registry and repository.

   2

   Indicates the registry and repository containing the content that is mirrored.

   3

   You can configure a namespace inside a registry to use any image in that namespace. If you use a registry domain as a source, the ImageContentSourcePolicy resource is applied to all repositories from the registry.

4. Create the new ImageContentSourcePolicy object:

   $ oc create -f registryrepomirror.yaml

   After the ImageContentSourcePolicy object is created, the new settings are deployed to each node and the cluster starts using the mirrored repository for requests to the source repository.

5. To check that the mirrored configuration settings, are applied, do the following on one of the nodes.

   1. List your nodes:

      $ oc get node

      Example output

      NAME                           STATUS                     ROLES    AGE  VERSION
      ip-10-0-137-44.ec2.internal    Ready                      worker   7m   v1.19.0
      ip-10-0-138-148.ec2.internal   Ready                      master   11m  v1.19.0
      ip-10-0-139-122.ec2.internal   Ready                      master   11m  v1.19.0
      ip-10-0-147-35.ec2.internal    Ready,SchedulingDisabled   worker   7m   v1.19.0
      ip-10-0-153-12.ec2.internal    Ready                      worker   7m   v1.19.0
      ip-10-0-154-10.ec2.internal    Ready                      master   11m  v1.19.0

      You can see that scheduling on each worker node is disabled as the change is being applied.

   2. Start the debugging process to access the node:

      $ oc debug node/ip-10-0-147-35.ec2.internal

      Example output

      Starting pod/ip-10-0-147-35ec2internal-debug ...
      To use host binaries, run `chroot /host`

   3. Access the node’s files:

      sh-4.2# chroot /host

   4. Check the /etc/containers/registries.conf file to make sure the changes were made:

      sh-4.2# cat /etc/containers/registries.conf

      Example output

      unqualified-search-registries = ["registry.access.redhat.com", "docker.io"]
      [[registry]]
        location = "registry.access.redhat.com/ubi8/"
        insecure = false
        blocked = false
        mirror-by-digest-only = true
        prefix = ""
      
        [[registry.mirror]]
          location = "example.io/example/ubi8-minimal"
          insecure = false
      
        [[registry.mirror]]
          location = "example.com/example/ubi8-minimal"
          insecure = false

   5. Pull an image digest to the node from the source and check if it is resolved by the mirror. ImageContentSourcePolicy objects support image digests only, not image tags.

      sh-4.2# podman pull --log-level=debug registry.access.redhat.com/ubi8/ubi-minimal@sha256:5cfbaf45ca96806917830c183e9f37df2e913b187adb32e89fd83fa455ebaa6

Procedure

1. Run the following command, specifying values for <local_registry>, <pull_spec>, and <pull_secret_file>:

   $ oc adm catalog mirror <local_registry>/<pull_spec> <local_registry> -a <pull_secret_file> --icsp-scope=registry

   where:

   <local_registry>

   is the local registry you have configured for your disconnected cluster, for example, local.registry:5000.

   <pull_spec>

   is the pull specification as configured in your disconnected registry, for example, redhat/redhat-operator-index:v4.6

   <pull_secret_file>

   is the registry.redhat.io pull secret in .json file format. You can download the pull secret from the Red Hat OpenShift Cluster Manager.

   The oc adm catalog mirror command creates a /redhat-operator-index-manifests directory and generates imageContentSourcePolicy.yaml, catalogSource.yaml, and mapping.txt files.

2. Apply the new ImageContentSourcePolicy resource to the cluster:

   $ oc apply -f imageContentSourcePolicy.yaml