Chapter 6. Updating a cluster in a disconnected environment

Procedure

1. Use the Red Hat OpenShift Container Platform Update Graph visualizer and update planner to plan an update from one version to another. The OpenShift Update Graph provides channel graphs and a way to confirm that there is an update path between your current and intended cluster versions.

2. Set the required environment variables:

   1. Export the release version:

      $ export OCP_RELEASE=<release_version>

      For <release_version>, specify the tag that corresponds to the version of OpenShift Container Platform to which you want to update, such as 4.5.4.

   2. Export the local registry name and host port:

      $ LOCAL_REGISTRY='<local_registry_host_name>:<local_registry_host_port>'

      For <local_registry_host_name>, specify the registry domain name for your mirror repository, and for <local_registry_host_port>, specify the port that it serves content on.

   3. Export the local repository name:

      $ LOCAL_REPOSITORY='<local_repository_name>'

      For <local_repository_name>, specify the name of the repository to create in your registry, such as ocp4/openshift4.

   4. If you are using the OpenShift Update Service, export an additional local repository name to contain the release images:

      $ LOCAL_RELEASE_IMAGES_REPOSITORY='<local_release_images_repository_name>'

      For <local_release_images_repository_name>, specify the name of the repository to create in your registry, such as ocp4/openshift4-release-images.

   5. Export the name of the repository to mirror:

      $ PRODUCT_REPO='openshift-release-dev'

      For a production release, you must specify openshift-release-dev.

   6. Export the path to your registry pull secret:

      $ LOCAL_SECRET_JSON='<path_to_pull_secret>'

      For <path_to_pull_secret>, specify the absolute path to and file name of the pull secret for your mirror registry that you created.

      Note

      If your cluster uses an ImageContentSourcePolicy object to configure repository mirroring, you can use only global pull secrets for mirrored registries. You cannot add a pull secret to a project.

   7. Export the release mirror:

      $ RELEASE_NAME="ocp-release"

      For a production release, you must specify ocp-release.

   8. Export the type of architecture for your cluster:

      $ ARCHITECTURE=<cluster_architecture> 1

      1

      Specify the architecture of the cluster, such as x86_64, aarch64, s390x, or ppc64le.

   9. Export the path to the directory to host the mirrored images:

      $ REMOVABLE_MEDIA_PATH=<path> 1

      1

      Specify the full path, including the initial forward slash (/) character.

3. Review the images and configuration manifests to mirror:

   $ oc adm release mirror -a ${LOCAL_SECRET_JSON} --to-dir=${REMOVABLE_MEDIA_PATH}/mirror quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-${ARCHITECTURE} --dry-run

4. Mirror the version images to the mirror registry.

   • If your mirror host does not have internet access, take the following actions:

     1. Connect the removable media to a system that is connected to the internet.

     2. Mirror the images and configuration manifests to a directory on the removable media:

        $ oc adm release mirror -a ${LOCAL_SECRET_JSON} --to-dir=${REMOVABLE_MEDIA_PATH}/mirror quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-${ARCHITECTURE}

        Note

        This command also generates and saves the mirrored release image signature config map onto the removable media.

     3. Take the media to the disconnected environment and upload the images to the local container registry.

        $ oc image mirror  -a ${LOCAL_SECRET_JSON} --from-dir=${REMOVABLE_MEDIA_PATH}/mirror "file://openshift/release:${OCP_RELEASE}*" ${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} 1

        1

        For REMOVABLE_MEDIA_PATH, you must use the same path that you specified when you mirrored the images.

     4. Use oc command-line interface (CLI) to log in to the cluster that you are updating.

     5. Apply the mirrored release image signature config map to the connected cluster:

        $ oc apply -f ${REMOVABLE_MEDIA_PATH}/mirror/config/<image_signature_file> 1

        1

        For <image_signature_file>, specify the path and name of the file, for example, signature-sha256-81154f5c03294534.yaml.

     6. If you are using the OpenShift Update Service, mirror the release image to a separate repository:

        $ oc image mirror -a ${LOCAL_SECRET_JSON} ${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE} ${LOCAL_REGISTRY}/${LOCAL_RELEASE_IMAGES_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE}

   • If the local container registry and the cluster are connected to the mirror host, take the following actions:

     1. Directly push the release images to the local registry and apply the config map to the cluster by using following command:

        $ oc adm release mirror -a ${LOCAL_SECRET_JSON} --from=quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-${ARCHITECTURE} \
          --to=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} --apply-release-image-signature

        Note

        If you include the --apply-release-image-signature option, do not create the config map for image signature verification.

     2. If you are using the OpenShift Update Service, mirror the release image to a separate repository:

        $ oc image mirror -a ${LOCAL_SECRET_JSON} ${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE} ${LOCAL_REGISTRY}/${LOCAL_RELEASE_IMAGES_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE}

Procedure

1. In the web console, click Operators OperatorHub.

   Note

   Enter Update Service into the Filter by keyword…​ field to find the Operator faster.

2. Choose OpenShift Update Service from the list of available Operators, and click Install.

   1. Select an Update channel.
   2. Select a Version.
   3. Select A specific namespace on the cluster under Installation Mode.
   4. Select a namespace for Installed Namespace or accept the recommended namespace openshift-update-service.

   5. Select an Update approval strategy:

      • The Automatic strategy allows Operator Lifecycle Manager (OLM) to automatically update the Operator when a new version is available.
      • The Manual strategy requires a cluster administrator to approve the Operator update.
   6. Click Install.
3. Go to Operators Installed Operators and verify that the OpenShift Update Service Operator is installed.
4. Ensure that OpenShift Update Service is listed in the correct namespace with a Status of Succeeded.

Procedure

1. Create a namespace for the OpenShift Update Service Operator:

   1. Create a Namespace object YAML file, for example, update-service-namespace.yaml, for the OpenShift Update Service Operator:

      apiVersion: v1
      kind: Namespace
      metadata:
        name: openshift-update-service
        annotations:
          openshift.io/node-selector: ""
        labels:
          openshift.io/cluster-monitoring: "true" 1

      1

      Set the openshift.io/cluster-monitoring label to enable Operator-recommended cluster monitoring on this namespace.

   2. Create the namespace:

      $ oc create -f <filename>.yaml

      For example:

      $ oc create -f update-service-namespace.yaml

2. Install the OpenShift Update Service Operator by creating the following objects:

   1. Create an OperatorGroup object YAML file, for example, update-service-operator-group.yaml:

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: update-service-operator-group
        namespace: openshift-update-service
      spec:
        targetNamespaces:
        - openshift-update-service

   2. Create an OperatorGroup object:

      $ oc -n openshift-update-service create -f <filename>.yaml

      For example:

      $ oc -n openshift-update-service create -f update-service-operator-group.yaml

   3. Create a Subscription object YAML file, for example, update-service-subscription.yaml:

      Example Subscription

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: update-service-subscription
        namespace: openshift-update-service
      spec:
        channel: v1
        installPlanApproval: "Automatic"
        source: "redhat-operators" 1
        sourceNamespace: "openshift-marketplace"
        name: "cincinnati-operator"

      1

      Specify the name of the catalog source that provides the Operator. For clusters that do not use a custom Operator Lifecycle Manager (OLM), specify redhat-operators. If your OpenShift Container Platform cluster is installed in a disconnected environment, specify the name of the CatalogSource object created when you configured Operator Lifecycle Manager (OLM).

   4. Create the Subscription object:

      $ oc create -f <filename>.yaml

      For example:

      $ oc -n openshift-update-service create -f update-service-subscription.yaml

      The OpenShift Update Service Operator is installed to the openshift-update-service namespace and targets the openshift-update-service namespace.

3. Verify the Operator installation:

   $ oc -n openshift-update-service get clusterserviceversions

   Example output

   NAME                             DISPLAY                    VERSION   REPLACES   PHASE
   update-service-operator.v4.6.0   OpenShift Update Service   4.6.0                Succeeded
   ...

   If the OpenShift Update Service Operator is listed, the installation was successful. The version number might be different than shown.

Procedure

1. In the web console, click Operators Installed Operators.
2. Choose OpenShift Update Service from the list of installed Operators.
3. Click the Update Service tab.
4. Click Create UpdateService.
5. Enter a name in the Name field, for example, service.
6. Enter the local pullspec in the Graph Data Image field to the graph data container image created in "Creating the OpenShift Update Service graph data container image", for example, registry.example.com/openshift/graph-data:latest.
7. In the Releases field, enter the registry and repository created to contain the release images in "Mirroring the OpenShift Container Platform image repository", for example, registry.example.com/ocp4/openshift4-release-images.
8. Enter 2 in the Replicas field.
9. Click Create to create the OpenShift Update Service application.

10. Verify the OpenShift Update Service application:

    • From the UpdateServices list in the Update Service tab, click the Update Service application just created.
    • Click the Resources tab.
    • Verify each application resource has a status of Created.

Procedure

1. Configure the OpenShift Update Service target namespace, for example, openshift-update-service:

   $ NAMESPACE=openshift-update-service

   The namespace must match the targetNamespaces value from the operator group.

2. Configure the name of the OpenShift Update Service application, for example, service:

   $ NAME=service

3. Configure the registry and repository for the release images as configured in "Mirroring the OpenShift Container Platform image repository", for example, registry.example.com/ocp4/openshift4-release-images:

   $ RELEASE_IMAGES=registry.example.com/ocp4/openshift4-release-images

4. Set the local pullspec for the graph data image to the graph data container image created in "Creating the OpenShift Update Service graph data container image", for example, registry.example.com/openshift/graph-data:latest:

   $ GRAPH_DATA_IMAGE=registry.example.com/openshift/graph-data:latest

5. Create an OpenShift Update Service application object:

   $ oc -n "${NAMESPACE}" create -f - <<EOF
   apiVersion: updateservice.operator.openshift.io/v1
   kind: UpdateService
   metadata:
     name: ${NAME}
   spec:
     replicas: 2
     releases: ${RELEASE_IMAGES}
     graphDataImage: ${GRAPH_DATA_IMAGE}
   EOF

6. Verify the OpenShift Update Service application:

   1. Use the following command to obtain a policy engine route:

      $ while sleep 1; do POLICY_ENGINE_GRAPH_URI="$(oc -n "${NAMESPACE}" get -o jsonpath='{.status.policyEngineURI}/api/upgrades_info/v1/graph{"\n"}' updateservice "${NAME}")"; SCHEME="${POLICY_ENGINE_GRAPH_URI%%:*}"; if test "${SCHEME}" = http -o "${SCHEME}" = https; then break; fi; done

      You might need to poll until the command succeeds.

   2. Retrieve a graph from the policy engine. Be sure to specify a valid version for channel. For example, if running in OpenShift Container Platform 4.17, use stable-4.17:

      $ while sleep 10; do HTTP_CODE="$(curl --header Accept:application/json --output /dev/stderr --write-out "%{http_code}" "${POLICY_ENGINE_GRAPH_URI}?channel=stable-4.6")"; if test "${HTTP_CODE}" -eq 200; then break; fi; echo "${HTTP_CODE}"; done

      This polls until the graph request succeeds; however, the resulting graph might be empty depending on which release images you have mirrored.

Procedure

1. Configure mirrored repositories, by either:

   • Setting up a mirrored repository with Red Hat Quay, as described in Red Hat Quay Repository Mirroring. Using Red Hat Quay allows you to copy images from one repository to another and also automatically sync those repositories repeatedly over time.

   • Using a tool such as skopeo to copy images manually from the source repository to the mirrored repository.

     For example, after installing the skopeo RPM package on a Red Hat Enterprise Linux (RHEL) 7 or RHEL 8 system, use the skopeo command as shown in this example:

     $ skopeo copy --all \
     docker://registry.access.redhat.com/ubi9/ubi-minimal:latest@sha256:5cf... \
     docker://example.io/example/ubi-minimal

     In this example, you have a container image registry that is named example.io with an image repository named example to which you want to copy the ubi9/ubi-minimal image from registry.access.redhat.com. After you create the mirrored registry, you can configure your OpenShift Container Platform cluster to redirect requests made of the source repository to the mirrored repository.

2. Create a postinstallation mirror configuration CR, by using one of the following examples:

   • Create an ImageDigestMirrorSet or ImageTagMirrorSet CR, as needed, replacing the source and mirrors with your own registry and repository pairs and images:

     apiVersion: config.openshift.io/v1 1
     kind: ImageDigestMirrorSet 2
     metadata:
       name: ubi9repo
     spec:
       imageDigestMirrors: 3
       - mirrors:
         - example.io/example/ubi-minimal 4
         - example.com/example/ubi-minimal 5
         source: registry.access.redhat.com/ubi9/ubi-minimal 6
         mirrorSourcePolicy: AllowContactingSource 7
       - mirrors:
         - mirror.example.com/redhat
         source: registry.example.com/redhat 8
         mirrorSourcePolicy: AllowContactingSource
       - mirrors:
         - mirror.example.com
         source: registry.example.com 9
         mirrorSourcePolicy: AllowContactingSource
       - mirrors:
         - mirror.example.net/image
         source: registry.example.com/example/myimage 10
         mirrorSourcePolicy: AllowContactingSource
       - mirrors:
         - mirror.example.net
         source: registry.example.com/example 11
         mirrorSourcePolicy: AllowContactingSource
       - mirrors:
         - mirror.example.net/registry-example-com
         source: registry.example.com 12
         mirrorSourcePolicy: AllowContactingSource

     1

     Indicates the API to use with this CR. This must be config.openshift.io/v1.

     2

     Indicates the kind of object according to the pull type:

     • ImageDigestMirrorSet: Pulls a digest reference image.
     • ImageTagMirrorSet: Pulls a tag reference image.

     3

     Indicates the type of image pull method, either:

     • imageDigestMirrors: Use for an ImageDigestMirrorSet CR.
     • imageTagMirrors: Use for an ImageTagMirrorSet CR.

     4

     Indicates the name of the mirrored image registry and repository.

     5

     Optional: Indicates a secondary mirror repository for each target repository. If one mirror is down, the target repository can use the secondary mirror.

     6

     Indicates the registry and repository source, which is the repository that is referred to in an image pull specification.

     7

     Optional: Indicates the fallback policy if the image pull fails:

     • AllowContactingSource: Allows continued attempts to pull the image from the source repository. This is the default.
     • NeverContactSource: Prevents continued attempts to pull the image from the source repository.

     8

     Optional: Indicates a namespace inside a registry, which allows you to use any image in that namespace. If you use a registry domain as a source, the object is applied to all repositories from the registry.

     9

     Optional: Indicates a registry, which allows you to use any image in that registry. If you specify a registry name, the object is applied to all repositories from a source registry to a mirror registry.

     10

     Pulls the image registry.example.com/example/myimage@sha256:…​ from the mirror mirror.example.net/image@sha256:...

     11

     Pulls the image registry.example.com/example/image@sha256:…​ in the source registry namespace from the mirror mirror.example.net/image@sha256:…​.

     12

     Pulls the image registry.example.com/myimage@sha256 from the mirror registry example.net/registry-example-com/myimage@sha256:…​.

   • Create an ImageContentSourcePolicy custom resource, replacing the source and mirrors with your own registry and repository pairs and images:

     apiVersion: operator.openshift.io/v1alpha1
     kind: ImageContentSourcePolicy
     metadata:
       name: mirror-ocp
     spec:
       repositoryDigestMirrors:
       - mirrors:
         - mirror.registry.com:443/ocp/release 1
         source: quay.io/openshift-release-dev/ocp-release 2
       - mirrors:
         - mirror.registry.com:443/ocp/release
         source: quay.io/openshift-release-dev/ocp-v4.0-art-dev

     1

     Specifies the name of the mirror image registry and repository.

     2

     Specifies the online registry and repository containing the content that is mirrored.

3. Create the new object:

   $ oc create -f registryrepomirror.yaml

   After the object is created, the Machine Config Operator (MCO) drains the nodes for ImageTagMirrorSet objects only. The MCO does not drain the nodes for ImageDigestMirrorSet and ImageContentSourcePolicy objects.

4. To check that the mirrored configuration settings are applied, do the following on one of the nodes.

   1. List your nodes:

      $ oc get node

      Example output

      NAME                           STATUS                     ROLES    AGE  VERSION
      ip-10-0-137-44.ec2.internal    Ready                      worker   7m   v1.30.3
      ip-10-0-138-148.ec2.internal   Ready                      master   11m  v1.30.3
      ip-10-0-139-122.ec2.internal   Ready                      master   11m  v1.30.3
      ip-10-0-147-35.ec2.internal    Ready                      worker   7m   v1.30.3
      ip-10-0-153-12.ec2.internal    Ready                      worker   7m   v1.30.3
      ip-10-0-154-10.ec2.internal    Ready                      master   11m  v1.30.3

   2. Start the debugging process to access the node:

      $ oc debug node/ip-10-0-147-35.ec2.internal

      Example output

      Starting pod/ip-10-0-147-35ec2internal-debug ...
      To use host binaries, run `chroot /host`

   3. Change your root directory to /host:

      sh-4.2# chroot /host

   4. Check the /etc/containers/registries.conf file to make sure the changes were made:

      sh-4.2# cat /etc/containers/registries.conf

      The following output represents a registries.conf file where postinstallation mirror configuration CRs were applied. The final two entries are marked digest-only and tag-only respectively.

      Example output

      unqualified-search-registries = ["registry.access.redhat.com", "docker.io"]
      short-name-mode = ""
      
      [[registry]]
        prefix = ""
        location = "registry.access.redhat.com/ubi9/ubi-minimal" 1
      
        [[registry.mirror]]
          location = "example.io/example/ubi-minimal" 2
          pull-from-mirror = "digest-only" 3
      
        [[registry.mirror]]
          location = "example.com/example/ubi-minimal"
          pull-from-mirror = "digest-only"
      
      [[registry]]
        prefix = ""
        location = "registry.example.com"
      
        [[registry.mirror]]
          location = "mirror.example.net/registry-example-com"
          pull-from-mirror = "digest-only"
      
      [[registry]]
        prefix = ""
        location = "registry.example.com/example"
      
        [[registry.mirror]]
          location = "mirror.example.net"
          pull-from-mirror = "digest-only"
      
      [[registry]]
        prefix = ""
        location = "registry.example.com/example/myimage"
      
        [[registry.mirror]]
          location = "mirror.example.net/image"
          pull-from-mirror = "digest-only"
      
      [[registry]]
        prefix = ""
        location = "registry.example.com"
      
        [[registry.mirror]]
          location = "mirror.example.com"
          pull-from-mirror = "digest-only"
      
      [[registry]]
        prefix = ""
        location = "registry.example.com/redhat"
      
        [[registry.mirror]]
          location = "mirror.example.com/redhat"
          pull-from-mirror = "digest-only"
      [[registry]]
        prefix = ""
        location = "registry.access.redhat.com/ubi9/ubi-minimal"
        blocked = true 4
      
        [[registry.mirror]]
          location = "example.io/example/ubi-minimal-tag"
          pull-from-mirror = "tag-only" 5

      1

      Indicates the repository that is referred to in a pull spec.

      2

      Indicates the mirror for that repository.

      3

      Indicates that the image pull from the mirror is a digest reference image.

      4

      Indicates that the NeverContactSource parameter is set for this repository.

      5

      Indicates that the image pull from the mirror is a tag reference image.

   5. Pull an image to the node from the source and check if it is resolved by the mirror.

      sh-4.2# podman pull --log-level=debug registry.access.redhat.com/ubi9/ubi-minimal@sha256:5cf...

Procedure

1. Use the following command to convert one or more ImageContentSourcePolicy YAML files to an ImageDigestMirrorSet YAML file:

   $ oc adm migrate icsp <file_name>.yaml <file_name>.yaml <file_name>.yaml --dest-dir <path_to_the_directory>

   where:

   <file_name>

   Specifies the name of the source ImageContentSourcePolicy YAML. You can list multiple file names.

   --dest-dir

   Optional: Specifies a directory for the output ImageDigestMirrorSet YAML. If unset, the file is written to the current directory.

   For example, the following command converts the icsp.yaml and icsp-2.yaml file and saves the new YAML files to the idms-files directory.

   $ oc adm migrate icsp icsp.yaml icsp-2.yaml --dest-dir idms-files

   Example output

   wrote ImageDigestMirrorSet to idms-files/imagedigestmirrorset_ubi8repo.5911620242173376087.yaml
   wrote ImageDigestMirrorSet to idms-files/imagedigestmirrorset_ubi9repo.6456931852378115011.yaml

2. Create the CR object by running the following command:

   $ oc create -f <path_to_the_directory>/<file-name>.yaml

   where:

   <path_to_the_directory>

   Specifies the path to the directory, if you used the --dest-dir flag.

   <file_name>

   Specifies the name of the ImageDigestMirrorSet YAML.

3. Remove the ICSP objects after the IDMS objects are rolled out.

Procedure

1. In the web console, click Operators Installed Operators.
2. Choose OpenShift Update Service from the list of installed Operators.
3. Click the Update Service tab.
4. From the list of installed OpenShift Update Service applications, select the application to be deleted and then click Delete UpdateService.
5. From the Delete UpdateService? confirmation dialog, click Delete to confirm the deletion.

Procedure

1. Get the OpenShift Update Service application name using the namespace the OpenShift Update Service application was created in, for example, openshift-update-service:

   $ oc get updateservice -n openshift-update-service

   Example output

   NAME      AGE
   service   6s

2. Delete the OpenShift Update Service application using the NAME value from the previous step and the namespace the OpenShift Update Service application was created in, for example, openshift-update-service:

   $ oc delete updateservice service -n openshift-update-service

   Example output

   updateservice.updateservice.operator.openshift.io "service" deleted

Procedure

1. In the web console, click Operators Installed Operators.
2. Select OpenShift Update Service from the list of installed Operators and click Uninstall Operator.
3. From the Uninstall Operator? confirmation dialog, click Uninstall to confirm the uninstallation.

Procedure

1. Change to the project containing the OpenShift Update Service Operator, for example, openshift-update-service:

   $ oc project openshift-update-service

   Example output

   Now using project "openshift-update-service" on server "https://example.com:6443".

2. Get the name of the OpenShift Update Service Operator operator group:

   $ oc get operatorgroup

   Example output

   NAME                             AGE
   openshift-update-service-fprx2   4m41s

3. Delete the operator group, for example, openshift-update-service-fprx2:

   $ oc delete operatorgroup openshift-update-service-fprx2

   Example output

   operatorgroup.operators.coreos.com "openshift-update-service-fprx2" deleted

4. Get the name of the OpenShift Update Service Operator subscription:

   $ oc get subscription

   Example output

   NAME                      PACKAGE                   SOURCE                        CHANNEL
   update-service-operator   update-service-operator   updateservice-index-catalog   v1

5. Using the Name value from the previous step, check the current version of the subscribed OpenShift Update Service Operator in the currentCSV field:

   $ oc get subscription update-service-operator -o yaml | grep " currentCSV"

   Example output

     currentCSV: update-service-operator.v0.0.1

6. Delete the subscription, for example, update-service-operator:

   $ oc delete subscription update-service-operator

   Example output

   subscription.operators.coreos.com "update-service-operator" deleted

7. Delete the CSV for the OpenShift Update Service Operator using the currentCSV value from the previous step:

   $ oc delete clusterserviceversion update-service-operator.v0.0.1

   Example output

   clusterserviceversion.operators.coreos.com "update-service-operator.v0.0.1" deleted