Chapter 5. Mirroring images for a disconnected installation by using the oc-mirror plugin v2

1. Create an image set configuration file.

2. Mirror the image set to the target mirror registry by using one of the following workflows:

   • Mirror an image set directly to the target mirror registry (mirror to mirror).
   • Mirror an image set to disk (mirror to disk), transfer the tar file to the target environment, then mirror the image set to the target mirror registry (disk to mirror).
3. Configure your cluster to use the resources generated by the oc-mirror plugin v2.
4. Repeat these steps to update your target mirror registry as necessary.

Procedure

1. Download the oc-mirror CLI plugin:

   1. Navigate to the Downloads page of the Red Hat Hybrid Cloud Console.
   2. In the OpenShift disconnected installation tools section, select the OS type and Architecture type of the OpenShift Client (oc) mirror plugin from the dropdown menus.
   3. Click Download to save the file.

2. Extract the archive by running the following command:

   $ tar xvzf oc-mirror.tar.gz

   Copy to Clipboard Toggle word wrap

3. If necessary, update the plugin file to be executable by running the following command:

   $ chmod +x oc-mirror

   Copy to Clipboard Toggle word wrap
   Note

   Do not rename the oc-mirror file.

4. Install the oc-mirror CLI plugin by placing the file in your PATH, for example /usr/local/bin, by running the following command:

   $ sudo mv oc-mirror /usr/local/bin/.

   Copy to Clipboard Toggle word wrap

1. Download your registry.redhat.io pull secret from Red Hat OpenShift Cluster Manager.

2. Make a copy of your pull secret in JSON format by running the following command:

   $ cat ./pull-secret | jq . > <path>/<pull_secret_file_in_json> 

   1

   Copy to Clipboard Toggle word wrap

   1

   Specify the path to the folder to store the pull secret in and a name for the JSON file that you create.

   Example pull secret

   {
     "auths": {
       "cloud.openshift.com": {
         "auth": "b3BlbnNo...",
         "email": "you@example.com"
       },
       "quay.io": {
         "auth": "b3BlbnNo...",
         "email": "you@example.com"
       },
       "registry.connect.redhat.com": {
         "auth": "NTE3Njg5Nj...",
         "email": "you@example.com"
       },
       "registry.redhat.io": {
         "auth": "NTE3Njg5Nj...",
         "email": "you@example.com"
       }
     }
   }

   Copy to Clipboard Toggle word wrap

3. If the $XDG_RUNTIME_DIR/containers directory does not exist, create one by entering the following command:

   $ mkdir -p $XDG_RUNTIME_DIR/containers

   Copy to Clipboard Toggle word wrap
4. Save the pull secret file as $XDG_RUNTIME_DIR/containers/auth.json.

5. Generate the base64-encoded user name and password or token for your mirror registry by running the following command:

   $ echo -n '<user_name>:<password>' | base64 -w0 

   1

   Copy to Clipboard Toggle word wrap

   1

   For <user_name> and <password>, specify the user name and password that you configured for your registry.

   Example output

   BGVtbYk3ZHAtqXs=

   Copy to Clipboard Toggle word wrap

6. Edit the JSON file and add a section that describes your registry to it:

     "auths": {
       "<mirror_registry>": { 

   1

   
         "auth": "<credentials>", 

   2

   
         "email": "you@example.com"
       }
     },

   Copy to Clipboard Toggle word wrap

   1

   Specify the registry domain name, and optionally the port, that your mirror registry uses to serve content. For example, registry.example.com or registry.example.com:8443

   2

   Specify the base64-encoded user name and password for the mirror registry.

   Example modified pull secret

   {
     "auths": {
       "registry.example.com": {
         "auth": "BGVtbYk3ZHAtqXs=",
         "email": "you@example.com"
       },
       "cloud.openshift.com": {
         "auth": "b3BlbnNo...",
         "email": "you@example.com"
       },
       "quay.io": {
         "auth": "b3BlbnNo...",
         "email": "you@example.com"
       },
       "registry.connect.redhat.com": {
         "auth": "NTE3Njg5Nj...",
         "email": "you@example.com"
       },
       "registry.redhat.io": {
         "auth": "NTE3Njg5Nj...",
         "email": "you@example.com"
       }
     }
   }

   Copy to Clipboard Toggle word wrap

Verification

1. Navigate to the working-dir/cluster-resources directory that was generated in the <file_path> directory.
2. Verify that the YAML files are present for the ImageDigestMirrorSet, ImageTagMirrorSet, and CatalogSource resources.

1. Mirror to disk: Mirror the image set to an archive.
2. Disk transfer: Manually transfer the archive to the network of the disconnected mirror registry.
3. Disk to mirror: Mirror the image set from the archive to the target disconnected registry.

Procedure

1. Log in to the OpenShift CLI as a user with the cluster-admin role.

2. Apply the YAML files from the results directory to the cluster by running the following command:

   $ oc apply -f <path_to_oc_mirror_workspace>/working-dir/cluster-resources

   Copy to Clipboard Toggle word wrap

3. If you mirrored release images, apply the release image signatures to the cluster by running the following command:

   $ oc apply -f working-dir/cluster-resources/signature-configmap.json

   Copy to Clipboard Toggle word wrap
   Important

   If you are mirroring Operators instead of clusters, do not run the preceding command. Running the command results in an error because there are no release image signatures to apply.

   Additionally, a YAML file is available in the same directory working-dir/cluster-resources/. You can use either the JSON or YAML format.

Verification

1. Verify that the ImageDigestMirrorSet resources are successfully installed by running the following command:

   $ oc get imagedigestmirrorset

   Copy to Clipboard Toggle word wrap

2. Verify that the ImageTagMirrorSet resources are successfully installed by running the following command:

   $ oc get imagetagmirrorset

   Copy to Clipboard Toggle word wrap

3. Verify that the CatalogSource resources are successfully installed by running the following command:

   $ oc get catalogsource -n openshift-marketplace

   Copy to Clipboard Toggle word wrap

4. Verify that the ClusterCatalog resources are successfully installed by running the following command:

   $ oc get clustercatalog

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a delete-image-set-config.yaml file and include the following content:

   DeleteImageSetConfiguration file

   apiVersion: mirror.openshift.io/v2alpha1
   kind: DeleteImageSetConfiguration
   delete:
     platform:
       channels:
         - name: <channel_name> 

   1

   
           minVersion: <channel_min_version> 

   2

   
           maxVersion: <channel_max_version> 

   3

   
     operators:
       - catalog: <operator_catalog_name> 

   4

   
         packages:
         - name: <operator_name> 

   5

   
            minVersion: <operator_max_version> 

   6

   
            maxVersion: <operator_min_version> 

   7

   
     additionalImages:
       - name: <additional_images>

   Copy to Clipboard Toggle word wrap

   1 1

   Specify the name of the OpenShift Container Platform channel to delete, for example stable-4.15.

   2 3

   Specify a version range of the images to delete within the channel, for example 4.15.0 for the minimum version and 4.15.1 for the maximum version. To delete only one version’s images, use that version number for both the minVersion and maxVersion fields.

   4

   Specify an Operator catalog image containing the Operators to delete, for example registry.redhat.io/redhat/redhat-operator-index:v4.14. The Operator catalog image will not get deleted. Its presence in the registry might be necessary for other Operators still remaining on the cluster.

   5

   Specify a specific Operator to delete, for example aws-load-balancer-operator.

   6 7

   Specify a version range of the images to delete for the Operator, for example 0.0.1 for the minimum version and 0.0.2 for the maximum version.

2. Create a delete-images.yaml file by running the following command:

   $ oc mirror delete --config delete-image-set-config.yaml --workspace file://<previously_mirrored_work_folder> --v2 --generate docker://<remote_registry>

   Copy to Clipboard Toggle word wrap

   where:

   <previously_mirrored_work_folder>

   Specifies the directory where images were previously mirrored to or stored during the mirroring process.

   <remote_registry>

   Specifies the URL or address of the remote container registry from which images will be deleted.

   Important

   When deleting images, specify the correct workspace directory. Modify or delete the cache directory only when starting mirroring from scratch, such as setting up a new cluster. Incorrect changes to the cache directory might disrupt further mirroring operations.

3. Go to the <previously_mirrored_work_folder>/delete directory that was created.
4. Verify that the delete-images.yaml file has been generated.
5. Manually ensure that each image listed in the file is no longer needed by the cluster and can be safely removed from the registry.

6. After you generate the delete-images YAML file, delete the images from the remote registry by running the following command:

   $ oc mirror delete --v2 --delete-yaml-file <previously_mirrored_work_folder>/working-dir/delete/delete-images.yaml docker://<remote_registry>

   Copy to Clipboard Toggle word wrap

   where:

   <previously_mirrored_work_folder>

   Specifies the directory where images were previously mirrored or stored during the mirroring process.

   <remote_registry>

   Specifies the URL or address of the remote container registry from which images will be deleted.

   Important

   When using the mirror-to-mirror method to mirror images, images are not cached locally, so you cannot delete images from a local cache.

Verification

1. Navigate to the workspace directory that was generated:

   $ cd <oc_mirror_workspace_path>

   Copy to Clipboard Toggle word wrap
2. Review the mapping.txt and missing.txt files that were generated. These files contain a list of all images that would be mirrored.

Procedure

1. Check for server-related issues:

   Example error

   [ERROR]  : [Worker] error mirroring image localhost:55000/openshift/graph-image:latest error: copying image 1/4 from manifest list: trying to reuse blob sha256:edab65b863aead24e3ed77cea194b6562143049a9307cd48f86b542db9eecb6e at destination: pinging container registry localhost:5000: Get "https://localhost:5000/v2/": http: server gave HTTP response to HTTPS client

   Copy to Clipboard Toggle word wrap

   1. Open the mirroring_error_date_time.log file in the working-dir/logs folder located in the oc-mirror plugin v2 output directory.
   2. Look for error messages that typically indicate server-side issues, such as HTTP 500 errors, expired tokens, or timeouts.
   3. Retry the mirroring process or contact support if the issue persists.

2. Check for incomplete mirroring of Operators:

   Example error

   error mirroring image docker://registry.redhat.io/3scale-amp2/zync-rhel9@sha256:8bb6b31e108d67476cc62622f20ff8db34efae5d58014de9502336fcc479d86d (Operator bundles: [3scale-operator.v0.11.12] - Operators: [3scale-operator]) error: initializing source docker://localhost:55000/3scale-amp2/zync-rhel9:8bb6b31e108d67476cc62622f20ff8db34efae5d58014de9502336fcc479d86d: reading manifest 8bb6b31e108d67476cc62622f20ff8db34efae5d58014de9502336fcc479d86d in localhost:55000/3scale-amp2/zync-rhel9: manifest unknown
   error mirroring image docker://registry.redhat.io/3scale-amp2/3scale-rhel7-operator-metadata@sha256:de0a70d1263a6a596d28bf376158056631afd0b6159865008a7263a8e9bf0c7d error: skipping operator bundle docker://registry.redhat.io/3scale-amp2/3scale-rhel7-operator-metadata@sha256:de0a70d1263a6a596d28bf376158056631afd0b6159865008a7263a8e9bf0c7d because one of its related images failed to mirror
   error mirroring image docker://registry.redhat.io/3scale-amp2/system-rhel7@sha256:fe77272021867cc6b6d5d0c9bd06c99d4024ad53f1ab94ec0ab69d0fda74588e (Operator bundles: [3scale-operator.v0.11.12] - Operators: [3scale-operator]) error: initializing source docker://localhost:55000/3scale-amp2/system-rhel7:fe77272021867cc6b6d5d0c9bd06c99d4024ad53f1ab94ec0ab69d0fda74588e: reading manifest fe77272021867cc6b6d5d0c9bd06c99d4024ad53f1ab94ec0ab69d0fda74588e in localhost:55000/3scale-amp2/system-rhel7: manifest unknown

   Copy to Clipboard Toggle word wrap

   1. Check for warnings in the console or log file indicating which Operators are incomplete.

      If an Operator is flagged as incomplete, the image related to that Operator likely failed to mirror.

   2. Manually mirror the missing image or retry the mirroring process.

3. Check for errors related to generated cluster resources. Even if some images fail to mirror, oc-mirror v2 will still generate cluster resources such as IDMS.yaml and ITMS.yaml files for the successfully mirrored images.

   1. Check the output directory for the generated files.
   2. If these files are missing for specific images, ensure that no critical errors occurred for those images during the mirroring process.

1. The user executes oc-mirror plugin v2 to mirror content from an online registry to a local disk directory.
2. The mirrored content is saved to the disk for transfer to offline environments.

Procedure

1. Before running oc-mirror plugin v2 in the disconnected environment, create a registries.conf file. The TOML format of the file is described in this specification:

   Note

   It is recommended to store the file under $HOME/.config/containers/registries.conf or /etc/containers/registries.conf.

   Example registries.conf

   [[registry]]
   location="registry.redhat.io"
   [[registry.mirror]]
   location="<enterprise-registry.in>"
   
   [[registry]]
   location="quay.io"
   [[registry.mirror]]
   location="<enterprise-registry.in>"

   Copy to Clipboard Toggle word wrap

2. Generate a mirror archive.

   1. To collect all the OpenShift Container Platform content into an archive on the disk under <file_path>/enterprise-content, run the following command:

      $ oc mirror --v2 -c isc.yaml file://<file_path>/enterprise-content

      Copy to Clipboard Toggle word wrap

      Example of isc.yaml

      apiVersion: mirror.openshift.io/v2alpha1
      kind: ImageSetConfiguration
      mirror:
        platform:
          architectures:
            - "amd64"
          channels:
            - name: stable-4.15
              minVersion: 4.15.0
              maxVersion: 4.15.3

      Copy to Clipboard Toggle word wrap

      After the archive is generated, it is transferred to the disconnected environment. The transport mechanism is not part of oc-mirror plugin v2. The enterprise network administrators determine the transfer strategy.

      In some cases, the transfer is done manually, in that the disk is physically unplugged from one location, and plugged to another computer in the disconnected environment. In other cases, the Secure File Transfer Protocol (SFTP) or other protocols are used.

3. After the transfer of the archive is done, you can execute oc-mirror plugin v2 again in order to mirror the relevant archive contents to the registry (entrerpise_registry.in in the example) as demonstrated in the following example:

   $ oc mirror --v2 -c isc.yaml --from file://<disconnected_environment_file_path>/enterprise-content docker://<enterprise_registry.in>/

   Copy to Clipboard Toggle word wrap

   Where:

   • --from points to the folder containing the archive. It starts with the file://.
   • docker:// is the destination of the mirroring is the final argument. Because it is a docker registry.
   • -c (--config) is a mandatory argument. It enables oc-mirror plugin v2 to eventually mirror only sub-parts of the archive to the registry. One archive might contain several OpenShift Container Platform releases, but the disconnected environment or an enclave might mirror only a few.

4. Prepare the imageSetConfig YAML file, which describes the content to mirror to the enclave:

   Example isc-enclave.yaml

   apiVersion: mirror.openshift.io/v2alpha1
   kind: ImageSetConfiguration
   mirror:
     platform:
       architectures:
         - "amd64"
       channels:
         - name: stable-4.15
           minVersion: 4.15.2
           maxVersion: 4.15.2

   Copy to Clipboard Toggle word wrap

   You must run oc-mirror plugin v2 on a machine with access to the disconnected registry. In the previous example, the disconnected environment, enterprise-registry.in, is accessible.

5. Update the graph URL

   If you are using graph:true, oc-mirror plugin v2 attempts to reach the cincinnati API endpoint. Because this environment is disconnected, be sure to export the environment variable UPDATE_URL_OVERRIDE to refer to the URL for the OpenShift Update Service (OSUS):

   $ export UPDATE_URL_OVERRIDE=https://<osus.enterprise.in>/graph

   Copy to Clipboard Toggle word wrap

   For more information on setting up OSUS on an OpenShift cluster, see "Updating a cluster in a disconnected environment using the OpenShift Update Service".

   Note

   When updating between OpenShift Container Platform Extended Update Support (EUS) versions, you must also include images for an intermediate minor version between the current and target versions. The oc-mirror plugin v2 might not always detect this requirement automatically, so check the Red Hat OpenShift Container Platform Update Graph page to confirm any required intermediate versions.

   Use the Update Graph page to find the intermediate minor versions suggested by the application, and include any of these versions in the ImageSetConfiguration file when using the oc-mirror plugin v2.

6. Generate a mirror archive from the enterprise registry for the enclave.

   To prepare an archive for the enclave1, the user executes oc-mirror plugin v2 in the enterprise disconnected environment by using the imageSetConfiguration specific for that enclave. This ensures that only images needed by that enclave are mirrored:

   $ oc mirror --v2 -c isc-enclave.yaml
   file:///disk-enc1/

   Copy to Clipboard Toggle word wrap

   This action collects all the OpenShift Container Platform content into an archive and generates an archive on disk.

7. After the archive is generated, it will be transferred to the enclave1 network. The transport mechanism is not the responsibility of oc-mirror plugin v2.

8. Mirror contents to the enclave registry

   After the transfer of the archive is done, the user can execute oc-mirror plugin v2 again in order to mirror the relevant archive contents to the registry.

   $ oc mirror --v2 -c isc-enclave.yaml --from file://local-disk docker://registry.enc1.in

   Copy to Clipboard Toggle word wrap

   The administrators of the OpenShift Container Platform cluster in enclave1 are now ready to install or upgrade that cluster.