Chapter 14. Converting a connected cluster to a disconnected cluster

Procedure

1. Configuring credentials that allow image mirroring:

   1. Add the CA certificate for the mirror registry, in the simple PEM or DER file formats, to the list of trusted CAs. For example:

      $ cp </path/to/cert.crt> /usr/share/pki/ca-trust-source/anchors/

      where, </path/to/cert.crt>

      Specifies the path to the certificate on your local file system.

   2. Update the CA trust. For example, in Linux:

      $ update-ca-trust

   3. Extract the .dockerconfigjson file from the global pull secret:

      $ oc extract secret/pull-secret -n openshift-config --confirm --to=.

      Example output

      .dockerconfigjson

   4. Edit the .dockerconfigjson file to add your mirror registry and authentication credentials and save it as a new file:

      {"auths":{"<local_registry>": {"auth": "<credentials>","email": "you@example.com"}}},"<registry>:<port>/<namespace>/":{"auth":"<token>"}}}

      where:

      <local_registry>

      Specifies the registry domain name, and optionally the port, that your mirror registry uses to serve content.

      auth

      Specifies the base64-encoded user name and password for your mirror registry.

      <registry>:<port>/<namespace>

      Specifies the mirror registry details.

      <token>

      Specifies the base64-encoded username:password for your mirror registry.

      For example:

      $ {"auths":{"cloud.openshift.com":{"auth":"b3BlbnNoaWZ0Y3UjhGOVZPT0lOMEFaUjdPUzRGTA==","email":"user@example.com"},
      "quay.io":{"auth":"b3BlbnNoaWZ0LXJlbGVhc2UtZGOVZPT0lOMEFaUGSTd4VGVGVUjdPUzRGTA==","email":"user@example.com"},
      "registry.connect.redhat.com"{"auth":"NTE3MTMwNDB8dWhjLTFEZlN3VHkxOSTd4VGVGVU1MdTpleUpoYkdjaUailA==","email":"user@example.com"},
      "registry.redhat.io":{"auth":"NTE3MTMwNDB8dWhjLTFEZlN3VH3BGSTd4VGVGVU1MdTpleUpoYkdjaU9fZw==","email":"user@example.com"},
      "registry.svc.ci.openshift.org":{"auth":"dXNlcjpyWjAwWVFjSEJiT2RKVW1pSmg4dW92dGp1SXRxQ3RGN1pwajJhN1ZXeTRV"},"my-registry:5000/my-namespace/":{"auth":"dXNlcm5hbWU6cGFzc3dvcmQ="}}}

Procedure

1. Mirror the Operator Lifecycle Manager (OLM) images:

   $ oc adm catalog mirror registry.redhat.io/redhat/redhat-operator-index:v{product-version} <mirror_registry>:<port>/olm -a <reg_creds>

   where:

   product-version

   Specifies the tag that corresponds to the version of OpenShift Container Platform to install, such as 4.8.

   mirror_registry

   Specifies the fully qualified domain name (FQDN) for the target registry and namespace to mirror the Operator content to, where <namespace> is any existing namespace on the registry.

   reg_creds

   Specifies the location of your modified .dockerconfigjson file.

   For example:

   $ oc adm catalog mirror registry.redhat.io/redhat/redhat-operator-index:v4.8 mirror.registry.com:443/olm -a ./.dockerconfigjson  --index-filter-by-os='.*'

2. Mirror the content for any other Red Hat-provided Operator:

   $ oc adm catalog mirror <index_image> <mirror_registry>:<port>/<namespace> -a <reg_creds>

   where:

   index_image

   Specifies the index image for the catalog that you want to mirror.

   mirror_registry

   Specifies the FQDN for the target registry and namespace to mirror the Operator content to, where <namespace> is any existing namespace on the registry.

   reg_creds

   Optional: Specifies the location of your registry credentials file, if required.

   For example:

   $ oc adm catalog mirror registry.redhat.io/redhat/community-operator-index:v4.8 mirror.registry.com:443/olm -a ./.dockerconfigjson  --index-filter-by-os='.*'

3. Mirror the OpenShift Container Platform image repository:

   $ oc adm release mirror -a .dockerconfigjson --from=quay.io/openshift-release-dev/ocp-release:v<product-version>-<architecture> --to=<local_registry>/<local_repository> --to-release-image=<local_registry>/<local_repository>:v<product-version>-<architecture>

   where:

   product-version

   Specifies the tag that corresponds to the version of OpenShift Container Platform to install, such as 4.8.15-x86_64.

   architecture

   Specifies the type of architecture for your server, such as x86_64.

   local_registry

   Specifies the registry domain name for your mirror repository.

   local_repository

   Specifies the name of the repository to create in your registry, such as ocp4/openshift4.

   For example:

   $ oc adm release mirror -a .dockerconfigjson --from=quay.io/openshift-release-dev/ocp-release:4.8.15-x86_64 --to=mirror.registry.com:443/ocp/release --to-release-image=mirror.registry.com:443/ocp/release:4.8.15-x86_64

   Example output

   info: Mirroring 109 images to mirror.registry.com/ocp/release ...
   mirror.registry.com:443/
     ocp/release
   	manifests:
     	sha256:086224cadce475029065a0efc5244923f43fb9bb3bb47637e0aaf1f32b9cad47 -> 4.8.15-x86_64-thanos
     	sha256:0a214f12737cb1cfbec473cc301aa2c289d4837224c9603e99d1e90fc00328db -> 4.8.15-x86_64-kuryr-controller
     	sha256:0cf5fd36ac4b95f9de506623b902118a90ff17a07b663aad5d57c425ca44038c -> 4.8.15-x86_64-pod
     	sha256:0d1c356c26d6e5945a488ab2b050b75a8b838fc948a75c0fa13a9084974680cb -> 4.8.15-x86_64-kube-client-agent
   
   …..
   sha256:66e37d2532607e6c91eedf23b9600b4db904ce68e92b43c43d5b417ca6c8e63c mirror.registry.com:443/ocp/release:4.5.41-multus-admission-controller
   sha256:d36efdbf8d5b2cbc4dcdbd64297107d88a31ef6b0ec4a39695915c10db4973f1 mirror.registry.com:443/ocp/release:4.5.41-cluster-kube-scheduler-operator
   sha256:bd1baa5c8239b23ecdf76819ddb63cd1cd6091119fecdbf1a0db1fb3760321a2 mirror.registry.com:443/ocp/release:4.5.41-aws-machine-controllers
   info: Mirroring completed in 2.02s (0B/s)
   
   Success
   Update image:  mirror.registry.com:443/ocp/release:4.5.41-x86_64
   Mirror prefix: mirror.registry.com:443/ocp/release

4. Mirror any other registries, as needed:

   $ oc image mirror <online_registry>/my/image:latest <mirror_registry>

Procedure

1. Edit your .dockerconfigjson file to remove the cloud.openshift.com entry, for example:

   "cloud.openshift.com":{"auth":"<hash>","email":"user@example.com"}

2. Save the file.

3. Update the cluster secret with the edited .dockerconfigjson file:

   $ oc set data secret/pull-secret -n openshift-config --from-file=.dockerconfigjson=./.dockerconfigjson

4. Verify that the Insights Operator is no longer degraded:

   $ oc get co insights

   Example output

   NAME       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
   insights   4.5.41    True        False         False      3d

Procedure

1. View the ICSP objects in your cluster:

   $ oc get imagecontentsourcepolicy

   Example output

   NAME                 AGE
   mirror-ocp           6d20h
   ocp4-index-0         6d18h
   qe45-index-0         6d15h

2. Delete all the ICSP objects you created when disconnecting your cluster:

   $ oc delete imagecontentsourcepolicy <icsp_name> <icsp_name> <icsp_name>

   For example:

   $ oc delete imagecontentsourcepolicy mirror-ocp ocp4-index-0 qe45-index-0

   Example output

   imagecontentsourcepolicy.operator.openshift.io "mirror-ocp" deleted
   imagecontentsourcepolicy.operator.openshift.io "ocp4-index-0" deleted
   imagecontentsourcepolicy.operator.openshift.io "qe45-index-0" deleted

3. Wait for all the nodes to restart and return to the READY status and verify that the registries.conf file is pointing to the original registries and not the mirror registries:

   1. Log into a node:

      $ oc debug node/<node_name>

   2. Set /host as the root directory within the debug shell:

      sh-4.4# chroot /host

   3. Examine the registries.conf file:

      sh-4.4# cat /etc/containers/registries.conf

      Example output

      unqualified-search-registries = ["registry.access.redhat.com", "docker.io"] 1

      1

      The registry and registry.mirror entries created by the ICSPs you deleted are removed.