Chapter 4. Administrator tasks

Procedure

1. Navigate in the web console to the Operators OperatorHub page.

2. Scroll or type a keyword into the Filter by keyword box to find the Operator you want. For example, type advanced to find the Advanced Cluster Management for Kubernetes Operator.

   You can also filter options by Infrastructure Features. For example, select Disconnected if you want to see Operators that work in disconnected environments, also known as restricted network environments.

3. Select the Operator to display additional information.

   Note

   Choosing a Community Operator warns that Red Hat does not certify Community Operators; you must acknowledge the warning before continuing.

4. Read the information about the Operator and click Install.

5. On the Install Operator page:

   1. Select one of the following:

      • All namespaces on the cluster (default) installs the Operator in the default openshift-operators namespace to watch and be made available to all namespaces in the cluster. This option is not always available.
      • A specific namespace on the cluster allows you to choose a specific, single namespace in which to install the Operator. The Operator will only watch and be made available for use in this single namespace.
   2. Choose a specific, single namespace in which to install the Operator. The Operator will only watch and be made available for use in this single namespace.
   3. Select an Update Channel (if more than one is available).
   4. Select Automatic or Manual approval strategy, as described earlier.

6. Click Install to make the Operator available to the selected namespaces on this OpenShift Container Platform cluster.

   1. If you selected a Manual approval strategy, the upgrade status of the subscription remains Upgrading until you review and approve the install plan.

      After approving on the Install Plan page, the subscription upgrade status moves to Up to date.

   2. If you selected an Automatic approval strategy, the upgrade status should resolve to Up to date without intervention.

7. After the upgrade status of the subscription is Up to date, select Operators Installed Operators to verify that the cluster service version (CSV) of the installed Operator eventually shows up. The Status should ultimately resolve to InstallSucceeded in the relevant namespace.

   Note

   For the All namespaces…​ installation mode, the status resolves to InstallSucceeded in the openshift-operators namespace, but the status is Copied if you check in other namespaces.

   If it does not:

   1. Check the logs in any pods in the openshift-operators project (or other relevant namespace if A specific namespace…​ installation mode was selected) on the Workloads Pods page that are reporting issues to troubleshoot further.

Procedure

1. View the list of Operators available to the cluster from OperatorHub:

   $ oc get packagemanifests -n openshift-marketplace

   Example output

   NAME                               CATALOG               AGE
   3scale-operator                    Red Hat Operators     91m
   advanced-cluster-management        Red Hat Operators     91m
   amq7-cert-manager                  Red Hat Operators     91m
   ...
   couchbase-enterprise-certified     Certified Operators   91m
   crunchy-postgres-operator          Certified Operators   91m
   mongodb-enterprise                 Certified Operators   91m
   ...
   etcd                               Community Operators   91m
   jaeger                             Community Operators   91m
   kubefed                            Community Operators   91m
   ...

   Note the catalog for your desired Operator.

2. Inspect your desired Operator to verify its supported install modes and available channels:

   $ oc describe packagemanifests <operator_name> -n openshift-marketplace

3. An Operator group, defined by an OperatorGroup object, selects target namespaces in which to generate required RBAC access for all Operators in the same namespace as the Operator group.

   The namespace to which you subscribe the Operator must have an Operator group that matches the install mode of the Operator, either the AllNamespaces or SingleNamespace mode. If the Operator you intend to install uses the AllNamespaces, then the openshift-operators namespace already has an appropriate Operator group in place.

   However, if the Operator uses the SingleNamespace mode and you do not already have an appropriate Operator group in place, you must create one.

   Note

   The web console version of this procedure handles the creation of the OperatorGroup and Subscription objects automatically behind the scenes for you when choosing SingleNamespace mode.

   1. Create an OperatorGroup object YAML file, for example operatorgroup.yaml:

      Example OperatorGroup object

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: <operatorgroup_name>
        namespace: <namespace>
      spec:
        targetNamespaces:
        - <namespace>

   2. Create the OperatorGroup object:

      $ oc apply -f operatorgroup.yaml

4. Create a Subscription object YAML file to subscribe a namespace to an Operator, for example sub.yaml:

   Example Subscription object

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: <subscription_name>
     namespace: openshift-operators 1
   spec:
     channel: <channel_name> 2
     name: <operator_name> 3
     source: redhat-operators 4
     sourceNamespace: openshift-marketplace 5
     config:
       env: 6
       - name: ARGS
         value: "-v=10"
       envFrom: 7
       - secretRef:
           name: license-secret
       volumes: 8
       - name: <volume_name>
         configMap:
           name: <configmap_name>
       volumeMounts: 9
       - mountPath: <directory_name>
         name: <volume_name>
       tolerations: 10
       - operator: "Exists"
       resources: 11
         requests:
           memory: "64Mi"
           cpu: "250m"
         limits:
           memory: "128Mi"
           cpu: "500m"
       nodeSelector: 12
         foo: bar

   1

   For AllNamespaces install mode usage, specify the openshift-operators namespace. Otherwise, specify the relevant single namespace for SingleNamespace install mode usage.

   2

   Name of the channel to subscribe to.

   3

   Name of the Operator to subscribe to.

   4

   Name of the catalog source that provides the Operator.

   5

   Namespace of the catalog source. Use openshift-marketplace for the default OperatorHub catalog sources.

   6

   The env parameter defines a list of Environment Variables that must exist in all containers in the pod created by OLM.

   7

   The envFrom parameter defines a list of sources to populate Environment Variables in the container.

   8

   The volumes parameter defines a list of Volumes that must exist on the pod created by OLM.

   9

   The volumeMounts parameter defines a list of VolumeMounts that must exist in all containers in the pod created by OLM. If a volumeMount references a volume that does not exist, OLM fails to deploy the Operator.

   10

   The tolerations parameter defines a list of Tolerations for the pod created by OLM.

   11

   The resources parameter defines resource constraints for all the containers in the pod created by OLM.

   12

   The nodeSelector parameter defines a NodeSelector for the pod created by OLM.

5. Create the Subscription object:

   $ oc apply -f sub.yaml

   At this point, OLM is now aware of the selected Operator. A cluster service version (CSV) for the Operator should appear in the target namespace, and APIs provided by the Operator should be available for creation.

Procedure

1. Create a Subscription object YAML file that subscribes a namespace to an Operator with a specific version by setting the startingCSV field. Set the installPlanApproval field to Manual to prevent the Operator from automatically upgrading if a later version exists in the catalog.

   For example, the following sub.yaml file can be used to install the Red Hat Quay Operator specifically to version 3.4.0:

   Subscription with a specific starting Operator version

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: quay-operator
     namespace: quay
   spec:
     channel: quay-v3.4
     installPlanApproval: Manual 1
     name: quay-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     startingCSV: quay-operator.v3.4.0 2

   1

   Set the approval strategy to Manual in case your specified version is superseded by a later version in the catalog. This plan prevents an automatic upgrade to a later version and requires manual approval before the starting CSV can complete the installation.

   2

   Set a specific version of an Operator CSV.

2. Create the Subscription object:

   $ oc apply -f sub.yaml

3. Manually approve the pending install plan to complete the Operator installation.

Procedure

1. In the Administrator perspective of the OpenShift Container Platform web console, navigate to Operators Installed Operators.
2. Click the name of the Operator you want to change the update channel for.
3. Click the Subscription tab.
4. Click the name of the update channel under Channel.
5. Click the newer update channel that you want to change to, then click Save.

6. For subscriptions with an Automatic approval strategy, the upgrade begins automatically. Navigate back to the Operators Installed Operators page to monitor the progress of the upgrade. When complete, the status changes to Succeeded and Up to date.

   For subscriptions with a Manual approval strategy, you can manually approve the upgrade from the Subscription tab.

Procedure

1. In the Administrator perspective of the OpenShift Container Platform web console, navigate to Operators Installed Operators.
2. Operators that have a pending upgrade display a status with Upgrade available. Click the name of the Operator you want to upgrade.
3. Click the Subscription tab. Any upgrades requiring approval are displayed next to Upgrade Status. For example, it might display 1 requires approval.
4. Click 1 requires approval, then click Preview Install Plan.
5. Review the resources that are listed as available for upgrade. When satisfied, click Approve.
6. Navigate back to the Operators Installed Operators page to monitor the progress of the upgrade. When complete, the status changes to Succeeded and Up to date.

Procedure

1. From the Operators Installed Operators page, scroll or type a keyword into the Filter by name to find the Operator you want. Then, click on it.

2. On the right side of the Operator Details page, select Uninstall Operator from the Actions list.

   An Uninstall Operator? dialog box is displayed, reminding you that:

   Removing the Operator will not remove any of its custom resource definitions or managed resources. If your Operator has deployed applications on the cluster or configured off-cluster resources, these will continue to run and need to be cleaned up manually.

   This action removes the Operator as well as the Operator deployments and pods, if any. Any Operands, and resources managed by the Operator, including CRDs and CRs, are not removed. The web console enables dashboards and navigation items for some Operators. To remove these after uninstalling the Operator, you might need to manually delete the Operator CRDs.

3. Select Uninstall. This Operator stops running and no longer receives updates.

Procedure

1. Check the current version of the subscribed Operator (for example, jaeger) in the currentCSV field:

   $ oc get subscription jaeger -n openshift-operators -o yaml | grep currentCSV

   Example output

     currentCSV: jaeger-operator.v1.8.2

2. Delete the subscription (for example, jaeger):

   $ oc delete subscription jaeger -n openshift-operators

   Example output

   subscription.operators.coreos.com "jaeger" deleted

3. Delete the CSV for the Operator in the target namespace using the currentCSV value from the previous step:

   $ oc delete clusterserviceversion jaeger-operator.v1.8.2 -n openshift-operators

   Example output

   clusterserviceversion.operators.coreos.com "jaeger-operator.v1.8.2" deleted

Procedure

1. Get the names of the Subscription and ClusterServiceVersion objects from the namespace where the Operator is installed:

   $ oc get sub,csv -n <namespace>

   Example output

   NAME                                                       PACKAGE                  SOURCE             CHANNEL
   subscription.operators.coreos.com/elasticsearch-operator   elasticsearch-operator   redhat-operators   5.0
   
   NAME                                                                         DISPLAY                            VERSION    REPLACES   PHASE
   clusterserviceversion.operators.coreos.com/elasticsearch-operator.5.0.0-65   OpenShift Elasticsearch Operator   5.0.0-65              Succeeded

2. Delete the subscription:

   $ oc delete subscription <subscription_name> -n <namespace>

3. Delete the cluster service version:

   $ oc delete csv <csv_name> -n <namespace>

4. Get the names of any failing jobs and related config maps in the openshift-marketplace namespace:

   $ oc get job,configmap -n openshift-marketplace

   Example output

   NAME                                                                        COMPLETIONS   DURATION   AGE
   job.batch/1de9443b6324e629ddf31fed0a853a121275806170e34c926d69e53a7fcbccb   1/1           26s        9m30s
   
   NAME                                                                        DATA   AGE
   configmap/1de9443b6324e629ddf31fed0a853a121275806170e34c926d69e53a7fcbccb   3      9m30s

5. Delete the job:

   $ oc delete job <job_name> -n openshift-marketplace

   This ensures pods that try to pull the inaccessible image are not recreated.

6. Delete the config map:

   $ oc delete configmap <configmap_name> -n openshift-marketplace

7. Reinstall the Operator using OperatorHub in the web console.

Procedure

1. Navigate in the web console to the Operators OperatorHub page.
2. Select the Operator and click Install.

3. On the Install Operator page, modify the Subscription object to include one or more of the following environment variables in the spec section:

   • HTTP_PROXY
   • HTTPS_PROXY
   • NO_PROXY

   For example:

   Subscription object with proxy setting overrides

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: etcd-config-test
     namespace: openshift-operators
   spec:
     config:
       env:
       - name: HTTP_PROXY
         value: test_http
       - name: HTTPS_PROXY
         value: test_https
       - name: NO_PROXY
         value: test
     channel: clusterwide-alpha
     installPlanApproval: Automatic
     name: etcd
     source: community-operators
     sourceNamespace: openshift-marketplace
     startingCSV: etcdoperator.v0.9.4-clusterwide

   Note

   These environment variables can also be unset using an empty value to remove any previously set cluster-wide or custom proxy settings.

   OLM handles these environment variables as a unit; if at least one of them is set, all three are considered overridden and the cluster-wide defaults are not used for the deployments of the subscribed Operator.

4. Click Install to make the Operator available to the selected namespaces.

5. After the CSV for the Operator appears in the relevant namespace, you can verify that custom proxy environment variables are set in the deployment. For example, using the CLI:

   $ oc get deployment -n openshift-operators \
       etcd-operator -o yaml \
       | grep -i "PROXY" -A 2

   Example output

           - name: HTTP_PROXY
             value: test_http
           - name: HTTPS_PROXY
             value: test_https
           - name: NO_PROXY
             value: test
           image: quay.io/coreos/etcd-operator@sha256:66a37fd61a06a43969854ee6d3e21088a98b93838e284a6086b13917f96b0d9c
   ...

Procedure

1. Create an empty config map in the namespace where the subscription for your Operator exists and include the following label:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: trusted-ca 1
     labels:
       config.openshift.io/inject-trusted-cabundle: "true" 2

   1

   Name of the config map.

   2

   Requests the Cluster Network Operator to inject the merged bundle.

   After creating this config map, it is immediately populated with the certificate contents of the merged bundle.

2. Update your the Subscription object to include a spec.config section that mounts the trusted-ca config map as a volume to each container within a pod that requires a custom CA:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: my-operator
   spec:
     package: etcd
     channel: alpha
     config: 1
       selector:
         matchLabels:
           <labels_for_pods> 2
       volumes: 3
       - name: trusted-ca
         configMap:
           name: trusted-ca
           items:
             - key: ca-bundle.crt 4
               path: tls-ca-bundle.pem 5
       volumeMounts: 6
       - name: trusted-ca
         mountPath: /etc/pki/ca-trust/extracted/pem
         readOnly: true

   1

   Add a config section if it does not exist.

   2

   Specify labels to match pods that are owned by the Operator.

   3

   Create a trusted-ca volume.

   4

   ca-bundle.crt is required as the config map key.

   5

   tls-ca-bundle.pem is required as the config map path.

   6

   Create a trusted-ca volume mount.

Procedure

1. List Operator subscriptions:

   $ oc get subs -n <operator_namespace>

2. Use the oc describe command to inspect a Subscription resource:

   $ oc describe sub <subscription_name> -n <operator_namespace>

3. In the command output, find the Conditions section for the status of Operator subscription condition types. In the following example, the CatalogSourcesUnhealthy condition type has a status of false because all available catalog sources are healthy:

   Example output

   Conditions:
      Last Transition Time:  2019-07-29T13:42:57Z
      Message:               all available catalogsources are healthy
      Reason:                AllCatalogSourcesHealthy
      Status:                False
      Type:                  CatalogSourcesUnhealthy

Procedure

1. List the catalog sources in a namespace. For example, you can check the openshift-marketplace namespace, which is used for cluster-wide catalog sources:

   $ oc get catalogsources -n openshift-marketplace

   Example output

   NAME                  DISPLAY               TYPE   PUBLISHER   AGE
   certified-operators   Certified Operators   grpc   Red Hat     55m
   community-operators   Community Operators   grpc   Red Hat     55m
   example-catalog       Example Catalog       grpc   Example Org 2m25s
   redhat-marketplace    Red Hat Marketplace   grpc   Red Hat     55m
   redhat-operators      Red Hat Operators     grpc   Red Hat     55m

2. Use the oc describe command to get more details and status about a catalog source:

   $ oc describe catalogsource example-catalog -n openshift-marketplace

   Example output

   Name:         example-catalog
   Namespace:    openshift-marketplace
   ...
   Status:
     Connection State:
       Address:              example-catalog.openshift-marketplace.svc:50051
       Last Connect:         2021-09-09T17:07:35Z
       Last Observed State:  TRANSIENT_FAILURE
     Registry Service:
       Created At:         2021-09-09T17:05:45Z
       Port:               50051
       Protocol:           grpc
       Service Name:       example-catalog
       Service Namespace:  openshift-marketplace

   In the preceding example output, the last observed state is TRANSIENT_FAILURE. This state indicates that there is a problem establishing a connection for the catalog source.

3. List the pods in the namespace where your catalog source was created:

   $ oc get pods -n openshift-marketplace

   Example output

   NAME                                    READY   STATUS             RESTARTS   AGE
   certified-operators-cv9nn               1/1     Running            0          36m
   community-operators-6v8lp               1/1     Running            0          36m
   marketplace-operator-86bfc75f9b-jkgbc   1/1     Running            0          42m
   example-catalog-bwt8z                   0/1     ImagePullBackOff   0          3m55s
   redhat-marketplace-57p8c                1/1     Running            0          36m
   redhat-operators-smxx8                  1/1     Running            0          36m

   When a catalog source is created in a namespace, a pod for the catalog source is created in that namespace. In the preceding example output, the status for the example-catalog-bwt8z pod is ImagePullBackOff. This status indicates that there is an issue pulling the catalog source’s index image.

4. Use the oc describe command to inspect a pod for more detailed information:

   $ oc describe pod example-catalog-bwt8z -n openshift-marketplace

   Example output

   Name:         example-catalog-bwt8z
   Namespace:    openshift-marketplace
   Priority:     0
   Node:         ci-ln-jyryyg2-f76d1-ggdbq-worker-b-vsxjd/10.0.128.2
   ...
   Events:
     Type     Reason          Age                From               Message
     ----     ------          ----               ----               -------
     Normal   Scheduled       48s                default-scheduler  Successfully assigned openshift-marketplace/example-catalog-bwt8z to ci-ln-jyryyf2-f76d1-fgdbq-worker-b-vsxjd
     Normal   AddedInterface  47s                multus             Add eth0 [10.131.0.40/23] from openshift-sdn
     Normal   BackOff         20s (x2 over 46s)  kubelet            Back-off pulling image "quay.io/example-org/example-catalog:v1"
     Warning  Failed          20s (x2 over 46s)  kubelet            Error: ImagePullBackOff
     Normal   Pulling         8s (x3 over 47s)   kubelet            Pulling image "quay.io/example-org/example-catalog:v1"
     Warning  Failed          8s (x3 over 47s)   kubelet            Failed to pull image "quay.io/example-org/example-catalog:v1": rpc error: code = Unknown desc = reading manifest v1 in quay.io/example-org/example-catalog: unauthorized: access to the requested resource is not authorized
     Warning  Failed          8s (x3 over 47s)   kubelet            Error: ErrImagePull

   In the preceding example output, the error messages indicate that the catalog source’s index image is failing to pull successfully because of an authorization issue. For example, the index image might be stored in a registry that requires login credentials.

Procedure

1. Create a new namespace:

   $ cat <<EOF | oc create -f -
   apiVersion: v1
   kind: Namespace
   metadata:
     name: scoped
   EOF

2. Allocate permissions that you want the Operator(s) to be confined to. This involves creating a new service account, relevant role(s), and role binding(s).

   $ cat <<EOF | oc create -f -
   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: scoped
     namespace: scoped
   EOF

   The following example grants the service account permissions to do anything in the designated namespace for simplicity. In a production environment, you should create a more fine-grained set of permissions:

   $ cat <<EOF | oc create -f -
   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: scoped
     namespace: scoped
   rules:
   - apiGroups: ["*"]
     resources: ["*"]
     verbs: ["*"]
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: scoped-bindings
     namespace: scoped
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: Role
     name: scoped
   subjects:
   - kind: ServiceAccount
     name: scoped
     namespace: scoped
   EOF

3. Create an OperatorGroup object in the designated namespace. This Operator group targets the designated namespace to ensure that its tenancy is confined to it.

   In addition, Operator groups allow a user to specify a service account. Specify the service account created in the previous step:

   $ cat <<EOF | oc create -f -
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: scoped
     namespace: scoped
   spec:
     serviceAccountName: scoped
     targetNamespaces:
     - scoped
   EOF

   Any Operator installed in the designated namespace is tied to this Operator group and therefore to the service account specified.

4. Create a Subscription object in the designated namespace to install an Operator:

   $ cat <<EOF | oc create -f -
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: etcd
     namespace: scoped
   spec:
     channel: singlenamespace-alpha
     name: etcd
     source: <catalog_source_name> 1
     sourceNamespace: <catalog_source_namespace> 2
   EOF

   1

   Specify a catalog source that already exists in the designated namespace or one that is in the global catalog namespace.

   2

   Specify a namespace where the catalog source was created.

   Any Operator tied to this Operator group is confined to the permissions granted to the specified service account. If the Operator requests permissions that are outside the scope of the service account, the installation fails with relevant errors.

Procedure

1. Review the Subscription object. Its status has an object reference installPlanRef that points to the InstallPlan object that attempted to create the necessary [Cluster]Role[Binding] object(s) for the Operator:

   apiVersion: operators.coreos.com/v1
   kind: Subscription
   metadata:
     name: etcd
     namespace: scoped
   status:
     installPlanRef:
       apiVersion: operators.coreos.com/v1
       kind: InstallPlan
       name: install-4plp8
       namespace: scoped
       resourceVersion: "117359"
       uid: 2c1df80e-afea-11e9-bce3-5254009c9c23

2. Check the status of the InstallPlan object for any errors:

   apiVersion: operators.coreos.com/v1
   kind: InstallPlan
   status:
     conditions:
     - lastTransitionTime: "2019-07-26T21:13:10Z"
       lastUpdateTime: "2019-07-26T21:13:10Z"
       message: 'error creating clusterrole etcdoperator.v0.9.4-clusterwide-dsfx4: clusterroles.rbac.authorization.k8s.io
         is forbidden: User "system:serviceaccount:scoped:scoped" cannot create resource
         "clusterroles" in API group "rbac.authorization.k8s.io" at the cluster scope'
       reason: InstallComponentFailed
       status: "False"
       type: Installed
     phase: Failed

   The error message tells you:

   • The type of resource it failed to create, including the API group of the resource. In this case, it was clusterroles in the rbac.authorization.k8s.io group.
   • The name of the resource.
   • The type of error: is forbidden tells you that the user does not have enough permission to do the operation.
   • The name of the user who attempted to create or update the resource. In this case, it refers to the service account specified in the Operator group.

   • The scope of the operation: cluster scope or not.

     The user can add the missing permission to the service account and then iterate.

     Note

     Operator Lifecycle Manager (OLM) does not currently provide the complete list of errors on the first try.

Procedure

1. In the Administrator perspective of the web console, navigate to Administration Cluster Settings.
2. Click the Global Configuration tab, and then click OperatorHub.
3. Click the Sources tab.
4. Select the Options menu for the catalog that you want to remove, and then click Delete CatalogSource.

Procedure

1. Authenticate with registry.redhat.io:

   $ podman login registry.redhat.io

2. Authenticate with your target registry:

   $ podman login <target_registry>

3. Determine the list of packages you want to include in your pruned index.

   1. Run the source index image that you want to prune in a container. For example:

      $ podman run -p50051:50051 \
          -it registry.redhat.io/redhat/redhat-operator-index:v4.6

      Example output

      Trying to pull registry.redhat.io/redhat/redhat-operator-index:v4.6...
      Getting image source signatures
      Copying blob ae8a0c23f5b1 done
      ...
      INFO[0000] serving registry                              database=/database/index.db port=50051

   2. In a separate terminal session, use the grpcurl command to get a list of the packages provided by the index:

      $ grpcurl -plaintext localhost:50051 api.Registry/ListPackages > packages.out

   3. Inspect the packages.out file and identify which package names from this list you want to keep in your pruned index. For example:

      Example snippets of packages list

      ...
      {
        "name": "advanced-cluster-management"
      }
      ...
      {
        "name": "jaeger-product"
      }
      ...
      {
      {
        "name": "quay-operator"
      }
      ...

   4. In the terminal session where you executed the podman run command, press Ctrl and C to stop the container process.

4. Run the following command to prune the source index of all but the specified packages:

   $ opm index prune \
       -f registry.redhat.io/redhat/redhat-operator-index:v4.6 \1
       -p advanced-cluster-management,jaeger-product,quay-operator \2
       [-i registry.redhat.io/openshift4/ose-operator-registry:v4.6] \3
       -t <target_registry>:<port>/<namespace>/redhat-operator-index:v4.6 4

   1

   Index to prune.

   2

   Comma-separated list of packages to keep.

   3

   Required only for IBM Power Systems and IBM Z images: Operator Registry base image with the tag that matches the target OpenShift Container Platform cluster major and minor version.

   4

   Custom tag for new index image being built.

5. Run the following command to push the new index image to your target registry:

   $ podman push <target_registry>:<port>/<namespace>/redhat-operator-index:v4.6

   where <namespace> is any existing namespace on the registry. For example, you might create an olm-mirror namespace to push all mirrored content to.

Procedure

1. If you want to mirror a Red Hat-provided catalog, run the following command on your workstation with unrestricted network access to authenticate with registry.redhat.io:

   $ podman login registry.redhat.io

2. The oc adm catalog mirror command extracts the contents of an index image to generate the manifests required for mirroring. The default behavior of the command generates manifests, then automatically mirrors all of the image content from the index image, as well as the index image itself, to your mirror registry. Alternatively, if your mirror registry is on a completely disconnected, or airgapped, host, you can first mirror the content to removable media, move the media to the disconnected environment, then mirror the content from the media to the registry.

   • Option A: If your mirror registry is on the same network as your workstation with unrestricted network access, take the following actions on your workstation:

     1. If your mirror registry requires authentication, run the following command to log in to the registry:

        $ podman login <mirror_registry>

     2. Run the following command to mirror the content:

        $ oc adm catalog mirror \
            <index_image> \1
            <mirror_registry>:<port>/<namespace> \2
            [-a ${REG_CREDS}] \3
            [--insecure] \4
            [--index-filter-by-os='<platform>/<arch>'] \5
            [--manifests-only] 6

        1

        Specify the index image for the catalog you want to mirror. For example, this might be a pruned index image that you created previously, or one of the source index images for the default catalogs, such as registry.redhat.io/redhat/redhat-operator-index:v4.6.

        2

        Specify the fully qualified domain name (FQDN) for the target registry and namespace to mirror the Operator content to, where <namespace> is any existing namespace on the registry. For example, you might create an olm-mirror namespace to push all mirrored content to.

        3

        Optional: If required, specify the location of your registry credentials file. {REG_CREDS} is required for registry.redhat.io.

        4

        Optional: If you do not want to configure trust for the target registry, add the --insecure flag.

        5

        Optional: Specify which platform and architecture of the index image to select when multiple variants are available. Images are passed as '<platform>/<arch>[/<variant>]'. This does not apply to images referenced by the index. Valid values are linux/amd64, linux/ppc64le, and linux/s390x.

        6

        Optional: Generate only the manifests required for mirroring, and do not actually mirror the image content to a registry. This option can be useful for reviewing what will be mirrored, and it allows you to make any changes to the mapping list if you require only a subset of packages. You can then use the mapping.txt file with the oc image mirror command to mirror the modified list of images in a later step. This flag is intended for only advanced selective mirroring of content from the catalog; the opm index prune command, if you used it previously to prune the index image, is suitable for most catalog management use cases.

        Example output

        src image has index label for database path: /database/index.db
        using database path mapping: /database/index.db:/tmp/153048078
        wrote database to /tmp/153048078 1
        ...
        wrote mirroring manifests to manifests-redhat-operator-index-1614211642 2

        1

        Directory for the temporary index.db database generated by the command.

        2

        Be sure to record the manifests directory name that is generated. This directory name is used in a later step.

        Note

        Red Hat Quay does not support nested repositories. As a result, running the oc adm catalog mirror command will fail with a 401 unauthorized error. As a workaround, you can use the --max-components=2 option when running the oc adm catalog mirror command to disable the creation of nested repositories. For more information on this workaround, see the Unauthorized error thrown while using catalog mirror command with Quay registry Knowledgebase Solution article.

   • Option B: If your mirror registry is on a disconnected host, take the following actions.

     1. Run the following command on your workstation with unrestricted network access to mirror the content to local files:

        $ oc adm catalog mirror \
            <index_image> \1
            file:///local/index \2
            [-a ${REG_CREDS}] \
            [--insecure]

        1

        Specify the index image for the catalog you want to mirror. For example, this might be a pruned index image that you created previously, or one of the source index images for the default catalogs, such as registry.redhat.io/redhat/redhat-operator-index:v4.6.

        2

        Mirrors content to local files in your current directory.

        Example output

        ...
        info: Mirroring completed in 5.93s (5.915MB/s)
        wrote mirroring manifests to manifests-my-index-1614985528 1
        
        To upload local images to a registry, run:
        
        	oc adm catalog mirror file://local/index/myrepo/my-index:v1 REGISTRY/REPOSITORY 2

        1

        Be sure to record the manifests directory name that is generated. This directory name is used in a later step.

        2

        Record the expanded file:// path that based on your provided index image. This path is used in a later step.

     2. Copy the v2/ directory that is generated in your current directory to removable media.
     3. Physically remove the media and attach it to a host in the disconnected environment that has access to the mirror registry.

     4. If your mirror registry requires authentication, run the following command on your host in the disconnected environment to log in to the registry:

        $ podman login <mirror_registry>

     5. Run the following command from the parent directory containing the v2/ directory to upload the images from local files to the mirror registry:

        $ oc adm catalog mirror \
            file://local/index/<repo>/<index_image>:<tag> \1
            <mirror_registry>:<port>/<namespace> \2
            [-a ${REG_CREDS}] \
            [--insecure]

        1

        Specify the file:// path from the previous command output.

        2

        Specify the fully qualified domain name (FQDN) for the target registry and namespace to mirror the Operator content to, where <namespace> is any existing namespace on the registry. For example, you might create an olm-mirror namespace to push all mirrored content to.

        Note

        Red Hat Quay does not support nested repositories. As a result, running the oc adm catalog mirror command will fail with a 401 unauthorized error. As a workaround, you can use the --max-components=2 option when running the oc adm catalog mirror command to disable the creation of nested repositories. For more information on this workaround, see the Unauthorized error thrown while using catalog mirror command with Quay registry Knowledgebase Solution article.

3. After mirroring the content to your registry, inspect the manifests directory that is generated in your current directory.

   Note

   The manifests directory name is used in a later step.

   If you mirrored content to a registry on the same network in the previous step, the directory name takes the following form:

   manifests-<index_image_name>-<random_number>

   If you mirrored content to a registry on a disconnected host in the previous step, the directory name takes the following form:

   manifests-index/<namespace>/<index_image_name>-<random_number>

   The manifests directory contains the following files, some of which might require further modification:

   • The catalogSource.yaml file is a basic definition for a CatalogSource object that is pre-populated with your index image tag and other relevant metadata. This file can be used as is or modified to add the catalog source to your cluster.

     Important

     If you mirrored the content to local files, you must modify your catalogSource.yaml file to remove any backslash (/) characters from the metadata.name field. Otherwise, when you attempt to create the object, it fails with an "invalid resource name" error.

   • The imageContentSourcePolicy.yaml file defines an ImageContentSourcePolicy object that can configure nodes to translate between the image references stored in Operator manifests and the mirrored registry.

     Note

     If your cluster uses an ImageContentSourcePolicy object to configure repository mirroring, you can use only global pull secrets for mirrored registries. You cannot add a pull secret to a project.

   • The mapping.txt file contains all of the source images and where to map them in the target registry. This file is compatible with the oc image mirror command and can be used to further customize the mirroring configuration.

     Important

     If you used the --manifests-only flag during the mirroring process and want to further trim the subset of packages to be mirrored, see the steps in the "Mirroring a Package Manifest Format catalog image" procedure about modifying your mapping.txt file and using the file with the oc image mirror command. After following those further actions, you can continue this procedure.

4. On a host with access to the disconnected cluster, create the ImageContentSourcePolicy object by running the following command to specify the imageContentSourcePolicy.yaml file in your manifests directory:

   $ oc create -f <path/to/manifests/dir>/imageContentSourcePolicy.yaml

   where <path/to/manifests/dir> is the path to the manifests directory for your mirrored content.

Procedure

1. Create a CatalogSource object that references your index image. If you used the oc adm catalog mirror command to mirror your catalog to a target registry, you can use the generated catalogSource.yaml file as a starting point.

   1. Modify the following to your specifications and save it as a catalogSource.yaml file:

      apiVersion: operators.coreos.com/v1alpha1
      kind: CatalogSource
      metadata:
        name: my-operator-catalog 1
        namespace: openshift-marketplace 2
      spec:
        sourceType: grpc
        image: <registry>:<port>/<namespace>/redhat-operator-index:v4.6 3
        displayName: My Operator Catalog
        publisher: <publisher_name> 4
        updateStrategy:
          registryPoll: 5
            interval: 30m

      1

      If you mirrored content to local files before uploading to a registry, remove any backslash (/) characters from the metadata.name field to avoid an "invalid resource name" error when you create the object.

      2

      If you want the catalog source to be available globally to users in all namespaces, specify the openshift-marketplace namespace. Otherwise, you can specify a different namespace for the catalog to be scoped and available only for that namespace.

      3

      Specify your index image.

      4

      Specify your name or an organization name publishing the catalog.

      5

      Catalog sources can automatically check for new versions to keep up to date.

   2. Use the file to create the CatalogSource object:

      $ oc apply -f catalogSource.yaml

2. Verify the following resources are created successfully.

   1. Check the pods:

      $ oc get pods -n openshift-marketplace

      Example output

      NAME                                    READY   STATUS    RESTARTS  AGE
      my-operator-catalog-6njx6               1/1     Running   0         28s
      marketplace-operator-d9f549946-96sgr    1/1     Running   0         26h

   2. Check the catalog source:

      $ oc get catalogsource -n openshift-marketplace

      Example output

      NAME                  DISPLAY               TYPE PUBLISHER  AGE
      my-operator-catalog   My Operator Catalog   grpc            5s

   3. Check the package manifest:

      $ oc get packagemanifest -n openshift-marketplace

      Example output

      NAME                          CATALOG               AGE
      jaeger-product                My Operator Catalog   93s

Procedure

1. Update the existing index by adding bundle images:

   $ opm index add \
       --bundles <registry>/<namespace>/<new_bundle_image>@sha256:<digest> \1
       --from-index <registry>/<namespace>/<existing_index_image>:<existing_tag> \2
       --tag <registry>/<namespace>/<existing_index_image>:<updated_tag> \3
       --pull-tool podman 4

   1

   The --bundles flag specifies a comma-separated list of additional bundle images to add to the index.

   2

   The --from-index flag specifies the previously pushed index.

   3

   The --tag flag specifies the image tag to apply to the updated index image.

   4

   The --pull-tool flag specifies the tool used to pull container images.

   where:

   <registry>

   Specifies the hostname of the registry, such as quay.io or mirror.example.com.

   <namespace>

   Specifies the namespace of the registry, such as ocs-dev or abc.

   <new_bundle_image>

   Specifies the new bundle image to add to the registry, such as ocs-operator.

   <digest>

   Specifies the SHA image ID, or digest, of the bundle image, such as c7f11097a628f092d8bad148406aa0e0951094a03445fd4bc0775431ef683a41.

   <existing_index_image>

   Specifies the previously pushed image, such as abc-redhat-operator-index.

   <existing_tag>

   Specifies a previously pushed image tag, such as 4.6.

   <updated_tag>

   Specifies the image tag to apply to the updated index image, such as 4.6.1.

   Example command

   $ opm index add \
       --bundles quay.io/ocs-dev/ocs-operator@sha256:c7f11097a628f092d8bad148406aa0e0951094a03445fd4bc0775431ef683a41 \
       --from-index mirror.example.com/abc/abc-redhat-operator-index:4.6 \
       --tag mirror.example.com/abc/abc-redhat-operator-index:4.6.1 \
       --pull-tool podman

2. Push the updated index image:

   $ podman push <registry>/<namespace>/<existing_index_image>:<updated_tag>

3. Follow the steps in the Mirroring an Operator catalog procedure again to mirror the updated content. However, when you get to the step about creating the ImageContentSourcePolicy (ICSP) object, use the oc replace command instead of the oc create command. For example:

   $ oc replace -f ./manifests-redhat-operator-index-<random_number>/imageContentSourcePolicy.yaml

   This change is required because the object already exists and must be updated.

   Note

   Normally, the oc apply command can be used to update existing objects that were previously created using oc apply. However, due to a known issue regarding the size of the metadata.annotations field in ICSP objects, the oc replace command must be used for this step currently.

4. After Operator Lifecycle Manager (OLM) automatically polls the index image referenced in the catalog source at its regular interval, verify that the new packages are successfully added:

   $ oc get packagemanifests -n openshift-marketplace