Chapter 4. Administrator tasks

Procedure

1. Navigate in the web console to the Operators OperatorHub page.

2. Scroll or type a keyword into the Filter by keyword box to find the Operator you want. For example, type

   advanced

   to find the Advanced Cluster Management for Kubernetes Operator.

   You can also filter options by Infrastructure Features. For example, select Disconnected if you want to see Operators that work in disconnected environments, also known as restricted network environments.

3. Select the Operator to display additional information.

   Note

   Choosing a Community Operator warns that Red Hat does not certify Community Operators; you must acknowledge the warning before continuing.

4. Read the information about the Operator and click Install.

5. On the Install Operator page:

   1. Select one of the following:

      • All namespaces on the cluster (default) installs the Operator in the default

        openshift-operators

        namespace to watch and be made available to all namespaces in the cluster. This option is not always available.
      • A specific namespace on the cluster allows you to choose a specific, single namespace in which to install the Operator. The Operator will only watch and be made available for use in this single namespace.
   2. Choose a specific, single namespace in which to install the Operator. The Operator will only watch and be made available for use in this single namespace.
   3. Select an Update Channel (if more than one is available).
   4. Select Automatic or Manual approval strategy, as described earlier.

6. Click Install to make the Operator available to the selected namespaces on this OpenShift Container Platform cluster.

   1. If you selected a Manual approval strategy, the upgrade status of the subscription remains Upgrading until you review and approve the install plan.

      After approving on the Install Plan page, the subscription upgrade status moves to Up to date.

   2. If you selected an Automatic approval strategy, the upgrade status should resolve to Up to date without intervention.

7. After the upgrade status of the subscription is Up to date, select Operators Installed Operators to verify that the cluster service version (CSV) of the installed Operator eventually shows up. The Status should ultimately resolve to InstallSucceeded in the relevant namespace.

   Note

   For the All namespaces…​ installation mode, the status resolves to InstallSucceeded in the

   openshift-operators

   namespace, but the status is Copied if you check in other namespaces.

   If it does not:

   1. Check the logs in any pods in the

      openshift-operators

      project (or other relevant namespace if A specific namespace…​ installation mode was selected) on the Workloads Pods page that are reporting issues to troubleshoot further.

Procedure

1. View the list of Operators available to the cluster from OperatorHub:

   $ oc get packagemanifests -n openshift-marketplace

   Example output

   NAME                               CATALOG               AGE
   3scale-operator                    Red Hat Operators     91m
   advanced-cluster-management        Red Hat Operators     91m
   amq7-cert-manager                  Red Hat Operators     91m
   ...
   couchbase-enterprise-certified     Certified Operators   91m
   crunchy-postgres-operator          Certified Operators   91m
   mongodb-enterprise                 Certified Operators   91m
   ...
   etcd                               Community Operators   91m
   jaeger                             Community Operators   91m
   kubefed                            Community Operators   91m
   ...

   Note the catalog for your desired Operator.

2. Inspect your desired Operator to verify its supported install modes and available channels:

   $ oc describe packagemanifests <operator_name> -n openshift-marketplace

3. An Operator group, defined by an

   OperatorGroup

   object, selects target namespaces in which to generate required RBAC access for all Operators in the same namespace as the Operator group.

   The namespace to which you subscribe the Operator must have an Operator group that matches the install mode of the Operator, either the

   AllNamespaces

   or

   SingleNamespace

   mode. If the Operator you intend to install uses the

   AllNamespaces

   mode, the

   openshift-operators

   namespace already has the appropriate

   global-operators

   Operator group in place.

   However, if the Operator uses the

   SingleNamespace

   mode and you do not already have an appropriate Operator group in place, you must create one.
   Note
   • The web console version of this procedure handles the creation of the

     OperatorGroup

     and

     Subscription

     objects automatically behind the scenes for you when choosing

     SingleNamespace

     mode.
   • You can only have one Operator group per namespace. For more information, see "Operator groups".

   1. Create an

      OperatorGroup

      object YAML file, for example

      operatorgroup.yaml

      :

      Example OperatorGroup object

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: <operatorgroup_name>
        namespace: <namespace>
      spec:
        targetNamespaces:
        - <namespace>

      Warning

      Operator Lifecycle Manager (OLM) creates the following cluster roles for each Operator group:

      • <operatorgroup_name>-admin

      • <operatorgroup_name>-edit

      • <operatorgroup_name>-view

      When you manually create an Operator group, you must specify a unique name that does not conflict with the existing cluster roles or other Operator groups on the cluster.

   2. Create the

      OperatorGroup

      object:

      $ oc apply -f operatorgroup.yaml

4. Create a

   Subscription

   object YAML file to subscribe a namespace to an Operator, for example

   sub.yaml

   :

   Example Subscription object

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: <subscription_name>
     namespace: openshift-operators 

   1

   
   spec:
     channel: <channel_name> 

   2

   
     name: <operator_name> 

   3

   
     source: redhat-operators 

   4

   
     sourceNamespace: openshift-marketplace 

   5

   
     config:
       env: 

   6

   
       - name: ARGS
         value: "-v=10"
       envFrom: 

   7

   
       - secretRef:
           name: license-secret
       volumes: 

   8

   
       - name: <volume_name>
         configMap:
           name: <configmap_name>
       volumeMounts: 

   9

   
       - mountPath: <directory_name>
         name: <volume_name>
       tolerations: 

   10

   
       - operator: "Exists"
       resources: 

   11

   
         requests:
           memory: "64Mi"
           cpu: "250m"
         limits:
           memory: "128Mi"
           cpu: "500m"
       nodeSelector: 

   12

   
         foo: bar

   1

   For default AllNamespaces install mode usage, specify the openshift-operators namespace. Alternatively, you can specify a custom global namespace, if you have created one. Otherwise, specify the relevant single namespace for SingleNamespace install mode usage.

   2

   Name of the channel to subscribe to.

   3

   Name of the Operator to subscribe to.

   4

   Name of the catalog source that provides the Operator.

   5

   Namespace of the catalog source. Use openshift-marketplace for the default OperatorHub catalog sources.

   6

   The env parameter defines a list of Environment Variables that must exist in all containers in the pod created by OLM.

   7

   The envFrom parameter defines a list of sources to populate Environment Variables in the container.

   8

   The volumes parameter defines a list of Volumes that must exist on the pod created by OLM.

   9

   The volumeMounts parameter defines a list of VolumeMounts that must exist in all containers in the pod created by OLM. If a volumeMount references a volume that does not exist, OLM fails to deploy the Operator.

   10

   The tolerations parameter defines a list of Tolerations for the pod created by OLM.

   11

   The resources parameter defines resource constraints for all the containers in the pod created by OLM.

   12

   The nodeSelector parameter defines a NodeSelector for the pod created by OLM.

5. Create the

   Subscription

   object:

   $ oc apply -f sub.yaml

   At this point, OLM is now aware of the selected Operator. A cluster service version (CSV) for the Operator should appear in the target namespace, and APIs provided by the Operator should be available for creation.

Procedure

1. Create a

   Subscription

   object YAML file that subscribes a namespace to an Operator with a specific version by setting the

   startingCSV

   field. Set the

   installPlanApproval

   field to

   Manual

   to prevent the Operator from automatically upgrading if a later version exists in the catalog.

   For example, the following

   sub.yaml

   file can be used to install the Red Hat Quay Operator specifically to version 3.4.0:

   Subscription with a specific starting Operator version

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: quay-operator
     namespace: quay
   spec:
     channel: quay-v3.4
     installPlanApproval: Manual 

   1

   
     name: quay-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     startingCSV: quay-operator.v3.4.0 

   2

   1

   Set the approval strategy to Manual in case your specified version is superseded by a later version in the catalog. This plan prevents an automatic upgrade to a later version and requires manual approval before the starting CSV can complete the installation.

   2

   Set a specific version of an Operator CSV.

2. Create the

   Subscription

   object:

   $ oc apply -f sub.yaml

3. Manually approve the pending install plan to complete the Operator installation.

Procedure

1. Before installing the Operator, create a namespace for the tenant Operator that is separate from the tenant’s namespace. For example, if the tenant’s namespace is

   team1

   , you might create a

   team1-operator

   namespace:

   1. Define a

      Namespace

      resource and save the YAML file, for example,

      team1-operator.yaml

      :

      apiVersion: v1
      kind: Namespace
      metadata:
        name: team1-operator

   2. Create the namespace by running the following command:

      $ oc create -f team1-operator.yaml

2. Create an Operator group for the tenant Operator scoped to the tenant’s namespace, with only that one namespace entry in the

   spec.targetNamespaces

   list:

   1. Define an

      OperatorGroup

      resource and save the YAML file, for example,

      team1-operatorgroup.yaml

      :

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: team1-operatorgroup
        namespace: team1-operator
      spec:
        targetNamespaces:
        - team1 

      1

      1

      Define only the tenant’s namespace in the spec.targetNamespaces list.

   2. Create the Operator group by running the following command:

      $ oc create -f team1-operatorgroup.yaml

Procedure

1. Before installing the Operator, create a namespace for the installation of your desired Operator. This installation namespace will become the custom global namespace:

   1. Define a

      Namespace

      resource and save the YAML file, for example,

      global-operators.yaml

      :

      apiVersion: v1
      kind: Namespace
      metadata:
        name: global-operators

   2. Create the namespace by running the following command:

      $ oc create -f global-operators.yaml

2. Create a custom global Operator group, which is an Operator group that watches all namespaces:

   1. Define an

      OperatorGroup

      resource and save the YAML file, for example,

      global-operatorgroup.yaml

      . Omit both the

      spec.selector

      and

      spec.targetNamespaces

      fields to make it a global Operator group, which selects all namespaces:

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: global-operatorgroup
        namespace: global-operators

      Note

      The

      status.namespaces

      of a created global Operator group contains the empty string (

      ""

      ), which signals to a consuming Operator that it should watch all namespaces.

   2. Create the Operator group by running the following command:

      $ oc create -f global-operatorgroup.yaml

1. Determine a node or set of nodes to target for the pods per your requirements. If available, note an existing label, such as

   node-role.kubernetes.io/app

   , that identifies the node or nodes. Otherwise, add a label, such as

   myoperator

   , by using a compute machine set or editing the node directly. You will use this label in a later step as the node selector on your project.
2. If you want to ensure that only pods with a certain label are allowed to run on the nodes, while steering unrelated workloads to other nodes, add a taint to the node or nodes by using a compute machine set or editing the node directly. Use an effect that ensures that new pods that do not match the taint cannot be scheduled on the nodes. For example, a

   myoperator:NoSchedule

   taint ensures that new pods that do not match the taint are not scheduled onto that node, but existing pods on the node are allowed to remain.
3. Create a project that is configured with a default node selector and, if you added a taint, a matching toleration.

1. Install the Operator as usual.
2. If needed, ensure that your nodes are labeled to properly respond to the affinity.

3. Edit the Operator

   Subscription

   object to add an affinity:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: openshift-custom-metrics-autoscaler-operator
     namespace: openshift-keda
   spec:
     name: my-package
     source: my-operators
     sourceNamespace: operator-registries
     config:
       affinity: 

   1

   
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: kubernetes.io/hostname
                 operator: In
                 values:
                 - ip-10-0-185-229.ec2.internal
   #...

   1

   Add a nodeAffinity, podAffinity, or podAntiAffinity. See the Additional resources section that follows for information about creating the affinity.

Procedure

1. In the Administrator perspective of the web console, navigate to Operators Installed Operators.
2. Click the name of the Operator you want to change the update channel for.
3. Click the Subscription tab.
4. Click the name of the update channel under Channel.
5. Click the newer update channel that you want to change to, then click Save.

6. For subscriptions with an Automatic approval strategy, the update begins automatically. Navigate back to the Operators Installed Operators page to monitor the progress of the update. When complete, the status changes to Succeeded and Up to date.

   For subscriptions with a Manual approval strategy, you can manually approve the update from the Subscription tab.

Procedure

1. In the Administrator perspective of the OpenShift Container Platform web console, navigate to Operators Installed Operators.
2. Operators that have a pending update display a status with Upgrade available. Click the name of the Operator you want to update.
3. Click the Subscription tab. Any update requiring approval are displayed next to Upgrade Status. For example, it might display 1 requires approval.
4. Click 1 requires approval, then click Preview Install Plan.
5. Review the resources that are listed as available for update. When satisfied, click Approve.
6. Navigate back to the Operators Installed Operators page to monitor the progress of the update. When complete, the status changes to Succeeded and Up to date.

Procedure

1. Navigate to the Operators Installed Operators page.
2. Scroll or enter a keyword into the Filter by name field to find the Operator that you want to remove. Then, click on it.

3. On the right side of the Operator Details page, select Uninstall Operator from the Actions list.

   An Uninstall Operator? dialog box is displayed.

4. Select Uninstall to remove the Operator, Operator deployments, and pods. Following this action, the Operator stops running and no longer receives updates.

   Note

   This action does not remove resources managed by the Operator, including custom resource definitions (CRDs) and custom resources (CRs). Dashboards and navigation items enabled by the web console and off-cluster resources that continue to run might need manual clean up. To remove these after uninstalling the Operator, you might need to manually delete the Operator CRDs.

Procedure

1. Ensure the latest version of the subscribed operator (for example,

   serverless-operator

   ) is identified in the

   currentCSV

   field.

   $ oc get subscription.operators.coreos.com serverless-operator -n openshift-serverless -o yaml | grep currentCSV

   Example output

     currentCSV: serverless-operator.v1.28.0

2. Delete the subscription (for example,

   serverless-operator

   ):

   $ oc delete subscription.operators.coreos.com serverless-operator -n openshift-serverless

   Example output

   subscription.operators.coreos.com "serverless-operator" deleted

3. Delete the CSV for the Operator in the target namespace using the

   currentCSV

   value from the previous step:

   $ oc delete clusterserviceversion serverless-operator.v1.28.0 -n openshift-serverless

   Example output

   clusterserviceversion.operators.coreos.com "serverless-operator.v1.28.0" deleted

Procedure

1. Get the names of the

   Subscription

   and

   ClusterServiceVersion

   objects from the namespace where the Operator is installed:

   $ oc get sub,csv -n <namespace>

   Example output

   NAME                                                       PACKAGE                  SOURCE             CHANNEL
   subscription.operators.coreos.com/elasticsearch-operator   elasticsearch-operator   redhat-operators   5.0
   
   NAME                                                                         DISPLAY                            VERSION    REPLACES   PHASE
   clusterserviceversion.operators.coreos.com/elasticsearch-operator.5.0.0-65   OpenShift Elasticsearch Operator   5.0.0-65              Succeeded

2. Delete the subscription:

   $ oc delete subscription <subscription_name> -n <namespace>

3. Delete the cluster service version:

   $ oc delete csv <csv_name> -n <namespace>

4. Get the names of any failing jobs and related config maps in the

   openshift-marketplace

   namespace:

   $ oc get job,configmap -n openshift-marketplace

   Example output

   NAME                                                                        COMPLETIONS   DURATION   AGE
   job.batch/1de9443b6324e629ddf31fed0a853a121275806170e34c926d69e53a7fcbccb   1/1           26s        9m30s
   
   NAME                                                                        DATA   AGE
   configmap/1de9443b6324e629ddf31fed0a853a121275806170e34c926d69e53a7fcbccb   3      9m30s

5. Delete the job:

   $ oc delete job <job_name> -n openshift-marketplace

   This ensures pods that try to pull the inaccessible image are not recreated.

6. Delete the config map:

   $ oc delete configmap <configmap_name> -n openshift-marketplace

7. Reinstall the Operator using OperatorHub in the web console.

Procedure

1. Navigate in the web console to the Operators OperatorHub page.
2. Select the Operator and click Install.

3. On the Install Operator page, modify the

   Subscription

   object to include one or more of the following environment variables in the

   spec

   section:

   • HTTP_PROXY

   • HTTPS_PROXY

   • NO_PROXY

   For example:

   Subscription object with proxy setting overrides

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: etcd-config-test
     namespace: openshift-operators
   spec:
     config:
       env:
       - name: HTTP_PROXY
         value: test_http
       - name: HTTPS_PROXY
         value: test_https
       - name: NO_PROXY
         value: test
     channel: clusterwide-alpha
     installPlanApproval: Automatic
     name: etcd
     source: community-operators
     sourceNamespace: openshift-marketplace
     startingCSV: etcdoperator.v0.9.4-clusterwide

   Note

   These environment variables can also be unset using an empty value to remove any previously set cluster-wide or custom proxy settings.

   OLM handles these environment variables as a unit; if at least one of them is set, all three are considered overridden and the cluster-wide defaults are not used for the deployments of the subscribed Operator.

4. Click Install to make the Operator available to the selected namespaces.

5. After the CSV for the Operator appears in the relevant namespace, you can verify that custom proxy environment variables are set in the deployment. For example, using the CLI:

   $ oc get deployment -n openshift-operators \
       etcd-operator -o yaml \
       | grep -i "PROXY" -A 2

   Example output

           - name: HTTP_PROXY
             value: test_http
           - name: HTTPS_PROXY
             value: test_https
           - name: NO_PROXY
             value: test
           image: quay.io/coreos/etcd-operator@sha256:66a37fd61a06a43969854ee6d3e21088a98b93838e284a6086b13917f96b0d9c
   ...

Procedure

1. Create an empty config map in the namespace where the subscription for your Operator exists and include the following label:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: trusted-ca 

   1

   
     labels:
       config.openshift.io/inject-trusted-cabundle: "true" 

   2

   1

   Name of the config map.

   2

   Requests the Cluster Network Operator to inject the merged bundle.

   After creating this config map, it is immediately populated with the certificate contents of the merged bundle.

2. Update your the

   Subscription

   object to include a

   spec.config

   section that mounts the

   trusted-ca

   config map as a volume to each container within a pod that requires a custom CA:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: my-operator
   spec:
     package: etcd
     channel: alpha
     config: 

   1

   
       selector:
         matchLabels:
           <labels_for_pods> 

   2

   
       volumes: 

   3

   
       - name: trusted-ca
         configMap:
           name: trusted-ca
           items:
             - key: ca-bundle.crt 

   4

   
               path: tls-ca-bundle.pem 

   5

   
       volumeMounts: 

   6

   
       - name: trusted-ca
         mountPath: /etc/pki/ca-trust/extracted/pem
         readOnly: true

   1

   Add a config section if it does not exist.

   2

   Specify labels to match pods that are owned by the Operator.

   3

   Create a trusted-ca volume.

   4

   ca-bundle.crt is required as the config map key.

   5

   tls-ca-bundle.pem is required as the config map path.

   6

   Create a trusted-ca volume mount.

   Note

   Deployments of an Operator can fail to validate the authority and display a

   x509 certificate signed by unknown authority

   error. This error can occur even after injecting a custom CA when using the subscription of an Operator. In this case, you can set the

   mountPath

   as

   /etc/ssl/certs

   for trusted-ca by using the subscription of an Operator.

Procedure

1. List Operator subscriptions:

   $ oc get subs -n <operator_namespace>

2. Use the

   oc describe

   command to inspect a

   Subscription

   resource:

   $ oc describe sub <subscription_name> -n <operator_namespace>

3. In the command output, find the

   Conditions

   section for the status of Operator subscription condition types. In the following example, the

   CatalogSourcesUnhealthy

   condition type has a status of

   false

   because all available catalog sources are healthy:

   Example output

   Name:         cluster-logging
   Namespace:    openshift-logging
   Labels:       operators.coreos.com/cluster-logging.openshift-logging=
   Annotations:  <none>
   API Version:  operators.coreos.com/v1alpha1
   Kind:         Subscription
   # ...
   Conditions:
      Last Transition Time:  2019-07-29T13:42:57Z
      Message:               all available catalogsources are healthy
      Reason:                AllCatalogSourcesHealthy
      Status:                False
      Type:                  CatalogSourcesUnhealthy
   # ...

Procedure

1. List the catalog sources in a namespace. For example, you can check the

   openshift-marketplace

   namespace, which is used for cluster-wide catalog sources:

   $ oc get catalogsources -n openshift-marketplace

   Example output

   NAME                  DISPLAY               TYPE   PUBLISHER   AGE
   certified-operators   Certified Operators   grpc   Red Hat     55m
   community-operators   Community Operators   grpc   Red Hat     55m
   example-catalog       Example Catalog       grpc   Example Org 2m25s
   redhat-marketplace    Red Hat Marketplace   grpc   Red Hat     55m
   redhat-operators      Red Hat Operators     grpc   Red Hat     55m

2. Use the

   oc describe

   command to get more details and status about a catalog source:

   $ oc describe catalogsource example-catalog -n openshift-marketplace

   Example output

   Name:         example-catalog
   Namespace:    openshift-marketplace
   Labels:       <none>
   Annotations:  operatorframework.io/managed-by: marketplace-operator
                 target.workload.openshift.io/management: {"effect": "PreferredDuringScheduling"}
   API Version:  operators.coreos.com/v1alpha1
   Kind:         CatalogSource
   # ...
   Status:
     Connection State:
       Address:              example-catalog.openshift-marketplace.svc:50051
       Last Connect:         2021-09-09T17:07:35Z
       Last Observed State:  TRANSIENT_FAILURE
     Registry Service:
       Created At:         2021-09-09T17:05:45Z
       Port:               50051
       Protocol:           grpc
       Service Name:       example-catalog
       Service Namespace:  openshift-marketplace
   # ...

   In the preceding example output, the last observed state is

   TRANSIENT_FAILURE

   . This state indicates that there is a problem establishing a connection for the catalog source.

3. List the pods in the namespace where your catalog source was created:

   $ oc get pods -n openshift-marketplace

   Example output

   NAME                                    READY   STATUS             RESTARTS   AGE
   certified-operators-cv9nn               1/1     Running            0          36m
   community-operators-6v8lp               1/1     Running            0          36m
   marketplace-operator-86bfc75f9b-jkgbc   1/1     Running            0          42m
   example-catalog-bwt8z                   0/1     ImagePullBackOff   0          3m55s
   redhat-marketplace-57p8c                1/1     Running            0          36m
   redhat-operators-smxx8                  1/1     Running            0          36m

   When a catalog source is created in a namespace, a pod for the catalog source is created in that namespace. In the preceding example output, the status for the

   example-catalog-bwt8z

   pod is

   ImagePullBackOff

   . This status indicates that there is an issue pulling the catalog source’s index image.

4. Use the

   oc describe

   command to inspect a pod for more detailed information:

   $ oc describe pod example-catalog-bwt8z -n openshift-marketplace

   Example output

   Name:         example-catalog-bwt8z
   Namespace:    openshift-marketplace
   Priority:     0
   Node:         ci-ln-jyryyg2-f76d1-ggdbq-worker-b-vsxjd/10.0.128.2
   ...
   Events:
     Type     Reason          Age                From               Message
     ----     ------          ----               ----               -------
     Normal   Scheduled       48s                default-scheduler  Successfully assigned openshift-marketplace/example-catalog-bwt8z to ci-ln-jyryyf2-f76d1-fgdbq-worker-b-vsxjd
     Normal   AddedInterface  47s                multus             Add eth0 [10.131.0.40/23] from openshift-sdn
     Normal   BackOff         20s (x2 over 46s)  kubelet            Back-off pulling image "quay.io/example-org/example-catalog:v1"
     Warning  Failed          20s (x2 over 46s)  kubelet            Error: ImagePullBackOff
     Normal   Pulling         8s (x3 over 47s)   kubelet            Pulling image "quay.io/example-org/example-catalog:v1"
     Warning  Failed          8s (x3 over 47s)   kubelet            Failed to pull image "quay.io/example-org/example-catalog:v1": rpc error: code = Unknown desc = reading manifest v1 in quay.io/example-org/example-catalog: unauthorized: access to the requested resource is not authorized
     Warning  Failed          8s (x3 over 47s)   kubelet            Error: ErrImagePull

   In the preceding example output, the error messages indicate that the catalog source’s index image is failing to pull successfully because of an authorization issue. For example, the index image might be stored in a registry that requires login credentials.

Procedure

1. Edit the

   OperatorCondition

   object for the Operator:

   $ oc edit operatorcondition <name>

2. Add a

   Spec.Overrides

   array to the object:

   Example Operator condition override

   apiVersion: operators.coreos.com/v1
   kind: OperatorCondition
   metadata:
     name: my-operator
     namespace: operators
   spec:
     overrides:
     - type: Upgradeable 

   1

   
       status: "True"
       reason: "upgradeIsSafe"
       message: "This is a known issue with the Operator where it always reports that it cannot be upgraded."
     conditions:
     - type: Upgradeable
       status: "False"
       reason: "migration"
       message: "The operator is performing a migration."
       lastTransitionTime: "2020-08-24T23:15:55Z"

   1

   Allows the cluster administrator to change the upgrade readiness to True.

Procedure

1. Create a new namespace:

   $ cat <<EOF | oc create -f -
   apiVersion: v1
   kind: Namespace
   metadata:
     name: scoped
   EOF

2. Allocate permissions that you want the Operator(s) to be confined to. This involves creating a new service account, relevant role(s), and role binding(s).

   $ cat <<EOF | oc create -f -
   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: scoped
     namespace: scoped
   EOF

   The following example grants the service account permissions to do anything in the designated namespace for simplicity. In a production environment, you should create a more fine-grained set of permissions:

   $ cat <<EOF | oc create -f -
   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: scoped
     namespace: scoped
   rules:
   - apiGroups: ["*"]
     resources: ["*"]
     verbs: ["*"]
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: scoped-bindings
     namespace: scoped
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: Role
     name: scoped
   subjects:
   - kind: ServiceAccount
     name: scoped
     namespace: scoped
   EOF

3. Create an

   OperatorGroup

   object in the designated namespace. This Operator group targets the designated namespace to ensure that its tenancy is confined to it.

   In addition, Operator groups allow a user to specify a service account. Specify the service account created in the previous step:

   $ cat <<EOF | oc create -f -
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: scoped
     namespace: scoped
   spec:
     serviceAccountName: scoped
     targetNamespaces:
     - scoped
   EOF

   Any Operator installed in the designated namespace is tied to this Operator group and therefore to the service account specified.

   Warning

   Operator Lifecycle Manager (OLM) creates the following cluster roles for each Operator group:

   • <operatorgroup_name>-admin

   • <operatorgroup_name>-edit

   • <operatorgroup_name>-view

   When you manually create an Operator group, you must specify a unique name that does not conflict with the existing cluster roles or other Operator groups on the cluster.

4. Create a

   Subscription

   object in the designated namespace to install an Operator:

   $ cat <<EOF | oc create -f -
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: etcd
     namespace: scoped
   spec:
     channel: singlenamespace-alpha
     name: etcd
     source: <catalog_source_name> 

   1

   
     sourceNamespace: <catalog_source_namespace> 

   2

   
   EOF

   1

   Specify a catalog source that already exists in the designated namespace or one that is in the global catalog namespace.

   2

   Specify a namespace where the catalog source was created.

   Any Operator tied to this Operator group is confined to the permissions granted to the specified service account. If the Operator requests permissions that are outside the scope of the service account, the installation fails with relevant errors.

1. Disable all of the default global catalogs.
2. Enable custom, curated catalogs in the same namespace where the relevant Operator groups have been preinstalled.

Procedure

1. Review the

   Subscription

   object. Its status has an object reference

   installPlanRef

   that points to the

   InstallPlan

   object that attempted to create the necessary

   [Cluster]Role[Binding]

   object(s) for the Operator:

   apiVersion: operators.coreos.com/v1
   kind: Subscription
   metadata:
     name: etcd
     namespace: scoped
   status:
     installPlanRef:
       apiVersion: operators.coreos.com/v1
       kind: InstallPlan
       name: install-4plp8
       namespace: scoped
       resourceVersion: "117359"
       uid: 2c1df80e-afea-11e9-bce3-5254009c9c23

2. Check the status of the

   InstallPlan

   object for any errors:

   apiVersion: operators.coreos.com/v1
   kind: InstallPlan
   status:
     conditions:
     - lastTransitionTime: "2019-07-26T21:13:10Z"
       lastUpdateTime: "2019-07-26T21:13:10Z"
       message: 'error creating clusterrole etcdoperator.v0.9.4-clusterwide-dsfx4: clusterroles.rbac.authorization.k8s.io
         is forbidden: User "system:serviceaccount:scoped:scoped" cannot create resource
         "clusterroles" in API group "rbac.authorization.k8s.io" at the cluster scope'
       reason: InstallComponentFailed
       status: "False"
       type: Installed
     phase: Failed

   The error message tells you:

   • The type of resource it failed to create, including the API group of the resource. In this case, it was

     clusterroles

     in the

     rbac.authorization.k8s.io

     group.
   • The name of the resource.
   • The type of error:

     is forbidden

     tells you that the user does not have enough permission to do the operation.
   • The name of the user who attempted to create or update the resource. In this case, it refers to the service account specified in the Operator group.

   • The scope of the operation:

     cluster scope

     or not.

     The user can add the missing permission to the service account and then iterate.

     Note

     Operator Lifecycle Manager (OLM) does not currently provide the complete list of errors on the first try.

Procedure

1. Create a

   CatalogSource

   object that references your index image.

   1. Modify the following to your specifications and save it as a

      catalogSource.yaml

      file:

      apiVersion: operators.coreos.com/v1alpha1
      kind: CatalogSource
      metadata:
        name: my-operator-catalog
        namespace: openshift-marketplace 

      1

      
        annotations:
          olm.catalogImageTemplate: 

      2

      
            "<registry>/<namespace>/<index_image_name>:v{kube_major_version}.{kube_minor_version}.{kube_patch_version}"
      spec:
        sourceType: grpc
        grpcPodConfig:
          securityContextConfig: <security_mode> 

      3

      
        image: <registry>/<namespace>/<index_image_name>:<tag> 

      4

      
        displayName: My Operator Catalog
        publisher: <publisher_name> 

      5

      
        updateStrategy:
          registryPoll: 

      6

      
            interval: 30m

      1

      If you want the catalog source to be available globally to users in all namespaces, specify the openshift-marketplace namespace. Otherwise, you can specify a different namespace for the catalog to be scoped and available only for that namespace.

      2

      Optional: Set the olm.catalogImageTemplate annotation to your index image name and use one or more of the Kubernetes cluster version variables as shown when constructing the template for the image tag.

      3

      Specify the value of legacy or restricted. If the field is not set, the default value is legacy. In a future OpenShift Container Platform release, it is planned that the default value will be restricted.

      Note

      If your catalog cannot run with

      restricted

      permissions, it is recommended that you manually set this field to

      legacy

      .

      4

      Specify your index image. If you specify a tag after the image name, for example :v4.12, the catalog source pod uses an image pull policy of Always, meaning the pod always pulls the image prior to starting the container. If you specify a digest, for example @sha256:<id>, the image pull policy is IfNotPresent, meaning the pod pulls the image only if it does not already exist on the node.

      5

      Specify your name or an organization name publishing the catalog.

      6

      Catalog sources can automatically check for new versions to keep up to date.

   2. Use the file to create the

      CatalogSource

      object:

      $ oc apply -f catalogSource.yaml

2. Verify the following resources are created successfully.

   1. Check the pods:

      $ oc get pods -n openshift-marketplace

      Example output

      NAME                                    READY   STATUS    RESTARTS  AGE
      my-operator-catalog-6njx6               1/1     Running   0         28s
      marketplace-operator-d9f549946-96sgr    1/1     Running   0         26h

   2. Check the catalog source:

      $ oc get catalogsource -n openshift-marketplace

      Example output

      NAME                  DISPLAY               TYPE PUBLISHER  AGE
      my-operator-catalog   My Operator Catalog   grpc            5s

   3. Check the package manifest:

      $ oc get packagemanifest -n openshift-marketplace

      Example output

      NAME                          CATALOG               AGE
      jaeger-product                My Operator Catalog   93s

Procedure

1. Create a secret for each required private registry.

   1. Log in to the private registry to create or update your registry credentials file:

      $ podman login <registry>:<port>

      Note

      The file path of your registry credentials can be different depending on the container tool used to log in to the registry. For the

      podman

      CLI, the default location is

      ${XDG_RUNTIME_DIR}/containers/auth.json

      . For the

      docker

      CLI, the default location is

      /root/.docker/config.json

      .

   2. It is recommended to include credentials for only one registry per secret, and manage credentials for multiple registries in separate secrets. Multiple secrets can be included in a

      CatalogSource

      object in later steps, and OpenShift Container Platform will merge the secrets into a single virtual credentials file for use during an image pull.

      A registry credentials file can, by default, store details for more than one registry or for multiple repositories in one registry. Verify the current contents of your file. For example:

      File storing credentials for multiple registries

      {
          "auths": {
              "registry.redhat.io": {
                  "auth": "FrNHNydQXdzclNqdg=="
              },
              "quay.io": {
                  "auth": "fegdsRib21iMQ=="
              },
              "https://quay.io/my-namespace/my-user/my-image": {
                  "auth": "eWfjwsDdfsa221=="
              },
              "https://quay.io/my-namespace/my-user": {
                  "auth": "feFweDdscw34rR=="
              },
              "https://quay.io/my-namespace": {
                  "auth": "frwEews4fescyq=="
              }
          }
      }

      Because this file is used to create secrets in later steps, ensure that you are storing details for only one registry per file. This can be accomplished by using either of the following methods:

      • Use the

        podman logout <registry>

        command to remove credentials for additional registries until only the one registry you want remains.

      • Edit your registry credentials file and separate the registry details to be stored in multiple files. For example:

        File storing credentials for one registry

        {
                "auths": {
                        "registry.redhat.io": {
                                "auth": "FrNHNydQXdzclNqdg=="
                        }
                }
        }

        File storing credentials for another registry

        {
                "auths": {
                        "quay.io": {
                                "auth": "Xd2lhdsbnRib21iMQ=="
                        }
                }
        }

   3. Create a secret in the

      openshift-marketplace

      namespace that contains the authentication credentials for a private registry:

      $ oc create secret generic <secret_name> \
          -n openshift-marketplace \
          --from-file=.dockerconfigjson=<path/to/registry/credentials> \
          --type=kubernetes.io/dockerconfigjson

      Repeat this step to create additional secrets for any other required private registries, updating the

      --from-file

      flag to specify another registry credentials file path.

2. Create or update an existing

   CatalogSource

   object to reference one or more secrets:

   apiVersion: operators.coreos.com/v1alpha1
   kind: CatalogSource
   metadata:
     name: my-operator-catalog
     namespace: openshift-marketplace
   spec:
     sourceType: grpc
     secrets: 

   1

   
     - "<secret_name_1>"
     - "<secret_name_2>"
     grpcPodConfig:
       securityContextConfig: <security_mode> 

   2

   
     image: <registry>:<port>/<namespace>/<image>:<tag>
     displayName: My Operator Catalog
     publisher: <publisher_name>
     updateStrategy:
       registryPoll:
         interval: 30m

   1

   Add a spec.secrets section and specify any required secrets.

   2

   Specify the value of legacy or restricted. If the field is not set, the default value is legacy. In a future OpenShift Container Platform release, it is planned that the default value will be restricted.

   Note

   If your catalog cannot run with

   restricted

   permissions, it is recommended that you manually set this field to

   legacy

   .

3. If any Operator or Operand images that are referenced by a subscribed Operator require access to a private registry, you can either provide access to all namespaces in the cluster, or individual target tenant namespaces.

   • To provide access to all namespaces in the cluster, add authentication details to the global cluster pull secret in the

     openshift-config

     namespace.
     Warning

     Cluster resources must adjust to the new global pull secret, which can temporarily limit the usability of the cluster.

     1. Extract the

        .dockerconfigjson

        file from the global pull secret:

        $ oc extract secret/pull-secret -n openshift-config --confirm

     2. Update the

        .dockerconfigjson

        file with your authentication credentials for the required private registry or registries and save it as a new file:

        $ cat .dockerconfigjson | \
            jq --compact-output '.auths["<registry>:<port>/<namespace>/"] |= . + {"auth":"<token>"}' \

        1

        
            > new_dockerconfigjson

        1

        Replace <registry>:<port>/<namespace> with the private registry details and <token> with your authentication credentials.

     3. Update the global pull secret with the new file:

        $ oc set data secret/pull-secret -n openshift-config \
            --from-file=.dockerconfigjson=new_dockerconfigjson

   • To update an individual namespace, add a pull secret to the service account for the Operator that requires access in the target tenant namespace.

     1. Recreate the secret that you created for the

        openshift-marketplace

        in the tenant namespace:

        $ oc create secret generic <secret_name> \
            -n <tenant_namespace> \
            --from-file=.dockerconfigjson=<path/to/registry/credentials> \
            --type=kubernetes.io/dockerconfigjson

     2. Verify the name of the service account for the Operator by searching the tenant namespace:

        $ oc get sa -n <tenant_namespace> 

        1

        1

        If the Operator was installed in an individual namespace, search that namespace. If the Operator was installed for all namespaces, search the openshift-operators namespace.

        Example output

        NAME            SECRETS   AGE
        builder         2         6m1s
        default         2         6m1s
        deployer        2         6m1s
        etcd-operator   2         5m18s 

        1

        1

        Service account for an installed etcd Operator.

     3. Link the secret to the service account for the Operator:

        $ oc secrets link <operator_sa> \
            -n <tenant_namespace> \
             <secret_name> \
            --for=pull

Procedure

1. In the Administrator perspective of the web console, navigate to Administration Cluster Settings.
2. Click the Configuration tab, and then click OperatorHub.
3. Click the Sources tab.
4. Select the Options menu for the catalog that you want to remove, and then click Delete CatalogSource.

Procedure

1. Create a

   CatalogSource

   object that references your index image. If you used the

   oc adm catalog mirror

   command to mirror your catalog to a target registry, you can use the generated

   catalogSource.yaml

   file in your manifests directory as a starting point.

   1. Modify the following to your specifications and save it as a

      catalogSource.yaml

      file:

      apiVersion: operators.coreos.com/v1alpha1
      kind: CatalogSource
      metadata:
        name: my-operator-catalog 

      1

      
        namespace: openshift-marketplace 

      2

      
      spec:
        sourceType: grpc
        grpcPodConfig:
          securityContextConfig: <security_mode> 

      3

      
        image: <registry>/<namespace>/redhat-operator-index:v4.12 

      4

      
        displayName: My Operator Catalog
        publisher: <publisher_name> 

      5

      
        updateStrategy:
          registryPoll: 

      6

      
            interval: 30m

      1

      If you mirrored content to local files before uploading to a registry, remove any backslash (/) characters from the metadata.name field to avoid an "invalid resource name" error when you create the object.

      2

      If you want the catalog source to be available globally to users in all namespaces, specify the openshift-marketplace namespace. Otherwise, you can specify a different namespace for the catalog to be scoped and available only for that namespace.

      3

      Specify the value of legacy or restricted. If the field is not set, the default value is legacy. In a future OpenShift Container Platform release, it is planned that the default value will be restricted.

      Note

      If your catalog cannot run with

      restricted

      permissions, it is recommended that you manually set this field to

      legacy

      .

      4

      Specify your index image. If you specify a tag after the image name, for example :v4.12, the catalog source pod uses an image pull policy of Always, meaning the pod always pulls the image prior to starting the container. If you specify a digest, for example @sha256:<id>, the image pull policy is IfNotPresent, meaning the pod pulls the image only if it does not already exist on the node.

      5

      Specify your name or an organization name publishing the catalog.

      6

      Catalog sources can automatically check for new versions to keep up to date.

   2. Use the file to create the

      CatalogSource

      object:

      $ oc apply -f catalogSource.yaml

2. Verify the following resources are created successfully.

   1. Check the pods:

      $ oc get pods -n openshift-marketplace

      Example output

      NAME                                    READY   STATUS    RESTARTS  AGE
      my-operator-catalog-6njx6               1/1     Running   0         28s
      marketplace-operator-d9f549946-96sgr    1/1     Running   0         26h

   2. Check the catalog source:

      $ oc get catalogsource -n openshift-marketplace

      Example output

      NAME                  DISPLAY               TYPE PUBLISHER  AGE
      my-operator-catalog   My Operator Catalog   grpc            5s

   3. Check the package manifest:

      $ oc get packagemanifest -n openshift-marketplace

      Example output

      NAME                          CATALOG               AGE
      jaeger-product                My Operator Catalog   93s

Procedure

1. Choose a platform Operator from the supported set of OLM-based Operators. For the list of this set and details on current limitations, see "Technology Preview restrictions for platform Operators".
2. Select a cluster installation method and follow the instructions through creating an

   install-config.yaml

   file. For more details on preparing for a cluster installation, see "Selecting a cluster installation method and preparing it for users".

3. After you have created the

   install-config.yaml

   file and completed any modifications to it, change to the directory that contains the installation program and create the manifests:

   $ ./openshift-install create manifests --dir <installation_directory> 

   1

   1

   For <installation_directory>, specify the name of the directory that contains the install-config.yaml file for your cluster.

4. Create a

   FeatureGate

   object YAML file in the

   <installation_directory>/manifests/

   directory that enables the

   TechPreviewNoUpgrade

   feature set, for example a

   feature-gate.yaml

   file:

   Example feature-gate.yaml file

   apiVersion: config.openshift.io/v1
   kind: FeatureGate
   metadata:
     annotations:
       include.release.openshift.io/self-managed-high-availability: "true"
       include.release.openshift.io/single-node-developer: "true"
       release.openshift.io/create-only: "true"
     name: cluster
   spec:
     featureSet: TechPreviewNoUpgrade 

   1

   1

   Enable the TechPreviewNoUpgrade feature set.

5. Create a

   PlatformOperator

   object YAML file for your chosen platform Operator in the

   <installation_directory>/manifests/

   directory, for example a

   service-mesh-po.yaml

   file for the Red Hat OpenShift Service Mesh Operator:

   Example service-mesh-po.yaml file

   apiVersion: platform.openshift.io/v1alpha1
   kind: PlatformOperator
   metadata:
     name: service-mesh-po
   spec:
     package:
       name: servicemeshoperator

6. When you are ready to complete the cluster install, refer to your chosen installation method and continue through running the

   openshift-install create cluster

   command.

   During cluster creation, your provided manifests are used to enable the

   TechPreviewNoUpgrade

   feature set and install your chosen platform Operator.
   Important

   Failure of the platform Operator to successfully install will block the cluster installation process.

Verification

1. Check the status of the

   service-mesh-po

   platform Operator by running the following command:

   $ oc get platformoperator service-mesh-po -o yaml

   Example output

   ...
   status:
     activeBundleDeployment:
       name: service-mesh-po
     conditions:
     - lastTransitionTime: "2022-10-24T17:24:40Z"
       message: Successfully applied the service-mesh-po BundleDeployment resource
       reason: InstallSuccessful
       status: "True" 

   1

   
       type: Installed

   1

   Wait until the Installed status condition reports True.

2. Verify that the

   platform-operators-aggregated

   cluster Operator is reporting an

   Available=True

   status:

   $ oc get clusteroperator platform-operators-aggregated -o yaml

   Example output

   ...
   status:
     conditions:
     - lastTransitionTime: "2022-10-24T17:43:26Z"
       message: All platform operators are in a successful state
       reason: AsExpected
       status: "False"
       type: Progressing
     - lastTransitionTime: "2022-10-24T17:43:26Z"
       status: "False"
       type: Degraded
     - lastTransitionTime: "2022-10-24T17:43:26Z"
       message: All platform operators are in a successful state
       reason: AsExpected
       status: "True"
       type: Available

Procedure

1. Choose a platform Operator from the supported set of OLM-based Operators. For the list of this set and details on current limitations, see "Technology Preview restrictions for platform Operators".

2. Create a

   PlatformOperator

   object YAML file for your chosen platform Operator, for example a

   service-mesh-po.yaml

   file for the Red Hat OpenShift Service Mesh Operator:

   Example sevice-mesh-po.yaml file

   apiVersion: platform.openshift.io/v1alpha1
   kind: PlatformOperator
   metadata:
     name: service-mesh-po
   spec:
     package:
       name: servicemeshoperator

3. Create the

   PlatformOperator

   object by running the following command:

   $ oc apply -f service-mesh-po.yaml

   Note

   If your cluster does not have the

   TechPreviewNoUpgrade

   feature set enabled, the object creation fails with the following message:

   error: resource mapping not found for name: "service-mesh-po" namespace: "" from "service-mesh-po.yaml": no matches for kind "PlatformOperator" in version "platform.openshift.io/v1alpha1"
   ensure CRDs are installed first

Verification

1. Check the status of the

   service-mesh-po

   platform Operator by running the following command:

   $ oc get platformoperator service-mesh-po -o yaml

   Example output

   ...
   status:
     activeBundleDeployment:
       name: service-mesh-po
     conditions:
     - lastTransitionTime: "2022-10-24T17:24:40Z"
       message: Successfully applied the service-mesh-po BundleDeployment resource
       reason: InstallSuccessful
       status: "True" 

   1

   
       type: Installed

   1

   Wait until the Installed status condition reports True.

2. Verify that the

   platform-operators-aggregated

   cluster Operator is reporting an

   Available=True

   status:

   $ oc get clusteroperator platform-operators-aggregated -o yaml

   Example output

   ...
   status:
     conditions:
     - lastTransitionTime: "2022-10-24T17:43:26Z"
       message: All platform operators are in a successful state
       reason: AsExpected
       status: "False"
       type: Progressing
     - lastTransitionTime: "2022-10-24T17:43:26Z"
       status: "False"
       type: Degraded
     - lastTransitionTime: "2022-10-24T17:43:26Z"
       message: All platform operators are in a successful state
       reason: AsExpected
       status: "True"
       type: Available