Chapter 4. Administrator tasks

Procedure

1. Navigate in the web console to the Operators OperatorHub page.

2. Scroll or type a keyword into the Filter by keyword box to find the Operator you want. For example, type advanced to find the Advanced Cluster Management for Kubernetes Operator.

   You can also filter options by Infrastructure Features. For example, select Disconnected if you want to see Operators that work in disconnected environments, also known as restricted network environments.

3. Select the Operator to display additional information.

   Note

   Choosing a Community Operator warns that Red Hat does not certify Community Operators; you must acknowledge the warning before continuing.

4. Read the information about the Operator and click Install.

5. On the Install Operator page, configure your Operator installation:

   1. If you want to install a specific version of an Operator, select an Update channel and Version from the lists. You can browse the various versions of an Operator across any channels it might have, view the metadata for that channel and version, and select the exact version you want to install.

      Note

      The version selection defaults to the latest version for the channel selected. If the latest version for the channel is selected, the Automatic approval strategy is enabled by default. Otherwise, Manual approval is required when not installing the latest version for the selected channel.

      Installing an Operator with Manual approval causes all Operators installed within the namespace to function with the Manual approval strategy and all Operators are updated together. If you want to update Operators independently, install Operators into separate namespaces.

   2. Confirm the installation mode for the Operator:

      • All namespaces on the cluster (default) installs the Operator in the default openshift-operators namespace to watch and be made available to all namespaces in the cluster. This option is not always available.
      • A specific namespace on the cluster allows you to choose a specific, single namespace in which to install the Operator. The Operator will only watch and be made available for use in this single namespace.

   3. For clusters on cloud providers with token authentication enabled:

      • If the cluster uses AWS STS (STS Mode in the web console), enter the Amazon Resource Name (ARN) of the AWS IAM role of your service account in the role ARN field. To create the role’s ARN, follow the procedure described in Preparing AWS account.
      • If the cluster uses Microsoft Entra Workload ID (Workload Identity / Federated Identity Mode in the web console), add the client ID, tenant ID, and subscription ID in the appropriate field.

   4. For Update approval, select either the Automatic or Manual approval strategy.

      Important

      If the web console shows that the cluster uses AWS STS or Microsoft Entra Workload ID, you must set Update approval to Manual.

      Subscriptions with automatic update approvals are not recommended because there might be permission changes to make prior to updating. Subscriptions with manual update approvals ensure that administrators have the opportunity to verify the permissions of the later version and take any necessary steps prior to update.

6. Click Install to make the Operator available to the selected namespaces on this OpenShift Container Platform cluster:

   1. If you selected a Manual approval strategy, the upgrade status of the subscription remains Upgrading until you review and approve the install plan.

      After approving on the Install Plan page, the subscription upgrade status moves to Up to date.

   2. If you selected an Automatic approval strategy, the upgrade status should resolve to Up to date without intervention.

Procedure

1. View the list of Operators available to the cluster from OperatorHub:

   $ oc get packagemanifests -n openshift-marketplace

   Example 4.1. Example output

   NAME                               CATALOG               AGE
   3scale-operator                    Red Hat Operators     91m
   advanced-cluster-management        Red Hat Operators     91m
   amq7-cert-manager                  Red Hat Operators     91m
   # ...
   couchbase-enterprise-certified     Certified Operators   91m
   crunchy-postgres-operator          Certified Operators   91m
   mongodb-enterprise                 Certified Operators   91m
   # ...
   etcd                               Community Operators   91m
   jaeger                             Community Operators   91m
   kubefed                            Community Operators   91m
   # ...

   Note the catalog for your desired Operator.

2. Inspect your desired Operator to verify its supported install modes and available channels:

   $ oc describe packagemanifests <operator_name> -n openshift-marketplace

   Example 4.2. Example output

   # ...
   Kind:         PackageManifest
   # ...
         Install Modes: 1
           Supported:  true
           Type:       OwnNamespace
           Supported:  true
           Type:       SingleNamespace
           Supported:  false
           Type:       MultiNamespace
           Supported:  true
           Type:       AllNamespaces
   # ...
       Entries:
         Name:       example-operator.v3.7.11
         Version:    3.7.11
         Name:       example-operator.v3.7.10
         Version:    3.7.10
       Name:         stable-3.7 2
   # ...
      Entries:
         Name:         example-operator.v3.8.5
         Version:      3.8.5
         Name:         example-operator.v3.8.4
         Version:      3.8.4
       Name:           stable-3.8 3
     Default Channel:  stable-3.8 4

   1

   Indicates which install modes are supported.

   2 3

   Example channel names.

   4

   The channel selected by default if one is not specified.

   Tip

   You can print an Operator’s version and channel information in YAML format by running the following command:

   $ oc get packagemanifests <operator_name> -n <catalog_namespace> -o yaml

   • If more than one catalog is installed in a namespace, run the following command to look up the available versions and channels of an Operator from a specific catalog:

     $ oc get packagemanifest \
        --selector=catalog=<catalogsource_name> \
        --field-selector metadata.name=<operator_name> \
        -n <catalog_namespace> -o yaml

     Important

     If you do not specify the Operator’s catalog, running the oc get packagemanifest and oc describe packagemanifest commands might return a package from an unexpected catalog if the following conditions are met:

     • Multiple catalogs are installed in the same namespace.
     • The catalogs contain the same Operators or Operators with the same name.

3. If the Operator you intend to install supports the AllNamespaces install mode, and you choose to use this mode, skip this step, because the openshift-operators namespace already has an appropriate Operator group in place by default, called global-operators.

   If the Operator you intend to install supports the SingleNamespace install mode, and you choose to use this mode, you must ensure an appropriate Operator group exists in the related namespace. If one does not exist, you can create create one by following these steps:

   Important

   You can only have one Operator group per namespace. For more information, see "Operator groups".

   1. Create an OperatorGroup object YAML file, for example operatorgroup.yaml, for SingleNamespace install mode:

      Example OperatorGroup object for SingleNamespace install mode

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: <operatorgroup_name>
        namespace: <namespace> 1
      spec:
        targetNamespaces:
        - <namespace> 2

      1 2

      For SingleNamespace install mode, use the same <namespace> value for both the metadata.namespace and spec.targetNamespaces fields.

   2. Create the OperatorGroup object:

      $ oc apply -f operatorgroup.yaml

4. Create a Subscription object to subscribe a namespace to an Operator:

   1. Create a YAML file for the Subscription object, for example subscription.yaml:

      Note

      If you want to subscribe to a specific version of an Operator, set the startingCSV field to the desired version and set the installPlanApproval field to Manual to prevent the Operator from automatically upgrading if a later version exists in the catalog. For details, see the following "Example Subscription object with a specific starting Operator version".

      Example 4.3. Example Subscription object

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: <subscription_name>
        namespace: <namespace_per_install_mode> 1
      spec:
        channel: <channel_name> 2
        name: <operator_name> 3
        source: <catalog_name> 4
        sourceNamespace: <catalog_source_namespace> 5
        config:
          env: 6
          - name: ARGS
            value: "-v=10"
          envFrom: 7
          - secretRef:
              name: license-secret
          volumes: 8
          - name: <volume_name>
            configMap:
              name: <configmap_name>
          volumeMounts: 9
          - mountPath: <directory_name>
            name: <volume_name>
          tolerations: 10
          - operator: "Exists"
          resources: 11
            requests:
              memory: "64Mi"
              cpu: "250m"
            limits:
              memory: "128Mi"
              cpu: "500m"
          nodeSelector: 12
            foo: bar

      1

      For default AllNamespaces install mode usage, specify the openshift-operators namespace. Alternatively, you can specify a custom global namespace, if you have created one. For SingleNamespace install mode usage, specify the relevant single namespace.

      2

      Name of the channel to subscribe to.

      3

      Name of the Operator to subscribe to.

      4

      Name of the catalog source that provides the Operator.

      5

      Namespace of the catalog source. Use openshift-marketplace for the default OperatorHub catalog sources.

      6

      The env parameter defines a list of environment variables that must exist in all containers in the pod created by OLM.

      7

      The envFrom parameter defines a list of sources to populate environment variables in the container.

      8

      The volumes parameter defines a list of volumes that must exist on the pod created by OLM.

      9

      The volumeMounts parameter defines a list of volume mounts that must exist in all containers in the pod created by OLM. If a volumeMount references a volume that does not exist, OLM fails to deploy the Operator.

      10

      The tolerations parameter defines a list of tolerations for the pod created by OLM.

      11

      The resources parameter defines resource constraints for all the containers in the pod created by OLM.

      12

      The nodeSelector parameter defines a NodeSelector for the pod created by OLM.

      Example 4.4. Example Subscription object with a specific starting Operator version

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: example-operator
        namespace: example-operator
      spec:
        channel: stable-3.7
        installPlanApproval: Manual 1
        name: example-operator
        source: custom-operators
        sourceNamespace: openshift-marketplace
        startingCSV: example-operator.v3.7.10 2

      1

      Set the approval strategy to Manual in case your specified version is superseded by a later version in the catalog. This plan prevents an automatic upgrade to a later version and requires manual approval before the starting CSV can complete the installation.

      2

      Set a specific version of an Operator CSV.

   2. For clusters on cloud providers with token authentication enabled, configure your Subscription object by following these steps:

      1. Ensure the Subscription object is set to manual update approvals:

         kind: Subscription
         # ...
         spec:
           installPlanApproval: Manual 1

         1

         Subscriptions with automatic update approvals are not recommended because there might be permission changes to make prior to updating. Subscriptions with manual update approvals ensure that administrators have the opportunity to verify the permissions of the later version and take any necessary steps prior to update.

      2. Include the relevant cloud provider-specific fields in the Subscription object’s config section:

         • If the cluster is in AWS STS mode, include the following fields:

           kind: Subscription
           # ...
           spec:
             config:
               env:
               - name: ROLEARN
                 value: "<role_arn>" 1

           1

           Include the role ARN details.

         • If the cluster is in Microsoft Entra Workload ID mode, include the following fields:

           kind: Subscription
           # ...
           spec:
            config:
              env:
              - name: CLIENTID
                value: "<client_id>" 1
              - name: TENANTID
                value: "<tenant_id>" 2
              - name: SUBSCRIPTIONID
                value: "<subscription_id>" 3

           1

           Include the client ID.

           2

           Include the tenant ID.

           3

           Include the subscription ID.

   3. Create the Subscription object by running the following command:

      $ oc apply -f subscription.yaml

5. If you set the installPlanApproval field to Manual, manually approve the pending install plan to complete the Operator installation. For more information, see "Manually approving a pending Operator update".

Verification

1. Check the status of the Subscription object for your installed Operator by running the following command:

   $ oc describe subscription <subscription_name> -n <namespace>

2. If you created an Operator group for SingleNamespace install mode, check the status of the OperatorGroup object by running the following command:

   $ oc describe operatorgroup <operatorgroup_name> -n <namespace>

Procedure

1. Before installing the Operator, create a namespace for the tenant Operator that is separate from the tenant’s namespace. For example, if the tenant’s namespace is team1, you might create a team1-operator namespace:

   1. Define a Namespace resource and save the YAML file, for example, team1-operator.yaml:

      apiVersion: v1
      kind: Namespace
      metadata:
        name: team1-operator

   2. Create the namespace by running the following command:

      $ oc create -f team1-operator.yaml

2. Create an Operator group for the tenant Operator scoped to the tenant’s namespace, with only that one namespace entry in the spec.targetNamespaces list:

   1. Define an OperatorGroup resource and save the YAML file, for example, team1-operatorgroup.yaml:

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: team1-operatorgroup
        namespace: team1-operator
      spec:
        targetNamespaces:
        - team1 1

      1

      Define only the tenant’s namespace in the spec.targetNamespaces list.

   2. Create the Operator group by running the following command:

      $ oc create -f team1-operatorgroup.yaml

Procedure

1. Before installing the Operator, create a namespace for the installation of your desired Operator. This installation namespace will become the custom global namespace:

   1. Define a Namespace resource and save the YAML file, for example, global-operators.yaml:

      apiVersion: v1
      kind: Namespace
      metadata:
        name: global-operators

   2. Create the namespace by running the following command:

      $ oc create -f global-operators.yaml

2. Create a custom global Operator group, which is an Operator group that watches all namespaces:

   1. Define an OperatorGroup resource and save the YAML file, for example, global-operatorgroup.yaml. Omit both the spec.selector and spec.targetNamespaces fields to make it a global Operator group, which selects all namespaces:

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: global-operatorgroup
        namespace: global-operators

      Note

      The status.namespaces of a created global Operator group contains the empty string (""), which signals to a consuming Operator that it should watch all namespaces.

   2. Create the Operator group by running the following command:

      $ oc create -f global-operatorgroup.yaml

1. Determine a node or set of nodes to target for the pods per your requirements. If available, note an existing label, such as node-role.kubernetes.io/app, that identifies the node or nodes. Otherwise, add a label, such as myoperator, by using a compute machine set or editing the node directly. You will use this label in a later step as the node selector on your project.
2. If you want to ensure that only pods with a certain label are allowed to run on the nodes, while steering unrelated workloads to other nodes, add a taint to the node or nodes by using a compute machine set or editing the node directly. Use an effect that ensures that new pods that do not match the taint cannot be scheduled on the nodes. For example, a myoperator:NoSchedule taint ensures that new pods that do not match the taint are not scheduled onto that node, but existing pods on the node are allowed to remain.
3. Create a project that is configured with a default node selector and, if you added a taint, a matching toleration.

1. Install the Operator as usual.
2. If needed, ensure that your nodes are labeled to properly respond to the affinity.

3. Edit the Operator Subscription object to add an affinity:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: openshift-custom-metrics-autoscaler-operator
     namespace: openshift-keda
   spec:
     name: my-package
     source: my-operators
     sourceNamespace: operator-registries
     config:
       affinity: 1
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: kubernetes.io/hostname
                 operator: In
                 values:
                 - ip-10-0-185-229.ec2.internal
   #...

   1

   Add a nodeAffinity, podAffinity, or podAntiAffinity. See the Additional resources section that follows for information about creating the affinity.

Procedure

1. In the Administrator perspective of the web console, navigate to Operators Installed Operators.
2. Click the name of the Operator you want to change the update channel for.
3. Click the Subscription tab.
4. Click the name of the update channel under Update channel.
5. Click the newer update channel that you want to change to, then click Save.

6. For subscriptions with an Automatic approval strategy, the update begins automatically. Navigate back to the Operators Installed Operators page to monitor the progress of the update. When complete, the status changes to Succeeded and Up to date.

   For subscriptions with a Manual approval strategy, you can manually approve the update from the Subscription tab.

Procedure

1. In the Administrator perspective of the OpenShift Container Platform web console, navigate to Operators Installed Operators.
2. Operators that have a pending update display a status with Upgrade available. Click the name of the Operator you want to update.
3. Click the Subscription tab. Any updates requiring approval are displayed next to Upgrade status. For example, it might display 1 requires approval.
4. Click 1 requires approval, then click Preview Install Plan.
5. Review the resources that are listed as available for update. When satisfied, click Approve.
6. Navigate back to the Operators Installed Operators page to monitor the progress of the update. When complete, the status changes to Succeeded and Up to date.

Procedure

1. Navigate to the Operators Installed Operators page.
2. Scroll or enter a keyword into the Filter by name field to find the Operator that you want to remove. Then, click on it.

3. On the right side of the Operator Details page, select Uninstall Operator from the Actions list.

   An Uninstall Operator? dialog box is displayed.

4. Select Uninstall to remove the Operator, Operator deployments, and pods. Following this action, the Operator stops running and no longer receives updates.

   Note

   This action does not remove resources managed by the Operator, including custom resource definitions (CRDs) and custom resources (CRs). Dashboards and navigation items enabled by the web console and off-cluster resources that continue to run might need manual clean up. To remove these after uninstalling the Operator, you might need to manually delete the Operator CRDs.

Procedure

1. Ensure the latest version of the subscribed operator (for example, serverless-operator) is identified in the currentCSV field.

   $ oc get subscription.operators.coreos.com serverless-operator -n openshift-serverless -o yaml | grep currentCSV

   Example output

     currentCSV: serverless-operator.v1.28.0

2. Delete the subscription (for example, serverless-operator):

   $ oc delete subscription.operators.coreos.com serverless-operator -n openshift-serverless

   Example output

   subscription.operators.coreos.com "serverless-operator" deleted

3. Delete the CSV for the Operator in the target namespace using the currentCSV value from the previous step:

   $ oc delete clusterserviceversion serverless-operator.v1.28.0 -n openshift-serverless

   Example output

   clusterserviceversion.operators.coreos.com "serverless-operator.v1.28.0" deleted

Procedure

1. Get the names of the Subscription and ClusterServiceVersion objects from the namespace where the Operator is installed:

   $ oc get sub,csv -n <namespace>

   Example output

   NAME                                                       PACKAGE                  SOURCE             CHANNEL
   subscription.operators.coreos.com/elasticsearch-operator   elasticsearch-operator   redhat-operators   5.0
   
   NAME                                                                         DISPLAY                            VERSION    REPLACES   PHASE
   clusterserviceversion.operators.coreos.com/elasticsearch-operator.5.0.0-65   OpenShift Elasticsearch Operator   5.0.0-65              Succeeded

2. Delete the subscription:

   $ oc delete subscription <subscription_name> -n <namespace>

3. Delete the cluster service version:

   $ oc delete csv <csv_name> -n <namespace>

4. Get the names of any failing jobs and related config maps in the openshift-marketplace namespace:

   $ oc get job,configmap -n openshift-marketplace

   Example output

   NAME                                                                        COMPLETIONS   DURATION   AGE
   job.batch/1de9443b6324e629ddf31fed0a853a121275806170e34c926d69e53a7fcbccb   1/1           26s        9m30s
   
   NAME                                                                        DATA   AGE
   configmap/1de9443b6324e629ddf31fed0a853a121275806170e34c926d69e53a7fcbccb   3      9m30s

5. Delete the job:

   $ oc delete job <job_name> -n openshift-marketplace

   This ensures pods that try to pull the inaccessible image are not recreated.

6. Delete the config map:

   $ oc delete configmap <configmap_name> -n openshift-marketplace

7. Reinstall the Operator using OperatorHub in the web console.

Procedure

1. Navigate in the web console to the Operators OperatorHub page.
2. Select the Operator and click Install.

3. On the Install Operator page, modify the Subscription object to include one or more of the following environment variables in the spec section:

   • HTTP_PROXY
   • HTTPS_PROXY
   • NO_PROXY

   For example:

   Subscription object with proxy setting overrides

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: etcd-config-test
     namespace: openshift-operators
   spec:
     config:
       env:
       - name: HTTP_PROXY
         value: test_http
       - name: HTTPS_PROXY
         value: test_https
       - name: NO_PROXY
         value: test
     channel: clusterwide-alpha
     installPlanApproval: Automatic
     name: etcd
     source: community-operators
     sourceNamespace: openshift-marketplace
     startingCSV: etcdoperator.v0.9.4-clusterwide

   Note

   These environment variables can also be unset using an empty value to remove any previously set cluster-wide or custom proxy settings.

   OLM handles these environment variables as a unit; if at least one of them is set, all three are considered overridden and the cluster-wide defaults are not used for the deployments of the subscribed Operator.

4. Click Install to make the Operator available to the selected namespaces.

5. After the CSV for the Operator appears in the relevant namespace, you can verify that custom proxy environment variables are set in the deployment. For example, using the CLI:

   $ oc get deployment -n openshift-operators \
       etcd-operator -o yaml \
       | grep -i "PROXY" -A 2

   Example output

           - name: HTTP_PROXY
             value: test_http
           - name: HTTPS_PROXY
             value: test_https
           - name: NO_PROXY
             value: test
           image: quay.io/coreos/etcd-operator@sha256:66a37fd61a06a43969854ee6d3e21088a98b93838e284a6086b13917f96b0d9c
   ...

Procedure

1. Create an empty config map in the namespace where the subscription for your Operator exists and include the following label:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: trusted-ca 1
     labels:
       config.openshift.io/inject-trusted-cabundle: "true" 2

   1

   Name of the config map.

   2

   Requests the Cluster Network Operator to inject the merged bundle.

   After creating this config map, it is immediately populated with the certificate contents of the merged bundle.

2. Update the Subscription object to include a spec.config section that mounts the trusted-ca config map as a volume to each container within a pod that requires a custom CA:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: my-operator
   spec:
     package: etcd
     channel: alpha
     config: 1
       selector:
         matchLabels:
           <labels_for_pods> 2
       volumes: 3
       - name: trusted-ca
         configMap:
           name: trusted-ca
           items:
             - key: ca-bundle.crt 4
               path: tls-ca-bundle.pem 5
       volumeMounts: 6
       - name: trusted-ca
         mountPath: /etc/pki/ca-trust/extracted/pem
         readOnly: true

   1

   Add a config section if it does not exist.

   2

   Specify labels to match pods that are owned by the Operator.

   3

   Create a trusted-ca volume.

   4

   ca-bundle.crt is required as the config map key.

   5

   tls-ca-bundle.pem is required as the config map path.

   6

   Create a trusted-ca volume mount.

   Note

   Deployments of an Operator can fail to validate the authority and display a x509 certificate signed by unknown authority error. This error can occur even after injecting a custom CA when using the subscription of an Operator. In this case, you can set the mountPath as /etc/ssl/certs for trusted-ca by using the subscription of an Operator.

Procedure

1. List Operator subscriptions:

   $ oc get subs -n <operator_namespace>

2. Use the oc describe command to inspect a Subscription resource:

   $ oc describe sub <subscription_name> -n <operator_namespace>

3. In the command output, find the Conditions section for the status of Operator subscription condition types. In the following example, the CatalogSourcesUnhealthy condition type has a status of false because all available catalog sources are healthy:

   Example output

   Name:         cluster-logging
   Namespace:    openshift-logging
   Labels:       operators.coreos.com/cluster-logging.openshift-logging=
   Annotations:  <none>
   API Version:  operators.coreos.com/v1alpha1
   Kind:         Subscription
   # ...
   Conditions:
      Last Transition Time:  2019-07-29T13:42:57Z
      Message:               all available catalogsources are healthy
      Reason:                AllCatalogSourcesHealthy
      Status:                False
      Type:                  CatalogSourcesUnhealthy
   # ...

Procedure

1. List the catalog sources in a namespace. For example, you can check the openshift-marketplace namespace, which is used for cluster-wide catalog sources:

   $ oc get catalogsources -n openshift-marketplace

   Example output

   NAME                  DISPLAY               TYPE   PUBLISHER   AGE
   certified-operators   Certified Operators   grpc   Red Hat     55m
   community-operators   Community Operators   grpc   Red Hat     55m
   example-catalog       Example Catalog       grpc   Example Org 2m25s
   redhat-marketplace    Red Hat Marketplace   grpc   Red Hat     55m
   redhat-operators      Red Hat Operators     grpc   Red Hat     55m

2. Use the oc describe command to get more details and status about a catalog source:

   $ oc describe catalogsource example-catalog -n openshift-marketplace

   Example output

   Name:         example-catalog
   Namespace:    openshift-marketplace
   Labels:       <none>
   Annotations:  operatorframework.io/managed-by: marketplace-operator
                 target.workload.openshift.io/management: {"effect": "PreferredDuringScheduling"}
   API Version:  operators.coreos.com/v1alpha1
   Kind:         CatalogSource
   # ...
   Status:
     Connection State:
       Address:              example-catalog.openshift-marketplace.svc:50051
       Last Connect:         2021-09-09T17:07:35Z
       Last Observed State:  TRANSIENT_FAILURE
     Registry Service:
       Created At:         2021-09-09T17:05:45Z
       Port:               50051
       Protocol:           grpc
       Service Name:       example-catalog
       Service Namespace:  openshift-marketplace
   # ...

   In the preceding example output, the last observed state is TRANSIENT_FAILURE. This state indicates that there is a problem establishing a connection for the catalog source.

3. List the pods in the namespace where your catalog source was created:

   $ oc get pods -n openshift-marketplace

   Example output

   NAME                                    READY   STATUS             RESTARTS   AGE
   certified-operators-cv9nn               1/1     Running            0          36m
   community-operators-6v8lp               1/1     Running            0          36m
   marketplace-operator-86bfc75f9b-jkgbc   1/1     Running            0          42m
   example-catalog-bwt8z                   0/1     ImagePullBackOff   0          3m55s
   redhat-marketplace-57p8c                1/1     Running            0          36m
   redhat-operators-smxx8                  1/1     Running            0          36m

   When a catalog source is created in a namespace, a pod for the catalog source is created in that namespace. In the preceding example output, the status for the example-catalog-bwt8z pod is ImagePullBackOff. This status indicates that there is an issue pulling the catalog source’s index image.

4. Use the oc describe command to inspect a pod for more detailed information:

   $ oc describe pod example-catalog-bwt8z -n openshift-marketplace

   Example output

   Name:         example-catalog-bwt8z
   Namespace:    openshift-marketplace
   Priority:     0
   Node:         ci-ln-jyryyg2-f76d1-ggdbq-worker-b-vsxjd/10.0.128.2
   ...
   Events:
     Type     Reason          Age                From               Message
     ----     ------          ----               ----               -------
     Normal   Scheduled       48s                default-scheduler  Successfully assigned openshift-marketplace/example-catalog-bwt8z to ci-ln-jyryyf2-f76d1-fgdbq-worker-b-vsxjd
     Normal   AddedInterface  47s                multus             Add eth0 [10.131.0.40/23] from openshift-sdn
     Normal   BackOff         20s (x2 over 46s)  kubelet            Back-off pulling image "quay.io/example-org/example-catalog:v1"
     Warning  Failed          20s (x2 over 46s)  kubelet            Error: ImagePullBackOff
     Normal   Pulling         8s (x3 over 47s)   kubelet            Pulling image "quay.io/example-org/example-catalog:v1"
     Warning  Failed          8s (x3 over 47s)   kubelet            Failed to pull image "quay.io/example-org/example-catalog:v1": rpc error: code = Unknown desc = reading manifest v1 in quay.io/example-org/example-catalog: unauthorized: access to the requested resource is not authorized
     Warning  Failed          8s (x3 over 47s)   kubelet            Error: ErrImagePull

   In the preceding example output, the error messages indicate that the catalog source’s index image is failing to pull successfully because of an authorization issue. For example, the index image might be stored in a registry that requires login credentials.

Procedure

1. Edit the OperatorCondition object for the Operator:

   $ oc edit operatorcondition <name>

2. Add a Spec.Overrides array to the object:

   Example Operator condition override

   apiVersion: operators.coreos.com/v2
   kind: OperatorCondition
   metadata:
     name: my-operator
     namespace: operators
   spec:
     overrides:
     - type: Upgradeable 1
       status: "True"
       reason: "upgradeIsSafe"
       message: "This is a known issue with the Operator where it always reports that it cannot be upgraded."
     conditions:
     - type: Upgradeable
       status: "False"
       reason: "migration"
       message: "The operator is performing a migration."
       lastTransitionTime: "2020-08-24T23:15:55Z"

   1

   Allows the cluster administrator to change the upgrade readiness to True.

Procedure

1. Create a new namespace:

   $ cat <<EOF | oc create -f -
   apiVersion: v1
   kind: Namespace
   metadata:
     name: scoped
   EOF

2. Allocate permissions that you want the Operator(s) to be confined to. This involves creating a new service account, relevant role(s), and role binding(s).

   $ cat <<EOF | oc create -f -
   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: scoped
     namespace: scoped
   EOF

   The following example grants the service account permissions to do anything in the designated namespace for simplicity. In a production environment, you should create a more fine-grained set of permissions:

   $ cat <<EOF | oc create -f -
   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: scoped
     namespace: scoped
   rules:
   - apiGroups: ["*"]
     resources: ["*"]
     verbs: ["*"]
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: scoped-bindings
     namespace: scoped
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: Role
     name: scoped
   subjects:
   - kind: ServiceAccount
     name: scoped
     namespace: scoped
   EOF

3. Create an OperatorGroup object in the designated namespace. This Operator group targets the designated namespace to ensure that its tenancy is confined to it.

   In addition, Operator groups allow a user to specify a service account. Specify the service account created in the previous step:

   $ cat <<EOF | oc create -f -
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: scoped
     namespace: scoped
   spec:
     serviceAccountName: scoped
     targetNamespaces:
     - scoped
   EOF

   Any Operator installed in the designated namespace is tied to this Operator group and therefore to the service account specified.

4. Create a Subscription object in the designated namespace to install an Operator:

   $ cat <<EOF | oc create -f -
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: etcd
     namespace: scoped
   spec:
     channel: singlenamespace-alpha
     name: etcd
     source: <catalog_source_name> 1
     sourceNamespace: <catalog_source_namespace> 2
   EOF

   1

   Specify a catalog source that already exists in the designated namespace or one that is in the global catalog namespace.

   2

   Specify a namespace where the catalog source was created.

   Any Operator tied to this Operator group is confined to the permissions granted to the specified service account. If the Operator requests permissions that are outside the scope of the service account, the installation fails with relevant errors.

1. Disable all of the default global catalogs.
2. Enable custom, curated catalogs in the same namespace where the relevant Operator groups have been preinstalled.

Procedure

1. Review the Subscription object. Its status has an object reference installPlanRef that points to the InstallPlan object that attempted to create the necessary [Cluster]Role[Binding] object(s) for the Operator:

   apiVersion: operators.coreos.com/v1
   kind: Subscription
   metadata:
     name: etcd
     namespace: scoped
   status:
     installPlanRef:
       apiVersion: operators.coreos.com/v1
       kind: InstallPlan
       name: install-4plp8
       namespace: scoped
       resourceVersion: "117359"
       uid: 2c1df80e-afea-11e9-bce3-5254009c9c23

2. Check the status of the InstallPlan object for any errors:

   apiVersion: operators.coreos.com/v1
   kind: InstallPlan
   status:
     conditions:
     - lastTransitionTime: "2019-07-26T21:13:10Z"
       lastUpdateTime: "2019-07-26T21:13:10Z"
       message: 'error creating clusterrole etcdoperator.v0.9.4-clusterwide-dsfx4: clusterroles.rbac.authorization.k8s.io
         is forbidden: User "system:serviceaccount:scoped:scoped" cannot create resource
         "clusterroles" in API group "rbac.authorization.k8s.io" at the cluster scope'
       reason: InstallComponentFailed
       status: "False"
       type: Installed
     phase: Failed

   The error message tells you:

   • The type of resource it failed to create, including the API group of the resource. In this case, it was clusterroles in the rbac.authorization.k8s.io group.
   • The name of the resource.
   • The type of error: is forbidden tells you that the user does not have enough permission to do the operation.
   • The name of the user who attempted to create or update the resource. In this case, it refers to the service account specified in the Operator group.

   • The scope of the operation: cluster scope or not.

     The user can add the missing permission to the service account and then iterate.

     Note

     Operator Lifecycle Manager (OLM) does not currently provide the complete list of errors on the first try.

Procedure

1. Create a CatalogSource object that references your index image.

   1. Modify the following to your specifications and save it as a catalogSource.yaml file:

      apiVersion: operators.coreos.com/v1alpha1
      kind: CatalogSource
      metadata:
        name: my-operator-catalog
        namespace: openshift-marketplace 1
        annotations:
          olm.catalogImageTemplate: 2
            "<registry>/<namespace>/<index_image_name>:v{kube_major_version}.{kube_minor_version}.{kube_patch_version}"
      spec:
        sourceType: grpc
        grpcPodConfig:
          securityContextConfig: <security_mode> 3
        image: <registry>/<namespace>/<index_image_name>:<tag> 4
        displayName: My Operator Catalog
        publisher: <publisher_name> 5
        updateStrategy:
          registryPoll: 6
            interval: 30m

      1

      If you want the catalog source to be available globally to users in all namespaces, specify the openshift-marketplace namespace. Otherwise, you can specify a different namespace for the catalog to be scoped and available only for that namespace.

      2

      Optional: Set the olm.catalogImageTemplate annotation to your index image name and use one or more of the Kubernetes cluster version variables as shown when constructing the template for the image tag.

      3

      Specify the value of legacy or restricted. If the field is not set, the default value is legacy. In a future OpenShift Container Platform release, it is planned that the default value will be restricted. If your catalog cannot run with restricted permissions, it is recommended that you manually set this field to legacy.

      4

      Specify your index image. If you specify a tag after the image name, for example :v4.15, the catalog source pod uses an image pull policy of Always, meaning the pod always pulls the image prior to starting the container. If you specify a digest, for example @sha256:<id>, the image pull policy is IfNotPresent, meaning the pod pulls the image only if it does not already exist on the node.

      5

      Specify your name or an organization name publishing the catalog.

      6

      Catalog sources can automatically check for new versions to keep up to date.

   2. Use the file to create the CatalogSource object:

      $ oc apply -f catalogSource.yaml

2. Verify the following resources are created successfully.

   1. Check the pods:

      $ oc get pods -n openshift-marketplace

      Example output

      NAME                                    READY   STATUS    RESTARTS  AGE
      my-operator-catalog-6njx6               1/1     Running   0         28s
      marketplace-operator-d9f549946-96sgr    1/1     Running   0         26h

   2. Check the catalog source:

      $ oc get catalogsource -n openshift-marketplace

      Example output

      NAME                  DISPLAY               TYPE PUBLISHER  AGE
      my-operator-catalog   My Operator Catalog   grpc            5s

   3. Check the package manifest:

      $ oc get packagemanifest -n openshift-marketplace

      Example output

      NAME                          CATALOG               AGE
      jaeger-product                My Operator Catalog   93s

Procedure

1. Create a secret for each required private registry.

   1. Log in to the private registry to create or update your registry credentials file:

      $ podman login <registry>:<port>

      Note

      The file path of your registry credentials can be different depending on the container tool used to log in to the registry. For the podman CLI, the default location is ${XDG_RUNTIME_DIR}/containers/auth.json. For the docker CLI, the default location is /root/.docker/config.json.

   2. It is recommended to include credentials for only one registry per secret, and manage credentials for multiple registries in separate secrets. Multiple secrets can be included in a CatalogSource object in later steps, and OpenShift Container Platform will merge the secrets into a single virtual credentials file for use during an image pull.

      A registry credentials file can, by default, store details for more than one registry or for multiple repositories in one registry. Verify the current contents of your file. For example:

      File storing credentials for multiple registries

      {
          "auths": {
              "registry.redhat.io": {
                  "auth": "FrNHNydQXdzclNqdg=="
              },
              "quay.io": {
                  "auth": "fegdsRib21iMQ=="
              },
              "https://quay.io/my-namespace/my-user/my-image": {
                  "auth": "eWfjwsDdfsa221=="
              },
              "https://quay.io/my-namespace/my-user": {
                  "auth": "feFweDdscw34rR=="
              },
              "https://quay.io/my-namespace": {
                  "auth": "frwEews4fescyq=="
              }
          }
      }

      Because this file is used to create secrets in later steps, ensure that you are storing details for only one registry per file. This can be accomplished by using either of the following methods:

      • Use the podman logout <registry> command to remove credentials for additional registries until only the one registry you want remains.

      • Edit your registry credentials file and separate the registry details to be stored in multiple files. For example:

        File storing credentials for one registry

        {
                "auths": {
                        "registry.redhat.io": {
                                "auth": "FrNHNydQXdzclNqdg=="
                        }
                }
        }

        File storing credentials for another registry

        {
                "auths": {
                        "quay.io": {
                                "auth": "Xd2lhdsbnRib21iMQ=="
                        }
                }
        }

   3. Create a secret in the openshift-marketplace namespace that contains the authentication credentials for a private registry:

      $ oc create secret generic <secret_name> \
          -n openshift-marketplace \
          --from-file=.dockerconfigjson=<path/to/registry/credentials> \
          --type=kubernetes.io/dockerconfigjson

      Repeat this step to create additional secrets for any other required private registries, updating the --from-file flag to specify another registry credentials file path.

2. Create or update an existing CatalogSource object to reference one or more secrets:

   apiVersion: operators.coreos.com/v1alpha1
   kind: CatalogSource
   metadata:
     name: my-operator-catalog
     namespace: openshift-marketplace
   spec:
     sourceType: grpc
     secrets: 1
     - "<secret_name_1>"
     - "<secret_name_2>"
     grpcPodConfig:
       securityContextConfig: <security_mode> 2
     image: <registry>:<port>/<namespace>/<image>:<tag>
     displayName: My Operator Catalog
     publisher: <publisher_name>
     updateStrategy:
       registryPoll:
         interval: 30m

   1

   Add a spec.secrets section and specify any required secrets.

   2

   Specify the value of legacy or restricted. If the field is not set, the default value is legacy. In a future OpenShift Container Platform release, it is planned that the default value will be restricted. If your catalog cannot run with restricted permissions, it is recommended that you manually set this field to legacy.

3. If any Operator or Operand images that are referenced by a subscribed Operator require access to a private registry, you can either provide access to all namespaces in the cluster, or individual target tenant namespaces.

   • To provide access to all namespaces in the cluster, add authentication details to the global cluster pull secret in the openshift-config namespace.

     Warning

     Cluster resources must adjust to the new global pull secret, which can temporarily limit the usability of the cluster.

     1. Extract the .dockerconfigjson file from the global pull secret:

        $ oc extract secret/pull-secret -n openshift-config --confirm

     2. Update the .dockerconfigjson file with your authentication credentials for the required private registry or registries and save it as a new file:

        $ cat .dockerconfigjson | \
            jq --compact-output '.auths["<registry>:<port>/<namespace>/"] |= . + {"auth":"<token>"}' \1
            > new_dockerconfigjson

        1

        Replace <registry>:<port>/<namespace> with the private registry details and <token> with your authentication credentials.

     3. Update the global pull secret with the new file:

        $ oc set data secret/pull-secret -n openshift-config \
            --from-file=.dockerconfigjson=new_dockerconfigjson

   • To update an individual namespace, add a pull secret to the service account for the Operator that requires access in the target tenant namespace.

     1. Recreate the secret that you created for the openshift-marketplace in the tenant namespace:

        $ oc create secret generic <secret_name> \
            -n <tenant_namespace> \
            --from-file=.dockerconfigjson=<path/to/registry/credentials> \
            --type=kubernetes.io/dockerconfigjson

     2. Verify the name of the service account for the Operator by searching the tenant namespace:

        $ oc get sa -n <tenant_namespace> 1

        1

        If the Operator was installed in an individual namespace, search that namespace. If the Operator was installed for all namespaces, search the openshift-operators namespace.

        Example output

        NAME            SECRETS   AGE
        builder         2         6m1s
        default         2         6m1s
        deployer        2         6m1s
        etcd-operator   2         5m18s 1

        1

        Service account for an installed etcd Operator.

     3. Link the secret to the service account for the Operator:

        $ oc secrets link <operator_sa> \
            -n <tenant_namespace> \
             <secret_name> \
            --for=pull

Procedure

1. In the Administrator perspective of the web console, navigate to Administration Cluster Settings.
2. Click the Configuration tab, and then click OperatorHub.
3. Click the Sources tab.
4. Select the Options menu for the catalog that you want to remove, and then click Delete CatalogSource.

Procedure

1. Create a CatalogSource object that references your index image. If you used the oc adm catalog mirror command to mirror your catalog to a target registry, you can use the generated catalogSource.yaml file in your manifests directory as a starting point.

   1. Modify the following to your specifications and save it as a catalogSource.yaml file:

      apiVersion: operators.coreos.com/v1alpha1
      kind: CatalogSource
      metadata:
        name: my-operator-catalog 1
        namespace: openshift-marketplace 2
      spec:
        sourceType: grpc
        grpcPodConfig:
          securityContextConfig: <security_mode> 3
        image: <registry>/<namespace>/redhat-operator-index:v4.15 4
        displayName: My Operator Catalog
        publisher: <publisher_name> 5
        updateStrategy:
          registryPoll: 6
            interval: 30m

      1

      If you mirrored content to local files before uploading to a registry, remove any backslash (/) characters from the metadata.name field to avoid an "invalid resource name" error when you create the object.

      2

      If you want the catalog source to be available globally to users in all namespaces, specify the openshift-marketplace namespace. Otherwise, you can specify a different namespace for the catalog to be scoped and available only for that namespace.

      3

      Specify the value of legacy or restricted. If the field is not set, the default value is legacy. In a future OpenShift Container Platform release, it is planned that the default value will be restricted. If your catalog cannot run with restricted permissions, it is recommended that you manually set this field to legacy.

      4

      Specify your index image. If you specify a tag after the image name, for example :v4.15, the catalog source pod uses an image pull policy of Always, meaning the pod always pulls the image prior to starting the container. If you specify a digest, for example @sha256:<id>, the image pull policy is IfNotPresent, meaning the pod pulls the image only if it does not already exist on the node.

      5

      Specify your name or an organization name publishing the catalog.

      6

      Catalog sources can automatically check for new versions to keep up to date.

   2. Use the file to create the CatalogSource object:

      $ oc apply -f catalogSource.yaml

2. Verify the following resources are created successfully.

   1. Check the pods:

      $ oc get pods -n openshift-marketplace

      Example output

      NAME                                    READY   STATUS    RESTARTS  AGE
      my-operator-catalog-6njx6               1/1     Running   0         28s
      marketplace-operator-d9f549946-96sgr    1/1     Running   0         26h

   2. Check the catalog source:

      $ oc get catalogsource -n openshift-marketplace

      Example output

      NAME                  DISPLAY               TYPE PUBLISHER  AGE
      my-operator-catalog   My Operator Catalog   grpc            5s

   3. Check the package manifest:

      $ oc get packagemanifest -n openshift-marketplace

      Example output

      NAME                          CATALOG               AGE
      jaeger-product                My Operator Catalog   93s

Procedure

1. Choose a platform Operator from the supported set of OLM-based Operators. For the list of this set and details on current limitations, see "Technology Preview restrictions for platform Operators".
2. Select a cluster installation method and follow the instructions through creating an install-config.yaml file. For more details on preparing for a cluster installation, see "Selecting a cluster installation method and preparing it for users".

3. After you have created the install-config.yaml file and completed any modifications to it, change to the directory that contains the installation program and create the manifests:

   $ ./openshift-install create manifests --dir <installation_directory> 1

   1

   For <installation_directory>, specify the name of the directory that contains the install-config.yaml file for your cluster.

4. Create a FeatureGate object YAML file in the <installation_directory>/manifests/ directory that enables the TechPreviewNoUpgrade feature set, for example a feature-gate.yaml file:

   Example feature-gate.yaml file

   apiVersion: config.openshift.io/v1
   kind: FeatureGate
   metadata:
     annotations:
       include.release.openshift.io/self-managed-high-availability: "true"
       include.release.openshift.io/single-node-developer: "true"
       release.openshift.io/create-only: "true"
     name: cluster
   spec:
     featureSet: TechPreviewNoUpgrade 1

   1

   Enable the TechPreviewNoUpgrade feature set.

5. Create a PlatformOperator object YAML file for your chosen platform Operator in the <installation_directory>/manifests/ directory, for example a service-mesh-po.yaml file for the Red Hat OpenShift Service Mesh Operator:

   Example service-mesh-po.yaml file

   apiVersion: platform.openshift.io/v1alpha1
   kind: PlatformOperator
   metadata:
     name: service-mesh-po
   spec:
     package:
       name: servicemeshoperator

6. When you are ready to complete the cluster install, refer to your chosen installation method and continue through running the openshift-install create cluster command.

   During cluster creation, your provided manifests are used to enable the TechPreviewNoUpgrade feature set and install your chosen platform Operator.

   Important

   Failure of the platform Operator to successfully install will block the cluster installation process.

Verification

1. Check the status of the service-mesh-po platform Operator by running the following command:

   $ oc get platformoperator service-mesh-po -o yaml

   Example output

   ...
   status:
     activeBundleDeployment:
       name: service-mesh-po
     conditions:
     - lastTransitionTime: "2022-10-24T17:24:40Z"
       message: Successfully applied the service-mesh-po BundleDeployment resource
       reason: InstallSuccessful
       status: "True" 1
       type: Installed

   1

   Wait until the Installed status condition reports True.

2. Verify that the platform-operators-aggregated cluster Operator is reporting an Available=True status:

   $ oc get clusteroperator platform-operators-aggregated -o yaml

   Example output

   ...
   status:
     conditions:
     - lastTransitionTime: "2022-10-24T17:43:26Z"
       message: All platform operators are in a successful state
       reason: AsExpected
       status: "False"
       type: Progressing
     - lastTransitionTime: "2022-10-24T17:43:26Z"
       status: "False"
       type: Degraded
     - lastTransitionTime: "2022-10-24T17:43:26Z"
       message: All platform operators are in a successful state
       reason: AsExpected
       status: "True"
       type: Available

Procedure

1. Choose a platform Operator from the supported set of OLM-based Operators. For the list of this set and details on current limitations, see "Technology Preview restrictions for platform Operators".

2. Create a PlatformOperator object YAML file for your chosen platform Operator, for example a service-mesh-po.yaml file for the Red Hat OpenShift Service Mesh Operator:

   Example sevice-mesh-po.yaml file

   apiVersion: platform.openshift.io/v1alpha1
   kind: PlatformOperator
   metadata:
     name: service-mesh-po
   spec:
     package:
       name: servicemeshoperator

3. Create the PlatformOperator object by running the following command:

   $ oc apply -f service-mesh-po.yaml

   Note

   If your cluster does not have the TechPreviewNoUpgrade feature set enabled, the object creation fails with the following message:

   error: resource mapping not found for name: "service-mesh-po" namespace: "" from "service-mesh-po.yaml": no matches for kind "PlatformOperator" in version "platform.openshift.io/v1alpha1"
   ensure CRDs are installed first

Verification

1. Check the status of the service-mesh-po platform Operator by running the following command:

   $ oc get platformoperator service-mesh-po -o yaml

   Example output

   ...
   status:
     activeBundleDeployment:
       name: service-mesh-po
     conditions:
     - lastTransitionTime: "2022-10-24T17:24:40Z"
       message: Successfully applied the service-mesh-po BundleDeployment resource
       reason: InstallSuccessful
       status: "True" 1
       type: Installed

   1

   Wait until the Installed status condition reports True.

2. Verify that the platform-operators-aggregated cluster Operator is reporting an Available=True status:

   $ oc get clusteroperator platform-operators-aggregated -o yaml

   Example output

   ...
   status:
     conditions:
     - lastTransitionTime: "2022-10-24T17:43:26Z"
       message: All platform operators are in a successful state
       reason: AsExpected
       status: "False"
       type: Progressing
     - lastTransitionTime: "2022-10-24T17:43:26Z"
       status: "False"
       type: Degraded
     - lastTransitionTime: "2022-10-24T17:43:26Z"
       message: All platform operators are in a successful state
       reason: AsExpected
       status: "True"
       type: Available

Procedure

1. Get a list of installed platform Operators and find the name for the Operator you want to delete:

   $ oc get platformoperator

2. Delete the PlatformOperator resource for the chosen Operator, for example, for the Quay Operator:

   $ oc delete platformoperator quay-operator

   Example output

   platformoperator.platform.openshift.io "quay-operator" deleted

Verification

1. Verify the namespace for the platform Operator is eventually deleted, for example, for the Quay Operator:

   $ oc get ns quay-operator-system

   Example output

   Error from server (NotFound): namespaces "quay-operator-system" not found

2. Verify the platform-operators-aggregated cluster Operator continues to report an Available=True status:

   $ oc get co platform-operators-aggregated

   Example output

   NAME                            VERSION     AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
   platform-operators-aggregated   4.15.0-0    True        False         False      70s

Procedure

1. List Operator subscriptions:

   $ oc get subs -n <operator_namespace>

2. Use the oc describe command to inspect a Subscription resource:

   $ oc describe sub <subscription_name> -n <operator_namespace>

3. In the command output, find the Conditions section for the status of Operator subscription condition types. In the following example, the CatalogSourcesUnhealthy condition type has a status of false because all available catalog sources are healthy:

   Example output

   Name:         cluster-logging
   Namespace:    openshift-logging
   Labels:       operators.coreos.com/cluster-logging.openshift-logging=
   Annotations:  <none>
   API Version:  operators.coreos.com/v1alpha1
   Kind:         Subscription
   # ...
   Conditions:
      Last Transition Time:  2019-07-29T13:42:57Z
      Message:               all available catalogsources are healthy
      Reason:                AllCatalogSourcesHealthy
      Status:                False
      Type:                  CatalogSourcesUnhealthy
   # ...

Procedure

1. List the catalog sources in a namespace. For example, you can check the openshift-marketplace namespace, which is used for cluster-wide catalog sources:

   $ oc get catalogsources -n openshift-marketplace

   Example output

   NAME                  DISPLAY               TYPE   PUBLISHER   AGE
   certified-operators   Certified Operators   grpc   Red Hat     55m
   community-operators   Community Operators   grpc   Red Hat     55m
   example-catalog       Example Catalog       grpc   Example Org 2m25s
   redhat-marketplace    Red Hat Marketplace   grpc   Red Hat     55m
   redhat-operators      Red Hat Operators     grpc   Red Hat     55m

2. Use the oc describe command to get more details and status about a catalog source:

   $ oc describe catalogsource example-catalog -n openshift-marketplace

   Example output

   Name:         example-catalog
   Namespace:    openshift-marketplace
   Labels:       <none>
   Annotations:  operatorframework.io/managed-by: marketplace-operator
                 target.workload.openshift.io/management: {"effect": "PreferredDuringScheduling"}
   API Version:  operators.coreos.com/v1alpha1
   Kind:         CatalogSource
   # ...
   Status:
     Connection State:
       Address:              example-catalog.openshift-marketplace.svc:50051
       Last Connect:         2021-09-09T17:07:35Z
       Last Observed State:  TRANSIENT_FAILURE
     Registry Service:
       Created At:         2021-09-09T17:05:45Z
       Port:               50051
       Protocol:           grpc
       Service Name:       example-catalog
       Service Namespace:  openshift-marketplace
   # ...

   In the preceding example output, the last observed state is TRANSIENT_FAILURE. This state indicates that there is a problem establishing a connection for the catalog source.

3. List the pods in the namespace where your catalog source was created:

   $ oc get pods -n openshift-marketplace

   Example output

   NAME                                    READY   STATUS             RESTARTS   AGE
   certified-operators-cv9nn               1/1     Running            0          36m
   community-operators-6v8lp               1/1     Running            0          36m
   marketplace-operator-86bfc75f9b-jkgbc   1/1     Running            0          42m
   example-catalog-bwt8z                   0/1     ImagePullBackOff   0          3m55s
   redhat-marketplace-57p8c                1/1     Running            0          36m
   redhat-operators-smxx8                  1/1     Running            0          36m

   When a catalog source is created in a namespace, a pod for the catalog source is created in that namespace. In the preceding example output, the status for the example-catalog-bwt8z pod is ImagePullBackOff. This status indicates that there is an issue pulling the catalog source’s index image.

4. Use the oc describe command to inspect a pod for more detailed information:

   $ oc describe pod example-catalog-bwt8z -n openshift-marketplace

   Example output

   Name:         example-catalog-bwt8z
   Namespace:    openshift-marketplace
   Priority:     0
   Node:         ci-ln-jyryyg2-f76d1-ggdbq-worker-b-vsxjd/10.0.128.2
   ...
   Events:
     Type     Reason          Age                From               Message
     ----     ------          ----               ----               -------
     Normal   Scheduled       48s                default-scheduler  Successfully assigned openshift-marketplace/example-catalog-bwt8z to ci-ln-jyryyf2-f76d1-fgdbq-worker-b-vsxjd
     Normal   AddedInterface  47s                multus             Add eth0 [10.131.0.40/23] from openshift-sdn
     Normal   BackOff         20s (x2 over 46s)  kubelet            Back-off pulling image "quay.io/example-org/example-catalog:v1"
     Warning  Failed          20s (x2 over 46s)  kubelet            Error: ImagePullBackOff
     Normal   Pulling         8s (x3 over 47s)   kubelet            Pulling image "quay.io/example-org/example-catalog:v1"
     Warning  Failed          8s (x3 over 47s)   kubelet            Failed to pull image "quay.io/example-org/example-catalog:v1": rpc error: code = Unknown desc = reading manifest v1 in quay.io/example-org/example-catalog: unauthorized: access to the requested resource is not authorized
     Warning  Failed          8s (x3 over 47s)   kubelet            Error: ErrImagePull

   In the preceding example output, the error messages indicate that the catalog source’s index image is failing to pull successfully because of an authorization issue. For example, the index image might be stored in a registry that requires login credentials.

Procedure

1. List Operators running in the cluster. The output includes Operator version, availability, and up-time information:

   $ oc get clusteroperators

2. List Operator pods running in the Operator’s namespace, plus pod status, restarts, and age:

   $ oc get pod -n <operator_namespace>

3. Output a detailed Operator pod summary:

   $ oc describe pod <operator_pod_name> -n <operator_namespace>

4. If an Operator issue is node-specific, query Operator container status on that node.

   1. Start a debug pod for the node:

      $ oc debug node/my-node

   2. Set /host as the root directory within the debug shell. The debug pod mounts the host’s root file system in /host within the pod. By changing the root directory to /host, you can run binaries contained in the host’s executable paths:

      # chroot /host

      Note

      OpenShift Container Platform 4.15 cluster nodes running Red Hat Enterprise Linux CoreOS (RHCOS) are immutable and rely on Operators to apply cluster changes. Accessing cluster nodes by using SSH is not recommended. However, if the OpenShift Container Platform API is not available, or the kubelet is not properly functioning on the target node, oc operations will be impacted. In such situations, it is possible to access nodes using ssh core@<node>.<cluster_name>.<base_domain> instead.

   3. List details about the node’s containers, including state and associated pod IDs:

      # crictl ps

   4. List information about a specific Operator container on the node. The following example lists information about the network-operator container:

      # crictl ps --name network-operator

   5. Exit from the debug shell.

Procedure

1. List the Operator pods that are running in the Operator’s namespace, plus the pod status, restarts, and age:

   $ oc get pods -n <operator_namespace>

2. Review logs for an Operator pod:

   $ oc logs pod/<pod_name> -n <operator_namespace>

   If an Operator pod has multiple containers, the preceding command will produce an error that includes the name of each container. Query logs from an individual container:

   $ oc logs pod/<operator_pod_name> -c <container_name> -n <operator_namespace>

3. If the API is not functional, review Operator pod and container logs on each control plane node by using SSH instead. Replace <master-node>.<cluster_name>.<base_domain> with appropriate values.

   1. List pods on each control plane node:

      $ ssh core@<master-node>.<cluster_name>.<base_domain> sudo crictl pods

   2. For any Operator pods not showing a Ready status, inspect the pod’s status in detail. Replace <operator_pod_id> with the Operator pod’s ID listed in the output of the preceding command:

      $ ssh core@<master-node>.<cluster_name>.<base_domain> sudo crictl inspectp <operator_pod_id>

   3. List containers related to an Operator pod:

      $ ssh core@<master-node>.<cluster_name>.<base_domain> sudo crictl ps --pod=<operator_pod_id>

   4. For any Operator container not showing a Ready status, inspect the container’s status in detail. Replace <container_id> with a container ID listed in the output of the preceding command:

      $ ssh core@<master-node>.<cluster_name>.<base_domain> sudo crictl inspect <container_id>

   5. Review the logs for any Operator containers not showing a Ready status. Replace <container_id> with a container ID listed in the output of the preceding command:

      $ ssh core@<master-node>.<cluster_name>.<base_domain> sudo crictl logs -f <container_id>

      Note

      OpenShift Container Platform 4.15 cluster nodes running Red Hat Enterprise Linux CoreOS (RHCOS) are immutable and rely on Operators to apply cluster changes. Accessing cluster nodes by using SSH is not recommended. Before attempting to collect diagnostic data over SSH, review whether the data collected by running oc adm must gather and other oc commands is sufficient instead. However, if the OpenShift Container Platform API is not available, or the kubelet is not properly functioning on the target node, oc operations will be impacted. In such situations, it is possible to access nodes using ssh core@<node>.<cluster_name>.<base_domain>.

Procedure

1. Get the names of the Subscription and ClusterServiceVersion objects from the namespace where the Operator is installed:

   $ oc get sub,csv -n <namespace>

   Example output

   NAME                                                       PACKAGE                  SOURCE             CHANNEL
   subscription.operators.coreos.com/elasticsearch-operator   elasticsearch-operator   redhat-operators   5.0
   
   NAME                                                                         DISPLAY                            VERSION    REPLACES   PHASE
   clusterserviceversion.operators.coreos.com/elasticsearch-operator.5.0.0-65   OpenShift Elasticsearch Operator   5.0.0-65              Succeeded

2. Delete the subscription:

   $ oc delete subscription <subscription_name> -n <namespace>

3. Delete the cluster service version:

   $ oc delete csv <csv_name> -n <namespace>

4. Get the names of any failing jobs and related config maps in the openshift-marketplace namespace:

   $ oc get job,configmap -n openshift-marketplace

   Example output

   NAME                                                                        COMPLETIONS   DURATION   AGE
   job.batch/1de9443b6324e629ddf31fed0a853a121275806170e34c926d69e53a7fcbccb   1/1           26s        9m30s
   
   NAME                                                                        DATA   AGE
   configmap/1de9443b6324e629ddf31fed0a853a121275806170e34c926d69e53a7fcbccb   3      9m30s

5. Delete the job:

   $ oc delete job <job_name> -n openshift-marketplace

   This ensures pods that try to pull the inaccessible image are not recreated.

6. Delete the config map:

   $ oc delete configmap <configmap_name> -n openshift-marketplace

7. Reinstall the Operator using OperatorHub in the web console.

Procedure

1. Check if there are any namespaces related to the Operator that are stuck in "Terminating" state:

   $ oc get namespaces

   Example output

   operator-ns-1                                       Terminating

2. Check if there are any CRDs related to the Operator that are still present after the failed uninstallation:

   $ oc get crds

   Note

   CRDs are global cluster definitions; the actual custom resource (CR) instances related to the CRDs could be in other namespaces or be global cluster instances.

3. If there are any CRDs that you know were provided or managed by the Operator and that should have been deleted after uninstallation, delete the CRD:

   $ oc delete crd <crd_name>

4. Check if there are any remaining CR instances related to the Operator that are still present after uninstallation, and if so, delete the CRs:

   1. The type of CRs to search for can be difficult to determine after uninstallation and can require knowing what CRDs the Operator manages. For example, if you are troubleshooting an uninstallation of the etcd Operator, which provides the EtcdCluster CRD, you can search for remaining EtcdCluster CRs in a namespace:

      $ oc get EtcdCluster -n <namespace_name>

      Alternatively, you can search across all namespaces:

      $ oc get EtcdCluster --all-namespaces

   2. If there are any remaining CRs that should be removed, delete the instances:

      $ oc delete <cr_name> <cr_instance_name> -n <namespace_name>

5. Check that the namespace deletion has successfully resolved:

   $ oc get namespace <namespace_name>

   Important

   If the namespace or other Operator resources are still not uninstalled cleanly, contact Red Hat Support.

6. Reinstall the Operator using OperatorHub in the web console.