Operators

Procedure

1. Create a new project in the OpenShift Container Platform web console for this procedure. This example uses a project called my-etcd.

2. Navigate to the Operators → Installed Operators page. The Operators that have been installed to the cluster by the cluster administrator and are available for use are shown here as a list of cluster service versions (CSVs). CSVs are used to launch and manage the software provided by the Operator.

   Tip

   You can get this list from the CLI using:

   $ oc get csv

3. On the Installed Operators page, click the etcd Operator to view more details and available actions.

   As shown under Provided APIs, this Operator makes available three new resource types, including one for an etcd Cluster (the EtcdCluster resource). These objects work similar to the built-in native Kubernetes ones, such as Deployment or ReplicaSet, but contain logic specific to managing etcd.

4. Create a new etcd cluster:

   1. In the etcd Cluster API box, click Create instance.
   2. The next screen allows you to make any modifications to the minimal starting template of an EtcdCluster object, such as the size of the cluster. For now, click Create to finalize. This triggers the Operator to start up the pods, services, and other components of the new etcd cluster.

5. Click on the example etcd cluster, then click the Resources tab to see that your project now contains a number of resources created and configured automatically by the Operator.

   Verify that a Kubernetes service has been created that allows you to access the database from other pods in your project.

6. All users with the edit role in a given project can create, manage, and delete application instances (an etcd cluster, in this example) managed by Operators that have already been created in the project, in a self-service manner, just like a cloud service. If you want to enable additional users with this ability, project administrators can add the role using the following command:

   $ oc policy add-role-to-user edit <user> -n <target_project>

Procedure

1. Navigate in the web console to the Operators → OperatorHub page.

2. Scroll or type a keyword into the Filter by keyword box to find the Operator you want. For example, type advanced to find the Advanced Cluster Management for Kubernetes Operator.

   You can also filter options by Infrastructure Features. For example, select Disconnected if you want to see Operators that work in disconnected environments, also known as restricted network environments.

3. Select the Operator to display additional information.

   Note

   Choosing a Community Operator warns that Red Hat does not certify Community Operators; you must acknowledge the warning before continuing.

4. Read the information about the Operator and click Install.

5. On the Install Operator page:

   1. Choose a specific, single namespace in which to install the Operator. The Operator will only watch and be made available for use in this single namespace.
   2. Select an Update Channel (if more than one is available).
   3. Select Automatic or Manual approval strategy, as described earlier.

6. Click Install to make the Operator available to the selected namespaces on this OpenShift Container Platform cluster.

   1. If you selected a Manual approval strategy, the upgrade status of the subscription remains Upgrading until you review and approve the install plan.

      After approving on the Install Plan page, the subscription upgrade status moves to Up to date.

   2. If you selected an Automatic approval strategy, the upgrade status should resolve to Up to date without intervention.

7. After the upgrade status of the subscription is Up to date, select Operators → Installed Operators to verify that the cluster service version (CSV) of the installed Operator eventually shows up. The Status should ultimately resolve to InstallSucceeded in the relevant namespace.

   Note

   For the All namespaces…​ installation mode, the status resolves to InstallSucceeded in the openshift-operators namespace, but the status is Copied if you check in other namespaces.

   If it does not:

   1. Check the logs in any pods in the openshift-operators project (or other relevant namespace if A specific namespace…​ installation mode was selected) on the Workloads → Pods page that are reporting issues to troubleshoot further.

Procedure

1. View the list of Operators available to the cluster from OperatorHub:

   $ oc get packagemanifests -n openshift-marketplace

   Example output

   NAME                               CATALOG               AGE
   3scale-operator                    Red Hat Operators     91m
   advanced-cluster-management        Red Hat Operators     91m
   amq7-cert-manager                  Red Hat Operators     91m
   ...
   couchbase-enterprise-certified     Certified Operators   91m
   crunchy-postgres-operator          Certified Operators   91m
   mongodb-enterprise                 Certified Operators   91m
   ...
   etcd                               Community Operators   91m
   jaeger                             Community Operators   91m
   kubefed                            Community Operators   91m
   ...

   Note the catalog for your desired Operator.

2. Inspect your desired Operator to verify its supported install modes and available channels:

   $ oc describe packagemanifests <operator_name> -n openshift-marketplace

3. An Operator group, defined by an OperatorGroup object, selects target namespaces in which to generate required RBAC access for all Operators in the same namespace as the Operator group.

   The namespace to which you subscribe the Operator must have an Operator group that matches the install mode of the Operator, either the AllNamespaces or SingleNamespace mode. If the Operator you intend to install uses the AllNamespaces, then the openshift-operators namespace already has an appropriate Operator group in place.

   However, if the Operator uses the SingleNamespace mode and you do not already have an appropriate Operator group in place, you must create one.

   Note

   The web console version of this procedure handles the creation of the OperatorGroup and Subscription objects automatically behind the scenes for you when choosing SingleNamespace mode.

   1. Create an OperatorGroup object YAML file, for example operatorgroup.yaml:

      Example OperatorGroup object

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: <operatorgroup_name>
        namespace: <namespace>
      spec:
        targetNamespaces:
        - <namespace>

   2. Create the OperatorGroup object:

      $ oc apply -f operatorgroup.yaml

4. Create a Subscription object YAML file to subscribe a namespace to an Operator, for example sub.yaml:

   Example Subscription object

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: <subscription_name>
     namespace: openshift-operators 1
   spec:
     channel: <channel_name> 2
     name: <operator_name> 3
     source: redhat-operators 4
     sourceNamespace: openshift-marketplace 5
     config:
       env: 6
       - name: ARGS
         value: "-v=10"
       envFrom: 7
       - secretRef:
           name: license-secret
       volumes: 8
       - name: <volume_name>
         configMap:
           name: <configmap_name>
       volumeMounts: 9
       - mountPath: <directory_name>
         name: <volume_name>
       tolerations: 10
       - operator: "Exists"
       resources: 11
         requests:
           memory: "64Mi"
           cpu: "250m"
         limits:
           memory: "128Mi"
           cpu: "500m"
       nodeSelector: 12
         foo: bar

   1

   For AllNamespaces install mode usage, specify the openshift-operators namespace. Otherwise, specify the relevant single namespace for SingleNamespace install mode usage.

   2

   Name of the channel to subscribe to.

   3

   Name of the Operator to subscribe to.

   4

   Name of the catalog source that provides the Operator.

   5

   Namespace of the catalog source. Use openshift-marketplace for the default OperatorHub catalog sources.

   6

   The env parameter defines a list of Environment Variables that must exist in all containers in the pod created by OLM.

   7

   The envFrom parameter defines a list of sources to populate Environment Variables in the container.

   8

   The volumes parameter defines a list of Volumes that must exist on the pod created by OLM.

   9

   The volumeMounts parameter defines a list of VolumeMounts that must exist in all containers in the pod created by OLM. If a volumeMount references a volume that does not exist, OLM fails to deploy the Operator.

   10

   The tolerations parameter defines a list of Tolerations for the pod created by OLM.

   11

   The resources parameter defines resource constraints for all the containers in the pod created by OLM.

   12

   The nodeSelector parameter defines a NodeSelector for the pod created by OLM.

5. Create the Subscription object:

   $ oc apply -f sub.yaml

   At this point, OLM is now aware of the selected Operator. A cluster service version (CSV) for the Operator should appear in the target namespace, and APIs provided by the Operator should be available for creation.

Procedure

1. Create a Subscription object YAML file that subscribes a namespace to an Operator with a specific version by setting the startingCSV field. Set the installPlanApproval field to Manual to prevent the Operator from automatically upgrading if a later version exists in the catalog.

   For example, the following sub.yaml file can be used to install the Red Hat Quay Operator specifically to version 3.4.0:

   Subscription with a specific starting Operator version

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: quay-operator
     namespace: quay
   spec:
     channel: quay-v3.4
     installPlanApproval: Manual 1
     name: quay-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     startingCSV: quay-operator.v3.4.0 2

   1

   Set the approval strategy to Manual in case your specified version is superseded by a later version in the catalog. This plan prevents an automatic upgrade to a later version and requires manual approval before the starting CSV can complete the installation.

   2

   Set a specific version of an Operator CSV.

2. Create the Subscription object:

   $ oc apply -f sub.yaml

3. Manually approve the pending install plan to complete the Operator installation.

Procedure

1. Navigate in the web console to the Operators → OperatorHub page.

2. Scroll or type a keyword into the Filter by keyword box to find the Operator you want. For example, type advanced to find the Advanced Cluster Management for Kubernetes Operator.

   You can also filter options by Infrastructure Features. For example, select Disconnected if you want to see Operators that work in disconnected environments, also known as restricted network environments.

3. Select the Operator to display additional information.

   Note

   Choosing a Community Operator warns that Red Hat does not certify Community Operators; you must acknowledge the warning before continuing.

4. Read the information about the Operator and click Install.

5. On the Install Operator page:

   1. Select one of the following:

      • All namespaces on the cluster (default) installs the Operator in the default openshift-operators namespace to watch and be made available to all namespaces in the cluster. This option is not always available.
      • A specific namespace on the cluster allows you to choose a specific, single namespace in which to install the Operator. The Operator will only watch and be made available for use in this single namespace.
   2. Choose a specific, single namespace in which to install the Operator. The Operator will only watch and be made available for use in this single namespace.
   3. Select an Update Channel (if more than one is available).
   4. Select Automatic or Manual approval strategy, as described earlier.

6. Click Install to make the Operator available to the selected namespaces on this OpenShift Container Platform cluster.

   1. If you selected a Manual approval strategy, the upgrade status of the subscription remains Upgrading until you review and approve the install plan.

      After approving on the Install Plan page, the subscription upgrade status moves to Up to date.

   2. If you selected an Automatic approval strategy, the upgrade status should resolve to Up to date without intervention.

7. After the upgrade status of the subscription is Up to date, select Operators → Installed Operators to verify that the cluster service version (CSV) of the installed Operator eventually shows up. The Status should ultimately resolve to InstallSucceeded in the relevant namespace.

   Note

   For the All namespaces…​ installation mode, the status resolves to InstallSucceeded in the openshift-operators namespace, but the status is Copied if you check in other namespaces.

   If it does not:

   1. Check the logs in any pods in the openshift-operators project (or other relevant namespace if A specific namespace…​ installation mode was selected) on the Workloads → Pods page that are reporting issues to troubleshoot further.

Procedure

1. View the list of Operators available to the cluster from OperatorHub:

   $ oc get packagemanifests -n openshift-marketplace

   Example output

   NAME                               CATALOG               AGE
   3scale-operator                    Red Hat Operators     91m
   advanced-cluster-management        Red Hat Operators     91m
   amq7-cert-manager                  Red Hat Operators     91m
   ...
   couchbase-enterprise-certified     Certified Operators   91m
   crunchy-postgres-operator          Certified Operators   91m
   mongodb-enterprise                 Certified Operators   91m
   ...
   etcd                               Community Operators   91m
   jaeger                             Community Operators   91m
   kubefed                            Community Operators   91m
   ...

   Note the catalog for your desired Operator.

2. Inspect your desired Operator to verify its supported install modes and available channels:

   $ oc describe packagemanifests <operator_name> -n openshift-marketplace

3. An Operator group, defined by an OperatorGroup object, selects target namespaces in which to generate required RBAC access for all Operators in the same namespace as the Operator group.

   The namespace to which you subscribe the Operator must have an Operator group that matches the install mode of the Operator, either the AllNamespaces or SingleNamespace mode. If the Operator you intend to install uses the AllNamespaces, then the openshift-operators namespace already has an appropriate Operator group in place.

   However, if the Operator uses the SingleNamespace mode and you do not already have an appropriate Operator group in place, you must create one.

   Note

   The web console version of this procedure handles the creation of the OperatorGroup and Subscription objects automatically behind the scenes for you when choosing SingleNamespace mode.

   1. Create an OperatorGroup object YAML file, for example operatorgroup.yaml:

      Example OperatorGroup object

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: <operatorgroup_name>
        namespace: <namespace>
      spec:
        targetNamespaces:
        - <namespace>

   2. Create the OperatorGroup object:

      $ oc apply -f operatorgroup.yaml

4. Create a Subscription object YAML file to subscribe a namespace to an Operator, for example sub.yaml:

   Example Subscription object

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: <subscription_name>
     namespace: openshift-operators 1
   spec:
     channel: <channel_name> 2
     name: <operator_name> 3
     source: redhat-operators 4
     sourceNamespace: openshift-marketplace 5
     config:
       env: 6
       - name: ARGS
         value: "-v=10"
       envFrom: 7
       - secretRef:
           name: license-secret
       volumes: 8
       - name: <volume_name>
         configMap:
           name: <configmap_name>
       volumeMounts: 9
       - mountPath: <directory_name>
         name: <volume_name>
       tolerations: 10
       - operator: "Exists"
       resources: 11
         requests:
           memory: "64Mi"
           cpu: "250m"
         limits:
           memory: "128Mi"
           cpu: "500m"
       nodeSelector: 12
         foo: bar

   1

   For AllNamespaces install mode usage, specify the openshift-operators namespace. Otherwise, specify the relevant single namespace for SingleNamespace install mode usage.

   2

   Name of the channel to subscribe to.

   3

   Name of the Operator to subscribe to.

   4

   Name of the catalog source that provides the Operator.

   5

   Namespace of the catalog source. Use openshift-marketplace for the default OperatorHub catalog sources.

   6

   The env parameter defines a list of Environment Variables that must exist in all containers in the pod created by OLM.

   7

   The envFrom parameter defines a list of sources to populate Environment Variables in the container.

   8

   The volumes parameter defines a list of Volumes that must exist on the pod created by OLM.

   9

   The volumeMounts parameter defines a list of VolumeMounts that must exist in all containers in the pod created by OLM. If a volumeMount references a volume that does not exist, OLM fails to deploy the Operator.

   10

   The tolerations parameter defines a list of Tolerations for the pod created by OLM.

   11

   The resources parameter defines resource constraints for all the containers in the pod created by OLM.

   12

   The nodeSelector parameter defines a NodeSelector for the pod created by OLM.

5. Create the Subscription object:

   $ oc apply -f sub.yaml

   At this point, OLM is now aware of the selected Operator. A cluster service version (CSV) for the Operator should appear in the target namespace, and APIs provided by the Operator should be available for creation.

Procedure

1. Create a Subscription object YAML file that subscribes a namespace to an Operator with a specific version by setting the startingCSV field. Set the installPlanApproval field to Manual to prevent the Operator from automatically upgrading if a later version exists in the catalog.

   For example, the following sub.yaml file can be used to install the Red Hat Quay Operator specifically to version 3.4.0:

   Subscription with a specific starting Operator version

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: quay-operator
     namespace: quay
   spec:
     channel: quay-v3.4
     installPlanApproval: Manual 1
     name: quay-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     startingCSV: quay-operator.v3.4.0 2

   1

   Set the approval strategy to Manual in case your specified version is superseded by a later version in the catalog. This plan prevents an automatic upgrade to a later version and requires manual approval before the starting CSV can complete the installation.

   2

   Set a specific version of an Operator CSV.

2. Create the Subscription object:

   $ oc apply -f sub.yaml

3. Manually approve the pending install plan to complete the Operator installation.

Procedure

1. In the Administrator perspective of the OpenShift Container Platform web console, navigate to Operators → Installed Operators.
2. Click the name of the Operator you want to change the update channel for.
3. Click the Subscription tab.
4. Click the name of the update channel under Channel.
5. Click the newer update channel that you want to change to, then click Save.

6. For subscriptions with an Automatic approval strategy, the upgrade begins automatically. Navigate back to the Operators → Installed Operators page to monitor the progress of the upgrade. When complete, the status changes to Succeeded and Up to date.

   For subscriptions with a Manual approval strategy, you can manually approve the upgrade from the Subscription tab.

Procedure

1. In the Administrator perspective of the OpenShift Container Platform web console, navigate to Operators → Installed Operators.
2. Operators that have a pending upgrade display a status with Upgrade available. Click the name of the Operator you want to upgrade.
3. Click the Subscription tab. Any upgrades requiring approval are displayed next to Upgrade Status. For example, it might display 1 requires approval.
4. Click 1 requires approval, then click Preview Install Plan.
5. Review the resources that are listed as available for upgrade. When satisfied, click Approve.
6. Navigate back to the Operators → Installed Operators page to monitor the progress of the upgrade. When complete, the status changes to Succeeded and Up to date.

Procedure

1. From the Operators → Installed Operators page, scroll or type a keyword into the Filter by name to find the Operator you want. Then, click on it.

2. On the right side of the Operator Details page, select Uninstall Operator from the Actions list.

   An Uninstall Operator? dialog box is displayed, reminding you that:

   Removing the Operator will not remove any of its custom resource definitions or managed resources. If your Operator has deployed applications on the cluster or configured off-cluster resources, these will continue to run and need to be cleaned up manually.

   This action removes the Operator as well as the Operator deployments and pods, if any. Any Operands, and resources managed by the Operator, including CRDs and CRs, are not removed. The web console enables dashboards and navigation items for some Operators. To remove these after uninstalling the Operator, you might need to manually delete the Operator CRDs.

3. Select Uninstall. This Operator stops running and no longer receives updates.

Procedure

1. Check the current version of the subscribed Operator (for example, jaeger) in the currentCSV field:

   $ oc get subscription jaeger -n openshift-operators -o yaml | grep currentCSV

   Example output

     currentCSV: jaeger-operator.v1.8.2

2. Delete the subscription (for example, jaeger):

   $ oc delete subscription jaeger -n openshift-operators

   Example output

   subscription.operators.coreos.com "jaeger" deleted

3. Delete the CSV for the Operator in the target namespace using the currentCSV value from the previous step:

   $ oc delete clusterserviceversion jaeger-operator.v1.8.2 -n openshift-operators

   Example output

   clusterserviceversion.operators.coreos.com "jaeger-operator.v1.8.2" deleted

Procedure

1. Get the names of the Subscription and ClusterServiceVersion objects from the namespace where the Operator is installed:

   $ oc get sub,csv -n <namespace>

   Example output

   NAME                                                       PACKAGE                  SOURCE             CHANNEL
   subscription.operators.coreos.com/elasticsearch-operator   elasticsearch-operator   redhat-operators   5.0
   
   NAME                                                                         DISPLAY                            VERSION    REPLACES   PHASE
   clusterserviceversion.operators.coreos.com/elasticsearch-operator.5.0.0-65   OpenShift Elasticsearch Operator   5.0.0-65              Succeeded

2. Delete the subscription:

   $ oc delete subscription <subscription_name> -n <namespace>

3. Delete the cluster service version:

   $ oc delete csv <csv_name> -n <namespace>

4. Get the names of any failing jobs and related config maps in the openshift-marketplace namespace:

   $ oc get job,configmap -n openshift-marketplace

   Example output

   NAME                                                                        COMPLETIONS   DURATION   AGE
   job.batch/1de9443b6324e629ddf31fed0a853a121275806170e34c926d69e53a7fcbccb   1/1           26s        9m30s
   
   NAME                                                                        DATA   AGE
   configmap/1de9443b6324e629ddf31fed0a853a121275806170e34c926d69e53a7fcbccb   3      9m30s

5. Delete the job:

   $ oc delete job <job_name> -n openshift-marketplace

   This ensures pods that try to pull the inaccessible image are not recreated.

6. Delete the config map:

   $ oc delete configmap <configmap_name> -n openshift-marketplace

7. Reinstall the Operator using OperatorHub in the web console.

Procedure

1. Navigate in the web console to the Operators → OperatorHub page.
2. Select the Operator and click Install.

3. On the Install Operator page, modify the Subscription object to include one or more of the following environment variables in the spec section:

   • HTTP_PROXY
   • HTTPS_PROXY
   • NO_PROXY

   For example:

   Subscription object with proxy setting overrides

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: etcd-config-test
     namespace: openshift-operators
   spec:
     config:
       env:
       - name: HTTP_PROXY
         value: test_http
       - name: HTTPS_PROXY
         value: test_https
       - name: NO_PROXY
         value: test
     channel: clusterwide-alpha
     installPlanApproval: Automatic
     name: etcd
     source: community-operators
     sourceNamespace: openshift-marketplace
     startingCSV: etcdoperator.v0.9.4-clusterwide

   Note

   These environment variables can also be unset using an empty value to remove any previously set cluster-wide or custom proxy settings.

   OLM handles these environment variables as a unit; if at least one of them is set, all three are considered overridden and the cluster-wide defaults are not used for the deployments of the subscribed Operator.

4. Click Install to make the Operator available to the selected namespaces.

5. After the CSV for the Operator appears in the relevant namespace, you can verify that custom proxy environment variables are set in the deployment. For example, using the CLI:

   $ oc get deployment -n openshift-operators \
       etcd-operator -o yaml \
       | grep -i "PROXY" -A 2

   Example output

           - name: HTTP_PROXY
             value: test_http
           - name: HTTPS_PROXY
             value: test_https
           - name: NO_PROXY
             value: test
           image: quay.io/coreos/etcd-operator@sha256:66a37fd61a06a43969854ee6d3e21088a98b93838e284a6086b13917f96b0d9c
   ...

Procedure

1. Create an empty config map in the namespace where the subscription for your Operator exists and include the following label:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: trusted-ca 1
     labels:
       config.openshift.io/inject-trusted-cabundle: "true" 2

   1

   Name of the config map.

   2

   Requests the Cluster Network Operator to inject the merged bundle.

   After creating this config map, it is immediately populated with the certificate contents of the merged bundle.

2. Update your the Subscription object to include a spec.config section that mounts the trusted-ca config map as a volume to each container within a pod that requires a custom CA:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: my-operator
   spec:
     package: etcd
     channel: alpha
     config: 1
       selector:
         matchLabels:
           <labels_for_pods> 2
       volumes: 3
       - name: trusted-ca
         configMap:
           name: trusted-ca
           items:
             - key: ca-bundle.crt 4
               path: tls-ca-bundle.pem 5
       volumeMounts: 6
       - name: trusted-ca
         mountPath: /etc/pki/ca-trust/extracted/pem
         readOnly: true

   1

   Add a config section if it does not exist.

   2

   Specify labels to match pods that are owned by the Operator.

   3

   Create a trusted-ca volume.

   4

   ca-bundle.crt is required as the config map key.

   5

   tls-ca-bundle.pem is required as the config map path.

   6

   Create a trusted-ca volume mount.

Procedure

1. List Operator subscriptions:

   $ oc get subs -n <operator_namespace>

2. Use the oc describe command to inspect a Subscription resource:

   $ oc describe sub <subscription_name> -n <operator_namespace>

3. In the command output, find the Conditions section for the status of Operator subscription condition types. In the following example, the CatalogSourcesUnhealthy condition type has a status of false because all available catalog sources are healthy:

   Example output

   Conditions:
      Last Transition Time:  2019-07-29T13:42:57Z
      Message:               all available catalogsources are healthy
      Reason:                AllCatalogSourcesHealthy
      Status:                False
      Type:                  CatalogSourcesUnhealthy

Procedure

1. List the catalog sources in a namespace. For example, you can check the openshift-marketplace namespace, which is used for cluster-wide catalog sources:

   $ oc get catalogsources -n openshift-marketplace

   Example output

   NAME                  DISPLAY               TYPE   PUBLISHER   AGE
   certified-operators   Certified Operators   grpc   Red Hat     55m
   community-operators   Community Operators   grpc   Red Hat     55m
   example-catalog       Example Catalog       grpc   Example Org 2m25s
   redhat-marketplace    Red Hat Marketplace   grpc   Red Hat     55m
   redhat-operators      Red Hat Operators     grpc   Red Hat     55m

2. Use the oc describe command to get more details and status about a catalog source:

   $ oc describe catalogsource example-catalog -n openshift-marketplace

   Example output

   Name:         example-catalog
   Namespace:    openshift-marketplace
   ...
   Status:
     Connection State:
       Address:              example-catalog.openshift-marketplace.svc:50051
       Last Connect:         2021-09-09T17:07:35Z
       Last Observed State:  TRANSIENT_FAILURE
     Registry Service:
       Created At:         2021-09-09T17:05:45Z
       Port:               50051
       Protocol:           grpc
       Service Name:       example-catalog
       Service Namespace:  openshift-marketplace

   In the preceding example output, the last observed state is TRANSIENT_FAILURE. This state indicates that there is a problem establishing a connection for the catalog source.

3. List the pods in the namespace where your catalog source was created:

   $ oc get pods -n openshift-marketplace

   Example output

   NAME                                    READY   STATUS             RESTARTS   AGE
   certified-operators-cv9nn               1/1     Running            0          36m
   community-operators-6v8lp               1/1     Running            0          36m
   marketplace-operator-86bfc75f9b-jkgbc   1/1     Running            0          42m
   example-catalog-bwt8z                   0/1     ImagePullBackOff   0          3m55s
   redhat-marketplace-57p8c                1/1     Running            0          36m
   redhat-operators-smxx8                  1/1     Running            0          36m

   When a catalog source is created in a namespace, a pod for the catalog source is created in that namespace. In the preceding example output, the status for the example-catalog-bwt8z pod is ImagePullBackOff. This status indicates that there is an issue pulling the catalog source’s index image.

4. Use the oc describe command to inspect a pod for more detailed information:

   $ oc describe pod example-catalog-bwt8z -n openshift-marketplace

   Example output

   Name:         example-catalog-bwt8z
   Namespace:    openshift-marketplace
   Priority:     0
   Node:         ci-ln-jyryyg2-f76d1-ggdbq-worker-b-vsxjd/10.0.128.2
   ...
   Events:
     Type     Reason          Age                From               Message
     ----     ------          ----               ----               -------
     Normal   Scheduled       48s                default-scheduler  Successfully assigned openshift-marketplace/example-catalog-bwt8z to ci-ln-jyryyf2-f76d1-fgdbq-worker-b-vsxjd
     Normal   AddedInterface  47s                multus             Add eth0 [10.131.0.40/23] from openshift-sdn
     Normal   BackOff         20s (x2 over 46s)  kubelet            Back-off pulling image "quay.io/example-org/example-catalog:v1"
     Warning  Failed          20s (x2 over 46s)  kubelet            Error: ImagePullBackOff
     Normal   Pulling         8s (x3 over 47s)   kubelet            Pulling image "quay.io/example-org/example-catalog:v1"
     Warning  Failed          8s (x3 over 47s)   kubelet            Failed to pull image "quay.io/example-org/example-catalog:v1": rpc error: code = Unknown desc = reading manifest v1 in quay.io/example-org/example-catalog: unauthorized: access to the requested resource is not authorized
     Warning  Failed          8s (x3 over 47s)   kubelet            Error: ErrImagePull

   In the preceding example output, the error messages indicate that the catalog source’s index image is failing to pull successfully because of an authorization issue. For example, the index image might be stored in a registry that requires login credentials.

Procedure

1. Create a new namespace:

   $ cat <<EOF | oc create -f -
   apiVersion: v1
   kind: Namespace
   metadata:
     name: scoped
   EOF

2. Allocate permissions that you want the Operator(s) to be confined to. This involves creating a new service account, relevant role(s), and role binding(s).

   $ cat <<EOF | oc create -f -
   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: scoped
     namespace: scoped
   EOF

   The following example grants the service account permissions to do anything in the designated namespace for simplicity. In a production environment, you should create a more fine-grained set of permissions:

   $ cat <<EOF | oc create -f -
   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: scoped
     namespace: scoped
   rules:
   - apiGroups: ["*"]
     resources: ["*"]
     verbs: ["*"]
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: scoped-bindings
     namespace: scoped
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: Role
     name: scoped
   subjects:
   - kind: ServiceAccount
     name: scoped
     namespace: scoped
   EOF

3. Create an OperatorGroup object in the designated namespace. This Operator group targets the designated namespace to ensure that its tenancy is confined to it.

   In addition, Operator groups allow a user to specify a service account. Specify the service account created in the previous step:

   $ cat <<EOF | oc create -f -
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: scoped
     namespace: scoped
   spec:
     serviceAccountName: scoped
     targetNamespaces:
     - scoped
   EOF

   Any Operator installed in the designated namespace is tied to this Operator group and therefore to the service account specified.

4. Create a Subscription object in the designated namespace to install an Operator:

   $ cat <<EOF | oc create -f -
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: etcd
     namespace: scoped
   spec:
     channel: singlenamespace-alpha
     name: etcd
     source: <catalog_source_name> 1
     sourceNamespace: <catalog_source_namespace> 2
   EOF

   1

   Specify a catalog source that already exists in the designated namespace or one that is in the global catalog namespace.

   2

   Specify a namespace where the catalog source was created.

   Any Operator tied to this Operator group is confined to the permissions granted to the specified service account. If the Operator requests permissions that are outside the scope of the service account, the installation fails with relevant errors.

Procedure

1. Review the Subscription object. Its status has an object reference installPlanRef that points to the InstallPlan object that attempted to create the necessary [Cluster]Role[Binding] object(s) for the Operator:

   apiVersion: operators.coreos.com/v1
   kind: Subscription
   metadata:
     name: etcd
     namespace: scoped
   status:
     installPlanRef:
       apiVersion: operators.coreos.com/v1
       kind: InstallPlan
       name: install-4plp8
       namespace: scoped
       resourceVersion: "117359"
       uid: 2c1df80e-afea-11e9-bce3-5254009c9c23

2. Check the status of the InstallPlan object for any errors:

   apiVersion: operators.coreos.com/v1
   kind: InstallPlan
   status:
     conditions:
     - lastTransitionTime: "2019-07-26T21:13:10Z"
       lastUpdateTime: "2019-07-26T21:13:10Z"
       message: 'error creating clusterrole etcdoperator.v0.9.4-clusterwide-dsfx4: clusterroles.rbac.authorization.k8s.io
         is forbidden: User "system:serviceaccount:scoped:scoped" cannot create resource
         "clusterroles" in API group "rbac.authorization.k8s.io" at the cluster scope'
       reason: InstallComponentFailed
       status: "False"
       type: Installed
     phase: Failed

   The error message tells you:

   • The type of resource it failed to create, including the API group of the resource. In this case, it was clusterroles in the rbac.authorization.k8s.io group.
   • The name of the resource.
   • The type of error: is forbidden tells you that the user does not have enough permission to do the operation.
   • The name of the user who attempted to create or update the resource. In this case, it refers to the service account specified in the Operator group.

   • The scope of the operation: cluster scope or not.

     The user can add the missing permission to the service account and then iterate.

     Note

     Operator Lifecycle Manager (OLM) does not currently provide the complete list of errors on the first try.

Procedure

1. In the Administrator perspective of the web console, navigate to Administration → Cluster Settings.
2. Click the Global Configuration tab, and then click OperatorHub.
3. Click the Sources tab.
4. Select the Options menu for the catalog that you want to remove, and then click Delete CatalogSource.

Procedure

1. Authenticate with registry.redhat.io:

   $ podman login registry.redhat.io

2. Authenticate with your target registry:

   $ podman login <target_registry>

3. Determine the list of packages you want to include in your pruned index.

   1. Run the source index image that you want to prune in a container. For example:

      $ podman run -p50051:50051 \
          -it registry.redhat.io/redhat/redhat-operator-index:v4.6

      Example output

      Trying to pull registry.redhat.io/redhat/redhat-operator-index:v4.6...
      Getting image source signatures
      Copying blob ae8a0c23f5b1 done
      ...
      INFO[0000] serving registry                              database=/database/index.db port=50051

   2. In a separate terminal session, use the grpcurl command to get a list of the packages provided by the index:

      $ grpcurl -plaintext localhost:50051 api.Registry/ListPackages > packages.out

   3. Inspect the packages.out file and identify which package names from this list you want to keep in your pruned index. For example:

      Example snippets of packages list

      ...
      {
        "name": "advanced-cluster-management"
      }
      ...
      {
        "name": "jaeger-product"
      }
      ...
      {
      {
        "name": "quay-operator"
      }
      ...

   4. In the terminal session where you executed the podman run command, press Ctrl and C to stop the container process.

4. Run the following command to prune the source index of all but the specified packages:

   $ opm index prune \
       -f registry.redhat.io/redhat/redhat-operator-index:v4.6 \1
       -p advanced-cluster-management,jaeger-product,quay-operator \2
       [-i registry.redhat.io/openshift4/ose-operator-registry:v4.6] \3
       -t <target_registry>:<port>/<namespace>/redhat-operator-index:v4.6 4

   1

   Index to prune.

   2

   Comma-separated list of packages to keep.

   3

   Required only for IBM Power Systems and IBM Z images: Operator Registry base image with the tag that matches the target OpenShift Container Platform cluster major and minor version.

   4

   Custom tag for new index image being built.

5. Run the following command to push the new index image to your target registry:

   $ podman push <target_registry>:<port>/<namespace>/redhat-operator-index:v4.6

   where <namespace> is any existing namespace on the registry. For example, you might create an olm-mirror namespace to push all mirrored content to.

Procedure

1. If you want to mirror a Red Hat-provided catalog, run the following command on your workstation with unrestricted network access to authenticate with registry.redhat.io:

   $ podman login registry.redhat.io

2. The oc adm catalog mirror command extracts the contents of an index image to generate the manifests required for mirroring. The default behavior of the command generates manifests, then automatically mirrors all of the image content from the index image, as well as the index image itself, to your mirror registry. Alternatively, if your mirror registry is on a completely disconnected, or airgapped, host, you can first mirror the content to removable media, move the media to the disconnected environment, then mirror the content from the media to the registry.

   • Option A: If your mirror registry is on the same network as your workstation with unrestricted network access, take the following actions on your workstation:

     1. If your mirror registry requires authentication, run the following command to log in to the registry:

        $ podman login <mirror_registry>

     2. Run the following command to mirror the content:

        $ oc adm catalog mirror \
            <index_image> \1
            <mirror_registry>:<port>/<namespace> \2
            [-a ${REG_CREDS}] \3
            [--insecure] \4
            [--index-filter-by-os='<platform>/<arch>'] \5
            [--manifests-only] 6

        1

        Specify the index image for the catalog you want to mirror. For example, this might be a pruned index image that you created previously, or one of the source index images for the default catalogs, such as registry.redhat.io/redhat/redhat-operator-index:v4.6.

        2

        Specify the fully qualified domain name (FQDN) for the target registry and namespace to mirror the Operator content to, where <namespace> is any existing namespace on the registry. For example, you might create an olm-mirror namespace to push all mirrored content to.

        3

        Optional: If required, specify the location of your registry credentials file. {REG_CREDS} is required for registry.redhat.io.

        4

        Optional: If you do not want to configure trust for the target registry, add the --insecure flag.

        5

        Optional: Specify which platform and architecture of the index image to select when multiple variants are available. Images are passed as '<platform>/<arch>[/<variant>]'. This does not apply to images referenced by the index. Valid values are linux/amd64, linux/ppc64le, and linux/s390x.

        6

        Optional: Generate only the manifests required for mirroring, and do not actually mirror the image content to a registry. This option can be useful for reviewing what will be mirrored, and it allows you to make any changes to the mapping list if you require only a subset of packages. You can then use the mapping.txt file with the oc image mirror command to mirror the modified list of images in a later step. This flag is intended for only advanced selective mirroring of content from the catalog; the opm index prune command, if you used it previously to prune the index image, is suitable for most catalog management use cases.

        Example output

        src image has index label for database path: /database/index.db
        using database path mapping: /database/index.db:/tmp/153048078
        wrote database to /tmp/153048078 1
        ...
        wrote mirroring manifests to manifests-redhat-operator-index-1614211642 2

        1

        Directory for the temporary index.db database generated by the command.

        2

        Be sure to record the manifests directory name that is generated. This directory name is used in a later step.

        Note

        Red Hat Quay does not support nested repositories. As a result, running the oc adm catalog mirror command will fail with a 401 unauthorized error. As a workaround, you can use the --max-components=2 option when running the oc adm catalog mirror command to disable the creation of nested repositories. For more information on this workaround, see the Unauthorized error thrown while using catalog mirror command with Quay registry Knowledgebase Solution article.

   • Option B: If your mirror registry is on a disconnected host, take the following actions.

     1. Run the following command on your workstation with unrestricted network access to mirror the content to local files:

        $ oc adm catalog mirror \
            <index_image> \1
            file:///local/index \2
            [-a ${REG_CREDS}] \
            [--insecure]

        1

        Specify the index image for the catalog you want to mirror. For example, this might be a pruned index image that you created previously, or one of the source index images for the default catalogs, such as registry.redhat.io/redhat/redhat-operator-index:v4.6.

        2

        Mirrors content to local files in your current directory.

        Example output

        ...
        info: Mirroring completed in 5.93s (5.915MB/s)
        wrote mirroring manifests to manifests-my-index-1614985528 1
        
        To upload local images to a registry, run:
        
        	oc adm catalog mirror file://local/index/myrepo/my-index:v1 REGISTRY/REPOSITORY 2

        1

        Be sure to record the manifests directory name that is generated. This directory name is used in a later step.

        2

        Record the expanded file:// path that based on your provided index image. This path is used in a later step.

     2. Copy the v2/ directory that is generated in your current directory to removable media.
     3. Physically remove the media and attach it to a host in the disconnected environment that has access to the mirror registry.

     4. If your mirror registry requires authentication, run the following command on your host in the disconnected environment to log in to the registry:

        $ podman login <mirror_registry>

     5. Run the following command from the parent directory containing the v2/ directory to upload the images from local files to the mirror registry:

        $ oc adm catalog mirror \
            file://local/index/<repo>/<index_image>:<tag> \1
            <mirror_registry>:<port>/<namespace> \2
            [-a ${REG_CREDS}] \
            [--insecure]

        1

        Specify the file:// path from the previous command output.

        2

        Specify the fully qualified domain name (FQDN) for the target registry and namespace to mirror the Operator content to, where <namespace> is any existing namespace on the registry. For example, you might create an olm-mirror namespace to push all mirrored content to.

        Note

        Red Hat Quay does not support nested repositories. As a result, running the oc adm catalog mirror command will fail with a 401 unauthorized error. As a workaround, you can use the --max-components=2 option when running the oc adm catalog mirror command to disable the creation of nested repositories. For more information on this workaround, see the Unauthorized error thrown while using catalog mirror command with Quay registry Knowledgebase Solution article.

3. After mirroring the content to your registry, inspect the manifests directory that is generated in your current directory.

   Note

   The manifests directory name is used in a later step.

   If you mirrored content to a registry on the same network in the previous step, the directory name takes the following form:

   manifests-<index_image_name>-<random_number>

   If you mirrored content to a registry on a disconnected host in the previous step, the directory name takes the following form:

   manifests-index/<namespace>/<index_image_name>-<random_number>

   The manifests directory contains the following files, some of which might require further modification:

   • The catalogSource.yaml file is a basic definition for a CatalogSource object that is pre-populated with your index image tag and other relevant metadata. This file can be used as is or modified to add the catalog source to your cluster.

     Important

     If you mirrored the content to local files, you must modify your catalogSource.yaml file to remove any backslash (/) characters from the metadata.name field. Otherwise, when you attempt to create the object, it fails with an "invalid resource name" error.

   • The imageContentSourcePolicy.yaml file defines an ImageContentSourcePolicy object that can configure nodes to translate between the image references stored in Operator manifests and the mirrored registry.

     Note

     If your cluster uses an ImageContentSourcePolicy object to configure repository mirroring, you can use only global pull secrets for mirrored registries. You cannot add a pull secret to a project.

   • The mapping.txt file contains all of the source images and where to map them in the target registry. This file is compatible with the oc image mirror command and can be used to further customize the mirroring configuration.

     Important

     If you used the --manifests-only flag during the mirroring process and want to further trim the subset of packages to be mirrored, see the steps in the "Mirroring a Package Manifest Format catalog image" procedure about modifying your mapping.txt file and using the file with the oc image mirror command. After following those further actions, you can continue this procedure.

4. On a host with access to the disconnected cluster, create the ImageContentSourcePolicy object by running the following command to specify the imageContentSourcePolicy.yaml file in your manifests directory:

   $ oc create -f <path/to/manifests/dir>/imageContentSourcePolicy.yaml

   where <path/to/manifests/dir> is the path to the manifests directory for your mirrored content.

Procedure

1. Create a CatalogSource object that references your index image. If you used the oc adm catalog mirror command to mirror your catalog to a target registry, you can use the generated catalogSource.yaml file as a starting point.

   1. Modify the following to your specifications and save it as a catalogSource.yaml file:

      apiVersion: operators.coreos.com/v1alpha1
      kind: CatalogSource
      metadata:
        name: my-operator-catalog 1
        namespace: openshift-marketplace 2
      spec:
        sourceType: grpc
        image: <registry>:<port>/<namespace>/redhat-operator-index:v4.6 3
        displayName: My Operator Catalog
        publisher: <publisher_name> 4
        updateStrategy:
          registryPoll: 5
            interval: 30m

      1

      If you mirrored content to local files before uploading to a registry, remove any backslash (/) characters from the metadata.name field to avoid an "invalid resource name" error when you create the object.

      2

      If you want the catalog source to be available globally to users in all namespaces, specify the openshift-marketplace namespace. Otherwise, you can specify a different namespace for the catalog to be scoped and available only for that namespace.

      3

      Specify your index image.

      4

      Specify your name or an organization name publishing the catalog.

      5

      Catalog sources can automatically check for new versions to keep up to date.

   2. Use the file to create the CatalogSource object:

      $ oc apply -f catalogSource.yaml

2. Verify the following resources are created successfully.

   1. Check the pods:

      $ oc get pods -n openshift-marketplace

      Example output

      NAME                                    READY   STATUS    RESTARTS  AGE
      my-operator-catalog-6njx6               1/1     Running   0         28s
      marketplace-operator-d9f549946-96sgr    1/1     Running   0         26h

   2. Check the catalog source:

      $ oc get catalogsource -n openshift-marketplace

      Example output

      NAME                  DISPLAY               TYPE PUBLISHER  AGE
      my-operator-catalog   My Operator Catalog   grpc            5s

   3. Check the package manifest:

      $ oc get packagemanifest -n openshift-marketplace

      Example output

      NAME                          CATALOG               AGE
      jaeger-product                My Operator Catalog   93s

Procedure

1. Update the existing index by adding bundle images:

   $ opm index add \
       --bundles <registry>/<namespace>/<new_bundle_image>@sha256:<digest> \1
       --from-index <registry>/<namespace>/<existing_index_image>:<existing_tag> \2
       --tag <registry>/<namespace>/<existing_index_image>:<updated_tag> \3
       --pull-tool podman 4

   1

   The --bundles flag specifies a comma-separated list of additional bundle images to add to the index.

   2

   The --from-index flag specifies the previously pushed index.

   3

   The --tag flag specifies the image tag to apply to the updated index image.

   4

   The --pull-tool flag specifies the tool used to pull container images.

   where:

   <registry>

   Specifies the hostname of the registry, such as quay.io or mirror.example.com.

   <namespace>

   Specifies the namespace of the registry, such as ocs-dev or abc.

   <new_bundle_image>

   Specifies the new bundle image to add to the registry, such as ocs-operator.

   <digest>

   Specifies the SHA image ID, or digest, of the bundle image, such as c7f11097a628f092d8bad148406aa0e0951094a03445fd4bc0775431ef683a41.

   <existing_index_image>

   Specifies the previously pushed image, such as abc-redhat-operator-index.

   <existing_tag>

   Specifies a previously pushed image tag, such as 4.6.

   <updated_tag>

   Specifies the image tag to apply to the updated index image, such as 4.6.1.

   Example command

   $ opm index add \
       --bundles quay.io/ocs-dev/ocs-operator@sha256:c7f11097a628f092d8bad148406aa0e0951094a03445fd4bc0775431ef683a41 \
       --from-index mirror.example.com/abc/abc-redhat-operator-index:4.6 \
       --tag mirror.example.com/abc/abc-redhat-operator-index:4.6.1 \
       --pull-tool podman

2. Push the updated index image:

   $ podman push <registry>/<namespace>/<existing_index_image>:<updated_tag>

3. Follow the steps in the Mirroring an Operator catalog procedure again to mirror the updated content. However, when you get to the step about creating the ImageContentSourcePolicy (ICSP) object, use the oc replace command instead of the oc create command. For example:

   $ oc replace -f ./manifests-redhat-operator-index-<random_number>/imageContentSourcePolicy.yaml

   This change is required because the object already exists and must be updated.

   Note

   Normally, the oc apply command can be used to update existing objects that were previously created using oc apply. However, due to a known issue regarding the size of the metadata.annotations field in ICSP objects, the oc replace command must be used for this step currently.

4. After Operator Lifecycle Manager (OLM) automatically polls the index image referenced in the catalog source at its regular interval, verify that the new packages are successfully added:

   $ oc get packagemanifests -n openshift-marketplace

1. Create an Operator project by using the Operator SDK command-line interface (CLI).
2. Define new resource APIs by adding custom resource definitions (CRDs).
3. Specify resources to watch by using the Operator SDK API.
4. Define the Operator reconciling logic in a designated handler and use the Operator SDK API to interact with resources.
5. Use the Operator SDK CLI to build and generate the Operator deployment manifests.

Procedure

1. Set the release version variable:

   $ RELEASE_VERSION=v0.19.4

2. Download the release binary.

   • For Linux:

     $ curl -OJL https://github.com/operator-framework/operator-sdk/releases/download/${RELEASE_VERSION}/operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu

   • For macOS:

     $ curl -OJL https://github.com/operator-framework/operator-sdk/releases/download/${RELEASE_VERSION}/operator-sdk-${RELEASE_VERSION}-x86_64-apple-darwin

3. Verify the downloaded release binary.

   1. Download the provided .asc file.

      • For Linux:

        $ curl -OJL https://github.com/operator-framework/operator-sdk/releases/download/${RELEASE_VERSION}/operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu.asc

      • For macOS:

        $ curl -OJL https://github.com/operator-framework/operator-sdk/releases/download/${RELEASE_VERSION}/operator-sdk-${RELEASE_VERSION}-x86_64-apple-darwin.asc

   2. Place the binary and corresponding .asc file into the same directory and run the following command to verify the binary:

      • For Linux:

        $ gpg --verify operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu.asc

      • For macOS:

        $ gpg --verify operator-sdk-${RELEASE_VERSION}-x86_64-apple-darwin.asc

      If you do not have the public key of the maintainer on your workstation, you will get the following error:

      Example output with error

      $ gpg: assuming signed data in 'operator-sdk-${RELEASE_VERSION}-x86_64-apple-darwin'
      $ gpg: Signature made Fri Apr  5 20:03:22 2019 CEST
      $ gpg:                using RSA key <key_id> 1
      $ gpg: Can't check signature: No public key

      1

      RSA key string.

      To download the key, run the following command, replacing <key_id> with the RSA key string provided in the output of the previous command:

      $ gpg [--keyserver keys.gnupg.net] --recv-key "<key_id>" 1

      1

      If you do not have a key server configured, specify one with the --keyserver option.

4. Install the release binary in your PATH:

   • For Linux:

     $ chmod +x operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu

     $ sudo cp operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu /usr/local/bin/operator-sdk

     $ rm operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu

   • For macOS:

     $ chmod +x operator-sdk-${RELEASE_VERSION}-x86_64-apple-darwin

     $ sudo cp operator-sdk-${RELEASE_VERSION}-x86_64-apple-darwin /usr/local/bin/operator-sdk

     $ rm operator-sdk-${RELEASE_VERSION}-x86_64-apple-darwin

5. Verify that the CLI tool was installed correctly:

   $ operator-sdk version

Procedure

1. Install the SDK CLI using the brew command:

   $ brew install operator-sdk

2. Verify that the CLI tool was installed correctly:

   $ operator-sdk version

Procedure

1. Clone the operator-sdk repository:

   $ git clone https://github.com/operator-framework/operator-sdk

2. Change to the directory for the cloned repository:

   $ cd operator-sdk

3. Check out the v0.19.4 release:

   $ git checkout tags/v0.19.4 -b v0.19.4

4. Update dependencies:

   $ make tidy

5. Compile and install the SDK CLI:

   $ make install

   This installs the CLI binary operator-sdk in the $GOPATH/bin/ directory.

6. Verify that the CLI tool was installed correctly:

   $ operator-sdk version

Procedure

1. Create an Operator project:

   1. Create a directory for the project:

      $ mkdir -p $HOME/projects/memcached-operator

   2. Change to the directory:

      $ cd $HOME/projects/memcached-operator

   3. Activate support for Go modules:

      $ export GO111MODULE=on

   4. Run the operator-sdk init command to initialize the project:

      $ operator-sdk init \
          --domain=example.com \
          --repo=github.com/example-inc/memcached-operator

      Note

      The operator-sdk init command uses the go.kubebuilder.io/v2 plug-in by default.

2. Update your Operator to use supported images:

   1. In the project root-level Dockerfile, change the default runner image reference from:

      FROM gcr.io/distroless/static:nonroot

      to:

      FROM registry.access.redhat.com/ubi8/ubi-minimal:latest

   2. Depending on the Go project version, your Dockerfile might contain a USER 65532:65532 or USER nonroot:nonroot directive. In either case, remove the line, as it is not required by the supported runner image.

   3. In the config/default/manager_auth_proxy_patch.yaml file, change the image value from:

      gcr.io/kubebuilder/kube-rbac-proxy:<tag>

      to use the supported image:

      registry.redhat.io/openshift4/ose-kube-rbac-proxy:v4.6

3. Update the test target in your Makefile to install dependencies required during later builds by replacing the following lines:

   Example 5.1. Existing test target

   test: generate fmt vet manifests
           go test ./... -coverprofile cover.out

   With the following lines:

   Example 5.2. Updated test target

   ENVTEST_ASSETS_DIR=$(shell pwd)/testbin
   test: manifests generate fmt vet ## Run tests.
   	mkdir -p ${ENVTEST_ASSETS_DIR}
   	test -f ${ENVTEST_ASSETS_DIR}/setup-envtest.sh || curl -sSLo ${ENVTEST_ASSETS_DIR}/setup-envtest.sh https://raw.githubusercontent.com/kubernetes-sigs/controller-runtime/v0.7.2/hack/setup-envtest.sh
   	source ${ENVTEST_ASSETS_DIR}/setup-envtest.sh; fetch_envtest_tools $(ENVTEST_ASSETS_DIR); setup_envtest_env $(ENVTEST_ASSETS_DIR); go test ./... -coverprofile cover.out

4. Create a custom resource definition (CRD) API and controller:

   1. Run the following command to create an API with group cache, version v1, and kind Memcached:

      $ operator-sdk create api \
          --group=cache \
          --version=v1 \
          --kind=Memcached

   2. When prompted, enter y for creating both the resource and controller:

      Create Resource [y/n]
      y
      Create Controller [y/n]
      y

      Example output

      Writing scaffold for you to edit...
      api/v1/memcached_types.go
      controllers/memcached_controller.go
      ...

      This process generates the Memcached resource API at api/v1/memcached_types.go and the controller at controllers/memcached_controller.go.

   3. Modify the Go type definitions at api/v1/memcached_types.go to have the following spec and status:

      // MemcachedSpec defines the desired state of Memcached
      type MemcachedSpec struct {
      	// +kubebuilder:validation:Minimum=0
      	// Size is the size of the memcached deployment
      	Size int32 `json:"size"`
      }
      
      // MemcachedStatus defines the observed state of Memcached
      type MemcachedStatus struct {
      	// Nodes are the names of the memcached pods
      	Nodes []string `json:"nodes"`
      }

   4. Add the +kubebuilder:subresource:status marker to add a status subresource to the CRD manifest:

      // Memcached is the Schema for the memcacheds API
      // +kubebuilder:subresource:status 1
      type Memcached struct {
      	metav1.TypeMeta   `json:",inline"`
      	metav1.ObjectMeta `json:"metadata,omitempty"`
      
      	Spec   MemcachedSpec   `json:"spec,omitempty"`
      	Status MemcachedStatus `json:"status,omitempty"`
      }

      1

      Add this line.

      This enables the controller to update the CR status without changing the rest of the CR object.

   5. Update the generated code for the resource type:

      $ make generate

      Tip

      After you modify a *_types.go file, you must run the make generate command to update the generated code for that resource type.

      The above Makefile target invokes the controller-gen utility to update the api/v1/zz_generated.deepcopy.go file. This ensures your API Go type definitions implement the runtime.Object interface that all Kind types must implement.

5. Generate and update CRD manifests:

   $ make manifests

   This Makefile target invokes the controller-gen utility to generate the CRD manifests in the config/crd/bases/cache.example.com_memcacheds.yaml file.

   1. Optional: Add custom validation to your CRD.

      OpenAPI v3.0 schemas are added to CRD manifests in the spec.validation block when the manifests are generated. This validation block allows Kubernetes to validate the properties in a Memcached custom resource (CR) when it is created or updated.

      As an Operator author, you can use annotation-like, single-line comments called Kubebuilder markers to configure custom validations for your API. These markers must always have a +kubebuilder:validation prefix. For example, adding an enum-type specification can be done by adding the following marker:

      // +kubebuilder:validation:Enum=Lion;Wolf;Dragon
      type Alias string

      Usage of markers in API code is discussed in the Kubebuilder Generating CRDs and Markers for Config/Code Generation documentation. A full list of OpenAPIv3 validation markers is also available in the Kubebuilder CRD Validation documentation.

      If you add any custom validations, run the following command to update the OpenAPI validation section for the CRD:

      $ make manifests

6. After creating a new API and controller, you can implement the controller logic. For this example, replace the generated controller file controllers/memcached_controller.go with following example implementation:

   Example 5.3. Example memcached_controller.go

   /*
   Licensed under the Apache License, Version 2.0 (the "License");
   you may not use this file except in compliance with the License.
   You may obtain a copy of the License at
   
       http://www.apache.org/licenses/LICENSE-2.0
   
   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.
   */
   
   package controllers
   
   import (
   	"context"
   	"reflect"
   
   	"github.com/go-logr/logr"
   	appsv1 "k8s.io/api/apps/v1"
   	corev1 "k8s.io/api/core/v1"
   	"k8s.io/apimachinery/pkg/api/errors"
   	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
   	"k8s.io/apimachinery/pkg/runtime"
   	"k8s.io/apimachinery/pkg/types"
   	ctrl "sigs.k8s.io/controller-runtime"
   	"sigs.k8s.io/controller-runtime/pkg/client"
   
   	cachev1 "github.com/example-inc/memcached-operator/api/v1"
   )
   
   // MemcachedReconciler reconciles a Memcached object
   type MemcachedReconciler struct {
   	client.Client
   	Log    logr.Logger
   	Scheme *runtime.Scheme
   }
   
   // +kubebuilder:rbac:groups=cache.example.com,resources=memcacheds,verbs=get;list;watch;create;update;patch;delete
   // +kubebuilder:rbac:groups=cache.example.com,resources=memcacheds/status,verbs=get;update;patch
   // +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
   // +kubebuilder:rbac:groups=core,resources=pods,verbs=get;list;
   
   func (r *MemcachedReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
   	ctx := context.Background()
   	log := r.Log.WithValues("memcached", req.NamespacedName)
   
   	// Fetch the Memcached instance
   	memcached := &cachev1.Memcached{}
   	err := r.Get(ctx, req.NamespacedName, memcached)
   	if err != nil {
   		if errors.IsNotFound(err) {
   			// Request object not found, could have been deleted after reconcile request.
   			// Owned objects are automatically garbage collected. For additional cleanup logic use finalizers.
   			// Return and don't requeue
   			log.Info("Memcached resource not found. Ignoring since object must be deleted")
   			return ctrl.Result{}, nil
   		}
   		// Error reading the object - requeue the request.
   		log.Error(err, "Failed to get Memcached")
   		return ctrl.Result{}, err
   	}
   
   	// Check if the deployment already exists, if not create a new one
   	found := &appsv1.Deployment{}
   	err = r.Get(ctx, types.NamespacedName{Name: memcached.Name, Namespace: memcached.Namespace}, found)
   	if err != nil && errors.IsNotFound(err) {
   		// Define a new deployment
   		dep := r.deploymentForMemcached(memcached)
   		log.Info("Creating a new Deployment", "Deployment.Namespace", dep.Namespace, "Deployment.Name", dep.Name)
   		err = r.Create(ctx, dep)
   		if err != nil {
   			log.Error(err, "Failed to create new Deployment", "Deployment.Namespace", dep.Namespace, "Deployment.Name", dep.Name)
   			return ctrl.Result{}, err
   		}
   		// Deployment created successfully - return and requeue
   		return ctrl.Result{Requeue: true}, nil
   	} else if err != nil {
   		log.Error(err, "Failed to get Deployment")
   		return ctrl.Result{}, err
   	}
   
   	// Ensure the deployment size is the same as the spec
   	size := memcached.Spec.Size
   	if *found.Spec.Replicas != size {
   		found.Spec.Replicas = &size
   		err = r.Update(ctx, found)
   		if err != nil {
   			log.Error(err, "Failed to update Deployment", "Deployment.Namespace", found.Namespace, "Deployment.Name", found.Name)
   			return ctrl.Result{}, err
   		}
   		// Spec updated - return and requeue
   		return ctrl.Result{Requeue: true}, nil
   	}
   
   	// Update the Memcached status with the pod names
   	// List the pods for this memcached's deployment
   	podList := &corev1.PodList{}
   	listOpts := []client.ListOption{
   		client.InNamespace(memcached.Namespace),
   		client.MatchingLabels(labelsForMemcached(memcached.Name)),
   	}
   	if err = r.List(ctx, podList, listOpts...); err != nil {
   		log.Error(err, "Failed to list pods", "Memcached.Namespace", memcached.Namespace, "Memcached.Name", memcached.Name)
   		return ctrl.Result{}, err
   	}
   	podNames := getPodNames(podList.Items)
   
   	// Update status.Nodes if needed
   	if !reflect.DeepEqual(podNames, memcached.Status.Nodes) {
   		memcached.Status.Nodes = podNames
   		err := r.Status().Update(ctx, memcached)
   		if err != nil {
   			log.Error(err, "Failed to update Memcached status")
   			return ctrl.Result{}, err
   		}
   	}
   
   	return ctrl.Result{}, nil
   }
   
   // deploymentForMemcached returns a memcached Deployment object
   func (r *MemcachedReconciler) deploymentForMemcached(m *cachev1.Memcached) *appsv1.Deployment {
   	ls := labelsForMemcached(m.Name)
   	replicas := m.Spec.Size
   
   	dep := &appsv1.Deployment{
   		ObjectMeta: metav1.ObjectMeta{
   			Name:      m.Name,
   			Namespace: m.Namespace,
   		},
   		Spec: appsv1.DeploymentSpec{
   			Replicas: &replicas,
   			Selector: &metav1.LabelSelector{
   				MatchLabels: ls,
   			},
   			Template: corev1.PodTemplateSpec{
   				ObjectMeta: metav1.ObjectMeta{
   					Labels: ls,
   				},
   				Spec: corev1.PodSpec{
   					Containers: []corev1.Container{{
   						Image:   "memcached:1.4.36-alpine",
   						Name:    "memcached",
   						Command: []string{"memcached", "-m=64", "-o", "modern", "-v"},
   						Ports: []corev1.ContainerPort{{
   							ContainerPort: 11211,
   							Name:          "memcached",
   						}},
   					}},
   				},
   			},
   		},
   	}
   	// Set Memcached instance as the owner and controller
   	ctrl.SetControllerReference(m, dep, r.Scheme)
   	return dep
   }
   
   // labelsForMemcached returns the labels for selecting the resources
   // belonging to the given memcached CR name.
   func labelsForMemcached(name string) map[string]string {
   	return map[string]string{"app": "memcached", "memcached_cr": name}
   }
   
   // getPodNames returns the pod names of the array of pods passed in
   func getPodNames(pods []corev1.Pod) []string {
   	var podNames []string
   	for _, pod := range pods {
   		podNames = append(podNames, pod.Name)
   	}
   	return podNames
   }
   
   func (r *MemcachedReconciler) SetupWithManager(mgr ctrl.Manager) error {
   	return ctrl.NewControllerManagedBy(mgr).
   		For(&cachev1.Memcached{}).
   		Owns(&appsv1.Deployment{}).
   		Complete(r)
   }

   The example controller runs the following reconciliation logic for each Memcached CR:

   • Create a Memcached deployment if it does not exist.
   • Ensure that the deployment size is the same as specified by the Memcached CR spec.
   • Update the Memcached CR status with the names of the memcached pods.

   The next two sub-steps inspect how the controller watches resources and how the reconcile loop is triggered. You can skip these steps to go directly to building and running the Operator.

   1. Inspect the controller implementation at the controllers/memcached_controller.go file to see how the controller watches resources.

      The SetupWithManager() function specifies how the controller is built to watch a CR and other resources that are owned and managed by that controller:

      Example 5.4. SetupWithManager() function

      import (
      	...
      	appsv1 "k8s.io/api/apps/v1"
      	...
      )
      
      func (r *MemcachedReconciler) SetupWithManager(mgr ctrl.Manager) error {
      	return ctrl.NewControllerManagedBy(mgr).
      		For(&cachev1.Memcached{}).
      		Owns(&appsv1.Deployment{}).
      		Complete(r)
      }

      NewControllerManagedBy() provides a controller builder that allows various controller configurations.

      For(&cachev1.Memcached{}) specifies the Memcached type as the primary resource to watch. For each Add, Update, or Delete event for a Memcached type, the reconcile loop is sent a reconcile Request argument, which consists of a namespace and name key, for that Memcached object.

      Owns(&appsv1.Deployment{}) specifies the Deployment type as the secondary resource to watch. For each Deployment type Add, Update, or Delete event, the event handler maps each event to a reconcile request for the owner of the deployment. In this case, the owner is the Memcached object for which the deployment was created.

   2. Every controller has a reconciler object with a Reconcile() method that implements the reconcile loop. The reconcile loop is passed the Request argument, which is a namespace and name key used to find the primary resource object, Memcached, from the cache:

      Example 5.5. Reconcile loop

      import (
      	ctrl "sigs.k8s.io/controller-runtime"
      
      	cachev1 "github.com/example-inc/memcached-operator/api/v1"
      	...
      )
      
      func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
        // Lookup the Memcached instance for this reconcile request
        memcached := &cachev1.Memcached{}
        err := r.Get(ctx, req.NamespacedName, memcached)
        ...
      }

      Based on the return value of the Reconcile() function, the reconcile Request might be requeued, and the loop might be triggered again:

      Example 5.6. Requeue logic

      // Reconcile successful - don't requeue
      return reconcile.Result{}, nil
      // Reconcile failed due to error - requeue
      return reconcile.Result{}, err
      // Requeue for any reason other than error
      return reconcile.Result{Requeue: true}, nil

      You can set the Result.RequeueAfter to requeue the request after a grace period:

      Example 5.7. Requeue after grace period

      import "time"
      
      // Reconcile for any reason other than an error after 5 seconds
      return ctrl.Result{RequeueAfter: time.Second*5}, nil

      Note

      You can return Result with RequeueAfter set to periodically reconcile a CR.

      For more on reconcilers, clients, and interacting with resource events, see the Controller Runtime Client API documentation.

Procedure

1. Change to the namespace where your Operator is installed. For example, if you deployed the Operator using the make deploy command:

   $ oc project memcached-operator-system

2. Edit the sample Memcached CR manifest at config/samples/cache_v1_memcached.yaml to contain the following specification:

   apiVersion: cache.example.com/v1
   kind: Memcached
   metadata:
     name: memcached-sample
   ...
   spec:
   ...
     size: 3

3. Create the CR:

   $ oc apply -f config/samples/cache_v1_memcached.yaml

4. Ensure that the Memcached Operator creates the deployment for the sample CR with the correct size:

   $ oc get deployments

   Example output

   NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
   memcached-operator-controller-manager   1/1     1            1           8m
   memcached-sample                        3/3     3            3           1m

5. Check the pods and CR status to confirm the status is updated with the Memcached pod names.

   1. Check the pods:

      $ oc get pods

      Example output

      NAME                                  READY     STATUS    RESTARTS   AGE
      memcached-sample-6fd7c98d8-7dqdr      1/1       Running   0          1m
      memcached-sample-6fd7c98d8-g5k7v      1/1       Running   0          1m
      memcached-sample-6fd7c98d8-m7vn7      1/1       Running   0          1m

   2. Check the CR status:

      $ oc get memcached/memcached-sample -o yaml

      Example output

      apiVersion: cache.example.com/v1
      kind: Memcached
      metadata:
      ...
        name: memcached-sample
      ...
      spec:
        size: 3
      status:
        nodes:
        - memcached-sample-6fd7c98d8-7dqdr
        - memcached-sample-6fd7c98d8-g5k7v
        - memcached-sample-6fd7c98d8-m7vn7

6. Update the deployment size.

   1. Update config/samples/cache_v1_memcached.yaml file to change the spec.size field in the Memcached CR from 3 to 5:

      $ oc patch memcached memcached-sample \
          -p '{"spec":{"size": 5}}' \
          --type=merge

   2. Confirm that the Operator changes the deployment size:

      $ oc get deployments

      Example output

      NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
      memcached-operator-controller-manager   1/1     1            1           10m
      memcached-sample                        5/5     5            5           3m

Procedure

1. Create a new Operator project. A namespace-scoped Operator watches and manages resources in a single namespace. Namespace-scoped Operators are preferred because of their flexibility. They enable decoupled upgrades, namespace isolation for failures and monitoring, and differing API definitions.

   To create a new Ansible-based, namespace-scoped memcached-operator project and change to the new directory, use the following commands:

   $ operator-sdk new memcached-operator \
       --api-version=cache.example.com/v1alpha1 \
       --kind=Memcached \
       --type=ansible

   $ cd memcached-operator

   This creates the memcached-operator project specifically for watching the Memcached resource with API version example.com/v1apha1 and kind Memcached.

2. Customize the Operator logic.

   For this example, the memcached-operator executes the following reconciliation logic for each Memcached custom resource (CR):

   • Create a memcached deployment if it does not exist.
   • Ensure that the deployment size is the same as specified by the Memcached CR.

   By default, the memcached-operator watches Memcached resource events as shown in the watches.yaml file and executes the Ansible role Memcached:

   - version: v1alpha1
     group: cache.example.com
     kind: Memcached

   You can optionally customize the following logic in the watches.yaml file:

   1. Specifying a role option configures the Operator to use this specified path when launching ansible-runner with an Ansible role. By default, the operator-sdk new command fills in an absolute path to where your role should go:

      - version: v1alpha1
        group: cache.example.com
        kind: Memcached
        role: /opt/ansible/roles/memcached

   2. Specifying a playbook option in the watches.yaml file configures the Operator to use this specified path when launching ansible-runner with an Ansible playbook:

      - version: v1alpha1
        group: cache.example.com
        kind: Memcached
        playbook: /opt/ansible/playbook.yaml

3. Build the Memcached Ansible role.

   Modify the generated Ansible role under the roles/memcached/ directory. This Ansible role controls the logic that is executed when a resource is modified.

   1. Define the Memcached spec.

      Defining the spec for an Ansible-based Operator can be done entirely in Ansible. The Ansible Operator passes all key-value pairs listed in the CR spec field along to Ansible as variables. The names of all variables in the spec field are converted to snake case (lowercase with an underscore) by the Operator before running Ansible. For example, serviceAccount in the spec becomes service_account in Ansible.

      Tip

      You should perform some type validation in Ansible on the variables to ensure that your application is receiving expected input.

      In case the user does not set the spec field, set a default by modifying the roles/memcached/defaults/main.yml file:

      size: 1

   2. Define the Memcached deployment.

      With the Memcached spec now defined, you can define what Ansible is actually executed on resource changes. Because this is an Ansible role, the default behavior executes the tasks in the roles/memcached/tasks/main.yml file.

      The goal is for Ansible to create a deployment if it does not exist, which runs the memcached:1.4.36-alpine image. Ansible 2.7+ supports the k8s Ansible module, which this example leverages to control the deployment definition.

      Modify the roles/memcached/tasks/main.yml to match the following:

      - name: start memcached
        k8s:
          definition:
            kind: Deployment
            apiVersion: apps/v1
            metadata:
              name: '{{ meta.name }}-memcached'
              namespace: '{{ meta.namespace }}'
            spec:
              replicas: "{{size}}"
              selector:
                matchLabels:
                  app: memcached
              template:
                metadata:
                  labels:
                    app: memcached
                spec:
                  containers:
                  - name: memcached
                    command:
                    - memcached
                    - -m=64
                    - -o
                    - modern
                    - -v
                    image: "docker.io/memcached:1.4.36-alpine"
                    ports:
                      - containerPort: 11211

      Note

      This example used the size variable to control the number of replicas of the Memcached deployment. This example sets the default to 1, but any user can create a CR that overwrites the default.

4. Deploy the CRD.

   Before running the Operator, Kubernetes needs to know about the new custom resource definition (CRD) that the Operator will be watching. Deploy the Memcached CRD:

   $ oc create -f deploy/crds/cache.example.com_memcacheds_crd.yaml

5. Build and run the Operator.

   There are two ways to build and run the Operator:

   • As a pod inside a Kubernetes cluster.
   • As a Go program outside the cluster using the operator-sdk up command.

   Choose one of the following methods:

   1. Run as a pod inside a Kubernetes cluster. This is the preferred method for production use.

      1. Build the memcached-operator image and push it to a registry:

         $ operator-sdk build quay.io/example/memcached-operator:v0.0.1

         $ podman push quay.io/example/memcached-operator:v0.0.1

      2. Deployment manifests are generated in the deploy/operator.yaml file. The deployment image in this file needs to be modified from the placeholder REPLACE_IMAGE to the previous built image. To do this, run:

         $ sed -i 's|REPLACE_IMAGE|quay.io/example/memcached-operator:v0.0.1|g' deploy/operator.yaml

      3. Deploy the memcached-operator manifests:

         $ oc create -f deploy/service_account.yaml

         $ oc create -f deploy/role.yaml

         $ oc create -f deploy/role_binding.yaml

         $ oc create -f deploy/operator.yaml

      4. Verify that the memcached-operator deployment is up and running:

         $ oc get deployment

         NAME                     DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
         memcached-operator       1         1         1            1           1m

   2. Run outside the cluster. This method is preferred during the development cycle to speed up deployment and testing.

      Ensure that Ansible Runner and Ansible Runner HTTP Plug-in are installed or else you will see unexpected errors from Ansible Runner when a CR is created.

      It is also important that the role path referenced in the watches.yaml file exists on your machine. Because normally a container is used where the role is put on disk, the role must be manually copied to the configured Ansible roles path (for example /etc/ansible/roles).

      1. To run the Operator locally with the default Kubernetes configuration file present at $HOME/.kube/config:

         $ operator-sdk run --local

         To run the Operator locally with a provided Kubernetes configuration file:

         $ operator-sdk run --local --kubeconfig=config

6. Create a Memcached CR.

   1. Modify the deploy/crds/cache_v1alpha1_memcached_cr.yaml file as shown and create a Memcached CR:

      $ cat deploy/crds/cache_v1alpha1_memcached_cr.yaml

      Example output

      apiVersion: "cache.example.com/v1alpha1"
      kind: "Memcached"
      metadata:
        name: "example-memcached"
      spec:
        size: 3

      $ oc apply -f deploy/crds/cache_v1alpha1_memcached_cr.yaml

   2. Ensure that the memcached-operator creates the deployment for the CR:

      $ oc get deployment

      Example output

      NAME                     DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
      memcached-operator       1         1         1            1           2m
      example-memcached        3         3         3            3           1m

   3. Check the pods to confirm three replicas were created:

      $ oc get pods

      NAME                                  READY     STATUS    RESTARTS   AGE
      example-memcached-6fd7c98d8-7dqdr     1/1       Running   0          1m
      example-memcached-6fd7c98d8-g5k7v     1/1       Running   0          1m
      example-memcached-6fd7c98d8-m7vn7     1/1       Running   0          1m
      memcached-operator-7cc7cfdf86-vvjqk   1/1       Running   0          2m

7. Update the size.

   1. Change the spec.size field in the memcached CR from 3 to 4 and apply the change:

      $ cat deploy/crds/cache_v1alpha1_memcached_cr.yaml

      Example output

      apiVersion: "cache.example.com/v1alpha1"
      kind: "Memcached"
      metadata:
        name: "example-memcached"
      spec:
        size: 4

      $ oc apply -f deploy/crds/cache_v1alpha1_memcached_cr.yaml

   2. Confirm that the Operator changes the deployment size:

      $ oc get deployment

      Example output

      NAME                 DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
      example-memcached    4         4         4            4           5m

8. Clean up the resources:

   $ oc delete -f deploy/crds/cache_v1alpha1_memcached_cr.yaml

   $ oc delete -f deploy/operator.yaml

   $ oc delete -f deploy/role_binding.yaml

   $ oc delete -f deploy/role.yaml

   $ oc delete -f deploy/service_account.yaml

   $ oc delete -f deploy/crds/cache_v1alpha1_memcached_crd.yaml