Chapter 5. Developing Operators

1. Create an Operator project by using the Operator SDK command-line interface (CLI).
2. Define new resource APIs by adding custom resource definitions (CRDs).
3. Specify resources to watch by using the Operator SDK API.
4. Define the Operator reconciling logic in a designated handler and use the Operator SDK API to interact with resources.
5. Use the Operator SDK CLI to build and generate the Operator deployment manifests.

Procedure

1. Set the release version variable:

   $ RELEASE_VERSION=v0.19.4

2. Download the release binary.

   • For Linux:

     $ curl -OJL https://github.com/operator-framework/operator-sdk/releases/download/${RELEASE_VERSION}/operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu

   • For macOS:

     $ curl -OJL https://github.com/operator-framework/operator-sdk/releases/download/${RELEASE_VERSION}/operator-sdk-${RELEASE_VERSION}-x86_64-apple-darwin

3. Verify the downloaded release binary.

   1. Download the provided .asc file.

      • For Linux:

        $ curl -OJL https://github.com/operator-framework/operator-sdk/releases/download/${RELEASE_VERSION}/operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu.asc

      • For macOS:

        $ curl -OJL https://github.com/operator-framework/operator-sdk/releases/download/${RELEASE_VERSION}/operator-sdk-${RELEASE_VERSION}-x86_64-apple-darwin.asc

   2. Place the binary and corresponding .asc file into the same directory and run the following command to verify the binary:

      • For Linux:

        $ gpg --verify operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu.asc

      • For macOS:

        $ gpg --verify operator-sdk-${RELEASE_VERSION}-x86_64-apple-darwin.asc

      If you do not have the public key of the maintainer on your workstation, you will get the following error:

      Example output with error

      $ gpg: assuming signed data in 'operator-sdk-${RELEASE_VERSION}-x86_64-apple-darwin'
      $ gpg: Signature made Fri Apr  5 20:03:22 2019 CEST
      $ gpg:                using RSA key <key_id> 

      1

      
      $ gpg: Can't check signature: No public key

      1

      RSA key string.

      To download the key, run the following command, replacing <key_id> with the RSA key string provided in the output of the previous command:

      $ gpg [--keyserver keys.gnupg.net] --recv-key "<key_id>" 

      1

      1

      If you do not have a key server configured, specify one with the --keyserver option.

4. Install the release binary in your PATH:

   • For Linux:

     $ chmod +x operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu

     $ sudo cp operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu /usr/local/bin/operator-sdk

     $ rm operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu

   • For macOS:

     $ chmod +x operator-sdk-${RELEASE_VERSION}-x86_64-apple-darwin

     $ sudo cp operator-sdk-${RELEASE_VERSION}-x86_64-apple-darwin /usr/local/bin/operator-sdk

     $ rm operator-sdk-${RELEASE_VERSION}-x86_64-apple-darwin

5. Verify that the CLI tool was installed correctly:

   $ operator-sdk version

Procedure

1. Install the SDK CLI using the brew command:

   $ brew install operator-sdk

2. Verify that the CLI tool was installed correctly:

   $ operator-sdk version

Procedure

1. Clone the operator-sdk repository:

   $ git clone https://github.com/operator-framework/operator-sdk

2. Change to the directory for the cloned repository:

   $ cd operator-sdk

3. Check out the v0.19.4 release:

   $ git checkout tags/v0.19.4 -b v0.19.4

4. Update dependencies:

   $ make tidy

5. Compile and install the SDK CLI:

   $ make install

   This installs the CLI binary operator-sdk in the $GOPATH/bin/ directory.

6. Verify that the CLI tool was installed correctly:

   $ operator-sdk version

Procedure

1. Create an Operator project:

   1. Create a directory for the project:

      $ mkdir -p $HOME/projects/memcached-operator

   2. Change to the directory:

      $ cd $HOME/projects/memcached-operator

   3. Activate support for Go modules:

      $ export GO111MODULE=on

   4. Run the operator-sdk init command to initialize the project:

      $ operator-sdk init \
          --domain=example.com \
          --repo=github.com/example-inc/memcached-operator

      Note

      The operator-sdk init command uses the go.kubebuilder.io/v2 plug-in by default.

2. Update your Operator to use supported images:

   1. In the project root-level Dockerfile, change the default runner image reference from:

      FROM gcr.io/distroless/static:nonroot

      to:

      FROM registry.access.redhat.com/ubi8/ubi-minimal:latest

   2. Depending on the Go project version, your Dockerfile might contain a USER 65532:65532 or USER nonroot:nonroot directive. In either case, remove the line, as it is not required by the supported runner image.

   3. In the config/default/manager_auth_proxy_patch.yaml file, change the image value from:

      gcr.io/kubebuilder/kube-rbac-proxy:<tag>

      to use the supported image:

      registry.redhat.io/openshift4/ose-kube-rbac-proxy:v4.6

3. Update the test target in your Makefile to install dependencies required during later builds by replacing the following lines:

   Example 5.1. Existing test target

   test: generate fmt vet manifests
           go test ./... -coverprofile cover.out

   With the following lines:

   Example 5.2. Updated test target

   ENVTEST_ASSETS_DIR=$(shell pwd)/testbin
   test: manifests generate fmt vet ## Run tests.
   	mkdir -p ${ENVTEST_ASSETS_DIR}
   	test -f ${ENVTEST_ASSETS_DIR}/setup-envtest.sh || curl -sSLo ${ENVTEST_ASSETS_DIR}/setup-envtest.sh https://raw.githubusercontent.com/kubernetes-sigs/controller-runtime/v0.7.2/hack/setup-envtest.sh
   	source ${ENVTEST_ASSETS_DIR}/setup-envtest.sh; fetch_envtest_tools $(ENVTEST_ASSETS_DIR); setup_envtest_env $(ENVTEST_ASSETS_DIR); go test ./... -coverprofile cover.out

4. Create a custom resource definition (CRD) API and controller:

   1. Run the following command to create an API with group cache, version v1, and kind Memcached:

      $ operator-sdk create api \
          --group=cache \
          --version=v1 \
          --kind=Memcached

   2. When prompted, enter y for creating both the resource and controller:

      Create Resource [y/n]
      y
      Create Controller [y/n]
      y

      Example output

      Writing scaffold for you to edit...
      api/v1/memcached_types.go
      controllers/memcached_controller.go
      ...

      This process generates the Memcached resource API at api/v1/memcached_types.go and the controller at controllers/memcached_controller.go.

   3. Modify the Go type definitions at api/v1/memcached_types.go to have the following spec and status:

      // MemcachedSpec defines the desired state of Memcached
      type MemcachedSpec struct {
      	// +kubebuilder:validation:Minimum=0
      	// Size is the size of the memcached deployment
      	Size int32 `json:"size"`
      }
      
      // MemcachedStatus defines the observed state of Memcached
      type MemcachedStatus struct {
      	// Nodes are the names of the memcached pods
      	Nodes []string `json:"nodes"`
      }

   4. Add the +kubebuilder:subresource:status marker to add a status subresource to the CRD manifest:

      // Memcached is the Schema for the memcacheds API
      // +kubebuilder:subresource:status 

      1

      
      type Memcached struct {
      	metav1.TypeMeta   `json:",inline"`
      	metav1.ObjectMeta `json:"metadata,omitempty"`
      
      	Spec   MemcachedSpec   `json:"spec,omitempty"`
      	Status MemcachedStatus `json:"status,omitempty"`
      }

      1

      Add this line.

      This enables the controller to update the CR status without changing the rest of the CR object.

   5. Update the generated code for the resource type:

      $ make generate

      Tip

      After you modify a *_types.go file, you must run the make generate command to update the generated code for that resource type.

      The above Makefile target invokes the controller-gen utility to update the api/v1/zz_generated.deepcopy.go file. This ensures your API Go type definitions implement the runtime.Object interface that all Kind types must implement.

5. Generate and update CRD manifests:

   $ make manifests

   This Makefile target invokes the controller-gen utility to generate the CRD manifests in the config/crd/bases/cache.example.com_memcacheds.yaml file.

   1. Optional: Add custom validation to your CRD.

      OpenAPI v3.0 schemas are added to CRD manifests in the spec.validation block when the manifests are generated. This validation block allows Kubernetes to validate the properties in a Memcached custom resource (CR) when it is created or updated.

      As an Operator author, you can use annotation-like, single-line comments called Kubebuilder markers to configure custom validations for your API. These markers must always have a +kubebuilder:validation prefix. For example, adding an enum-type specification can be done by adding the following marker:

      // +kubebuilder:validation:Enum=Lion;Wolf;Dragon
      type Alias string

      Usage of markers in API code is discussed in the Kubebuilder Generating CRDs and Markers for Config/Code Generation documentation. A full list of OpenAPIv3 validation markers is also available in the Kubebuilder CRD Validation documentation.

      If you add any custom validations, run the following command to update the OpenAPI validation section for the CRD:

      $ make manifests

6. After creating a new API and controller, you can implement the controller logic. For this example, replace the generated controller file controllers/memcached_controller.go with following example implementation:

   Example 5.3. Example memcached_controller.go

   /*
   Licensed under the Apache License, Version 2.0 (the "License");
   you may not use this file except in compliance with the License.
   You may obtain a copy of the License at
   
       http://www.apache.org/licenses/LICENSE-2.0
   
   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.
   */
   
   package controllers
   
   import (
   	"context"
   	"reflect"
   
   	"github.com/go-logr/logr"
   	appsv1 "k8s.io/api/apps/v1"
   	corev1 "k8s.io/api/core/v1"
   	"k8s.io/apimachinery/pkg/api/errors"
   	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
   	"k8s.io/apimachinery/pkg/runtime"
   	"k8s.io/apimachinery/pkg/types"
   	ctrl "sigs.k8s.io/controller-runtime"
   	"sigs.k8s.io/controller-runtime/pkg/client"
   
   	cachev1 "github.com/example-inc/memcached-operator/api/v1"
   )
   
   // MemcachedReconciler reconciles a Memcached object
   type MemcachedReconciler struct {
   	client.Client
   	Log    logr.Logger
   	Scheme *runtime.Scheme
   }
   
   // +kubebuilder:rbac:groups=cache.example.com,resources=memcacheds,verbs=get;list;watch;create;update;patch;delete
   // +kubebuilder:rbac:groups=cache.example.com,resources=memcacheds/status,verbs=get;update;patch
   // +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
   // +kubebuilder:rbac:groups=core,resources=pods,verbs=get;list;
   
   func (r *MemcachedReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
   	ctx := context.Background()
   	log := r.Log.WithValues("memcached", req.NamespacedName)
   
   	// Fetch the Memcached instance
   	memcached := &cachev1.Memcached{}
   	err := r.Get(ctx, req.NamespacedName, memcached)
   	if err != nil {
   		if errors.IsNotFound(err) {
   			// Request object not found, could have been deleted after reconcile request.
   			// Owned objects are automatically garbage collected. For additional cleanup logic use finalizers.
   			// Return and don't requeue
   			log.Info("Memcached resource not found. Ignoring since object must be deleted")
   			return ctrl.Result{}, nil
   		}
   		// Error reading the object - requeue the request.
   		log.Error(err, "Failed to get Memcached")
   		return ctrl.Result{}, err
   	}
   
   	// Check if the deployment already exists, if not create a new one
   	found := &appsv1.Deployment{}
   	err = r.Get(ctx, types.NamespacedName{Name: memcached.Name, Namespace: memcached.Namespace}, found)
   	if err != nil && errors.IsNotFound(err) {
   		// Define a new deployment
   		dep := r.deploymentForMemcached(memcached)
   		log.Info("Creating a new Deployment", "Deployment.Namespace", dep.Namespace, "Deployment.Name", dep.Name)
   		err = r.Create(ctx, dep)
   		if err != nil {
   			log.Error(err, "Failed to create new Deployment", "Deployment.Namespace", dep.Namespace, "Deployment.Name", dep.Name)
   			return ctrl.Result{}, err
   		}
   		// Deployment created successfully - return and requeue
   		return ctrl.Result{Requeue: true}, nil
   	} else if err != nil {
   		log.Error(err, "Failed to get Deployment")
   		return ctrl.Result{}, err
   	}
   
   	// Ensure the deployment size is the same as the spec
   	size := memcached.Spec.Size
   	if *found.Spec.Replicas != size {
   		found.Spec.Replicas = &size
   		err = r.Update(ctx, found)
   		if err != nil {
   			log.Error(err, "Failed to update Deployment", "Deployment.Namespace", found.Namespace, "Deployment.Name", found.Name)
   			return ctrl.Result{}, err
   		}
   		// Spec updated - return and requeue
   		return ctrl.Result{Requeue: true}, nil
   	}
   
   	// Update the Memcached status with the pod names
   	// List the pods for this memcached's deployment
   	podList := &corev1.PodList{}
   	listOpts := []client.ListOption{
   		client.InNamespace(memcached.Namespace),
   		client.MatchingLabels(labelsForMemcached(memcached.Name)),
   	}
   	if err = r.List(ctx, podList, listOpts...); err != nil {
   		log.Error(err, "Failed to list pods", "Memcached.Namespace", memcached.Namespace, "Memcached.Name", memcached.Name)
   		return ctrl.Result{}, err
   	}
   	podNames := getPodNames(podList.Items)
   
   	// Update status.Nodes if needed
   	if !reflect.DeepEqual(podNames, memcached.Status.Nodes) {
   		memcached.Status.Nodes = podNames
   		err := r.Status().Update(ctx, memcached)
   		if err != nil {
   			log.Error(err, "Failed to update Memcached status")
   			return ctrl.Result{}, err
   		}
   	}
   
   	return ctrl.Result{}, nil
   }
   
   // deploymentForMemcached returns a memcached Deployment object
   func (r *MemcachedReconciler) deploymentForMemcached(m *cachev1.Memcached) *appsv1.Deployment {
   	ls := labelsForMemcached(m.Name)
   	replicas := m.Spec.Size
   
   	dep := &appsv1.Deployment{
   		ObjectMeta: metav1.ObjectMeta{
   			Name:      m.Name,
   			Namespace: m.Namespace,
   		},
   		Spec: appsv1.DeploymentSpec{
   			Replicas: &replicas,
   			Selector: &metav1.LabelSelector{
   				MatchLabels: ls,
   			},
   			Template: corev1.PodTemplateSpec{
   				ObjectMeta: metav1.ObjectMeta{
   					Labels: ls,
   				},
   				Spec: corev1.PodSpec{
   					Containers: []corev1.Container{{
   						Image:   "memcached:1.4.36-alpine",
   						Name:    "memcached",
   						Command: []string{"memcached", "-m=64", "-o", "modern", "-v"},
   						Ports: []corev1.ContainerPort{{
   							ContainerPort: 11211,
   							Name:          "memcached",
   						}},
   					}},
   				},
   			},
   		},
   	}
   	// Set Memcached instance as the owner and controller
   	ctrl.SetControllerReference(m, dep, r.Scheme)
   	return dep
   }
   
   // labelsForMemcached returns the labels for selecting the resources
   // belonging to the given memcached CR name.
   func labelsForMemcached(name string) map[string]string {
   	return map[string]string{"app": "memcached", "memcached_cr": name}
   }
   
   // getPodNames returns the pod names of the array of pods passed in
   func getPodNames(pods []corev1.Pod) []string {
   	var podNames []string
   	for _, pod := range pods {
   		podNames = append(podNames, pod.Name)
   	}
   	return podNames
   }
   
   func (r *MemcachedReconciler) SetupWithManager(mgr ctrl.Manager) error {
   	return ctrl.NewControllerManagedBy(mgr).
   		For(&cachev1.Memcached{}).
   		Owns(&appsv1.Deployment{}).
   		Complete(r)
   }

   The example controller runs the following reconciliation logic for each Memcached CR:

   • Create a Memcached deployment if it does not exist.
   • Ensure that the deployment size is the same as specified by the Memcached CR spec.
   • Update the Memcached CR status with the names of the memcached pods.

   The next two sub-steps inspect how the controller watches resources and how the reconcile loop is triggered. You can skip these steps to go directly to building and running the Operator.

   1. Inspect the controller implementation at the controllers/memcached_controller.go file to see how the controller watches resources.

      The SetupWithManager() function specifies how the controller is built to watch a CR and other resources that are owned and managed by that controller:

      Example 5.4. SetupWithManager() function

      import (
      	...
      	appsv1 "k8s.io/api/apps/v1"
      	...
      )
      
      func (r *MemcachedReconciler) SetupWithManager(mgr ctrl.Manager) error {
      	return ctrl.NewControllerManagedBy(mgr).
      		For(&cachev1.Memcached{}).
      		Owns(&appsv1.Deployment{}).
      		Complete(r)
      }

      NewControllerManagedBy() provides a controller builder that allows various controller configurations.

      For(&cachev1.Memcached{}) specifies the Memcached type as the primary resource to watch. For each Add, Update, or Delete event for a Memcached type, the reconcile loop is sent a reconcile Request argument, which consists of a namespace and name key, for that Memcached object.

      Owns(&appsv1.Deployment{}) specifies the Deployment type as the secondary resource to watch. For each Deployment type Add, Update, or Delete event, the event handler maps each event to a reconcile request for the owner of the deployment. In this case, the owner is the Memcached object for which the deployment was created.

   2. Every controller has a reconciler object with a Reconcile() method that implements the reconcile loop. The reconcile loop is passed the Request argument, which is a namespace and name key used to find the primary resource object, Memcached, from the cache:

      Example 5.5. Reconcile loop

      import (
      	ctrl "sigs.k8s.io/controller-runtime"
      
      	cachev1 "github.com/example-inc/memcached-operator/api/v1"
      	...
      )
      
      func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
        // Lookup the Memcached instance for this reconcile request
        memcached := &cachev1.Memcached{}
        err := r.Get(ctx, req.NamespacedName, memcached)
        ...
      }

      Based on the return value of the Reconcile() function, the reconcile Request might be requeued, and the loop might be triggered again:

      Example 5.6. Requeue logic

      // Reconcile successful - don't requeue
      return reconcile.Result{}, nil
      // Reconcile failed due to error - requeue
      return reconcile.Result{}, err
      // Requeue for any reason other than error
      return reconcile.Result{Requeue: true}, nil

      You can set the Result.RequeueAfter to requeue the request after a grace period:

      Example 5.7. Requeue after grace period

      import "time"
      
      // Reconcile for any reason other than an error after 5 seconds
      return ctrl.Result{RequeueAfter: time.Second*5}, nil

      Note

      You can return Result with RequeueAfter set to periodically reconcile a CR.

      For more on reconcilers, clients, and interacting with resource events, see the Controller Runtime Client API documentation.

Procedure

1. Change to the namespace where your Operator is installed. For example, if you deployed the Operator using the make deploy command:

   $ oc project memcached-operator-system

2. Edit the sample Memcached CR manifest at config/samples/cache_v1_memcached.yaml to contain the following specification:

   apiVersion: cache.example.com/v1
   kind: Memcached
   metadata:
     name: memcached-sample
   ...
   spec:
   ...
     size: 3

3. Create the CR:

   $ oc apply -f config/samples/cache_v1_memcached.yaml

4. Ensure that the Memcached Operator creates the deployment for the sample CR with the correct size:

   $ oc get deployments

   Example output

   NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
   memcached-operator-controller-manager   1/1     1            1           8m
   memcached-sample                        3/3     3            3           1m

5. Check the pods and CR status to confirm the status is updated with the Memcached pod names.

   1. Check the pods:

      $ oc get pods

      Example output

      NAME                                  READY     STATUS    RESTARTS   AGE
      memcached-sample-6fd7c98d8-7dqdr      1/1       Running   0          1m
      memcached-sample-6fd7c98d8-g5k7v      1/1       Running   0          1m
      memcached-sample-6fd7c98d8-m7vn7      1/1       Running   0          1m

   2. Check the CR status:

      $ oc get memcached/memcached-sample -o yaml

      Example output

      apiVersion: cache.example.com/v1
      kind: Memcached
      metadata:
      ...
        name: memcached-sample
      ...
      spec:
        size: 3
      status:
        nodes:
        - memcached-sample-6fd7c98d8-7dqdr
        - memcached-sample-6fd7c98d8-g5k7v
        - memcached-sample-6fd7c98d8-m7vn7

6. Update the deployment size.

   1. Update config/samples/cache_v1_memcached.yaml file to change the spec.size field in the Memcached CR from 3 to 5:

      $ oc patch memcached memcached-sample \
          -p '{"spec":{"size": 5}}' \
          --type=merge

   2. Confirm that the Operator changes the deployment size:

      $ oc get deployments

      Example output

      NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
      memcached-operator-controller-manager   1/1     1            1           10m
      memcached-sample                        5/5     5            5           3m

Procedure

1. Create a new Operator project. A namespace-scoped Operator watches and manages resources in a single namespace. Namespace-scoped Operators are preferred because of their flexibility. They enable decoupled upgrades, namespace isolation for failures and monitoring, and differing API definitions.

   To create a new Ansible-based, namespace-scoped memcached-operator project and change to the new directory, use the following commands:

   $ operator-sdk new memcached-operator \
       --api-version=cache.example.com/v1alpha1 \
       --kind=Memcached \
       --type=ansible

   $ cd memcached-operator

   This creates the memcached-operator project specifically for watching the Memcached resource with API version example.com/v1apha1 and kind Memcached.

2. Customize the Operator logic.

   For this example, the memcached-operator executes the following reconciliation logic for each Memcached custom resource (CR):

   • Create a memcached deployment if it does not exist.
   • Ensure that the deployment size is the same as specified by the Memcached CR.

   By default, the memcached-operator watches Memcached resource events as shown in the watches.yaml file and executes the Ansible role Memcached:

   - version: v1alpha1
     group: cache.example.com
     kind: Memcached

   You can optionally customize the following logic in the watches.yaml file:

   1. Specifying a role option configures the Operator to use this specified path when launching ansible-runner with an Ansible role. By default, the operator-sdk new command fills in an absolute path to where your role should go:

      - version: v1alpha1
        group: cache.example.com
        kind: Memcached
        role: /opt/ansible/roles/memcached

   2. Specifying a playbook option in the watches.yaml file configures the Operator to use this specified path when launching ansible-runner with an Ansible playbook:

      - version: v1alpha1
        group: cache.example.com
        kind: Memcached
        playbook: /opt/ansible/playbook.yaml

3. Build the Memcached Ansible role.

   Modify the generated Ansible role under the roles/memcached/ directory. This Ansible role controls the logic that is executed when a resource is modified.

   1. Define the Memcached spec.

      Defining the spec for an Ansible-based Operator can be done entirely in Ansible. The Ansible Operator passes all key-value pairs listed in the CR spec field along to Ansible as variables. The names of all variables in the spec field are converted to snake case (lowercase with an underscore) by the Operator before running Ansible. For example, serviceAccount in the spec becomes service_account in Ansible.

      Tip

      You should perform some type validation in Ansible on the variables to ensure that your application is receiving expected input.

      In case the user does not set the spec field, set a default by modifying the roles/memcached/defaults/main.yml file:

      size: 1

   2. Define the Memcached deployment.

      With the Memcached spec now defined, you can define what Ansible is actually executed on resource changes. Because this is an Ansible role, the default behavior executes the tasks in the roles/memcached/tasks/main.yml file.

      The goal is for Ansible to create a deployment if it does not exist, which runs the memcached:1.4.36-alpine image. Ansible 2.7+ supports the k8s Ansible module, which this example leverages to control the deployment definition.

      Modify the roles/memcached/tasks/main.yml to match the following:

      - name: start memcached
        k8s:
          definition:
            kind: Deployment
            apiVersion: apps/v1
            metadata:
              name: '{{ meta.name }}-memcached'
              namespace: '{{ meta.namespace }}'
            spec:
              replicas: "{{size}}"
              selector:
                matchLabels:
                  app: memcached
              template:
                metadata:
                  labels:
                    app: memcached
                spec:
                  containers:
                  - name: memcached
                    command:
                    - memcached
                    - -m=64
                    - -o
                    - modern
                    - -v
                    image: "docker.io/memcached:1.4.36-alpine"
                    ports:
                      - containerPort: 11211

      Note

      This example used the size variable to control the number of replicas of the Memcached deployment. This example sets the default to 1, but any user can create a CR that overwrites the default.

4. Deploy the CRD.

   Before running the Operator, Kubernetes needs to know about the new custom resource definition (CRD) that the Operator will be watching. Deploy the Memcached CRD:

   $ oc create -f deploy/crds/cache.example.com_memcacheds_crd.yaml

5. Build and run the Operator.

   There are two ways to build and run the Operator:

   • As a pod inside a Kubernetes cluster.
   • As a Go program outside the cluster using the operator-sdk up command.

   Choose one of the following methods:

   1. Run as a pod inside a Kubernetes cluster. This is the preferred method for production use.

      1. Build the memcached-operator image and push it to a registry:

         $ operator-sdk build quay.io/example/memcached-operator:v0.0.1

         $ podman push quay.io/example/memcached-operator:v0.0.1

      2. Deployment manifests are generated in the deploy/operator.yaml file. The deployment image in this file needs to be modified from the placeholder REPLACE_IMAGE to the previous built image. To do this, run:

         $ sed -i 's|REPLACE_IMAGE|quay.io/example/memcached-operator:v0.0.1|g' deploy/operator.yaml

      3. Deploy the memcached-operator manifests:

         $ oc create -f deploy/service_account.yaml

         $ oc create -f deploy/role.yaml

         $ oc create -f deploy/role_binding.yaml

         $ oc create -f deploy/operator.yaml

      4. Verify that the memcached-operator deployment is up and running:

         $ oc get deployment

         NAME                     DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
         memcached-operator       1         1         1            1           1m

   2. Run outside the cluster. This method is preferred during the development cycle to speed up deployment and testing.

      Ensure that Ansible Runner and Ansible Runner HTTP Plug-in are installed or else you will see unexpected errors from Ansible Runner when a CR is created.

      It is also important that the role path referenced in the watches.yaml file exists on your machine. Because normally a container is used where the role is put on disk, the role must be manually copied to the configured Ansible roles path (for example /etc/ansible/roles).

      1. To run the Operator locally with the default Kubernetes configuration file present at $HOME/.kube/config:

         $ operator-sdk run --local

         To run the Operator locally with a provided Kubernetes configuration file:

         $ operator-sdk run --local --kubeconfig=config

6. Create a Memcached CR.

   1. Modify the deploy/crds/cache_v1alpha1_memcached_cr.yaml file as shown and create a Memcached CR:

      $ cat deploy/crds/cache_v1alpha1_memcached_cr.yaml

      Example output

      apiVersion: "cache.example.com/v1alpha1"
      kind: "Memcached"
      metadata:
        name: "example-memcached"
      spec:
        size: 3

      $ oc apply -f deploy/crds/cache_v1alpha1_memcached_cr.yaml

   2. Ensure that the memcached-operator creates the deployment for the CR:

      $ oc get deployment

      Example output

      NAME                     DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
      memcached-operator       1         1         1            1           2m
      example-memcached        3         3         3            3           1m

   3. Check the pods to confirm three replicas were created:

      $ oc get pods

      NAME                                  READY     STATUS    RESTARTS   AGE
      example-memcached-6fd7c98d8-7dqdr     1/1       Running   0          1m
      example-memcached-6fd7c98d8-g5k7v     1/1       Running   0          1m
      example-memcached-6fd7c98d8-m7vn7     1/1       Running   0          1m
      memcached-operator-7cc7cfdf86-vvjqk   1/1       Running   0          2m

7. Update the size.

   1. Change the spec.size field in the memcached CR from 3 to 4 and apply the change:

      $ cat deploy/crds/cache_v1alpha1_memcached_cr.yaml

      Example output

      apiVersion: "cache.example.com/v1alpha1"
      kind: "Memcached"
      metadata:
        name: "example-memcached"
      spec:
        size: 4

      $ oc apply -f deploy/crds/cache_v1alpha1_memcached_cr.yaml

   2. Confirm that the Operator changes the deployment size:

      $ oc get deployment

      Example output

      NAME                 DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
      example-memcached    4         4         4            4           5m

8. Clean up the resources:

   $ oc delete -f deploy/crds/cache_v1alpha1_memcached_cr.yaml

   $ oc delete -f deploy/operator.yaml

   $ oc delete -f deploy/role_binding.yaml

   $ oc delete -f deploy/role.yaml

   $ oc delete -f deploy/service_account.yaml

   $ oc delete -f deploy/crds/cache_v1alpha1_memcached_crd.yaml

Procedure

1. To track CR status manually from your application, update the watches.yaml file with a manageStatus field set to false:

   - version: v1
     group: api.example.com
     kind: Test1
     role: Test1
     manageStatus: false

2. Use the operator_sdk.util.k8s_status Ansible module to update the subresource. For example, to update with key test1 and value test2, operator_sdk.util can be used as shown:

   - operator_sdk.util.k8s_status:
       api_version: app.example.com/v1
       kind: Test1
       name: "{{ meta.name }}"
       namespace: "{{ meta.namespace }}"
       status:
         test1: test2

   Collections can also be declared in the meta/main.yml for the role, which is included for new scaffolded Ansible Operators:

   collections:
     - operator_sdk.util

   Declaring collections in the role meta allows you to invoke the k8s_status module directly:

   k8s_status:
     <snip>
     status:
       test1: test2

Procedure

1. Create a new Operator project. A namespace-scoped Operator watches and manages resources in a single namespace. Namespace-scoped Operators are preferred because of their flexibility. They enable decoupled upgrades, namespace isolation for failures and monitoring, and differing API definitions.

   To create a new Helm-based, namespace-scoped nginx-operator project, use the following command:

   $ operator-sdk new nginx-operator \
     --api-version=example.com/v1alpha1 \
     --kind=Nginx \
     --type=helm

   $ cd nginx-operator

   This creates the nginx-operator project specifically for watching the Nginx resource with API version example.com/v1apha1 and kind Nginx.

2. Customize the Operator logic.

   For this example, the nginx-operator executes the following reconciliation logic for each Nginx custom resource (CR):

   • Create an Nginx deployment if it does not exist.
   • Create an Nginx service if it does not exist.
   • Create an Nginx ingress if it is enabled and does not exist.
   • Ensure that the deployment, service, and optional ingress match the desired configuration (for example, replica count, image, service type) as specified by the Nginx CR.

   By default, the nginx-operator watches Nginx resource events as shown in the watches.yaml file and executes Helm releases using the specified chart:

   - version: v1alpha1
     group: example.com
     kind: Nginx
     chart: /opt/helm/helm-charts/nginx

   1. Review the Nginx Helm chart.

      When a Helm Operator project is created, the Operator SDK creates an example Helm chart that contains a set of templates for a simple Nginx release.

      For this example, templates are available for deployment, service, and ingress resources, along with a NOTES.txt template, which Helm chart developers use to convey helpful information about a release.

      If you are not already familiar with Helm Charts, review the Helm Chart developer documentation.

   2. Understand the Nginx CR spec.

      Helm uses a concept called values to provide customizations to the defaults of a Helm chart, which are defined in the values.yaml file.

      Override these defaults by setting the desired values in the CR spec. You can use the number of replicas as an example:

      1. First, inspect the helm-charts/nginx/values.yaml file to find that the chart has a value called replicaCount and it is set to 1 by default. To have 2 Nginx instances in your deployment, your CR spec must contain replicaCount: 2.

         Update the deploy/crds/example.com_v1alpha1_nginx_cr.yaml file to look like the following:

         apiVersion: example.com/v1alpha1
         kind: Nginx
         metadata:
           name: example-nginx
         spec:
           replicaCount: 2

      2. Similarly, the default service port is set to 80. To instead use 8080, update the deploy/crds/example.com_v1alpha1_nginx_cr.yaml file again by adding the service port override:

         apiVersion: example.com/v1alpha1
         kind: Nginx
         metadata:
           name: example-nginx
         spec:
           replicaCount: 2
           service:
             port: 8080

         The Helm Operator applies the entire spec as if it was the contents of a values file, just like the helm install -f ./overrides.yaml command works.

3. Deploy the CRD.

   Before running the Operator, Kubernetes must know about the new custom resource definition (CRD) that the Operator will be watching. Deploy the following CRD:

   $ oc create -f deploy/crds/example_v1alpha1_nginx_crd.yaml

4. Build and run the Operator.

   There are two ways to build and run the Operator:

   • As a pod inside a Kubernetes cluster.
   • As a Go program outside the cluster using the operator-sdk up command.

   Choose one of the following methods:

   1. Run as a pod inside a Kubernetes cluster. This is the preferred method for production use.

      1. Build the nginx-operator image and push it to a registry:

         $ operator-sdk build quay.io/example/nginx-operator:v0.0.1

         $ podman push quay.io/example/nginx-operator:v0.0.1

      2. Deployment manifests are generated in the deploy/operator.yaml file. The deployment image in this file needs to be modified from the placeholder REPLACE_IMAGE to the previous built image. To do this, run:

         $ sed -i 's|REPLACE_IMAGE|quay.io/example/nginx-operator:v0.0.1|g' deploy/operator.yaml

      3. Deploy the nginx-operator manifests:

         $ oc create -f deploy/service_account.yaml

         $ oc create -f deploy/role.yaml

         $ oc create -f deploy/role_binding.yaml

         $ oc create -f deploy/operator.yaml

      4. Verify that the nginx-operator deployment is up and running:

         $ oc get deployment

         Example output

         NAME                 DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
         nginx-operator       1         1         1            1           1m

   2. Run outside the cluster. This method is preferred during the development cycle to speed up deployment and testing.

      It is important that the chart path referenced in the watches.yaml file exists on your machine. By default, the watches.yaml file is scaffolded to work with an Operator image built with the operator-sdk build command. When developing and testing your Operator with the operator-sdk run --local command, the SDK looks in your local file system for this path.

      1. Create a symlink at this location to point to the path of your Helm chart:

         $ sudo mkdir -p /opt/helm/helm-charts

         $ sudo ln -s $PWD/helm-charts/nginx /opt/helm/helm-charts/nginx

      2. To run the Operator locally with the default Kubernetes configuration file present at $HOME/.kube/config:

         $ operator-sdk run --local

         To run the Operator locally with a provided Kubernetes configuration file:

         $ operator-sdk run --local --kubeconfig=<path_to_config>

5. Deploy the Nginx CR.

   Apply the Nginx CR that you modified earlier:

   $ oc apply -f deploy/crds/example.com_v1alpha1_nginx_cr.yaml

   Ensure that the nginx-operator creates the deployment for the CR:

   $ oc get deployment

   Example output

   NAME                                           DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
   example-nginx-b9phnoz9spckcrua7ihrbkrt1        2         2         2            2           1m

   Check the pods to confirm two replicas were created:

   $ oc get pods

   Example output

   NAME                                                      READY     STATUS    RESTARTS   AGE
   example-nginx-b9phnoz9spckcrua7ihrbkrt1-f8f9c875d-fjcr9   1/1       Running   0          1m
   example-nginx-b9phnoz9spckcrua7ihrbkrt1-f8f9c875d-ljbzl   1/1       Running   0          1m

   Check that the service port is set to 8080:

   $ oc get service

   Example output

   NAME                                      TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
   example-nginx-b9phnoz9spckcrua7ihrbkrt1   ClusterIP   10.96.26.3   <none>        8080/TCP   1m

6. Update the replicaCount and remove the port.

   Change the spec.replicaCount field from 2 to 3, remove the spec.service field, and apply the change:

   $ cat deploy/crds/example.com_v1alpha1_nginx_cr.yaml

   Example output

   apiVersion: "example.com/v1alpha1"
   kind: "Nginx"
   metadata:
     name: "example-nginx"
   spec:
     replicaCount: 3

   $ oc apply -f deploy/crds/example.com_v1alpha1_nginx_cr.yaml

   Confirm that the Operator changes the deployment size:

   $ oc get deployment

   Example output

   NAME                                           DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
   example-nginx-b9phnoz9spckcrua7ihrbkrt1        3         3         3            3           1m

   Check that the service port is set to the default 80:

   $ oc get service

   Example output

   NAME                                      TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)  AGE
   example-nginx-b9phnoz9spckcrua7ihrbkrt1   ClusterIP   10.96.26.3   <none>        80/TCP   1m

7. Clean up the resources:

   $ oc delete -f deploy/crds/example.com_v1alpha1_nginx_cr.yaml

   $ oc delete -f deploy/operator.yaml

   $ oc delete -f deploy/role_binding.yaml

   $ oc delete -f deploy/role.yaml

   $ oc delete -f deploy/service_account.yaml

   $ oc delete -f deploy/crds/example_v1alpha1_nginx_crd.yaml

Procedure

1. In your Operator project, configure your CSV composition by modifying the deploy/olm-catalog/csv-config.yaml file, if desired.

2. Generate the CSV:

   $ operator-sdk generate csv --csv-version <version>

3. In the new CSV generated in the deploy/olm-catalog/ directory, ensure all required, manually-defined fields are set appropriately.

Procedure

1. Use SHA references to related images in two places in the CSV for your Operator:

   1. Update spec.relatedImages:

      ...
      spec:
        relatedImages: 

      1

      
          - name: etcd-operator 

      2

      
            image: quay.io/etcd-operator/operator@sha256:d134a9865524c29fcf75bbc4469013bc38d8a15cb5f41acfddb6b9e492f556e4 

      3

      
          - name: etcd-image
            image: quay.io/etcd-operator/etcd@sha256:13348c15263bd8838ec1d5fc4550ede9860fcbb0f843e48cbccec07810eebb68
      ...

      1

      Create a relatedImages section and set the list of related images.

      2

      Specify a unique identifier for the image.

      3

      Specify each image by a digest (SHA), not by an image tag.

   2. Update the env section in the deployment when declaring environment variables that inject the image that the Operator should use:

      spec:
        install:
          spec:
            deployments:
            - name: etcd-operator-v3.1.1
              spec:
                replicas: 1
                selector:
                  matchLabels:
                    name: etcd-operator
                strategy:
                  type: Recreate
                template:
                  metadata:
                    labels:
                      name: etcd-operator
                  spec:
                    containers:
                    - args:
                      - /opt/etcd/bin/etcd_operator_run.sh
                      env:
                      - name: WATCH_NAMESPACE
                        valueFrom:
                          fieldRef:
                            fieldPath: metadata.annotations['olm.targetNamespaces']
                      - name: ETCD_OPERATOR_DEFAULT_ETCD_IMAGE 

      1

      
                        value: quay.io/etcd-operator/etcd@sha256:13348c15263bd8838ec1d5fc4550ede9860fcbb0f843e48cbccec07810eebb68 

      2

      
                      - name: ETCD_LOG_LEVEL
                        value: INFO
                      image: quay.io/etcd-operator/operator@sha256:d134a9865524c29fcf75bbc4469013bc38d8a15cb5f41acfddb6b9e492f556e4 

      3

      
                      imagePullPolicy: IfNotPresent
                      livenessProbe:
                        httpGet:
                          path: /healthy
                          port: 8080
                        initialDelaySeconds: 10
                        periodSeconds: 30
                      name: etcd-operator
                      readinessProbe:
                        httpGet:
                          path: /ready
                          port: 8080
                        initialDelaySeconds: 10
                        periodSeconds: 30
                      resources: {}
                    serviceAccountName: etcd-operator
          strategy: deployment

      1

      Inject the images referenced by the Operator by using environment variables.

      2

      Specify each image by a digest (SHA), not by an image tag.

      3

      Also reference the Operator container image by a digest (SHA), not by an image tag.

      Note

      When configuring probes, the timeoutSeconds value must be lower than the periodSeconds value. The timeoutSeconds default value is 1. The periodSeconds default value is 10.

2. Add the disconnected annotation, which indicates that the Operator works in a disconnected environment:

   metadata:
     annotations:
       operators.openshift.io/infrastructure-features: '["disconnected"]'

   Operators can be filtered in OperatorHub by this infrastructure feature.

Procedure

1. Run the following make commands in your Operator project directory to build and push your Operator image. Modify the IMG argument in the following steps to reference a repository that you have access to. You can obtain an account for storing containers at repository sites such as Quay.io.

   1. Build the image:

      $ make docker-build IMG=<registry>/<user>/<operator_image_name>:<tag>

      Note

      The Dockerfile generated by the SDK for the Operator explicitly references GOARCH=amd64 for go build. This can be amended to GOARCH=$TARGETARCH for non-AMD64 architectures. Docker will automatically set the environment variable to the value specified by –platform. With Buildah, the –build-arg will need to be used for the purpose. For more information, see Multiple Architectures.

   2. Push the image to a repository:

      $ make docker-push IMG=<registry>/<user>/<operator_image_name>:<tag>

2. Update your Makefile by setting the IMG URL to your Operator image name and tag that you pushed:

   $ # Image URL to use all building/pushing image targets
   IMG ?= <registry>/<user>/<operator_image_name>:<tag>

   This value is used for subsequent operations.

3. Create your Operator bundle manifest by running the make bundle command, which invokes several commands, including the Operator SDK generate bundle and bundle validate subcommands:

   $ make bundle

   Bundle manifests for an Operator describe how to display, create, and manage an application. The make bundle command creates the following files and directories in your Operator project:

   • A bundle manifests directory named bundle/manifests that contains a ClusterServiceVersion object
   • A bundle metadata directory named bundle/metadata
   • All custom resource definitions (CRDs) in a config/crd directory
   • A Dockerfile bundle.Dockerfile

   These files are then automatically validated by using operator-sdk bundle validate to ensure the on-disk bundle representation is correct.

4. Build and push your bundle image by running the following commands. OLM consumes Operator bundles using an index image, which reference one or more bundle images.

   1. Build the bundle image. Set BUNDLE_IMG with the details for the registry, user namespace, and image tag where you intend to push the image:

      $ make bundle-build BUNDLE_IMG=<registry>/<user>/<bundle_image_name>:<tag>

   2. Push the bundle image:

      $ docker push <registry>/<user>/<bundle_image_name>:<tag>

Procedure

1. Define a .osdk-scorecard.yaml configuration file in your Operator project.
2. Create the namespace defined in the RBAC files (role_binding).

3. Run the scorecard from the root directory of your Operator project:

   $ operator-sdk scorecard

   The scorecard return code is 1 if any of the executed texts did not pass and 0 if all selected tests passed.

Procedure

1. The scorecard requires a proxy container in the deployment pod of the Operator to read Operator logs. A few modifications to your CSV and creation of one extra object are required to run the proxy before deploying your Operator with Operator Lifecycle Manager (OLM).

   This step can be performed manually or automated using bash functions. Choose one of the following methods.

   • Manual method:

     1. Create a proxy server secret containing a local kubeconfig file`.

        1. Generate a user name using the namespaced owner reference of the scorecard proxy.

           $ echo '{"apiVersion":"","kind":"","name":"scorecard","uid":"","Namespace":"'<namespace>'"}' | base64 -w 0 

           1

           1

           Replace <namespace> with the namespace your Operator will deploy in.

        2. Write a Config manifest scorecard-config.yaml using the following template, replacing <username> with the base64 user name generated in the previous step:

           apiVersion: v1
           kind: Config
           clusters:
           - cluster:
               insecure-skip-tls-verify: true
               server: http://<username>@localhost:8889
             name: proxy-server
           contexts:
           - context:
               cluster: proxy-server
               user: admin/proxy-server
             name: <namespace>/proxy-server
           current-context: <namespace>/proxy-server
           preferences: {}
           users:
           - name: admin/proxy-server
             user:
               username: <username>
               password: unused

        3. Encode the Config as base64:

           $ cat scorecard-config.yaml | base64 -w 0

        4. Create a Secret manifest scorecard-secret.yaml:

           apiVersion: v1
           kind: Secret
           metadata:
             name: scorecard-kubeconfig
             namespace: <namespace> 

           1

           
           data:
             kubeconfig: <kubeconfig_base64> 

           2

           1

           Replace <namespace> with the namespace your Operator will deploy in.

           2

           Replace <kubeconfig_base64> with the Config encoded as base64.

        5. Apply the secret:

           $ oc apply -f scorecard-secret.yaml

        6. Insert a volume referring to the secret into the deployment for the Operator:

           spec:
             install:
               spec:
                 deployments:
                 - name: memcached-operator
                   spec:
                     ...
                     template:
                       ...
                       spec:
                         containers:
                         ...
                         volumes:
                         - name: scorecard-kubeconfig 

           1

           
                           secret:
                             secretName: scorecard-kubeconfig
                             items:
                             - key: kubeconfig
                               path: config

           1

           Scorecard kubeconfig volume.

     2. Insert a volume mount and KUBECONFIG environment variable into each container in the deployment of your Operator:

        spec:
          install:
            spec:
              deployments:
              - name: memcached-operator
                spec:
                  ...
                  template:
                    ...
                    spec:
                      containers:
                      - name: container1
                        ...
                        volumeMounts:
                        - name: scorecard-kubeconfig 

        1

        
                          mountPath: /scorecard-secret
                        env:
                        - name: KUBECONFIG 

        2

        
                          value: /scorecard-secret/config
                      - name: container2 

        3

        
                        ...

        1

        Scorecard kubeconfig volume mount.

        2

        Scorecard kubeconfig environment variable.

        3

        Repeat the same for this and all other containers.

     3. Insert the scorecard proxy container into the deployment of your Operator:

        spec:
          install:
            spec:
              deployments:
              - name: memcached-operator
                spec:
                  ...
                  template:
                    ...
                    spec:
                      containers:
                      ...
                      - name: scorecard-proxy 

        1

        
                        command:
                        - scorecard-proxy
                        env:
                        - name: WATCH_NAMESPACE
                          valueFrom:
                            fieldRef:
                              apiVersion: v1
                              fieldPath: metadata.namespace
                        image: quay.io/operator-framework/scorecard-proxy:master
                        imagePullPolicy: Always
                        ports:
                        - name: proxy
                          containerPort: 8889

        1

        Scorecard proxy container.

   • Automated method:

     The community-operators repository has several bash functions that can perform the previous steps in the procedure for you.

     1. Run the following curl command:

        $ curl -Lo csv-manifest-modifiers.sh \
            https://raw.githubusercontent.com/operator-framework/community-operators/master/scripts/lib/file

     2. Source the csv-manifest-modifiers.sh file:

        $ . ./csv-manifest-modifiers.sh

     3. Create the kubeconfig secret file:

        $ create_kubeconfig_secret_file scorecard-secret.yaml "<namespace>" 

        1

        1

        Replace <namespace> with the namespace your Operator will deploy in.

     4. Apply the secret:

        $ oc apply -f scorecard-secret.yaml

     5. Insert the kubeconfig volume:

        $ insert_kubeconfig_volume "<csv_file>" 

        1

        1

        Replace <csv_file> with the path to your CSV manifest.

     6. Insert the kubeconfig secret mount:

        $ insert_kubeconfig_secret_mount "<csv_file>"

     7. Insert the proxy container:

        $ insert_proxy_container "<csv_file>" "quay.io/operator-framework/scorecard-proxy:master"

2. After inserting the proxy container, follow the steps in the Getting started with the Operator SDK guide to bundle your CSV and custom resource definitions (CRDs) and deploy your Operator on OLM.
3. After your Operator has been deployed on OLM, define a .osdk-scorecard.yaml configuration file in your Operator project and ensure both the csv-path: <csv_manifest_path> and olm-deployed options are set.

4. Run the scorecard with both the csv-path: <csv_manifest_path> and olm-deployed options set in your scorecard configuration file:

   $ operator-sdk scorecard