Chapter 5. Developing Operators

Procedure

1. Create a directory for the project:

   $ mkdir -p $HOME/projects/memcached-operator

2. Change to the directory:

   $ cd $HOME/projects/memcached-operator

3. Activate support for Go modules:

   $ export GO111MODULE=on

4. Run the operator-sdk init command to initialize the project:

   $ operator-sdk init \
       --domain=example.com \
       --repo=github.com/example-inc/memcached-operator

   Note

   The operator-sdk init command uses the Go plugin by default.

   The operator-sdk init command generates a go.mod file to be used with Go modules. The --repo flag is required when creating a project outside of $GOPATH/src/, because generated files require a valid module path.

Procedure

1. Run the following command to create an API with group cache, version, v1, and kind Memcached:

   $ operator-sdk create api \
       --group=cache \
       --version=v1 \
       --kind=Memcached

2. When prompted, enter y for creating both the resource and controller:

   Create Resource [y/n]
   y
   Create Controller [y/n]
   y

   Example output

   Writing scaffold for you to edit...
   api/v1/memcached_types.go
   controllers/memcached_controller.go
   ...

Procedure

1. Edit the controllers/memcached_controller.go file to include the following:

   1. Import the proxy package from the operator-lib library:

      import (
        ...
         "github.com/operator-framework/operator-lib/proxy"
      )

   2. Add the proxy.ReadProxyVarsFromEnv helper function to the reconcile loop and append the results to the Operand environments:

      for i, container := range dep.Spec.Template.Spec.Containers {
      		dep.Spec.Template.Spec.Containers[i].Env = append(container.Env, proxy.ReadProxyVarsFromEnv()...)
      }
      ...

2. Set the environment variable on the Operator deployment by adding the following to the config/manager/manager.yaml file:

   containers:
    - args:
      - --leader-elect
      - --leader-election-id=ansible-proxy-demo
      image: controller:latest
      name: manager
      env:
        - name: "HTTP_PROXY"
          value: "http_proxy_test"

Procedure

1. Change to the namespace where your Operator is installed. For example, if you deployed the Operator using the make deploy command:

   $ oc project memcached-operator-system

2. Edit the sample Memcached CR manifest at config/samples/cache_v1_memcached.yaml to contain the following specification:

   apiVersion: cache.example.com/v1
   kind: Memcached
   metadata:
     name: memcached-sample
   ...
   spec:
   ...
     size: 3

3. Create the CR:

   $ oc apply -f config/samples/cache_v1_memcached.yaml

4. Ensure that the Memcached Operator creates the deployment for the sample CR with the correct size:

   $ oc get deployments

   Example output

   NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
   memcached-operator-controller-manager   1/1     1            1           8m
   memcached-sample                        3/3     3            3           1m

5. Check the pods and CR status to confirm the status is updated with the Memcached pod names.

   1. Check the pods:

      $ oc get pods

      Example output

      NAME                                  READY     STATUS    RESTARTS   AGE
      memcached-sample-6fd7c98d8-7dqdr      1/1       Running   0          1m
      memcached-sample-6fd7c98d8-g5k7v      1/1       Running   0          1m
      memcached-sample-6fd7c98d8-m7vn7      1/1       Running   0          1m

   2. Check the CR status:

      $ oc get memcached/memcached-sample -o yaml

      Example output

      apiVersion: cache.example.com/v1
      kind: Memcached
      metadata:
      ...
        name: memcached-sample
      ...
      spec:
        size: 3
      status:
        nodes:
        - memcached-sample-6fd7c98d8-7dqdr
        - memcached-sample-6fd7c98d8-g5k7v
        - memcached-sample-6fd7c98d8-m7vn7

6. Update the deployment size.

   1. Update config/samples/cache_v1_memcached.yaml file to change the spec.size field in the Memcached CR from 3 to 5:

      $ oc patch memcached memcached-sample \
          -p '{"spec":{"size": 5}}' \
          --type=merge

   2. Confirm that the Operator changes the deployment size:

      $ oc get deployments

      Example output

      NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
      memcached-operator-controller-manager   1/1     1            1           10m
      memcached-sample                        5/5     5            5           3m

7. Delete the CR by running the following command:

   $ oc delete -f config/samples/cache_v1_memcached.yaml

8. Clean up the resources that have been created as part of this tutorial.

   • If you used the make deploy command to test the Operator, run the following command:

     $ make undeploy

   • If you used the operator-sdk run bundle command to test the Operator, run the following command:

     $ operator-sdk cleanup <project_name>

Procedure

1. Edit your Operator project’s Makefile to update the Operator SDK version to v1.36.1-ocp, as shown in the following example:

   Example Makefile

   # Set the Operator SDK version to use. By default, what is installed on the system is used.
   # This is useful for CI or a project to utilize a specific version of the operator-sdk toolkit.
   OPERATOR_SDK_VERSION ?= v1.36.1-ocp

2. Update the kube-rbac-proxy container to use the Red Hat Enterprise Linux (RHEL) 9-based image:

   1. Find the entry for the kube-rbac-proxy container in the following files:

      • config/default/manager_auth_proxy_patch.yaml
      • bundle/manifests/<operator_name>.clusterserviceversion.yaml for your Operator project, for example memcached-operator.clusterserviceversion.yaml from the tutorials

   2. Update the image name in the pull spec from ose-kube-rbac-proxy to ose-kube-rbac-proxy-rhel9, and update the tag to v4:

      Example ose-kube-rbac-proxy-rhel9 pull spec with v4 image tag

      # ...
            containers:
            - name: kube-rbac-proxy
              image: registry.redhat.io/openshift4/ose-kube-rbac-proxy-rhel9:v4
      # ...

3. The go/v4 plugin is now stable and is the default version used when scaffolding a Go-based Operator. The transition from Golang v2 and v3 plugins to the new Golang v4 plugin introduces significant changes. This migration is designed to enhance your project’s functionality and compatibility, reflecting the evolving landscape of Golang development.

   For more information on the reasoning behind these changes, see go/v3 vs go/v4 in the Kubebuilder documentation.

   For a comprehensive understanding of the migration process to the v4 plugin format and detailed migration steps, see Migration from go/v3 to go/v4 by updating the files manually in the Kubebuilder documentation.

4. The kustomize/v2 plugin is now stable and is the default version used in the plugin chain when using go/v4, ansible/v1, helm/v1, and hybrid/v1-alpha plugins. For more information on this default scaffold, see Kustomize v2 in the Kubebuilder documentation.

5. If your Operator project uses a multi-platform, or multi-archicture, build, replace the existing docker-buildx target with following definition in your project Makefile:

   Example Makefile

   docker-buildx:
   ## Build and push the Docker image for the manager for multi-platform support
   	- docker buildx create --name project-v3-builder
   	docker buildx use project-v3-builder
   	- docker buildx build --push --platform=$(PLATFORMS) --tag ${IMG} -f Dockerfile .
   	- docker buildx rm project-v3-builder

6. You must upgrade the Kubernetes versions in your Operator project to use 1.29. The following changes must be made in your project structure, Makefile, and go.mod files.

   Important

   The go/v3 plugin is being deprecated by Kubebuilder, therefore Operator SDK is also migrating to go/v4 in a future release.

   1. Update your go.mod file to upgrade your dependencies:

      k8s.io/api v0.29.2
      k8s.io/apimachinery v0.29.2
      k8s.io/client-go v0.29.2
      sigs.k8s.io/controller-runtime v0.17.3

   2. Download the upgraded dependencies by running the following command:

      $ go mod tidy

   3. You can now generate a file that contains all the resources built with Kustomize, which are necessary to install this project without its dependencies. Update your Makefile by making the following changes:

      + .PHONY: build-installer
      +   build-installer: manifests generate kustomize ## Generate a consolidated YAML with CRDs and deployment.
      +   	mkdir -p dist
      +   	cd config/manager && $(KUSTOMIZE) edit set image controller=${IMG}
      +   	$(KUSTOMIZE) build config/default > dist/install.yaml

   4. Update the ENVTEST_K8S_VERSION variable in your Makefile by making the following changes:

      - ENVTEST_K8S_VERSION = 1.28.3
      + ENVTEST_K8S_VERSION = 1.29.0

   5. Remove the following section from your Makefile:

      - GOLANGCI_LINT = $(shell pwd)/bin/golangci-lint
      - GOLANGCI_LINT_VERSION ?= v1.54.2
      - golangci-lint:
      - 	@[ -f $(GOLANGCI_LINT) ] || { \
      - 	set -e ;\
      - 	curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(shell dirname $(GOLANGCI_LINT)) $(GOLANGCI_LINT_VERSION) ;\
      - 	}

   6. Update your Makefile by making the following changes:

      Example 5.2. Makefile changes

      - ## Tool Binaries
      - KUBECTL ?= kubectl
      - KUSTOMIZE ?= $(LOCALBIN)/kustomize
      - CONTROLLER_GEN ?= $(LOCALBIN)/controller-gen
      - ENVTEST ?= $(LOCALBIN)/setup-envtest
      -
      - ## Tool Versions
      - KUSTOMIZE_VERSION ?= v5.2.1
      - CONTROLLER_TOOLS_VERSION ?= v0.13.0
      -
      - .PHONY: kustomize
      - kustomize: $(KUSTOMIZE) ## Download kustomize locally if necessary. If wrong version is installed, it will be removed before downloading.
      - $(KUSTOMIZE): $(LOCALBIN)
      -   @if test -x $(LOCALBIN)/kustomize && ! $(LOCALBIN)/kustomize version | grep -q $(KUSTOMIZE_VERSION); then \
      -   echo "$(LOCALBIN)/kustomize version is not expected $(KUSTOMIZE_VERSION). Removing it before installing."; \
      -   rm -rf $(LOCALBIN)/kustomize; \
      -   fi
      -   test -s $(LOCALBIN)/kustomize || GOBIN=$(LOCALBIN) GO111MODULE=on go install sigs.k8s.io/kustomize/kustomize/v5@$(KUSTOMIZE_VERSION)
      -
      - .PHONY: controller-gen
      - controller-gen: $(CONTROLLER_GEN) ## Download controller-gen locally if necessary. If wrong version is installed, it will be overwritten.
      - $(CONTROLLER_GEN): $(LOCALBIN)
      -   test -s $(LOCALBIN)/controller-gen && $(LOCALBIN)/controller-gen --version | grep -q $(CONTROLLER_TOOLS_VERSION) || \
      -   GOBIN=$(LOCALBIN) go install sigs.k8s.io/controller-tools/cmd/controller-gen@$(CONTROLLER_TOOLS_VERSION)
      -
      - .PHONY: envtest
      - envtest: $(ENVTEST) ## Download envtest-setup locally if necessary.
      - $(ENVTEST): $(LOCALBIN)
      -   test -s $(LOCALBIN)/setup-envtest || GOBIN=$(LOCALBIN) go install sigs.k8s.io/controller-runtime/tools/setup-envtest@latest
      + ## Tool Binaries
      + KUBECTL ?= kubectl
      + KUSTOMIZE ?= $(LOCALBIN)/kustomize-$(KUSTOMIZE_VERSION)
      + CONTROLLER_GEN ?= $(LOCALBIN)/controller-gen-$(CONTROLLER_TOOLS_VERSION)
      + ENVTEST ?= $(LOCALBIN)/setup-envtest-$(ENVTEST_VERSION)
      + GOLANGCI_LINT = $(LOCALBIN)/golangci-lint-$(GOLANGCI_LINT_VERSION)
      +
      + ## Tool Versions
      + KUSTOMIZE_VERSION ?= v5.3.0
      + CONTROLLER_TOOLS_VERSION ?= v0.14.0
      + ENVTEST_VERSION ?= release-0.17
      + GOLANGCI_LINT_VERSION ?= v1.57.2
      +
      + .PHONY: kustomize
      + kustomize: $(KUSTOMIZE) ## Download kustomize locally if necessary.
      + $(KUSTOMIZE): $(LOCALBIN)
      + 	$(call go-install-tool,$(KUSTOMIZE),sigs.k8s.io/kustomize/kustomize/v5,$(KUSTOMIZE_VERSION))
      +
      + .PHONY: controller-gen
      + controller-gen: $(CONTROLLER_GEN) ## Download controller-gen locally if necessary.
      + $(CONTROLLER_GEN): $(LOCALBIN)
      + 	$(call go-install-tool,$(CONTROLLER_GEN),sigs.k8s.io/controller-tools/cmd/controller-gen,$(CONTROLLER_TOOLS_VERSION))
      +
      + .PHONY: envtest
      + envtest: $(ENVTEST) ## Download setup-envtest locally if necessary.
      + $(ENVTEST): $(LOCALBIN)
      + 	$(call go-install-tool,$(ENVTEST),sigs.k8s.io/controller-runtime/tools/setup-envtest,$(ENVTEST_VERSION))
      +
      + .PHONY: golangci-lint
      + golangci-lint: $(GOLANGCI_LINT) ## Download golangci-lint locally if necessary.
      + $(GOLANGCI_LINT): $(LOCALBIN)
      + 	$(call go-install-tool,$(GOLANGCI_LINT),github.com/golangci/golangci-lint/cmd/golangci-lint,${GOLANGCI_LINT_VERSION})
      +
      + # go-install-tool will 'go install' any package with custom target and name of binary, if it doesn't exist
      + # $1 - target path with name of binary (ideally with version)
      + # $2 - package url which can be installed
      + # $3 - specific version of package
      + define go-install-tool
      + @[ -f $(1) ] || { \
      + set -e; \
      + package=$(2)@$(3) ;\
      + echo "Downloading $${package}" ;\
      + GOBIN=$(LOCALBIN) go install $${package} ;\
      + mv "$$(echo "$(1)" | sed "s/-$(3)$$//")" $(1) ;\
      + }
      + endef

Procedure

1. Create a directory for the project:

   $ mkdir -p $HOME/projects/memcached-operator

2. Change to the directory:

   $ cd $HOME/projects/memcached-operator

3. Run the operator-sdk init command with the ansible plugin to initialize the project:

   $ operator-sdk init \
       --plugins=ansible \
       --domain=example.com

Procedure

1. Update the roles/memcached/tasks/main.yml file with the following structure:

   ---
   - name: start memcached
     k8s:
       definition:
         kind: Deployment
         apiVersion: apps/v1
         metadata:
           name: '{{ ansible_operator_meta.name }}-memcached'
           namespace: '{{ ansible_operator_meta.namespace }}'
         spec:
           replicas: "{{size}}"
           selector:
             matchLabels:
               app: memcached
           template:
             metadata:
               labels:
                 app: memcached
             spec:
               containers:
               - name: memcached
                 command:
                 - memcached
                 - -m=64
                 - -o
                 - modern
                 - -v
                 image: "docker.io/memcached:1.4.36-alpine"
                 ports:
                   - containerPort: 11211

   This memcached role ensures a memcached deployment exist and sets the deployment size.

2. Set default values for variables used in your Ansible role by editing the roles/memcached/defaults/main.yml file:

   ---
   # defaults file for Memcached
   size: 1

3. Update the Memcached sample resource in the config/samples/cache_v1_memcached.yaml file with the following structure:

   apiVersion: cache.example.com/v1
   kind: Memcached
   metadata:
     labels:
       app.kubernetes.io/name: memcached
       app.kubernetes.io/instance: memcached-sample
       app.kubernetes.io/part-of: memcached-operator
       app.kubernetes.io/managed-by: kustomize
       app.kubernetes.io/created-by: memcached-operator
     name: memcached-sample
   spec:
     size: 3

   The key-value pairs in the custom resource (CR) spec are passed to Ansible as extra variables.

Procedure

1. Add the environment variables to the deployment by updating the roles/memcached/tasks/main.yml file with the following:

   ...
   env:
      - name: HTTP_PROXY
        value: '{{ lookup("env", "HTTP_PROXY") | default("", True) }}'
      - name: http_proxy
        value: '{{ lookup("env", "HTTP_PROXY") | default("", True) }}'
   ...

2. Set the environment variable on the Operator deployment by adding the following to the config/manager/manager.yaml file:

   containers:
    - args:
      - --leader-elect
      - --leader-election-id=ansible-proxy-demo
      image: controller:latest
      name: manager
      env:
        - name: "HTTP_PROXY"
          value: "http_proxy_test"

Procedure

1. Change to the namespace where your Operator is installed. For example, if you deployed the Operator using the make deploy command:

   $ oc project memcached-operator-system

2. Edit the sample Memcached CR manifest at config/samples/cache_v1_memcached.yaml to contain the following specification:

   apiVersion: cache.example.com/v1
   kind: Memcached
   metadata:
     name: memcached-sample
   ...
   spec:
   ...
     size: 3

3. Create the CR:

   $ oc apply -f config/samples/cache_v1_memcached.yaml

4. Ensure that the Memcached Operator creates the deployment for the sample CR with the correct size:

   $ oc get deployments

   Example output

   NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
   memcached-operator-controller-manager   1/1     1            1           8m
   memcached-sample                        3/3     3            3           1m

5. Check the pods and CR status to confirm the status is updated with the Memcached pod names.

   1. Check the pods:

      $ oc get pods

      Example output

      NAME                                  READY     STATUS    RESTARTS   AGE
      memcached-sample-6fd7c98d8-7dqdr      1/1       Running   0          1m
      memcached-sample-6fd7c98d8-g5k7v      1/1       Running   0          1m
      memcached-sample-6fd7c98d8-m7vn7      1/1       Running   0          1m

   2. Check the CR status:

      $ oc get memcached/memcached-sample -o yaml

      Example output

      apiVersion: cache.example.com/v1
      kind: Memcached
      metadata:
      ...
        name: memcached-sample
      ...
      spec:
        size: 3
      status:
        nodes:
        - memcached-sample-6fd7c98d8-7dqdr
        - memcached-sample-6fd7c98d8-g5k7v
        - memcached-sample-6fd7c98d8-m7vn7

6. Update the deployment size.

   1. Update config/samples/cache_v1_memcached.yaml file to change the spec.size field in the Memcached CR from 3 to 5:

      $ oc patch memcached memcached-sample \
          -p '{"spec":{"size": 5}}' \
          --type=merge

   2. Confirm that the Operator changes the deployment size:

      $ oc get deployments

      Example output

      NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
      memcached-operator-controller-manager   1/1     1            1           10m
      memcached-sample                        5/5     5            5           3m

7. Delete the CR by running the following command:

   $ oc delete -f config/samples/cache_v1_memcached.yaml

8. Clean up the resources that have been created as part of this tutorial.

   • If you used the make deploy command to test the Operator, run the following command:

     $ make undeploy

   • If you used the operator-sdk run bundle command to test the Operator, run the following command:

     $ operator-sdk cleanup <project_name>

Procedure

1. Edit your Operator project’s Makefile to update the Operator SDK version to v1.36.1-ocp, as shown in the following example:

   Example Makefile

   # Set the Operator SDK version to use. By default, what is installed on the system is used.
   # This is useful for CI or a project to utilize a specific version of the operator-sdk toolkit.
   OPERATOR_SDK_VERSION ?= v1.36.1-ocp

2. Update the kube-rbac-proxy container to use the Red Hat Enterprise Linux (RHEL) 9-based image:

   1. Find the entry for the kube-rbac-proxy container in the following files:

      • config/default/manager_auth_proxy_patch.yaml
      • bundle/manifests/<operator_name>.clusterserviceversion.yaml for your Operator project, for example memcached-operator.clusterserviceversion.yaml from the tutorials

   2. Update the image name in the pull spec from ose-kube-rbac-proxy to ose-kube-rbac-proxy-rhel9, and update the tag to v4:

      Example ose-kube-rbac-proxy-rhel9 pull spec with v4 image tag

      # ...
            containers:
            - name: kube-rbac-proxy
              image: registry.redhat.io/openshift4/ose-kube-rbac-proxy-rhel9:v4
      # ...

3. Edit your Operator’s Dockerfile to update the ose-ansible-rhel9-operator image tag to 4, as shown in the following example:

   Example Dockerfile

   FROM registry.redhat.io/openshift4/ose-ansible-rhel9-operator:v4

4. The kustomize/v2 plugin is now stable and is the default version used in the plugin chain when using go/v4, ansible/v1, helm/v1, and hybrid/v1-alpha plugins. For more information on this default scaffold, see Kustomize v2 in the Kubebuilder documentation.

5. If your Operator project uses a multi-platform, or multi-archicture, build, replace the existing docker-buildx target with following definition in your project Makefile:

   Example Makefile

   docker-buildx:
   ## Build and push the Docker image for the manager for multi-platform support
   	- docker buildx create --name project-v3-builder
   	docker buildx use project-v3-builder
   	- docker buildx build --push --platform=$(PLATFORMS) --tag ${IMG} -f Dockerfile .
   	- docker buildx rm project-v3-builder

6. You must upgrade the Kubernetes versions in your Operator project to use 1.29. The following changes must be made in your project structure, Makefile, and go.mod files.

   Important

   The go/v3 plugin is being deprecated by Kubebuilder, therefore Operator SDK is also migrating to go/v4 in a future release.

   1. Update your go.mod file to upgrade your dependencies:

      k8s.io/api v0.29.2
      k8s.io/apimachinery v0.29.2
      k8s.io/client-go v0.29.2
      sigs.k8s.io/controller-runtime v0.17.3

   2. Download the upgraded dependencies by running the following command:

      $ go mod tidy

Procedure

1. Install Ansible 2.15+:

   $ sudo dnf install ansible

2. Install the Python Kubernetes client package:

   $ pip install kubernetes

3. Install the Kubernetes Collection using one of the following methods:

   • You can install the collection directly from Ansible Galaxy:

     $ ansible-galaxy collection install community.kubernetes

   • If you have already initialized your Operator, you might have a requirements.yml file at the top level of your project. This file specifies Ansible dependencies that must be installed for your Operator to function. By default, this file installs the community.kubernetes collection as well as the operator_sdk.util collection, which provides modules and plugins for Operator-specific functions.

     To install the dependent modules from the requirements.yml file:

     $ ansible-galaxy collection install -r requirements.yml

Procedure

1. In your Ansible-based Operator project directory, modify the roles/<kind>/tasks/main.yml file with the Ansible logic that you want. The roles/<kind>/ directory is created when you use the --generate-role flag while creating an API. The <kind> replaceable matches the kind that you specified for the API.

   The following example creates and deletes a config map based on the value of a variable named state:

   ---
   - name: set ConfigMap example-config to {{ state }}
     community.kubernetes.k8s:
       api_version: v1
       kind: ConfigMap
       name: example-config
       namespace: <operator_namespace> 1
       state: "{{ state }}"
     ignore_errors: true 2

   1

   Specify the namespace where you want the config map created.

   2

   Setting ignore_errors: true ensures that deleting a nonexistent config map does not fail.

2. Modify the roles/<kind>/defaults/main.yml file to set state to present by default:

   ---
   state: present

3. Create an Ansible playbook by creating a playbook.yml file in the top-level of your project directory, and include your <kind> role:

   ---
   - hosts: localhost
     roles:
       - <kind>

4. Run the playbook:

   $ ansible-playbook playbook.yml

   Example output

   [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'
   
   PLAY [localhost] ********************************************************************************
   
   TASK [Gathering Facts] ********************************************************************************
   ok: [localhost]
   
   TASK [memcached : set ConfigMap example-config to present] ********************************************************************************
   changed: [localhost]
   
   PLAY RECAP ********************************************************************************
   localhost                  : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

5. Verify that the config map was created:

   $ oc get configmaps

   Example output

   NAME               DATA   AGE
   example-config     0      2m1s

6. Rerun the playbook setting state to absent:

   $ ansible-playbook playbook.yml --extra-vars state=absent

   Example output

   [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'
   
   PLAY [localhost] ********************************************************************************
   
   TASK [Gathering Facts] ********************************************************************************
   ok: [localhost]
   
   TASK [memcached : set ConfigMap example-config to absent] ********************************************************************************
   changed: [localhost]
   
   PLAY RECAP ********************************************************************************
   localhost                  : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

7. Verify that the config map was deleted:

   $ oc get configmaps

Procedure

1. Install your custom resource definition (CRD) and proper role-based access control (RBAC) definitions for your custom resource (CR):

   $ make install

   Example output

   /usr/bin/kustomize build config/crd | kubectl apply -f -
   customresourcedefinition.apiextensions.k8s.io/memcacheds.cache.example.com created

2. Run the make run command:

   $ make run

   Example output

   /home/user/memcached-operator/bin/ansible-operator run
   {"level":"info","ts":1612739145.2871568,"logger":"cmd","msg":"Version","Go Version":"go1.15.5","GOOS":"linux","GOARCH":"amd64","ansible-operator":"v1.10.1","commit":"1abf57985b43bf6a59dcd18147b3c574fa57d3f6"}
   ...
   {"level":"info","ts":1612739148.347306,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"}
   {"level":"info","ts":1612739148.3488882,"logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE_VERBOSITY_MEMCACHED_CACHE_EXAMPLE_COM","default":2}
   {"level":"info","ts":1612739148.3490262,"logger":"cmd","msg":"Environment variable not set; using default value","Namespace":"","envVar":"ANSIBLE_DEBUG_LOGS","ANSIBLE_DEBUG_LOGS":false}
   {"level":"info","ts":1612739148.3490646,"logger":"ansible-controller","msg":"Watching resource","Options.Group":"cache.example.com","Options.Version":"v1","Options.Kind":"Memcached"}
   {"level":"info","ts":1612739148.350217,"logger":"proxy","msg":"Starting to serve","Address":"127.0.0.1:8888"}
   {"level":"info","ts":1612739148.3506632,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
   {"level":"info","ts":1612739148.350784,"logger":"controller-runtime.manager.controller.memcached-controller","msg":"Starting EventSource","source":"kind source: cache.example.com/v1, Kind=Memcached"}
   {"level":"info","ts":1612739148.5511978,"logger":"controller-runtime.manager.controller.memcached-controller","msg":"Starting Controller"}
   {"level":"info","ts":1612739148.5512562,"logger":"controller-runtime.manager.controller.memcached-controller","msg":"Starting workers","worker count":8}

   With the Operator now watching your CR for events, the creation of a CR will trigger your Ansible role to run.

   Note

   Consider an example config/samples/<gvk>.yaml CR manifest:

   apiVersion: <group>.example.com/v1alpha1
   kind: <kind>
   metadata:
     name: "<kind>-sample"

   Because the spec field is not set, Ansible is invoked with no extra variables. Passing extra variables from a CR to Ansible is covered in another section. It is important to set reasonable defaults for the Operator.

3. Create an instance of your CR with the default variable state set to present:

   $ oc apply -f config/samples/<gvk>.yaml

4. Check that the example-config config map was created:

   $ oc get configmaps

   Example output

   NAME                    STATUS    AGE
   example-config          Active    3s

5. Modify your config/samples/<gvk>.yaml file to set the state field to absent. For example:

   apiVersion: cache.example.com/v1
   kind: Memcached
   metadata:
     name: memcached-sample
   spec:
     state: absent

6. Apply the changes:

   $ oc apply -f config/samples/<gvk>.yaml

7. Confirm that the config map is deleted:

   $ oc get configmap

Procedure

1. Run the following make commands to build and push the Operator image. Modify the IMG argument in the following steps to reference a repository that you have access to. You can obtain an account for storing containers at repository sites such as Quay.io.

   1. Build the image:

      $ make docker-build IMG=<registry>/<user>/<image_name>:<tag>

      Note

      The Dockerfile generated by the SDK for the Operator explicitly references GOARCH=amd64 for go build. This can be amended to GOARCH=$TARGETARCH for non-AMD64 architectures. Docker will automatically set the environment variable to the value specified by –platform. With Buildah, the –build-arg will need to be used for the purpose. For more information, see Multiple Architectures.

   2. Push the image to a repository:

      $ make docker-push IMG=<registry>/<user>/<image_name>:<tag>

      Note

      The name and tag of the image, for example IMG=<registry>/<user>/<image_name>:<tag>, in both the commands can also be set in your Makefile. Modify the IMG ?= controller:latest value to set your default image name.

2. Run the following command to deploy the Operator:

   $ make deploy IMG=<registry>/<user>/<image_name>:<tag>

   By default, this command creates a namespace with the name of your Operator project in the form <project_name>-system and is used for the deployment. This command also installs the RBAC manifests from config/rbac.

3. Run the following command to verify that the Operator is running:

   $ oc get deployment -n <project_name>-system

   Example output

   NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
   <project_name>-controller-manager       1/1     1            1           8m

Procedure

1. Update the watches.yaml file with a manageStatus field set to false:

   - version: v1
     group: api.example.com
     kind: <kind>
     role: <role>
     manageStatus: false

2. Use the operator_sdk.util.k8s_status Ansible module to update the subresource. For example, to update with key test and value data, operator_sdk.util can be used as shown:

   - operator_sdk.util.k8s_status:
       api_version: app.example.com/v1
       kind: <kind>
       name: "{{ ansible_operator_meta.name }}"
       namespace: "{{ ansible_operator_meta.namespace }}"
       status:
         test: data

3. You can declare collections in the meta/main.yml file for the role, which is included for scaffolded Ansible-based Operators:

   collections:
     - operator_sdk.util

4. After declaring collections in the role meta, you can invoke the k8s_status module directly:

   k8s_status:
     ...
     status:
       key1: value1

Procedure

1. Create a directory for the project:

   $ mkdir -p $HOME/projects/nginx-operator

2. Change to the directory:

   $ cd $HOME/projects/nginx-operator

3. Run the operator-sdk init command with the helm plugin to initialize the project:

   $ operator-sdk init \
       --plugins=helm \
       --domain=example.com \
       --group=demo \
       --version=v1 \
       --kind=Nginx

   Note

   By default, the helm plugin initializes a project using a boilerplate Helm chart. You can use additional flags, such as the --helm-chart flag, to initialize a project using an existing Helm chart.

   The init command creates the nginx-operator project specifically for watching a resource with API version example.com/v1 and kind Nginx.

4. For Helm-based projects, the init command generates the RBAC rules in the config/rbac/role.yaml file based on the resources that would be deployed by the default manifest for the chart. Verify that the rules generated in this file meet the permission requirements of the Operator.

Procedure

1. Edit the watches.yaml file to include overrides based on an environment variable by adding the overrideValues field:

   ...
   - group: demo.example.com
     version: v1alpha1
     kind: Nginx
     chart: helm-charts/nginx
     overrideValues:
       proxy.http: $HTTP_PROXY
   ...

2. Add the proxy.http value in the helm-charts/nginx/values.yaml file:

   ...
   proxy:
     http: ""
     https: ""
     no_proxy: ""

3. To make sure the chart template supports using the variables, edit the chart template in the helm-charts/nginx/templates/deployment.yaml file to contain the following:

   containers:
     - name: {{ .Chart.Name }}
       securityContext:
         - toYaml {{ .Values.securityContext | nindent 12 }}
       image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
       imagePullPolicy: {{ .Values.image.pullPolicy }}
       env:
         - name: http_proxy
           value: "{{ .Values.proxy.http }}"

4. Set the environment variable on the Operator deployment by adding the following to the config/manager/manager.yaml file:

   containers:
    - args:
      - --leader-elect
      - --leader-election-id=ansible-proxy-demo
      image: controller:latest
      name: manager
      env:
        - name: "HTTP_PROXY"
          value: "http_proxy_test"

Procedure

1. Change to the namespace where your Operator is installed. For example, if you deployed the Operator using the make deploy command:

   $ oc project nginx-operator-system

2. Edit the sample Nginx CR manifest at config/samples/demo_v1_nginx.yaml to contain the following specification:

   apiVersion: demo.example.com/v1
   kind: Nginx
   metadata:
     name: nginx-sample
   ...
   spec:
   ...
     replicaCount: 3

3. The Nginx service account requires privileged access to run in OpenShift Dedicated. Add the following security context constraint (SCC) to the service account for the nginx-sample pod:

   $ oc adm policy add-scc-to-user \
       anyuid system:serviceaccount:nginx-operator-system:nginx-sample

4. Create the CR:

   $ oc apply -f config/samples/demo_v1_nginx.yaml

5. Ensure that the Nginx Operator creates the deployment for the sample CR with the correct size:

   $ oc get deployments

   Example output

   NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
   nginx-operator-controller-manager       1/1     1            1           8m
   nginx-sample                            3/3     3            3           1m

6. Check the pods and CR status to confirm the status is updated with the Nginx pod names.

   1. Check the pods:

      $ oc get pods

      Example output

      NAME                                  READY     STATUS    RESTARTS   AGE
      nginx-sample-6fd7c98d8-7dqdr          1/1       Running   0          1m
      nginx-sample-6fd7c98d8-g5k7v          1/1       Running   0          1m
      nginx-sample-6fd7c98d8-m7vn7          1/1       Running   0          1m

   2. Check the CR status:

      $ oc get nginx/nginx-sample -o yaml

      Example output

      apiVersion: demo.example.com/v1
      kind: Nginx
      metadata:
      ...
        name: nginx-sample
      ...
      spec:
        replicaCount: 3
      status:
        nodes:
        - nginx-sample-6fd7c98d8-7dqdr
        - nginx-sample-6fd7c98d8-g5k7v
        - nginx-sample-6fd7c98d8-m7vn7

7. Update the deployment size.

   1. Update config/samples/demo_v1_nginx.yaml file to change the spec.size field in the Nginx CR from 3 to 5:

      $ oc patch nginx nginx-sample \
          -p '{"spec":{"replicaCount": 5}}' \
          --type=merge

   2. Confirm that the Operator changes the deployment size:

      $ oc get deployments

      Example output

      NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
      nginx-operator-controller-manager       1/1     1            1           10m
      nginx-sample                            5/5     5            5           3m

8. Delete the CR by running the following command:

   $ oc delete -f config/samples/demo_v1_nginx.yaml

9. Clean up the resources that have been created as part of this tutorial.

   • If you used the make deploy command to test the Operator, run the following command:

     $ make undeploy

   • If you used the operator-sdk run bundle command to test the Operator, run the following command:

     $ operator-sdk cleanup <project_name>

Procedure

1. Edit your Operator project’s Makefile to update the Operator SDK version to v1.36.1-ocp, as shown in the following example:

   Example Makefile

   # Set the Operator SDK version to use. By default, what is installed on the system is used.
   # This is useful for CI or a project to utilize a specific version of the operator-sdk toolkit.
   OPERATOR_SDK_VERSION ?= v1.36.1-ocp

2. Update the kube-rbac-proxy container to use the Red Hat Enterprise Linux (RHEL) 9-based image:

   1. Find the entry for the kube-rbac-proxy container in the following files:

      • config/default/manager_auth_proxy_patch.yaml
      • bundle/manifests/<operator_name>.clusterserviceversion.yaml for your Operator project, for example memcached-operator.clusterserviceversion.yaml from the tutorials

   2. Update the image name in the pull spec from ose-kube-rbac-proxy to ose-kube-rbac-proxy-rhel9, and update the tag to v4:

      Example ose-kube-rbac-proxy-rhel9 pull spec with v4 image tag

      # ...
            containers:
            - name: kube-rbac-proxy
              image: registry.redhat.io/openshift4/ose-kube-rbac-proxy-rhel9:v4
      # ...

3. Edit your Operator’s Dockerfile to update the ose-helm-rhel9-operator image tag to 4, as shown in the following example:

   Example Dockerfile

   FROM registry.redhat.io/openshift4/ose-helm-rhel9-operator:v4

4. The kustomize/v2 plugin is now stable and is the default version used in the plugin chain when using go/v4, ansible/v1, helm/v1, and hybrid/v1-alpha plugins. For more information on this default scaffold, see Kustomize v2 in the Kubebuilder documentation.

5. If your Operator project uses a multi-platform, or multi-archicture, build, replace the existing docker-buildx target with following definition in your project Makefile:

   Example Makefile

   docker-buildx:
   ## Build and push the Docker image for the manager for multi-platform support
   	- docker buildx create --name project-v3-builder
   	docker buildx use project-v3-builder
   	- docker buildx build --push --platform=$(PLATFORMS) --tag ${IMG} -f Dockerfile .
   	- docker buildx rm project-v3-builder

6. You must upgrade the Kubernetes versions in your Operator project to use 1.29. The following changes must be made in your project structure, Makefile, and go.mod files.

   Important

   The go/v3 plugin is being deprecated by Kubebuilder, therefore Operator SDK is also migrating to go/v4 in a future release.

   1. Update your go.mod file to upgrade your dependencies:

      k8s.io/api v0.29.2
      k8s.io/apimachinery v0.29.2
      k8s.io/client-go v0.29.2
      sigs.k8s.io/controller-runtime v0.17.3

   2. Download the upgraded dependencies by running the following command:

      $ go mod tidy

7. Update the Kustomize version in your Makefile by making the following changes:

   - curl -sSLo - https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize/v5.2.1/kustomize_v5.2.1_$(OS)_$(ARCH).tar.gz | \
   + curl -sSLo - https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize/v5.3.0/kustomize_v5.3.0_$(OS)_$(ARCH).tar.gz | \

1. Valid values are shown intentionally with double quotes, because Kubernetes annotations must be strings.

Procedure

1. Before marking one of your CRDs as internal, ensure that any debugging information or configuration that might be required to manage the application is reflected on the status or spec block of your CR, if applicable to your Operator.

2. Add the operators.operatorframework.io/internal-objects annotation to the CSV of your Operator to specify any internal objects to hide in the user interface:

   Internal object annotation

   apiVersion: operators.coreos.com/v1alpha1
   kind: ClusterServiceVersion
   metadata:
     name: my-operator-v1.2.3
     annotations:
       operators.operatorframework.io/internal-objects: '["my.internal.crd1.io","my.internal.crd2.io"]' 1
   ...

   1

   Set any internal CRDs as an array of strings.

1. The restricted security context is not compatible with the default namespace. To configure your Operator’s pod security admission in your production environment, see "Complying with pod security admission". For more information about pod security admission, see "Understanding and managing pod security admission".