Chapter 5. Developing Operators

Procedure

1. Create a project.

   1. Create your project directory:

      $ mkdir memcached-operator

   2. Change into the project directory:

      $ cd memcached-operator

   3. Run the operator-sdk init command to initialize the project:

      $ operator-sdk init \
          --domain=example.com \
          --repo=github.com/example-inc/memcached-operator

      The command uses the Go plugin by default.

2. Create an API.

   Create a simple Memcached API:

   $ operator-sdk create api \
       --resource=true \
       --controller=true \
       --group cache \
       --version v1 \
       --kind Memcached

3. Build and push the Operator image.

   Use the default Makefile targets to build and push your Operator. Set IMG with a pull spec for your image that uses a registry you can push to:

   $ make docker-build docker-push IMG=<registry>/<user>/<image_name>:<tag>

4. Run the Operator.

   1. Install the CRD:

      $ make install

   2. Deploy the project to the cluster. Set IMG to the image that you pushed:

      $ make deploy IMG=<registry>/<user>/<image_name>:<tag>

5. Create a sample custom resource (CR).

   1. Create a sample CR:

      $ oc apply -f config/samples/cache_v1_memcached.yaml \
          -n memcached-operator-system

   2. Watch for the CR to reconcile the Operator:

      $ oc logs deployment.apps/memcached-operator-controller-manager \
          -c manager \
          -n memcached-operator-system

6. Delete a CR.

   Delete a CR by running the following command:

   $ oc delete -f config/samples/cache_v1_memcached.yaml -n memcached-operator-system

7. Clean up.

   Run the following command to clean up the resources that have been created as part of this procedure:

   $ make undeploy

Procedure

1. Create a directory for the project:

   $ mkdir -p $HOME/projects/memcached-operator

2. Change to the directory:

   $ cd $HOME/projects/memcached-operator

3. Activate support for Go modules:

   $ export GO111MODULE=on

4. Run the operator-sdk init command to initialize the project:

   $ operator-sdk init \
       --domain=example.com \
       --repo=github.com/example-inc/memcached-operator

   Note

   The operator-sdk init command uses the Go plugin by default.

   The operator-sdk init command generates a go.mod file to be used with Go modules. The --repo flag is required when creating a project outside of $GOPATH/src/, because generated files require a valid module path.

Procedure

1. Run the following command to create an API with group cache, version, v1, and kind Memcached:

   $ operator-sdk create api \
       --group=cache \
       --version=v1 \
       --kind=Memcached

2. When prompted, enter y for creating both the resource and controller:

   Create Resource [y/n]
   y
   Create Controller [y/n]
   y

   Example output

   Writing scaffold for you to edit...
   api/v1/memcached_types.go
   controllers/memcached_controller.go
   ...

Procedure

1. Edit the controllers/memcached_controller.go file to include the following:

   1. Import the proxy package from the operator-lib library:

      import (
        ...
         "github.com/operator-framework/operator-lib/proxy"
      )

   2. Add the proxy.ReadProxyVarsFromEnv helper function to the reconcile loop and append the results to the Operand environments:

      for i, container := range dep.Spec.Template.Spec.Containers {
      		dep.Spec.Template.Spec.Containers[i].Env = append(container.Env, proxy.ReadProxyVarsFromEnv()...)
      }
      ...

2. Set the environment variable on the Operator deployment by adding the following to the config/manager/manager.yaml file:

   containers:
    - args:
      - --leader-elect
      - --leader-election-id=ansible-proxy-demo
      image: controller:latest
      name: manager
      env:
        - name: "HTTP_PROXY"
          value: "http_proxy_test"

Procedure

1. Change to the namespace where your Operator is installed. For example, if you deployed the Operator using the make deploy command:

   $ oc project memcached-operator-system

2. Edit the sample Memcached CR manifest at config/samples/cache_v1_memcached.yaml to contain the following specification:

   apiVersion: cache.example.com/v1
   kind: Memcached
   metadata:
     name: memcached-sample
   ...
   spec:
   ...
     size: 3

3. Create the CR:

   $ oc apply -f config/samples/cache_v1_memcached.yaml

4. Ensure that the Memcached Operator creates the deployment for the sample CR with the correct size:

   $ oc get deployments

   Example output

   NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
   memcached-operator-controller-manager   1/1     1            1           8m
   memcached-sample                        3/3     3            3           1m

5. Check the pods and CR status to confirm the status is updated with the Memcached pod names.

   1. Check the pods:

      $ oc get pods

      Example output

      NAME                                  READY     STATUS    RESTARTS   AGE
      memcached-sample-6fd7c98d8-7dqdr      1/1       Running   0          1m
      memcached-sample-6fd7c98d8-g5k7v      1/1       Running   0          1m
      memcached-sample-6fd7c98d8-m7vn7      1/1       Running   0          1m

   2. Check the CR status:

      $ oc get memcached/memcached-sample -o yaml

      Example output

      apiVersion: cache.example.com/v1
      kind: Memcached
      metadata:
      ...
        name: memcached-sample
      ...
      spec:
        size: 3
      status:
        nodes:
        - memcached-sample-6fd7c98d8-7dqdr
        - memcached-sample-6fd7c98d8-g5k7v
        - memcached-sample-6fd7c98d8-m7vn7

6. Update the deployment size.

   1. Update config/samples/cache_v1_memcached.yaml file to change the spec.size field in the Memcached CR from 3 to 5:

      $ oc patch memcached memcached-sample \
          -p '{"spec":{"size": 5}}' \
          --type=merge

   2. Confirm that the Operator changes the deployment size:

      $ oc get deployments

      Example output

      NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
      memcached-operator-controller-manager   1/1     1            1           10m
      memcached-sample                        5/5     5            5           3m

7. Delete the CR by running the following command:

   $ oc delete -f config/samples/cache_v1_memcached.yaml

8. Clean up the resources that have been created as part of this tutorial.

   • If you used the make deploy command to test the Operator, run the following command:

     $ make undeploy

   • If you used the operator-sdk run bundle command to test the Operator, run the following command:

     $ operator-sdk cleanup <project_name>

Procedure

1. Create a project.

   1. Create your project directory:

      $ mkdir memcached-operator

   2. Change into the project directory:

      $ cd memcached-operator

   3. Run the operator-sdk init command with the ansible plugin to initialize the project:

      $ operator-sdk init \
          --plugins=ansible \
          --domain=example.com

2. Create an API.

   Create a simple Memcached API:

   $ operator-sdk create api \
       --group cache \
       --version v1 \
       --kind Memcached \
       --generate-role 1

   1

   Generates an Ansible role for the API.

3. Build and push the Operator image.

   Use the default Makefile targets to build and push your Operator. Set IMG with a pull spec for your image that uses a registry you can push to:

   $ make docker-build docker-push IMG=<registry>/<user>/<image_name>:<tag>

4. Run the Operator.

   1. Install the CRD:

      $ make install

   2. Deploy the project to the cluster. Set IMG to the image that you pushed:

      $ make deploy IMG=<registry>/<user>/<image_name>:<tag>

5. Create a sample custom resource (CR).

   1. Create a sample CR:

      $ oc apply -f config/samples/cache_v1_memcached.yaml \
          -n memcached-operator-system

   2. Watch for the CR to reconcile the Operator:

      $ oc logs deployment.apps/memcached-operator-controller-manager \
          -c manager \
          -n memcached-operator-system

      Example output

      ...
      I0205 17:48:45.881666       7 leaderelection.go:253] successfully acquired lease memcached-operator-system/memcached-operator
      {"level":"info","ts":1612547325.8819902,"logger":"controller-runtime.manager.controller.memcached-controller","msg":"Starting EventSource","source":"kind source: cache.example.com/v1, Kind=Memcached"}
      {"level":"info","ts":1612547325.98242,"logger":"controller-runtime.manager.controller.memcached-controller","msg":"Starting Controller"}
      {"level":"info","ts":1612547325.9824686,"logger":"controller-runtime.manager.controller.memcached-controller","msg":"Starting workers","worker count":4}
      {"level":"info","ts":1612547348.8311093,"logger":"runner","msg":"Ansible-runner exited successfully","job":"4037200794235010051","name":"memcached-sample","namespace":"memcached-operator-system"}

6. Delete a CR.

   Delete a CR by running the following command:

   $ oc delete -f config/samples/cache_v1_memcached.yaml -n memcached-operator-system

7. Clean up.

   Run the following command to clean up the resources that have been created as part of this procedure:

   $ make undeploy

Procedure

1. Create a directory for the project:

   $ mkdir -p $HOME/projects/memcached-operator

2. Change to the directory:

   $ cd $HOME/projects/memcached-operator

3. Run the operator-sdk init command with the ansible plugin to initialize the project:

   $ operator-sdk init \
       --plugins=ansible \
       --domain=example.com

Procedure

1. Update the roles/memcached/tasks/main.yml file with the following structure:

   ---
   - name: start memcached
     k8s:
       definition:
         kind: Deployment
         apiVersion: apps/v1
         metadata:
           name: '{{ ansible_operator_meta.name }}-memcached'
           namespace: '{{ ansible_operator_meta.namespace }}'
         spec:
           replicas: "{{size}}"
           selector:
             matchLabels:
               app: memcached
           template:
             metadata:
               labels:
                 app: memcached
             spec:
               containers:
               - name: memcached
                 command:
                 - memcached
                 - -m=64
                 - -o
                 - modern
                 - -v
                 image: "docker.io/memcached:1.4.36-alpine"
                 ports:
                   - containerPort: 11211

   This memcached role ensures a memcached deployment exist and sets the deployment size.

2. Set default values for variables used in your Ansible role by editing the roles/memcached/defaults/main.yml file:

   ---
   # defaults file for Memcached
   size: 1

3. Update the Memcached sample resource in the config/samples/cache_v1_memcached.yaml file with the following structure:

   apiVersion: cache.example.com/v1
   kind: Memcached
   metadata:
     labels:
       app.kubernetes.io/name: memcached
       app.kubernetes.io/instance: memcached-sample
       app.kubernetes.io/part-of: memcached-operator
       app.kubernetes.io/managed-by: kustomize
       app.kubernetes.io/created-by: memcached-operator
     name: memcached-sample
   spec:
     size: 3

   The key-value pairs in the custom resource (CR) spec are passed to Ansible as extra variables.

Procedure

1. Add the environment variables to the deployment by updating the roles/memcached/tasks/main.yml file with the following:

   ...
   env:
      - name: HTTP_PROXY
        value: '{{ lookup("env", "HTTP_PROXY") | default("", True) }}'
      - name: http_proxy
        value: '{{ lookup("env", "HTTP_PROXY") | default("", True) }}'
   ...

2. Set the environment variable on the Operator deployment by adding the following to the config/manager/manager.yaml file:

   containers:
    - args:
      - --leader-elect
      - --leader-election-id=ansible-proxy-demo
      image: controller:latest
      name: manager
      env:
        - name: "HTTP_PROXY"
          value: "http_proxy_test"

Procedure

1. Change to the namespace where your Operator is installed. For example, if you deployed the Operator using the make deploy command:

   $ oc project memcached-operator-system

2. Edit the sample Memcached CR manifest at config/samples/cache_v1_memcached.yaml to contain the following specification:

   apiVersion: cache.example.com/v1
   kind: Memcached
   metadata:
     name: memcached-sample
   ...
   spec:
   ...
     size: 3

3. Create the CR:

   $ oc apply -f config/samples/cache_v1_memcached.yaml

4. Ensure that the Memcached Operator creates the deployment for the sample CR with the correct size:

   $ oc get deployments

   Example output

   NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
   memcached-operator-controller-manager   1/1     1            1           8m
   memcached-sample                        3/3     3            3           1m

5. Check the pods and CR status to confirm the status is updated with the Memcached pod names.

   1. Check the pods:

      $ oc get pods

      Example output

      NAME                                  READY     STATUS    RESTARTS   AGE
      memcached-sample-6fd7c98d8-7dqdr      1/1       Running   0          1m
      memcached-sample-6fd7c98d8-g5k7v      1/1       Running   0          1m
      memcached-sample-6fd7c98d8-m7vn7      1/1       Running   0          1m

   2. Check the CR status:

      $ oc get memcached/memcached-sample -o yaml

      Example output

      apiVersion: cache.example.com/v1
      kind: Memcached
      metadata:
      ...
        name: memcached-sample
      ...
      spec:
        size: 3
      status:
        nodes:
        - memcached-sample-6fd7c98d8-7dqdr
        - memcached-sample-6fd7c98d8-g5k7v
        - memcached-sample-6fd7c98d8-m7vn7

6. Update the deployment size.

   1. Update config/samples/cache_v1_memcached.yaml file to change the spec.size field in the Memcached CR from 3 to 5:

      $ oc patch memcached memcached-sample \
          -p '{"spec":{"size": 5}}' \
          --type=merge

   2. Confirm that the Operator changes the deployment size:

      $ oc get deployments

      Example output

      NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
      memcached-operator-controller-manager   1/1     1            1           10m
      memcached-sample                        5/5     5            5           3m

7. Delete the CR by running the following command:

   $ oc delete -f config/samples/cache_v1_memcached.yaml

8. Clean up the resources that have been created as part of this tutorial.

   • If you used the make deploy command to test the Operator, run the following command:

     $ make undeploy

   • If you used the operator-sdk run bundle command to test the Operator, run the following command:

     $ operator-sdk cleanup <project_name>

Procedure

1. Make the following changes to your Operator’s Dockerfile:

   1. Replace the ansible-operator-2.11-preview base image with the ansible-operator base image and update the version to 1.31.0, as shown in the following example:

      Example Dockerfile

      FROM quay.io/operator-framework/ansible-operator:v1.31.0

   2. The update to Ansible 2.15.0 in version 1.30.0 of the Ansible Operator removed the following preinstalled Python modules:

      • ipaddress
      • openshift
      • jmespath
      • cryptography
      • oauthlib

      If your Operator depends on one of these removed Python modules, update your Dockerfile to install the required modules using the pip install command.

2. Edit your Operator project’s makefile to update the Operator SDK version to 1.31.0, as shown in the following example:

   Example makefile

   # Set the Operator SDK version to use. By default, what is installed on the system is used.
   # This is useful for CI or a project to utilize a specific version of the operator-sdk toolkit.
   OPERATOR_SDK_VERSION ?= v1.31.0 1

   1

   Change the version from 1.28.0 to 1.31.0.

3. Update your requirements.yaml and requirements.go files to remove the community.kubernetes collection and update the operator_sdk.util collection to version 0.5.0, as shown in the following example:

   Example requirements.yaml file

     collections:
   -  - name: community.kubernetes 1
   -    version: "2.0.1"
      - name: operator_sdk.util
   -    version: "0.4.0"
   +    version: "0.5.0" 2
      - name: kubernetes.core
        version: "2.4.0"
      - name: cloud.common

   1

   Remove the community.kubernetes collection

   2

   Update the operator_sdk.util collection to version 0.5.0.

4. Remove all instances of the lint field from your molecule/kind/molecule.yml and molecule/default/molecule.yml files, as shown in the following example:

     ---
     dependency:
       name: galaxy
     driver:
       name: delegated
   -   lint: |
   -     set -e
   -     yamllint -d "{extends: relaxed, rules: {line-length: {max: 120}}}" .
     platforms:
       - name: cluster
         groups:
   	- k8s
     provisioner:
       name: ansible
   -     lint: |
   -       set -e
         ansible-lint
       inventory:
         group_vars:
   	all:
   	  namespace: ${TEST_OPERATOR_NAMESPACE:-osdk-test}
         host_vars:
   	localhost:
   	  ansible_python_interpreter: '{{ ansible_playbook_python }}'
   	  config_dir: ${MOLECULE_PROJECT_DIRECTORY}/config
   	  samples_dir: ${MOLECULE_PROJECT_DIRECTORY}/config/samples
   	  operator_image: ${OPERATOR_IMAGE:-""}
   	  operator_pull_policy: ${OPERATOR_PULL_POLICY:-"Always"}
   	  kustomize: ${KUSTOMIZE_PATH:-kustomize}
       env:
         K8S_AUTH_KUBECONFIG: ${KUBECONFIG:-"~/.kube/config"}
     verifier:
       name: ansible
   -     lint: |
   -       set -e
   -      ansible-lint

Procedure

1. Install Ansible 2.15+:

   $ sudo dnf install ansible

2. Install the Python Kubernetes client package:

   $ pip install kubernetes

3. Install the Kubernetes Collection using one of the following methods:

   • You can install the collection directly from Ansible Galaxy:

     $ ansible-galaxy collection install community.kubernetes

   • If you have already initialized your Operator, you might have a requirements.yml file at the top level of your project. This file specifies Ansible dependencies that must be installed for your Operator to function. By default, this file installs the community.kubernetes collection as well as the operator_sdk.util collection, which provides modules and plugins for Operator-specific functions.

     To install the dependent modules from the requirements.yml file:

     $ ansible-galaxy collection install -r requirements.yml

Procedure

1. In your Ansible-based Operator project directory, modify the roles/<kind>/tasks/main.yml file with the Ansible logic that you want. The roles/<kind>/ directory is created when you use the --generate-role flag while creating an API. The <kind> replaceable matches the kind that you specified for the API.

   The following example creates and deletes a config map based on the value of a variable named state:

   ---
   - name: set ConfigMap example-config to {{ state }}
     community.kubernetes.k8s:
       api_version: v1
       kind: ConfigMap
       name: example-config
       namespace: <operator_namespace> 1
       state: "{{ state }}"
     ignore_errors: true 2

   1

   Specify the namespace where you want the config map created.

   2

   Setting ignore_errors: true ensures that deleting a nonexistent config map does not fail.

2. Modify the roles/<kind>/defaults/main.yml file to set state to present by default:

   ---
   state: present

3. Create an Ansible playbook by creating a playbook.yml file in the top-level of your project directory, and include your <kind> role:

   ---
   - hosts: localhost
     roles:
       - <kind>

4. Run the playbook:

   $ ansible-playbook playbook.yml

   Example output

   [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'
   
   PLAY [localhost] ********************************************************************************
   
   TASK [Gathering Facts] ********************************************************************************
   ok: [localhost]
   
   TASK [memcached : set ConfigMap example-config to present] ********************************************************************************
   changed: [localhost]
   
   PLAY RECAP ********************************************************************************
   localhost                  : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

5. Verify that the config map was created:

   $ oc get configmaps

   Example output

   NAME               DATA   AGE
   example-config     0      2m1s

6. Rerun the playbook setting state to absent:

   $ ansible-playbook playbook.yml --extra-vars state=absent

   Example output

   [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'
   
   PLAY [localhost] ********************************************************************************
   
   TASK [Gathering Facts] ********************************************************************************
   ok: [localhost]
   
   TASK [memcached : set ConfigMap example-config to absent] ********************************************************************************
   changed: [localhost]
   
   PLAY RECAP ********************************************************************************
   localhost                  : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

7. Verify that the config map was deleted:

   $ oc get configmaps

Procedure

1. Install your custom resource definition (CRD) and proper role-based access control (RBAC) definitions for your custom resource (CR):

   $ make install

   Example output

   /usr/bin/kustomize build config/crd | kubectl apply -f -
   customresourcedefinition.apiextensions.k8s.io/memcacheds.cache.example.com created

2. Run the make run command:

   $ make run

   Example output

   /home/user/memcached-operator/bin/ansible-operator run
   {"level":"info","ts":1612739145.2871568,"logger":"cmd","msg":"Version","Go Version":"go1.15.5","GOOS":"linux","GOARCH":"amd64","ansible-operator":"v1.10.1","commit":"1abf57985b43bf6a59dcd18147b3c574fa57d3f6"}
   ...
   {"level":"info","ts":1612739148.347306,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"}
   {"level":"info","ts":1612739148.3488882,"logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE_VERBOSITY_MEMCACHED_CACHE_EXAMPLE_COM","default":2}
   {"level":"info","ts":1612739148.3490262,"logger":"cmd","msg":"Environment variable not set; using default value","Namespace":"","envVar":"ANSIBLE_DEBUG_LOGS","ANSIBLE_DEBUG_LOGS":false}
   {"level":"info","ts":1612739148.3490646,"logger":"ansible-controller","msg":"Watching resource","Options.Group":"cache.example.com","Options.Version":"v1","Options.Kind":"Memcached"}
   {"level":"info","ts":1612739148.350217,"logger":"proxy","msg":"Starting to serve","Address":"127.0.0.1:8888"}
   {"level":"info","ts":1612739148.3506632,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
   {"level":"info","ts":1612739148.350784,"logger":"controller-runtime.manager.controller.memcached-controller","msg":"Starting EventSource","source":"kind source: cache.example.com/v1, Kind=Memcached"}
   {"level":"info","ts":1612739148.5511978,"logger":"controller-runtime.manager.controller.memcached-controller","msg":"Starting Controller"}
   {"level":"info","ts":1612739148.5512562,"logger":"controller-runtime.manager.controller.memcached-controller","msg":"Starting workers","worker count":8}

   With the Operator now watching your CR for events, the creation of a CR will trigger your Ansible role to run.

   Note

   Consider an example config/samples/<gvk>.yaml CR manifest:

   apiVersion: <group>.example.com/v1alpha1
   kind: <kind>
   metadata:
     name: "<kind>-sample"

   Because the spec field is not set, Ansible is invoked with no extra variables. Passing extra variables from a CR to Ansible is covered in another section. It is important to set reasonable defaults for the Operator.

3. Create an instance of your CR with the default variable state set to present:

   $ oc apply -f config/samples/<gvk>.yaml

4. Check that the example-config config map was created:

   $ oc get configmaps

   Example output

   NAME                    STATUS    AGE
   example-config          Active    3s

5. Modify your config/samples/<gvk>.yaml file to set the state field to absent. For example:

   apiVersion: cache.example.com/v1
   kind: Memcached
   metadata:
     name: memcached-sample
   spec:
     state: absent

6. Apply the changes:

   $ oc apply -f config/samples/<gvk>.yaml

7. Confirm that the config map is deleted:

   $ oc get configmap

Procedure

1. Run the following make commands to build and push the Operator image. Modify the IMG argument in the following steps to reference a repository that you have access to. You can obtain an account for storing containers at repository sites such as Quay.io.

   1. Build the image:

      $ make docker-build IMG=<registry>/<user>/<image_name>:<tag>

      Note

      The Dockerfile generated by the SDK for the Operator explicitly references GOARCH=amd64 for go build. This can be amended to GOARCH=$TARGETARCH for non-AMD64 architectures. Docker will automatically set the environment variable to the value specified by –platform. With Buildah, the –build-arg will need to be used for the purpose. For more information, see Multiple Architectures.

   2. Push the image to a repository:

      $ make docker-push IMG=<registry>/<user>/<image_name>:<tag>

      Note

      The name and tag of the image, for example IMG=<registry>/<user>/<image_name>:<tag>, in both the commands can also be set in your Makefile. Modify the IMG ?= controller:latest value to set your default image name.

2. Run the following command to deploy the Operator:

   $ make deploy IMG=<registry>/<user>/<image_name>:<tag>

   By default, this command creates a namespace with the name of your Operator project in the form <project_name>-system and is used for the deployment. This command also installs the RBAC manifests from config/rbac.

3. Run the following command to verify that the Operator is running:

   $ oc get deployment -n <project_name>-system

   Example output

   NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
   <project_name>-controller-manager       1/1     1            1           8m

Procedure

1. Update the watches.yaml file with a manageStatus field set to false:

   - version: v1
     group: api.example.com
     kind: <kind>
     role: <role>
     manageStatus: false

2. Use the operator_sdk.util.k8s_status Ansible module to update the subresource. For example, to update with key test and value data, operator_sdk.util can be used as shown:

   - operator_sdk.util.k8s_status:
       api_version: app.example.com/v1
       kind: <kind>
       name: "{{ ansible_operator_meta.name }}"
       namespace: "{{ ansible_operator_meta.namespace }}"
       status:
         test: data

3. You can declare collections in the meta/main.yml file for the role, which is included for scaffolded Ansible-based Operators:

   collections:
     - operator_sdk.util

4. After declaring collections in the role meta, you can invoke the k8s_status module directly:

   k8s_status:
     ...
     status:
       key1: value1

Procedure

1. Create a project.

   1. Create your project directory:

      $ mkdir nginx-operator

   2. Change into the project directory:

      $ cd nginx-operator

   3. Run the operator-sdk init command with the helm plugin to initialize the project:

      $ operator-sdk init \
          --plugins=helm

2. Create an API.

   Create a simple Nginx API:

   $ operator-sdk create api \
       --group demo \
       --version v1 \
       --kind Nginx

   This API uses the built-in Helm chart boilerplate from the helm create command.

3. Build and push the Operator image.

   Use the default Makefile targets to build and push your Operator. Set IMG with a pull spec for your image that uses a registry you can push to:

   $ make docker-build docker-push IMG=<registry>/<user>/<image_name>:<tag>

4. Run the Operator.

   1. Install the CRD:

      $ make install

   2. Deploy the project to the cluster. Set IMG to the image that you pushed:

      $ make deploy IMG=<registry>/<user>/<image_name>:<tag>

5. Add a security context constraint (SCC).

   The Nginx service account requires privileged access to run in OpenShift Container Platform. Add the following SCC to the service account for the nginx-sample pod:

   $ oc adm policy add-scc-to-user \
       anyuid system:serviceaccount:nginx-operator-system:nginx-sample

6. Create a sample custom resource (CR).

   1. Create a sample CR:

      $ oc apply -f config/samples/demo_v1_nginx.yaml \
          -n nginx-operator-system

   2. Watch for the CR to reconcile the Operator:

      $ oc logs deployment.apps/nginx-operator-controller-manager \
          -c manager \
          -n nginx-operator-system

7. Delete a CR.

   Delete a CR by running the following command:

   $ oc delete -f config/samples/demo_v1_nginx.yaml -n nginx-operator-system

8. Clean up.

   Run the following command to clean up the resources that have been created as part of this procedure:

   $ make undeploy

Procedure

1. Create a directory for the project:

   $ mkdir -p $HOME/projects/nginx-operator

2. Change to the directory:

   $ cd $HOME/projects/nginx-operator

3. Run the operator-sdk init command with the helm plugin to initialize the project:

   $ operator-sdk init \
       --plugins=helm \
       --domain=example.com \
       --group=demo \
       --version=v1 \
       --kind=Nginx

   Note

   By default, the helm plugin initializes a project using a boilerplate Helm chart. You can use additional flags, such as the --helm-chart flag, to initialize a project using an existing Helm chart.

   The init command creates the nginx-operator project specifically for watching a resource with API version example.com/v1 and kind Nginx.

4. For Helm-based projects, the init command generates the RBAC rules in the config/rbac/role.yaml file based on the resources that would be deployed by the default manifest for the chart. Verify that the rules generated in this file meet the permission requirements of the Operator.

Procedure

1. Edit the watches.yaml file to include overrides based on an environment variable by adding the overrideValues field:

   ...
   - group: demo.example.com
     version: v1alpha1
     kind: Nginx
     chart: helm-charts/nginx
     overrideValues:
       proxy.http: $HTTP_PROXY
   ...

2. Add the proxy.http value in the helm-charts/nginx/values.yaml file:

   ...
   proxy:
     http: ""
     https: ""
     no_proxy: ""

3. To make sure the chart template supports using the variables, edit the chart template in the helm-charts/nginx/templates/deployment.yaml file to contain the following:

   containers:
     - name: {{ .Chart.Name }}
       securityContext:
         - toYaml {{ .Values.securityContext | nindent 12 }}
       image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
       imagePullPolicy: {{ .Values.image.pullPolicy }}
       env:
         - name: http_proxy
           value: "{{ .Values.proxy.http }}"

4. Set the environment variable on the Operator deployment by adding the following to the config/manager/manager.yaml file:

   containers:
    - args:
      - --leader-elect
      - --leader-election-id=ansible-proxy-demo
      image: controller:latest
      name: manager
      env:
        - name: "HTTP_PROXY"
          value: "http_proxy_test"

Procedure

1. Change to the namespace where your Operator is installed. For example, if you deployed the Operator using the make deploy command:

   $ oc project nginx-operator-system

2. Edit the sample Nginx CR manifest at config/samples/demo_v1_nginx.yaml to contain the following specification:

   apiVersion: demo.example.com/v1
   kind: Nginx
   metadata:
     name: nginx-sample
   ...
   spec:
   ...
     replicaCount: 3

3. The Nginx service account requires privileged access to run in OpenShift Container Platform. Add the following security context constraint (SCC) to the service account for the nginx-sample pod:

   $ oc adm policy add-scc-to-user \
       anyuid system:serviceaccount:nginx-operator-system:nginx-sample

4. Create the CR:

   $ oc apply -f config/samples/demo_v1_nginx.yaml

5. Ensure that the Nginx Operator creates the deployment for the sample CR with the correct size:

   $ oc get deployments

   Example output

   NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
   nginx-operator-controller-manager       1/1     1            1           8m
   nginx-sample                            3/3     3            3           1m

6. Check the pods and CR status to confirm the status is updated with the Nginx pod names.

   1. Check the pods:

      $ oc get pods

      Example output

      NAME                                  READY     STATUS    RESTARTS   AGE
      nginx-sample-6fd7c98d8-7dqdr          1/1       Running   0          1m
      nginx-sample-6fd7c98d8-g5k7v          1/1       Running   0          1m
      nginx-sample-6fd7c98d8-m7vn7          1/1       Running   0          1m

   2. Check the CR status:

      $ oc get nginx/nginx-sample -o yaml

      Example output

      apiVersion: demo.example.com/v1
      kind: Nginx
      metadata:
      ...
        name: nginx-sample
      ...
      spec:
        replicaCount: 3
      status:
        nodes:
        - nginx-sample-6fd7c98d8-7dqdr
        - nginx-sample-6fd7c98d8-g5k7v
        - nginx-sample-6fd7c98d8-m7vn7

7. Update the deployment size.

   1. Update config/samples/demo_v1_nginx.yaml file to change the spec.size field in the Nginx CR from 3 to 5:

      $ oc patch nginx nginx-sample \
          -p '{"spec":{"replicaCount": 5}}' \
          --type=merge

   2. Confirm that the Operator changes the deployment size:

      $ oc get deployments

      Example output

      NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
      nginx-operator-controller-manager       1/1     1            1           10m
      nginx-sample                            5/5     5            5           3m

8. Delete the CR by running the following command:

   $ oc delete -f config/samples/demo_v1_nginx.yaml

9. Clean up the resources that have been created as part of this tutorial.

   • If you used the make deploy command to test the Operator, run the following command:

     $ make undeploy

   • If you used the operator-sdk run bundle command to test the Operator, run the following command:

     $ operator-sdk cleanup <project_name>

Procedure

1. Edit your Operator’s Dockerfile to update the Helm Operator version to 1.31.0, as shown in the following example:

   Example Dockerfile

   FROM quay.io/operator-framework/helm-operator:v1.31.0 1

   1

   Update the Helm Operator version from 1.28.0 to 1.31.0

2. Edit your Operator project’s makefile to update the Operator SDK to 1.31.0, as shown in the following example:

   Example makefile

   # Set the Operator SDK version to use. By default, what is installed on the system is used.
   # This is useful for CI or a project to utilize a specific version of the operator-sdk toolkit.
   OPERATOR_SDK_VERSION ?= v1.31.0 1

   1

   Change the version from 1.28.0 to 1.31.0.

3. If you use a custom service account for deployment, define the following role to require a watch operation on your secrets resource, as shown in the following example:

   Example config/rbac/role.yaml file

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: <operator_name>-admin
   subjects:
   - kind: ServiceAccount
     name: <operator_name>
     namespace: <operator_namespace>
   roleRef:
     kind: ClusterRole
     name: cluster-admin
     apiGroup: ""
   rules: 1
     - apiGroups:
         - ""
       resources:
         - secrets
       verbs:
         - watch

   1

   Add the rules stanza to create a watch operation for your secrets resource.

Procedure

1. Create a directory for the project:

   $ mkdir -p $HOME/github.com/example/memcached-operator

2. Change to the directory:

   $ cd $HOME/github.com/example/memcached-operator

3. Run the operator-sdk init command to initialize the project. This example uses a domain of my.domain so that all API groups are <group>.my.domain:

   $ operator-sdk init \
       --plugins=hybrid.helm.sdk.operatorframework.io \
       --project-version="3" \
       --domain my.domain \
       --repo=github.com/example/memcached-operator

   The init command generates the RBAC rules in the config/rbac/role.yaml file based on the resources that would be deployed by the chart’s default manifests. Verify that the rules generated in the config/rbac/role.yaml file meet your Operator’s permission requirements.

Procedure

1. Run the following command to create a Go API with group cache, version v1, and kind MemcachedBackup:

   $ operator-sdk create api \
       --group=cache \
       --version v1 \
       --kind MemcachedBackup \
       --resource \
       --controller \
       --plugins=go/v3

2. When prompted, enter y for creating both resource and controller:

   $ Create Resource [y/n]
   y
   Create Controller [y/n]
   y

Procedure

1. Run the following make commands to build and push the Operator image. Modify the IMG argument in the following steps to reference a repository that you have access to. You can obtain an account for storing containers at repository sites such as Quay.io.

   1. Build the image:

      $ make docker-build IMG=<registry>/<user>/<image_name>:<tag>

      Note

      The Dockerfile generated by the SDK for the Operator explicitly references GOARCH=amd64 for go build. This can be amended to GOARCH=$TARGETARCH for non-AMD64 architectures. Docker will automatically set the environment variable to the value specified by –platform. With Buildah, the –build-arg will need to be used for the purpose. For more information, see Multiple Architectures.

   2. Push the image to a repository:

      $ make docker-push IMG=<registry>/<user>/<image_name>:<tag>

      Note

      The name and tag of the image, for example IMG=<registry>/<user>/<image_name>:<tag>, in both the commands can also be set in your Makefile. Modify the IMG ?= controller:latest value to set your default image name.

2. Run the following command to deploy the Operator:

   $ make deploy IMG=<registry>/<user>/<image_name>:<tag>

   By default, this command creates a namespace with the name of your Operator project in the form <project_name>-system and is used for the deployment. This command also installs the RBAC manifests from config/rbac.

3. Run the following command to verify that the Operator is running:

   $ oc get deployment -n <project_name>-system

   Example output

   NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
   <project_name>-controller-manager       1/1     1            1           8m

Procedure

1. Change to the namespace where your Operator is installed:

   $ oc project <project_name>-system

2. Update the sample Memcached CR manifest at the config/samples/cache_v1_memcached.yaml file by updating the replicaCount field to 3:

   Example 5.7. Example config/samples/cache_v1_memcached.yaml file

   apiVersion: cache.my.domain/v1
   kind: Memcached
   metadata:
     name: memcached-sample
   spec:
     # Default values copied from <project_dir>/helm-charts/memcached/values.yaml
     affinity: {}
     autoscaling:
       enabled: false
       maxReplicas: 100
       minReplicas: 1
       targetCPUUtilizationPercentage: 80
     fullnameOverride: ""
     image:
       pullPolicy: IfNotPresent
       repository: nginx
       tag: ""
     imagePullSecrets: []
     ingress:
       annotations: {}
       className: ""
       enabled: false
       hosts:
       - host: chart-example.local
         paths:
         - path: /
           pathType: ImplementationSpecific
       tls: []
     nameOverride: ""
     nodeSelector: {}
     podAnnotations: {}
     podSecurityContext: {}
     replicaCount: 3
     resources: {}
     securityContext: {}
     service:
       port: 80
       type: ClusterIP
     serviceAccount:
       annotations: {}
       create: true
       name: ""
     tolerations: []

3. Create the Memcached CR:

   $ oc apply -f config/samples/cache_v1_memcached.yaml

4. Ensure that the Memcached Operator creates the deployment for the sample CR with the correct size:

   $ oc get pods

   Example output

   NAME                                  READY     STATUS    RESTARTS   AGE
   memcached-sample-6fd7c98d8-7dqdr      1/1       Running   0          18m
   memcached-sample-6fd7c98d8-g5k7v      1/1       Running   0          18m
   memcached-sample-6fd7c98d8-m7vn7      1/1       Running   0          18m

5. Update the sample MemcachedBackup CR manifest at the config/samples/cache_v1_memcachedbackup.yaml file by updating the size to 2:

   Example 5.8. Example config/samples/cache_v1_memcachedbackup.yaml file

   apiVersion: cache.my.domain/v1
   kind: MemcachedBackup
   metadata:
     name: memcachedbackup-sample
   spec:
     size: 2

6. Create the MemcachedBackup CR:

   $ oc apply -f config/samples/cache_v1_memcachedbackup.yaml

7. Ensure that the count of memcachedbackup pods is the same as specified in the CR:

   $ oc get pods

   Example output

   NAME                                        READY     STATUS    RESTARTS   AGE
   memcachedbackup-sample-8649699989-4bbzg     1/1       Running   0          22m
   memcachedbackup-sample-8649699989-mq6mx     1/1       Running   0          22m

8. You can update the spec in each of the above CRs, and then apply them again. The controller reconciles again and ensures that the size of the pods is as specified in the spec of the respective CRs.

9. Clean up the resources that have been created as part of this tutorial:

   1. Delete the Memcached resource:

      $ oc delete -f config/samples/cache_v1_memcached.yaml

   2. Delete the MemcachedBackup resource:

      $ oc delete -f config/samples/cache_v1_memcachedbackup.yaml

   3. If you used the make deploy command to test the Operator, run the following command:

      $ make undeploy

Procedure

1. Create a project.

   1. Create your project directory:

      $ mkdir memcached-operator

   2. Change into the project directory:

      $ cd memcached-operator

   3. Run the operator-sdk init command with the quarkus plugin to initialize the project:

      $ operator-sdk init \
          --plugins=quarkus \
          --domain=example.com \
          --project-name=memcached-operator

2. Create an API.

   Create a simple Memcached API:

   $ operator-sdk create api \
       --plugins quarkus \
       --group cache \
       --version v1 \
       --kind Memcached

3. Build and push the Operator image.

   Use the default Makefile targets to build and push your Operator. Set IMG with a pull spec for your image that uses a registry you can push to:

   $ make docker-build docker-push IMG=<registry>/<user>/<image_name>:<tag>

4. Run the Operator.

   1. Install the CRD:

      $ make install

   2. Deploy the project to the cluster. Set IMG to the image that you pushed:

      $ make deploy IMG=<registry>/<user>/<image_name>:<tag>

5. Create a sample custom resource (CR).

   1. Create a sample CR:

      $ oc apply -f config/samples/cache_v1_memcached.yaml \
          -n memcached-operator-system

   2. Watch for the CR to reconcile the Operator:

      $ oc logs deployment.apps/memcached-operator-controller-manager \
          -c manager \
          -n memcached-operator-system

6. Delete a CR.

   Delete a CR by running the following command:

   $ oc delete -f config/samples/cache_v1_memcached.yaml -n memcached-operator-system

7. Clean up.

   Run the following command to clean up the resources that have been created as part of this procedure:

   $ make undeploy

Procedure

1. Create a directory for the project:

   $ mkdir -p $HOME/projects/memcached-operator

2. Change to the directory:

   $ cd $HOME/projects/memcached-operator

3. Run the operator-sdk init command with the quarkus plugin to initialize the project:

   $ operator-sdk init \
       --plugins=quarkus \
       --domain=example.com \
       --project-name=memcached-operator

Procedure

1. Run the following command to create an API:

   $ operator-sdk create api \
       --plugins=quarkus \1
       --group=cache \2
       --version=v1 \3
       --kind=Memcached 4

   1

   Set the plugin flag to quarkus.

   2

   Set the group flag to cache.

   3

   Set the version flag to v1.

   4

   Set the kind flag to Memcached.

Verification

1. Run the tree command to view the file structure:

   $ tree

   Example output

   .
   ├── Makefile
   ├── PROJECT
   ├── pom.xml
   └── src
       └── main
           ├── java
           │   └── com
           │       └── example
           │           ├── Memcached.java
           │           ├── MemcachedReconciler.java
           │           ├── MemcachedSpec.java
           │           └── MemcachedStatus.java
           └── resources
               └── application.properties
   
   6 directories, 8 files

Procedure

1. Append the following dependency to the pom.xml file:

       <dependency>
         <groupId>commons-collections</groupId>
         <artifactId>commons-collections</artifactId>
         <version>3.2.2</version>
       </dependency>

2. For this example, replace the generated controller file MemcachedReconciler.java with following example implementation:

   Example 5.9. Example MemcachedReconciler.java

   package com.example;
   
   import io.fabric8.kubernetes.client.KubernetesClient;
   import io.javaoperatorsdk.operator.api.reconciler.Context;
   import io.javaoperatorsdk.operator.api.reconciler.Reconciler;
   import io.javaoperatorsdk.operator.api.reconciler.UpdateControl;
   import io.fabric8.kubernetes.api.model.ContainerBuilder;
   import io.fabric8.kubernetes.api.model.ContainerPortBuilder;
   import io.fabric8.kubernetes.api.model.LabelSelectorBuilder;
   import io.fabric8.kubernetes.api.model.ObjectMetaBuilder;
   import io.fabric8.kubernetes.api.model.OwnerReferenceBuilder;
   import io.fabric8.kubernetes.api.model.Pod;
   import io.fabric8.kubernetes.api.model.PodSpecBuilder;
   import io.fabric8.kubernetes.api.model.PodTemplateSpecBuilder;
   import io.fabric8.kubernetes.api.model.apps.Deployment;
   import io.fabric8.kubernetes.api.model.apps.DeploymentBuilder;
   import io.fabric8.kubernetes.api.model.apps.DeploymentSpecBuilder;
   import org.apache.commons.collections.CollectionUtils;
   import java.util.HashMap;
   import java.util.List;
   import java.util.Map;
   import java.util.stream.Collectors;
   
   public class MemcachedReconciler implements Reconciler<Memcached> {
     private final KubernetesClient client;
   
     public MemcachedReconciler(KubernetesClient client) {
       this.client = client;
     }
   
     // TODO Fill in the rest of the reconciler
   
     @Override
     public UpdateControl<Memcached> reconcile(
         Memcached resource, Context context) {
         // TODO: fill in logic
         Deployment deployment = client.apps()
                 .deployments()
                 .inNamespace(resource.getMetadata().getNamespace())
                 .withName(resource.getMetadata().getName())
                 .get();
   
         if (deployment == null) {
             Deployment newDeployment = createMemcachedDeployment(resource);
             client.apps().deployments().create(newDeployment);
             return UpdateControl.noUpdate();
         }
   
         int currentReplicas = deployment.getSpec().getReplicas();
         int requiredReplicas = resource.getSpec().getSize();
   
         if (currentReplicas != requiredReplicas) {
             deployment.getSpec().setReplicas(requiredReplicas);
             client.apps().deployments().createOrReplace(deployment);
             return UpdateControl.noUpdate();
         }
   
         List<Pod> pods = client.pods()
             .inNamespace(resource.getMetadata().getNamespace())
             .withLabels(labelsForMemcached(resource))
             .list()
             .getItems();
   
         List<String> podNames =
             pods.stream().map(p -> p.getMetadata().getName()).collect(Collectors.toList());
   
   
         if (resource.getStatus() == null
                  || !CollectionUtils.isEqualCollection(podNames, resource.getStatus().getNodes())) {
              if (resource.getStatus() == null) resource.setStatus(new MemcachedStatus());
              resource.getStatus().setNodes(podNames);
              return UpdateControl.updateResource(resource);
         }
   
         return UpdateControl.noUpdate();
     }
   
     private Map<String, String> labelsForMemcached(Memcached m) {
       Map<String, String> labels = new HashMap<>();
       labels.put("app", "memcached");
       labels.put("memcached_cr", m.getMetadata().getName());
       return labels;
     }
   
     private Deployment createMemcachedDeployment(Memcached m) {
         Deployment deployment = new DeploymentBuilder()
             .withMetadata(
                 new ObjectMetaBuilder()
                     .withName(m.getMetadata().getName())
                     .withNamespace(m.getMetadata().getNamespace())
                     .build())
             .withSpec(
                 new DeploymentSpecBuilder()
                     .withReplicas(m.getSpec().getSize())
                     .withSelector(
                         new LabelSelectorBuilder().withMatchLabels(labelsForMemcached(m)).build())
                     .withTemplate(
                         new PodTemplateSpecBuilder()
                             .withMetadata(
                                 new ObjectMetaBuilder().withLabels(labelsForMemcached(m)).build())
                             .withSpec(
                                 new PodSpecBuilder()
                                     .withContainers(
                                         new ContainerBuilder()
                                             .withImage("memcached:1.4.36-alpine")
                                             .withName("memcached")
                                             .withCommand("memcached", "-m=64", "-o", "modern", "-v")
                                             .withPorts(
                                                 new ContainerPortBuilder()
                                                     .withContainerPort(11211)
                                                     .withName("memcached")
                                                     .build())
                                             .build())
                                     .build())
                             .build())
                     .build())
             .build();
       deployment.addOwnerReference(m);
       return deployment;
     }
   }

   The example controller runs the following reconciliation logic for each Memcached custom resource (CR):

   • Creates a Memcached deployment if it does not exist.
   • Ensures that the deployment size matches the size specified by the Memcached CR spec.
   • Updates the Memcached CR status with the names of the memcached pods.

1. Valid values are shown intentionally with double quotes, because Kubernetes annotations must be strings.

Procedure

1. Before marking one of your CRDs as internal, ensure that any debugging information or configuration that might be required to manage the application is reflected on the status or spec block of your CR, if applicable to your Operator.

2. Add the operators.operatorframework.io/internal-objects annotation to the CSV of your Operator to specify any internal objects to hide in the user interface:

   Internal object annotation

   apiVersion: operators.coreos.com/v1alpha1
   kind: ClusterServiceVersion
   metadata:
     name: my-operator-v1.2.3
     annotations:
       operators.operatorframework.io/internal-objects: '["my.internal.crd1.io","my.internal.crd2.io"]' 1
   ...

   1

   Set any internal CRDs as an array of strings.