第 4 章 Developing Operators

Procedure

1. Create a new project.

   Use the CLI to create a new memcached-operator project:

   $ mkdir -p $GOPATH/src/github.com/example-inc/

   Copy to Clipboard Toggle word wrap

   $ cd $GOPATH/src/github.com/example-inc/

   Copy to Clipboard Toggle word wrap

   $ operator-sdk new memcached-operator

   Copy to Clipboard Toggle word wrap

   $ cd memcached-operator

   Copy to Clipboard Toggle word wrap

2. Add a new custom resource definition (CRD).

   1. Use the CLI to add a new CRD API called Memcached, with APIVersion set to cache.example.com/v1apha1 and Kind set to Memcached:

      $ operator-sdk add api \
          --api-version=cache.example.com/v1alpha1 \
          --kind=Memcached

      Copy to Clipboard Toggle word wrap

      This scaffolds the Memcached resource API under pkg/apis/cache/v1alpha1/.

   2. Modify the spec and status of the Memcached custom resource (CR) at the pkg/apis/cache/v1alpha1/memcached_types.go file:

      type MemcachedSpec struct {
      	// Size is the size of the memcached deployment
      	Size int32 `json:"size"`
      }
      type MemcachedStatus struct {
      	// Nodes are the names of the memcached pods
      	Nodes []string `json:"nodes"`
      }

      Copy to Clipboard Toggle word wrap

   3. After modifying the *_types.go file, always run the following command to update the generated code for that resource type:

      $ operator-sdk generate k8s

      Copy to Clipboard Toggle word wrap

3. Optional: Add custom validation to your CRD.

   OpenAPI v3.0 schemas are added to CRD manifests in the spec.validation block when the manifests are generated. This validation block allows Kubernetes to validate the properties in a Memcached CR when it is created or updated.

   Additionally, a pkg/apis/<group>/<version>/zz_generated.openapi.go file is generated. This file contains the Go representation of this validation block if the +k8s:openapi-gen=true annotation is present above the Kind type declaration, which is present by default. This auto-generated code is the OpenAPI model of your Go Kind type, from which you can create a full OpenAPI Specification and generate a client.

   As an Operator author, you can use Kubebuilder markers (annotations) to configure custom validations for your API. These markers must always have a +kubebuilder:validation prefix. For example, adding an enum-type specification can be done by adding the following marker:

   // +kubebuilder:validation:Enum=Lion;Wolf;Dragon
   type Alias string

   Copy to Clipboard Toggle word wrap

   Usage of markers in API code is discussed in the Kubebuilder Generating CRDs and Markers for Config/Code Generation documentation. A full list of OpenAPIv3 validation markers is also available in the Kubebuilder CRD Validation documentation.

   If you add any custom validations, run the following command to update the OpenAPI validation section in the deploy/crds/cache.example.com_memcacheds_crd.yaml file for the CRD:

   $ operator-sdk generate crds

   Copy to Clipboard Toggle word wrap

   Example generated YAML

   spec:
     validation:
       openAPIV3Schema:
         properties:
           spec:
             properties:
               size:
                 format: int32
                 type: integer

   Copy to Clipboard Toggle word wrap

4. Add a new controller.

   1. Add a new controller to the project to watch and reconcile the Memcached resource:

      $ operator-sdk add controller \
          --api-version=cache.example.com/v1alpha1 \
          --kind=Memcached

      Copy to Clipboard Toggle word wrap

      This scaffolds a new controller implementation under pkg/controller/memcached/.

   2. For this example, replace the generated controller file pkg/controller/memcached/memcached_controller.go with the example implementation.

      The example controller executes the following reconciliation logic for each Memcached resource:

      • Create a Memcached deployment if it does not exist.
      • Ensure that the Deployment size is the same as specified by the Memcached CR spec.
      • Update the Memcached resource status with the names of the Memcached pods.

      The next two sub-steps inspect how the controller watches resources and how the reconcile loop is triggered. You can skip these steps to go directly to building and running the Operator.

   3. Inspect the controller implementation at the pkg/controller/memcached/memcached_controller.go file to see how the controller watches resources.

      The first watch is for the Memcached type as the primary resource. For each add, update, or delete event, the reconcile loop is sent a reconcile Request (a <namespace>:<name> key) for that Memcached object:

      err := c.Watch(
        &source.Kind{Type: &cachev1alpha1.Memcached{}}, &handler.EnqueueRequestForObject{})

      Copy to Clipboard Toggle word wrap

      The next watch is for Deployment objects, but the event handler maps each event to a reconcile Request for the owner of the deployment. In this case, this is the Memcached object for which the deployment was created. This allows the controller to watch deployments as a secondary resource:

      err := c.Watch(&source.Kind{Type: &appsv1.Deployment{}}, &handler.EnqueueRequestForOwner{
      		IsController: true,
      		OwnerType:    &cachev1alpha1.Memcached{},
      	})

      Copy to Clipboard Toggle word wrap

   4. Every controller has a Reconciler object with a Reconcile() method that implements the reconcile loop. The reconcile loop is passed the Request argument which is a <namespace>:<name> key used to lookup the primary resource object, Memcached, from the cache:

      func (r *ReconcileMemcached) Reconcile(request reconcile.Request) (reconcile.Result, error) {
        // Lookup the Memcached instance for this reconcile request
        memcached := &cachev1alpha1.Memcached{}
        err := r.client.Get(context.TODO(), request.NamespacedName, memcached)
        ...
      }

      Copy to Clipboard Toggle word wrap

      Based on the return value of the Reconcile() function, the reconcile Request might be requeued, and the loop might be triggered again:

      // Reconcile successful - don't requeue
      return reconcile.Result{}, nil
      // Reconcile failed due to error - requeue
      return reconcile.Result{}, err
      // Requeue for any reason other than error
      return reconcile.Result{Requeue: true}, nil

      Copy to Clipboard Toggle word wrap

5. Build and run the Operator.

   1. Before running the Operator, the CRD must be registered with the Kubernetes API server:

      $ oc create \
          -f deploy/crds/cache_v1alpha1_memcached_crd.yaml

      Copy to Clipboard Toggle word wrap

   2. After registering the CRD, there are two options for running the Operator:

      • As a Deployment inside a Kubernetes cluster
      • As Go program outside a cluster

      Choose one of the following methods.

      1. Option A: Running as a deployment inside the cluster.

         1. Build the memcached-operator image and push it to a registry:

            $ operator-sdk build quay.io/example/memcached-operator:v0.0.1

            Copy to Clipboard Toggle word wrap

         2. The deployment manifest is generated at deploy/operator.yaml. Update the deployment image as follows since the default is just a placeholder:

            $ sed -i 's|REPLACE_IMAGE|quay.io/example/memcached-operator:v0.0.1|g' deploy/operator.yaml

            Copy to Clipboard Toggle word wrap
         3. Ensure you have an account on Quay.io for the next step, or substitute your preferred container registry. On the registry, create a new public image repository named memcached-operator.

         4. Push the image to the registry:

            $ podman push quay.io/example/memcached-operator:v0.0.1

            Copy to Clipboard Toggle word wrap

         5. Set up RBAC and create the memcached-operator manifests:

            $ oc create -f deploy/role.yaml

            Copy to Clipboard Toggle word wrap

            $ oc create -f deploy/role_binding.yaml

            Copy to Clipboard Toggle word wrap

            $ oc create -f deploy/service_account.yaml

            Copy to Clipboard Toggle word wrap

            $ oc create -f deploy/operator.yaml

            Copy to Clipboard Toggle word wrap

         6. Verify that the memcached-operator deploy is up and running:

            $ oc get deployment

            Copy to Clipboard Toggle word wrap

            Example output

            NAME                     DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
            memcached-operator       1         1         1            1           1m

            Copy to Clipboard Toggle word wrap

      2. Option B: Running locally outside the cluster.

         This method is preferred during development cycle to deploy and test faster.

         Run the Operator locally with the default Kubernetes configuration file present at $HOME/.kube/config:

         $ operator-sdk run --local --namespace=default

         Copy to Clipboard Toggle word wrap

         You can use a specific kubeconfig using the flag --kubeconfig=<path/to/kubeconfig>.

6. Verify that the Operator can deploy a Memcached application by creating a Memcached CR.

   1. Create the example Memcached CR that was generated at deploy/crds/cache_v1alpha1_memcached_cr.yaml.

   2. View the file:

      $ cat deploy/crds/cache_v1alpha1_memcached_cr.yaml

      Copy to Clipboard Toggle word wrap

      Example output

      apiVersion: "cache.example.com/v1alpha1"
      kind: "Memcached"
      metadata:
        name: "example-memcached"
      spec:
        size: 3

      Copy to Clipboard Toggle word wrap

   3. Create the object:

      $ oc apply -f deploy/crds/cache_v1alpha1_memcached_cr.yaml

      Copy to Clipboard Toggle word wrap

   4. Ensure that memcached-operator creates the deployment for the CR:

      $ oc get deployment

      Copy to Clipboard Toggle word wrap

      Example output

      NAME                     DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
      memcached-operator       1         1         1            1           2m
      example-memcached        3         3         3            3           1m

      Copy to Clipboard Toggle word wrap

   5. Check the pods and CR to confirm the CR status is updated with the pod names:

      $ oc get pods

      Copy to Clipboard Toggle word wrap

      Example output

      NAME                                  READY     STATUS    RESTARTS   AGE
      example-memcached-6fd7c98d8-7dqdr     1/1       Running   0          1m
      example-memcached-6fd7c98d8-g5k7v     1/1       Running   0          1m
      example-memcached-6fd7c98d8-m7vn7     1/1       Running   0          1m
      memcached-operator-7cc7cfdf86-vvjqk   1/1       Running   0          2m

      Copy to Clipboard Toggle word wrap

      $ oc get memcached/example-memcached -o yaml

      Copy to Clipboard Toggle word wrap

      Example output

      apiVersion: cache.example.com/v1alpha1
      kind: Memcached
      metadata:
        clusterName: ""
        creationTimestamp: 2018-03-31T22:51:08Z
        generation: 0
        name: example-memcached
        namespace: default
        resourceVersion: "245453"
        selfLink: /apis/cache.example.com/v1alpha1/namespaces/default/memcacheds/example-memcached
        uid: 0026cc97-3536-11e8-bd83-0800274106a1
      spec:
        size: 3
      status:
        nodes:
        - example-memcached-6fd7c98d8-7dqdr
        - example-memcached-6fd7c98d8-g5k7v
        - example-memcached-6fd7c98d8-m7vn7

      Copy to Clipboard Toggle word wrap

7. Verify that the Operator can manage a deployed Memcached application by updating the size of the deployment.

   1. Change the spec.size field in the memcached CR from 3 to 4:

      $ cat deploy/crds/cache_v1alpha1_memcached_cr.yaml

      Copy to Clipboard Toggle word wrap

      Example output

      apiVersion: "cache.example.com/v1alpha1"
      kind: "Memcached"
      metadata:
        name: "example-memcached"
      spec:
        size: 4

      Copy to Clipboard Toggle word wrap

   2. Apply the change:

      $ oc apply -f deploy/crds/cache_v1alpha1_memcached_cr.yaml

      Copy to Clipboard Toggle word wrap

   3. Confirm that the Operator changes the deployment size:

      $ oc get deployment

      Copy to Clipboard Toggle word wrap

      Example output

      NAME                 DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
      example-memcached    4         4         4            4           5m

      Copy to Clipboard Toggle word wrap

8. Clean up the resources:

   $ oc delete -f deploy/crds/cache_v1alpha1_memcached_cr.yaml

   Copy to Clipboard Toggle word wrap

   $ oc delete -f deploy/crds/cache_v1alpha1_memcached_crd.yaml

   Copy to Clipboard Toggle word wrap

   $ oc delete -f deploy/operator.yaml

   Copy to Clipboard Toggle word wrap

   $ oc delete -f deploy/role.yaml

   Copy to Clipboard Toggle word wrap

   $ oc delete -f deploy/role_binding.yaml

   Copy to Clipboard Toggle word wrap

   $ oc delete -f deploy/service_account.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Generate an Operator manifest.

   An Operator manifest describes how to display, create, and manage the application, in this case Memcached, as a whole. It is defined by a ClusterServiceVersion (CSV) object and is required for OLM to function.

   From the memcached-operator/ directory that was created when you built the Memcached Operator, generate the CSV manifest:

   $ operator-sdk generate csv --csv-version 0.0.1

   Copy to Clipboard Toggle word wrap
   注意

   See Building a CSV for the Operator Framework for more information on manually defining a manifest file.

2. Create an Operator group that specifies the namespaces that the Operator will target. Create the following Operator group in the namespace where you will create the CSV. In this example, the default namespace is used:

   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: memcached-operator-group
     namespace: default
   spec:
     targetNamespaces:
     - default

   Copy to Clipboard Toggle word wrap

3. Deploy the Operator. Use the files that were generated into the deploy/ directory by the Operator SDK when you built the Memcached Operator.

   1. Edit the generated CSV manifest file by adding displayName fields for each custom resource definition (CRD) kind in the spec.customresourcedefinitions.owned section:

      deploy/olm-catalog/memcached-operator/0.0.1/memcached-operator.v0.0.1.clusterserviceversion.yaml file

      ...
      spec:
        customresourcedefinitions:
          owned:
          - kind: Memcached
            name: memcacheds.cache.example.com
            version: v1alpha1
            description: Memcached is the Schema for the memcacheds API
            displayName: Memcached 

      1

      
      ...

      Copy to Clipboard Toggle word wrap

      1

      Specify a display name for the CRD.

   2. Apply the CSV manifest to the specified namespace in the cluster:

      $ oc apply -f deploy/olm-catalog/memcached-operator/0.0.1/memcached-operator.v0.0.1.clusterserviceversion.yaml

      Copy to Clipboard Toggle word wrap

      When you apply this manifest, the cluster does not immediately update because it does not yet meet the requirements specified in the manifest.

   3. Create the role, role binding, and service account to grant resource permissions to the Operator, and the custom resource definition (CRD) to create the Memcached custom resource that the Operator manages:

      $ oc create -f deploy/crds/cache.example.com_memcacheds_crd.yaml

      Copy to Clipboard Toggle word wrap

      $ oc create -f deploy/service_account.yaml

      Copy to Clipboard Toggle word wrap

      $ oc create -f deploy/role.yaml

      Copy to Clipboard Toggle word wrap

      $ oc create -f deploy/role_binding.yaml

      Copy to Clipboard Toggle word wrap

      Because OLM creates Operators in a particular namespace when a manifest is applied, administrators can leverage the native Kubernetes RBAC permission model to restrict which users are allowed to install Operators.

4. Create an application instance.

   The Memcached Operator is now running in the default namespace. Users interact with Operators via instances of custom resources; in this case, the resource has the kind Memcached. Native Kubernetes RBAC also applies to custom resources, providing administrators control over who can interact with each Operator.

   Creating instances of Memcached objects in this namespace will now trigger the Memcached Operator to instantiate pods running the memcached server that are managed by the Operator. The more custom resources you create, the more unique Memcached application instances are managed by the Memcached Operator running in this namespace.

   $ cat <<EOF | oc apply -f -
   apiVersion: "cache.example.com/v1alpha1"
   kind: "Memcached"
   metadata:
     name: "memcached-for-wordpress"
   spec:
     size: 1
   EOF

   Copy to Clipboard Toggle word wrap

   $ cat <<EOF | oc apply -f -
   apiVersion: "cache.example.com/v1alpha1"
   kind: "Memcached"
   metadata:
     name: "memcached-for-drupal"
   spec:
     size: 1
   EOF

   Copy to Clipboard Toggle word wrap

   $ oc get Memcached

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                      AGE
   memcached-for-drupal      22s
   memcached-for-wordpress   27s

   Copy to Clipboard Toggle word wrap

   $ oc get pods

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                                       READY     STATUS    RESTARTS   AGE
   memcached-app-operator-66b5777b79-pnsfj    1/1       Running   0          14m
   memcached-for-drupal-5476487c46-qbd66      1/1       Running   0          3s
   memcached-for-wordpress-65b75fd8c9-7b9x7   1/1       Running   0          8s

   Copy to Clipboard Toggle word wrap