Specialized hardware and driver enablement

Procedure

1. The image URL of the driver-toolkit corresponding to a certain release can be extracted from the release image using the oc adm command:

   • For an x86 image, the command is as follows:

     $ oc adm release info quay.io/openshift-release-dev/ocp-release:{product-version}.z-x86_64 --image-for=driver-toolkit

   • For an ARM image, the command is as follows:

     $ oc adm release info quay.io/openshift-release-dev/ocp-release:{product-version}.z-aarch64 --image-for=driver-toolkit

   Example output

   The output for the ocp-release:4.11.0-x86_64 image is as follows:

   quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:514e256367e8707e2c30f18a4f06fbd6c821ab9776602d2488e861f577a357d0

2. This image can be pulled using a valid pull secret, such as the pull secret required to install OpenShift Container Platform.

   $ podman pull --authfile=path/to/pullsecret.json quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:<SHA>

1. The YAML defines an ImageStream for storing the simple-kmod driver container image, and a BuildConfig for building the container. Save this YAML as 0000-buildconfig.yaml.template.

   apiVersion: image.openshift.io/v1
   kind: ImageStream
   metadata:
     labels:
       app: simple-kmod-driver-container
     name: simple-kmod-driver-container
     namespace: simple-kmod-demo
   spec: {}
   ---
   apiVersion: build.openshift.io/v1
   kind: BuildConfig
   metadata:
     labels:
       app: simple-kmod-driver-build
     name: simple-kmod-driver-build
     namespace: simple-kmod-demo
   spec:
     nodeSelector:
       node-role.kubernetes.io/worker: ""
     runPolicy: "Serial"
     triggers:
       - type: "ConfigChange"
       - type: "ImageChange"
     source:
       git:
         ref: "master"
         uri: "https://github.com/openshift-psap/kvc-simple-kmod.git"
       type: Git
       dockerfile: |
         FROM DRIVER_TOOLKIT_IMAGE
   
         WORKDIR /build/
   
         # Expecting kmod software version as an input to the build
         ARG KMODVER
   
         # Grab the software from upstream
         RUN git clone https://github.com/openshift-psap/simple-kmod.git
         WORKDIR simple-kmod
   
         # Build and install the module
         RUN make all       KVER=$(rpm -q --qf "%{VERSION}-%{RELEASE}.%{ARCH}"  kernel-core) KMODVER=${KMODVER} \
         && make install   KVER=$(rpm -q --qf "%{VERSION}-%{RELEASE}.%{ARCH}"  kernel-core) KMODVER=${KMODVER}
   
         # Add the helper tools
         WORKDIR /root/kvc-simple-kmod
         ADD Makefile .
         ADD simple-kmod-lib.sh .
         ADD simple-kmod-wrapper.sh .
         ADD simple-kmod.conf .
         RUN mkdir -p /usr/lib/kvc/ \
         && mkdir -p /etc/kvc/ \
         && make install
   
         RUN systemctl enable kmods-via-containers@simple-kmod
     strategy:
       dockerStrategy:
         buildArgs:
           - name: KMODVER
             value: DEMO
     output:
       to:
         kind: ImageStreamTag
         name: simple-kmod-driver-container:demo

2. Substitute the correct driver toolkit image for the OpenShift Container Platform version you are running in place of “DRIVER_TOOLKIT_IMAGE” with the following commands.

   $ OCP_VERSION=$(oc get clusterversion/version -ojsonpath={.status.desired.version})

   $ DRIVER_TOOLKIT_IMAGE=$(oc adm release info $OCP_VERSION --image-for=driver-toolkit)

   $ sed "s#DRIVER_TOOLKIT_IMAGE#${DRIVER_TOOLKIT_IMAGE}#" 0000-buildconfig.yaml.template > 0000-buildconfig.yaml

3. Create the image stream and build config with

   $ oc create -f 0000-buildconfig.yaml

4. After the builder pod completes successfully, deploy the driver container image as a DaemonSet.

   1. The driver container must run with the privileged security context in order to load the kernel modules on the host. The following YAML file contains the RBAC rules and the DaemonSet for running the driver container. Save this YAML as 1000-drivercontainer.yaml.

      apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: simple-kmod-driver-container
      ---
      apiVersion: rbac.authorization.k8s.io/v1
      kind: Role
      metadata:
        name: simple-kmod-driver-container
      rules:
      - apiGroups:
        - security.openshift.io
        resources:
        - securitycontextconstraints
        verbs:
        - use
        resourceNames:
        - privileged
      ---
      apiVersion: rbac.authorization.k8s.io/v1
      kind: RoleBinding
      metadata:
        name: simple-kmod-driver-container
      roleRef:
        apiGroup: rbac.authorization.k8s.io
        kind: Role
        name: simple-kmod-driver-container
      subjects:
      - kind: ServiceAccount
        name: simple-kmod-driver-container
      userNames:
      - system:serviceaccount:simple-kmod-demo:simple-kmod-driver-container
      ---
      apiVersion: apps/v1
      kind: DaemonSet
      metadata:
        name: simple-kmod-driver-container
      spec:
        selector:
          matchLabels:
            app: simple-kmod-driver-container
        template:
          metadata:
            labels:
              app: simple-kmod-driver-container
          spec:
            serviceAccount: simple-kmod-driver-container
            serviceAccountName: simple-kmod-driver-container
            containers:
            - image: image-registry.openshift-image-registry.svc:5000/simple-kmod-demo/simple-kmod-driver-container:demo
              name: simple-kmod-driver-container
              imagePullPolicy: Always
              command: ["/sbin/init"]
              lifecycle:
                preStop:
                  exec:
                    command: ["/bin/sh", "-c", "systemctl stop kmods-via-containers@simple-kmod"]
              securityContext:
                privileged: true
            nodeSelector:
              node-role.kubernetes.io/worker: ""

   2. Create the RBAC rules and daemon set:

      $ oc create -f 1000-drivercontainer.yaml

5. After the pods are running on the worker nodes, verify that the simple_kmod kernel module is loaded successfully on the host machines with lsmod.

   1. Verify that the pods are running:

      $ oc get pod -n simple-kmod-demo

      Example output

      NAME                                 READY   STATUS      RESTARTS   AGE
      simple-kmod-driver-build-1-build     0/1     Completed   0          6m
      simple-kmod-driver-container-b22fd   1/1     Running     0          40s
      simple-kmod-driver-container-jz9vn   1/1     Running     0          40s
      simple-kmod-driver-container-p45cc   1/1     Running     0          40s

   2. Execute the lsmod command in the driver container pod:

      $ oc exec -it pod/simple-kmod-driver-container-p45cc -- lsmod | grep simple

      Example output

      simple_procfs_kmod     16384  0
      simple_kmod            16384  0

Procedure

1. Install the SRO in the openshift-operators namespace:

   1. Create the following Subscription CR and save the YAML in the sro-sub.yaml file:

      Example Subscription CR

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: openshift-special-resource-operator
        namespace: openshift-operators
      spec:
        channel: "stable"
        installPlanApproval: Automatic
        name: openshift-special-resource-operator
        source: redhat-operators
        sourceNamespace: openshift-marketplace

   2. Create the subscription object by running the following command:

      $ oc create -f sro-sub.yaml

   3. Switch to the openshift-operators project:

      $ oc project openshift-operators

Procedure

1. Log in to the OpenShift Container Platform web console.

2. Install the Special Resource Operator:

   1. In the OpenShift Container Platform web console, click Operators → OperatorHub.
   2. Choose Special Resource Operator from the list of available Operators, and then click Install.
   3. On the Install Operator page, select a specific namespace on the cluster, select the namespace created in the previous section, and then click Install.

1. Navigate to the Operators → Installed Operators page.

2. Ensure that Special Resource Operator is listed in the openshift-operators project with a Status of InstallSucceeded.

   Note

   During installation, an Operator might display a Failed status. If the installation later succeeds with an InstallSucceeded message, you can ignore the Failed message.

3. If the Operator does not appear as installed, to troubleshoot further:

   1. Navigate to the Operators → Installed Operators page and inspect the Operator Subscriptions and Install Plans tabs for any failure or errors under Status.
   2. Navigate to the Workloads → Pods page and check the logs for pods in the openshift-operators project.

Procedure

1. To create a simple-kmod SpecialResource object, define an image stream and build config to build the image, and a service account, role, role binding, and daemon set to run the container. The service account, role, and role binding are required to run the daemon set with the privileged security context so that the kernel module can be loaded.

   1. Create a templates directory, and change into it:

      $ mkdir -p chart/simple-kmod-0.0.1/templates

      $ cd chart/simple-kmod-0.0.1/templates

   2. Save this YAML template for the image stream and build config in the templates directory as 0000-buildconfig.yaml:

      apiVersion: image.openshift.io/v1
      kind: ImageStream
      metadata:
        labels:
          app: {{.Values.specialresource.metadata.name}}-{{.Values.groupName.driverContainer}} 1
        name: {{.Values.specialresource.metadata.name}}-{{.Values.groupName.driverContainer}} 2
      spec: {}
      ---
      apiVersion: build.openshift.io/v1
      kind: BuildConfig
      metadata:
        labels:
          app: {{.Values.specialresource.metadata.name}}-{{.Values.groupName.driverBuild}}  3
        name: {{.Values.specialresource.metadata.name}}-{{.Values.groupName.driverBuild}} 4
        annotations:
          specialresource.openshift.io/wait: "true"
          specialresource.openshift.io/driver-container-vendor: simple-kmod
          specialresource.openshift.io/kernel-affine: "true"
      spec:
        nodeSelector:
          node-role.kubernetes.io/worker: ""
        runPolicy: "Serial"
        triggers:
          - type: "ConfigChange"
          - type: "ImageChange"
        source:
          git:
            ref: {{.Values.specialresource.spec.driverContainer.source.git.ref}}
            uri: {{.Values.specialresource.spec.driverContainer.source.git.uri}}
          type: Git
        strategy:
          dockerStrategy:
            dockerfilePath: Dockerfile.SRO
            buildArgs:
              - name: "IMAGE"
                value: {{ .Values.driverToolkitImage  }}
              {{- range $arg := .Values.buildArgs }}
              - name: {{ $arg.name }}
                value: {{ $arg.value }}
              {{- end }}
              - name: KVER
                value: {{ .Values.kernelFullVersion }}
        output:
          to:
            kind: ImageStreamTag
            name: {{.Values.specialresource.metadata.name}}-{{.Values.groupName.driverContainer}}:v{{.Values.kernelFullVersion}} 5

      1 2 3 4 5

      The templates such as {{.Values.specialresource.metadata.name}} are filled in by the SRO, based on fields in the SpecialResource CR and variables known to the Operator such as {{.Values.KernelFullVersion}}.

   3. Save the following YAML template for the RBAC resources and daemon set in the templates directory as 1000-driver-container.yaml:

      apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: {{.Values.specialresource.metadata.name}}-{{.Values.groupName.driverContainer}}
      ---
      apiVersion: rbac.authorization.k8s.io/v1
      kind: Role
      metadata:
        name: {{.Values.specialresource.metadata.name}}-{{.Values.groupName.driverContainer}}
      rules:
      - apiGroups:
        - security.openshift.io
        resources:
        - securitycontextconstraints
        verbs:
        - use
        resourceNames:
        - privileged
      ---
      apiVersion: rbac.authorization.k8s.io/v1
      kind: RoleBinding
      metadata:
        name: {{.Values.specialresource.metadata.name}}-{{.Values.groupName.driverContainer}}
      roleRef:
        apiGroup: rbac.authorization.k8s.io
        kind: Role
        name: {{.Values.specialresource.metadata.name}}-{{.Values.groupName.driverContainer}}
      subjects:
      - kind: ServiceAccount
        name: {{.Values.specialresource.metadata.name}}-{{.Values.groupName.driverContainer}}
        namespace: {{.Values.specialresource.spec.namespace}}
      ---
      apiVersion: apps/v1
      kind: DaemonSet
      metadata:
        labels:
          app: {{.Values.specialresource.metadata.name}}-{{.Values.groupName.driverContainer}}
        name: {{.Values.specialresource.metadata.name}}-{{.Values.groupName.driverContainer}}
        annotations:
          specialresource.openshift.io/wait: "true"
          specialresource.openshift.io/state: "driver-container"
          specialresource.openshift.io/driver-container-vendor: simple-kmod
          specialresource.openshift.io/kernel-affine: "true"
          specialresource.openshift.io/from-configmap: "true"
      spec:
        updateStrategy:
          type: OnDelete
        selector:
          matchLabels:
            app: {{.Values.specialresource.metadata.name}}-{{.Values.groupName.driverContainer}}
        template:
          metadata:
            labels:
              app: {{.Values.specialresource.metadata.name}}-{{.Values.groupName.driverContainer}}
          spec:
            priorityClassName: system-node-critical
            serviceAccount: {{.Values.specialresource.metadata.name}}-{{.Values.groupName.driverContainer}}
            serviceAccountName: {{.Values.specialresource.metadata.name}}-{{.Values.groupName.driverContainer}}
            containers:
            - image: image-registry.openshift-image-registry.svc:5000/{{.Values.specialresource.spec.namespace}}/{{.Values.specialresource.metadata.name}}-{{.Values.groupName.driverContainer}}:v{{.Values.kernelFullVersion}}
              name: {{.Values.specialresource.metadata.name}}-{{.Values.groupName.driverContainer}}
              imagePullPolicy: Always
              command: ["/sbin/init"]
              lifecycle:
                preStop:
                  exec:
                    command: ["/bin/sh", "-c", "systemctl stop kmods-via-containers@{{.Values.specialresource.metadata.name}}"]
              securityContext:
                privileged: true
            nodeSelector:
              node-role.kubernetes.io/worker: ""
              feature.node.kubernetes.io/kernel-version.full: "{{.Values.KernelFullVersion}}"

   4. Change into the chart/simple-kmod-0.0.1 directory:

      $ cd ..

   5. Save the following YAML for the chart as Chart.yaml in the chart/simple-kmod-0.0.1 directory:

      apiVersion: v2
      name: simple-kmod
      description: Simple kmod will deploy a simple kmod driver-container
      icon: https://avatars.githubusercontent.com/u/55542927
      type: application
      version: 0.0.1
      appVersion: 1.0.0

2. From the chart directory, create the chart using the helm package command:

   $ helm package simple-kmod-0.0.1/

   Example output

   Successfully packaged chart and saved it to: /data/<username>/git/<github_username>/special-resource-operator/yaml-for-docs/chart/simple-kmod-0.0.1/simple-kmod-0.0.1.tgz

3. Create a config map to store the chart files:

   1. Create a directory for the config map files:

      $ mkdir cm

   2. Copy the Helm chart into the cm directory:

      $ cp simple-kmod-0.0.1.tgz cm/simple-kmod-0.0.1.tgz

   3. Create an index file specifying the Helm repo that contains the Helm chart:

      $ helm repo index cm --url=cm://simple-kmod/simple-kmod-chart

   4. Create a namespace for the objects defined in the Helm chart:

      $ oc create namespace simple-kmod

   5. Create the config map object:

      $ oc create cm simple-kmod-chart --from-file=cm/index.yaml --from-file=cm/simple-kmod-0.0.1.tgz -n simple-kmod

4. Use the following SpecialResource manifest to deploy the simple-kmod object using the Helm chart that you created in the config map. Save this YAML as simple-kmod-configmap.yaml:

   apiVersion: sro.openshift.io/v1beta1
   kind: SpecialResource
   metadata:
     name: simple-kmod
   spec:
     #debug: true 1
     namespace: simple-kmod
     chart:
       name: simple-kmod
       version: 0.0.1
       repository:
         name: example
         url: cm://simple-kmod/simple-kmod-chart 2
     set:
       kind: Values
       apiVersion: sro.openshift.io/v1beta1
       kmodNames: ["simple-kmod", "simple-procfs-kmod"]
       buildArgs:
       - name: "KMODVER"
         value: "SRO"
     driverContainer:
       source:
         git:
           ref: "master"
           uri: "https://github.com/openshift-psap/kvc-simple-kmod.git"

   1

   Optional: Uncomment the #debug: true line to have the YAML files in the chart printed in full in the Operator logs and to verify that the logs are created and templated properly.

   2

   The spec.chart.repository.url field tells the SRO to look for the chart in a config map.

5. From a command line, create the SpecialResource file:

   $ oc create -f simple-kmod-configmap.yaml

1. Use oc get pods command to display the status of the build pods:

   $ oc get pods -n simple-kmod

   Example output

   NAME                                                  READY   STATUS      RESTARTS   AGE
   simple-kmod-driver-build-12813789169ac0ee-1-build     0/1     Completed   0          7m12s
   simple-kmod-driver-container-12813789169ac0ee-mjsnh   1/1     Running     0          8m2s
   simple-kmod-driver-container-12813789169ac0ee-qtkff   1/1     Running     0          8m2s

2. Use the oc logs command, along with the build pod name obtained from the oc get pods command above, to display the logs of the simple-kmod driver container image build:

   $ oc logs pod/simple-kmod-driver-build-12813789169ac0ee-1-build -n simple-kmod

3. To verify that the simple-kmod kernel modules are loaded, execute the lsmod command in one of the driver container pods that was returned from the oc get pods command above:

   $ oc exec -n simple-kmod -it pod/simple-kmod-driver-container-12813789169ac0ee-mjsnh -- lsmod | grep simple

   Example output

   simple_procfs_kmod     16384  0
   simple_kmod            16384  0

Procedure

1. Create a templates directory by running the following command:

   $ mkdir -p charts/acm-simple-kmod-0.0.1/templates

2. Change to the templates directory by running the following command:

   $ cd charts/acm-simple-kmod-0.0.1/templates

3. Create templates files for the BuildConfig, Policy, and PlacementRule resources.

   1. Save this YAML template for the image stream and build config in the templates directory as 0001-buildconfig.yaml.

      apiVersion: build.openshift.io/v1
      kind: BuildConfig
      metadata:
          labels:
              app: {{ printf "%s-%s" .Values.specialResourceModule.metadata.name .Values.kernelFullVersion | replace "." "-" | replace "_" "-" | trunc 63 }}
          name: {{ printf "%s-%s" .Values.specialResourceModule.metadata.name .Values.kernelFullVersion | replace "." "-" | replace "_" "-" | trunc 63 }}
          annotations:
      specialresource.openshift.io/wait: "true"
      spec:
          nodeSelector:
              node-role.kubernetes.io/worker: ""
          runPolicy: "Serial"
          triggers:
              - type: "ConfigChange"
              - type: "ImageChange"
          source:
          dockerfile: |
              FROM {{ .Values.driverToolkitImage  }} as builder
              WORKDIR /build/
              RUN git clone -b {{.Values.specialResourceModule.spec.set.git.ref}} {{.Values.specialResourceModule.spec.set.git.uri}}
              WORKDIR /build/simple-kmod
              RUN make all install KVER={{ .Values.kernelFullVersion }}
              FROM registry.redhat.io/ubi8/ubi-minimal
              RUN microdnf -y install kmod
              COPY --from=builder /etc/driver-toolkit-release.json /etc/
              COPY --from=builder /lib/modules/{{ .Values.kernelFullVersion }}/* /lib/modules/{{ .Values.kernelFullVersion }}/
          strategy:
              dockerStrategy:
                  dockerfilePath: Dockerfile.SRO
                  buildArgs:
                      - name: "IMAGE"
                        value: {{ .Values.driverToolkitImage  }}
                      {{- range $arg := .Values.buildArgs }}
                      - name: {{ $arg.name }}
                        value: {{ $arg.value }}
                      {{- end }}
                      - name: KVER
                        value: {{ .Values.kernelFullVersion }}
          output:
              to:
                  kind: DockerImage
                  name: {{.Values.registry}}/{{.Values.specialResourceModule.metadata.name}}-{{.Values.groupName.driverContainer}}:{{.Values.kernelFullVersion}}

   2. Save this YAML template for the ACM policy in the templates directory as 0002-policy.yaml.

      apiVersion: policy.open-cluster-management.io/v1
      kind: Policy
      metadata:
          name: policy-{{.Values.specialResourceModule.metadata.name}}-ds
          annotations:
              policy.open-cluster-management.io/categories: CM Configuration Management
              policy.open-cluster-management.io/controls: CM-2 Baseline Configuration
              policy.open-cluster-management.io/standards: NIST-CSF
      spec:
          remediationAction: enforce
          disabled: false
          policy-templates:
              - objectDefinition:
                  apiVersion: policy.open-cluster-management.io/v1
                  kind: ConfigurationPolicy
                  metadata:
                      name: config-{{.Values.specialResourceModule.metadata.name}}-ds
                  spec:
                      remediationAction: enforce
                      severity: low
                      namespaceselector:
                          exclude:
                              - kube-*
                          include:
                              - '*'
                      object-templates:
                          - complianceType: musthave
                            objectDefinition:
                              apiVersion: v1
                              kind: Namespace
                              metadata:
                                  name: {{.Values.specialResourceModule.spec.namespace}}
                          - complianceType: mustonlyhave
                            objectDefinition:
                              apiVersion: v1
                              kind: ServiceAccount
                              metadata:
                                  name: {{.Values.specialResourceModule.metadata.name}}
                                  namespace: {{.Values.specialResourceModule.spec.namespace}}
                          - complianceType: mustonlyhave
                            objectDefinition:
                              apiVersion: rbac.authorization.k8s.io/v1
                              kind: Role
                              metadata:
                                  name: {{.Values.specialResourceModule.metadata.name}}
                                  namespace: {{.Values.specialResourceModule.spec.namespace}}
                              rules:
                              - apiGroups:
                                  - security.openshift.io
                                resources:
                                  - securitycontextconstraints
                                verbs:
                                  - use
                                resourceNames:
                                  - privileged
                          - complianceType: mustonlyhave
                            objectDefinition:
                              apiVersion: rbac.authorization.k8s.io/v1
                              kind: RoleBinding
                              metadata:
                                  name: {{.Values.specialResourceModule.metadata.name}}
                                  namespace: {{.Values.specialResourceModule.spec.namespace}}
                              roleRef:
                                  apiGroup: rbac.authorization.k8s.io
                                  kind: Role
                                  name: {{.Values.specialResourceModule.metadata.name}}
                              subjects:
                              - kind: ServiceAccount
                                name: {{.Values.specialResourceModule.metadata.name}}
                                namespace: {{.Values.specialResourceModule.spec.namespace}}
                          - complianceType: musthave
                            objectDefinition:
                              apiVersion: apps/v1
                              kind: DaemonSet
                              metadata:
                                  labels:
                                      app: {{ printf "%s-%s" .Values.specialResourceModule.metadata.name .Values.kernelFullVersion | replace "." "-" | replace "_" "-" | trunc 63 }}
                                  name: {{ printf "%s-%s" .Values.specialResourceModule.metadata.name .Values.kernelFullVersion | replace "." "-" | replace "_" "-" | trunc 63 }}
                                  namespace: {{.Values.specialResourceModule.spec.namespace}}
                              spec:
                                  updateStrategy:
                                      type: OnDelete
                                  selector:
                                      matchLabels:
                                          app: {{ printf "%s-%s" .Values.specialResourceModule.metadata.name .Values.kernelFullVersion | replace "." "-" | replace "_" "-" | trunc 63 }}
                                  template:
                                      metadata:
                                          labels:
                                              app: {{ printf "%s-%s" .Values.specialResourceModule.metadata.name .Values.kernelFullVersion | replace "." "-" | replace "_" "-" | trunc 63 }}
                                      spec:
                                          priorityClassName: system-node-critical
                                          serviceAccount: {{.Values.specialResourceModule.metadata.name}}
                                          serviceAccountName: {{.Values.specialResourceModule.metadata.name}}
                                          containers:
                                          - image: {{.Values.registry}}/{{.Values.specialResourceModule.metadata.name}}-{{.Values.groupName.driverContainer}}:{{.Values.kernelFullVersion}}
                                            name: {{.Values.specialResourceModule.metadata.name}}
                                            imagePullPolicy: Always
                                            command: [sleep, infinity]
                                            lifecycle:
                                              preStop:
                                                  exec:
                                                      command: ["modprobe", "-r", "-a" , "simple-kmod", "simple-procfs-kmod"]
                                            securityContext:
                                              privileged: true

   3. Save this YAML template for the placement of policies in the templates directory as 0003-policy.yaml.

      apiVersion: apps.open-cluster-management.io/v1
      kind: PlacementRule
      metadata:
          name: {{.Values.specialResourceModule.metadata.name}}-placement
      spec:
          clusterConditions:
          - status: "True"
            type: ManagedClusterConditionAvailable
          clusterSelector:
            matchExpressions:
            - key: name
              operator: NotIn
              values:
              - local-cluster
      ---
      apiVersion: policy.open-cluster-management.io/v1
      kind: PlacementBinding
      metadata:
          name: {{.Values.specialResourceModule.metadata.name}}-binding
      placementRef:
          apiGroup: apps.open-cluster-management.io
          kind: PlacementRule
          name: {{.Values.specialResourceModule.metadata.name}}-placement
      subjects:
          - apiGroup: policy.open-cluster-management.io
            kind: Policy
            name: policy-{{.Values.specialResourceModule.metadata.name}}-ds

   4. Change into the charts/acm-simple-kmod-0.0.1 directory by running the following command:

      cd ..

   5. Save the following YAML template for the chart as Chart.yaml in the charts/acm-simple-kmod-0.0.1 directory:

      apiVersion: v2
      name: acm-simple-kmod
      description: Build ACM enabled simple-kmod driver with SpecialResourceOperator
      icon: https://avatars.githubusercontent.com/u/55542927
      type: application
      version: 0.0.1
      appVersion: 1.6.4

4. From the charts directory, create the chart using the command:

   $ helm package acm-simple-kmod-0.0.1/

   Example output

   Successfully packaged chart and saved it to: <directory>/charts/acm-simple-kmod-0.0.1.tgz

5. Create a config map to store the chart files.

   1. Create a directory for the config map files by running the following command:

      $ mkdir cm

   2. Copy the Helm chart into the cm directory by running the following command:

      $ cp acm-simple-kmod-0.0.1.tgz cm/acm-simple-kmod-0.0.1.tgz

   3. Create an index file specifying the Helm repository that contains the Helm chart by running the following command:

      $ helm repo index cm --url=cm://acm-simple-kmod/acm-simple-kmod-chart

   4. Create a namespace for the objects defined in the Helm chart by running the following command:

      $ oc create namespace acm-simple-kmod

   5. Create the config map object by running the following command:

      $ oc create cm acm-simple-kmod-chart --from-file=cm/index.yaml --from-file=cm/acm-simple-kmod-0.0.1.tgz -n acm-simple-kmod

6. Use the following SpecialResourceModule manifest to deploy the simple-kmod object using the Helm chart that you created in the config map. Save this YAML file as acm-simple-kmod.yaml:

   apiVersion: sro.openshift.io/v1beta1
   kind: SpecialResourceModule
   metadata:
       name: acm-simple-kmod
   spec:
       namespace: acm-simple-kmod
       chart:
           name: acm-simple-kmod
           version: 0.0.1
           repository:
               name: acm-simple-kmod
               url: cm://acm-simple-kmod/acm-simple-kmod-chart
       set:
           kind: Values
           apiVersion: sro.openshift.io/v1beta1
           buildArgs:
               - name: "KMODVER"
                 value: "SRO"
           registry: <your_registry>  1
           git:
               ref: master
               uri: https://github.com/openshift-psap/kvc-simple-kmod.git
       watch:
               - path: "$.metadata.labels.openshiftVersion"
                 apiVersion: cluster.open-cluster-management.io/v1
                 kind: ManagedCluster
                 name: spoke1

   1

   Specify the URL for a registry that you have configured.

7. Create the special resource module by running the following command:

   $ oc apply -f charts/examples/acm-simple-kmod.yaml

Verification

1. Check the status of the build pods by running the following command:

   $ KUBECONFIG=~/hub/auth/kubeconfig oc get pod -n acm-simple-kmod

   Example output

   NAME                                                   READY   STATUS      RESTARTS   AGE
   acm-simple-kmod-4-18-0-305-34-2-el8-4-x86-64-1-build   0/1     Completed   0          42m

2. Check that the policies have been created by running the following command:

   $ KUBECONFIG=~/hub/auth/kubeconfig oc get placementrules,placementbindings,policies -n acm-simple-kmod

   Example output

   NAME                                                                      AGE   REPLICAS
   placementrule.apps.open-cluster-management.io/acm-simple-kmod-placement   40m
   
   NAME                                                                         AGE
   placementbinding.policy.open-cluster-management.io/acm-simple-kmod-binding   40m
   
   NAME                                                                 REMEDIATION ACTION   COMPLIANCE STATE   AGE
   policy.policy.open-cluster-management.io/policy-acm-simple-kmod-ds   enforce              Compliant          40m

3. Check that the resources have been reconciled by running the following command:

   $ KUBECONFIG=~/hub/auth/kubeconfig oc get specialresourcemodule acm-simple-kmod -o json | jq -r '.status'

   Example output

   {
     "versions": {
       "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a3330ef5a178435721ff4efdde762261a9c55212e9b4534385e04037693fbe4": {
         "complete": true
       }
     }
   }

4. Check that the resources are running in the spoke by running the following command:

   $ KUBECONFIG=~/spoke1/kubeconfig oc get ds,pod -n acm-simple-kmod

   Example output

   AME                                                          DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
   daemonset.apps/acm-simple-kmod-4-18-0-305-45-1-el8-4-x86-64   3         3         3       3            3           <none>          26m
   
   NAME                                                     READY   STATUS    RESTARTS   AGE
   pod/acm-simple-kmod-4-18-0-305-45-1-el8-4-x86-64-brw78   1/1     Running   0          26m
   pod/acm-simple-kmod-4-18-0-305-45-1-el8-4-x86-64-fqh5h   1/1     Running   0          26m
   pod/acm-simple-kmod-4-18-0-305-45-1-el8-4-x86-64-m9sfd   1/1     Running   0          26m

Procedure

1. Create a namespace for the NFD Operator.

   1. Create the following Namespace custom resource (CR) that defines the openshift-nfd namespace, and then save the YAML in the nfd-namespace.yaml file:

      apiVersion: v1
      kind: Namespace
      metadata:
        name: openshift-nfd

   2. Create the namespace by running the following command:

      $ oc create -f nfd-namespace.yaml

2. Install the NFD Operator in the namespace you created in the previous step by creating the following objects:

   1. Create the following OperatorGroup CR and save the YAML in the nfd-operatorgroup.yaml file:

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        generateName: openshift-nfd-
        name: openshift-nfd
        namespace: openshift-nfd
      spec:
        targetNamespaces:
        - openshift-nfd

   2. Create the OperatorGroup CR by running the following command:

      $ oc create -f nfd-operatorgroup.yaml

   3. Create the following Subscription CR and save the YAML in the nfd-sub.yaml file:

      Example Subscription

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: nfd
        namespace: openshift-nfd
      spec:
        channel: "stable"
        installPlanApproval: Automatic
        name: nfd
        source: redhat-operators
        sourceNamespace: openshift-marketplace

   4. Create the subscription object by running the following command:

      $ oc create -f nfd-sub.yaml

   5. Change to the openshift-nfd project:

      $ oc project openshift-nfd

Procedure

1. In the OpenShift Container Platform web console, click Operators → OperatorHub.
2. Choose Node Feature Discovery from the list of available Operators, and then click Install.
3. On the Install Operator page, select A specific namespace on the cluster, and then click Install. You do not need to create a namespace because it is created for you.

1. Navigate to the Operators → Installed Operators page.

2. Ensure that Node Feature Discovery is listed in the openshift-nfd project with a Status of InstallSucceeded.

   Note

   During installation an Operator might display a Failed status. If the installation later succeeds with an InstallSucceeded message, you can ignore the Failed message.

1. Navigate to the Operators → Installed Operators page and inspect the Operator Subscriptions and Install Plans tabs for any failure or errors under Status.
2. Navigate to the Workloads → Pods page and check the logs for pods in the openshift-nfd project.

Procedure

1. Create the following NodeFeatureDiscovery Custom Resource (CR), and then save the YAML in the NodeFeatureDiscovery.yaml file:

   apiVersion: nfd.openshift.io/v1
   kind: NodeFeatureDiscovery
   metadata:
     name: nfd-instance
     namespace: openshift-nfd
   spec:
     instance: "" # instance is empty by default
     topologyupdater: false # False by default
     operand:
       image: registry.redhat.io/openshift4/ose-node-feature-discovery:v4.11
       imagePullPolicy: Always
     workerConfig:
       configData: |
         core:
         #  labelWhiteList:
         #  noPublish: false
           sleepInterval: 60s
         #  sources: [all]
         #  klog:
         #    addDirHeader: false
         #    alsologtostderr: false
         #    logBacktraceAt:
         #    logtostderr: true
         #    skipHeaders: false
         #    stderrthreshold: 2
         #    v: 0
         #    vmodule:
         ##   NOTE: the following options are not dynamically run-time configurable
         ##         and require a nfd-worker restart to take effect after being changed
         #    logDir:
         #    logFile:
         #    logFileMaxSize: 1800
         #    skipLogHeaders: false
         sources:
           cpu:
             cpuid:
         #     NOTE: whitelist has priority over blacklist
               attributeBlacklist:
                 - "BMI1"
                 - "BMI2"
                 - "CLMUL"
                 - "CMOV"
                 - "CX16"
                 - "ERMS"
                 - "F16C"
                 - "HTT"
                 - "LZCNT"
                 - "MMX"
                 - "MMXEXT"
                 - "NX"
                 - "POPCNT"
                 - "RDRAND"
                 - "RDSEED"
                 - "RDTSCP"
                 - "SGX"
                 - "SSE"
                 - "SSE2"
                 - "SSE3"
                 - "SSE4.1"
                 - "SSE4.2"
                 - "SSSE3"
               attributeWhitelist:
           kernel:
             kconfigFile: "/path/to/kconfig"
             configOpts:
               - "NO_HZ"
               - "X86"
               - "DMI"
           pci:
             deviceClassWhitelist:
               - "0200"
               - "03"
               - "12"
             deviceLabelFields:
               - "class"
     customConfig:
       configData: |
             - name: "more.kernel.features"
               matchOn:
               - loadedKMod: ["example_kmod3"]

1. Create the NodeFeatureDiscovery CR instance by running the following command:

   $ oc create -f NodeFeatureDiscovery.yaml

Procedure

1. Navigate to the Operators → Installed Operators page.
2. Find Node Feature Discovery and see a box under Provided APIs.
3. Click Create instance.
4. Edit the values of the NodeFeatureDiscovery CR.
5. Click Create.