Specialized hardware and driver enablement

Procedure

1. Use the

   oc adm

   command to extract the image URL of the

   driver-toolkit

   corresponding to a certain release:

   • For an x86 image, the command is as follows:

     $ oc adm release info quay.io/openshift-release-dev/ocp-release:4.19.z-x86_64 --image-for=driver-toolkit

   • For an ARM image, the command is as follows:

     $ oc adm release info quay.io/openshift-release-dev/ocp-release:4.19.z-aarch64 --image-for=driver-toolkit

   Example output

   quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b53883ca2bac5925857148c4a1abc300ced96c222498e3bc134fe7ce3a1dd404

2. Obtain this image using a valid pull secret, such as the pull secret required to install OpenShift Container Platform:

   $ podman pull --authfile=path/to/pullsecret.json quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:<SHA>

1. The YAML defines an

   ImageStream

   for storing the

   simple-kmod

   driver container image, and a

   BuildConfig

   for building the container. Save this YAML as

   0000-buildconfig.yaml.template

   .

   apiVersion: image.openshift.io/v1
   kind: ImageStream
   metadata:
     labels:
       app: simple-kmod-driver-container
     name: simple-kmod-driver-container
     namespace: simple-kmod-demo
   spec: {}
   ---
   apiVersion: build.openshift.io/v1
   kind: BuildConfig
   metadata:
     labels:
       app: simple-kmod-driver-build
     name: simple-kmod-driver-build
     namespace: simple-kmod-demo
   spec:
     nodeSelector:
       node-role.kubernetes.io/worker: ""
     runPolicy: "Serial"
     triggers:
       - type: "ConfigChange"
       - type: "ImageChange"
     source:
       dockerfile: |
         ARG DTK
         FROM ${DTK} as builder
   
         ARG KVER
   
         WORKDIR /build/
   
         RUN git clone https://github.com/openshift-psap/simple-kmod.git
   
         WORKDIR /build/simple-kmod
   
         RUN make all install KVER=${KVER}
   
         FROM registry.redhat.io/ubi8/ubi-minimal
   
         ARG KVER
   
         # Required for installing `modprobe`
         RUN microdnf install kmod
   
         COPY --from=builder /lib/modules/${KVER}/simple-kmod.ko /lib/modules/${KVER}/
         COPY --from=builder /lib/modules/${KVER}/simple-procfs-kmod.ko /lib/modules/${KVER}/
         RUN depmod ${KVER}
     strategy:
       dockerStrategy:
         buildArgs:
           - name: KMODVER
             value: DEMO
             # $ oc adm release info quay.io/openshift-release-dev/ocp-release:<cluster version>-x86_64 --image-for=driver-toolkit
           - name: DTK
             value: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:34864ccd2f4b6e385705a730864c04a40908e57acede44457a783d739e377cae
           - name: KVER
             value: 4.18.0-372.26.1.el8_6.x86_64
     output:
       to:
         kind: ImageStreamTag
         name: simple-kmod-driver-container:demo

2. Substitute the correct driver toolkit image for the OpenShift Container Platform version you are running in place of “DRIVER_TOOLKIT_IMAGE” with the following commands.

   $ OCP_VERSION=$(oc get clusterversion/version -ojsonpath={.status.desired.version})

   $ DRIVER_TOOLKIT_IMAGE=$(oc adm release info $OCP_VERSION --image-for=driver-toolkit)

   $ sed "s#DRIVER_TOOLKIT_IMAGE#${DRIVER_TOOLKIT_IMAGE}#" 0000-buildconfig.yaml.template > 0000-buildconfig.yaml

3. Create the image stream and build config with

   $ oc create -f 0000-buildconfig.yaml

4. After the builder pod completes successfully, deploy the driver container image as a

   DaemonSet

   .

   1. The driver container must run with the privileged security context in order to load the kernel modules on the host. The following YAML file contains the RBAC rules and the

      DaemonSet

      for running the driver container. Save this YAML as

      1000-drivercontainer.yaml

      .

      apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: simple-kmod-driver-container
      ---
      apiVersion: rbac.authorization.k8s.io/v1
      kind: Role
      metadata:
        name: simple-kmod-driver-container
      rules:
      - apiGroups:
        - security.openshift.io
        resources:
        - securitycontextconstraints
        verbs:
        - use
        resourceNames:
        - privileged
      ---
      apiVersion: rbac.authorization.k8s.io/v1
      kind: RoleBinding
      metadata:
        name: simple-kmod-driver-container
      roleRef:
        apiGroup: rbac.authorization.k8s.io
        kind: Role
        name: simple-kmod-driver-container
      subjects:
      - kind: ServiceAccount
        name: simple-kmod-driver-container
      userNames:
      - system:serviceaccount:simple-kmod-demo:simple-kmod-driver-container
      ---
      apiVersion: apps/v1
      kind: DaemonSet
      metadata:
        name: simple-kmod-driver-container
      spec:
        selector:
          matchLabels:
            app: simple-kmod-driver-container
        template:
          metadata:
            labels:
              app: simple-kmod-driver-container
          spec:
            serviceAccount: simple-kmod-driver-container
            serviceAccountName: simple-kmod-driver-container
            containers:
            - image: image-registry.openshift-image-registry.svc:5000/simple-kmod-demo/simple-kmod-driver-container:demo
              name: simple-kmod-driver-container
              imagePullPolicy: Always
              command: [sleep, infinity]
              lifecycle:
                postStart:
                  exec:
                    command: ["modprobe", "-v", "-a" , "simple-kmod", "simple-procfs-kmod"]
                preStop:
                  exec:
                    command: ["modprobe", "-r", "-a" , "simple-kmod", "simple-procfs-kmod"]
              securityContext:
                privileged: true
            nodeSelector:
              node-role.kubernetes.io/worker: ""

   2. Create the RBAC rules and daemon set:

      $ oc create -f 1000-drivercontainer.yaml

5. After the pods are running on the worker nodes, verify that the

   simple_kmod

   kernel module is loaded successfully on the host machines with

   lsmod

   .

   1. Verify that the pods are running:

      $ oc get pod -n simple-kmod-demo

      Example output

      NAME                                 READY   STATUS      RESTARTS   AGE
      simple-kmod-driver-build-1-build     0/1     Completed   0          6m
      simple-kmod-driver-container-b22fd   1/1     Running     0          40s
      simple-kmod-driver-container-jz9vn   1/1     Running     0          40s
      simple-kmod-driver-container-p45cc   1/1     Running     0          40s

   2. Execute the

      lsmod

      command in the driver container pod:

      $ oc exec -it pod/simple-kmod-driver-container-p45cc -- lsmod | grep simple

      Example output

      simple_procfs_kmod     16384  0
      simple_kmod            16384  0

Procedure

1. Create a namespace for the NFD Operator.

   1. Create the following

      Namespace

      custom resource (CR) that defines the

      openshift-nfd

      namespace, and then save the YAML in the

      nfd-namespace.yaml

      file. Set

      cluster-monitoring

      to

      "true"

      .

      apiVersion: v1
      kind: Namespace
      metadata:
        name: openshift-nfd
        labels:
          name: openshift-nfd
          openshift.io/cluster-monitoring: "true"

   2. Create the namespace by running the following command:

      $ oc create -f nfd-namespace.yaml

2. Install the NFD Operator in the namespace you created in the previous step by creating the following objects:

   1. Create the following

      OperatorGroup

      CR and save the YAML in the

      nfd-operatorgroup.yaml

      file:

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        generateName: openshift-nfd-
        name: openshift-nfd
        namespace: openshift-nfd
      spec:
        targetNamespaces:
        - openshift-nfd

   2. Create the

      OperatorGroup

      CR by running the following command:

      $ oc create -f nfd-operatorgroup.yaml

   3. Create the following

      Subscription

      CR and save the YAML in the

      nfd-sub.yaml

      file:

      Example Subscription

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: nfd
        namespace: openshift-nfd
      spec:
        channel: "stable"
        installPlanApproval: Automatic
        name: nfd
        source: redhat-operators
        sourceNamespace: openshift-marketplace

   4. Create the subscription object by running the following command:

      $ oc create -f nfd-sub.yaml

   5. Change to the

      openshift-nfd

      project:

      $ oc project openshift-nfd

Procedure

1. In the OpenShift Container Platform web console, click Operators → OperatorHub.
2. Choose Node Feature Discovery from the list of available Operators, and then click Install.
3. On the Install Operator page, select A specific namespace on the cluster, and then click Install. You do not need to create a namespace because it is created for you.

1. Navigate to the Operators → Installed Operators page.

2. Ensure that Node Feature Discovery is listed in the openshift-nfd project with a Status of InstallSucceeded.

   Note

   During installation an Operator might display a Failed status. If the installation later succeeds with an InstallSucceeded message, you can ignore the Failed message.

1. Navigate to the Operators → Installed Operators page and inspect the Operator Subscriptions and Install Plans tabs for any failure or errors under Status.
2. Navigate to the Workloads → Pods page and check the logs for pods in the

   openshift-nfd

   project.

Procedure

1. Create a

   NodeFeatureDiscovery

   CR:

   Example NodeFeatureDiscovery CR

   apiVersion: nfd.openshift.io/v1
   kind: NodeFeatureDiscovery
   metadata:
     name: nfd-instance
     namespace: openshift-nfd
   spec:
     instance: "" # instance is empty by default
     topologyupdater: false # False by default
     operand:
       image: registry.redhat.io/openshift4/ose-node-feature-discovery-rhel9:v4.19 

   1

   
       imagePullPolicy: Always
     workerConfig:
       configData: |
         core:
         #  labelWhiteList:
         #  noPublish: false
           sleepInterval: 60s
         #  sources: [all]
         #  klog:
         #    addDirHeader: false
         #    alsologtostderr: false
         #    logBacktraceAt:
         #    logtostderr: true
         #    skipHeaders: false
         #    stderrthreshold: 2
         #    v: 0
         #    vmodule:
         ##   NOTE: the following options are not dynamically run-time configurable
         ##         and require a nfd-worker restart to take effect after being changed
         #    logDir:
         #    logFile:
         #    logFileMaxSize: 1800
         #    skipLogHeaders: false
         sources:
           cpu:
             cpuid:
         #     NOTE: whitelist has priority over blacklist
               attributeBlacklist:
                 - "BMI1"
                 - "BMI2"
                 - "CLMUL"
                 - "CMOV"
                 - "CX16"
                 - "ERMS"
                 - "F16C"
                 - "HTT"
                 - "LZCNT"
                 - "MMX"
                 - "MMXEXT"
                 - "NX"
                 - "POPCNT"
                 - "RDRAND"
                 - "RDSEED"
                 - "RDTSCP"
                 - "SGX"
                 - "SSE"
                 - "SSE2"
                 - "SSE3"
                 - "SSE4.1"
                 - "SSE4.2"
                 - "SSSE3"
               attributeWhitelist:
           kernel:
             kconfigFile: "/path/to/kconfig"
             configOpts:
               - "NO_HZ"
               - "X86"
               - "DMI"
           pci:
             deviceClassWhitelist:
               - "0200"
               - "03"
               - "12"
             deviceLabelFields:
               - "class"
     customConfig:
       configData: |
             - name: "more.kernel.features"
               matchOn:
               - loadedKMod: ["example_kmod3"]

   1

   The operand.image field is mandatory.

2. Create the

   NodeFeatureDiscovery

   CR by running the following command:

   $ oc apply -f <filename>

Verification

1. Check that the

   NodeFeatureDiscovery

   CR was created by running the following command:

   $ oc get pods

   Example output

   NAME                                      READY   STATUS    RESTARTS   AGE
   nfd-controller-manager-7f86ccfb58-vgr4x   2/2     Running   0          11m
   nfd-master-hcn64                          1/1     Running   0          60s
   nfd-master-lnnxx                          1/1     Running   0          60s
   nfd-master-mp6hr                          1/1     Running   0          60s
   nfd-worker-vgcz9                          1/1     Running   0          60s
   nfd-worker-xqbws                          1/1     Running   0          60s

   A successful deployment shows a

   Running

   status.

Procedure

1. Determine the digest of the registry image:

   1. Run the following command:

      $ skopeo inspect docker://registry.redhat.io/openshift4/ose-node-feature-discovery:<openshift_version>

      Example command

      $ skopeo inspect docker://registry.redhat.io/openshift4/ose-node-feature-discovery:v4.12

   2. Inspect the output to identify the image digest:

      Example output

      {
        ...
        "Digest": "sha256:1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef",
        ...
      }

2. Use the

   skopeo

   CLI tool to copy the image from

   registry.redhat.io

   to your mirror registry, by running the following command:

   skopeo copy docker://registry.redhat.io/openshift4/ose-node-feature-discovery@<image_digest> docker://<mirror_registry>/openshift4/ose-node-feature-discovery@<image_digest>

   Example command

   skopeo copy docker://registry.redhat.io/openshift4/ose-node-feature-discovery@sha256:1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef docker://<your-mirror-registry>/openshift4/ose-node-feature-discovery@sha256:1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef

3. Create a

   NodeFeatureDiscovery

   CR:

   Example NodeFeatureDiscovery CR

   apiVersion: nfd.openshift.io/v1
   kind: NodeFeatureDiscovery
   metadata:
     name: nfd-instance
   spec:
     operand:
       image: <mirror_registry>/openshift4/ose-node-feature-discovery@<image_digest> 

   1

   
       imagePullPolicy: Always
     workerConfig:
       configData: |
         core:
         #  labelWhiteList:
         #  noPublish: false
           sleepInterval: 60s
         #  sources: [all]
         #  klog:
         #    addDirHeader: false
         #    alsologtostderr: false
         #    logBacktraceAt:
         #    logtostderr: true
         #    skipHeaders: false
         #    stderrthreshold: 2
         #    v: 0
         #    vmodule:
         ##   NOTE: the following options are not dynamically run-time configurable
         ##         and require a nfd-worker restart to take effect after being changed
         #    logDir:
         #    logFile:
         #    logFileMaxSize: 1800
         #    skipLogHeaders: false
         sources:
           cpu:
             cpuid:
         #     NOTE: whitelist has priority over blacklist
               attributeBlacklist:
                 - "BMI1"
                 - "BMI2"
                 - "CLMUL"
                 - "CMOV"
                 - "CX16"
                 - "ERMS"
                 - "F16C"
                 - "HTT"
                 - "LZCNT"
                 - "MMX"
                 - "MMXEXT"
                 - "NX"
                 - "POPCNT"
                 - "RDRAND"
                 - "RDSEED"
                 - "RDTSCP"
                 - "SGX"
                 - "SSE"
                 - "SSE2"
                 - "SSE3"
                 - "SSE4.1"
                 - "SSE4.2"
                 - "SSSE3"
               attributeWhitelist:
           kernel:
             kconfigFile: "/path/to/kconfig"
             configOpts:
               - "NO_HZ"
               - "X86"
               - "DMI"
           pci:
             deviceClassWhitelist:
               - "0200"
               - "03"
               - "12"
             deviceLabelFields:
               - "class"
     customConfig:
       configData: |
             - name: "more.kernel.features"
               matchOn:
               - loadedKMod: ["example_kmod3"]

   1

   The operand.image field is mandatory.

4. Create the

   NodeFeatureDiscovery

   CR by running the following command:

   $ oc apply -f <filename>

Verification

1. Check the status of the

   NodeFeatureDiscovery

   CR by running the following command:

   $ oc get nodefeaturediscovery nfd-instance -o yaml

2. Check that the pods are running without

   ImagePullBackOff

   errors by running the following command:

   $ oc get pods -n <nfd_namespace>

Procedure

1. Navigate to the Operators → Installed Operators page.
2. In the Node Feature Discovery section, under Provided APIs, click Create instance.
3. Edit the values of the

   NodeFeatureDiscovery

   CR.
4. Click Create.

Procedure

1. Log in to the OpenShift Container Platform web console.

2. Install the Kernel Module Management Operator:

   1. In the OpenShift Container Platform web console, click Operators → OperatorHub.
   2. Select Kernel Module Management Operator from the list of available Operators, and then click Install.
   3. From the Installed Namespace list, select the

      openshift-kmm

      namespace.
   4. Click Install.

1. Navigate to the Operators → Installed Operators page.

2. Ensure that Kernel Module Management Operator is listed in the openshift-kmm project with a Status of InstallSucceeded.

   Note

   During installation, an Operator might display a Failed status. If the installation later succeeds with an InstallSucceeded message, you can ignore the Failed message.

Troubleshooting

1. To troubleshoot issues with Operator installation:

   1. Navigate to the Operators → Installed Operators page and inspect the Operator Subscriptions and Install Plans tabs for any failure or errors under Status.
   2. Navigate to the Workloads → Pods page and check the logs for pods in the

      openshift-kmm

      project.

Procedure

1. Install KMM in the

   openshift-kmm

   namespace:

   1. Create the following

      Namespace

      CR and save the YAML file, for example,

      kmm-namespace.yaml

      :

      apiVersion: v1
      kind: Namespace
      metadata:
        name: openshift-kmm

   2. Create the following

      OperatorGroup

      CR and save the YAML file, for example,

      kmm-op-group.yaml

      :

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: kernel-module-management
        namespace: openshift-kmm

   3. Create the following

      Subscription

      CR and save the YAML file, for example,

      kmm-sub.yaml

      :

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: kernel-module-management
        namespace: openshift-kmm
      spec:
        channel: release-1.0
        installPlanApproval: Automatic
        name: kernel-module-management
        source: redhat-operators
        sourceNamespace: openshift-marketplace
        startingCSV: kernel-module-management.v1.0.0

   4. Create the subscription object by running the following command:

      $ oc create -f kmm-sub.yaml

Procedure

1. Install KMM in the

   openshift-kmm

   namespace:

   1. Create the following

      Namespace

      CR and save the YAML file, for example,

      kmm-namespace.yaml

      file:

      apiVersion: v1
      kind: Namespace
      metadata:
        name: openshift-kmm

   2. Create the following

      SecurityContextConstraint

      object and save the YAML file, for example,

      kmm-security-constraint.yaml

      :

      allowHostDirVolumePlugin: false
      allowHostIPC: false
      allowHostNetwork: false
      allowHostPID: false
      allowHostPorts: false
      allowPrivilegeEscalation: false
      allowPrivilegedContainer: false
      allowedCapabilities:
        - NET_BIND_SERVICE
      apiVersion: security.openshift.io/v1
      defaultAddCapabilities: null
      fsGroup:
        type: MustRunAs
      groups: []
      kind: SecurityContextConstraints
      metadata:
        name: restricted-v2
      priority: null
      readOnlyRootFilesystem: false
      requiredDropCapabilities:
        - ALL
      runAsUser:
        type: MustRunAsRange
      seLinuxContext:
        type: MustRunAs
      seccompProfiles:
        - runtime/default
      supplementalGroups:
        type: RunAsAny
      users: []
      volumes:
        - configMap
        - downwardAPI
        - emptyDir
        - persistentVolumeClaim
        - projected
        - secret

   3. Bind the

      SecurityContextConstraint

      object to the Operator’s

      ServiceAccount

      by running the following commands:

      $ oc apply -f kmm-security-constraint.yaml

      $ oc adm policy add-scc-to-user kmm-security-constraint -z kmm-operator-controller -n openshift-kmm

   4. Create the following

      OperatorGroup

      CR and save the YAML file, for example,

      kmm-op-group.yaml

      :

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: kernel-module-management
        namespace: openshift-kmm

   5. Create the following

      Subscription

      CR and save the YAML file, for example,

      kmm-sub.yaml

      :

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: kernel-module-management
        namespace: openshift-kmm
      spec:
        channel: release-1.0
        installPlanApproval: Automatic
        name: kernel-module-management
        source: redhat-operators
        sourceNamespace: openshift-marketplace
        startingCSV: kernel-module-management.v1.0.0

   6. Create the subscription object by running the following command:

      $ oc create -f kmm-sub.yaml

1. List all nodes matching

   .spec.selector

   .
2. Build a set of all kernel versions running on those nodes.

3. For each kernel version:

   1. Go through

      .spec.moduleLoader.container.kernelMappings

      and find the appropriate container image name. If the kernel mapping has

      build

      or

      sign

      defined and the container image does not already exist, run the build, the signing pod, or both, as needed.
   2. Create a worker pod to pull the container image determined in the previous step and run

      modprobe

      .
   3. If

      .spec.devicePlugin

      is defined, create a device plugin daemon set using the configuration specified under

      .spec.devicePlugin.container

      .

4. Run

   garbage-collect

   on:
   1. Obsolete device plugin

      DaemonSets

      that do not target any node.
   2. Successful build pods.
   3. Successful signing pods.

1. Build the kernel modules.
2. Copy the

   .ko

   files into a smaller end-user image such as ubi-minimal.

3. To leverage DTK in your in-cluster build, use the

   DTK_AUTO

   build argument. The value is automatically set by KMM when creating the

   Build

   resource. See the following example.

   ARG DTK_AUTO
   FROM ${DTK_AUTO} as builder
   ARG KERNEL_FULL_VERSION
   WORKDIR /usr/src
   RUN ["git", "clone", "https://github.com/rh-ecosystem-edge/kernel-module-management.git"]
   WORKDIR /usr/src/kernel-module-management/ci/kmm-kmod
   RUN KERNEL_SRC_DIR=/lib/modules/${KERNEL_FULL_VERSION}/build make all
   FROM ubi9/ubi-minimal
   ARG KERNEL_FULL_VERSION
   RUN microdnf install kmod
   COPY --from=builder /usr/src/kernel-module-management/ci/kmm-kmod/kmm_ci_a.ko /opt/lib/modules/${KERNEL_FULL_VERSION}/
   COPY --from=builder /usr/src/kernel-module-management/ci/kmm-kmod/kmm_ci_b.ko /opt/lib/modules/${KERNEL_FULL_VERSION}/
   RUN depmod -b /opt ${KERNEL_FULL_VERSION}

Procedure

1. Check to ensure the public key secret is set correctly:

   $ oc get secret -o yaml <certificate secret name> | awk '/cert/{print $2; exit}' | base64 -d  | openssl x509 -inform der -text

   This should display a certificate with a Serial Number, Issuer, Subject, and more.

2. Check to ensure the private key secret is set correctly:

   $ oc get secret -o yaml <private key secret name> | awk '/key/{print $2; exit}' | base64 -d

   This should display the key enclosed in the

   -----BEGIN PRIVATE KEY-----

   and

   -----END PRIVATE KEY-----

   lines.

Procedure

1. Apply a

   MachineConfig

   resource to the existing running cluster. In order to identify the necessary nodes that need to be updated, you must create an appropriate

   MachineConfigPool

   resource.
2. MCO applies the reboots node by node. On any rebooted node, two new

   systemd

   services are deployed:

   pull

   service and

   load

   service.
3. The

   load

   service is configured to run prior to the

   NetworkConfiguration

   service. The service tries to pull a predefined kernel module image and then, using that image, to unload an in-tree module and load an OOT kernel module.
4. The

   pull

   service is configured to run after NetworkManager service. The service checks if the preconfigured kernel module image is located on the node’s filesystem. If it is, the service exists normally, and the server continues with the boot process. If not, it pulls the image onto the node and reboots the node afterwards.