Specialized hardware and driver enablement

Procedure

1. Use the oc adm command to extract the image URL of the driver-toolkit corresponding to a certain release:

   • For an x86 image, the command is as follows:

     $ oc adm release info quay.io/openshift-release-dev/ocp-release:4.16.z-x86_64 --image-for=driver-toolkit

   • For an ARM image, the command is as follows:

     $ oc adm release info quay.io/openshift-release-dev/ocp-release:4.16.z-aarch64 --image-for=driver-toolkit

   Example output

   quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b53883ca2bac5925857148c4a1abc300ced96c222498e3bc134fe7ce3a1dd404

2. Obtain this image using a valid pull secret, such as the pull secret required to install OpenShift Container Platform:

   $ podman pull --authfile=path/to/pullsecret.json quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:<SHA>

1. The YAML defines an ImageStream for storing the simple-kmod driver container image, and a BuildConfig for building the container. Save this YAML as 0000-buildconfig.yaml.template.

   apiVersion: image.openshift.io/v1
   kind: ImageStream
   metadata:
     labels:
       app: simple-kmod-driver-container
     name: simple-kmod-driver-container
     namespace: simple-kmod-demo
   spec: {}
   ---
   apiVersion: build.openshift.io/v1
   kind: BuildConfig
   metadata:
     labels:
       app: simple-kmod-driver-build
     name: simple-kmod-driver-build
     namespace: simple-kmod-demo
   spec:
     nodeSelector:
       node-role.kubernetes.io/worker: ""
     runPolicy: "Serial"
     triggers:
       - type: "ConfigChange"
       - type: "ImageChange"
     source:
       dockerfile: |
         ARG DTK
         FROM ${DTK} as builder
   
         ARG KVER
   
         WORKDIR /build/
   
         RUN git clone https://github.com/openshift-psap/simple-kmod.git
   
         WORKDIR /build/simple-kmod
   
         RUN make all install KVER=${KVER}
   
         FROM registry.redhat.io/ubi8/ubi-minimal
   
         ARG KVER
   
         # Required for installing `modprobe`
         RUN microdnf install kmod
   
         COPY --from=builder /lib/modules/${KVER}/simple-kmod.ko /lib/modules/${KVER}/
         COPY --from=builder /lib/modules/${KVER}/simple-procfs-kmod.ko /lib/modules/${KVER}/
         RUN depmod ${KVER}
     strategy:
       dockerStrategy:
         buildArgs:
           - name: KMODVER
             value: DEMO
             # $ oc adm release info quay.io/openshift-release-dev/ocp-release:<cluster version>-x86_64 --image-for=driver-toolkit
           - name: DTK
             value: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:34864ccd2f4b6e385705a730864c04a40908e57acede44457a783d739e377cae
           - name: KVER
             value: 4.18.0-372.26.1.el8_6.x86_64
     output:
       to:
         kind: ImageStreamTag
         name: simple-kmod-driver-container:demo

2. Substitute the correct driver toolkit image for the OpenShift Container Platform version you are running in place of “DRIVER_TOOLKIT_IMAGE” with the following commands.

   $ OCP_VERSION=$(oc get clusterversion/version -ojsonpath={.status.desired.version})

   $ DRIVER_TOOLKIT_IMAGE=$(oc adm release info $OCP_VERSION --image-for=driver-toolkit)

   $ sed "s#DRIVER_TOOLKIT_IMAGE#${DRIVER_TOOLKIT_IMAGE}#" 0000-buildconfig.yaml.template > 0000-buildconfig.yaml

3. Create the image stream and build config with

   $ oc create -f 0000-buildconfig.yaml

4. After the builder pod completes successfully, deploy the driver container image as a DaemonSet.

   1. The driver container must run with the privileged security context in order to load the kernel modules on the host. The following YAML file contains the RBAC rules and the DaemonSet for running the driver container. Save this YAML as 1000-drivercontainer.yaml.

      apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: simple-kmod-driver-container
      ---
      apiVersion: rbac.authorization.k8s.io/v1
      kind: Role
      metadata:
        name: simple-kmod-driver-container
      rules:
      - apiGroups:
        - security.openshift.io
        resources:
        - securitycontextconstraints
        verbs:
        - use
        resourceNames:
        - privileged
      ---
      apiVersion: rbac.authorization.k8s.io/v1
      kind: RoleBinding
      metadata:
        name: simple-kmod-driver-container
      roleRef:
        apiGroup: rbac.authorization.k8s.io
        kind: Role
        name: simple-kmod-driver-container
      subjects:
      - kind: ServiceAccount
        name: simple-kmod-driver-container
      userNames:
      - system:serviceaccount:simple-kmod-demo:simple-kmod-driver-container
      ---
      apiVersion: apps/v1
      kind: DaemonSet
      metadata:
        name: simple-kmod-driver-container
      spec:
        selector:
          matchLabels:
            app: simple-kmod-driver-container
        template:
          metadata:
            labels:
              app: simple-kmod-driver-container
          spec:
            serviceAccount: simple-kmod-driver-container
            serviceAccountName: simple-kmod-driver-container
            containers:
            - image: image-registry.openshift-image-registry.svc:5000/simple-kmod-demo/simple-kmod-driver-container:demo
              name: simple-kmod-driver-container
              imagePullPolicy: Always
              command: [sleep, infinity]
              lifecycle:
                postStart:
                  exec:
                    command: ["modprobe", "-v", "-a" , "simple-kmod", "simple-procfs-kmod"]
                preStop:
                  exec:
                    command: ["modprobe", "-r", "-a" , "simple-kmod", "simple-procfs-kmod"]
              securityContext:
                privileged: true
            nodeSelector:
              node-role.kubernetes.io/worker: ""

   2. Create the RBAC rules and daemon set:

      $ oc create -f 1000-drivercontainer.yaml

5. After the pods are running on the worker nodes, verify that the simple_kmod kernel module is loaded successfully on the host machines with lsmod.

   1. Verify that the pods are running:

      $ oc get pod -n simple-kmod-demo

      Example output

      NAME                                 READY   STATUS      RESTARTS   AGE
      simple-kmod-driver-build-1-build     0/1     Completed   0          6m
      simple-kmod-driver-container-b22fd   1/1     Running     0          40s
      simple-kmod-driver-container-jz9vn   1/1     Running     0          40s
      simple-kmod-driver-container-p45cc   1/1     Running     0          40s

   2. Execute the lsmod command in the driver container pod:

      $ oc exec -it pod/simple-kmod-driver-container-p45cc -- lsmod | grep simple

      Example output

      simple_procfs_kmod     16384  0
      simple_kmod            16384  0

Procedure

1. Create a namespace for the NFD Operator.

   1. Create the following Namespace custom resource (CR) that defines the openshift-nfd namespace, and then save the YAML in the nfd-namespace.yaml file. Set cluster-monitoring to "true".

      apiVersion: v1
      kind: Namespace
      metadata:
        name: openshift-nfd
        labels:
          name: openshift-nfd
          openshift.io/cluster-monitoring: "true"

   2. Create the namespace by running the following command:

      $ oc create -f nfd-namespace.yaml

2. Install the NFD Operator in the namespace you created in the previous step by creating the following objects:

   1. Create the following OperatorGroup CR and save the YAML in the nfd-operatorgroup.yaml file:

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        generateName: openshift-nfd-
        name: openshift-nfd
        namespace: openshift-nfd
      spec:
        targetNamespaces:
        - openshift-nfd

   2. Create the OperatorGroup CR by running the following command:

      $ oc create -f nfd-operatorgroup.yaml

   3. Create the following Subscription CR and save the YAML in the nfd-sub.yaml file:

      Example Subscription

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: nfd
        namespace: openshift-nfd
      spec:
        channel: "stable"
        installPlanApproval: Automatic
        name: nfd
        source: redhat-operators
        sourceNamespace: openshift-marketplace

   4. Create the subscription object by running the following command:

      $ oc create -f nfd-sub.yaml

   5. Change to the openshift-nfd project:

      $ oc project openshift-nfd

Procedure

1. In the OpenShift Container Platform web console, click Operators → OperatorHub.
2. Choose Node Feature Discovery from the list of available Operators, and then click Install.
3. On the Install Operator page, select A specific namespace on the cluster, and then click Install. You do not need to create a namespace because it is created for you.

1. Navigate to the Operators → Installed Operators page.

2. Ensure that Node Feature Discovery is listed in the openshift-nfd project with a Status of InstallSucceeded.

   Note

   During installation an Operator might display a Failed status. If the installation later succeeds with an InstallSucceeded message, you can ignore the Failed message.

1. Navigate to the Operators → Installed Operators page and inspect the Operator Subscriptions and Install Plans tabs for any failure or errors under Status.
2. Navigate to the Workloads → Pods page and check the logs for pods in the openshift-nfd project.

Procedure

1. Create a NodeFeatureDiscovery CR:

   Example NodeFeatureDiscovery CR

   apiVersion: nfd.openshift.io/v1
   kind: NodeFeatureDiscovery
   metadata:
     name: nfd-instance
     namespace: openshift-nfd
   spec:
     instance: "" # instance is empty by default
     topologyupdater: false # False by default
     operand:
       image: registry.redhat.io/openshift4/ose-node-feature-discovery-rhel9:v4.16
       imagePullPolicy: Always
     workerConfig:
       configData: |
         core:
         #  labelWhiteList:
         #  noPublish: false
           sleepInterval: 60s
         #  sources: [all]
         #  klog:
         #    addDirHeader: false
         #    alsologtostderr: false
         #    logBacktraceAt:
         #    logtostderr: true
         #    skipHeaders: false
         #    stderrthreshold: 2
         #    v: 0
         #    vmodule:
         ##   NOTE: the following options are not dynamically run-time configurable
         ##         and require a nfd-worker restart to take effect after being changed
         #    logDir:
         #    logFile:
         #    logFileMaxSize: 1800
         #    skipLogHeaders: false
         sources:
           cpu:
             cpuid:
         #     NOTE: whitelist has priority over blacklist
               attributeBlacklist:
                 - "BMI1"
                 - "BMI2"
                 - "CLMUL"
                 - "CMOV"
                 - "CX16"
                 - "ERMS"
                 - "F16C"
                 - "HTT"
                 - "LZCNT"
                 - "MMX"
                 - "MMXEXT"
                 - "NX"
                 - "POPCNT"
                 - "RDRAND"
                 - "RDSEED"
                 - "RDTSCP"
                 - "SGX"
                 - "SSE"
                 - "SSE2"
                 - "SSE3"
                 - "SSE4.1"
                 - "SSE4.2"
                 - "SSSE3"
               attributeWhitelist:
           kernel:
             kconfigFile: "/path/to/kconfig"
             configOpts:
               - "NO_HZ"
               - "X86"
               - "DMI"
           pci:
             deviceClassWhitelist:
               - "0200"
               - "03"
               - "12"
             deviceLabelFields:
               - "class"
     customConfig:
       configData: |
             - name: "more.kernel.features"
               matchOn:
               - loadedKMod: ["example_kmod3"]

2. Create the NodeFeatureDiscovery CR by running the following command:

   $ oc apply -f <filename>

Verification

1. Check that the NodeFeatureDiscovery CR was created by running the following command:

   $ oc get pods

   Example output

   NAME                                      READY   STATUS    RESTARTS   AGE
   nfd-controller-manager-7f86ccfb58-vgr4x   2/2     Running   0          11m
   nfd-master-hcn64                          1/1     Running   0          60s
   nfd-master-lnnxx                          1/1     Running   0          60s
   nfd-master-mp6hr                          1/1     Running   0          60s
   nfd-worker-vgcz9                          1/1     Running   0          60s
   nfd-worker-xqbws                          1/1     Running   0          60s

   A successful deployment shows a Running status.

Procedure

1. Determine the digest of the registry image:

   1. Run the following command:

      $ skopeo inspect docker://registry.redhat.io/openshift4/ose-node-feature-discovery:<openshift_version>

      Example command

      $ skopeo inspect docker://registry.redhat.io/openshift4/ose-node-feature-discovery:v4.12

   2. Inspect the output to identify the image digest:

      Example output

      {
        ...
        "Digest": "sha256:1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef",
        ...
      }

2. Use the skopeo CLI tool to copy the image from registry.redhat.io to your mirror registry, by running the following command:

   skopeo copy docker://registry.redhat.io/openshift4/ose-node-feature-discovery@<image_digest> docker://<mirror_registry>/openshift4/ose-node-feature-discovery@<image_digest>

   Example command

   skopeo copy docker://registry.redhat.io/openshift4/ose-node-feature-discovery@sha256:1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef docker://<your-mirror-registry>/openshift4/ose-node-feature-discovery@sha256:1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef

3. Create a NodeFeatureDiscovery CR:

   Example NodeFeatureDiscovery CR

   apiVersion: nfd.openshift.io/v1
   kind: NodeFeatureDiscovery
   metadata:
     name: nfd-instance
   spec:
     operand:
       image: <mirror_registry>/openshift4/ose-node-feature-discovery@<image_digest>
       imagePullPolicy: Always
     workerConfig:
       configData: |
         core:
         #  labelWhiteList:
         #  noPublish: false
           sleepInterval: 60s
         #  sources: [all]
         #  klog:
         #    addDirHeader: false
         #    alsologtostderr: false
         #    logBacktraceAt:
         #    logtostderr: true
         #    skipHeaders: false
         #    stderrthreshold: 2
         #    v: 0
         #    vmodule:
         ##   NOTE: the following options are not dynamically run-time configurable
         ##         and require a nfd-worker restart to take effect after being changed
         #    logDir:
         #    logFile:
         #    logFileMaxSize: 1800
         #    skipLogHeaders: false
         sources:
           cpu:
             cpuid:
         #     NOTE: whitelist has priority over blacklist
               attributeBlacklist:
                 - "BMI1"
                 - "BMI2"
                 - "CLMUL"
                 - "CMOV"
                 - "CX16"
                 - "ERMS"
                 - "F16C"
                 - "HTT"
                 - "LZCNT"
                 - "MMX"
                 - "MMXEXT"
                 - "NX"
                 - "POPCNT"
                 - "RDRAND"
                 - "RDSEED"
                 - "RDTSCP"
                 - "SGX"
                 - "SSE"
                 - "SSE2"
                 - "SSE3"
                 - "SSE4.1"
                 - "SSE4.2"
                 - "SSSE3"
               attributeWhitelist:
           kernel:
             kconfigFile: "/path/to/kconfig"
             configOpts:
               - "NO_HZ"
               - "X86"
               - "DMI"
           pci:
             deviceClassWhitelist:
               - "0200"
               - "03"
               - "12"
             deviceLabelFields:
               - "class"
     customConfig:
       configData: |
             - name: "more.kernel.features"
               matchOn:
               - loadedKMod: ["example_kmod3"]

4. Create the NodeFeatureDiscovery CR by running the following command:

   $ oc apply -f <filename>

Verification

1. Check the status of the NodeFeatureDiscovery CR by running the following command:

   $ oc get nodefeaturediscovery nfd-instance -o yaml

2. Check that the pods are running without ImagePullBackOff errors by running the following command:

   $ oc get pods -n <nfd_namespace>

Procedure

1. Navigate to the Operators → Installed Operators page.
2. In the Node Feature Discovery section, under Provided APIs, click Create instance.
3. Edit the values of the NodeFeatureDiscovery CR.
4. Click Create.

Procedure

1. Log in to the OpenShift Container Platform web console.

2. Install the Kernel Module Management Operator:

   1. In the OpenShift Container Platform web console, click Operators → OperatorHub.
   2. Select Kernel Module Management Operator from the list of available Operators, and then click Install.
   3. From the Installed Namespace list, select the openshift-kmm namespace.
   4. Click Install.

1. Navigate to the Operators → Installed Operators page.

2. Ensure that Kernel Module Management Operator is listed in the openshift-kmm project with a Status of InstallSucceeded.

   Note

   During installation, an Operator might display a Failed status. If the installation later succeeds with an InstallSucceeded message, you can ignore the Failed message.

Troubleshooting

1. To troubleshoot issues with Operator installation:

   1. Navigate to the Operators → Installed Operators page and inspect the Operator Subscriptions and Install Plans tabs for any failure or errors under Status.
   2. Navigate to the Workloads → Pods page and check the logs for pods in the openshift-kmm project.

Procedure

1. Install KMM in the openshift-kmm namespace:

   1. Create the following Namespace CR and save the YAML file, for example, kmm-namespace.yaml:

      apiVersion: v1
      kind: Namespace
      metadata:
        name: openshift-kmm

   2. Create the following OperatorGroup CR and save the YAML file, for example, kmm-op-group.yaml:

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: kernel-module-management
        namespace: openshift-kmm

   3. Create the following Subscription CR and save the YAML file, for example, kmm-sub.yaml:

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: kernel-module-management
        namespace: openshift-kmm
      spec:
        channel: release-1.0
        installPlanApproval: Automatic
        name: kernel-module-management
        source: redhat-operators
        sourceNamespace: openshift-marketplace
        startingCSV: kernel-module-management.v1.0.0

   4. Create the subscription object by running the following command:

      $ oc create -f kmm-sub.yaml

Procedure

1. Install KMM in the openshift-kmm namespace:

   1. Create the following Namespace CR and save the YAML file, for example, kmm-namespace.yaml file:

      apiVersion: v1
      kind: Namespace
      metadata:
        name: openshift-kmm

   2. Create the following SecurityContextConstraint object and save the YAML file, for example, kmm-security-constraint.yaml:

      allowHostDirVolumePlugin: false
      allowHostIPC: false
      allowHostNetwork: false
      allowHostPID: false
      allowHostPorts: false
      allowPrivilegeEscalation: false
      allowPrivilegedContainer: false
      allowedCapabilities:
        - NET_BIND_SERVICE
      apiVersion: security.openshift.io/v1
      defaultAddCapabilities: null
      fsGroup:
        type: MustRunAs
      groups: []
      kind: SecurityContextConstraints
      metadata:
        name: restricted-v2
      priority: null
      readOnlyRootFilesystem: false
      requiredDropCapabilities:
        - ALL
      runAsUser:
        type: MustRunAsRange
      seLinuxContext:
        type: MustRunAs
      seccompProfiles:
        - runtime/default
      supplementalGroups:
        type: RunAsAny
      users: []
      volumes:
        - configMap
        - downwardAPI
        - emptyDir
        - persistentVolumeClaim
        - projected
        - secret

   3. Bind the SecurityContextConstraint object to the Operator’s ServiceAccount by running the following commands:

      $ oc apply -f kmm-security-constraint.yaml

      $ oc adm policy add-scc-to-user kmm-security-constraint -z kmm-operator-controller -n openshift-kmm

   4. Create the following OperatorGroup CR and save the YAML file, for example, kmm-op-group.yaml:

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: kernel-module-management
        namespace: openshift-kmm

   5. Create the following Subscription CR and save the YAML file, for example, kmm-sub.yaml:

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: kernel-module-management
        namespace: openshift-kmm
      spec:
        channel: release-1.0
        installPlanApproval: Automatic
        name: kernel-module-management
        source: redhat-operators
        sourceNamespace: openshift-marketplace
        startingCSV: kernel-module-management.v1.0.0

   6. Create the subscription object by running the following command:

      $ oc create -f kmm-sub.yaml

1. List all nodes matching .spec.selector.
2. Build a set of all kernel versions running on those nodes.

3. For each kernel version:

   1. Go through .spec.moduleLoader.container.kernelMappings and find the appropriate container image name. If the kernel mapping has build or sign defined and the container image does not already exist, run the build, the signing pod, or both, as needed.
   2. Create a worker pod to pull the container image determined in the previous step and run modprobe.
   3. If .spec.devicePlugin is defined, create a device plugin daemon set using the configuration specified under .spec.devicePlugin.container.

4. Run garbage-collect on:

   1. Obsolete device plugin DaemonSets that do not target any node.
   2. Successful build pods.
   3. Successful signing pods.

1. Build the kernel modules.
2. Copy the .ko files into a smaller end-user image such as ubi-minimal.

3. To leverage DTK in your in-cluster build, use the DTK_AUTO build argument. The value is automatically set by KMM when creating the Build resource. See the following example.

   ARG DTK_AUTO
   FROM ${DTK_AUTO} as builder
   ARG KERNEL_FULL_VERSION
   WORKDIR /usr/src
   RUN ["git", "clone", "https://github.com/rh-ecosystem-edge/kernel-module-management.git"]
   WORKDIR /usr/src/kernel-module-management/ci/kmm-kmod
   RUN KERNEL_SRC_DIR=/lib/modules/${KERNEL_FULL_VERSION}/build make all
   FROM ubi9/ubi-minimal
   ARG KERNEL_FULL_VERSION
   RUN microdnf install kmod
   COPY --from=builder /usr/src/kernel-module-management/ci/kmm-kmod/kmm_ci_a.ko /opt/lib/modules/${KERNEL_FULL_VERSION}/
   COPY --from=builder /usr/src/kernel-module-management/ci/kmm-kmod/kmm_ci_b.ko /opt/lib/modules/${KERNEL_FULL_VERSION}/
   RUN depmod -b /opt ${KERNEL_FULL_VERSION}

Procedure

1. Check to ensure the public key secret is set correctly:

   $ oc get secret -o yaml <certificate secret name> | awk '/cert/{print $2; exit}' | base64 -d  | openssl x509 -inform der -text

   This should display a certificate with a Serial Number, Issuer, Subject, and more.

2. Check to ensure the private key secret is set correctly:

   $ oc get secret -o yaml <private key secret name> | awk '/key/{print $2; exit}' | base64 -d

   This should display the key enclosed in the -----BEGIN PRIVATE KEY----- and -----END PRIVATE KEY----- lines.

Procedure

1. Apply a MachineConfig resource to the existing running cluster. In order to identify the necessary nodes that need to be updated, you must create an appropriate MachineConfigPool resource.
2. MCO applies the reboots node by node. On any rebooted node, two new systemd services are deployed: pull service and load service.
3. The load service is configured to run prior to the NetworkConfiguration service. The service tries to pull a predefined kernel module image and then, using that image, to unload an in-tree module and load an OOT kernel module.
4. The pull service is configured to run after NetworkManager service. The service checks if the preconfigured kernel module image is located on the node’s filesystem. If it is, the service exists normally, and the server continues with the boot process. If not, it pulls the image onto the node and reboots the node afterwards.