Chapter 6. IBM Spyre integration
The IBM Spyre Operator integrates IBM Spyre accelerators directly into OpenShift AI workflows.
To enable IBM Spyre in OpenShift AI, you must perform the following tasks:
- Install all necessary dependencies before you enable IBM Spyre accelerators in OpenShift AI.
- Install the latest version of the IBM Spyre Operator from the software catalog.
- After you install the IBM Spyre Operator, you must create an accelerator profile or hardware profile in OpenShift AI to expose IBM Spyre as an available accelerator resource for workloads.
aiu-smi metrics collection tool in OpenShift on IBM Power
To run aiu-smi in an OpenShift environment on IBM Power, use the command oc exec <pod> -- aiu-smi, where <pod> is the pod starting with the model deployment name.
For more information on the aiu-smi tool, refer to the IBM Spyre for Power documentation on aiu-smi (except the View the command usage and output section).
For more detailed guidance on configuring IBM Spyre accelerators for production-ready deployments, contact IBM support.
6.1. Enabling IBM Spyre
Before you can use IBM Spyre AI accelerators in OpenShift AI, you must install the Spyre Operator.
Prerequisites
- You have logged in to the OpenShift cluster.
-
You have the
cluster-adminrole in the OpenShift cluster. Your worker nodes equipped with IBM Spyre accelerators meet the following hardware requirements:
- A minimum of 512 GB of RAM.
- A minimum of 500 GB of local disk space.
- You have configured the IBM Spyre accelerators and verified that the cluster detects them.
-
You have applied the required
MachineConfigobjects as described in Specifying IBM SpyreMachineConfigs.
Procedure
- To enable IBM Spyre support on an OpenShift cluster, follow the instructions in IBM Spyre accelerator on Red Hat OpenShift Container Platform in the IBM documentation.
-
After you install the Node Feature Discovery (NFD) Operator, create a
NodeFeatureDiscoveryinstance. -
After you install the IBM Spyre Operator, create a
SpyreClusterPolicyinstance and populate it with default values. -
To use the default scheduler for IBM Spyre workloads, remove the
externalDeviceReservationfield from theSpyreClusterPolicyobject under thespec.experimentalModesection. - Create a hardware profile for the IBM Spyre accelerators. For more information, see Working with hardware profiles.
Verification
Go to the Installed Operators page. The navigation path depends on your OpenShift version:
-
On OpenShift 4.20 and later, click Ecosystem
Installed Operators. -
On OpenShift 4.19, click Operators
Installed Operators.
-
On OpenShift 4.20 and later, click Ecosystem
Verify that the following operators appear with a status of
Succeeded:- IBM Spyre Operator
- Node Feature Discovery (NFD)
- cert-manager Operator for Red Hat OpenShift
- Secondary Scheduler Operator
Verify that the cluster detects the IBM Spyre accelerators.
Wait a few minutes after the installation completes, and then run the following command to describe a worker node:
$ oc describe node <node_name>In the
Capacitysection of the output, verify that theibm.comresources appear, similar to the following example:Capacity: cpu: 16 ephemeral-storage: 523823084Ki hugepages-1Gi: 0 hugepages-2Mi: 0 ibm.com/spyre_pf: 4 ibm.com/spyre_pf_0481_50_00.0: 1 ibm.com/spyre_pf_0482_60_00.0: 1 ibm.com/spyre_pf_0483_70_00.0: 1 ibm.com/spyre_pf_0484_80_00.0: 0 ibm.com/spyre_pf_tier0: 3 ibm.com/spyre_pf_tier1: 3 ibm.com/spyre_pf_tier2: 3 memory: 1038738560Ki pods: 250
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}