6.4. 对动态加速器 Slicer Operator 进行故障排除

流程

1. 运行以下命令，检查所有 DAS Operator 组件的状态：

   $ oc get pods -n das-operator

   Copy to Clipboard Toggle word wrap

   输出示例

   NAME                                    READY   STATUS    RESTARTS   AGE
   das-daemonset-6rsfd                     1/1     Running   0          5m16s
   das-daemonset-8qzgf                     1/1     Running   0          5m16s
   das-operator-5946478b47-cjfcp           1/1     Running   0          5m18s
   das-operator-5946478b47-npwmn           1/1     Running   0          5m18s
   das-operator-webhook-59949d4f85-5n9qt   1/1     Running   0          68s
   das-operator-webhook-59949d4f85-nbtdl   1/1     Running   0          68s
   das-scheduler-6cc59dbf96-4r85f          1/1     Running   0          68s
   das-scheduler-6cc59dbf96-bf6ml          1/1     Running   0          68s

   Copy to Clipboard Toggle word wrap

2. 运行以下命令，检查 DAS Operator 控制器的日志：

   $ oc logs -n das-operator deployment/das-operator

   Copy to Clipboard Toggle word wrap

3. 运行以下命令，检查 webhook 服务器的日志：

   $ oc logs -n das-operator deployment/das-operator-webhook

   Copy to Clipboard Toggle word wrap

4. 运行以下命令，检查调度程序插件的日志：

   $ oc logs -n das-operator deployment/das-scheduler

   Copy to Clipboard Toggle word wrap

5. 运行以下命令，检查设备插件 daemonset 的日志：

   $ oc logs -n das-operator daemonset/das-daemonset

   Copy to Clipboard Toggle word wrap

流程

1. 运行以下命令检查活跃的 AllocationClaim 资源：

   $ oc get allocationclaims -n das-operator

   Copy to Clipboard Toggle word wrap

   输出示例

   NAME                                                                                           AGE
   13950288-57df-4ab5-82bc-6138f646633e-harpatil000034jma-qh5fm-worker-f-57md9-cuda-vectoradd-0   5m
   ce997b60-a0b8-4ea4-9107-cf59b425d049-harpatil000034jma-qh5fm-worker-f-fl4wg-cuda-vectoradd-0   5m

   Copy to Clipboard Toggle word wrap

2. 运行以下命令，查看有关特定 AllocationClaim 的详细信息：

   $ oc get allocationclaims -n das-operator -o yaml

   Copy to Clipboard Toggle word wrap

   输出示例（截断）

   apiVersion: inference.redhat.com/v1alpha1
   kind: AllocationClaim
   metadata:
     name: 13950288-57df-4ab5-82bc-6138f646633e-harpatil000034jma-qh5fm-worker-f-57md9-cuda-vectoradd-0
     namespace: das-operator
   spec:
     gpuUUID: GPU-9003fd9c-1ad1-c935-d8cd-d1ae69ef17c0
     migPlacement:
       size: 1
       start: 0
     nodename: harpatil000034jma-qh5fm-worker-f-57md9
     podRef:
       kind: Pod
       name: cuda-vectoradd-f4b84b678-l2m69
       namespace: default
       uid: 13950288-57df-4ab5-82bc-6138f646633e
     profile: 1g.5gb
   status:
     conditions:
     - lastTransitionTime: "2025-08-06T19:28:48Z"
       message: Allocation is inUse
       reason: inUse
       status: "True"
       type: State
     state: inUse

   Copy to Clipboard Toggle word wrap

3. 运行以下命令，检查不同状态中的声明：

   $ oc get allocationclaims -n das-operator -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.state}{"\n"}{end}'

   Copy to Clipboard Toggle word wrap

   输出示例

   13950288-57df-4ab5-82bc-6138f646633e-harpatil000034jma-qh5fm-worker-f-57md9-cuda-vectoradd-0	inUse
   ce997b60-a0b8-4ea4-9107-cf59b425d049-harpatil000034jma-qh5fm-worker-f-fl4wg-cuda-vectoradd-0	inUse

   Copy to Clipboard Toggle word wrap

4. 运行以下命令，查看与 AllocationClaim 资源相关的事件：

   $ oc get events -n das-operator --field-selector involvedObject.kind=AllocationClaim

   Copy to Clipboard Toggle word wrap

5. 运行以下命令，检查 NodeAccelerator 资源以验证 GPU 硬件检测：

   $ oc get nodeaccelerator -n das-operator

   Copy to Clipboard Toggle word wrap

   输出示例

   NAME                                     AGE
   harpatil000034jma-qh5fm-worker-f-57md9   96m
   harpatil000034jma-qh5fm-worker-f-fl4wg   96m

   Copy to Clipboard Toggle word wrap

   NodeAccelerator 资源代表 DAS Operator 检测到的 GPU 功能的节点。

流程

1. 在使用 GPU 硬件的节点上，运行以下命令来验证 CDI 设备是否已创建：

   $ oc debug node/<node-name>

   Copy to Clipboard Toggle word wrap

   sh-4.4# chroot /host
   sh-4.4# ls -l /var/run/cdi/

   Copy to Clipboard Toggle word wrap

2. 运行以下命令检查 NVIDIA GPU Operator 状态：

   $ oc get clusterpolicies.nvidia.com -o jsonpath='{.items[0].status.state}'

   Copy to Clipboard Toggle word wrap

   输出应显示 ready。

1. 运行以下命令来编辑 DASOperator 资源来提高日志详细程度：

   $ oc edit dasoperator -n das-operator

   Copy to Clipboard Toggle word wrap

2. 将 operatorLogLevel 字段设置为 Debug 或 Trace ：

   spec:
     operatorLogLevel: Debug

   Copy to Clipboard Toggle word wrap
3. 保存更改，并验证 Operator pod 重启并增加详细程度。