海光DCU Kubernetes集成实战：Device Plugin部署与推理服务调度

海光 DCU 在 Kubernetes 中的集成相比昇腾更简单，因为可以直接复用 AMD GPU 的 Device Plugin（amd.com/gpu 资源）和 ROCm 生态的监控工具。本文讲解 DCU K8s 集成的完整流程。

DCU K8s 集成架构

Kubernetes API Server
       ↓
   Scheduler
       ↓ 分配 amd.com/gpu 资源
  kubelet
       ↓ 通过 Device Plugin API
  AMD GPU Device Plugin（DaemonSet）
  ↓ 识别 /dev/dri/renderD* 设备
  海光 DCU 硬件

监控链路：
  hy-smi → rocm-smi-exporter → Prometheus → Grafana

一、安装 DCU Device Plugin

# 前置：所有 DCU 节点已安装 DTK 驱动
# 验证
ls /dev/kfd /dev/dri/renderD*

# 给 DCU 节点打标签
kubectl label node dcu-node-01 dcu-node-02 \
    accelerator=hygon-dcu \
    dcu-model=Z100L

# 打污点（隔离 DCU 节点）
kubectl taint node dcu-node-01 dcu-node-02 \
    accelerator=hygon-dcu:NoSchedule

# dcu-device-plugin.yaml
# 使用 ROCm k8s-device-plugin（DCU 兼容）
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: dcu-device-plugin
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: dcu-device-plugin
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        name: dcu-device-plugin
    spec:
      nodeSelector:
        accelerator: hygon-dcu
      tolerations:
        - key: accelerator
          value: hygon-dcu
          effect: NoSchedule
      priorityClassName: system-node-critical
      containers:
        - name: dcu-device-plugin
          image: rocm/k8s-device-plugin:1.25.2.6
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
          volumeMounts:
            - name: dp
              mountPath: /var/lib/kubelet/device-plugins
            - name: sys
              mountPath: /sys
            - name: dev
              mountPath: /dev
      volumes:
        - name: dp
          hostPath:
            path: /var/lib/kubelet/device-plugins
        - name: sys
          hostPath:
            path: /sys
        - name: dev
          hostPath:
            path: /dev

kubectl apply -f dcu-device-plugin.yaml

# 验证
kubectl rollout status ds/dcu-device-plugin -n kube-system

# 检查 DCU 资源是否出现
kubectl describe node dcu-node-01 | grep -A5 Capacity
# Capacity:
#   amd.com/gpu:    8   ← DCU 设备数量
#   cpu:            128
#   memory:         512Gi

二、DCU 推理服务部署

# vllm-dcu-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-dcu-qwen
  namespace: inference
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm-dcu-qwen
  template:
    metadata:
      labels:
        app: vllm-dcu-qwen
    spec:
      nodeSelector:
        accelerator: hygon-dcu
      tolerations:
        - key: accelerator
          value: hygon-dcu
          effect: NoSchedule
      containers:
        - name: vllm-server
          image: vllm-rocm:latest    # ROCm 版 vLLM
          ports:
            - containerPort: 8000
          command:
            - python3
            - -m
            - vllm.entrypoints.openai.api_server
            - --model=/models/Qwen2.5-7B-Instruct
            - --device=cuda          # DCU 兼容 CUDA 接口
            - --tensor-parallel-size=4
            - --dtype=bfloat16
            - --gpu-memory-utilization=0.90
            - --port=8000
          resources:
            limits:
              amd.com/gpu: 4        # 申请 4 个 DCU
          securityContext:
            capabilities:
              add: ["SYS_PTRACE"]   # ROCm 需要 ptrace 权限
          env:
            - name: HIP_VISIBLE_DEVICES
              value: "0,1,2,3"
            - name: ROCR_VISIBLE_DEVICES
              value: "0,1,2,3"
          volumeMounts:
            - name: models
              mountPath: /models
            - name: dev-kfd
              mountPath: /dev/kfd
            - name: dri
              mountPath: /dev/dri
      volumes:
        - name: models
          persistentVolumeClaim:
            claimName: models-pvc
        - name: dev-kfd
          hostPath:
            path: /dev/kfd
        - name: dri
          hostPath:
            path: /dev/dri

---
apiVersion: v1
kind: Service
metadata:
  name: vllm-dcu-qwen
  namespace: inference
spec:
  selector:
    app: vllm-dcu-qwen
  ports:
    - port: 8000
      targetPort: 8000

三、DCU 监控接入 Prometheus

# ROCm SMI Exporter（类似 DCGM Exporter，但用于 AMD/DCU）
# GitHub：https://github.com/ROCm/rocm_smi_lib/tree/develop/rocm_smi_exporter

# rocm-smi-exporter-daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: rocm-smi-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      name: rocm-smi-exporter
  template:
    metadata:
      labels:
        name: rocm-smi-exporter
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "2021"
    spec:
      nodeSelector:
        accelerator: hygon-dcu
      tolerations:
        - key: accelerator
          value: hygon-dcu
          effect: NoSchedule
      containers:
        - name: rocm-smi-exporter
          image: rocm/rocm-smi-exporter:latest
          ports:
            - containerPort: 2021
          securityContext:
            privileged: true
          volumeMounts:
            - name: dev
              mountPath: /dev
      volumes:
        - name: dev
          hostPath:
            path: /dev

DCU 监控 PromQL

# DCU 利用率
rocm_gpu_use_percent{job="hygon-dcu"}

# DCU 显存使用率
rocm_gpu_memory_vram_used_bytes / rocm_gpu_memory_vram_total_bytes * 100

# DCU 温度
rocm_gpu_temp_celsius{sensor="edge"}

# DCU 功耗
rocm_gpu_power_draw_watts

# 按 Pod 统计 DCU 使用（需要结合 K8s 标签）
rocm_gpu_use_percent * on(node) group_left(pod)
    kube_pod_info{pod=~"vllm.*"}

四、多 DCU 分布式训练

# 使用 PyTorch Job（需要安装 kubeflow/training-operator）
apiVersion: kubeflow.org/v1
kind: PyTorchJob
metadata:
  name: dcu-training-job
  namespace: training
spec:
  pytorchReplicaSpecs:
    Master:
      replicas: 1
      restartPolicy: OnFailure
      template:
        spec:
          nodeSelector:
            accelerator: hygon-dcu
          tolerations:
            - key: accelerator
              value: hygon-dcu
              effect: NoSchedule
          containers:
            - name: pytorch
              image: rocm/pytorch:rocm6.0_ubuntu22.04_py3.11_pytorch_2.3.0
              command:
                - torchrun
                - --nproc_per_node=8
                - --nnodes=2
                - train.py
              resources:
                limits:
                  amd.com/gpu: 8
    Worker:
      replicas: 1    # 2 节点共 16 DCU
      restartPolicy: OnFailure
      template:
        spec:
          nodeSelector:
            accelerator: hygon-dcu
          containers:
            - name: pytorch
              image: rocm/pytorch:rocm6.0_ubuntu22.04_py3.11_pytorch_2.3.0
              command:
                - torchrun
                - --nproc_per_node=8
                - --nnodes=2
                - train.py
              resources:
                limits:
                  amd.com/gpu: 8

五、资源配额与 LimitRange

# DCU 资源配额（限制命名空间使用量）
apiVersion: v1
kind: ResourceQuota
metadata:
  name: dcu-quota
  namespace: inference
spec:
  hard:
    requests.amd.com/gpu: "16"    # 推理命名空间最多使用 16 个 DCU
    limits.amd.com/gpu: "16"

---
# LimitRange（设置 DCU Pod 的默认资源限制）
apiVersion: v1
kind: LimitRange
metadata:
  name: dcu-limits
  namespace: inference
spec:
  limits:
    - type: Container
      max:
        amd.com/gpu: "8"     # 单容器最多 8 个 DCU
      min:
        amd.com/gpu: "1"     # 单容器至少 1 个 DCU

六、节点故障处理

# 场景：DCU 节点故障，需要快速迁移推理服务

# 1. 驱逐故障节点上的 Pod
kubectl drain dcu-node-01 \
    --ignore-daemonsets \
    --delete-emptydir-data \
    --grace-period=30

# 2. 标记不可调度
kubectl cordon dcu-node-01

# Deployment 会自动在其他 DCU 节点重新调度（确保有足够资源）

# 3. 检查服务恢复
kubectl get pods -n inference -o wide
kubectl get endpoints inference/vllm-dcu-qwen

# 4. 排查故障
ssh dcu-node-01
hy-smi                    # 检查 DCU 状态
dmesg | grep -i "amdgpu\|error" | tail -20  # 检查内核日志
journalctl -u kubelet --since "1 hour ago"  # 检查 kubelet 日志

# 5. 修复后重新接入
kubectl uncordon dcu-node-01

# 可选：配置 PodDisruptionBudget 确保迁移期间服务可用性
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: vllm-dcu-pdb
  namespace: inference
spec:
  minAvailable: 1     # 至少保留 1 个副本
  selector:
    matchLabels:
      app: vllm-dcu-qwen

小结

海光 DCU 的 K8s 集成相对简单，复用 AMD GPU 生态（amd.com/gpu 资源、ROCm SMI Exporter、rocm/pytorch 镜像）即可完成大部分配置。注意事项：Pod 需要挂载 /dev/kfd 和 /dev/dri 设备文件（ROCm 运行时依赖）；容器内用 HIP_VISIBLE_DEVICES 控制可见 DCU（类似 CUDA_VISIBLE_DEVICES）；多卡通信使用 RCCL（兼容 NCCL 语义）。整体来说，DCU 的 K8s 集成比昇腾简单，是混合集群中国产 GPU 接入成本最低的选择。