NVIDIA+昇腾双栈Kubernetes调度实战：灰度迁移、流量分割与统一运维

NVIDIA + 昇腾双栈集群是国产化过渡期最典型的架构：保留 NVIDIA 跑现有业务，逐步将工作负载迁移到昇腾。本文聚焦如何让两套硬件共存调度，以及如何安全地做灰度迁移，而不是仅仅让两个 Device Plugin 同时运行。

双栈集群面临的核心挑战

挑战1：同一服务，双份代码
  同一个推理服务需要维护两个版本：
  - CUDA 版本（给 NVIDIA）
  - torch_npu 版本（给昇腾）
  → 需要双镜像构建、双部署流水线

挑战2：调度意图不明确
  "这个 Job 应该跑在哪个硬件上？"
  → 需要明确的调度标签语言

挑战3：迁移期间的服务连续性
  从 NVIDIA 迁移到昇腾，中途不能断服务
  → 需要流量灰度切换能力

挑战4：统一监控指标不对齐
  DCGM 指标名 vs NPU Exporter 指标名完全不同
  → 需要归一化层

挑战5：成本不透明
  两套硬件并存，需要知道各自的成本贡献
  → 需要按硬件类型的资源账单

一、双栈集群节点规划

# 节点池划分
# Pool A: NVIDIA GPU（现有）
kubectl label node nvidia-01 nvidia-02 nvidia-03 \
    pool=nvidia \
    accelerator=nvidia-gpu \
    gpu-model=A100-80G \
    migration-status=stable      # 稳定运行的工作负载

# Pool B: 昇腾 NPU（新增）
kubectl label node ascend-01 ascend-02 \
    pool=ascend \
    accelerator=huawei-npu \
    npu-model=910B \
    migration-status=testing     # 正在验证的工作负载

# 污点隔离（防止误调度）
kubectl taint node nvidia-01 nvidia-02 nvidia-03 \
    pool=nvidia:NoSchedule

kubectl taint node ascend-01 ascend-02 \
    pool=ascend:NoSchedule

# 查看集群硬件分布
kubectl get nodes -L pool,accelerator,gpu-model,npu-model
# NAME         STATUS   POOL     ACCELERATOR    GPU-MODEL     NPU-MODEL
# nvidia-01    Ready    nvidia   nvidia-gpu     A100-80G      <none>
# nvidia-02    Ready    nvidia   nvidia-gpu     A100-80G      <none>
# nvidia-03    Ready    nvidia   nvidia-gpu     A100-80G      <none>
# ascend-01    Ready    ascend   huawei-npu     <none>        910B
# ascend-02    Ready    ascend   huawei-npu     <none>        910B

二、双镜像构建流水线

# CI/CD 双镜像构建（GitLab CI 示例）
# .gitlab-ci.yml

stages:
  - build
  - test
  - deploy

# 构建 NVIDIA 版本镜像
build:nvidia:
  stage: build
  script:
    - docker build -f Dockerfile.cuda -t registry/inference-service:${CI_COMMIT_SHA}-cuda .
    - docker push registry/inference-service:${CI_COMMIT_SHA}-cuda
  only:
    - main

# 构建昇腾版本镜像
build:ascend:
  stage: build
  script:
    - docker build -f Dockerfile.npu -t registry/inference-service:${CI_COMMIT_SHA}-npu .
    - docker push registry/inference-service:${CI_COMMIT_SHA}-npu
  only:
    - main

# 自动化精度对比测试
test:precision:
  stage: test
  script:
    - python3 scripts/compare_outputs.py \
        --cuda-image registry/inference-service:${CI_COMMIT_SHA}-cuda \
        --npu-image registry/inference-service:${CI_COMMIT_SHA}-npu \
        --tolerance 0.01
  only:
    - main

# Dockerfile.cuda（NVIDIA 版本）
FROM pytorch/pytorch:2.3.0-cuda12.1-cudnn8-runtime
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . /app
WORKDIR /app
CMD ["python3", "serve.py"]

# Dockerfile.npu（昇腾版本）
FROM ascendai/cann:8.0.RC3-ubuntu22.04-aarch64
RUN pip install torch==2.3.1 torch_npu==2.3.1.post2
COPY requirements-npu.txt .
RUN pip install -r requirements-npu.txt
COPY . /app
WORKDIR /app
# 挂载路径在 K8s 中通过 Volume 提供
CMD ["python3", "serve_npu.py"]

三、蓝绿切换：NVIDIA → 昇腾渐进迁移

# 阶段1：两套 Deployment 同时运行，Service 指向 NVIDIA（默认）

# inference-service-nvidia.yaml（蓝，稳定）
apiVersion: apps/v1
kind: Deployment
metadata:
  name: inference-blue    # 蓝 = 当前生产
  namespace: inference
  labels:
    app: inference-service
    version: blue
    hardware: nvidia
spec:
  replicas: 4
  selector:
    matchLabels:
      app: inference-service
      version: blue
  template:
    metadata:
      labels:
        app: inference-service
        version: blue
        hardware: nvidia
    spec:
      nodeSelector:
        pool: nvidia
      tolerations:
        - key: pool
          value: nvidia
          effect: NoSchedule
      containers:
        - name: inference
          image: registry/inference-service:v1.2.3-cuda
          resources:
            limits:
              nvidia.com/gpu: 1

---
# inference-service-ascend.yaml（绿，新版本）
apiVersion: apps/v1
kind: Deployment
metadata:
  name: inference-green   # 绿 = 新部署，待验证
  namespace: inference
  labels:
    app: inference-service
    version: green
    hardware: ascend
spec:
  replicas: 1             # 初始只有 1 个副本（流量很少）
  selector:
    matchLabels:
      app: inference-service
      version: green
  template:
    metadata:
      labels:
        app: inference-service
        version: green
        hardware: ascend
    spec:
      nodeSelector:
        pool: ascend
      tolerations:
        - key: pool
          value: ascend
          effect: NoSchedule
      containers:
        - name: inference
          image: registry/inference-service:v1.2.3-npu
          resources:
            limits:
              huawei.com/Ascend910: 1
          volumeMounts:
            - name: driver
              mountPath: /usr/local/Ascend/driver
              readOnly: true
      volumes:
        - name: driver
          hostPath:
            path: /usr/local/Ascend/driver

Istio 流量分割

# 使用 Istio VirtualService 控制流量比例（不需要改 Deployment）
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: inference-vs
  namespace: inference
spec:
  hosts:
    - inference-service
  http:
    - route:
        # 初期：95% NVIDIA，5% 昇腾
        - destination:
            host: inference-service
            subset: nvidia
          weight: 95
        - destination:
            host: inference-service
            subset: ascend
          weight: 5

---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: inference-dr
  namespace: inference
spec:
  host: inference-service
  subsets:
    - name: nvidia
      labels:
        hardware: nvidia
    - name: ascend
      labels:
        hardware: ascend

---
# Service（两个 Deployment 共享同一个 Service，通过 Istio 分流）
apiVersion: v1
kind: Service
metadata:
  name: inference-service
  namespace: inference
spec:
  selector:
    app: inference-service     # 选择两个版本的 Pod
  ports:
    - port: 8000
      targetPort: 8000

# 灰度迁移进度表
# 每次调整 VirtualService 的 weight，观察 1-2 天后再继续

# 第1周：5% 昇腾（验证基本可用性）
kubectl patch virtualservice inference-vs -n inference \
    --type=merge -p '{"spec":{"http":[{"route":[
        {"destination":{"host":"inference-service","subset":"nvidia"},"weight":95},
        {"destination":{"host":"inference-service","subset":"ascend"},"weight":5}
    ]}]}}'

# 第2周：20% 昇腾（扩大验证范围）
# 第3周：50% 昇腾（双活状态，持续对比监控）
# 第4周：80% 昇腾（主力切换）
# 第5周：100% 昇腾（完成迁移，NVIDIA 进入缩减状态）

四、统一监控指标归一化

# Prometheus Recording Rules：归一化 NVIDIA 和昇腾的指标名

groups:
  - name: unified_gpu_metrics
    interval: 30s
    rules:
      # ==== 利用率 ====
      # NVIDIA（来自 DCGM Exporter）
      - record: cluster:gpu_utilization:percent
        expr: |
          label_replace(
            DCGM_FI_DEV_GPU_UTIL,
            "hardware", "nvidia", "", ""
          )
      # 昇腾（来自 NPU Exporter）
      - record: cluster:gpu_utilization:percent
        expr: |
          label_replace(
            container_npu_utilization,
            "hardware", "ascend", "", ""
          )

      # ==== 显存使用率 ====
      - record: cluster:gpu_memory_used_ratio:percent
        expr: |
          label_replace(
            DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_TOTAL * 100,
            "hardware", "nvidia", "", ""
          )
      - record: cluster:gpu_memory_used_ratio:percent
        expr: |
          label_replace(
            container_npu_memory_used_bytes / container_npu_memory_total_bytes * 100,
            "hardware", "ascend", "", ""
          )

      # ==== 温度 ====
      - record: cluster:gpu_temperature:celsius
        expr: |
          label_replace(DCGM_FI_DEV_GPU_TEMP, "hardware", "nvidia", "", "")
      - record: cluster:gpu_temperature:celsius
        expr: |
          label_replace(container_npu_temperature, "hardware", "ascend", "", "")

# Grafana 双栈对比 Panel

# 利用率对比（NVIDIA vs 昇腾）
avg by (hardware) (cluster:gpu_utilization:percent)

# 两套硬件的推理延迟对比（用于验证昇腾的服务质量）
histogram_quantile(0.99,
  sum by (hardware, le) (
    rate(http_request_duration_seconds_bucket{
      app="inference-service"
    }[5m])
  )
)

# 昇腾副本的错误率（是否高于 NVIDIA）
sum by (hardware) (
  rate(http_requests_total{app="inference-service", status=~"5.."}[5m])
)
/
sum by (hardware) (
  rate(http_requests_total{app="inference-service"}[5m])
)

# 迁移进度（昇腾流量占比）
sum(rate(http_requests_total{app="inference-service", hardware="ascend"}[5m]))
/
sum(rate(http_requests_total{app="inference-service"}[5m]))

五、双栈运维 SOP

# === SOP 1：昇腾副本出现故障，快速切回 NVIDIA ===

# 紧急回切（1分钟内完成）
kubectl patch virtualservice inference-vs -n inference \
    --type=merge -p '{"spec":{"http":[{"route":[
        {"destination":{"host":"inference-service","subset":"nvidia"},"weight":100}
    ]}]}}'

# 确认流量已切回
kubectl exec -it $(kubectl get pod -n inference -l hardware=nvidia -o name | head -1) -- \
    curl -s http://localhost:8000/metrics | grep requests_total

# 分析昇腾故障
kubectl logs -n inference -l hardware=ascend --since=1h | tail -100
kubectl exec -it $(kubectl get pod -n inference -l hardware=ascend -o name | head -1) -- \
    npu-smi info

# === SOP 2：验证昇腾精度（每次版本升级后执行）===

# 对同一批请求发送到两套硬件，对比响应
python3 - << 'EOF'
import requests, json, numpy as np

test_prompts = ["请介绍深度学习"] * 50
nvidia_responses = []
ascend_responses = []

for prompt in test_prompts:
    # NVIDIA 副本（直接访问，绕过 Istio 分流）
    r = requests.post("http://inference-nvidia:8000/v1/completions",
                      json={"prompt": prompt, "max_tokens": 100, "temperature": 0})
    nvidia_responses.append(r.json()["choices"][0]["text"])
    
    # 昇腾副本
    r = requests.post("http://inference-ascend:8000/v1/completions",
                      json={"prompt": prompt, "max_tokens": 100, "temperature": 0})
    ascend_responses.append(r.json()["choices"][0]["text"])

# 对比（温度为0时，输出应该完全一致）
mismatches = sum(1 for n, a in zip(nvidia_responses, ascend_responses) if n != a)
print(f"不一致数量: {mismatches}/{len(test_prompts)}")
if mismatches > 0:
    print("⚠️  精度验证失败，暂停迁移")
else:
    print("✅ 精度验证通过")
EOF

六、资源成本分析

# 统计各硬件类型的 GPU 时使用量（用于成本分配）

# NVIDIA GPU 时（过去 24 小时）
sum_over_time(
  count by (node) (DCGM_FI_DEV_GPU_UTIL > 0)[24h:1m]
) * 60   # 分钟 → 小时

# 昇腾 NPU 时（过去 24 小时）
sum_over_time(
  count by (node) (container_npu_utilization > 0)[24h:1m]
) * 60

# 各 namespace（业务团队）的 GPU 使用量
sum by (namespace, hardware) (
  cluster:gpu_utilization:percent
) / 100

小结

NVIDIA + 昇腾双栈集群的核心是渐进迁移而非一次性切换。关键机制：双镜像 CI/CD（每次发版同时构建 CUDA 和 NPU 版本）、Istio 流量分割（精确控制 NVIDIA vs 昇腾的流量比例）、统一指标归一化（让 Grafana 可以横向对比两套硬件的服务质量）。每次迁移决策都应该基于数据：昇腾副本的错误率是否与 NVIDIA 相当？P99 延迟是否可接受？精度验证是否通过？这三个指标都满足后再扩大昇腾的流量比例，最终实现零风险迁移。