在 A4 虚拟机上使用 Dynamo 和 SGLang，通过多主机 PD 解耦在 GKE 上部署 Qwen3.8-2.4T

设置环境变量

导出您在本教程中使用的环境变量：

export PROJECT_ID="YOUR_PROJECT_ID"
export REGION="YOUR_REGION"
export ZONE="YOUR_ZONE"
export CLUSTER_NAME="YOUR_CLUSTER_NAME"
export RESERVATION_NAME="YOUR_RESERVATION_NAME"
export GCS_BUCKET="YOUR_BUCKET_NAME"
export HF_TOKEN="YOUR_HF_TOKEN"
export CLUSTER_TOOLKIT_VERSION="v1.104.0"
export DYNAMO_RELEASE_VERSION="1.5.0"

gcloud config set project "${PROJECT_ID}"
gcloud config set billing/quota_project "${PROJECT_ID}"

替换以下内容：

• YOUR_PROJECT_ID：您的 Google Cloud项目的 ID。
• YOUR_REGION：A4 容量所在的区域以及您要部署集群的区域，例如 us-central1。
• YOUR_ZONE：A4 容量所在的可用区，例如 us-central1-b。
• YOUR_CLUSTER_NAME：GKE 集群和 Cluster Toolkit 部署的名称。请使用不超过 20 个字符的名称，以便生成的节点池服务账号 ID (YOUR_CLUSTER_NAME-gke-np-sa) 保持在 30 个字符的 Identity and Access Management (IAM) 限制范围内。
• YOUR_RESERVATION_NAME：A4 预留的名称，或格式为 RESERVATION_NAME/reservationBlocks/RESERVATION_BLOCK_NAME 的扩展预留块路径。
• YOUR_BUCKET_NAME：Cloud Storage 存储桶的全球唯一名称，该存储桶用于存储 Cluster Toolkit 的 Terraform 状态和暂存的 Qwen3.8-2.4T 模型权重。
• YOUR_HF_TOKEN：您的 Hugging Face 访问令牌。

创建 GKE 集群

下载 Cluster Toolkit，并在 Standard 模式下预配一个 GKE 集群，该集群包含四个 a4-highgpu-8g 节点、GPUDirect RDMA 网络和 DRA 支持：

1. 登录 Google Cloud CLI 并为 Terraform 生成应用默认凭证：

   gcloud auth login
   gcloud auth application-default login

2. 下载并解压缩 CLUSTER_TOOLKIT_VERSION (v1.104.0) 中指定的版本的预构建 Cluster Toolkit 软件包：

   TAG="${CLUSTER_TOOLKIT_VERSION:-v1.104.0}"
   OS="linux"
   ARCH="amd64"
   mkdir -p cluster-toolkit && curl -L "https://github.com/GoogleCloudPlatform/cluster-toolkit/releases/download/${TAG}/gcluster_bundle_${OS}_${ARCH}.tgz" | tar -xz -C cluster-toolkit

   如果您在 macOS 或 arm64 工作站上运行此命令，请将 OS="mac" 或 ARCH="arm64" 设置为与您的平台匹配。

3. 如需存储 Cluster Toolkit 部署的 Terraform 状态和暂存的模型权重，请在 A4 预留所在的区域中创建一个 Cloud Storage 存储桶，并启用统一存储桶级访问权限和对象版本控制：

   gcloud storage buckets create "gs://${GCS_BUCKET}" \
       --default-storage-class=STANDARD \
       --project="${PROJECT_ID}" \
       --location="${REGION}" \
       --uniform-bucket-level-access
   gcloud storage buckets update "gs://${GCS_BUCKET}" --versioning

4. 为四个 a4-highgpu-8g 节点创建 cluster-toolkit/examples/gke-a4/gke-a4-deployment.yaml 部署配置文件：

   cat <<EOF > cluster-toolkit/examples/gke-a4/gke-a4-deployment.yaml
   terraform_backend_defaults:
     type: gcs
     configuration:
       bucket: ${GCS_BUCKET}
   
   vars:
     project_id: ${PROJECT_ID}
     deployment_name: ${CLUSTER_NAME}
     region: ${REGION}
     zone: ${ZONE}
     static_node_count: 4
     authorized_cidr: 0.0.0.0/0
     reservation_affinity:
       consume_reservation_type: SPECIFIC_RESERVATION
       specific_reservations:
       - name: ${RESERVATION_NAME}
   EOF

   默认情况下，部署文件配置为使用特定预留。如果您没有特定预留，则可以在部署集群之前更新 cluster-toolkit/examples/gke-a4/gke-a4-deployment.yaml，以使用其他使用选项，例如 Spot 虚拟机或灵活启动型虚拟机。如需了解配置说明，请参阅 GKE 使用选项。

5. 切换到 cluster-toolkit 目录，使用 gcluster deploy 命令部署 GKE 集群，然后返回到工作目录：

   cd cluster-toolkit
   ./gcluster deploy \
       -d examples/gke-a4/gke-a4-deployment.yaml \
       examples/gke-a4/gke-a4.yaml \
       --auto-approve -w
   cd ..

   创建集群大约需要 20 到 30 分钟。如果您在不使用 --auto-approve 标志的情况下运行 ./gcluster deploy，则在系统提示时输入 a 即可应用建议的更改。

6. 配置 kubectl 凭据以与您的集群通信：

   gcloud container clusters get-credentials "${CLUSTER_NAME}" \
       --location="${REGION}" \
       --project="${PROJECT_ID}"

安装 KAI 调度器和 NVIDIA Dynamo

通过创建 Hugging Face Secret 并安装 KAI 调度器和 NVIDIA Dynamo 平台（采用 Grove 帮派调度）来准备集群：

1. 创建一个名为 hf-secret 的 Kubernetes Secret，用于存储您的 Hugging Face 访问令牌：

   kubectl create secret generic hf-secret \
       --from-literal=hf_api_token="${HF_TOKEN}" \
       --dry-run=client -o yaml | kubectl apply -f -

2. 下载 NVIDIA Dynamo 平台 Helm chart (v1.5.0)，安装 KAI 调度器，然后安装启用了 Grove 的 Dynamo 平台：

   RELEASE_VERSION="${DYNAMO_RELEASE_VERSION:-1.5.0}"
   helm pull "https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-platform-${RELEASE_VERSION}.tgz"
   tar -zxf "dynamo-platform-${RELEASE_VERSION}.tgz"
   
   helm upgrade --install kai-scheduler ./dynamo-platform/charts/kai-scheduler \
       --set scheduler.args.default-staleness-grace-period="-1s" \
       --set topologyMigration.enabled=false
   
   kubectl wait --for=condition=Available deployment/kai-operator --timeout=120s
   until kubectl get deployment/queue-controller >/dev/null 2>&1; do sleep 2; done
   kubectl wait --for=condition=Available deployment/queue-controller --timeout=120s
   
   helm upgrade --install dynamo-platform ./dynamo-platform \
       --set global.grove.install=true \
       --set global.grove.enabled=true \
       --set global.kai-scheduler.enabled=true

在 Cloud Storage 中暂存模型权重

将 nvidia/Qwen3.8-2.4T-A95B-NVFP4 检查点暂存到区域内 Cloud Storage 存储桶 (gs://${GCS_BUCKET}) 中，以便推理工作器 Pod 可以使用 Cloud Storage FUSE 装载权重，而无需在每次冷启动时从 Hugging Face 下载权重：

1. 向与 workload-identity-k8s-sa Kubernetes 服务账号绑定的 Google 服务账号授予您在创建 GKE 集群中创建的 gs://${GCS_BUCKET} 存储桶的 Storage Object User 角色 (roles/storage.objectUser)：

   GSA=$(kubectl get sa workload-identity-k8s-sa -o jsonpath='{.metadata.annotations.iam\.gke\.io/gcp-service-account}')
   gcloud storage buckets add-iam-policy-binding "gs://${GCS_BUCKET}" \
       --member="serviceAccount:${GSA}" \
       --role="roles/storage.objectUser"

2. 下载 qwen3.8-2.4t-nvfp4-stage-weights-job.yaml 和 qwen3.8-2.4t-nvfp4-multihost-a4-gcsfuse.yaml 清单：

   curl -O https://raw.githubusercontent.com/GoogleCloudSamples/AIHypercomputerSamples/refs/heads/main/gpu/inference/qwen38-2t/qwen3.8-2.4t-nvfp4-stage-weights-job.yaml
   curl -O https://raw.githubusercontent.com/GoogleCloudSamples/AIHypercomputerSamples/refs/heads/main/gpu/inference/qwen38-2t/qwen3.8-2.4t-nvfp4-multihost-a4-gcsfuse.yaml

   或者，创建一个名为 qwen3.8-2.4t-nvfp4-stage-weights-job.yaml 的文件，其中包含以下内容。此索引作业在四个 A4 节点上运行四个并行 Pod。每个 Pod 都会将 nvidia/Qwen3.8-2.4T-A95B-NVFP4 检查点的大小均衡切片下载到节点的本地 NVMe SSD，并使用 gcloud storage rsync 将其上传到 gs://${GCS_BUCKET}/hf_models/models--nvidia--qwen3_8-2_4t-a95b-nvfp4：

   # ==============================================================================
   # Stage nvidia/Qwen3.8-2.4T-A95B-NVFP4 (~1.44 TB, 210 files) from Hugging Face Hub
   # into an in-region Cloud Storage bucket for qwen3.8-2.4t-nvfp4-multihost-a4-gcsfuse.yaml
   #
   # Each Indexed Job pod downloads a size-balanced slice of the repo onto the node's
   # local NVMe SSD, then uploads that slice with `gcloud storage rsync`.
   # ==============================================================================
   apiVersion: batch/v1
   kind: Job
   metadata:
     name: qwen3-8-2-4t-stage-weights
   spec:
     completionMode: Indexed
     completions: 4
     parallelism: 4
     backoffLimitPerIndex: 2
     template:
       metadata:
         labels:
           app: qwen3-8-2-4t-stage-weights
       spec:
         restartPolicy: Never
         serviceAccountName: workload-identity-k8s-sa
         nodeSelector:
           cloud.google.com/gke-accelerator: nvidia-b200
         tolerations:
           - key: "nvidia.com/gpu"
             operator: "Exists"
             effect: "NoSchedule"
         topologySpreadConstraints:
           - maxSkew: 1
             topologyKey: kubernetes.io/hostname
             whenUnsatisfiable: DoNotSchedule
             labelSelector:
               matchLabels:
                 app: qwen3-8-2-4t-stage-weights
         initContainers:
           - name: download
             image: nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.0
             command: ["python3", "-c"]
             args:
               - |
                 import os, time
                 from huggingface_hub import HfApi, snapshot_download
                 repo = os.environ["REPO_ID"]
                 n = int(os.environ["NUM_SHARDS"])
                 i = int(os.environ["JOB_COMPLETION_INDEX"])
                 info = HfApi().model_info(repo, files_metadata=True)
                 files = sorted(((s.size or 0, s.rfilename) for s in info.siblings), reverse=True)
                 bins = [[0, []] for _ in range(n)]
                 for size, name in files:
                     b = min(bins, key=lambda x: x[0])
                     b[0] += size
                     b[1].append(name)
                 size, mine = bins[i]
                 print(f"shard {i}/{n}: {len(mine)} files, {size / 1e9:.1f} GB", flush=True)
                 for attempt in range(1, 7):
                     try:
                         snapshot_download(repo, local_dir="/ssd/model", allow_patterns=mine, max_workers=16)
                         break
                     except Exception as e:
                         print(f"shard {i}/{n}: attempt {attempt} failed: {e!r}", flush=True)
                         if attempt == 6:
                             raise
                         time.sleep(30)
                 print(f"shard {i}/{n}: download complete", flush=True)
             env:
               - name: REPO_ID
                 value: "nvidia/Qwen3.8-2.4T-A95B-NVFP4"
               - name: NUM_SHARDS
                 value: "4"
               - name: HF_HOME
                 value: "/ssd/hf"
               - name: HF_HUB_DISABLE_XET
                 value: "1"
               - name: HF_TOKEN
                 valueFrom:
                   secretKeyRef:
                     name: hf-secret
                     key: hf_api_token
                     optional: true
             resources:
               requests:
                 cpu: "16"
                 memory: "64Gi"
                 ephemeral-storage: "450Gi"
               limits:
                 cpu: "16"
                 memory: "64Gi"
                 ephemeral-storage: "700Gi"
             volumeMounts:
               - name: ssd
                 mountPath: /ssd
         containers:
           - name: upload
             image: gcr.io/google.com/cloudsdktool/google-cloud-cli:slim
             command: ["/bin/bash", "-c"]
             args:
               - |
                 set -euo pipefail
                 gcloud storage rsync --recursive --exclude='^\.cache/.*' /ssd/model "${DEST}"
                 echo "shard ${JOB_COMPLETION_INDEX}: upload complete"
             env:
               - name: DEST
                 value: "gs://${GCS_BUCKET}/hf_models/models--nvidia--qwen3_8-2_4t-a95b-nvfp4"
             resources:
               requests:
                 cpu: "16"
                 memory: "32Gi"
               limits:
                 cpu: "16"
                 memory: "32Gi"
             volumeMounts:
               - name: ssd
                 mountPath: /ssd
         volumes:
           - name: ssd
             emptyDir: {}

3. 将 Cloud Storage 存储桶名称替换为 qwen3.8-2.4t-nvfp4-stage-weights-job.yaml，启动暂存作业，等待所有四个 Pod 完成权重上传，然后删除已完成的作业：

   envsubst '${GCS_BUCKET}' < qwen3.8-2.4t-nvfp4-stage-weights-job.yaml | kubectl apply -f -
   
   kubectl wait --for=condition=complete job/qwen3-8-2-4t-stage-weights --timeout=2h
   
   kubectl delete job qwen3-8-2-4t-stage-weights

   将模型权重下载到本地 NVMe SSD 并将其上传到 Cloud Storage 大约需要 25 分钟。

4. 可选：在暂存作业运行时，在第二个终端会话中流式传输暂存 Pod 的日志，以监控下载进度：

   POD_NAME="YOUR_POD_NAME"
   kubectl logs -f "${POD_NAME}"

   将 YOUR_POD_NAME 替换为 kubectl get pods 中其中一个 qwen3-8-2-4t-stage-weights Pod 的名称。

部署多主机分离式 SGLang 服务器

工作负载清单 (qwen3.8-2.4t-nvfp4-multihost-a4-gcsfuse.yaml) 定义了一个 Kubernetes ResourceClaimTemplate (all-mrdma)，该模板通过 DRA 为 workload-identity-k8s-sa 服务账号分配节点的多个 NIC RDMA 设备 (mrdma.google.com)，以及 Role 和 RoleBinding 资源，并定义了一个启用了 Grove 组调度的 NVIDIA Dynamo DynamoGraphDeployment (qwen3-8-2-4t-a4-disagg)。部署包括以下内容：

• 前端 (Frontend)：运行一个路由器副本，用于接受客户端请求、发现已注册的预填充和解码工作器，以及协调解耦请求路由。

• 预填充工作器 (Prefill)：在两个 a4-highgpu-8g 节点上运行一个双节点工作器池，总共配备 16 个 NVIDIA B200 GPU。每个 Pod 都会使用 Cloud Storage FUSE CSI 驱动程序 (gcsfuse.csi.storage.gke.io) 和本地 NVMe SSD 文件缓存，从 Cloud Storage 存储桶中的 /model-weights 装载暂存的 nvidia/Qwen3.8-2.4T-A95B-NVFP4 权重，获取 NCCL/gIB 环境，并使用以下标志在 --disaggregation-mode prefill 中启动 python3 -m dynamo.sglang：

  • --served-model-name "nvidia/Qwen3.8-2.4T-A95B-NVFP4"
  • --tensor-parallel-size 16
  • --ep-size 16
  • --nnodes 2
  • --moe-runner-backend flashinfer_trtllm
  • --attention-backend triton
  • --kv-cache-dtype fp8_e4m3
  • --disable-flashinfer-autotune

• 解码工作器 (Decode)：在其余两个 a4-highgpu-8g 节点上运行一个双节点工作器池，总共包含 16 个 NVIDIA B200 GPU。每个 Pod 都会通过 Cloud Storage FUSE CSI 驱动程序在 /model-weights 处装载 Cloud Storage 中的暂存模型权重，并使用以下标志在 --disaggregation-mode decode 中启动 python3 -m dynamo.sglang：

  • --served-model-name "nvidia/Qwen3.8-2.4T-A95B-NVFP4"
  • --tensor-parallel-size 16
  • --ep-size 16
  • --nnodes 2
  • --moe-runner-backend flashinfer_trtllm
  • --attention-backend triton
  • --kv-cache-dtype fp8_e4m3

部署和监控解耦推理服务器：

1. 查看您之前下载的 qwen3.8-2.4t-nvfp4-multihost-a4-gcsfuse.yaml 工作负载清单，或创建一个名为 qwen3.8-2.4t-nvfp4-multihost-a4-gcsfuse.yaml 的文件，其中包含以下内容：

   # ==============================================================================
   # Multihost Disaggregated Serving of Qwen3.8-2.4T-A95B (NVFP4 MoE) on GKE
   # using NVIDIA Dynamo Platform (v1.5.0) and SGLang on GPU A4 (4 x a4-highgpu-8g)
   # ==============================================================================
   apiVersion: resource.k8s.io/v1
   kind: ResourceClaimTemplate
   metadata:
     name: all-mrdma
   spec:
     spec:
       devices:
         requests:
           - name: req-mrdma
             exactly:
               deviceClassName: mrdma.google.com
               allocationMode: All
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: dynamo-workload-identity-discovery-role
     namespace: default
   rules:
     - apiGroups: [""]
       resources: ["endpoints", "pods"]
       verbs: ["get", "list", "watch"]
     - apiGroups: ["discovery.k8s.io"]
       resources: ["endpointslices"]
       verbs: ["get", "list", "watch"]
     - apiGroups: ["nvidia.com"]
       resources: ["dynamoworkermetadatas"]
       verbs: ["create", "get", "list", "watch", "update", "patch", "delete"]
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: dynamo-workload-identity-discovery-binding
     namespace: default
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: Role
     name: dynamo-workload-identity-discovery-role
   subjects:
     - kind: ServiceAccount
       name: workload-identity-k8s-sa
       namespace: default
   ---
   apiVersion: nvidia.com/v1alpha1
   kind: DynamoGraphDeployment
   metadata:
     name: qwen3-8-2-4t-a4-disagg
     annotations:
       nvidia.com/enable-grove: "true"
   spec:
     backendFramework: sglang
     services:
       # --------------------------------------------------------------------------
       # FRONTEND: Dynamo OpenAI-Compatible Router (`python3 -m dynamo.frontend`)
       # --------------------------------------------------------------------------
       Frontend:
         componentType: frontend
         replicas: 1
         resources:
           requests:
             cpu: "8"
             memory: "32Gi"
           limits:
             cpu: "8"
             memory: "32Gi"
         extraPodSpec:
           tolerations:
             - key: "nvidia.com/gpu"
               operator: "Exists"
               effect: "NoSchedule"
           mainContainer:
             image: nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.0
             command: ["/bin/bash", "-c"]
             args:
               - |
                 set -ex
                 exec python3 -m dynamo.frontend --http-port 8000
   
       # --------------------------------------------------------------------------
       # PREFILL WORKER: 2 x GPU A4 (`a4-highgpu-8g`, 16 x B200 GPUs, TP=16, EP=16)
       # --------------------------------------------------------------------------
       Prefill:
         componentType: worker
         subComponentType: prefill
         replicas: 1
         multinode:
           nodeCount: 2
         extraPodMetadata:
           annotations:
             gke-gcsfuse/volumes: "true"
         livenessProbe:
           httpGet:
             path: /live
             port: system
           periodSeconds: 60
           timeoutSeconds: 30
           failureThreshold: 60
         readinessProbe:
           httpGet:
             path: /health
             port: system
           initialDelaySeconds: 120
           periodSeconds: 30
           timeoutSeconds: 30
           failureThreshold: 60
         sharedMemory:
           size: 128Gi
         resources:
           requests:
             gpu: "8"
             cpu: "128"
             memory: "1500Gi"
           limits:
             gpu: "8"
             cpu: "128"
             memory: "2000Gi"
         envs:
           - name: MODEL_PATH
             value: "/model-weights/hf_models/models--nvidia--qwen3_8-2_4t-a95b-nvfp4"
           - name: DYN_SYSTEM_PORT
             value: "9090"
           - name: LD_LIBRARY_PATH
             value: "/usr/local/gib/lib64:/usr/local/nvidia/lib64"
           - name: GLOO_SOCKET_IFNAME
             value: "eth0"
           - name: NCCL_ALGO
             value: "Ring,Tree"
           - name: WITH_NVIDIA_PEERMEM
             value: "0"
         extraPodSpec:
           serviceAccountName: workload-identity-k8s-sa
           automountServiceAccountToken: true
           resourceClaims:
             - name: rdma
               resourceClaimTemplateName: all-mrdma
           nodeSelector:
             cloud.google.com/gke-accelerator: nvidia-b200
           tolerations:
             - key: "nvidia.com/gpu"
               operator: "Exists"
               effect: "NoSchedule"
           mainContainer:
             image: nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.0
             resources:
               claims:
                 - name: rdma
             startupProbe:
               httpGet:
                 path: /live
                 port: system
               initialDelaySeconds: 120
               periodSeconds: 30
               timeoutSeconds: 10
               failureThreshold: 120
             command: ["/bin/bash", "-c"]
             args:
               - |
                 set -ex
                 if [ -f /usr/local/gib/scripts/set_nccl_env.sh ]; then
                   . /usr/local/gib/scripts/set_nccl_env.sh
                 fi
   
                 echo "=== [Dynamo Prefill Worker] Launching dynamo.sglang ==="
                 exec python3 -m dynamo.sglang \
                   --dist-init-addr "${GROVE_PCSG_NAME}-${GROVE_PCSG_INDEX}-prefill-ldr-0.${GROVE_HEADLESS_SERVICE}.:29500" \
                   --model-path "${MODEL_PATH}" \
                   --served-model-name "nvidia/Qwen3.8-2.4T-A95B-NVFP4" \
                   --host 0.0.0.0 \
                   --disaggregation-mode prefill \
                   --disaggregation-bootstrap-port 30001 \
                   --tensor-parallel-size 16 \
                   --ep-size 16 \
                   --moe-runner-backend flashinfer_trtllm \
                   --attention-backend triton \
                   --disable-flashinfer-autotune \
                   --nnodes 2 \
                   --kv-cache-dtype fp8_e4m3 \
                   --context-length 16384 \
                   --mem-fraction-static 0.85 \
                   --enable-metrics
             volumeMounts:
               - name: library-dir-host
                 mountPath: /usr/local/nvidia
               - name: gib
                 mountPath: /usr/local/gib
               - name: gcs-model-dir
                 mountPath: /model-weights
                 readOnly: true
           volumes:
             - name: library-dir-host
               hostPath:
                 path: /home/kubernetes/bin/nvidia
             - name: gib
               hostPath:
                 path: /home/kubernetes/bin/gib
             - name: gcs-model-dir
               csi:
                 driver: gcsfuse.csi.storage.gke.io
                 readOnly: true
                 volumeAttributes:
                   bucketName: ${GCS_BUCKET}
                   mountOptions: "implicit-dirs,file-cache:max-size-mb:-1"
   
       # --------------------------------------------------------------------------
       # DECODE WORKER: 2 x GPU A4 (`a4-highgpu-8g`, 16 x B200 GPUs, TP=16, EP=16)
       # --------------------------------------------------------------------------
       Decode:
         componentType: worker
         subComponentType: decode
         replicas: 1
         multinode:
           nodeCount: 2
         extraPodMetadata:
           annotations:
             gke-gcsfuse/volumes: "true"
         livenessProbe:
           httpGet:
             path: /live
             port: system
           periodSeconds: 60
           timeoutSeconds: 30
           failureThreshold: 60
         readinessProbe:
           httpGet:
             path: /health
             port: system
           initialDelaySeconds: 120
           periodSeconds: 30
           timeoutSeconds: 30
           failureThreshold: 60
         sharedMemory:
           size: 128Gi
         resources:
           requests:
             gpu: "8"
             cpu: "128"
             memory: "1500Gi"
           limits:
             gpu: "8"
             cpu: "128"
             memory: "2000Gi"
         envs:
           - name: MODEL_PATH
             value: "/model-weights/hf_models/models--nvidia--qwen3_8-2_4t-a95b-nvfp4"
           - name: DYN_SYSTEM_PORT
             value: "9090"
           - name: LD_LIBRARY_PATH
             value: "/usr/local/gib/lib64:/usr/local/nvidia/lib64"
           - name: GLOO_SOCKET_IFNAME
             value: "eth0"
           - name: NCCL_ALGO
             value: "Ring,Tree"
           - name: WITH_NVIDIA_PEERMEM
             value: "0"
         extraPodSpec:
           serviceAccountName: workload-identity-k8s-sa
           automountServiceAccountToken: true
           resourceClaims:
             - name: rdma
               resourceClaimTemplateName: all-mrdma
           nodeSelector:
             cloud.google.com/gke-accelerator: nvidia-b200
           tolerations:
             - key: "nvidia.com/gpu"
               operator: "Exists"
               effect: "NoSchedule"
           mainContainer:
             image: nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.0
             resources:
               claims:
                 - name: rdma
             startupProbe:
               httpGet:
                 path: /live
                 port: system
               initialDelaySeconds: 120
               periodSeconds: 30
               timeoutSeconds: 10
               failureThreshold: 120
             command: ["/bin/bash", "-c"]
             args:
               - |
                 set -ex
                 if [ -f /usr/local/gib/scripts/set_nccl_env.sh ]; then
                   . /usr/local/gib/scripts/set_nccl_env.sh
                 fi
   
                 echo "=== [Dynamo Decode Worker] Launching dynamo.sglang ==="
                 exec python3 -m dynamo.sglang \
                   --dist-init-addr "${GROVE_PCSG_NAME}-${GROVE_PCSG_INDEX}-decode-ldr-0.${GROVE_HEADLESS_SERVICE}.:29500" \
                   --model-path "${MODEL_PATH}" \
                   --served-model-name "nvidia/Qwen3.8-2.4T-A95B-NVFP4" \
                   --host 0.0.0.0 \
                   --disaggregation-mode decode \
                   --disaggregation-bootstrap-port 30001 \
                   --tensor-parallel-size 16 \
                   --ep-size 16 \
                   --moe-runner-backend flashinfer_trtllm \
                   --attention-backend triton \
                   --nnodes 2 \
                   --kv-cache-dtype fp8_e4m3 \
                   --context-length 16384 \
                   --mem-fraction-static 0.85 \
                   --enable-metrics
             volumeMounts:
               - name: library-dir-host
                 mountPath: /usr/local/nvidia
               - name: gib
                 mountPath: /usr/local/gib
               - name: gcs-model-dir
                 mountPath: /model-weights
                 readOnly: true
           volumes:
             - name: library-dir-host
               hostPath:
                 path: /home/kubernetes/bin/nvidia
             - name: gib
               hostPath:
                 path: /home/kubernetes/bin/gib
             - name: gcs-model-dir
               csi:
                 driver: gcsfuse.csi.storage.gke.io
                 readOnly: true
                 volumeAttributes:
                   bucketName: ${GCS_BUCKET}
                   mountOptions: "implicit-dirs,file-cache:max-size-mb:-1"

2. 将您的 Cloud Storage 存储桶名称替换为 qwen3.8-2.4t-nvfp4-multihost-a4-gcsfuse.yaml，并将工作负载清单应用于集群：

   envsubst '${GCS_BUCKET}' < qwen3.8-2.4t-nvfp4-multihost-a4-gcsfuse.yaml | kubectl apply -f -

3. 如需监控权重加载和服务器初始化，请列出集群中的 Pod，并流式传输预填充领导者 Pod 和解码领导者 Pod 的日志：

   kubectl get pods
   POD_NAME="YOUR_POD_NAME"
   kubectl logs -f "${POD_NAME}"

   将 YOUR_POD_NAME 替换为 kubectl get pods 输出中的预填充领导者 Pod 或解码领导者 Pod 的名称。

   在冷启动时，通过 Cloud Storage FUSE 从 Cloud Storage 流式传输 2,400,000,000,000 个 NVFP4 模型权重（200 个分片，约 1.44 TB）并初始化多主机 SGLang 服务器大约需要 4 到 5 分钟。在发送任何请求之前，请等待前端 Pod 在 kubectl get pods 的 READY 列中显示 1/1，并且预填充和解码 Pod 显示 2/2：

   NAME                                                     READY   STATUS    RESTARTS   AGE
   qwen3-8-2-4t-a4-disagg-0-decode-0-decode-ldr-wz92h       2/2     Running   0          5m
   qwen3-8-2-4t-a4-disagg-0-decode-0-decode-wkr-fb8g6       2/2     Running   0          5m
   qwen3-8-2-4t-a4-disagg-0-frontend-jrx4s                  1/1     Running   0          5m
   qwen3-8-2-4t-a4-disagg-0-prefill-0-prefill-ldr-7nz25     2/2     Running   0          5m
   qwen3-8-2-4t-a4-disagg-0-prefill-0-prefill-wkr-7d5cw     2/2     Running   0          5m
   

发送推理请求

向已部署的 Qwen3.8-2.4T 服务发送对话补全请求：

1. 将本地端口 8000 转发到 qwen3-8-2-4t-a4-disagg-frontend 服务：

   kubectl port-forward svc/qwen3-8-2-4t-a4-disagg-frontend 8000:8000

   让端口转发在此终端会话中运行，或在后台运行该命令。

2. 在第二个终端会话中，向 /v1/chat/completions 端点发送对话补全请求：

   curl -s http://127.0.0.1:8000/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{
       "model": "nvidia/Qwen3.8-2.4T-A95B-NVFP4",
       "messages": [{"role": "user", "content": "Explain KV-cache disaggregation in one sentence."}],
       "max_tokens": 64
     }' | jq .

运行基准工作负载

使用 sglang.bench_serving，通过 Dynamo 前端 Pod 对已部署的 Qwen3.8-2.4T 服务器进行基准测试，配置为 1,024 个输入词元、1,024 个输出词元 (1k/1k)、256 个提示和 128 个最大并发数：

ROUTER_POD=$(kubectl get pod -l 'nvidia.com/dynamo-component-type=frontend' -o jsonpath='{.items[0].metadata.name}')

kubectl exec "${ROUTER_POD}" -- python3 -m sglang.bench_serving \
  --backend sglang-oai-chat \
  --base-url http://localhost:8000 \
  --model nvidia/Qwen3.8-2.4T-A95B-NVFP4 \
  --dataset-name random \
  --num-prompts 256 \
  --random-input-len 1024 \
  --random-output-len 1024 \
  --random-range-ratio 1.0 \
  --max-concurrency 128 | tee benchmark.log

基准测试完成后，查找包含如下指标的输出：

• Output token throughput
• Total Token throughput
• TTFT
• TPOT