在 A4 VM 上使用 Dynamo 和 SGLang，透過多主機 PD 分散式架構在 GKE 上提供 Qwen3.8-2.4T

設定環境變數

匯出本教學課程中使用的環境變數：

export PROJECT_ID="YOUR_PROJECT_ID"
export REGION="YOUR_REGION"
export ZONE="YOUR_ZONE"
export CLUSTER_NAME="YOUR_CLUSTER_NAME"
export RESERVATION_NAME="YOUR_RESERVATION_NAME"
export GCS_BUCKET="YOUR_BUCKET_NAME"
export HF_TOKEN="YOUR_HF_TOKEN"
export CLUSTER_TOOLKIT_VERSION="v1.104.0"
export DYNAMO_RELEASE_VERSION="1.5.0"

gcloud config set project "${PROJECT_ID}"
gcloud config set billing/quota_project "${PROJECT_ID}"

更改下列內容：

• YOUR_PROJECT_ID： Google Cloud專案的 ID。
• YOUR_REGION：A4 容量所在及要部署叢集的區域，例如 us-central1。
• YOUR_ZONE：A4 運算資源所在的可用區，例如 us-central1-b。
• YOUR_CLUSTER_NAME：GKE 叢集和 Cluster Toolkit 部署作業的名稱。請使用 20 個字元以內的名稱，確保產生的節點集區服務帳戶 ID (YOUR_CLUSTER_NAME-gke-np-sa) 不會超過 Identity and Access Management (IAM) 的 30 個字元限制。
• YOUR_RESERVATION_NAME：A4 預留項目的名稱，或格式為 RESERVATION_NAME/reservationBlocks/RESERVATION_BLOCK_NAME 的擴充預留區塊路徑。
• YOUR_BUCKET_NAME：Cloud Storage bucket 的全域不重複名稱，用於儲存 Cluster Toolkit 的 Terraform 狀態，以及暫存的 Qwen3.8-2.4T 模型權重。
• YOUR_HF_TOKEN：您的 Hugging Face 存取權杖。

建立 GKE 叢集

下載 Cluster Toolkit，並以 Standard 模式佈建 GKE 叢集，其中包含四個 a4-highgpu-8g 節點、GPUDirect RDMA 網路和 DRA 支援：

1. 登入 Google Cloud CLI，並為 Terraform 產生應用程式預設憑證：

   gcloud auth login
   gcloud auth application-default login

2. 下載並解壓縮預先建構的 Cluster Toolkit 套件，版本請依 CLUSTER_TOOLKIT_VERSION (v1.104.0) 指定：

   TAG="${CLUSTER_TOOLKIT_VERSION:-v1.104.0}"
   OS="linux"
   ARCH="amd64"
   mkdir -p cluster-toolkit && curl -L "https://github.com/GoogleCloudPlatform/cluster-toolkit/releases/download/${TAG}/gcluster_bundle_${OS}_${ARCH}.tgz" | tar -xz -C cluster-toolkit

   如果您在 macOS 或 arm64 工作站上執行這個指令，請將 OS="mac" 或 ARCH="arm64" 設為與平台相符。

3. 如要儲存 Cluster Toolkit 部署作業的 Terraform 狀態和暫存的模型權重，請在 A4 預留項目所在區域建立 Cloud Storage bucket，並啟用統一 bucket 層級存取權和物件版本管理功能：

   gcloud storage buckets create "gs://${GCS_BUCKET}" \
       --default-storage-class=STANDARD \
       --project="${PROJECT_ID}" \
       --location="${REGION}" \
       --uniform-bucket-level-access
   gcloud storage buckets update "gs://${GCS_BUCKET}" --versioning

4. 為四個 a4-highgpu-8g 節點建立 cluster-toolkit/examples/gke-a4/gke-a4-deployment.yaml 部署設定檔：

   cat <<EOF > cluster-toolkit/examples/gke-a4/gke-a4-deployment.yaml
   terraform_backend_defaults:
     type: gcs
     configuration:
       bucket: ${GCS_BUCKET}
   
   vars:
     project_id: ${PROJECT_ID}
     deployment_name: ${CLUSTER_NAME}
     region: ${REGION}
     zone: ${ZONE}
     static_node_count: 4
     authorized_cidr: 0.0.0.0/0
     reservation_affinity:
       consume_reservation_type: SPECIFIC_RESERVATION
       specific_reservations:
       - name: ${RESERVATION_NAME}
   EOF

   根據預設，部署檔案會設為使用特定預留項目。如果沒有特定預留項目，請先更新 cluster-toolkit/examples/gke-a4/gke-a4-deployment.yaml，改用其他消耗選項 (例如 Spot VM 或彈性啟動型 VM)，再部署叢集。如需設定操作說明，請參閱「GKE 消耗選項」。

5. 切換至 cluster-toolkit 目錄，使用 gcluster deploy 指令部署 GKE 叢集，然後返回目前使用的目錄：

   cd cluster-toolkit
   ./gcluster deploy \
       -d examples/gke-a4/gke-a4-deployment.yaml \
       examples/gke-a4/gke-a4.yaml \
       --auto-approve -w
   cd ..

   建立叢集約需 20 到 30 分鐘。如果您執行 ./gcluster deploy 時未使用 --auto-approve 旗標，請在系統提示時輸入 a，套用建議的變更。

6. 設定 kubectl 憑證，以便與叢集通訊：

   gcloud container clusters get-credentials "${CLUSTER_NAME}" \
       --location="${REGION}" \
       --project="${PROJECT_ID}"

安裝 KAI Scheduler 和 NVIDIA Dynamo

建立 Hugging Face Secret，並安裝 KAI 排程器和 NVIDIA Dynamo 平台 (採用 Grove Gang 排程)，為叢集做好準備：

1. 建立名為 hf-secret 的 Kubernetes Secret，用於儲存 Hugging Face 存取權杖：

   kubectl create secret generic hf-secret \
       --from-literal=hf_api_token="${HF_TOKEN}" \
       --dry-run=client -o yaml | kubectl apply -f -

2. 下載 NVIDIA Dynamo 平台 Helm 圖表 (v1.5.0)，安裝 KAI 排程器，並安裝已啟用 Grove 的 Dynamo 平台：

   RELEASE_VERSION="${DYNAMO_RELEASE_VERSION:-1.5.0}"
   helm pull "https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-platform-${RELEASE_VERSION}.tgz"
   tar -zxf "dynamo-platform-${RELEASE_VERSION}.tgz"
   
   helm upgrade --install kai-scheduler ./dynamo-platform/charts/kai-scheduler \
       --set scheduler.args.default-staleness-grace-period="-1s" \
       --set topologyMigration.enabled=false
   
   kubectl wait --for=condition=Available deployment/kai-operator --timeout=120s
   until kubectl get deployment/queue-controller >/dev/null 2>&1; do sleep 2; done
   kubectl wait --for=condition=Available deployment/queue-controller --timeout=120s
   
   helm upgrade --install dynamo-platform ./dynamo-platform \
       --set global.grove.install=true \
       --set global.grove.enabled=true \
       --set global.kai-scheduler.enabled=true

將模型權重暫存至 Cloud Storage

將 nvidia/Qwen3.8-2.4T-A95B-NVFP4 檢查點暫存至區域內 Cloud Storage bucket (gs://${GCS_BUCKET})，這樣推論 worker Pod 就能使用 Cloud Storage FUSE 掛接權重，不必在每次冷啟動時從 Hugging Face 下載：

1. 在建立 GKE 叢集時建立的 gs://${GCS_BUCKET} bucket，將 Storage 物件使用者角色 (roles/storage.objectUser) 授予繫結至 workload-identity-k8s-sa Kubernetes 服務帳戶的 Google 服務帳戶：

   GSA=$(kubectl get sa workload-identity-k8s-sa -o jsonpath='{.metadata.annotations.iam\.gke\.io/gcp-service-account}')
   gcloud storage buckets add-iam-policy-binding "gs://${GCS_BUCKET}" \
       --member="serviceAccount:${GSA}" \
       --role="roles/storage.objectUser"

2. 下載 qwen3.8-2.4t-nvfp4-stage-weights-job.yaml 和 qwen3.8-2.4t-nvfp4-multihost-a4-gcsfuse.yaml 資訊清單：

   curl -O https://raw.githubusercontent.com/GoogleCloudSamples/AIHypercomputerSamples/refs/heads/main/gpu/inference/qwen38-2t/qwen3.8-2.4t-nvfp4-stage-weights-job.yaml
   curl -O https://raw.githubusercontent.com/GoogleCloudSamples/AIHypercomputerSamples/refs/heads/main/gpu/inference/qwen38-2t/qwen3.8-2.4t-nvfp4-multihost-a4-gcsfuse.yaml

   或者，您也可以建立名為 qwen3.8-2.4t-nvfp4-stage-weights-job.yaml 的檔案，並在其中加入下列內容。這項索引工作會在四個 A4 節點上執行四個平行 Pod。每個 Pod 都會將大小平衡的 nvidia/Qwen3.8-2.4T-A95B-NVFP4 檢查點切片下載至節點的本機 NVMe SSD，並使用 gcloud storage rsync 上傳至 gs://${GCS_BUCKET}/hf_models/models--nvidia--qwen3_8-2_4t-a95b-nvfp4：

   # ==============================================================================
   # Stage nvidia/Qwen3.8-2.4T-A95B-NVFP4 (~1.44 TB, 210 files) from Hugging Face Hub
   # into an in-region Cloud Storage bucket for qwen3.8-2.4t-nvfp4-multihost-a4-gcsfuse.yaml
   #
   # Each Indexed Job pod downloads a size-balanced slice of the repo onto the node's
   # local NVMe SSD, then uploads that slice with `gcloud storage rsync`.
   # ==============================================================================
   apiVersion: batch/v1
   kind: Job
   metadata:
     name: qwen3-8-2-4t-stage-weights
   spec:
     completionMode: Indexed
     completions: 4
     parallelism: 4
     backoffLimitPerIndex: 2
     template:
       metadata:
         labels:
           app: qwen3-8-2-4t-stage-weights
       spec:
         restartPolicy: Never
         serviceAccountName: workload-identity-k8s-sa
         nodeSelector:
           cloud.google.com/gke-accelerator: nvidia-b200
         tolerations:
           - key: "nvidia.com/gpu"
             operator: "Exists"
             effect: "NoSchedule"
         topologySpreadConstraints:
           - maxSkew: 1
             topologyKey: kubernetes.io/hostname
             whenUnsatisfiable: DoNotSchedule
             labelSelector:
               matchLabels:
                 app: qwen3-8-2-4t-stage-weights
         initContainers:
           - name: download
             image: nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.0
             command: ["python3", "-c"]
             args:
               - |
                 import os, time
                 from huggingface_hub import HfApi, snapshot_download
                 repo = os.environ["REPO_ID"]
                 n = int(os.environ["NUM_SHARDS"])
                 i = int(os.environ["JOB_COMPLETION_INDEX"])
                 info = HfApi().model_info(repo, files_metadata=True)
                 files = sorted(((s.size or 0, s.rfilename) for s in info.siblings), reverse=True)
                 bins = [[0, []] for _ in range(n)]
                 for size, name in files:
                     b = min(bins, key=lambda x: x[0])
                     b[0] += size
                     b[1].append(name)
                 size, mine = bins[i]
                 print(f"shard {i}/{n}: {len(mine)} files, {size / 1e9:.1f} GB", flush=True)
                 for attempt in range(1, 7):
                     try:
                         snapshot_download(repo, local_dir="/ssd/model", allow_patterns=mine, max_workers=16)
                         break
                     except Exception as e:
                         print(f"shard {i}/{n}: attempt {attempt} failed: {e!r}", flush=True)
                         if attempt == 6:
                             raise
                         time.sleep(30)
                 print(f"shard {i}/{n}: download complete", flush=True)
             env:
               - name: REPO_ID
                 value: "nvidia/Qwen3.8-2.4T-A95B-NVFP4"
               - name: NUM_SHARDS
                 value: "4"
               - name: HF_HOME
                 value: "/ssd/hf"
               - name: HF_HUB_DISABLE_XET
                 value: "1"
               - name: HF_TOKEN
                 valueFrom:
                   secretKeyRef:
                     name: hf-secret
                     key: hf_api_token
                     optional: true
             resources:
               requests:
                 cpu: "16"
                 memory: "64Gi"
                 ephemeral-storage: "450Gi"
               limits:
                 cpu: "16"
                 memory: "64Gi"
                 ephemeral-storage: "700Gi"
             volumeMounts:
               - name: ssd
                 mountPath: /ssd
         containers:
           - name: upload
             image: gcr.io/google.com/cloudsdktool/google-cloud-cli:slim
             command: ["/bin/bash", "-c"]
             args:
               - |
                 set -euo pipefail
                 gcloud storage rsync --recursive --exclude='^\.cache/.*' /ssd/model "${DEST}"
                 echo "shard ${JOB_COMPLETION_INDEX}: upload complete"
             env:
               - name: DEST
                 value: "gs://${GCS_BUCKET}/hf_models/models--nvidia--qwen3_8-2_4t-a95b-nvfp4"
             resources:
               requests:
                 cpu: "16"
                 memory: "32Gi"
               limits:
                 cpu: "16"
                 memory: "32Gi"
             volumeMounts:
               - name: ssd
                 mountPath: /ssd
         volumes:
           - name: ssd
             emptyDir: {}

3. 將 Cloud Storage bucket 名稱代入 qwen3.8-2.4t-nvfp4-stage-weights-job.yaml，啟動暫存工作，等待所有四個 Pod 完成權重上傳，然後刪除已完成的工作：

   envsubst '${GCS_BUCKET}' < qwen3.8-2.4t-nvfp4-stage-weights-job.yaml | kubectl apply -f -
   
   kubectl wait --for=condition=complete job/qwen3-8-2-4t-stage-weights --timeout=2h
   
   kubectl delete job qwen3-8-2-4t-stage-weights

   將模型權重下載至本機 NVMe SSD，並上傳至 Cloud Storage 約需 25 分鐘。

4. 選用：在暫存工作執行期間，在第二個終端機工作階段中串流暫存 Pod 的記錄，監控下載進度：

   POD_NAME="YOUR_POD_NAME"
   kubectl logs -f "${POD_NAME}"

   將 YOUR_POD_NAME 替換為 kubectl get pods 中其中一個 qwen3-8-2-4t-stage-weights Pod 的名稱。

部署多主機解構式 SGLang 伺服器

工作負載資訊清單 (qwen3.8-2.4t-nvfp4-multihost-a4-gcsfuse.yaml) 定義了 Kubernetes ResourceClaimTemplate (all-mrdma)，透過 DRA、workload-identity-k8s-sa 服務帳戶的角色和 RoleBinding 資源，以及啟用 Grove Gang 排程的 NVIDIA Dynamo DynamoGraphDeployment (qwen3-8-2-4t-a4-disagg)，分配節點的多 NIC RDMA 裝置 (mrdma.google.com)。部署作業包含下列項目：

• 前端 (Frontend)：執行一個路由器副本，接受用戶端要求、探索已註冊的預先填入和解碼 worker，並協調分散式要求轉送。

• 預先填入 worker (Prefill)：在兩個 a4-highgpu-8g 節點上執行雙節點 worker 集區，共有 16 個 NVIDIA B200 GPU。每個 Pod 都會使用 Cloud Storage FUSE CSI 驅動程式 (gcsfuse.csi.storage.gke.io) 搭配本機 NVMe SSD 檔案快取，從 Cloud Storage bucket 的 /model-weights 掛接暫存的 nvidia/Qwen3.8-2.4T-A95B-NVFP4 權重，並提供 NCCL/gIB 環境，然後在 --disaggregation-mode prefill 中啟動 python3 -m dynamo.sglang，並使用下列旗標：

  • --served-model-name "nvidia/Qwen3.8-2.4T-A95B-NVFP4"
  • --tensor-parallel-size 16
  • --ep-size 16
  • --nnodes 2
  • --moe-runner-backend flashinfer_trtllm
  • --attention-backend triton
  • --kv-cache-dtype fp8_e4m3
  • --disable-flashinfer-autotune

• 解碼 worker (Decode)：在剩餘的兩個 a4-highgpu-8g 節點上執行雙節點 worker 集區，共有 16 個 NVIDIA B200 GPU。每個 Pod 都會透過 Cloud Storage FUSE CSI 驅動程式，從 Cloud Storage 的 /model-weights 掛接暫存的模型權重，並使用下列旗標在 --disaggregation-mode decode 中啟動 python3 -m dynamo.sglang：

  • --served-model-name "nvidia/Qwen3.8-2.4T-A95B-NVFP4"
  • --tensor-parallel-size 16
  • --ep-size 16
  • --nnodes 2
  • --moe-runner-backend flashinfer_trtllm
  • --attention-backend triton
  • --kv-cache-dtype fp8_e4m3

部署及監控分解推論伺服器：

1. 查看先前下載的 qwen3.8-2.4t-nvfp4-multihost-a4-gcsfuse.yaml 工作負載資訊清單，或建立名為 qwen3.8-2.4t-nvfp4-multihost-a4-gcsfuse.yaml 的檔案，並加入下列內容：

   # ==============================================================================
   # Multihost Disaggregated Serving of Qwen3.8-2.4T-A95B (NVFP4 MoE) on GKE
   # using NVIDIA Dynamo Platform (v1.5.0) and SGLang on GPU A4 (4 x a4-highgpu-8g)
   # ==============================================================================
   apiVersion: resource.k8s.io/v1
   kind: ResourceClaimTemplate
   metadata:
     name: all-mrdma
   spec:
     spec:
       devices:
         requests:
           - name: req-mrdma
             exactly:
               deviceClassName: mrdma.google.com
               allocationMode: All
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: dynamo-workload-identity-discovery-role
     namespace: default
   rules:
     - apiGroups: [""]
       resources: ["endpoints", "pods"]
       verbs: ["get", "list", "watch"]
     - apiGroups: ["discovery.k8s.io"]
       resources: ["endpointslices"]
       verbs: ["get", "list", "watch"]
     - apiGroups: ["nvidia.com"]
       resources: ["dynamoworkermetadatas"]
       verbs: ["create", "get", "list", "watch", "update", "patch", "delete"]
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: dynamo-workload-identity-discovery-binding
     namespace: default
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: Role
     name: dynamo-workload-identity-discovery-role
   subjects:
     - kind: ServiceAccount
       name: workload-identity-k8s-sa
       namespace: default
   ---
   apiVersion: nvidia.com/v1alpha1
   kind: DynamoGraphDeployment
   metadata:
     name: qwen3-8-2-4t-a4-disagg
     annotations:
       nvidia.com/enable-grove: "true"
   spec:
     backendFramework: sglang
     services:
       # --------------------------------------------------------------------------
       # FRONTEND: Dynamo OpenAI-Compatible Router (`python3 -m dynamo.frontend`)
       # --------------------------------------------------------------------------
       Frontend:
         componentType: frontend
         replicas: 1
         resources:
           requests:
             cpu: "8"
             memory: "32Gi"
           limits:
             cpu: "8"
             memory: "32Gi"
         extraPodSpec:
           tolerations:
             - key: "nvidia.com/gpu"
               operator: "Exists"
               effect: "NoSchedule"
           mainContainer:
             image: nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.0
             command: ["/bin/bash", "-c"]
             args:
               - |
                 set -ex
                 exec python3 -m dynamo.frontend --http-port 8000
   
       # --------------------------------------------------------------------------
       # PREFILL WORKER: 2 x GPU A4 (`a4-highgpu-8g`, 16 x B200 GPUs, TP=16, EP=16)
       # --------------------------------------------------------------------------
       Prefill:
         componentType: worker
         subComponentType: prefill
         replicas: 1
         multinode:
           nodeCount: 2
         extraPodMetadata:
           annotations:
             gke-gcsfuse/volumes: "true"
         livenessProbe:
           httpGet:
             path: /live
             port: system
           periodSeconds: 60
           timeoutSeconds: 30
           failureThreshold: 60
         readinessProbe:
           httpGet:
             path: /health
             port: system
           initialDelaySeconds: 120
           periodSeconds: 30
           timeoutSeconds: 30
           failureThreshold: 60
         sharedMemory:
           size: 128Gi
         resources:
           requests:
             gpu: "8"
             cpu: "128"
             memory: "1500Gi"
           limits:
             gpu: "8"
             cpu: "128"
             memory: "2000Gi"
         envs:
           - name: MODEL_PATH
             value: "/model-weights/hf_models/models--nvidia--qwen3_8-2_4t-a95b-nvfp4"
           - name: DYN_SYSTEM_PORT
             value: "9090"
           - name: LD_LIBRARY_PATH
             value: "/usr/local/gib/lib64:/usr/local/nvidia/lib64"
           - name: GLOO_SOCKET_IFNAME
             value: "eth0"
           - name: NCCL_ALGO
             value: "Ring,Tree"
           - name: WITH_NVIDIA_PEERMEM
             value: "0"
         extraPodSpec:
           serviceAccountName: workload-identity-k8s-sa
           automountServiceAccountToken: true
           resourceClaims:
             - name: rdma
               resourceClaimTemplateName: all-mrdma
           nodeSelector:
             cloud.google.com/gke-accelerator: nvidia-b200
           tolerations:
             - key: "nvidia.com/gpu"
               operator: "Exists"
               effect: "NoSchedule"
           mainContainer:
             image: nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.0
             resources:
               claims:
                 - name: rdma
             startupProbe:
               httpGet:
                 path: /live
                 port: system
               initialDelaySeconds: 120
               periodSeconds: 30
               timeoutSeconds: 10
               failureThreshold: 120
             command: ["/bin/bash", "-c"]
             args:
               - |
                 set -ex
                 if [ -f /usr/local/gib/scripts/set_nccl_env.sh ]; then
                   . /usr/local/gib/scripts/set_nccl_env.sh
                 fi
   
                 echo "=== [Dynamo Prefill Worker] Launching dynamo.sglang ==="
                 exec python3 -m dynamo.sglang \
                   --dist-init-addr "${GROVE_PCSG_NAME}-${GROVE_PCSG_INDEX}-prefill-ldr-0.${GROVE_HEADLESS_SERVICE}.:29500" \
                   --model-path "${MODEL_PATH}" \
                   --served-model-name "nvidia/Qwen3.8-2.4T-A95B-NVFP4" \
                   --host 0.0.0.0 \
                   --disaggregation-mode prefill \
                   --disaggregation-bootstrap-port 30001 \
                   --tensor-parallel-size 16 \
                   --ep-size 16 \
                   --moe-runner-backend flashinfer_trtllm \
                   --attention-backend triton \
                   --disable-flashinfer-autotune \
                   --nnodes 2 \
                   --kv-cache-dtype fp8_e4m3 \
                   --context-length 16384 \
                   --mem-fraction-static 0.85 \
                   --enable-metrics
             volumeMounts:
               - name: library-dir-host
                 mountPath: /usr/local/nvidia
               - name: gib
                 mountPath: /usr/local/gib
               - name: gcs-model-dir
                 mountPath: /model-weights
                 readOnly: true
           volumes:
             - name: library-dir-host
               hostPath:
                 path: /home/kubernetes/bin/nvidia
             - name: gib
               hostPath:
                 path: /home/kubernetes/bin/gib
             - name: gcs-model-dir
               csi:
                 driver: gcsfuse.csi.storage.gke.io
                 readOnly: true
                 volumeAttributes:
                   bucketName: ${GCS_BUCKET}
                   mountOptions: "implicit-dirs,file-cache:max-size-mb:-1"
   
       # --------------------------------------------------------------------------
       # DECODE WORKER: 2 x GPU A4 (`a4-highgpu-8g`, 16 x B200 GPUs, TP=16, EP=16)
       # --------------------------------------------------------------------------
       Decode:
         componentType: worker
         subComponentType: decode
         replicas: 1
         multinode:
           nodeCount: 2
         extraPodMetadata:
           annotations:
             gke-gcsfuse/volumes: "true"
         livenessProbe:
           httpGet:
             path: /live
             port: system
           periodSeconds: 60
           timeoutSeconds: 30
           failureThreshold: 60
         readinessProbe:
           httpGet:
             path: /health
             port: system
           initialDelaySeconds: 120
           periodSeconds: 30
           timeoutSeconds: 30
           failureThreshold: 60
         sharedMemory:
           size: 128Gi
         resources:
           requests:
             gpu: "8"
             cpu: "128"
             memory: "1500Gi"
           limits:
             gpu: "8"
             cpu: "128"
             memory: "2000Gi"
         envs:
           - name: MODEL_PATH
             value: "/model-weights/hf_models/models--nvidia--qwen3_8-2_4t-a95b-nvfp4"
           - name: DYN_SYSTEM_PORT
             value: "9090"
           - name: LD_LIBRARY_PATH
             value: "/usr/local/gib/lib64:/usr/local/nvidia/lib64"
           - name: GLOO_SOCKET_IFNAME
             value: "eth0"
           - name: NCCL_ALGO
             value: "Ring,Tree"
           - name: WITH_NVIDIA_PEERMEM
             value: "0"
         extraPodSpec:
           serviceAccountName: workload-identity-k8s-sa
           automountServiceAccountToken: true
           resourceClaims:
             - name: rdma
               resourceClaimTemplateName: all-mrdma
           nodeSelector:
             cloud.google.com/gke-accelerator: nvidia-b200
           tolerations:
             - key: "nvidia.com/gpu"
               operator: "Exists"
               effect: "NoSchedule"
           mainContainer:
             image: nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.0
             resources:
               claims:
                 - name: rdma
             startupProbe:
               httpGet:
                 path: /live
                 port: system
               initialDelaySeconds: 120
               periodSeconds: 30
               timeoutSeconds: 10
               failureThreshold: 120
             command: ["/bin/bash", "-c"]
             args:
               - |
                 set -ex
                 if [ -f /usr/local/gib/scripts/set_nccl_env.sh ]; then
                   . /usr/local/gib/scripts/set_nccl_env.sh
                 fi
   
                 echo "=== [Dynamo Decode Worker] Launching dynamo.sglang ==="
                 exec python3 -m dynamo.sglang \
                   --dist-init-addr "${GROVE_PCSG_NAME}-${GROVE_PCSG_INDEX}-decode-ldr-0.${GROVE_HEADLESS_SERVICE}.:29500" \
                   --model-path "${MODEL_PATH}" \
                   --served-model-name "nvidia/Qwen3.8-2.4T-A95B-NVFP4" \
                   --host 0.0.0.0 \
                   --disaggregation-mode decode \
                   --disaggregation-bootstrap-port 30001 \
                   --tensor-parallel-size 16 \
                   --ep-size 16 \
                   --moe-runner-backend flashinfer_trtllm \
                   --attention-backend triton \
                   --nnodes 2 \
                   --kv-cache-dtype fp8_e4m3 \
                   --context-length 16384 \
                   --mem-fraction-static 0.85 \
                   --enable-metrics
             volumeMounts:
               - name: library-dir-host
                 mountPath: /usr/local/nvidia
               - name: gib
                 mountPath: /usr/local/gib
               - name: gcs-model-dir
                 mountPath: /model-weights
                 readOnly: true
           volumes:
             - name: library-dir-host
               hostPath:
                 path: /home/kubernetes/bin/nvidia
             - name: gib
               hostPath:
                 path: /home/kubernetes/bin/gib
             - name: gcs-model-dir
               csi:
                 driver: gcsfuse.csi.storage.gke.io
                 readOnly: true
                 volumeAttributes:
                   bucketName: ${GCS_BUCKET}
                   mountOptions: "implicit-dirs,file-cache:max-size-mb:-1"

2. 將 Cloud Storage bucket 名稱替換為 qwen3.8-2.4t-nvfp4-multihost-a4-gcsfuse.yaml，並將工作負載資訊清單套用至叢集：

   envsubst '${GCS_BUCKET}' < qwen3.8-2.4t-nvfp4-multihost-a4-gcsfuse.yaml | kubectl apply -f -

3. 如要監控權重載入和伺服器初始化作業，請列出叢集中的 Pod，並串流預先填入前導 Pod 和解碼前導 Pod 的記錄：

   kubectl get pods
   POD_NAME="YOUR_POD_NAME"
   kubectl logs -f "${POD_NAME}"

   將 YOUR_POD_NAME 替換為 kubectl get pods 輸出內容中的預先填補前導 Pod 或解碼前導 Pod 名稱。

   冷啟動時，系統需要約 4 到 5 分鐘，才能透過 Cloud Storage FUSE 從 Cloud Storage 串流 2,400,000,000,000 個 NVFP4 模型權重 (200 個 shard 約 1.44 TB)，並初始化多主機 SGLang 伺服器。請先確認前端 Pod 顯示 1/1，且預填和解碼 Pod 在 kubectl get pods 的 READY 欄中顯示 2/2，再傳送要求：

   NAME                                                     READY   STATUS    RESTARTS   AGE
   qwen3-8-2-4t-a4-disagg-0-decode-0-decode-ldr-wz92h       2/2     Running   0          5m
   qwen3-8-2-4t-a4-disagg-0-decode-0-decode-wkr-fb8g6       2/2     Running   0          5m
   qwen3-8-2-4t-a4-disagg-0-frontend-jrx4s                  1/1     Running   0          5m
   qwen3-8-2-4t-a4-disagg-0-prefill-0-prefill-ldr-7nz25     2/2     Running   0          5m
   qwen3-8-2-4t-a4-disagg-0-prefill-0-prefill-wkr-7d5cw     2/2     Running   0          5m
   

傳送推論要求

向已部署的 Qwen3.8-2.4T 服務傳送對話完成要求：

1. 將本機通訊埠 8000 轉送至 qwen3-8-2-4t-a4-disagg-frontend 服務：

   kubectl port-forward svc/qwen3-8-2-4t-a4-disagg-frontend 8000:8000

   請讓通訊埠轉送功能在這個終端機工作階段中執行，或在背景執行指令。

2. 在第二個終端機工作階段中，將對話完成要求傳送至 /v1/chat/completions 端點：

   curl -s http://127.0.0.1:8000/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{
       "model": "nvidia/Qwen3.8-2.4T-A95B-NVFP4",
       "messages": [{"role": "user", "content": "Explain KV-cache disaggregation in one sentence."}],
       "max_tokens": 64
     }' | jq .

執行基準工作負載

使用 sglang.bench_serving，從 Dynamo 前端 Pod 基準測試已部署的 Qwen3.8-2.4T 伺服器，設定為 1,024 個輸入權杖、1,024 個輸出權杖 (1k/1k)、256 個提示，並行數量上限為 128：

ROUTER_POD=$(kubectl get pod -l 'nvidia.com/dynamo-component-type=frontend' -o jsonpath='{.items[0].metadata.name}')

kubectl exec "${ROUTER_POD}" -- python3 -m sglang.bench_serving \
  --backend sglang-oai-chat \
  --base-url http://localhost:8000 \
  --model nvidia/Qwen3.8-2.4T-A95B-NVFP4 \
  --dataset-name random \
  --num-prompts 256 \
  --random-input-len 1024 \
  --random-output-len 1024 \
  --random-range-ratio 1.0 \
  --max-concurrency 128 | tee benchmark.log

基準測試完成後，請查看包含下列指標的輸出內容：

• Output token throughput
• Total Token throughput
• TTFT
• TPOT