Serve Qwen3-Coder-480B on GKE with multi-host SGLang TorchTPU on TPU7x accelerators

Set up your environment variables

Export the environment variables that you use throughout this tutorial:

export CLUSTER_NAME="YOUR_CLUSTER_NAME"
export PROJECT_ID="YOUR_PROJECT_ID"
export REGION="YOUR_REGION"
export ZONE="YOUR_ZONE"
export RESERVATION="YOUR_RESERVATION_NAME"
export HF_TOKEN="YOUR_HF_TOKEN"
export SGLANG_IMAGE="lmsysorg/sglang:v0.1.0-tpu"

Replace the following:

• YOUR_CLUSTER_NAME: the name of your GKE cluster.
• YOUR_PROJECT_ID: the ID of your Google Cloud project.
• YOUR_REGION: the region where you want to deploy your cluster, such as us-central1.
• YOUR_ZONE: the zone where you want to deploy your TPU7x node pools, such as us-central1-c.
• YOUR_RESERVATION_NAME: if you're using a reservation, the name of your TPU7x reservation.
• YOUR_HF_TOKEN: your Hugging Face access token.
• YOUR_SGLANG_TPU_IMAGE: if you're using a custom SGLang TorchTPU container image, the URI of your image; otherwise, the default SGLang TorchTPU image.

Create the GKE cluster

To create a GKE cluster in Standard mode and attach two multi-host TPU7x node pools for prefill and decode disaggregation, complete the following steps:

1. Create a regional GKE cluster that uses e2-standard-4 nodes for the default node pool, which provides sufficient CPU and memory resources for the LeaderWorkerSet controller and the SGLang disaggregation router:

   gcloud container clusters create $CLUSTER_NAME \
     --project=$PROJECT_ID \
     --location=$REGION \
     --machine-type=e2-standard-4

2. Create a high-throughput workload placement policy with a 2x2x2 accelerator topology, which is required for multi-host TPU7x node pools:

   gcloud compute resource-policies create workload-policy tpu7x-2x2x2-workload-policy \
       --type=HIGH_THROUGHPUT \
       --accelerator-topology=2x2x2 \
       --project=${PROJECT_ID} \
       --region=${REGION}

3. Create two multi-host TPU7x node pools (tpu7x-pool-0 and tpu7x-pool-1), each containing two tpu7x-standard-4t nodes (eight TPU7x chips in a 2x2x2 topology per node pool, for a total of 16 TPU7x chips):

   gcloud container node-pools create tpu7x-pool-0 \
       --cluster=$CLUSTER_NAME \
       --location=$REGION \
       --node-locations=$ZONE \
       --machine-type=tpu7x-standard-4t \
       --num-nodes=2 \
       --placement-policy=tpu7x-2x2x2-workload-policy \
       --reservation-affinity=specific \
       --reservation=$RESERVATION \
       --project=$PROJECT_ID
   
   gcloud container node-pools create tpu7x-pool-1 \
       --cluster=$CLUSTER_NAME \
       --location=$REGION \
       --node-locations=$ZONE \
       --machine-type=tpu7x-standard-4t \
       --num-nodes=2 \
       --placement-policy=tpu7x-2x2x2-workload-policy \
       --reservation-affinity=specific \
       --reservation=$RESERVATION \
       --project=$PROJECT_ID

   Instead of using a specific reservation, you can also create the node pools using Spot capacity or Flex-start VMs by passing the following flags:

   • Spot VMs: replace --reservation-affinity=specific and --reservation=${RESERVATION} with --spot to use Spot capacity. Instances using Spot capacity might take time to be created after the node pool is created, and they might be preempted during workload runs.
   • Flex-start VMs: replace --num-nodes=2, --reservation-affinity=specific, and --reservation=${RESERVATION} with --flex-start, --enable-autoscaling, --num-nodes=0, --min-nodes=0, and --max-nodes=2 to provision the node pool using the Flex-start consumption option. For more information, see Run a small batch workload with TPUs and Flex-start VMs.

4. Configure kubectl credentials to communicate with your cluster:

   gcloud container clusters get-credentials $CLUSTER_NAME --location=${REGION} --project=${PROJECT_ID}

Configure credentials and install LeaderWorkerSet

To prepare your cluster for the multi-host SGLang deployment, complete the following steps:

1. Create a Kubernetes Secret named hf-secret to store your Hugging Face access token:

   kubectl create secret generic hf-secret \
       --from-literal=hf_api_token=${HF_TOKEN}

2. Install the LeaderWorkerSet CustomResourceDefinition and controller on your cluster:

   kubectl apply --server-side -k github.com/kubernetes-sigs/lws/config/default?ref=main
   echo "Waiting for LWS controller to be ready..."
   kubectl wait --for=condition=Available --timeout=300s deployment/lws-controller-manager -n lws-system

Deploy the multi-host SGLang server

The workload manifest (qwen3-coder-480b-multihost-disagg-v7x.yaml) defines the following resources to coordinate disaggregated prefill and decode serving across the two 2x2x2 TPU7x node pools:

• StorageClass (hyperdisk-balanced-tpu): provisions ephemeral 700 GiB ReadWriteOnce volumes backed by Google Cloud Hyperdisk Balanced storage. Each of the four prefill and decode Pods mounts its own volume at /data to store the downloaded Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 checkpoint locally.
• Prefill pool (sglang-prefill-service and sglang-tpu-prefill): runs a LeaderWorkerSet of size: 2 (one leader Pod and one worker Pod) pinned exclusively to one 2x2x2 TPU7x node pool (eight TPU chips, which provide 16 TensorCores). The LeaderWorkerSet uses the leaderworkerset.sigs.k8s.io/exclusive-topology annotation set to "cloud.google.com/gke-nodepool". The Pods launch sglang.launch_server with the --disaggregation-mode=prefill, --tensor-parallel-size=16, and --ep-size=16 flags, exposing HTTP port 8000 and bootstrap port 8998.
• Decode pool (sglang-decode-service and sglang-tpu-decode): runs a LeaderWorkerSet of size: 2 pinned to the second 2x2x2 TPU7x node pool (eight TPU chips, which provide 16 TensorCores). The LeaderWorkerSet launches sglang.launch_server with the --disaggregation-mode=decode, --tensor-parallel-size=16, and --ep-size=16 flags.
• Disaggregation router (sglang-router and sglang-service): runs a CPU Deployment that executes the sglang_router.launch_router module with the --pd-disaggregation flag and exposes a ClusterIP Service on port 8000. The router directs prompts and coordinates KV cache transfer between the sglang-prefill-service and sglang-decode-service services.

To deploy and monitor the multi-host SGLang servers, complete the following steps:

1. Download the workload manifest:

   curl -LO https://raw.githubusercontent.com/GoogleCloudSamples/AIHypercomputerSamples/main/tpu/inference/qwen3-coder/qwen3-coder-480b-multihost-disagg-v7x.yaml

   Alternatively, create a file named qwen3-coder-480b-multihost-disagg-v7x.yaml with the following content:

   # ==============================================================================
   # Serve Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 on GKE / Ironwood (TPU v7x)
   # using SGLang TorchTPU with Prefill/Decode Disaggregation, multi-host
   # ==============================================================================
   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   metadata:
     name: hyperdisk-balanced-tpu
   provisioner: pd.csi.storage.gke.io
   volumeBindingMode: WaitForFirstConsumer
   allowVolumeExpansion: true
   parameters:
     type: hyperdisk-balanced
   ---
   # ==============================================================================
   # PREFILL POOL: Internal ClusterIP Service exposing SGLang Prefill Endpoint
   # ==============================================================================
   apiVersion: v1
   kind: Service
   metadata:
     name: sglang-prefill-service
     labels:
       app: sglang-prefill
   spec:
     type: ClusterIP
     selector:
       app: sglang-prefill
       role: leader
     ports:
       - name: http
         protocol: TCP
         port: 8000
         targetPort: 8000
       - name: bootstrap
         protocol: TCP
         port: 8998
         targetPort: 8998
   ---
   # ==============================================================================
   # PREFILL POOL: LeaderWorkerSet (TPU v7x, 1 slice = 2 nodes, 8 chips / 16 TensorCores)
   # ==============================================================================
   apiVersion: leaderworkerset.x-k8s.io/v1
   kind: LeaderWorkerSet
   metadata:
     name: sglang-tpu-prefill
     annotations:
       leaderworkerset.sigs.k8s.io/exclusive-topology: "cloud.google.com/gke-nodepool"
     labels:
       app: sglang-prefill
   spec:
     replicas: 1
     rolloutStrategy:
       type: RollingUpdate
     leaderWorkerTemplate:
       size: 2
       restartPolicy: RecreateGroupOnPodRestart
       leaderTemplate:
         metadata:
           labels:
             app: sglang-prefill
             role: leader
         spec: &prefill-pod-spec
           securityContext:
             fsGroup: 65534
           nodeSelector:
             cloud.google.com/gke-tpu-accelerator: tpu7x
             cloud.google.com/gke-tpu-topology: 2x2x2
           tolerations:
             - key: "google.com/tpu"
               operator: "Exists"
               effect: "NoSchedule"
           containers:
             - name: sglang
               image: $SGLANG_IMAGE
               command:
                 - /bin/bash
                 - -c
                 - |
                   set -ex
                   ulimit -c 0
                   LEADER_POD="${LWS_LEADER_ADDRESS%%.*}"
                   DNS_SUFFIX="${LWS_LEADER_ADDRESS#*.}"
                   HOSTS="${LWS_LEADER_ADDRESS}"
                   for ((i = 1; i < LWS_GROUP_SIZE; i++)); do
                     HOSTS="${HOSTS},${LEADER_POD}-${i}.${DNS_SUFFIX}"
                   done
                   export TPU_WORKER_HOSTNAMES="${HOSTS}"
                   export TPU_WORKER_ID="${LWS_WORKER_INDEX}"
                   export TPU_TOPOLOGY="${TPU_TOPOLOGY:-2x2x2}"
                   echo "=== [prefill rank ${LWS_WORKER_INDEX}/${LWS_GROUP_SIZE}] hosts=${TPU_WORKER_HOSTNAMES} ==="
                   exec python3 -m sglang.launch_server \
                     --device=tpu \
                     --host=0.0.0.0 \
                     --port=8000 \
                     --model-path=Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \
                     --tensor-parallel-size=16 \
                     --ep-size=16 \
                     --nnodes="${LWS_GROUP_SIZE}" \
                     --node-rank="${LWS_WORKER_INDEX}" \
                     --dist-init-addr="${LWS_LEADER_ADDRESS}:20000" \
                     --attention-backend=pallas_tpu_rpa_batched \
                     --mem-fraction-static=0.8 \
                     --max-running-requests=256 \
                     --chunked-prefill-size=8192 \
                     --max-total-tokens=786432 \
                     --context-length=9216 \
                     --enable-torch-compile \
                     --torch-compile-max-bs=256 \
                     --tpu-compile-tracer=direct \
                     --sampling-defaults=openai \
                     --enable-fused-qk-norm-rope \
                     --disaggregation-mode=prefill \
                     --disaggregation-transfer-backend=tpu \
                     --download-dir=/data \
                     --tpu-compile-tokens=8192 \
                     --disable-radix-cache
               env:
                 - name: HF_TOKEN
                   valueFrom:
                     secretKeyRef:
                       name: hf-secret
                       key: hf_api_token
                       optional: true
               resources:
                 requests:
                   cpu: "16"
                   memory: 128Gi
                   google.com/tpu: "4"
                 limits:
                   memory: 768Gi
                   google.com/tpu: "4"
               ports:
                 - containerPort: 8000
                   name: http
                 - containerPort: 8998
                   name: bootstrap
               startupProbe:
                 httpGet:
                   path: /health
                   port: 8000
                 initialDelaySeconds: 60
                 periodSeconds: 15
                 timeoutSeconds: 10
                 failureThreshold: 236
               readinessProbe:
                 httpGet:
                   path: /health
                   port: 8000
                 periodSeconds: 10
                 timeoutSeconds: 5
                 failureThreshold: 3
               volumeMounts:
                 - name: model-cache
                   mountPath: /data
                 - name: dshm
                   mountPath: /dev/shm
           volumes:
             - name: dshm
               emptyDir:
                 medium: Memory
                 sizeLimit: 32Gi
             - name: model-cache
               ephemeral:
                 volumeClaimTemplate:
                   spec:
                     accessModes: ["ReadWriteOnce"]
                     storageClassName: hyperdisk-balanced-tpu
                     resources:
                       requests:
                         storage: 700Gi
       workerTemplate:
         metadata:
           labels:
             app: sglang-prefill
             role: worker
         spec: *prefill-pod-spec
   ---
   # ==============================================================================
   # DECODE POOL: Internal ClusterIP Service exposing SGLang Decode Endpoint
   # ==============================================================================
   apiVersion: v1
   kind: Service
   metadata:
     name: sglang-decode-service
     labels:
       app: sglang-decode
   spec:
     type: ClusterIP
     selector:
       app: sglang-decode
       role: leader
     ports:
       - name: http
         protocol: TCP
         port: 8000
         targetPort: 8000
       - name: bootstrap
         protocol: TCP
         port: 8998
         targetPort: 8998
   ---
   # ==============================================================================
   # DECODE POOL: LeaderWorkerSet (TPU v7x, 1 slice = 2 nodes, 8 chips / 16 TensorCores)
   # ==============================================================================
   apiVersion: leaderworkerset.x-k8s.io/v1
   kind: LeaderWorkerSet
   metadata:
     name: sglang-tpu-decode
     annotations:
       leaderworkerset.sigs.k8s.io/exclusive-topology: "cloud.google.com/gke-nodepool"
     labels:
       app: sglang-decode
   spec:
     replicas: 1
     rolloutStrategy:
       type: RollingUpdate
     leaderWorkerTemplate:
       size: 2
       restartPolicy: RecreateGroupOnPodRestart
       leaderTemplate:
         metadata:
           labels:
             app: sglang-decode
             role: leader
         spec: &decode-pod-spec
           securityContext:
             fsGroup: 65534
           nodeSelector:
             cloud.google.com/gke-tpu-accelerator: tpu7x
             cloud.google.com/gke-tpu-topology: 2x2x2
           tolerations:
             - key: "google.com/tpu"
               operator: "Exists"
               effect: "NoSchedule"
           containers:
             - name: sglang
               image: $SGLANG_IMAGE
               command:
                 - /bin/bash
                 - -c
                 - |
                   set -ex
                   ulimit -c 0
                   LEADER_POD="${LWS_LEADER_ADDRESS%%.*}"
                   DNS_SUFFIX="${LWS_LEADER_ADDRESS#*.}"
                   HOSTS="${LWS_LEADER_ADDRESS}"
                   for ((i = 1; i < LWS_GROUP_SIZE; i++)); do
                     HOSTS="${HOSTS},${LEADER_POD}-${i}.${DNS_SUFFIX}"
                   done
                   export TPU_WORKER_HOSTNAMES="${HOSTS}"
                   export TPU_WORKER_ID="${LWS_WORKER_INDEX}"
                   export TPU_TOPOLOGY="${TPU_TOPOLOGY:-2x2x2}"
                   echo "=== [decode rank ${LWS_WORKER_INDEX}/${LWS_GROUP_SIZE}] hosts=${TPU_WORKER_HOSTNAMES} ==="
                   exec python3 -m sglang.launch_server \
                     --device=tpu \
                     --host=0.0.0.0 \
                     --port=8000 \
                     --model-path=Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \
                     --tensor-parallel-size=16 \
                     --ep-size=16 \
                     --nnodes="${LWS_GROUP_SIZE}" \
                     --node-rank="${LWS_WORKER_INDEX}" \
                     --dist-init-addr="${LWS_LEADER_ADDRESS}:20000" \
                     --attention-backend=pallas_tpu_rpa_batched \
                     --mem-fraction-static=0.8 \
                     --max-running-requests=256 \
                     --chunked-prefill-size=8192 \
                     --max-total-tokens=786432 \
                     --context-length=9216 \
                     --enable-torch-compile \
                     --torch-compile-max-bs=256 \
                     --tpu-compile-tracer=direct \
                     --sampling-defaults=openai \
                     --enable-fused-qk-norm-rope \
                     --disaggregation-mode=decode \
                     --disaggregation-transfer-backend=tpu \
                     --download-dir=/data
               env:
                 - name: HF_TOKEN
                   valueFrom:
                     secretKeyRef:
                       name: hf-secret
                       key: hf_api_token
                       optional: true
               resources:
                 requests:
                   cpu: "16"
                   memory: 128Gi
                   google.com/tpu: "4"
                 limits:
                   memory: 768Gi
                   google.com/tpu: "4"
               ports:
                 - containerPort: 8000
                   name: http
                 - containerPort: 8998
                   name: bootstrap
               startupProbe:
                 httpGet:
                   path: /health
                   port: 8000
                 initialDelaySeconds: 60
                 periodSeconds: 15
                 timeoutSeconds: 10
                 failureThreshold: 236
               readinessProbe:
                 httpGet:
                   path: /health
                   port: 8000
                 periodSeconds: 10
                 timeoutSeconds: 5
                 failureThreshold: 3
               volumeMounts:
                 - name: model-cache
                   mountPath: /data
                 - name: dshm
                   mountPath: /dev/shm
           volumes:
             - name: dshm
               emptyDir:
                 medium: Memory
                 sizeLimit: 32Gi
             - name: model-cache
               ephemeral:
                 volumeClaimTemplate:
                   spec:
                     accessModes: ["ReadWriteOnce"]
                     storageClassName: hyperdisk-balanced-tpu
                     resources:
                       requests:
                         storage: 700Gi
       workerTemplate:
         metadata:
           labels:
             app: sglang-decode
             role: worker
         spec: *decode-pod-spec
   ---
   # ==============================================================================
   # SGLANG DISAGGREGATION ROUTER / PROXY DEPLOYMENT
   # ==============================================================================
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: sglang-router
     labels:
       app: sglang-router
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: sglang-router
     template:
       metadata:
         labels:
           app: sglang-router
       spec:
         containers:
           - name: router
             image: $SGLANG_IMAGE
             command:
               - python3
               - -m
               - sglang_router.launch_router
             args:
               - --pd-disaggregation
               - --prefill
               - http://sglang-prefill-service:8000
               - "8998"
               - --decode
               - http://sglang-decode-service:8000
               - --host
               - 0.0.0.0
               - --port
               - "8000"
               - --model-path
               - Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8
               - --request-timeout-secs
               - "3600"
               - --worker-startup-timeout-secs
               - "5400"
             env:
               - name: HF_TOKEN
                 valueFrom:
                   secretKeyRef:
                     name: hf-secret
                     key: hf_api_token
                     optional: true
             ports:
               - containerPort: 8000
                 name: http
             readinessProbe:
               httpGet:
                 path: /health_generate
                 port: 8000
               initialDelaySeconds: 15
               periodSeconds: 30
               timeoutSeconds: 10
               failureThreshold: 3
             resources:
               requests:
                 cpu: "1"
                 memory: 4Gi
               limits:
                 cpu: "4"
                 memory: 12Gi
   ---
   # ==============================================================================
   # SGLANG EXTERNAL SERVICE: Public / Client Endpoint for OpenAI-compatible Serving
   # ==============================================================================
   apiVersion: v1
   kind: Service
   metadata:
     name: sglang-service
     labels:
       app: sglang-router
   spec:
     type: ClusterIP
     selector:
       app: sglang-router
     ports:
       - name: http
         protocol: TCP
         port: 8000
         targetPort: 8000

2. Apply the workload manifest to your cluster:

   envsubst '$SGLANG_IMAGE' < qwen3-coder-480b-multihost-disagg-v7x.yaml | kubectl apply -f -

3. Verify that the router, prefill, and decode Pods are created:

   kubectl get pods

   When all Pods are ready, the output is similar to the following:

   NAME                             READY   STATUS    RESTARTS   AGE
   sglang-router-6d87cb8c7d-gv4wr   1/1     Running   0          39m
   sglang-tpu-decode-0              1/1     Running   0          40m
   sglang-tpu-decode-0-1            1/1     Running   0          40m
   sglang-tpu-prefill-0             1/1     Running   0          40m
   sglang-tpu-prefill-0-1           1/1     Running   0          40m
   

4. Monitor the prefill and decode leader Pod logs until startup completes:

   kubectl logs -f -l role=leader --prefix=true &
   LOG_PID=$!
   kubectl wait --for=condition=Ready --timeout=3600s pod -l role=leader
   kill "${LOG_PID}" 2>/dev/null || true

   Initial model download, weight loading, and compilation take approximately 15 to 30 minutes on the first run.

   If you're using preemptible capacity such as Spot capacity or Flex-start VMs, it might take longer after node pool creation for your TPU slice nodes to be created, and they might be preempted during workload runs. For more information, see Preemptible VM instances. To check whether your TPU slice nodes have been created, see Check the node's status and conditions.

   Wait until both the prefill and decode leader Pod logs output the following message and kubectl get pods shows 1/1 in the READY column for all Pods:

   The server is fired up and ready to roll!
   

5. Open a second terminal session and forward local port 8000 to sglang-service:

   kubectl port-forward service/sglang-service 8000:8000

   Leave port forwarding running in this terminal session.

Send inference requests

Return to your first terminal session and send a chat completion request to the /v1/chat/completions endpoint exposed by the SGLang router:

curl -s -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8",
    "messages": [
      {"role": "user", "content": "Explain quicksort in two sentences."}
    ],
    "max_tokens": 60,
    "temperature": 0.0
  }'

Run a benchmark workload

To benchmark the deployed Qwen3-Coder model using a 1,024-token input and 1,024-token output (1k/1k) workload from the sglang-router Pod, run the following command:

kubectl exec deployment/sglang-router -- python3 -m sglang.bench_serving \
  --backend sglang \
  --base-url http://localhost:8000 \
  --model Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \
  --dataset-name random \
  --random-input-len 1024 \
  --random-output-len 1024 \
  --random-range-ratio 1.0 \
  --num-prompts 256 \
  --max-concurrency 64