Serve Qwen3.5-397B on GKE with vLLM TorchTPU on TPU7x accelerators

Set up your environment variables

Set up your authentication:

gcloud auth login
gcloud auth application-default login

Set up your environment variables:

export PROJECT_ID="YOUR_PROJECT_ID"
export REGION="YOUR_REGION"
export ZONE="YOUR_ZONE"
export CLUSTER_NAME="YOUR_CLUSTER_NAME"
export MACHINE_TYPE="tpu7x-standard-4t"
export NODE_POOL_NAME="np-0"
export RESERVATION_NAME="YOUR_RESERVATION_NAME"
export HF_TOKEN="YOUR_HF_TOKEN"
export NETWORK="YOUR_NETWORK_NAME"
export SUBNETWORK="YOUR_SUBNETWORK_NAME"
export VLLM_IMAGE="YOUR_VLLM_TPU_IMAGE"

gcloud config set project "${PROJECT_ID}"
gcloud config set billing/quota_project "${PROJECT_ID}"

Replace the following:

• YOUR_PROJECT_ID: the ID of your Google Cloud project.
• YOUR_REGION: the region where you want to deploy your cluster, such as us-central1.
• YOUR_ZONE: the zone where you want to deploy your TPU7x node pool, such as us-central1-c.
• YOUR_CLUSTER_NAME: the name of your GKE cluster, such as tpu7x-tutorial-cluster.
• YOUR_RESERVATION_NAME: if you're using a reservation, the name of your TPU7x reservation.
• YOUR_HF_TOKEN: your Hugging Face access token.
• YOUR_NETWORK_NAME: the name of your Google Cloud network. If you created a custom network, then specify that network. Otherwise, specify default.
• YOUR_SUBNETWORK_NAME: the name of your Google Cloud subnetwork. If you created a custom subnetwork, then specify that subnetwork. Otherwise, specify default.
• YOUR_VLLM_TPU_IMAGE: the vLLM TorchTPU image to use. For public images, refer to the vLLM TorchTPU repository.

Create the GKE cluster

To create a GKE cluster in Standard mode and attach a single-host TPU7x node pool, complete the following steps:

1. Create a regional GKE cluster:

   gcloud container clusters create "${CLUSTER_NAME}" \
       --project="${PROJECT_ID}" \
       --location="${REGION}" \
       --node-locations="${ZONE}" \
       --network="${NETWORK}" \
       --subnetwork="${SUBNETWORK}"

   The output is similar to the following:

   NAME                    LOCATION       MASTER_VERSION      MASTER_IP      MACHINE_TYPE    NODE_VERSION        NUM_NODES  STATUS
   tpu7x-tutorial-cluster  us-central1    1.35.7-gke.1027000  34.63.222.246  n2-standard-16  1.35.7-gke.1027000  3          RUNNING
   

2. Create a single-host TPU7x node pool with one tpu7x-standard-4t node (four TPU7x chips in a 2x2x1 topology) using your reservation:

   gcloud container node-pools create "${NODE_POOL_NAME}" \
       --cluster="${CLUSTER_NAME}" \
       --machine-type="${MACHINE_TYPE}" \
       --location="${REGION}" \
       --node-locations="${ZONE}" \
       --project="${PROJECT_ID}" \
       --num-nodes=1 \
       --reservation-affinity=specific \
       --reservation="${RESERVATION_NAME}"

   Instead of using a specific reservation, you can also create the node pool using Spot capacity or Flex-start VMs by passing the following flags:

   • Spot VMs: replace the --reservation-affinity and --reservation flags with --spot to use Spot capacity. Instances using Spot capacity might take time to be created after the node pool is created, and they might be preempted during workload runs. If your Pods take time to be created, or workloads stop in the middle of running, this could be the cause.
   • Flex-start VMs: pass the --flex-start flag to provision the node pool using the Flex-start consumption option. For more information, see Run a small batch workload with TPUs and Flex-start VMs.

   The output is similar to the following:

   NAME  MACHINE_TYPE       DISK_SIZE_GB  NODE_VERSION
   np-0  tpu7x-standard-4t  100           1.35.7-gke.1027000
   

3. Configure kubectl credentials to communicate with your cluster:

   gcloud container clusters get-credentials "${CLUSTER_NAME}" \
       --location="${REGION}" \
       --project="${PROJECT_ID}"

Configure credentials

Create a Kubernetes Secret named hf-secret to store your Hugging Face access token:

kubectl create secret generic hf-secret \
    --from-literal=hf_api_token="${HF_TOKEN}" \
    --dry-run=client -o yaml | kubectl apply -f -

Deploy the vLLM TorchTPU server

The workload manifest (qwen3_5_server.yaml) defines the following resources:

• A StorageClass (hyperdisk-balanced-qwen3-5) and a 1,500 GiB PersistentVolumeClaim (hd-claim-qwen3-5) backed by Google Cloud Hyperdisk Balanced storage to store the downloaded model weights, XLA compilation cache, and PyTorch compile cache.
• A Deployment (vllm-qwen3-5) scheduled on a single TPU7x node with a 2x2x1 topology (google.com/tpu: '4'). The container runs vllm serve for Qwen/Qwen3.5-397B-A17B-FP8 with --data-parallel-size=8, --tensor-parallel-size=1, --enable-expert-parallel, --kv-cache-dtype=fp8, --max-model-len=9236, and --max-num-seqs=64.
• A ClusterIP Service (vllm-qwen3-5-service) that exposes port 8000.

To deploy and monitor the workload, complete the following steps:

1. Fetch the server manifest:

   curl -O https://raw.githubusercontent.com/GoogleCloudSamples/AIHypercomputerSamples/refs/heads/main/tpu/inference/qwen3-5-397b/qwen3_5_server.yaml

   Alternately, you can choose to create a file named qwen3_5_server.yaml with the following content:

   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   metadata:
     name: hyperdisk-balanced-qwen3-5
   provisioner: pd.csi.storage.gke.io
   parameters:
     type: hyperdisk-balanced
   reclaimPolicy: Delete
   volumeBindingMode: WaitForFirstConsumer
   allowVolumeExpansion: true
   ---
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: hd-claim-qwen3-5
     namespace: default
   spec:
     storageClassName: hyperdisk-balanced-qwen3-5
     accessModes:
       - ReadWriteOnce
     resources:
       requests:
         storage: 1500Gi
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: vllm-qwen3-5
     namespace: default
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: vllm-qwen3-5
     template:
       metadata:
         labels:
           app: vllm-qwen3-5
       spec:
         nodeSelector:
           cloud.google.com/gke-tpu-accelerator: tpu7x
           cloud.google.com/gke-tpu-topology: 2x2x1
         tolerations:
         - key: "cloud.google.com/gke-spot"
           operator: "Equal"
           value: "true"
           effect: "NoSchedule"
         containers:
         - name: vllm-tpu
           image: $VLLM_IMAGE
           command: ["vllm", "serve"]
           args:
           - --host=0.0.0.0
           - --port=8000
           - --model=Qwen/Qwen3.5-397B-A17B-FP8
           - --tokenizer=Qwen/Qwen3.5-397B-A17B-FP8
           - --max-model-len=9236
           - --max-num-batched-tokens=1024
           - --max-num-seqs=64
           - --no-enable-prefix-caching
           - --gpu-memory-utilization=0.9
           - --data-parallel-size=8
           - --tensor-parallel-size=1
           - --async-scheduling
           - --language-model-only
           - --enable-auto-tool-choice
           - --tool-call-parser=qwen3_coder
           - --reasoning-parser=qwen3
           - '--limit-mm-per-prompt={"image": 0, "video": 0}'
           - --kv-cache-dtype=fp8
           - --enable-expert-parallel
           - --download-dir=/data
           - --block-size=128
           env:
           - name: MODEL_IMPL_TYPE
             value: "vllm"
           - name: DP_SCHED_ENABLED
             value: "1"
           - name: ONEHOT_MOE_PERMUTE_THRESHOLD
             value: "32768"
           - name: VLLM_ENGINE_READY_TIMEOUT_S
             value: "9600"
           - name: VLLM_ENGINE_ITERATION_TIMEOUT_S
             value: "1800"
           - name: USE_MOE_SPARSE_CORE
             value: "1"
           - name: TPU_MOE_SKIP_PADDED_TOKENS
             value: "1"
           - name: VLLM_CACHE_ROOT
             value: "/data/vllm_cache"
           - name: VLLM_TORCH_COMPILE_CACHE_DIR
             value: "/data/torch_compile_cache"
           - name: XLA_CACHE_DIR
             value: "/data/xla_cache"
           - name: HF_HOME
             value: /data
           - name: HF_TOKEN
             valueFrom:
               secretKeyRef:
                 name: hf-secret
                 key: hf_api_token
           ports:
           - containerPort: 8000
           resources:
             limits:
               google.com/tpu: '4'
             requests:
               google.com/tpu: '4'
           readinessProbe:
             tcpSocket:
               port: 8000
             initialDelaySeconds: 15
             periodSeconds: 10
           volumeMounts:
           - mountPath: "/data"
             name: data-volume
           - mountPath: /dev/shm
             name: dshm
         volumes:
         - emptyDir:
             medium: Memory
           name: dshm
         - name: data-volume
           persistentVolumeClaim:
             claimName: hd-claim-qwen3-5
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: vllm-qwen3-5-service
     namespace: default
   spec:
     selector:
       app: vllm-qwen3-5
     type: ClusterIP
     ports:
     - name: http
       protocol: TCP
       port: 8000
       targetPort: 8000

2. Apply the vLLM TorchTPU server manifest to your cluster:

   envsubst '$VLLM_IMAGE' < "qwen3_5_server.yaml" | kubectl apply -f -

3. Monitor the server deployment logs until application startup completes:

   kubectl wait --for=jsonpath='{.status.phase}'=Running pod -l app=vllm-qwen3-5 --timeout=900s
   kubectl logs deployment/vllm-qwen3-5 -c vllm-tpu -f

   Initial service startup might take approximately 20 to 30 minutes while the Pod downloads the model weights and compiles the model.

   If you're using preemptive capacity like Spot or Flex-start, it might take longer after node pool creation for your TPU slice nodes to be created, and they can be preempted during workload runs. If your Pods are taking longer than expected to start, or your workloads stop in the middle, this could be the cause. For more information, see Preemption of Spot VMs. To check if your TPU slice nodes have been created, see Check the node's status and conditions.

   Wait until the logs report Application startup complete before you send traffic:

   (APIServer pid=1) INFO:     Started server process [1]
   (APIServer pid=1) INFO:     Waiting for application startup.
   (APIServer pid=1) INFO:     Application startup complete.
   

4. Open a second terminal session.

5. In your second terminal session, forward local port 8000 to vllm-qwen3-5-service:

   kubectl port-forward service/vllm-qwen3-5-service 8000:8000

   Leave port forwarding running in this terminal session.

Send inference requests

Return to your first terminal session and send a chat completion request to the /v1/chat/completions endpoint:

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen/Qwen3.5-397B-A17B-FP8",
        "messages": [
            {"role": "user", "content": "Hello! Tell me a joke."}
        ],
        "max_tokens": 512,
        "temperature": 0.7
    }' | jq .

Run a benchmark workload

To benchmark the deployed Qwen3.5 397B model using an 8,192-token input and 1,024-token output (8k/1k) workload with the InferenceX benchmark suite, complete the following steps:

1. Fetch the benchmark manifest:

   curl -O https://raw.githubusercontent.com/GoogleCloudSamples/AIHypercomputerSamples/refs/heads/main/tpu/inference/qwen3-5-397b/qwen3_5-benchmark-8k1k.yaml

   Alternately, create a file named qwen3_5-benchmark-8k1k.yaml with the following content:

   apiVersion: v1
   kind: Pod
   metadata:
     name: qwen3-5-bench-8k1k
     namespace: default
   spec:
     restartPolicy: OnFailure
     terminationGracePeriodSeconds: 60
     containers:
     - name: vllm-bench
       image: $VLLM_IMAGE
       command: ["/bin/bash", "-c"]
       args:
       - |
         while ! curl http://vllm-qwen3-5-service:8000/ping; do sleep 30 && echo 'Waiting for server...'; done
         apt-get update && apt-get install -y git && \
         git clone https://github.com/SemiAnalysisAI/InferenceX.git /ubench/inferencex && \
         cd /ubench/inferencex && \
         git checkout 89ce6098ef2bc4576a735c43f39c7d972b091cfc && \
         python3 /ubench/inferencex/utils/bench_serving/benchmark_serving.py \
           --backend=vllm \
           --percentile-metrics='ttft,tpot,itl,e2el' \
           --host=vllm-qwen3-5-service \
           --port=8000 \
           --model=Qwen/Qwen3.5-397B-A17B-FP8 \
           --tokenizer=Qwen/Qwen3.5-397B-A17B-FP8 \
           --dataset-name=random \
           --random-input-len=8192 \
           --random-output-len=1024 \
           --random-range-ratio=1.0 \
           --num-prompts=2560 \
           --num-warmups=512 \
           --max-concurrency=256 \
           --ignore-eos
       env:
       - name: HUGGING_FACE_HUB_TOKEN
         valueFrom:
           secretKeyRef:
             key: hf_api_token
             name: hf-secret

2. Apply the benchmark Pod manifest:

   envsubst '$VLLM_IMAGE' < "qwen3_5-benchmark-8k1k.yaml" | kubectl apply -f -

3. Monitor the benchmark logs:

   kubectl logs -f qwen3-5-bench-8k1k

   The benchmark takes approximately 25 to 30 minutes to complete.