Use vLLM on GKE to run inference with DeepSeek-V3.1-Base

Access DeepSeek by using Hugging Face

To use Hugging Face to access DeepSeek, do the following:

1. Sign in to Hugging Face and explore the DeepSeek-V3.1-Base model.
2. Create a Hugging Face read access token.
3. Copy and save the read access token value. You use it later in this tutorial.

Prepare your environment

To prepare your environment, set the default environment variables:

export PROJECT_ID="YOUR_PROJECT_ID"
export RESERVATION_URL="YOUR_RESERVATION_URL"
export REGION="YOUR_REGION"
export CLUSTER_NAME="YOUR_CLUSTER_NAME"
export GCS_BUCKET="YOUR_GCS_BUCKET"
export HUGGING_FACE_TOKEN="YOUR_HF_TOKEN"
export NETWORK="NETWORK_NAME"
export SUBNETWORK="SUBNETWORK_NAME"
export PROJECT_NUMBER=$(gcloud projects describe "${PROJECT_ID}" --format="value(projectNumber)")

gcloud config set project "${PROJECT_ID}"
gcloud config set billing/quota_project "${PROJECT_ID}"

Replace the following:

• YOUR_PROJECT_ID: the ID of the Google Cloud project where you want to create the GKE cluster.

• YOUR_RESERVATION_URL: the URL of the reservation that you want to use to create your GKE cluster. Based on the project in which the reservation exists, specify one of the following values:

  • The reservation exists in your project: RESERVATION_NAME

  • The reservation exists in a different project, and your project can use the reservation: projects/RESERVATION_PROJECT_ID/reservations/RESERVATION_NAME

• YOUR_REGION: the region where you want to create your GKE cluster. You can only create the cluster in the region where your reservation exists.

• YOUR_CLUSTER_NAME: the name of the GKE cluster to create.

• YOUR_GCS_BUCKET: the name of the Cloud Storage bucket where you download the model.

• YOUR_HF_TOKEN: the Hugging Face access token that you created in the previous section.

• NETWORK_NAME: the network that the GKE cluster uses. Specify one of the following values:

  • If you created a custom network, then specify the name of your network.

  • Otherwise, specify default.

• SUBNETWORK_NAME: the subnetwork that the GKE cluster uses. Specify one of the following values:

  • If you created a custom subnetwork, then specify the name of your subnetwork. You can only specify a subnetwork that exists in the same region as the reservation.

  • Otherwise, specify default.

Create a GKE cluster in Autopilot mode

To create a GKE cluster in Autopilot mode, run the following command:

gcloud container clusters create-auto $CLUSTER_NAME \
    --project=$PROJECT_ID \
    --region=$REGION \
    --release-channel=rapid \
    --network=$NETWORK \
    --subnetwork=$SUBNETWORK

Creating the GKE cluster might take some time to complete. To verify that Google Cloud has finished creating your cluster, go to Kubernetes clusters on the Google Cloud console.

Create a Kubernetes secret for Hugging Face credentials

To create a Kubernetes secret for Hugging Face credentials, do the following:

1. Configure kubectl to communicate with your GKE cluster:

   gcloud container clusters get-credentials $CLUSTER_NAME \
       --location=$REGION

2. Create a Kubernetes secret to store your Hugging Face token:

   kubectl create secret generic hf-secret \
       --from-literal=hf_token=${HUGGING_FACE_TOKEN} \
       --dry-run=client -o yaml | kubectl apply -f -

Create a Cloud Storage bucket

If you want to use a new bucket to store the model, run the following:

gcloud storage buckets create gs://$GCS_BUCKET --location=$REGION --uniform-bucket-level-access

Grant write permissions on the Cloud Storage bucket to the default service account:

gcloud storage buckets add-iam-policy-binding gs://$GCS_BUCKET \
    --member="principal://iam.googleapis.com/projects/$PROJECT_NUMBER/locations/global/workloadIdentityPools/$PROJECT_ID.svc.id.goog/subject/ns/default/sa/default" \
    --role="roles/storage.objectAdmin"

If you want to use an existing Cloud Storage bucket, you can skip this step. However, you must ensure that your bucket is in the same region as your cluster and that the service account has the required write permissions to it.

Download the model to your Cloud Storage bucket

1. Create a deepseek-download-job.yaml file:

   apiVersion: batch/v1
   kind: Job
   metadata:
     name: deepseek-download-job
     namespace: default
   spec:
     template:
       metadata:
         labels:
           app: deepseek-oss-downloader
       spec:
         restartPolicy: OnFailure
         containers:
         - name: downloader
           image: google/cloud-sdk:slim
           resources:
             requests:
               cpu: "8"
               memory: "32Gi"
             limits:
               cpu: "8"
               memory: "32Gi"
           env:
           - name: HUGGING_FACE_HUB_TOKEN
             valueFrom:
               secretKeyRef:
                 name: hf-secret
                 key: hf_token
           - name: HF_XET_HIGH_PERFORMANCE
             value: "0"
           - name: PIP_BREAK_SYSTEM_PACKAGES
             value: "1"
           command: ["/bin/sh", "-c"]
           args:
           - |
             set -e
             echo "Installing Hugging Face Hub CLI..."
             pip install -U "huggingface_hub[cli]"
   
             LOCAL_DIR="/mnt/model-download/deepseek-v3-1"
             mkdir -p "$LOCAL_DIR"
   
             echo "Downloading model from HF to local SSD using parallel workers..."
             hf download deepseek-ai/DeepSeek-V3.1-Base \
               --token "$HUGGING_FACE_HUB_TOKEN" \
               --local-dir "$LOCAL_DIR" \
               --max-workers 8
   
             DOWNLOAD_STATUS=$?
   
             if [ $DOWNLOAD_STATUS -ne 0 ]; then
               echo "ERROR: Model download failed with exit code $DOWNLOAD_STATUS"
               exit $DOWNLOAD_STATUS
             fi
   
             echo "Download to SSD complete. Syncing to GCS..."
             gcloud storage rsync "$LOCAL_DIR" "gs://$GCS_BUCKET/deepseek-v3-1" --recursive
   
             echo "Process successfully completed!"
           volumeMounts:
           - mountPath: /mnt/model-download
             name: download-storage
         volumes:
         - name: download-storage
           ephemeral:
             volumeClaimTemplate:
               spec:
                 accessModes: [ "ReadWriteOnce" ]
                 storageClassName: premium-rwo
                 resources:
                   requests:
                     storage: 800Gi

2. Apply the deepseek-download-job.yaml manifest to initialize the download job:

   envsubst '$GCS_BUCKET' < deepseek-download-job.yaml | kubectl apply -f -

3. To see the completion status, run the following command:

   kubectl wait \
       --for=condition=Complete \
       --timeout=7200s job/deepseek-download-job

   The --timeout flag specifies how long the command monitors the job before timing out.

   The job resource downloads the DeepSeek-V3.1-Base model weights from Hugging Face to your Google Cloud Storage bucket using SSD volume. The download takes around 40 minutes to complete. Once the download finishes, proceed to the next section to launch the model deployment.

4. To delete the job, run the following command:

   kubectl delete job deepseek-download-job

Deploy a vLLM container to your GKE cluster

After you've downloaded the model to your Cloud Storage bucket, deploy a vLLM container to your GKE cluster by completing the following steps:

1. Create a vllm-deepseek3-1-base.yaml file with your chosen vLLM deployment:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: deepseek3-1-deploy
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: deepseek
     template:
       metadata:
         labels:
           app: deepseek
           ai.gke.io/model: deepseek-v3-1-base
           ai.gke.io/inference-server: vllm
           examples.ai.gke.io/source: user-guide
         annotations:
           gke-gcsfuse/volumes: "true"
       spec:
         containers:
         - name: vllm-inference
           image: us-docker.pkg.dev/vertex-ai/vertex-vision-model-garden-dockers/pytorch-vllm-serve:20250819_0916_RC01
           resources:
             requests:
               cpu: "10"
               memory: "1000Gi"
               ephemeral-storage: "1Ti"
               nvidia.com/gpu: "8"
             limits:
               cpu: "10"
               memory: "1000Gi"
               ephemeral-storage: "1Ti"
               nvidia.com/gpu: "8"
           command: ["python3", "-m", "vllm.entrypoints.openai.api_server"]
           args:
           - --model=/mnt/gcs/deepseek-v3-1
           - --tensor-parallel-size=8
           - --host=0.0.0.0
           - --port=8000
           - --max-model-len=8192
           - --max-num-seqs=4
           volumeMounts:
           - mountPath: /dev/shm
             name: dshm
           - mountPath: /mnt/gcs
             name: gcs-bucket-volume
             readOnly: true
           livenessProbe:
             httpGet:
               path: /health
               port: 8000
             initialDelaySeconds: 1800
             periodSeconds: 10
           readinessProbe:
             httpGet:
               path: /health
               port: 8000
             initialDelaySeconds: 1800
             periodSeconds: 5
         volumes:
         - name: dshm
           emptyDir:
               medium: Memory
         - name: gcs-bucket-volume
           csi:
             driver: gcsfuse.csi.storage.gke.io
             volumeAttributes:
               bucketName: $GCS_BUCKET
               mountOptions: "implicit-dirs,file-cache:max-size-mb:-1,file-cache:enable-parallel-downloads:true,file-cache:max-parallel-downloads:32,file-cache:parallel-downloads-per-file:8,file-cache:download-chunk-size-mb:16"
         nodeSelector:
           cloud.google.com/gke-accelerator: nvidia-b200
           cloud.google.com/reservation-name: $RESERVATION_URL
           cloud.google.com/reservation-affinity: "specific"
           cloud.google.com/gke-gpu-driver-version: latest
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: deepseek-service
   spec:
     selector:
       app: deepseek
     type: ClusterIP
     ports:
       - protocol: TCP
         port: 8000
         targetPort: 8000
   ---
   apiVersion: monitoring.googleapis.com/v1
   kind: PodMonitoring
   metadata:
     name: deepseek-monitoring
   spec:
     selector:
       matchLabels:
         app: deepseek
     endpoints:
     - port: 8000
       path: /metrics
       interval: 30s

2. Apply the vllm-deepseek3-1-base.yaml file to your GKE cluster:

   envsubst < vllm-deepseek3-1-base.yaml | kubectl apply -f -

3. To see the completion status, run the following command:

   kubectl wait \
     --for=condition=Available \
     --timeout=7200s deployment/deepseek3-1-deploy

   The --timeout flag allows the command to monitor the deployment for the specified period of time.

Interact with DeepSeek-V3.1-Base by using curl

To verify the DeepSeek-V3.1-Base model that you deployed, do the following:

1. Set up port forwarding to DeepSeek-V3.1-Base:

   kubectl port-forward service/deepseek-service 8000:8000

2. Open a new terminal window. You can then chat with your model by usingcurl:

   curl http://127.0.0.1:8000/v1/chat/completions \
   -X POST \
   -H "Content-Type: application/json" \
   -d '{
     "model": "deepseek-ai/DeepSeek-V3.1-Base",
     "messages": [
       {
         "role": "user",
         "content": "Describe how generative AI works in one short and easy to understand sentence"
       }
     ],
   "stream":false
   }' | jq .

3. The output that you see is similar to the following:

   {
     "id": "chatcmpl-1a47172070544a5d83199ed5548befca",
     "object": "chat.completion",
     "created": 1755891024,
     "model": "deepseek-ai/DeepSeek-V3.1-Base",
     "choices": [
       {
         "index": 0,
         "message": {
           "role": "assistant",
           "content": "\nGenerative AI uses patterns from existing data to create new, similar content, like text, images, or music.\n",
           "refusal": null,
           "annotations": null,
           "audio": null,
           "function_call": null,
           "tool_calls": [],
           "reasoning_content": null
         },
         "logprobs": null,
         "finish_reason": "stop",
         "stop_reason": null
       }
     ],
     "service_tier": null,
     "system_fingerprint": null,
     "usage": {
       "prompt_tokens": 17,
       "total_tokens": 42,
       "completion_tokens": 25,
       "prompt_tokens_details": null
     },
     "prompt_logprobs": null,
     "kv_transfer_params": null
   }
   

Observe the performance of the model

If you want to observe your model's performance, then you can use the vLLM dashboard integration in Cloud Monitoring. This dashboard helps you view critical performance metrics for your model like token throughput, network latency, and error rates. For information, see vLLM in the Monitoring documentation.