Fine-tune and scale reinforcement learning with verl on GKE

This tutorial shows you how to orchestrate a distributed training environment for reinforcement learning on Google Kubernetes Engine (GKE). You use Ray and the verl (Volcano Engine Reinforcement Learning) framework to set up a distributed training environment to fine-tune a Qwen2.5-32B-Instruct model on the GSM8K dataset.

This tutorial focuses on the Group Relative Policy Optimization (GRPO) training pipeline on GKE with Ray and verl. GRPO is a reinforcement learning algorithm designed to improve a model's reasoning ability. This memory-efficient algorithm simplifies the reinforcement learning (RL) process by eliminating the Critic, or value model, and using a relative group-based calculation instead.

This tutorial is a good starting point if you need to set up a distributed training environment where data, model weights, and the training engine are decoupled for efficiency.

This tutorial supports the following GPU architectures:

• Intel or AMD-based GPU nodes: Set up and scale using NVIDIA B200 or H200 GPUs.
• Arm-based A4X (GB200) nodes: Set up and scale using NVIDIA GB200 Grace Blackwell Superchips, using GKE Dynamic Resource Allocation (DRA) and Multi-Node NVLink (IMEX).

Background

The following sections provide a brief overview of the concepts used in this tutorial.

Reinforcement learning (RL)

RL teaches models through experience, exploration, and feedback rather than static imitation. Although pre-training teaches a model what to say, reinforcement learning from human feedback (RLHF) teaches it how to be helpful, safe, and logical. RL serves as the bridge between a base model and a fine-tuned model for a specialized use case.

For more information, see What is reinforcement learning?

Group Relative Policy Optimization (GRPO)

GRPO, an algorithm popularized by DeepSeek, offers a memory-efficient alternative to Proximal Policy Optimization (PPO) for LLM alignment by removing the Critic model. Instead of a Critic network, GRPO generates a group of responses for the same prompt and uses the average reward of that group as the baseline.

For more information, see GRPO.

Volcano Engine Reinforcement Learning (verl)

verl is a high-performance framework designed to handle the complex memory and compute patterns of LLM-based RL.

For more information, see verl.

Objectives

This tutorial shows you how set up reinforcement learning on GKE with verl, by completing the following steps:

1. Set up a GKE cluster with A4X (GB200 Superchips), A4 (B200 GPUs), or A3 Ultra (H200 GPUs).
2. Configure KubeRay to manage a distributed Ray cluster.
3. Use Cloud Storage FUSE to mount a Cloud Storage bucket across all nodes.
4. Run a GRPO training job using verl to align the Qwen2.5-32B-Instruct model with the GSM8K dataset.

Before you begin

Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

Install the Google Cloud CLI.

If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

To initialize the gcloud CLI, run the following command:

gcloud init

Create or select a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

• Create a Google Cloud project:

  gcloud projects create PROJECT_ID

  Replace PROJECT_ID with a name for the Google Cloud project you are creating.

• Select the Google Cloud project that you created:

  gcloud config set project PROJECT_ID

  Replace PROJECT_ID with your Google Cloud project name.

Verify that billing is enabled for your Google Cloud project.

Enable the required APIs:

Roles required to enable APIs

To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

gcloud services enable container.googleapis.com storage.googleapis.com compute.googleapis.com

Install the Google Cloud CLI.

If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

To initialize the gcloud CLI, run the following command:

gcloud init

Create or select a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

• Create a Google Cloud project:

  gcloud projects create PROJECT_ID

  Replace PROJECT_ID with a name for the Google Cloud project you are creating.

• Select the Google Cloud project that you created:

  gcloud config set project PROJECT_ID

  Replace PROJECT_ID with your Google Cloud project name.

Verify that billing is enabled for your Google Cloud project.

Enable the required APIs:

Roles required to enable APIs

To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

gcloud services enable container.googleapis.com storage.googleapis.com compute.googleapis.com

• Grant roles to your user account. Run the following command once for each of the following IAM roles: roles/container.admin, roles/iam.serviceAccountAdmin, roles/storage.admin

  gcloud projects add-iam-policy-binding PROJECT_ID --member="user:USER_IDENTIFIER" --role=ROLE

  Replace the following:

  • PROJECT_ID: Your project ID.
  • USER_IDENTIFIER: The identifier for your user account. For example, myemail@example.com.
  • ROLE: The IAM role that you grant to your user account.

• Create a Hugging Face account, if you don't already have one.
• Ensure that you have a Hugging Face token.
• Ensure your project has sufficient quota for A4X (GB200 Superchips), A4 (B200 GPUs), or A3 Ultra (H200 GPUs). To learn more, see Plan GPU quota and GPU quota.
• Ensure that you have an active capacity reservation for your GPU machine type. For more information, see Reserve capacity through your account team.
• For the A4X (GB200) setup, ensure that you have Helm installed.

Prepare your environment

In this tutorial, you use Cloud Shell.

1. Go to the Google Cloud console.

2. At the top of the Google Cloud console window, click the Activate Cloud Shell button.

3. Set the environment variables:

   export PROJECT_ID=$(gcloud config get project)
   export PROJECT_NUMBER=$(gcloud projects describe ${PROJECT_ID} --format="value(projectNumber)")
   export CONTROL_PLANE_REGION=CONTROL_PLANE_REGION
   export NODE_ZONE=NODE_ZONE
   export CLUSTER_NAME=CLUSTER_NAME
   export KSA_NAME=KSA_NAME
   export GS_BUCKET=BUCKET_NAME-${PROJECT_ID}
   export NAMESPACE=default
   export GPU_TYPE=GPU_TYPE
   export MACHINE_TYPE=MACHINE_TYPE
   export RESERVATION=RESERVATION
   export HF_TOKEN=YOUR_HUGGING_FACE_TOKEN
   
   # A4 (B200 GPUs) or A3 Ultra (H200 GPUs) only variables
   export GVNIC_NETWORK_PREFIX=YGVNIC_NAME
   export RDMA_NETWORK_PREFIX=RDMA_NAME
   
   # A4X (GB200 Superchips) only variables
   export NUM_GPU_NODES=4
   export VERL_IMAGE=verlai/verl:vllm023.aarch64.dev1
   export VERL_REF=ddbcdb7
   

   Replace the following values:

   • CONTROL_PLANE_REGION: the Compute Engine region for the GKE cluster control plane.
   • NODE_ZONE: the zone where the nodes are reserved. For more information, see GPU availability.
   • CLUSTER_NAME: the name of your GKE cluster.
   • KSA_NAME: the name of your Kubernetes service account.
   • BUCKET_NAME: the base name for your Cloud Storage bucket. You don't need to specify the gs:// prefix.
   • GPU_TYPE: the accelerator that you reserved in the Compute Engine capacity reservation. Must be one of the following values:
     • nvidia-gb200: A4X (GB200 Superchips)
     • nvidia-b200: A4 (B200 GPUs)
     • nvidia-h200-141gb: A3 Ultra (H200 GPUs)
   • MACHINE_TYPE: the type of machine to use:
     • For A4X (GB200 Superchips), use a4x-highgpu-4g.
     • For A4 (B200 GPUs), use a4-highgpu-8g or later.
     • For A3 Ultra (H200 GPUs), use a3-ultragpu-8g or later.
   • RESERVATION: the name of your capacity reservation.
   • YOUR_HUGGING_FACE_TOKEN: your Hugging Face token.
   • GVNIC_NAME (A4 or A3 Ultra only): the prefix for the gVNIC network name. You can use any prefix you want.
   • RDMA_NAME (A4 or A3 Ultra only): the prefix for the remote direct memory access (RDMA) network. You can use any prefix you want.

Set up infrastructure

In this section, you create standard VPC networks and the GKE cluster.

Create RDMA network and subnets (A4 and A3 Ultra only)

This section is required for A4 and A3 Ultra GPUs only.

If you use A4X (GB200) GPUs, skip this section and proceed directly to Create the GKE cluster. For A4X (GB200) GPUs, GKE creates the networks automatically when the node pool uses the auto accelerator network profile. The Cluster Toolkit blueprint enables this profile by using the enable_dranet:true flag.

1. Create a VPC network for the gVNIC interface:

   gcloud compute networks create ${GVNIC_NETWORK_PREFIX}-net \
       --subnet-mode=custom \
       --project=${PROJECT_ID}
   gcloud compute networks subnets create ${GVNIC_NETWORK_PREFIX}-sub \
       --network=${GVNIC_NETWORK_PREFIX}-net \
       --region=${CONTROL_PLANE_REGION} \
       --range=192.168.0.0/24
   gcloud compute firewall-rules create ${GVNIC_NETWORK_PREFIX}-internal \
       --network=${GVNIC_NETWORK_PREFIX}-net \
       --action=ALLOW \
       --rules=tcp:0-65535,udp:0-65535,icmp \
       --source-ranges=192.168.0.0/16
   

2. Create a VPC network and subnets for RDMA with 8 subnets for 8 GPUs:

   gcloud beta compute networks create ${RDMA_NETWORK_PREFIX}-net \
       --network-profile=${NODE_ZONE}-vpc-roce \
       --subnet-mode=custom
   
   for N in $(seq 0 7); do
     gcloud compute networks subnets create ${RDMA_NETWORK_PREFIX}-sub-$N \
       --network=${RDMA_NETWORK_PREFIX}-net \
       --region=${CONTROL_PLANE_REGION} \
       --range=192.168.$((N+1)).0/24 &
   done
   wait
   

3. Clone the sample repository:

   git clone https://github.com/GoogleCloudPlatform/kubernetes-engine-samples.git
   cd kubernetes-engine-samples
   

4. Navigate to the working directory:

   cd ai-ml/verl-on-gke
   

Create the GKE cluster

Create the GKE cluster corresponding to your GPU architecture:

Configure network mappings (A4 and A3 Ultra only)

This step is required for standard GPU setups (A4 and A3 Ultra only). If you use A4X (GB200), GKE manages network interfaces automatically, so skip this section.

1. Inspect the network-mapping.yaml manifest:

   # Copyright 2026 Google LLC. All rights reserved.
   #
   # Licensed under the Apache License, Version 2.0 (the "License");
   # you may not use this file except in compliance with the License.
   # You may obtain a copy of the License at
   #
   #     http://www.apache.org/licenses/LICENSE-2.0
   #
   # Unless required by applicable law or agreed to in writing, software
   # distributed under the License is distributed on an "AS IS" BASIS,
   # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   # See the License for the specific language governing permissions and
   # limitations under the License.
   apiVersion: networking.gke.io/v1
   kind: GKENetworkParamSet
   metadata:
     name: gvnic-1
   spec:
     vpc: ${GVNIC_NETWORK_PREFIX}-net
     vpcSubnet: ${GVNIC_NETWORK_PREFIX}-sub
     deviceMode: NetDevice
   ---
   apiVersion: networking.gke.io/v1
   kind: Network
   metadata:
     name: gvnic-1
   spec:
     type: "Device"
     parametersRef:
       group: networking.gke.io
       kind: GKENetworkParamSet
       name: gvnic-1
   ---
   apiVersion: networking.gke.io/v1
   kind: GKENetworkParamSet
   metadata:
     name: rdma-0
   spec:
     vpc: ${RDMA_NETWORK_PREFIX}-net
     vpcSubnet: ${RDMA_NETWORK_PREFIX}-sub-0
     deviceMode: RDMA
   ---
   apiVersion: networking.gke.io/v1
   kind: Network
   metadata:
     name: rdma-0
   spec:
     type: "Device"
     parametersRef:
       group: networking.gke.io
       kind: GKENetworkParamSet
       name: rdma-0
   ---
   apiVersion: networking.gke.io/v1
   kind: GKENetworkParamSet
   metadata:
     name: rdma-1
   spec:
     vpc: ${RDMA_NETWORK_PREFIX}-net
     vpcSubnet: ${RDMA_NETWORK_PREFIX}-sub-1
     deviceMode: RDMA
   ---
   apiVersion: networking.gke.io/v1
   kind: Network
   metadata:
     name: rdma-1
   spec:
     type: "Device"
     parametersRef:
       group: networking.gke.io
       kind: GKENetworkParamSet
       name: rdma-1
   ---
   apiVersion: networking.gke.io/v1
   kind: GKENetworkParamSet
   metadata:
     name: rdma-2
   spec:
     vpc: ${RDMA_NETWORK_PREFIX}-net
     vpcSubnet: ${RDMA_NETWORK_PREFIX}-sub-2
     deviceMode: RDMA
   ---
   apiVersion: networking.gke.io/v1
   kind: Network
   metadata:
     name: rdma-2
   spec:
     type: "Device"
     parametersRef:
       group: networking.gke.io
       kind: GKENetworkParamSet
       name: rdma-2
   ---
   apiVersion: networking.gke.io/v1
   kind: GKENetworkParamSet
   metadata:
     name: rdma-3
   spec:
     vpc: ${RDMA_NETWORK_PREFIX}-net
     vpcSubnet: ${RDMA_NETWORK_PREFIX}-sub-3
     deviceMode: RDMA
   ---
   apiVersion: networking.gke.io/v1
   kind: Network
   metadata:
     name: rdma-3
   spec:
     type: "Device"
     parametersRef:
       group: networking.gke.io
       kind: GKENetworkParamSet
       name: rdma-3
   ---
   apiVersion: networking.gke.io/v1
   kind: GKENetworkParamSet
   metadata:
     name: rdma-4
   spec:
     vpc: ${RDMA_NETWORK_PREFIX}-net
     vpcSubnet: ${RDMA_NETWORK_PREFIX}-sub-4
     deviceMode: RDMA
   ---
   apiVersion: networking.gke.io/v1
   kind: Network
   metadata:
     name: rdma-4
   spec:
     type: "Device"
     parametersRef:
       group: networking.gke.io
       kind: GKENetworkParamSet
       name: rdma-4
   ---
   apiVersion: networking.gke.io/v1
   kind: GKENetworkParamSet
   metadata:
     name: rdma-5
   spec:
     vpc: ${RDMA_NETWORK_PREFIX}-net
     vpcSubnet: ${RDMA_NETWORK_PREFIX}-sub-5
     deviceMode: RDMA
   ---
   apiVersion: networking.gke.io/v1
   kind: Network
   metadata:
     name: rdma-5
   spec:
     type: "Device"
     parametersRef:
       group: networking.gke.io
       kind: GKENetworkParamSet
       name: rdma-5
   ---
   apiVersion: networking.gke.io/v1
   kind: GKENetworkParamSet
   metadata:
     name: rdma-6
   spec:
     vpc: ${RDMA_NETWORK_PREFIX}-net
     vpcSubnet: ${RDMA_NETWORK_PREFIX}-sub-6
     deviceMode: RDMA
   ---
   apiVersion: networking.gke.io/v1
   kind: Network
   metadata:
     name: rdma-6
   spec:
     type: "Device"
     parametersRef:
       group: networking.gke.io
       kind: GKENetworkParamSet
       name: rdma-6
   ---
   apiVersion: networking.gke.io/v1
   kind: GKENetworkParamSet
   metadata:
     name: rdma-7
   spec:
     vpc: ${RDMA_NETWORK_PREFIX}-net
     vpcSubnet: ${RDMA_NETWORK_PREFIX}-sub-7
     deviceMode: RDMA
   ---
   apiVersion: networking.gke.io/v1
   kind: Network
   metadata:
     name: rdma-7
   spec:
     type: "Device"
     parametersRef:
       group: networking.gke.io
       kind: GKENetworkParamSet
       name: rdma-7
   

2. Apply the manifest:

   envsubst < network-mapping.yaml > network-mapping-updated.yaml
   kubectl apply -f network-mapping-updated.yaml
   

Prepare data and storage

Configure Cloud Storage and Kubernetes resources:

1. Create a Cloud Storage bucket:

     gcloud storage buckets create gs://${GS_BUCKET} \
         --location=${CONTROL_PLANE_REGION}  \
         --enable-hierarchical-namespace \
         --uniform-bucket-level-access
   

2. Create a Kubernetes Service Account (KSA) and bind it to the bucket:

     kubectl create serviceaccount ${KSA_NAME} --namespace ${NAMESPACE}
   
     gcloud storage buckets add-iam-policy-binding gs://${GS_BUCKET} \
         --member "principal://iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${PROJECT_ID}.svc.id.goog/subject/ns/${NAMESPACE}/sa/${KSA_NAME}" \
         --role "roles/storage.objectUser"
   

3. Create the Secret for Hugging Face:

     kubectl create secret generic hf-secret --from-literal=hf_api_token=${HF_TOKEN}
   

4. Inspect the gcsfuse-storage.yaml manifest:

   # Copyright 2026 Google LLC. All rights reserved.
   #
   # Licensed under the Apache License, Version 2.0 (the "License");
   # you may not use this file except in compliance with the License.
   # You may obtain a copy of the License at
   #
   #     http://www.apache.org/licenses/LICENSE-2.0
   #
   # Unless required by applicable law or agreed to in writing, software
   # distributed under the License is distributed on an "AS IS" BASIS,
   # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   # See the License for the specific language governing permissions and
   # limitations under the License.
   
   apiVersion: v1
   kind: PersistentVolume
   metadata:
     name: training-bucket-pv
   spec:
     accessModes:
     -   ReadWriteMany
     capacity:
       storage: 768Gi
     persistentVolumeReclaimPolicy: Delete
     storageClassName: gcsfuse-sc
     mountOptions:
     -   implicit-dirs
     -   metadata-cache:negative-ttl-secs:0
     -   metadata-cache:ttl-secs:0
     -   metadata-cache:stat-cache-max-size-mb:-1
     -   metadata-cache:type-cache-max-size-mb:-1
     -   file-cache:max-size-mb:-1
     -   file-cache:cache-file-for-range-read:true
     -   file-cache:enable-parallel-downloads:true
     -   read_ahead_kb=1024
     -   write:enable-streaming-writes:true
     -   write:global-max-blocks:200000
     csi:
       driver: gcsfuse.csi.storage.gke.io
       volumeHandle: ${GS_BUCKET}
       volumeAttributes:
         skipCSIBucketAccessCheck: "true"
         gcsfuseMetadataPrefetchOnMount: "true"
   ---
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: training-bucket-pvc
   spec:
     accessModes:
     -   ReadWriteMany
     resources:
       requests:
         storage: 768Gi
     storageClassName: gcsfuse-sc
   

5. Apply the manifest:

     envsubst < gcsfuse-storage.yaml > gcsfuse-storage-updated.yaml
     kubectl apply -f gcsfuse-storage-updated.yaml
   

Prepare model and data

Populate your Cloud Storage bucket with the model weights and datasets. You can run these commands locally or on a GKE Pod to populate the bucket:

1. Clone the verl repository, prepare virtual environment and process the GSM8K dataset:

   git clone https://github.com/volcengine/verl.git
   git -C verl checkout ${VERL_REF}
   
   VENV_DIR=.venv
   python3 -m venv $VENV_DIR
   source $VENV_DIR/bin/activate
   pip install verl
   
   python verl/examples/data_preprocess/gsm8k.py --local_save_dir ~/data/gsm8k
   

2. Download the Qwen2.5-32B-Instruct model using the Hugging Face CLI (this requires around 66 GB of disk space):

   hf download Qwen/Qwen2.5-32B-Instruct --local-dir Qwen2.5-32B-Instruct
   

3. Upload the model, data, and the verl code to your Cloud Storage bucket:

   gcloud storage cp --recursive verl gs://${GS_BUCKET}/verl
   gcloud storage cp --recursive Qwen2.5-32B-Instruct gs://${GS_BUCKET}/Qwen2.5-32B-Instruct
   gcloud storage cp --recursive ~/data/gsm8k/* gs://${GS_BUCKET}/gsm8k/
   

Deploy RayCluster custom resource

Deploy a RayCluster custom resource, which consists of one system head Pod and multiple GPU-backed worker Pods.

Launch the GRPO Job

Configure and submit the reinforcement learning training job:

Clean up

To avoid incurring charges, delete the resources:

kubectl delete raycluster b200-ray-cluster
gcloud container clusters delete ${CLUSTER_NAME} --location=${CONTROL_PLANE_REGION}
gcloud storage rm -r gs://${GS_BUCKET}

kubectl delete raycluster gb200-ray-cluster
kubectl delete computedomain verl-compute-domain
gcloud storage rm -r gs://${GS_BUCKET}
gcloud container clusters delete ${CLUSTER_NAME} --location=${CONTROL_PLANE_REGION}

What's next

• Learn about Ray on GKE
• Learn about A4X cluster creation on GKE
• Explore verl documentation