Start RayJobs faster across multiple resource flavors

Standard

This document describes how to configure and deploy ephemeral RayJobs on Google Kubernetes Engine (GKE) using Kueue concurrent admission to place and migrate the job across compute SKUs.

Running an ephemeral RayJob means submitting a job definition to Kubernetes where the operator automatically provisions a dedicated Ray cluster, runs your workload to completion, retrieves the results and status, and immediately tears down the cluster to release resources.

Concurrent admission is a Kueue feature that allows ephemeral RayJobs to start immediately on available resources and fall back across different SKU types—such as Reservations, DWS Calendar, DWS Flex, On-demand, and Spot VMs—within a single job request. Rather than waiting idly for a specific preferred SKU to become available, a RayJob can start immediately using any available compute type, reducing time-to-start while maximizing overall fleet utilization across your provisioned capacity.

How concurrent admission works

Concurrent admission is an alpha Kueue feature (available in Kueue v0.18 or later) that changes how jobs are placed across compute SKUs:

Multi-flavor pursuit: Rather than evaluating resource flavors serially, Kueue pursues multiple acceptable ResourceFlavors concurrently. By generating parallel workload variants for each SKU option, Kueue schedules and runs the entire job on whichever flavor first has available capacity.
Optional migration: If enabled (concurrentAdmissionPolicy.migration.mode: TryPreferredFlavors), Kueue continues pursuing more preferred SKUs concurrently. If a more preferred flavor becomes available later, Kueue migrates the running job to it. If migration is disabled or omitted, the job remains on its initial flavor to avoid restart disruption.
Workload structure: Kueue implements concurrent admission by marking the original workload as a parent and creating one variant workload per ResourceFlavor. Each variant pursues a single flavor independently. When one is admitted, the parent is admitted, and the remaining variants are deactivated.

Note: The order of ResourceFlavors in the ClusterQueue defines preference (the first flavor listed is the most preferred).

Before you begin

Before you start, make sure that you have performed the following tasks:

Enable the Google Kubernetes Engine API.

Enable Google Kubernetes Engine API

To use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running the gcloud components update command. Earlier gcloud CLI versions might not support running the commands in this document.
Note: For existing gcloud CLI installations, make sure to set the compute/region property. If you use primarily zonal clusters, set the compute/zone instead. By setting a default location, you can avoid errors in the gcloud CLI like the following: One of [--zone, --region] must be supplied: Please specify location. You might need to specify the location in certain commands if the location of your cluster differs from the default that you set.

Make sure you have the following command-line tools installed:

The Google Cloud CLI (gcloud)
kubectl

Requirements and limitations

Kueue version: Concurrent admission requires Kueue v0.18 or later.
Single-flavor constraint: Concurrent admission requires all PodSets in a workload—including the Ray head and worker group—to be pinned to the same single ResourceFlavor. Decoupled flavor assignment (such as scheduling the Ray head node on a standard CPU flavor while placing worker nodes on GPU flavors) is not supported under concurrent admission policies.

Step 1: Create and connect to the GKE cluster

In this step, you define your environment variables, create a Standard cluster with the Ray operator add-on enabled, create a Compute Engine GPU reservation, add the GPU node pools, and connect to your cluster.

Set environment variables for your project, zone, and cluster name:
```
export PROJECT_ID=PROJECT_ID
export ZONE=ZONE
export CLUSTER_NAME=gpu-cluster
```
Replace PROJECT_ID with your Google Cloud project ID, and ZONE with your target compute zone (for example, us-central1-a).

Create a Standard GKE cluster with the Ray operator add-on enabled:

gcloud container clusters create ${CLUSTER_NAME} \
    --project=${PROJECT_ID} \
    --zone=${ZONE} \
    --machine-type=n2-standard-4 \
    --num-nodes=2 \
    --addons=RayOperator

Create a Compute Engine GPU reservation for the GPU node pool to consume. Specify the --require-specific-reservation flag to prevent non-GKE VMs from consuming the capacity:
```
gcloud compute reservations create my-gpu-reservation \
    --project=${PROJECT_ID} \
    --zone=${ZONE} \
    --vm-count=3 \
    --machine-type=n1-standard-4 \
    --accelerator=type=nvidia-tesla-t4,count=1 \
    --require-specific-reservation
```
Note: Size the reservation to hold the entire worker group (3 VMs for 3 workers requesting 1 GPU each), because concurrent admission migrates all 3 GPU workers onto the reservation together.

Create the GPU node pools. In this example, you create a Reservation GPU pool and a DWS Flex GPU pool using NVIDIA T4 GPUs:

Create the Reservation GPU node pool:

gcloud container node-pools create gpu-pool-reservation \
    --cluster=${CLUSTER_NAME} \
    --project=${PROJECT_ID} \
    --zone=${ZONE} \
    --machine-type=n1-standard-4 \
    --accelerator=type=nvidia-tesla-t4,count=1 \
    --reservation-affinity=specific \
    --reservation=my-gpu-reservation \
    --num-nodes=0 \
    --enable-autoscaling --min-nodes=0 --max-nodes=3

Create the DWS Flex GPU node pool:

gcloud container node-pools create gpu-pool-dws-flex \
    --cluster=${CLUSTER_NAME} \
    --project=${PROJECT_ID} \
    --zone=${ZONE} \
    --machine-type=n1-standard-4 \
    --accelerator=type=nvidia-tesla-t4,count=1 \
    --num-nodes=0 \
    --enable-autoscaling --min-nodes=0 --max-nodes=3 \
    --flex-start \
    --location-policy=ANY \
    --reservation-affinity=none

Retrieve cluster credentials to configure kubectl:

gcloud container clusters get-credentials ${CLUSTER_NAME} \
    --project=${PROJECT_ID} \
    --zone=${ZONE}

Step 2: Install Kueue

Install the Kueue manifests:

kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/download/v0.18.0/manifests.yaml

Enable the ConcurrentAdmission feature gate by editing the kueue-manager-config ConfigMap in the kueue-system namespace:

kubectl edit configmap kueue-manager-config -n kueue-system

Add the featureGates entry to the controller_manager_config.yaml data:

apiVersion: v1
kind: ConfigMap
metadata:
  name: kueue-manager-config
  namespace: kueue-system
data:
  controller_manager_config.yaml: |
    apiVersion: config.kueue.x-k8s.io/v1beta2
    kind: Configuration
    featureGates:
      ConcurrentAdmission: true
    # ... keep the rest of the existing configuration ...

Restart the Kueue controller manager deployment to apply the configuration:

kubectl rollout restart deployment kueue-controller-manager -n kueue-system

Step 3: Configure Kueue

Create the ResourceFlavor resources (representing Reservation and DWS Flex capacity), a ClusterQueue with a concurrent admission policy, and a LocalQueue.

Key considerations for this configuration include:

The ClusterQueue must use queueingStrategy: BestEffortFIFO. StrictFIFO is not supported with concurrent admission.
The ClusterQueue must have exactly one resourceGroup containing at most 16 flavors.
Flavors are listed in order of preference: reservation-flavor first, followed by dws-flex-flavor.
spec.concurrentAdmissionPolicy is immutable once the ClusterQueue is created. To modify the policy, delete and recreate the ClusterQueue.

Caution: Concurrent admission requires all PodSets in a workload—including the Ray head and worker group—to be pinned to the same single ResourceFlavor. Decoupled flavor assignment (such as scheduling the Ray head node on a standard CPU flavor while placing worker nodes on GPU/TPU flavors) is not supported under concurrent admission policies. If your workloads require scheduling a standalone CPU Ray head under concurrent admission, file a feature request on the Kueue GitHub repository.

Create a file named kueue-setup.yaml:

apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor
metadata:
  name: reservation-flavor
spec:
  nodeLabels:
    cloud.google.com/gke-nodepool: gpu-pool-reservation
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor
metadata:
  name: dws-flex-flavor
spec:
  nodeLabels:
    cloud.google.com/gke-nodepool: gpu-pool-dws-flex
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
  name: cluster-queue
spec:
  namespaceSelector: {}
  queueingStrategy: BestEffortFIFO
  # Concurrent admission lets Kueue provision capacity in several flavors at
  # the same time instead of trying them one after another.
  concurrentAdmissionPolicy:
    migration:
      # Move the workload to an earlier (more preferred) flavor if that flavor
      # becomes ready first.
      mode: TryPreferredFlavors
      constraints:
        # Stop trying additional flavors once this flavor is reached.
        lastAcceptableFlavorName: reservation-flavor
  resourceGroups:
  - coveredResources: ["nvidia.com/gpu", "cpu", "memory"]
    flavors:
    - name: reservation-flavor
      resources:
      - name: "nvidia.com/gpu"
        nominalQuota: "3"
      - name: "cpu"
        nominalQuota: "8"
      - name: "memory"
        nominalQuota: "32Gi"
    - name: dws-flex-flavor
      resources:
      - name: "nvidia.com/gpu"
        nominalQuota: "3"
      - name: "cpu"
        nominalQuota: "8"
      - name: "memory"
        nominalQuota: "32Gi"
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: LocalQueue
metadata:
  name: user-queue
  namespace: default
spec:
  clusterQueue: cluster-queue

Apply the configuration to your cluster:
```
kubectl apply -f kueue-setup.yaml
```

Step 4: Define the ephemeral RayJob

With concurrent admission, each admitted variant pins the entire workload to one flavor, and Kueue injects that flavor's node selector onto the head and worker pods. You define a single worker group (with 3 worker replicas) and let Kueue decide and migrate which GPU SKU runs it.

Key settings in this definition include:

No SKU node selectors: Do not pin the worker group to a specific node pool. Kueue automatically adds the node selector based on the admitted flavor.
Head GPU toleration: Because GKE taints GPU nodes with nvidia.com/gpu=present:NoSchedule, the head pod requires a matching toleration to colocate on a GPU node with a worker.
Ephemeral behavior: shutdownAfterJobFinishes: true instructs KubeRay to delete the Ray cluster when the job finishes.
Disabled autoscaling: Use fixed replicas (minReplicas == maxReplicas). Autoscaling is not recommended for ephemeral jobs.

Create a file named ephemeral-rayjob.yaml:

apiVersion: ray.io/v1
kind: RayJob
metadata:
  name: ephemeral-gpu-job
  labels:
    # Kueue admits this RayJob through the LocalQueue defined in kueue-setup.yaml.
    kueue.x-k8s.io/queue-name: user-queue
spec:
  entrypoint: |
    python -c "
    import ray, time, socket
    ray.init()
    print('Cluster resources:', ray.cluster_resources())
    @ray.remote(num_gpus=1)
    def gpu_task(task_id):
        ip = socket.gethostbyname(socket.gethostname())
        print(f'Task {task_id} running on node with IP: {ip}')
        time.sleep(5)
        return ip
    futures = [gpu_task.remote(i) for i in range(3)]
    results = ray.get(futures)
    print('Tasks successfully executed on nodes:', set(results))
    "
  # Delete the RayCluster after the job finishes, which makes the job ephemeral.
  shutdownAfterJobFinishes: true
  rayClusterSpec:
    rayVersion: '2.58.0'
    headGroupSpec:
      rayStartParams:
        dashboard-host: '0.0.0.0'
      template:
        spec:
          tolerations:
          - key: nvidia.com/gpu
            operator: Exists
            effect: NoSchedule
          containers:
          - name: ray-head
            image: rayproject/ray:2.58.0
            resources:
              requests:
                cpu: "1"
                memory: "2Gi"
              limits:
                cpu: "1"
                memory: "2Gi"
    workerGroupSpecs:
    - groupName: worker-group
      replicas: 3
      minReplicas: 3
      maxReplicas: 3
      rayStartParams: {}
      template:
        spec:
          nodeSelector:
            cloud.google.com/gke-accelerator: nvidia-tesla-t4
          containers:
          - name: ray-worker
            image: rayproject/ray:2.58.0
            resources:
              requests:
                nvidia.com/gpu: "1"
                cpu: "2"
                memory: "8Gi"
              limits:
                nvidia.com/gpu: "1"
                cpu: "2"
                memory: "8Gi"

Submit the RayJob to the cluster:
```
kubectl apply -f ephemeral-rayjob.yaml
```

Step 5: Monitor and verify concurrent admission

Inspect the parent and variant workload objects:

kubectl get workloads

To list only the parent workload:

kubectl get workloads -l kueue.x-k8s.io/concurrent-admission-parent=true

Watch pod creation and placement:
```
kubectl get pods -w -o wide
```
The head and worker pods land on nodes matching the admitted SKU flavor. The head colocates on a GPU node with one of the workers, while the short-lived submitter pod runs on the default CPU pool.
Check the status of the RayJob:
```
kubectl get rayjobs
```

View the output logs from the running job:

kubectl logs -l job-name=ephemeral-gpu-job

The output is similar to the following:

Cluster resources: {'node:10.52.3.6': 1.0, 'node:__internal_head__': 1.0, 'memory': 27917287424.0, 'object_store_memory': 8194336357.0, 'CPU': 7.0, 'GPU': 3.0, 'node:10.52.3.7': 1.0, 'node:10.52.4.6': 1.0, 'node:10.52.2.6': 1.0}
(gpu_task pid=277, ip=10.52.4.6) Task 0 running on node with IP: 10.52.4.6
(gpu_task pid=275, ip=10.52.2.6) Task 2 running on node with IP: 10.52.2.6
(gpu_task pid=276, ip=10.52.3.7) Task 1 running on node with IP: 10.52.3.7
Tasks successfully executed on nodes: {'10.52.2.6', '10.52.4.6', '10.52.3.7'}
Job 'ephemeral-gpu-job' succeeded

Verify cluster teardown. Once the job succeeds, KubeRay automatically removes the Ray cluster pods:
```
kubectl get pods
```

Step 6: (Optional) Start on DWS Flex and migrate to reservation

This scenario demonstrates how a job starts on fallback capacity (DWS Flex) and subsequently upgrades to a reservation when quota becomes available.

Simulate initial condition (reservation occupied): If the reservation quota is fully occupied by other workloads when the RayJob is submitted, the reservation-flavor variant cannot be admitted.

Job starts on fallback flavor (T1): Kueue admits dws-flex-flavor, and the RayCluster is created on gpu-pool-dws-flex:

kubectl get workloads

The output shows dws-flex-flavor admitted as True:

NAME                                                          ADMITTED
rayjob-ephemeral-gpu-job-[hash]                               True
rayjob-ephemeral-gpu-job-variant-dws-flex-flavor-[hash]        True
rayjob-ephemeral-gpu-job-variant-reservation-flavor-[hash]

Reservation capacity frees up (T2): When capacity on the reservation becomes available, Kueue admits the reservation-flavor variant:

kubectl get workloads -w

NAME                                                          ADMITTED
rayjob-ephemeral-gpu-job-[hash]                               True
rayjob-ephemeral-gpu-job-variant-reservation-flavor-[hash]   True
rayjob-ephemeral-gpu-job-variant-dws-flex-flavor-[hash]        False

Job migrates to reservation (T3): Because reservation-flavor has higher priority, Kueue migrates the parent workload. Migration restarts the RayCluster on the new flavor, and the pods are recreated on gpu-pool-reservation:
```
kubectl get pods -o wide -w
```
Note: Migration restarts the job on the new flavor. Concurrent admission with migration is recommended for workloads that can tolerate restart disruption in exchange for reduced initial wait time.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this guide, delete the cluster and reservation:

Delete the GKE cluster:

gcloud container clusters delete ${CLUSTER_NAME} \
    --project=${PROJECT_ID} \
    --zone=${ZONE} \
    --quiet

Delete the Compute Engine reservation:

gcloud compute reservations delete my-gpu-reservation \
    --project=${PROJECT_ID} \
    --zone=${ZONE} \
    --quiet

What's next

Learn more about Ray on GKE.
Read about Kueue concurrent admission.
Explore GPU sharing and provisioning strategies on GKE.

Start RayJobs faster across multiple resource flavors Stay organized with collections Save and categorize content based on your preferences.

How concurrent admission works

Before you begin

Requirements and limitations

Step 1: Create and connect to the GKE cluster

Step 2: Install Kueue

Step 3: Configure Kueue

Step 4: Define the ephemeral RayJob

Step 5: Monitor and verify concurrent admission

Step 6: (Optional) Start on DWS Flex and migrate to reservation

Clean up

What's next

Start RayJobs faster across multiple resource flavors