gcluster job submit

NAME

gcluster job submit - submits a workload to a cluster.

SYNOPSIS

gcluster job submit [FLAGS]

DESCRIPTION

The gcluster job submit command submits a workload to a cluster. On GKE clusters, Cluster Toolkit deploys workloads as Kubernetes JobSets integrated with Kueue.

FLAGS

  • -i, --image IMAGE: The name of the prebuilt container image to run, including the registry path. Specify either the --image flag or the --base-image flag, unless you submit a headless Pathways job with the --pathways-headless flag.
  • -B, --base-image IMAGE: The name of the base image used to build a new container image and push it to Artifact Registry by using Crane. This flag requires the --build-context flag and can't be combined with the --image flag.
  • -b, --build-context DIR: The path to the build context directory for Crane. This flag is required when you use the --base-image flag and also requires you to set the GCLUSTER_IMAGE_REPO environment variable to your Artifact Registry repository name.
  • -e, --command COMMAND: The command to run in the workload container, such as python3 train.py. This flag is required for standard job submissions, unless you submit a headless Pathways job with the --pathways-headless flag.
  • --compute-type TYPE: The type of compute to request, such as n2-standard-32, nvidia-l4, or v5litepod-32. This flag is required.
  • -o, --dry-run-out PATH: The path to output the generated Kubernetes manifest instead of applying it.
  • -f, --platform PLATFORM: The target platform for the image build, such as linux/amd64 or linux/arm64. Use this flag with the --base-image flag.
  • --mount VOLUME: The volumes to mount.
  • --env KEY=VALUE: Custom environment variables in KEY=VALUE format to pass to the workload container. You can specify this flag multiple times.
  • -n, --name NAME: The name of the workload to create. The name can't exceed 28 characters for standard jobs or 22 characters for Pathways jobs (--pathways). This flag is required.
  • -q, --queue QUEUE: The name of the Kueue LocalQueue.

  • --num-nodes N: The number of nodes to use per group or slice.

    Note: The `--num-nodes` flag can't be used with TPU jobs because the
  node count is calculated automatically from the `--topology` value.

  • --num-slices N: The number of independent groups or slices to use.

  • --restarts N: The maximum number of restarts for the JobSet.

  • --gke-ttl-after-finished DURATION: The time to retain the JobSet after it finishes.

  • --grace-period DURATION: The time to wait before forcefully terminating a pod.

  • --gke-disable-parallel-containers: Disables parallel containers for TPU7x on GKE.

  • --placement-policy POLICY: The name of the GKE placement policy to use.

  • --node-constraint KEY=VALUE: Key=value pairs for node labels to target specific nodes.

  • --cpu-affinity AFFINITY: CPU affinity rules.

  • --restart-on-exit-codes CODES: A list of exit codes that don't trigger a job failure.

  • --image-pull-secret SECRET: A comma-separated list of secrets for pulling images.

  • --service-account ACCOUNT: The service account name for the pods.

  • --topology TOPOLOGY: The TPU slice topology.

  • --gke-scheduler SCHEDULER: The Kubernetes Scheduler name.

  • --await-job-completion: Waits for the submitted job to complete.

  • --timeout TIMEOUT: The time to wait for the job.

  • --priority PRIORITY: A priority class name.

  • --verbose: Turns on verbose logging for the workload.

  • --gke-nap-provisioning MODEL: The compute provisioning model for GKE node auto-provisioning, such as on-demand, spot, or reservation.

  • --gke-nap-reservation RESERVATION: The name of the Google Cloud reservation for GKE node auto-provisioning. This field is required when you set the value --gke-nap-provisioning=reservation.

  • --pathways: Generates a manifest for a Pathways job.

  • --pathways-proxy-server-image PW_PROXY_IMAGE: The image for the Pathways proxy server.

  • --pathways-server-image PW_SERVER_IMAGE: The image for the Pathways server.

  • --pathways-worker-image IMAGE: The image for the Pathways worker.

  • --pathways-headless: Prevents deploying the user's workload container within the pathways-head job.

  • --pathways-gcs-location PW_GCS_LOCATION: The Cloud Storage location to store Pathways artifacts. This flag is required when the --pathways flag is set.

  • --pathways-elastic-slices N: Configures the number of elastic slices.

  • --pathways-max-slice-restarts N: The maximum times the workers in a slice can be restarted.

  • --pathways-proxy-args ARGS: Additional arguments for pathways-proxy.

  • --pathways-server-args ARGS: Additional arguments for pathways-rm.

  • --pathways-worker-args ARGS: Additional arguments for pathways-worker.

  • --pathways-proxy-env KEY=VALUE: Environment variables for the Pathways proxy.

  • --pathways-server-env KEY=VALUE: Environment variables for the Pathways server.

  • --pathways-worker-env KEY=VALUE: Environment variables for the Pathways worker.

  • --pathways-colocated-python-sidecar-image IMAGE: The image for the Python-based sidecar.

  • --pathways-head-np NP: The node pool to use for the Pathways head job.

  • --gke-mtc-enabled: Turns on Multi-Tier Checkpointing (MTC).

  • --gke-mtc-ramdisk-dir DIR: The ramdisk directory path for MTC. This flag is required when the --gke-mtc-enabled flag is set.

  • --gke-custom-templates-path PATH: The path to the local directory containing custom GKE templates.

PERSISTENT FLAGS

These flags are available to all gcluster job commands:

  • -c, --cluster CLUSTER_NAME: The name of the GKE cluster.
  • -l, --location LOCATION: The location, either a region or a zone, of the GKE cluster.
  • -p, --project PROJECT_ID: The Google Cloud project ID.
  • --gke-namespace NAMESPACE: The target GKE namespace for the operation.
  • --skip-prereqs: Skips local environment prerequisite checks, such as gcloud, kubectl, and docker authentication.

EXAMPLES

To submit a workload named train-job to a cluster:

gcluster job submit \
    --cluster=CLUSTER_NAME \
    --project=PROJECT_ID \
    --location=LOCATION \
    --name=train-job \
    --image=IMAGE \
    --compute-type=TYPE \
    --command="python3 train.py"

To submit a multi-node workload:

gcluster job submit \
    --cluster=CLUSTER_NAME \
    --project=PROJECT_ID \
    --location=LOCATION \
    --name=multi-node-job \
    --image=IMAGE \
    --command="python3 train.py" \
    --compute-type=TYPE \
    --num-nodes=N

To submit a multi-slice workload:

gcluster job submit \
    --cluster=CLUSTER_NAME \
    --project=PROJECT_ID \
    --location=LOCATION \
    --name=multi-slice-job \
    --image=IMAGE \
    --command="python3 train.py" \
    --compute-type=TYPE \
    --num-slices=N

To submit a Pathways workload:

gcluster job submit \
    --cluster=CLUSTER_NAME \
    --project=PROJECT_ID \
    --location=LOCATION \
    --name=pw-train-job \
    --compute-type=TYPE \
    --image=IMAGE \
    --pathways-server-image=PW_SERVER_IMAGE \
    --pathways-proxy-server-image=PW_PROXY_IMAGE \
    --command="python3 train.py" \
    --pathways \
    --pathways-gcs-location=PW_GCS_LOCATION