Enable Agent Sandbox on GKE

This document explains how to enable the Agent Sandbox feature in a Google Kubernetes Engine (GKE) cluster. It also explains how to create a sandboxed environment on the cluster to safely execute untrusted code.

For an overview of how the Agent Sandbox feature isolates untrusted AI-generated code, see About GKE Agent Sandbox.

Costs

Agent Sandbox is offered at no extra charge in GKE. GKE pricing applies to the resources that you create.

To avoid unnecessary charges, ensure that you disable GKE or delete the project after you have completed this document.

Before you begin

1. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Roles required to select or create a project

   • Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
   • Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

   Go to project selector

2. Verify that billing is enabled for your Google Cloud project.

3. Enable the Artifact Registry, Google Kubernetes Engine APIs.

   Roles required to enable APIs

   To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

   Enable the APIs

4. In the Google Cloud console, activate Cloud Shell.

   Activate Cloud Shell

5. Ensure that your cluster is running GKE version 1.36.3-gke.1767000 or later (supports the v1beta1 API).

Define environment variables

To simplify the commands that you run in this document, you can set environment variables in Cloud Shell. In Cloud Shell, define the following useful environment variables by running the following commands:

export PROJECT_ID=$(gcloud config get project)
export CLUSTER_NAME="agent-sandbox-cluster"
export LOCATION="us-central1"
export CLUSTER_VERSION="1.36.3-gke.1767000"
export NODE_POOL_NAME="agent-sandbox-pool"
export MACHINE_TYPE="e2-standard-2"

Here's an explanation of these environment variables:

• PROJECT_ID: the ID of your current Google Cloud project. Defining this variable helps ensure that all resources, like your GKE cluster, are created in the correct project.
• CLUSTER_NAME: the name of your GKE cluster—for example, agent-sandbox-cluster.
• LOCATION: the Google Cloud region or zone where your GKE cluster is created. Set this to the region (for example, us-central1) if you create an Autopilot cluster, or the zone (for example, us-central1-a) if you create a Standard cluster.
• CLUSTER_VERSION: the version of GKE your cluster will run (1.36.3-gke.1767000 or later).
• NODE_POOL_NAME: the name of the node pool that will run sandboxed workloads—for example, agent-sandbox-pool. This variable is only required if you are creating a GKE Standard cluster.
• MACHINE_TYPE: the machine type of the nodes in your node pool—for example, e2-standard-2. For details about different machine series and choosing between different options, see the Machine families resource and comparison guide. This variable is only required if you are creating a GKE Standard cluster.

Enable Agent Sandbox

You can enable the Agent Sandbox feature when you create a new cluster, or when updating an existing cluster.

Enable Agent Sandbox when creating a new GKE cluster

We recommend that you use a Autopilot cluster for a fully managed Kubernetes experience. To choose the GKE mode of operation that's the best fit for your workloads, see Choose a GKE mode of operation.

gcloud beta container clusters create-auto ${CLUSTER_NAME} \
    --location=${LOCATION} \
    --cluster-version=${CLUSTER_VERSION} \
    --enable-agent-sandbox

Enable Agent Sandbox when updating an existing GKE cluster

To enable Agent Sandbox on an existing cluster, the cluster must be running version 1.36.3-gke.1767000 or later, which supports the v1beta1 API.

Ensure that your LOCATION environment variable is set to the region or zone where your existing cluster is located.

1. If you are using a GKE Standard cluster, Agent Sandbox relies on gVisor. If your Standard cluster doesn't have a gVisor-enabled node pool, you must create one first:

   gcloud container node-pools create ${NODE_POOL_NAME} \
       --cluster=${CLUSTER_NAME} \
       --machine-type=${MACHINE_TYPE} \
       --location=${LOCATION} \
       --image-type=cos_containerd \
       --sandbox=type=gvisor
   

2. Update the cluster to enable the Agent Sandbox feature:

   gcloud beta container clusters update ${CLUSTER_NAME} \
       --location=${LOCATION} \
       --enable-agent-sandbox
   

Verify the configuration

You can verify whether the Agent Sandbox feature is enabled by inspecting the cluster description.

gcloud beta container clusters describe ${CLUSTER_NAME} \
    --location=${LOCATION} \
    --format="value(addonsConfig.agentSandboxConfig.enabled)"

If you created an Autopilot cluster, the location is the region (for example, us-central1). If you created a Standard cluster, the location is the zone (for example, us-central1-a).

If the feature is successfully enabled, the command returns True.

Agent Sandbox deployment requirements

To successfully deploy a workload, such as a Sandbox or SandboxTemplate, your YAML manifest must include specific security and configuration settings. GKE enforces these requirements using a Validating Admission Policy (VAP). If these requirements are not met, the admission controller rejects the deployment.

Required configuration

Your deployment manifest must include the following settings:

• runtimeClassName: gvisor: ensures the Pod runs in a gVisor sandbox.
• automountServiceAccountToken: false: prevents the Pod from automatically mounting the default service account token.
• securityContext.runAsNonRoot: true: ensures the container does not run as the root user.
• securityContext.capabilities.drop: ["ALL"]: drops all Linux capabilities from the container.
• resources.limits: you must specify CPU and memory limits to prevent potential denial-of-service (DoS) scenarios.
• nodeSelector: must target sandbox.gke.io/runtime: gvisor.
• tolerations: must include a toleration for the sandbox.gke.io/runtime=gvisor:NoSchedule taint.

Prohibited configuration

Your deployment manifest must not include any of the following:

• hostNetwork: true, hostPID: true, or hostIPC: true.
• privileged: true in container security contexts.
• HostPath volumes.
• Added capabilities (capabilities.add).
• hostPort settings.
• Custom sysctls.
• Projected volumes for service account tokens or certificates.

Deploy a sandboxed environment

We recommend deploying a sandboxed environment by defining a SandboxTemplate and keeping pre-warmed instances ready using a SandboxWarmPool. You can then request an instance from this warm node pool using a SandboxClaim. Alternatively, you can create a Sandbox directly, but this approach doesn't support warm pools.

SandboxTemplate, SandboxWarmPool, SandboxClaim, and Sandbox are Kubernetes custom resources.

Recommended: Create a SandboxTemplate and SandboxWarmPool

The SandboxTemplate acts as a reusable blueprint. The SandboxWarmPool helps ensure that a specified number of pre-warmed Pods are always running and ready to be claimed. Use of this custom resource minimizes startup latency.

To deploy a sandboxed environment by creating the SandboxTemplate and SandboxWarmPool, complete the following steps:

1. In Cloud Shell, create a file named sandbox-template.yaml with the following content:

   apiVersion: extensions.agents.x-k8s.io/v1beta1
   kind: SandboxTemplate
   metadata:
     name: python-runtime-template
     namespace: default
   spec:
     podTemplate:
       metadata:
         labels:
           sandbox-type: python-runtime
       spec:
         runtimeClassName: gvisor # Required
         automountServiceAccountToken: false # Required
         securityContext:
           runAsNonRoot: true # Required
         nodeSelector:
           sandbox.gke.io/runtime: gvisor # Required
         tolerations:
         - key: "sandbox.gke.io/runtime"
           value: "gvisor"
           effect: "NoSchedule" # Required
         containers:
         - name: runtime
           image: registry.k8s.io/agent-sandbox/python-runtime-sandbox:v0.1.0
           ports:
           - containerPort: 8888
           resources:
             requests:
               cpu: "250m"
               memory: "512Mi"
             limits:
               cpu: "500m"
               memory: "1Gi" # Required
           securityContext:
             capabilities:
               drop: ["ALL"] # Required
         restartPolicy: OnFailure
   

2. Apply the SandboxTemplate manifest:

   kubectl apply -f sandbox-template.yaml
   

3. Create a file named sandbox-warmpool.yaml with the following content:

   apiVersion: extensions.agents.x-k8s.io/v1beta1
   kind: SandboxWarmPool
   metadata:
     name: python-runtime-warmpool
     namespace: default
     labels:
       app: python-runtime-warmpool
   spec:
     replicas: 2
     sandboxTemplateRef:
       # This must match the name of the SandboxTemplate.
       name: python-runtime-template
   

4. Apply the SandboxWarmPool manifest:

   kubectl apply -f sandbox-warmpool.yaml
   

Create a SandboxClaim

The SandboxClaim requests a sandbox from the warm pool. Because you created a warm pool, the created Sandbox adopts a running Pod from the pool instead of starting a fresh Pod.

To request a sandbox from the warm pool by creating a SandboxClaim, complete the following steps:

1. Create a file named sandbox-claim.yaml with the following content:

   apiVersion: extensions.agents.x-k8s.io/v1beta1
   kind: SandboxClaim
   metadata:
     name: sandbox-claim
     namespace: default
   spec:
     warmPoolRef:
       # This must match the name of the SandboxWarmPool.
       name: python-runtime-warmpool
   

2. Apply the SandboxClaim manifest:

   kubectl apply -f sandbox-claim.yaml
   

3. Verify that the sandbox, claim, and warm pool are ready:

   kubectl get sandboxwarmpool,sandboxclaim,sandbox,pod
   

Alternative: Create a Sandbox directly

If you don't need the fast startup times provided by warm pools, you can deploy a Sandbox directly without using templates.

To deploy a sandboxed environment by creating a Sandbox directly, complete the following steps:

1. Create a file named sandbox.yaml with the following content:

   apiVersion: agents.x-k8s.io/v1beta1
   kind: Sandbox
   metadata:
     name: sandbox-example-2
   spec:
     replicas: 1
     podTemplate:
       metadata:
         labels:
           sandbox: sandbox-example
       spec:
         runtimeClassName: gvisor
         restartPolicy: OnFailure
         automountServiceAccountToken: false # Required
         securityContext:
           runAsNonRoot: true # Required
           runAsUser: 1000 # Required if image defaults to root (e.g. busybox)
         nodeSelector:
           sandbox.gke.io/runtime: gvisor
         tolerations:
         - key: "sandbox.gke.io/runtime"
           value: "gvisor"
           effect: "NoSchedule" # Required
         containers:
         - name: my-container
           image: busybox
           command: ["/bin/sh", "-c"]
           args: ["sleep 3600000; echo 'Container finished successfully'; exit 0"]
           securityContext:
             capabilities:
               drop: ["ALL"] # Required
             allowPrivilegeEscalation: false
           resources:
             limits:
               cpu: "100m"
               memory: "128Mi" # Required
   

2. Apply the Sandbox manifest:

   kubectl apply -f sandbox.yaml
   

3. Verify that the sandbox is running:

   kubectl get sandbox
   

Migrate Agent Sandbox from v1alpha1 to v1beta1

If your cluster was deployed with an earlier version of Agent Sandbox by using v1alpha1 custom resources, you can upgrade to GKE version 1.36.3-gke.1767000 or later with near-zero workload downtime.

Note: This migration procedure applies to clusters using the managed GKE Agent Sandbox feature (--enable-agent-sandbox). If you deployed Agent Sandbox using open-source manifests, refer to the upstream migration guide.

Main API differences between v1alpha1 and v1beta1

Migrate using the migration tool

To automatically create shadow warm pools and re-persist storage, use the canonical migration script from the Agent Sandbox repository.

Download and prepare the script:

curl -LO https://raw.githubusercontent.com/kubernetes-sigs/agent-sandbox/v0.5.6/helm/files/migrate.sh
chmod +x migrate.sh

Step-by-step migration runbook

To migrate an existing cluster with near-zero workload downtime, complete the following three phases in order:

Phase 1: Pre-upgrade bootstrap phase

1. Back up existing resources: Save a YAML backup of all Agent Sandbox custom resources in your cluster:

   kubectl get sandboxtemplates,sandboxwarmpools,sandboxclaims,sandboxes \
       --all-namespaces -o yaml > agent-sandbox-v1alpha1-backup.yaml
   

2. Verify template security compliance: ensure that existing SandboxTemplate resources meet the Agent Sandbox deployment requirements. During the storage migration in Phase 3, the admission controller rejects updates to any templates that do not comply with these security policies.

3. Preview the shadow pools to be created:

   ./migrate.sh --phase=bootstrap --dry-run
   

4. Execute the bootstrap phase:

   ./migrate.sh --phase=bootstrap
   

5. Verify the created shadow pools:

   kubectl get sandboxwarmpools --all-namespaces \
       -o custom-columns="NAMESPACE:.metadata.namespace,NAME:.metadata.name,REPLICAS:.spec.replicas,SHADOW:.metadata.annotations.agents\.x-k8s\.io/migration-shadow"
   

Phase 2: Upgrade the GKE control plane

Upgrade the GKE control plane to version 1.36.3-gke.1767000 or later:

gcloud container clusters upgrade ${CLUSTER_NAME} \
    --location=${LOCATION} \
    --master \
    --cluster-version=1.36.3-gke.1767000

During the control plane rollout, note the following:

• Pods experience no restarts or downtime.
• The new controller and /convert webhook endpoint are deployed on the control plane.

Phase 3: Post-upgrade storage migration

After the control plane upgrade completes, refresh your credentials and rewrite the stored etcd objects by running the storage migration phase:

./migrate.sh --phase=migrate

Post-migration verification checklist

Disable Agent Sandbox

To disable the Agent Sandbox feature, use the gcloud beta container clusters update command with the --no-enable-agent-sandbox flag.

gcloud beta container clusters update ${CLUSTER_NAME} \
    --location=${LOCATION} \
    --no-enable-agent-sandbox

If you created an Autopilot cluster, the location is the region (for example, us-central1). If you created a Standard cluster, the location is the zone (for example, us-central1-a).

Clean up resources

To avoid incurring charges to your Google Cloud account, delete the GKE cluster that you created.

gcloud container clusters delete $CLUSTER_NAME \
    --location=${LOCATION} \
    --quiet

If you created an Autopilot cluster, the location is the region (for example, us-central1). If you created a Standard cluster, the location is the zone (for example, us-central1-a).

What's next

• Learn how to save and restore Agent Sandbox environments with Pod snapshots.
• Learn about the underlying technology used by Agent Sandbox.
• Learn more about GKE security.
• Explore the Agent Sandbox open-source project on GitHub.
• Learn how to use open source Kata Containers with Agent Sandbox. Kata Containers isn't a Google Cloud product. If you install and use this software, then you're responsible for management and troubleshooting. Google's support and SLAs don't apply to Kata Containers.