Credential vending overview

Credential vending for the Lakehouse runtime catalog lets you delegate Cloud Storage access to query engines and enforce fine-grained Identity and Access Management (IAM) policies at the catalog, namespace, and table levels using the Google Cloud console or the Google Cloud CLI. Credential vending is also required to enable automatic table management for background maintenance tasks such as compaction, snapshot expiration, and garbage collection.

How credential vending works

When you use credential vending, the sequence of operations changes slightly to enforce policies before storage is accessed. The following diagram illustrates how query engines obtain downscoped storage tokens from the catalog to securely read and write data directly in Cloud Storage:

Query engines obtain downscoped storage tokens from the Lakehouse runtime catalog to read and write data in Cloud Storage.
Credential vending workflow delegating Cloud Storage access.

The sequence of operations shown in the preceding diagram is as follows:

  1. Request: A user submits a SQL statement, such as a query, DML, or DDL operation, to a supported engine such as Apache Spark or BigQuery.
  2. Metadata lookup: The engine sends a request to the Lakehouse runtime catalog to resolve the table.
  3. Authentication and policy: The catalog authenticates the user and checks their IAM permissions on Lakehouse resources.
  4. Response: Because credential vending is enabled, the catalog returns the metadata and a short-lived storage token (downscoped storage credentials) to the engine. The token is scoped to the paths associated with the table and downscoped to the minimum required permissions (for example, read-only access for queries that don't need to write files).
  5. Access: The engine uses this token to read or write the specific authorized files directly from Cloud Storage.
  6. Compute: The engine processes the data or executes the operation, and returns the results.

When automatic table management (gcp.biglake.table-management.enabled) is enabled, background maintenance tasks such as compaction, snapshot expiration, and garbage collection use the catalog's auto-provisioned service account to authenticate directly to Cloud Storage and update the underlying data files.

Supported engines

To use credential vending with query engines, you must configure the Apache Iceberg REST catalog endpoint in your Lakehouse runtime catalog to support credential vending. Credential vending isn't supported for Apache Hive catalog endpoints.

  • Open source engines: Supported engines like Spark and Trino use short-lived storage tokens vended by the catalog. Your client application must specify support for credential vending in the X-Iceberg-Access-Delegation header.
  • BigQuery: BigQuery uses vended credentials for Cloud Storage access instead of end-user credentials.

Required service account permissions

When credential vending is enabled, ensure the following service accounts have the necessary roles:

  • Auto-provisioned Lakehouse runtime catalog service account: You must grant the Storage Object User role (roles/storage.objectUser) to this account on all associated Cloud Storage buckets. Without this access, vended credentials can't read or write to storage. If you use the Google Cloud console, clicking Set bucket permissions verifies this role. For the gcloud CLI, Terraform, or the API, you must grant this role manually.
  • Query engine service accounts: Service accounts running query engine jobs (such as Managed Service for Apache Spark, Managed Service for Apache Spark, or Dataflow) require the BigLake Editor role (roles/biglake.editor) to obtain vended credentials with write scope.

What's next