Sequence the rollout of cluster upgrades with custom stages

If you manage rollouts across multiple fleets, we recommend that you use a dedicated project to host your RolloutSequence objects. This project acts as the parent and coordinator for the rollouts across the sequence. This project is typically not part of the sequence; that is, the project doesn't contain fleets or clusters which are part of the sequence.

Before you begin

• Install the Google Cloud CLI. After installation, initialize the Google Cloud CLI by running the following command:

  gcloud init

  If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

• Ensure that you have existing Autopilot or Standard clusters. To create a new cluster, see Create an Autopilot cluster.
• (Optional) If you don't already have a dedicated Google Cloud project to host your RolloutSequence configuration, create one by using the Google Cloud console or another method.
• Ensure that you have enabled the required APIs for fleets. These APIs must be enabled in your fleet host projects to create any type of rollout sequence. For your rollout sequence host project, enable the gkehub.googleapis.com API.
• For Terraform instructions, ensure that you use version 7.39.0 or later of the google provider.

Required roles

To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

To create or modify a rollout sequence, you must have the Fleet Editor IAM role (roles/gkehub.editor) on each fleet host project in the rollout sequence and your rollout sequence host project. This role provides the following permissions:

• gkehub.rolloutsequences.create
• gkehub.rolloutsequences.get
• gkehub.rolloutsequences.list
• gkehub.rolloutsequences.update
• gkehub.rolloutsequences.delete
• gkehub.fleet.update

These permissions let you create, access, and modify RolloutSequence objects, and use fleets in the rollout sequence.

If you need to register or unregister clusters to a fleet, you need all of the following permissions:

• Cluster registration permissions in your fleet host projects.
• Cluster admin permissions for any GKE clusters to be registered.
• Cross-project cluster registration permissions for any GKE clusters to be registered to a fleet in a different project.

You can create rollout sequences using fleets hosted in projects in one or more organizations, if you have the necessary permissions for those projects.

For more information about the least-privileged IAM roles required for different tasks, see Get predefined role suggestions with Gemini assistance.

Configure a rollout sequence

To create a rollout sequence, your clusters must be organized into groups of fleets. You can also create granular stages that can target specific subsets of clusters within a fleet by using Kubernetes labels. For guidance on how to organize your clusters, see the Community bank example. After you organize the clusters into groups and optionally label them, you create a rollout sequence by defining the ordered list of stages and the soak time for each group.

Organize clusters into fleets

In a rollout sequence, we recommend enrolling all clusters in the same release channel. If the clusters are not enrolled in the same channel, GKE selects a version from the most conservative channel in the sequence. For example, if clusters are enrolled in both the Stable and Regular channels, GKE chooses the version from the Stable channel. We also recommend that all clusters run the same minor version in order to be eligible for the same auto-upgrade target version.

If you have already organized your clusters into fleets, you can skip the following steps and proceed to the Create subsets of clusters section.

1. Group your clusters into fleets. You can organize your clusters by deployment environments such as Testing, Staging, and Production (recommended).
2. Register each cluster with a fleet based on your chosen grouping.

Create subsets of clusters (optional)

To make a stage in your rollout sequence target specific clusters, you must label those clusters.

For example, to test a new version on a small subset of clusters before a full rollout, you might apply a canary label to those clusters. To add the canary label with value true to a cluster using the Google Cloud CLI, run the following command:

gcloud container clusters update CLUSTER_NAME \
    --location=CLUSTER_LOCATION\
    --update-labels=canary=true

Replace the following:

• CLUSTER_NAME: the name of the existing cluster.
• CLUSTER_LOCATION: the Compute Engine region or zone of the cluster.

The --update-labels=canary=true flag instructs GKE to apply the canary label to the cluster .

For more information about adding a label to a cluster, see Add or update labels for existing clusters.

Create a rollout sequence with custom stages

A rollout sequence with custom stages defines the order of upgrades by declaratively defining the stages. With the gcloud CLI, you use a YAML file, and with Terraform, you add a resource block to your configuration. In performing the steps described in this section, you create a RolloutSequence.

To ensure that the sequence captures all clusters, each fleet must include a catch-all stage (a stage without a label selector). This catch-all stage captures all remaining clusters that GKE didn't select in earlier stages. If you assign a single cluster to multiple stages within one RolloutSequence, to resolve conflicts, GKE implicitly assigns the cluster only to the earliest stage.

The following example configurations creates three stages:

• The first stage targets all clusters in the dev fleet. After the upgrade is complete, there is a soak time of 7 days (7d).
• The second stage targets clusters in the prod fleet that have the label canary=true. After the upgrade is complete, there is a soak time of 7 days (7d).

• The third stage targets the remaining clusters in the prod fleet. After the upgrade is complete, there is a soak time of 7 days (7d).

Roll out a specific version

You can roll out a specific version to your rollout sequence. For more information, see Rolling out a specific version.

You can roll out a version to either the control planes or nodes of the clusters in your rollout sequence.

Roll out a version to the control planes in a rollout sequence

Run the following command:

gcloud container fleet rolloutsequences upgrade ROLLOUT_SEQUENCE_NAME\
    --control-plane-version="CONTROL_PLANE_VERSION"

Replace the following:

• ROLLOUT_SEQUENCE_NAME: the name of the rollout sequence.
• CONTROL_PLANE_VERSION: the version to roll out. You can choose a specific version, such as 1.34.7-gke.1055000, or use a version alias. For more information, see Specifying cluster version.

Roll out a version to the nodes in a rollout sequence

Run the following command:

gcloud container fleet rolloutsequences upgrade ROLLOUT_SEQUENCE_NAME \
    --node-version="NODE_VERSION"

Replace the following:

• ROLLOUT_SEQUENCE_NAME: the name of the rollout sequence.
• NODE_VERSION: the version to roll out, for example, 1.34.7-gke.1055000.

Check the status of a rollout

After you configure a rollout sequence, the system automatically creates Rollout objects to manage upgrades. You can observe and track the progress of these objects using Google Cloud CLI commands.

List rollouts

To list all active and historical rollouts in your rollout sequence host project, run the following command:

gcloud container fleet rollouts list --project=HOST_PROJECT_ID

Replace HOST_PROJECT_ID with the ID of your rollout sequence host project.

You can omit the --project=HOST_PROJECT_ID flag if you are already in the project where your rollout sequence is hosted.

The output is similar to the following:

NAME                                              STATE      CREATE_TIME
05eb251e4f19269e23-node-1-33-5-gke-1201000-t7mqd  COMPLETED  2025-10-30T20:07:46
05eb251e4f19269e23-kcp-1-33-5-gke-1201000-djwst   COMPLETED  2025-10-30T18:07:06
05eb251e4f19269e23-node-1-33-5-gke-1125000-6bxvu  COMPLETED  2025-10-23T17:46:54
05eb251e4f19269e23-kcp-1-33-5-gke-1125000-2f6ct   RUNNING    2025-10-23T16:41:33

In the preceding output, rollout names that contain kcp refer to control plane upgrades, and names that contain node refer to node upgrades. The segment of the rollout name after kcp or node is derived from the GKE version.

Describe a rollout

To get detailed information about a particular rollout, including the target version, state, and which clusters have been upgraded, use the describe command with the rollout ID that you obtained from the preceding command:

gcloud container fleet rollouts describe ROLLOUT_ID \
  --project=HOST_PROJECT_ID

Replace the following:

• ROLLOUT_ID: the rollout ID that you obtained when you listed the rollouts.
• HOST_PROJECT_ID: the project ID where your rollout sequence is hosted.

For example:

gcloud container fleet rollouts describe 927e9a989930cf3b55-kcp-1-32-4-gke-1106006 \
  --project=my-hostfleet

The output is similar to the following:

createTime: '2025-05-26T11:47:29.909959672Z'
membershipStates:
  projects/dev-project-id/locations/us-central1/memberships/c-1:
    lastUpdateTime: '2025-05-26T12:20:55.601542481Z'
    targets:
    - cluster:   projects/dev-project-id/locations/us-central1/clusters/c-1
      operation: //container.googleapis.com/v1/projects/dev-project-id/locations/us-central1/operations/operation-1234567890-abcdefg-hijklm-nopqrst
      state: SUCCEEDED
    stageAssignment: 1
  projects/dev-project-id/locations/us-central1/memberships/c-2:
    lastUpdateTime: '2025-05-26T12:22:57.151203493Z'
    targets:
    - cluster:   projects/dev-project-id/locations/us-central1/clusters/c-2
      operation: //container.googleapis.com/v1/projects/dev-project-id/locations/us-central1/operations/operation-987654321-ghijkl-mno-pqr-stu-vwxyz
      state: SUCCEEDED
    stageAssignment: 1
  projects/prod-project-id/locations/us-central1/memberships/c-1:
    lastUpdateTime: '2025-05-26T13:03:34.134308942Z'
    targets:
    - cluster: projects/prod-project-id/locations/us-central1/clusters/c-1
      operation: //container.googleapis.com/v1/projects/prod-project-id/locations/us-central1/operations/operation-567891234-efghij-klm-nopq-rstu-vwxyz
      state: SUCCEEDED
    stageAssignment: 2
  projects/prod-project-id/locations/us-central1/memberships/c-2:
    lastUpdateTime: '2025-05-26T13:06:34.025261641Z'
    targets:
    - cluster: projects/prod-project-id/locations/us-central1/clusters/c-1
      operation: //container.googleapis.com/v1/projects/prod-project-id/locations/us-central1/operations/operation-765432198-01a7b896-67c2-523-6fjjh4-icmdydh
      state: SUCCEEDED
    stageAssignment: 2
name: projects/user-hostfleet/locations/global/rollouts/05eb251e4f19269e23-kcp-1-32-4-gke-1106006
rolloutSequence: projects/project-id/locations/global/rolloutSequences/my-sequence
state: COMPLETED
updateTime: '2025-07-22T07:36:51.052691989Z'
versionUpgrade:
  desiredVersion: 1.32.4-gke.1106006
  type: TYPE_CONTROL_PLANE
stages:
- state: COMPLETED
  endTime: '2025-05-26T12:22:28.828506491Z'
  stageNumber: 1
  startTime: '2025-05-26T11:48:28.772658427Z'
  soakDuration: 600s
- state: COMPLETED
  endTime: '2025-05-26T13:06:20.026390832Z'
  stageNumber: 2
  startTime: '2025-05-26T12:32:38.419372153Z'
  soakDuration: 600s

Status information for a rollout

When you describe a rollout, the stages and membershipStates fields of the output provide the progress status of each stage and cluster within that stage, respectively.

The following table lists the potential statuses of a stage:

The following table lists the potential statuses of a cluster within a sequence:

Manage a rollout

You can manage an individual rollout by using the instructions provided in the following sections. For more information about managing a rollout and the corresponding actions that you can take, see Managing a rollout.

For managing an entire rollout sequence, and not only one specific rollout, see the following section, Manage a rollout sequence.

Pause a rollout

You can pause an active rollout. For more information about this action, see Pausing a rollout.

To pause a rollout, run the following command:

gcloud container fleet rollouts pause ROLLOUT_ID

Replace ROLLOUT_ID with the ID of the rollout that you want to pause. To get the rollout IDs for the rollouts in your rollout sequence, see List rollouts.

Resume a rollout

You can resume a paused rollout that's been paused for less than 90 days. For more information about this action, see Resuming a rollout.

To resume a paused rollout, run the following command:

gcloud container fleet rollouts resume ROLLOUT_ID

Replace ROLLOUT_ID with the ID of the rollout that you want to resume. To get the rollout IDs for the rollouts in your rollout sequence, see List rollouts.

Cancel a rollout

You can cancel a rollout, including both rollouts that are active or have been paused. You can't cancel rollouts for mandatory automatic upgrades. For more information about canceling rollouts, see Canceling a rollout.

To cancel a rollout, run the following command:

gcloud container fleet rollouts cancel ROLLOUT_ID

Replace ROLLOUT_ID with the ID of the rollout that you want to cancel. To get the rollout IDs for the rollouts in your rollout sequence, see List rollouts.

Complete a rollout stage

You can complete a rollout stage if you're ready for upgrades to proceed to the next stage. For more information about this action, see Completing a rollout stage.

To complete a rollout stage, run the following command:

gcloud container fleet rollouts force-complete-stage ROLLOUT_ID \
    --stage=STAGE

Replace the following:

• ROLLOUT_ID: the ID of the rollout where you want to complete a stage. To get the rollout IDs for the rollouts in your rollout sequence, see List rollouts.
• STAGE: the stage that you want to complete now. For example, if you want to complete the second stage, enter 2. The stage must be running or soaking to use this command.

Ignore a cluster in a rollout sequence for a specific rollout

You can have GKE ignore a cluster in a rollout sequence if you don't want the cluster to be upgraded as part of an active rollout. However, this action removes the cluster from the rollout sequence for active and future rollouts, not only active rollouts.

To perform this action to prevent a cluster from being upgraded in an active rollout, see Ignore a cluster in a rollout sequence.

Manage a rollout sequence

You can control automatic cluster upgrades with rollout sequencing in several ways, as explained in the following sections.

If you want to manage one rollout from a rollout sequence instead of the entire rollout sequence and all its rollouts, see Manage a rollout.

List your rollout sequences

To list all rollout sequences in your host project, run the following command:

gcloud container fleet rolloutsequences list --project=HOST_PROJECT_ID

Replace HOST_PROJECT_ID with the ID of your rollout sequence host project.

Describe a rollout sequence

To see the details of a specific rollout sequence, run the following command:

gcloud container fleet rolloutsequences describe ROLLOUT_SEQUENCE_NAME \
  --project=HOST_PROJECT_ID

Replace the following:

• ROLLOUT_SEQUENCE_NAME: the name of the rollout sequence.
• HOST_PROJECT_ID: the ID of your rollout sequence host project.

The output is similar to the following:

createTime: '2025-10-23T16:40:16.403871189Z'
displayName: my-display-name
name: projects/HOST_PROJECT_ID/locations/global/rolloutSequences/ROLLOUT_SEQUENCE_NAME
stages:
- clusterSelector:
    labelSelector: resource.labels.canary=='true'
  fleetProjects:
  - projects/FLEET_PROJECT_ID
  soakDuration: 600s
- fleetProjects:
  - projects/FLEET_PROJECT_ID
  soakDuration: 300s
uid: 5c5b2ac8-9d76-45f9-92ca-5e6bd3fbcaef
updateTime: '2025-10-23T17:11:57.285678399Z'

Choose what types of upgrades GKE performs in a rollout sequence

By default, GKE performs all types of cluster upgrades in a rollout sequence. You can restrict the scope of cluster upgrades in a rollout sequence to perform only specific types of upgrades. For information, see Choosing what types of upgrades GKE performs in a rollout sequence.

To designate what types of upgrades you want GKE to perform in a rollout sequence, run the following command:

gcloud container fleet rolloutsequences update ROLLOUT_SEQUENCE_NAME \
    --auto-rollout-scope=TYPES_OF_UPGRADES

Replace the following:

• ROLLOUT_SEQUENCE_NAME: the name of the rollout sequence.

• TYPES_OF_UPGRADES: the list of upgrades to perform, which can be the following:

  • all: the default, GKE performs all types of cluster upgrades in the rollout sequence.
  • none: GKE doesn't perform any cluster upgrades in the rollout sequence, except for rollouts for mandatory automatic upgrades.

  • One or more of the following; if plural, put the values in a comma-separated list with no spaces:

    • control-plane-patch: GKE performs automatic cluster control plane upgrades to later patch versions.
    • control-plane-minor: GKE performs automatic cluster control plane upgrades to later minor versions.
    • nodes-patch: GKE performs automatic cluster node upgrades to later patch versions.
    • nodes-minor: GKE performs automatic cluster node upgrades to later minor versions.

    For example, if you want only patch versions to roll out in the rollout sequence, specify control-plane-patch,nodes-patch.

Modify a rollout sequence

You can modify an existing rollout sequence after you've created it. Modify your rollout sequence with the following instructions, depending on how you're managing your rollout sequence.

gcloud container fleet rolloutsequences update test-rollout-sequence \
  --display-name="My Updated Rollout Sequence" \
  --stage-config=rollout-sequence.yaml

Ignore a cluster in a rollout sequence

You can have GKE ignore a cluster in a rollout sequence. You can ignore one or more clusters in a rollout sequence. For more information, see Ignoring a cluster in a rollout sequence.

To have GKE ignore a cluster in a rollout sequence, do the following steps:

1. Add a label to the cluster that you want GKE to ignore, or use an existing label. You can have GKE ignore, for example, either a label such as status, or ignore a label when it has a specific value, such as status=quarantine.

2. Set the rollout sequence to ignore clusters with this label:

   gcloud container fleet rolloutsequences update ROLLOUT_SEQUENCE_NAME \
       --ignored-clusters-selector="resource.labels.LABEL_TO_IGNORE"
   

   Replace the following:

   • ROLLOUT_SEQUENCE_NAME: the name of the rollout sequence.
   • LABEL_TO_IGNORE: the name of the label for clusters that GKE should ignore for the rollout sequence.

These instructions configure GKE to ignore one type of label in the rollout sequence. If you want to ignore multiple labels or create more complex expressions for how to ignore clusters with specific labels, you can pass values to this flag by using the CEL expression syntax. All labels must use the resource.labels. prefix.

Add a cluster back to a rollout sequence

If you followed the instructions in the previous section for GKE to ignore a cluster in a rollout sequence, you can undo this action. If you want GKE to not ignore clusters anymore, remove the ignore selector with the following command:

gcloud container fleet rolloutsequences update ROLLOUT_SEQUENCE_NAME \
    --ignored-clusters-selector

This command resets the ignore selector so that GKE won't ignore any clusters in the sequence.

Or, if you want GKE to not ignore a specific cluster, remove the labels from the cluster.

Delete a rollout sequence

To delete a rollout sequence, run the following command:

gcloud container fleet rolloutsequences delete ROLLOUT_SEQUENCE_NAME \
  --project=HOST_PROJECT_ID

Replace the following:

• ROLLOUT_SEQUENCE_NAME with the name of the rollout sequence.
• HOST_PROJECT_ID with the ID of your rollout sequence host project.

When you delete a rollout sequence, any rollouts that are in progress for that sequence are cancelled. The clusters that were part of the sequence revert to the default auto-upgrade behavior for their enrolled release channel.

Migrate an existing rollout sequence to use custom stages

If you use the fleet-based version of rollout sequencing, you can migrate to a sequence that uses custom stages by creating a new RolloutSequence that references your existing fleets. Note that this version of rollout sequencing doesn't have Google Cloud console support.

To migrate your rollout sequence, complete the following steps:

1. Optionally, before you migrate the sequence, make a copy of your current rollout sequence configuration. To make a copy of the configuration, do one of the following:

   • gcloud CLI: run the command to check the status of a fleet-based rollout sequence. Save the output of the command.
   • Terraform: save the Terraform configuration.

   We recommend that you don't delete your fleet-based rollout sequence, to make it easier to revert to the earlier version of rollout sequencing, if necessary. You don't need to delete the fleet-based rollout sequence for the rollout sequence with custom stages to work, because GKE adheres only to the rollout sequence with custom stages when both exist.

2. Create a dedicated Google Cloud project to host your rollout sequence. This project is typically not part of the sequence; that is, the project doesn't contain fleets or clusters which are part of the sequence.

3. If you want the rollout sequence to include specific clusters within a fleet, add labels to those clusters. This step is optional.

4. Follow the instructions in Create a rollout sequence with custom stages.

   For example, the following manifest, which is named rollout-sequence-migrate.yaml, references the existing fleets in a previous rollout sequence. This manifest describes three stages, including a canary stage in the prod fleet:

   - stage:
     fleet-projects:
     - projects/dev
     soak-duration: 604800s
   - stage:
     fleet-projects:
     - projects/prod
     soak-duration: 604800s
     label-selector: canary=true
   - stage:
     fleet-projects:
     - projects/prod
     soak-duration: 604800s
   

Immediately after you define a new RolloutSequence for your fleets, GKE begins upgrading the fleets according to the new sequence, and removes the previous configuration.

Migrate a rollout sequence with custom stages to the previous rollout sequence

This section describes how to revert from rollout sequencing with custom stages back to the fleet-based rollout sequencing model. This process involves deleting the new RolloutSequence and restoring your original fleet-based configuration.

Prevent out-of-order upgrade during migration

To avoid unintended or out-of-order upgrades while you are reconfiguring your sequence, apply a maintenance exclusion to your production clusters. This step temporarily pauses all automatic upgrades on those clusters. For example, you can configure a maintenance exclusion of type no upgrades on your production clusters.

Delete the rollout sequence

Delete the RolloutSequence object that manages your clusters. This deletion disengages the custom stages feature.

To delete the RolloutSequence, run the following command:

gcloud container fleet rolloutsequences delete ROLLOUT_SEQUENCE_NAME

Replace ROLLOUT_SEQUENCE_NAME with the name of your rollout sequence.

Restore your earlier rollout sequence configuration (without custom stages)

After you delete the RolloutSequence, you can restore your original fleet-based configuration. This process involves re-creating the clusterupgrade features with their original parameters, including the upstreamFleet links and soak times for each fleet in your sequence. For more information, see Create a rollout sequence.

Remove the maintenance exclusions

After you restore your original fleet-based rollout sequencing configuration, remove the maintenance exclusion you applied in the first step in this section. GKE resumes the automatic upgrades, now governed by your restored fleet-based sequence.

What's Next

• Learn more about cluster upgrades with rollout sequencing.