Report and repair faulty TPU hosts in All Capacity mode

If a TPU VM running in All Capacity mode experiences an unresolvable hardware issue (such as high inter-chip interconnect (ICI) latency or high temperatures), you can report its host as faulty. Reporting a faulty host prompts Compute Engine to mark the host for automatic repair.

Limitations

When you report a faulty host, the following limitations apply:

• Repair duration: Repairing the faulty host typically takes 3 to 14 days, during which the TPU VM remains stopped.
• Prerequisites: You can only report a faulty host if the TPU VM that runs on the host meets all of the following conditions:
  • The TPU VM is running.
  • The TPU VM is configured in All Capacity mode.
  • The TPU VM uses the reservation-bound provisioning model.
• Deletion constraint: If you delete a TPU VM while the reportHostAsFaulty operation is in progress, then the operation fails.

Before you begin

Select the tab for how you plan to use the samples on this page:

Required roles

To get the permissions that you need to report a faulty host, ask your administrator to grant you the following IAM roles:

• Compute Instance Admin (v1) (roles/compute.instanceAdmin.v1) on the TPU VM or the project
• To view the state of a faulty host report operation by using Cloud Logging: Logs Viewer (roles/logging.viewer) on the project

For more information about granting roles, see Manage access to projects, folders, and organizations.

These predefined roles contain the permissions required to report a faulty host. To see the exact permissions that are required, expand the Required permissions section:

You might also be able to get these permissions with custom roles or other predefined roles.

Understand the faulty host report process

After you report a faulty host for a TPU VM, the host repair process begins. The following details outline the scope, rate limits, and step-by-step process for All Capacity mode.

Specifications

• Supported machine types: TPU v6e and later.
• Rate limiting: No rate limits apply to the report faulty host API.

Process timeline

The following diagram illustrates the lifecycle of the host reporting and repair process:

When you report a faulty host, the following sequence occurs (corresponding to the numbered steps in the diagram):

1. Report the faulty host: You initiate the report using the gcloud command or REST API. The TPU VM remains in the RUNNING state during this initial check, which typically takes 10–12 minutes. To track the status, see Review report faulty host operations.
2. Mark host as faulty: Compute Engine confirms the request, creates the report operation, and marks the underlying physical host as faulty.
3. Stop the TPU VM: Once the host is marked, the TPU VM is stopped, and its state changes to REPAIRING.

4. Begin repair: The automatic host repair operation starts (usually within a minute after the VM stops).

5. Complete repair: The hardware repair process finishes on the physical host.

Report a faulty host

Report the faulty host. Select one of the following options.

gcloud compute instances report-host-as-faulty TPU_NAME \
  --fault-reasons=behavior=FAULT_BEHAVIOR,description=DESCRIPTION \
  --disruption-schedule=DISRUPTION_SCHEDULE

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instances/TPU_NAME/reportHostAsFaulty
{
  "disruptionSchedule": "DISRUPTION_SCHEDULE",
  "faultReasons": [
    {
      "behavior": "FAULT_BEHAVIOR",
      "description": "DESCRIPTION"
    }
  ]
}

Replace the following:

• PROJECT_ID: The ID of the project where the TPU VM exists.
• ZONE: The zone where the TPU VM exists.
• TPU_NAME: The name of the TPU VM.
• FAULT_BEHAVIOR: The type of fault observed. Use one of the following values:
  • PERFORMANCE: The host is experiencing performance degradation, such as high inter-chip-interconnect (ICI) latency.
  • SILENT_DATA_CORRUPTION: The host is experiencing silent data corruption (SDC).
  • BEHAVIOR_UNSPECIFIED: Use this to report an issue that does not fit in the other behavior groups.
• DESCRIPTION: A brief description of the issue, which is useful for debugging and tracking.
• DISRUPTION_SCHEDULE: The timing of the host disruption for repair. Use one of the following values:
  • IMMEDIATE: Stops the TPU VM and starts the host repair process immediately.
  • FUTURE: Schedules the host repair for a later time, allowing the VM to continue running temporarily if the workload can tolerate the fault.

Review report faulty host operations

After you report a faulty host, Compute Engine starts a series of operations to mark the host as faulty and prepares the host for repair. Specifically, during a report faulty host operation, the following process happens:

1. Mark the host as faulty. Compute Engine creates the report faulty host operation. The report faulty host operation then creates a sequence of sub-operations. These sub-operations mark the underlying host as faulty.

2. Prepare the host for repairs. After all sub-operations complete, the report faulty host operation starts. Compute Engine stops the TPU VM and starts the repair faulty host operation.

3. Report completion and repair the host. Compute Engine completes the report faulty host operation, and the host repair operation runs.

To track the status of report faulty host (compute.instances.reportHostAsFaulty) operations in your project, select one of the following options.

GET https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/operations?filter=items.operationType:compute.instances.reportHostAsFaulty

Replace the following:

• PROJECT_ID: the ID of your project.
• ZONE: the zone where the operations exist.

What's next?

• If you encounter issues when reporting a faulty host, then see Troubleshoot faulty host API.