Monitor Cloud TPU health

Cloud TPU health monitoring provides real-time information on the health status of TPU VMs and slices. Using either the Cloud Monitoring console or Google Cloud CLI, you can identify hardware failures as they occur and reallocate resources, avoiding job failures and minimizing performance degradation.

TPU health monitoring assigns either a HEALTHY or UNHEALTHY status to each TPU instance in use. TPUs functioning as expected are assigned a HEALTHY status, and those that are no longer functional or performing at a severely reduced level are assigned an UNHEALTHY status.

Along with a health status, TPU health monitoring provides logging metrics for specific jobs. You can use these metrics to further diagnose TPU health and performance issues that are not caused by hardware issues.

Unhealthy TPU detection

When TPU health monitoring assigns an UNHEALTHY status to a TPU, it provides identifying information of the failing TPU through the compute.googleapis.com/instance/tpu/infra_health service endpoint.

TPU health status causes

A machine is categorized as UNHEALTHY when there are issues with the physical hardware. When a machine is categorized as UNHEALTHY, TPU health monitoring provides one of the following causes:

• BAD_TPU_CHIP: A physical failure within the TPU hardware itself. The VM may be running as intended, but a TPU connected to the VM is faulty.
• NETWORKING_ISSUE: Hardware issues in the underlying network communication between the different workers or components of the distributed TPU system, such as dropped transfer Remote Procedure Calls (RPCs).
• BAD_SC_CHIP: A physical failure within a SparseCore of a TPU chip.

Metric labels

TPU health monitoring provides the following metric labels to identify the specific unhealthy TPU:

TPU health monitoring also provides the following resource labels:

TPU health monitoring provides the following additional metric labels for All Capacity mode reservations through TPU Cluster Director:

Logging

A machine without any physical issues may be impaired or non-functional due to issues unrelated to hardware. In these cases, TPU health monitoring continues to categorize the machine as HEALTHY, but also logs potential causes of the issue with Cloud Logging in the associated project. The logs provide details about the issue, including a brief description, the suspected root cause, potential culprits, and actionable next steps.

The logs contain one of the following causes:

• FINGERPRINT_MISMATCH: Programs expected to be running the same TPU module contain different binary fingerprints. This is likely due to inconsistencies caused by compiler issues.
• DATA_INPUT_STALL: TPUs are stuck during Direct Memory Access (DMA) operations or data transfers. TPUs cannot continue until DMA operations complete.
• UNRECOVERABLE_ERROR: One or more workers encountered an unrecoverable error and are in an undefined state.
• DIFFERENT_MODULE: Workers are executing different TPU modules when they are expected to be running the same module.
• UNKNOWN_CAUSE: Undetermined cause of error.

You can view logs using the Logs Explorer or Logs API. For more information about Google Cloud logging, see Google Cloud Logging.

Monitoring with the console

The Monitoring Dashboard in the Google Cloud console provides real-time visualizations of machine health status, historical trends, and total counts. The Failure Event table links job failures to their root causes.

To view a prebuilt a dashboard in Cloud Monitoring:

1. In the Google Cloud console, go to the Cloud Monitoring page.
   Go to the Monitoring console
2. In the navigation pane, click Dashboards.
3. In the Filter search field, enter "TPU Instance Health Dashboard".

The dashboard displays the following metrics by default:

• Overall TPU instance health: The percentage of TPU instances that are categorized as HEALTHY.
• Number of instances: The total number of TPU instances in use.
• Number of healthy TPU instances: The total number of TPU instances categorized as HEALTHY.
• Number of unhealthy TPU instances: The total number of TPU instances categorized as UNHEALTHY.
• VM infra health distribution: The number of HEALTHY instances over time.

To add a custom query, click Add Query and write a PromQL query. For example, the following query retrieves all unhealthy TPU instances:

count({__name__="compute.googleapis.com/instance/tpu/infra_health", monitored_resource="gce_instance", health_status="UNHEALTHY"})

For more information on monitoring with the Google Cloud console, see the Cloud Monitoring documentation.

Monitoring with the CLI

The Google Cloud CLI provides real-time TPU health data. The following example request retrieves all UNHEALTHY TPU:

export TOKEN=$(gcloud auth application-default print-access-token)

export PROMQL_QUERY='count({__name__="compute.googleapis.com/instance/tpu/infra_health", monitored_resource="gce_instance", health_status="UNHEALTHY"})'

export LOCATION="global"

curl -G \
  --header "Authorization: Bearer ${TOKEN}" \
  --header "Content-Type: application/json" \
  --header "X-Goog-User-Project: ${PROJECT_ID}" \
  --data-urlencode "query=${PROMQL_QUERY}" \
  "https://monitoring.googleapis.com/v1/projects/${PROJECT_ID}/location/${LOCATION}/prometheus/api/v1/query"

You should expect to see a response similar to the following:

{"status":"success","data":{"resultType":"vector","result":[65]}}

For more information on gcloud CLI, see gcloud CLI overview.