Troubleshoot Dataflow autoscaling

This page shows you how to resolve issues with the Dataflow autoscaling features and provides information about how to manage autoscaling.

Job doesn't scale up or down

This section provides information about scenarios that might prevent workers from scaling up or down.

Streaming job doesn't scale up

When your streaming pipeline has a backlog, the workers don't scale up.

This issue occurs when the backlog lasts less than a few minutes or when parallelism is limited.

Sometimes, backlog is elevated but parallelism is low. In that case, Dataflow doesn't scale up, because the work can't be distributed across more workers, so adding more workers won't help with processing. For more information, see Streaming autoscaling.

Batch and streaming jobs don't scale up

Your batch or streaming job runs as expected, but when more workers are needed, the job doesn't scale up.

This issue might occur for one of the following reasons:

• The staging or temp files are inaccessible. If your job uses a Cloud Storage bucket, the bucket might have a lifecycle configuration that deletes objects in the bucket. The deleted objects include staging and temp folders and files. To verify whether files have been deleted, check the lifecycle configuration for the bucket. If the staging or temp folders or files were deleted after the job started, the packages required to create new workers might not exist. To resolve this issue, recreate the folders and files in the bucket.
• Firewall rules prevent workers from sending and receiving traffic on the necessary TCP ports. Firewall rules might prevent workers from starting. Dataflow workers need to be able to send and receive traffic on TCP ports 12345 and 12346. For more information, including steps to resolve this issue, see Firewall rules for Dataflow.
• A custom source has a getProgress() method that returns a NULL value. When you use a custom source, the backlog metrics rely on the return value of your custom source's getProgress() method to start collecting data. The default implementation for getProgress() returns a NULL value. To resolve this issue, ensure that your custom source overrides the default getProgress() method to return a non-NULL value.
• An update triggered by Vertical Autoscaling temporarily deactivates Horizontal Autoscaling. For more information, see Effect on Horizontal Autoscaling.
• If you're using a map operation in a Python pipeline and your job doesn't scale up, you might need to add a Reshuffle transform to your pipeline code. For more information, see Reshuffle in the Apache Beam documentation.

Streaming job doesn't scale down

When your streaming job has a low backlog and low CPU utilization, the workers don't scale down. This issue can occur for various reasons.

• When jobs don't use Streaming Engine, Dataflow balances the number of persistent disks between the workers. As a result, each worker must have an equal number of persistent disks. For example, if there are 100 disks and 100 workers, each worker has one disk. When the job scales down, the job can have 50 workers with two persistent disks per worker. The job doesn't scale down again until it can have 25 workers with four persistent disks per worker. In addition, the minimum number of workers is the value assigned to maxNumWorkers divided by 15. For more information, see Scaling range for streaming autoscaling pipelines.

• When jobs use Streaming Engine, the downscaling target is based on a target CPU utilization of 75%. When this CPU utilization can't be achieved, downscaling is disabled.

• The backlog time estimate needs to stay below ten seconds for at least two minutes before workers scale down. Fluctuations in backlog time might disable scaling down. In addition, low throughput can skew the time estimate.

• PeriodicImpulse is supported in the Apache Beam SDK versions 2.60.0 and later. When your pipeline uses PeriodicImpulse with the Apache Beam SDK versions 2.59.0 and earlier, Dataflow workers don't scale down as expected.

Scaling up stops

Your batch or streaming job starts scaling up, but the workers stop scaling up even though a backlog remains.

This issue occurs when quota limits are reached.

• Compute Engine quotas: Dataflow jobs are subject to the Compute Engine quota of the project. If multiple jobs are running, the project might be at the limit of its Compute Engine quota. In that case, Dataflow can't increase the number of workers.
• CPU quotas: Dataflow jobs are also subject to the CPU quota of the project. If the worker type uses more than one CPU, the project might be at the limit of the CPU quota.
• External IP address quotas: When your job uses external IP addresses to communicate with resources, you need as many external IPs addresses as workers. When the number of workers scales up, the number of external IP addresses also increases. When you reach the IP address limit, the workers stop scaling up.

In addition, if the region you choose is out of a resource, you can't create new resources of that type, even if you have remaining quota in your region or project. For example, you might still have quota to create external IP addresses in us-central1, but that region might not have available IP addresses. For more information, see Quotas and resource availability.

To resolve this issue, request a quota increase or run the job in a different region.

The worker utilization hint has no effect

You set the worker utilization hint but the autoscaling behavior does not change.

To understand this issue, go to the Worker CPU utilization chart and check whether the worker utilization hint is actively used. If the hint is being used, the chart shows CPU utilization hint (actively used by autoscaler). Otherwise, it shows CPU utilization hint (not actively used by autoscaler).

The utilization hint is only one factor that affects autoscaling. The following table lists some reasons why the autoscaler might not actively use the hint:

For more information, see Streaming autoscaling heuristics.

Troubleshoot autoscaling schedules

If you encounter issues when creating, updating, or running autoscaling schedules, review the following common scenarios and troubleshooting steps.

Unable to set or update an autoscaling schedule

If an attempt to create or update an autoscaling schedule fails, verify that your schedule configuration complies with the following constraints:

• Schedule limit: A single job can have a maximum of 10 autoscaling schedules. To add a new schedule when the job already has 10 schedules, you must first remove an existing schedule.
• Minimum duration: The schedule duration must be at least 5 minutes (300 seconds).
• Parameter constraints:
  • The CPU utilization target must fall within [0.1, 0.9].
  • The maximum worker count must be greater than or equal to the minimum worker count (max_workers ≥ min_workers).
• Crontab format restrictions at job start: Crontab expressions specified at job launch using the --experiments=autoscaling_schedule=... pipeline option cannot contain commas (such as 1,15 * * * * or * * * * MON,WED,FRI). Commas are reserved to delimit pipeline options. If your schedule requires commas, launch the job without the schedule and configure it after launch using an in-flight update, or specify multiple separate schedules.

Schedule doesn't take effect or a different schedule is applied

If an autoscaling schedule doesn't take effect during its expected window, or if a different schedule's settings are applied, review the following:

• Only one schedule is active at a time: If two or more schedules have overlapping time windows, Dataflow determines which schedule to apply based on schedule priority:
  • The schedule with the higher priority value (priority) is applied.
  • If multiple active schedules share the same priority value, the most recently updated or modified schedule is applied.
  • When the active schedule's duration ends, Dataflow applies any other schedule that is still active, or reverts autoscaling behavior to the job's default settings.
• Time zone discrepancies: Verify that the time zone specified for your schedule matches the intended region and daylight saving time rules. By default, schedule start times are interpreted in UTC.
• Verify active schedules in the console: In the Google Cloud console, navigate to the Autoscaling tab on the Dataflow job details page to see which schedule is active. You can also inspect the configured schedules and their parameters by running the gcloud dataflow jobs describe command or by querying the REST API.

Pipeline doesn't scale up to the scheduled worker count

If a schedule activates with higher worker limits or lower utilization hints, but the pipeline doesn't scale up to the expected number of workers, check the following factors:

• Resource and quota limits: Scaling up requires sufficient resource quota in your Google Cloud project and region. Worker allocation might be prevented by:
  • Compute Engine instance quotas: Your project might reach its instance limit for the region.
  • CPU quotas: If your workers use multiple CPUs (for example, n1-standard-4), ensure that your project has sufficient CPU quota.
  • IP address quotas: Each worker VM requires an IP address. If the IP address quota in your VPC network or region is exhausted, new workers can't be provisioned.
  • Resource availability: If the region or zone experiences temporary resource exhaustion, Dataflow might be unable to create new worker instances even if quota is available. For more information, see Scaling up stops.

• Latency tier sensitivity: If the active schedule configures a latency tier of medium_latency or high_latency, Dataflow intentionally waits for more backlog to accumulate before triggering upscale events:

  • low_latency (default): scales up when backlog exceeds 15 seconds.
  • medium_latency: scales up when backlog exceeds 180 seconds (3 minutes).
  • high_latency: scales up when backlog exceeds 600 seconds (10 minutes).

  If your job appears to lag or delay scaling up during a scheduled window, verify whether the active schedule configured a higher latency tier.

• Key-based parallelism limits: Dataflow can't scale beyond the number of distinct processing keys available at that stage. If your data stream contains few unique keys, adding workers can't improve throughput, and Dataflow caps the worker count to avoid idle resources.

• Worker pool stability: If worker instances encounter crashes or out-of-memory (OOM) errors, Dataflow temporarily pauses autoscaling resize requests until the worker pool stabilizes.

Gaps in autoscaling metrics

There are short, temporary gaps in autoscaling metrics.

This issue can occur if backend tasks are restarted. These gaps in the metrics don't indicate a problem with autoscaling or the health of your streaming job.

CPU is unevenly distributed

When the job is autoscaling, CPU utilization is unevenly distributed among workers. Some workers have higher CPU utilization, system latency, or data freshness than others.

This issue can occur if your data contains a hot key. A hot key is a key with enough elements to negatively affect pipeline performance. Each key must be processed by a single worker, so the work can't be split between workers.

For more information, see the hot key error guidance.

The work item requesting state read is no longer valid on the backend

During communication between worker VM instances and Streaming Engine tasks in a streaming pipeline, the following error occurs:

The work item requesting state read is no longer valid on the backend.
The work has already completed or will be retried.
This is expected during autoscaling events.

During autoscaling, worker VM instances communicate with multiple Streaming Engine tasks, and each task serves multiple worker VM instances. Item keys are used to distribute the work. Each task and worker VM instance have a collection of key ranges, and the distribution of these ranges can change dynamically. For example, during autoscaling, job resizing can cause the key range distribution to change. When a key range changes, this error can occur. The error is expected, and unless you see a correlation between these messages and an underperforming pipeline, you can ignore it.

Insufficient Streaming Engine resources

If Streaming Engine cannot allocate the minimum number of workers that you request, the following error is returned:

Streaming Engine does not currently have enough resources available to fulfill
the request.

To resolve this issue, try setting a smaller minimum number of workers. See Set the autoscaling range.

Scaling range for streaming autoscaling pipelines

This section provides details about the scaling range for streaming autoscaling pipelines.

Maximum number of workers streaming autoscaling might use

Limit autoscaling to reduce the impact on billing

If you don't want autoscaling to increase your bill, you can limit the maximum number of workers that your streaming job can use.

Change the scaling range

For information about changing the scaling range on a streaming pipeline, see Set the autoscaling range.

Turn off autoscaling on streaming pipelines

To turn off autoscaling on streaming pipeline, follow these steps.

Use a fixed number of workers

For streaming jobs that don't use Streaming Engine, the default behavior is to use a fixed number of workers. To use streaming autoscaling with these pipelines, you must explicitly opt in as it's not on by default.