Google Distributed Cloud air-gapped 1.16.2 hotfixes

Hotfix 3

The 1.16.2-gdch.1353-3 hotfix is available.

Storage:

• Enhanced internal storage monitoring and diagnostics by introducing the following:

  • End-to-end file storage user journey probers.
  • Orphaned ONTAP volume detection.
  • New CSI and user journey health dashboards.
  • Secure CSI and Trident metrics scraping.

Storage

• Provided the following updates to fix bugs and improve control plane stability:

  • Upgraded NetApp Trident and CSI sidecar components.
  • Increased CSI worker concurrency.
  • Added configurable ONTAP API timeouts.
  • Restricted Trident controller scheduling to control plane nodes to improve storage control plane throughput and prevent connectivity issues on worker and GPU nodes.

Hotfix 2

Backup

• Added a deactivate option for virtual machine backup plans.

• Conditionally enabled liveness probes for backup cluster and virtual machine controllers on non-root-admin clusters.

CLI

• Set --auto-diff as the default run mode for the gdcloud system assets add command.

Cluster management

• Fixed an issue to populate subnet status with the correct cluster namespace.

Documentation

• Updated the GDC offline documentation hosted in the GDC console to fix an issue causing tabbed content to not render.

Identity and access management

• Added a missing rotatableSecret resource for Identity and Access Management.

• Optimized reconciliation by skipping zonalRoleRef updates when the role name is unchanged.

• Updated the ZonalRoleBindingReplica reconciler to properly handle user and administrator role combinations.

Inventory

• Updated inventory ingestion validation for B300 servers.

• Added inventory support for trunk cables, unified management ports, and an arbitrary number of data ports.

• Added support for placing power distribution units (PDUs) in any rack unit (U) slot instead of restricting them to U02.

• Removed dangling patch panel checks and added top-of-rack (ToR) switch pairing exceptions for dangling patch panels.

• Removed username and password fields from inventory switch configuration objects.

Logging

• Fixed log routing to direct standard Kubernetes cluster system logs to the application operator (AO) tenant.

Monitoring

• Added the node_name label to node exporter metrics.

Network Time Protocol (NTP)

• Increased memory limits and added caching for NTP services to improve stability.

Networking

• Added support for host network (HNETUpgrade) data plane RPM hotpatch upgrades, including OCI artifact packaging for onetd, libgdc-ipsec-offload, and libstrongswan-doca, Kubernetes Job-based Ansible execution, and upgrade failure alerts.

• Skipped checking the use verb on subnets that are propagated from the global API server.

• Enabled subnet consumer tracking in production and added cluster watch reconciliation for subnets in use.

• Fixed an issue in SwapEndFrontBacks where string replacement corrupted device names during port validation.

• Added support for three-letter rack names in management switch configurations.

• Added the commonName field to the apiserver-proxy-lancer-mgmt certificate manifest.

Platform authentication

• Enabled the Certificate Authority Service data plane feature flag in production and added CAS-Proxy service level objectives (SLOs).

Servers

• Updated server provisioning to install host network deliverables and complete ConnectX-7 (CX7) firmware provisioning jobs before marking servers as ready.

• Enabled the SERVAllowDellIDRACDowngrade feature flag in production to allow Dell iDRAC and BIOS firmware downgrades for the d4-highgpu2 machine class.

• Added the SERV-R0051 runbook to document procedures for downgrading Dell server firmware.

Storage

• Added on-demand Range-Read Helper (RRH) reconstruction, automatic recovery, orphan cleanup, and metrics to improve range read and multipart upload reliability.

• Optimized bucket reconciliation by skipping redundant StorageGRID updates for CORS, lifecycle, and locking policies when configurations are unchanged.

• Improved presigned URL lifecycle security and redacted S3 signatures and credentials from error logs.

• Fixed a request pointer race condition in the object storage proxy handler, improved response filter error logging, and added payload transfer instrumentation.

• Updated the StorageGRID hotfix version to 11.9.0.14 and skipped ObjectStorageUpgradeRequest custom resource creation on non-anchor zones.

Virtual machines

• Updated the virtual machine package repository gateway to use STRIP_LEADING_DIRECTORY_PATH for operating system (OS) package routing.

Hotfix 1

CLI

• Added support for diff-based OCI package loading in the gdcloud system container-registry load-oci command to reduce download and upload sizes for hotfix releases. Diffs are always compared against the base release, not against preceding hotfix versions. When loading a diff package, the command skips duplicate image layers and blobs that already exist in the base release. Because hotfixes are cumulative, each hotfix release package contains all fixes introduced since the base release, excluding duplicate artifacts from the base release.

Database service

• Updated Database Service to version 0.16.6 and Database Service console UI to version 2.16.0-beta.31 to apply the latest bug fixes and stability improvements.

Documentation

• Updated the GDC offline documentation hosted in the GDC console Help menu with the latest content. For more information, see GDC console access.

Firewall

• Added the ZoneProtectionProfile custom resource data model, custom profile generation and claims, validation webhooks, and reconnaissance protection support for firewall nodes. Using the ZoneProtectionProfile custom resource (zoneprotectionprofiles.firewall.private.gdc.goog/v1alpha2), operators can configure protection at the organization level, including the root organization, to block malicious traffic and port scans. Operators can also customize scan thresholds, define IP address allowlists (exclusionIPs), change protection actions (alert, allow, block, block-ip), or selectively disable scan protections (enabled: false).

Gemini

• Updated the Gemini Inference Stack operator and model server templates to remove unsupported server configs and apply stability fixes.

Identity and access management

• Fixed a timeout issue with rotatableSecret in Anthos Identity Service (AIS) that caused identity service reconciliation delays.

• Enabled cookie splitting support in AIS to resolve large authentication token issues, and updated identity service container images.

Inventory

• Added support for B300 rack expansion.

• Fixed an issue where auto-generated passwords containing unsupported special characters caused switch provisioning and object storage bootstrap failures.

• Updated rack and instance IDs to support four-character identifiers for GPU pods, and removed unnecessary cable, adapter, and transceiver validations during preflight checks.

• Added support for management IP address and CIDR expansion after decommissioning a middle rack (a rack deployed after racks A and B, but before racks X and Y). Previously, if this middle rack was decommissioned, deploying or expanding to a new rack later would trigger IP address allocation issues. This fix resolves those conflicts, ensuring seamless rack expansion.

• Added support for iDRAC and OCP port names in inventory webhook validation.

Logging

• Optimized LoggingRule filter expressions to prevent high query rates (QPS) on object storage.

Monitoring

• Fixed an issue where incorrect metrics port configurations on the EZ infrastructure controller caused high-frequency EZ-E1001, EZ-E1003, and EZ-E1004 alerts to fire erroneously.

• Improved MonitoringTarget reconciler resilience by automatically requeueing targets after certificate retrieval failures and preventing unnecessary patches on persistent volume claims (PVCs).

• Fixed an issue where restartedAt annotations caused unnecessary restarts during Prometheus StatefulSet comparison.

• Removed legacy Google-Built OpenTelemetry Collector (GBOC) network policies in the monitoring namespace.

Networking

• Fixed an issue where subnetClaimRef wasn't populated when a server was located in a different rack from its management switch.

• Fixed an issue where device category CIDRs were missing for existing management switches during vertical expansion.

• Added support and generic regular expression validation for B300 server management and data port naming formats (such as fe-s6p1 and ocp-s1p1).

• Fixed JSON parsing errors in VLAN interface tables when the interface list wraps across multiple lines.

• Added a host network stage to rollout policies to ensure reliable network staging during component updates.

• Disabled ABM Gateway resource installation to avoid reconciliation errors in unsupported environments.

• Fixed connectivity tests to treat media converters as passive devices during validation.

• Improved error handling for invalid port formats during switch port connection validation.

Operating system

• Improved file access policy daemon (fapolicyd) performance and stability on nodes by increasing cache size and optimizing trust database rules.

Security

• Added support for multiple root certificates during package validation preflight checks.

Servers

• Added support for Dell iDRAC and BIOS firmware downgrades for the d4-highgpu2 machine class.

Upgrade

• Added safeguards to prevent re-evaluation of completed upgrades and improved client validation error handling during upgrade orchestration.

• Added support for a new diff-download mode for hotfixes, to download only the changed artifacts compared to the base release, drastically reducing the download size.

Vertex AI

• Updated Vertex AI Search container images and prediction_proxy to apply the latest security and stability updates.

Virtual machines

• Added support for running workloads that require GPUs on standard Kubernetes clusters.

• Marked shared VM and virtual machine image resources as audit-relevant in AuditlogInventory to ensure comprehensive audit logging.

• Fixed an issue in image-controller where missing cache checks caused excessive HTTP 409 conflict errors during concurrent image creation.

• Fixed an issue affecting virtual machine image import stability.

Vulnerability management

• Fixed vulnerability report import workflows to wait for completion before proceeding.

• Skipped unnecessary Security Compliance Framework for Google Cloud (SCFGM) generation for Windows OIC nodes.

• Made database CPU and memory resource allocations configurable and increased default database resource limits for the vulnerability scanning database.