Configure Tier 1 networking for a VM

This document describes how to configure Tier 1 networking performance for certain virtual machine (VM) instances in Google Distributed Cloud (GDC) air-gapped. VMs with Tier 1 networking configurations are especially useful for large, distributed compute workloads with lots of heavy internode communications, such as high performance computing (HPC), machine learning (ML), and deep learning (DL).

This document is for developers in platform administrator or application operator groups that configure networking for VMs in a Distributed Cloud environment. For more information, see Audiences for GDC air-gapped documentation.

Before you begin

To use gdcloud command-line interface (CLI) commands, ensure that you have downloaded, installed, and configured the gdcloud CLI. All commands for Distributed Cloud use the gdcloud or kubectl CLI, and require an operating system (OS) environment.

Get the kubeconfig file path

To run commands against the Management API server, ensure you have the following resources:

1. Sign in and generate the kubeconfig file for the Management API server if you don't have one.

2. Use the path to the kubeconfig file of the Management API server to replace MANAGEMENT_API_SERVER in these instructions.

Request IAM roles

Contact your Project IAM Admin to request the following roles on your project:

• Virtual Machine Project Admin (project-vm-admin): create, modify, list, and delete VMs in the project namespace.

• Project Viewer(project-viewer): view all resources within the project namespaces.

All VM roles must bind to the namespace of the project where the VM resides. Follow the steps to verify your access.

Limitations

Tier 1 networking is supported with N2, N3, N4, A2, and A3 machine types that have at least 4 vCPUs.

Bandwidth tiers

The ingress and egress bandwidth limit represents the maximum possible amount of data per unit of time, such as gigabits per second (Gbps), that GDC allows a VM instance to transmit or receive from its network interfaces (NICs). The bandwidth includes data transferred between VM instances and object storage.

GDC has the following bandwidth limits:

• The default bandwidth limit is 10 Gbps.
• Tier 1 networking increases the maximum egress bandwidth limit for compute instances. The maximum egress bandwidth limit ranges from 16 Gbps to 25 Gbps, depending on the size and machine type of your instance.
• The actual egress bandwidth is always less than or equal to the egress bandwidth limit.

To achieve the highest possible egress bandwidth, all of the following must be true:

• The sending and receiving compute instances must be in the same zone.
• Packets sent between the instances must use internal IP address destinations.
• The number of TCP flows must be between 4 and 128 flows.
• The sending instances must have TCP Segmentation Offload (TSO) enabled, and the receiving instances must have Generic Receive Offload (GRO) enabled.
• The guest OS on the sending instance has Transmit Packet Steering (XPS) and Interrupt Request (IRQ) pinning configured. For more information, see Configure guest OS pinning.

N2 and N3 VMs

N4 VMs

A2 VMs

A3 VMs

Configure a VM with Tier 1 networking

Create instances and containers that use Tier 1 networking:

gdcloud compute instances create VM_NAME \
    --machine-type=MACHINE_TYPE \
    --image=BOOT_DISK_IMAGE_NAME --image-project=vm-system \
    --boot-disk-size=BOOT_DISK_SIZE \
    --network-performance-tier=tier_1

kubectl --kubeconfig MANAGEMENT_API_SERVER \
    apply -n PROJECT -f - <<EOF
apiVersion: virtualmachine.gdc.goog/v1
kind: VirtualMachineDisk
metadata:
  name: VM_BOOT_DISK_NAME
spec:
  source:
    image:
      name: BOOT_DISK_IMAGE_NAME
      namespace: vm-system
  size: BOOT_DISK_SIZE
---
apiVersion: virtualmachine.gdc.goog/v1
kind: VirtualMachine
metadata:
  name: VM_NAME
spec:
  compute:
    virtualMachineType: MACHINE_TYPE
  disks:
    - virtualMachineDiskRef:
        name: VM_BOOT_DISK_NAME
      boot: true
      autoDelete: BOOT_DISK_AUTO_DELETE
  network:
    networkPerformanceTier: Tier_1
EOF

Configure guest OS pinning for consistent performance

To prevent network throughput drops and maintain consistent network bandwidth on guest VMs using Tier 1 networking, configure Transmit Packet Steering (XPS) and Interrupt Request (IRQ) pinning in the guest operating system.

Without these configurations, the guest OS scheduler might consolidate all transmit queues and network interrupts onto a single CPU core under heavy network load, limiting throughput.

Before configuring XPS and IRQ pinning, find the number of virtual CPUs (vCPUs) assigned to your VM by running the following command on the guest VM:

nproc

Use this number to replace NUM_VCPUS in the following configurations.

Configure XPS pinning

Distribute the transmit (TX) queues across the virtual CPUs (vCPUs) assigned to the VM. A uniform distribution improves cache locality and balances processing.

To configure XPS pinning on the guest VM, run the following script:

NUM_VCPUS=NUM_VCPUS

for i in $(seq 0 $((NUM_VCPUS - 1))); do
    cpu=$(( i % (NUM_VCPUS / 4) ))
    mask=$(printf "%x" $((1 << cpu)))
    echo "Queue tx-$i -> CPU $cpu (mask $mask)"
    echo $mask > /sys/class/net/enp1s0/queues/tx-$i/xps_cpus
done

Configure IRQ pinning

Bind the network interface interrupts directly to the specific CPU cores responsible for processing those queues. This alignment minimizes Inter-Processor Interrupts (IPIs) and cache invalidations.

To configure IRQ pinning on the guest VM, run the following script:

NUM_VCPUS=NUM_VCPUS
irqs=$(grep "virtio0-" /proc/interrupts | awk -F: '{print $1}' | tr -d ' ')

idx=0
for irq in $irqs; do
    cpu=$(( idx % (NUM_VCPUS / 4) ))
    mask=$(printf "%x" $((1 << cpu)))
    echo "Mapping IRQ $irq to CPU $cpu (mask $mask)"
    echo $mask > /proc/irq/$irq/smp_affinity
    idx=$(( idx + 1 ))
done