Create a mixed-protocol load balancer

This document shows you how to expose an application running in a Google Kubernetes Engine (GKE) cluster by using a mixed-protocol external or internal LoadBalancer Service for both TCP and UDP traffic.

To learn more about L4 load balancers, see About LoadBalancer Services.

Overview

You can expose applications that use both TCP and UDP protocols by using two separate GKE LoadBalancer Services with a manually coordinated, shared IP address. However, this approach is inefficient because it requires managing multiple Services for a single application and might lead to issues such as configuration errors or exhausted IP address quotas.

Mixed-protocol LoadBalancer Services let you use a single Service to manage traffic for both TCP and UDP. Using a single Service simplifies your configuration by letting you use a single IP address and a consolidated set of forwarding rules for both protocols. This feature is supported for both external passthrough Network Load Balancers and internal passthrough Network Load Balancers.

GKE supports mixed-protocol Services with IPv4, IPv6, and dual-stack configurations. Mixed-protocol Services use Layer 3 (L3) forwarding rules. By design, these rules forward all traffic that arrive on the load balancer's virtual IP (VIP) address directly to the cluster nodes.

To maintain security, GKE automatically creates and manages firewall rules on priority 999 that allow only the specific TCP and UDP traffic defined in your Service manifest. All other unauthorized traffic directed to the load balancer's VIP address is blocked by GKE managed firewall rules by using priority 1000. If you create higher-priority firewall rules, ensure that the rules don't accidentally allow unauthorized traffic to reach your nodes. For more information, see GKE Service firewall rules.

Before you begin

Before you start, make sure that you have performed the following tasks:

  • Enable the Google Kubernetes Engine API.
    • Enable Google Kubernetes Engine API
  • To use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running the gcloud components update command. Earlier gcloud CLI versions might not support running the commands in this document.

Requirements

To create a LoadBalancer Service that uses mixed protocols, your cluster must meet the following requirements:

  • Mixed-protocol load balancing is generally available from GKE version 1.36.2-gke.1498000 and later. The generally available version supports both external and internal load balancers with IPv4, IPv6, and dual-stack configurations.
  • For versions 1.34.1-gke.2190000 through to 1.36.2-gke.1498000, mixed-protocol load balancing is supported only for external load balancers that use IPv4 addresses.
  • You must have the HttpLoadBalancing addon enabled on your cluster.
  • For internal load balancers, the cluster must have GKE subsetting enabled.
  • For new internal LoadBalancer Services, in the Service manifest, set the value of the spec.loadBalancerClass field to networking.gke.io/l4-regional-internal. For existing internal Services, your manifest already has the networking.gke.io/load-balancer-type: "Internal" annotation, and you can leave the annotation as is.
  • For new external LoadBalancer Services, set the spec.loadBalancerClass field to networking.gke.io/l4-regional-external in the Service manifest. For existing external Services, your manifest already has the cloud.google.com/l4-rbs: "enabled" annotation, and you can leave the annotation as is.

Limitations

  • On versions 1.34.1-gke.2190000 through to 1.36.2-gke.1498000, mixed-protocol load balancers support only IPv4 addresses.
  • Existing Services that have the gke.networking.io/l4-ilb-v1 or gke.networking.io/l4-netlb-v1 finalizers cannot be used for mixed-protocol load balancing. If you want to use mixed protocols on these Services, you must delete and re-create the Service according to the preceding requirements.
  • Updating ports on the Service might cause a brief traffic disruption for all traffic routed through the load balancer.

Pricing

Google Cloud bills you per forwarding rule, for any external IP addresses, and for data sent. The following table describes the number of forwarding rules and external IP addresses used for specified configurations. For more information, see VPC network pricing.

Type Transport Layer Internet Layer Number of Forwarding Rules Number of External IP Addresses
Internal Single or mixed (TCP, UDP, or both) IPv4 1 0
IPv6 1 0
IPv4 and IPv6 (DualStack) 2 0
External Single or mixed (TCP, UDP, or both) IPv4 1 1
IPv6 1 1
IPv4 and IPv6 (DualStack) 2 2

Deploy a workload

This section shows how to deploy a sample workload that listens on both TCP and UDP ports. Note that the Deployment configuration is the same whether you are using a mixed-protocol LoadBalancer Service or two separate single-protocol LoadBalancer Services.

  1. The following manifest is for a sample application that listens on port 8080 for both TCP and UDP traffic. Save the following manifest as mixed-app-deployment.yaml:

    apiVersion: apps/v1
kind: Deployment
metadata:
  name: mixed-app-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: mixed-app
  template:
    metadata:
      labels:
        app: mixed-app
    spec:
      containers:
      - image: gcr.io/kubernetes-e2e-test-images/agnhost:2.6
        name: agnhost
        args: ["serve-hostname", "--port=8080", "--tcp=true", "--udp=true", "--http=false"]
        ports:
          - name: tcp8080
            protocol: TCP
            containerPort: 8080
          - name: udp8080
            protocol: UDP
            containerPort: 8080

  2. Apply the manifest to your cluster:

    kubectl apply -f mixed-app-deployment.yaml

Create a mixed-protocol load balancer

Create a Service of type LoadBalancer that exposes the deployment to both TCP and UDP traffic. You can create either an external or an internal load balancer.

  1. To create an external load balancer, save the following manifest as mixed-protocol-lb.yaml:

    apiVersion: v1
kind: Service
metadata:
  name: mixed-protocol-external-lb
spec:
  loadBalancerClass: "networking.gke.io/l4-regional-external"
  type: LoadBalancer
  selector:
    app: mixed-app
  ports:
  - name: tcp-port
    protocol: TCP
    port: 8080
  - name: udp-port
    protocol: UDP
    port: 8080

    To create an internal load balancer, set the value of the spec.loadBalancerClass field to networking.gke.io/l4-regional-internal.

    The preceding Service has two ports, one for TCP and one for UDP, both on port 8080.

  2. Apply the manifest to your cluster:

    kubectl apply --server-side -f mixed-protocol-lb.yaml

Verify the mixed-protocol load balancer

After you create the Service, verify that GKE created the load balancer successfully.

  1. Inspect the Service:

    kubectl describe service SERVICE_NAME

    Replace SERVICE_NAME with the name of your Service (for example, mixed-protocol-lb).

    The output shows the IP address of the load balancer and the forwarding rules. Verify the following details in the output:

    • The status.loadBalancer.ingress.ip field is populated.
    • For clusters on versions 1.34.1-gke.2190000 through to 1.36.2-gke.1498000, verify that the following annotations for your external load balancer are present:
      • service.kubernetes.io/tcp-forwarding-rule
      • service.kubernetes.io/udp-forwarding-rule
    • For clusters that are created on versions after 1.36.2-gke.1498000, verify that the following annotations for your load balancer are present, depending on your configuration:
      • For IPv4: service.kubernetes.io/l3-forwarding-rule
      • For IPv6: service.kubernetes.io/l3-forwarding-rule-ipv6
      • For dual-stack: both annotations.
    • The Events section contains no error messages.

Update the mixed-protocol load balancer

You can update the ports on a mixed-protocol load balancer by editing the Service manifest. To edit the Service, run the following command:

kubectl edit service SERVICE_NAME

Replace SERVICE_NAME with the name of your Service.

Update ports

To update the ports on a mixed-protocol load balancer, modify the ports section of the Service manifest. You can add, remove, or modify ports.

The following example adds a UDP port for streaming and a TCP port for game-server metadata:

apiVersion: v1
kind: Service
metadata:
  name: mixed-protocol-lb
spec:
  loadBalancerClass: "networking.gke.io/l4-regional-external"  # for internal LB, use: "networking.gke.io/l4-regional-internal"
  type: LoadBalancer
  selector:
    app: mixed-app
  ports:
  - name: tcp-port
    protocol: TCP
    port: 8080
  - name: streaming
    protocol: UDP
    port: 10100
  - name: gameserver-metadata
    protocol: TCP
    port: 10400
  - name: https
    protocol: TCP
    port: 443

Update a single-protocol load balancer to mixed-protocol

To change a single-protocol load balancer to a mixed-protocol load balancer, edit the Service to include ports for both the TCP and UDP protocols.

The following example adds a UDP port for DNS to an existing TCP-only load balancer:

apiVersion: v1
kind: Service
metadata:
  name: already-existing-single-protocol-lb
spec:
  loadBalancerClass: "networking.gke.io/l4-regional-external" # for internal LB, use: "networking.gke.io/l4-regional-internal"
  type: LoadBalancer
  selector:
    app: mixed-app
  ports:
  - name: http
    protocol: TCP
    port: 80
  - name: https
    protocol: TCP
    port: 443
  - name: dns
    protocol: UDP
    port: 53

Update a mixed-protocol load balancer to single-protocol

To change a mixed-protocol load balancer to a single-protocol load balancer, remove all ports for one of the protocols.

The following example removes the UDP port for DNS, which converts the load balancer to TCP-only:

apiVersion: v1
kind: Service
metadata:
  name: already-existing-mixed-protocol-lb
spec:
  loadBalancerClass: "networking.gke.io/l4-regional-external" # for internal LB, use: "networking.gke.io/l4-regional-internal"
  type: LoadBalancer
  selector:
    app: mixed-app
  ports:
  - name: http
    protocol: TCP
    port: 80
  - name: https
    protocol: TCP
    port: 443

Delete the mixed-protocol LoadBalancer

To delete the mixed-protocol LoadBalancer Service, run the following command:

kubectl delete service SERVICE_NAME

Replace SERVICE_NAME with the name of your Service (for example, mixed-protocol-external-lb).

GKE automatically removes all load balancer resources created for the Service.

Troubleshooting

This section describes how to resolve common issues with mixed-protocol LoadBalancer Services.

Check for error events

The first step in troubleshooting is to check the events associated with your Service.

  1. Get the details of your Service:

    kubectl describe service SERVICE_NAME

    Replace SERVICE_NAME with the name of your Service.

  2. Review the Events section at the end of the output for any error messages.

Error: Mixed Protocol is not supported for LoadBalancer

If you created the Service with the cloud.google.com/l4-rbs: "enabled" annotation, you might see a warning event from the original service controller after you create the mixed-protocol load balancer: mixed-protocol is not supported for LoadBalancer.

You can safely ignore this message because the new controller, which supports mixed-protocols, correctly provisions the load balancer.

Port definition is missing after an update

Symptom:

When you update a Service that uses the same port for both TCP and UDP (for example, port 8080), one of the port definitions is missing from the updated Service.

Cause:

This is a known issue in Kubernetes. When you update a Service with multiple protocols on the same port, the client-side patch calculation can incorrectly merge the port list, which causes one of the port definitions to be removed. This issue affects clients that use client-side patching, such as kubectl apply and the Go client with merge patches.

Solution:

The workaround for this issue depends on your client.

  • For kubectl: Use the --server-side flag with kubectl apply:

    kubectl apply --server-side -f YOUR_SERVICE_MANIFEST.yaml

    Replace YOUR_SERVICE_MANIFEST with the name of your Service manifest.

  • For go-client: Don't use merge patches. Instead, use an update call to replace the Service. This requires an HTTP PUT request with the entire Service object specification.

What's next