Reconciliation strategy

With declarative configuration, you define the desired state of the system. The system then works constantly to remain as close as possible to this state. See Declarative management of Kubernetes objects using configuration files for more information.

With Config Connector, you can create and update resources in any order, regardless of dependency relationships. GKE moves your declared configuration towards eventual consistency with the desired state.

For example, if you create a PubSubSubscription before the corresponding PubSubTopic, Config Connector waits until the topic is created before creating the associated subscription.

The duration your Config Connector installation remains inconsistent depends on the number and types of resources it manages. Changes to a GKE cluster are typically executed in seconds. However, the time to create Google Cloud resources can vary based on the type of resource. For example, a single PubSubTopic takes seconds to create. Google Cloud resources do not reach consistency until they are created. For example, when creating an SQLInstance and an SQLDatabase, the system is inconsistent for a period of minutes while the database is created.

GKE and Config Connector reconcile each resource with every update or after a jitter period with an average based on the preconfigured default interval. You can find the default interval in "Config Connector Default Average Reconcile Interval In Seconds" from each resource's reference page. For links to each resource, see the Resource overview. When there is an error in reconciling, Config Connector retries with exponential backoff where maximum backoff is 2 minutes. For resources managed by direct controllers, you can configure this maximum by using the cnrm.cloud.google.com/backoff-max-delay-in-seconds annotation. You can view any errors in the Events of a given resource.

Configuring the reconciliation interval

Starting from Config Connector 1.102, you can configure the average reconcile interval for resources managed by Config Connector with the cnrm.cloud.google.com/reconcile-interval-in-seconds annotation. The value of the annotation overwrites the default value of "Config Connector Default Average Reconcile Interval In Seconds" from the resource reference page. The value of the annotation should be a non-negative integer representing time in seconds. If the value is set to 0, Config Connector stops initiating reconciliations for the resource once it reaches the UpToDate status. This annotation controls periodic reconciliation after the resource reaches the UpToDate status. The annotation does not affect retry intervals when reconciliation fails. For more information, see Configuring exponential backoff for reconciliation errors.

For example, if you want Config Connector to reconcile a resource less frequently to avoid hitting underlying Google Cloud API quota issues, you can set the average reconciliation interval value to 1 hour.

cnrm.cloud.google.com/reconcile-interval-in-seconds: "3600"

If you want Config Connector to reconcile a resource more frequently to correct drifts sooner, you can set the annotation with a small value.

You can annotate all resources of a particular type that shares the same Group Version Kind (GVK) with the following script:

#!/bin/bash

KIND=RESOURCE_KIND
NAMESPACE=RESOURCE_NAMESPACE
ANNOTATION_KEY="cnrm.cloud.google.com/reconcile-interval-in-seconds"
ANNOTATION_VALUE=RECONCILE_INTERVAL

kubectl annotate --overwrite --all ${KIND} ${ANNOTATION_KEY}=${ANNOTATION_VALUE} -n ${NAMESPACE}
echo "Annotation added to all ${KIND} RESOURCE"

Replace the following:

  • RESOURCE_KIND: the resource kind that you want to annotate.
  • RESOURCE_NAMESPACE: the namespace that contains the resources that you want to annotate.
  • RECONCILE_INTERVAL: the reconcile interval in seconds.

You can set the reconciliation interval to 0 to disable drift correction for a resource, but it does not disable resource actuation. If you make changes to the resource Spec, the resource will be reconciled again.

Setting the reconciliation interval to 0 is irreversible. This means changing the value back to a non-zero number does not make Config Connector reconcile the resource again.

If you want to revert the 0 reconciliation interval, you have the following options:

  • Modify the resource spec with the reconciliation interval value to enable new reconciliations.
  • Abandon the resource by setting the annotation cnrm.cloud.google.com/deletion-policy: "abandon" and recreate the resource with a reconciliation interval value other than 0.

Configuring exponential backoff for reconciliation errors

When a resource fails to reconcile, Config Connector retries reconciliation by using exponential backoff. The delay starts at 250 milliseconds and doubles after each failed attempt (for example, 500 milliseconds, then 1 second, then 2 seconds), up to a default maximum of 2 minutes.

The cnrm.cloud.google.com/reconcile-interval-in-seconds annotation controls periodic drift correction after a resource reaches the UpToDate status. In Config Connector version 1.160.0 and later, you can use the cnrm.cloud.google.com/backoff-max-delay-in-seconds annotation to control retry behavior when reconciliation fails. Set the annotation to one of the following values:

  • A positive integer ("1" or greater): sets the maximum delay, in seconds, between retries. For example, "600" caps the retry interval at 10 minutes, and "3600" caps it at 1 hour.
  • "0": stops automatic retries after a failed reconciliation. The resource stays in a failed state until a change triggers reconciliation, for example, an update to spec, metadata.labels, metadata.annotations, or a referenced resource or Secret.

For example, to cap retry intervals at 30 minutes and reduce Google Cloud API quota consumption on failing resources, add the following annotation:

metadata:
  annotations:
    cnrm.cloud.google.com/backoff-max-delay-in-seconds: "1800"

To stop automatic retries on reconciliation errors until you update the resource, add the following annotation:

metadata:
  annotations:
    cnrm.cloud.google.com/backoff-max-delay-in-seconds: "0"

Mutable but unreadable fields are actuated on change only

Some APIs expose fields that are not readable, but are mutable (for example, the password for a SQL user). Due to the inability to see if these fields have been modified, mutable but unreadable fields are updated only when the custom resource is modified.

Resources are not recreated when modifying immutable fields

Some fields in a resource are immutable, and can't be reconciled without deleting, then re-creating the target resource.

In these situations, Config Connector emits an "UpdatedFailed" Kubernetes event for the resource rather than perform this re-creation. You must then delete and re-create the resource.

Example event:

Warning  UpdateFailed  37m (x643 over 15d)    computeinstance-controller  Update call failed: the desired mutation for the following field(s) is invalid: [bootDisk.0.InitializeParams.0.Image networkInterface.0.NetworkIp]