Troubleshoot Transparent Data Encryption (TDE)

Select a documentation version: This page describes how to troubleshoot Transparent Data Encryption (TDE) issues in AlloyDB Omni.

Retrieve troubleshooting logs

To troubleshoot TDE issues, you can collect logs depending on your deployment type.

Kubernetes

Collect diagnostics from the following three sources:

  1. Cluster configuration and events:

    Use the describe database cluster command to check for high-level errors or status messages:

    kubectl describe dbcluster CLUSTER_NAME -n NAMESPACE

  2. Pod logs:

    Get the logs for the database pod:

    kubectl logs POD_NAME -n NAMESPACE

  3. Internal database logs:

    Inspect the PostgreSQL diagnostic log file inside the pod, filtering for TDE-related messages:

    kubectl exec -it POD_NAME -c database -n NAMESPACE -- cat /obs/diagnostic/postgresql.log | grep "TDE"

Docker

Collect the container logs and filter for TDE-related messages:

docker logs CONTAINER_NAME | grep "TDE"

Error scenarios

This section describes common TDE error scenarios and how to resolve them.

JWT errors

If you see HashiCorp Vault authentication or initialization failures in the collected logs, they are often related to JSON Web Token (JWT) issues.

Description

You might see errors like the following:

Error Description Reason
TDE: Failed to register Key Store with URL: VAULT_URL, error code: 3, error message: JWT cannot be empty JWT not found or empty The JWT token is missing or empty.
TDE: Failed to register Key Store with URL: VAULT_URL, error code: 3, error message: Invalid JWT format. Expected JWS compact serialization (three dot-separated segments: header, payload, and signature) with base64url encoded values. Malformed JWT format The JWT token is not in the correct JWS compact serialization format.
TDE: Failed to register Key Store with URL: VAULT_URL, error code: 3, error message: Vault request failed with HTTP code: 400, Vault errors: ["error validating token: invalid expiration time (exp) claim: token is expired"] Expired JWT The JWT token has expired.
TDE: Failed to register Key Store with URL: VAULT_URL, error code: 3, error message: Vault request failed with HTTP code: 400, Vault errors: ["error validating token: error verifying token signature: failed to verify id token signature"] Invalid JWT signature The Vault server couldn't verify the signature of the JWT token.
TDE: Failed to register Key Store with URL: VAULT_URL, error code: 9, error message: Failed to get Vault configuration parameter 'VAULT_JWT' Permission issue AlloyDB Omni lacks permission to read the JWT token file, or the VAULT_JWT environment variable is not set.

To resolve JWT issues:

  1. Retrieve the active JWT:

    Kubernetes

    Run the following command to retrieve the JWT token from the pod:

    kubectl exec -it POD_NAME -c database -n NAMESPACE -- cat /var/run/secrets/vault/token

    Docker

    Run the following command to print the VAULT_JWT environment variable from the container:

    docker exec -it CONTAINER_NAME sh -c 'echo $VAULT_JWT'

  2. Examine JWT structure: make sure that the token is valid and conforms to the JWT compact serialization format.

  3. Verify token state: make sure that the token is active and has not expired.

KMS URL errors

If the database fails to initialize or boot, it might be due to a misconfigured Key Management Service (KMS) Key Encryption Key (KEK) URL.

Description

You might see errors like the following:

Error Description Reason
TDE: Failed to register Key Store with URL: KMS_URL, error code: 12, error message: Unsupported key store type Unsupported key store type Occurs when the KEK URL protocol is not vault. For example, using http:// or https:// instead of vault://, or omitting the scheme entirely.
TDE: Failed to register Key Store with URL: KMS_URL, error code: 3, error message: Invalid Vault address format. Expected hostname or hostname:port Invalid Vault address format Occurs when the Vault address part of the URL is incorrectly formatted. For example, if it includes the protocol scheme again, such as vault://https://vault.example.com:8200/....
TDE: Failed to read KEK, error code: 3, error message: Invalid key path format. Must start with '/', not end with '/', and contain only valid characters Invalid key path format Occurs when the path part of the Vault URL is malformed. The path must start with a forward slash / and must not end with a trailing slash.
TDE: Failed to register Key Store with URL: KMS_URL, error code: 13, error message: curl_easy_perform() failed: Could not resolve hostname Hostname resolution failure Occurs when the Vault hostname in the KEK URL can't be resolved. This can happen if the URL is malformed (for example, vault://https:) or if there is a DNS configuration issue.
  • Verify KEK URL scheme: make sure that the URL starts with vault://. Don't use https:// as the prefix.
    • Correct: vault://vault.example.com:8200/v1/secret/data/key
    • Incorrect: https://vault.example.com:8200/v1/secret/data/key
  • Check address format: make sure that the host and port are correctly specified immediately after the vault:// prefix.
  • Verify path formatting: make sure that the path starts with / and doesn't have a trailing slash (for example, use /v1/secret/data/key, not /v1/secret/data/key/).
  • Verify host resolution: make sure that the Vault hostname specified in the URL is correct and can be resolved from within the database container or pod.

KEK not base64-encoded

If the KEK stored in HashiCorp Vault is not correctly base64-encoded, AlloyDB Omni fails to decode it upon retrieval. This prevents it from unwrapping the local Data Encryption Keys (DEKs) and causes a boot failure.

Description

You might see errors like the following in the database diagnostic logs (postgresql.log):

Error Description Reason
TDE: Failed to read KEK, error code: 13, error message: Error decoding base64 key string Error decoding base64 key string Occurs when the key manager attempts to decode the retrieved secret from Vault using Base64, but the string is not valid Base64.
  • Retrieve KEK value: retrieve and inspect the KEK value directly from HashiCorp Vault.
  • Update value in Vault: make sure that the KEK stored in Vault is base64-encoded. If it is stored as raw binary or plain text, update the value in Vault with its base64-encoded equivalent.
    1. Convert the raw key to Base64: 
      echo -n "RAW_KEY" | base64
    2. Update the secret in HashiCorp Vault to the generated base64-encoded string.

KEK version missing in Vault

The KEK version missing in Vault error occurs when AlloyDB Omni attempts to unwrap local DEKs using a specific historical version of the KEK, but that version is deleted, destroyed, or missing in HashiCorp Vault. This is most common during backup restore operations or standby replica initialization.

Description

You might see errors like the following in the database diagnostic logs (postgresql.log):

Error Description Reason
TDE: Failed to read KEK, error code: 5, error message: Vault request failed with HTTP code: 404, Response body: {... "destroyed":true, "version":4 ...} Key version destroyed The requested KEK version is permanently destroyed in Vault.
  • Verify KEK version in Vault: make sure that the requested KEK version is present and active (not deleted or destroyed) in HashiCorp Vault.
  • Restore missing KEK version: restore or undelete the requested KEK version in Vault if it has been deleted but not destroyed.
  • Handle destroyed keys: if the requested key version is permanently destroyed in Vault, the data or backup encrypted under that version is permanently unrecoverable.

Bad or missing TLS or SSL certificates

The bad or missing TLS or SSL certificate error occurs when AlloyDB Omni can't establish a secure TLS connection to HashiCorp Vault because certificates are missing, expired, untrusted, or misconfigured.

Description

You might see errors like the following in the database diagnostic logs (postgresql.log):

Error Description Reason
TDE: Failed to read KEK, error code: 13, error message: curl_easy_perform() failed: SSL peer certificate or SSH remote key was not OK
Or
TDE: Failed to read KEK, error code: 13, error message: curl_easy_perform() failed: Peer certificate cannot be authenticated with given CA certificates		 Vault server certificate verification failed The Vault server's certificate can't be verified using the provided CA certificate.
TDE: Failed to register Key Store with URL: VAULT_URL, error code: 13, error message: curl_easy_perform() failed: Problem with the local SSL certificate Local SSL certificate problem Occurs if a client certificate is required and has issues.

To resolve TLS verification issues, make sure that the correct certificates are supplied, mounted, and readable by the postgres user.

Kubernetes

Kubernetes expects certificates to be supplied as a Secret mounted into the database pod at /tde-tls. To locate and inspect the certificate secret:

  1. Extract the referenced TdeConfig name directly from your database cluster:

    kubectl get dbcluster CLUSTER_NAME -n NAMESPACE -o jsonpath='{.spec.primarySpec.features.transparentDataEncryption.tdeConfigRef.name}'

  2. Extract the secret name directly from the TdeConfig resource:

    kubectl get tdeconfig TDE_CONFIG_NAME -n NAMESPACE -o jsonpath='{.spec.tls.certSecret.name}'

  3. Verify that the secret exists and contains the required keys:

    kubectl describe secret CERT_SECRET_NAME -n NAMESPACE

  4. Confirm that the secret is correctly mounted and that the files are readable by the postgres user:

    kubectl exec -it POD_NAME -c database -n NAMESPACE -- ls -la /tde-tls

Docker

Docker expects certificates to be supplied using a mounted host volume.

  1. Confirm that VAULT_CERT_PATH is set and points to the correct directory inside the container:

    docker inspect CONTAINER_NAME | grep VAULT_CERT_PATH

  2. Check that the certificate files exist and are readable by the database user (uid 999):

    docker exec -it CONTAINER_NAME ls -la PATH_TO_CERTS_DIRECTORY

    Make sure that ca.crt, tls.crt and tls.key are present and have read permissions for the database user.