Semantic caching with a private (Private Service Connect) endpoint

This page applies to Apigee, but not to Apigee hybrid.

View Apigee Edge documentation.

This page describes how to configure and use the Apigee semantic caching policies to enable intelligent response reuse based on semantic similarity. In this example, the policies run their similarity search against a Vector Search index that is deployed on a private (Private Service Connect) endpoint. Using these policies in your Apigee API proxy minimizes redundant backend API calls, reduces latency, and lowers operational costs.

Before you begin

Before you begin, complete the following tasks:

Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

Go to project selector

Verify that billing is enabled for your Google Cloud project.

Enable the Compute Engine, AI Platform, and Cloud Storage APIs.

Roles required to enable APIs

To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

Enable the APIs

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

Go to project selector

Verify that billing is enabled for your Google Cloud project.

Enable the Compute Engine, AI Platform, and Cloud Storage APIs.

Roles required to enable APIs

Enable the APIs

Enable and configure the Vertex AI Text embeddings API in your Google Cloud project.
Create (or have access to) a Vector Search index deployed on a private (Private Service Connect) endpoint. This tutorial does not duplicate the Vector Search setup steps; see Vector Search index prerequisites for the SemanticCacheLookup-specific requirements and links to the Vector Search documentation.
Confirm that you have an Intermediate or Comprehensive environment available in your Apigee instance. Semantic caching policies can only be deployed in Intermediate or Comprehensive environments.
Confirm that you have a environment group with a runtime hostname that you can use to send requests to your API proxy.

Required roles

To get the permissions that you need to create and use the semantic caching policies, ask your administrator to grant you the AI Platform User (roles/aiplatform.user) IAM role on the service account you use to deploy Apigee proxies. For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

Note: The identity deploying the proxy needs the roles/iam.serviceAccountUser (Service Account User) role on SERVICE_ACCOUNT (or at the project level). See Step 4: Import and deploy the API proxy.

Set environment variables

In the Google Cloud project that contains your Apigee instance, use the following command to set environment variables:

export PROJECT_ID=PROJECT_ID
export REGION=REGION
export RUNTIME_HOSTNAME=RUNTIME_HOSTNAME

Where:

PROJECT_ID is the ID of the project with your Apigee instance.
REGION is the Google Cloud region of your Apigee instance.
RUNTIME_HOSTNAME is the hostname of your Apigee runtime.

To confirm that the environment variables are set correctly, run the following command and review the output:

echo $PROJECT_ID $REGION $RUNTIME_HOSTNAME

Set the project

Set the Google Cloud project in your development environment:

    gcloud auth login
    gcloud config set project $PROJECT_ID

Vector Search index prerequisites

This tutorial assumes that you already have (or will create) a Vector Search index deployed on a private (Private Service Connect) endpoint. Creating, formatting, and deploying a Vector Search index is documented in the Vector Search guides, so this tutorial does not duplicate those steps. Follow the Vector Search documentation to:

When you create the index, it must meet the following SemanticCacheLookup-specific requirements:

The index must use STREAM_UPDATE ("indexUpdateMethod": "STREAM_UPDATE") so that the SemanticCachePopulate policy's upsertDatapoints calls become queryable in near real time.
The index dimensions must match the output dimensionality of the embeddings model you use in the SemanticCacheLookup policy. This tutorial uses gemini-embedding-001, which produces 3072-dimensional embeddings by default. If you truncate the output to a lower dimensionality (for example, 768 or 1536), set dimensions to the same value.
Create the index with the distance measure (distanceMeasureType) that matches your policy's <DistanceMeasureType>. The <SimilaritySearch><VertexAI><DistanceMeasureType> element in the SemanticCacheLookup policy is optional and defaults to DOT_PRODUCT_DISTANCE; COSINE_DISTANCE is also supported. The index distance measure and the policy <DistanceMeasureType> must be the same.

The following minimal example creates a compatible index. For the full request body and all available options, see Create and manage an index:

ACCESS_TOKEN=$(gcloud auth print-access-token) && curl -X POST \
  "https://$REGION-aiplatform.googleapis.com/v1/projects/$PROJECT_ID/locations/$REGION/indexes" \
  -H "Authorization: Bearer $ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "displayName": "semantic-cache-index",
    "metadata": {
      "config": {
        "dimensions": 3072,
        "distanceMeasureType": "DOT_PRODUCT_DISTANCE"
      }
    },
    "indexUpdateMethod": "STREAM_UPDATE"
  }'

Note the numeric INDEX_ID returned in the response; you use it in the SemanticCachePopulate policy. After you create the index, create a Private Service Connect index endpoint and deploy the index to it.

When you create the Private Service Connect index endpoint, it must meet the following SemanticCacheLookup-specific requirements:

The projectAllowlist must include the Apigee project that initiates the connection:
- Apigee: Use the Apigee tenant project. Get the tenant project ID from the Organizations API (apigeeProjectId field).
The projectAllowlist cannot be modified after the index endpoint is created. If you allowlist the wrong project, you must delete and re-create the index endpoint.

Note the numeric INDEX_ENDPOINT_ID of the index endpoint.

Configure the service account for the Apigee proxy

The Apigee proxy uses a service account for its Vertex AI REST calls: the Embeddings API in the SemanticCacheLookup policy, upsertDatapoints in the SemanticCachePopulate policy, and the model target. Grant that service account the AI Platform User (roles/aiplatform.user) role:

gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:SERVICE_ACCOUNT" \
  --role="roles/aiplatform.user"

Where SERVICE_ACCOUNT is the email address of the service account that the proxy uses. You reference this service account when you deploy the proxy in Step 4: Import and deploy the API proxy.

Note: The identity that deploys the proxy needs the roles/iam.serviceAccountUser (Service Account User) role on SERVICE_ACCOUNT (or at the project level) to deploy a proxy with that service account attached. See Step 4: Import and deploy the API proxy.

Overview

The semantic caching policies help Apigee users with LLM models to intelligently serve identical or semantically similar prompts efficiently, minimizing backend API calls and reducing resource consumption.

The SemanticCacheLookup and SemanticCachePopulate policies attach to the request and response flows, respectively, of an Apigee API proxy. When the proxy receives a request, the SemanticCacheLookup policy extracts the user prompt from the request and converts the prompt into a numerical representation using the Text embeddings API. A semantic similarity search performs using Vector Search to find similar prompts. If a similar prompt data point is found, a cache lookup performs. If cached data is found, the cached response returns to the client.

If the similarity search does not return a similar previous prompt, the LLM model generates content in response to the user prompt and populates the Apigee cache with the response. A feedback loop creates to update the Vector Search index entries in preparation for future requests.

In this scenario, the Vector Search index is deployed on a private (Private Service Connect) endpoint, over gRPC. See more details about Vector Search Private Service Connect support in Query Private Services Access or Private Service Connect indexes.

The following sections describe the steps to create and configure the semantic caching policies:

Verify your resources and get the values Apigee needs.
Connect to the service attachment.
Build the API proxy bundle.
Import and deploy the API proxy.
Test the semantic caching policies.

Step 1: Verify your resources and get the values Apigee needs

Before you configure Apigee, confirm that your Vector Search index endpoint is Private Service Connect-enabled and that your index is deployed. Then read the two values that the Apigee proxy consumes: the service attachment and the DEPLOYED_INDEX_ID.

Confirm that the index is deployed and that the endpoint exposes a Private Service Connect service attachment:

gcloud ai index-endpoints describe INDEX_ENDPOINT_ID \
  --project=$PROJECT_ID --region=$REGION \
  --format="value(deployedIndexes.privateEndpoints.serviceAttachment)"

The command returns a service attachment resource name of the form projects/TENANT_PROJECT/regions/REGION/serviceAttachments/SERVICE_ATTACHMENT_NAME. This guide refers to that value as SERVICE_ATTACHMENT. If the command returns an empty value, the index is not yet deployed on a Private Service Connect endpoint. Return to Vector Search index prerequisites and finish deploying the index before you continue.

Read the DEPLOYED_INDEX_ID of the index deployed on the endpoint:

gcloud ai index-endpoints describe INDEX_ENDPOINT_ID \
  --project=$PROJECT_ID --region=$REGION \
  --format="value(deployedIndexes.id)"

This guide refers to that value as DEPLOYED_INDEX_ID. You use it in the SemanticCacheLookup policy in Step 3: Build the API proxy bundle.

For more information on deploying and querying private index endpoints, see Deploy an index to a Private Service Connect endpoint and Query Private Services Access or Private Service Connect indexes.

Step 2: Connect to the service attachment

This step gives you the private host that the proxy's <GrpcEndpoint> calls. On Apigee, create an Apigee endpoint attachment. The endpoint attachment is Apigee's Private Service Connect consumer side: it connects to the Vector Search service attachment and gives you a private host that the proxy calls.

curl -X POST \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "Content-Type: application/json" \
  -d '{
        "location": "'"$REGION"'",
        "serviceAttachment": "SERVICE_ATTACHMENT"
      }' \
  "https://apigee.googleapis.com/v1/organizations/$PROJECT_ID/endpointAttachments?endpointAttachmentId=ENDPOINT_ATTACHMENT"

Poll until the attachment's state is ACTIVE and its connectionState is ACCEPTED, then note the host:

curl -s -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  "https://apigee.googleapis.com/v1/organizations/$PROJECT_ID/endpointAttachments/ENDPOINT_ATTACHMENT"

The response contains the host in the host field. This guide refers to that value as TARGET_HOST.

To connect to the Vector Search service attachment from your proxy, you can use either:

The IP address: Use the IP address returned in the host field directly as TARGET_HOST (for example, 7.0.3.4).
A private DNS record: If you configured a private Cloud DNS zone in your Google Cloud project with DNS peering to Apigee, you can create an A record in your private zone pointing to the endpoint attachment IP address and use that domain name (such as vectorsearch.example.com) as TARGET_HOST. For more information, see Use a DNS record and Connecting with private DNS peering zones.

Note: If connectionState stays PENDING, the Vector Search service attachment is not accepting the connection—usually because the tenant project is not allowlisted. Because the index endpoint's projectAllowlist cannot be modified after creation, return to Vector Search index prerequisites and re-create the index endpoint with the correct privateServiceConnectConfig.projectAllowlist value.

Step 3: Build the API proxy bundle

Create the proxy bundle

Create the following directory layout:

apiproxy/
├── PROXY_NAME.xml
├── proxies/default.xml
├── targets/default.xml
└── policies/
    ├── SCL-1.xml
    └── SCP-1.xml

policies/SCL-1.xml—the SemanticCacheLookup policy. The <SimilaritySearch> block uses <PrivateServiceConnect><GrpcEndpoint> (no <URL>).

Note: <GrpcEndpoint> rules:

The format is grpc://TARGET_HOST:PORT; the scheme must be grpc://. grpcs:// (TLS) is not supported in this version.
The port is 10000 for Vector Search. Private Service Connect data-plane endpoints serve gRPC on port 10000, so the endpoint is always grpc://TARGET_HOST:10000.
TARGET_HOST can be either the endpoint attachment IP address (from Step 2) or a custom DNS record created in your private DNS zone.
The gRPC hop is plaintext and unauthenticated (secured by network isolation).

<SemanticCacheLookup async="false" continueOnError="false" enabled="true" name="SCL-1">
  <DisplayName>SCL-1</DisplayName>
  <IgnoreUnresolvedVariables>false</IgnoreUnresolvedVariables>
  <UserPromptSource>{jsonPath('$.contents[-1].parts[-1].text',request.content,true)}</UserPromptSource>
  <Embeddings>
    <VertexAI>
      <URL>https://REGION-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/REGION/publishers/google/models/gemini-embedding-001:predict</URL>
    </VertexAI>
  </Embeddings>
  <SimilaritySearch>
    <VertexAI>
      <PrivateServiceConnect>
        <GrpcEndpoint>grpc://TARGET_HOST:10000</GrpcEndpoint>
      </PrivateServiceConnect>
      <DeployedIndexID>DEPLOYED_INDEX_ID</DeployedIndexID>
      <Threshold>0.95</Threshold>
    </VertexAI>
  </SimilaritySearch>
</SemanticCacheLookup>

policies/SCP-1.xml—the SemanticCachePopulate policy. Populate is REST-only and must use <URL> (it rejects <PrivateServiceConnect> at deployment time):

<SemanticCachePopulate async="false" continueOnError="true" enabled="true" name="SCP-1">
  <DisplayName>SCP-1</DisplayName>
  <IgnoreUnresolvedVariables>true</IgnoreUnresolvedVariables>
  <SimilaritySearch>
    <VertexAI>
      <URL>https://REGION-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/REGION/indexes/INDEX_ID:upsertDatapoints</URL>
    </VertexAI>
  </SimilaritySearch>
  <TTLInSeconds>3600</TTLInSeconds>
</SemanticCachePopulate>

targets/default.xml—the model target. The target calls a Google API, so it needs a token; <GoogleAccessToken> uses the deployment's service account:

<TargetEndpoint name="default">
  <PreFlow name="PreFlow"><Request/><Response/></PreFlow>
  <PostFlow name="PostFlow"><Request/><Response/></PostFlow>
  <HTTPTargetConnection>
    <Authentication>
      <GoogleAccessToken>
        <Scopes>
          <Scope>https://www.googleapis.com/auth/cloud-platform</Scope>
        </Scopes>
      </GoogleAccessToken>
    </Authentication>
    <URL>https://REGION-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/REGION/publishers/google/models/gemini-2.5-flash:generateContent</URL>
  </HTTPTargetConnection>
</TargetEndpoint>

proxies/default.xml—run the SemanticCacheLookup policy on the request and the SemanticCachePopulate policy on the response:

<ProxyEndpoint name="default">
  <PreFlow name="PreFlow">
    <Request><Step><Name>SCL-1</Name></Step></Request>
    <Response><Step><Name>SCP-1</Name></Step></Response>
  </PreFlow>
  <PostFlow name="PostFlow"><Request/><Response/></PostFlow>
  <HTTPProxyConnection>
    <BasePath>/PROXY_NAME</BasePath>
  </HTTPProxyConnection>
  <RouteRule name="default">
    <TargetEndpoint>default</TargetEndpoint>
  </RouteRule>
</ProxyEndpoint>

PROXY_NAME.xml—the bundle descriptor:

<APIProxy name="PROXY_NAME">
  <BasePaths>/PROXY_NAME</BasePaths>
  <Policies><Policy>SCL-1</Policy><Policy>SCP-1</Policy></Policies>
  <ProxyEndpoints><ProxyEndpoint>default</ProxyEndpoint></ProxyEndpoints>
  <TargetEndpoints><TargetEndpoint>default</TargetEndpoint></TargetEndpoints>
</APIProxy>

Step 4: Import and deploy the API proxy

Zip the bundle, import it to create a new revision, and deploy the revision with your service account:

TOKEN=$(gcloud auth print-access-token)
(cd BUNDLE_DIR && zip -r ../PROXY_NAME.zip apiproxy)
curl -X POST -H "Authorization: Bearer $TOKEN" \
  -F "file=@PROXY_NAME.zip" \
  "https://apigee.googleapis.com/v1/organizations/$PROJECT_ID/apis?action=import&name=PROXY_NAME"
curl -X POST -H "Authorization: Bearer $TOKEN" \
  "https://apigee.googleapis.com/v1/organizations/$PROJECT_ID/environments/ENV/apis/PROXY_NAME/revisions/REVISION/deployments?override=true&serviceAccount=SERVICE_ACCOUNT"

Where:

BUNDLE_DIR is the directory that contains the apiproxy/ folder. The archive must contain the apiproxy/ folder at its root.
ENV is the Apigee environment where you deploy the proxy. The environment must be an Intermediate or Comprehensive environment.
REVISION is the revision number returned by the import call.
SERVICE_ACCOUNT is the email address of the service account you use to deploy the proxy.

Note: The identity deploying the proxy needs the roles/iam.serviceAccountUser (Service Account User) role on SERVICE_ACCOUNT (or at the project level) to deploy a proxy with that service account attached. Project Owners and Editors have this permission by default. If your deploying identity is not an Owner or Editor, grant the role on the service account using:

gcloud iam service-accounts add-iam-policy-binding SERVICE_ACCOUNT \
  --member="user:DEPLOYER_EMAIL" \
  --role="roles/iam.serviceAccountUser"

Wait until the deployment reports READY:

curl -s -H "Authorization: Bearer $TOKEN" \
  "https://apigee.googleapis.com/v1/organizations/$PROJECT_ID/environments/ENV/apis/PROXY_NAME/revisions/REVISION/deployments" | jq .state

Step 5: Test the semantic caching policies

Send a new prompt. This is a cache miss: the model is called and the answer is cached.

curl -X POST "https://$RUNTIME_HOSTNAME/PROXY_NAME" \
  -H "Content-Type: application/json" \
  -d '{"contents":[{"role":"user","parts":[{"text":"Explain in one sentence why the sky appears blue."}]}]}'

Send the same prompt again. This is a cache hit: the response is served from the cache and the model is not called.

curl -i -X POST "https://$RUNTIME_HOSTNAME/PROXY_NAME" \
  -H "Content-Type: application/json" \
  -d '{"contents":[{"role":"user","parts":[{"text":"Explain in one sentence why the sky appears blue."}]}]}'

On the hit, the response includes the Cached-content: true header, the same answer, and a noticeably lower latency.

You can also verify caching with a debug session. On a hit, the SemanticCacheLookup policy sets the following flow variables:

Variable	Value on a hit
`SemanticCacheLookup.SCL-1.dense_embeddings`	The prompt's embedding vector.
`SemanticCacheLookup.SCL-1.is_nearest_neighbor_hit`	`true`
`SemanticCacheLookup.SCL-1.cache_hit`	`true`
`SemanticCacheLookup.SCL-1.cached_llm_response`	The cached answer.

On a hit, the model target is not invoked—the flow short-circuits and returns the cached response.

Troubleshooting

For the full error reference, see the SemanticCacheLookup policy.

What's next

Learn how to query Private Services Access or Private Service Connect indexes in Vector Search.
Learn how to configure semantic caching against a public endpoint in Get started with semantic caching policies.
Learn how to Get started with Model Armor policies.

Semantic caching with a private (Private Service Connect) endpoint Stay organized with collections Save and categorize content based on your preferences.

Before you begin

Required roles

Set environment variables

Set the project

Vector Search index prerequisites

Configure the service account for the Apigee proxy

Overview

Step 1: Verify your resources and get the values Apigee needs

Step 2: Connect to the service attachment

Step 3: Build the API proxy bundle

Create the proxy bundle

Step 4: Import and deploy the API proxy

Step 5: Test the semantic caching policies

Troubleshooting

What's next

Semantic caching with a private (Private Service Connect) endpoint