在 RAG 引擎中使用 Vertex AI 特徵儲存庫

本頁面說明如何設定 Vertex AI 特徵儲存庫,做為 RAG Engine 的向量資料庫。

您也可以使用 RAG Engine 搭配 Vertex AI 特徵儲存庫 筆記本,按照當中的步驟操作。

整合 Vertex AI 特徵儲存庫 做為額外的向量資料庫後,RAG Engine 就能使用 Vertex AI 特徵儲存庫 處理大量資料,同時維持低延遲,有助於提升 RAG 應用程式的效能和擴充性。

設定 Vertex AI 特徵儲存庫

Vertex AI 特徵儲存庫是代管的雲端原生服務,也是 Gemini Enterprise Agent Platform 的重要元件。您可以在 BigQuery 資料表或檢視表中管理特徵資料,簡化機器學習 (ML) 特徵管理和線上提供作業。這可實現低延遲的線上特徵供應。

對於使用最佳化線上提供功能建立的 FeatureOnlineStore 執行個體,您可以利用向量相似度搜尋功能,擷取語意相似或相關的實體清單,也就是近似最鄰近項目。

以下各節說明如何為 RAG 應用程式設定 Vertex AI 特徵儲存庫 執行個體。

建立 BigQuery 資料表結構定義

使用 Google Cloud 控制台建立 BigQuery 資料表結構定義。其中必須包含下列欄位,才能做為資料來源。

欄位名稱 資料類型 狀態
corpus_id String 必填
file_id String 必填
chunk_id String 必填
chunk_data_type String 是否可為空值
chunk_data String 是否可為空值
file_original_uri String 是否可為空值
embeddings Float 重複

這個程式碼範例示範如何定義 BigQuery 資料表結構定義。

SQL

  CREATE TABLE `PROJECT_ID.input_us_central1.rag_source_new` (
    `corpus_id` STRING NOT NULL,
    `file_id` STRING NOT NULL,
    `chunk_id` STRING NOT NULL,
    `chunk_data_type` STRING,
    `chunk_data` STRING,
    `embeddings` ARRAY<FLOAT64>,
    `file_original_uri` STRING
  );

佈建 FeatureOnlineStore 執行個體

如要啟用特徵的線上提供功能,請使用 Vertex AI 特徵儲存庫 CreateFeatureOnlineStore API 設定 FeatureOnlineStore 執行個體。如果是首次佈建 FeatureOnlineStore,作業可能需要約五分鐘才能完成。

REST

如要建立網路商店執行個體,請使用 featureOnlineStores.create 方法傳送 POST 要求。

使用任何要求資料之前,請先修改下列項目的值:

  • LOCATION_ID:要建立 FeatureOnlineStore 執行個體的區域,例如 us-central1。
  • PROJECT_ID:專案 ID。
  • FEATUREONLINESTORE_NAME:新 FeatureOnlineStore 執行個體的名稱。

HTTP 方法和網址:

POST https://LOCATION_ID-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION_ID/featureOnlineStores?feature_online_store_id=FEATUREONLINESTORE_NAME

JSON 要求內文:

{
  "optimized": {}
}

如要傳送要求,請選擇以下其中一個選項:

curl

將要求主體儲存在名為 request.json 的檔案中,然後執行下列指令:

curl -X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json; charset=utf-8" \
-d @request.json \
"https://LOCATION_ID-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION_ID/featureOnlineStores?feature_online_store_id=FEATUREONLINESTORE_NAME"

PowerShell

將要求主體儲存在名為 request.json 的檔案中,然後執行下列指令:

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
-Method POST `
-Headers $headers `
-ContentType: "application/json; charset=utf-8" `
-InFile request.json `
-Uri "https://LOCATION_ID-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION_ID/featureOnlineStores?feature_online_store_id=FEATUREONLINESTORE_NAME" | Select-Object -Expand Content

您應該會收到如下的 JSON 回覆:

{
  "name": "projects/PROJECT_NUMBER/locations/LOCATION_ID/featureOnlineStores/FEATUREONLINESTORE_NAME/operations/OPERATION_ID",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.aiplatform.v1.CreateFeatureOnlineStoreOperationMetadata",
    "genericMetadata": {
      "createTime": "2023-09-18T17:49:23.847496Z",
      "updateTime": "2023-09-18T17:49:23.847496Z"
    }
  }
}

Python

如要瞭解如何安裝或更新 Vertex AI SDK for Python,請參閱「安裝 Vertex AI SDK for Python」。詳情請參閱 Python API 參考文件。


from google.cloud import aiplatform
from vertexai.resources.preview import feature_store


def create_optimized_public_feature_online_store_sample(
    project: str,
    location: str,
    feature_online_store_id: str,
):
    aiplatform.init(project=project, location=location)
    fos = feature_store.FeatureOnlineStore.create_optimized_store(
        feature_online_store_id
    )
    return fos

  • project:專案 ID。
  • location:要建立 FeatureOnlineStore 執行個體的區域,例如 us-central1。
  • feature_online_store_id:新 FeatureOnlineStore 執行個體的名稱。

建立 FeatureView 資源

如要將儲存特徵資料來源的 BigQuery 資料表連結至 FeatureOnlineStore 執行個體,請呼叫 CreateFeatureView API 來建立 FeatureView 資源。建立 FeatureView 資源時,請選擇預設距離指標 DOT_PRODUCT_DISTANCE,其定義為點積的負值 (DOT_PRODUCT_DISTANCE 越小,相似度越高)。

這個程式碼範例示範如何建立 FeatureView 資源。

REST

  # TODO(developer): Update and uncomment the following lines:
  # Set feature_view_id
  # Example: "feature_view_test"
  # FEATURE_VIEW_ID = "your-feature-view-id"
  #
  # The big_query_uri generated in the above BigQuery table schema creation step
  # The format should be "bq://" + BigQuery table ID
  # Example: "bq://tester.ragtest1.rag_testdata"
  # BIG_QUERY_URI=YOUR_BIG_QUERY_URI

  # Call CreateFeatureView API to create a FeatureView
  curl -X POST -H "Authorization: Bearer $(gcloud auth print-access-token)" -H "Content-Type: application/json" \
  https://us-central1-aiplatform.googleapis.com/v1/projects/${PROJECT_ID}/locations/us-central1/featureOnlineStores/${FEATURE_ONLINE_STORE_ID}/featureViews?feature_view_id=${FEATURE_VIEW_ID} \
    -d '{
          "vertex_rag_source": {
            "uri": '\""${BIG_QUERY_URI}"\"'
          }
      }'

  # Call ListFeatureViews API to verify the FeatureView is created successfully
  curl -X GET -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" -H "Content-Type: application/json" https://us-central1-aiplatform.googleapis.com/v1/projects/${PROJECT_ID}/locations/us-central1/featureOnlineStores/${FEATURE_ONLINE_STORE_ID}/featureViews

Python

如要瞭解如何安裝或更新 Vertex AI SDK for Python,請參閱「安裝 Vertex AI SDK for Python」。詳情請參閱 Python API 參考文件。


from google.cloud import aiplatform
from vertexai.resources.preview import feature_store


def create_feature_view_from_rag_source(
    project: str,
    location: str,
    existing_feature_online_store_id: str,
    feature_view_id: str,
    bq_table_uri: str,
):
    aiplatform.init(project=project, location=location)
    fos = feature_store.FeatureOnlineStore(existing_feature_online_store_id)
    fv = fos.create_feature_view(
        name=feature_view_id,
        source=feature_store.utils.FeatureViewVertexRagSource(uri=bq_table_uri),
    )
    return fv

上傳資料和線上提供

RAG API 會處理資料上傳和線上提供作業。如要進一步瞭解 Gemini Enterprise Agent Platform API 的 RAG Engine,請參閱下列資源:

在 RAG Engine 中使用 Vertex AI 特徵儲存庫

設定 Vertex AI 特徵儲存庫 執行個體後,請按照下列各節說明,將其設為要與 RAG 應用程式搭配使用的向量資料庫。

使用 Vertex AI 特徵儲存庫執行個體做為向量資料庫,建立 RAG 語料庫

如要建立 RAG 語料庫,請務必使用 FEATURE_VIEW_RESOURCE_NAME。RAG 語料庫建立後,會自動與 Vertex AI 特徵儲存庫執行個體建立關聯。RAG API 會使用產生的 rag_corpus_id,將資料上傳至 Vertex AI 特徵儲存庫 執行個體,並從 rag_corpus_id 擷取相關內容。

這個程式碼範例示範如何使用 Vertex AI 特徵儲存庫執行個體做為向量資料庫,建立 RAG 語料庫。

REST

# TODO(developer): Update and uncomment the following lines:
# CORPUS_DISPLAY_NAME = "your-corpus-display-name"
#
# Full feature view resource name
# Format: projects/${PROJECT_ID}/locations/us-central1/featureOnlineStores/${FEATURE_ONLINE_STORE_ID}/featureViews/${FEATURE_VIEW_ID}
# FEATURE_VIEW_RESOURCE_NAME = "your-feature-view-resource-name"

# Call CreateRagCorpus API to create a new RAG corpus
curl -X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
  https://us-central1-aiplatform.googleapis.com/v1beta1/projects/{PROJECT_ID}/locations/us-central1/ragCorpora -d '{
    "display_name" : '\""${CORPUS_DISPLAY_NAME}"\"',
    "rag_vector_db_config" : {
      "vertex_feature_store": {
        "feature_view_resource_name":'\""${FEATURE_VIEW_RESOURCE_NAME}"\"'
      }
    }
  }'

# Call ListRagCorpora API to verify the RAG corpus is created successfully
curl -sS -X GET \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
"https://us-central1-aiplatform.googleapis.com/v1beta1/projects/${PROJECT_ID}/locations/us-central1/ragCorpora"

Python

如要瞭解如何安裝或更新 Vertex AI SDK for Python,請參閱「安裝 Vertex AI SDK for Python」。詳情請參閱 Python API 參考文件。


import agentplatform
from agentplatform import types

# TODO(developer): Update and un-comment below lines
# PROJECT_ID = "your-project-id"
# feature_view_name = "projects/{PROJECT_ID}/locations/{LOCATION}/featureOnlineStores/{FEATURE_ONLINE_STORE_ID}/featureViews/{FEATURE_VIEW_ID}"
# display_name = "test_corpus"
# description = "Corpus Description"

# Initialize Agent Platform client once per session.
# See https://docs.cloud.google.com/gemini-enterprise-agent-platform/build/rag-engine/rag-overview#supported-regions
client = agentplatform.Client(project=PROJECT_ID, location="us-east5")

# Configure embedding model (Optional)
backend_config = types.RagVectorDbConfig(
    rag_embedding_model_config=types.RagEmbeddingModelConfig(
        vertex_prediction_endpoint=types.RagEmbeddingModelConfigVertexPredictionEndpoint(
            endpoint="publishers/google/models/text-embedding-005"
        ),
    ),
    vertex_feature_store=types.RagVectorDbConfigVertexFeatureStore(
        feature_view_resource_name=feature_view_name
    )
)

corpus = client.rag.create_corpus(
    rag_corpus=types.RagCorpus(
        display_name=display_name,
        description=description,
        rag_vector_db_config=backend_config,
    )
)
print(corpus)
# Example response:
# RagCorpus(name='projects/1234567890/locations/us-central1/ragCorpora/1234567890',
# display_name='test_corpus', description='Corpus Description', embedding_model_config=...
# ...

使用 RAG API 將檔案匯入 BigQuery 資料表

使用 ImportRagFiles API,將 Google Cloud Storage 或 Google 雲端硬碟中的檔案,匯入 Vertex AI 特徵儲存庫 執行個體的 BigQuery 資料表。檔案會嵌入並儲存在 BigQuery 資料表中。

這個程式碼範例示範如何使用 RAG API 將檔案匯入 BigQuery 資料表。

REST

# TODO(developer): Update and uncomment the following lines:
# RAG_CORPUS_ID = "your-rag-corpus-id"
#
# Google Cloud Storage bucket/file location.
# For example, "gs://rag-fos-test/"
# GCS_URIS= "your-gcs-uris"

# Call ImportRagFiles API to embed files and store in the BigQuery table
curl -X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
https://us-central1-aiplatform.googleapis.com/v1beta1/projects/${PROJECT_ID}/locations/us-central1/ragCorpora/${RAG_CORPUS_ID}/ragFiles:import \
-d '{
  "import_rag_files_config": {
    "gcs_source": {
      "uris": '\""${GCS_URIS}"\"'
    },
    "rag_file_chunking_config": {
      "chunk_size": 512
    }
  }
}'

# Call ListRagFiles API to verify the files are imported successfully
curl -X GET \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
https://us-central1-aiplatform.googleapis.com/v1beta1/projects/${PROJECT_ID}/locations/us-central1/ragCorpora/${RAG_CORPUS_ID}/ragFiles

Python

如要瞭解如何安裝或更新 Vertex AI SDK for Python,請參閱「安裝 Vertex AI SDK for Python」。詳情請參閱 Python API 參考文件。


import agentplatform
from agentplatform import types

from google.genai import types as genai_types

# TODO(developer): Update and un-comment below lines
# PROJECT_ID = "your-project-id"
# corpus_name = "projects/{PROJECT_ID}/locations/us-east5/ragCorpora/{rag_corpus_id}"
# paths = ["gs://my_bucket/my_files_dir/*"]

# Initialize Agent Platform client once per session.
# See https://docs.cloud.google.com/gemini-enterprise-agent-platform/build/rag-engine/rag-overview#supported-regions
client = agentplatform.Client(project=PROJECT_ID, location="us-east5")

response = client.rag.import_files(
    name=corpus_name,
    import_config=types.ImportRagFilesConfig(
        gcs_source=genai_types.GcsSource(uris=paths),
        rag_file_transformation_config=types.RagFileTransformationConfig(
            rag_file_chunking_config=types.RagFileChunkingConfig(
                chunk_size=512,
                chunk_overlap=100,
            )
        ),  # optional
        max_embedding_requests_per_min=900,  # optional
    ),
)

print(f"Imported {response.imported_rag_files_count} files.")
# Example response:
# Imported 2 files.

執行同步處理程序,建構 FeatureOnlineStore 索引

將資料上傳至 BigQuery 資料表後,請執行同步處理程序,讓資料可供線上提供。您必須使用 FeatureView 產生 FeatureOnlineStore 索引,同步處理程序可能需要 20 分鐘才能完成。

這段程式碼範例示範如何執行同步處理程序,建構 FeatureOnlineStore 索引。

REST

使用任何要求資料之前,請先修改下列項目的值:

  • LOCATION_ID:網路商店所在區域,例如 us-central1。
  • PROJECT_ID:專案 ID。
  • FEATUREONLINESTORE_NAME::含有特徵檢視的網路商店名稱。
  • FEATUREVIEW_NAME:要手動啟動資料同步作業的特徵檢視表名稱。

HTTP 方法和網址:

POST https://LOCATION_ID-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION_ID/featureOnlineStores/FEATUREONLINESTORE_NAME/featureViews/FEATUREVIEW_NAME:sync

如要傳送要求,請選擇以下其中一個選項:

curl

執行下列指令:

curl -X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json; charset=utf-8" \
-d "" \
"https://LOCATION_ID-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION_ID/featureOnlineStores/FEATUREONLINESTORE_NAME/featureViews/FEATUREVIEW_NAME:sync"

PowerShell

執行下列指令:

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
-Method POST `
-Headers $headers `
-Uri "https://LOCATION_ID-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION_ID/featureOnlineStores/FEATUREONLINESTORE_NAME/featureViews/FEATUREVIEW_NAME:sync" | Select-Object -Expand Content

您應該會收到如下的 JSON 回覆:

{
  "featureViewSync": "projects/PROJECT_ID/locations/LOCATION_ID/featureOnlineStores/FEATUREONLINESTORE_NAME/featureViews/FEATUREVIEW_NAME/featureViewSyncs/OPERATION_ID"
}

使用 RAG API 擷取相關內容

同步處理程序完成後,您就可以透過 RetrieveContexts API 從 FeatureOnlineStore 索引擷取相關內容。

REST

# TODO(developer): Update and uncomment the following lines:
# RETRIEVAL_QUERY="your-retrieval-query"
#
# Full RAG corpus resource name
# Format:
# "projects/${PROJECT_ID}/locations/us-central1/ragCorpora/${RAG_CORPUS_ID}"
# RAG_CORPUS_RESOURCE="your-rag-corpus-resource"

# Call RetrieveContexts API to retrieve relevant contexts
curl -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
https://us-central1-aiplatform.googleapis.com/v1beta1/projects/${PROJECT_ID}/locations/us-central1:retrieveContexts \
  -d '{
    "vertex_rag_store": {
      "rag_resources": {
          "rag_corpus": '\""${RAG_CORPUS_RESOURCE}"\"',
        },
    },
    "query": {
      "text": '\""${RETRIEVAL_QUERY}"\"',
      "similarity_top_k": 10
    }
  }'

Python

如要瞭解如何安裝或更新 Vertex AI SDK for Python,請參閱「安裝 Vertex AI SDK for Python」。詳情請參閱 Python API 參考文件。


import agentplatform

from agentplatform import types
from google.genai import types as genai_types

# TODO(developer): Update and un-comment below lines
# PROJECT_ID = "your-project-id"
# corpus_name = "projects/[PROJECT_ID]/locations/us-east5/ragCorpora/[rag_corpus_id]"

# Initialize Agent Platform client once per session.
# See https://docs.cloud.google.com/gemini-enterprise-agent-platform/build/rag-engine/rag-overview#supported-regions
client = agentplatform.Client(project=PROJECT_ID, location="us-east5")

response = client.rag.retrieve_contexts(
    vertex_rag_store=genai_types.VertexRagStore(
        rag_resources=[
            genai_types.VertexRagStoreRagResource(
                rag_corpus=corpus_name,
                # Optional: supply IDs from `rag.list_files()`.
                # rag_file_ids=["rag-file-1", "rag-file-2", ...],
            )
        ],
    ),
    query=types.RagQuery(
        text="Hello World!",
        rag_retrieval_config=genai_types.RagRetrievalConfig(
            top_k=10,
            filter=genai_types.RagRetrievalConfigFilter(
                vector_distance_threshold=0.5
            ),
        ),
    )
)
print(response)
# Example response:
# contexts {
#   contexts {
#     source_uri: "gs://your-bucket-name/file.txt"
#     text: "....
#   ....

使用 Agent Platform Gemini API 生成內容

呼叫 Agent Platform GenerateContent API,使用 Gemini 模型生成內容,並在要求中指定 RAG_CORPUS_RESOURCE,從 FeatureOnlineStore 索引擷取資料。

REST

# TODO(developer): Update and uncomment the following lines:
# MODEL_ID=gemini-2.5-flash
# GENERATE_CONTENT_PROMPT="your-generate-content-prompt"

# GenerateContent with contexts retrieved from the FeatureStoreOnline index
curl -X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json"  https://us-central1-aiplatform.googleapis.com/v1beta1/projects/${PROJECT_ID}/locations/us-central1/publishers/google/models/${MODEL_ID}:generateContent \
-d '{
  "contents": {
    "role": "user",
    "parts": {
      "text": '\""${GENERATE_CONTENT_PROMPT}"\"'
    }
  },
  "tools": {
    "retrieval": {
      "vertex_rag_store": {
        "rag_resources": {
            "rag_corpus": '\""${RAG_CORPUS_RESOURCE}"\"',
          },
        "similarity_top_k": 8,
      }
    }
  }
}'

Python

如要瞭解如何安裝或更新 Vertex AI SDK for Python,請參閱「安裝 Vertex AI SDK for Python」。詳情請參閱 Python API 參考文件。


from google import genai
from google.genai import types as genai_types

# TODO(developer): Update and un-comment below lines
# PROJECT_ID = "your-project-id"
# corpus_name = "projects/{PROJECT_ID}/locations/us-east5/ragCorpora/{rag_corpus_id}"

rag_retrieval_tool = genai_types.Tool(
    retrieval=genai_types.Retrieval(
        vertex_rag_store=genai_types.VertexRagStore(
            rag_resources=[
                genai_types.VertexRagStoreRagResource(
                    rag_corpus=corpus_name
                )
            ],
            rag_retrieval_config=genai_types.RagRetrievalConfig(
                top_k=10,
                filter=genai_types.RagRetrievalConfigFilter(
                    vector_distance_threshold=0.5
                ),
            ),
        ),
    )
)

# Create a GenAI SDK client to make a generate_content request
genai_client = genai.Client(vertexai=True, project=PROJECT_ID, location="global")

response = genai_client.models.generate_content(
    model="gemini-3.8-flash",
    contents="Why is the sky blue?",
    config=genai_types.GenerateContentConfig(
        tools=[rag_retrieval_tool]
    ),
)
print(response.text)
# Example response:
#   The sky appears blue due to a phenomenon called Rayleigh scattering.
#   Sunlight, which contains all colors of the rainbow, is scattered
#   by the tiny particles in the Earth's atmosphere....
#   ...

後續步驟