Create and use a glossary

You can use a glossary to define terminology specific to your domain. With a glossary, you can add term pairs, including a source and target language term. The term pairs ensure that the Vertex AI Translation service consistently translates your terminology.

The following are examples of cases where you can define glossary entries:

• Product names: Identify product names to keep them in the translation. For example, Google Home must translate to Google Home.
• Ambiguous words: Specify the meaning of vague words and homonyms. For example, bat can mean a piece of sports equipment or an animal.
• Borrowed words: Clarify the meaning of words adopted from a different language. For example, bouillabaisse in French translates to bouillabaisse in English, a fish stew dish.

The terms in a glossary can be single words (also referred to as tokens) or short phrases, usually shorter than five words. Vertex AI Translation ignores matching glossary entries if the words are stopwords.

Vertex AI Translation offers the following glossary methods available in Google Distributed Cloud (GDC) air-gapped:

Before you begin

Before creating a glossary to define your terminology for translation, you must have a project named translation-glossary-project. The custom resource of the project must look like in the following example:

  apiVersion: resourcemanager.gdc.goog/v1
  kind: Project
  metadata:
    labels:
      atat.config.google.com/clin-number: CLIN_NUMBER
      atat.config.google.com/task-order-number: TASK_ORDER_NUMBER
    name: translation-glossary-project
    namespace: platform

To get the permissions you need to use a glossary, ask your IAM Project Admin to grant you the following roles in your project namespace:

• AI Translation Developer: Get the AI Translation Developer (ai-translation-developer) role to access the Vertex AI Translation service.
• Project Bucket Admin: Get the Project Bucket Admin (project-bucket-admin) role to manage storage buckets and objects within buckets, which lets you create and upload files.

For more information about prerequisites, see Set up a translation project.

Create a glossary file

You must create a glossary file to store your source language and target language terms. This section contains the two different glossary layouts you can use to define your terms.

The following table describes the limits supported on Distributed Cloud for glossary files:

Choose one of the following layouts for your glossary file:

• Unidirectional glossary: Specify the expected translation for a pair of source and target terms in a specific language. Unidirectional glossaries support TSV, CSV, and TMX file formats.
• Equivalent term set glossary: Specify the expected translation in multiple languages on each row. Equivalent term set glossaries support CSV file formats.

Unidirectional glossary

The Vertex AI Translation API accepts tab-separated values (TSV) and comma-separated values (CSV). Each row contains a pair of terms separated by a tab (\t) or a comma (,) for these file formats.

The Vertex AI Translation API also accepts the Translation Memory eXchange (TMX) format, a standard XML format for providing the translation's source and target term pairs. The supported input files are in a format based on TMX version 1.4.

The following examples show the required structure for TSV, CSV, and TMX file formats of unidirectional glossaries:

<?xml version='1.0' encoding='utf-8'?>
<!DOCTYPE tmx SYSTEM "tmx14.dtd">
<tmx version="1.4">
  <header segtype="sentence" o-tmf="UTF-8" adminlang="en" srclang="en" datatype="PlainText"/>
  <body>
    <tu>
      <tuv xml:lang="en">
        <seg>account</seg>
      </tuv>
      <tuv xml:lang="es">
        <seg>cuenta</seg>
      </tuv>
    </tu>
    <tu>
      <tuv xml:lang="en">
        <seg>directions</seg>
      </tuv>
      <tuv xml:lang="es">
        <seg>indicaciones</seg>
      </tuv>
    </tu>
  </body>
</tmx>

After identifying the glossary terms in your unidirectional glossary, upload the file to a storage bucket and make it available to the Vertex AI Translation API by creating and importing a glossary.

Equivalent term set glossary

The Vertex AI Translation API accepts glossary files for equivalent term sets using the CSV format. To define equivalent term sets, create a multi-column CSV file in which each row lists a single glossary term in multiple languages. See the list of supported languages and their respective language codes.

The following image shows an example of a multi-column CSV file. Each row represents a glossary term, and each column represents a translation of the term into different languages.

The header is the first row in the file, which identifies the language for each column. The header row uses the ISO-639-1 or BCP-47 standard language codes. The Vertex AI Translation API doesn't use part-of-speech (pos) information, and specific position values aren't validated.

Each subsequent row contains equivalent glossary terms in the languages identified in the header. You can leave columns blank if the term isn't available in all languages.

After identifying the glossary terms in your equivalent term set, upload the file to a storage bucket and make it available to the Vertex AI Translation API by creating and importing a glossary.

Upload your glossary file to a storage bucket

Follow these steps to upload your glossary file to a storage bucket:

1. Configure the gdcloud CLI for object storage.

2. Create a storage bucket in your project namespace. Use a Standard storage class.

   You can create the storage bucket by deploying a Bucket resource in your project namespace:

     apiVersion: object.gdc.goog/v1
     kind: Bucket
     metadata:
       name: glossary-bucket
       namespace: translation-glossary-project
     spec:
       description: bucket for translation glossary
       storageClass: Standard
       bucketPolicy:
         lockingPolicy:
           defaultObjectRetentionDays: 90
   

3. Grant read permissions on the bucket and get, list, and watch permissions on secrets in the project namespace to the service accounts (ai-translation-system-sa and g-vai-translation-sie-sa) used by the Vertex AI Translation service.

   Follow these steps to create the role and role binding using custom resources on the management API server. A user who already holds the secrets permissions must apply these resources, or the request fails with attempting to grant RBAC permissions not currently held:

   1. Create the role by deploying a Role resource in the project namespace:

        apiVersion: rbac.authorization.k8s.io/v1
        kind: Role
        metadata:
          name: ai-translation-glossary-reader
          namespace: translation-glossary-project
        rules:
          -
            apiGroups:
              - object.gdc.goog
            resources:
              - buckets
            verbs:
              - read-object
          -
            apiGroups:
              - ""
            resources:
              - secrets
            verbs:
              - get
              - list
              - watch
      

   2. Create the role binding by deploying a RoleBinding resource in the project namespace:

        apiVersion: rbac.authorization.k8s.io/v1
        kind: RoleBinding
        metadata:
          name: ai-translation-glossary-reader-rolebinding
          namespace: translation-glossary-project
        roleRef:
          apiGroup: rbac.authorization.k8s.io
          kind: Role
          name: ai-translation-glossary-reader
        subjects:
          -
            kind: ServiceAccount
            name: ai-translation-system-sa
            namespace: g-vai-translation-sie
          -
            kind: ServiceAccount
            name: g-vai-translation-sie-sa
            namespace: g-vai-translation-sie
      

4. Upload your glossary file to the storage bucket you created. For more information, see Upload and download storage objects in projects.

Create a glossary

The CreateGlossary method creates a glossary and returns the identifier to the long-running operation that generates the glossary.

To create a glossary, replace the following before using any of the request data:

• ENDPOINT: the Vertex AI Translation endpoint that you use for your organization. For more information, view service status and endpoints.
• PROJECT_ID: your project ID.
• BUCKET_NAME: the name of the storage bucket where your glossary file is located.
• GLOSSARY_FILENAME: the name of your glossary file in the storage bucket.

The following is the syntax for an HTTP request to create a glossary:

POST https://ENDPOINT/v3/projects/PROJECT_ID/glossaries

According to the glossary file you created, choose one of the following options to create a glossary:

curl -X POST \
-H "Authorization: Bearer TOKEN" \
-H "Content-Type: application/json; charset=utf-8" \
-d @request.json \
"https://ENDPOINT/v3/projects/PROJECT_ID/glossaries"

$cred = TOKEN
$headers = @{ "Authorization" = "Bearer $cred" }
Invoke-WebRequest
  -Method POST
  -Headers $headers
  -ContentType: "application/json; charset=utf-8"
  -InFile request.json
  -Uri "https://ENDPOINT/v3/projects/PROJECT_ID/glossaries"
  | Select-Object -Expand Content

{
"name": "projects/PROJECT_ID/operations/OPERATION_ID",
"metadata": {
    "@type": "type.googleapis.com/google.cloud.translation.v3.CreateGlossaryMetadata",
    "name": "projects/PROJECT_ID/glossaries/GLOSSARY_ID",
    "state": "RUNNING",
    "submitTime": TIME
  }
}

curl -X POST \
    -H "Authorization: Bearer TOKEN" \
    -H "Content-Type: application/json; charset=utf-8" \
    -d @request.json \
    "https://ENDPOINT/v3/projects/PROJECT_ID/glossaries"

{
  "name": "projects/PROJECT_ID/operations/OPERATION_ID",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.translation.v3.CreateGlossaryMetadata",
    "name": "projects/PROJECT_ID/glossaries/GLOSSARY_ID",
    "state": "RUNNING",
    "submitTime": TIME
  }
}

$cred = TOKEN
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
  -Method POST `
  -Headers $headers `
  -ContentType: "application/json; charset=utf-8" `
  -InFile request.json `
  -Uri "https://ENDPOINT/v3/projects/PROJECT_ID/glossaries"
  | Select-Object -Expand Content

{
  "name": "projects/PROJECT_ID/operations/OPERATION_ID",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.translation.v3.CreateGlossaryMetadata",
    "name": "projects/PROJECT_ID/glossaries/GLOSSARY_ID",
    "state": "RUNNING",
    "submitTime": TIME
  }
}

Depending on the size of the glossary file, creating a glossary typically takes less than 10 minutes to complete. You can retrieve the status of this operation to know when it is finished.

Get a glossary

The GetGlossary method returns a stored glossary. If the glossary doesn't exist, the output returns the NOT_FOUND value. To call the GetGlossary method, specify your project ID and the glossary ID. Both the CreateGlossary and ListGlossaries methods return the glossary ID.

For example, the following requests return information about a specific glossary in your project:

curl -X GET \
   -H "Authorization: Bearer TOKEN" \
   "http://ENDPOINT/v3/projects/PROJECT_ID/glossaries/GLOSSARY_ID"

from google.cloud import translate_v3 as translate
from google.api_core.client_options import ClientOptions

def get_glossary(creds, name="projects/PROJECT_ID/glossaries/GLOSSARY_ID"):
    """Get a glossary by its full name. The service assigns the glossary ID."""
    client = translate.TranslationServiceClient(
        credentials=creds, client_options=ClientOptions(api_endpoint="ENDPOINT:443"))
    response = client.get_glossary(name=name)
    print(f"Glossary name: {response.name}")

List glossaries

The ListGlossaries method returns a list of Glossary resources in a project, each containing the full name (projects/PROJECT_ID/glossaries/GLOSSARY_ID). If the project has no glossaries, the output returns {}. If the project doesn't exist, the output returns the NOT_FOUND value. To call the ListGlossaries method, specify your project ID and the Vertex AI Translation endpoint.

For example, the following request returns the glossaries in your project:

curl -X GET \
    -H "Authorization: Bearer TOKEN" \
    "http://ENDPOINT/v3/projects/PROJECT_ID/glossaries?page_size=10"

Replace TOKEN with the authentication token you obtained.

Delete a glossary

The DeleteGlossary method deletes a glossary. If the glossary doesn't exist, the output returns the NOT_FOUND value. To call the DeleteGlossary method, specify your project ID, glossary ID, and the Vertex AI Translation endpoint. Both the CreateGlossary and ListGlossaries methods return the glossary ID.

For example, the following request deletes a glossary from your project:

curl -X DELETE \
    -H "Authorization: Bearer TOKEN" \
    "http://ENDPOINT/v3/projects/PROJECT_ID/glossaries/GLOSSARY_ID"

Replace TOKEN with the authentication token you obtained.