About metadata connectors

As data landscapes grow more complex, metadata can become siloed across different platforms, making it difficult for teams to find, trust, and manage their data. Knowledge Catalog (formerly Dataplex Universal Catalog) connectors help solve this by letting you automatically import technical, operational, business, and semantic metadata from third-party data sources directly into Google Cloud.

Through these connectors, you can let analysts search external data assets without switching tools, let data stewards enforce centralized governance across a multi-cloud environment, and monitor cross-platform data quality in one place. By scheduling recurring runs, you can keep your catalog synchronized with your source systems so that your teams have an accurate, up-to-date view of your data ecosystem.

Supported data sources

You can import metadata from the following sources using Knowledge Catalog connectors:

  • Databricks Unity Catalog
  • dbt Core and Cloud
  • MySQL
  • Oracle
  • PostgreSQL
  • Snowflake Horizon
  • SQL Server

Integrations with data governance partners

You can also integrate Knowledge Catalog with other data cataloging and governance platforms. Instead of pulling from raw data sources, these integrations let parallel catalog tools share information. You can enrich metadata within Knowledge Catalog or export Knowledge Catalog metadata to view it within those third-party tools.

The following partners provide integrations for Knowledge Catalog:

The remainder of this page describes importing metadata through connectors. For more information about partner integrations, see the linked partner pages.

How it works

Knowledge Catalog connectors use prebuilt integrations to connect to your data sources and automatically extract metadata. BigQuery Data Transfer Service manages these transfers and maps the extracted metadata to a specific Knowledge Catalog entry group that you choose.

The connectors extract the following types of metadata:

  • Technical metadata: Schema, data type, and column name definitions.
  • Operational metadata: Created and modified timestamps, lineage, usage statistics, and data quality results.
  • Business metadata: Tags, owners, and annotations.
  • Semantic metadata: Dimensions, measures, metrics, golden queries, and relationships.

After you configure a connector, Knowledge Catalog schedules metadata import runs on the schedule that you specify, ensuring that the catalog stays synchronized with the source system. Each time a metadata import run occurs, it performs a full overwrite of the entries in the destination entry group managed by that configuration. Metadata objects that are no longer present in the source are removed from the entry group, and new objects are added. Objects in the entry group that aren't imported by the connector configuration are unaffected.

Terminology

The following terms are used in this documentation:

  • Metadata import configuration: The settings for a scheduled or on-demand task that uses BigQuery Data Transfer Service to extract metadata from a source and load it into Knowledge Catalog.
  • Run: A single execution instance of a metadata import configuration.
  • Entry group: A resource in Knowledge Catalog that contains metadata entries. When you configure a connector, you designate an entry group as the destination for the imported metadata.
  • Entry: A resource in Knowledge Catalog that represents a data asset, such as a table or view.

Pricing

There are no additional charges from Knowledge Catalog or BigQuery Data Transfer Service for importing metadata using connectors.

Standard charges apply for other Knowledge Catalog usage (such as metadata storage). For more information, see Knowledge Catalog pricing.

Limitations

  • Only metadata is imported, not the underlying data from the source tables or views. To transfer the underlying data, you can use the BigQuery Data Transfer Service.
  • Each run performs a full overwrite of connector-managed entries in the destination entry group. Incremental metadata updates are not supported.
  • Run history is retained for 90 days.

What's next