[Yandex Cloud documentation](../../index.md) > [Yandex MetaData Hub](../index.md) > Data Catalog > Concepts > Data sources

# Data sources

A data source is a connection to a database or service used to ingest metadata into a [catalog](data-catalog.md). User data is never ingested. When profiling is enabled, the system runs additional read queries against data or samples to calculate column and table statistics.

The metadata ingested into a catalog contains the following source information:

| Source information | Description  |
| ----------------------- | --------- |
| Stored objects | Databases, schemas, tables, views, and indexes |
| Data structure | Column or field names, their data types and support for storing `NULL` values (nullability), nested columns or fields |
| Comments | Descriptions of tables and columns |
| Data lineage | Links between tables and views |
| Data profiling | Statistics on tables and columns |
| Deleted objects | Objects removed from the source since the previous ingestion |


## Source metadata

The scope of ingested metadata depends on the data source:

* [PostgreSQL](#source-pg)
* [MySQL®](#source-my)
* [ClickHouse®](#source-ch)
* [OpenSearch](#source-os)
* [Yandex MPP Analytics for PostgreSQL/Greenplum®](#source-gp)
* [Yandex StoreDoc/MongoDB](#source-mg)
* [Yandex Data Transfer](#source-dt)
* [Yandex WebSQL](#source-websql)
* [Yandex DataLens](#source-datalens)


### PostgreSQL {#source-pg}

Metadata with the following source information is ingested from PostgreSQL into the catalog:

| Source information | Description  |
| ----------------------- | --------- |
| Databases and schemas | Databases and schemas available in the source |
| Tables | Tables, their technical names, and parent schema mapping |
| Views | Views and their SQL definitions |
| Stored procedures | Stored procedures and their SQL definitions |
| Columns | Names, data types, and parent schema mapping |
| Comments | Table and column descriptions defined via `COMMENT ON` |
| Data lineage | Links between tables and views, as well as between views |
| Column-level data lineage | Links between columns in views |
| Data profiling | Statistics on tables, rows, and columns |
| Deleted objects | Objects removed from the source since the previous ingestion |

Within a catalog, metadata has the following structure: database → schema → table/view → column.

There are two methods for data lineage tracing:

* Parsing the SQL definitions of views.
* Analyzing SQL queries.
  
  SQL query analysis requires PostgreSQL version `13` or higher and the `pg_stat_statements` extension.

For data profiling, the user needs the `SELECT` permissions for the tables.


### MySQL® {#source-my}

Metadata with the following source information is ingested from MySQL® into the catalog:

| Source information | Description  |
| ----------------------- | --------- |
| Databases | Databases available in the source |
| Tables | Tables and their parent database mapping |
| Views | Views and their SQL definitions |
| Stored procedures | Stored procedures and their SQL definitions |
| Columns | Names and data types |
| Comments | Comments to tables and columns |
| Data lineage | Links between tables and views, as well as between views |
| Column-level data lineage | Links between source columns and view columns |
| Data profiling | Statistics on tables and columns |
| Deleted objects | Objects removed from the source since the previous ingestion |

Metadata is ingested from the MySQL® system tables. Within a catalog, metadata has the following structure: database → table/view → column.

Data lineage is traced by parsing SQL definitions of views.

To collect metadata, the user needs the following privileges:

* `SHOW VIEW` for views.
* `SELECT` for tables to access data profiling metrics.

Learn more about [user privileges in MySQL®](../../managed-mysql/concepts/user-rights.md).


### ClickHouse® {#source-ch}

Metadata with the following source information is ingested from ClickHouse® into the catalog:

| Source information | Description  |
| ----------------------- | --------- |
| Databases | Databases available in the source |
| Tables | User tables |
| Views | Views and their SQL definitions |
| Materialized views | Materialized views and their SQL definitions |
| [Dictionaries](../../managed-clickhouse/concepts/dictionaries.md) | Dictionaries and their structure |
| Columns | Names and data types |
| Comments | Comments to tables and columns |
| Data lineage | Links between tables and views, as well as between views |
| Column-level data lineage | Links between columns |
| Use of objects | Object access statistics |
| Data profiling | Statistics on tables and columns |
| Deleted objects | Objects removed from the source since the previous ingestion |

Metadata is ingested from the ClickHouse® system tables using `SQLAlchemy`. Within a catalog, metadata has the following structure: database → table/view → column.

Data lineage is traced by parsing SQL definitions of views stored in `system.tables.create_table_query`.

For data profiling, the user needs the `SELECT` permissions for the tables.


### OpenSearch {#source-os}

Metadata with the following source information is ingested from OpenSearch into the catalog:

| Source information | Description  |
| ----------------------- | --------- |
| [Indexes](../../managed-opensearch/concepts/indexing.md) | Indexes available in the source |
| Index fields | Fields included in indexes |
| Field types | Field data types |
| Index templates | Rules applied when creating indexes |
| Data profiling | Statistics on indexes and fields |
| Deleted objects | Indexes removed from the source since the previous ingestion |

Metadata is ingested from OpenSearch via the API. Within a catalog, each index is represented as a dataset with fields. The field name and type are determined based on the index schema.

For data profiling, the user needs permissions to read indexes.


### Yandex MPP Analytics for PostgreSQL/Greenplum® {#source-gp}

{% note warning %}

Export of metadata from Yandex MPP Analytics for PostgreSQL with Apache Cloudberry™ is not supported.

{% endnote %}

Metadata with the following source information is ingested from Greenplum® and [Yandex MPP Analytics for PostgreSQL](../../managed-greenplum/index.md) into the catalog:

| Source information | Description  |
| ----------------------- | --------- |
| Databases and schemas | Databases and schemas available in the source |
| Tables | Tables, their technical names, and parent schema mapping |
| Views | Views and their SQL definitions |
| Materialized views | Materialized views and their SQL definitions |
| [External tables](../../managed-greenplum/concepts/external-tables.md) | Information about external tables |
| Columns | Names, data types, and parent schema mapping |
| Limitations | Rules for maintaining data integrity |
| Indexes | Table indexes |
| Comments | Table and column descriptions defined via `COMMENT ON` |
| Data lineage | Links between tables, views, and materialized views |
| Data profiling | Statistics on tables and columns |
| Deleted objects | Objects removed from the source since the previous ingestion |

Within a catalog, metadata has the following structure: database → schema → table/view → column.

There are two methods for data lineage tracing:

* Parsing the SQL definitions of views.
* Analyzing SQL queries.

For data profiling, the user needs the `SELECT` permissions for the tables.


### Yandex StoreDoc/MongoDB {#source-mg}

Metadata with the following source information is ingested from MongoDB and [Yandex StoreDoc](../../storedoc/index.md) into the catalog:

| Source information | Description  |
| ----------------------- | --------- |
| Databases | Databases available in the source |
| Collections | Names, parent database mapping, list of fields and their types, document structure, technical attributes |
| Fields | Names, nested fields, types, support for storing `NULL` values, mixed types |
| Deleted objects | Collections removed from the source since the previous ingestion |

Within a catalog, metadata has the following structure: database → collection → field.

In Yandex StoreDoc and MongoDB, there is no single collection schema, as individual documents may have varying structures. If schema inference is enabled, a unified collection schema is generated based on a sample of documents. To build the schema, you need read permissions for the databases metadata is ingested from.


### Yandex Data Transfer {#source-dt}

The metadata ingested from [Data Transfer](../../data-transfer/index.md) into a catalog constitutes metadata of [transfers](../../data-transfer/concepts/index.md#transfer) running within the selected [Yandex Cloud folders](../../resource-manager/concepts/resources-hierarchy.md#folder).

This metadata includes the following transfer information:

* Links between storages, tables, and columns.
* [Data transformations](../../data-transfer/concepts/data-transformation.md).

For each transfer participant without a corresponding [source](../operations/data-catalog/create-source.md) in the catalog, a storage without an active source is created. In this case, the catalog will only store metadata obtained during the transfer. You can later create a source for each storage and configure metadata ingestion.


### Yandex WebSQL {#source-websql}

The metadata ingested from [WebSQL](../../websql/index.md) into a catalog constitutes the metadata of SQL queries and database objects used in queries. Query results are not ingested. The ingested queries are restricted to databases which have a corresponding source in the catalog.

In the catalog, a user will only see their queries or queries for which they have view permissions.


### Yandex DataLens {#source-datalens}

A catalog ingests metadata for the following [DataLens](../../datalens/index.md) objects:

* [Datasets](../../datalens/dataset/index.md)
* [Charts](../../datalens/concepts/chart/index.md)
* [Dashboards](../../datalens/concepts/dashboard.md)
* [Reports](../../datalens/reports/index.md)
* [Workbooks and collections](../../datalens/workbooks-collections/index.md)

The ingested metadata is limited to the pre-selected [connections](../../datalens/concepts/connection/index.md). If no connections are selected, the system will ingest metadata from all available connections.

To ingest metadata, the [service account](../../iam/concepts/users/service-accounts.md) needs the `datalens.visitor` and `datalens.admin` roles for the organization. Instead of the `datalens.admin` role, you can grant granular permissions to access individual DataLens resources (e.g., `datalens.workbooks.viewer`). In this scenario, the ingested metadata will be limited to the specific resources to which the service account has access.

[Learn more about roles in DataLens](../../datalens/security/roles.md).

_ClickHouse® is a registered trademark of [ClickHouse, Inc](https://clickhouse.com)._

_Greenplum® and Greenplum Database® are registered trademarks or trademarks of Broadcom Inc. in the United States and/or other countries._