[Yandex Cloud documentation](../../index.md) > [Tutorials](../index.md) > [Machine learning and artificial intelligence](index.md) > Using Yandex Cloud AI Studio generative models > Integrating the AI data analyst with Data Catalog

# Integrating AI Studio with Yandex Data Catalog

# Integration with Yandex Data Catalog

You can use an AI assistant to search and analyze patterns in [metadata catalogs](../../metadata-hub/concepts/data-catalog.md) deployed in Data Catalog. To do that, you need to connect the Data Catalog MCP server to MCP Hub. The server allows you to request the list of metadata catalogs, search through metadata, and obtain its lineage graph at the table and column level for use in the context of conversation with agents.

To set up integration with Data Catalog in AI Studio:

1. [Set up your infrastructure](#infra).
1. [Prepare the metadata catalog](#prepare-data-catalog).
1. [Connect an external MCP server](#connect-mcp).
1. [Test a conversation with the agent](#test-dialog).

## Getting started {#before-you-begin}

Sign up for Yandex Cloud and create a [billing account](../../billing/concepts/billing-account.md):
1. Navigate to the [management console](https://console.yandex.cloud) and log in to Yandex Cloud or create a new account.
1. On the **[Yandex Cloud Billing](https://center.yandex.cloud/billing/accounts)** page, make sure you have a billing account linked and it has the `ACTIVE` or `TRIAL_ACTIVE` [status](../../billing/concepts/billing-account-statuses.md). If you do not have a billing account, [create one](../../billing/quickstart/index.md) and [link](../../billing/operations/pin-cloud.md) a cloud to it.

If you have an active billing account, you can create or select a [folder](../../resource-manager/concepts/resources-hierarchy.md#folder) for your infrastructure on the [cloud page](https://console.yandex.cloud/cloud).

[Learn more about clouds and folders here](../../resource-manager/concepts/resources-hierarchy.md).


### Required paid resources {#paid-resources}

The integration infrastructure cost includes a fee for Agent Atelier based on the number of tokens in request and response (see [Yandex Cloud AI Studio pricing](https://aistudio.yandex.ru/docs/en/ai-studio/pricing)). You start paying for the agent as soon as you activate it.


## Set up your infrastructure {#infra}

### Create a folder and network {#create-folder}

Create a resource folder to host your metadata catalog.

{% list tabs group=instructions %}

- Management console {#console}

   1. In the [management console](https://console.yandex.cloud), select a cloud and click ![create](../../_assets/console-icons/plus.svg) **Create folder**.
   1. Name your folder, e.g., `data-folder`.
   1. Select **Create a default network**. This will create a [network](../../vpc/concepts/network.md#network) with subnets in each [availability zone](../../overview/concepts/geo-scope.md).
   1. Click **Create**.

{% endlist %}

[Learn more about clouds and folders](../../resource-manager/concepts/resources-hierarchy.md).

### Create a service account {#create-sa}

{% list tabs group=instructions %}

- Management console {#console}

   1. Navigate to `data-folder`.
   1. Navigate to **Identity and Access Management**.
   1. Click **Create service account**.
   1. Name the [service account](../../iam/concepts/users/service-accounts.md), e.g., `sa-for-mcp-server`.
   1. Click **Add role** and assign the following [roles](../../iam/concepts/access-control/roles.md) to the service account:
      * `data-catalog.user` for access to the metadata catalog resources.
      * `serverless.mcpGateways.invoker` for access to the MCP server in MCP Hub.
      * `serverless.mcpGateways.anonymousInvoker` for access to the external MCP server.

   1. Click **Create**.

{% endlist %}

## Prepare the metadata catalog {#prepare-data-catalog}

### Create a metadata catalog {#create-catalog}

{% list tabs group=instructions %}

- Management console {#console}

    1. In the [management console](https://console.yandex.cloud), select the [resource folder](../../resource-manager/concepts/resources-hierarchy.md#folder) where you want to create a metadata catalog.
    1. Navigate to **Yandex MetaData Hub**.
    1. In the left-hand panel, select ![image](../../_assets/console-icons/folder-magnifier.svg) **Data Catalog**.
    1. Click **Creating a catalog**.
    1. In the **Name** field, enter the catalog name: `test-sales`.
    1. Click **Create**.

    {% note info %}
    
    When you create a metadata catalog, the **metadata AI markup** is on by default.
    
    With this option enabled, the AI assistant suggests descriptions, [domains](../../metadata-hub/concepts/data-catalog.md#domains-and-subdomains), [classifications and tags](../../metadata-hub/concepts/data-catalog.md#classifications-and-tags), [glossaries and terms](../../metadata-hub/concepts/data-catalog.md#glossaries-and-terms), and marks up your metadata using them. You can confirm, edit, or reject any suggestion your AI assistant makes by hovering over the **AI** icon next to the suggestion and selecting the action.
    
    After the catalog is created, you can manage the AI markup on the **Overview** page or when [updating](../../metadata-hub/operations/data-catalog/update-catalog.md) the catalog.
    
    {% endnote %}

{% endlist %}

### Create a metadata source {#create-source}

{% list tabs group=instructions %}

- Management console {#console}

    1. In the left-hand panel, select ![image](../../_assets/console-icons/cloud-arrow-up-in.svg) **Data sources**.
    1. Click **Create data source**.
    1. Specify `test-sales-source` as the source name.
    1. Select the type of the backend that will supply metadata for analysis. Once the source is created, you cannot change the database type. Available backends:

       * PostgreSQL
       * MySQL®
       * ClickHouse®
       * Yandex StoreDoc/MongoDB
       * OpenSearch
       * Greenplum®
       * Yandex Data Transfer
       * WebSQL
       * DataLens

    1. Specify the source parameters for the selected database type:

        * **Connection ID**: Managed connection ID in [Yandex Connection Manager](../../metadata-hub/quickstart/connection-manager.md).
        * **Database name**: Name of the database to ingest metadata from.

    1. Click **Create**.

{% endlist %}

### Create and start a data ingestion {#create-ingestion}

{% list tabs group=instructions %}

- Management console {#console}

  1. In the left-hand panel, select ![image](../../_assets/console-icons/arrow-up-from-square.svg) **Ingestions**.
  1. Click **Create ingestion**.
  1. Specify the ingestion settings:

      * In the **Name** field, enter `load-sales` as the ingestion name.
      * Select the metadata source you [created earlier](#create-source).
      * Specify the ingestion configuration for the data source:

         * Select **Manually** for the ingestion schedule.
         * Optionally, under **Data Filters**, use regular expressions to specify which databases and database objects to include in or exclude from the ingestion.

         * Under **Metadata Types**, select the metadata types to extract from the source.
         * Optionally, under **Data Profiling**:
         
           * Select **Enable Profiling** to perform data profiling, i.e., analysis and collection of statistics on the data being extracted.
           * Select **Table level only** to skip data profiling in every table column. With this option on, data characteristics will only be collected for the table as a whole.
           * In the **Max Workers** field, specify the number of computing threads for profiling.
           * In the **Sample Size** field, specify the number of rows for sampling for column profiling. This setting applies when the **Use Sampling** option is enabled.
           * In the **Table size limit** field, specify the table size in GB above which the table will be excluded from profiling.
           * In the **Table row limit** field, specify the number of rows above which the table will be excluded from profiling.
           * Select **Enable field null count** to get the number of rows with `NULL` for each column.
           * Select **Enable distinct value count** to get the number of unique values for each column.
           * Select **Enable field min value** to get the minimum value for each numeric column.
           * Select **Enable field max value** to get the maximum value for each numeric column.
           * Select **Enable field mean value** to get the mean value for each numeric column.
           * Select **Enable field median value** to get the median value for each numeric column.
           * Select **Enables field value stddev** to get the standard deviation value for each numeric column.
           * Select **Enables field quintiles** to get quantiles for each numeric column.
           * Select **Enable distinct value frequency count** to get the frequency of unique values for each column.
           * Select **Enable field histogram** to get a histogram for each numeric column.
           * Select **Enable field sample values** to get sample values for each column.
           * Select **Enable query joining** to dynamically combine SQL queries for faster profiling.
           * In the **Limit** field, specify the maximum number of rows to profile. If set to `0`, all rows will be profiled.
         
         * Under **Metadata Processing**, select the image for metadata processing:
           * Enable **Use File Cache** to improve ingestion performance.

  1. Click **Create**.
  1. In the list of ingestions, click ![image](../../_assets/console-icons/ellipsis.svg) in the line with your new ingestion and select **Start**.
  
     During ingestion, the AI assistant will automatically mark up the data. Once successfully completed, the ingestion will get the **Success** status.

  1. To view ingested and marked-up data, select ![image](../../_assets/console-icons/database-magnifier.svg) **Metadata search** in the left-hand panel.

     The page displays the info about the data, i.e., data source, database, and tables.

     {% note info %}

     The AI assistant automatically creates entities for metadata markup (domains, glossaries, tags, classifications, and terms) and their descriptions. You can confirm, edit, or reject the markup suggested by your AI assistant by hovering over the **AI** icon next to the suggestion and selecting the action.

    {% endnote %}

{% endlist %}

## Connect an external MCP server {#connect-mcp}

### Connecting in AI Studio {#mcp-ai-studio}

{% list tabs group=instructions %}

- Management console {#console}

  1. Navigate to `data-folder`.
  1. Navigate to **AI Studio**.
  1. In the left-hand panel, select ![logo-mcp](../../_assets/console-icons/logo-mcp.svg) **MCP servers** and click **Create MCP server**. In the window that opens:

      1. Under **Add method**, select ![plug-connection](../../_assets/console-icons/plug-connection.svg) **Connect**.
      1. Under **Tools**, click **Add tools**. In the window that opens, configure the MCP server connection:

         * **Transport**: **Streamable HTTP**.

         * **URL**: `https://datacatalog-consumer.mcp.cloud.yandex.net/mcp`.

         * **Authorization type**: `Access token`.

         * Under **Authorization header**, set the **Value** field to `Bearer <IAM_token>`. To do it, get an [IAM token](../../iam/concepts/authorization/iam-token.md) for the service account [created earlier](#create-sa), then paste it into the field.

            {% note info %}

            The IAM token [lifetime](../../iam/concepts/authorization/iam-token.md#lifetime) does not exceed 12 hours; however, we recommend requesting a token more often, e.g., every hour.
            
            To have the IAM token reissued automatically, the `export IAM_TOKEN=$(yc iam create-token)` script can be used.

            {% endnote %}

      1. Click **Connect**.
      1. In the **Add tools** window that opens, select all tools and click **Add**.

      1. Under **Server parameters**:

          1. In the **Name** field, enter a name for the new MCP server. Follow these naming requirements:
             
             * Length: between 3 and 63 characters.
             * It can only contain lowercase Latin letters, numbers, and hyphens.
             * It must start with a letter and cannot end with a hyphen.
          1. Optionally, add a description and labels for the server you are creating by using the corresponding buttons.
          1. In the **Access** field, select **Private**.
          1. In the **Service account** field, select the previously created service account.
          1. Optionally, turn on the **Enable logging** option and configure the logging settings to keep a log of the MCP server you are creating.
      1. Click **Save**.

  1. In the left-hand panel, select ![logo-agent](../../_assets/console-icons/face-robot.svg) **Agents** and click **Create agent**.
  1. Specify the agent settings:
     * **Name**: Agent name.
     * **Model**: Language model.
     * Under **Instruction**, select a ready-made system instruction template for the agent or describe the agent's desired behavior and actions.
     * Under **Tools**:
       * Click **Add** and select **Add MCP**.
       * In the list, select the MCP server you created earlier and click **Select**.
       * In the **Default behavior for all tools** field, select **Confirmation not needed**.
       * Click **Create and continue**.

{% endlist %}

### Connecting to an external AI agent {#mcp-connect-agent}

1. Get an [IAM token](../../iam/concepts/authorization/iam-token.md) for the service account you [created earlier](#create-sa).
   
   {% note info %}

   The IAM token [lifetime](../../iam/concepts/authorization/iam-token.md#lifetime) does not exceed 12 hours; however, we recommend requesting a token more often, e.g., every hour.
   
   To have the IAM token reissued automatically, the `export IAM_TOKEN=$(yc iam create-token)` script can be used.

   {% endnote %}

1. Specify the Data Catalog MCP server configuration for your agent:

   ```json
   {
     "mcpServers": {
       "yandex-cloud-datacatalog-consumer": {
         "type": "streamableHttp",
         "url": "https://datacatalog-consumer.mcp.cloud.yandex.net/mcp",
         "headers": {
           "Authorization": "Bearer <IAM_token>"
         }
       }
     }
   }
   ```

## Test a conversation with the agent {#test-dialog}

{% note tip %}

If using the agent in AI Studio, do the test in the right-hand **Agent testing** panel.

{% endnote %}

1. Start a conversation with the agent by specifying the data catalog ID as shown below:

   ```text
   Use the marked-up data in the apah36iavgh5******** data catalog.
   ```

1. Use the examples of prompts to respond to which the agent will be analyzing the marked-up data from Data Catalog. It is assumed that the data contains sales-related information:

   * `Write an SQL query to generate YoY sales analytics`.
   * `Find all tables with user payment information`.
   * `Which tables are marked as containing sensitive data?`
   * `Where does the customer_transactions table get its data from?`
   * `Help find the tables needed to calculate the user retention metric`.
   * `Where can I find the website users' behavior data?`
   * `Which data should I use to analyze sales funnel conversion rate?`
   * `Show all dependencies of the transactions table to see how schema changes affect it`.