[Yandex Cloud documentation](../../index.md) > [Yandex Data Transfer](../index.md) > [Tutorials](index.md) > Change data capture and delivery to the queue > YDB and delivery to YDS

# YDB change data capture and delivery to YDS

# YDB change data capture and delivery to YDS


A Data Streams stream can get data from Managed Service for YDB databases in real time using the [Change Data Capture](../concepts/cdc.md) (CDC) technology.

{% note info %}

In YDB, CDC mode is supported starting from version 22.5.

{% endnote %}

To set up CDC using Data Transfer:

1. [Prepare the Managed Service for YDB source database](#prepare-source-ydb).
1. [Create a target stream in Data Streams](#create-target-yds).
1. [Prepare and activate the transfer](#prepare-transfer).
1. [Test the transfer](#verify-transfer).

If you no longer need the resources you created, [delete them](#clear-out).


## Required paid resources {#paid-resources}

* Managed Service for YDB databases (see [Managed Service for YDB pricing](../../ydb/pricing/index.md)). The cost depends on the deployment mode:

	* In serverless mode, you pay for data operations as well as the amount of stored data and backups.
  	* In dedicated instance mode, you pay for the use of computing resources allocated to the database, storage size, and backups.

* Data Streams (see [Data Streams pricing](../../data-streams/pricing.md)). The cost depends on the pricing model:

    * [Based on allocated resources](../../data-streams/pricing.md#rules): You pay a fixed hourly rate for the established throughput limit and message retention period, and additionally for the number of units of actually written data.
    * [On-demand](../../data-streams/pricing.md#on-demand): You pay for the performed read/write operations, the amount of read or written data, and the actual storage used for messages that are still within their retention period.


## Getting started {#before-you-begin}

Set up your infrastructure:

{% list tabs group=instructions %}

- Manually {#manual}

    1. [Create a Managed Service for YDB source database](../../ydb/operations/manage-databases.md) in any suitable configuration.

    1. If you selected Dedicated DB mode for the source, [create](../../vpc/operations/security-group-create.md) and [configure](../../ydb/operations/connection.md#configuring-security-groups) a security group in the network hosting the DB.

    1. [Create a Managed Service for YDB database](../../ydb/operations/manage-databases.md) in any suitable configuration for the Data Streams target stream.

- Terraform {#tf}

    1. If you do not have Terraform yet, [install it](../../tutorials/infrastructure-management/terraform-quickstart.md#install-terraform).
    1. [Get the authentication credentials](../../tutorials/infrastructure-management/terraform-quickstart.md#get-credentials). You can add them to environment variables or specify them later in the provider configuration file.
    1. [Configure and initialize a provider](../../tutorials/infrastructure-management/terraform-quickstart.md#configure-provider). There is no need to create a provider configuration file manually, you can [download it](https://github.com/yandex-cloud-examples/yc-terraform-provider-settings/blob/main/provider.tf).
    1. Place the configuration file in a separate working directory and [specify the parameter values](../../tutorials/infrastructure-management/terraform-quickstart.md#configure-provider). If you did not add the authentication credentials to environment variables, specify them in the configuration file.

    1. Download the [data-transfer-ydb-yds.tf](https://github.com/yandex-cloud-examples/yc-data-transfer-cdc-from-ydb-to-yds/blob/main/data-transfer-ydb-yds.tf) configuration file to the same working directory.

        This file describes:

        * [Network](../../vpc/concepts/network.md#network).
        * [Subnet](../../vpc/concepts/network.md#subnet).
        * [Security group](../../vpc/concepts/security-groups.md) and the rule required to connect to the Managed Service for YDB database.
        * Managed Service for YDB source database.
        * Managed Service for YDB database for the target stream.
        * Transfer.

    1. In the `data-transfer-ydb-yds.tf` file, specify these variables:

        * `source_db_name`: Managed Service for YDB source database name.
        * `target_db_name`: Name of the Managed Service for YDB database for the target data stream.
        * `transfer_enabled`: `0` to ensure that no transfer is created before you [create the endpoints](#prepare-transfer).

    1. Validate your Terraform configuration files using this command:

        ```bash
        terraform validate
        ```

        Terraform will display any configuration errors detected in your files.

    1. Create the required infrastructure:

        1. Run this command to view the planned changes:
        
           ```bash
           terraform plan
           ```
        
           If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.
        
        1. If everything looks correct, apply the changes:
           1. Run this command:
        
              ```bash
              terraform apply
              ```
        
           1. Confirm updating the resources.
           1. Wait for the operation to complete.

        All the required resources will be created in the specified folder. You can check resource availability and their settings in the [management console](https://console.yandex.cloud).

{% endlist %}

## Prepare the Managed Service for YDB source database {#prepare-source-ydb}

1. Get ready for running SQL queries in the Managed Service for YDB source database:

    {% list tabs group=instructions %}

    - Management console {#console}

        1. In the [management console](https://console.yandex.cloud), select the folder containing your database.
        1. Navigate to **Managed Service for YDB**.
        1. Select the database from the list and go to the **Navigation** tab.
        1. Click **New SQL query**.

    - YDB CLI {#cli}

        1. [Set up a connection to the Managed Service for YDB database](../../ydb/operations/connection.md).
        1. Make sure you can run queries using the YDB CLI with the selected authentication mode. For example, for an [IAM token](../../iam/concepts/authorization/iam-token.md):

            ```bash
            ydb \
              --endpoint <endpoint> \
              --database <DB_name> \
              --iam-token-file <path_to_IAM_token> \
              yql -s "SELECT 1;"
            ```

            Result:

            ```text
            ┌─────────┐
            | column0 |
            ├─────────┤
            | 1       |
            └─────────┘
            ```

    {% endlist %}

1. [Create a table](../../ydb/operations/schema.md#create-table) for test data:

    ```sql
    CREATE TABLE test
    (
        id Uint64,
        text Utf8,
        PRIMARY KEY (id)
    );
    ```

## Create a target stream in Data Streams {#create-target-yds}

[Create a stream for the Data Streams](../../data-streams/operations/manage-streams.md#create-data-stream).

## Prepare and activate a transfer {#prepare-transfer}

1. [Create](../operations/endpoint/index.md#create) an [`YDB` source endpoint](../operations/endpoint/source/ydb.md):

    * **Connection settings**:

        * **Database**: Select the Managed Service for YDB database from the list.
        * **Service account ID**: Select an existing service account or create a new one with the `ydb.editor` role.

    * **Included paths list**:

        * **Path 1**: `test`.

1. [Create an endpoint](../operations/endpoint/index.md#create) for the [`Yandex Data Streams` target](../operations/endpoint/target/data-streams.md):

    * **Connection settings**:

        * **Database**: Select the Managed Service for YDB database for the target stream from the list.
        * **Stream**: Specify the name of the stream in Data Streams.
        * **Service account**: Select an existing service account or create a new one with the `yds.editor` role.

    * **Serializing settings**: `Auto`.

1. Create a transfer:

    {% list tabs group=instructions %}

    - Manually {#manual}

        1. [Create a transfer](../operations/transfer.md#create) of the **_Replication_**-type that will use the endpoints you created.
        1. [Activate](../operations/transfer.md#activate) the transfer.

    - Terraform {#tf}

        1. In the `data-transfer-ydb-yds.tf` file, specify the following variables:

            * `source_endpoint_id`: Source endpoint ID.
            * `target_endpoint_id`: Target endpoint ID.
            * `transfer_enabled`: Set to `1` to create the transfer.

        1. Validate your Terraform configuration files using this command:

            ```bash
            terraform validate
            ```

            Terraform will display any configuration errors detected in your files.

        1. Create the required infrastructure:

            1. Run this command to view the planned changes:
            
               ```bash
               terraform plan
               ```
            
               If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.
            
            1. If everything looks correct, apply the changes:
               1. Run this command:
            
                  ```bash
                  terraform apply
                  ```
            
               1. Confirm updating the resources.
               1. Wait for the operation to complete.

            The transfer will be activated automatically upon creation.

    {% endlist %}

## Test the transfer {#verify-transfer}

1. Wait for the transfer status to change to **Replicating**.

1. [Add the test data](../../ydb/operations/crud.md) to the `test` table in the Managed Service for YDB source database:

    ```sql
    INSERT INTO test
    (
        id,
        text
    )
    VALUES
    (
        1,
        'text 1'
    ),
    (
        2,
        'text 2'
    ),
    (
        3,
        'text 3'
    );
    ```

Make sure the data from the source has been moved to the Data Streams stream:

  {% list tabs group=instructions %}

  - Management console {#console}

      1. In the [management console](https://console.yandex.cloud), select a folder.
      1. Navigate to **Data Streams**.
      1. Select the target stream from the list and navigate to ![image](../../_assets/console-icons/bars.svg) **Data viewer**.
      1. Make sure `shard-000000` now contains messages with the source table rows. For a closer look at the messages, click ![image](../../_assets/console-icons/eye.svg).

  - AWS CLI {#cli}

      1. Install the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html).
      1. [Configure the environment](../../data-streams/quickstart/index.md) for Data Streams.
      1. Read the stream data using:

          * [AWS CLI](../../data-streams/operations/aws-cli/get-records.md).
          * [AWS SDK](../../data-streams/operations/aws-sdk/get-records.md).

  {% endlist %}

## Delete the resources you created {#clear-out}

{% note info %}

Before deleting the created resources, [deactivate the transfer](../operations/transfer.md#deactivate).

{% endnote %}

To reduce the consumption of resources, delete those you do not need:

1. [Delete the transfer](../operations/transfer.md#delete).
1. [Delete the source and target endpoints](../operations/endpoint/index.md#delete).
1. If you created service accounts together with the endpoints, [delete them](../../iam/operations/sa/delete.md).
1. Delete the rest of the resources depending on how you created them:

   {% list tabs group=instructions %}

   - Manually {#manual}

       1. [Delete the Managed Service for YDB source database](../../ydb/operations/manage-databases.md#delete-db).
       1. [Delete the Managed Service for YDB database](../../ydb/operations/manage-databases.md#delete-db) used for the target stream.

   - Terraform {#tf}

       1. In the terminal window, go to the directory containing the infrastructure plan.
       
           {% note warning %}
       
           Make sure the directory has no Terraform manifests with the resources you want to keep. Terraform deletes all resources that were created using the manifests in the current directory.
       
           {% endnote %}
       
       1. Delete resources:
       
           1. Run this command:
       
               ```bash
               terraform destroy
               ```
       
           1. Confirm deleting the resources and wait for the operation to complete.
       
           All the resources described in the Terraform manifests will be deleted.

   {% endlist %}