[Yandex Cloud documentation](../../../../index.md) > [Yandex Data Transfer](../../../index.md) > [Step-by-step guides](../../index.md) > [Configuring endpoints](../index.md) > YDB > Source

# Transferring data from a Managed Service for YDB source endpoint


Yandex Data Transfer enables you to migrate data from a Managed Service for YDB database and implement various data transfer, processing, and transformation scenarios. To implement a transfer:

1. [Explore possible data transfer scenarios](#scenarios).
1. [Prepare the Managed Service for YDB](#prepare) database for the transfer.
1. [Set up a source endpoint](#endpoint-settings) in Yandex Data Transfer.
1. [Set up one of the supported data targets](#supported-targets).
1. [Create](../../transfer.md#create) a transfer and [start](../../transfer.md#activate) it.
1. Perform required operations with the database and [control the transfer](../../monitoring.md).
1. In case of any issues, [use ready-made solutions](#troubleshooting) to resolve them.

## Scenarios for transferring data from Managed Service for YDB {#scenarios}

1. Data change capture means tracking changes to a database and delivering those changes to consumers. It is used for applications that are sensitive to real-time data changes.

    * [Capturing changes from PostgreSQL and delivering to YDS](../../../tutorials/ydb-to-yds.md).
    * [Capturing changes from PostgreSQL and delivering to Apache Kafka®](../../../tutorials/cdc-ydb.md).

1. Uploading data to data marts is a process of transferring prepared data to storage for subsequent visualization.

    * [Loading data from YDB to the ClickHouse®](../../../tutorials/ydb-to-clickhouse.md) data mart.

1. Uploading data to scalable Object Storage storage allows you to save on data storage and simplifies the exchange with contractors.

    * [Loading data from YDB to Object Storage](../../../tutorials/ydb-to-object-storage.md).

For a detailed description of possible Yandex Data Transfer scenarios, see [Tutorials](../../../tutorials/index.md).

## Preparing the source database {#prepare}

If you selected Dedicated database mode, [create](../../../../vpc/operations/security-group-create.md) and [configure](../../../../ydb/operations/connection.md#configuring-security-groups) a security group in the network hosting the DB.

## Configuring the Managed Service for YDB source endpoint {#endpoint-settings}

When [creating](../index.md#create) or [updating](../index.md#update) an endpoint, you can define:
* Yandex Managed Service for YDB DB connection settings. These are required parameters.
* Transfer path list (for tables and directories).


{% note warning %}

To create or edit an endpoint of a managed database, you will need the [`ydb.viewer`](../../../../ydb/security/index.md#ydb-viewer) role or the primitive [`viewer`](../../../../iam/roles-reference.md#viewer) role for the folder the cluster of this managed database resides in.

{% endnote %}


{% list tabs group=instructions %}

- Management console {#console}

  * **Database**: Select a [Managed Service for YDB](../../../../ydb/index.md) database from the list.

  
  * **Service account ID**: Select or [create](../../../../iam/operations/sa/create.md) a [service account](../../../../iam/concepts/users/service-accounts.md) with the `ydb.editor` role that Data Transfer will use to connect to the database.

  * **Security groups**: Select the cloud network to host the endpoint and security groups for network traffic.
    
    Thus, you will be able to apply the specified security group rules to the VMs and clusters in the selected network without changing the settings of these VMs and clusters. For more information, see [Networking in Yandex Data Transfer](../../../concepts/network.md).


  * **Included paths list**: Add paths to the Managed Service for YDB tables and directories to transfer.

      Table and directory names must meet the ID naming rules in YDB. Escaping double quotes is not required. The `/` character is used as a path delimiter. For more information, see [this YDB guide](https://ydb.tech/docs/en//concepts/connect#database).

      Adding new paths when editing an endpoint used in **Snapshot and increment** transfers in the **Replicating** status will not result in uploading the data history of these tables. To add a table with its historical data, use the **List of objects for transfer** field in the [transfer settings](../../transfer.md#update).

      For **Snapshot** transfers, you may skip specifying paths. In this case, all tables will be transferred.

      For **Replication** or **Snapshot and increment** transfers, specifying paths is required, including when you replicate all tables.

  * **Name of pre-created changefeed**: Specify the name of the update stream if already created. Otherwise, leave the field blank.

  * **Name of pre-created consumer**: Specify the data consumer you created for the update stream. The default consumer is `__data_transfer_consumer`.


      
      {% note info %}

      If a data consumer is specified, the `ydb.viewer` role is enough for the service account specified in the endpoint settings to connect the transfer to YDB. If no data consumer is specified, the service account must have the `ydb.editor` role to create the default consumer.

      {% endnote %}
      

  * **Parallel copy settings**:

    {% note warning %}

    For consistent data migration during parallel copying, the transfer first copies the original tables from the source to a temporary folder, reads the data from these copies, and then deletes the temporary copies.
    The data is copied at the metadata level, which does not take much time. However, the copied tables are fully counted towards the Managed Service for YDB disk quota.
    **Therefore, by running a transfer with the sharded snapshot option on you may deplete your disk quota for the source database. So make sure your database has enough disk quota left before you run a transfer.**

    {% endnote %}

    * **Use sharded snapshot**: Enable to speed up the transfer with the help of a sharded snapshot.

      During the copying stage, tables are divided into partitions. Copying will proceed faster if the number of workers multiplied by the number of threads inside a worker is proportional to the number of partitions.

    * **Copy folder**: Specify the name of the directory to store the copies of the tables during the copying stage. The directory is created in the root directory of the source base and gets the default name of `data-transfer`. As soon as the copying stage is over, the directory will be deleted.
    
    
    To control parallel copying, the service account specified in the endpoint settings must have the [`ydb.editor`](../../../../ydb/security/index.md#ydb-editor) role.


    To maximize the parallel copying speed:

      * In the source endpoint parameters, enable **Use sharded snapshot**.

      
      * In the transfer parameters, under **Parallel snapshot settings**, set the number of workers and threads per worker so that the total number of threads equals the number of partitions. For example, to copy a table of 6 partitions faster, you can allocate three workers with two threads per worker.
      

    [More on parallel copying](../../../concepts/sharded.md).

- Terraform {#tf}

  * Endpoint type: `ydb_source`.

  * `name`: Managed Service for YDB endpoint name.
  
  * `database`: Path to a Managed Service for YDB database.
    
      You can view the DB path in the [management console](https://console.yandex.cloud) on the YDB database information page.
    
      > Example of a DB path: `/ru-central1/b1gia87mbaomkfvs6rgl/etnudu2n9ri35luqe4h1`.
  
  
  * `service_account_id`: [Service account](../../../../iam/concepts/users/service-accounts.md) with the `ydb.editor` role that Data Transfer will use to connect to the database.
  
  
  * `paths`: List of paths to the Managed Service for YDB tables or directories to transfer.
    
      Table and directory names must meet the ID naming rules in YDB. Escaping double quotes is not required. The `/` character is used as a path delimiter. For more information, see the [YDB](https://ydb.tech/docs/en//concepts/connect#database) documentation.
    
      For **Replication** or **Snapshot and increment** transfers, specifying paths is required, including when you replicate all tables. For **Snapshot** transfers, you may skip specifying paths. In this case, all tables will be transferred.
    
      Adding new paths when editing an endpoint used in **Snapshot and increment** transfers in the **Replicating** status will not result in uploading the data history of these tables. To add a table with its historical data, use the **List of objects for transfer** field in the [transfer settings](../../transfer.md#update).
  
  * `security_groups`: [Security groups](../../../../vpc/concepts/security-groups.md) for the network traffic.
    
      Security group rules apply to a transfer. They allow opening network access from the transfer VM to the cluster. For more information, see [Networking in Yandex Data Transfer](../../../concepts/network.md).
    
      Security groups must belong to the same network as the cluster.
    
      {% note info %}
    
      In Terraform, it is not required to specify a network for security groups.
    
      {% endnote %}
  
  * `changefeed_custom_name`: Here, specify the name of the update stream if already created. Otherwise, omit this parameter.

  Here is an example of the configuration file structure (not all parameters are given):

  
  ```hcl
  resource "yandex_datatransfer_endpoint" "ydb-source" { 
    name = "<endpoint_name>"
    settings {
      ydb_source {
        database               = "<YDB_database_name>"
        service_account_id     = "<service_account_ID>"
        paths                  = ["<list_of_paths_to_transferrable_YDB_objects>"]
        changefeed_custom_name = "<changefeed_name>"
      }
    }
  }
  ```


  For more information, see [this Terraform provider guide](../../../../terraform/resources/datatransfer_endpoint.md).

{% endlist %}


## Configuring the data target {#supported-targets}

Configure one of the supported data targets:

* [ClickHouse®](../target/clickhouse.md)
* [Yandex Object Storage](../target/object-storage.md)
* [Apache Kafka®](../target/kafka.md)
* [YDS](../target/data-streams.md)
* [YTsaurus](yt.md)
* [PostgreSQL](../target/postgresql.md)
* [Yandex Managed Service for YDB](../target/yandex-database.md).

For a complete list of supported sources and targets in Yandex Data Transfer, see [Available transfers](../../../transfer-matrix.md).

After configuring the data source and target, [create and start the transfer](../../transfer.md#create).

## Troubleshooting data transfer issues {#troubleshooting}

Known issues when using a Managed Service for YDB endpoint:

### Transfer failure {#overloaded}

A _**Replication**_ or _**Snapshot and increment**_ transfer is interrupted with an error.

Error message:

```text
/Ydb.PersQueue.V1.PersQueueService/AddReadRule failed: OVERLOADED
```

Transfers are aborted due to the cloud [quota](https://console.yandex.cloud/cloud?section=quotas) on the number of operations with Managed Service for YDB.

**Solution:**

1. Increase the **Number of schema transactions per minute** property in the [Managed Service for YDB quotas](../../../../ydb/concepts/limits.md) for the cloud with the required database and [reactivate](../../transfer.md#activate) the transfer.

For more troubleshooting tips, see [Troubleshooting](../../../troubleshooting/index.md).