# Data transfer

In YDB, a transfer is an asynchronous mechanism that moves data from a [topic](https://ydb.tech/docs/en//concepts/glossary#topic) to a [table](https://ydb.tech/docs/en//concepts/glossary#table). A transfer instance is [created](https://ydb.tech/docs/en//yql/reference/syntax/create-transfer), [edited](https://ydb.tech/docs/en//yql/reference/syntax/alter-transfer), and [deleted](https://ydb.tech/docs/en//yql/reference/syntax/drop-transfer) using YQL. A transfer runs inside the database in background mode. It serves the function of supplying data from a topic to a table.

In Managed Service for YDB, transfers are only available in the [dedicated](serverless-and-dedicated.md#dedicated) database operating mode. An use case of creating a transfer that moves data within the same database is described in the article titled [Transfer: quick start](https://ydb.tech/docs/en//recipes/transfer/quickstart).

A transfer can read data from topics located either in the same [database](https://ydb.tech/docs/en//concepts/glossary#database) the transfer is created in or in another YDB database or YDB [cluster](https://ydb.tech/docs/en//concepts/glossary#cluster). If you need to read a topic from another database, you must specify its connection parameters when creating a transfer. The target table must always reside in the database the transfer itself is created in.

To read a topic from another Managed Service for YDB database in Yandex Cloud, there is support for authorization via [Identity and Access Management](../../iam/index.md).

## Creating a transfer to read data from another Managed Service for YDB database in Yandex Cloud {#other-database}

### Prepare an API key to access the topic

{% note info %}

You should follow the steps to prepare an API key in the same cloud the source topic database is in.

{% endnote %}

1. [Create a service account](../../iam/operations/sa/create.md) where the topic is located.
1. [Assign the following roles to the service account](../../iam/operations/sa/assign-role-for-sa.md):
   * To read from the data stream: `ydb.viewer`.
   * To automatically add a [reader](https://ydb.tech/docs/en//concepts/glossary#consumer), if applicable: `ydb.admin`.
1. [Create an API key](../../iam/operations/authentication/manage-api-keys.md) with the `yc.ydb.topics.manage` scope.

### Create a secret to access the topic in the source database

Add the secret in the cloud you are going to create your transfer in:

```yql
CREATE OBJECT example_secret (TYPE SECRET) WITH value="ApiKey <your_api_key>";
```

Where:

* `<your_api_key>`: API key you created earlier.

### Create a topic

Create a topic the transfer will be reading data from. You can do it using this SQL query:

```yql
CREATE TOPIC `transfer_recipe/source_topic`;
```

The `transfer_recipe/source_topic` topic allows transferring any unstructured data.

### Create a table

After you create the topic, create a table to receive data from `source_topic`. You can do it using this SQL query:

```yql
CREATE TABLE `transfer_recipe/target_table` (
  partition Uint32 NOT NULL,
  offset Uint64 NOT NULL,
  data String,
  PRIMARY KEY (partition, offset)
);
```

The `transfer_recipe/target_table` table has three columns:

* `partition`: ID of the topic [partition](https://ydb.tech/docs/en//concepts/glossary.md#partition) the message was received from.
* `offset`: [Sequence number](https://ydb.tech/docs/en//concepts/glossary.md#offset) that identifies the message within the partition.
* `data`: Message body.

### Create a transfer

After you create the topic and the table, you need to add a data transfer that will move your messages from the one to the other. You can do it using this SQL query:

```yql
$transformation_lambda = ($msg) -> {
    return [
        <|
            partition: $msg._partition,
            offset: $msg._offset,
            data: $msg._data
        |>
    ];
};
CREATE TRANSFER `transfer_recipe/example_transfer`
  FROM `transfer_recipe/source_topic` TO `transfer_recipe/target_table`
  USING $transformation_lambda
WITH (
    CONNECTION_STRING = '<endpoint>',
    TOKEN_SECRET_NAME = 'example_secret'
)
```

Where:

* `<endpoint>`: Endpoint of connection to the source database the topic is in. It has the following format: `grpcs://lb.etn952fh3eo2jd2mrIhK.ydb.mdb.yandexcloud.net:2135/?database=/global/b1gvcqr959dbmi0e5c1B/etn77atb9o1epqUsCGoY`. The endpoint appears in the Endpoint field on the Overview tab of the data stream page in the [management console](https://console.yandex.cloud).
* `example_secret`: Secret you created earlier.

### Test the transfer

To test the transfer for correct operation, write several messages into the topic.

Some time after the messages are added to source_topic, the relevant records will appear in transfer_recipe/target_table. To check this, run the following SQL query:

```yql
SELECT *
FROM `transfer_recipe/target_table`;
```

## See also

* [Transfer](https://ydb.tech/docs/en//concepts/transfer)
* [Transfer: quick start](https://ydb.tech/docs/en//recipes/transfer/quickstart)
* [Transfer: delivering NGINX access logs into a table](https://ydb.tech/docs/en//recipes/transfer/nginx)
* [CREATE TRANSFER](https://ydb.tech/docs/en//yql/reference/syntax/create-transfer)
* [ALTER TRANSFER](https://ydb.tech/docs/en//yql/reference/syntax/alter-transfer)
* [DROP TRANSFER](https://ydb.tech/docs/en//yql/reference/syntax/drop-transfer)