[Yandex Cloud documentation](../../index.md) > [Yandex Data Transfer](../index.md) > [Step-by-step guides](index.md) > Preparing for a transfer

# Preparing for a transfer



## Preparing a source {#source}

### Airbyte® sources {#source-airbyte}

#### AWS CloudTrail source {#source-aws}

Get an AWS key ID and secret access key by following the [AWS guide](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html).

For more information, see [this Airbyte® guide](https://docs.airbyte.com/integrations/sources/aws-cloudtrail/).

#### BigQuery source {#source-bigquery}

To prepare a BigQuery data source:

1. [Create a Google Cloud service account](https://cloud.google.com/iam/docs/creating-managing-service-accounts).
1. [Add the service account](https://cloud.google.com/iam/docs/granting-changing-revoking-access#granting-console) as a participant with the `BigQuery User` role to the Google Cloud project.
1. [Create a Google Cloud service account key](https://cloud.google.com/iam/docs/creating-managing-service-account-keys).

For more information, see [this Airbyte® guide](https://docs.airbyte.com/integrations/sources/bigquery).

#### Microsoft SQL Server source {#source-mssql}

Airbyte® has the following requirements for a Microsoft SQL Server data source:

1. Make sure your database is accessible from the computer running Airbyte®.
1. Create a dedicated read-only Airbyte® user with access to all tables that require replication.

For more information, see [this Airbyte® guide](https://docs.airbyte.com/integrations/sources/mssql).

Airbyte® is already built into Data Transfer, so you do not have to create a separate VM to deploy it and add a user. All you have to do is grant Data Transfer [network access](../concepts/network.md) to the source database.

#### S3 source {#source-s3}

If using a private bucket as a source, grant the `read` and `list` permissions to the account you are going to use for connection.

For more information, see [this Airbyte® guide](https://docs.airbyte.com/integrations/sources/s3/).


### Apache Kafka® source {#source-kf}

{% list tabs %}

- Managed Service for Apache Kafka®
    
    
    [Create a user](../../managed-kafka/operations/cluster-accounts.md#create-account) with the `ACCESS_ROLE_CONSUMER` role for the source topic.


- Apache Kafka®
    
    1. If not planning to use [Cloud Interconnect](../../interconnect/concepts/index.md) or [VPN](https://en.wikipedia.org/wiki/Virtual_private_network) for connections to an external cluster, make such cluster accessible from the Internet from [IP addresses used by Data Transfer](../../overview/concepts/public-ips.md#virtual-private-cloud).
       
       For details on linking your network up with external resources, see [this concept](../concepts/network.md#source-external).
    
    1. [Configure access to the source cluster from Yandex Cloud](../concepts/network.md#source-external).
    
    1. [Configure user access rights](https://kafka.apache.org/42/security/authorization-and-acls/#examples) to the topic you need.
    
    1. Grant the `READ` permissions to the consumer group whose ID matches the transfer ID.
    
        ```text
        bin/kafka-acls --bootstrap-server localhost:9092 \
          --command-config adminclient-configs.conf \
          --add \
          --allow-principal User:username \
          --operation Read \
          --group <transfer_ID>
        ```
    
    1. Optionally, to log in with a username and password, [configure SASL authentication](https://kafka.apache.org/42/security/authentication-using-sasl/).

{% endlist %}

### ClickHouse® source {#source-ch}

{% note info %}

Yandex Data Transfer cannot transfer a ClickHouse® database if its name contains a hyphen.


If transferring tables with engines other than `ReplicatedMergeTree` and `Distributed` in a ClickHouse® multi-host cluster, the transfer will fail with the following error: `the following tables have not Distributed or Replicated engines and are not yet supported`.

{% endnote %}

{% list tabs %}

* Managed Service for ClickHouse®

    
    1. Make sure the tables you are transferring use the `MergeTree` family engines. Only these tables and [materialized views](https://clickhouse.com/docs/enen/materialized-views) (MaterializedView) will be transferred.

       In case of a multi-host cluster, only tables and materialized views with the `ReplicatedMergeTree` or `Distributed` engines will be transferred. Make sure these tables and views are present on all the cluster hosts.

    1. [Create a user](../../managed-clickhouse/operations/cluster-users.md) with access to the source database. In the user settings, specify a value of at least `1000000` for the **Max execution time** [parameter](../../managed-clickhouse/concepts/settings-list.md#setting-max-execution-time).

* ClickHouse®

    1. Make sure the tables you are transferring use the `MergeTree` family engines. Only these tables and [materialized views](https://clickhouse.com/docs/enen/materialized-views) (MaterializedView) will be transferred.

       In case of a multi-host cluster, only tables and materialized views with the `ReplicatedMergeTree` or `Distributed` engines will be transferred. Make sure these tables and views are present on all the cluster hosts.

    1. If not planning to use [Cloud Interconnect](../../interconnect/concepts/index.md) or [VPN](https://en.wikipedia.org/wiki/Virtual_private_network) for connections to an external cluster, make such cluster accessible from the Internet from [IP addresses used by Data Transfer](../../overview/concepts/public-ips.md#virtual-private-cloud).
       
       For details on linking your network up with external resources, see [this concept](../concepts/network.md#source-external).

    1. [Configure access to the source cluster from Yandex Cloud](../concepts/network.md#source-external).

    1. Create a user with access to the source database. In the user settings, specify a value of at least `1000000` for the **Max execution time** parameter.

{% endlist %}

### Greenplum® source {#source-gp}


{% note info %}

Data stored in a `MATERIALIZED VIEW` is not transferred. To transfer `MATERIALIZED VIEW` data, create an ordinary `VIEW` that refers to the `MATERIALIZED VIEW` to be transferred.

{% endnote %}

{% list tabs %}

- Yandex MPP Analytics for PostgreSQL

    
    1. Create a user account the transfer will use to connect to the source. To do this, run the following command:
    
        ```pgsql
        CREATE ROLE <username> LOGIN ENCRYPTED PASSWORD '<password>';
        ```

    1. Configure the source cluster to enable the user you created to connect to all the cluster [master hosts](../../managed-greenplum/concepts/index.md).
    
    1. If you are going to use [parallel copy](../concepts/sharded.md), configure the source cluster to enable the user you created to connect to all the cluster's [segment hosts](../../managed-greenplum/concepts/index.md) in utility mode. To do this, make sure that the "Access from Data Transfer" setting is enabled for the cluster.
    
    1. Grant the user you created the `SELECT` privilege for the tables to transfer and the `USAGE` privilege for the schemas these tables are in.
    
       Privileges must be granted to entire tables. Access to certain table columns only is not supported.
    
       Tables without the required privileges are unavailable to Data Transfer. These tables are processed as if they did not exist.
    
       This example issues privileges to all the tables in the selected schema:
    
        ```pgsql
        GRANT SELECT ON ALL TABLES IN SCHEMA <schema_name> TO <username>;
        GRANT USAGE ON SCHEMA <schema_name> TO <username>;
        ```

- Greenplum®
    
    1. If not planning to use [Cloud Interconnect](../../interconnect/concepts/index.md) or [VPN](https://en.wikipedia.org/wiki/Virtual_private_network) for connections to an external cluster, make such cluster accessible from the Internet from [IP addresses used by Data Transfer](../../overview/concepts/public-ips.md#virtual-private-cloud).
       
       For details on linking your network up with external resources, see [this concept](../concepts/network.md#source-external).
    
    1. Create a user account the transfer will use to connect to the source. To do this, run the following command:
    
        ```pgsql
        CREATE ROLE <username> LOGIN ENCRYPTED PASSWORD '<password>';
        ```

    1. Configure the source cluster to enable the user you created to connect to all the cluster [master hosts](../../managed-greenplum/concepts/index.md).
    
    1. If you are going to use [parallel copy](../concepts/sharded.md), configure the source cluster to enable the user you created to connect to all the cluster's [segment hosts](../../managed-greenplum/concepts/index.md) in utility mode.
    
    1. Grant the user you created the `SELECT` privilege for the tables to transfer and the `USAGE` privilege for the schemas these tables are in.
    
       Privileges must be granted to entire tables. Access to certain table columns only is not supported.
    
       Tables without the required privileges are unavailable to Data Transfer. These tables are processed as if they did not exist.
    
       This example grants privileges to all the database tables:
    
        ```pgsql
        GRANT SELECT ON ALL TABLES IN SCHEMA <schema_name> TO <username>;
        GRANT USAGE ON SCHEMA <schema_name> TO <username>;
        ```

{% endlist %}

Data Transfer works with Greenplum® differently depending on the transfer configuration and the source cluster contents. Detailed information is available in the section on [Greenplum® source endpoint settings](endpoint/source/greenplum.md).


### MongoDB source {#source-mg}


{% list tabs %}

- Yandex StoreDoc (Managed Service for MongoDB)


    1. Estimate the total number of databases for transfer and the total Yandex StoreDoc workload. If the workload on the databases exceeds 10,000 writes per second, create multiple endpoints and transfers. For more information, see [Transferring data from a MongoDB/Yandex StoreDoc (Managed Service for MongoDB) source endpoint](endpoint/source/mongodb.md).
    
    1. [Create a user](../../storedoc/operations/cluster-users.md#adduser) with the `readWrite` role for each source database to replicate. The `readWrite` role is required to enable the transfer to write data to the `__data_transfer.__dt_cluster_time` service collection.

- MongoDB
    
    1. Estimate the total number of databases for transfer and the total MongoDB workload. If the workload on the databases exceeds 10,000 writes per second, create multiple endpoints and transfers. For more information, see [Transferring data from a MongoDB/Yandex StoreDoc (Managed Service for MongoDB) source endpoint](endpoint/source/mongodb.md).
    
    1. If not planning to use [Cloud Interconnect](../../interconnect/concepts/index.md) or [VPN](https://en.wikipedia.org/wiki/Virtual_private_network) for connections to an external cluster, make such cluster accessible from the Internet from [IP addresses used by Data Transfer](../../overview/concepts/public-ips.md#virtual-private-cloud).
       
       For details on linking your network up with external resources, see [this concept](../concepts/network.md#source-external).
    
    1. Make sure the MongoDB version on the target is `4.0` or higher.
    
    1. Make sure the MongoDB cluster is configured so that it returns correctly resolving IP addresses or FQDNs (fully qualified domain names) in response to requests.
    
    1. [Configure access to the source cluster from Yandex Cloud](../concepts/network.md#source-external). To configure a source cluster for connections from the internet:
        
        1. In the configuration file, change `net.bindIp` from `127.0.0.1` to `0.0.0.0`:
        
            ```yaml
            # network interfaces
            net:
              port: 27017
              bindIp: 0.0.0.0
            ```
        
        1. Restart `mongod`:
        
            ```bash
            sudo systemctl restart mongod.service
            ```
    
    1. If the source cluster does not use replication, enable it:
        
        1. Add the replication settings to the `/etc/mongod.conf` configuration file:
        
            ```yaml
            replication:
              replSetName: <replica_set_name>
            ```
        
        1. Restart `mongod`:
        
            ```bash
            sudo systemctl restart mongod.service
            ```
        
        1. Connect to MongoDB and initialize the replica set with this command:
        
            ```javascript
            rs.initiate({
                _id: "<replica_set_name>",
                members: [{
                    _id: 0,
                    host: "<IP_address_Yandex_StoreDoc_listens_on>:<port>"
                }]
            });
            ```
    
    1. Create a user with the `readWrite` role for all the source databases to replicate:
    
        ```javascript
        use admin
        db.createUser({
            user: "<username>",
            pwd: "<password>",
            mechanisms: ["SCRAM-SHA-1"],
            roles: [
                {
                    db: "<source_database_1_name>",
                    role: "readWrite"
                },
                {
                    db: "<source_database_2_name>",
                    role: "readWrite"
                },
                ...
            ]
        });
        ```
    
       Once started, the transfer will connect to the source on behalf of this user. The `readWrite` role is required to enable the transfer to write data to the `__data_transfer.__dt_cluster_time` service collection.
    
       {% note info %}
    
       For MongoDB 3.6 or higher, you only need to assign the created user the `read` role for the databases to replicate.
    
       {% endnote %}
    
    1. When using MongoDB 3.6 or higher, to run the transfer, the user must have the read permission for the `local.oplog.rs` collection and the read and write permissions for the `__data_transfer.__dt_cluster_time` collection. To assign a user the `clusterAdmin` role granting these permissions, connect to MongoDB and run the following commands:
    
        ```js
        use admin;
        db.grantRolesToUser("<username>", ["clusterAdmin"]);
        ```
    
       To grant more granular permissions, you can assign the `clusterMonitor` role required for reading the `local.oplog.rs` collection and grant read and write access to the `__data_transfer.__dt_cluster_time` system collection.

{% endlist %}

### MySQL® source {#source-my}

{% list tabs %}

- Managed Service for MySQL®

    
    1. [Enable full binary logging](../../managed-mysql/operations/update.md#change-mysql-config) on the source by setting the [**Binlog row image**](https://dev.mysql.com/doc/refman/5.7/en/replication-options-binary-log.html#sysvar_binlog_row_image) parameter to `FULL` or `NOBLOB`.
 
    1. Optionally, [set a limit](../../managed-mysql/operations/update.md#change-mysql-config) on the size of data chunks to send using the **Max allowed packet** parameter.
    
    1. [Create a user](../../managed-mysql/operations/cluster-users.md#adduser) for connecting to the source.
        
        1. [Grant](../../managed-mysql/operations/grant.md#grant-privilege) the user `ALL_PRIVILEGES` for the source database.
        
        1. [Grant](../../managed-mysql/concepts/settings-list.md#setting-administrative-privileges) the user the `REPLICATION CLIENT` and `REPLICATION SLAVE` administrative privileges.
    
    1. For _**Replication**_ and _**Snapshot and increment**_ transfers, tables without unique indexes are not transferred. 
       
       If a table containing a row has a primary key, only the primary key columns get written to the binary log when the row is modified. If there is no primary key but there is a unique index where all columns are not `NULL`, only these columns get written to the binary log. If there is no primary key or unique index without `NULL` columns, all columns in the row get written to the binary log.
       
       To make sure the transfer works correctly if the database you need to migrate contains tables with no unique indexes:
       
       * Do not transfer such tables. You can add them to the list of excluded tables in [source endpoint settings](endpoint/source/mysql.md#additional-settings).
       
       * Create unique indexes, e.g., primary keys (`PRIMARY KEY`), in the tables which do not have them.
       
           1. To get a list of tables without a primary key, run the query:
       
               ```sql
               SELECT
                   tab.table_schema AS database_name,
                   tab.table_name AS table_name,
                   tab.table_rows AS table_rows,
                   tco.*
               FROM information_schema.tables tab
                   LEFT JOIN information_schema.table_constraints tco
                       ON (tab.table_schema = tco.table_schema
                           AND tab.table_name = tco.table_name
               )
               WHERE
                   tab.table_schema NOT IN ('mysql', 'information_schema', 'performance_schema', 'sys')
                   AND tco.constraint_type IS NULL
                   AND tab.table_type = 'BASE TABLE';
               ```
       
           1. Study the structure of tables without a primary key that need to be transferred to the target:
       
               ```sql
               SHOW CREATE TABLE <database_name>.<table_name>;
               ```
       
           1. Add a simple or complex primary key to the tables that need to be transferred to the target:
       
               ```sql
               ALTER TABLE <table_name> ADD PRIMARY KEY (<column_or_group_of_columns>);
               ```
       
           1. If the table being transferred to the target has no column or group of columns suitable for the role of the primary key, create a new column:
       
               ```sql
               ALTER TABLE <table_name> ADD id BIGINT PRIMARY KEY AUTO_INCREMENT;
               ```
    
       {% note info %}
    
       If, when creating a primary key, you get an error saying _`Creating index 'PRIMARY' required more than 'innodb_online_alter_log_max_size' bytes of modification log. Please try again`_, [increase](../../managed-mysql/operations/update.md#change-mysql-config) the [`Innodb log file size`](https://dev.mysql.com/doc/refman/8.0/en/innodb-parameters.html#sysvar_innodb_log_file_size) parameter value in the DBMS settings.
    
       {% endnote %}
    
    1. Deactivate trigger transfer at the transfer initiation stage and reactivate it at the completion stage (for the _**Replication**_ and the _**Snapshot and increment**_ transfer types). For more information, see the [description of the additional endpoint settings for the MySQL® source](endpoint/source/mysql.md#additional-settings).

- MySQL®
    
    1. If not planning to use [Cloud Interconnect](../../interconnect/concepts/index.md) or [VPN](https://en.wikipedia.org/wiki/Virtual_private_network) for connections to an external cluster, make such cluster accessible from the Internet from [IP addresses used by Data Transfer](../../overview/concepts/public-ips.md#virtual-private-cloud).
       
       For details on linking your network up with external resources, see [this concept](../concepts/network.md#source-external).
    
    1. Make sure the source uses the MyISAM or InnoDB low-level storage subsystem. If you use other subsystems, the transfer may fail.
    
    1. [Enable full binary logging](https://dev.mysql.com/doc/refman/8.0/en/replication-options-binary-log.html#sysvar_binlog_row_image) on the source by setting the `binlog_row_image` parameter to `FULL` or `NOBLOB`.

       {% note info %}

       For MariaDB 11.4 or higher, set `binlog_legacy_event_pos = true`.

       {% endnote %}    
    
    1. [Specify row format for the binary log](https://dev.mysql.com/doc/refman/5.7/en/replication-options-binary-log.html#sysvar_binlog_format) on the source by setting the `binlog_format` parameter to `ROW`.

    1. For the _**Replication**_ and _**Snapshot and increment**_ transfer types:

        * In the `log_bin` parameter, [specify the path to the binary log file](https://dev.mysql.com/doc/refman/8.0/en/replication-options-binary-log.html#option_mysqld_log-bin).

        * Enter the binary log information using the [SHOW MASTER STATUS](https://dev.mysql.com/doc/refman/8.0/en/show-master-status.html) request (for MySQL® 5.7 and 8.0) or the [SHOW BINARY LOG STATUS](https://dev.mysql.com/doc/refman/8.4/en/show-binary-log-status.html) request (for MySQL® 8.4). The request should return a string with the information, not an empty response.

    1. If the replication source is a cluster that is behind the load balancer, enable GTID mode for it (`GTID-MODE = ON`).
    
       If it is not possible to enable GTID mode for any reason, make sure the binary log name template contains the host name.
    
       In both cases, this will allow replication to continue even after changing the master host.
    
    1. Optionally, [set a limit](https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_max_allowed_packet) on the size of outbound data chunks using the `max_allowed_packet` parameter.
    
    1. Create a user to connect to the source and grant them the required privileges:
    
        ```sql
        CREATE USER '<username>'@'%' IDENTIFIED BY '<password>';
        GRANT ALL PRIVILEGES ON <database_name>.* TO '<username>'@'%';
        GRANT REPLICATION CLIENT, REPLICATION SLAVE ON *.* TO '<username>'@'%';
        ```

    1. For _**Replication**_ and _**Snapshot and increment**_ transfers, tables without unique indexes are not transferred. 
       
       If a table containing a row has a primary key, only the primary key columns get written to the binary log when the row is modified. If there is no primary key but there is a unique index where all columns are not `NULL`, only these columns get written to the binary log. If there is no primary key or unique index without `NULL` columns, all columns in the row get written to the binary log.
       
       To make sure the transfer works correctly if the database you need to migrate contains tables with no unique indexes:
       
       * Do not transfer such tables. You can add them to the list of excluded tables in [source endpoint settings](endpoint/source/mysql.md#additional-settings).
       
       * Create unique indexes, e.g., primary keys (`PRIMARY KEY`), in the tables which do not have them.
       
           1. To get a list of tables without a primary key, run the query:
       
               ```sql
               SELECT
                   tab.table_schema AS database_name,
                   tab.table_name AS table_name,
                   tab.table_rows AS table_rows,
                   tco.*
               FROM information_schema.tables tab
                   LEFT JOIN information_schema.table_constraints tco
                       ON (tab.table_schema = tco.table_schema
                           AND tab.table_name = tco.table_name
               )
               WHERE
                   tab.table_schema NOT IN ('mysql', 'information_schema', 'performance_schema', 'sys')
                   AND tco.constraint_type IS NULL
                   AND tab.table_type = 'BASE TABLE';
               ```
       
           1. Study the structure of tables without a primary key that need to be transferred to the target:
       
               ```sql
               SHOW CREATE TABLE <database_name>.<table_name>;
               ```
       
           1. Add a simple or complex primary key to the tables that need to be transferred to the target:
       
               ```sql
               ALTER TABLE <table_name> ADD PRIMARY KEY (<column_or_group_of_columns>);
               ```
       
           1. If the table being transferred to the target has no column or group of columns suitable for the role of the primary key, create a new column:
       
               ```sql
               ALTER TABLE <table_name> ADD id BIGINT PRIMARY KEY AUTO_INCREMENT;
               ```
    
       {% note info %}
    
       If, when creating a primary key, you get an error saying _`Creating index 'PRIMARY' required more than 'innodb_online_alter_log_max_size' bytes of modification log. Please try again`_, increase the [`inno_db_log_file_size`](https://dev.mysql.com/doc/refman/8.0/en/innodb-parameters.html#sysvar_innodb_log_file_size) parameter value in the DBMS settings.
    
       {% endnote %}
    
    1. Deactivate trigger transfer at the transfer initiation stage and reactivate it at the completion stage (for the _**Replication**_ and the _**Snapshot and increment**_ transfer types). For more information, see the [description of the additional endpoint settings for the MySQL® source](endpoint/source/mysql.md#additional-settings).

{% endlist %}

### Elasticsearch source {#source-es}


If not planning to use [Cloud Interconnect](../../interconnect/concepts/index.md) or [VPN](https://en.wikipedia.org/wiki/Virtual_private_network) for connections to an external cluster, make such cluster accessible from the Internet from [IP addresses used by Data Transfer](../../overview/concepts/public-ips.md#virtual-private-cloud).

For details on linking your network up with external resources, see [this concept](../concepts/network.md#source-external).

### OpenSearch source {#source-os}

{% list tabs %}

- OpenSearch

  
  If not planning to use [Cloud Interconnect](../../interconnect/concepts/index.md) or [VPN](https://en.wikipedia.org/wiki/Virtual_private_network) for connections to an external cluster, make such cluster accessible from the Internet from [IP addresses used by Data Transfer](../../overview/concepts/public-ips.md#virtual-private-cloud).
  
  For details on linking your network up with external resources, see [this concept](../concepts/network.md#source-external).


{% endlist %}

### Oracle source {#source-oracle}

{% note info %}

Some versions of Oracle use `V_$` instead of `V$` as the prefix for system objects. For example, `V_$DATABASE` instead of `V$DATABASE`.

If you get an error like "`can only select from fixed tables/views`" when granting permissions to system objects, try changing the prefixes.

{% endnote %}

{% list tabs %}

- Oracle

   * To prepare the source for the _**Snapshot**_ transfer:

      1. Create a user account the transfer will use to connect to the source:

         ```sql
         CREATE USER <username> IDENTIFIED BY <password>;
         GRANT CREATE SESSION TO <username>;
         ```

      1. Grant privileges to the created user:

         ```sql
         GRANT SELECT ON V$DATABASE TO <username>;
         GRANT SELECT ON DBA_EXTENTS TO <username>;
         GRANT SELECT ON DBA_OBJECTS TO <username>;
         GRANT FLASHBACK ANY TABLE TO <username>;
         ```

         If required, you can only grant the `FLASHBACK` privileges to the tables you need to copy rather than to `ANY TABLE`.

      1. Grant the user the [privilege to read the tables](https://docs.oracle.com/en/database/oracle/oracle-database/21/sqlrf/GRANT.html) to be copied.

   * To prepare the source for the _**Replication**_ transfer:

      1. Create a user account the transfer will use to connect to the source:

         ```sql
         CREATE USER <username> IDENTIFIED BY <password>;
         ALTER USER <username> DEFAULT tablespace USERS TEMPORARY tablespace TEMP;
         ALTER USER <username> quote unlimited on USERS;

         GRANT
             CREATE SESSION,
             execute_catalog_role,
             SELECT ANY TRANSACTION,
             SELECT ANY DISCTIONARY,
             CREATE PROCEDURE,
             LOGMINING
         TO <username>;
         ```

      1. Grant privileges to the created user:

         ```sql
         GRANT SELECT ON V$DATABASE TO <username>;
         GRANT SELECT ON V$LOG TO <username>;
         GRANT SELECT ON V$LOGFILE TO <username>;
         GRANT SELECT ON V$ARCHIVED_LOG TO <username>;

         GRANT SELECT ON dba_objects TO <username>;
         GRANT SELECT ON dba_extents TO <username>;

         GRANT EXECUTE ON SYS.DBMS_LOGMNR TO <username>;
         GRANT SELECT ON SYSTEM.LOGMNR_COL$ TO <username>;
         GRANT SELECT ON SYSTEM.LOGMNR_OBJ$ TO <username>;
         GRANT SELECT ON SYSTEM.LOGMNR_USER$ TO <username>;
         GRANT SELECT ON SYSTEM.LOGMNR_UID$ TO <username>;
         ```

      1. Grant the user the [privilege to read the tables](https://docs.oracle.com/en/database/oracle/oracle-database/21/sqlrf/GRANT.html) to be replicated.
      1. Enable [Minimal Supplemental Logging](https://docs.oracle.com/database/121/SUTIL/GUID-D2DDD67C-E1CC-45A6-A2A7-198E4C142FA3.htm#SUTIL1583) with primary keys as follows:

         ```sql
         ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (PRIMARY KEY) COLUMNS;
         ```

   * If you are using the [CDB environment](https://docs.oracle.com/database/121/CNCPT/cdbovrvw.htm#CNCPT89234), configure the following settings:

      1. Create a [user](https://docs.oracle.com/en/database/oracle/oracle-database/19/multi/overview-of-managing-a-multitenant-environment.html#GUID-7D303718-2D59-495F-90FB-E51A377B1AD2) `Common User`:

         ```sql
         CREATE USER C##<username> IDENTIFIED BY <password> CONTAINER=all;
         ALTER USER C##<username> DEFAULT TABLESPACE USERS temporary tablespace TEMP CONTAINER=all;
         ALTER USER C##<username> quota unlimited on USERS CONTAINER=all;
         ALTER USER C##<username> SET container_data = (cdb$root, <your_PCB_name>) CONTAINER=current;

         GRANT
             CREATE SESSION,
             execute_catalog_role,
             SELECT ANY TRANSACTION,
             SELECT ANY DICTIONALY,
             CREATE PROCEDURE,
             LOGMINING,
             SET CONTAINER
         TO C##<username> CONTAINER=ALL;
         ```

         If required, you can only specify the `cdb$root` container and the container with the tables you need to transfer.

      1. To allow the user to switch to the `cdb$root` container, grant them the `ALTER SESSION` privileges:

         ```sql
         GRANT ALTER SESSION TO C##<username>;
         ```

      1. Grant privileges to the created user:

         ```sql
         GRANT SELECT ON V$DATABASE TO C##<username> CONTAINER=ALL;
         GRANT SELECT ON V$LOG TO C##<username> CONTAINER=ALL;
         GRANT SELECT ON V$LOGFILE TO C##<username> CONTAINER=ALL;
         GRANT SELECT ON V$ARCHIVED_LOG TO C##<username> CONTAINER=ALL;

         GRANT SELECT ON dba_objects TO C##<username> CONTAINER=ALL;
         GRANT SELECT ON dba_extents TO C##<username> CONTAINER=ALL;

         GRANT EXECUTE ON SYS.DBMS_LOGMNR TO C##<username> CONTAINER=ALL;
         GRANT SELECT ON SYSTEM.LOGMNR_COL$ TO C##<username> CONTAINER=ALL;
         GRANT SELECT ON SYSTEM.LOGMNR_OBJ$ TO C##<username> CONTAINER=ALL;
         GRANT SELECT ON SYSTEM.LOGMNR_USER$ TO C##<username> CONTAINER=ALL;
         GRANT SELECT ON SYSTEM.LOGMNR_UID$ TO C##<username> CONTAINER=ALL;
         ```

{% endlist %}

### PostgreSQL source {#source-pg}

{% note info %}

When performing a transfer from PostgreSQL to a target of any type, objects of the [large object](https://www.postgresql.org/docs/current/largeobjects.html) type will not get transferred.

When transferring data of the `TIMESTAMP WITHOUT TIME ZONE` type, the time zone set in the `timezone` parameter of the PostgreSQL source database applies.

{% cut "Example" %}

The column with the `TIMESTAMP WITHOUT TIME ZONE` data type contains the `1970-01-01 00:00:00` value. The way the value will be transferred depends on the `timezone` parameter in the database:

* If the parameter is set to `Etc/UTC`, the time will be transferred as `1970-01-01 00:00:00+00`.
* If the parameter is set to `Europe/Moscow`, the time will be transferred as `1970-01-01 00:00:00+03`.

{% endcut %}

Data stored in a `MATERIALIZED VIEW` is not transferred. To transfer `MATERIALIZED VIEW` data, create an ordinary `VIEW` that refers to the `MATERIALIZED VIEW` to be transferred.

If the definition of the `VIEW` to be transferred contains an invocation of the `VOLATILE` [function](https://www.postgresql.org/docs/current/xfunc-volatility.html), the transfer reads data from this `VIEW` with the `READ UNCOMMITTED` isolation level. No consistency between the `VIEW` data and the data of other objects being transferred is guaranteed. Reading data from a `MATERIALIZED VIEW` in the `VIEW` definition are equivalent to invoking the `VOLATILE` function.

Large objects in the [TOAST storage system](https://www.postgresql.org/docs/12/storage-toast.html) and those of the [bytea](https://www.postgresql.org/docs/12/datatype-binary.html) type get transferred without restrictions.

{% endnote %}

{% list tabs %}

- Managed Service for PostgreSQL

    
    1. Configure the user account the transfer will use to connect to the source:
        
        1. [Create a user](../../managed-postgresql/operations/cluster-users.md#adduser).
        
        1. For the _**Replication**_ and _**Snapshot and increment**_ transfer types, [assign the `mdb_replication` role](../../managed-postgresql/operations/grant.md#grant-role) to this user.
        
        1. [Connect to the database](../../managed-postgresql/operations/connect/index.md) you want to migrate as the database owner and [configure privileges](../../managed-postgresql/operations/grant.md#grant-privilege):

            * Grant the `SELECT` privilege for all the database tables within the transfer:

              ```sql
              GRANT SELECT ON ALL TABLES IN SCHEMA <schema_name> TO <DB_owner_name>;
              ```

            * Grant the `SELECT` privilege for all the database sequences within the transfer:

              ```sql
              GRANT SELECT ON ALL SEQUENCES IN SCHEMA <schema_name> TO <DB_owner_name>;
              ```

            * Grant the `USAGE` privilege for all schemas storing these tables and sequences:

              ```sql
              GRANT USAGE ON SCHEMA <schema_name> TO <DB_owner_name>;
              ```

            * Grant `ALL PRIVILEGES` (`CREATE` and `USAGE`) for the service table (`__consumer_keeper` and `__data_transfer_mole_finder`) schema specified by the [endpoint parameter](endpoint/source/postgresql.md#additional-settings) if the endpoint is going to be used for the _**Replication**_ or _**Snapshot and increment**_ transfer types.

              ```sql
              GRANT ALL PRIVILEGES ON SCHEMA <schema_name> TO <DB_owner_name>;
              ```

    1. Configure the [number of user connections](../concepts/work-with-endpoints.md#postgresql-connection-limit) to the database.

    1. If the replication source is a cluster, configure it as follows:

        * [Enable](../../managed-postgresql/operations/extensions/cluster-extensions.md) the `pg_tm_aux` extension for it. This will allow replication to continue even after changing the master host. In some cases, a transfer may end in an error after you replace a master in your cluster. For more information, see [Troubleshooting](../troubleshooting/index.md#master-change).

        * In the cluster, specify the `Wal keep size` [setting](../../managed-postgresql/concepts/settings-list.md#setting-wal-keep-size) value. When you change the master host, [WAL](https://www.postgresql.org/docs/current/wal-intro.html) should have enough records on the new master to resume replication from the same place. If there are not enough records, the transfer may fail with an [error](endpoint/source/postgresql.md#no-wal-story). As the minimum `Wal keep size`, use the average value from the **Source Buffer Size** chart in the [Data Transfer monitoring dashboard](monitoring.md). If disk space permits, provide a generous safety margin.

    1. To transfer tables without primary keys for the _**Replication**_ and _**Snapshot and increment**_ transfer types, you must add the `REPLICA IDENTITY`:
       
       * Do not transfer tables without primary keys. For this purpose, add them to the list of excluded tables in [source endpoint settings](endpoint/source/postgresql.md#additional-settings.md).
       * Add the replica ID to tables without `primary keys`:
          * For tables with an index, set `REPLICA IDENTITY` by `unique key`:
       
             ```sql
             ALTER TABLE MY_TBL REPLICA IDENTITY USING INDEX MY_IDX;
             ```
       
          * For tables without an index, change `REPLICA IDENTITY`:
       
             ```sql
             ALTER TABLE MY_TBL REPLICA IDENTITY FULL;
             ```
             In this case, Data Transfer will treat such tables as tables where the primary key is a composite key that includes all columns of the table.
       
       If there are no primary keys in a table, logical replication [will not include any changes in the rows](https://www.postgresql.org/docs/current/logical-replication-publication.html) (`UPDATE` or `DELETE`).
       
       * If, while trying to transfer tables from PostgreSQL to PostgreSQL, you do not exclude a table without primary keys on the transfer source, the following error message will be returned:
       
          ```text
           failed to run (abstract1 source): Cannot parse logical replication message: failed to reload schema: primary key check failed: Tables: n / N check failed: "public"."MY_TBL": no key columns found
          ```
       
       * If, while trying to transfer tables from PostgreSQL to a different database, you add a table without primary keys, the following error message will be returned:
       
          ```text
          failed to run (abstract1 source): Cannot parse logical replication message: failed to reload schema: primary key check failed: Tables: n / N check failed:
          "public"."MY_TBL": no key columns found
          ```
    
    1. Disable the transfer of external keys when creating a source endpoint. Recreate them once the transfer is completed.

    1. If a database contains tables with generated columns, such tables will not be migrated and the transfer will end with an error. For more information, see [Troubleshooting](../troubleshooting/index.md#generated-columns). To make sure the transfer is running properly when migrating a database with such tables, add them to the list of excluded tables in the [source endpoint settings](endpoint/source/postgresql.md#additional-settings.md).

    1. Find and terminate DDL queries that are running for too long. To do this, make a selection from the PostgreSQL `pg_stat_activity` system table:
    
        ```sql
        SELECT NOW() - query_start AS duration, query, state
        FROM pg_stat_activity
        WHERE state != 'idle' ORDER BY 1 DESC;
        ```
    
       You will see a list of queries running on the server. Look for the queries with a high `duration` value.
    
    1. Deactivate trigger transfer at the transfer initiation stage and reactivate it at the completion stage (for the _**Replication**_ and the _**Snapshot and increment**_ transfer types). For more information, see the [description of additional endpoint settings for the PostgreSQL source](endpoint/source/postgresql.md#additional-settings).
    
    1. To enable parallel data reads from the table by ranges, make sure its primary key is specified. Then specify the number of workers and threads in the [transfer parameters](transfer.md#create) under **Runtime environment**.
    
    1. Configure WAL monitoring. {#wal-setup-recommendation}

       _**Replication**_ and _**Snapshot and increment**_ transfers use [logical replication](https://www.postgresql.org/docs/current/logicaldecoding.html). To perform the replication, the transfer creates a replication slot where `slot_name` matches the transfer ID, which you can get by selecting the transfer in the list of your transfers. Your WAL may grow due to different reasons: a long-running transaction or a transfer issue. Therefore, we recommend you to configure WAL monitoring on the source side.

        1. To monitor storage or disk space usage, [use monitoring tools to set up an alert](../../managed-postgresql/operations/monitoring.md#monitoring-hosts) (see the `disk.used_bytes` description).
        
        1. Specify the maximum WAL size for replication in the `Max slot wal keep size` [setting](../../managed-postgresql/concepts/settings-list.md#setting-max-slot-wal-keep-size). The value of this setting can be edited as of PostgreSQL version 13. To urgently disable a transfer to perform data reads, [delete the replication slot](../../managed-postgresql/operations/replication-slots.md#delete). 
        
           {% note warning %}
        
           If set to `-1` (unlimited size), you will not be able to delete WAL files due to open logical replication slots the information is not read from. As a result, the WAL files will take up the entire disk space and you will not be able to connect to the cluster.
        
           {% endnote %}
        
        1. [Set up an alert](../../managed-postgresql/operations/monitoring.md) with the Yandex Monitoring tools for the metric used for `Total size of WAL files`. Make sure the threshold values are less than those specified for the `disk.used_bytes` metric because, apart from the data, the disk stores temporary files, the WAL, and other types of data. You can monitor the current slot size by running this DB query with the correct `slot_name`, which matches the transfer ID:
        
            ```sql
            SELECT slot_name, pg_size_pretty(pg_current_wal_lsn() - restart_lsn), active_pid, catalog_xmin, restart_lsn, confirmed_flush_lsn
            FROM pg_replication_slots WHERE slot_name = '<transfer_ID>'
            ```

- PostgreSQL
    
    1. If not planning to use [Cloud Interconnect](../../interconnect/concepts/index.md) or [VPN](https://en.wikipedia.org/wiki/Virtual_private_network) for connections to an external cluster, make such cluster accessible from the Internet from [IP addresses used by Data Transfer](../../overview/concepts/public-ips.md#virtual-private-cloud).
       
       For details on linking your network up with external resources, see [this concept](../concepts/network.md#source-external).
    
    1. Configure the user account the transfer will use to connect to the source:

        1. Create a new user:
            
            * For the _**Snapshot**_ transfer type, use this command to create a user:
            
                ```sql
                CREATE ROLE <username> LOGIN ENCRYPTED PASSWORD '<password>';
                ```
            
            * For the _**Replication**_ and _**Snapshot and increment**_ transfer types, create a user with the `REPLICATION` privilege by running this command:
            
                ```sql
                CREATE ROLE <username> WITH REPLICATION LOGIN ENCRYPTED PASSWORD '<password>';
                ```
        
        1. Grant the new user the `SELECT` privilege for all the database tables within the transfer:
        
            ```sql
            GRANT SELECT ON ALL TABLES IN SCHEMA <schema_name> TO <username>;
            ```
        
        1. Grant the new user a privilege for the schemas of the transferred DB:

            * Grant the `USAGE` privilege for the _**Snapshot**_ transfer type:
        
                ```sql
                GRANT USAGE ON SCHEMA <schema_name> TO <username>;
                ```

            * For the _**Replication**_ and _**Snapshot and increment**_ transfer types, grant the `CREATE` and `USAGE` (`ALL PRIVILEGES`) privileges required to create [service tables](endpoint/source/postgresql.md#additional-settings):

                ```sql
                GRANT ALL PRIVILEGES ON SCHEMA <schema_name> TO <username>;
                ```

        1. Grant the new user the `SELECT` privilege for all the database sequences within the transfer:

            ```sql
            GRANT SELECT ON ALL SEQUENCES IN SCHEMA <schema_name> TO <username>;
            ```

        1. Grant the new user the `CONNECT` privilege if the source cluster's default settings do not allow connections for new users:

            ```sql
            GRANT CONNECT ON DATABASE <database_name> TO <username>;
            ```

    1. Configure PostgreSQL:
    
        1. Make changes to the source cluster configuration and authentication settings. To do this, edit the `postgresql.conf` and `pg_hba.conf` files. On Debian and Ubuntu distributions, these files reside in the `/etc/postgresql/<PostgreSQL_version>/main/` directory by default:
        
            1. Set the maximum number of user connections. To do this, update the `max_connections` setting in `postgresql.conf`:
        
                ```ini
                max_connections = <number_of_connections>
                ```
                
                Where `<number_of_connections>` is the maximum number of connections. For more information on how to configure this setting, see [Specifics of working with endpoints](../concepts/work-with-endpoints.md#postgresql-connection-limit).
        
                You can check the current number of connections using the `pg_stat_activity` system table:
        
                ```sql
                SELECT count(*) FROM pg_stat_activity;
                ```
        
            1. Configure the [Write Ahead Log (WAL)](https://www.postgresql.org/docs/current/static/wal-intro.html) log level by setting [wal_level](https://www.postgresql.org/docs/current/runtime-config-wal.html#RUNTIME-CONFIG-WAL-SETTINGS) to `logical` in `postgresql.conf`:
        
                ```ini
                wal_level = logical
                ```
        
            1. Optionally, configure SSL to enable data encryption and compression. To turn SSL on, specify the following in `postgresql.conf`:
        
                ```ini
                ssl = on
                ```
            
            1. Enable cluster access by updating the `listen_addresses` [setting](https://www.postgresql.org/docs/current/runtime-config-connection.html#GUC-LISTEN-ADDRESSES) in `postgresql.conf`. For example, to make the source cluster accept connection requests from all IP addresses, specify the following:
        
                ```ini
                listen_addresses = '*'
                ```
        
            1. Configure authentication in the `pg_hba.conf` file:
        
                {% list tabs %}
        
                - SSL
        
                    ```txt
                    hostssl         all            all             <connection_IP_address>      md5
                    hostssl         replication    all             <connection_IP_address>      md5
                    ```
        
                - Without SSL
        
                    ```txt
                    host         all            all             <connection_IP_address>      md5
                    host         replication    all             <connection_IP_address>      md5
                    ```
        
                {% endlist %}
        
                Where `<connection_IP_address>` can be either an exact IP address or a range of IP addresses. For example, to allow access from the Yandex Cloud network, you can specify [all public IP addresses](../../overview/concepts/public-ips.md) in Yandex Cloud.
        1. If a firewall is enabled in the source cluster, allow incoming connections from the relevant addresses.
        
        1. Restart PostgreSQL for the settings to apply:
        
           ```bash
           sudo systemctl restart postgresql
           ```
        
        1. Check the PostgreSQL status after restart:
        
           ```bash
           sudo systemctl status postgresql
           ```

    1. Install and enable the [wal2json](https://github.com/eulerto/wal2json) extension.

        * Linux
            
            1. Add the [PostgreSQL official repository](https://www.postgresql.org/download/) for your distribution.
            1. Update the list of available packages and install `wal2json` for your PostgreSQL version.
        
        * Windows 10, 11
            
            1. If you do not have Microsoft Visual Studio installed yet, download and install it. To build the wal2json extension, the [Community Edition](https://visualstudio.microsoft.com/vs/community/) is sufficient. During installation, select the following components:
                
                * MSBuild
                * MSVC v141 x86/x64 build tools
                * C++\CLI support for v141 build tools
                * MSVC v141 - VS 2017 C++ x64\x86 build tools
                * MSVC v141 - VS 2017 C++ x64\x86 Spectre-mitigated libs
                * The latest version of the Windows SDK for the active OS version
                * Other dependencies that are installed automatically for selected components

                Take note of the version number of the installed Windows SDK. You will need it while setting the wal2json build parameters.
            
            1. Download the wal2json source code from the [project page](https://github.com/eulerto/wal2json/releases).
            1. Unpack the archive with the source code to the `C:\wal2json\` folder.
            1. Navigate to `C:\wal2json`.
            1. Within one PowerShell session, make changes to the `wal2json.vcxproj` file as follows:
                
                * Replace the `C:\postgres\pg103` lines with the path to the folder housing your installed PostgreSQL version, for example:
                
                    ```powershell
                    (Get-Content .\wal2json.vcxproj).replace('C:\postgres\pg103', 'C:\PostgreSQL\14') | `
                        Set-Content .\wal2json.vcxproj
                    ```
                
                * Replace the `/MP` build parameter with `/MT`, for example:
                
                    ```powershell
                    (Get-Content .\wal2json.vcxproj).replace('/MP', '/MT') | Set-Content .\wal2json.vcxproj
                    ```
                
                * Specify the version number of the installed Windows SDK in `<WindowsTargetPlatformVersion>`:
                
                    ```powershell
                    (Get-Content .\wal2json.vcxproj).replace('<WindowsTargetPlatformVersion>8.1', '<WindowsTargetPlatformVersion><installed_Windows_SDK_version>') | `
                        Set-Content .\wal2json.vcxproj
                    ```
                
                1. Enter the value of the extension variable required for building wal2json. For example, for Visual Studio Community Edition 2022:
                
                    ```powershell
                    $VCTargetsPath='C:\Program Files\Microsoft Visual Studio\2022\Comminuty\MSBuild\Microsoft\VC\v150'
                    ```
                
                1. Build the project:
                
                    ```powershell
                    & 'C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Current\Bin\MSBuild.exe' /p:Configuration=Release /p:Platform=x64
                    ```
                
                1. Copy the `wal2json.dll` file from the `build/release` folder to the `lib` folder of your PostgreSQL version.
    
    1. If the replication source is a cluster, install and enable the [pg_tm_aux](https://github.com/x4m/pg_tm_aux) extension on its hosts. This will allow replication to continue even after changing the master host. In some cases, a transfer may end in an error after you replace a master in your cluster. For more information, see [Troubleshooting](../troubleshooting/index.md#master-change).
    
    1. To transfer tables without primary keys for the _**Replication**_ and _**Snapshot and increment**_ transfer types, you must add the `REPLICA IDENTITY`:
       
       * Do not transfer tables without primary keys. For this purpose, add them to the list of excluded tables in [source endpoint settings](endpoint/source/postgresql.md#additional-settings.md).
       * Add the replica ID to tables without `primary keys`:
          * For tables with an index, set `REPLICA IDENTITY` by `unique key`:
       
             ```sql
             ALTER TABLE MY_TBL REPLICA IDENTITY USING INDEX MY_IDX;
             ```
       
          * For tables without an index, change `REPLICA IDENTITY`:
       
             ```sql
             ALTER TABLE MY_TBL REPLICA IDENTITY FULL;
             ```
             In this case, Data Transfer will treat such tables as tables where the primary key is a composite key that includes all columns of the table.
       
       If there are no primary keys in a table, logical replication [will not include any changes in the rows](https://www.postgresql.org/docs/current/logical-replication-publication.html) (`UPDATE` or `DELETE`).
       
       * If, while trying to transfer tables from PostgreSQL to PostgreSQL, you do not exclude a table without primary keys on the transfer source, the following error message will be returned:
       
          ```text
           failed to run (abstract1 source): Cannot parse logical replication message: failed to reload schema: primary key check failed: Tables: n / N check failed: "public"."MY_TBL": no key columns found
          ```
       
       * If, while trying to transfer tables from PostgreSQL to a different database, you add a table without primary keys, the following error message will be returned:
       
          ```text
          failed to run (abstract1 source): Cannot parse logical replication message: failed to reload schema: primary key check failed: Tables: n / N check failed:
          "public"."MY_TBL": no key columns found
          ```
    
    1. Disable the transfer of external keys when creating a source endpoint. Recreate them once the transfer is completed.

    1. If a database contains tables with generated columns, such tables will not be migrated and the transfer will end with an error. For more information, see [Troubleshooting](../troubleshooting/index.md#generated-columns). To make sure the transfer is running properly when migrating a database with such tables, add them to the list of excluded tables in the [source endpoint settings](endpoint/source/postgresql.md#additional-settings.md).

    1. Find and terminate DDL queries that are running for too long. To do this, make a selection from the PostgreSQL `pg_stat_activity` system table:
    
        ```sql
        SELECT NOW() - query_start AS duration, query, state
        FROM pg_stat_activity
        WHERE state != 'idle' ORDER BY 1 DESC;
        ```
    
       You will see a list of queries running on the server. Look for the queries with a high `duration` value.
    
    1. Deactivate trigger transfer at the transfer initiation stage and reactivate it at the completion stage (for the _**Replication**_ and the _**Snapshot and increment**_ transfer types). For more information, see the [description of additional endpoint settings for the PostgreSQL source](endpoint/source/postgresql.md#additional-settings).
    
    1. To enable parallel data reads from the table by ranges, make sure its primary key is specified. Then specify the number of [workers](../concepts/index.md#worker) and threads in the [transfer parameters](transfer.md#create) under **Runtime environment**.
    
    1. If replication via [Patroni](https://github.com/zalando/patroni) is configured on the source, add an [ignore_slots](https://patroni.readthedocs.io/en/latest/SETTINGS.html?highlight=ignore_slots#dynamic-configuration-settings) section to the source configuration:
    
       ```yaml
       ignore_slots:
         - database: <database>
           name: <replication_slot>
           plugin: wal2json
           type: logical
       ```
    
       Where:
        
        * `database`: Name of the database the transfer is configured for.
        * `name`: Replication slot name.

       The database and the replication slot names must match the values specified in the [source endpoint settings](endpoint/source/postgresql.md). By default, the `replication slot name` is the same as the `transfer ID`.

       Otherwise, the start of the replication phase will fail:
    
       ```
       Warn(Termination): unable to create new pg source: Replication slotID <replication_slot_name> does not exist.
       ```
    
    1. Configure WAL monitoring. _**Replication**_ and _**Snapshot and increment**_ transfers use [logical replication](https://www.postgresql.org/docs/current/logicaldecoding.html). To perform the replication, the transfer creates a replication slot where `slot_name` matches the transfer ID, which you can get by selecting the transfer in the list of your transfers. Your WAL may grow due to different reasons: a long-running transaction or a transfer issue. Therefore, we recommend you to configure WAL monitoring on the source side.
        
        1. Set up alerts as described in the [disk usage recommendations](https://www.postgresql.org/docs/current/diskusage.html).
        
        1. [Set the maximum WAL size](https://www.postgresql.org/docs/current/runtime-config-replication.html#GUC-MAX-SLOT-WAL-KEEP-SIZE). This feature is available starting with PostgreSQL version 13.
        
        1. You can monitor the current slot size by running this DB query with the correct `slot_name`, which matches the transfer ID:
        
            ```sql
            SELECT slot_name, pg_size_pretty(pg_current_wal_lsn() - restart_lsn), active_pid, catalog_xmin, restart_lsn, confirmed_flush_lsn
            FROM pg_replication_slots WHERE slot_name = '<transfer_ID>'
            ```    

{% endlist %}

{% note info %}

For things to note about data transfer from PostgreSQL to ClickHouse® using _**Replication**_ and _**Snapshot and increment**_ transfers, see [Asynchronously replicating data from PostgreSQL to ClickHouse®](../tutorials/rdbms-to-clickhouse.md).

{% endnote %}



### Yandex Data Streams source {#source-yds}

1. [Create a service account](../../iam/operations/sa/create.md) with the `yds.editor` role.
1. [Create a data stream](../../data-streams/operations/manage-streams.md#create-data-stream).
1. (Optional) [Create a processing function](../../functions/operations/function/function-create.md).

   {% cut "Processing function example" %}

   ```javascript
   const yc = require("yandex-cloud");
   const { Parser } = require("@robojones/nginx-log-parser");
   module.exports.handler = async function (event, context) {
       const schema =
           '$remote_addr - $remote_user [$time_local] "$request" $status $bytes_sent "$http_referer" "$http_user_agent"';
       const parser = new Parser(schema);
       return {
           Records: event.Records.map((record) => {
               const decodedData = new Buffer(record.kinesis.data, "base64")
                   .toString("ascii")
                   .trim();
               try {
                   const result = parser.parseLine(decodedData);
                   if (result.request == "") {
                       // empty request - drop message
                       return {
                           eventID: record.eventID,
                           invokeIdentityArn: record.invokeIdentityArn,
                           eventVersion: record.eventVersion,
                           eventName: record.eventName,
                           eventSourceARN: record.eventSourceARN,
                           result: "Dropped"
                       };
                   }
                   return {
                       // successfully parsed message
                       eventID: record.eventID,
                       invokeIdentityArn: record.invokeIdentityArn,
                       eventVersion: record.eventVersion,
                       eventName: record.eventName,
                       eventSourceARN: record.eventSourceARN,
                       kinesis: {
                           data: new Buffer(JSON.stringify(result)).toString(
                               "base64"
                           ),
                       },
                       result: "Ok"
                   };
               } catch (err) {
                   // error - fail message
                   return {
                       eventID: record.eventID,
                       invokeIdentityArn: record.invokeIdentityArn,
                       eventVersion: record.eventVersion,
                       eventName: record.eventName,
                       eventSourceARN: record.eventSourceARN,
                       result: "ProcessingFailed",
                   };
               }
           })
       };
   };
   ```

   {% endcut %}

1. (Optional) Prepare a data schema file in JSON format.

   Sample file with a data schema:

   ```json
   [
       {
           "name": "<field_name>",
           "type": "<type>"
       },
       ...
       {
           "name": "<field_name>",
           "type": "<type>"
       }
   ]
   ```

   Supported types:

   * `any`
   * `boolean`
   * `datetime`
   * `double`
   * `int8`
   * `int16`
   * `int32`
   * `int64`
   * `string`
   * `uint8`
   * `uint16`
   * `uint32`
   * `uint64`
   * `utf8`


### Yandex Managed Service for YDB source {#source-ydb}

If you selected Dedicated database mode, [create](../../vpc/operations/security-group-create.md) and [configure](../../ydb/operations/connection.md#configuring-security-groups) a security group in the network hosting the DB.


## Preparing a target {#target}

### ClickHouse® target {#target-ch}

{% list tabs %}

- Managed Service for ClickHouse®

    1. [Create a target database](../../managed-clickhouse/operations/databases.md#add-db).

       If you need to transfer multiple databases, create a separate transfer for each one of them.

    1. [Create a user](../../managed-clickhouse/operations/cluster-users.md#adduser) with access to the target database.

       Once started, the transfer will connect to the target on behalf of this user.

    1. If [user management via SQL](../../managed-clickhouse/operations/cluster-users.md#sql-user-management) is enabled in the cluster, grant the new user the following permissions:

        ```sql
        GRANT CLUSTER ON *.* TO <username>
        GRANT SELECT, INSERT, ALTER DELETE, CREATE TABLE, CREATE VIEW, DROP TABLE, TRUNCATE, dictGet ON <DB_name>.* TO <username>
        GRANT SELECT(macro, substitution) ON system.macros TO <username>
        ``` 

       If user management via SQL is disabled, [permissions are assigned](../../managed-clickhouse/operations/cluster-users.md) via the management console and CLI.

    1. [Create a security group](../../vpc/operations/security-group-create.md) and [configure it](../../managed-clickhouse/operations/connect/index.md#configuring-security-groups).

    1. Assign the created security group to the Managed Service for ClickHouse® cluster.

- ClickHouse®

    1. If not planning to use [Cloud Interconnect](../../interconnect/concepts/index.md) or [VPN](https://en.wikipedia.org/wiki/Virtual_private_network) for connections to an external cluster, make such cluster accessible from the Internet from [IP addresses used by Data Transfer](../../overview/concepts/public-ips.md#virtual-private-cloud).
       
       For details on linking your network up with external resources, see [this concept](../concepts/network.md#source-external).

    1. Create a target database. Its name must be the same as the source database name. If you need to transfer multiple databases, create a separate transfer for each one of them.

    1. Create a user with access to the target database.

       Once started, the transfer will connect to the target on behalf of this user.

    1. Grant the new user the following permissions:

        ```sql
        GRANT CLUSTER ON *.* TO <username>
        GRANT SELECT, INSERT, ALTER DELETE, CREATE TABLE, CREATE VIEW, DROP TABLE, TRUNCATE, dictGet ON <DB_name>.* TO <username>
        GRANT SELECT(macro, substitution) ON system.macros TO <username>
        ```

{% endlist %}

### Greenplum® target {#target-gp}


{% list tabs %}

- Yandex MPP Analytics for PostgreSQL
    
    1. Disable the following settings on the target:
        
        * Integrity checks for foreign keys
        * Triggers
        * Other constraints

       {% note warning %}

       Do not reactivate these settings before the transfer is complete. This will ensure data integrity with respect to foreign keys.

       {% endnote %}
    
    1. Create a user:
    
        ```sql
        CREATE ROLE <username> LOGIN ENCRYPTED PASSWORD '<password>';
        ```
    
    1. Grant the user all privileges for the database, schemas, and tables to be transferred:
    
        ```sql
        GRANT ALL PRIVILEGES ON DATABASE <database_name> TO <username>;
        ```
    
       If the database is not empty, the user must be its owner:
    
        ```sql
        ALTER DATABASE <database_name> OWNER TO <username>;
        ```
    
       Once started, the transfer will connect to the target on behalf of this user.

- Greenplum®
    
    1. If not planning to use [Cloud Interconnect](../../interconnect/concepts/index.md) or [VPN](https://en.wikipedia.org/wiki/Virtual_private_network) for connections to an external cluster, make such cluster accessible from the Internet from [IP addresses used by Data Transfer](../../overview/concepts/public-ips.md#virtual-private-cloud).
       
       For details on linking your network up with external resources, see [this concept](../concepts/network.md#source-external).
    
    1. Disable the following settings on the target:
        
        * Integrity checks for foreign keys
        * Triggers
        * Other constraints

       {% note warning %}

       Do not reactivate these settings before the transfer is complete. This will ensure data integrity with respect to foreign keys.

       {% endnote %}
    
    1. Create a user:
    
        ```sql
        CREATE ROLE <username> LOGIN ENCRYPTED PASSWORD '<password>';
        ```
    
    1. Grant the user all privileges for the database, schemas, and tables to be transferred:
    
        ```sql
        GRANT ALL PRIVILEGES ON DATABASE <database_name> TO <username>;
        ```
    
       If the database is not empty, the user must be its owner:
    
        ```sql
        ALTER DATABASE <database_name> OWNER TO <username>;
        ```
    
       Once started, the transfer will connect to the target on behalf of this user.

{% endlist %}


### MongoDB target {#target-mg}


{% list tabs %}

- Yandex StoreDoc (Managed Service for MongoDB)
    
    1. [Create a database](../../storedoc/operations/databases.md#add-db).
    1. [Create a user](../../storedoc/operations/cluster-users.md#adduser) with the [`readWrite`](../../storedoc/concepts/users-and-roles.md#readWrite) role for the new database.
    1. To shard the migrated collections in the Yandex StoreDoc target cluster:
        1. Use [this guide](../../storedoc/tutorials/sharding.md) to create and configure empty sharded collections in the target database.
        
           Data Transfer does not automatically shard the migrated collections. Sharding large collections may take a long time and slow down the transfer.
        
        1. If sharding uses any key other than `_id` (default), [assign](../../storedoc/operations/cluster-users.md#updateuser) the `mdbShardingManager` role to the user.
        
        1. When [creating a target endpoint](endpoint/target/mongodb.md), select `DISABLED` or `TRUNCATE` as your cleanup policy.
        
           Selecting the `DROP` policy will result in the service deleting all the data from the target database, including sharded collections, and replacing them with new unsharded ones when a transfer is activated.


- MongoDB
    
    1. If not planning to use [Cloud Interconnect](../../interconnect/concepts/index.md) or [VPN](https://en.wikipedia.org/wiki/Virtual_private_network) for connections to an external cluster, make such cluster accessible from the Internet from [IP addresses used by Data Transfer](../../overview/concepts/public-ips.md#virtual-private-cloud).
       
       For details on linking your network up with external resources, see [this concept](../concepts/network.md#source-external).
    
    1. Make sure the MongoDB version on the target is not lower than that on the source.
    
    1. Make sure the MongoDB cluster is configured so that it returns correctly resolving IP addresses or FQDNs (fully qualified domain names) in response to requests.
    
    1. Configure the target cluster to allow connections from the internet:
        
        1. In the configuration file, change `net.bindIp` from `127.0.0.1` to `0.0.0.0`:
        
            ```yaml
            # network interfaces
            net:
              port: 27017
              bindIp: 0.0.0.0
            ```
        
        1. Restart `mongod`:
        
            ```bash
            sudo systemctl restart mongod.service
            ```
    
    1. If the target cluster does not use replication, enable it:
        
        1. Add the replication settings to the `/etc/mongod.conf` configuration file:
        
            ```yaml
            replication:
              replSetName: <replica_set_name>
            ```
        
        1. Restart `mongod`:
        
            ```bash
            sudo systemctl restart mongod.service
            ```
        
        1. Connect to MongoDB and initialize the replica set with this command:
        
            ```javascript
            rs.initiate({
                _id: "<replica_set_name>",
                members: [{
                    _id: 0,
                    host: "<IP_address_Yandex_StoreDoc_listens_on>:<port>"
                }]
            });
            ```
    
    1. Connect to the cluster and create a target database:
    
        ```javascript
        use <database_name>
        ```
    
    1. Create a user with the `readWrite` role for the target database:
    
        ```javascript
        use admin;
        db.createUser({
            user: "<username>",
            pwd: "<password>",
            mechanisms: ["SCRAM-SHA-1"],
            roles: [
                {
                    db: "<target_database_name>",
                    role: "readWrite"
                }
            ]
        });
        ```
    
       Once started, the transfer will connect to the target on behalf of this user.
    
    1. To shard the migrated collections in the target cluster:
        
        1. Set up a database and populate it with empty collections with the same names as those in the source.
        
           Data Transfer does not automatically shard the migrated collections. Sharding large collections may take a long time and slow down the transfer.
        
        1. Enable target database sharding:
        
            ```javascript
            sh.enableSharding("<target_database_name>")
            ```
        
        1. Shard every collection based on its namespace:
        
            ```javascript
            sh.shardCollection("<target_database_name>.<collection_name>", { <field_name>: <1|"hashed">, ... });
            ```
        
        
        1. To make sure that sharding is set up and enabled, get a list of available shards:
        
            ```javascript
            sh.status()
            ```
        
        1. If sharding uses any key other than `_id` (default), assign the `clusterManager` system role to the user Data Transfer will use for connection to the target cluster:
        
            ```javascript
            use admin;
            db.grantRolesToUser("<username>", ["clusterManager"]);
            ```
        
        1. When [creating a target endpoint](endpoint/target/mongodb.md), select `DISABLED` or `TRUNCATE` as your cleanup policy.
        
           Selecting the `DROP` policy will result in the service deleting all the data from the target database, including sharded collections, and replacing them with new unsharded ones when a transfer is activated.


{% endlist %}

### MySQL® target {#target-my}

{% list tabs %}

- Managed Service for MySQL®

   1. Make sure that the MySQL® major version on the target is not lower than that on the source.

   1. [Set an SQL Mode](../../managed-mysql/operations/update.md#change-mysql-config) matching the source.

   1. [Create a user](../../managed-mysql/operations/cluster-users.md#adduser) for connecting to the source.

      1. [Assign the user](../../managed-mysql/operations/grant.md#grant-role) the `ALL_PRIVILEGES` role for the source database.

- MySQL®

   1. If not planning to use [Cloud Interconnect](../../interconnect/concepts/index.md) or [VPN](https://en.wikipedia.org/wiki/Virtual_private_network) for connections to an external cluster, make such cluster accessible from the Internet from [IP addresses used by Data Transfer](../../overview/concepts/public-ips.md#virtual-private-cloud).
      
      For details on linking your network up with external resources, see [this concept](../concepts/network.md#source-external).

   1. Make sure that the MySQL® major version on the target is not lower than that on the source.

   1. Make sure the target uses the MyISAM or InnoDB low-level storage subsystem.

   1. [Set an SQL Mode](https://dev.mysql.com/doc/refman/8.0/en/sql-mode.html#sql-mode-setting) matching the source.

   1. Create a user to connect to the target and grant them the required privileges:

      ```sql
      CREATE USER '<username>'@'%' IDENTIFIED BY '<password>';
      GRANT ALL PRIVILEGES ON <database_name>.* TO '<username>'@'%';
      ```

{% endlist %}

### Yandex Object Storage target {#target-storage}

1. [Create a bucket](../../storage/operations/buckets/create.md) with a configuration you need.
1. [Create a service account](../../iam/operations/sa/create.md) with the `storage.uploader` role.

### Elasticsearch target {#target-es}


If not planning to use [Cloud Interconnect](../../interconnect/concepts/index.md) or [VPN](https://en.wikipedia.org/wiki/Virtual_private_network) for connections to an external cluster, make such cluster accessible from the Internet from [IP addresses used by Data Transfer](../../overview/concepts/public-ips.md#virtual-private-cloud).

For details on linking your network up with external resources, see [this concept](../concepts/network.md#source-external).

* Make sure the number of columns in the source does not exceed the maximum number of fields in Elasticsearch indexes. The maximum number of fields is provided in the `index.mapping.total_fields.limit` [parameter](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-settings-limit.html#mapping-settings-limit); the default value is `1000`.

   To increase the parameter value, [set up a template](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-put-template.html#indices-put-template) that makes the maximum number of fields in new indexes equal to the specified value.

   {% cut "Sample template setup request" %}

   ```bash
   curl \
       --user <Elasticsearch_username>:<password> \
       --header 'Content-Type: application/json' \
       --request PUT "https://<Elasticsearch_cluster_FQDN>:9200/_template/index_defaults" \
       --data '
           {
               "index_patterns": "cdc*",
               "settings": {
                   "index": {
                       "mapping": {
                           "total_fields": {
                               "limit": "2000"
                           }
                       }
                   }
               }
           }'
   ```

   With this template setup, all new indexes with the `cdc*` mask may contain up to `2000` fields.

   {% endcut %}

   You can also set up templates using the [Kibana interface](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-mgmt.html#manage-index-templates).

   To check the current `index.mapping.total_fields.limit` parameter value, use the [Kibana interface](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-mgmt.html#view-edit-indices) or execute the following request:

   ```bash
   curl \
       --user <Elasticsearch_username>:<password> \
       --header 'Content-Type: application/json' \
       --request GET 'https://<Elasticsearch_cluster_FQDN>:9200/<index_name>/_settings/*total_fields.limit?include_defaults=true'
   ```

* By default, when transferring data to a single index, only one host is used. To distribute the load across hosts when transferring large amounts of data, [set up a template](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-put-template.html#indices-put-template) to split new indexes into shards in advance.

   {% cut "Sample template setup request" %}

   ```bash
   curl \
       --user <Elasticsearch_username>:<password> \
       --header 'Content-Type: application/json' \
       --request PUT 'https://<Elasticsearch_cluster_FQDN>:9200/_template/index_defaults' \
       --data '
           {
               "index_patterns": "cdc*",
               "settings" : {
                   "index" : {
                       "number_of_shards" : 15,
                       "number_of_replicas" : 1
                   }
               }
           }'
   ```

   {% endcut %}

   With this template setup, all new indexes with the `cdc*` mask will be split into `15` shards.

   You can also set up templates using the [Kibana interface](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-mgmt.html#manage-index-templates).

### OpenSearch target {#target-os}

{% list tabs %}

- Managed Service for OpenSearch

  * Make sure the number of columns in the source does not exceed the maximum number of fields in OpenSearch indexes. The maximum number of fields is set in the `index.mapping.total_fields.limit` parameter. Its default value is `1,000`.
  
     {% note warning %}
  
     Exceeding the limit will result in the `Limit of total fields [1000] has been exceeded` error and the transfer will be stopped.
  
     {% endnote %}
  
     To increase the parameter value, [set up a template](https://opensearch.org/docs/latest/im-plugin/index-templates/) that makes the maximum number of fields in new indexes equal to the specified value.
  
     {% cut "Sample template setup request" %}
  
     ```bash
     curl \
     --user <OpenSearch_username>:<password> \
     --header 'Content-Type: application/json' \
     --request PUT "https://<URL_of_OpenSearch_host_with_DATA_role>:9200/_template/index_defaults" \
     --data '
         {
             "index_patterns": "cdc*",
             "settings": {
                 "index": {
                     "mapping": {
                         "total_fields": {
                             "limit": "2000"
                         }
                     }
                 }
             }
         }'
     ```
  
     With this template setup, all new indexes with the `cdc*` mask will be able to contain up to `2,000` fields.
  
     {% endcut %}
  
     You can also set up templates using the [OpenSearch Dashboards interface](https://opensearch.org/docs/latest/dashboards/im-dashboards/component-templates/).
  
     To check the current `index.mapping.total_fields.limit` parameter value, run this request:
  
     ```bash
     curl \
         --user <OpenSearch_username>:<password> \
         --header 'Content-Type: application/json' \
         --request GET 'https://<URL_of_OpenSearch_host_with_DATA_role>:9200/<index_name>/_settings/*total_fields.limit?include_defaults=true'
     ```
  
  * By default, when transferring data to a single index, only one host is used. To distribute the load across hosts when transferring large amounts of data, [set up a template](https://opensearch.org/docs/latest/im-plugin/index-templates/) to split new indexes into shards in advance.
  
     {% cut "Sample template setup request" %}
  
     ```bash
     curl \
     --user <OpenSearch_username>:<password> \
     --header 'Content-Type: application/json' \
     --request PUT 'https://<URL_of_OpenSearch_host_with_DATA_role>:9200/_template/index_defaults' \
     --data '
         {
             "index_patterns": "cdc*",
             "settings" : {
                 "index" : {
                     "number_of_shards" : 15,
                     "number_of_replicas" : 1
                 }
             }
         }'
     ```
  
     {% endcut %}
  
     With this template setup, all new indexes with the `cdc*` mask will be split into `15` shards.
  
     You can also set up templates using the [OpenSearch Dashboards interface](https://opensearch.org/docs/latest/dashboards/im-dashboards/component-templates/).
  
  * To enhance data security and availability, set up a policy that will create a new index if at least one of the following conditions is met (recommended values):
  
     * Index is over 50 GB in size.
     * Index is over 30 days old.
  
     You can create and enable a policy using requests. For more on policies, see [this OpenSearch guide](https://opensearch.org/docs/latest/im-plugin/ism/policies/).
  
     {% cut "Example of a policy creation request" %}
  
     ```bash
     curl \
     --user <OpenSearch_username>:<password> \
     --header 'Content-Type: application/json' \
     --request PUT 'https://<address_of_OpenSearch_host_with_DATA_role>:9200/_plugins/_ism/policies/rollover_policy' \
     --data '
         {
             "policy": {
                 "description": "Example rollover policy",
                 "default_state": "rollover",
                 "schema_version": 1,
                 "states": [
                     {
                         "name": "rollover",
                         "actions": [
                             {
                                 "rollover": {
                                     "min_index_age": "30d",
                                     "min_primary_shard_size": "50gb"
                                 }
                             }
                         ],
                         "transitions": []
                     }
                 ],
                 "ism_template": {
                     "index_patterns": ["log*"],
                     "priority": 100
                 }
             }
         }'
     ```
  
     {% endcut %}
  
     {% cut "Example of a request to assign an alias to a policy" %}
  
     ```bash
     curl \
     --user <OpenSearch_username>:<password> \
     --header 'Content-Type: application/json' \
     --request PUT 'https://<address_of_OpenSearch_host_with_DATA_role>:9200/_index_template/ism_rollover' \
     --data '
         {
             "index_patterns": ["log*"],
             "template": {
                 "settings": {
                     "plugins.index_state_management.rollover_alias": "log"
                 }
             }
         }'
     ```
  
     {% endcut %}
  
     {% cut "Example of a request to create an index with a policy alias" %}
  
     ```bash
     curl \
     --user <OpenSearch_username>:<password> \
     --header 'Content-Type: application/json' \
     --request PUT 'https://<address_of_OpenSearch_host_with_DATA_role>:9200/log-000001' \
     --data '
         {
             "aliases": {
                 "log": {
                     "is_write_index": true
                 }
             }
         }'
     ```
  
     {% endcut %}
  
     {% cut "Example of a request to check if a policy is attached to an index" %}
  
     ```bash
     curl \
     --user <OpenSearch_username>:<password> \
     --header 'Content-Type: application/json' \
     --request GET 'https://<address_of_OpenSearch_host_with_DATA_role>:9200/_plugins/_ism/explain/log-000001?pretty'
     ```
  
     {% endcut %}

- OpenSearch
    
    * If not planning to use [Cloud Interconnect](../../interconnect/concepts/index.md) or [VPN](https://en.wikipedia.org/wiki/Virtual_private_network) for connections to an external cluster, make such cluster accessible from the Internet from [IP addresses used by Data Transfer](../../overview/concepts/public-ips.md#virtual-private-cloud).
      
      For details on linking your network up with external resources, see [this concept](../concepts/network.md#source-external).

  * Make sure the number of columns in the source does not exceed the maximum number of fields in OpenSearch indexes. The maximum number of fields is set in the `index.mapping.total_fields.limit` parameter. Its default value is `1,000`.
  
     {% note warning %}
  
     Exceeding the limit will result in the `Limit of total fields [1000] has been exceeded` error and the transfer will be stopped.
  
     {% endnote %}
  
     To increase the parameter value, [set up a template](https://opensearch.org/docs/latest/im-plugin/index-templates/) that makes the maximum number of fields in new indexes equal to the specified value.
  
     {% cut "Sample template setup request" %}
  
     ```bash
     curl \
     --user <OpenSearch_username>:<password> \
     --header 'Content-Type: application/json' \
     --request PUT "https://<URL_of_OpenSearch_host_with_DATA_role>:9200/_template/index_defaults" \
     --data '
         {
             "index_patterns": "cdc*",
             "settings": {
                 "index": {
                     "mapping": {
                         "total_fields": {
                             "limit": "2000"
                         }
                     }
                 }
             }
         }'
     ```
  
     With this template setup, all new indexes with the `cdc*` mask will be able to contain up to `2,000` fields.
  
     {% endcut %}
  
     You can also set up templates using the [OpenSearch Dashboards interface](https://opensearch.org/docs/latest/dashboards/im-dashboards/component-templates/).
  
     To check the current `index.mapping.total_fields.limit` parameter value, run this request:
  
     ```bash
     curl \
         --user <OpenSearch_username>:<password> \
         --header 'Content-Type: application/json' \
         --request GET 'https://<URL_of_OpenSearch_host_with_DATA_role>:9200/<index_name>/_settings/*total_fields.limit?include_defaults=true'
     ```
  
  * By default, when transferring data to a single index, only one host is used. To distribute the load across hosts when transferring large amounts of data, [set up a template](https://opensearch.org/docs/latest/im-plugin/index-templates/) to split new indexes into shards in advance.
  
     {% cut "Sample template setup request" %}
  
     ```bash
     curl \
     --user <OpenSearch_username>:<password> \
     --header 'Content-Type: application/json' \
     --request PUT 'https://<URL_of_OpenSearch_host_with_DATA_role>:9200/_template/index_defaults' \
     --data '
         {
             "index_patterns": "cdc*",
             "settings" : {
                 "index" : {
                     "number_of_shards" : 15,
                     "number_of_replicas" : 1
                 }
             }
         }'
     ```
  
     {% endcut %}
  
     With this template setup, all new indexes with the `cdc*` mask will be split into `15` shards.
  
     You can also set up templates using the [OpenSearch Dashboards interface](https://opensearch.org/docs/latest/dashboards/im-dashboards/component-templates/).
  
  * To enhance data security and availability, set up a policy that will create a new index if at least one of the following conditions is met (recommended values):
  
     * Index is over 50 GB in size.
     * Index is over 30 days old.
  
     You can create and enable a policy using requests. For more on policies, see [this OpenSearch guide](https://opensearch.org/docs/latest/im-plugin/ism/policies/).
  
     {% cut "Example of a policy creation request" %}
  
     ```bash
     curl \
     --user <OpenSearch_username>:<password> \
     --header 'Content-Type: application/json' \
     --request PUT 'https://<address_of_OpenSearch_host_with_DATA_role>:9200/_plugins/_ism/policies/rollover_policy' \
     --data '
         {
             "policy": {
                 "description": "Example rollover policy",
                 "default_state": "rollover",
                 "schema_version": 1,
                 "states": [
                     {
                         "name": "rollover",
                         "actions": [
                             {
                                 "rollover": {
                                     "min_index_age": "30d",
                                     "min_primary_shard_size": "50gb"
                                 }
                             }
                         ],
                         "transitions": []
                     }
                 ],
                 "ism_template": {
                     "index_patterns": ["log*"],
                     "priority": 100
                 }
             }
         }'
     ```
  
     {% endcut %}
  
     {% cut "Example of a request to assign an alias to a policy" %}
  
     ```bash
     curl \
     --user <OpenSearch_username>:<password> \
     --header 'Content-Type: application/json' \
     --request PUT 'https://<address_of_OpenSearch_host_with_DATA_role>:9200/_index_template/ism_rollover' \
     --data '
         {
             "index_patterns": ["log*"],
             "template": {
                 "settings": {
                     "plugins.index_state_management.rollover_alias": "log"
                 }
             }
         }'
     ```
  
     {% endcut %}
  
     {% cut "Example of a request to create an index with a policy alias" %}
  
     ```bash
     curl \
     --user <OpenSearch_username>:<password> \
     --header 'Content-Type: application/json' \
     --request PUT 'https://<address_of_OpenSearch_host_with_DATA_role>:9200/log-000001' \
     --data '
         {
             "aliases": {
                 "log": {
                     "is_write_index": true
                 }
             }
         }'
     ```
  
     {% endcut %}
  
     {% cut "Example of a request to check if a policy is attached to an index" %}
  
     ```bash
     curl \
     --user <OpenSearch_username>:<password> \
     --header 'Content-Type: application/json' \
     --request GET 'https://<address_of_OpenSearch_host_with_DATA_role>:9200/_plugins/_ism/explain/log-000001?pretty'
     ```
  
     {% endcut %}

{% endlist %}

### PostgreSQL target {#target-pg}

{% list tabs %}

- Managed Service for PostgreSQL

    1. Make sure that the PostgreSQL major version on the target is not lower than that on the source.

    1. For transfers from PostgreSQL, [enable the same extensions](../../managed-postgresql/operations/extensions/cluster-extensions.md) in the target database as in the source database.

        If extensions in the source database are installed in a custom schema and are used in the DDLs of the objects you are moving, create DDLs in the target manually. In these DDLs, a function call must not include the schema name. Set the target endpoint cleanup policy to `Truncate` to prevent the transfer from deleting these objects.

    1. Select the `Drop` cleanup policy for transfer tables.

        If you have created DDLs in the target manually, use the `Truncate` policy. The `Truncate` policy will not delete these DDLs.

    1. [Create a user](../../managed-postgresql/operations/cluster-users.md#adduser) with access to the target database.

    1. Grant the user all privileges for the database, schemas, and tables to be transferred:

        ```sql
        GRANT ALL PRIVILEGES ON DATABASE <database_name> TO <username>;
        ```

       If the database is not empty, the user must be its owner:

        ```sql
        ALTER DATABASE <database_name> OWNER TO <username>;
        ```

       Once started, the transfer will connect to the target on behalf of this user.
    1. If the target has the [Save transaction boundaries](endpoint/target/postgresql.md#additional-settings) option enabled, grant the new user all the privileges needed to create the `__data_transfer_lsn` auxiliary table in the [current schema](https://www.postgresql.org/docs/current/ddl-schemas.html#DDL-SCHEMAS-PATH) (usually `public`) in the target:

       ```sql
       GRANT ALL PRIVILEGES ON SCHEMA <schema_name> TO <username>;
       ```

    1. Configure the [number of user connections](../concepts/work-with-endpoints.md#postgresql-connection-limit) to the database.

- PostgreSQL

    1. If not planning to use [Cloud Interconnect](../../interconnect/concepts/index.md) or [VPN](https://en.wikipedia.org/wiki/Virtual_private_network) for connections to an external cluster, make such cluster accessible from the Internet from [IP addresses used by Data Transfer](../../overview/concepts/public-ips.md#virtual-private-cloud).
       
       For details on linking your network up with external resources, see [this concept](../concepts/network.md#source-external).

    1. Make sure that the PostgreSQL major version on the target is not lower than that on the source.

    1. In the target database, enable the same extensions that are enabled in the source database.

    1. Make sure the target has the `DROP transfer tables` cleanup policy selected.

    1. Create a user:

        ```sql
        CREATE ROLE <username> LOGIN ENCRYPTED PASSWORD '<password>';
        ```

    1. Grant the user all privileges for the database, schemas, and tables to be transferred:

        ```sql
        GRANT ALL PRIVILEGES ON DATABASE <database_name> TO <username>;
        ```

       If the database is not empty, the user must be its owner:

        ```sql
        ALTER DATABASE <database_name> OWNER TO <username>;
        ```

       Once started, the transfer will connect to the target on behalf of this user.

    1. If the target has the [Save transaction boundaries](endpoint/target/postgresql.md#additional-settings) option enabled, grant the new user all the privileges needed to create the `__data_transfer_lsn` auxiliary table in the [current schema](https://www.postgresql.org/docs/current/ddl-schemas.html#DDL-SCHEMAS-PATH) (usually `public`) in the target:

        ```sql
        GRANT ALL PRIVILEGES ON SCHEMA <schema_name> TO <username>;
        ```

    1. Configure the [number of user connections](../concepts/work-with-endpoints.md#postgresql-connection-limit) to the database.

{% endlist %}

Data stored in a `MATERIALIZED VIEW` is not transferred. To transfer `MATERIALIZED VIEW` data, create an ordinary `VIEW` that refers to the `MATERIALIZED VIEW` to be transferred.

If the definition of the `VIEW` to be transferred contains an invocation of the `VOLATILE` [function](https://www.postgresql.org/docs/current/xfunc-volatility.html), the transfer reads data from this `VIEW` with the `READ UNCOMMITTED` isolation level. No consistency between the `VIEW` data and the data of other objects being transferred is guaranteed. Reading data from a `MATERIALIZED VIEW` in the `VIEW` definition are equivalent to invoking the `VOLATILE` function.


### Yandex Managed Service for YDB target {#target-ydb}

1. [Create a service account](../../iam/operations/sa/create.md) with the `ydb.editor` role.
1. If your database is running in Dedicated mode, [create](../../vpc/operations/security-group-create.md) and [configure](../../ydb/operations/connection.md#configuring-security-groups) a security group in the network hosting the DB.


_Airbyte® is a registered trademark of Airbyte, Inc in the United States and/or other countries._

_Greenplum® and Greenplum Database® are registered trademarks or trademarks of Broadcom Inc. in the United States and/or other countries._

_ClickHouse® is a registered trademark of [ClickHouse, Inc](https://clickhouse.com)._