[Yandex Cloud documentation](../../index.md) > [Yandex StoreDoc](../index.md) > [Tutorials](index.md) > Migrating collections from a third-party Yandex StoreDoc cluster

# Migrating collections from MongoDB to Yandex StoreDoc

# Migrating collections from MongoDB to Yandex StoreDoc


To migrate collections from a third-party MongoDB cluster to your Yandex StoreDoc cluster, follow these steps: transfer the data, switch legacy databases to read-only mode, switch the workload to your Yandex Cloud cluster.


## Required paid resources {#paid-resources}

* Yandex StoreDoc cluster: computing resources allocated to hosts, storage and backup size (see [Yandex StoreDoc pricing](../pricing.md)).
* Public IP addresses if public access is enabled for cluster hosts (see [Virtual Private Cloud pricing](../../vpc/pricing.md)).
* Each transfer: use of computing resources and the number of transferred data rows (see [Data Transfer pricing](../../data-transfer/pricing.md)).


## Getting started {#before-you-begin}

Make sure you can connect to the source cluster’s hosts from the internet.

## Migrating data using Yandex Data Transfer {#data-transfer}

# Migrating data using Yandex Data Transfer {#data-transfer}

1. [Prepare the source cluster](../../data-transfer/operations/prepare.md#source-mg).
1. [Prepare the target cluster](../../data-transfer/operations/prepare.md#target-mg).
1. [Create a source endpoint](../../data-transfer/operations/endpoint/index.md#create) with the following settings:
   
   * **Database type**: `MongoDB`.
   * **Endpoint parameters** → **Connection settings**: `Custom installation`.
   
       Configure the source cluster connection settings.
   
   {% note info %}
   
   Transferring of `Time Series` collections is not supported, so you should [exclude](../../data-transfer/operations/endpoint/source/mongodb.md#additional-settings) such collections in the endpoint settings.
   
   {% endnote %}
1. [Create a target endpoint](../../data-transfer/operations/endpoint/index.md#create) with the following settings:
   
   * **Database type**: `MongoDB`.
   * **Endpoint parameters** → **Connection settings**: `Yandex StoreDoc cluster`.
   
       Specify the target cluster ID.
1. [Create a transfer](../../data-transfer/operations/transfer.md#create) of the _**Snapshot and increment**_ type that will use the new endpoints.
   
   To make large collections (over 1 GB) copy more quickly, enable [parallel copy](../../data-transfer/concepts/sharded.md) in the transfer settings. Specify two or more [workers](../../data-transfer/concepts/index.md#worker). The collection will be split into the specified number of parts that will be copied concurrently.
   
   For parallel copy to work, the `_id` fields in all documents within the collection must have the same data type. If a transfer detects a type mismatch, the collection will not be split but transferred in a single thread instead. If needed, remove documents with mismatched data types from the collection prior to transfer.
   
   {% note info %}
   
   If a document with a different data type is added to the collection after the transfer starts, the transfer will migrate it at the replication stage after parallel copying. However, when reactivated, the transfer will not be able to split the collection into parts, since the requirement for the same `_id` field type in all documents of the collection will not be met.
   
   {% endnote %}
1. [Activate the transfer](../../data-transfer/operations/transfer.md#activate).
1. Wait for the transfer status to change to **Replicating**.
1. Switch the source cluster to "read-only" mode and move the workload to the target cluster.
1. On the [transfer monitoring](../../data-transfer/operations/monitoring.md) page, wait until the **Maximum data transfer delay** value drops to zero. This means that all changes made in the source cluster after the initial data copy have been transferred to the target cluster.
1. [Deactivate](../../data-transfer/operations/transfer.md#deactivate) the transfer and wait for its status to change to **Stopped**.

    For more information about transfer statuses, see [Transfer lifecycle](../../data-transfer/concepts/transfer-lifecycle.md#statuses).

1. [Ensure the transfer is stopped, then delete it](../../data-transfer/operations/transfer.md#delete).
1. [Delete the source and target endpoints](../../data-transfer/operations/endpoint/index.md#delete).

## Collection migration example {#example}

1. [Configure the source cluster](#source-setup).
1. [Create and configure the target cluster](#target-setup).
1. [Set up the endpoints and transfer](#prepare-endpoints-transfer).
1. [Transfer data](#transfer).
1. [Delete the resources you created](#clear-out).

### Configure the source cluster {#source-setup}
  
1. Connect to the `mongos` host on the MongoDB source cluster via `mongosh`.
1. Create a database `db1`.
1. Create a user with owner privileges for `db1` and log in as that user:

    ```javascript
    use admin
    db.createUser({
      user: "user1",
      pwd: passwordPrompt(),
      roles: [
          {
              role: "dbOwner",
              db: "db1"
          }
      ]
    })

    db.auth("user1")
    ```

1. Fill in `collection1` in `db1` with random values. 200,000 entries will be added to the collection:
  
    ```javascript
    use db1
    function randomName() {
      // Base 36 uses letters and digits to represent a number:
      return (Math.random()+1).toString(36).substring(2);
    }

    var day = 1000 * 60 * 60 * 24;
    function randomDate () {
      return new Date(Date.now() - (Math.floor(Math.random() * day)));
    }

    for (var i = 1; i <= 200000; ++i) {
      db.collection1.insertOne({
       name: randomName(),
       creationDate: randomDate(),
       uid: i
      });
    }
    ```

### Create and configure the target cluster {#target-setup}

{% list tabs group=cluster %}

- Non-sharded target cluster {#non-sharded}

  1. [Create](../operations/cluster-create.md) a Yandex StoreDoc cluster with your preferred configuration.
  1. [Create](../operations/databases.md#add-db) a database `db1`.
  1. [Create a user](../operations/cluster-users.md#adduser) named `user_transfer` with the [`readWrite`](../concepts/users-and-roles.md#readWrite) role for the new database.
  
- Sharded target cluster {#sharded}

  1. [Create](../operations/cluster-create.md) a Yandex StoreDoc cluster with your preferred configuration. The cluster must have at least two hosts.
  1. [Enable sharding](../operations/shards.md).
  1. [Create](../operations/databases.md#add-db) a database `db1`.
  1. [Create a user](../operations/cluster-users.md#adduser) named `user_transfer` with the [`readWrite`](../concepts/users-and-roles.md#readWrite) role for your new database and the [`mdbShardingManager`](../concepts/users-and-roles.md#mdbShardingManager) role for the `admin` database.
  1. Follow [these steps](sharding.md) to create and configure an empty sharded collection named `collection1` in the `db1` database.

{% endlist %}

### Set up the endpoints and transfer {#prepare-endpoints-transfer}

{% list tabs group=cluster %}

- Non-sharded target cluster {#non-sharded}

  1. [Create the endpoint for the source cluster](../../data-transfer/operations/endpoint/index.md#create):
      * **Database type**: `MongoDB`.
      * **Connection settings**: `Custom installation`.
        * **CA certificate**: If connecting to the source cluster requires encryption, upload the certificate file.
        * **Host list**: Specify the FQDNs of the source cluster’s hosts.
        * **Port**: Specify the port for connecting to the hosts.
        * **Authentication source**: `db1`.
        * **User**: `user1`.
        * **Password**: Enter the `user1` password.
  1. [Create the endpoint for the target cluster](../../data-transfer/operations/endpoint/index.md#create):
      * **Database type**: `MongoDB`.
      * **Connection settings**: `MDB cluster`.
        * Specify the target cluster ID.
        * **User**: `user_transfer`.
        * **Password**: Enter the `user_transfer` password.
        * **Cleanup policy**: `Drop`.
  1. [Create a transfer](../../data-transfer/operations/transfer.md#create):
     * **Transfer type**: **Snapshot and increment**.
     * **Source**: Select the endpoint you created for the source cluster.
     * **Target**: Select the endpoint you created for the target cluster.

- Sharded target cluster {#sharded}

  1. [Create the endpoint for the source cluster](../../data-transfer/operations/endpoint/index.md#create):
      * **Database type**: `MongoDB`.
      * **Connection settings**: `Custom installation`.
        * **CA certificate**: If connecting to the source cluster requires encryption, upload the certificate file.
        * **Host list**: Specify the FQDNs of the source cluster’s hosts.
        * **Port**: Specify the port for connecting to the hosts.
        * **Authentication source**: `db1`.
        * **User**: `user1`.
        * **Password**: Enter the `user1` password.
  1. [Create the endpoint for the target cluster](../../data-transfer/operations/endpoint/index.md#create):
      * **Database type**: `MongoDB`.
      * **Connection settings**: `MDB cluster`.
         * Specify the target cluster ID.
         * **User**: `user_transfer`.
         * **Password**: Enter the `user_transfer` password.
         * **Cleanup policy**: `Don't cleanup` or `Truncate`.
  1. [Create a transfer](../../data-transfer/operations/transfer.md#create):
      * **Transfer type**: **Snapshot and increment**.
      * **Source**: Select the endpoint you created for the source cluster.
      * **Target**: Select the endpoint you created for the target cluster.

{% endlist %}

### Transfer the data {#transfer}

{% list tabs group=cluster %}

- Non-sharded target cluster {#non-sharded}

  1. [Activate](../../data-transfer/operations/transfer.md#activate) the transfer.
  1. Wait for the transfer status to change to **Replicating**.
  1. Switch the source cluster to <q>read-only</q> mode and transfer the workload over to the target cluster.
  1. On the [transfer monitoring](../../data-transfer/operations/monitoring.md) page, wait until the **Maximum data transfer delay** value drops to zero. This means that all changes made in the source cluster after the initial data copy have been transferred to the target cluster.
  1. [Connect](../operations/connect/index.md) to the target cluster.
  1. Verify that `collection1` has been transferred and contains 200,000 documents, matching the count in the source cluster:
  
     ```javascript
     use db1
     db.collection1.countDocuments()
     ```

     ```text
     200000
     ```

- Sharded target cluster {#sharded}

  1. [Activate](../../data-transfer/operations/transfer.md#activate) the transfer.
  1. Wait for the transfer status to change to **Replicating**.
  1. Switch the source cluster to <q>read-only</q> mode and transfer the workload over to the target cluster.
  1. On the [transfer monitoring](../../data-transfer/operations/monitoring.md) page, wait until the **Maximum data transfer delay** value drops to zero. This means that all changes made in the source cluster after the initial data copy have been transferred to the target cluster.
  1. [Connect](../operations/connect/index.md) to the target cluster.
  1. Verify that `collection1` has been transferred,  that it contains 200,000 documents as in the source cluster, and that the documents are distributed across shards:
       
      ```javascript
      use db1
      db.collection1.countDocuments()
      ```

      ```text
      200000
      ```

      ```javascript
      db.collection1.getShardDistribution()
      ```

      ```text
      Shard rs01 at rs01/<host_1>
      {
        data: '7.05MiB',
        docs: 99752,
        chunks: 2,
        'estimated data per chunk': '3.52MiB',
        'estimated docs per chunk': 49876
      }
      ---
      Shard rs02 at rs02/<host_2>
      {
        data: '7.08MiB',
        docs: 100248,
        chunks: 2,
        'estimated data per chunk': '3.54MiB',
        'estimated docs per chunk': 50124
      }
      ---
      ...
      ```

{% endlist %}

### Delete the resources you created {#clear-out}

Some resources are not free of charge. Delete the resources you no longer need to avoid paying for them:

1. [Deactivate](../../data-transfer/operations/transfer.md#deactivate) the transfer and wait for its status to change to **Stopped**.

    To learn more about the transfer lifecycle, see [this Yandex Data Transfer guide](../../data-transfer/concepts/transfer-lifecycle.md).

1. Once the transfer is stopped, [delete it](../../data-transfer/operations/transfer.md#delete).
1. [Delete the source and target endpoints](../../data-transfer/operations/endpoint/index.md#delete).
1. [Delete the Yandex StoreDoc cluster](../operations/cluster-delete.md).