[Yandex Cloud documentation](../../index.md) > [Yandex StoreDoc](../index.md) > [Tutorials](index.md) > Migrating data to Yandex StoreDoc

# Migrating data to Yandex StoreDoc


To migrate your data to Yandex StoreDoc, follow these steps: transfer the data, switch the legacy database to read-only mode, transfer the workload over to the target cluster in Yandex Cloud.

You can migrate data from a third-party _source cluster_ to a Yandex StoreDoc _target cluster_ using the following two methods:

* [Migrating data via Yandex Data Transfer](#data-transfer).

    This migration method allows you to:

    * Migrate your database with zero downtime.
    * Migrate from older to newer versions of MongoDB.
    * Eliminate the need for an intermediate VM or for exposing your Yandex StoreDoc target cluster to the internet.

    To use this method, enable public access to the source cluster.

* [Migration via database dump](#dump-and-restore).

    A _dump_ is a collection of files allowing you to restore a database to a specific state. To migrate data to a Yandex StoreDoc cluster, create a database dump using `mongodump` and restore it on the target cluster using `mongorestore`. To ensure dump integrity, switch the source cluster to <q>read-only</q> mode before creating the dump.

    Use this method only if data migration via Yandex Data Transfer is impossible.

## Migrating data using Yandex Data Transfer {#data-transfer}

To transfer data:

1. [Create a target cluster](#create-cluster).
1. [Prepare source and target clusters](#prepare-source-target).
1. [Set up the endpoints and transfer](#prepare-endpoints-transfer).
1. [Transfer data](#transfer).

If you no longer need the resources you created, [delete them](#clear-out).


### Getting started {#before-you-begin-transfer}

Sign up for Yandex Cloud and create a [billing account](../../billing/concepts/billing-account.md):
1. Navigate to the [management console](https://console.yandex.cloud) and log in to Yandex Cloud or create a new account.
1. On the **[Yandex Cloud Billing](https://center.yandex.cloud/billing/accounts)** page, make sure you have a billing account linked and it has the `ACTIVE` or `TRIAL_ACTIVE` [status](../../billing/concepts/billing-account-statuses.md). If you do not have a billing account, [create one](../../billing/quickstart/index.md) and [link](../../billing/operations/pin-cloud.md) a cloud to it.

If you have an active billing account, you can create or select a [folder](../../resource-manager/concepts/resources-hierarchy.md#folder) for your infrastructure on the [cloud page](https://console.yandex.cloud/cloud).

[Learn more about clouds and folders here](../../resource-manager/concepts/resources-hierarchy.md).

##### **Required paid resources** {#paid-resources-transfer}

* Yandex StoreDoc cluster: use of computing resources allocated to hosts, storage and backup size (see [Yandex StoreDoc pricing](../pricing.md)).
* Public IP addresses if public access is enabled for cluster hosts (see [Virtual Private Cloud pricing](../../vpc/pricing.md)).
* Each transfer: use of computing resources and the number of transferred data rows (see [Data Transfer pricing](../../data-transfer/pricing.md)).


### Create a target cluster {#create-cluster}

[Create a Yandex StoreDoc target cluster](../operations/cluster-create.md) with computing capacity and storage size matching the source database’s environment.

The source and target database names must be the same.


### Prepare source and target clusters {#prepare-source-target}

1. [Prepare the source cluster](../../data-transfer/operations/prepare.md#source-mg).
1. Make sure the source cluster’s network settings allow cluster connections from the internet.
1. [Prepare the target cluster](../../data-transfer/operations/prepare.md#target-mg).


### Set up the endpoints and transfer {#prepare-endpoints-transfer}

1. [Create a source endpoint](../../data-transfer/operations/endpoint/index.md#create) with the following settings:
   
   * **Database type**: `MongoDB`.
   * **Endpoint parameters** → **Connection settings**: `Custom installation`.
   
       Configure the source cluster connection settings.
   
   {% note info %}
   
   Transferring of `Time Series` collections is not supported, so you should [exclude](../../data-transfer/operations/endpoint/source/mongodb.md#additional-settings) such collections in the endpoint settings.
   
   {% endnote %}
1. [Create a target endpoint](../../data-transfer/operations/endpoint/index.md#create) with the following settings:
   
   * **Database type**: `MongoDB`.
   * **Endpoint parameters** → **Connection settings**: `Yandex StoreDoc cluster`.
   
       Specify the target cluster ID.
1. [Create a transfer](../../data-transfer/operations/transfer.md#create) of the _**Snapshot and increment**_ type that will use the new endpoints.
   
   To make large collections (over 1 GB) copy more quickly, enable [parallel copy](../../data-transfer/concepts/sharded.md) in the transfer settings. Specify two or more [workers](../../data-transfer/concepts/index.md#worker). The collection will be split into the specified number of parts that will be copied concurrently.
   
   For parallel copy to work, the `_id` fields in all documents within the collection must have the same data type. If a transfer detects a type mismatch, the collection will not be split but transferred in a single thread instead. If needed, remove documents with mismatched data types from the collection prior to transfer.
   
   {% note info %}
   
   If a document with a different data type is added to the collection after the transfer starts, the transfer will migrate it at the replication stage after parallel copying. However, when reactivated, the transfer will not be able to split the collection into parts, since the requirement for the same `_id` field type in all documents of the collection will not be met.
   
   {% endnote %}

### Transfer the data {#transfer}

1. [Activate the transfer](../../data-transfer/operations/transfer.md#activate).
1. Wait for the transfer status to change to **Replicating**.
1. Switch the source cluster to <q>read-only</q> mode and transfer the workload over to the target cluster.
1. On the [transfer monitoring](../../data-transfer/operations/monitoring.md) page, wait until the **Maximum data transfer delay** value drops to zero. This means that all changes made in the source cluster after the initial data copy have been transferred to the target cluster.

### Delete the resources you created {#clear-out}

Some resources are not free of charge. Delete the resources you no longer need to avoid paying for them:

1. [Deactivate](../../data-transfer/operations/transfer.md#deactivate) the transfer and wait for its status to change to **Stopped**.

    For more information about transfer statuses, see [Transfer lifecycle](../../data-transfer/concepts/transfer-lifecycle.md#statuses).

1. [Ensure the transfer is stopped, then delete it](../../data-transfer/operations/transfer.md#delete).
1. [Delete](../../data-transfer/operations/endpoint/index.md#delete) the source and target endpoints.
1. [Delete the Yandex StoreDoc cluster](../operations/cluster-delete.md).

## Migration via database dump {#dump-and-restore}

To transfer data via database dump:

1. [Create a target cluster](#create-cluster-dump).
1. [Create a dump](#dump) of the source database using `mongodump`.
1. If necessary, [create a VM](#create-vm) in Compute Cloud to restore the database from the dump within the Yandex Cloud infrastructure.
1. [Restore the data from the dump](#restore) to the cluster using `mongorestore`.

If you no longer need the resources you created, [delete them](#clear-out-dump).


### Getting started {#before-you-begin-dump}

Sign up for Yandex Cloud and create a [billing account](../../billing/concepts/billing-account.md):
1. Navigate to the [management console](https://console.yandex.cloud) and log in to Yandex Cloud or create a new account.
1. On the **[Yandex Cloud Billing](https://center.yandex.cloud/billing/accounts)** page, make sure you have a billing account linked and it has the `ACTIVE` or `TRIAL_ACTIVE` [status](../../billing/concepts/billing-account-statuses.md). If you do not have a billing account, [create one](../../billing/quickstart/index.md) and [link](../../billing/operations/pin-cloud.md) a cloud to it.

If you have an active billing account, you can create or select a [folder](../../resource-manager/concepts/resources-hierarchy.md#folder) for your infrastructure on the [cloud page](https://console.yandex.cloud/cloud).

[Learn more about clouds and folders here](../../resource-manager/concepts/resources-hierarchy.md).

##### **Required paid resources** {#paid-resources-dump}

* Yandex StoreDoc cluster: use of computing resources allocated to hosts, storage and backup size (see [Yandex StoreDoc pricing](../pricing.md)).
* Public IP addresses if public access is enabled for cluster hosts (see [Virtual Private Cloud pricing](../../vpc/pricing.md)).
* VM instance: use of computing resources, storage, public IP address, and OS (see [Compute Cloud pricing](../../compute/pricing.md)).


### Create a target cluster {#create-cluster-dump}

[Create a Yandex StoreDoc target cluster](../operations/cluster-create.md) with computing capacity and storage size matching the source database’s environment.

The source and target database names must be the same.

### Create a dump {#dump}

Use `mongodump` to create a database dump.

1. Install `mongodump` and other MongoDB tools. Example for [Ubuntu 20.04 LTS](https://yandex.cloud/en/marketplace/products/yc/ubuntu-20-04-lts):

    ```bash
    wget -qO - https://www.mongodb.org/static/pgp/server-4.4.asc | sudo apt-key add -
    echo "deb [ arch=amd64 ] https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/4.4 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.4.list
    sudo apt update
    sudo apt install mongodb-org-shell mongodb-org-tools
    ```


1. Before creating a dump, we recommend switching your database to read-only mode to avoid losing any data that might be written during the dump process.

1. Create a database dump:

    ```bash
    mongodump --host <DBMS_server_address> \
              --port <port> \
              --username <username> \
              --password "<password>" \
              --db <DB_name> \
              --out ~/db_dump
    ```

   If you can use multiple CPU cores for the dump, specify the `-j` flag with the number of available cores:

    ```bash
    mongodump --host <DBMS_server_address> \
              --port <port> \
              --username <username> \
              --password "<password>" \
              -j <number_of_cores> \
              --db <DB_name> \
              --out ~/db_dump
    ```

1. Archive the dump:

    ```bash
    tar -cvzf db_dump.tar.gz ~/db_dump
    ```

### Optionally, create a VM for dump upload {#create-vm}

You will need an intermediate VM in Yandex Compute Cloud under the following conditions:

* Your Yandex StoreDoc cluster is not reachable from the internet.
* Your hardware or connection to the cluster in Yandex Cloud is not very reliable.

To prepare your virtual machine for dump recovery:

1. In the management console, [create a new VM](../../compute/operations/vm-create/create-linux-vm.md) from an [Ubuntu 20.04 LTS](https://yandex.cloud/en/marketplace/products/yc/ubuntu-20-04-lts) image. The necessary amount of RAM and the number of CPU cores depend on the volume of data transferred and the required transfer speed.


   The minimum configuration (1 core, 2 GB RAM, 10 GB disk space) should be sufficient for migrating a database of up to 1 GB. The larger the database being migrated, the more disk space and RAM are required, with the available disk space at least twice the database size.

   The VM must reside in the same network and availability zone as the Yandex StoreDoc cluster’s master host. The VM must have an external IP address, which will allow you to upload the dump from outside Yandex Cloud.

1. Install the MongoDB client and additional database utilities:

    ```bash
    wget -qO - https://www.mongodb.org/static/pgp/server-4.4.asc | sudo apt-key add -
    echo "deb [ arch=amd64 ] https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/4.4 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.4.list
    sudo apt update
    sudo apt install mongodb-org-shell mongodb-org-tools
    ```

1. Move the database dump from your server to the VM with the help of `scp` or a similar tool:

    ```bash
    scp ~/db_dump.tar.gz <VM_username>@<VM_public_address>:/tmp/db_dump.tar.gz
    ```

1. Extract the dump on the virtual machine:

    ```bash
    tar -xzf /tmp/db_dump.tar.gz
    ```

Now you have a VM with a database dump, ready to be restored to the Yandex StoreDoc cluster.


### Restore the data {#restore}

Restore your database from the dump via `mongorestore`.

* If you are restoring a dump from a VM located in Yandex Cloud:

    ```bash
    mongorestore --host <DBMS_server_address> \
                 --port <port> \
                 --username <username> \
                 --password "<password>" \
                 -j <number_of_streams> \
                 --authenticationDatabase <DB_name> \
                 --nsInclude '*.*' /tmp/db_dump
    ```

* If you are restoring a dump from a server outside Yandex Cloud, you must explicitly specify the SSL settings for `mongorestore`:

    ```bash
    mongorestore --host <DBMS_server_address> \
                 --port <port> \
                 --ssl \
                 --sslCAFile <path_to_certificate_file> \
                 --username <username> \
                 --password "<password>" \
                 -j <number_of_streams> \
                 --authenticationDatabase <DB_name> \
                 --nsInclude '*.*' ~/db_dump
    ```

* To transfer only specific collections, use the `--nsInclude` and `--nsExclude` flags to specify the namespaces that should be included and excluded from the collections being restored.


### Delete the resources you created {#clear-out-dump}

Some resources are not free of charge. Delete the resources you no longer need to avoid paying for them:

1. [Delete the Yandex StoreDoc cluster](../operations/cluster-delete.md).
1. If you created a virtual machine to upload the dump to, [delete it](../../compute/operations/vm-control/vm-delete.md).
1. If you reserved a public static IP address, release and [delete it](../../vpc/operations/address-delete.md).