[Yandex Cloud documentation](../../index.md) > [Tutorials](../index.md) > [Building a data platform](index.md) > Migrating a database from a third-party Valkey™ cluster to Yandex Managed Service for Valkey™

# Migrating a database from a third-party Valkey™ cluster to Yandex Managed Service for Valkey™


# Migrating a database from a third-party Valkey™ cluster to Yandex Managed Service for Valkey™


For data migration, Valkey™ uses a _logical dump_. A _logical dump_ is a file with a sequence of commands that restore the database state. There are several ways to create a dump. The following example will use [redis-dump-go](https://github.com/yannh/redis-dump-go/).

{% note info %}

You cannot use a binary RDB dump for migration, as Yandex Managed Service for Valkey™ does not provide access to the file system on cluster hosts.

{% endnote %}

To migrate Valkey™ databases from the _source cluster_ to the _target cluster_:

1. [Connect to the source cluster and create a logical dump](#create-dump).
1. Optionally, [upload the dump to the intermediate VM](#load-vm).

    You need to use a VM in Yandex Compute Cloud to transfer data if:

    * Your Yandex Managed Service for Valkey™ cluster is not accessible from the internet.
    * Your local hardware or Yandex Cloud cluster connection is not reliable enough.

1. [Restore the dump in the target cluster](#restore-dump).
1. [Make sure the dump is fully restored](#check-data).

If you no longer need the resources you created, [delete them](#clear-out).


## Required paid resources {#paid-resources}

* Yandex Managed Service for Valkey™ target cluster: use of computing resources allocated to hosts, storage and backup size (see [Valkey™ pricing](../../managed-valkey/pricing.md)).
* Public IP addresses if public access is enabled for cluster hosts (see [Virtual Private Cloud pricing](../../vpc/pricing.md)).
* Virtual machine if created to upload a dump: use of computing resources, storage, public IP address, and OS (see [Compute Cloud pricing](../../compute/pricing.md)).


## Getting started {#before-you-begin}

### Set up your infrastructure {#deploy-infrastructure}

{% list tabs group=instructions %}

- Manually {#manual}

    1. [Create](../../managed-valkey/operations/cluster-create.md) a Yandex Managed Service for Valkey™ cluster with any suitable configuration. To connect to the cluster from your local machine outside Yandex Cloud, enable TLS and public access for cluster hosts when creating the cluster.

        {% note info %}
        
        Public access to cluster hosts is required if you plan to connect to the cluster via the internet. This connection option is simpler and is recommended for the purposes of this guide. You can connect to non-public hosts as well but only from Yandex Cloud virtual machines located in the same cloud network as the cluster.
        
        {% endnote %}

    1. Optionally, [create an intermediate Linux VM](../../compute/operations/vm-create/create-linux-vm.md) in Yandex Compute Cloud in the same network as the Yandex Managed Service for Valkey™ cluster using the following configuration:

        * Under **Boot disk image**, select [Ubuntu 20.04](https://yandex.cloud/en/marketplace/products/yc/ubuntu-20-04-lts).
        * Under **Network settings**:

            * **Public IP address**: `Auto`.
            * **Internal IPv4 address**: `Auto`.
            * **Security groups**: Same as for the Yandex Managed Service for Valkey™ cluster.

    
    1. If using Virtual Private Cloud security groups, [configure them](../../managed-valkey/operations/connect/index.md#configuring-security-groups).


- Terraform {#tf}

    1. If you do not have Terraform yet, [install it](../infrastructure-management/terraform-quickstart.md#install-terraform).
    1. [Get the authentication credentials](../infrastructure-management/terraform-quickstart.md#get-credentials). You can add them to environment variables or specify them later in the provider configuration file.
    1. [Configure and initialize a provider](../infrastructure-management/terraform-quickstart.md#configure-provider). There is no need to create a provider configuration file manually, you can [download it](https://github.com/yandex-cloud-examples/yc-terraform-provider-settings/blob/main/provider.tf).
    1. Place the configuration file in a separate working directory and [specify the parameter values](../infrastructure-management/terraform-quickstart.md#configure-provider). If you did not add the authentication credentials to environment variables, specify them in the configuration file.

    1. Download the configuration file matching your cluster type to the same working directory:

        * [redis-cluster-non-sharded.tf](https://github.com/yandex-cloud-examples/yc-redis-data-migration-from-on-premise/blob/main/redis-cluster-non-sharded.tf): For a non-sharded cluster.
        * [redis-cluster-sharded.tf](https://github.com/yandex-cloud-examples/yc-redis-data-migration-from-on-premise/blob/main/redis-cluster-sharded.tf): For a [sharded](../../managed-valkey/concepts/sharding.md) cluster.

        Each file describes the following:

        * Network.
        * Subnet.
        * Default security group and rules for connecting to your cluster and VM.
        * Yandex Managed Service for Valkey™ cluster with public internet access.
        * VM with public internet access, if required.

    1. Specify the following in the configuration file:

        * Password to access the Yandex Managed Service for Valkey™ cluster.
        * Optionally, VM settings:

            * VM public [image](../../compute/operations/images-with-pre-installed-software/get-list.md) ID, e.g., for [Ubuntu 20.04 LTS](https://yandex.cloud/en/marketplace/products/yc/ubuntu-20-04-lts).
            * Username and absolute path to the [public SSH key](../../compute/operations/vm-connect/ssh.md#creating-ssh-keys) for accessing the virtual machine. By default, [Ubuntu 20.04 LTS](https://yandex.cloud/en/marketplace/products/yc/ubuntu-20-04-lts) ignores the specified username and automatically creates a user named `ubuntu`. Use it to connect to the VM.

    1. Validate your Terraform configuration files using this command:

        ```bash
        terraform validate
        ```

        Terraform will display any configuration errors detected in your files.

    1. Create the required infrastructure:

        1. Run this command to view the planned changes:
        
           ```bash
           terraform plan
           ```
        
           If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.
        
        1. If everything looks correct, apply the changes:
           1. Run this command:
        
              ```bash
              terraform apply
              ```
        
           1. Confirm updating the resources.
           1. Wait for the operation to complete.

        All the required resources will be created in the specified folder. You can check resource availability and their settings in the [management console](https://console.yandex.cloud).

{% endlist %}

### Install additional software {#install-extra-software}

1. Optionally, install SSH file transfer tools on your local machine, such as the following:

    * [WinSCP](https://winscp.net/eng/index.php)
    * [Putty SCP](https://www.putty.org/)

1. Make sure [GNU Screen](https://www.gnu.org/software/screen/) is installed on the source cluster.

    Creating and restoring a dump may take a long time. To keep these processes running when your SSH session times out, start them with this tool. If your SSH connection drops during these operations, reconnect and resume the session using this command:

    ```bash
    screen -R
    ```

## Connect to the source cluster and create a logical dump {#create-dump}

1. Connect to the source cluster master host over SSH.
1. Download the archive with `redis-dump-go` from the [project page](https://github.com/yannh/redis-dump-go/releases). The examples below use version `0.5.1`.

    ```bash
    wget https://github.com/yannh/redis-dump-go/releases/download/v0.5.1/redis-dump-go_0.5.1_linux_amd64.tar.gz
    ```

1. Unpack the archive to the current directory:

    ```bash
    tar xf redis-dump-go_0.5.1_linux_amd64.tar.gz
    ```

1. Review the tool’s command-line options:

    ```bash
    ./redis-dump-go -h
    ```

1. If connecting to the Valkey™ cluster requires a password, enter it in the `REDISDUMPGO_AUTH` environment variable:

    ```bash
    export REDISDUMPGO_AUTH="<Valkey™_password>"
    ```

1. Start an interactive `screen` session:

    ```bash
    screen
    ```

1. Start creating a logical dump:

    ```bash
    ./redis-dump-go \
        -host <master_host_IP_address_or_FQDN_in_Valkey™_cluster> \
        -port <Valkey™_port> > <dump_file>
    ```

    {% note tip %}

    While the dump is being created, the number of processed keys will appear on the screen. Remember or write down the last output value, as you will need it to check whether the dump has been fully restored in the target cluster.

    {% endnote %}

1. Once the dump is created, download it to your computer.

1. Terminate the interactive `screen` session:

    ```bash
    exit
    ```

## Optionally, upload the dump to the intermediate VM {#load-vm}


1. [Connect to the intermediate VM over SSH](../../compute/operations/vm-connect/ssh.md).


1. Upload the dump from your computer to the intermediate VM in any way you like.

## Restore the dump in the target cluster {#restore-dump}

1. [Connect to the cluster](../../managed-valkey/operations/connect/index.md) and run an interactive `screen` session:

    ```bash
    screen
    ```

1. Start the dump recovery:

    {% list tabs group=connection %}

    - Connecting without TLS {#without-tls}

        **Before connecting, install the required dependencies:**
        
        ```bash
        sudo apt update && sudo apt install -y redis-tools
        ```

        **Connecting via Sentinel**

        ```bash
        host=$(redis-cli \
          -h <FQDN_of_any_Valkey™_host> \
          -p 26379 \
          sentinel \
          get-master-addr-by-name \
          no-shards-no-tls | head -n 1)
        redis-cli \
          -h ${host} \
          -p 6379 \
          -a <target_cluster_password> \
          --pipe < <dump_file>
        ```

        **Connecting directly to the master host**

        ```bash
        redis-cli \
          -h <master_host_FQDN> \
          -p 6379 \
          -a <target_cluster_password> \
          --pipe < <dump_file>
        ```

        When connecting to a non-sharded cluster, you can use [special FQDNs](../../managed-valkey/operations/connect/fqdn.md#special-fqdns) instead of the master host FQDN.

        **Connecting to a sharded cluster**

        1. Create a script with commands to load the dump:

            `load-dump.sh`

            ```bash
            shards=('<FQDN_of_master_host_in_shard_1>' \
                    ...
                    '<FQDN_of_master_host_in_shard_N>')

            for shard in "${shards[@]}" ; do
              redis-cli -h "${shard}" \
                        -p 6379 \
                        -a "<target_cluster_password>" \
                        --pipe < <dump_file>
            done
            ```

        1. Run the script:

            ```bash
            bash ./load-dump.sh
            ```

            While the script runs, you may see data insertion error messages. This is normal behavior for the `redis-cli` command, as each shard in a sharded cluster stores only a portion of the data. For more information, see [Sharding in Yandex Managed Service for Valkey™](../../managed-valkey/concepts/sharding.md).

    - Connecting with TLS {#with-tls}

        **Before connecting, install the dependencies:**
        
        Build `redis-tools` with TLS support using one of the two methods:
        
        * **From a repository**
        
            1. Add a repository:
        
                ```bash
                sudo apt-add-repository ppa:redislabs/redis
                ```
        
                The repository’s packages come pre-built with `BUILD_TLS=yes`.
        
            1. Install the tool:
        
                ```bash
                sudo apt update && sudo apt install -y redis-tools
                ```
        
        * **Manually**
        
            Go to the directory you want to download the distribution to. Download the stable tool version, then build and install it:
        
            ```bash
            wget https://download.redis.io/redis-stable.tar.gz && \
            tar -xzvf redis-stable.tar.gz && \
            cd redis-stable && \
            make BUILD_TLS=yes && \
            sudo make install && \
            sudo cp ./src/redis-cli /usr/bin/
            ```

        **Connecting via Sentinel**

        ```bash
        host=$(redis-cli \
               -h <FQDN_of_any_Valkey™_host> \
               -p 26379 \
               sentinel \
               get-master-addr-by-name \
               no-shards-tls | head -n 1)
        redis-cli \
            -h ${host} \
            -p 6380 \
            -a <target_cluster_password> \
            --tls \
            --cacert ~/.redis/YandexInternalRootCA.crt \
            --pipe < <dump_file>
        ```

        **Connecting directly to the master host**

        ```bash
        redis-cli \
            -h c-<cluster_ID>.rw.mdb.yandexcloud.net \
            -p 6380 \
            -a <target_cluster_password> \
            --tls \
            --cacert ~/.redis/YandexInternalRootCA.crt \
            --pipe < <dump_file>
        ```

        When connecting to a non-sharded cluster, you can use [special FQDNs](../../managed-valkey/operations/connect/fqdn.md#special-fqdns) instead of the master host FQDN.

        **Connecting to a sharded cluster**

        1. Create a script with commands to load the dump:

            `load-dump.sh`

            ```bash
            shards=('<FQDN_of_master_host_in_shard_1>' \
                    ...
                    '<FQDN_of_master_host_in_shard_N>')

            for shard in "${shards[@]}" ; do
              redis-cli -h "${shard}" \
                        -p 6380 \
                        -a "<target_cluster_password>" \
                        --tls \
                        --cacert ~/.redis/YandexInternalRootCA.crt \
                        --pipe < <dump_file>
            done
            ```

        1. Run the script:

            ```bash
            bash ./load-dump.sh
            ```

            While the script runs, you may see data insertion error messages. This is normal behavior for the `redis-cli` command, as each shard in a sharded cluster stores only a portion of the data. For more information, see [Sharding in Yandex Managed Service for Valkey™](../../managed-valkey/concepts/sharding.md).

        {% endcut %}

    {% endlist %}

1. Terminate the interactive `screen` session:

    ```bash
    exit
    ```

## Make sure the dump is fully restored {#check-data}

1. In the [management console](https://console.yandex.cloud), select the folder where you want to restore the cluster.
1. Navigate to **Yandex Managed Service for&nbsp;Valkey™**.
1. Click the cluster name and open the [Monitoring](../../managed-valkey/operations/monitoring.md) tab.

Pay attention to the **DB Keys** chart showing the number of keys stored in the cluster. If the cluster is [sharded](../../managed-valkey/concepts/sharding.md), the chart will show the number of keys in each shard. In this case, the number of keys in the cluster is equal to the total number of keys across all shards.

The total number of keys in the cluster must match the number of keys processed by `redis-dump-go` when creating the dump.

## Delete the resources you created {#clear-out}

Delete the resources you no longer need to avoid paying for them:

{% list tabs group=instructions %}

- Manually {#manual}

    * [Delete the Yandex Managed Service for Valkey™ cluster](../../managed-valkey/operations/cluster-delete.md).
    * If you created a staging virtual machine, [delete it](../../compute/operations/vm-control/vm-delete.md).
    * If you reserved public static IP addresses, release and [delete them](../../vpc/operations/address-delete.md).

- Terraform {#tf}

    1. In the terminal window, go to the directory containing the infrastructure plan.
    
        {% note warning %}
    
        Make sure the directory has no Terraform manifests with the resources you want to keep. Terraform deletes all resources that were created using the manifests in the current directory.
    
        {% endnote %}
    
    1. Delete resources:
    
        1. Run this command:
    
            ```bash
            terraform destroy
            ```
    
        1. Confirm deleting the resources and wait for the operation to complete.
    
        All the resources described in the Terraform manifests will be deleted.

{% endlist %}