[Yandex Cloud documentation](../../index.md) > [Yandex Managed Service for OpenSearch](../index.md) > [Tutorials](index.md) > Migrating and copying data > Migrating data from a third-party OpenSearch cluster

# Migrating data from a third-party OpenSearch cluster using Yandex Data Transfer

# Migrating data from OpenSearch to Yandex Managed Service for OpenSearch using Yandex Data Transfer


With Data Transfer, you can transfer data from a third-party OpenSearch source cluster's indexes to Managed Service for OpenSearch indexes. Proceed as follows:

1. [Configure the source cluster](#configure-source).
1. [Prepare your test data](#prepare-data).
1. [Configure the target cluster](#configure-target).
1. [Prepare and activate your transfer](#prepare-transfer).
1. [Test the transfer](#verify-transfer).

If you no longer need the resources you created, [delete them](#clear-out).


## Required paid resources {#paid-resources}

* Managed Service for OpenSearch cluster: use of computing resources, storage and backup size (see [Managed Service for OpenSearch pricing](../pricing.md)).
* Public IP addresses if public access is enabled for cluster hosts (see [Yandex Virtual Private Cloud pricing](../../vpc/pricing.md)).


## Getting started {#before-you-begin}

1. [Enable access to the OpenSearch source cluster from Yandex Cloud](../../data-transfer/concepts/network.md#source-external).

1. Create a Managed Service for OpenSearch target cluster:

    {% list tabs group=instructions %}

    - Manually {#manual}

        [Create a Managed Service for OpenSearch target cluster](../operations/cluster-create.md) in any suitable configuration with publicly accessible hosts.

        {% note info %}
        
        Public access to cluster hosts is required if you plan to connect to the cluster via the internet. This connection option is simpler and is recommended for the purposes of this guide. You can connect to non-public hosts as well but only from Yandex Cloud virtual machines located in the same cloud network as the cluster.
        
        {% endnote %}

    - Terraform {#tf}

        1. If you do not have Terraform yet, [install it and configure the Yandex Cloud provider](../../tutorials/infrastructure-management/terraform-quickstart.md#install-terraform).
           
           
           To manage infrastructure using Terraform under a service account or user accounts (a Yandex account, a federated account, or a local user), [authenticate](../../terraform/authentication.md) using the appropriate method.
        1. Download the [file with provider settings](https://github.com/yandex-cloud-examples/yc-terraform-provider-settings/blob/main/provider.tf). Place it in a separate working directory and specify the parameter values.
        1. Download the [data-transfer-os-mos.tf](https://github.com/yandex-cloud-examples/yc-data-transfer-opensearch-from-onprem-to-cloud/blob/main/data-transfer-os-mos.tf) configuration file to the same working directory.

            This file describes:

            * [Network](../../vpc/concepts/network.md#network).
            * [Subnet](../../vpc/concepts/network.md#subnet).
            * [Security group](../../vpc/concepts/security-groups.md) and rules for connecting to a Managed Service for OpenSearch cluster.
            * Managed Service for OpenSearch target cluster.
            * Transfer.

        1. In the `data-transfer-os-mos.tf` file, specify the following variables:

            * `mos_version`: OpenSearch version.
            * `mos_admin_password`: Managed Service for OpenSearch admin password.
            * `transfer_enabled`: Set to `0` to ensure that no transfer is created until you [create endpoints manually](#prepare-transfer).
            * `profile_name`: Name of your CLI profile.

              If you do not have the Yandex Cloud CLI yet, [install and initialize it](../../cli/quickstart.md#install).

        1. Run the `terraform init` command in the directory with the configuration file. This command initializes the provider specified in the configuration files and enables you to use its resources and data sources.
        1. Validate your Terraform configuration files using this command:

            ```bash
            terraform validate
            ```

            Terraform will display any configuration errors detected in your files.

        1. Create the required infrastructure:

            1. Run this command to view the planned changes:
            
               ```bash
               terraform plan
               ```
            
               If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.
            
            1. If everything looks correct, apply the changes:
               1. Run this command:
            
                  ```bash
                  terraform apply
                  ```
            
               1. Confirm updating the resources.
               1. Wait for the operation to complete.

            All the required resources will be created in the specified folder. You can check resource availability and their settings in the [management console](https://console.yandex.cloud).

    {% endlist %}

1. Install the following tools:

    * [curl](https://curl.se/) to make requests to clusters.

        ```bash
        sudo apt update && sudo apt install --yes curl
        ```

    * [jq](https://stedolan.github.io/jq/) for stream processing of JSON files.

        ```bash
        sudo apt update && sudo apt install --yes jq
        ```

## Configure the source cluster {#configure-source}

Create a user to run the transfer.

You can deliver data from the OpenSearch cluster as `admin` with the `superuser` role; however, a more secure strategy is to create dedicated users with limited privileges for each job.

1. Optionally, in the source cluster, [create a role](https://opensearch.org/docs/latest/security-plugin/access-control/users-roles/#create-roles) with the `create_index` and `write` privileges for all indexes (`*`).

1. Optionally, in the source cluster, [create a user](https://opensearch.org/docs/latest/security-plugin/access-control/users-roles) to run the transfer and assign them the role you created.

## Prepare your test data {#prepare-data}

1. In the source cluster, create a test index named `people` and define its schema:

    ```bash
    curl --user <user_name_in_source_cluster>:<user_password_in_source_cluster> \
         --header 'Content-Type: application/json' \
         --request PUT 'https://<address_of_OpenSearch_host_with_DATA_role>:9200/people' && \
    curl --user <user_name_in_source_cluster>:<user_password_in_source_cluster> \
         --header 'Content-Type: application/json' \
         --request PUT 'https://<address_of_OpenSearch_host_with_DATA_role>:9200/people/_mapping?pretty' \
         --data'
         {
               "properties": {
                  "name": {"type": "text"},
                  "age": {"type": "integer"}
               }
         }
         '
    ```

1. Populate the test index with data:

    ```bash
    curl --user <user_name_in_source_cluster>:<user_password_in_source_cluster> \
         --header 'Content-Type: application/json' \
         --request POST 'https://<address_of_OpenSearch_host_with_DATA_role>:9200/people/_doc/?pretty' \
         --data'
         {
               "name": "Alice",
               "age": "30"
         }
         ' && \
    curl --user <user_name_in_source_cluster>:<user_password_in_source_cluster> \
         --header 'Content-Type: application/json' \
         --request POST 'https://<address_of_OpenSearch_host_with_DATA_role>:9200/people/_doc/?pretty' \
         --data'
         {
               "name": "Robert",
               "age": "32"
         }
         '
    ```

1. Optionally, check the data in the test index:

    ```bash
    curl --user <user_name_in_target_cluster>:<user_password_in_target_cluster> \
         --header 'Content-Type: application/json' \
         --request GET 'https://<address_of_OpenSearch_host_with_DATA_role>:9200/people/_search?pretty'
    ```

## Configure the target cluster {#configure-target}

1. [Get an SSL certificate](../operations/connect/index.md#ssl-certificate) to connect to the Managed Service for OpenSearch cluster.

1. Optionally, create a user to run the transfer.

    You can deliver data to the Managed Service for OpenSearch cluster as `admin` with the `superuser` role; however, a more secure strategy is to create dedicated users with limited privileges for each job.

    1. [Create a role](https://opensearch.org/docs/latest/security-plugin/access-control/users-roles/#create-roles) with the `create_index` and `write` privileges for all indexes (`*`).

    1. [Create a user](../operations/cluster-users.md) and assign this role to them.

## Prepare and activate a transfer {#prepare-transfer}

1. [Create an endpoint](../../data-transfer/operations/endpoint/index.md#create) for the [OpenSearch source cluster](../../data-transfer/operations/endpoint/source/opensearch.md#on-premise).

1. [Create an endpoint](../../data-transfer/operations/endpoint/index.md#create) for the [Managed Service for OpenSearch target cluster](../../data-transfer/operations/endpoint/target/opensearch.md).

1. Create a transfer:

    {% list tabs group=instructions %}

    - Manually {#manual}

        1. [Create a transfer](../../data-transfer/operations/transfer.md#create) of the **_Snapshot_** type that will use the endpoints you created.
        1. [Activate the transfer](../../data-transfer/operations/transfer.md#activate).

    - Terraform {#tf}

        1. In the `data-transfer-os-mos.tf` file, specify the following variables:

            * `source_endpoint_id`: Source endpoint ID.
            * `target_endpoint_id`: Target endpoint ID.
            * `transfer_enabled`: Set to `1` to create a transfer.

        1. Validate your Terraform configuration files using this command:

            ```bash
            terraform validate
            ```

            Terraform will display any configuration errors detected in your files.

        1. Create the required infrastructure:

            1. Run this command to view the planned changes:
            
               ```bash
               terraform plan
               ```
            
               If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.
            
            1. If everything looks correct, apply the changes:
               1. Run this command:
            
                  ```bash
                  terraform apply
                  ```
            
               1. Confirm updating the resources.
               1. Wait for the operation to complete.

            The transfer will be activated automatically as soon as it is created.

    {% endlist %}

## Test the transfer {#verify-transfer}

1. Wait for the transfer status to change to **Completed**.
1. Check that the Managed Service for OpenSearch cluster's `people` index contains the data you sent:

    {% list tabs group=programming_language %}

    - Bash {#bash}

      Run this command:

      ```bash
      curl --user <user_name_in_target_cluster>:<user_password_in_target_cluster> \
           --cacert ~/.opensearch/root.crt \
           --header 'Content-Type: application/json' \
           --request GET 'https://<FQDN_of_OpenSearch_host_with_DATA_role>:9200/people/_search?pretty'
      ```

    - OpenSearch Dashboards {#opensearch}

      1. [Connect](../operations/connect/clients.md#dashboards) to the target cluster using OpenSearch Dashboards.
      1. Select the `Global` tenant.
      1. Open the management panel by clicking ![os-dashboards-sandwich](../../_assets/console-icons/bars.svg).
      1. Under **OpenSearch Dashboards**, select **Discover**.
      1. In the **CHANGE INDEX PATTERN** field, select the `people` index.

    {% endlist %}

## Delete the resources you created {#clear-out}

To minimize resource consumption, delete the resources you no longer need:

1. [Delete the transfer](../../data-transfer/operations/transfer.md#delete).
1. [Delete the source and target endpoints](../../data-transfer/operations/endpoint/index.md#delete).
1. Delete the rest of the resources depending on how you created them:

    {% list tabs group=instructions %}

    - Manually {#manual}

        1. [Delete the Managed Service for OpenSearch cluster](../operations/cluster-delete.md).

        1. [Delete the subnet](../../vpc/operations/subnet-delete.md) and [network](../../vpc/operations/network-delete.md).

    - Terraform {#tf}

        1. In the terminal window, go to the directory containing the infrastructure plan.
        
            {% note warning %}
        
            Make sure the directory has no Terraform manifests with the resources you want to keep. Terraform deletes all resources that were created using the manifests in the current directory.
        
            {% endnote %}
        
        1. Delete resources:
        
            1. Run this command:
        
                ```bash
                terraform destroy
                ```
        
            1. Confirm deleting the resources and wait for the operation to complete.
        
            All the resources described in the Terraform manifests will be deleted.

    {% endlist %}