[Yandex Cloud documentation](../../index.md) > [Yandex Data Transfer](../index.md) > [Tutorials](index.md) > Migration with data storage type changed > Elasticsearch in OpenSearch

# Migrating data from Elasticsearch to Yandex Managed Service for OpenSearch

# Migrating data from Elasticsearch to Yandex Managed Service for OpenSearch using Yandex Data Transfer


You can set up data transfer from Elasticsearch to Managed Service for OpenSearch indexes using Data Transfer. Follow these steps:

1. [Configure the source cluster](#configure-source).
1. [Prepare your test data](#prepare-data).
1. [Configure the target cluster](#configure-target).
1. [Prepare and activate your transfer](#prepare-transfer).
1. [Test the transfer](#verify-transfer).

If you no longer need the resources you created, [delete them](#clear-out).


## Required paid resources {#paid-resources}

* Managed Service for OpenSearch cluster: use of computing resources, storage and backup size (see [Managed Service for OpenSearch pricing](../../managed-opensearch/pricing.md)).
* Public IP addresses if public access is enabled for cluster hosts (see [Virtual Private Cloud pricing](../../vpc/pricing.md)).


## Getting started {#before-you-begin}

1. [Create a custom Elasticsearch installation](https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started.html).
1. [Enable access to the cluster from Yandex Cloud](../concepts/network.md#source-external).
1. Create a Managed Service for OpenSearch target cluster:

    {% list tabs group=instructions %}

    - Manually {#manual}

        [Create a Managed Service for OpenSearch target cluster](../../managed-opensearch/operations/cluster-create.md) in any suitable configuration with publicly accessible hosts.

        {% note info %}
        
        Public access to cluster hosts is required if you plan to connect to the cluster via the internet. This connection option is simpler and is recommended for the purposes of this guide. You can connect to non-public hosts as well but only from Yandex Cloud virtual machines located in the same cloud network as the cluster.
        
        {% endnote %}

    - Terraform {#tf}

        1. If you do not have Terraform yet, [install it and configure the Yandex Cloud provider](../../tutorials/infrastructure-management/terraform-quickstart.md#install-terraform).
           
           
           To manage infrastructure using Terraform under a service account or user accounts (a Yandex account, a federated account, or a local user), [authenticate](../../terraform/authentication.md) using the appropriate method.
        1. Download the [file with provider settings](https://github.com/yandex-cloud-examples/yc-terraform-provider-settings/blob/main/provider.tf). Place it in a separate working directory and specify the parameter values.
        1. Download the [data-transfer-es-mos.tf](https://github.com/yandex-cloud-examples/yc-data-transfer-elasticsearch-to-opensearch/blob/main/data-transfer-es-mos.tf) configuration file to your current working directory.

            This file describes:

            * [Network](../../vpc/concepts/network.md#network).
            * [Subnet](../../vpc/concepts/network.md#subnet).
            * [Security group](../../vpc/concepts/security-groups.md) and rules for connecting to a Managed Service for OpenSearch cluster.
            * Managed Service for OpenSearch target cluster.
            * Transfer.

        1. In the `data-transfer-es-mos.tf` file, specify the following variables:

            * `mos_version`: OpenSearch version.
            * `mos_admin_password`: Managed Service for OpenSearch admin password.
            * `transfer_enabled`: Set to `0` to prevent transfer creation until you [create endpoints manually](#prepare-transfer).

        1. Run the `terraform init` command in the directory with the configuration file. This command initializes the provider specified in the configuration files and enables you to use its resources and data sources.
        1. Validate your Terraform configuration files using this command:

            ```bash
            terraform validate
            ```

            Terraform will display any configuration errors detected in your files.

        1. Create the required infrastructure:

            1. Run this command to view the planned changes:
            
               ```bash
               terraform plan
               ```
            
               If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.
            
            1. If everything looks correct, apply the changes:
               1. Run this command:
            
                  ```bash
                  terraform apply
                  ```
            
               1. Confirm updating the resources.
               1. Wait for the operation to complete.

            All the required resources will be created in the specified folder. You can check resource availability and their settings in the [management console](https://console.yandex.cloud).

    {% endlist %}

1. Install the following tools:

    * [curl](https://curl.se/) to make requests to clusters.

        ```bash
        sudo apt update && sudo apt install --yes curl
        ```

    * [jq](https://stedolan.github.io/jq/) for stream processing of JSON files.

        ```bash
        sudo apt update && sudo apt install --yes jq
        ```

## Configure the source cluster {#configure-source}


You can deliver data from the Elasticsearch cluster as `admin` with the `superuser` role; however, however, a more secure strategy is to create dedicated users with limited privileges for each job. To do this, create a user to execute the transfer:

1. In the source cluster, [create a role](https://www.elastic.co/guide/en/elasticsearch/reference/current/defining-roles.html) with the `create_index` and `write` [privileges](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-privileges.html#privileges-list-indices) for all indexes (`*`).

1. In the source cluster, create a user to run the transfer and assign the user the role you created.

## Prepare your test data {#prepare-data}

1. In the source cluster, create a test index named `people` and define its schema:

    ```bash
    curl --user <username_in_source_cluster>:<user_password_in_source_cluster> \
         --header 'Content-Type: application/json' \
         --request PUT 'https://<address_of_Elasticsearch_host_with_Data_role>:9200/people' && \
    curl --user <username_in_source_cluster>:<user_password_in_source_cluster> \
         --header 'Content-Type: application/json' \
         --request PUT 'https://<address_of_Elasticsearch_host_with_Data_role>:9200/people/_mapping?pretty' \
         --data'
         {
               "properties": {
                  "name": {"type": "text"},
                  "age": {"type": "integer"}
               }
         }
         '
    ```

1. Populate the test index with data:

    ```bash
    curl --user <username_in_source_cluster>:<user_password_in_source_cluster> \
         --header 'Content-Type: application/json' \
         --request POST 'https://<address_of_Elasticsearch_host_with_Data_role>:9200/people/_doc/?pretty' \
         --data'
         {
               "name" : "Alice",
               "age" : "30"
         }
         ' && \
    curl --user <user_name_in_source_cluster>:<user_password_in_source_cluster> \
         --header 'Content-Type: application/json' \
         --request POST 'https://<address_of_Elasticsearch_host_with_Data_role>:9200/people/_doc/?pretty' \
         --data'
         {
               "name" : "Robert",
               "age" : "32"
         }
         '
    ```

1. Optionally, check the data in the test index:

    ```bash
    curl --user <username_in_target_cluster>:<user_password_in_target_cluster> \
         --header 'Content-Type: application/json' \
         --request GET 'https://<address_of_Elasticsearch_host_with_Data_role>:9200/people/_search?pretty'
    ```

## Configure the target cluster {#configure-target}

1. [Get an SSL certificate](../../managed-opensearch/operations/connect/index.md#ssl-certificate) to connect to the Managed Service for OpenSearch cluster.

1. Optionally, create a user to run the transfer.

    You can deliver data to the Managed Service for OpenSearch cluster as `admin` with the `superuser` role; however, a more secure strategy is to create dedicated users with limited privileges for each job.

    1. [Create a role](https://opensearch.org/docs/latest/security-plugin/access-control/users-roles/#create-roles) with the `create_index` and `write` privileges for all indexes (`*`).

    1. [Create a user](../../managed-opensearch/operations/cluster-users.md) and assign this role to them.

## Prepare and activate your transfer {#prepare-transfer}

1. [Create an endpoint](../operations/endpoint/index.md#create) for the [Elasticsearch source](../operations/endpoint/source/elasticsearch.md).

1. [Create an endpoint](../operations/endpoint/index.md#create) for the [OpenSearch target](../operations/endpoint/target/opensearch.md).

1. Create a transfer:

    {% list tabs group=instructions %}

    - Manually {#manual}

        1. [Create a transfer](../operations/transfer.md#create) of the **_Snapshot_**-type that will use the endpoints you created.
        1. [Activate the transfer](../operations/transfer.md#activate) and wait for its status to change to **Completed**.

    - Terraform {#tf}

        1. In the `data-transfer-mes-mos.tf` file, specify the following variables:

            * `source_endpoint_id`: Source endpoint ID.
            * `target_endpoint_id`: Target endpoint ID.
            * `transfer_enabled`: Set to `1` to create a transfer.

        1. Validate your Terraform configuration files using this command:

            ```bash
            terraform validate
            ```

            Terraform will display any configuration errors detected in your files.

        1. Create the required infrastructure:

            1. Run this command to view the planned changes:
            
               ```bash
               terraform plan
               ```
            
               If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.
            
            1. If everything looks correct, apply the changes:
               1. Run this command:
            
                  ```bash
                  terraform apply
                  ```
            
               1. Confirm updating the resources.
               1. Wait for the operation to complete.

        1. [Activate the transfer](../operations/transfer.md#activate) and wait for its status to change to **Completed**.

    {% endlist %}

## Test the transfer {#verify-transfer}

Check that the Managed Service for OpenSearch cluster's `people` index contains the data you sent:

{% list tabs group=programming_language %}

- Bash {#bash}

    Run this command:

    ```bash
    curl --user <username_in_target_cluster>:<user_password_in_target_cluster> \
         --cacert ~/.opensearch/root.crt \
         --header 'Content-Type: application/json' \
         --request GET 'https://<address_of_OpenSearch_host_with_Data_Node_role>:9200/people/_search?pretty'
    ```

- OpenSearch Dashboards {#opensearch}

    1. [Connect](../../managed-opensearch/operations/connect/clients.md#dashboards) to the target cluster using OpenSearch Dashboards.
    1. Select the `Global` tenant.
    1. Open the management panel by clicking ![os-dashboards-sandwich](../../_assets/console-icons/bars.svg).
    1. Under **OpenSearch Dashboards**, select **Discover**.
    1. In the **CHANGE INDEX PATTERN** field, select the `people` index.

{% endlist %}

## Delete the resources you created {#clear-out}

To minimize resource consumption, delete the resources you no longer need:

1. [Delete the transfer](../operations/transfer.md#delete).
1. [Delete the source and target endpoints](../operations/endpoint/index.md#delete).
1. Delete the rest of the resources depending on how you created them:

    {% list tabs group=instructions %}

    - Manually {#manual}

        [Delete the Managed Service for OpenSearch cluster](../../managed-opensearch/operations/cluster-delete.md).

    - Terraform {#tf}

        1. In the terminal window, go to the directory containing the infrastructure plan.
        
            {% note warning %}
        
            Make sure the directory has no Terraform manifests with the resources you want to keep. Terraform deletes all resources that were created using the manifests in the current directory.
        
            {% endnote %}
        
        1. Delete resources:
        
            1. Run this command:
        
                ```bash
                terraform destroy
                ```
        
            1. Confirm deleting the resources and wait for the operation to complete.
        
            All the resources described in the Terraform manifests will be deleted.

    {% endlist %}