[Yandex Cloud documentation](../../../../index.md) > [Yandex Data Transfer](../../../index.md) > [Step-by-step guides](../../index.md) > [Configuring endpoints](../index.md) > Elasticsearch > Target

# Transferring data to an Elasticsearch target endpoint


Yandex Data Transfer enables you to migrate data to a Elasticsearch database and implement various data transfer, processing, and transformation scenarios. To implement a transfer:

1. [Configure one of the supported data sources](#supported-sources).
1. [Prepare the Elasticsearch](#prepare) database for the transfer.
1. [Configure the target endpoint](#endpoint-settings) in Yandex Data Transfer.
1. [Create](../../transfer.md#create) a transfer and [start](../../transfer.md#activate) it.
1. Perform required operations with the database and [control the transfer](../../monitoring.md).
1. In case of any issues, [use ready-made solutions](#troubleshooting) to resolve them.

## Configuring the data source {#supported-sources}

Configure one of the supported data sources:

* [Apache Kafka®](../source/kafka.md)
* [YDS](../source/data-streams.md)
* [PostgreSQL](../source/postgresql.md)
* [Elasticsearch](../source/elasticsearch.md)
* [OpenSearch](../source/opensearch.md)

For a complete list of supported sources and targets in Yandex Data Transfer, see [Available transfers](../../../transfer-matrix.md).

## Preparing the target database {#prepare}

{% note info %}

Elasticsearch versions lower than 7.x are not supported.

{% endnote %}

If not planning to use [Cloud Interconnect](../../../../interconnect/concepts/index.md) or [VPN](https://en.wikipedia.org/wiki/Virtual_private_network) for connections to an external cluster, make such cluster accessible from the Internet from [IP addresses used by Data Transfer](../../../../overview/concepts/public-ips.md#virtual-private-cloud).

For details on linking your network up with external resources, see [this concept](../../../concepts/network.md#source-external).

* Make sure the number of columns in the source does not exceed the maximum number of fields in Elasticsearch indexes. The maximum number of fields is provided in the `index.mapping.total_fields.limit` [parameter](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-settings-limit.html#mapping-settings-limit); the default value is `1000`.

   To increase the parameter value, [set up a template](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-put-template.html#indices-put-template) that makes the maximum number of fields in new indexes equal to the specified value.

   {% cut "Sample template setup request" %}

   ```bash
   curl \
       --user <Elasticsearch_username>:<password> \
       --header 'Content-Type: application/json' \
       --request PUT "https://<Elasticsearch_cluster_FQDN>:9200/_template/index_defaults" \
       --data '
           {
               "index_patterns": "cdc*",
               "settings": {
                   "index": {
                       "mapping": {
                           "total_fields": {
                               "limit": "2000"
                           }
                       }
                   }
               }
           }'
   ```

   With this template setup, all new indexes with the `cdc*` mask may contain up to `2000` fields.

   {% endcut %}

   You can also set up templates using the [Kibana interface](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-mgmt.html#manage-index-templates).

   To check the current `index.mapping.total_fields.limit` parameter value, use the [Kibana interface](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-mgmt.html#view-edit-indices) or execute the following request:

   ```bash
   curl \
       --user <Elasticsearch_username>:<password> \
       --header 'Content-Type: application/json' \
       --request GET 'https://<Elasticsearch_cluster_FQDN>:9200/<index_name>/_settings/*total_fields.limit?include_defaults=true'
   ```

* By default, when transferring data to a single index, only one host is used. To distribute the load across hosts when transferring large amounts of data, [set up a template](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-put-template.html#indices-put-template) to split new indexes into shards in advance.

   {% cut "Sample template setup request" %}

   ```bash
   curl \
       --user <Elasticsearch_username>:<password> \
       --header 'Content-Type: application/json' \
       --request PUT 'https://<Elasticsearch_cluster_FQDN>:9200/_template/index_defaults' \
       --data '
           {
               "index_patterns": "cdc*",
               "settings" : {
                   "index" : {
                       "number_of_shards" : 15,
                       "number_of_replicas" : 1
                   }
               }
           }'
   ```

   {% endcut %}

   With this template setup, all new indexes with the `cdc*` mask will be split into `15` shards.

   You can also set up templates using the [Kibana interface](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-mgmt.html#manage-index-templates).

## Configuring the Elasticsearch target endpoint {#endpoint-settings}

When [creating](../index.md#create) or [updating](../index.md#update) an endpoint, you can define:

* [Elasticsearch connection settings](#on-premise). These are required parameters.
* [Additional parameters](#additional-settings).


### Custom installation {#on-premise}

Connection to nodes with explicitly specified network addresses and ports.

{% list tabs group=instructions %}

- Management console {#console}

    * **Data nodes**: Click ![image](../../../../_assets/console-icons/plus.svg) to add a new data node. For each node, specify:

    * **Host**: IP address or FQDN of the host with the `DATA` role you need to connect to.

    * **Port**: Port number Data Transfer will use to connect to the host with the `DATA` role.

    * **SSL**: Select this option if a secure SSL connection is used.

    * **CA certificate**: Upload the certificate file or add its contents as text if data encryption is required, e.g., to comply with the PCI DSS requirements.

    * **Subnet ID**: Select or [create](../../../../vpc/operations/subnet-create.md) a subnet in the required [availability zone](../../../../overview/concepts/geo-scope.md).
      If this field has a value specified for both endpoints, both subnets must be hosted in the same availability zone.
      
   * **User**: Specify the username Data Transfer will use to connect to the cluster.

   * **Password**: Enter the user password to the cluster.


   * **Security groups**: Select the cloud network to host the endpoint and security groups for network traffic.

     Thus, you will be able to apply the specified security group rules to the VMs and clusters in the selected network without changing the settings of these VMs and clusters. For more information, see [Networking in Yandex Data Transfer](../../../concepts/network.md).


{% endlist %}

### Additional settings {#additional-settings}

{% list tabs group=instructions %}

- Management console {#console}

    * **Cleanup policy**: Select a way to clean up data in the target database before the transfer:

        * `Don't cleanup`: Select this option if you are only going to do replication without copying data.

        * `Drop`: Completely delete the tables included in the transfer (default).

            Use this option to always transfer the latest version of the table schema to the target database from the source whenever the transfer is activated.

    * **Sanitize documents keys**: Use this option to automatically replace keys that are not valid for Elasticsearch in the target fields.

        The autocorrect rules are as follows:
        
        * Empty keys or keys consisting of spaces and periods will be replaced with underscores: `""`, `" "`, `"."` → `"_"`.
        * Leading and trailing periods will be removed: `"somekey."`, `".somekey"` → `"somekey"`.
        * If there are two periods in a row or there is nothing but spaces between them, the entire fragment will be replaced with a period: `" some . . key"` → `" some . key"`.
        
        Here is an example of how the autocorrect works: `". s o m e  ..incorrect....key.    . . "` → `" s o m e  .incorrect.key"`.

{% endlist %}


After configuring the data source and target, [create and start the transfer](../../transfer.md#create).