[Yandex Cloud documentation](../../index.md) > [Yandex Data Streams](../index.md) > [Tutorials](index.md) > Data migration to Yandex Object Storage using Yandex Data Transfer

# Migrating data to Yandex Object Storage using Yandex Data Transfer

With Data Transfer, you can transfer data from a [stream in Data Streams](../concepts/glossary.md#stream-concepts) to a Yandex Object Storage bucket.

1. [Set up a data stream in Data Streams](#prepare-source).
1. [Prepare and activate the transfer](#prepare-transfer).
1. [Test the transfer](#verify-transfer).

If you no longer need the resources you created, [delete them](#clear-out).


## Required paid resources {#paid-resources}

* Managed Service for YDB database (see [Managed Service for YDB pricing](../../ydb/pricing/index.md)). Its cost depends on the deployment mode:

	* In serverless mode, you pay for data operations as well as the amount of stored data and backups.
  	* In dedicated instance mode, you pay for the use of computing resources allocated to the database, storage size, and backups.

* Data Streams (see [Data Streams pricing](../pricing.md)). The cost depends on the pricing model:

    * [Based on allocated resources](../pricing.md#rules): You pay a fixed hourly rate for the established throughput limit and message retention period, and additionally for the number of units of actually written data.
    * [On-demand](../pricing.md#on-demand): You pay for the performed read/write operations, the amount of read or written data, and the actual storage used for messages that are still within their retention period.

* Object Storage bucket: use of storage, data operations (see [Object Storage pricing](../../storage/pricing.md)).


## Getting started {#before-you-begin}

Set up your infrastructure:

{% list tabs group=instructions %}

- Manually {#manual}

    1. [Create a Managed Service for YDB database](../../ydb/operations/manage-databases.md) of any suitable configuration.

    1. [Create a bucket in Yandex Object Storage](../../storage/operations/buckets/create.md).

- Terraform {#tf}

    1. If you do not have Terraform yet, [install it](../../tutorials/infrastructure-management/terraform-quickstart.md#install-terraform).
    1. [Get the authentication credentials](../../tutorials/infrastructure-management/terraform-quickstart.md#get-credentials). You can add them to environment variables or specify them later in the provider configuration file.
    1. [Configure and initialize a provider](../../tutorials/infrastructure-management/terraform-quickstart.md#configure-provider). There is no need to create a provider configuration file manually, you can [download it](https://github.com/yandex-cloud-examples/yc-terraform-provider-settings/blob/main/provider.tf).
    1. Place the configuration file in a separate working directory and [specify the parameter values](../../tutorials/infrastructure-management/terraform-quickstart.md#configure-provider). If you did not add the authentication credentials to environment variables, specify them in the configuration file.

    1. Download the [data-transfer-yds-obj.tf](https://github.com/yandex-cloud-examples/yc-data-transfer-from-yds-to-object-storage/blob/main/data-transfer-yds-obj.tf) configuration file to the same working directory.

        This file describes:

        * Database: Managed Service for YDB.
        * Service account with the `yds.editor`, `storage.editor`, and `storage.uploader` roles.
        * Bucket in Object Storage.
        * Transfer.

    1. In the `data-transfer-yds-obj.tf` file, specify the following:

        * `folder_id`: [ID of the folder](../../resource-manager/operations/folder/get-id.md) for the new resources.
        * `sa_name`: Name of the service account for creating a bucket and for use in endpoints.
        * `source_db_name`: Managed Service for YDB database name.
        * `bucket_name`: Object Storage bucket name.

    1. Make sure the Terraform configuration files are correct using this command:

        ```bash
        terraform validate
        ```

        Terraform will display any configuration errors detected in your files.

    1. Create the required infrastructure:

        1. Run this command to view the planned changes:
        
           ```bash
           terraform plan
           ```
        
           If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.
        
        1. If everything looks correct, apply the changes:
           1. Run this command:
        
              ```bash
              terraform apply
              ```
        
           1. Confirm updating the resources.
           1. Wait for the operation to complete.

        All the required resources will be created in the specified folder. You can check resource availability and their settings in the [management console](https://console.yandex.cloud).

{% endlist %}

## Set up a data stream in Data Streams {#prepare-source}

1. [Create a data stream in Data Streams](../operations/aws-cli/create.md).

1. [Send test data to this stream](../operations/aws-cli/send.md). Use the car sensor data in JSON format as the message:

```json
{
    "device_id":"iv9a94th6rzt********",
    "datetime":"2020-06-05T17:27:00",
    "latitude":"55.70329032",
    "longitude":"37.65472196",
    "altitude":"427.5",
    "speed":"0",
    "battery_voltage":"23.5",
    "cabin_temperature":"17",
    "fuel_level":null
}
```

## Prepare and activate the transfer {#prepare-transfer}

1. [Create a source endpoint](../../data-transfer/operations/endpoint/index.md#create):

    * **Database type**: `Yandex Data Streams`.
    * **Endpoint settings**:

        * **Connection settings**:

            * **Database**: Select the Managed Service for YDB database from the list.
            * **Stream**: Specify the name of the stream in Data Streams.
            * **Service account**: Select an existing service account or create a new one with the `yds.editor` role.

        * **Advanced settings**:

            * **Conversion rules**: `JSON`.
            * **Data scheme**: `JSON specification`:

                Create and upload a data schema file in JSON format, `json_schema.json`:

                {% cut "json_schema.json" %}

                ```json
                [
                    {
                        "name": "device_id",
                        "type": "string"
                    },
                    {
                        "name": "datetime",
                        "type": "datetime"
                    },
                    {
                        "name": "latitude",
                        "type": "double"
                    },
                    {
                        "name": "longitude",
                        "type": "double"
                    },
                    {
                        "name": "altitude",
                        "type": "double"
                    },
                    {
                        "name": "speed",
                        "type": "double"
                    },
                    {
                        "name": "battery_voltage",
                        "type": "any"
                    },
                    {
                        "name": "cabin_temperature",
                        "type": "double"
                    },
                    {
                        "name": "fuel_level",
                        "type": "any"
                    }
                ]
                ```

                {% endcut %}

1. [Create a target endpoint](../../data-transfer/operations/endpoint/index.md#create):

    * **Database type**: `Object Storage`.
    * **Endpoint settings**:

        * **Connection settings**:

            * **Bucket**: Enter the Object Storage bucket name.
            * **Service account**: Select an existing service account or create a new one with the `storage.uploader` role.

        * **Serialization format**: Select `JSON` or `CSV` if you have enabled **Conversion rules** in the advanced settings for the source endpoint.

1. Create a transfer:

    {% list tabs group=instructions %}

    - Manually {#manual}

        1. [Create a transfer](../../data-transfer/operations/transfer.md#create) of the **Replication** type that will use the endpoints you created.
        1. [Activate](../../data-transfer/operations/transfer.md#activate) the transfer.

    - Terraform {#tf}

        1. In the `data-transfer-yds-obj.tf` file, uncomment the following:

            * Variables:

                * `source_endpoint_id`, and set it to the source endpoint ID.
                * `target_endpoint_id`, and set it to the target endpoint ID.

            * `yandex_datatransfer_transfer` resource.

        1. Make sure the Terraform configuration files are correct using this command:

            ```bash
            terraform validate
            ```

            Terraform will display any configuration errors detected in your files.

        1. Create the required infrastructure:

            1. Run this command to view the planned changes:
            
               ```bash
               terraform plan
               ```
            
               If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.
            
            1. If everything looks correct, apply the changes:
               1. Run this command:
            
                  ```bash
                  terraform apply
                  ```
            
               1. Confirm updating the resources.
               1. Wait for the operation to complete.

            The transfer will be activated automatically upon creation.

    {% endlist %}

## Test the transfer {#verify-transfer}

1. Wait for the transfer status to change to **Replicating**.

1. Make sure the data from the stream in Data Streams has moved to the Object Storage bucket:

    1. In the [management console](https://console.yandex.cloud), select the folder with the bucket.
    1. Navigate to **Object Storage**.
    1. Select the bucket from the list.
    1. Make sure the bucket contains the `<stream_name>_0.raw` file (`.json` or `.csv`, depending on the selected output format) with the test data.

1. [Send a new message to your stream in Data Streams](../operations/aws-cli/send.md):

    ```json
    {
        "device_id": "rhibbh3y08qm********",
        "datetime": "2020-06-06 09:49:54",
        "latitude": 55.71294467,
        "longitude": 37.66542005,
        "altitude": 429.13,
        "speed": 55.5,
        "battery_voltage": null,
        "cabin_temperature": 18,
        "fuel_level": 32
    }
    ```

1. Make sure the Object Storage bucket contains the new data:

    1. In the [management console](https://console.yandex.cloud), select the folder with the bucket.
    1. Navigate to **Object Storage**.
    1. Select the bucket from the list.
    1. Make sure the bucket now contains the `<stream_name>_0-1_1.raw` file (`.json` or `.csv`, depending on the selected output format) with the new data.

## Delete the resources you created {#clear-out}

{% note info %}

Before deleting any resources, [deactivate the transfer](../../data-transfer/operations/transfer.md#deactivate).

{% endnote %}

To minimize resource consumption, delete the resources you no longer need:

1. [Delete the transfer](../../data-transfer/operations/transfer.md#delete).
1. [Delete](../../data-transfer/operations/endpoint/index.md#delete) the source and target endpoints.
1. [Delete the objects](../../storage/operations/objects/delete.md) from the Object Storage bucket.
1. Delete the other resources depending on how you created them:

   {% list tabs group=instructions %}

   - Manually {#manual}

       1. [Delete the Managed Service for YDB database](../../ydb/operations/manage-databases.md#delete-db).
       1. [Delete the bucket in Object Storage](../../storage/operations/buckets/delete.md).
       1. If you created any service accounts when setting up your endpoints, [delete them](../../iam/operations/sa/delete.md).

   - Terraform {#tf}

       1. In the terminal window, go to the directory containing the infrastructure plan.
       
           {% note warning %}
       
           Make sure the directory has no Terraform manifests with the resources you want to keep. Terraform deletes all resources that were created using the manifests in the current directory.
       
           {% endnote %}
       
       1. Delete resources:
       
           1. Run this command:
       
               ```bash
               terraform destroy
               ```
       
           1. Confirm deleting the resources and wait for the operation to complete.
       
           All the resources described in the Terraform manifests will be deleted.

   {% endlist %}