[Yandex Cloud documentation](../../index.md) > [Tutorials](../index.md) > [Serverless technologies](index.md) > Serverless-based ETL > Ingesting data into storage systems

# Ingesting data into storage systems


Data from mobile phones, various smart devices, or external services can come in massive numbers of small batches. The communication circuits used for transmission are often slow, and the communication time, limited. Yandex Data Streams receives data arriving at high frequency and speed and forms outbound data batches for the target systems, thus maintaining optimal operating modes for sources and targets. Using an API gateway to receive messages enables you to implement a custom data transmission protocol.

In this use case, the [API gateway](../../api-gateway/concepts/index.md) receives incoming data and sends it to the [data stream](../../data-streams/concepts/index.md). The stream buffers the data and sends it to the [ClickHouse®](../../managed-clickhouse/concepts/index.md) database cluster through a [transfer](../../data-transfer/concepts/index.md).

To set up data ingestion:

1. [Get your cloud ready](#before-you-begin).
1. [Set up your environment](#setup).
1. [Create a ClickHouse® cluster](#create-cluster).
1. [Create a data stream](#create-stream).
1. [Create an API gateway](#create-api-gw).
1. [Create a transfer](#create-transfer).
1. [Test sending and receiving data](#test-ingestion).

If you no longer need to ingest data, [delete the associated resources](#clear-out).

## Get your cloud ready {#before-you-begin}

Sign up for Yandex Cloud and create a [billing account](../../billing/concepts/billing-account.md):
1. Navigate to the [management console](https://console.yandex.cloud) and log in to Yandex Cloud or create a new account.
1. On the **[Yandex Cloud Billing](https://center.yandex.cloud/billing/accounts)** page, make sure you have a billing account linked and it has the `ACTIVE` or `TRIAL_ACTIVE` [status](../../billing/concepts/billing-account-statuses.md). If you do not have a billing account, [create one](../../billing/quickstart/index.md) and [link](../../billing/operations/pin-cloud.md) a cloud to it.

If you have an active billing account, you can create or select a [folder](../../resource-manager/concepts/resources-hierarchy.md#folder) for your infrastructure on the [cloud page](https://console.yandex.cloud/cloud).

[Learn more about clouds and folders here](../../resource-manager/concepts/resources-hierarchy.md).


### Required paid resources {#paid-resources}

* API gateway: Number of API gateway requests (see [Yandex API Gateway pricing](../../api-gateway/pricing.md)).
* Data Streams (see [Data Streams pricing](../../data-streams/pricing.md)). The cost depends on the pricing model:

    * [Based on allocated resources](../../data-streams/pricing.md#rules): You pay a fixed hourly rate for the established throughput limit and message retention period, and additionally for the number of units of actually written data.

    * [On-demand](../../data-streams/pricing.md#on-demand): You pay for the performed read/write operations, the amount of read or written data, and the actual storage used for messages that are still within their retention period.

* Managed Service for YDB database, operating in serverless mode: data operations, amount of stored data and backups (see [Managed Service for YDB pricing](../../ydb/pricing/index.md)).
* Managed Service for ClickHouse® cluster: use of computing resources allocated to hosts, storage and backup size (see [Managed Service for ClickHouse® pricing](../../managed-clickhouse/pricing.md)).
* Public IP addresses if public access is enabled for cluster hosts (see [Virtual Private Cloud pricing](../../vpc/pricing.md)).
* Each transfer: use of computing resources and the number of transferred data rows (see [Yandex Data Transfer pricing](../../data-transfer/pricing.md)).


## Set up your environment {#setup}

[Create](../../iam/operations/sa/create.md) a service account and [assign](../../iam/operations/sa/assign-role-for-sa.md) it the `editor` role for your folder.

## Create a ClickHouse® cluster {#create-cluster}

1. In the [management console](https://console.yandex.cloud), select the folder where you want to create your database cluster.
1. Navigate to **Managed Service for ClickHouse**.
1. Click **Create cluster**.
1. Configure your ClickHouse® cluster:
    1. Under **DBMS settings**, specify the database name and a user name along with their password.
    1. Under **Basic parameters**:
      * Enter a name for the cluster.
      * Select the service account you created earlier.
    1. Under **Hosts**, click ![pencil](../../_assets/pencil.svg). Enable the **Public access** option and click **Save**.

        {% note info %}
        
        Public access to cluster hosts is required if you plan to connect to the cluster via the internet. This connection option is simpler and is recommended for the purposes of this guide. You can connect to non-public hosts as well but only from Yandex Cloud virtual machines located in the same cloud network as the cluster.
        
        {% endnote %}
    1. Specify the remaining cluster settings by following [this guide](../../managed-clickhouse/operations/cluster-create.md).
1. Click **Create cluster**.

Wait for the cluster to start. When the cluster is ready for use, its status will change to `Alive`.

## Create a data stream {#create-stream}

1. In the [management console](https://console.yandex.cloud), select the folder where you want to create a data stream.
1. Navigate to **Data Streams**.
1. Click **Create stream**.
1. Specify an existing [serverless](../../ydb/concepts/serverless-and-dedicated.md#serverless) database in YDB or [create](../../ydb/quickstart.md#serverless) a new one. If you chose to create a new database, click ![refresh-button](../../_assets/data-streams/refresh-button.svg) after creating it to refresh the list of databases.
1. Name the data stream.
1. Click **Create**.

Wait for the stream to start. When the stream is ready for use, its status will change from `CREATING` to `ACTIVE`.

## Create an API gateway {#create-api-gw}

1. On the page of the stream you created, click **Actions** and select **API Gateway**.
1. Name your API gateway.
1. In the **Specification** field, replace the `service_account_id` key value with the ID of the service account you created earlier.

    Save the **Name** and ***Default domain** values, as you will need them in the next steps.
1. Click **Create**.

Wait for the API gateway to start. When the API gateway is ready for use, its status will change from `CREATING` to `ACTIVE`.

## Create a transfer {#create-transfer}

1. In the [management console](https://console.yandex.cloud), select the folder where you want to create a transfer.
1. Navigate to **Data Transfer**.
1. Click **Create transfer**.
1. Name the transfer.
1. [Create](../../data-transfer/operations/endpoint/index.md#create) a source endpoint:
    1. Name the endpoint.
    1. From the **Database type** list, select `Yandex Data Streams`.
    1. Select a database for the source.
    1. Enter the name of the stream you created earlier.
    1. Select the service account you created earlier.
    1. Click **Create**.
1. [Create](../../data-transfer/operations/endpoint/index.md#create) a target endpoint:
    1. Name the endpoint.
    1. From the **Database type** list, select `ClickHouse`.
    1. Select the MDB cluster you created earlier.
    1. Enter the database name, username, and password of the cluster you created earlier.
    1. Click **Create**.
1. Click **Create**.
1. Click ![ellipsis](../../_assets/horizontal-ellipsis.svg) next to the name of the created transfer and select **Activate**.

Wait until the transfer gets activated. Once the transfer is ready for use, its status will change from **Creating** to **Replicating**.

## Test sending and receiving data {#test-ingestion}

1. Send data to the storage system:

    ```bash
    curl --request POST --data 'test massage' https://<url>/<paths>
    ```

    Where:

    * `<url>`: API gateway **Service domain** you saved earlier.
    * `<paths>`: API gateway **Name** you saved earlier.
1. In the [management console](https://console.yandex.cloud), select the Managed Service for ClickHouse® cluster you created earlier.
1. On the left-hand panel, select **SQL**.
1. Enter the username and password and click **Connect**.
1. In the list, select the previously created database.
1. Select the database table.

If everything is set up properly, the table will show a new entry containing system data and the sent message.

## Delete the resources you created {#clear-out}

To minimize resource consumption, delete the resources you no longer need:

1. [Delete the API gateway](../../api-gateway/operations/api-gw-delete.md).
1. [Delete the transfer](../../data-transfer/operations/transfer.md#delete).
1. [Delete the endpoints](../../data-transfer/operations/endpoint/index.md#delete).
1. [Delete the data stream](../../data-streams/operations/manage-streams.md#delete-data-stream).
1. [Delete the ClickHouse® cluster](../../managed-clickhouse/operations/cluster-delete.md).

_ClickHouse® is a registered trademark of [ClickHouse, Inc](https://clickhouse.com)._