# Replicating logs to Yandex Object Storage using Yandex Data Streams

With [Yandex Data Streams](../../data-streams/index.md), you can set up automatic replication of service and user application logs to [Yandex Object Storage](../../storage/index.md).

The solution works as follows:
1. Your Cloud Logging [log group](../../logging/concepts/log-group.md) receives logs, e.g., from a [VM](../../compute/concepts/vm.md).
1. The log group is set up to automatically forward logs to a specific [data stream](../../data-streams/concepts/glossary.md#stream-concepts) in Data Streams.
1. A Data Transfer [transfer](../concepts/index.md#transfer) is set up to fetch data from the stream and save it to an Object Storage [bucket](../../storage/concepts/bucket.md).

To set up log replication:

1. [Get your cloud ready](#before-you-begin).
1. [Set up your environment](#prepare-environment).
1. [Create a bucket](#create-bucket).
1. [Create a data stream](#create-stream).
1. [Connect the stream to the log group](#stream-log-connect).
1. [Create a transfer](#create-transfer).
1. [Test sending and receiving data](#check-ingestion).

If you no longer want to store logs, [delete the resources allocated to them](#clear-out).

## Getting started {#before-you-begin}

Sign up for Yandex Cloud and create a [billing account](../../billing/concepts/billing-account.md):
1. Navigate to the [management console](https://console.yandex.cloud) and log in to Yandex Cloud or create a new account.
1. On the **[Yandex Cloud Billing](https://center.yandex.cloud/billing/accounts)** page, make sure you have a billing account linked and it has the `ACTIVE` or `TRIAL_ACTIVE` [status](../../billing/concepts/billing-account-statuses.md). If you do not have a billing account, [create one](../../billing/quickstart/index.md) and [link](../../billing/operations/pin-cloud.md) a cloud to it.

If you have an active billing account, you can create or select a [folder](../../resource-manager/concepts/resources-hierarchy.md#folder) for your infrastructure on the [cloud page](https://console.yandex.cloud/cloud).

[Learn more about clouds and folders here](../../resource-manager/concepts/resources-hierarchy.md).


### Required paid resources {#paid-resources}

* Data Streams (see [Data Streams pricing](../../data-streams/pricing.md)). The cost depends on the pricing model:

    * [Based on allocated resources](../../data-streams/pricing.md#rules): You pay a fixed hourly rate for the established throughput limit and message retention period, and additionally for the number of units of actually written data.
    * [On-demand](../../data-streams/pricing.md#on-demand): You pay for the performed read/write operations, the amount of read or written data, and the actual storage used for messages that are still within their retention period.

* Managed Service for YDB database, operating in serverless mode: data operations, amount of stored data and backups (see [Managed Service for YDB pricing](../../ydb/pricing/index.md)).
* Object Storage bucket: use of storage, data operations (see [Object Storage pricing](../../storage/pricing.md)).


## Set up your environment {#prepare-environment}

1. [Create](../../iam/operations/sa/create.md) a service account, e.g., `logs-sa`, with the `editor` [role](../../iam/roles-reference.md#editor) for the folder.
1. [Set up](../../logging/tutorials/index.md) the transfer of logs to the log group. For example, you can [transfer](../../logging/tutorials/vm-fluent-bit-logging.md) logs from a VM or [add](../../logging/operations/write-logs.md) test records to the log group.

## Create a bucket {#create-bucket}

{% list tabs group=instructions %}

- Management console {#console}

  1. In the [management console](https://console.yandex.cloud), select the [folder](../../resource-manager/concepts/resources-hierarchy.md#folder) where you want to create a [bucket](../../storage/concepts/bucket.md).
  1. Navigate to **Object Storage**.
  1. Click **Create bucket**.
  1. Enter a name for the bucket.
  1. In the **Storage class** field, select `Cold`.
  1. Click **Create bucket**.

{% endlist %}

## Create a data stream {#create-stream}

{% list tabs group=instructions %}

- Management console {#console}

  1. In the [management console](https://console.yandex.cloud), select the [folder](../../resource-manager/concepts/resources-hierarchy.md#folder) where you want to create a [data stream](../../data-streams/concepts/glossary.md#stream-concepts).
  1. Navigate to **Data Streams**.
  1. Click **Create stream**.
  1. Specify an existing [serverless](../../ydb/concepts/serverless-and-dedicated.md#serverless) database in YDB or [create](../../ydb/quickstart.md#serverless) a new one. If you have created a new database, click ![refresh-button](../../_assets/data-streams/refresh-button.svg) to update the database list.
  1. Name the data stream, e.g., `logs-stream`.
  1. Click **Create**.

  Wait for the stream to start. Once the stream is ready for use, its status will change from `Creating` to `Active`.

{% endlist %}

## Connect the stream to the log group {#stream-log-connect}

{% list tabs group=instructions %}

- Management console {#console}

  1. In the [management console](https://console.yandex.cloud), select the folder with the log group.
  1. [Go](../../console/operations/select-service.md#select-service) to **Cloud Logging**.
  1. Next to the log group receiving the logs, click ![options](../../_assets/console-icons/ellipsis.svg) and select **Edit**.
  1. In the **Data stream** field, select `logs-stream` you created earlier.
  1. Click **Save**.

{% endlist %}

## Create a transfer {#create-transfer}

{% list tabs group=instructions %}

- Management console {#console}

  1. In the [management console](https://console.yandex.cloud), select the [folder](../../resource-manager/concepts/resources-hierarchy.md#folder) where you want to create a [transfer](../concepts/index.md#transfer).
  1. Navigate to Data Transfer.
  1. Create a source [endpoint](../concepts/index.md#endpoint):
     1. In the ![endpoint](../../_assets/console-icons/aperture.svg) **Endpoints** tab, click **Create endpoint**.
     1. In the **Direction** field, select `Source`.
     1. Enter the endpoint name, e.g., `logs-source`.
     1. From the **Database type** list, select `Yandex Data Streams`.
     1. Select the database you specified in the settings of the [stream](../../data-streams/concepts/glossary.md#stream-concepts) you created earlier.
     1. Name the data stream: `logs-stream`.
     1. Select the `logs-sa` [service account](../../iam/concepts/users/service-accounts.md) you created earlier.
     1. Under **Advanced settings**, specify the conversion rules for the `CloudLogging parser` data.
     1. Click **Create**.
  1. Create a target endpoint:
     1. In the ![endpoint](../../_assets/console-icons/aperture.svg) **Endpoints** tab, click **Create endpoint**.
     1. In the **Direction** field, select `Target`.
     1. Enter the endpoint name, e.g., `logs-receiver`.
     1. From the **Database type** list, select `Object Storage`.
     1. Enter the name of the previously created [bucket](../../storage/concepts/bucket.md).
     1. Select the `logs-sa` service account you created earlier.
     1. In the **Serialization format** field, select `JSON`.
     1. Click **Create**.
  1. Create a transfer:
     1. In the ![image](../../_assets/console-icons/arrow-right-arrow-left.svg) **Transfers** tab, click **Create transfer**.
     1. Enter the transfer name, e.g., `logs-transfer`.
     1. Select the `logs-source` source endpoint you created earlier.
     1. Select the `logs-receiver` target endpoint you created earlier.
     1. Click **Create**.
  1. Click ![ellipsis](../../_assets/console-icons/ellipsis.svg) next to the new transfer and select **Activate**.

  Wait until the transfer gets activated. Once the transfer is ready for use, its [status](../concepts/transfer-lifecycle.md#statuses) will change from `Creating` to `Replicating`.

{% endlist %}

## Test sending and receiving data {#check-ingestion}

{% list tabs group=instructions %}

- Management console {#console}

  1. In the [management console](https://console.yandex.cloud), navigate to the [folder](../../resource-manager/concepts/resources-hierarchy.md#folder) with the new [data stream](../../data-streams/concepts/glossary.md#stream-concepts), [transfer](../concepts/index.md##transfer), and [bucket](../../storage/concepts/bucket.md).
  1. Navigate to **Data Streams**.
  1. Select the data stream named `logs-stream`.
  1. Go to the **Monitoring** tab and check the stream activity charts.
  1. Navigate to Data Transfer.
  1. Select the `logs-transfer` transfer.
  1. Go to the **Monitoring** tab and check the transfer activity charts.
  1. Navigate to **Object Storage**.
  1. Select the previously created bucket.
  1. Make sure that you have objects in the bucket. Download and review the log files you got.

{% endlist %}

## Delete the resources you created {#clear-out}

To reduce the consumption of resources, delete those you do not need:

1. [Delete the transfer](../operations/transfer.md#delete).
1. [Delete the endpoints](../operations/endpoint/index.md#delete).
1. [Delete the data stream](../../data-streams/operations/manage-streams.md#delete-data-stream).
1. [Delete the objects from the bucket](../../storage/operations/objects/delete.md).
1. [Delete the bucket](../../storage/operations/buckets/delete.md).