[Yandex Cloud documentation](../../index.md) > [Yandex Virtual Private Cloud](../index.md) > [Tutorials](index.md) > Connecting to services > Synchronizing Apache Kafka® topics in Object Storage with no web access

# Syncing data from Apache Kafka® topics to an Object Storage bucket without using the internet

# Syncing data from Apache Kafka® topics to a Yandex Object Storage bucket without using the internet


You can synchronize data from Apache Kafka® topics to a Yandex Object Storage bucket without using the internet via a service connection in the user network hosting the Managed Service for Apache Kafka® cluster. Proceed as follows:

1. [Send data to the topic](#send-data).
1. [Make sure the bucket is not accessible from the external network](#check-bucket-access).
1. [Check if there is data in the bucket](#check-bucket-data).

If you no longer need the resources you created, [delete them](#clear-out).


## Required paid resources {#paid-resources}

The support cost for this solution includes:

* Object Storage bucket fee: data storage and data operations (see [Object Storage pricing](../../storage/pricing.md)).
* Managed Service for Apache Kafka® cluster fee: use of computing resources allocated to hosts and disk space (see [Managed Service for Apache Kafka® pricing](../../managed-kafka/pricing.md)).
* Fee for the use of public IP addresses for cluster hosts (see [Virtual Private Cloud pricing](../pricing.md)).


## Getting started {#before-you-begin}


1. Set up your infrastructure:

    {% list tabs group=resources %}

    - Manually {#manual}

        1. [Create a network](../operations/network-create.md) named `my-private-network`. Disable **Create subnets** when creating it.
        1. [Create a subnet](../operations/subnet-create.md) in any availability zone.
        1. [Create a service connection to Object Storage](../operations/private-endpoint-create.md) in `my-private-network` and write down its ID.
        1. [Create a service account](../../iam/operations/sa/create.md#create-sa) named `storage-pe-admin` with the `storage.admin` role. The Managed Service for Apache Kafka® cluster will use it to access the bucket.
        1. [Create a static access key](../../iam/operations/authentication/manage-access-keys.md#create-access-key) for the `storage-pe-admin` service account.
        1. [Create a Yandex Object Storage bucket](../../storage/operations/buckets/create.md) and configure its bucket policy:

            * **Result**: `Allow`.
            * **Action**: `All actions`.
            * **Resource**: `<bucket_name>` and `<bucket_name>/*`.
            * **Condition**: Select the `yc:private-endpoint-id` key from the list and specify the ID of the created service connection as its value.

        1. [Create a Managed Service for Apache Kafka® cluster](../../managed-kafka/operations/cluster-create.md) of any suitable [configuration](../../managed-kafka/concepts/instance-types.md) with the following settings:

            * Network: `my-private-network`.
            * Broker hosts: One
            * Access to cluster hosts: Public

                {% note info %}
                
                Public access to cluster hosts is required if you plan to connect to the cluster via the internet. This connection option is simpler and is recommended for the purposes of this guide. You can connect to non-public hosts as well but only from Yandex Cloud virtual machines located in the same cloud network as the cluster.
                
                {% endnote %}

        
        1. If you are using security groups in a cluster, make sure they are [configured correctly](../../managed-kafka/operations/connect/index.md#configuring-security-groups) and allow connections to it.


        1. [In the cluster, create a topic](../../managed-kafka/operations/cluster-topics.md#create-topic) named `my-private-topic`.
        1. [In the cluster, create a user](../../managed-kafka/operations/cluster-accounts.md#create-account) named `mkf-user` with the `ACCESS_ROLE_PRODUCER` access permission for the new topic.
        1. [In the cluster, create a connector](../../managed-kafka/operations/cluster-connector.md#create) with the following settings:

            * Under **Additional properties**, specify the connector properties:
               * `key.converter`: `org.apache.kafka.connect.storage.StringConverter`
               * `value.converter`: `org.apache.kafka.connect.converters.ByteArrayConverter`
               * `format.output.fields.value.encoding`: `none`
            * Select the **S3 Sink** connector type.
            * In the **Topics** field, specify `my-private-topic`.
            * Under **S3 connection**, specify the following settings:
               * **Bucket**: Bucket you created earlier.
               * **Endpoint**: `storage.pe.yandexcloud.net`.
               * **Access key ID**, **Secret access key**: ID and secret key of the previously created static access key.

        1. [Create a VM](../../compute/operations/vm-create/create-linux-vm.md) with a public IP address in the created `my-private-network` to connect to the bucket.

    - Terraform {#tf}

        1. If you do not have Terraform yet, [install it](../../tutorials/infrastructure-management/terraform-quickstart.md#install-terraform).
        1. [Get the authentication credentials](../../tutorials/infrastructure-management/terraform-quickstart.md#get-credentials). You can add them to environment variables or specify them later in the provider configuration file.
        1. [Configure and initialize a provider](../../tutorials/infrastructure-management/terraform-quickstart.md#configure-provider). There is no need to create a provider configuration file manually, you can [download it](https://github.com/yandex-cloud-examples/yc-terraform-provider-settings/blob/main/provider.tf).
        1. Place the configuration file in a separate working directory and [specify the parameter values](../../tutorials/infrastructure-management/terraform-quickstart.md#configure-provider). If you did not add the authentication credentials to environment variables, specify them in the configuration file.

        1. Download the [kafka-objstorage-sync-private-network.tf](https://github.com/yandex-cloud-examples/yc-sync-kafka-to-s3-private-endpoint/blob/main/kafka-objstorage-sync-private-network.tf) configuration file to the same working directory.

            This file describes:

            * [Network](../concepts/network.md#network).
            * [Subnet](../concepts/network.md#subnet).
            * [Service connection](../concepts/private-endpoint.md).
            * [Security group](../concepts/security-groups.md) required for cluster connection.
            * Service account for creating and accessing the bucket.
            * Object Storage bucket.
            * Managed Service for Apache Kafka® cluster.
            * Apache Kafka® topic.
            * Apache Kafka® user.
            * Apache Kafka® connector.
            * VM to read data from the bucket.

        1. In the `kafka-objstorage-sync-private-network.tf` file, specify the following:

            * `tf_account_name`: Service account name, the same as the one in the provider settings.
            * `bucket_name`: Bucket name consistent with the [naming conventions](../../storage/concepts/bucket.md#naming).
            * `mkf_version`: Apache Kafka® version.
            * `mkf_user_password`: Apache Kafka® password.
            * `vm_image_id`: ID of the public VM [image](../../compute/operations/images-with-pre-installed-software/get-list.md).
            * `vm_username` and `vm_ssh_key`: Username and absolute path to the [public key](../../compute/operations/vm-connect/ssh.md#creating-ssh-keys), for access to the VM.

        1. Validate your Terraform configuration files using this command:

            ```bash
            terraform validate
            ```

            Terraform will display any configuration errors detected in your files.

        1. Create the required infrastructure:

            1. Run this command to view the planned changes:
            
               ```bash
               terraform plan
               ```
            
               If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.
            
            1. If everything looks correct, apply the changes:
               1. Run this command:
            
                  ```bash
                  terraform apply
                  ```
            
               1. Confirm updating the resources.
               1. Wait for the operation to complete.

            All the required resources will be created in the specified folder. You can check resource availability and their settings in the [management console](https://console.yandex.cloud).

    {% endlist %}

1. Make sure Yandex Cloud DNS now has the `*.storage.pe.yandexcloud.net` record in the `.` [service zone](../../dns/concepts/dns-zone.md#service-zones) of the new network.
1. Install [kafkacat](https://github.com/edenhill/kcat) to write data to the Apache Kafka® topic.

    ```bash
    sudo apt update && sudo apt install --yes kafkacat
    ```

    Make sure you can use it to [connect to the previously created Managed Service for Apache Kafka® cluster over SSL](../../managed-kafka/operations/connect/clients.md#bash-zsh).

## Send data to the topic {#send-data}

1. Send test data to `my-private-topic` using `kafkacat`:

   ```bash
   for i in $(echo {1..50})
       do
          echo "test message_"$i | kafkacat -P \
             -b <broker_host_FQDN>:9091 \
             -t my-private-topic \
             -k key \
             -X security.protocol=SASL_SSL \
             -X sasl.username="<username_in_source_cluster>" \
             -X sasl.password="<user_password_in_source_cluster>" \
             -X ssl.ca.location=/usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt -Z
       done
   ```

1. Make sure data is replicated to the bucket. To do this, [open the bucket upload statistics](../../storage/operations/buckets/get-stats.md#storage-ui) and check the **Modify Requests** chart for requests.

## Make sure the bucket is not accessible from the external network {#check-bucket-access}

1. If you do not have the AWS CLI yet, [install and configure it](../../storage/tools/aws-cli.md).
1. Run this command to get a list of bucket objects at a public network address:

   ```bash
   aws s3 ls s3://<bucket_name> \
       --endpoint-url=https://storage.yandexcloud.net \
       --recursive
   ```

   Result:

   ```text
   An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied
   ```

1. Run this command to get a list of bucket objects at a private network address:

   ```bash
   aws s3 ls s3://<bucket_name> \
       --endpoint-url=https://storage.pe.yandexcloud.net \
       --recursive
   ```

   Result:

   ```text
   Could not connect to the endpoint URL: "https://storage.pe.yandexcloud.net/<bucket_name>?list-type=2&prefix=&encoding-type=url"
   ```

## Check if there is data in the bucket via the service connection {#check-bucket-data}

[Connect](../../compute/operations/vm-connect/ssh.md) to the previously created VM over SSH and do the following:

1. [Install and configure the AWS CLI](../../storage/tools/aws-cli.md).
1. Run this command to get a list of bucket objects at a private network address:

    ```bash
    aws s3 ls s3://<bucket_name> \
        --endpoint-url=https://storage.pe.yandexcloud.net \
        --recursive
    ```

    Result:

    ```text
    2025-08-01 14:38:23         20 my-private-topic-1-0
    2025-08-01 14:38:23        587 my-private-topic-1-1
    2025-08-01 14:38:24        440 my-private-topic-1-29
    ```

This result means Apache Kafka® topic data has been successfully synchronized via the service connection.

## Delete the resources you created {#clear-out}

Some resources are not free of charge. Delete the resources you no longer need to avoid paying for them:

{% list tabs group=resources %}

- Manually {#manual}

    * [Delete the Managed Service for Apache Kafka® cluster](../../managed-kafka/operations/cluster-delete.md).
    * [Delete the Object Storage bucket](../../storage/operations/buckets/delete.md). Before deleting the bucket, [delete](../../storage/operations/objects/delete.md) all its objects.
    * [Delete the VM](../../compute/operations/vm-control/vm-delete.md).

- Terraform {#tf}

    First, [delete](../../storage/operations/objects/delete.md) all objects from the bucket you [created earlier](#before-you-begin).

    1. In the terminal window, go to the directory containing the infrastructure plan.
    
        {% note warning %}
    
        Make sure the directory has no Terraform manifests with the resources you want to keep. Terraform deletes all resources that were created using the manifests in the current directory.
    
        {% endnote %}
    
    1. Delete resources:
    
        1. Run this command:
    
            ```bash
            terraform destroy
            ```
    
        1. Confirm deleting the resources and wait for the operation to complete.
    
        All the resources described in the Terraform manifests will be deleted.

    {% endlist %}