[Yandex Cloud documentation](../../index.md) > [Yandex Managed Service for Apache Kafka®](../index.md) > [Tutorials](index.md) > Moving data from Apache Kafka® > Migrating a database from a third-party Apache Kafka® cluster

# Migrating a database from a third-party Apache Kafka® cluster


There are two ways to migrate topics from an Apache Kafka® _source cluster_ to a Managed Service for Apache Kafka® _target cluster_:

* [Using the built-in Yandex Managed Service for Apache Kafka® MirrorMaker connector](#kf-connector).

  This method is easy to configure and does not require creating an intermediate VM.

* [Using MirrorMaker 2.0](#kf-mirrormaker).

  This requires setting up the utility manually on an intermediate virtual machine. Use this method only if it is not possible to migrate data using the built-in MirrorMaker connector for whatever reason.

Both methods are also suitable for [migrating](../operations/host-migration.md#one-host) a single-host Managed Service for Apache Kafka® cluster to a different availability zone.


## Migrating data using Yandex Managed Service for Apache Kafka® Connector {#kf-connector}

To transfer data using Yandex Managed Service for Apache Kafka® Connector:

1. [Prepare the source cluster](#prepare-source).
1. [Create a target cluster and a connector](#create-cluster-connector).
1. [Check the target cluster topic for data](#check-data-mkf-connector).

If you no longer need the resources you created, [delete them](#clear-out-connector).


### Getting started {#before-you-begin}

Sign up for Yandex Cloud and create a [billing account](../../billing/concepts/billing-account.md):
1. Navigate to the [management console](https://console.yandex.cloud) and log in to Yandex Cloud or create a new account.
1. On the **[Yandex Cloud Billing](https://center.yandex.cloud/billing/accounts)** page, make sure you have a billing account linked and it has the `ACTIVE` or `TRIAL_ACTIVE` [status](../../billing/concepts/billing-account-statuses.md). If you do not have a billing account, [create one](../../billing/quickstart/index.md) and [link](../../billing/operations/pin-cloud.md) a cloud to it.

If you have an active billing account, you can create or select a [folder](../../resource-manager/concepts/resources-hierarchy.md#folder) for your infrastructure on the [cloud page](https://console.yandex.cloud/cloud).

[Learn more about clouds and folders here](../../resource-manager/concepts/resources-hierarchy.md).

##### **Required paid resources** {#paid-resources-connector}

* Managed Service for Apache Kafka® cluster: use of computing resources allocated to hosts, storage and backup size (see [Managed Service for Apache Kafka® pricing](../pricing.md)).
* Public IP addresses if public access is enabled for cluster hosts (see [Virtual Private Cloud pricing](../../vpc/pricing.md)).


### Prepare the source cluster {#prepare-source}

1. Create the `admin-source` user and assign them the `ACCESS_ROLE_ADMIN` role for all topics (`*`).
1. Make sure the source cluster’s network settings allow cluster connections from the internet.

### Create a target cluster and a connector {#create-cluster-connector}

{% list tabs group=instructions %}

- Manually {#manual}

    1. [Create a target cluster](../operations/cluster-create.md).
    1. Set up the target cluster:
        1. Create a [user](../operations/cluster-accounts.md#create-account) named `admin-cloud`.
        1. [Create a topic](../operations/cluster-topics.md#create-topic) in any configuration. You will only need it to configure user access to topics.
        1. [Assign](../operations/cluster-accounts.md#grant-permission) to the user the `ACCESS_ROLE_ADMIN` role for all topics (`*`).
        1. Enable the [Auto create topics enable](../concepts/settings-list.md#settings-auto-create-topics) property.
        1. Configure [security groups](../operations/connect/index.md#configuring-security-groups) to connect to the target cluster.

    1. [For the target cluster, create a connector](../operations/cluster-connector.md#create-connector) of the `MirrorMaker` type, configured as follows:

        * **Topics**: List of topics to migrate. You can also specify a regular expression for selecting topics. To migrate all topics, specify `.*`.
        * Under **Source cluster**, specify the parameters for connecting to the source cluster:
          * **Alias**: Source cluster prefix in the connector settings. The default value is `source`. Topics in the target cluster will be created with the specified prefix.
          * **Bootstrap servers**: Comma-separated list of the FQDNs of the source cluster broker hosts with the port numbers, such as follows:

              ```text
              FQDN_1:9091,FQDN_2:9091,...,FQDN_N:9091
              ```

          * **SASL mechanism**: Authentication mechanism for username and password validation, `SCRAM-SHA-512`.
          * **SASL username** and **SASL password**: Username and password of the previously created `admin-source` user.
          * **Security protocol**: Select the connection protocol for the connector:
            * `SASL_PLAINTEXT`: For connecting to the source cluster without SSL.
            * `SASL_SSL`: For SSL connections to the source cluster.

        * Under **Target cluster**, select **Use this cluster**.


- Terraform {#tf}

    1. If you do not have Terraform yet, [install it](../../tutorials/infrastructure-management/terraform-quickstart.md#install-terraform).
    1. [Get the authentication credentials](../../tutorials/infrastructure-management/terraform-quickstart.md#get-credentials). You can add them to environment variables or specify them later in the provider configuration file.
    1. [Configure and initialize a provider](../../tutorials/infrastructure-management/terraform-quickstart.md#configure-provider). There is no need to create a provider configuration file manually, you can [download it](https://github.com/yandex-cloud-examples/yc-terraform-provider-settings/blob/main/provider.tf).
    1. Place the configuration file in a separate working directory and [specify the parameter values](../../tutorials/infrastructure-management/terraform-quickstart.md#configure-provider). If you did not add the authentication credentials to environment variables, specify them in the configuration file.

    1. Download the [kafka-mirrormaker-connector.tf](https://github.com/yandex-cloud-examples/yc-kafka-data-migration-from-on-premise/blob/main/kafka-mirrormaker-connector.tf) configuration file to the same working directory.

        This file describes:

        * Network.
        * Subnet.
        * Default security group and inbound internet rules for the cluster.
        * Managed Service for Apache Kafka® target cluster with [Auto create topics enable](../concepts/settings-list.md#settings-auto-create-topics) set to `true`.
        * `admin-cloud` [admin user](../operations/cluster-accounts.md#create-account) for the target cluster.
        * MirrorMaker connector for the target cluster.

    1. In the `kafka-mirrormaker-connector.tf` file, specify the following:

        * Source cluster username and passwords for the source and target cluster users.
        * FQDNs of the source cluster broker hosts.
        * Source and target cluster aliases.
        * Filter pattern for topics to migrate.
        * Apache Kafka® version.

    1. Validate your Terraform configuration files using this command:

        ```bash
        terraform validate
        ```

        Terraform will display any configuration errors detected in your files.

    1. Create the required infrastructure:

        1. Run this command to view the planned changes:
        
           ```bash
           terraform plan
           ```
        
           If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.
        
        1. If everything looks correct, apply the changes:
           1. Run this command:
        
              ```bash
              terraform apply
              ```
        
           1. Confirm updating the resources.
           1. Wait for the operation to complete.

        All the required resources will be created in the specified folder. You can check resource availability and their settings in the [management console](https://console.yandex.cloud).


{% endlist %}

{% note info %}

Once created, the connector is automatically activated and data transfer starts.

{% endnote %}

### Check the target cluster topic for data {#check-data-mkf-connector}

1. In the [management console](https://console.yandex.cloud), open the target cluster.
1. Make sure the migrated topic is displayed on the **Topics** tab.

   A prefix (`source` by default) will be added to the topic name. For example, a topic named `mytopic` will be moved to the target cluster as `source.mytopic`.

1. [Connect](../operations/connect/clients.md#bash-zsh) to the target cluster topic using kafkacat. Add the prefix to the source cluster topic name.
1. Make sure the console displays messages from the source cluster topic.

### Delete the resources you created {#clear-out-connector}

Some resources are not free of charge. Delete the resources you no longer need to avoid paying for them:

{% list tabs group=instructions %}

- Manually {#manual}

    [Delete the Yandex Managed Service for Apache Kafka® cluster](../operations/cluster-delete.md). The connector will be deleted together with the cluster.


- Terraform {#tf}

    1. In the terminal window, go to the directory containing the infrastructure plan.
    
        {% note warning %}
    
        Make sure the directory has no Terraform manifests with the resources you want to keep. Terraform deletes all resources that were created using the manifests in the current directory.
    
        {% endnote %}
    
    1. Delete resources:
    
        1. Run this command:
    
            ```bash
            terraform destroy
            ```
    
        1. Confirm deleting the resources and wait for the operation to complete.
    
        All the resources described in the Terraform manifests will be deleted.


{% endlist %}


## Migrating data via MirrorMaker {#kf-mirrormaker}

To transfer data using MirrorMaker:

1. [Set up your infrastructure](#prepare-infrastructure).
1. [Configure the source cluster and VM](#additional-settings).
1. [Configure MirrorMaker](#configure-mirrormaker).
1. [Start replication](#replication-start).
1. [Check the target cluster topic for data](#check-data-mkf).

If you no longer need the resources you created, [delete them](#clear-out-mm).


### Getting started {#before-you-begin}

Sign up for Yandex Cloud and create a [billing account](../../billing/concepts/billing-account.md):
1. Navigate to the [management console](https://console.yandex.cloud) and log in to Yandex Cloud or create a new account.
1. On the **[Yandex Cloud Billing](https://center.yandex.cloud/billing/accounts)** page, make sure you have a billing account linked and it has the `ACTIVE` or `TRIAL_ACTIVE` [status](../../billing/concepts/billing-account-statuses.md). If you do not have a billing account, [create one](../../billing/quickstart/index.md) and [link](../../billing/operations/pin-cloud.md) a cloud to it.

If you have an active billing account, you can create or select a [folder](../../resource-manager/concepts/resources-hierarchy.md#folder) for your infrastructure on the [cloud page](https://console.yandex.cloud/cloud).

[Learn more about clouds and folders here](../../resource-manager/concepts/resources-hierarchy.md).

##### **Required paid resources** {#paid-resources-mirrormaker}

* Managed Service for Apache Kafka® cluster: use of computing resources allocated to hosts, storage and backup size (see [Managed Service for Apache Kafka® pricing](../pricing.md)).
* Public IP addresses if public access is enabled for cluster hosts (see [Virtual Private Cloud pricing](../../vpc/pricing.md)).
* VM instance: use of computing resources, storage, public IP address, and OS (see [Compute Cloud pricing](../../compute/pricing.md)).


### Set up your infrastructure {#prepare-infrastructure}

{% list tabs group=instructions %}

- Manually {#manual}

  1. [Create a Managed Service for Apache Kafka®](../operations/cluster-create.md) target cluster.
  1. Set up the target cluster:
      * [Create a user](../operations/cluster-accounts.md#create-account) named `admin-cloud`.
      * [Create a topic](../operations/cluster-topics.md#create-topic) in any configuration. You will only need it to configure user access to topics.
      * [Assign](../operations/cluster-accounts.md#grant-permission) to the user the `ACCESS_ROLE_ADMIN` role for all topics (`*`).
      * Enable the [Auto create topics enable](../concepts/settings-list.md#settings-auto-create-topics) property.
      * Configure [security groups](../operations/connect/index.md#configuring-security-groups) to connect to the target cluster.

  1. [Create a new Linux VM](../../compute/operations/vm-create/create-linux-vm.md) for MirrorMaker in the same network as the target cluster. 

      To connect to the VM via the internet:

      * Enable public access when creating the VM.
      * Make sure the VM's security group allows internet connections.

- Terraform {#tf}

  1. If you do not have Terraform yet, [install it](../../tutorials/infrastructure-management/terraform-quickstart.md#install-terraform).
  1. [Get the authentication credentials](../../tutorials/infrastructure-management/terraform-quickstart.md#get-credentials). You can add them to environment variables or specify them later in the provider configuration file.
  1. [Configure and initialize a provider](../../tutorials/infrastructure-management/terraform-quickstart.md#configure-provider). There is no need to create a provider configuration file manually, you can [download it](https://github.com/yandex-cloud-examples/yc-terraform-provider-settings/blob/main/provider.tf).
  1. Place the configuration file in a separate working directory and [specify the parameter values](../../tutorials/infrastructure-management/terraform-quickstart.md#configure-provider). If you did not add the authentication credentials to environment variables, specify them in the configuration file.

  1. Download the [kafka-mirror-maker.tf](https://github.com/yandex-cloud-examples/yc-kafka-data-migration-from-on-premise/blob/main/kafka-mirror-maker.tf) configuration file to the same working directory.

      This file describes:

      * Network.
      * Subnet.
      * Default security group and inbound internet rules for your cluster and VM.
      * Managed Service for Apache Kafka® cluster with [Auto create topics enable](../concepts/settings-list.md#settings-auto-create-topics) set to `true`.
      * Apache Kafka® [administrator user](../operations/cluster-accounts.md#create-account) named `admin-cloud` with the `ACCESS_ROLE_ADMIN` role for all cluster topics.
      * Virtual machine with public internet access.

  1. In `kafka-mirror-maker.tf`, specify the following:

      * Managed Service for Apache Kafka® cluster name.
      * Apache Kafka® admin user password.
      * Public Ubuntu [image](../../compute/operations/images-with-pre-installed-software/get-list.md) ID (non-GPU), e.g., [Ubuntu 24.04 LTS](https://yandex.cloud/en/marketplace/products/yc/ubuntu-24-04-lts).
      * Username and path to the [public key](../../compute/operations/vm-connect/ssh.md#creating-ssh-keys) for VM access.

  1. Validate your Terraform configuration files using this command:

      ```bash
      terraform validate
      ```

      Terraform will display any configuration errors detected in your files.

  1. Create the required infrastructure:

      1. Run this command to view the planned changes:
      
         ```bash
         terraform plan
         ```
      
         If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.
      
      1. If everything looks correct, apply the changes:
         1. Run this command:
      
            ```bash
            terraform apply
            ```
      
         1. Confirm updating the resources.
         1. Wait for the operation to complete.

      All the required resources will be created in the specified folder. You can check resource availability and their settings in the [management console](https://console.yandex.cloud).

{% endlist %}

### Configure the source cluster and VM {#additional-settings}

1. Prepare the source cluster:

    1. In the source cluster, create the `admin-source` user and assign them the `ACCESS_ROLE_ADMIN` role for all topics (`*`).
    1. Enable the `Auto create topics enable` setting.

1. [Connect](../../compute/operations/vm-connect/ssh.md) to the VM over SSH.

    1. Install the JDK:

        ```bash
        sudo apt update && sudo apt install --yes default-jdk
        ```

    1. [Download](https://kafka.apache.org/community/downloads/) and unpack the Apache Kafka® archive with the same version as installed on the target cluster, e.g., for version 3.9:

        ```bash
        wget https://archive.apache.org/dist/kafka/3.9.0/kafka_2.12-3.9.0.tgz && \
        tar -xvf kafka_2.12-3.9.0.tgz
        ```

    1. Download an [SSL certificate](../operations/connect/index.md#get-ssl-cert) for connecting to the Managed Service for Apache Kafka® cluster:

        ```bash
        sudo mkdir -p /usr/local/share/ca-certificates/Yandex && \
        sudo wget "https://storage.yandexcloud.net/cloud-certs/CA.pem" \
            --output-document /usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt && \
        sudo chmod 0655 /usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt
        ```

    1. Install [kafkacat](https://github.com/edenhill/kcat):

        ```bash
        sudo apt update && sudo apt install --yes kafkacat
        ```

1. Configure a firewall and [security groups](../operations/connect/index.md#configuring-security-groups) if required for MirrorMaker connection to the target and source clusters.

### Configure MirrorMaker {#configure-mirrormaker}

1. [Connect to the MirrorMaker VM over SSH](../../compute/operations/vm-connect/ssh.md).
1. In the home directory, create a folder named `mirror-maker` to store Java Keystore certificates and MirrorMaker configuration files:

   ```bash
   mkdir --parents /home/<home_directory>/mirror-maker
   ```

1. Choose a password of at least 6 characters for a certificate store, create the store, and add the SSL certificate for cluster connection:

   ```bash
   sudo keytool --noprompt -importcert -alias YandexCA \
      -file /usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt \
      -keystore /home/<home_directory>/mirror-maker/keystore \
      -storepass <certificate_store_password>
   ```

1. Create a MirrorMaker configuration file named `mm2.properties` in the `mirror-maker` folder:

   ```text
   # Kafka clusters
   clusters=source, cloud
   source.bootstrap.servers=<source_cluster_broker_FQDN>:9091
   cloud.bootstrap.servers=<source_cluster_broker_1_FQDN>:9091, ..., <source_cluster_broker_N_FQDN>:9091

   # Source and target cluster settings
   source->cloud.enabled=true
   cloud->source.enabled=false
   source.cluster.alias=source
   cloud.cluster.alias=cloud

   # Internal topics settings
   source.config.storage.replication.factor=<R>
   source.status.storage.replication.factor=<R>
   source.offset.storage.replication.factor=<R>
   source.offsets.topic.replication.factor=<R>
   source.errors.deadletterqueue.topic.replication.factor=<R>
   source.offset-syncs.topic.replication.factor=<R>
   source.heartbeats.topic.replication.factor=<R>
   source.checkpoints.topic.replication.factor=<R>
   source.transaction.state.log.replication.factor=<R>
   cloud.config.storage.replication.factor=<R>
   cloud.status.storage.replication.factor=<R>
   cloud.offset.storage.replication.factor=<R>
   cloud.offsets.topic.replication.factor=<R>
   cloud.errors.deadletterqueue.topic.replication.factor=<R>
   cloud.offset-syncs.topic.replication.factor=<R>
   cloud.heartbeats.topic.replication.factor=<R>
   cloud.checkpoints.topic.replication.factor=<R>
   cloud.transaction.state.log.replication.factor=<R>

   # Topics
   topics=.*
   groups=.*
   topics.blacklist=.*[\-\.]internal, .*\replica, __consumer_offsets
   groups.blacklist=console-consumer-.*, connect-.*, __.*
   replication.factor=<M>
   refresh.topics.enable=true
   sync.topic.configs.enabled=true
   refresh.topics.interval.seconds=10

   # Tasks
   tasks.max=<T>

   # Source cluster authentication parameters. Comment out if no authentication required
   source.client.id=mm2_consumer_test
   source.group.id=mm2_consumer_group
   source.security.protocol=SASL_PLAINTEXT
   source.sasl.mechanism=SCRAM-SHA-512
   source.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="admin-source" password="<password>";

   # Target cluster authentication parameters
   cloud.client.id=mm2_producer_test
   cloud.group.id=mm2_producer_group
   cloud.ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
   cloud.ssl.truststore.location=/home/<home_directory>/mirror-maker/keystore
   cloud.ssl.truststore.password=<certificate_store_password>
   cloud.ssl.protocol=TLS
   cloud.security.protocol=SASL_SSL
   cloud.sasl.mechanism=SCRAM-SHA-512
   cloud.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="admin-cloud" password="<password>";

   # Enable heartbeats and checkpoints
   source->target.emit.heartbeats.enabled=true
   source->target.emit.checkpoints.enabled=true
   ```

   MirrorMaker configuration notes:

   * It performs one-way replication (`source->cloud.enabled = true`, `cloud->source.enabled = false`).
   * In the `topics` parameter, list the topics you want to migrate. You can also specify a regular expression for selecting topics. To migrate all topics, specify `.*`. This configuration replicates all topics.
   * Topic names in the target cluster cluster match those in the source cluster.
   * `<R>` stands for the [replication factor](../concepts/settings-list.md#settings-topic-replication-factor) for MirrorMaker service topics. Its value should not exceed the lesser of the broker counts in the source and target clusters.
   * `<M>` stands for the [default replication factor](../concepts/settings-list.md#settings-topic-replication-factor) defined for topics in the target cluster.
   * `<T>` stands for the number of concurrent MirrorMaker processes. To distribute replication load evenly, we recommend a value of at least `2`. For more information, see [this Apache Kafka® guide](https://kafka.apache.org/42/operations/geo-replication-cross-cluster-data-mirroring/#configuration-file-syntax).

   You can get the Managed Service for Apache Kafka® broker FQDNs with the [list of hosts in the cluster](../operations/cluster-hosts.md).

### Start replication {#replication-start}

1. [Connect to the MirrorMaker VM over SSH](../../compute/operations/vm-connect/ssh.md).
1. Run MirrorMaker on the VM as follows:

    ```bash
    <Apache_Kafka_installation_path>/bin/connect-mirror-maker.sh /home/<home_directory>/mirror-maker/mm2.properties
    ```

### Check the target cluster topic for data {#check-data-mkf}

1. In the [management console](https://console.yandex.cloud), open the target cluster.
1. Make sure the migrated topic is displayed on the **Topics** tab.

   A prefix (`source` by default) will be added to the topic name. For example, a topic named `mytopic` will be moved to the target cluster as `source.mytopic`.

1. [Connect](../operations/connect/clients.md#bash-zsh) to the target cluster topic using kafkacat. Add the prefix to the source cluster topic name.
1. Make sure the console displays messages from the source cluster topic.

To learn more about MirrorMaker 2.0, see [this Apache Kafka® article](https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0).

### Delete the resources you created {#clear-out-mm}

Some resources are not free of charge. Delete the resources you no longer need to avoid paying for them:

{% list tabs group=instructions %}

- Manually {#manual}

    * [Delete the Yandex Managed Service for Apache Kafka® cluster](../operations/cluster-delete.md).
    * [Delete the VM](../../compute/operations/vm-control/vm-delete.md).
    * If you reserved public static IP addresses, release and [delete them](../../vpc/operations/address-delete.md).

- Terraform {#tf}

    1. In the terminal window, go to the directory containing the infrastructure plan.
    
        {% note warning %}
    
        Make sure the directory has no Terraform manifests with the resources you want to keep. Terraform deletes all resources that were created using the manifests in the current directory.
    
        {% endnote %}
    
    1. Delete resources:
    
        1. Run this command:
    
            ```bash
            terraform destroy
            ```
    
        1. Confirm deleting the resources and wait for the operation to complete.
    
        All the resources described in the Terraform manifests will be deleted.

{% endlist %}