[Yandex Cloud documentation](../../index.md) > [Yandex Managed Service for Apache Kafka®](../index.md) > [Tutorials](index.md) > Upgrading a Managed Service for Apache Kafka® cluster to migrate from ZooKeeper to KRaft

# Upgrading a Managed Service for Apache Kafka® cluster to migrate from ZooKeeper to KRaft

# Upgrading a Managed Service for Apache Kafka® cluster to migrate from ZooKeeper to KRaft


Managed Service for Apache Kafka® multi-host clusters version 3.5 and lower use ZooKeeper to manage metadata. ZooKeeper support will be [discontinued](../concepts/update-policy.md#version-schedule) starting from Apache Kafka® 4.0. You can migrate clusters with ZooKeeper hosts to the KRaft protocol. Starting with version 3.6, Apache Kafka® uses [KRaft](../concepts/kraft.md) as the main metadata synchronization protocol.

To switch to KRaft in a ZooKeeper cluster:

1. [Upgrade the cluster version](#update-version).
1. [Migrate the cluster to KRaft](#migrate-to-kraft).

If you no longer need the resources you created, [delete them](#clear-out).


## Required paid resources {#paid-resources}

The support cost for this solution includes:

* Managed Service for Apache Kafka® cluster fee: use of computing resources allocated to hosts (including KRaft hosts) and disk space (see [Managed Service for Apache Kafka® pricing](../pricing.md)).
* Fee for the use of public IP addresses for cluster hosts (see [Virtual Private Cloud pricing](../../vpc/pricing.md)).


## Upgrade the cluster version {#update-version}

Upgrade your Apache Kafka® cluster with ZooKeeper to version `3.9` step by step, without skipping any versions in the following order: 3.5 → 3.6 → 3.7 → 3.8 → 3.9. If your cluster’s version is lower than `3.5`, first, [upgrade the cluster](../operations/cluster-version-update.md) to this version.

{% list tabs group=instructions %}

- Management console {#console}

    1. In the [management console](https://console.yandex.cloud), select a folder.
    1. Navigate to **Managed Service for Kafka**.
    1. In the cluster row, click ![image](../../_assets/console-icons/ellipsis.svg) and select **Edit**.
    1. In the **Version** field, select `3.6`.
    1. Click **Save**.
    1. Repeat the steps for the remaining Apache Kafka® versions in the given order.

- CLI {#cli}

    If you do not have the Yandex Cloud CLI yet, [install and initialize it](../../cli/quickstart.md#install).

    The folder used by default is the one specified when [creating](../../cli/operations/profile/profile-create.md) the CLI profile. To change the default folder, use the `yc config set folder-id <folder_ID>` command. You can also specify a different folder for any command using `--folder-name` or `--folder-id`. If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions.

    1. Start the Apache Kafka® upgrade for your cluster with the following command:

        ```bash
        yc managed-kafka cluster update <cluster_name_or_ID> \
           --version=3.6
        ```

    1. Repeat the command for the remaining versions in the given order.

- Terraform {#tf}

    1. Open the current Terraform configuration file describing your infrastructure.
    1. In the `config` section of the Managed Service for Apache Kafka® cluster, specify `3.6` as the new Apache Kafka® version in the `version` field:

        ```hcl
        resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
          ...
          config {
            version = "3.6"
          }
        }
        ```

    1. Make sure the settings are correct.

        1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.
        1. Run this command:
        
           ```bash
           terraform validate
           ```
        
           Terraform will show any errors found in your configuration files.

    1. Confirm updating the resources.

        1. Run this command to view the planned changes:
        
           ```bash
           terraform plan
           ```
        
           If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.
        
        1. If everything looks correct, apply the changes:
           1. Run this command:
        
              ```bash
              terraform apply
              ```
        
           1. Confirm updating the resources.
           1. Wait for the operation to complete.

    1. Repeat the steps for the remaining Apache Kafka® versions in the given order.

- REST API {#api}

    1. [Get an IAM token for API authentication](../api-ref/authentication.md) and put it into an environment variable:

        ```bash
        export IAM_TOKEN="<IAM_token>"
        ```

    1. Call the [Cluster.update](../api-ref/Cluster/update.md) method, e.g., via the following [cURL](https://curl.se/) request:

        ```bash
        curl \
            --request PATCH \
            --header "Authorization: Bearer $IAM_TOKEN" \
            --header "Content-Type: application/json" \
            -url 'https://mdb.api.cloud.yandex.net/managed-kafka/v1/clusters/<cluster_ID>' \
            --data '{
                      "updateMask": "configSpec.version",
                      "configSpec": {
                        "version": "3.6"
                      }
                    }'
        ```

        Where:

        * `updateMask`: Comma-separated string of settings you want to update.

          Here, we only specified a single setting, `configSpec.version`.

        * `configSpec.version`: Apache Kafka® version.

       You can get the cluster ID with the [list of clusters in the folder](../operations/cluster-list.md#list-clusters).

    1. Check the [server response](../api-ref/Cluster/update.md#yandex.cloud.operation.Operation) to make sure your request was successful.
    1. Repeat the steps for the remaining Apache Kafka® versions in the given order.

- gRPC API {#grpc-api}

    1. [Get an IAM token for API authentication](../api-ref/authentication.md) and put it into an environment variable:

        ```bash
        export IAM_TOKEN="<IAM_token>"
        ```

    1. Call the [ClusterService/Update](../api-ref/grpc/Cluster/update.md) method, e.g., via the following [gRPCurl](https://github.com/fullstorydev/grpcurl) request:

        ```bash
        grpcurl \
            -format json \
            -import-path ~/cloudapi/ \
            -import-path ~/cloudapi/third_party/googleapis/ \
            -proto ~/cloudapi/yandex/cloud/mdb/kafka/v1/cluster_service.proto \
            -rpc-header "Authorization: Bearer $IAM_TOKEN" \
            -d '{
                  "cluster_id": "<cluster_ID>",
                  "update_mask": {
                    "paths": [
                      "config_spec.version"
                    ]
                  },
                  "config_spec": {
                    "version": "3.6"
                  }
                }' \
            mdb.api.cloud.yandex.net:443 \
            yandex.cloud.mdb.kafka.v1.ClusterService.Update
        ```

        Where:

        * `update_mask`: List of settings to update as an array of strings (`paths[]`).

          Here, we only specified a single setting, `config_spec.version`.

        * `config_spec.version`: Apache Kafka® version.

        You can get the cluster ID with the [list of clusters in the folder](../operations/cluster-list.md#list-clusters).

    1. Check the [server response](../api-ref/grpc/Cluster/update.md#yandex.cloud.operation.Operation) to make sure your request was successful.
    1. Repeat the steps for the remaining Apache Kafka® versions in the given order.

{% endlist %}

## Migrate the cluster to KRaft {#migrate-to-kraft}

To migrate a Managed Service for Apache Kafka® cluster with ZooKeeper hosts to the KRaft protocol, configure resources for the KRaft controllers.

{% list tabs group=instructions %}

- Management console {#console}

    1. In the [management console](https://console.yandex.cloud), select a folder.
    1. Navigate to **Managed Service for&nbsp;Kafka**.
    1. Click the cluster name.
    1. At the top of the screen, click **Migrate**.
    1. Select the [platform](../../compute/concepts/vm-platforms.md), host type, and host class for the KRaft controllers.
    1. Click **Save**.
    1. Wait for the migration to complete.

- CLI {#cli}

    Run this command to start the cluster migration:

     ```bash
     yc managed-kafka cluster update <cluster_name_or_ID> \
        --controller-resource-preset "<KRaft_host_class>" \
        --controller-disk-size <storage_size> \
        --controller-disk-type <disk_type>
     ```

     Where:

     * `--controller-resource-preset`: [KRaft host class](../concepts/instance-types.md).
     * `--controller-disk-type`: Disk type of KRaft hosts.

     {% note info %}
     
     For KRaft controllers:
     
     * Only the `network-ssd` and `network-ssd-nonreplicated` disk types are available.
     * The Intel Broadwell platform is not available.
     
     {% endnote %}

     To find out the cluster name or ID, [get the list of clusters in the folder](../operations/cluster-list.md#list-clusters).

- Terraform {#tf}

    1. Open the current Terraform configuration file with the infrastructure plan.
    1. Delete the `config.zookeeper` section for the Managed Service for Apache Kafka® cluster.
    1. Add the `config.kraft` section with the KRaft controller resource description:

        ```hcl
        resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
          ...
          config {
            ...
            kraft {
              resources {
                disk_size          = <storage_size_in_GB>
                disk_type_id       = "<disk_type>"
                resource_preset_id = "<KRaft_host_class>"
              }
            }
          }
        }
        ```

        Where:

        * `kraft.resources.resource_preset_id`: [KRaft host class](../concepts/instance-types.md).
        * `kraft.resources.disk_type_id`: Disk type of KRaft hosts.

        {% note info %}
        
        For KRaft controllers:
        
        * Only the `network-ssd` and `network-ssd-nonreplicated` disk types are available.
        * The Intel Broadwell platform is not available.
        
        {% endnote %}

    1. Make sure the settings are correct.

        1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.
        1. Run this command:
        
           ```bash
           terraform validate
           ```
        
           Terraform will show any errors found in your configuration files.

    1. Confirm updating the resources.

        1. Run this command to view the planned changes:
        
           ```bash
           terraform plan
           ```
        
           If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.
        
        1. If everything looks correct, apply the changes:
           1. Run this command:
        
              ```bash
              terraform apply
              ```
        
           1. Confirm updating the resources.
           1. Wait for the operation to complete.

- REST API {#api}

    1. [Get an IAM token for API authentication](../api-ref/authentication.md) and put it into an environment variable:

        ```bash
        export IAM_TOKEN="<IAM_token>"
        ```

    1. Call the [Cluster.update](../api-ref/Cluster/update.md) method, e.g., via the following [cURL](https://curl.se/) request:

        ```bash
        curl \
            --request PATCH \
            --header "Authorization: Bearer $IAM_TOKEN" \
            --header "Content-Type: application/json" \
            -url 'https://mdb.api.cloud.yandex.net/managed-kafka/v1/clusters/<cluster_ID>' \
            --data '{
                      "updateMask": "configSpec.kraft.resources.resourcePresetId,configSpec.kraft.resources.diskSize,configSpec.kraft.resources.diskTypeId",
                      "configSpec": {
                        "kraft": {
                          "resources": {
                            "resourcePresetId": "<KRaft_host_class>",
                            "diskSize": "<storage_size_in_bytes>",
                            "diskTypeId": "<disk_type>"
                          }
                        }
                      }
                    }'
        ```

        Where:

        * `updateMask`: Comma-separated string of settings to update.

          Here, you need to specify all settings of the resources you want to add: `configSpec.kraft.resources.resourcePresetId`, `configSpec.kraft.resources.diskSize`, and `configSpec.kraft.resources.diskTypeId`.

        * `configSpec.kraft`: KRaft controller configuration:

            * `resources.resourcePresetId`: Host class ID. You can get the list of available host classes with their IDs using the [ResourcePreset.list](../api-ref/ResourcePreset/list.md) method.
            * `resources.diskSize`: Disk size, in bytes.
            * `resources.diskTypeId`: Disk type.

            {% note info %}
            
            For KRaft controllers:
            
            * Only the `network-ssd` and `network-ssd-nonreplicated` disk types are available.
            * The Intel Broadwell platform is not available.
            
            {% endnote %}

       You can get the cluster ID with the [list of clusters in the folder](../operations/cluster-list.md#list-clusters).

    1. Check the [server response](../api-ref/Cluster/update.md#yandex.cloud.operation.Operation) to make sure your request was successful.

- gRPC API {#grpc-api}

    1. [Get an IAM token for API authentication](../api-ref/authentication.md) and put it into an environment variable:

        ```bash
        export IAM_TOKEN="<IAM_token>"
        ```

    1. Call the [ClusterService/Update](../api-ref/grpc/Cluster/update.md) method, e.g., via the following [gRPCurl](https://github.com/fullstorydev/grpcurl) request:

        ```bash
        grpcurl \
            -format json \
            -import-path ~/cloudapi/ \
            -import-path ~/cloudapi/third_party/googleapis/ \
            -proto ~/cloudapi/yandex/cloud/mdb/kafka/v1/cluster_service.proto \
            -rpc-header "Authorization: Bearer $IAM_TOKEN" \
            -d '{
                  "cluster_id": "<cluster_ID>",
                  "update_mask": {
                    "paths": [
                      "config_spec.kraft.resources.resource_preset_id",
                      "config_spec.kraft.resources.disk_size",
                      "config_spec.kraft.resources.disk_type_id"
                    ]
                  },
                  "config_spec": {
                    "kraft": {
                      "resources": {
                        "resource_preset_id": "<KRaft_host_class>",
                        "disk_size": "<storage_size_in_bytes>",
                        "disk_type_id": "<disk_type>"
                      }
                    }
                  }
                }' \
            mdb.api.cloud.yandex.net:443 \
            yandex.cloud.mdb.kafka.v1.ClusterService.Update
        ```

        Where:

        * `update_mask`: List of settings to update as an array of strings (`paths[]`).

          Here, you need to specify all settings of the resources you want to add: `config_spec.kraft.resources.resource_preset_id`, `config_spec.kraft.resources.disk_size`, and `config_spec.kraft.resources.disk_type_id`.

        * `config_spec.kraft`: KRaft controller configuration:

            * `resources.resource_preset_id`: Host class ID. You can get the list of available host classes with their IDs using the [ResourcePreset.list](../api-ref/grpc/ResourcePreset/list.md) method.
            * `resources.disk_size`: Disk size, in bytes.
            * `resources.disk_type_id`: Disk type.

            {% note info %}
            
            For KRaft controllers:
            
            * Only the `network-ssd` and `network-ssd-nonreplicated` disk types are available.
            * The Intel Broadwell platform is not available.
            
            {% endnote %}

        You can get the cluster ID with the [list of clusters in the folder](../operations/cluster-list.md#list-clusters).

    1. Check the [server response](../api-ref/grpc/Cluster/update.md#yandex.cloud.operation.Operation) to make sure your request was successful.

{% endlist %}

## Delete the resources you created {#clear-out}

Some resources are not free of charge. Delete the resources you no longer need to avoid paying for them:

{% list tabs group=instructions %}

- Manually {#manual}

    [Delete the Managed Service for Apache Kafka® cluster](../operations/cluster-delete.md).

- Terraform {#tf}

    1. In the terminal window, go to the directory containing the infrastructure plan.
    
        {% note warning %}
    
        Make sure the directory has no Terraform manifests with the resources you want to keep. Terraform deletes all resources that were created using the manifests in the current directory.
    
        {% endnote %}
    
    1. Delete resources:
    
        1. Run this command:
    
            ```bash
            terraform destroy
            ```
    
        1. Confirm deleting the resources and wait for the operation to complete.
    
        All the resources described in the Terraform manifests will be deleted.

{% endlist %}