[Yandex Cloud documentation](../../index.md) > [Yandex Managed Service for Apache Kafka®](../index.md) > [Step-by-step guides](index.md) > Clusters > Managing disk space

# Managing disk space in a Managed Service for Apache Kafka® cluster

When the [storage](../concepts/storage.md) usage exceeds 97%, the host automatically switches to read-only mode. To prevent issues with topic writes, use one of the following methods:


* [Set up alerts in Yandex Monitoring](#set-alert) to monitor storage usage.


* [Increase the storage size](#change-disk-size) to automatically disable read-only mode.
* [Set up automatic storage expansion](#disk-size-autoscale) to prevent the disk from running out of free space and the host from switching to read-only mode.


## Setting up alerts in Yandex Monitoring {#set-alert}

1. In the [management console](https://console.yandex.cloud), navigate to the relevant folder.
1. Navigate to **Monitoring**.
1. [Create a notification channel](../../monitoring/operations/alert/create-channel.md).
1. [Create an alert](../../monitoring/operations/alert/create-alert.md) with the following settings:

    1. **Metrics**: Configure the following metric settings:

        * Cloud.
        * Folder.
        * **Managed Service for Kafka**.
        * Managed Service for Apache Kafka® cluster ID.

            You can get the ID with the [list of clusters in the folder](cluster-list.md#list-clusters).

        * `disk.free_bytes` label.

    1. **Alert condition**: Set the condition for free disk space usage to trigger the alert:

        * **Aggregation function**: `Minimum` (metric’s minimum value over the period).
        * **Comparison function**: `Less than or equals`.
        * **Warning**: `95` (95% of the storage size).
        * **Alarm**: `90` (90% of the storage size).
        * **Evaluation window**: Preferred metric update period.
        * **Evaluation delay**: Preferred time shift backward, in seconds. It allows to keep the alert from triggering when multiple metrics are specified and collected at different intervals. Learn more about the calculation delay, see [this Yandex Monitoring guide](../../monitoring/concepts/alerting/alert.md#evaluation-delay).

    1. **Notifications**: Add the notification channel you created earlier.


## Increasing your storage size {#change-disk-size}

Make sure the cloud has enough quota to increase the storage size. Open the cloud's [Quotas](https://console.yandex.cloud/cloud?section=quotas) page and check the **HDD storage capacity** and **SSD storage capacity** lines under **Managed Databases** to make sure there is available quota for storage space.

{% list tabs group=instructions %}

* Management console {#console}

    To increase your cluster storage size:

    1. In the [management console](https://console.yandex.cloud), navigate to the relevant folder.
    1. Navigate to **Managed Service for Kafka**.
    1. In the cluster row, click ![image](../../_assets/console-icons/ellipsis.svg) and select ![pencil](../../_assets/console-icons/pencil.svg) **Edit**.
    1. Edit the settings under **Storage**.

        You cannot change the disk type for an Apache Kafka® cluster once the cluster is created.

    1. Click **Save**.

* CLI {#cli}

    If you do not have the Yandex Cloud CLI yet, [install and initialize it](../../cli/quickstart.md#install).

    The folder used by default is the one specified when [creating](../../cli/operations/profile/profile-create.md) the CLI profile. To change the default folder, use the `yc config set folder-id <folder_ID>` command. You can also specify a different folder for any command using `--folder-name` or `--folder-id`. If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions.

    To increase your host storage size:

    1. View the description of the CLI command for updating a cluster:

        ```bash
        yc managed-kafka cluster update --help
        ```

    1. To resize your broker host storage, run this command:

        ```bash
        yc managed-kafka cluster update <cluster_name_or_ID> \
           --disk-size <storage_size>
        ```

        If you specify no size units, gigabytes are used.

    1. To resize ZooKeeper host storage, run this command:

        ```bash
        yc managed-kafka cluster update <cluster_name_or_ID> \
           --zookeeper-disk-size <disk_size>
        ```

        If you specify no size units, gigabytes are used.

    You cannot change the disk type for an Apache Kafka® cluster once the cluster is created.

* Terraform {#tf}

  To increase your cluster storage size:

    1. Open the current Terraform configuration file with the infrastructure plan.

        For information about creating this file, see [Creating a cluster Apache Kafka®](cluster-create.md).

    1. In the Managed Service for Apache Kafka® cluster description, change the `disk_size` value in the `kafka.resources` and `zookeeper.resources` sections for Apache Kafka® and ZooKeeper hosts, respectively:

        ```hcl
        resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
          ...
          kafka {
            resources {
              disk_size = <storage_size_in_GB>
              ...
            }
            ...
          }
          zookeeper {
            resources {
              disk_size = <storage_size_in_GB>
              ...
            }
          }
        }
        ```

        You cannot change the disk type for an Apache Kafka® cluster once the cluster is created.

    1. Make sure the settings are correct.

        1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.
        1. Run this command:
        
           ```bash
           terraform validate
           ```
        
           Terraform will show any errors found in your configuration files.

    1. Confirm resource changes.

        1. Run this command to view the planned changes:
        
           ```bash
           terraform plan
           ```
        
           If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.
        
        1. If everything looks correct, apply the changes:
           1. Run this command:
        
              ```bash
              terraform apply
              ```
        
           1. Confirm updating the resources.
           1. Wait for the operation to complete.

    For more information, see [this Terraform provider guide](../../terraform/resources/mdb_kafka_cluster.md).

    {% note warning "Timeouts" %}
    
    The Terraform provider limits the time for all operations with the Managed Service for Apache Kafka® cluster to 60 minutes.
    
    Operations exceeding the timeout are aborted.
    
    {% cut "How do I change these limits?" %}
    
    Add the `timeouts` section to your cluster description, such as the following:
    
    ```hcl
    resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
      ...
      timeouts {
        create = "1h30m" # 1 hour 30 minutes
        update = "2h"    # 2 hours
        delete = "30m"   # 30 minutes
      }
    }
    ```
    
    {% endcut %}
    
    {% endnote %}

* REST API {#api}

    1. [Get an IAM token for API authentication](../api-ref/authentication.md) and put it into an environment variable:

        ```bash
        export IAM_TOKEN="<IAM_token>"
        ```

    1. Call the [Cluster.update](../api-ref/Cluster/update.md) method, e.g., via the following [cURL](https://curl.se/) request:

        {% note warning %}
        
        The API method will assign default values to all the parameters of the object you are modifying unless you explicitly provide them in your request. To avoid this, list the settings you want to change in the `updateMask` parameter as a single comma-separated string.
        
        {% endnote %}

        ```bash
        curl \
            --request PATCH \
            --header "Authorization: Bearer $IAM_TOKEN" \
            --header "Content-Type: application/json" \
            --url 'https://mdb.api.cloud.yandex.net/managed-kafka/v1/clusters/<cluster_ID>' \
            --data '{
                      "updateMask": "configSpec.kafka.resources.diskSize,configSpec.zookeeper.resources.diskSize",
                      "configSpec": {
                        "kafka": {
                          "resources": {
                            "diskSize": "<storage_size_in_bytes>"
                          }
                        },
                        "zookeeper": {
                          "resources": {
                            "diskSize": "<storage_size_in_bytes>"
                          }
                        }
                      }
                    }'
        ```

        Where:

        * `updateMask`: Comma-separated string of settings you want to update.

            Specify the relevant parameters:
            * `configSpec.kafka.resources.diskSize`: To resize the broker host storage.
            * `configSpec.zookeeper.resources.diskSize`: To resize the ZooKeeper host storage. Use only for Apache Kafka® 3.5 clusters.
        * `configSpec.kafka.resources.diskSize`: Broker host storage size, in bytes.
        * `configSpec.zookeeper.resources.diskSize`: ZooKeeper host storage size, in bytes. Use only for Apache Kafka® 3.5 clusters.

        You can get the cluster ID with the [list of clusters in the folder](cluster-list.md#list-clusters).

    1. View the [server response](../api-ref/Cluster/update.md#responses) to make sure your request was successful.

* gRPC API {#grpc-api}

    1. [Get an IAM token for API authentication](../api-ref/authentication.md) and put it into an environment variable:

        ```bash
        export IAM_TOKEN="<IAM_token>"
        ```

    1. Clone the [cloudapi](https://github.com/yandex-cloud/cloudapi) repository:
       
       ```bash
       cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
       ```
       
       Below, we assume that the repository contents reside in the `~/cloudapi/` directory.

    1. Call the [ClusterService/Update](../api-ref/grpc/Cluster/update.md) method, e.g., via the following [gRPCurl](https://github.com/fullstorydev/grpcurl) request:

        {% note warning %}
        
        The API method will assign default values to all the parameters of the object you are modifying unless you explicitly provide them in your request. To avoid this, list the settings you want to change in the `update_mask` parameter as an array of `paths[]` strings.
        
        {% cut "Format for listing settings" %}
        
        ```yaml
        "update_mask": {
            "paths": [
                "<setting_1>",
                "<setting_2>",
                ...
                "<setting_N>"
            ]
        }
        ```
        
        {% endcut %}
        
        {% endnote %}

        ```bash
        grpcurl \
            -format json \
            -import-path ~/cloudapi/ \
            -import-path ~/cloudapi/third_party/googleapis/ \
            -proto ~/cloudapi/yandex/cloud/mdb/kafka/v1/cluster_service.proto \
            -rpc-header "Authorization: Bearer $IAM_TOKEN" \
            -d '{
                  "cluster_id": "<cluster_ID>",
                  "update_mask": {
                    "paths": [
                      "config_spec.kafka.resources.disk_size",
                      "config_spec.zookeeper.resources.disk_size"
                    ]
                  },
                  "config_spec": {
                    "kafka": {
                      "resources": {
                        "disk_size": "<storage_size_in_bytes>"
                      }
                    },
                    "zookeeper": {
                      "resources": {
                        "disk_size": "<storage_size_in_bytes>"
                      }
                    }
                  }
                }' \
            mdb.api.cloud.yandex.net:443 \
            yandex.cloud.mdb.kafka.v1.ClusterService.Update
        ```

        Where:

        * `update_mask`: List of settings you want to update as an array of strings (`paths[]`).

            Specify the relevant parameters:
            * `config_spec.kafka.resources.disk_size`: To resize the broker host storage.
            * `config_spec.brokers_count`: To resize the ZooKeeper host storage. Use only for Apache Kafka® 3.5 clusters.
        * `config_spec.kafka.resources.disk_size`: Broker host storage size, in bytes.
        * `config_spec.zookeeper.resources.disk_size`: ZooKeeper host storage size, in bytes. Use only for Apache Kafka® 3.5 clusters.

        You can get the cluster ID with the [list of clusters in the folder](cluster-list.md#list-clusters). Earlier, you already obtained the list of available host classes with their IDs.

    1. Check the [server response](../api-ref/grpc/Cluster/update.md#yandex.cloud.operation.Operation) to make sure your request was successful.

{% endlist %}

## Set up automatic storage expansion {#disk-size-autoscale}

Set up [automatic storage expansion](../concepts/storage.md#auto-rescale) to prevent the cluster from running out of disk space and the hosts from switching to read-only mode.

Make sure the cloud has enough quota to increase the storage size. Open the cloud's [Quotas](https://console.yandex.cloud/cloud?section=quotas) page and check the **HDD storage capacity** and **SSD storage capacity** lines under **Managed Databases** to make sure there is available quota for storage space.


{% note warning %}

You cannot reduce the storage size following an automatic expansion.

{% endnote %}


{% list tabs group=instructions %}

* Management console {#console}

    1. In the [management console](https://console.yandex.cloud), navigate to the relevant folder.
    1. Navigate to **Managed Service for&nbsp;Kafka**.
    1. In the cluster row, click ![image](../../_assets/console-icons/ellipsis.svg) and select **Edit**.
    1. Under **Automatic increase of storage size**, set the storage [utilization thresholds](../concepts/storage.md#auto-rescale) that will trigger storage expansion when reached: 
    
        1. In the **Increase size** field, select one or both thresholds:
            * **In the maintenance window when full at more than**: Scheduled expansion threshold. When reached, the storage expands during the next [maintenance window](../concepts/maintenance.md#maintenance-window).
            * **Immediately when full at more than**: Immediate expansion threshold. When reached, the storage expands immediately.
        1. Specify a threshold value (as a percentage of the total storage size). If you select both thresholds, make sure the immediate expansion threshold is higher than the scheduled one.
        1. Set the **Maximum storage size**.

* CLI {#cli}

    If you do not have the Yandex Cloud CLI yet, [install and initialize it](../../cli/quickstart.md#install).

    The folder used by default is the one specified when [creating](../../cli/operations/profile/profile-create.md) the CLI profile. To change the default folder, use the `yc config set folder-id <folder_ID>` command. You can also specify a different folder for any command using `--folder-name` or `--folder-id`. If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions.

    To set up automatic storage expansion:

    1. View the description of the CLI command for updating a cluster:

        ```bash
        yc managed-kafka cluster update --help
        ```

    1. Run this command, specifying the maximum storage size and the conditions for expansion:

        ```bash
        yc managed-kafka cluster update <cluster_ID_or_name> \
           --disk-size-autoscaling planned-usage-threshold=<scheduled_expansion_percentage>,`
                                  `emergency-usage-threshold=<immediate_expansion_percentage>,`
                                  `disk-size-limit=<maximum_storage_size_in_bytes>
        ```

        Where:
        
        * `planned-usage-threshold`: Storage usage percentage to trigger a storage expansion during the next [maintenance window](../concepts/maintenance.md#maintenance-window).
        
           Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled.
        
           If you set this condition, configure the maintenance schedule.
        
        * `emergency-usage-threshold`: Storage usage percentage to trigger an immediate storage expansion.
        
           Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. The value of this setting must be greater than or equal to `planned-usage-threshold`.
        
        * `disk-size-limit`: Maximum storage size, in bytes, that can be set when storage usage reaches one of the specified thresholds.
        
           If you set it to `0`, automatic storage expansion will be disabled.

* REST API {#api}

    1. [Get an IAM token for API authentication](../api-ref/authentication.md) and put it into an environment variable:

        ```bash
        export IAM_TOKEN="<IAM_token>"
        ```

    1. Call the [Cluster.update](../api-ref/Cluster/update.md) method, e.g., via the following [cURL](https://curl.se/) request:

        {% note warning %}
        
        The API method will assign default values to all the parameters of the object you are modifying unless you explicitly provide them in your request. To avoid this, list the settings you want to change in the `updateMask` parameter as a single comma-separated string.
        
        {% endnote %}

        ```bash
        curl \
            --request PATCH \
            --header "Authorization: Bearer $IAM_TOKEN" \
            --header "Content-Type: application/json" \
            --url 'https://mdb.api.cloud.yandex.net/managed-kafka/v1/clusters/<cluster_ID>' \
            --data '{
                      "updateMask": "configSpec.diskSizeAutoscaling.plannedUsageThreshold,configSpec.diskSizeAutoscaling.plannedUsageThreshold,configSpec.diskSizeAutoscaling.plannedUsageThreshold",
                      "configSpec": {
                        "diskSizeAutoscaling": {
                          "plannedUsageThreshold": "<scheduled_expansion_percentage>",
                          "emergencyUsageThreshold": "<immediate_expansion_percentage>",
                          "diskSizeLimit": "<maximum_storage_size_in_bytes>"
                        }
                      }
                    }'
        ```

        Where:

        * `updateMask`: Comma-separated string of settings you want to update.

            Specify the relevant parameters:
            * `configSpec.diskSizeAutoscaling.plannedUsageThreshold`: To change the storage usage percentage to trigger a scheduled expansion.
            * `configSpec.diskSizeAutoscaling.emergencyUsageThreshold`: To change the storage usage percentage to trigger a non-scheduled expansion.
            * `configSpec.diskSizeAutoscaling.diskSizeLimit`: To change the maximum storage size during automatic expansion.

        * `plannedUsageThreshold`: Storage usage percentage to trigger a storage expansion during the next [maintenance window](../concepts/maintenance.md#maintenance-window).
        
            Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled.
        
            If you set this condition, configure the maintenance window schedule.
        
        * `emergencyUsageThreshold`: Storage usage percentage to trigger an immediate storage expansion.
        
          Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. The value of this setting must be greater than or equal to `plannedUsageThreshold`.
        
        * `diskSizeLimit`: Maximum storage size, in bytes, that can be set when storage usage reaches one of the specified thresholds.

        You can request the cluster ID with the [list of clusters in the folder](cluster-list.md#list-clusters).

    1. View the [server response](../api-ref/Cluster/update.md#responses) to make sure your request was successful.

* gRPC API {#grpc-api}

    1. [Get an IAM token for API authentication](../api-ref/authentication.md) and put it into an environment variable:

        ```bash
        export IAM_TOKEN="<IAM_token>"
        ```

    1. Clone the [cloudapi](https://github.com/yandex-cloud/cloudapi) repository:
       
       ```bash
       cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
       ```
       
       Below, we assume that the repository contents reside in the `~/cloudapi/` directory.

    1. Call the [ClusterService/Update](../api-ref/grpc/Cluster/update.md) method, e.g., via the following [gRPCurl](https://github.com/fullstorydev/grpcurl) request:

        {% note warning %}
        
        The API method will assign default values to all the parameters of the object you are modifying unless you explicitly provide them in your request. To avoid this, list the settings you want to change in the `update_mask` parameter as an array of `paths[]` strings.
        
        {% cut "Format for listing settings" %}
        
        ```yaml
        "update_mask": {
            "paths": [
                "<setting_1>",
                "<setting_2>",
                ...
                "<setting_N>"
            ]
        }
        ```
        
        {% endcut %}
        
        {% endnote %}

        ```bash
        grpcurl \
            -format json \
            -import-path ~/cloudapi/ \
            -import-path ~/cloudapi/third_party/googleapis/ \
            -proto ~/cloudapi/yandex/cloud/mdb/kafka/v1/cluster_service.proto \
            -rpc-header "Authorization: Bearer $IAM_TOKEN" \
            -d '{
                  "cluster_id": "<cluster_ID>",
                  "update_mask": {
                    "paths": [
                      "config_spec.disk_size_autoscaling.planned_usage_threshold",
                      "config_spec.disk_size_autoscaling.emergency_usage_threshold",
                      "config_spec.disk_size_autoscaling.disk_size_limit"
                    ]
                  },
                  "config_spec": {
                    "disk_size_autoscaling": {
                        "planned_usage_threshold": "<scheduled_expansion_percentage>",
                        "emergency_usage_threshold": "<immediate_expansion_percentage>",
                        "disk_size_limit": "<maximum_storage_size_in_bytes>"
                    }
                  }
                }' \
            mdb.api.cloud.yandex.net:443 \
            yandex.cloud.mdb.kafka.v1.ClusterService.Update
        ```

        Where:

        * `update_mask`: List of settings you want to update as an array of strings (`paths[]`).

            Specify the relevant parameters:
            * `config_spec.disk_size_autoscaling.planned_usage_threshold`: To change the storage usage percentage to trigger a scheduled expansion.
            * `config_spec.disk_size_autoscaling.emergency_usage_threshold`: To change the storage usage percentage to trigger a non-scheduled expansion.
            * `config_spec.disk_size_autoscaling.disk_size_limit`: To change the maximum storage size during automatic expansion.

        * `planned_usage_threshold`: Storage usage percentage to trigger a storage expansion during the next [maintenance window](../concepts/maintenance.md#maintenance-window).
        
            Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled.
        
            If you set this condition, configure the maintenance window schedule.
        
        * `emergency_usage_threshold`: Storage usage percentage to trigger an immediate storage expansion.
        
          Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. The value of this setting must be greater than or equal to `planned_usage_threshold`.
        
        * `disk_size_limit`: Maximum storage size, in bytes, that can be set when storage usage reaches one of the specified thresholds.

        You can get the cluster ID with the [list of clusters in the folder](cluster-list.md#list-clusters). Earlier, you already obtained the list of available host classes with their IDs.

    1. Check the [server response](../api-ref/grpc/Cluster/update.md#yandex.cloud.operation.Operation) to make sure your request was successful.

{% endlist %}

Upon reaching the specified threshold, the storage expands differently depending on the disk type:

* For network HDDs and SSDs, by the higher of the two values: 20 GB or 20% of the current disk size.
* For non-replicated SSDs, by 93 GB.
* For local SSDs, in a cluster based on:

    * **Intel Cascade Lake**, by 100 GB.
    * **Intel Ice Lake** or **AMD Zen 4**, by 368 GB.


If the threshold is reached again, the storage will be automatically expanded until it reaches the specified maximum. After that, you can set a new maximum storage size manually.


{% note warning %}

You cannot reduce the storage size following an automatic expansion.

{% endnote %}