# Resource relationships in Managed Service for Apache Kafka®


Apache Kafka® is a distributed cross-application communication system that uses the _publication-subscription_ principle.

Yandex Managed Service for Apache Kafka® helps you deploy and maintain clusters of 3.6, 3.7, 3.8, 3.9 or 4.0 Apache Kafka® servers in the Yandex Cloud infrastructure.


Each Managed Service for Apache Kafka® cluster consists of one or more [broker hosts](brokers.md) that host [topics](topics.md) and their partitions. Cluster hosts may reside in different availability zones. You can learn more about the Yandex Cloud availability zones in the [Platform overview](../../overview/concepts/geo-scope.md).

If all broker hosts are created in the same availability zone, you cannot change their number.


Cluster hosts accept [connections](../operations/connect/clients.md) from clients, such as data [producer](producers-consumers.md) and [consumer](producers-consumers.md) applications. Producers send messages to specific cluster topics and consumers read messages from them. Thus, you will not need to reconfigure consumers if a producer changes.

Be mindful of what is managed by the service and what, by the Yandex Cloud customer. Understanding these zones of control makes you an efficient user of cloud resources and helps avoid potential database-related issues. For more information, see [Zones of control between managed database (MDB) service users and Yandex Cloud](../../overview/concepts/mdb-responsibilities.md).

## Host configuration depending on Apache Kafka® version {#version}

Different Apache Kafka® versions use different tools to store cluster metadata, state, and configuration:

* Versions 3.5 and lower support [ZooKeeper](index.md#zookeeper).
* Versions 3.6 to (and including) 3.9 support ZooKeeper and [Apache Kafka® Raft](index.md#kraft).
* Versions 4.0 or higher support [Apache Kafka® Raft](index.md#kraft) only.

{% note info %}

## Environment {#environment}

Select the environment where you want to create your Managed Service for Apache Kafka® cluster (note that you cannot change the environment once the cluster is created):
  * `PRODUCTION`: For stable versions of your apps.
  * `PRESTABLE`: For testing purposes. The prestable environment is similar to the production environment and likewise covered by an SLA, but it is the first to get new features, improvements, and bug fixes. In the prestable environment, you can test new versions for compatibility with your application.

## Host class {#broker-class}

Under **Host class**, select the [platform](../../compute/concepts/vm-platforms.md), host type, and host class.

The host class defines the specifications of [virtual machines](../../compute/concepts/vm.md) Apache Kafka® nodes will be deployed on. All available options are listed under [Host classes](instance-types.md).

When you [change the host class](../operations/cluster-update.md#change-brokers) for a Managed Service for Apache Kafka® cluster, the specifications of all existing instances also change.

## Default host configuration {#default-config}

ZooKeeper and KRaft hosts have the following configuration by default:

* [Host class](instance-types.md): `s3-c2-m8`
* Disk size: `10` GB
* [Disk type](storage.md): `network-ssd`

## Automatic storage expansion {#auto-rescale}

To prevent the cluster from running out of disk space, create a cluster with [automatic storage expansion](storage.md#auto-rescale):

```bash
yc managed-kafka cluster create \
...
--disk-size-autoscaling disk-size-limit=<maximum_storage_size_in_bytes>,`
                        `planned-usage-threshold=<scheduled_expansion_percentage>,`
                        `emergency-usage-threshold=<immediate_expansion_percentage> \
...
```

Where:

* `planned-usage-threshold`: Storage usage percentage to trigger a storage expansion during the next [maintenance window](maintenance.md#maintenance-window).

   Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled.

   If you set this condition, configure the maintenance schedule.

* `emergency-usage-threshold`: Storage usage percentage to trigger an immediate storage expansion.

   Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. The value of this setting must be greater than or equal to `planned-usage-threshold`.

* `disk-size-limit`: Maximum storage size, in bytes, that can be set when storage usage reaches one of the specified thresholds.

   If you set it to `0`, automatic storage expansion will be disabled.

{% note warning %}

You cannot reduce the storage size following an automatic expansion.

{% endnote %}

## Cluster status {#status}

Wait until the Managed Service for Apache Kafka® cluster is ready: its status on the Managed Service for Apache Kafka® dashboard will change to `Running`, and its state, to `Alive`. This may take a while.

## Availability zone and cluster creation warning {#note-warning}

{% note warning %}
        
If you create a Managed Service for Apache Kafka® cluster with a single [availability zone](../../overview/concepts/geo-scope.md), you will not be able to increase the number of zones and broker hosts later.
        
{% endnote %}

## Maintenance window setup {#maintenance}

To set up a [maintenance window](maintenance.md) (including for stopped Managed Service for Apache Kafka® clusters), provide the relevant value in the `--maintenance-window` parameter when creating your cluster:

```bash
yc managed-kafka cluster create \
...
--maintenance-window type=<maintenance_type>,`
                    `day=<day_of_week>,`
                    `hour=<hour> \
...
```

Where `type` is the maintenance type:

* `anytime`: Any time (default).
* `weekly`: On a schedule. For this value, also specify the following:
    * `day`: Day of week, i.e., `MON`, `TUE`, `WED`, `THU`, `FRI`, `SAT`, or `SUN`.
    * `hour`: Sequence number of UTC hour interval, from `1` to `24`.

      > For example, `1` stands for the interval from `00:00` to `01:00`, and `5`, from `04:00` to `05:00`.

## Combined mode and host configuration transfer warning {#note-warning-combined-mode}

{% note warning %}
                    
If you are creating a [KRaft (combined mode)](kraft.md#cluster-topology) cluster, do not provide the KRaft host configuration.
                    
{% endnote %}

## Cluster deletion protection {#protect-from-deletion}

{% note info %}

Even with cluster deletion protection enabled, one can still delete a user or topic or connect manually and delete the data.

{% endnote %}

{% endnote %}

### Host configuration in clusters with ZooKeeper {#zookeeper}

ZooKeeper is automatically configured and placed in a cluster based on the number of broker hosts:

* If the Apache Kafka® cluster consists of one broker host, ZooKeeper is hosted on the same host.
* If the Apache Kafka® cluster consists of two or more hosts, ZooKeeper is hosted separately from brokers on three additional hosts. These hosts are added to the cluster automatically.

You cannot delete ZooKeeper hosts. The number of ZooKeeper hosts is fixed.

For more information on creating a Managed Service for Apache Kafka® cluster with ZooKeeper, see [this guide](../operations/cluster-create.md#create-cluster).

### Host configuration in clusters with Apache Kafka® Raft {#kraft}

The [Apache Kafka® Raft protocol](kraft.md) (abbreviated KRaft) is used to store metadata instead of ZooKeeper.

KRaft is automatically configured and placed in a cluster based on the selected mode and the number of broker hosts:

* **KRaft (combined mode)**: One Apache Kafka® host accommodates a [broker](brokers.md) and a KRaft metadata controller at the same time. 


Only three Apache Kafka® broker hosts are created in the cluster in one of these configurations:
  * Three broker hosts in the same [availability zone](../../overview/concepts/geo-scope.md).
  * Each broker host in a separate availability zone.

  
  You cannot set the number of broker hosts manually.

* **KRaft (on separate hosts)**: Broker and KRaft metadata controller are on separate hosts. A new cluster with multiple broker hosts will automatically get three separate KRaft hosts.

  The number of broker hosts is set manually.

You cannot delete KRaft hosts. The number of KRaft hosts is fixed.

For more information on creating a Managed Service for Apache Kafka® cluster with KRaft, see [this guide](../operations/cluster-create.md#create-cluster-kraft).

For more information, see the [Apache Kafka®](https://kafka.apache.org/42/operations/kraft/) documentation.