[Yandex Cloud documentation](../../index.md) > [Yandex Managed Service for Apache Kafka®](../index.md) > [Step-by-step guides](index.md) > Clusters > Creating a cluster # Creating a cluster Apache Kafka® A [Managed Service for Apache Kafka® cluster](../concepts/index.md) consists of one or more [broker hosts](../concepts/brokers.md) that hold [topics and their partitions](../concepts/topics.md). [Producers and consumers](../concepts/producers-consumers.md) can work with these topics by connecting to Managed Service for Apache Kafka® cluster hosts. {% note info %} The available disk types [depend](../concepts/storage.md) on the selected [host class](../concepts/instance-types.md). {% endnote %} ## Differences in configurations of clusters with ZooKeeper and clusters that use the Apache Kafka® Raft protocol {#zk-vs-kraft} Different Apache Kafka® versions use different tools to store cluster metadata, state, and configuration: * Versions 3.5 and lower support [ZooKeeper](../concepts/index.md#zookeeper). * Versions 3.6 to (and including) 3.9 support ZooKeeper and [Apache Kafka® Raft](../concepts/index.md#kraft). * Versions 4.0 or higher support [Apache Kafka® Raft](../concepts/index.md#kraft) only. **Hosts with ZooKeeper** {#host-zookeeper} When selecting Apache Kafka® 3.5 or lower, only ZooKeeper is supported. A new multi-host cluster will automatically get three separate ZooKeeper hosts. **Hosts with KRaft** {#host-kraft} When selecting Apache Kafka® 3.6 or higher, the KRaft protocol is additionally supported. The KRaft protocol is available in one of the following modes: * **KRaft (combined mode)**: One Apache Kafka® host accommodates a [broker](../concepts/brokers.md) and a KRaft metadata controller at the same time. Only three Apache Kafka® broker hosts are created in the cluster in one of these configurations: * Three broker hosts in the same [availability zone](../../overview/concepts/geo-scope.md). * Each broker host in a separate availability zone. You cannot set the number of broker hosts manually. * **KRaft (on separate hosts)**: Broker and KRaft metadata controller are on separate hosts. A new cluster with multiple broker hosts will automatically get three separate KRaft hosts. The number of broker hosts is set manually. You cannot delete KRaft hosts. The number of KRaft hosts is fixed. For more information about the differences in cluster configurations with ZooKeeper and KRaft, see [Resource relationships in Managed Service for Apache Kafka®](../concepts/index.md). ## Getting started {#before-you-begin} 1. Calculate the [minimum storage size](../concepts/storage.md#minimal-storage-size) for topics. 1. [Assign](../../iam/operations/roles/grant.md) the following roles to your Yandex Cloud account: * [managed-kafka.editor or higher](../security/index.md#roles-list): To create a cluster. * [vpc.user](../../vpc/security/index.md#vpc-user): To use the cluster [network](../../vpc/concepts/network.md#network). * [kms.keys.user](../../kms/security/index.md#kms-keys-user): To manage [disk encryption](../concepts/storage.md#disk-encryption). If you specify security group IDs when creating a Managed Service for Apache Kafka® cluster, you may also need to [configure security groups](connect/index.md#configuring-security-groups) to connect to the cluster. ## Creating a cluster with ZooKeeper {#create-cluster} {% note warning %} When creating a cluster with ZooKeeper, do not specify the KRaft settings. {% endnote %} {% list tabs group=instructions %} - Management console {#console} To create a Managed Service for Apache Kafka® cluster: 1. In the [management console](https://console.yandex.cloud), go to the appropriate [folder](../../resource-manager/concepts/resources-hierarchy.md#folder). 1. Navigate to **Managed Service for Kafka**. 1. Click **Create cluster**. 1. Under **Basic parameters**: 1. Enter a name and description for the Managed Service for Apache Kafka® cluster. The Managed Service for Apache Kafka® cluster name must be unique within the folder. 1. ## Environment {#environment} Select the environment where you want to create your Managed Service for Apache Kafka® cluster (note that you cannot change the environment once the cluster is created): * `PRODUCTION`: For stable versions of your apps. * `PRESTABLE`: For testing purposes. The prestable environment is similar to the production environment and likewise covered by an SLA, but it is the first to get new features, improvements, and bug fixes. In the prestable environment, you can test new versions for compatibility with your application. ## Host class {#broker-class} Under **Host class**, select the [platform](../../compute/concepts/vm-platforms.md), host type, and host class. The host class defines the specifications of [virtual machines](../../compute/concepts/vm.md) Apache Kafka® nodes will be deployed on. All available options are listed under [Host classes](../concepts/instance-types.md). When you [change the host class](cluster-update.md#change-brokers) for a Managed Service for Apache Kafka® cluster, the specifications of all existing instances also change. ## Default host configuration {#default-config} ZooKeeper and KRaft hosts have the following configuration by default: * [Host class](../concepts/instance-types.md): `s3-c2-m8` * Disk size: `10` GB * [Disk type](../concepts/storage.md): `network-ssd` ## Automatic storage expansion {#auto-rescale} To prevent the cluster from running out of disk space, create a cluster with [automatic storage expansion](../concepts/storage.md#auto-rescale): ```bash yc managed-kafka cluster create \ ... --disk-size-autoscaling disk-size-limit=,` `planned-usage-threshold=,` `emergency-usage-threshold= \ ... ``` Where: * `planned-usage-threshold`: Storage usage percentage to trigger a storage expansion during the next [maintenance window](../concepts/maintenance.md#maintenance-window). Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. If you set this condition, configure the maintenance schedule. * `emergency-usage-threshold`: Storage usage percentage to trigger an immediate storage expansion. Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. The value of this setting must be greater than or equal to `planned-usage-threshold`. * `disk-size-limit`: Maximum storage size, in bytes, that can be set when storage usage reaches one of the specified thresholds. If you set it to `0`, automatic storage expansion will be disabled. {% note warning %} You cannot reduce the storage size following an automatic expansion. {% endnote %} ## Cluster status {#status} Wait until the Managed Service for Apache Kafka® cluster is ready: its status on the Managed Service for Apache Kafka® dashboard will change to `Running`, and its state, to `Alive`. This may take a while. ## Availability zone and cluster creation warning {#note-warning} {% note warning %} If you create a Managed Service for Apache Kafka® cluster with a single [availability zone](../../overview/concepts/geo-scope.md), you will not be able to increase the number of zones and broker hosts later. {% endnote %} ## Maintenance window setup {#maintenance} To set up a [maintenance window](../concepts/maintenance.md) (including for stopped Managed Service for Apache Kafka® clusters), provide the relevant value in the `--maintenance-window` parameter when creating your cluster: ```bash yc managed-kafka cluster create \ ... --maintenance-window type=,` `day=,` `hour= \ ... ``` Where `type` is the maintenance type: * `anytime`: Any time (default). * `weekly`: On a schedule. For this value, also specify the following: * `day`: Day of week, i.e., `MON`, `TUE`, `WED`, `THU`, `FRI`, `SAT`, or `SUN`. * `hour`: Sequence number of UTC hour interval, from `1` to `24`. > For example, `1` stands for the interval from `00:00` to `01:00`, and `5`, from `04:00` to `05:00`. ## Combined mode and host configuration transfer warning {#note-warning-combined-mode} {% note warning %} If you are creating a [KRaft (combined mode)](../concepts/kraft.md#cluster-topology) cluster, do not provide the KRaft host configuration. {% endnote %} ## Cluster deletion protection {#protect-from-deletion} {% note info %} Even with cluster deletion protection enabled, one can still delete a user or topic or connect manually and delete the data. {% endnote %} 1. Select the Apache Kafka® version. 1. ## Environment {#environment} Select the environment where you want to create your Managed Service for Apache Kafka® cluster (note that you cannot change the environment once the cluster is created): * `PRODUCTION`: For stable versions of your apps. * `PRESTABLE`: For testing purposes. The prestable environment is similar to the production environment and likewise covered by an SLA, but it is the first to get new features, improvements, and bug fixes. In the prestable environment, you can test new versions for compatibility with your application. ## Host class {#broker-class} Under **Host class**, select the [platform](../../compute/concepts/vm-platforms.md), host type, and host class. The host class defines the specifications of [virtual machines](../../compute/concepts/vm.md) Apache Kafka® nodes will be deployed on. All available options are listed under [Host classes](../concepts/instance-types.md). When you [change the host class](cluster-update.md#change-brokers) for a Managed Service for Apache Kafka® cluster, the specifications of all existing instances also change. ## Default host configuration {#default-config} ZooKeeper and KRaft hosts have the following configuration by default: * [Host class](../concepts/instance-types.md): `s3-c2-m8` * Disk size: `10` GB * [Disk type](../concepts/storage.md): `network-ssd` ## Automatic storage expansion {#auto-rescale} To prevent the cluster from running out of disk space, create a cluster with [automatic storage expansion](../concepts/storage.md#auto-rescale): ```bash yc managed-kafka cluster create \ ... --disk-size-autoscaling disk-size-limit=,` `planned-usage-threshold=,` `emergency-usage-threshold= \ ... ``` Where: * `planned-usage-threshold`: Storage usage percentage to trigger a storage expansion during the next [maintenance window](../concepts/maintenance.md#maintenance-window). Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. If you set this condition, configure the maintenance schedule. * `emergency-usage-threshold`: Storage usage percentage to trigger an immediate storage expansion. Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. The value of this setting must be greater than or equal to `planned-usage-threshold`. * `disk-size-limit`: Maximum storage size, in bytes, that can be set when storage usage reaches one of the specified thresholds. If you set it to `0`, automatic storage expansion will be disabled. {% note warning %} You cannot reduce the storage size following an automatic expansion. {% endnote %} ## Cluster status {#status} Wait until the Managed Service for Apache Kafka® cluster is ready: its status on the Managed Service for Apache Kafka® dashboard will change to `Running`, and its state, to `Alive`. This may take a while. ## Availability zone and cluster creation warning {#note-warning} {% note warning %} If you create a Managed Service for Apache Kafka® cluster with a single [availability zone](../../overview/concepts/geo-scope.md), you will not be able to increase the number of zones and broker hosts later. {% endnote %} ## Maintenance window setup {#maintenance} To set up a [maintenance window](../concepts/maintenance.md) (including for stopped Managed Service for Apache Kafka® clusters), provide the relevant value in the `--maintenance-window` parameter when creating your cluster: ```bash yc managed-kafka cluster create \ ... --maintenance-window type=,` `day=,` `hour= \ ... ``` Where `type` is the maintenance type: * `anytime`: Any time (default). * `weekly`: On a schedule. For this value, also specify the following: * `day`: Day of week, i.e., `MON`, `TUE`, `WED`, `THU`, `FRI`, `SAT`, or `SUN`. * `hour`: Sequence number of UTC hour interval, from `1` to `24`. > For example, `1` stands for the interval from `00:00` to `01:00`, and `5`, from `04:00` to `05:00`. ## Combined mode and host configuration transfer warning {#note-warning-combined-mode} {% note warning %} If you are creating a [KRaft (combined mode)](../concepts/kraft.md#cluster-topology) cluster, do not provide the KRaft host configuration. {% endnote %} ## Cluster deletion protection {#protect-from-deletion} {% note info %} Even with cluster deletion protection enabled, one can still delete a user or topic or connect manually and delete the data. {% endnote %} 1. Under **Storage**: * Select the disk type. {% note warning %} You cannot change disk type after you create a cluster. {% endnote %} The type you select determines the increments for changing the disk size: * Network HDDs and SSDs: In increments of 1 GB. * Local SSDs: * For **Intel Cascade Lake**: In increments of 100 GB. * For **Intel Ice Lake** and **AMD Zen 4**: In 368 GB increments. * Non-replicated SSDs: In increments of 93 GB. You cannot change the disk type for a Managed Service for Apache Kafka® cluster once the cluster is created. * Select the storage size to use for data. 1. Under **Automatic increase of storage size**, set the storage [utilization thresholds](../concepts/storage.md#auto-rescale) that will trigger storage expansion when reached: 1. In the **Increase size** field, select one or both thresholds: * **In the maintenance window when full at more than**: Scheduled expansion threshold. When reached, the storage expands during the next [maintenance window](../concepts/maintenance.md#maintenance-window). * **Immediately when full at more than**: Immediate expansion threshold. When reached, the storage expands immediately. 1. Specify a threshold value (as a percentage of the total storage size). If you select both thresholds, make sure the immediate expansion threshold is higher than the scheduled one. 1. Set the **Maximum storage size**. 1. Under **Network settings**: 1. Select one or more availability zones for your Apache Kafka® broker hosts. ## Environment {#environment} Select the environment where you want to create your Managed Service for Apache Kafka® cluster (note that you cannot change the environment once the cluster is created): * `PRODUCTION`: For stable versions of your apps. * `PRESTABLE`: For testing purposes. The prestable environment is similar to the production environment and likewise covered by an SLA, but it is the first to get new features, improvements, and bug fixes. In the prestable environment, you can test new versions for compatibility with your application. ## Host class {#broker-class} Under **Host class**, select the [platform](../../compute/concepts/vm-platforms.md), host type, and host class. The host class defines the specifications of [virtual machines](../../compute/concepts/vm.md) Apache Kafka® nodes will be deployed on. All available options are listed under [Host classes](../concepts/instance-types.md). When you [change the host class](cluster-update.md#change-brokers) for a Managed Service for Apache Kafka® cluster, the specifications of all existing instances also change. ## Default host configuration {#default-config} ZooKeeper and KRaft hosts have the following configuration by default: * [Host class](../concepts/instance-types.md): `s3-c2-m8` * Disk size: `10` GB * [Disk type](../concepts/storage.md): `network-ssd` ## Automatic storage expansion {#auto-rescale} To prevent the cluster from running out of disk space, create a cluster with [automatic storage expansion](../concepts/storage.md#auto-rescale): ```bash yc managed-kafka cluster create \ ... --disk-size-autoscaling disk-size-limit=,` `planned-usage-threshold=,` `emergency-usage-threshold= \ ... ``` Where: * `planned-usage-threshold`: Storage usage percentage to trigger a storage expansion during the next [maintenance window](../concepts/maintenance.md#maintenance-window). Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. If you set this condition, configure the maintenance schedule. * `emergency-usage-threshold`: Storage usage percentage to trigger an immediate storage expansion. Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. The value of this setting must be greater than or equal to `planned-usage-threshold`. * `disk-size-limit`: Maximum storage size, in bytes, that can be set when storage usage reaches one of the specified thresholds. If you set it to `0`, automatic storage expansion will be disabled. {% note warning %} You cannot reduce the storage size following an automatic expansion. {% endnote %} ## Cluster status {#status} Wait until the Managed Service for Apache Kafka® cluster is ready: its status on the Managed Service for Apache Kafka® dashboard will change to `Running`, and its state, to `Alive`. This may take a while. ## Availability zone and cluster creation warning {#note-warning} {% note warning %} If you create a Managed Service for Apache Kafka® cluster with a single [availability zone](../../overview/concepts/geo-scope.md), you will not be able to increase the number of zones and broker hosts later. {% endnote %} ## Maintenance window setup {#maintenance} To set up a [maintenance window](../concepts/maintenance.md) (including for stopped Managed Service for Apache Kafka® clusters), provide the relevant value in the `--maintenance-window` parameter when creating your cluster: ```bash yc managed-kafka cluster create \ ... --maintenance-window type=,` `day=,` `hour= \ ... ``` Where `type` is the maintenance type: * `anytime`: Any time (default). * `weekly`: On a schedule. For this value, also specify the following: * `day`: Day of week, i.e., `MON`, `TUE`, `WED`, `THU`, `FRI`, `SAT`, or `SUN`. * `hour`: Sequence number of UTC hour interval, from `1` to `24`. > For example, `1` stands for the interval from `00:00` to `01:00`, and `5`, from `04:00` to `05:00`. ## Combined mode and host configuration transfer warning {#note-warning-combined-mode} {% note warning %} If you are creating a [KRaft (combined mode)](../concepts/kraft.md#cluster-topology) cluster, do not provide the KRaft host configuration. {% endnote %} ## Cluster deletion protection {#protect-from-deletion} {% note info %} Even with cluster deletion protection enabled, one can still delete a user or topic or connect manually and delete the data. {% endnote %} 1. Select the [network](../../vpc/concepts/network.md#network). 1. Select [subnets](../../vpc/concepts/network.md#subnet) in each availability zone for this network. To [create a new subnet](../../vpc/operations/subnet-create.md), click **Create** next to the availability zone. {% note info %} For an Apache Kafka® cluster with multiple broker hosts, specify subnets in each availability zone even if you plan to place broker hosts only in some of them. You need these subnets to deploy three [ZooKeeper hosts](../concepts/index.md), one per availability zone. For more information, see [Resource relationships in Managed Service for Apache Kafka®](../concepts/index.md). {% endnote %} 1. Select [security groups](../../vpc/concepts/security-groups.md) for the Managed Service for Apache Kafka® cluster’s network traffic. 1. To enable internet access to broker hosts, select **Public access**. In this case, you can only connect to them using SSL. For more information, see [Connecting to an Apache Kafka® cluster from applications](connect/clients.md). 1. Under **Hosts**: 1. Specify the number of Apache Kafka® broker hosts to reside in each selected availability zone. When selecting the number of broker hosts, consider the following: * If you add more than one broker host to the cluster, the system will automatically add three ZooKeeper hosts. * You need at least two broker hosts to enable replication in a Managed Service for Apache Kafka® cluster. * High availability of a Managed Service for Apache Kafka® cluster depends on certain [prerequisites](../concepts/ha-cluster.md). 1. Select **ZooKeeper (on separate hosts)** as the coordination service. 1. If you specified more than one broker host, under **ZooKeeper host class**, specify the properties of the [ZooKeeper hosts](../concepts/index.md) to reside in each selected availability zone. 1. Specify additional Managed Service for Apache Kafka® cluster settings, if required: * **Maintenance**: [Maintenance window](../concepts/maintenance.md) settings: * To enable maintenance at any time, select **At any time** (default). * To specify the preferred maintenance start time, select **By schedule** and specify the day of the week and the UTC time interval. For example, you can choose a time when the cluster is least loaded. Both active and stopped clusters are subject to maintenance. Maintenance operations include DBMS updates, patches, etc. * **Deletion protection**: Manages cluster protection against accidental deletion. Even with cluster deletion protection enabled, one can still delete a user or topic or connect manually and delete the data. * **Disk encryption**: Enable this setting to encrypt the disks with a [custom KMS key](../../kms/concepts/key.md). * To [create](../../kms/operations/key.md#create) a new key, click **Create**. * To use a previously created key, select it in the **KMS key** field. To learn more about disk encryption, see [Storage](../concepts/storage.md#disk-encryption). {% note warning %} You can enable disk encryption only when creating a cluster. {% endnote %} * **Schema registry**: Enable this setting to manage data schemas using [Managed Schema Registry](../concepts/managed-schema-registry.md). {% note warning %} You cannot disable data schema management using Managed Schema Registry after enabling it. {% endnote %} * **Kafka Rest API**: Enable this setting to allow sending requests to the Apache Kafka® API. It is implemented using [Karapace](https://github.com/Aiven-Open/karapace), an open-source tool. The Karapace API is compatible with the [Confluent REST Proxy API](https://docs.confluent.io/platform/current/kafka-rest/api.html) with only minor exceptions. {% note warning %} You cannot disable **Kafka Rest API** once it is enabled. {% endnote %} * **Kafka UI**: Enable this setting to use the [Apache Kafka®](../concepts/kafka-ui.md) web UI. 1. Configure the [Apache Kafka® settings](../concepts/settings-list.md#cluster-settings), if required. 1. Click **Create**. 1. ## Environment {#environment} Select the environment where you want to create your Managed Service for Apache Kafka® cluster (note that you cannot change the environment once the cluster is created): * `PRODUCTION`: For stable versions of your apps. * `PRESTABLE`: For testing purposes. The prestable environment is similar to the production environment and likewise covered by an SLA, but it is the first to get new features, improvements, and bug fixes. In the prestable environment, you can test new versions for compatibility with your application. ## Host class {#broker-class} Under **Host class**, select the [platform](../../compute/concepts/vm-platforms.md), host type, and host class. The host class defines the specifications of [virtual machines](../../compute/concepts/vm.md) Apache Kafka® nodes will be deployed on. All available options are listed under [Host classes](../concepts/instance-types.md). When you [change the host class](cluster-update.md#change-brokers) for a Managed Service for Apache Kafka® cluster, the specifications of all existing instances also change. ## Default host configuration {#default-config} ZooKeeper and KRaft hosts have the following configuration by default: * [Host class](../concepts/instance-types.md): `s3-c2-m8` * Disk size: `10` GB * [Disk type](../concepts/storage.md): `network-ssd` ## Automatic storage expansion {#auto-rescale} To prevent the cluster from running out of disk space, create a cluster with [automatic storage expansion](../concepts/storage.md#auto-rescale): ```bash yc managed-kafka cluster create \ ... --disk-size-autoscaling disk-size-limit=,` `planned-usage-threshold=,` `emergency-usage-threshold= \ ... ``` Where: * `planned-usage-threshold`: Storage usage percentage to trigger a storage expansion during the next [maintenance window](../concepts/maintenance.md#maintenance-window). Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. If you set this condition, configure the maintenance schedule. * `emergency-usage-threshold`: Storage usage percentage to trigger an immediate storage expansion. Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. The value of this setting must be greater than or equal to `planned-usage-threshold`. * `disk-size-limit`: Maximum storage size, in bytes, that can be set when storage usage reaches one of the specified thresholds. If you set it to `0`, automatic storage expansion will be disabled. {% note warning %} You cannot reduce the storage size following an automatic expansion. {% endnote %} ## Cluster status {#status} Wait until the Managed Service for Apache Kafka® cluster is ready: its status on the Managed Service for Apache Kafka® dashboard will change to `Running`, and its state, to `Alive`. This may take a while. ## Availability zone and cluster creation warning {#note-warning} {% note warning %} If you create a Managed Service for Apache Kafka® cluster with a single [availability zone](../../overview/concepts/geo-scope.md), you will not be able to increase the number of zones and broker hosts later. {% endnote %} ## Maintenance window setup {#maintenance} To set up a [maintenance window](../concepts/maintenance.md) (including for stopped Managed Service for Apache Kafka® clusters), provide the relevant value in the `--maintenance-window` parameter when creating your cluster: ```bash yc managed-kafka cluster create \ ... --maintenance-window type=,` `day=,` `hour= \ ... ``` Where `type` is the maintenance type: * `anytime`: Any time (default). * `weekly`: On a schedule. For this value, also specify the following: * `day`: Day of week, i.e., `MON`, `TUE`, `WED`, `THU`, `FRI`, `SAT`, or `SUN`. * `hour`: Sequence number of UTC hour interval, from `1` to `24`. > For example, `1` stands for the interval from `00:00` to `01:00`, and `5`, from `04:00` to `05:00`. ## Combined mode and host configuration transfer warning {#note-warning-combined-mode} {% note warning %} If you are creating a [KRaft (combined mode)](../concepts/kraft.md#cluster-topology) cluster, do not provide the KRaft host configuration. {% endnote %} ## Cluster deletion protection {#protect-from-deletion} {% note info %} Even with cluster deletion protection enabled, one can still delete a user or topic or connect manually and delete the data. {% endnote %} - CLI {#cli} If you do not have the Yandex Cloud CLI yet, [install and initialize it](../../cli/quickstart.md#install). The folder used by default is the one specified when [creating](../../cli/operations/profile/profile-create.md) the CLI profile. To change the default folder, use the `yc config set folder-id ` command. You can also specify a different folder for any command using `--folder-name` or `--folder-id`. If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions. To create a Managed Service for Apache Kafka® cluster: 1. View the description of the CLI command to create a Managed Service for Apache Kafka® cluster: ```bash yc managed-kafka cluster create --help ``` 1. Specify the Managed Service for Apache Kafka® cluster parameters in the create command (not all parameters are given in the example): ```bash yc managed-kafka cluster create \ --name \ --environment \ --version \ --schema-registry \ --network-name \ --subnet-ids \ --zone-ids \ --brokers-count \ --resource-preset \ --disk-type \ --disk-size \ --assign-public-ip \ --security-group-ids \ --deletion-protection \ --kafka-ui-enabled \ --disk-encryption-key-id ``` Where: * `--environment`: Cluster environment, `prestable` or `production`. * `--version`: Apache Kafka® version, 3.6, 3.7, 3.8 or 3.9. Additionally, provide the [ZooKeeper host configuration](../concepts/index.md#zookeeper). * `--schema-registry`: Manage data schemas using [Managed Schema Registry](../concepts/managed-schema-registry.md). {% note warning %} You cannot disable data schema management using Managed Schema Registry after enabling it. {% endnote %} * `--zone-ids` and `--brokers-count`: Availability zones and number of broker hosts per zone. * `--resource-preset`: [Host class](../concepts/instance-types.md). * `--disk-type`: [Disk type](../concepts/storage.md). {% note warning %} You cannot change disk type after you create a cluster. {% endnote %} * `--deletion-protection`: Cluster protection from accidental deletion, `true` or `false`. ## Environment {#environment} Select the environment where you want to create your Managed Service for Apache Kafka® cluster (note that you cannot change the environment once the cluster is created): * `PRODUCTION`: For stable versions of your apps. * `PRESTABLE`: For testing purposes. The prestable environment is similar to the production environment and likewise covered by an SLA, but it is the first to get new features, improvements, and bug fixes. In the prestable environment, you can test new versions for compatibility with your application. ## Host class {#broker-class} Under **Host class**, select the [platform](../../compute/concepts/vm-platforms.md), host type, and host class. The host class defines the specifications of [virtual machines](../../compute/concepts/vm.md) Apache Kafka® nodes will be deployed on. All available options are listed under [Host classes](../concepts/instance-types.md). When you [change the host class](cluster-update.md#change-brokers) for a Managed Service for Apache Kafka® cluster, the specifications of all existing instances also change. ## Default host configuration {#default-config} ZooKeeper and KRaft hosts have the following configuration by default: * [Host class](../concepts/instance-types.md): `s3-c2-m8` * Disk size: `10` GB * [Disk type](../concepts/storage.md): `network-ssd` ## Automatic storage expansion {#auto-rescale} To prevent the cluster from running out of disk space, create a cluster with [automatic storage expansion](../concepts/storage.md#auto-rescale): ```bash yc managed-kafka cluster create \ ... --disk-size-autoscaling disk-size-limit=,` `planned-usage-threshold=,` `emergency-usage-threshold= \ ... ``` Where: * `planned-usage-threshold`: Storage usage percentage to trigger a storage expansion during the next [maintenance window](../concepts/maintenance.md#maintenance-window). Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. If you set this condition, configure the maintenance schedule. * `emergency-usage-threshold`: Storage usage percentage to trigger an immediate storage expansion. Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. The value of this setting must be greater than or equal to `planned-usage-threshold`. * `disk-size-limit`: Maximum storage size, in bytes, that can be set when storage usage reaches one of the specified thresholds. If you set it to `0`, automatic storage expansion will be disabled. {% note warning %} You cannot reduce the storage size following an automatic expansion. {% endnote %} ## Cluster status {#status} Wait until the Managed Service for Apache Kafka® cluster is ready: its status on the Managed Service for Apache Kafka® dashboard will change to `Running`, and its state, to `Alive`. This may take a while. ## Availability zone and cluster creation warning {#note-warning} {% note warning %} If you create a Managed Service for Apache Kafka® cluster with a single [availability zone](../../overview/concepts/geo-scope.md), you will not be able to increase the number of zones and broker hosts later. {% endnote %} ## Maintenance window setup {#maintenance} To set up a [maintenance window](../concepts/maintenance.md) (including for stopped Managed Service for Apache Kafka® clusters), provide the relevant value in the `--maintenance-window` parameter when creating your cluster: ```bash yc managed-kafka cluster create \ ... --maintenance-window type=,` `day=,` `hour= \ ... ``` Where `type` is the maintenance type: * `anytime`: Any time (default). * `weekly`: On a schedule. For this value, also specify the following: * `day`: Day of week, i.e., `MON`, `TUE`, `WED`, `THU`, `FRI`, `SAT`, or `SUN`. * `hour`: Sequence number of UTC hour interval, from `1` to `24`. > For example, `1` stands for the interval from `00:00` to `01:00`, and `5`, from `04:00` to `05:00`. ## Combined mode and host configuration transfer warning {#note-warning-combined-mode} {% note warning %} If you are creating a [KRaft (combined mode)](../concepts/kraft.md#cluster-topology) cluster, do not provide the KRaft host configuration. {% endnote %} ## Cluster deletion protection {#protect-from-deletion} {% note info %} Even with cluster deletion protection enabled, one can still delete a user or topic or connect manually and delete the data. {% endnote %} * `--kafka-ui-enabled`: This setting defines whether to use [Kafka UI for Apache Kafka®](../concepts/kafka-ui.md) and can be `true` or `false`. * `--disk-encryption-key-id`: ID of the [custom KMS key](../../kms/concepts/key.md). To encrypt the disk, provide the KMS key ID in this parameter. Learn more about disk encryption in [Storage](../concepts/storage.md#disk-encryption). {% note warning %} You can enable disk encryption only when creating a cluster. {% endnote %} {% note tip %} You can also configure the [Apache Kafka® settings](../concepts/settings-list.md#cluster-settings) here, if required. {% endnote %} 1. To use ZooKeeper in your cluster, provide the [ZooKeeper host configuration](../concepts/index.md#zookeeper): ```bash yc managed-kafka cluster create \ ... --zookeeper-resource-preset \ --zookeeper-disk-size \ --zookeeper-disk-type \ ... ``` Where: * `--zookeeper-resource-preset`: ZooKeeper [host class](../concepts/instance-types.md). * `--zookeeper-disk-size`: Storage size. * `--zookeeper-disk-type`: ZooKeeper [disk type](../concepts/storage.md). 1. ## Environment {#environment} Select the environment where you want to create your Managed Service for Apache Kafka® cluster (note that you cannot change the environment once the cluster is created): * `PRODUCTION`: For stable versions of your apps. * `PRESTABLE`: For testing purposes. The prestable environment is similar to the production environment and likewise covered by an SLA, but it is the first to get new features, improvements, and bug fixes. In the prestable environment, you can test new versions for compatibility with your application. ## Host class {#broker-class} Under **Host class**, select the [platform](../../compute/concepts/vm-platforms.md), host type, and host class. The host class defines the specifications of [virtual machines](../../compute/concepts/vm.md) Apache Kafka® nodes will be deployed on. All available options are listed under [Host classes](../concepts/instance-types.md). When you [change the host class](cluster-update.md#change-brokers) for a Managed Service for Apache Kafka® cluster, the specifications of all existing instances also change. ## Default host configuration {#default-config} ZooKeeper and KRaft hosts have the following configuration by default: * [Host class](../concepts/instance-types.md): `s3-c2-m8` * Disk size: `10` GB * [Disk type](../concepts/storage.md): `network-ssd` ## Automatic storage expansion {#auto-rescale} To prevent the cluster from running out of disk space, create a cluster with [automatic storage expansion](../concepts/storage.md#auto-rescale): ```bash yc managed-kafka cluster create \ ... --disk-size-autoscaling disk-size-limit=,` `planned-usage-threshold=,` `emergency-usage-threshold= \ ... ``` Where: * `planned-usage-threshold`: Storage usage percentage to trigger a storage expansion during the next [maintenance window](../concepts/maintenance.md#maintenance-window). Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. If you set this condition, configure the maintenance schedule. * `emergency-usage-threshold`: Storage usage percentage to trigger an immediate storage expansion. Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. The value of this setting must be greater than or equal to `planned-usage-threshold`. * `disk-size-limit`: Maximum storage size, in bytes, that can be set when storage usage reaches one of the specified thresholds. If you set it to `0`, automatic storage expansion will be disabled. {% note warning %} You cannot reduce the storage size following an automatic expansion. {% endnote %} ## Cluster status {#status} Wait until the Managed Service for Apache Kafka® cluster is ready: its status on the Managed Service for Apache Kafka® dashboard will change to `Running`, and its state, to `Alive`. This may take a while. ## Availability zone and cluster creation warning {#note-warning} {% note warning %} If you create a Managed Service for Apache Kafka® cluster with a single [availability zone](../../overview/concepts/geo-scope.md), you will not be able to increase the number of zones and broker hosts later. {% endnote %} ## Maintenance window setup {#maintenance} To set up a [maintenance window](../concepts/maintenance.md) (including for stopped Managed Service for Apache Kafka® clusters), provide the relevant value in the `--maintenance-window` parameter when creating your cluster: ```bash yc managed-kafka cluster create \ ... --maintenance-window type=,` `day=,` `hour= \ ... ``` Where `type` is the maintenance type: * `anytime`: Any time (default). * `weekly`: On a schedule. For this value, also specify the following: * `day`: Day of week, i.e., `MON`, `TUE`, `WED`, `THU`, `FRI`, `SAT`, or `SUN`. * `hour`: Sequence number of UTC hour interval, from `1` to `24`. > For example, `1` stands for the interval from `00:00` to `01:00`, and `5`, from `04:00` to `05:00`. ## Combined mode and host configuration transfer warning {#note-warning-combined-mode} {% note warning %} If you are creating a [KRaft (combined mode)](../concepts/kraft.md#cluster-topology) cluster, do not provide the KRaft host configuration. {% endnote %} ## Cluster deletion protection {#protect-from-deletion} {% note info %} Even with cluster deletion protection enabled, one can still delete a user or topic or connect manually and delete the data. {% endnote %} 1. ## Environment {#environment} Select the environment where you want to create your Managed Service for Apache Kafka® cluster (note that you cannot change the environment once the cluster is created): * `PRODUCTION`: For stable versions of your apps. * `PRESTABLE`: For testing purposes. The prestable environment is similar to the production environment and likewise covered by an SLA, but it is the first to get new features, improvements, and bug fixes. In the prestable environment, you can test new versions for compatibility with your application. ## Host class {#broker-class} Under **Host class**, select the [platform](../../compute/concepts/vm-platforms.md), host type, and host class. The host class defines the specifications of [virtual machines](../../compute/concepts/vm.md) Apache Kafka® nodes will be deployed on. All available options are listed under [Host classes](../concepts/instance-types.md). When you [change the host class](cluster-update.md#change-brokers) for a Managed Service for Apache Kafka® cluster, the specifications of all existing instances also change. ## Default host configuration {#default-config} ZooKeeper and KRaft hosts have the following configuration by default: * [Host class](../concepts/instance-types.md): `s3-c2-m8` * Disk size: `10` GB * [Disk type](../concepts/storage.md): `network-ssd` ## Automatic storage expansion {#auto-rescale} To prevent the cluster from running out of disk space, create a cluster with [automatic storage expansion](../concepts/storage.md#auto-rescale): ```bash yc managed-kafka cluster create \ ... --disk-size-autoscaling disk-size-limit=,` `planned-usage-threshold=,` `emergency-usage-threshold= \ ... ``` Where: * `planned-usage-threshold`: Storage usage percentage to trigger a storage expansion during the next [maintenance window](../concepts/maintenance.md#maintenance-window). Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. If you set this condition, configure the maintenance schedule. * `emergency-usage-threshold`: Storage usage percentage to trigger an immediate storage expansion. Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. The value of this setting must be greater than or equal to `planned-usage-threshold`. * `disk-size-limit`: Maximum storage size, in bytes, that can be set when storage usage reaches one of the specified thresholds. If you set it to `0`, automatic storage expansion will be disabled. {% note warning %} You cannot reduce the storage size following an automatic expansion. {% endnote %} ## Cluster status {#status} Wait until the Managed Service for Apache Kafka® cluster is ready: its status on the Managed Service for Apache Kafka® dashboard will change to `Running`, and its state, to `Alive`. This may take a while. ## Availability zone and cluster creation warning {#note-warning} {% note warning %} If you create a Managed Service for Apache Kafka® cluster with a single [availability zone](../../overview/concepts/geo-scope.md), you will not be able to increase the number of zones and broker hosts later. {% endnote %} ## Maintenance window setup {#maintenance} To set up a [maintenance window](../concepts/maintenance.md) (including for stopped Managed Service for Apache Kafka® clusters), provide the relevant value in the `--maintenance-window` parameter when creating your cluster: ```bash yc managed-kafka cluster create \ ... --maintenance-window type=,` `day=,` `hour= \ ... ``` Where `type` is the maintenance type: * `anytime`: Any time (default). * `weekly`: On a schedule. For this value, also specify the following: * `day`: Day of week, i.e., `MON`, `TUE`, `WED`, `THU`, `FRI`, `SAT`, or `SUN`. * `hour`: Sequence number of UTC hour interval, from `1` to `24`. > For example, `1` stands for the interval from `00:00` to `01:00`, and `5`, from `04:00` to `05:00`. ## Combined mode and host configuration transfer warning {#note-warning-combined-mode} {% note warning %} If you are creating a [KRaft (combined mode)](../concepts/kraft.md#cluster-topology) cluster, do not provide the KRaft host configuration. {% endnote %} ## Cluster deletion protection {#protect-from-deletion} {% note info %} Even with cluster deletion protection enabled, one can still delete a user or topic or connect manually and delete the data. {% endnote %} - Terraform {#tf} With [Terraform](https://www.terraform.io/), you can quickly create a cloud infrastructure in Yandex Cloud and manage it using configuration files. These files store the infrastructure description written in HashiCorp Configuration Language (HCL). If you change the configuration files, Terraform automatically detects which part of your configuration is already deployed, and what should be added or removed. Terraform is distributed under the [Business Source License](https://github.com/hashicorp/terraform/blob/main/LICENSE). The [Yandex Cloud provider for Terraform](https://github.com/yandex-cloud/terraform-provider-yandex) is distributed under the [MPL-2.0](https://www.mozilla.org/en-US/MPL/2.0/) license. For more information about the provider resources, see the guides on the [Terraform](https://www.terraform.io/docs/providers/yandex/index.html) website or [its mirror](../../terraform/index.md). If you do not have Terraform yet, [install it and configure the Yandex Cloud provider](../../tutorials/infrastructure-management/terraform-quickstart.md#install-terraform). To manage infrastructure using Terraform under a service account or user accounts (a Yandex account, a federated account, or a local user), [authenticate](../../terraform/authentication.md) using the appropriate method. To create a Managed Service for Apache Kafka® cluster: 1. In the configuration file, describe the resources you are creating: * Managed Service for Apache Kafka® cluster: Description of the cluster and its hosts. You can also configure the [Apache Kafka® settings](../concepts/settings-list.md#cluster-settings) here, if required. * Network: Description of the [cloud network](../../vpc/concepts/network.md#network) where a cluster will be located. If you already have a suitable network, you don't have to describe it again. * Subnets: Description of the [subnets](../../vpc/concepts/network.md#subnet) to connect the cluster hosts to. If you already have suitable subnets, you don't have to describe them again. Here is an example of the configuration file structure: ```hcl resource "yandex_mdb_kafka_cluster" "" { environment = "" name = "" network_id = "" subnet_ids = [""] security_group_ids = [""] deletion_protection = config { version = "" zones = [""] brokers_count = assign_public_ip = "" schema_registry = "" kafka_ui { enabled = } kafka { resources { disk_size = disk_type_id = "" resource_preset_id = "" } kafka_config {} } } } resource "yandex_vpc_network" "" { name = "" } resource "yandex_vpc_subnet" "" { name = "" zone = "" network_id = "" v4_cidr_blocks = [""] } ``` Where: * `environment`: Cluster environment, `PRESTABLE` or `PRODUCTION`. * `version`: Apache Kafka® version, 3.6, 3.7, 3.8 or 3.9. Additionally, provide the [ZooKeeper host configuration](../concepts/index.md#zookeeper). * `zones` and `brokers_count`: Availability zones and number of broker hosts per zone. * `deletion_protection`: Cluster deletion protection, `true` or `false`. ## Environment {#environment} Select the environment where you want to create your Managed Service for Apache Kafka® cluster (note that you cannot change the environment once the cluster is created): * `PRODUCTION`: For stable versions of your apps. * `PRESTABLE`: For testing purposes. The prestable environment is similar to the production environment and likewise covered by an SLA, but it is the first to get new features, improvements, and bug fixes. In the prestable environment, you can test new versions for compatibility with your application. ## Host class {#broker-class} Under **Host class**, select the [platform](../../compute/concepts/vm-platforms.md), host type, and host class. The host class defines the specifications of [virtual machines](../../compute/concepts/vm.md) Apache Kafka® nodes will be deployed on. All available options are listed under [Host classes](../concepts/instance-types.md). When you [change the host class](cluster-update.md#change-brokers) for a Managed Service for Apache Kafka® cluster, the specifications of all existing instances also change. ## Default host configuration {#default-config} ZooKeeper and KRaft hosts have the following configuration by default: * [Host class](../concepts/instance-types.md): `s3-c2-m8` * Disk size: `10` GB * [Disk type](../concepts/storage.md): `network-ssd` ## Automatic storage expansion {#auto-rescale} To prevent the cluster from running out of disk space, create a cluster with [automatic storage expansion](../concepts/storage.md#auto-rescale): ```bash yc managed-kafka cluster create \ ... --disk-size-autoscaling disk-size-limit=,` `planned-usage-threshold=,` `emergency-usage-threshold= \ ... ``` Where: * `planned-usage-threshold`: Storage usage percentage to trigger a storage expansion during the next [maintenance window](../concepts/maintenance.md#maintenance-window). Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. If you set this condition, configure the maintenance schedule. * `emergency-usage-threshold`: Storage usage percentage to trigger an immediate storage expansion. Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. The value of this setting must be greater than or equal to `planned-usage-threshold`. * `disk-size-limit`: Maximum storage size, in bytes, that can be set when storage usage reaches one of the specified thresholds. If you set it to `0`, automatic storage expansion will be disabled. {% note warning %} You cannot reduce the storage size following an automatic expansion. {% endnote %} ## Cluster status {#status} Wait until the Managed Service for Apache Kafka® cluster is ready: its status on the Managed Service for Apache Kafka® dashboard will change to `Running`, and its state, to `Alive`. This may take a while. ## Availability zone and cluster creation warning {#note-warning} {% note warning %} If you create a Managed Service for Apache Kafka® cluster with a single [availability zone](../../overview/concepts/geo-scope.md), you will not be able to increase the number of zones and broker hosts later. {% endnote %} ## Maintenance window setup {#maintenance} To set up a [maintenance window](../concepts/maintenance.md) (including for stopped Managed Service for Apache Kafka® clusters), provide the relevant value in the `--maintenance-window` parameter when creating your cluster: ```bash yc managed-kafka cluster create \ ... --maintenance-window type=,` `day=,` `hour= \ ... ``` Where `type` is the maintenance type: * `anytime`: Any time (default). * `weekly`: On a schedule. For this value, also specify the following: * `day`: Day of week, i.e., `MON`, `TUE`, `WED`, `THU`, `FRI`, `SAT`, or `SUN`. * `hour`: Sequence number of UTC hour interval, from `1` to `24`. > For example, `1` stands for the interval from `00:00` to `01:00`, and `5`, from `04:00` to `05:00`. ## Combined mode and host configuration transfer warning {#note-warning-combined-mode} {% note warning %} If you are creating a [KRaft (combined mode)](../concepts/kraft.md#cluster-topology) cluster, do not provide the KRaft host configuration. {% endnote %} ## Cluster deletion protection {#protect-from-deletion} {% note info %} Even with cluster deletion protection enabled, one can still delete a user or topic or connect manually and delete the data. {% endnote %} * `disk_encryption_key_id`: Disk encryption using a [custom KMS key](../../kms/concepts/key.md). Provide the KMS key ID to encrypt the disk. Learn more about disk encryption in [Disk encryption](../concepts/storage.md#disk-encryption). * `assign_public_ip`: Public access to the cluster, `true` or `false`. * `schema_registry`: Manage data schemas using [Managed Schema Registry](../concepts/managed-schema-registry.md), `true` or `false`. The default value is `false`. {% note warning %} You cannot disable data schema management using Managed Schema Registry after enabling it. {% endnote %} * `kafka_ui`: This setting defines whether to use [Kafka UI for Apache Kafka®](../concepts/kafka-ui.md) and can be `true` or `false`. The default value is `false`. To use ZooKeeper in the cluster, add the `zookeeper` section to the cluster description: ```hcl resource "yandex_mdb_kafka_cluster" "" { ... zookeeper { resources { resource_preset_id = "" disk_type_id = "" disk_size = } } } ``` To set up the [maintenance window](../concepts/maintenance.md) that will also apply to stopped clusters, add the `maintenance_window` section to the cluster description: ```hcl resource "yandex_mdb_kafka_cluster" "" { ... maintenance_window { type = "" day = "" hour = } ... } ``` Where: * `type`: Maintenance type. The possible values include: * `ANYTIME`: Any time. * `WEEKLY`: On a schedule. * `day`: Day of week for the `WEEKLY` type, i.e., `MON`, `TUE`, `WED`, `THU`, `FRI`, `SAT`, or `SUN`. * `hour`: Sequence number of UTC hour interval for the `WEEKLY` type, from `1` to `24`. > For example, `1` stands for the interval from `00:00` to `01:00`, and `5`, from `04:00` to `05:00`. To encrypt disks with a [custom KMS key](../../kms/concepts/key.md), add the `disk_encryption_key_id` parameter: ```hcl resource "yandex_mdb_kafka_cluster" "" { ... disk_encryption_key_id = ... } ``` To learn more about disk encryption, see [Storage](../concepts/storage.md#disk-encryption). 1. Make sure the settings are correct. 1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure. 1. Run this command: ```bash terraform validate ``` Terraform will show any errors found in your configuration files. 1. Create a Managed Service for Apache Kafka® cluster. 1. Run this command to view the planned changes: ```bash terraform plan ``` If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources. 1. If everything looks correct, apply the changes: 1. Run this command: ```bash terraform apply ``` 1. Confirm updating the resources. 1. Wait for the operation to complete. This will create all the resources you need in the specified folder, and the terminal will display the [FQDNs of the Managed Service for Apache Kafka® cluster hosts](../concepts/network.md#hostname). You can check the new resources and their settings in the [management console](https://console.yandex.cloud). For more information, see [this Terraform provider guide](../../terraform/resources/mdb_kafka_cluster.md). {% note warning "Timeouts" %} The Terraform provider limits the time for all operations with the Managed Service for Apache Kafka® cluster to 60 minutes. Operations exceeding the timeout are aborted. {% cut "How do I change these limits?" %} Add the `timeouts` section to your cluster description, such as the following: ```hcl resource "yandex_mdb_kafka_cluster" "" { ... timeouts { create = "1h30m" # 1 hour 30 minutes update = "2h" # 2 hours delete = "30m" # 30 minutes } } ``` {% endcut %} {% endnote %} - REST API {#api} 1. [Get an IAM token for API authentication](../api-ref/authentication.md) and place it in an environment variable: ```bash export IAM_TOKEN="" ``` 1. Call the [Cluster.create](../api-ref/Cluster/create.md) method, e.g., via the following [cURL](https://curl.se/) request: 1. Create a file named `body.json` and paste the following code into it: {% note info %} This example does not use all available parameters. {% endnote %} ```json { "folderId": "", "name": "", "environment": "", "networkId": "", "securityGroupIds": [ "", "", ... "" ], "configSpec": { "version": "", "kafka": { "resources": { "resourcePresetId": "", "diskSize": "", "diskTypeId": "" } }, "zookeeper": { "resources": { "resourcePresetId": "", "diskSize": "", "diskTypeId": "" } }, "zoneId": [ ], "brokersCount": "", "assignPublicIp": , "schemaRegistry": , "restApiConfig": { "enabled": }, "diskSizeAutoscaling": { }, "kafkaUiConfig": { "enabled": } }, "topicSpecs": [ { "name": "", "partitions": "", "replicationFactor": "" }, { }, { ... }, { } ], "userSpecs": [ { "name": "", "password": "", "permissions": [ { "topicName": "", "role": "" } ] }, { }, { ... }, { } ], "maintenanceWindow": { "anytime": {}, "weeklyMaintenanceWindow": { "day": "", "hour": "" } }, "deletionProtection": , "diskEncryptionKeyId": "" } ``` Where: * `name`: Cluster name. * `environment`: Cluster environment, `PRODUCTION` or `PRESTABLE`. * `networkId`: ID of the [network](../../vpc/concepts/network.md) where the cluster will be deployed. * `securityGroupIds`: [Security group](../../vpc/concepts/security-groups.md) IDs as an array of strings. Each string is a security group ID. * `configSpec`: Cluster configuration: * `version`: Apache Kafka® version, 3.6, 3.7, 3.8 or 3.9. Additionally, provide the [ZooKeeper host configuration](../concepts/index.md#zookeeper). * `kafka`: Apache Kafka® configuration: * `resources.resourcePresetId`: [Host class](../concepts/instance-types.md) ID. You can get the list of available host classes with their IDs by calling the [ResourcePreset.list](../api-ref/ResourcePreset/list.md) method. * `resources.diskSize`: Disk size, in bytes. * `resources.diskTypeId`: [Disk type](../concepts/storage.md). * `zookeeper`: ZooKeeper configuration: * `resources.resourcePresetId`: Host class ID. You can get the list of available host classes with their IDs using the [ResourcePreset.list](../api-ref/ResourcePreset/list.md) method. * `resources.diskSize`: Disk size, in bytes. * `resources.diskTypeId`: Disk type. * `zoneId` and `brokersCount`: Availability zones and number of broker hosts per zone. * `assignPublicIp`: Access to broker hosts from the internet, `true` or `false`. * `schemaRegistry`: Manage data schemas using [Managed Schema Registry](../concepts/managed-schema-registry.md), `true` or `false`. The default value is `false`. You will not be able to edit this setting once you create the Managed Service for Apache Kafka® cluster. * `restApiConfig`: Apache Kafka® REST API configuration. To enable sending Apache Kafka® REST API requests, specify `enabled: true`. * `diskSizeAutoscaling`: Storage [utilization thresholds](../concepts/storage.md#auto-rescale) (as a percentage of the total storage size) that will trigger storage expansion when reached: * `plannedUsageThreshold`: Storage usage percentage to trigger a storage expansion during the next [maintenance window](../concepts/maintenance.md#maintenance-window). Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. If you set this condition, configure the maintenance window schedule. * `emergencyUsageThreshold`: Storage usage percentage to trigger an immediate storage expansion. Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. The value of this setting must be greater than or equal to `plannedUsageThreshold`. * `diskSizeLimit`: Maximum storage size, in bytes, that can be set when storage usage reaches one of the specified thresholds. * `kafkaUiConfig`: Use [Kafka UI](../concepts/kafka-ui.md). For access to Kafka UI, specify `enabled: true`. * `topicSpecs`: Topic settings as an array of elements, one per topic. Each element has the following structure: * `name`: Topic name. {% note info %} Use the [Apache Kafka® Admin API](cluster-topics.md#admin-api) if you need to create a topic that starts with `_`. You cannot create such a topic using the Yandex Cloud interfaces. {% endnote %} * `partitions`: Number of partitions. * `replicationFactor`: Replication factor. * `userSpecs`: User settings as an array of elements, one per user. Each element has the following structure: * `name`: Username. It may contain Latin letters, numbers, hyphens, and underscores, and must start with a letter or underscore. * `password`: User password. The password must be from 8 to 128 characters long. * `permissions`: List of topics the user should have access to. The list appears as an array of elements one per topic. Each element has the following structure: * `topicName`: Topic name or name pattern: * `*`: To allow access to all topics. * Full topic name: To allow access to a specific topic. * `*`: To grant access to topics whose names start with the specified prefix. Let’s assume you have topics named `topic_a1`, `topic_a2`, and `a3`. If you specify `topic*`, access will be granted to `topic_a1` and `topic_a2`. To cover all cluster's topics, use `*`. * `role`: User’s role, `ACCESS_ROLE_CONSUMER`, `ACCESS_ROLE_PRODUCER`, or `ACCESS_ROLE_ADMIN`. The `ACCESS_ROLE_ADMIN` role is only available if all topics are selected (`topicName: "*"`). * `allowHosts` (optional): List of IP addresses the user is allowed to access the topic from. * `maintenanceWindow`: [Maintenance window](../concepts/maintenance.md) settings, including for stopped clusters. Select one of these options: * `anytime`: Any time (default). * `weeklyMaintenanceWindow`: On schedule: * `day`: Day of week in `DDD` format, i.e., `MON`, `TUE`, `WED`, `THU`, `FRI`, `SAT`, or `SUN`. * `hour`: Time of day (UTC) in `HH` format, from `1` to `24`. * `deletionProtection`: Cluster protection against accidental deletion, `true` or `false`. The default value is `false`. ## Environment {#environment} Select the environment where you want to create your Managed Service for Apache Kafka® cluster (note that you cannot change the environment once the cluster is created): * `PRODUCTION`: For stable versions of your apps. * `PRESTABLE`: For testing purposes. The prestable environment is similar to the production environment and likewise covered by an SLA, but it is the first to get new features, improvements, and bug fixes. In the prestable environment, you can test new versions for compatibility with your application. ## Host class {#broker-class} Under **Host class**, select the [platform](../../compute/concepts/vm-platforms.md), host type, and host class. The host class defines the specifications of [virtual machines](../../compute/concepts/vm.md) Apache Kafka® nodes will be deployed on. All available options are listed under [Host classes](../concepts/instance-types.md). When you [change the host class](cluster-update.md#change-brokers) for a Managed Service for Apache Kafka® cluster, the specifications of all existing instances also change. ## Default host configuration {#default-config} ZooKeeper and KRaft hosts have the following configuration by default: * [Host class](../concepts/instance-types.md): `s3-c2-m8` * Disk size: `10` GB * [Disk type](../concepts/storage.md): `network-ssd` ## Automatic storage expansion {#auto-rescale} To prevent the cluster from running out of disk space, create a cluster with [automatic storage expansion](../concepts/storage.md#auto-rescale): ```bash yc managed-kafka cluster create \ ... --disk-size-autoscaling disk-size-limit=,` `planned-usage-threshold=,` `emergency-usage-threshold= \ ... ``` Where: * `planned-usage-threshold`: Storage usage percentage to trigger a storage expansion during the next [maintenance window](../concepts/maintenance.md#maintenance-window). Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. If you set this condition, configure the maintenance schedule. * `emergency-usage-threshold`: Storage usage percentage to trigger an immediate storage expansion. Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. The value of this setting must be greater than or equal to `planned-usage-threshold`. * `disk-size-limit`: Maximum storage size, in bytes, that can be set when storage usage reaches one of the specified thresholds. If you set it to `0`, automatic storage expansion will be disabled. {% note warning %} You cannot reduce the storage size following an automatic expansion. {% endnote %} ## Cluster status {#status} Wait until the Managed Service for Apache Kafka® cluster is ready: its status on the Managed Service for Apache Kafka® dashboard will change to `Running`, and its state, to `Alive`. This may take a while. ## Availability zone and cluster creation warning {#note-warning} {% note warning %} If you create a Managed Service for Apache Kafka® cluster with a single [availability zone](../../overview/concepts/geo-scope.md), you will not be able to increase the number of zones and broker hosts later. {% endnote %} ## Maintenance window setup {#maintenance} To set up a [maintenance window](../concepts/maintenance.md) (including for stopped Managed Service for Apache Kafka® clusters), provide the relevant value in the `--maintenance-window` parameter when creating your cluster: ```bash yc managed-kafka cluster create \ ... --maintenance-window type=,` `day=,` `hour= \ ... ``` Where `type` is the maintenance type: * `anytime`: Any time (default). * `weekly`: On a schedule. For this value, also specify the following: * `day`: Day of week, i.e., `MON`, `TUE`, `WED`, `THU`, `FRI`, `SAT`, or `SUN`. * `hour`: Sequence number of UTC hour interval, from `1` to `24`. > For example, `1` stands for the interval from `00:00` to `01:00`, and `5`, from `04:00` to `05:00`. ## Combined mode and host configuration transfer warning {#note-warning-combined-mode} {% note warning %} If you are creating a [KRaft (combined mode)](../concepts/kraft.md#cluster-topology) cluster, do not provide the KRaft host configuration. {% endnote %} ## Cluster deletion protection {#protect-from-deletion} {% note info %} Even with cluster deletion protection enabled, one can still delete a user or topic or connect manually and delete the data. {% endnote %} * `diskEncryptionKeyId`: ID of the [custom KMS key](../../kms/concepts/key.md). Provide the KMS key ID to encrypt the disks. Learn more about disk encryption in [Disk encryption](../concepts/storage.md#disk-encryption). {% note warning %} You can enable disk encryption only when creating a cluster. {% endnote %} You can get the folder ID with the [list of folders in the cloud](../../resource-manager/operations/folder/get-id.md). 1. Run this request: ```bash curl \ --request POST \ --header "Authorization: Bearer $IAM_TOKEN" \ --header "Content-Type: application/json" \ --url 'https://mdb.api.cloud.yandex.net/managed-kafka/v1/clusters' \ --data '@body.json' ``` 1. View the [server response](../api-ref/Cluster/create.md#yandex.cloud.operation.Operation) to make sure your request was successful. - gRPC API {#grpc-api} 1. [Get an IAM token for API authentication](../api-ref/authentication.md) and put it into an environment variable: ```bash export IAM_TOKEN="" ``` 1. Clone the [cloudapi](https://github.com/yandex-cloud/cloudapi) repository: ```bash cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi ``` Below, we assume that the repository contents reside in the `~/cloudapi/` directory. 1. Call the [ClusterService/Create](../api-ref/grpc/Cluster/create.md) method, e.g., via the following [gRPCurl](https://github.com/fullstorydev/grpcurl) request: 1. Create a file named `body.json` and paste the following code into it: {% note info %} This example does not use all available parameters. {% endnote %} ```json { "folder_id": "", "name": "", "environment": "", "network_id": "", "security_group_ids": [ "", "", ... "" ], "config_spec": { "version": "", "kafka": { "resources": { "resource_preset_id": "", "disk_size": "", "disk_type_id": "" } }, "zookeeper": { "resources": { "resource_preset_id": "", "disk_size": "", "disk_type_id": "" } }, "zone_id": [ ], "brokers_count": { "value": "" }, "assign_public_ip": , "schema_registry": , "rest_api_config": { "enabled": }, "disk_size_autoscaling": { }, "kafka_ui_config": { "enabled": } }, "topic_specs": [ { "name": "", "partitions": { "value": "" }, "replication_factor": { "value": "" } }, { }, { ... }, { } ], "user_specs": [ { "name": "", "password": "", "permissions": [ { "topic_name": "", "role": "" } ] }, { }, { ... }, { } ], "maintenance_window": { "anytime": {}, "weekly_maintenance_window": { "day": "", "hour": "" } }, "deletion_protection": , "disk_encryption_key_id": "" } ``` Where: * `name`: Cluster name. * `environment`: Cluster environment, `PRODUCTION` or `PRESTABLE`. * `network_id`: ID of the [network](../../vpc/concepts/network.md) where the cluster will be deployed. * `security_group_ids`: [Security group](../../vpc/concepts/security-groups.md) IDs as an array of strings. Each string is a security group ID. * `config_spec`: Cluster configuration: * `version`: Apache Kafka® version, 3.6, 3.7, 3.8 or 3.9. Additionally, provide the [ZooKeeper host configuration](../concepts/index.md#zookeeper). * `kafka`: Apache Kafka® configuration: * `resources.resource_preset_id`: [Host class](../concepts/instance-types.md) ID. You can get the list of available host classes with their IDs by calling the [ResourcePreset.list](../api-ref/grpc/ResourcePreset/list.md) method. * `resources.disk_size`: Disk size, in bytes. * `resources.disk_type_id`: [Disk type](../concepts/storage.md). * `zookeeper`: ZooKeeper configuration: * `resources.resource_preset_id`: Host class ID. You can get the list of available host classes with their IDs by calling the [ResourcePreset.list](../api-ref/grpc/ResourcePreset/list.md) method. * `resources.disk_size`: Disk size, in bytes. * `resources.disk_type_id`: Disk type. * `zone_id` and `brokers_count`: Availability zones and number of broker hosts (provided as an object with the `value` field) per zone. * `assign_public_ip`: Access to broker hosts from the internet, `true` or `false`. * `schema_registry`: Manage data schemas using [Managed Schema Registry](../concepts/managed-schema-registry.md), `true` or `false`. The default value is `false`. You will not be able to edit this setting once you create the Managed Service for Apache Kafka® cluster. * `rest_api_config`: Apache Kafka® REST API configuration. To enable sending Apache Kafka® REST API requests, specify `enabled: true`. * `disk_size_autoscaling`: To prevent the cluster disk space from running out, set the storage [utilization thresholds](../concepts/storage.md#auto-rescale) (as a percentage of the total storage size) that will trigger storage expansion when reached: * `planned_usage_threshold`: Storage usage percentage to trigger a storage expansion during the next [maintenance window](../concepts/maintenance.md#maintenance-window). Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. If you set this condition, configure the maintenance window schedule. * `emergency_usage_threshold`: Storage usage percentage to trigger an immediate storage expansion. Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. The value of this setting must be greater than or equal to `planned_usage_threshold`. * `disk_size_limit`: Maximum storage size, in bytes, that can be set when storage usage reaches one of the specified thresholds. * `kafka_ui_config`: Use [Kafka UI](../concepts/kafka-ui.md). For access to Kafka UI, specify `enabled: true`. * `topic_specs`: Topic settings as an array of elements, Each element is for a separate topic and has the following structure: * `name`: Topic name. {% note info %} Use the [Apache Kafka® Admin API](cluster-topics.md#admin-api) if you need to create a topic that starts with `_`. You cannot create such a topic using the Yandex Cloud interfaces. {% endnote %} * `partitions`: Number of partitions, provided as an object with the `value` field. * `replication_factor`: Replication factor, provided as an object with the `value` field. * `user_specs`: User settings as an array of elements, one per user. Each element has the following structure: * `name`: Username. It may contain Latin letters, numbers, hyphens, and underscores, and must start with a letter or underscore. * `password`: User password. The password must be from 8 to 128 characters long. * `permissions`: List of topics the user should have access to. The list appears as an array of elements. one per topic. Each element has the following structure: * `topic_name`: Topic name or name pattern: * `*`: To allow access to all topics. * Full topic name: To allow access to a specific topic. * `*`: To grant access to topics whose names start with the specified prefix. Let’s assume you have topics named `topic_a1`, `topic_a2`, and `a3`. If you specify `topic*`, access will be granted to `topic_a1` and `topic_a2`. To cover all cluster's topics, use `*`. * `role`: User’s role, `ACCESS_ROLE_CONSUMER`, `ACCESS_ROLE_PRODUCER`, or `ACCESS_ROLE_ADMIN`. The `ACCESS_ROLE_ADMIN` role is only available if all topics are selected (`topicName: "*"`). * `allow_hosts` (optional): List of IP addresses the user is allowed to access the topic from, as an array of elements. * `maintenance_window`: [Maintenance window](../concepts/maintenance.md) settings, including for stopped clusters. Select one of these options: * `anytime`: Any time (default). * `weekly_maintenance_window`: On schedule: * `day`: Day of week in `DDD` format, i.e., `MON`, `TUE`, `WED`, `THU`, `FRI`, `SAT`, or `SUN`. * `hour`: Time of day (UTC) in `HH` format, from `1` to `24`. * `deletion_protection`: Cluster protection against accidental deletion, `true` or `false`. The default value is `false`. ## Environment {#environment} Select the environment where you want to create your Managed Service for Apache Kafka® cluster (note that you cannot change the environment once the cluster is created): * `PRODUCTION`: For stable versions of your apps. * `PRESTABLE`: For testing purposes. The prestable environment is similar to the production environment and likewise covered by an SLA, but it is the first to get new features, improvements, and bug fixes. In the prestable environment, you can test new versions for compatibility with your application. ## Host class {#broker-class} Under **Host class**, select the [platform](../../compute/concepts/vm-platforms.md), host type, and host class. The host class defines the specifications of [virtual machines](../../compute/concepts/vm.md) Apache Kafka® nodes will be deployed on. All available options are listed under [Host classes](../concepts/instance-types.md). When you [change the host class](cluster-update.md#change-brokers) for a Managed Service for Apache Kafka® cluster, the specifications of all existing instances also change. ## Default host configuration {#default-config} ZooKeeper and KRaft hosts have the following configuration by default: * [Host class](../concepts/instance-types.md): `s3-c2-m8` * Disk size: `10` GB * [Disk type](../concepts/storage.md): `network-ssd` ## Automatic storage expansion {#auto-rescale} To prevent the cluster from running out of disk space, create a cluster with [automatic storage expansion](../concepts/storage.md#auto-rescale): ```bash yc managed-kafka cluster create \ ... --disk-size-autoscaling disk-size-limit=,` `planned-usage-threshold=,` `emergency-usage-threshold= \ ... ``` Where: * `planned-usage-threshold`: Storage usage percentage to trigger a storage expansion during the next [maintenance window](../concepts/maintenance.md#maintenance-window). Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. If you set this condition, configure the maintenance schedule. * `emergency-usage-threshold`: Storage usage percentage to trigger an immediate storage expansion. Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. The value of this setting must be greater than or equal to `planned-usage-threshold`. * `disk-size-limit`: Maximum storage size, in bytes, that can be set when storage usage reaches one of the specified thresholds. If you set it to `0`, automatic storage expansion will be disabled. {% note warning %} You cannot reduce the storage size following an automatic expansion. {% endnote %} ## Cluster status {#status} Wait until the Managed Service for Apache Kafka® cluster is ready: its status on the Managed Service for Apache Kafka® dashboard will change to `Running`, and its state, to `Alive`. This may take a while. ## Availability zone and cluster creation warning {#note-warning} {% note warning %} If you create a Managed Service for Apache Kafka® cluster with a single [availability zone](../../overview/concepts/geo-scope.md), you will not be able to increase the number of zones and broker hosts later. {% endnote %} ## Maintenance window setup {#maintenance} To set up a [maintenance window](../concepts/maintenance.md) (including for stopped Managed Service for Apache Kafka® clusters), provide the relevant value in the `--maintenance-window` parameter when creating your cluster: ```bash yc managed-kafka cluster create \ ... --maintenance-window type=,` `day=,` `hour= \ ... ``` Where `type` is the maintenance type: * `anytime`: Any time (default). * `weekly`: On a schedule. For this value, also specify the following: * `day`: Day of week, i.e., `MON`, `TUE`, `WED`, `THU`, `FRI`, `SAT`, or `SUN`. * `hour`: Sequence number of UTC hour interval, from `1` to `24`. > For example, `1` stands for the interval from `00:00` to `01:00`, and `5`, from `04:00` to `05:00`. ## Combined mode and host configuration transfer warning {#note-warning-combined-mode} {% note warning %} If you are creating a [KRaft (combined mode)](../concepts/kraft.md#cluster-topology) cluster, do not provide the KRaft host configuration. {% endnote %} ## Cluster deletion protection {#protect-from-deletion} {% note info %} Even with cluster deletion protection enabled, one can still delete a user or topic or connect manually and delete the data. {% endnote %} * `disk_encryption_key_id`: ID of the [custom KMS key](../../kms/concepts/key.md). Provide the KMS key ID to encrypt the disks. Learn more about disk encryption in [Disk encryption](../concepts/storage.md#disk-encryption). {% note warning %} You can enable disk encryption only when creating a cluster. {% endnote %} You can get the folder ID with the [list of folders in the cloud](../../resource-manager/operations/folder/get-id.md). 1. Run this request: ```bash curl \ --request POST \ --header "Authorization: Bearer $IAM_TOKEN" \ --header "Content-Type: application/json" \ --url 'https://mdb.api.cloud.yandex.net/managed-kafka/v1/clusters' \ --data '@body.json' ``` 1. Check the [server response](../api-ref/Cluster/create.md#yandex.cloud.operation.Operation) to make sure your request was successful. {% endlist %} ## Creating a cluster with KRaft {#create-cluster-kraft} {% note warning %} When creating a cluster with KRaft, do not specify the ZooKeeper settings. {% endnote %} {% list tabs group=instructions %} - Management console {#console} To create a Managed Service for Apache Kafka® cluster: 1. In the [management console](https://console.yandex.cloud), go to the appropriate [folder](../../resource-manager/concepts/resources-hierarchy.md#folder). 1. Navigate to **Managed Service for Kafka**. 1. Click **Create cluster**. 1. Under **Basic parameters**: 1. Enter a name and description for the Managed Service for Apache Kafka® cluster. The Managed Service for Apache Kafka® cluster name must be unique within the folder. 1. ## Environment {#environment} Select the environment where you want to create your Managed Service for Apache Kafka® cluster (note that you cannot change the environment once the cluster is created): * `PRODUCTION`: For stable versions of your apps. * `PRESTABLE`: For testing purposes. The prestable environment is similar to the production environment and likewise covered by an SLA, but it is the first to get new features, improvements, and bug fixes. In the prestable environment, you can test new versions for compatibility with your application. ## Host class {#broker-class} Under **Host class**, select the [platform](../../compute/concepts/vm-platforms.md), host type, and host class. The host class defines the specifications of [virtual machines](../../compute/concepts/vm.md) Apache Kafka® nodes will be deployed on. All available options are listed under [Host classes](../concepts/instance-types.md). When you [change the host class](cluster-update.md#change-brokers) for a Managed Service for Apache Kafka® cluster, the specifications of all existing instances also change. ## Default host configuration {#default-config} ZooKeeper and KRaft hosts have the following configuration by default: * [Host class](../concepts/instance-types.md): `s3-c2-m8` * Disk size: `10` GB * [Disk type](../concepts/storage.md): `network-ssd` ## Automatic storage expansion {#auto-rescale} To prevent the cluster from running out of disk space, create a cluster with [automatic storage expansion](../concepts/storage.md#auto-rescale): ```bash yc managed-kafka cluster create \ ... --disk-size-autoscaling disk-size-limit=,` `planned-usage-threshold=,` `emergency-usage-threshold= \ ... ``` Where: * `planned-usage-threshold`: Storage usage percentage to trigger a storage expansion during the next [maintenance window](../concepts/maintenance.md#maintenance-window). Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. If you set this condition, configure the maintenance schedule. * `emergency-usage-threshold`: Storage usage percentage to trigger an immediate storage expansion. Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. The value of this setting must be greater than or equal to `planned-usage-threshold`. * `disk-size-limit`: Maximum storage size, in bytes, that can be set when storage usage reaches one of the specified thresholds. If you set it to `0`, automatic storage expansion will be disabled. {% note warning %} You cannot reduce the storage size following an automatic expansion. {% endnote %} ## Cluster status {#status} Wait until the Managed Service for Apache Kafka® cluster is ready: its status on the Managed Service for Apache Kafka® dashboard will change to `Running`, and its state, to `Alive`. This may take a while. ## Availability zone and cluster creation warning {#note-warning} {% note warning %} If you create a Managed Service for Apache Kafka® cluster with a single [availability zone](../../overview/concepts/geo-scope.md), you will not be able to increase the number of zones and broker hosts later. {% endnote %} ## Maintenance window setup {#maintenance} To set up a [maintenance window](../concepts/maintenance.md) (including for stopped Managed Service for Apache Kafka® clusters), provide the relevant value in the `--maintenance-window` parameter when creating your cluster: ```bash yc managed-kafka cluster create \ ... --maintenance-window type=,` `day=,` `hour= \ ... ``` Where `type` is the maintenance type: * `anytime`: Any time (default). * `weekly`: On a schedule. For this value, also specify the following: * `day`: Day of week, i.e., `MON`, `TUE`, `WED`, `THU`, `FRI`, `SAT`, or `SUN`. * `hour`: Sequence number of UTC hour interval, from `1` to `24`. > For example, `1` stands for the interval from `00:00` to `01:00`, and `5`, from `04:00` to `05:00`. ## Combined mode and host configuration transfer warning {#note-warning-combined-mode} {% note warning %} If you are creating a [KRaft (combined mode)](../concepts/kraft.md#cluster-topology) cluster, do not provide the KRaft host configuration. {% endnote %} ## Cluster deletion protection {#protect-from-deletion} {% note info %} Even with cluster deletion protection enabled, one can still delete a user or topic or connect manually and delete the data. {% endnote %} 1. Select Apache Kafka® 3.6 or higher. 1. ## Environment {#environment} Select the environment where you want to create your Managed Service for Apache Kafka® cluster (note that you cannot change the environment once the cluster is created): * `PRODUCTION`: For stable versions of your apps. * `PRESTABLE`: For testing purposes. The prestable environment is similar to the production environment and likewise covered by an SLA, but it is the first to get new features, improvements, and bug fixes. In the prestable environment, you can test new versions for compatibility with your application. ## Host class {#broker-class} Under **Host class**, select the [platform](../../compute/concepts/vm-platforms.md), host type, and host class. The host class defines the specifications of [virtual machines](../../compute/concepts/vm.md) Apache Kafka® nodes will be deployed on. All available options are listed under [Host classes](../concepts/instance-types.md). When you [change the host class](cluster-update.md#change-brokers) for a Managed Service for Apache Kafka® cluster, the specifications of all existing instances also change. ## Default host configuration {#default-config} ZooKeeper and KRaft hosts have the following configuration by default: * [Host class](../concepts/instance-types.md): `s3-c2-m8` * Disk size: `10` GB * [Disk type](../concepts/storage.md): `network-ssd` ## Automatic storage expansion {#auto-rescale} To prevent the cluster from running out of disk space, create a cluster with [automatic storage expansion](../concepts/storage.md#auto-rescale): ```bash yc managed-kafka cluster create \ ... --disk-size-autoscaling disk-size-limit=,` `planned-usage-threshold=,` `emergency-usage-threshold= \ ... ``` Where: * `planned-usage-threshold`: Storage usage percentage to trigger a storage expansion during the next [maintenance window](../concepts/maintenance.md#maintenance-window). Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. If you set this condition, configure the maintenance schedule. * `emergency-usage-threshold`: Storage usage percentage to trigger an immediate storage expansion. Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. The value of this setting must be greater than or equal to `planned-usage-threshold`. * `disk-size-limit`: Maximum storage size, in bytes, that can be set when storage usage reaches one of the specified thresholds. If you set it to `0`, automatic storage expansion will be disabled. {% note warning %} You cannot reduce the storage size following an automatic expansion. {% endnote %} ## Cluster status {#status} Wait until the Managed Service for Apache Kafka® cluster is ready: its status on the Managed Service for Apache Kafka® dashboard will change to `Running`, and its state, to `Alive`. This may take a while. ## Availability zone and cluster creation warning {#note-warning} {% note warning %} If you create a Managed Service for Apache Kafka® cluster with a single [availability zone](../../overview/concepts/geo-scope.md), you will not be able to increase the number of zones and broker hosts later. {% endnote %} ## Maintenance window setup {#maintenance} To set up a [maintenance window](../concepts/maintenance.md) (including for stopped Managed Service for Apache Kafka® clusters), provide the relevant value in the `--maintenance-window` parameter when creating your cluster: ```bash yc managed-kafka cluster create \ ... --maintenance-window type=,` `day=,` `hour= \ ... ``` Where `type` is the maintenance type: * `anytime`: Any time (default). * `weekly`: On a schedule. For this value, also specify the following: * `day`: Day of week, i.e., `MON`, `TUE`, `WED`, `THU`, `FRI`, `SAT`, or `SUN`. * `hour`: Sequence number of UTC hour interval, from `1` to `24`. > For example, `1` stands for the interval from `00:00` to `01:00`, and `5`, from `04:00` to `05:00`. ## Combined mode and host configuration transfer warning {#note-warning-combined-mode} {% note warning %} If you are creating a [KRaft (combined mode)](../concepts/kraft.md#cluster-topology) cluster, do not provide the KRaft host configuration. {% endnote %} ## Cluster deletion protection {#protect-from-deletion} {% note info %} Even with cluster deletion protection enabled, one can still delete a user or topic or connect manually and delete the data. {% endnote %} 1. Under **Storage**: * Select the disk type. {% note warning %} You cannot change disk type after you create a cluster. {% endnote %} The type you select determines the increments for changing the disk size: * Network HDDs and SSDs: In increments of 1 GB. * Local SSDs: * For **Intel Cascade Lake**: In increments of 100 GB. * For **Intel Ice Lake** and **AMD Zen 4**: In 368 GB increments. * Non-replicated SSDs: In increments of 93 GB. You cannot change the disk type for a Managed Service for Apache Kafka® cluster once the cluster is created. * Select the storage size to use for data. 1. Under **Automatic increase of storage size**, set the storage [utilization thresholds](../concepts/storage.md#auto-rescale) that will trigger storage expansion when reached: 1. In the **Increase size** field, select one or both thresholds: * **In the maintenance window when full at more than**: Scheduled expansion threshold. When reached, the storage expands during the next [maintenance window](../concepts/maintenance.md#maintenance-window). * **Immediately when full at more than**: Immediate expansion threshold. When reached, the storage expands immediately. 1. Specify a threshold value (as a percentage of the total storage size). If you select both thresholds, make sure the immediate expansion threshold is higher than the scheduled one. 1. Set the **Maximum storage size**. 1. Under **Network settings**: 1. Select one or more availability zones for your Apache Kafka® broker hosts. ## Environment {#environment} Select the environment where you want to create your Managed Service for Apache Kafka® cluster (note that you cannot change the environment once the cluster is created): * `PRODUCTION`: For stable versions of your apps. * `PRESTABLE`: For testing purposes. The prestable environment is similar to the production environment and likewise covered by an SLA, but it is the first to get new features, improvements, and bug fixes. In the prestable environment, you can test new versions for compatibility with your application. ## Host class {#broker-class} Under **Host class**, select the [platform](../../compute/concepts/vm-platforms.md), host type, and host class. The host class defines the specifications of [virtual machines](../../compute/concepts/vm.md) Apache Kafka® nodes will be deployed on. All available options are listed under [Host classes](../concepts/instance-types.md). When you [change the host class](cluster-update.md#change-brokers) for a Managed Service for Apache Kafka® cluster, the specifications of all existing instances also change. ## Default host configuration {#default-config} ZooKeeper and KRaft hosts have the following configuration by default: * [Host class](../concepts/instance-types.md): `s3-c2-m8` * Disk size: `10` GB * [Disk type](../concepts/storage.md): `network-ssd` ## Automatic storage expansion {#auto-rescale} To prevent the cluster from running out of disk space, create a cluster with [automatic storage expansion](../concepts/storage.md#auto-rescale): ```bash yc managed-kafka cluster create \ ... --disk-size-autoscaling disk-size-limit=,` `planned-usage-threshold=,` `emergency-usage-threshold= \ ... ``` Where: * `planned-usage-threshold`: Storage usage percentage to trigger a storage expansion during the next [maintenance window](../concepts/maintenance.md#maintenance-window). Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. If you set this condition, configure the maintenance schedule. * `emergency-usage-threshold`: Storage usage percentage to trigger an immediate storage expansion. Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. The value of this setting must be greater than or equal to `planned-usage-threshold`. * `disk-size-limit`: Maximum storage size, in bytes, that can be set when storage usage reaches one of the specified thresholds. If you set it to `0`, automatic storage expansion will be disabled. {% note warning %} You cannot reduce the storage size following an automatic expansion. {% endnote %} ## Cluster status {#status} Wait until the Managed Service for Apache Kafka® cluster is ready: its status on the Managed Service for Apache Kafka® dashboard will change to `Running`, and its state, to `Alive`. This may take a while. ## Availability zone and cluster creation warning {#note-warning} {% note warning %} If you create a Managed Service for Apache Kafka® cluster with a single [availability zone](../../overview/concepts/geo-scope.md), you will not be able to increase the number of zones and broker hosts later. {% endnote %} ## Maintenance window setup {#maintenance} To set up a [maintenance window](../concepts/maintenance.md) (including for stopped Managed Service for Apache Kafka® clusters), provide the relevant value in the `--maintenance-window` parameter when creating your cluster: ```bash yc managed-kafka cluster create \ ... --maintenance-window type=,` `day=,` `hour= \ ... ``` Where `type` is the maintenance type: * `anytime`: Any time (default). * `weekly`: On a schedule. For this value, also specify the following: * `day`: Day of week, i.e., `MON`, `TUE`, `WED`, `THU`, `FRI`, `SAT`, or `SUN`. * `hour`: Sequence number of UTC hour interval, from `1` to `24`. > For example, `1` stands for the interval from `00:00` to `01:00`, and `5`, from `04:00` to `05:00`. ## Combined mode and host configuration transfer warning {#note-warning-combined-mode} {% note warning %} If you are creating a [KRaft (combined mode)](../concepts/kraft.md#cluster-topology) cluster, do not provide the KRaft host configuration. {% endnote %} ## Cluster deletion protection {#protect-from-deletion} {% note info %} Even with cluster deletion protection enabled, one can still delete a user or topic or connect manually and delete the data. {% endnote %} 1. Select the [network](../../vpc/concepts/network.md#network). 1. Select [subnets](../../vpc/concepts/network.md#subnet) in each availability zone for this network. To [create a new subnet](../../vpc/operations/subnet-create.md), click **Create** next to the availability zone. {% note info %} For an Apache Kafka® cluster with multiple broker hosts, specify subnets in each availability zone even if you plan to place broker hosts only in some of them. You need these subnets to deploy three [KRaft hosts](../concepts/index.md), one per availability zone. For more information, see [Resource relationships in Managed Service for Apache Kafka®](../concepts/index.md). {% endnote %} 1. Select [security groups](../../vpc/concepts/security-groups.md) for the Managed Service for Apache Kafka® cluster’s network traffic. 1. To enable internet access to broker hosts, select **Public access**. In this case, you can only connect to them using SSL. For more information, see [Connecting to an Apache Kafka® cluster from applications](connect/clients.md). 1. Under **Hosts**: 1. Specify the number of Apache Kafka® broker hosts to reside in each selected availability zone. When selecting the number of hosts, consider the following: * You cannot set the number of broker hosts manually if using [KRaft (combined mode)](../concepts/kraft.md#cluster-topology) as the coordination service. * You need to set the number of brokers manually if using [KRaft (on separate hosts)](../concepts/kraft.md#cluster-topology) as the coordination service. A new multi-host cluster will automatically get three separate KRaft hosts. * You need at least two hosts to enable replication in a Managed Service for Apache Kafka® cluster. * High availability of a Managed Service for Apache Kafka® cluster depends on certain [prerequisites](../concepts/ha-cluster.md). 1. Under **Coordination service**, select one of these options: * **KRaft (on separate hosts)**: Broker and KRaft metadata controller are on separate hosts. * **KRaft (combined mode)**: One Apache Kafka® host accommodates a broker and a KRaft metadata controller at the same time. You can create a cluster either in one or in three [availability zones](../../overview/concepts/geo-scope.md): * One availability zone: Three broker hosts. * Three availability zones: One broker host in each availability zone. You cannot set the number of broker hosts manually. 1. If you specified more than one broker host, under **KRaft host class**, specify the properties of the [KRaft hosts](../concepts/index.md) to reside in each selected availability zone. 1. Specify additional Managed Service for Apache Kafka® cluster settings, if required: * **Maintenance**: [Maintenance window](../concepts/maintenance.md) settings: * To enable maintenance at any time, select **At any time** (default). * To specify the preferred maintenance start time, select **By schedule** and specify the day of the week and the UTC time interval. For example, you can choose a time when the cluster is least loaded. Both active and stopped clusters are subject to maintenance. Maintenance operations include DBMS updates, patches, etc. * **Deletion protection**: Manages cluster protection against accidental deletion. Even with cluster deletion protection enabled, one can still delete a user or topic or connect manually and delete the data. * **Disk encryption**: Enable this setting to encrypt the disks with a [custom KMS key](../../kms/concepts/key.md). * To [create](../../kms/operations/key.md#create) a new key, click **Create**. * To use a previously created key, select it in the **KMS key** field. To learn more about disk encryption, see [Storage](../concepts/storage.md#disk-encryption). {% note warning %} You can enable disk encryption only when creating a cluster. {% endnote %} * **Schema registry**: Enable this setting to manage data schemas using [Managed Schema Registry](../concepts/managed-schema-registry.md). {% note warning %} You cannot disable data schema management using Managed Schema Registry after enabling it. {% endnote %} * **Kafka Rest API**: Enable this setting to allow sending requests to the Apache Kafka® API. It is implemented using [Karapace](https://github.com/Aiven-Open/karapace), an open-source tool. The Karapace API is compatible with the [Confluent REST Proxy API](https://docs.confluent.io/platform/current/kafka-rest/api.html) with only minor exceptions. {% note warning %} You cannot disable **Kafka Rest API** once it is enabled. {% endnote %} * **Kafka UI**: Enable this setting to use the [Apache Kafka®](../concepts/kafka-ui.md) web UI. 1. Configure the [Apache Kafka® settings](../concepts/settings-list.md#cluster-settings), if required. 1. Click **Create**. 1. ## Environment {#environment} Select the environment where you want to create your Managed Service for Apache Kafka® cluster (note that you cannot change the environment once the cluster is created): * `PRODUCTION`: For stable versions of your apps. * `PRESTABLE`: For testing purposes. The prestable environment is similar to the production environment and likewise covered by an SLA, but it is the first to get new features, improvements, and bug fixes. In the prestable environment, you can test new versions for compatibility with your application. ## Host class {#broker-class} Under **Host class**, select the [platform](../../compute/concepts/vm-platforms.md), host type, and host class. The host class defines the specifications of [virtual machines](../../compute/concepts/vm.md) Apache Kafka® nodes will be deployed on. All available options are listed under [Host classes](../concepts/instance-types.md). When you [change the host class](cluster-update.md#change-brokers) for a Managed Service for Apache Kafka® cluster, the specifications of all existing instances also change. ## Default host configuration {#default-config} ZooKeeper and KRaft hosts have the following configuration by default: * [Host class](../concepts/instance-types.md): `s3-c2-m8` * Disk size: `10` GB * [Disk type](../concepts/storage.md): `network-ssd` ## Automatic storage expansion {#auto-rescale} To prevent the cluster from running out of disk space, create a cluster with [automatic storage expansion](../concepts/storage.md#auto-rescale): ```bash yc managed-kafka cluster create \ ... --disk-size-autoscaling disk-size-limit=,` `planned-usage-threshold=,` `emergency-usage-threshold= \ ... ``` Where: * `planned-usage-threshold`: Storage usage percentage to trigger a storage expansion during the next [maintenance window](../concepts/maintenance.md#maintenance-window). Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. If you set this condition, configure the maintenance schedule. * `emergency-usage-threshold`: Storage usage percentage to trigger an immediate storage expansion. Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. The value of this setting must be greater than or equal to `planned-usage-threshold`. * `disk-size-limit`: Maximum storage size, in bytes, that can be set when storage usage reaches one of the specified thresholds. If you set it to `0`, automatic storage expansion will be disabled. {% note warning %} You cannot reduce the storage size following an automatic expansion. {% endnote %} ## Cluster status {#status} Wait until the Managed Service for Apache Kafka® cluster is ready: its status on the Managed Service for Apache Kafka® dashboard will change to `Running`, and its state, to `Alive`. This may take a while. ## Availability zone and cluster creation warning {#note-warning} {% note warning %} If you create a Managed Service for Apache Kafka® cluster with a single [availability zone](../../overview/concepts/geo-scope.md), you will not be able to increase the number of zones and broker hosts later. {% endnote %} ## Maintenance window setup {#maintenance} To set up a [maintenance window](../concepts/maintenance.md) (including for stopped Managed Service for Apache Kafka® clusters), provide the relevant value in the `--maintenance-window` parameter when creating your cluster: ```bash yc managed-kafka cluster create \ ... --maintenance-window type=,` `day=,` `hour= \ ... ``` Where `type` is the maintenance type: * `anytime`: Any time (default). * `weekly`: On a schedule. For this value, also specify the following: * `day`: Day of week, i.e., `MON`, `TUE`, `WED`, `THU`, `FRI`, `SAT`, or `SUN`. * `hour`: Sequence number of UTC hour interval, from `1` to `24`. > For example, `1` stands for the interval from `00:00` to `01:00`, and `5`, from `04:00` to `05:00`. ## Combined mode and host configuration transfer warning {#note-warning-combined-mode} {% note warning %} If you are creating a [KRaft (combined mode)](../concepts/kraft.md#cluster-topology) cluster, do not provide the KRaft host configuration. {% endnote %} ## Cluster deletion protection {#protect-from-deletion} {% note info %} Even with cluster deletion protection enabled, one can still delete a user or topic or connect manually and delete the data. {% endnote %} - CLI {#cli} If you do not have the Yandex Cloud CLI yet, [install and initialize it](../../cli/quickstart.md#install). The folder used by default is the one specified when [creating](../../cli/operations/profile/profile-create.md) the CLI profile. To change the default folder, use the `yc config set folder-id ` command. You can also specify a different folder for any command using `--folder-name` or `--folder-id`. If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions. To create a Managed Service for Apache Kafka® cluster: 1. View the description of the CLI command to create a Managed Service for Apache Kafka® cluster: ```bash yc managed-kafka cluster create --help ``` 1. Specify the Managed Service for Apache Kafka® cluster parameters in the create command (not all parameters are given in the example): ```bash yc managed-kafka cluster create \ --name \ --environment \ --version \ --schema-registry \ --network-name \ --subnet-ids \ --resource-preset \ --disk-type \ --disk-size \ --assign-public-ip \ --security-group-ids \ --deletion-protection \ --kafka-ui-enabled \ --disk-encryption-key-id ``` Where: * `--environment`: Cluster environment, `prestable` or `production`. * `--version`: Apache Kafka® version. Specify 3.6 or higher. * `--schema-registry`: Manage data schemas using [Managed Schema Registry](../concepts/managed-schema-registry.md). {% note warning %} You cannot disable data schema management using Managed Schema Registry after enabling it. {% endnote %} * `--zone-ids` and `--brokers-count`: Availability zones and number of broker hosts per zone. If you are creating a cluster with [KRaft (combined mode)](../concepts/kraft.md#cluster-topology), specify one of the available configurations: * `--zone-ids=ru-central1-a,ru-central1-b,ru-central1-d --brokers-count=1`: Three availability zones with one broker host per zone. * `--zone-ids= --brokers-count=3`: One availability zone with three broker hosts. * `--resource-preset`: [Host class](../concepts/instance-types.md). * `--disk-type`: [Disk type](../concepts/storage.md). {% note warning %} You cannot change disk type after you create a cluster. {% endnote %} * `--deletion-protection`: Cluster protection from accidental deletion, `true` or `false`. ## Environment {#environment} Select the environment where you want to create your Managed Service for Apache Kafka® cluster (note that you cannot change the environment once the cluster is created): * `PRODUCTION`: For stable versions of your apps. * `PRESTABLE`: For testing purposes. The prestable environment is similar to the production environment and likewise covered by an SLA, but it is the first to get new features, improvements, and bug fixes. In the prestable environment, you can test new versions for compatibility with your application. ## Host class {#broker-class} Under **Host class**, select the [platform](../../compute/concepts/vm-platforms.md), host type, and host class. The host class defines the specifications of [virtual machines](../../compute/concepts/vm.md) Apache Kafka® nodes will be deployed on. All available options are listed under [Host classes](../concepts/instance-types.md). When you [change the host class](cluster-update.md#change-brokers) for a Managed Service for Apache Kafka® cluster, the specifications of all existing instances also change. ## Default host configuration {#default-config} ZooKeeper and KRaft hosts have the following configuration by default: * [Host class](../concepts/instance-types.md): `s3-c2-m8` * Disk size: `10` GB * [Disk type](../concepts/storage.md): `network-ssd` ## Automatic storage expansion {#auto-rescale} To prevent the cluster from running out of disk space, create a cluster with [automatic storage expansion](../concepts/storage.md#auto-rescale): ```bash yc managed-kafka cluster create \ ... --disk-size-autoscaling disk-size-limit=,` `planned-usage-threshold=,` `emergency-usage-threshold= \ ... ``` Where: * `planned-usage-threshold`: Storage usage percentage to trigger a storage expansion during the next [maintenance window](../concepts/maintenance.md#maintenance-window). Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. If you set this condition, configure the maintenance schedule. * `emergency-usage-threshold`: Storage usage percentage to trigger an immediate storage expansion. Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. The value of this setting must be greater than or equal to `planned-usage-threshold`. * `disk-size-limit`: Maximum storage size, in bytes, that can be set when storage usage reaches one of the specified thresholds. If you set it to `0`, automatic storage expansion will be disabled. {% note warning %} You cannot reduce the storage size following an automatic expansion. {% endnote %} ## Cluster status {#status} Wait until the Managed Service for Apache Kafka® cluster is ready: its status on the Managed Service for Apache Kafka® dashboard will change to `Running`, and its state, to `Alive`. This may take a while. ## Availability zone and cluster creation warning {#note-warning} {% note warning %} If you create a Managed Service for Apache Kafka® cluster with a single [availability zone](../../overview/concepts/geo-scope.md), you will not be able to increase the number of zones and broker hosts later. {% endnote %} ## Maintenance window setup {#maintenance} To set up a [maintenance window](../concepts/maintenance.md) (including for stopped Managed Service for Apache Kafka® clusters), provide the relevant value in the `--maintenance-window` parameter when creating your cluster: ```bash yc managed-kafka cluster create \ ... --maintenance-window type=,` `day=,` `hour= \ ... ``` Where `type` is the maintenance type: * `anytime`: Any time (default). * `weekly`: On a schedule. For this value, also specify the following: * `day`: Day of week, i.e., `MON`, `TUE`, `WED`, `THU`, `FRI`, `SAT`, or `SUN`. * `hour`: Sequence number of UTC hour interval, from `1` to `24`. > For example, `1` stands for the interval from `00:00` to `01:00`, and `5`, from `04:00` to `05:00`. ## Combined mode and host configuration transfer warning {#note-warning-combined-mode} {% note warning %} If you are creating a [KRaft (combined mode)](../concepts/kraft.md#cluster-topology) cluster, do not provide the KRaft host configuration. {% endnote %} ## Cluster deletion protection {#protect-from-deletion} {% note info %} Even with cluster deletion protection enabled, one can still delete a user or topic or connect manually and delete the data. {% endnote %} * `--kafka-ui-enabled`: This setting defines whether to use [Kafka UI for Apache Kafka®](../concepts/kafka-ui.md) and can be `true` or `false`. * `--disk-encryption-key-id`: ID of the [custom KMS key](../../kms/concepts/key.md). To encrypt the disk, provide the KMS key ID in this parameter. Learn more about disk encryption in [Storage](../concepts/storage.md#disk-encryption). {% note warning %} You can enable disk encryption only when creating a cluster. {% endnote %} {% note tip %} You can also configure the [Apache Kafka® settings](../concepts/settings-list.md#cluster-settings) here, if required. {% endnote %} 1. To use [KRaft (on separate hosts)](../concepts/kraft.md#cluster-topology), provide the KRaft host configuration. ```bash yc managed-kafka cluster create \ ... --controller-resource-preset \ --controller-disk-size \ --controller-disk-type \ ... ``` Where: * `--controller-resource-preset`: KRaft [host class](../concepts/instance-types.md). * `--controller-disk-size`: Storage size. * `--controller-disk-type`: KRaft [disk type](../concepts/storage.md). 1. ## Environment {#environment} Select the environment where you want to create your Managed Service for Apache Kafka® cluster (note that you cannot change the environment once the cluster is created): * `PRODUCTION`: For stable versions of your apps. * `PRESTABLE`: For testing purposes. The prestable environment is similar to the production environment and likewise covered by an SLA, but it is the first to get new features, improvements, and bug fixes. In the prestable environment, you can test new versions for compatibility with your application. ## Host class {#broker-class} Under **Host class**, select the [platform](../../compute/concepts/vm-platforms.md), host type, and host class. The host class defines the specifications of [virtual machines](../../compute/concepts/vm.md) Apache Kafka® nodes will be deployed on. All available options are listed under [Host classes](../concepts/instance-types.md). When you [change the host class](cluster-update.md#change-brokers) for a Managed Service for Apache Kafka® cluster, the specifications of all existing instances also change. ## Default host configuration {#default-config} ZooKeeper and KRaft hosts have the following configuration by default: * [Host class](../concepts/instance-types.md): `s3-c2-m8` * Disk size: `10` GB * [Disk type](../concepts/storage.md): `network-ssd` ## Automatic storage expansion {#auto-rescale} To prevent the cluster from running out of disk space, create a cluster with [automatic storage expansion](../concepts/storage.md#auto-rescale): ```bash yc managed-kafka cluster create \ ... --disk-size-autoscaling disk-size-limit=,` `planned-usage-threshold=,` `emergency-usage-threshold= \ ... ``` Where: * `planned-usage-threshold`: Storage usage percentage to trigger a storage expansion during the next [maintenance window](../concepts/maintenance.md#maintenance-window). Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. If you set this condition, configure the maintenance schedule. * `emergency-usage-threshold`: Storage usage percentage to trigger an immediate storage expansion. Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. The value of this setting must be greater than or equal to `planned-usage-threshold`. * `disk-size-limit`: Maximum storage size, in bytes, that can be set when storage usage reaches one of the specified thresholds. If you set it to `0`, automatic storage expansion will be disabled. {% note warning %} You cannot reduce the storage size following an automatic expansion. {% endnote %} ## Cluster status {#status} Wait until the Managed Service for Apache Kafka® cluster is ready: its status on the Managed Service for Apache Kafka® dashboard will change to `Running`, and its state, to `Alive`. This may take a while. ## Availability zone and cluster creation warning {#note-warning} {% note warning %} If you create a Managed Service for Apache Kafka® cluster with a single [availability zone](../../overview/concepts/geo-scope.md), you will not be able to increase the number of zones and broker hosts later. {% endnote %} ## Maintenance window setup {#maintenance} To set up a [maintenance window](../concepts/maintenance.md) (including for stopped Managed Service for Apache Kafka® clusters), provide the relevant value in the `--maintenance-window` parameter when creating your cluster: ```bash yc managed-kafka cluster create \ ... --maintenance-window type=,` `day=,` `hour= \ ... ``` Where `type` is the maintenance type: * `anytime`: Any time (default). * `weekly`: On a schedule. For this value, also specify the following: * `day`: Day of week, i.e., `MON`, `TUE`, `WED`, `THU`, `FRI`, `SAT`, or `SUN`. * `hour`: Sequence number of UTC hour interval, from `1` to `24`. > For example, `1` stands for the interval from `00:00` to `01:00`, and `5`, from `04:00` to `05:00`. ## Combined mode and host configuration transfer warning {#note-warning-combined-mode} {% note warning %} If you are creating a [KRaft (combined mode)](../concepts/kraft.md#cluster-topology) cluster, do not provide the KRaft host configuration. {% endnote %} ## Cluster deletion protection {#protect-from-deletion} {% note info %} Even with cluster deletion protection enabled, one can still delete a user or topic or connect manually and delete the data. {% endnote %} 1. ## Environment {#environment} Select the environment where you want to create your Managed Service for Apache Kafka® cluster (note that you cannot change the environment once the cluster is created): * `PRODUCTION`: For stable versions of your apps. * `PRESTABLE`: For testing purposes. The prestable environment is similar to the production environment and likewise covered by an SLA, but it is the first to get new features, improvements, and bug fixes. In the prestable environment, you can test new versions for compatibility with your application. ## Host class {#broker-class} Under **Host class**, select the [platform](../../compute/concepts/vm-platforms.md), host type, and host class. The host class defines the specifications of [virtual machines](../../compute/concepts/vm.md) Apache Kafka® nodes will be deployed on. All available options are listed under [Host classes](../concepts/instance-types.md). When you [change the host class](cluster-update.md#change-brokers) for a Managed Service for Apache Kafka® cluster, the specifications of all existing instances also change. ## Default host configuration {#default-config} ZooKeeper and KRaft hosts have the following configuration by default: * [Host class](../concepts/instance-types.md): `s3-c2-m8` * Disk size: `10` GB * [Disk type](../concepts/storage.md): `network-ssd` ## Automatic storage expansion {#auto-rescale} To prevent the cluster from running out of disk space, create a cluster with [automatic storage expansion](../concepts/storage.md#auto-rescale): ```bash yc managed-kafka cluster create \ ... --disk-size-autoscaling disk-size-limit=,` `planned-usage-threshold=,` `emergency-usage-threshold= \ ... ``` Where: * `planned-usage-threshold`: Storage usage percentage to trigger a storage expansion during the next [maintenance window](../concepts/maintenance.md#maintenance-window). Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. If you set this condition, configure the maintenance schedule. * `emergency-usage-threshold`: Storage usage percentage to trigger an immediate storage expansion. Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. The value of this setting must be greater than or equal to `planned-usage-threshold`. * `disk-size-limit`: Maximum storage size, in bytes, that can be set when storage usage reaches one of the specified thresholds. If you set it to `0`, automatic storage expansion will be disabled. {% note warning %} You cannot reduce the storage size following an automatic expansion. {% endnote %} ## Cluster status {#status} Wait until the Managed Service for Apache Kafka® cluster is ready: its status on the Managed Service for Apache Kafka® dashboard will change to `Running`, and its state, to `Alive`. This may take a while. ## Availability zone and cluster creation warning {#note-warning} {% note warning %} If you create a Managed Service for Apache Kafka® cluster with a single [availability zone](../../overview/concepts/geo-scope.md), you will not be able to increase the number of zones and broker hosts later. {% endnote %} ## Maintenance window setup {#maintenance} To set up a [maintenance window](../concepts/maintenance.md) (including for stopped Managed Service for Apache Kafka® clusters), provide the relevant value in the `--maintenance-window` parameter when creating your cluster: ```bash yc managed-kafka cluster create \ ... --maintenance-window type=,` `day=,` `hour= \ ... ``` Where `type` is the maintenance type: * `anytime`: Any time (default). * `weekly`: On a schedule. For this value, also specify the following: * `day`: Day of week, i.e., `MON`, `TUE`, `WED`, `THU`, `FRI`, `SAT`, or `SUN`. * `hour`: Sequence number of UTC hour interval, from `1` to `24`. > For example, `1` stands for the interval from `00:00` to `01:00`, and `5`, from `04:00` to `05:00`. ## Combined mode and host configuration transfer warning {#note-warning-combined-mode} {% note warning %} If you are creating a [KRaft (combined mode)](../concepts/kraft.md#cluster-topology) cluster, do not provide the KRaft host configuration. {% endnote %} ## Cluster deletion protection {#protect-from-deletion} {% note info %} Even with cluster deletion protection enabled, one can still delete a user or topic or connect manually and delete the data. {% endnote %} - Terraform {#tf} With [Terraform](https://www.terraform.io/), you can quickly create a cloud infrastructure in Yandex Cloud and manage it using configuration files. These files store the infrastructure description written in HashiCorp Configuration Language (HCL). If you change the configuration files, Terraform automatically detects which part of your configuration is already deployed, and what should be added or removed. Terraform is distributed under the [Business Source License](https://github.com/hashicorp/terraform/blob/main/LICENSE). The [Yandex Cloud provider for Terraform](https://github.com/yandex-cloud/terraform-provider-yandex) is distributed under the [MPL-2.0](https://www.mozilla.org/en-US/MPL/2.0/) license. For more information about the provider resources, see the guides on the [Terraform](https://www.terraform.io/docs/providers/yandex/index.html) website or [its mirror](../../terraform/index.md). If you do not have Terraform yet, [install it and configure the Yandex Cloud provider](../../tutorials/infrastructure-management/terraform-quickstart.md#install-terraform). To manage infrastructure using Terraform under a service account or user accounts (a Yandex account, a federated account, or a local user), [authenticate](../../terraform/authentication.md) using the appropriate method. To create a Managed Service for Apache Kafka® cluster: 1. In the configuration file, describe the resources you are creating: * Managed Service for Apache Kafka® cluster: Description of the cluster and its hosts. You can also configure the [Apache Kafka® settings](../concepts/settings-list.md#cluster-settings) here, if required. * Network: Description of the [cloud network](../../vpc/concepts/network.md#network) where a cluster will be located. If you already have a suitable network, you don't have to describe it again. * Subnets: Description of the [subnets](../../vpc/concepts/network.md#subnet) to connect the cluster hosts to. If you already have suitable subnets, you don't have to describe them again. Here is an example of the configuration file structure: ```hcl resource "yandex_mdb_kafka_cluster" "" { environment = "" name = "" network_id = "" subnet_ids = [""] security_group_ids = [""] deletion_protection = config { version = "" zones = [""] brokers_count = assign_public_ip = "" schema_registry = "" kafka_ui { enabled = } kafka { resources { disk_size = disk_type_id = "" resource_preset_id = "" } kafka_config {} } } } resource "yandex_vpc_network" "" { name = "" } resource "yandex_vpc_subnet" "" { name = "" zone = "" network_id = "" v4_cidr_blocks = [""] } ``` Where: * `environment`: Cluster environment, `PRESTABLE` or `PRODUCTION`. * `version`: Apache Kafka® version. Specify 3.6 or higher. * `zones` and `brokers_count`: Availability zones and number of broker hosts per zone. If you are creating a cluster with [KRaft (combined mode)](../concepts/kraft.md#cluster-topology), specify one of the available configurations: * `zones = ["ru-central1-a","ru-central1-b","ru-central1-d"] brokers_count = 1`: Three availability zones with one broker host per zone. * `zones = [""] brokers_count = 3`: One availability zone with three broker hosts. * `deletion_protection`: Cluster deletion protection, `true` or `false`. ## Environment {#environment} Select the environment where you want to create your Managed Service for Apache Kafka® cluster (note that you cannot change the environment once the cluster is created): * `PRODUCTION`: For stable versions of your apps. * `PRESTABLE`: For testing purposes. The prestable environment is similar to the production environment and likewise covered by an SLA, but it is the first to get new features, improvements, and bug fixes. In the prestable environment, you can test new versions for compatibility with your application. ## Host class {#broker-class} Under **Host class**, select the [platform](../../compute/concepts/vm-platforms.md), host type, and host class. The host class defines the specifications of [virtual machines](../../compute/concepts/vm.md) Apache Kafka® nodes will be deployed on. All available options are listed under [Host classes](../concepts/instance-types.md). When you [change the host class](cluster-update.md#change-brokers) for a Managed Service for Apache Kafka® cluster, the specifications of all existing instances also change. ## Default host configuration {#default-config} ZooKeeper and KRaft hosts have the following configuration by default: * [Host class](../concepts/instance-types.md): `s3-c2-m8` * Disk size: `10` GB * [Disk type](../concepts/storage.md): `network-ssd` ## Automatic storage expansion {#auto-rescale} To prevent the cluster from running out of disk space, create a cluster with [automatic storage expansion](../concepts/storage.md#auto-rescale): ```bash yc managed-kafka cluster create \ ... --disk-size-autoscaling disk-size-limit=,` `planned-usage-threshold=,` `emergency-usage-threshold= \ ... ``` Where: * `planned-usage-threshold`: Storage usage percentage to trigger a storage expansion during the next [maintenance window](../concepts/maintenance.md#maintenance-window). Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. If you set this condition, configure the maintenance schedule. * `emergency-usage-threshold`: Storage usage percentage to trigger an immediate storage expansion. Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. The value of this setting must be greater than or equal to `planned-usage-threshold`. * `disk-size-limit`: Maximum storage size, in bytes, that can be set when storage usage reaches one of the specified thresholds. If you set it to `0`, automatic storage expansion will be disabled. {% note warning %} You cannot reduce the storage size following an automatic expansion. {% endnote %} ## Cluster status {#status} Wait until the Managed Service for Apache Kafka® cluster is ready: its status on the Managed Service for Apache Kafka® dashboard will change to `Running`, and its state, to `Alive`. This may take a while. ## Availability zone and cluster creation warning {#note-warning} {% note warning %} If you create a Managed Service for Apache Kafka® cluster with a single [availability zone](../../overview/concepts/geo-scope.md), you will not be able to increase the number of zones and broker hosts later. {% endnote %} ## Maintenance window setup {#maintenance} To set up a [maintenance window](../concepts/maintenance.md) (including for stopped Managed Service for Apache Kafka® clusters), provide the relevant value in the `--maintenance-window` parameter when creating your cluster: ```bash yc managed-kafka cluster create \ ... --maintenance-window type=,` `day=,` `hour= \ ... ``` Where `type` is the maintenance type: * `anytime`: Any time (default). * `weekly`: On a schedule. For this value, also specify the following: * `day`: Day of week, i.e., `MON`, `TUE`, `WED`, `THU`, `FRI`, `SAT`, or `SUN`. * `hour`: Sequence number of UTC hour interval, from `1` to `24`. > For example, `1` stands for the interval from `00:00` to `01:00`, and `5`, from `04:00` to `05:00`. ## Combined mode and host configuration transfer warning {#note-warning-combined-mode} {% note warning %} If you are creating a [KRaft (combined mode)](../concepts/kraft.md#cluster-topology) cluster, do not provide the KRaft host configuration. {% endnote %} ## Cluster deletion protection {#protect-from-deletion} {% note info %} Even with cluster deletion protection enabled, one can still delete a user or topic or connect manually and delete the data. {% endnote %} * `assign_public_ip`: Public access to the cluster, `true` or `false`. * `schema_registry`: Manage data schemas using [Managed Schema Registry](../concepts/managed-schema-registry.md), `true` or `false`. The default value is `false`. {% note warning %} You cannot disable data schema management using Managed Schema Registry after enabling it. {% endnote %} * `kafka_ui`: This setting defines whether to use [Kafka UI for Apache Kafka®](../concepts/kafka-ui.md) and can be `true` or `false`. The default value is `false`. Add the `kraft` section to the cluster description to use [KRaft (on separate hosts)](../concepts/kraft.md#cluster-topology). ```hcl resource "yandex_mdb_kafka_cluster" "" { ... kraft { resources { resource_preset_id = "" disk_type_id = "" disk_size = } } } ``` To set up the [maintenance window](../concepts/maintenance.md) that will also apply to stopped clusters, add the `maintenance_window` section to the cluster description: ```hcl resource "yandex_mdb_kafka_cluster" "" { ... maintenance_window { type = "" day = "" hour = } ... } ``` Where: * `type`: Maintenance type. The possible values include: * `ANYTIME`: Any time. * `WEEKLY`: On a schedule. * `day`: Day of week for the `WEEKLY` type, i.e., `MON`, `TUE`, `WED`, `THU`, `FRI`, `SAT`, or `SUN`. * `hour`: Sequence number of UTC hour interval for the `WEEKLY` type, from `1` to `24`. > For example, `1` stands for the interval from `00:00` to `01:00`, and `5`, from `04:00` to `05:00`. To encrypt disks with a [custom KMS key](../../kms/concepts/key.md), add the `disk_encryption_key_id` parameter: ```hcl resource "yandex_mdb_kafka_cluster" "" { ... disk_encryption_key_id = ... } ``` To learn more about disk encryption, see [Storage](../concepts/storage.md#disk-encryption). 1. Make sure the settings are correct. 1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure. 1. Run this command: ```bash terraform validate ``` Terraform will show any errors found in your configuration files. 1. Create a Managed Service for Apache Kafka® cluster. 1. Run this command to view the planned changes: ```bash terraform plan ``` If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources. 1. If everything looks correct, apply the changes: 1. Run this command: ```bash terraform apply ``` 1. Confirm updating the resources. 1. Wait for the operation to complete. This will create all the resources you need in the specified folder, and the terminal will display the [FQDNs of the Managed Service for Apache Kafka® cluster hosts](../concepts/network.md#hostname). You can check the new resources and their settings in the [management console](https://console.yandex.cloud). For more information, see [this Terraform provider guide](../../terraform/resources/mdb_kafka_cluster.md). {% note warning "Timeouts" %} The Terraform provider limits the time for all operations with the Managed Service for Apache Kafka® cluster to 60 minutes. Operations exceeding the timeout are aborted. {% cut "How do I change these limits?" %} Add the `timeouts` section to your cluster description, such as the following: ```hcl resource "yandex_mdb_kafka_cluster" "" { ... timeouts { create = "1h30m" # 1 hour 30 minutes update = "2h" # 2 hours delete = "30m" # 30 minutes } } ``` {% endcut %} {% endnote %} - REST API {#api} 1. [Get an IAM token for API authentication](../api-ref/authentication.md) and place it in an environment variable: ```bash export IAM_TOKEN="" ``` 1. Call the [Cluster.create](../api-ref/Cluster/create.md) method, e.g., via the following [cURL](https://curl.se/) request: 1. Create a file named `body.json` and paste the following code into it: {% note info %} This example does not use all available parameters. {% endnote %} ```json { "folderId": "", "name": "", "environment": "", "networkId": "", "securityGroupIds": [ "", "", ... "" ], "configSpec": { "version": "", "kafka": { "resources": { "resourcePresetId": "", "diskSize": "", "diskTypeId": "" } }, "kraft": { "resources": { "resourcePresetId": "", "diskSize": "", "diskTypeId": "" } }, "zoneId": [ ], "brokersCount": "", "assignPublicIp": , "schemaRegistry": , "restApiConfig": { "enabled": }, "diskSizeAutoscaling": { }, "kafkaUiConfig": { "enabled": } }, "topicSpecs": [ { "name": "", "partitions": "", "replicationFactor": "" }, { }, { ... }, { } ], "userSpecs": [ { "name": "", "password": "", "permissions": [ { "topicName": "", "role": "" } ] }, { }, { ... }, { } ], "maintenanceWindow": { "anytime": {}, "weeklyMaintenanceWindow": { "day": "", "hour": "" } }, "deletionProtection": , "diskEncryptionKeyId": "" } ``` Where: * `name`: Cluster name. * `environment`: Cluster environment, `PRODUCTION` or `PRESTABLE`. * `networkId`: ID of the [network](../../vpc/concepts/network.md) where the cluster will be deployed. * `securityGroupIds`: [Security group](../../vpc/concepts/security-groups.md) IDs as an array of strings. Each string is a security group ID. * `configSpec`: Cluster configuration: * `version`: Apache Kafka® version. Specify 3.6 or higher. * `kafka`: Apache Kafka® configuration: * `resources.resourcePresetId`: [Host class](../concepts/instance-types.md) ID. You can get the list of available host classes with their IDs by calling the [ResourcePreset.list](../api-ref/ResourcePreset/list.md) method. * `resources.diskSize`: Disk size, in bytes. * `resources.diskTypeId`: [Disk type](../concepts/storage.md). * `kraft`: [KRaft](../concepts/kraft.md) configuration: * `resources.resourcePresetId`: Host class ID. You can get the list of available host classes with their IDs using the [ResourcePreset.list](../api-ref/ResourcePreset/list.md) method. * `resources.diskSize`: Disk size, in bytes. * `resources.diskTypeId`: Disk type. ## Environment {#environment} Select the environment where you want to create your Managed Service for Apache Kafka® cluster (note that you cannot change the environment once the cluster is created): * `PRODUCTION`: For stable versions of your apps. * `PRESTABLE`: For testing purposes. The prestable environment is similar to the production environment and likewise covered by an SLA, but it is the first to get new features, improvements, and bug fixes. In the prestable environment, you can test new versions for compatibility with your application. ## Host class {#broker-class} Under **Host class**, select the [platform](../../compute/concepts/vm-platforms.md), host type, and host class. The host class defines the specifications of [virtual machines](../../compute/concepts/vm.md) Apache Kafka® nodes will be deployed on. All available options are listed under [Host classes](../concepts/instance-types.md). When you [change the host class](cluster-update.md#change-brokers) for a Managed Service for Apache Kafka® cluster, the specifications of all existing instances also change. ## Default host configuration {#default-config} ZooKeeper and KRaft hosts have the following configuration by default: * [Host class](../concepts/instance-types.md): `s3-c2-m8` * Disk size: `10` GB * [Disk type](../concepts/storage.md): `network-ssd` ## Automatic storage expansion {#auto-rescale} To prevent the cluster from running out of disk space, create a cluster with [automatic storage expansion](../concepts/storage.md#auto-rescale): ```bash yc managed-kafka cluster create \ ... --disk-size-autoscaling disk-size-limit=,` `planned-usage-threshold=,` `emergency-usage-threshold= \ ... ``` Where: * `planned-usage-threshold`: Storage usage percentage to trigger a storage expansion during the next [maintenance window](../concepts/maintenance.md#maintenance-window). Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. If you set this condition, configure the maintenance schedule. * `emergency-usage-threshold`: Storage usage percentage to trigger an immediate storage expansion. Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. The value of this setting must be greater than or equal to `planned-usage-threshold`. * `disk-size-limit`: Maximum storage size, in bytes, that can be set when storage usage reaches one of the specified thresholds. If you set it to `0`, automatic storage expansion will be disabled. {% note warning %} You cannot reduce the storage size following an automatic expansion. {% endnote %} ## Cluster status {#status} Wait until the Managed Service for Apache Kafka® cluster is ready: its status on the Managed Service for Apache Kafka® dashboard will change to `Running`, and its state, to `Alive`. This may take a while. ## Availability zone and cluster creation warning {#note-warning} {% note warning %} If you create a Managed Service for Apache Kafka® cluster with a single [availability zone](../../overview/concepts/geo-scope.md), you will not be able to increase the number of zones and broker hosts later. {% endnote %} ## Maintenance window setup {#maintenance} To set up a [maintenance window](../concepts/maintenance.md) (including for stopped Managed Service for Apache Kafka® clusters), provide the relevant value in the `--maintenance-window` parameter when creating your cluster: ```bash yc managed-kafka cluster create \ ... --maintenance-window type=,` `day=,` `hour= \ ... ``` Where `type` is the maintenance type: * `anytime`: Any time (default). * `weekly`: On a schedule. For this value, also specify the following: * `day`: Day of week, i.e., `MON`, `TUE`, `WED`, `THU`, `FRI`, `SAT`, or `SUN`. * `hour`: Sequence number of UTC hour interval, from `1` to `24`. > For example, `1` stands for the interval from `00:00` to `01:00`, and `5`, from `04:00` to `05:00`. ## Combined mode and host configuration transfer warning {#note-warning-combined-mode} {% note warning %} If you are creating a [KRaft (combined mode)](../concepts/kraft.md#cluster-topology) cluster, do not provide the KRaft host configuration. {% endnote %} ## Cluster deletion protection {#protect-from-deletion} {% note info %} Even with cluster deletion protection enabled, one can still delete a user or topic or connect manually and delete the data. {% endnote %} * `zoneId` and `brokersCount`: Availability zones and number of broker hosts per zone. If you are creating a cluster with [KRaft (combined mode)](../concepts/kraft.md#cluster-topology), specify one of the available configurations: * `"zoneId": ["ru-central1-a","ru-central1-b","ru-central1-d"], "brokersCount": "1"`: Three availability zones with one broker host per zone. * `"zoneId": [""], "brokersCount": "3"`: One availability zone, three broker hosts. * `assignPublicIp`: Access to broker hosts from the internet, `true` or `false`. * `schemaRegistry`: Manage data schemas using [Managed Schema Registry](../concepts/managed-schema-registry.md), `true` or `false`. The default value is `false`. You will not be able to edit this setting once you create a Managed Service for Apache Kafka® cluster. * `restApiConfig`: Apache Kafka® REST API configuration. For access to sending requests to the Apache Kafka® REST API, specify `enabled: true`. * `diskSizeAutoscaling`: Storage [utilization thresholds](../concepts/storage.md#auto-rescale) (as a percentage of the total storage size) that will trigger storage expansion when reached: * `plannedUsageThreshold`: Storage usage percentage to trigger a storage expansion during the next [maintenance window](../concepts/maintenance.md#maintenance-window). Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. If you set this condition, configure the maintenance window schedule. * `emergencyUsageThreshold`: Storage usage percentage to trigger an immediate storage expansion. Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. The value of this setting must be greater than or equal to `plannedUsageThreshold`. * `diskSizeLimit`: Maximum storage size, in bytes, that can be set when storage usage reaches one of the specified thresholds. * `kafkaUiConfig`: Use [Kafka UI](../concepts/kafka-ui.md). For access to Kafka UI, specify `enabled: true`. * `topicSpecs`: Topic settings as an array of elements, one per topic. Each element has the following structure: * `name`: Topic name. {% note info %} Use the [Apache Kafka® Admin API](cluster-topics.md#admin-api) if you need to create a topic that starts with `_`. You cannot create such a topic using the Yandex Cloud interfaces. {% endnote %} * `partitions`: Number of partitions. * `replicationFactor`: Replication factor. * `userSpecs`: User settings as an array of elements, one per user. Each element has the following structure: * `name`: Username. It may contain Latin letters, numbers, hyphens, and underscores, and must start with a letter or underscore. * `password`: User password. The password must be from 8 to 128 characters long. * `permissions`: List of topics the user should have access to. The list appears as an array of elements one per topic. Each element has the following structure: * `topicName`: Topic name or name pattern: * `*`: To allow access to all topics. * Full topic name: To allow access to a specific topic. * `*`: To grant access to topics whose names start with the specified prefix. Let’s assume you have topics named `topic_a1`, `topic_a2`, and `a3`. If you specify `topic*`, access will be granted to `topic_a1` and `topic_a2`. To cover all cluster's topics, use `*`. * `role`: User’s role, `ACCESS_ROLE_CONSUMER`, `ACCESS_ROLE_PRODUCER`, or `ACCESS_ROLE_ADMIN`. The `ACCESS_ROLE_ADMIN` role is only available if all topics are selected (`topicName: "*"`). * `allowHosts` (optional): List of IP addresses the user is allowed to access the topic from. * `maintenanceWindow`: [Maintenance window](../concepts/maintenance.md) settings, applying to both running and stopped clusters. Select one of these options: * `anytime`: At any time (default). * `weeklyMaintenanceWindow`: On schedule: * `day`: Day of week in `DDD` format, i.e., `MON`, `TUE`, `WED`, `THU`, `FRI`, `SAT`, or `SUN`. * `hour`: Time of day (UTC) in `HH` format, from `1` to `24`. * `deletionProtection`: Cluster protection against accidental deletion, `true` or `false`. The default value is `false`. ## Environment {#environment} Select the environment where you want to create your Managed Service for Apache Kafka® cluster (note that you cannot change the environment once the cluster is created): * `PRODUCTION`: For stable versions of your apps. * `PRESTABLE`: For testing purposes. The prestable environment is similar to the production environment and likewise covered by an SLA, but it is the first to get new features, improvements, and bug fixes. In the prestable environment, you can test new versions for compatibility with your application. ## Host class {#broker-class} Under **Host class**, select the [platform](../../compute/concepts/vm-platforms.md), host type, and host class. The host class defines the specifications of [virtual machines](../../compute/concepts/vm.md) Apache Kafka® nodes will be deployed on. All available options are listed under [Host classes](../concepts/instance-types.md). When you [change the host class](cluster-update.md#change-brokers) for a Managed Service for Apache Kafka® cluster, the specifications of all existing instances also change. ## Default host configuration {#default-config} ZooKeeper and KRaft hosts have the following configuration by default: * [Host class](../concepts/instance-types.md): `s3-c2-m8` * Disk size: `10` GB * [Disk type](../concepts/storage.md): `network-ssd` ## Automatic storage expansion {#auto-rescale} To prevent the cluster from running out of disk space, create a cluster with [automatic storage expansion](../concepts/storage.md#auto-rescale): ```bash yc managed-kafka cluster create \ ... --disk-size-autoscaling disk-size-limit=,` `planned-usage-threshold=,` `emergency-usage-threshold= \ ... ``` Where: * `planned-usage-threshold`: Storage usage percentage to trigger a storage expansion during the next [maintenance window](../concepts/maintenance.md#maintenance-window). Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. If you set this condition, configure the maintenance schedule. * `emergency-usage-threshold`: Storage usage percentage to trigger an immediate storage expansion. Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. The value of this setting must be greater than or equal to `planned-usage-threshold`. * `disk-size-limit`: Maximum storage size, in bytes, that can be set when storage usage reaches one of the specified thresholds. If you set it to `0`, automatic storage expansion will be disabled. {% note warning %} You cannot reduce the storage size following an automatic expansion. {% endnote %} ## Cluster status {#status} Wait until the Managed Service for Apache Kafka® cluster is ready: its status on the Managed Service for Apache Kafka® dashboard will change to `Running`, and its state, to `Alive`. This may take a while. ## Availability zone and cluster creation warning {#note-warning} {% note warning %} If you create a Managed Service for Apache Kafka® cluster with a single [availability zone](../../overview/concepts/geo-scope.md), you will not be able to increase the number of zones and broker hosts later. {% endnote %} ## Maintenance window setup {#maintenance} To set up a [maintenance window](../concepts/maintenance.md) (including for stopped Managed Service for Apache Kafka® clusters), provide the relevant value in the `--maintenance-window` parameter when creating your cluster: ```bash yc managed-kafka cluster create \ ... --maintenance-window type=,` `day=,` `hour= \ ... ``` Where `type` is the maintenance type: * `anytime`: Any time (default). * `weekly`: On a schedule. For this value, also specify the following: * `day`: Day of week, i.e., `MON`, `TUE`, `WED`, `THU`, `FRI`, `SAT`, or `SUN`. * `hour`: Sequence number of UTC hour interval, from `1` to `24`. > For example, `1` stands for the interval from `00:00` to `01:00`, and `5`, from `04:00` to `05:00`. ## Combined mode and host configuration transfer warning {#note-warning-combined-mode} {% note warning %} If you are creating a [KRaft (combined mode)](../concepts/kraft.md#cluster-topology) cluster, do not provide the KRaft host configuration. {% endnote %} ## Cluster deletion protection {#protect-from-deletion} {% note info %} Even with cluster deletion protection enabled, one can still delete a user or topic or connect manually and delete the data. {% endnote %} * `diskEncryptionKeyId`: ID of the [custom KMS key](../../kms/concepts/key.md). Provide the KMS key ID to encrypt the disks. Learn more about disk encryption in [Disk encryption](../concepts/storage.md#disk-encryption). {% note warning %} You can enable disk encryption only when creating a cluster. {% endnote %} You can get the folder ID with the [list of folders in the cloud](../../resource-manager/operations/folder/get-id.md). 1. Run this request: ```bash curl \ --request POST \ --header "Authorization: Bearer $IAM_TOKEN" \ --header "Content-Type: application/json" \ --url 'https://mdb.api.cloud.yandex.net/managed-kafka/v1/clusters' \ --data '@body.json' ``` 1. View the [server response](../api-ref/Cluster/create.md#yandex.cloud.operation.Operation) to make sure your request was successful. - gRPC API {#grpc-api} 1. [Get an IAM token for API authentication](../api-ref/authentication.md) and put it into an environment variable: ```bash export IAM_TOKEN="" ``` 1. Clone the [cloudapi](https://github.com/yandex-cloud/cloudapi) repository: ```bash cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi ``` Below, we assume that the repository contents reside in the `~/cloudapi/` directory. 1. Call the [ClusterService/Create](../api-ref/grpc/Cluster/create.md) method, e.g., via the following [gRPCurl](https://github.com/fullstorydev/grpcurl) request: 1. Create a file named `body.json` and paste the following code into it: {% note info %} This example does not use all available parameters. {% endnote %} ```json { "folder_id": "", "name": "", "environment": "", "network_id": "", "security_group_ids": [ "", "", ... "" ], "config_spec": { "version": "", "kafka": { "resources": { "resource_preset_id": "", "disk_size": "", "disk_type_id": "" } }, "kraft": { "resources": { "resource_preset_id": "", "disk_size": "", "disk_type_id": "" } }, "zone_id": [ ], "brokers_count": { "value": "" }, "assign_public_ip": , "schema_registry": , "rest_api_config": { "enabled": }, "disk_size_autoscaling": { }, "kafka_ui_config": { "enabled": } }, "topic_specs": [ { "name": "", "partitions": { "value": "" }, "replication_factor": { "value": "" } }, { }, { ... }, { } ], "user_specs": [ { "name": "", "password": "", "permissions": [ { "topic_name": "", "role": "" } ] }, { }, { ... }, { } ], "maintenance_window": { "anytime": {}, "weekly_maintenance_window": { "day": "", "hour": "" } }, "deletion_protection": , "disk_encryption_key_id": "" } ``` Where: * `name`: Cluster name. * `environment`: Cluster environment, `PRODUCTION` or `PRESTABLE`. * `network_id`: ID of the [network](../../vpc/concepts/network.md) where the cluster will be deployed. * `security_group_ids`: [Security group](../../vpc/concepts/security-groups.md) IDs as an array of strings. Each string is a security group ID. * `config_spec`: Cluster configuration: * `version`: Apache Kafka® version. Specify 3.6 or higher. * `kafka`: Apache Kafka® configuration: * `resources.resource_preset_id`: [Host class](../concepts/instance-types.md) ID. You can get the list of available host classes with their IDs by calling the [ResourcePreset.list](../api-ref/grpc/ResourcePreset/list.md) method. * `resources.disk_size`: Disk size, in bytes. * `resources.disk_type_id`: [Disk type](../concepts/storage.md). * `kraft`: [KRaft](../concepts/kraft.md) configuration: * `resources.resource_preset_id`: Host class ID. You can get the list of available host classes with their IDs by calling the [ResourcePreset.list](../api-ref/grpc/ResourcePreset/list.md) method. * `resources.disk_size`: Disk size, in bytes. * `resources.disk_type_id`: Disk type. ## Environment {#environment} Select the environment where you want to create your Managed Service for Apache Kafka® cluster (note that you cannot change the environment once the cluster is created): * `PRODUCTION`: For stable versions of your apps. * `PRESTABLE`: For testing purposes. The prestable environment is similar to the production environment and likewise covered by an SLA, but it is the first to get new features, improvements, and bug fixes. In the prestable environment, you can test new versions for compatibility with your application. ## Host class {#broker-class} Under **Host class**, select the [platform](../../compute/concepts/vm-platforms.md), host type, and host class. The host class defines the specifications of [virtual machines](../../compute/concepts/vm.md) Apache Kafka® nodes will be deployed on. All available options are listed under [Host classes](../concepts/instance-types.md). When you [change the host class](cluster-update.md#change-brokers) for a Managed Service for Apache Kafka® cluster, the specifications of all existing instances also change. ## Default host configuration {#default-config} ZooKeeper and KRaft hosts have the following configuration by default: * [Host class](../concepts/instance-types.md): `s3-c2-m8` * Disk size: `10` GB * [Disk type](../concepts/storage.md): `network-ssd` ## Automatic storage expansion {#auto-rescale} To prevent the cluster from running out of disk space, create a cluster with [automatic storage expansion](../concepts/storage.md#auto-rescale): ```bash yc managed-kafka cluster create \ ... --disk-size-autoscaling disk-size-limit=,` `planned-usage-threshold=,` `emergency-usage-threshold= \ ... ``` Where: * `planned-usage-threshold`: Storage usage percentage to trigger a storage expansion during the next [maintenance window](../concepts/maintenance.md#maintenance-window). Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. If you set this condition, configure the maintenance schedule. * `emergency-usage-threshold`: Storage usage percentage to trigger an immediate storage expansion. Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. The value of this setting must be greater than or equal to `planned-usage-threshold`. * `disk-size-limit`: Maximum storage size, in bytes, that can be set when storage usage reaches one of the specified thresholds. If you set it to `0`, automatic storage expansion will be disabled. {% note warning %} You cannot reduce the storage size following an automatic expansion. {% endnote %} ## Cluster status {#status} Wait until the Managed Service for Apache Kafka® cluster is ready: its status on the Managed Service for Apache Kafka® dashboard will change to `Running`, and its state, to `Alive`. This may take a while. ## Availability zone and cluster creation warning {#note-warning} {% note warning %} If you create a Managed Service for Apache Kafka® cluster with a single [availability zone](../../overview/concepts/geo-scope.md), you will not be able to increase the number of zones and broker hosts later. {% endnote %} ## Maintenance window setup {#maintenance} To set up a [maintenance window](../concepts/maintenance.md) (including for stopped Managed Service for Apache Kafka® clusters), provide the relevant value in the `--maintenance-window` parameter when creating your cluster: ```bash yc managed-kafka cluster create \ ... --maintenance-window type=,` `day=,` `hour= \ ... ``` Where `type` is the maintenance type: * `anytime`: Any time (default). * `weekly`: On a schedule. For this value, also specify the following: * `day`: Day of week, i.e., `MON`, `TUE`, `WED`, `THU`, `FRI`, `SAT`, or `SUN`. * `hour`: Sequence number of UTC hour interval, from `1` to `24`. > For example, `1` stands for the interval from `00:00` to `01:00`, and `5`, from `04:00` to `05:00`. ## Combined mode and host configuration transfer warning {#note-warning-combined-mode} {% note warning %} If you are creating a [KRaft (combined mode)](../concepts/kraft.md#cluster-topology) cluster, do not provide the KRaft host configuration. {% endnote %} ## Cluster deletion protection {#protect-from-deletion} {% note info %} Even with cluster deletion protection enabled, one can still delete a user or topic or connect manually and delete the data. {% endnote %} * `zone_id` and `brokers_count`: Availability zones and number of broker hosts (provided as an object with the `value` field) per zone. If you are creating a cluster with [KRaft (combined mode)](../concepts/kraft.md#cluster-topology), specify one of the available configurations: * `"zone_id": ["ru-central1-a","ru-central1-b","ru-central1-d"], "brokers_count": {"value":"1"}`: Three availability zones with one broker host per zone. * `"zone_id": [""], "brokers_count": {"value":"3"}`: One availability zone with three broker hosts. * `assign_public_ip`: Access to broker hosts from the internet, `true` or `false`. * `schema_registry`: Manage data schemas using [Managed Schema Registry](../concepts/managed-schema-registry.md), `true` or `false`. The default value is `false`. You will not be able to edit this setting once you create the Managed Service for Apache Kafka® cluster. * `rest_api_config`: Apache Kafka® REST API configuration. For access to sending requests to the Apache Kafka® REST API, specify `enabled: true`. * `disk_size_autoscaling`: To prevent the cluster disk space from running out, set the storage [utilization thresholds](../concepts/storage.md#auto-rescale) (as a percentage of the total storage size) that will trigger storage expansion when reached: * `planned_usage_threshold`: Storage usage percentage to trigger a storage expansion during the next [maintenance window](../concepts/maintenance.md#maintenance-window). Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. If you set this condition, configure the maintenance window schedule. * `emergency_usage_threshold`: Storage usage percentage to trigger an immediate storage expansion. Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. The value of this setting must be greater than or equal to `planned_usage_threshold`. * `disk_size_limit`: Maximum storage size, in bytes, that can be set when storage usage reaches one of the specified thresholds. * `kafka_ui_config`: Use [Kafka UI](../concepts/kafka-ui.md). For access to Kafka UI, specify `enabled: true`. * `topic_specs`: Topic settings as an array of elements, Each element is for a separate topic and has the following structure: * `name`: Topic name. {% note info %} Use the [Apache Kafka® Admin API](cluster-topics.md#admin-api) if you need to create a topic that starts with `_`. You cannot create such a topic using the Yandex Cloud interfaces. {% endnote %} * `partitions`: Number of partitions, provided as an object with the `value` field. * `replication_factor`: Replication factor, provided as an object with the `value` field. * `user_specs`: User settings as an array of elements, one per user. Each element has the following structure: * `name`: Username. It may contain Latin letters, numbers, hyphens, and underscores, and must start with a letter or underscore. * `password`: User password. The password must be from 8 to 128 characters long. * `permissions`: List of topics the user should have access to. The list appears as an array of elements. one per topic. Each element has the following structure: * `topic_name`: Topic name or name pattern: * `*`: To allow access to all topics. * Full topic name: To allow access to a specific topic. * `*`: To grant access to topics whose names start with the specified prefix. Let’s assume you have topics named `topic_a1`, `topic_a2`, and `a3`. If you specify `topic*`, access will be granted to `topic_a1` and `topic_a2`. To cover all cluster's topics, use `*`. * `role`: User’s role, `ACCESS_ROLE_CONSUMER`, `ACCESS_ROLE_PRODUCER`, or `ACCESS_ROLE_ADMIN`. The `ACCESS_ROLE_ADMIN` role is only available if all topics are selected (`topicName: "*"`). * `allow_hosts` (optional): List of IP addresses the user is allowed to access the topic from, as an array of elements. * `maintenance_window`: [Maintenance window](../concepts/maintenance.md) settings, applying to both running and stopped clusters. Select one of these options: * `anytime`: At any time (default). * `weekly_maintenance_window`: On schedule: * `day`: Day of week in `DDD` format, i.e., `MON`, `TUE`, `WED`, `THU`, `FRI`, `SAT`, or `SUN`. * `hour`: Time of day (UTC) in `HH` format, from `1` to `24`. * `deletion_protection`: Cluster protection against accidental deletion, `true` or `false`. The default value is `false`. ## Environment {#environment} Select the environment where you want to create your Managed Service for Apache Kafka® cluster (note that you cannot change the environment once the cluster is created): * `PRODUCTION`: For stable versions of your apps. * `PRESTABLE`: For testing purposes. The prestable environment is similar to the production environment and likewise covered by an SLA, but it is the first to get new features, improvements, and bug fixes. In the prestable environment, you can test new versions for compatibility with your application. ## Host class {#broker-class} Under **Host class**, select the [platform](../../compute/concepts/vm-platforms.md), host type, and host class. The host class defines the specifications of [virtual machines](../../compute/concepts/vm.md) Apache Kafka® nodes will be deployed on. All available options are listed under [Host classes](../concepts/instance-types.md). When you [change the host class](cluster-update.md#change-brokers) for a Managed Service for Apache Kafka® cluster, the specifications of all existing instances also change. ## Default host configuration {#default-config} ZooKeeper and KRaft hosts have the following configuration by default: * [Host class](../concepts/instance-types.md): `s3-c2-m8` * Disk size: `10` GB * [Disk type](../concepts/storage.md): `network-ssd` ## Automatic storage expansion {#auto-rescale} To prevent the cluster from running out of disk space, create a cluster with [automatic storage expansion](../concepts/storage.md#auto-rescale): ```bash yc managed-kafka cluster create \ ... --disk-size-autoscaling disk-size-limit=,` `planned-usage-threshold=,` `emergency-usage-threshold= \ ... ``` Where: * `planned-usage-threshold`: Storage usage percentage to trigger a storage expansion during the next [maintenance window](../concepts/maintenance.md#maintenance-window). Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. If you set this condition, configure the maintenance schedule. * `emergency-usage-threshold`: Storage usage percentage to trigger an immediate storage expansion. Use a value between `0` and `100`%. The default value is `0`, i.e., automatic expansion is disabled. The value of this setting must be greater than or equal to `planned-usage-threshold`. * `disk-size-limit`: Maximum storage size, in bytes, that can be set when storage usage reaches one of the specified thresholds. If you set it to `0`, automatic storage expansion will be disabled. {% note warning %} You cannot reduce the storage size following an automatic expansion. {% endnote %} ## Cluster status {#status} Wait until the Managed Service for Apache Kafka® cluster is ready: its status on the Managed Service for Apache Kafka® dashboard will change to `Running`, and its state, to `Alive`. This may take a while. ## Availability zone and cluster creation warning {#note-warning} {% note warning %} If you create a Managed Service for Apache Kafka® cluster with a single [availability zone](../../overview/concepts/geo-scope.md), you will not be able to increase the number of zones and broker hosts later. {% endnote %} ## Maintenance window setup {#maintenance} To set up a [maintenance window](../concepts/maintenance.md) (including for stopped Managed Service for Apache Kafka® clusters), provide the relevant value in the `--maintenance-window` parameter when creating your cluster: ```bash yc managed-kafka cluster create \ ... --maintenance-window type=,` `day=,` `hour= \ ... ``` Where `type` is the maintenance type: * `anytime`: Any time (default). * `weekly`: On a schedule. For this value, also specify the following: * `day`: Day of week, i.e., `MON`, `TUE`, `WED`, `THU`, `FRI`, `SAT`, or `SUN`. * `hour`: Sequence number of UTC hour interval, from `1` to `24`. > For example, `1` stands for the interval from `00:00` to `01:00`, and `5`, from `04:00` to `05:00`. ## Combined mode and host configuration transfer warning {#note-warning-combined-mode} {% note warning %} If you are creating a [KRaft (combined mode)](../concepts/kraft.md#cluster-topology) cluster, do not provide the KRaft host configuration. {% endnote %} ## Cluster deletion protection {#protect-from-deletion} {% note info %} Even with cluster deletion protection enabled, one can still delete a user or topic or connect manually and delete the data. {% endnote %} * `disk_encryption_key_id`: ID of the [custom KMS key](../../kms/concepts/key.md). Provide the KMS key ID to encrypt the disks. Learn more about disk encryption in [Disk encryption](../concepts/storage.md#disk-encryption). {% note warning %} You can enable disk encryption only when creating a cluster. {% endnote %} You can get the folder ID with the [list of folders in the cloud](../../resource-manager/operations/folder/get-id.md). 1. Run this request: ```bash curl \ --request POST \ --header "Authorization: Bearer $IAM_TOKEN" \ --header "Content-Type: application/json" \ --url 'https://mdb.api.cloud.yandex.net/managed-kafka/v1/clusters' \ --data '@body.json' ``` 1. Check the [server response](../api-ref/Cluster/create.md#yandex.cloud.operation.Operation) to make sure your request was successful. {% endlist %} ## Creating a cluster copy {#duplicate} You can create an Apache Kafka® cluster with the settings of another one created earlier. Do this by importing the original Apache Kafka® cluster configuration into Terraform. Then you can either create an identical copy or use the imported configuration as the baseline and modify it as needed. The import feature is handy when the original Apache Kafka® cluster has a lot of settings and you need to create a similar one. To create an Apache Kafka® cluster copy: {% list tabs group=instructions %} - Terraform {#tf} 1. If you do not have Terraform yet, [install it](../../tutorials/infrastructure-management/terraform-quickstart.md#install-terraform). 1. [Get the authentication credentials](../../tutorials/infrastructure-management/terraform-quickstart.md#get-credentials). You can add them to environment variables or specify them later in the provider configuration file. 1. [Configure and initialize a provider](../../tutorials/infrastructure-management/terraform-quickstart.md#configure-provider). There is no need to create a provider configuration file manually, you can [download it](https://github.com/yandex-cloud-examples/yc-terraform-provider-settings/blob/main/provider.tf). 1. Place the configuration file in a separate working directory and [specify the parameter values](../../tutorials/infrastructure-management/terraform-quickstart.md#configure-provider). If you did not add the authentication credentials to environment variables, specify them in the configuration file. 1. In the same working directory, place a `.tf` file with the following contents: ```hcl resource "yandex_mdb_kafka_cluster" "old" { } ``` 1. Save the ID of the original Apache Kafka® cluster to an environment variable: ```bash export KAFKA_CLUSTER_ID= ``` You can get the ID with the [list of clusters in the folder](cluster-list.md#list-clusters). 1. Import the original Apache Kafka® cluster settings to the Terraform configuration: ```bash terraform import yandex_mdb_kafka_cluster.old ${KAFKA_CLUSTER_ID} ``` 1. Get the imported configuration: ```bash terraform show ``` 1. Copy it from your terminal and paste it into the `.tf` file. 1. Create a new directory `imported-cluster` and move your configuration file there. 1. Modify the configuration so that you can use it to create a new cluster: * Specify the new cluster name in the `resource` string and the `name` parameter. * Delete `created_at`, `health`, `host`, `id`, and `status`. * Add the `subnet_ids` argument with the list of subnet IDs for each availability zone. * If the `maintenance_window` section contains `type = "ANYTIME"`, delete the `hour` setting. * Optionally, make further changes if you need a customized configuration. 1. [Get the authentication credentials](../../tutorials/infrastructure-management/terraform-quickstart.md#get-credentials) in the `imported-cluster` directory. 1. In the same directory, [configure and initialize the provider](../../tutorials/infrastructure-management/terraform-quickstart.md#configure-provider). Instead of manually creating the provider configuration file, you can [download it](https://github.com/yandex-cloud-examples/yc-terraform-provider-settings/blob/main/provider.tf). 1. Move the configuration file to the `imported-cluster` directory and [specify the arguments](../../tutorials/infrastructure-management/terraform-quickstart.md#configure-provider). If you have not set the authentication credentials as environment variables, specify them in the configuration file. 1. Validate your Terraform configuration: ```bash terraform validate ``` Terraform will display any configuration errors detected in your files. 1. Create the required infrastructure: 1. Run this command to view the planned changes: ```bash terraform plan ``` If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources. 1. If everything looks correct, apply the changes: 1. Run this command: ```bash terraform apply ``` 1. Confirm updating the resources. 1. Wait for the operation to complete. All the required resources will be created in the specified folder. You can check resource availability and their settings in the [management console](https://console.yandex.cloud). {% note warning "Timeouts" %} The Terraform provider limits the time for all operations with the Managed Service for Apache Kafka® cluster to 60 minutes. Operations exceeding the timeout are aborted. {% cut "How do I change these limits?" %} Add the `timeouts` section to your cluster description, such as the following: ```hcl resource "yandex_mdb_kafka_cluster" "" { ... timeouts { create = "1h30m" # 1 hour 30 minutes update = "2h" # 2 hours delete = "30m" # 30 minutes } } ``` {% endcut %} {% endnote %} {% endlist %} ## Examples {#examples} ### Creating a single-host cluster {#creating-a-single-host-cluster} {% list tabs group=instructions %} - CLI {#cli} Create a Managed Service for Apache Kafka® cluster with the following test specifications: * Name: `mykf`. * Environment: `production`. * Apache Kafka® version: 3.9. * Network: `default`. * Subnet ID: `b0rcctk2rvtr8efcch64`. * Security group: `enp6saqnq4ie244g67sb`. * Broker host class: `s2.micro` (one host), availability zone: `ru-central1-a`. * Network SSD storage (`network-ssd`): 10 GB. * Public access: Enabled. * Deletion protection: Enabled. Run this command: ```bash yc managed-kafka cluster create \ --name mykf \ --environment production \ --version 3.9 \ --network-name default \ --subnet-ids b0rcctk2rvtr8efcch64 \ --zone-ids ru-central1-a \ --brokers-count 1 \ --resource-preset s2.micro \ --disk-size 10 \ --disk-type network-ssd \ --assign-public-ip \ --security-group-ids enp6saqnq4ie244g67sb \ --deletion-protection ``` - Terraform {#tf} Create a Managed Service for Apache Kafka® cluster with the following test specifications: * Cloud ID: `b1gq90dgh25bebiu75o`. * Folder ID: `b1gia87mbaomkfvsleds`. * Name: `mykf`. * Environment: `PRODUCTION`. * Apache Kafka® version: 3.9. * New network: `mynet` (new one), subnet: `mysubnet`, address range: `10.5.0.0/24`. * Security group: `mykf-sg` allowing internet connections to the Managed Service for Apache Kafka® cluster on port `9091`. * Broker host class: `s2.micro` (one host), availability zone: `ru-central1-a`. * Network SSD storage (`network-ssd`): 10 GB. * Public access: Enabled. * Deletion protection: Enabled. The configuration file for this Managed Service for Apache Kafka® cluster is as follows: ```hcl resource "yandex_mdb_kafka_cluster" "mykf" { environment = "PRODUCTION" name = "mykf" network_id = yandex_vpc_network.mynet.id subnet_ids = [ yandex_vpc_subnet.mysubnet.id ] security_group_ids = [ yandex_vpc_security_group.mykf-sg.id ] deletion_protection = true config { assign_public_ip = true brokers_count = 1 version = "3.9" kafka { resources { disk_size = 10 disk_type_id = "network-ssd" resource_preset_id = "s2.micro" } kafka_config {} } zones = [ "ru-central1-a" ] } } resource "yandex_vpc_network" "mynet" { name = "mynet" } resource "yandex_vpc_subnet" "mysubnet" { name = "mysubnet" zone = "ru-central1-a" network_id = yandex_vpc_network.mynet.id v4_cidr_blocks = ["10.5.0.0/24"] } resource "yandex_vpc_security_group" "mykf-sg" { name = "mykf-sg" network_id = yandex_vpc_network.mynet.id ingress { description = "Kafka" port = 9091 protocol = "TCP" v4_cidr_blocks = [ "0.0.0.0/0" ] } } ``` {% endlist %} ### Creating a cluster with KRaft in combined mode {#kafka-kraft} In our example, we use the [configuration](../concepts/kraft.md#cluster-topology) with three availability zones, each hosting a single broker. {% list tabs group=instructions %} - CLI {#cli} Create a Managed Service for Apache Kafka® cluster with the following test specifications: * Name: `kafka-kraft`. * Environment: `production`. * Apache Kafka® version: `3.9`. * Network ID: `enpc6eqfhmj2********`. * Subnet IDs: * `e9bhbia2scnk********` * `e2lfqbm5nt9r********` * `fl8beqmjckv8********` * One broker host in each of the following availability zones: * `ru-central1-a` * `ru-central1-b` * `ru-central1-d` * `ru-central1-e` * [Host class](../concepts/instance-types.md): `s2.micro`. * Network HDD storage (`network-hdd`): `10` GB. * Public access: Enabled. * Security group: `enp68jq81uun********`. Run this command: ```bash yc managed-kafka cluster create \ --name kafka-kraft \ --environment production \ --version 3.9 \ --network-id enpc6eqfhmj2******** \ --subnet-ids e9bhbia2scnk********,e2lfqbm5nt9r********,fl8beqmjckv8******** \ --zone-ids ru-central1-a,ru-central1-b,ru-central1-d,ru-central1-e \ --brokers-count 1 \ --resource-preset s2.micro \ --disk-size 10 \ --disk-type network-hdd \ --assign-public-ip \ --security-group-ids enp68jq81uun******** ``` - Terraform {#tf} Create a Managed Service for Apache Kafka® cluster with the following test specifications: * Name: `kafka-kraft`. * Environment: `production`. * Apache Kafka® version: `3.9`. * New network: `kafka-net`, subnets in each availability zone: * `kafka-subnet-a` with the `10.1.0.0/24` address range. * `kafka-subnet-b` with the `10.2.0.0/24` address range. * `kafka-subnet-d` with the `10.3.0.0/24` address range. * One broker host in each of the following availability zones: * `ru-central1-a` * `ru-central1-b` * `ru-central1-d` * `ru-central1-e` * [Host class](../concepts/instance-types.md): `s2.micro`. * Network HDD storage (`network-hdd`): `10` GB. * Public access: Enabled. * Security group: `kafka-sg` allowing all incoming and outgoing traffic. The configuration file for this Managed Service for Apache Kafka® cluster is as follows: ```hcl resource "yandex_mdb_kafka_cluster" "kafka-kraft" { name = "kafka-kraft" environment = "PRODUCTION" network_id = yandex_vpc_network.kafka-net.id subnet_ids = [yandex_vpc_subnet.kafka-subnet-a.id, yandex_vpc_subnet.kafka-subnet-b.id, yandex_vpc_subnet.kafka-subnet-d.id] security_group_ids = [yandex_vpc_security_group.kafka-sg.id] config { version = "3.9" brokers_count = 1 zones = ["ru-central1-a", "ru-central1-b", "ru-central1-d", "ru-central1-e"] assign_public_ip = true kafka { resources { disk_size = 10 disk_type_id = "network-hdd" resource_preset_id = "s2.micro" } kafka_config {} } } } resource "yandex_vpc_network" "kafka-net" { name = "kafka-net" } resource "yandex_vpc_subnet" "kafka-subnet-a" { name = "kafka-subnet-a" zone = "ru-central1-a" network_id = yandex_vpc_network.kafka-net.id v4_cidr_blocks = ["10.1.0.0/24"] } resource "yandex_vpc_subnet" "kafka-subnet-b" { name = "kafka-subnet-b" zone = "ru-central1-b" network_id = yandex_vpc_network.kafka-net.id v4_cidr_blocks = ["10.2.0.0/24"] } resource "yandex_vpc_subnet" "kafka-subnet-d" { name = "kafka-subnet-d" zone = "ru-central1-d" network_id = yandex_vpc_network.kafka-net.id v4_cidr_blocks = ["10.3.0.0/24"] } resource "yandex_vpc_security_group" "kafka-sg" { name = "kafka-sg" network_id = yandex_vpc_network.kafka-net.id ingress { protocol = "ANY" v4_cidr_blocks = ["0.0.0.0/0"] } egress { protocol = "ANY" v4_cidr_blocks = ["0.0.0.0/0"] } } ``` {% endlist %} ### Creating a cluster with KRaft on separate hosts (multi-host cluster) {#kafka-kraft-mh} {% list tabs group=instructions %} - CLI {#cli} Create a Managed Service for Apache Kafka® cluster with the following test specifications: * Name: `kafka-kraft-mh`. * Environment: `production`. * Apache Kafka® version: `3.9`. * Network ID: `enpc6eqfhmj2********`. * Subnet IDs: * `e9bhbia2scnk********` * `e2lfqbm5nt9r********` * `fl8beqmjckv8********` * Hosts: Two broker hosts in the `ru-central1-a` availability zone. * [Host class](../concepts/instance-types.md): `s2.micro`. * Network HDD storage (`network-hdd`): `10` GB. * [KRaft](../concepts/kraft.md) controller [host class](../concepts/instance-types.md): `s2.micro`. * [KRaft](../concepts/kraft.md) controller network SSD storage (`network-ssd`): `10` GB. * Public access: Enabled. * Security group: `enp68jq81uun********`. Run this command: ```bash yc managed-kafka cluster create \ --name kafka-kraft-mh \ --environment production \ --version 3.9 \ --network-id enpc6eqfhmj2******** \ --subnet-ids e9bhbia2scnk********,e2lfqbm5nt9r********,fl8beqmjckv8******** \ --zone-ids ru-central1-a \ --brokers-count 2 \ --resource-preset s2.micro \ --disk-size 10 \ --disk-type network-hdd \ --controller-resource-preset s2.micro \ --controller-disk-size 10 \ --controller-disk-type network-ssd \ --assign-public-ip \ --security-group-ids enp68jq81uun******** ``` - Terraform {#tf} Create a Managed Service for Apache Kafka® cluster with the following test specifications: * Name: `kafka-kraft-mh`. * Environment: `production`. * Apache Kafka® version: `3.9`. * New network: `kafka-net`, subnets in each availability zone: * `kafka-subnet-a` with the `10.1.0.0/24` address range. * `kafka-subnet-b` with the `10.2.0.0/24` address range. * `kafka-subnet-d` with the `10.3.0.0/24` address range. * Hosts: Two broker hosts in the `ru-central1-a` availability zone. * [Host class](../concepts/instance-types.md): `s2.micro`. * Network HDD storage (`network-hdd`): `10` GB. * [KRaft](../concepts/kraft.md) controller [host class](../concepts/instance-types.md): `s2.micro`. * [KRaft](../concepts/kraft.md) controller network SSD storage (`network-ssd`): `10` GB. * Public access: Enabled. * Security group: `kafka-sg` allowing all incoming and outgoing traffic. The configuration file for this Managed Service for Apache Kafka® cluster is as follows: ```hcl resource "yandex_mdb_kafka_cluster" "kafka-kraft-mh" { name = "kafka-kraft-mh" environment = "PRODUCTION" network_id = yandex_vpc_network.kafka-net.id subnet_ids = [yandex_vpc_subnet.kafka-subnet-a.id,yandex_vpc_subnet.kafka-subnet-b.id,yandex_vpc_subnet.kafka-subnet-d.id] security_group_ids = [yandex_vpc_security_group.kafka-sg.id] config { version = "3.9" brokers_count = 2 zones = ["ru-central1-a"] assign_public_ip = true kafka { resources { disk_size = 10 disk_type_id = "network-hdd" resource_preset_id = "s2.micro" } kafka_config {} } kraft { resources { resource_preset_id = "s2.micro" disk_type_id = "network-ssd" disk_size = 10 } } } } resource "yandex_vpc_network" "kafka-net" { name = "kafka-net" } resource "yandex_vpc_subnet" "kafka-subnet-a" { name = "kafka-subnet-a" zone = "ru-central1-a" network_id = yandex_vpc_network.kafka-net.id v4_cidr_blocks = ["10.1.0.0/24"] } resource "yandex_vpc_subnet" "kafka-subnet-b" { name = "kafka-subnet-b" zone = "ru-central1-b" network_id = yandex_vpc_network.kafka-net.id v4_cidr_blocks = ["10.2.0.0/24"] } resource "yandex_vpc_subnet" "kafka-subnet-d" { name = "kafka-subnet-d" zone = "ru-central1-d" network_id = yandex_vpc_network.kafka-net.id v4_cidr_blocks = ["10.3.0.0/24"] } resource "yandex_vpc_security_group" "kafka-sg" { name = "kafka-sg" network_id = yandex_vpc_network.kafka-net.id ingress { protocol = "ANY" v4_cidr_blocks = ["0.0.0.0/0"] } egress { protocol = "ANY" v4_cidr_blocks = ["0.0.0.0/0"] } } ``` {% endlist %} ### Creating a cluster with ZooKeeper on separate hosts (multi-host cluster) {#kafka-zk-mh} {% list tabs group=instructions %} - CLI {#cli} Create a Managed Service for Apache Kafka® cluster with the following test specifications: * Name: `kafka-zk-mh`. * Environment: `production`. * Apache Kafka® version: `3.9`. * Network ID: `enpc6eqfhmj2********`. * Subnet IDs: * `e9bhbia2scnk********` * `e2lfqbm5nt9r********` * `fl8beqmjckv8********` * Hosts: Two broker hosts in the `ru-central1-a` availability zone. * [Host class](../concepts/instance-types.md): `s2.micro`. * Network HDD storage (`network-hdd`): `10` GB. * ZooKeeper [host class](../concepts/instance-types.md): `s2.micro`. * [ZooKeeper](../concepts/index.md#zookeeper) network SSD storage (`network-ssd`): `10` GB. * Public access: Enabled. * Security group: `enp68jq81uun********`. Run this command: ```bash yc managed-kafka cluster create \ --name kafka-zk-mh \ --environment production \ --version 3.9 \ --network-id enpc6eqfhmj2******** \ --subnet-ids e9bhbia2scnk********,e2lfqbm5nt9r********,fl8beqmjckv8******** \ --zone-ids ru-central1-a \ --brokers-count 2 \ --resource-preset s2.micro \ --disk-size 10 \ --disk-type network-hdd \ --zookeeper-resource-preset s2.micro \ --zookeeper-disk-size 10 \ --zookeeper-disk-type network-ssd \ --assign-public-ip \ --security-group-ids enp68jq81uun******** ``` - Terraform {#tf} Create a Managed Service for Apache Kafka® cluster with the following test specifications: * Name: `kafka-zk-mh`. * Environment: `production`. * Apache Kafka® version: `3.9`. * New network: `kafka-net`, subnets in each availability zone: * `kafka-subnet-a` with the `10.1.0.0/24` address range. * `kafka-subnet-b` with the `10.2.0.0/24` address range. * `kafka-subnet-d` with the `10.3.0.0/24` address range. * Hosts: Two broker hosts in the `ru-central1-a` availability zone. * [Host class](../concepts/instance-types.md): `s2.micro`. * Network HDD storage (`network-hdd`): `10` GB. * ZooKeeper [host class](../concepts/instance-types.md): `s2.micro`. * [ZooKeeper](../concepts/index.md#zookeeper) network SSD storage (`network-ssd`): `10` GB. * Public access: Enabled. * Security group: `kafka-sg` allowing all incoming and outgoing traffic. The configuration file for this Managed Service for Apache Kafka® cluster is as follows: ```hcl resource "yandex_mdb_kafka_cluster" "kafka-zk-mh" { name = "kafka-zk-mh" environment = "PRODUCTION" network_id = yandex_vpc_network.kafka-net.id subnet_ids = [yandex_vpc_subnet.kafka-subnet-a.id,yandex_vpc_subnet.kafka-subnet-b.id,yandex_vpc_subnet.kafka-subnet-d.id] security_group_ids = [yandex_vpc_security_group.kafka-sg.id] config { version = "3.9" brokers_count = 2 zones = ["ru-central1-a"] assign_public_ip = true kafka { resources { disk_size = 10 disk_type_id = "network-hdd" resource_preset_id = "s2.micro" } kafka_config {} } zookeeper { resources { resource_preset_id = "s2.micro" disk_type_id = "network-ssd" disk_size = 10 } } } } resource "yandex_vpc_network" "kafka-net" { name = "kafka-net" } resource "yandex_vpc_subnet" "kafka-subnet-a" { name = "kafka-subnet-a" zone = "ru-central1-a" network_id = yandex_vpc_network.kafka-net.id v4_cidr_blocks = ["10.1.0.0/24"] } resource "yandex_vpc_subnet" "kafka-subnet-b" { name = "kafka-subnet-b" zone = "ru-central1-b" network_id = yandex_vpc_network.kafka-net.id v4_cidr_blocks = ["10.2.0.0/24"] } resource "yandex_vpc_subnet" "kafka-subnet-d" { name = "kafka-subnet-d" zone = "ru-central1-d" network_id = yandex_vpc_network.kafka-net.id v4_cidr_blocks = ["10.3.0.0/24"] } resource "yandex_vpc_security_group" "kafka-sg" { name = "kafka-sg" network_id = yandex_vpc_network.kafka-net.id ingress { protocol = "ANY" v4_cidr_blocks = ["0.0.0.0/0"] } egress { protocol = "ANY" v4_cidr_blocks = ["0.0.0.0/0"] } } ``` {% endlist %}