[Yandex Cloud documentation](../../index.md) > [Yandex Data Processing](../index.md) > [Step-by-step guides](index.md) > Yandex Data Processing subclusters > Managing subclusters

# Managing subclusters Yandex Data Processing

In addition to [updating the settings](subcluster-update.md) of a particular subcluster, you can create new and delete existing subclusters.

{% note warning %}

Each cluster can only have one subcluster with a master host, so you cannot create or delete subclusters with this role. You cannot delete data storage subclusters either.

{% endnote %}

## Getting a list of subclusters in a cluster {#list-subclusters}

{% list tabs group=instructions %}

- Management console {#console}

  1. Open the [folder dashboard](https://console.yandex.cloud).
  1. Navigate to **Yandex Data Processing**.
  1. Click the name of your cluster and select the **Subclusters** tab.

- CLI {#cli}

  If you do not have the Yandex Cloud CLI yet, [install and initialize it](../../cli/quickstart.md#install).

  The folder used by default is the one specified when [creating](../../cli/operations/profile/profile-create.md) the CLI profile. To change the default folder, use the `yc config set folder-id <folder_ID>` command. You can also specify a different folder for any command using `--folder-name` or `--folder-id`. If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions.

  To get a list of Yandex Data Processing subclusters in a cluster, run the following command:

  ```bash
  yc dataproc subcluster list --cluster-name=<cluster_name>
  ```

  You can get the cluster name with the [list of clusters in the folder](cluster-list.md#list).

{% endlist %}

## Creating a subcluster {#add-subcluster}

The number of hosts in Yandex Data Processing clusters is limited by [quotas](https://console.yandex.cloud/cloud?section=quotas) on the number of vCPUs and the amount of RAM available to virtual machines in your cloud. To view the available resources, navigate to [Quotas](https://console.yandex.cloud/cloud?section=quotas) and find **Compute Cloud**.

{% list tabs group=instructions %}

- Management console {#console}

  1. In the [management console](https://console.yandex.cloud), select the relevant folder.
  1. Navigate to **Yandex Data Processing** and select the cluster.
  1. Navigate to **Subclusters**.
  1. Click **Create subcluster**.
  1. Specify the subcluster settings:
     1. **Hosts**: Select the number of hosts.
     1. **Roles**: Select the subcluster roles depending on the services to deploy on the hosts:

         * `COMPUTENODE`: Role for processing data. In subclusters with this role, you can deploy YARN NodeManager and Spark libraries.
         * `DATANODE`: Role for storing data. In subclusters with this role, you can deploy YARN NodeManager, Spark libraries, HBase RegionServer, and HDFS Datanode.

     1. Under [**Host class**](../concepts/instance-types.md), select a platform and computing resources available to the host.
     1. Under **Storage size**, specify the storage type and size.
     1. Under **Network settings**:

         * Select **Network ID format**.
         * Specify **Subnet** (subnet of the network where your cluster resides).
         * Optionally, enable **Public access** to allow access to subcluster hosts from the internet.

             You will not be able to change this setting after the subcluster is created.

             {% note tip %}

             You can delete data processing subclusters and recreate them with the required configuration.

             {% endnote %}

     1. Optionally, enable **Autoscaling**.
  1. Click **Add subcluster**.

- CLI {#cli}

  If you do not have the Yandex Cloud CLI yet, [install and initialize it](../../cli/quickstart.md#install).

  The folder used by default is the one specified when [creating](../../cli/operations/profile/profile-create.md) the CLI profile. To change the default folder, use the `yc config set folder-id <folder_ID>` command. You can also specify a different folder for any command using `--folder-name` or `--folder-id`. If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions.

  To create a subcluster:
  1. See the description of the CLI command for creating a subcluster:

     ```bash
     yc dataproc subcluster create --help
     ```

  1. Specify the subcluster settings in the create command below (the example does not show all available settings):

     ```bash
     yc dataproc subcluster create <subcluster_name> \
       --cluster-name=<cluster_name> \
       --role=<subcluster_role> \
       --resource-preset=<host_class> \
       --disk-type=<storage_type> \
       --disk-size=<storage_size_in_GB> \
       --subnet-name=<subnet_name> \
       --hosts-count=<number_of_hosts>
     ```

     Where:
     * `--cluster-name`: Cluster name. You can get the cluster name with the [list of clusters in the folder](cluster-list.md#list).
     * `--role`: Subcluster role, `datanode` or `computenode`.
     * `--resource-preset`: [Host class](../concepts/instance-types.md).
     * `--disk-type`: [Storage type](../concepts/storage.md), which can be `network-ssd`, `network-hdd`, or `network-ssd-nonreplicated`.
     * `--disk-size`: Storage size, in GB.
     * `--subnet-name`: [Subnet name](../../vpc/concepts/network.md#subnet).
     * `--hosts-count`: Number of subcluster hosts. The minimum value is `1`, and the maximum value is `32`.

- Terraform {#tf}

  1. Open the current Terraform configuration file with the infrastructure plan.

     To learn how to create this file, refer to [Creating a cluster](cluster-create.md).
  1. In the Yandex Data Processing cluster description, add a `subcluster_spec` section containing the settings for the new subcluster:

     ```hcl
     resource "yandex_dataproc_cluster" "data_cluster" {
       ...
       cluster_config {
         ...
         subcluster_spec {
           name = "<subcluster_name>"
           role = "<subcluster_role>"
           resources {
             resource_preset_id = "<host_class>"
             disk_type_id       = "<storage_type>"
             disk_size          = <storage_size_in_GB>
           }
           subnet_id   = "<subnet_ID>"
           hosts_count = <number_of_hosts_in_subcluster>
           ...
         }
       }
     }
     ```

     Where `role` is the subcluster role, `COMPUTENODE` or `DATANODE`.

  1. Make sure the settings are correct.

     1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.
     1. Run this command:
     
        ```bash
        terraform validate
        ```
     
        Terraform will show any errors found in your configuration files.

  1. Confirm resource changes.

     1. Run this command to view the planned changes:
     
        ```bash
        terraform plan
        ```
     
        If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.
     
     1. If everything looks correct, apply the changes:
        1. Run this command:
     
           ```bash
           terraform apply
           ```
     
        1. Confirm updating the resources.
        1. Wait for the operation to complete.

  For more detailed information about the resources you can create with Terraform, see [this provider guide](../../terraform/resources/dataproc_cluster.md).

{% endlist %}

## Deleting a subcluster {#remove-host}

{% note warning %}

You cannot delete data storage subclusters.

{% endnote %}

{% list tabs group=instructions %}

- Management console {#console}

  To delete a subcluster:
  1. In the [management console](https://console.yandex.cloud), select the relevant folder.
  1. Navigate to **Yandex Data Processing** and select the cluster.
  1. Navigate to **Subclusters**.
  1. Click ![image](../../_assets/console-icons/ellipsis.svg) for the subcluster in question and select **Delete**.
  1. Optionally, specify the [decommissioning](../concepts/decommission.md) timeout.
  1. In the window that opens, click **Delete**.

- CLI {#cli}

  If you do not have the Yandex Cloud CLI yet, [install and initialize it](../../cli/quickstart.md#install).

  The folder used by default is the one specified when [creating](../../cli/operations/profile/profile-create.md) the CLI profile. To change the default folder, use the `yc config set folder-id <folder_ID>` command. You can also specify a different folder for any command using `--folder-name` or `--folder-id`. If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions.

  To delete a subcluster in a Yandex Data Processing cluster, run this command:

  ```bash
  yc dataproc subcluster delete <subcluster_name_or_ID> \
    --cluster-name=<cluster_name>
  ```

  You can request the subcluster name or ID with the [list of cluster subclusters](#list-subclusters), and the cluster name, with the [list of folder clusters](cluster-list.md#list).

- Terraform {#tf}

  1. Open the current Terraform configuration file with the infrastructure plan.

     To learn how to create this file, refer to [Creating a cluster](cluster-create.md).
  1. Delete the `subcluster_spec` section of the relevant subcluster from the Yandex Data Processing cluster description.
  1. Make sure the settings are correct.

     1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.
     1. Run this command:
     
        ```bash
        terraform validate
        ```
     
        Terraform will show any errors found in your configuration files.

  1. Type `yes` and press **Enter**.

     1. Run this command to view the planned changes:
     
        ```bash
        terraform plan
        ```
     
        If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.
     
     1. If everything looks correct, apply the changes:
        1. Run this command:
     
           ```bash
           terraform apply
           ```
     
        1. Confirm updating the resources.
        1. Wait for the operation to complete.

  For more detailed information about the resources you can create with Terraform, see [this provider guide](../../terraform/resources/dataproc_cluster.md).

{% endlist %}