[Yandex Cloud documentation](../../index.md) > [Yandex Managed Service for Apache Spark™](../index.md) > [Step-by-step guides](index.md) > Clusters > Updating a cluster

# Updating an Apache Spark™ cluster

After creating a cluster, you can edit its basic and advanced settings.

{% list tabs group=instructions %}

- Management console {#console}

    To change the cluster settings:

    1. Open the [folder dashboard](https://console.yandex.cloud).
    1. Navigate to **Managed Service for Apache Spark™**.

    1. Select your cluster and click **Edit** in the top panel.

    1. Under **Basic parameters**:
       * Edit the cluster name and description.
       * Delete or add new [labels](../../resource-manager/concepts/labels.md).
       * Select a service account or [create a new one](../../iam/operations/sa/create.md#create-sa) with the `managed-spark.integrationProvider` role. The cluster will thus get the permissions it needs to work with other resources.

    1. Under **Network settings**, select a [security group](../../vpc/concepts/security-groups.md) for cluster network traffic.

    1. Under **Driver configuration** and **Executor configuration**, specify the number of instances and [computing resource configuration](../concepts/instance-types.md). The number of instances can be either fixed or autoscalable.

    1. Under **Advanced settings**:

        1. Delete or add names of pip and deb packages.

           The package name format and version are defined by the install command: `pip install` for pip packages and `apt install` for deb packages.
        1. In the **Maintenance** setting, update cluster [maintenance](../concepts/maintenance.md) time:

           * To enable maintenance at any time, select **At any time** (default).
           * To specify the preferred maintenance start time, select **By schedule** and specify the day of the week and the UTC time interval. For example, you can choose a time when the cluster is least loaded.
           
           Both active and stopped clusters are subject to maintenance. Maintenance operations include DBMS updates, patches, etc.

        1. Select a [Apache Hive™ Metastore cluster](../../metadata-hub/concepts/metastore.md) to connect as a metadata storage.
        1. Enable or disable cluster deletion protection.
        1. Enable or disable **History Server**. This option allows using the service to monitor [Spark History Server](https://spark.apache.org/docs/latest/monitoring.html) applications.
        1. Enable or disable **Write logs**. This option enables logging of Spark applications in the cluster:
            1. Select the log destination:
                * **Folder**: Select a folder from the list.
                * **Group**: Select a [log group](../../logging/concepts/log-group.md) from the list or create a new one.
            1. Select **Min. logging level** from the list.

    1. Click **Save**.

- CLI {#cli}

    If you do not have the Yandex Cloud CLI yet, [install and initialize it](../../cli/quickstart.md#install).

    The folder used by default is the one specified when [creating](../../cli/operations/profile/profile-create.md) the CLI profile. To change the default folder, use the `yc config set folder-id <folder_ID>` command. You can also specify a different folder for any command using `--folder-name` or `--folder-id`. If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions.

    To change the cluster settings:

    1. View the description of the CLI command for updating a cluster:

        ```bash
        yc managed-spark cluster update --help
        ```

    1. Provide a list of settings to update in the update cluster command:

        ```bash
        yc managed-spark cluster update <cluster_name_or_ID> \
          --new-name <cluster_name> \
          --description <cluster_description> \
          --labels <label_list> \
          --service-account-id <service_account_ID> \
          --security-group-ids <list_of_security_group_IDs> \
          --driver-preset-id <driver_resource_ID> \
          --driver-fixed-size <number_of_driver_instances> \
          --executor-preset-id <executor_resource_ID> \
          --executor-fixed-size <number_of_executor_instances> \
          --history-server-enabled <use_Spark_History_Server> \
          --metastore-cluster-id <Apache_Hive™_Metastore_cluster_ID> \
          --pip-packages <list_of_pip_packages> \
          --deb-packages <list_of_deb_packages> \
          --log-enabled \
          --log-folder-id <folder_ID> \
          --maintenance-window type=<maintenance_type>,`
                               `day=<day_of_week>,`
                               `hour=<hour> \
          --deletion-protection
        ```

        Where:

        * `--new-name`: Cluster name, unique within the cloud.
        * `--description`: Cluster description.
        * `--labels`: List of labels. Provide labels in `<key>=<value>` format.
        * `--service-account-id`: ID of the service account for access to Yandex Cloud services. Make sure to assign the `managed-spark.integrationProvider` role to this service account.
        * `--security-group-ids`: List of security group IDs.

        * Host configuration to run Apache Spark™ drivers:

            * `--driver-preset-id`: Driver [host class](../concepts/instance-types.md).
            * `--driver-fixed-size`: Fixed number of driver hosts.
            * `--driver-min-size`: Minimum number of driver hosts for autoscaling.
            * `--driver-max-size`: Maximum number of driver hosts for autoscaling.

            Specify either a fixed number of hosts (`--driver-fixed-size`) or minimum and maximum number of hosts (`--driver-min-size` and `--driver-max-size`) for autoscaling.

        * Host configuration to run Apache Spark™ executors:

            * `--executor-preset-id`: Executor [host class](../concepts/instance-types.md).
            * `--executor-fixed-size`: Fixed number of executor hosts.
            * `--executor-min-size`: Minimum number of executor hosts for autoscaling.
            * `--executor-max-size`: Maximum number of executor hosts for autoscaling.

            Specify either a fixed number of hosts (`--executor-fixed-size`) or minimum and maximum number of hosts (`--executor-min-size` and `--executor-max-size`) for autoscaling.

        * `--history-server-enabled`: Enables the [Spark History Server](https://spark.apache.org/docs/latest/monitoring.html) monitoring service.
        * `--metastore-cluster-id`: Apache Hive™ Metastore cluster ID. This setting connects the [Apache Hive™ Metastore](../../metadata-hub/concepts/metastore.md) metadata storage.

        * Lists of packages enabling you to install additional libraries and applications in the cluster:

            * `--pip-packages`: List of pip packages.
            * `--deb-packages`: List of deb packages.

            You can set version restrictions for the installed packages, e.g.:

            ```bash
            --pip-packages pandas==2.1.1,scikit-learn>=1.0.0,clickhouse-driver~=0.2.0
            ```

            The package name format and version are defined by the install command: `pip install` for pip packages and `apt install` for deb packages.

        * Logging parameters:

            * `--log-enabled`: Enables logging.
            * `--log-folder-id`: Folder ID. Logs will be written to the default [log group](../../logging/concepts/log-group.md) for this folder.
            * `--log-group-id`: Custom log group ID. Logs will be written to this group.

            Specify either `--log-folder-id` or `--log-group-id`.

        * `--maintenance-window`: [Maintenance window](../concepts/maintenance.md) settings that apply to both running and stopped clusters. The `type` setting defines the maintenance type:

            * `anytime`: Any time (default).
            * `weekly`: On a schedule. For this value, also specify the following:
                * `day`: Day of week, i.e., `MON`, `TUE`, `WED`, `THU`, `FRI`, `SAT`, or `SUN`.
                * `hour`: Sequence number of UTC hour interval, from `1` to `24`.
            
                  > For example, `1` stands for the interval from `00:00` to `01:00`, and `5`, from `04:00` to `05:00`.

        * `--deletion-protection`: Enables cluster protection against accidental deletion.

            Even with deletion protection on, one can still connect to the cluster manually and delete it.

        You can get the cluster name and ID with the [list of clusters](cluster-list.md#list-clusters) in the folder.

- Terraform {#tf}

    To change the cluster settings:

    1. Open the current Terraform configuration file with the infrastructure plan.

        To learn how to create this file, refer to [Creating a cluster](cluster-create.md).

    1. To change cluster settings, change the required field values in the configuration file.

        {% note alert %}

        Do not change the cluster name using Terraform. This will delete the existing cluster and create a new one.

        {% endnote %}

        Here is an example of the configuration file structure:

        ```hcl
        resource "yandex_spark_cluster" "my_spark_cluster" {
          description         = "<cluster_description>"
          name                = "my-spark-cluster"
          folder_id           = "<folder_ID>"
          service_account_id  = "<service_account_ID>"
          deletion_protection = <protect_cluster_from_deletion>

          labels = {
            <label_list>
          }

          network = {
            subnet_ids         = ["<list_of_subnet_IDs>"]
            security_group_ids = ["<list_of_security_group_IDs>"]
          }

          config = {
            resource_pools = {
              driver = {
                resource_preset_id = "<host_class>"
                size               = <fixed_number_of_instances>
              }
              executor = {
                resource_preset_id = "<host_class>"
                size               = <fixed_number_of_instances>
              }
            }
            history_server = {
              enabled = <use_Spark_History_Server>
            } 
            metastore = {
              cluster_id = "<Apache_Hive™_Metastore_cluster_ID>"
            }
            dependencies = {
              deb_packages = ["<list_of_deb_packages>"]
              pip_packages = ["<list_of_pip_packages>"]
            }
          }

          maintenance_window = {
            type = "<maintenance_type>"
            day  = "<day_of_week>"
            hour = "<hour>"
          }

          logging = {
            enabled      = <enable_logging>
            folder_id    = "<folder_ID>"
          }

        }
        ```

        Where:

        * `description`: Cluster description.
        * `service_account_id`: Service account ID.
        * `deletion_protection`: Cluster deletion protection, `true` or `false`.
        * `labels`: List of labels. Provide labels in `<key> = "<value>"` format.
        * `security_group_ids`: List of security group IDs.
        * `driver`: Host configuration to run Apache Spark™ drivers. In this section, specify:

          * [Host class](../concepts/instance-types.md) in the `resource_preset_id` parameter.
          * Number of instances. Specify a fixed number in the `size` parameter or the minimum and maximum number for autoscaling in the `min_size` and `max_size` parameters.

        * `executor`: Host configuration to run Apache Spark™ executors. In this section, specify:

          * [Host class](../concepts/instance-types.md) in the `resource_preset_id` parameter.
          * Number of instances. Specify a fixed number in the `size` parameter or the minimum and maximum number for autoscaling in the `min_size` and `max_size` parameters.

        * `maintenance_window`: [Maintenance](../concepts/maintenance.md) window settings (including for disabled clusters). In this section, specify:

          * Maintenance type in the `type` parameter. The possible values include:
            * `ANYTIME`: Any time.
            * `WEEKLY`: On a schedule.
          * Day of week for the `WEEKLY` maintenance type in the `day` parameter, i.e., `MON`, `TUE`, `WED`, `THU`, `FRI`, `SAT`, or `SUN`.
          * UTC hour from `1` to `24` for the `WEEKLY` maintenance type in the `hour` parameter.

        * `history_server`: Connecting Apache Spark™ History Server. To use the service, set the `enabled` parameter to `true`.

        * `metastore`: Connecting a [Apache Hive™ Metastore](../../metadata-hub/concepts/metastore.md) metadata storage. Specify a Apache Hive™ Metastore cluster ID in the `cluster_id` parameter.

        * `dependencies`: Additional deb and pip packages for running Apache Spark™ jobs. In this section, specify:
          * `deb_packages`: Names of deb packages. Their format depends on the `apt install` installation command.
          * `pip_packages`: Names of pip packages. Their format depends on the `pip install` installation command.

        * `logging`: Logging parameters. Logs generated by Apache Spark™ components will be sent to Yandex Cloud Logging. To enable logging:

          * Set it to `enabled = true`.
          * Specify one of two log storage locations:

             * `folder_id`: Folder ID. Logs will be written to the default [log group](../../logging/concepts/log-group.md) for this folder.
             * `log_group_id`: Custom log group ID. Logs will be written to this group.

    1. Make sure the settings are correct.

        1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.
        1. Run this command:
        
           ```bash
           terraform validate
           ```
        
           Terraform will show any errors found in your configuration files.

    1. Confirm resource changes.

        1. Run this command to view the planned changes:
        
           ```bash
           terraform plan
           ```
        
           If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.
        
        1. If everything looks correct, apply the changes:
           1. Run this command:
        
              ```bash
              terraform apply
              ```
        
           1. Confirm updating the resources.
           1. Wait for the operation to complete.

    For more information, see [this Terraform provider guide](../../terraform/resources/airflow_cluster.md).

- gRPC API {#grpc-api}

    To change the cluster settings:

    1. [Get an IAM token for API authentication](../api-ref/authentication.md) and put it into an environment variable:

        ```bash
        export IAM_TOKEN="<IAM_token>"
        ```

    1. Clone the [cloudapi](https://github.com/yandex-cloud/cloudapi) repository:
       
       ```bash
       cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
       ```
       
       Below, we assume that the repository contents reside in the `~/cloudapi/` directory.

    1. Create a file named `body.json` and paste the following code into it:

        ```json
        {
          "cluster_id": "<cluster_ID>",
          "update_mask": "<list_of_settings_to_update>",
          "name": "<cluster_name>",
          "description": "<cluster_description>",
          "labels": { <label_list> },
          "config_spec": {
           "resource_pools": {
             "driver": {
               "resource_preset_id": "<driver_resource_ID>",
               "scale_policy": {
                 "fixed_scale": {
                   "size": "<number_of_driver_instances>"
                 }
               }
             },
             "executor": {
               "resource_preset_id": "<executor_resource_ID>",
               "scale_policy": {
                 "auto_scale": {
                   "min_size": "<minimum_number_of_executor_instances>",
                   "max_size": "<maximum_number_of_executor_instances>"
                 }
               }
             }
           },
           "history_server": {
             "enabled": <use_Spark_History_Server>
           },
            "dependencies": {
              "pip_packages": [ <list_of_pip_packages> ],
              "deb_packages": [ <list_of_deb_packages> ]
            },
            "metastore": {
              "cluster_id": "<Apache_Hive™_Metastore_cluster_ID>"
            }
          },
          "network_spec": {
            "security_group_ids": [ <list_of_security_group_IDs> ]
          },
          "deletion_protection": <deletion_protection>,
          "service_account_id": "<service_account_ID>",
          "logging": {
            "enabled": <use_of_logging>,
            "log_group_id": "<log_group_ID>",
            "folder_id": "<folder_ID>"
          }
        }
        ```

        Where:

        * `cluster_id`: Cluster ID. You can get it with the [list of clusters in the folder](cluster-list.md#list-clusters).
        * `update_mask`: List of settings you want to update as an array of strings (`paths[]`).

            {% cut "Format for listing settings" %}

            ```yaml
            "update_mask": {
                "paths": [
                    "<setting_1>",
                    "<setting_2>",
                    ...
                    "<setting_N>"
                ]
            }
            ```

            {% endcut %}

            {% note warning %}

            When you update a cluster, all its parameters will reset to their defaults unless explicitly provided in the request. To avoid this, list the settings you want to change in the `update_mask` parameter.

            {% endnote %}

        * `name`: Cluster name.
        * `description`: Cluster description.
        * `labels`: List of labels provided in `"<key>": "<value>"` format.
        * `config_spec`: Cluster configuration:

           * `resource_pools`: Resource pool configuration:

               * `driver`: Host configuration to run Apache Spark™ drivers.

                   * `resource_preset_id`: Driver [host class](../concepts/instance-types.md).
                   * `scale_policy`: Host group scaling policy for the driver:
                       * `fixed_scale`: Fixed scaling policy.

                          * `size`: Number of driver hosts.

                       * `auto_scale`: Automatic scaling policy.

                           * `min_size`: Minimum number of driver hosts.
                           * `max_size`: Maximum number of driver hosts.

                       Specify either `fixed_scale` or `auto_scale`.

               * `executor`: Host configuration to run Apache Spark™ executors.

                   * `resource_preset_id`: Executor [host class](../concepts/instance-types.md).
                   * `scale_policy`: Host group scaling policy for the executor:

                       * `fixed_scale`: Fixed scaling policy.

                           * `size`: Number of executor hosts.

                       * `auto_scale`: Automatic scaling policy.

                           * `min_size`: Minimum number of executor hosts.
                           * `max_size`: Maximum number of executor hosts.

                       Specify either `fixed_scale` or `auto_scale`.

               * `history_server`: History server parameters.

                   * `enabled`: Flag to enable history server. It allows using the service to monitor Spark History Server applications.

               * `dependencies`: Lists of packages enabling you to install additional libraries and applications on the cluster.

                   * `pip_packages`: List of pip packages.
                   * `deb_packages`: List of deb packages.

                   You can set version restrictions for the installed packages, e.g.:

                   ```bash
                   "dependencies": {
                     "pip_packages": [
                       "pandas==2.1.1",
                       "scikit-learn>=1.0.0",
                       "clickhouse-driver~=0.2.0"
                     ]
                   }
                   ```

                   The package name format and version are defined by the install command: `pip install` for pip packages and `apt install` for deb packages.

               * `metastore`: Parameters of the cluster’s metadata storage.

                   * `cluster_id`: [Apache Hive™ Metastore](../../metadata-hub/concepts/metastore.md) cluster ID.

           * `network`: Network settings:

               * `security_group_ids`: List of [security group](../../vpc/concepts/security-groups.md) IDs.

           * `deletion_protection`: Enables cluster protection against accidental deletion. The possible values are `true` or `false`.

              Even with deletion protection on, one can still connect to the cluster manually and delete it.

           * `service_account_id`: ID of the service account for access to Yandex Cloud services. Make sure to assign the `managed-spark.integrationProvider` role to this service account:

           * `logging`: Logging parameters:
               * `enabled`: Enables logging. The possible values are `true` or `false`. Logs generated by Apache Spark™ components will be sent to Yandex Cloud Logging. The possible values are `true` or `false`.
               * `folder_id`: Folder ID. Logs will be written to the default [log group](../../logging/concepts/log-group.md) for this folder.
               * `log_group_id`: Custom log group ID. Logs will be written to this group.

               Specify either `folder_id` or `log_group_id`.

    1. Call the [ClusterService.Update](../api-ref/grpc/Cluster/update.md) method, e.g., via the following [gRPCurl](https://github.com/fullstorydev/grpcurl) request:

        ```bash
        grpcurl \
            -format json \
            -import-path ~/cloudapi/ \
            -import-path ~/cloudapi/third_party/googleapis/ \
            -proto ~/cloudapi/yandex/cloud/spark/v1/cluster_service.proto \
            -rpc-header "Authorization: Bearer $IAM_TOKEN" \
            -d @ \
            spark.api.cloud.yandex.net:443 \
            yandex.cloud.spark.v1.ClusterService.Update \
            < body.json
        ```

    1. Check the [server response](../api-ref/grpc/Cluster/create.md#yandex.cloud.operation.Operation) to make sure your request was successful.

{% endlist %}