[Yandex Cloud documentation](../../../index.md) > [Yandex Data Processing](../../index.md) > [Step-by-step guides](../index.md) > Apache and other third-party services > Delta Lake > Configuring Delta Lake in multi-cluster mode

# Configuring Delta Lake in multi-cluster mode

When in multi-cluster mode, Yandex Data Processing uses a [Yandex Managed Service for YDB](../../../ydb/index.md) database to manage access to Delta Lake tables from different clusters and Apache Spark™ jobs.

For more information about Delta Lake, see [Delta Lake in Yandex Data Processing](../../concepts/deltalake.md) and [this Delta Lake guide](https://docs.delta.io/latest/index.html).


{% note info %}

Delta Lake is not part of Yandex Data Processing. It is not covered by Yandex Cloud support, and its usage is not governed by the [Yandex Data Processing Terms of Use](https://yandex.ru/legal/cloud_termsofuse/?lang=en).

{% endnote %}


## Set up your infrastructure {#prereq}

1. [Create a serverless Managed Service for YDB database](../../../ydb/operations/manage-databases.md#create-db-serverless).
1. [Create a service account](../../../iam/operations/sa/create.md) with the `ydb.editor` role to access YDB.
1. [Create a static access key](../../../iam/operations/authentication/manage-access-keys.md#create-access-key) for the service account.
1. [Create a Yandex Lockbox secret](../../../lockbox/operations/secret-create.md) containing the static key data as two `key-value` pairs:

    * `key-id` as the key and `<static_key_ID>` as the value.
    * `key-secret` as the key and `<static_key_secret_part>` as the value.

1. Configure one or more Yandex Data Processing clusters to work with Delta Lake:

    1. If you do not have a Yandex Data Processing cluster, [create one](../cluster-create.md).
    1. If you attached a [Yandex Object Storage bucket](../../../storage/concepts/bucket.md) to your cluster:

        1. Create a folder named `warehouse` in the bucket.
        1. [Set](../../concepts/settings-list.md#change-properties) `spark.sql.warehouse.dir` to `s3a://<bucket_name>/warehouse/`.

    1. [Create a Apache Hive™ Metastore cluster](../../../metadata-hub/operations/metastore/cluster-create.md) and [connect](../../../metadata-hub/operations/metastore/data-processing-connect.md) it to your Yandex Data Processing cluster.

1. Assign the `lockbox.payloadViewer` role to the service account you used to create the Yandex Data Processing clusters. You can do this:

    * [Only for the secret you created earlier](../../../lockbox/operations/secret-access.md).
    * [At the folder level](../../../iam/operations/sa/assign-role-for-sa.md).

## Set the component properties to work with Delta Lake {#settings}

1. Download the archive with the required Delta Lake libraries and add-ons to connect to Managed Service for YDB:

    * [Delta Lake 2.0.2](https://github.com/yandex-cloud/yc-delta/releases/download/v1.1/yc-delta20-multi-dp21-1.1-fatjar.jar) for Yandex Data Processing 2.1.0 or 2.1.3
    * [Delta Lake 2.3.0](https://github.com/yandex-cloud/yc-delta/releases/download/v1.1/yc-delta23-multi-dp21-1.1-fatjar.jar) for Yandex Data Processing 2.1.4 and higher

    You can check out the source code for add-ons to connect to YDB in these repositories:

    * [Add-ons for Delta Lake 2.0.2](https://github.com/yandex-cloud/yc-delta/blob/develop/yc-delta20)
    * [Add-ons for Delta Lake 2.3.0](https://github.com/yandex-cloud/yc-delta/blob/develop/yc-delta23)

1. Add the downloaded archive to the dependencies of all clusters or individual jobs that need access to Delta Lake tables. There are two ways to do this:

    * Save the archive to the Object Storage bucket and provide the file URL in the `spark.jars` property:

        `spark.jars=s3a://<bucket_name>/<file_path>`

        Make sure the cluster service account has read access to the bucket.

    * Copy the archive to all cluster nodes manually or using [initialization scripts](../../concepts/init-action.md) and provide the full file path in the `spark.driver.extraClassPath` and `spark.executor.extraClassPath` properties.

1. Set the following [properties](../../concepts/settings-list.md) at the level of clusters or individual Apache Spark™ jobs that need access to Delta Lake tables:

    * Set `spark.sql.extensions` to `io.delta.sql.DeltaSparkSessionExtension`.
    * Set `spark.sql.catalog.spark_catalog` to `org.apache.spark.sql.delta.catalog.DeltaCatalog`.
    * Set `spark.delta.logStore.s3a.impl` to `ru.yandex.cloud.custom.delta.YcS3YdbLogStore`.
    * Set `spark.io.delta.storage.S3DynamoDBLogStore.ddb.endpoint` to the Document API endpoint value. You can find it on the **Overview** tab of your database in the [management console](https://console.yandex.cloud/cloud).
    * Set `spark.io.delta.storage.S3DynamoDBLogStore.ddb.lockbox` to the Lockbox secret ID value. You can find it on the **Overview** tab of your Lockbox in the [management console](https://console.yandex.cloud/cloud).

You can now use Delta Lake in multi-cluster mode.

## Delta Lake use case {#example}

This use case was tested on a Yandex Data Processing 2.1.7 cluster.

1. [Use SSH to connect](../connect-ssh.md) to the Yandex Data Processing cluster master host.

1. Run an Apache Spark™ session in the cluster by providing the required parameters:

    ```bash
    spark-sql  \
        --conf spark.jars=s3a://<bucket_name>/yc-delta23-multi-dp21-1.1-fatjar.jar \
        --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \
        --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.YcDeltaCatalog \
        --conf spark.delta.logStore.s3a.impl=ru.yandex.cloud.custom.delta.YcS3YdbLogStore \
        --conf spark.io.delta.storage.S3DynamoDBLogStore.ddb.endpoint=<Document_API_endpoint> \
        --conf spark.io.delta.storage.S3DynamoDBLogStore.ddb.lockbox=<secret_ID>
    ```

1. In your active session, create a database and switch to it:

    ```sql
    CREATE DATABASE testdelta;
    USE testdelta;
    ```

1. Create a test table and populate it with data:

    ```sql
    CREATE TABLE tab1(a INTEGER NOT NULL, b VARCHAR(100)) USING DELTA;
    INSERT INTO tab1 VALUES (1,'One'), (2,'Two'), (3,'Three');
    ```

1. Update the `b` column by concatenating it with the `a` column values converted to a string:

    ```sql
    UPDATE tab1 SET b=b || ' ** ' || CAST(a AS VARCHAR(10));
    ```

1. Check the result:

    ```sql
    SELECT * FROM tab1;
    ```

    ```sql
    3	Three ** 3
    2	Two ** 2
    1	One ** 1
    ```

## Additional multi-cluster mode settings for production clusters in Yandex Data Processing {#prod-configs}

To improve Delta Lake performance and streamline data storage when using multi-cluster mode, configure additional YDB settings.

### Setting up Managed Service for YDB throughput {#throughput}

By default, a YDB in serverless mode is created with a [throughput](../../../ydb/concepts/serverless-and-dedicated.md#capacity) of 10 request units per second. This may not be enough for heavy use of Delta Lake tables.

To avoid Delta Lake performance degradation due to insufficient YDB throughput, monitor the **Document API units overflow** metric on the [YDB](../../../ydb/operations/monitoring.md) monitoring chart. Increase the throughput limit, if required.

There is a total throughput limit that applies to all YDB databases in the cloud, which depends on the quota. Contact [support](https://center.yandex.cloud/support) to request a quota increase, if required.

### Setting up automatic cleanup {#auto-vacuum}

When working with Delta Lake, metadata versions that are not in use may accumulate in a YDB table and Object Storage buckets. You can streamline storage use and boost Delta Lake performance using a [ready-made script](https://github.com/yandex-cloud/yc-delta/blob/develop/cf-cleanup/cfunc.py) that will automatically clean up outdated metadata from the YDB table and bucket on a regular basis.

The script is installed in the cloud as two [serverless functions](../../../functions/concepts/index.md):

* Function for cleaning up data in the YDB table. It is called automatically once an hour.
* Function for cleaning up data in the buckets. It is called automatically once a day.

To add these cleanup functions to your cloud:

1. If you do not have the Yandex Cloud CLI yet, [install and initialize it](../../../cli/quickstart.md#install).
1. Download these files from the [cf-cleanup](https://github.com/yandex-cloud/yc-delta/tree/develop/cf-cleanup) folder:

    * `cfunc.py`: Cleanup script source code.
    * `delta-prefixes.txt`: File with prefixes of paths to temporary Delta Lake files in buckets.
    * `pack.sh`: ZIP archive creation script.
    * `requirements.txt`: File with environment requirements to set up functions.

    Save these files to the `cf-cleanup` folder in the working directory.

1. Make the `pack.sh` file executable:

    ```bash
    chmod +x ./cf-cleanup/pack.sh
    ```

1. In the `delta-prefixes.txt` file, specify the paths to the Object Storage bucket folders with temporary Delta Lake files. Make sure to provide each path on a new line in the following format:

    ```txt
    BucketName Mode PathPrefix
    ```

    The `Mode` field can take the following values:

    * `W`: Warehouse, path for storing multiple databases.
    * `D`: Database, path for storing a single database.
    * `T`: Table, path for storing a specific table.

    Here is an example:

    ```txt
    mybucket1 W warehouse/
    mybucket2 D warehouse/testdelta2.db/
    mybucket3 T warehouse/testdelta3.db/tab1/
    ```

1. Place the `delta-prefixes.txt` file to your Object Storage bucket.
1. Download and save the following files for managing the cleanup functions to the working directory:

    * [ddb-maint-config.sh](https://github.com/yandex-cloud/yc-delta/blob/develop/ddb-maint-config.sh): Setup parameters
    * [ddb-maint-setup.sh](https://github.com/yandex-cloud/yc-delta/blob/develop/ddb-maint-setup.sh): Setup script
    * [ddb-maint-remove.sh](https://github.com/yandex-cloud/yc-delta/blob/develop/ddb-maint-remove.sh): Removal script

1. In the `ddb-maint-config.sh` file, specify the following parameters:

    * `sa_name`: Name of the service account that will be created for running the functions.
    * `cf_ddb_name`: Name of the serverless database cleanup function. It must be unique within the folder.
    * `cf_s3_name`: Name of the serverless bucket cleanup function. It must be unique within the folder.
    * `docapi_endpoint`: Document API endpoint. You can find it on the **Overview** tab of your YDB database in the [management console](https://console.yandex.cloud/cloud).
    * `docapi_table`: Name of the Delta Lake table to clean up.
    * `s3_prefix_file`: Path to the `delta-prefixes.txt` file in the Object Storage bucket, e.g., `s3://<bucket_name>/delta-prefixes.txt`.

1. Run the setup script in your local directory:

    ```bash
    bash ./ddb-maint-setup.sh
    ```

This will add to the cloud the functions for cleaning up temporary files in YDB tables and Object Storage buckets. You can check the new functions using the [management console](https://console.yandex.cloud/cloud).

If you no longer need the cleanup functions, run this script to remove them:

```bash
bash ./ddb-maint-remove.sh
```

The `spark.io.delta.storage.S3DynamoDBLogStore.ddb.ttl` Spark property sets the TTL for metadata records, which is `86400` seconds (24 hours) by default. The actual TTL for a specific record may be longer as it depends on when the cleanup function runs.