[Yandex Cloud documentation](../../index.md) > [Yandex Data Processing](../index.md) > [Step-by-step guides](index.md) > Apache and other third-party services > Apache Iceberg™ configuration

# Setting up Apache Iceberg™ in a Yandex Data Processing cluster

Yandex Data Processing 2.0 or higher supports using Apache Iceberg™ tables together with the Apache Spark™ engine.

For more information about Apache Iceberg™, see [Apache Iceberg™ in Yandex Data Processing](../concepts/apache-iceberg.md) and [this official guide](https://iceberg.apache.org/docs/latest/).


{% note info %}

Apache Iceberg™ is not part of Yandex Data Processing. It is not covered by Yandex Cloud support and its usage is not governed by the [Yandex Data Processing Terms of Use](https://yandex.ru/legal/cloud_termsofuse/?lang=en).

{% endnote %}


## Set the component properties to work with Apache Iceberg™ {#settings}

1. [Set](../concepts/settings-list.md#change-properties) the `spark:spark.sql.extensions` property to `org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions`. You can do this either at the cluster or job level.

1. Add the Apache Iceberg™ libraries to the cluster or job dependencies. Library versions must be [compatible with the Yandex Data Processing version](../concepts/apache-iceberg.md#compatibility).

    {% list tabs %}

    - Image 2.0.x

        To add the required library versions, use one of the following methods:

        * Set up access to the [Maven](https://maven.apache.org/index.html) repository and [set](../concepts/settings-list.md#change-properties) the `spark:spark.jars.packages` property to `org.apache.iceberg:iceberg-spark-runtime-3.0_2.12:1.0.0`.

            You can set up Maven access in two ways:
            
            * In your cluster's [security group](../../vpc/concepts/security-groups.md), allow network access to the [Maven Central](https://repo.maven.apache.org/maven2/) repository.
            * Configure an [alternative Maven repository](https://maven.apache.org/guides/mini/guide-mirror-settings.html) and allow traffic to it in the cluster [security group](../../vpc/concepts/security-groups.md).

        * Download the [iceberg-spark-runtime-3.0_2.12-1.0.0.jar](https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-spark-runtime-3.0_2.12/1.0.0/iceberg-spark-runtime-3.0_2.12-1.0.0.jar) library file and grant access to it using one of the following methods:

            * Save the file to a Yandex Object Storage bucket and provide the file's URL in the `spark:spark.jars` property.
            
                The file's URL has the following format: `s3a://<bucket_name>/<file_path>`.
            
                This bucket must be specified in the cluster settings. Make sure the cluster service account has read access to the bucket.
            
            * Copy the file to all the cluster nodes manually or using [initialization scripts](../concepts/init-action.md) and provide the full file path in the `spark:spark.driver.extraClassPath` and `spark:spark.executor.extraClassPath` properties.

    - Image 2.1.0–2.1.3

        To add the required library versions, use one of the following methods:

        * Set up access to the [Maven](https://maven.apache.org/index.html) repository and [set](../concepts/settings-list.md#change-properties) the `spark:spark.jars.packages` property to `org.apache.iceberg:iceberg-spark-runtime-3.2_2.12-1.4.3`.

            You can set up Maven access in two ways:
            
            * In your cluster's [security group](../../vpc/concepts/security-groups.md), allow network access to the [Maven Central](https://repo.maven.apache.org/maven2/) repository.
            * Configure an [alternative Maven repository](https://maven.apache.org/guides/mini/guide-mirror-settings.html) and allow traffic to it in the cluster [security group](../../vpc/concepts/security-groups.md).

        * Download the [iceberg-spark-runtime-3.2_2.12-1.4.3.jar](https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-spark-runtime-3.2_2.12/1.4.3/iceberg-spark-runtime-3.2_2.12-1.4.3.jar) library file and grant access to it using one of the following methods:

            * Save the file to a Yandex Object Storage bucket and provide the file's URL in the `spark:spark.jars` property.
            
                The file's URL has the following format: `s3a://<bucket_name>/<file_path>`.
            
                This bucket must be specified in the cluster settings. Make sure the cluster service account has read access to the bucket.
            
            * Copy the file to all the cluster nodes manually or using [initialization scripts](../concepts/init-action.md) and provide the full file path in the `spark:spark.driver.extraClassPath` and `spark:spark.executor.extraClassPath` properties.

    - Image 2.1.4–2.1.x

        To add the required library versions, use one of the following methods:

        * Set up access to the [Maven](https://maven.apache.org/index.html) repository and [set](../concepts/settings-list.md#change-properties) the `spark:spark.jars.packages` property to `org.apache.iceberg:iceberg-spark-runtime-3.3_2.12:1.5.2`.

            You can set up Maven access in two ways:
            
            * In your cluster's [security group](../../vpc/concepts/security-groups.md), allow network access to the [Maven Central](https://repo.maven.apache.org/maven2/) repository.
            * Configure an [alternative Maven repository](https://maven.apache.org/guides/mini/guide-mirror-settings.html) and allow traffic to it in the cluster [security group](../../vpc/concepts/security-groups.md).

        * Download the [iceberg-spark-runtime-3.3_2.12-1.5.2.jar](https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-spark-runtime-3.3_2.12/1.5.2/iceberg-spark-runtime-3.3_2.12-1.5.2.jar) library file and grant access to it using one of the following methods:

            * Save the file to a Yandex Object Storage bucket and provide the file's URL in the `spark:spark.jars` property.
            
                The file's URL has the following format: `s3a://<bucket_name>/<file_path>`.
            
                This bucket must be specified in the cluster settings. Make sure the cluster service account has read access to the bucket.
            
            * Copy the file to all the cluster nodes manually or using [initialization scripts](../concepts/init-action.md) and provide the full file path in the `spark:spark.driver.extraClassPath` and `spark:spark.executor.extraClassPath` properties.

    - Image 2.2.x

        To add the required library versions, use one of the following methods:

        * Set up access to the [Maven](https://maven.apache.org/index.html) repository and [set](../concepts/settings-list.md#change-properties) the `spark:spark.jars.packages` property to `org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.5.2`.

            You can set up Maven access in two ways:
            
            * In your cluster's [security group](../../vpc/concepts/security-groups.md), allow network access to the [Maven Central](https://repo.maven.apache.org/maven2/) repository.
            * Configure an [alternative Maven repository](https://maven.apache.org/guides/mini/guide-mirror-settings.html) and allow traffic to it in the cluster [security group](../../vpc/concepts/security-groups.md).

        * Download the [iceberg-spark-runtime-3.5_2.12-1.5.2.jar](https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-spark-runtime-3.5_2.12/1.5.2/iceberg-spark-runtime-3.5_2.12-1.5.2.jar) library file and grant access to it using one of the following methods:

            * Save the file to a Yandex Object Storage bucket and provide the file's URL in the `spark:spark.jars` property.
            
                The file's URL has the following format: `s3a://<bucket_name>/<file_path>`.
            
                This bucket must be specified in the cluster settings. Make sure the cluster service account has read access to the bucket.
            
            * Copy the file to all the cluster nodes manually or using [initialization scripts](../concepts/init-action.md) and provide the full file path in the `spark:spark.driver.extraClassPath` and `spark:spark.executor.extraClassPath` properties.

    {% endlist %}

You can now use Apache Iceberg™ in your Yandex Data Processing cluster.

## Apache Iceberg™ use case {#example}

This use case was tested on a Yandex Data Processing cluster version 2.0 with:

* Spark and Hadoop components installed.
* Object Storage bucket connected, and the cluster's service account having read and write permissions for this bucket.
* Access to the Maven Central repository configured.
* Component properties [configured](#settings) to enable downloading Apache Iceberg™ libraries from Maven Central.

To create an Apache Iceberg™ table and start working with it:

1. Specify the [settings for the folder](https://iceberg.apache.org/docs/latest/spark-configuration/#catalogs) that will contain the table.

    Apache Iceberg™ operates with tables at the individual folder level. Folder settings are specified at the individual folder level; you cannot specify settings for all folders at the same time.

    To configure the `sample` [Hadoop](https://iceberg.apache.org/docs/latest/spark-configuration/#catalog-configuration) folder, [set up these properties](../concepts/settings-list.md#change-properties) at the cluster or individual job level as follows:

    * `spark:spark.sql.catalog.sample` property to `org.apache.iceberg.spark.SparkCatalog`
    * `spark:spark.sql.catalog.sample.type` property to `hadoop`
    * `spark:spark.sql.catalog.sample.warehouse` property to `s3a://<bucket_name>/warehouse/`

        Table data will be stored in the bucket at the `warehouse/` path.

    For more information about the properties affecting folder settings, this [Apache Iceberg™ guide](https://iceberg.apache.org/docs/latest/spark-configuration/#catalog-configuration).

1. [Use SSH to connect](connect-ssh.md) to the Yandex Data Processing cluster master host.

1. Start a Spark SQL session:

    ```bash
    spark-sql
    ```

    You will perform all further actions within this session.

1. Create a database named `db` in the `sample` folder:

    ```sql
    CREATE DATABASE sample.db;
    ```

1. Switch to the `db` database in the `sample` folder:

    ```sql
    USE sample.db;
    ```

1. Create a two-column table named `mytable`:

    ```sql
    CREATE TABLE mytable (id bigint, data string) USING iceberg;
    ```

1. View table details:

    ```sql
    DESC FORMATTED mytable;
    ```

    Result example:

    ```sql
    id      bigint
    data    string

    # Partitioning
    Not partitioned

    # Detailed Table Information
    Name    sample.db.mytable
    Location        s3a://<bucket_name>/warehouse/db/mytable
    Provider        iceberg
    Owner   ubuntu
    Table Properties        [current-snapshot-id=none,format=iceberg/parquet]
    ```

1. Insert some entries to the table:

    ```sql
    INSERT INTO mytable VALUES (1, 'a'), (2, 'b'), (3, 'c');
    ```

1. Run a test query against the table:

    ```sql
    SELECT count(1), data FROM mytable GROUP BY data;
    ```

    Result example:

    ```sql
    1       a
    1       b
    1       c
    ```