[Yandex Cloud documentation](../../index.md) > [Yandex DataSphere](../index.md) > [Tutorials](index.md) > Data analytics > Using the service Yandex Managed Service for Apache Spark™

# Using Yandex Managed Service for Apache Spark™ in DataSphere

# Using Yandex Managed Service for Apache Spark™ in DataSphere

In DataSphere, you can use [Apache Spark™](../../managed-spark/index.md) clusters deployed in Yandex Managed Service for Apache Spark™. Apache Spark™ clusters are connected via [Spark connectors](../concepts/spark-connector.md). To run computations, a `SparkConnect` job is initiated in a cluster.

To set up integration with Managed Service for Apache Spark™ in DataSphere:

1. [Set up your infrastructure](#infra).
1. [Set up the DataSphere project](#project).
1. [Create a Managed Service for Apache Spark™ cluster and a Spark connector](#create-spark).
1. [Run your computations](#run-code).

If you no longer need the resources you created, [delete them](#clear-out).

## Getting started {#before-you-begin}

Before getting started, register in Yandex Cloud, set up a [community](../concepts/community.md), and link your [billing account](../../billing/concepts/billing-account.md) to it.
1. [On the DataSphere home page](https://datasphere.yandex.cloud), click **Try for free** and select an account to log in with: Yandex ID or your working account with the identity federation (SSO).
1. Select the [Yandex Identity Hub organization](../../organization/index.md) you are going to use in Yandex Cloud.
1. [Create a community](../operations/community/create.md).
1. [Link your billing account](../operations/community/link-ba.md) to the DataSphere community you are going to work in. Make sure you have a linked billing account and its [status](../../billing/concepts/billing-account-statuses.md) is `ACTIVE` or `TRIAL_ACTIVE`. If you do not have a billing account yet, create one in the DataSphere interface.

### Required paid resources {#paid-resources}

* Managed Service for Apache Spark™ cluster: computing resources of the cluster components (see [Managed Service for Apache Spark™ pricing](../../managed-spark/pricing.md)).
* NAT gateway: hourly use of the gateway and its outgoing traffic (see [Virtual Private Cloud pricing](../../vpc/pricing.md)).

## Set up your infrastructure {#infra}

### Create a folder {#create-folder}

Create a folder where your Apache Spark™ cluster will run.

{% list tabs group=instructions %}

- Management console {#console}

   1. In the [management console](https://console.yandex.cloud), select a cloud and click ![create](../../_assets/console-icons/plus.svg) **Create folder**.
   1. Name your folder, e.g., `data-folder`.
   1. Disable **Create a default network** to create your network and subnets manually.
   1. Click **Create**.

{% endlist %}

[Learn more about clouds and folders](../../resource-manager/concepts/resources-hierarchy.md).

### Create a network {#create-network}

Create a network the Apache Spark™ cluster will operate in.

{% list tabs group=instructions %}

- Management console {#console}

  1. In the [management console](https://console.yandex.cloud), navigate to `data-folder` you [created earlier](#create-folder).
  1. In the list of services, select **Virtual Private Cloud**.
  1. In the top-right corner, click **Create network**.
  1. In the **Name** field, enter a name for the network: `data-network`.

      This will automatically create three subnets in different availability zones.

  1. Click **Create network**.

{% endlist %}

### Create and set up a NAT gateway for internet access {#create-nat}

{% list tabs group=instructions %}

- Management console {#console}

   1. In `data-folder`, select **Virtual Private Cloud**.
   1. In the left-hand panel, select ![image](../../_assets/console-icons/arrows-opposite-to-dots.svg) **Gateways**.
   1. Click **Create** and set the gateway parameters:
      1. Name the gateway, e.g., `nat-for-cluster`.
      1. Select the gateway **Type**: **Egress NAT**.
      1. Click **Save**.
   1. In the left-hand panel, select ![image](../../_assets/console-icons/route.svg) **Routing tables**.
   1. Click **Create** and specify the route table parameters:
      1. Enter a name, e.g., `route-table`.
      1. Select `data-network`.
      1. Click **Add**.
      1. In the window that opens, select **Next hop** in the **Gateway** field.
      1. In the **Gateway** field, select the NAT gateway you created. The destination prefix will apply automatically.
      1. Click **Add**.
      1. Click **Create routing table**.
   1. Associate the route table with a subnet to route traffic from it through the NAT gateway:
      1. In the left-hand panel, select ![image](../../_assets/console-icons/nodes-right.svg) **Subnets**.
      1. In the row with the subnet you need, click ![image](../../_assets/console-icons/ellipsis.svg).
      1. In the menu that opens, select **Link routing table**.
      1. In the window that opens, select your route table from the list.
      1. Click **Link**.

{% endlist %}

### Create a service account for the Yandex Managed Service for Apache Spark™ cluster {#create-sa}

{% list tabs group=instructions %}

- Management console {#console}

   1. Navigate to `data-folder`.
   1. In the list of services, select **Identity and Access Management**.
   1. Click **Create service account**.
   1. Name the [service account](../../iam/concepts/users/service-accounts.md), e.g., `sa-for-spark`.
   1. Click **Add role** and assign the following [roles](../../iam/concepts/access-control/roles.md) to the service account:
      * `managed-spark.user` to use Apache Spark™ clusters.
      * `dataproc.agent` to get job information.
      * `dataproc.user` to run jobs in Apache Spark™ clusters.
      * `vpc.user` to use the Apache Spark™ cluster network.
      * `iam.serviceAccounts.user` to create resources in the folder on behalf of the service account.

   1. Click **Create**.

{% endlist %}

## Configure DataSphere {#project}

To work with Apache Spark™ clusters in DataSphere, create and set up a project.

### Create a project {#create-project}

1. Open the DataSphere [home page](https://datasphere.yandex.cloud).
1. In the left-hand panel, select ![image](../../_assets/console-icons/circles-concentric.svg) **Communities**.
1. Select the community where you want to create a project.
1. On the community page, click ![image](../../_assets/console-icons/folder-plus.svg) **Create project**.
1. In the window that opens, enter a name for the project. You can also add a description as needed.
1. Click **Create**.

### Edit the project settings {#change-settings}

1. Navigate to the **Settings** tab.
1. Under **Advanced settings**, click ![pencil](../../_assets/console-icons/pencil-to-line.svg) **Edit**.
1. Specify the parameters:
   * **Default folder**: `data-folder`.
   * **Service account**: `sa-for-spark`.
   * **Subnet**: `data-network-ru-central1-a`.

     {% note info %}
     
     If you specify a subnet in the project settings, the VM preparation during the first computation run may take longer.
     
     {% endnote %}

   * [Security groups](../../vpc/concepts/security-groups.md), if used in your organization.

1. Click **Save**.

### Edit the community settings {#change-settings-community}

To set up a connection to Apache Spark™ clusters:

1. Select the community where you have [created a project](#create-project).
1. Navigate to the **Settings** tab.
1. Under **Service agent**, click **Add service account**.
1. In the window that opens, select the service account you [created earlier](#create-sa) and click **Add**.
1. Under **Spark clusters**, click **Add service account** and select the service account you created earlier.

## Create a Managed Service for Apache Spark™ cluster and a Spark connector {#create-spark}

1. [Create a Managed Service for Apache Spark™ cluster](../../managed-spark/operations/cluster-create.md) with your preferred configuration and the following properties:

    * **Service account**: `sa-for-spark`.
    * **Network**: `data-network`.
    * **Subnet**: `data-network-ru-central1-a`.

1. Go to your [DataSphere workspace](https://datasphere.yandex.cloud).
1. In the left-hand panel, select ![image](../../_assets/console-icons/circles-concentric.svg) **Communities**.
1. Open the community you [created the project](#create-project) in and select the project you need.
1. Under **Project resources**, click ![spark-connector](../../_assets/console-icons/route.svg) **Spark connector**.
1. Click **Create connector**.
1. In the **Name** field, enter a name for your connector. The name format is as follows:

   * The name must be from 3 to 63 characters long.
   * It may contain uppercase and lowercase Latin and Cyrillic letters, numbers, hyphens, underscores, and spaces.
   * The first character must be a letter. The last character cannot be a hyphen, underscore, or space.

1. Under **Yandex Data Processing cluster**:

   1. Click the **Select cluster** tile.
   1. Select the previously created Managed Service for Apache Spark™ cluster from the list.

1. Optionally, to use an Object Storage bucket for computations, under **S3 settings**, specify the [static access key](../../iam/operations/authentication/manage-access-keys.md#create-access-key) ID and the [secret](../concepts/secrets.md) storing the secret part of the static key.
1. Under **Spark settings**, specify the SparkConnect job settings:

    * To use the standard Apache Spark™ cluster settings for computations, select **Use default settings**.
    * To manually add or update job settings, specify one or more *Key* and *Value* settings.

1. Click **Create**. You will see a page with detailed info on the connector you created.

## Run the computations {#run-code}

1. Open the DataSphere project:
   
   1. Select the project in your community or on the DataSphere [home page](https://datasphere.yandex.cloud) in the **Recent projects** tab.
   1. Click **Open project in JupyterLab** and wait for the loading to complete.
   1. Open the notebook tab.
1. In the cell, insert the code to compute, e.g.:

   ```python
   df = spark.createDataFrame([(1, "Sarah"), (2, "Maria")]).toDF(*["id", "name"])
   df.show()
   ```

1. Select **Run** → **Run Selected Cells** from the menu or press **Shift** + **Enter**.
1. In the **Notebook VM configurations** window that opens, go to the **With Yandex Data Processing cluster** tab.
1. Select the required configuration and connector.
1. Click **Select**.

   A [local PySpark](https://spark.apache.org/docs/3.5.6/api/python/reference/pyspark.sql/api/pyspark.sql.SparkSession.html) session will become available in the notebook via the `spark` variable. To run code in the notebook cells, the system will create and initiate a SparkConnect [job](../../managed-spark/concepts/index.md#job) in the Apache Spark™ cluster.

To terminate the job in the Apache Spark™ cluster, stop the notebook VM.

## Delete the resources you created {#clear-out}

Some resources are not free of charge. Delete the resources you no longer need to avoid paying for them:

1. [Managed Service for Apache Spark™ cluster](../../managed-spark/operations/cluster-delete.md).
1. [NAT gateway](../../vpc/operations/delete-nat-gateway.md#delete-nat-gateway).

{% endlist %}