[Yandex Cloud documentation](../../index.md) > [Tutorials](../index.md) > [Machine learning and artificial intelligence](index.md) > Development with DataSphere > DataSphere integration with Yandex Data Processing

# Yandex DataSphere integration with Yandex Data Processing

# Integration with Yandex Data Processing

In Yandex DataSphere projects, you can use Apache Spark™ clusters deployed in Yandex Managed Service for Apache Spark™. To set up integration with Yandex Data Processing in DataSphere:

1. [Set up your infrastructure](#infra).
1. [Set up the DataSphere project](#project).
1. [Create a bucket](#create-bucket).
1. [Create a Yandex Data Processing cluster](#create-cluster).
1. [Run your computations](#run-code).

If you no longer need the resources you created, [delete them](#clear-out).

## Getting started {#before-you-begin}

Before getting started, register in Yandex Cloud, set up a [community](../../datasphere/concepts/community.md), and link your [billing account](../../billing/concepts/billing-account.md) to it.
1. [On the DataSphere home page](https://datasphere.yandex.cloud), click **Try for free** and select an account to log in with: Yandex ID or your working account with the identity federation (SSO).
1. Select the [Yandex Identity Hub organization](../../organization/index.md) you are going to use in Yandex Cloud.
1. [Create a community](../../datasphere/operations/community/create.md).
1. [Link your billing account](../../datasphere/operations/community/link-ba.md) to the DataSphere community you are going to work in. Make sure you have a linked billing account and its [status](../../billing/concepts/billing-account-statuses.md) is `ACTIVE` or `TRIAL_ACTIVE`. If you do not have a billing account yet, create one in the DataSphere interface.

Log in to the Yandex Cloud [management console](https://console.yandex.cloud) and select the organization you use to access DataSphere. On the [**Yandex Cloud Billing**](https://center.yandex.cloud/billing/accounts) page, make sure you have a billing account linked.

If you have an active billing account, you can go to the [cloud page](https://console.yandex.cloud/cloud) to create or select a folder to run your infrastructure.

{% note info %}

If you are using an [identity federation](../../organization/concepts/add-federation.md) to work with Yandex Cloud, you might not have access to billing details. In this case, contact your Yandex Cloud organization administrator.

{% endnote %}

### Required paid resources {#paid-resources}

* Yandex Data Processing cluster: use of computing resources with a Yandex Data Processing markup, use of network drives, retrieval and storage of logs, volume of outgoing traffic (see [Yandex Data Processing pricing](../../data-proc/pricing.md)).
* NAT gateway: hourly use of the gateway and its outgoing traffic (see [Virtual Private Cloud pricing](../../vpc/pricing.md)).
* Yandex Object Storage bucket: use of storage, data operations (see [Object Storage pricing](../../storage/pricing.md)).

## Set up your infrastructure {#infra}

### Create a folder and network {#create-folder}

Create a folder where your Yandex Data Processing cluster will run.

{% list tabs group=instructions %}

- Management console {#console}

   1. In the [management console](https://console.yandex.cloud), select a cloud and click ![create](../../_assets/console-icons/plus.svg)**Create folder**.
   1. Name your folder, e.g., `data-folder`.
   1. Disable **Create a default network** to create a network and subnet manually.
   1. Click **Create**.

{% endlist %}

[Learn more about clouds and folders](../../resource-manager/concepts/resources-hierarchy.md).

#### Create a network {#create-network}

Create a network the Yandex Data Processing cluster will operate in.

{% list tabs group=instructions %}

- Management console {#console}

  1. In the [management console](https://console.yandex.cloud), navigate to `data-folder` you [created earlier](#create-folder).
  1. In the list of services, select **Virtual Private Cloud**.
  1. In the top-right corner, click **Create network**.
  1. In the **Name** field, enter a name for the network: `data-network`.
  1. Disable **Create subnets**.
  1. Click **Create network**.

{% endlist %}

#### Create a subnet {#create-subnet}

{% list tabs group=instructions %}

- Management console {#console}

    1. In `data-folder`, [navigate](../../console/operations/select-service.md#select-service) to **Virtual Private Cloud**.
    1. Select the `data-network` cloud network.
    1. Click **Create subnet**.
    1. Enter `data-subnet` as the subnet name.
    1. Select the `ru-central1-a` [availability zone](../../overview/concepts/geo-scope.md).
    1. Enter the subnet **CIDR**, e.g., `10.1.1.0/24`.
    1. Click **Create subnet**.

{% endlist %}

#### Create an egress NAT gateway {#create-nat}

{% list tabs group=instructions %}

- Management console {#console}

   1. In `data-folder`, [navigate](../../console/operations/select-service.md#select-service) to **Virtual Private Cloud**.
   1. In the left-hand panel, select ![image](../../_assets/console-icons/arrows-opposite-to-dots.svg) **Gateways**.
   1. Click **Create** and set the gateway parameters:
      * Name the gateway, e.g., `nat-for-cluster`.
      * Select the gateway **Type**: **Egress NAT**.
      * Click **Save**.
   1. In the left-hand panel, select ![image](../../_assets/console-icons/route.svg) **Routing tables**.
   1. Click **Create** and specify the route table parameters:
      1. Enter a name, e.g., `route-table`.
      1. Select `data-network`.
      1. Click **Add**.
         * In the window that opens, select **Gateway** in the **Next hop** field.
         * In the **Gateway** field, select the NAT gateway you created. The destination prefix will apply automatically.
         * Click **Add**.
   1. Click **Create routing table**.

  Next, associate the route table with the `data-subnet` subnet to route traffic from it through the NAT gateway:

   1. In the left-hand panel, select ![image](../../_assets/console-icons/nodes-right.svg) **Subnets**.
   1. In the `data-subnet` row, click ![image](../../_assets/console-icons/ellipsis.svg).
   1. In the menu that opens, select **Link routing table**.
   1. In the window that opens, select your route table from the list.
   1. Click **Link**.

{% endlist %}

### Create a service account for the Yandex Data Processing cluster {#create-sa}

{% list tabs group=instructions %}

- Management console {#console}

   1. Navigate to `data-folder`.
   1. [Navigate](../../console/operations/select-service.md#select-service) to **Identity and Access Management**.
   1. Click **Create service account**.
   1. Name the [service account](../../iam/concepts/users/service-accounts.md), e.g., `sa-for-data-proc`.
   1. Click **Add role** and assign the following [roles](../../iam/concepts/access-control/roles.md) to the service account:
      * `dataproc.agent` to create and use Yandex Data Processing clusters.
      * `dataproc.provisioner` to enable [subcluster autoscaling](../../data-proc/concepts/autoscaling.md).
      * `dataproc.user` to access Yandex Data Processing clusters as a [service agent](../../iam/concepts/service-control.md#service-agent).
      * `vpc.user` to use the Yandex Data Processing cluster network.
      * `iam.serviceAccounts.user` to create resources in the folder on behalf of the service account.

   1. Click **Create**.

{% endlist %}

### Create an SSH key pair {#ssh}

To ensure a safe connection to the Yandex Data Processing cluster hosts, you will need SSH keys. You may skip this step if you have already generated your SSH keys.

{% cut "How to generate an SSH key pair" %}

{% list tabs group=operating_system %}

- Linux/macOS {#linux-macos}

  1. Open the terminal.
  1. Use the `ssh-keygen` command to create a new key:
  
      ```bash
      ssh-keygen -t ed25519 -C "<optional_comment>"
      ```
  
      You can specify an empty string in the `-C` parameter to avoid adding a comment, or you may not specify the `-C` parameter at all: in this case, a default comment will be added.
  
      After running this command, you will be prompted to specify the name and path to the key files, as well as enter the password for the private key. If you only specify the name, the key pair will be created in the current directory. The public key will be saved in a file with the `.pub` extension, while the private key, in a file without extension.
  
      By default, the command prompts you to save the key under the `id_ed25519` name in the following directory: `/home/<username>/.ssh`. If there is already an SSH key named `id_ed25519` in this directory, you may accidentally overwrite it and lose access to the resources it is used in. Therefore, you may want to use unique names for all SSH keys.

- Windows 10/11 {#windows}

  If you do not have [OpenSSH](https://en.wikipedia.org/wiki/OpenSSH) installed yet, follow this [guide](https://learn.microsoft.com/en-us/windows-server/administration/openssh/openssh_install_firstuse?tabs=gui) to install it.
  
  1. Run `cmd.exe` or `powershell.exe` (make sure to update PowerShell before doing so).
  1. Use the `ssh-keygen` command to create a new key:
  
      ```shell
      ssh-keygen -t ed25519 -C "<optional_comment>"
      ```
  
      You can specify an empty string in the `-C` parameter to avoid adding a comment, or you may not specify the `-C` parameter at all: in this case, a default comment will be added.
  
      After running this command, you will be prompted to specify the name and path to the key files, as well as enter the password for the private key. If you only specify the name, the key pair will be created in the current directory. The public key will be saved in a file with the `.pub` extension, while the private key, in a file without extension.
  
      By default, the command prompts you to save the key under the `id_ed25519` name in the following folder: `C:\Users\<username>/.ssh`. If there is already an SSH key named `id_ed25519` in this directory, you may accidentally overwrite it and lose access to the resources it is used in. Therefore, you may want to use unique names for all SSH keys.

- Windows 7/8 {#windows7-8}

  Create keys using the PuTTY app:
  
  1. [Download](https://www.putty.org) and install PuTTY.
  1. Add the folder with PuTTY to the `PATH` variable:
  
      1. Click **Start** and type **Change system environment variables** in the Windows search bar.
      1. Click **Environment Variables...** at the bottom right.
      1. In the window that opens, find the `PATH` parameter and click **Edit**.
      1. Add your folder path to the list.
      1. Click **OK**.
  
  1. Launch the PuTTYgen app.
  1. Select **EdDSA** as the pair type to generate. Click **Generate** and move the cursor in the field above it until key creation is complete.
  
      ![ssh_generate_key](../../_assets/compute/ssh-putty/ssh_generate_key.png)
  
  1. In **Key passphrase**, enter a strong password. Enter it again in the field below.
  1. Click **Save private key** and save the private key. Do not share its key phrase with anyone.
  1. Click **Save public key** and save the public key to a file named `<key_name>.pub`.

{% endlist %}

{% note warning %}

Store your private key securely, as you will not be able to connect to the VM without it.

{% endnote %}

{% endcut %}

## Configure DataSphere {#project}

To work with Yandex Data Processing clusters in DataSphere, create and set up a project.

### Create a project {#create-project}

1. Open the DataSphere [home page](https://datasphere.yandex.cloud).
1. In the left-hand panel, select ![image](../../_assets/console-icons/circles-concentric.svg) **Communities**.
1. Select the community where you want to create a project.
1. On the community page, click ![image](../../_assets/console-icons/folder-plus.svg) **Create project**.
1. In the window that opens, enter a name for the project. You can also add a description as needed.
1. Click **Create**.

### Edit the project settings {#change-settings}

1. Navigate to the **Settings** tab.
1. Under **Advanced settings**, click **![pencil](../../_assets/console-icons/pencil-to-line.svg) Edit**.
1. Specify the parameters:
   * **Default folder**: `data-folder`.
   * **Service account**: `sa-for-data-proc`.
   * **Subnet**: `data-subnet`.
   * [Security groups](../../vpc/concepts/security-groups.md), if used in your organization.

1. Click **Save**.

### Edit the community settings {#change-settings-community}

To set up a connection to Yandex Data Processing clusters:

1. Select the community you [created the project](#create-project) in.
1. Navigate to the **Settings** tab.
1. Under **Service agent**, click **Add service account**.
1. In the window that opens, select the service account you [created earlier](#create-sa) and click **Add**.
1. Under **Spark clusters**, click **Add service account** and select the service account you created earlier.

## Create a bucket {#create-bucket}

{% list tabs group=instructions %}

- Management console {#console}

  1. In the [management console](https://console.yandex.cloud), select the folder you want to create a bucket in.
  1. [Navigate](../../console/operations/select-service.md#select-service) to **Object Storage**.
  1. Click **Create bucket**.
  1. In the ** Name** field, enter a name for the bucket.
  1. In the **Read objects**, **Read object list**, and **Read settings** fields, select **With authorization**.
  1. Click **Create bucket**.

{% endlist %}

## Create a Yandex Data Processing cluster {#create-cluster}

Before creating a cluster, make sure that your cloud has enough total SSD space (200 GB is allocated for a new cloud by default).

You can view your current resources under [Quotas](https://console.yandex.cloud/cloud?section=quotas) in the management console.

{% list tabs group=instructions %}

- Management console {#console}

   1. In the [management console](https://console.yandex.cloud), select the folder where you want to create a cluster.
   1. Click **Create resource** and select **Yandex Data Processing cluster** from the drop-down list.
   1. Enter a name for the cluster in the **Cluster name** field. It must be unique within the folder.
   1. In the **Environment** field, select `PRODUCTION`.
   1. In the **Version** field, select `2.1`.
   1. In the **Services** field, select `LIVY`, `SPARK`, `YARN`, and `HDFS`.
   1. Enter the public part of your SSH key in the **SSH key** field.
   1. In the **Service account** field, select `sa-for-data-proc`.
   1. In the **Availability zone** field, select `ru-central1-a`.
   1. In the **Properties** field, set up cluster integration with DataSphere:

      ```text
      livy:livy.spark.deploy-mode : client
      ```

      If necessary, set the properties of Hadoop and its components, for example:

      ```text
      hdfs:dfs.replication : 2
      hdfs:dfs.blocksize : 1073741824
      spark:spark.driver.cores : 1
      ```

      {% cut "Available properties from the official documentation for the components" %}

      * [Hadoop](https://hadoop.apache.org/docs/r3.3.2/hadoop-project-dist/hadoop-common/core-default.xml)
      * [HDFS](https://hadoop.apache.org/docs/r3.3.2/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml)
      * [Spark](https://archive.apache.org/dist/spark/docs/3.3.2/configuration.html#available-properties)
      * [YARN](https://hadoop.apache.org/docs/r3.3.2/hadoop-yarn/hadoop-yarn-common/yarn-default.xml)

      {% endcut %}

   1. Select the created bucket in the **Bucket name** field.
   1. Select `data-network`.
   1. Enable **UI Proxy** to access the [web interfaces of Yandex Data Processing components](../../data-proc/concepts/interfaces.md).
   1. Configure subclusters: no more than one main subcluster with a **Master** host and subclusters for data storage or computing.

      {% note info %}

      To run computations on clusters, make sure you have at least one `Compute` or `Data` subcluster.

      {% endnote %}

      The `Compute` and `Data` subcluster roles are different: you can deploy data storage components on `Data` subclusters and data processing components, on `Compute` subclusters. The `Compute` subcluster storage is only used to temporarily store processed files.
   1. For each subcluster, you can configure:
      * Number of hosts.
      * [Host class](../../data-proc/concepts/instance-types.md), i.e., the platform and computing resources available to the host.
      * Storage size and type.
      * Subnet of the network where the cluster is located.
   1. For `Compute` subclusters, you can specify the [autoscaling](../../data-proc/concepts/autoscaling.md) parameters.
   1. When you have set up all the subclusters, click **Create cluster**.

{% endlist %}

Yandex Data Processing will run the cluster create operation. After the cluster status changes to **Running**, you can [connect](../../data-proc/operations/connect-ssh.md) to any active subcluster using the specified SSH key.

The Yandex Data Processing cluster you created will be added to your DataSphere project under **Project resources** ⟶ **Yandex Data&nbsp;Processing** ⟶ **Available clusters**.

## Run your computations on the cluster {#run-code}

1. Open the DataSphere project:
   
   1. Select the project in your community or on the DataSphere [home page](https://datasphere.yandex.cloud) in the **Recent projects** tab.
   1. Click **Open project in JupyterLab** and wait for the loading to complete.
   1. Open the notebook tab.
1. In the cell, insert the code to compute, e.g.:

   ```python
   #!spark --cluster <cluster_name>
   import random

   def inside(p):
       x, y = random.random(), random.random()
       return x*x + y*y < 1
   
   NUM_SAMPLES = 1_000_000
   
   count = sc.parallelize(range(0, NUM_SAMPLES)) \
       .filter(inside).count()
   print("Pi is roughly %f" % (4.0 * count / NUM_SAMPLES))
   ```

   Where `#!spark --cluster <cluster_name>` is a required system command to run computations on a cluster.

1. Create another cell and paste into it the code to write data to S3 stating the bucket name:

   ```python
   #!spark --cluster <cluster_name>
   data = [[1, "tiger"], [2, "lion"], [3, "snow leopard"]]
   df = spark.createDataFrame(data, schema="id LONG, name STRING")
   df.repartition(1).write.option("header", True).mode("overwrite").csv("s3a://<bucket_name>/test")
   ```

1. Run all cells by selecting **Run** ⟶ **Run All Cells**.
1. In the **Notebook VM configurations** window that opens, select a VM configuration and click **Select**.

   Wait for computations to run. While they are in progress, logs will be displayed under the cell.

The file will appear in the bucket's `test` folder. To view bucket contents in the JupyterLab interface, create and activate an [S3 connector](../../datasphere/operations/data/s3-connectors.md) in your project.

{% note info %}

To get more than 100 MB of Yandex Data Processing cluster data, use an [S3 connector](../../datasphere/operations/data/s3-connectors.md).

{% endnote %}

To learn more about running computations on Yandex Data Processing clusters in DataSphere, see [this concept](../../datasphere/concepts/data-processing.md#existing-clusters).

## Delete the resources you created {#clear-out}

{% note warning %}

As a user of a cluster deployed in Yandex Data Processing, you manage its lifecycle yourself. The cluster will run, and fees will be [charged](../../data-proc/pricing.md), until you shut it down.

{% endnote %}

Some resources are not free of charge. Delete the resources you no longer need to avoid paying for them:

* [Objects](../../storage/operations/objects/delete-all.md) from the bucket
* [Bucket](../../storage/operations/buckets/delete.md)
* [Yandex Data Processing cluster](../../data-proc/operations/cluster-delete.md)
* [DataSphere project](../../datasphere/operations/projects/delete.md)
* [Subnet](../../vpc/operations/subnet-delete.md)
* [Route table](../../vpc/operations/delete-route-table.md)
* [NAT gateway](../../vpc/operations/delete-nat-gateway.md)
* [Network](../../vpc/operations/network-delete.md)
* [Service account](../../iam/operations/sa/delete.md)

#### See also {#see-also}

[Temporary Yandex Data Processing clusters deployed in DataSphere](../../datasphere/concepts/data-processing-template.md)