[Yandex Cloud documentation](../../index.md) > [Yandex DataSphere](../index.md) > [Concepts](index.md) > DataSphere Notebook > Computing on Apache Spark™ clusters > Ways to use Apache Spark™ clusters

# Ways to use Apache Spark™ clusters in DataSphere

[Yandex Data Processing](../../data-proc/index.md) allows you to deploy Apache Spark™ clusters. You can use Yandex Data Processing clusters to run distributed training.

## Cluster deployment options {#types}

To work with Yandex Data Processing clusters in DataSphere, you can use the following:

* [Spark connector](data-processing-operations.md#spark-with-existing-cluster)
* [Livy session](data-processing-operations.md#livy-sessions)

If you have no existing Yandex Data Processing clusters or you need a cluster for a short time, use temporary Yandex Data Processing clusters. You can create them using the following:

* [Spark connector](temporary-data-processing-clusters.md#spark-with-temporary-cluster) (preferred)
* [Yandex Data Processing template](temporary-data-processing-clusters.md#template)

Regardless of the deployment option, all Yandex Data Processing clusters are charged based on the [Yandex Data Processing pricing policy](../../data-proc/pricing.md).

## Setting up a DataSphere project to work with Yandex Data Processing clusters {#settings}

To work with Yandex Data Processing clusters:

1. [In the project settings](../operations/projects/update.md), specify these parameters:

   * [Default folder](../../resource-manager/concepts/resources-hierarchy.md#folder) for integrating with other Yandex Cloud services. It will house a Yandex Data Processing cluster based on the current cloud [quotas](../../data-proc/concepts/limits.md). A fee for using the cluster will be debited from your cloud billing account.
   * [Service account](../../iam/concepts/users/service-accounts.md) with the `vpc.user` [role](../../iam/concepts/access-control/roles.md). DataSphere will use for this account to work with the Yandex Data Processing cluster network.
   * [Subnet](../../vpc/concepts/network.md#subnet) for DataSphere to communicate with the Yandex Data Processing cluster. The subnet must be in the same [availability zone](../../overview/concepts/geo-scope.md) as the community you created. Since the Yandex Data Processing cluster needs to access the internet, make sure to [configure a NAT gateway](../../vpc/operations/create-nat-gateway.md) in this subnet. After you specify a subnet, the time for computing resource allocation may increase.

1. Create a [service agent](../../iam/concepts/service-control.md#service-agent):

   1. To allow a service agent to operate in DataSphere, ask your cloud admin or owner to run the following command in the Yandex Cloud CLI:
   
      ```bash
      yc iam service-control enable datasphere --cloud-id <cloud_ID>
      ```
   
      Where `--cloud-id` is the [ID of the cloud](../../resource-manager/operations/cloud/get-id.md) you are going to use in the DataSphere community.

   1. Create a service account with the following roles:

      * `dataproc.agent` to use Yandex Data Processing clusters.
      * `dataproc.admin` to create clusters from Yandex Data Processing templates.
      * `vpc.user` to use the Yandex Data Processing cluster network.
      * `iam.serviceAccounts.user` to create resources in the folder on behalf of the service account.

   1. Under **Spark clusters** in the community settings, click **Add service account** and select the service account you created.
   
{% note warning %}

The Yandex Data Processing persistent cluster must have the `livy:livy.spark.deploy-mode : client` setting.

{% endnote %}

#### Useful links {#see-also}

* [Yandex Data Processing templates](data-processing-template.md)
* [Integration with Yandex Data Processing](../tutorials/data-processing-integration.md)
* [Spark connector](spark-connector.md)