[Yandex Cloud documentation](../../index.md) > [Yandex Data Processing](../index.md) > [Tutorials](index.md) > Working with jobs > Working with Spark jobs

# Working with Spark jobs

[Apache Spark](https://spark.apache.org/) is a distributed processing framework for unstructured and semi-structured data and a part of the Hadoop project ecosystem.

In this section, we provide a simple example that demonstrates how to use the Spark interface for Scala and Java in Yandex Data Processing. In the example, we use Spark to count the number of times each word appears in a short text.

## Getting started {#before-you-begin}

1. [Create a service account](../../iam/operations/sa/create.md) with the `dataproc.agent` and `dataproc.provisioner` roles.

1. In Object Storage, [create buckets](../../storage/operations/buckets/create.md) and [configure access](../../storage/operations/buckets/edit-acl.md) to them:
   
   1. Create a bucket for the input data and grant the `READ` permission for this bucket to the cluster service account.
   1. Create a bucket for the processing output and grant the cluster service account `READ and WRITE` permissions for this bucket.

1. [Create a Yandex Data Processing cluster](../operations/cluster-create.md) with the following settings:

    * **Environment**: `PRODUCTION`.
    * **Services**:
        * `HDFS`
        * `SPARK`
        * `YARN`
    * **Service account**: Select the service account you created earlier.
    * **Bucket name**: Select a bucket for the processing results.

## Create a Spark job {#create-job}

1. Upload a file for processing:
   
   1. Copy and save the following to a file named `text.txt`:
   
       {% cut "text.txt" %}
   
       ```text
       she sells sea shells on the sea shore
       the shells that she sells are sea shells I am sure
       so if she sells sea shells on the sea shore
       I am sure that the shells are sea shore shells
       ```
   
       {% endcut %}
   
   1. [Upload](../../storage/operations/objects/upload.md) the `text.txt` file to the source data bucket.

1. Download the `spark-app_2.11-0.1.0-SNAPSHOT.jar` file containing the Scala code of the [word_count.scala](https://storage.yandexcloud.net/examples/scala-spark/word_count.scala) analysis program and upload it to the input data bucket:

    {% cut "word_count.scala" %}

    ```scala
    package com.yandex.cloud.dataproc.scala

    import org.apache.spark.{SparkConf, SparkContext}


    object Main {
        def main(args: Array[String]) {
            if (args.length != 2){ // check number of args
                System.err.println("Usage spark-app.jar <input_directory> <output_directory>");
                System.exit(-1);
            }


            val inDir = args(0); //input URI
            val outDir = args(1); //output URI

            val conf = new SparkConf().setAppName("Word count - Scala App")
            val sc = new SparkContext(conf)

            val text_file = sc.textFile(inDir)
            val counts = text_file.flatMap(line => line.split(" "))
            .map(word => (word, 1))
            .reduceByKey(_ + _)

            val defaultFS = sc.hadoopConfiguration.get("fs.defaultFS")

            if (outDir.toLowerCase().startsWith("s3a://")) {
                counts.saveAsTextFile(outDir)
            } else {
                counts.saveAsTextFile(defaultFS + "/" + outDir)
            }

            sc.stop()
        }
    }
    ```

    {% endcut %}

    For more information about building an application written in Scala for Spark, see [Using Spark Submit](run-spark-job.md#spark-submit).

1. [Create a Spark job](../operations/jobs-spark.md#create) with the following parameters:

    * **Main jar**: `s3a://<input_data_bucket_name>/spark-app_2.11-0.1.0-SNAPSHOT.jar`
    * **Main class**: `com.yandex.cloud.dataproc.scala.Main`
    * **Arguments**:
        * `s3a://<input_data_bucket_name>/text.txt`
        * `s3a://<output_bucket_name>/<output_directory>`

1. Wait for the [job status](../operations/jobs-spark.md#get-info) to change to `Done`.

1. [Download](../../storage/operations/objects/download.md) the files with the results from the bucket and review them:

    {% cut "part-00000" %}

    ```text
    (are,2)
    (am,2)
    (she,3)
    (so,1)
    ```

    {% endcut %}

    {% cut "part-00001" %}

    ```text
    (shore,3)
    (if,1)
    (that,2)
    (on,2)
    (shells,6)
    (I,2)
    (sure,2)
    (sea,6)
    (the,4)
    (sells,3)
    ```

    {% endcut %}

{% note info %}

You can view the job logs and search data in them using [Yandex Cloud Logging](../../logging/index.md). For more information, see [Working with logs](../operations/logging.md).

{% endnote %}

## Delete the resources you created {#clear-out}

Some resources are not free of charge. To avoid paying for them, delete the resources you no longer need:

1. [Delete the cluster](../operations/cluster-delete.md).
1. [Delete the buckets](../../storage/operations/buckets/delete.md).
1. [Delete the service account](../../iam/operations/sa/delete.md).