[Yandex Cloud documentation](../../index.md) > [Tutorials](../index.md) > [Building a data platform](index.md) > Working with jobs Yandex Data Processing > Working with MapReduce jobs

# Working with MapReduce jobs

[MapReduce](http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html) is a parallel processing tool for large datasets (a few dozen TB or more) on clusters in the Hadoop ecosystem. It allows you to work with data in different formats. Job input and output are stored in Yandex Object Storage. MapReduce uses a number of libraries, while [Apache Bigtop](https://github.com/apache/bigtop) determines the path to them.

In this article, we use a simple example to show how MapReduce works in Yandex Data Processing. We will use MapReduce to compute the population of the world's 500 largest cities based on the cities dataset.

To run MapReduce on Hadoop, we use the Streaming interface. At the same time, the data preprocessing (map) and the final output computation (reduce) stages use programs that read data from a standard program input (`stdin`) and write their result to a standard output (`stdout`).

To execute a MapReduce job:

1. [Set up your infrastructure](#infra).
1. [Create a MapReduce job](#create-job).

If you no longer need the resources you created, [delete them](#clear-out).


## Getting started {#before-you-begin}

Sign up for Yandex Cloud and create a [billing account](../../billing/concepts/billing-account.md):
1. Navigate to the [management console](https://console.yandex.cloud) and log in to Yandex Cloud or create a new account.
1. On the **[Yandex Cloud Billing](https://center.yandex.cloud/billing/accounts)** page, make sure you have a billing account linked and it has the `ACTIVE` or `TRIAL_ACTIVE` [status](../../billing/concepts/billing-account-statuses.md). If you do not have a billing account, [create one](../../billing/quickstart/index.md) and [link](../../billing/operations/pin-cloud.md) a cloud to it.

If you have an active billing account, you can create or select a [folder](../../resource-manager/concepts/resources-hierarchy.md#folder) for your infrastructure on the [cloud page](https://console.yandex.cloud/cloud).

[Learn more about clouds and folders here](../../resource-manager/concepts/resources-hierarchy.md).


### Required paid resources {#paid-resources}

* Yandex Data Processing cluster: use of computing resources with a Yandex Data Processing markup, use of network drives, retrieval and storage of logs, amount of outgoing traffic (see [Yandex Data Processing pricing](../../data-proc/pricing.md)).
* Public IP addresses if public access is enabled for cluster hosts (see [Yandex Virtual Private Cloud pricing](../../vpc/pricing.md)).
* Yandex Object Storage buckets: use of storage, data operations (see [Object Storage pricing](../../storage/pricing.md)).


## Set up your infrastructure {#infra}

1. [Create a service account](../../iam/operations/sa/create.md) with the `dataproc.agent` and `dataproc.provisioner` roles.

1. In Object Storage, [create buckets](../../storage/operations/buckets/create.md) and [configure access](../../storage/operations/buckets/edit-acl.md) to them:
   
   1. Create a bucket for the input data and grant the `READ` permission for this bucket to the cluster service account.
   1. Create a bucket for the processing output and grant the cluster service account `READ and WRITE` permissions for this bucket.

1. [Create a Yandex Data Processing cluster](../../data-proc/operations/cluster-create.md) with the following settings:

    * **Environment**: `PRODUCTION`.
    * **Services**:
        * `HDFS`
        * `MAPREDUCE`
        * `YARN`
    * **Service account**: Select the service account you created earlier.
    * **Bucket name**: Select a bucket for the processing results.

## Create a MapReduce job {#create-job}

1. [Download](http://download.geonames.org/export/dump/cities500.zip) an archived CSV file with the cities dataset and [upload it to the input data bucket](../../storage/operations/objects/upload.md).
1. Upload Python files to the input data bucket: `mapper.py`, which contains the code for data preprocessing (map stage), and `reducer.py`, which contains the code for the final computations (reduce stage):

    `mapper.py`

    ```python
    #!/usr/bin/python
    import sys
    
    population = sum(int(line.split('\t')[14]) for line in sys.stdin)
    print(population)
    ```

    `reducer.py`

    ```python
    #!/usr/bin/python
    import sys
    
    population = sum(int(value) for value in sys.stdin)
    print(population)
    ```

1. [Create a MapReduce job](../../data-proc/operations/jobs-mapreduce.md#create) with the following parameters:

    * **Main class**: `org.apache.hadoop.streaming.HadoopStreaming`
    * **Arguments**:
       * `-mapper`
       * `mapper.py`
       * `-reducer`
       * `reducer.py`
       * `-numReduceTasks`
       * `1`
       * `-input`
       * `s3a://<input_data_bucket_name>/cities500.txt`
       * `-output`
       * `s3a://<output_bucket_name>/<output_directory>`
    * **Files**:
       * `s3a://<input_data_bucket_name>/mapper.py`
       * `s3a://<input_data_bucket_name>/reducer.py`
    * **Properties**:
       * `mapreduce.job.maps: 6`
       * `yarn.app.mapreduce.am.resource.mb: 2048`
       * `yarn.app.mapreduce.am.command-opts: -Xmx2048m`

1. Wait for the [job status](../../data-proc/operations/jobs-mapreduce.md#get-info) to change to `Done`.

1. [Download](../../storage/operations/objects/download.md) the file with the result from the bucket and review it:

    `part-00000`

    ```text
    3157107417
    ```

{% note info %}

You can view the job logs and search data in them using [Yandex Cloud Logging](../../logging/index.md). For more information, see [Working with logs](../../data-proc/operations/logging.md).

{% endnote %}

## Delete the resources you created {#clear-out}

Some resources are not free of charge. To avoid paying for them, delete the resources you no longer need:

1. [Delete the cluster](../../data-proc/operations/cluster-delete.md).
1. [Delete the buckets](../../storage/operations/buckets/delete.md).
1. [Delete the service account](../../iam/operations/sa/delete.md).