[Yandex Cloud documentation](../../index.md) > [Yandex Managed Service for Apache Spark™](../index.md) > [Step-by-step guides](index.md) > Jobs > PySpark jobs

# Managing PySpark jobs

## Creating a job {#create}

{% note warning %}

Once created, the job will run automatically.

{% endnote %}

To create a job:

{% list tabs group=instructions %}

- Management console {#console}

    1. Open the [folder dashboard](https://console.yandex.cloud).
    1. Navigate to **Managed Service for Apache Spark™**.
    1. Click the name of your cluster and select the **Jobs** tab.
    1. Click **Create job**.
    1. Enter the job name.
    1. In the **Job type** field, select `PySpark`.
    1. In the **Main python file** field, specify the path to the main PY application file in the following format:

        | File location                                                | Path format                                          |
        |-----------------------------------------------------------------|------------------------------------------------------|
        | Instance file system                                       | `file:///<file_path>`                             |
        | [Object Storage](../../storage/concepts/bucket.md) bucket | `s3a://<bucket_name>/<file_path>`                  |
        | Internet                                                        | `http://<path_to_file>` or `https://<path_to_file>` |
        
        Archives in standard Linux formats, such as `zip`, `gz`, `xz`, `bz2`, etc., are supported.
        
        The cluster service account needs read access to all the files in the bucket. Step-by-step guides on how to set up access to Object Storage are provided in [Editing a bucket ACL](../../storage/operations/buckets/edit-acl.md).

    1. Optionally, specify the paths to PY files, if any.
    1. Specify job arguments.

       If an argument, variable, or property is in several space-separated parts, specify each part separately. At the same time, it is important to preserve the order in which you declare arguments, variables, and properties.
       
       The `-n 1000` argument, for instance, must be converted into two arguments, `-n` and `1000`, in that order.

    1. Optionally, specify the paths to JAR files, if any.
    1. Optionally, configure advanced settings:

        * Specify paths to the required files and archives.
        * In the **Properties** field, specify [Apache Spark™ properties](../concepts/spark-properties.md) as `key-value` pairs.
        * Specify the coordinates of included and excluded Maven packages as well as URLs of additional repositories for package search.

    1. Click **Submit job**.

- CLI {#cli}

    If you do not have the Yandex Cloud CLI yet, [install and initialize it](../../cli/quickstart.md#install).

    The folder used by default is the one specified when [creating](../../cli/operations/profile/profile-create.md) the CLI profile. To change the default folder, use the `yc config set folder-id <folder_ID>` command. You can also specify a different folder for any command using `--folder-name` or `--folder-id`. If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions.

    To create a PySpark job:

    1. See the description of the CLI command for creating a job:

        ```bash
        yc managed-spark job create-pyspark --help
        ```
    
    1. Create a job by running this command:

        ```bash
        yc managed-spark job create-pyspark \
          --cluster-id <cluster_ID> \
          --name <job_name> \
          --main-python-file-uri <path_to_main_PY_file> \
          --python-file-uris <list_of_paths_to_PY_files> \
          --jar-file-uris <list_of_paths_to_JAR_files> \
          --file-uris <list_of_paths_to_files> \
          --archive-uris <list_of_paths_to_archives> \
          --packages <list_of_package_Maven_coordinates> \
          --repositories <list_of_URLs_of_repositories_for_package_search> \
          --exclude-packages <list_of_Maven_coordinates_of_excluded_packages> \
          --properties <list_of_Apache_Spark™_properties> \
          --args <list_of_arguments> 
        ```

        Where:

        * `--cluster-id`: Cluster ID.

          You can get the cluster ID with the [list of clusters in the folder](cluster-list.md#list-clusters).

        * `--name` (optional): Job name.
        * `--main-python-file-uri`: Path to the application's main PY file.
        * `--python-file-uris`: List of paths to PY files.
        * `--jar-file-uris`: List of paths to JAR files.
        * `--file-uris`: List of paths to files.
        * `--archive-uris`: List of paths to archives.
        * `--packages`: List of Maven coordinates of packages in `groupId:artifactId:version` format.
        * `--repositories`: List of URLs of additional repositories for package search.
        * `--exclude-packages`: List of Maven coordinates of the packages to exclude, in `groupId:artifactId` format.
        * `--properties`: List of [Apache Spark™ properties](../concepts/spark-properties.md) in `key=value` format.
        * `--args`: List of application arguments.

        Available file formats:
          
          | File location                                                | Path format                                          |
          |-----------------------------------------------------------------|------------------------------------------------------|
          | Instance file system                                       | `file:///<file_path>`                             |
          | [Object Storage](../../storage/concepts/bucket.md) bucket | `s3a://<bucket_name>/<file_path>`                  |
          | Internet                                                        | `http://<path_to_file>` or `https://<path_to_file>` |
          
          Archives in standard Linux formats, such as `zip`, `gz`, `xz`, `bz2`, etc., are supported.
          
          The cluster service account needs read access to all the files in the bucket. Step-by-step guides on how to set up access to Object Storage are provided in [Editing a bucket ACL](../../storage/operations/buckets/edit-acl.md).

- gRPC API {#grpc-api}

    1. [Get an IAM token for API authentication](../api-ref/authentication.md) and put it into an environment variable:

       ```bash
       export IAM_TOKEN="<IAM_token>"
       ```

    1. Clone the [cloudapi](https://github.com/yandex-cloud/cloudapi) repository:
       
       ```bash
       cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
       ```
       
       Below, we assume that the repository contents reside in the `~/cloudapi/` directory.

    1. Call the [JobService.Create](../api-ref/grpc/Job/create.md) method, e.g., via the following [gRPCurl](https://github.com/fullstorydev/grpcurl) request:

        ```bash
        grpcurl \
            -format json \
            -import-path ~/cloudapi/ \
            -import-path ~/cloudapi/third_party/googleapis/ \
            -proto ~/cloudapi/yandex/cloud/spark/v1/job_service.proto \
            -rpc-header "Authorization: Bearer $IAM_TOKEN" \
            -d '{
                   "cluster_id": "<cluster_ID>",
                   "name": "<job_name>",
                   "pyspark_job": {
                     "args": [
                       <list_of_arguments>
                     ],
                     "jar_file_uris": [
                       <list_of_paths_to_JAR_files>
                     ],
                     "file_uris": [
                       <list_of_paths_to_files>
                     ],
                     "archive_uris": [
                       <list_of_paths_to_archives>
                     ],
                     "properties": {
                       <list_of_Apache_Spark™_properties>
                     },
                     "main_python_file_uri": "<path_to_main_PY_file>",
                     "python_file_uris": [
                       <list_of_paths_to_PY_files>
                     ],
                     "packages": [
                       <list_of_package_Maven_coordinates>
                     ],
                     "repositories": [
                       <URLs_of_repositories_for_package_search>
                     ],
                     "exclude_packages": [
                       <list_of_Maven_coordinates_of_excluded_packages>
                     ]
                   }
               }' \
            spark.api.cloud.yandex.net:443 \
            yandex.cloud.spark.v1.JobService.Create
        ```

        Where:

        * `name`: PySpark job name.
        * `spark_job`: PySpark job parameters:

            * `args`: Job arguments.
            * `jar_file_uris`: Paths to JAR files.
            * `file_uris`: Paths to files.
            * `archive_uris`: Paths to archives.
            * `properties`: [Apache Spark™ properties](../concepts/spark-properties.md) as `"key":"value"` pairs.
            * `main_python_file_uri`: Path to the main PY application file in the following format:

                | File location                                                | Path format                                          |
                |-----------------------------------------------------------------|------------------------------------------------------|
                | Instance file system                                       | `file:///<file_path>`                             |
                | [Object Storage](../../storage/concepts/bucket.md) bucket | `s3a://<bucket_name>/<file_path>`                  |
                | Internet                                                        | `http://<path_to_file>` or `https://<path_to_file>` |
                
                Archives in standard Linux formats, such as `zip`, `gz`, `xz`, `bz2`, etc., are supported.
                
                The cluster service account needs read access to all the files in the bucket. Step-by-step guides on how to set up access to Object Storage are provided in [Editing a bucket ACL](../../storage/operations/buckets/edit-acl.md).

            * `python_file_uris`: Paths to PY files.
            * `packages`: Maven coordinates of packages in `groupId:artifactId:version` format.
            * `repositories`: URLs of additional repositories for package search.
            * `exclude_packages`: Maven coordinates of the packages to exclude, in `groupId:artifactId` format.

        You can get the cluster ID with the [list of clusters in the folder](cluster-list.md#list-clusters).

    1. Check the [server response](../api-ref/grpc/Job/create.md#yandex.cloud.operation.Operation) to make sure your request was successful.

{% endlist %}

## Canceling a job {#cancel}

{% note info %}

You cannot cancel jobs with the `ERROR`, `DONE`, or `CANCELLED` status. To find out the job status, get the [list of jobs](#list) in the cluster.

{% endnote %}

{% list tabs group=instructions %}

- Management console {#console}

    1. Open the [folder dashboard](https://console.yandex.cloud).
    1. Navigate to **Managed Service for Apache Spark™**.
    1. Click the name of your cluster and select the **Jobs** tab.
    1. Click the job name.
    1. Click **Cancel** in the top-right corner of the page.
    1. In the window that opens, select **Cancel job**.

- CLI {#cli}
    
    If you do not have the Yandex Cloud CLI yet, [install and initialize it](../../cli/quickstart.md#install).
    
    The folder used by default is the one specified when [creating](../../cli/operations/profile/profile-create.md) the CLI profile. To change the default folder, use the `yc config set folder-id <folder_ID>` command. You can also specify a different folder for any command using `--folder-name` or `--folder-id`. If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions.
    
    To cancel a job, do the following:

    1. View the description of the CLI command for canceling a job:

        ```bash
        yc managed-spark job cancel --help
        ```
    
    1. Cancel a job by running this command:

        ```bash
        yc managed-spark job cancel <job_name_or_ID> \
          --cluster-id <cluster_ID>
        ```

        You can get the cluster ID with the [list of clusters in the folder](cluster-list.md#list-clusters).

        You can get the job name and ID with the [list of cluster jobs](#list).

- gRPC API {#grpc-api}

    1. [Get an IAM token for API authentication](../api-ref/authentication.md) and put it into an environment variable:

       ```bash
       export IAM_TOKEN="<IAM_token>"
       ```

    1. Clone the [cloudapi](https://github.com/yandex-cloud/cloudapi) repository:
       
       ```bash
       cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
       ```
       
       Below, we assume that the repository contents reside in the `~/cloudapi/` directory.

    1. Call the [JobService.Cancel](../api-ref/grpc/Job/cancel.md) method, e.g., via the following [gRPCurl](https://github.com/fullstorydev/grpcurl) request:

        ```bash
        grpcurl \
            -format json \
            -import-path ~/cloudapi/ \
            -import-path ~/cloudapi/third_party/googleapis/ \
            -proto ~/cloudapi/yandex/cloud/spark/v1/job_service.proto \
            -rpc-header "Authorization: Bearer $IAM_TOKEN" \
            -d '{
                   "cluster_id": "<cluster_ID>",
                   "job_id": "<job_ID>"
               }' \
            spark.api.cloud.yandex.net:443 \
            yandex.cloud.spark.v1.JobService.Cancel
        ```

        You can get the cluster ID with the [list of clusters in the folder](cluster-list.md#list-clusters), and the job ID, with the [list of cluster jobs](#list).

    1. Check the [server response](../api-ref/grpc/Job/cancel.md#yandex.cloud.operation.Operation) to make sure your request was successful.

{% endlist %}

## Getting a list of jobs {#list}

{% list tabs group=instructions %}

- Management console {#console}

    1. Open the [folder dashboard](https://console.yandex.cloud).
    1. Navigate to **Managed Service for Apache Spark™**.
    1. Click the name of your cluster and select the **Jobs** tab.

- CLI {#cli}
    
    If you do not have the Yandex Cloud CLI yet, [install and initialize it](../../cli/quickstart.md#install).
    
    The folder used by default is the one specified when [creating](../../cli/operations/profile/profile-create.md) the CLI profile. To change the default folder, use the `yc config set folder-id <folder_ID>` command. You can also specify a different folder for any command using `--folder-name` or `--folder-id`. If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions.
    
    To get a list of cluster jobs:

    1. See the description of the CLI command for getting a list of jobs:

        ```bash
        yc managed-spark job list --help
        ```
    
    1. Get the list of jobs by running this command:

        ```bash
        yc managed-spark job list \
          --cluster-id <cluster_ID>
        ```

        You can get the cluster ID with the [list of clusters in the folder](cluster-list.md#list-clusters).

- gRPC API {#grpc-api}

    1. [Get an IAM token for API authentication](../api-ref/authentication.md) and put it into an environment variable:

       ```bash
       export IAM_TOKEN="<IAM_token>"
       ```

    1. Clone the [cloudapi](https://github.com/yandex-cloud/cloudapi) repository:
       
       ```bash
       cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
       ```
       
       Below, we assume that the repository contents reside in the `~/cloudapi/` directory.

    1. Call the [JobService.List](../api-ref/grpc/Job/list.md) method, e.g., via the following [gRPCurl](https://github.com/fullstorydev/grpcurl) request:

        ```bash
        grpcurl \
            -format json \
            -import-path ~/cloudapi/ \
            -import-path ~/cloudapi/third_party/googleapis/ \
            -proto ~/cloudapi/yandex/cloud/spark/v1/job_service.proto \
            -rpc-header "Authorization: Bearer $IAM_TOKEN" \
            -d '{
                   "cluster_id": "<cluster_ID>"
               }' \
            spark.api.cloud.yandex.net:443 \
            yandex.cloud.spark.v1.JobService.List
        ```

        You can get the cluster ID with the [list of clusters in the folder](cluster-list.md#list-clusters).

    1. Check the [server response](../api-ref/grpc/Job/list.md#yandex.cloud.spark.v1.ListJobsResponse) to make sure your request was successful.

{% endlist %}

## Getting general info about a job {#get-info}

{% list tabs group=instructions %}

- Management console {#console}

    1. Open the [folder dashboard](https://console.yandex.cloud).
    1. Navigate to **Managed Service for Apache Spark™**.
    1. Click the name of your cluster and select the **Jobs** tab.
    1. Click the job name.

- CLI {#cli}
    
    If you do not have the Yandex Cloud CLI yet, [install and initialize it](../../cli/quickstart.md#install).
    
    The folder used by default is the one specified when [creating](../../cli/operations/profile/profile-create.md) the CLI profile. To change the default folder, use the `yc config set folder-id <folder_ID>` command. You can also specify a different folder for any command using `--folder-name` or `--folder-id`. If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions.
    
    To get information about a job:

    1. View the description of the CLI command for getting information about a job:

        ```bash
        yc managed-spark job get --help
        ```
    
    1. Get information about the job by running this command:

        ```bash
        yc managed-spark job get <job_ID> \
          --cluster-id <cluster_ID>
        ```

        You can get the cluster ID with the [list of clusters in the folder](cluster-list.md#list-clusters).

        You can get the job ID with the [list of cluster jobs](#list).

- gRPC API {#grpc-api}

    1. [Get an IAM token for API authentication](../api-ref/authentication.md) and put it into an environment variable:

       ```bash
       export IAM_TOKEN="<IAM_token>"
       ```

    1. Clone the [cloudapi](https://github.com/yandex-cloud/cloudapi) repository:
       
       ```bash
       cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
       ```
       
       Below, we assume that the repository contents reside in the `~/cloudapi/` directory.

    1. Call the [JobService.Get](../api-ref/grpc/Job/get.md) method, e.g., via the following [gRPCurl](https://github.com/fullstorydev/grpcurl) request:

        ```bash
        grpcurl \
            -format json \
            -import-path ~/cloudapi/ \
            -import-path ~/cloudapi/third_party/googleapis/ \
            -proto ~/cloudapi/yandex/cloud/spark/v1/job_service.proto \
            -rpc-header "Authorization: Bearer $IAM_TOKEN" \
            -d '{
                   "cluster_id": "<cluster_ID>",
                   "job_id": "<job_ID>"
               }' \
            spark.api.cloud.yandex.net:443 \
            yandex.cloud.spark.v1.JobService.Get
        ```

        You can get the cluster ID with the [list of clusters in the folder](cluster-list.md#list-clusters), and the job ID, with the [list of cluster jobs](#list).

    1. Check the [server response](../api-ref/grpc/Job/get.md#yandex.cloud.spark.v1.Job) to make sure your request was successful.

{% endlist %}

## Getting job execution logs {#get-logs}

{% note warning %}

To get job execution logs, enable logging in your cluster while [creating](cluster-create.md) it.

{% endnote %}

{% list tabs group=instructions %}

- Management console {#console}

    1. Open the [folder dashboard](https://console.yandex.cloud).
    1. Navigate to **Managed Service for Apache Spark™**.
    1. Click the name of your cluster and select the **Jobs** tab.
    1. Click the job name.
    1. In the **Output logs** field, click the link.

- CLI {#cli}
    
    If you do not have the Yandex Cloud CLI yet, [install and initialize it](../../cli/quickstart.md#install).
    
    The folder used by default is the one specified when [creating](../../cli/operations/profile/profile-create.md) the CLI profile. To change the default folder, use the `yc config set folder-id <folder_ID>` command. You can also specify a different folder for any command using `--folder-name` or `--folder-id`. If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions.
    
    To get job execution logs:

    1. See the description of the CLI command for getting job logs:

        ```bash
        yc managed-spark job log --help
        ```
    
    1. Get job logs by running this command:

        ```bash
        yc managed-spark job log <job_ID> \
          --cluster-id <cluster_ID>
        ```

        You can get the cluster ID with the [list of clusters in the folder](cluster-list.md#list-clusters).

        You can get the job ID with the [list of cluster jobs](#list).

        To get logs for multiple jobs, list their IDs separated by spaces, e.g.:
        
        ```bash
        yc managed-spark job log c9q9veov4uql******** c9qu8uftedte******** \
          --cluster-id c9q8ml85r1oh********
        ```

- gRPC API {#grpc-api}

    1. [Get an IAM token for API authentication](../api-ref/authentication.md) and put it into an environment variable:

       ```bash
       export IAM_TOKEN="<IAM_token>"
       ```

    1. Clone the [cloudapi](https://github.com/yandex-cloud/cloudapi) repository:
       
       ```bash
       cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
       ```
       
       Below, we assume that the repository contents reside in the `~/cloudapi/` directory.

    1. Call the [JobService.ListLog](../api-ref/grpc/Job/listLog.md) method, e.g., via the following [gRPCurl](https://github.com/fullstorydev/grpcurl) request:

        ```bash
        grpcurl \
            -format json \
            -import-path ~/cloudapi/ \
            -import-path ~/cloudapi/third_party/googleapis/ \
            -proto ~/cloudapi/yandex/cloud/spark/v1/job_service.proto \
            -rpc-header "Authorization: Bearer $IAM_TOKEN" \
            -d '{
                   "cluster_id": "<cluster_ID>",
                   "job_id": "<job_ID>"
               }' \
            spark.api.cloud.yandex.net:443 \
            yandex.cloud.spark.v1.JobService.ListLog
        ```

        You can request the cluster ID with the [list of clusters in the folder](cluster-list.md#list-clusters), and the job ID, with the [list of cluster jobs](#list).

    1. Check the [server response](../api-ref/grpc/Job/listLog.md#yandex.cloud.spark.v1.ListJobLogResponse) to make sure your request was successful.

{% endlist %}