# Monitoring the state of Spark applications

To assess the performance of Spark applications in a Yandex Data Processing cluster, you can check the following:

* [Application list](#list)
* [Application logs](#logs)
* [Application queue](#queue)
* [Application details](#info)
* [Resources allocated to the application](#resources)
* [Persisted RDDs](#tables)
* [List of SQL queries and their plans](#sql)

{% note info %}

Make sure the cluster has the [component web interfaces](../concepts/interfaces.md) enabled. If not, [enable them](connect-interfaces.md#ui-proxy-enable).

{% endnote %}

## Checking the application list {#list}

1. Open the [folder dashboard](https://console.yandex.cloud).
1. Navigate to **Yandex Data Processing**.
1. Click the cluster name.
1. Under **UI Proxy**, select `YARN Resource Manager Web UI`.

It shows information about all running and completed applications.

## Checking application logs {#logs}

1. Open the [folder dashboard](https://console.yandex.cloud).
1. Navigate to **Yandex Data Processing**.
1. Click the cluster name.
1. Under **UI Proxy**, select `YARN Resource Manager Web UI`.
1. Find the application you need and click its ID in the **ID** column.

    This will open a window with info on the application performance and a table listing its run attempts.

1. Click the link next to the attempt in question in the **Logs** column.

## Checking the application queue {#queue}

1. Open the [folder dashboard](https://console.yandex.cloud).
1. Navigate to **Yandex Data Processing**.
1. Click the cluster name.
1. Under **UI Proxy**, select `YARN Resource Manager Web UI`.
1. In the left-hand menu, navigate to **Scheduler**.

The **Application Queues** section shows the queue of applications and the resources they use.

## Checking application details {#info}

{% list tabs %}

* YARN Resource Manager Web UI

    1. Open the [folder dashboard](https://console.yandex.cloud).
    1. Navigate to **Yandex Data Processing**.
    1. Click the cluster name.
    1. Under **UI Proxy**, select `YARN Resource Manager Web UI`.
    1. Find the application in question and click the link in the **Tracking UI** column. The link name depends on the application status:

        * **ApplicationMaster**: For running applications.
        * **History**: For completed applications.

* Spark History Server Web UI

    1. Open the [folder dashboard](https://console.yandex.cloud).
    1. Navigate to **Yandex Data Processing**.
    1. Click the cluster name.
    1. Under **UI Proxy**, select `Spark History Server Web UI`.

        This will open the list of completed applications. To switch to the list of running applications, click **Show incomplete applications** at the bottom of the table.

    1. Find the application in question and click the link in the **App ID** column.

{% endlist %}

This will open the **Spark History Server Web UI** window with details of the application you selected:

* **Event Timeline**: History of job runs with info on added and removed [executors](../concepts/spark-sql.md#tasks).
* **Active Jobs**: List of running or pending jobs.
* **Completed Jobs**: List of completed jobs.

For each job, the table specifies:

* **Submitted** time
* **Duration**
* **Stages: Succeeded/Total**
* **Tasks: Succeeded/Total**

## Checking resources allocated to the application {#resources}

1. Open the [folder dashboard](https://console.yandex.cloud).
1. Navigate to **Yandex Data Processing**.
1. Click the cluster name.
1. Under **UI Proxy**, select `Spark History Server Web UI`.
1. In the top menu, navigate to **Executors**.

The UI displays two tables:

* **Summary**: High-level information, such as the number and status of [executors](../concepts/spark-sql.md#tasks) and resources in use.
* **Executors**: Information about each executor.

The tables specify the following:

* Amount of resources available per executor.
* Number of running and completed [tasks](../concepts/spark-sql.md#tasks).
* Task duration (**Task Time**), including the time spent on garbage collection (**GC Time**).

{% note tip %}

If garbage collection takes much time:

* Make sure the executor has enough memory allocated.
* Configure the garbage collector manually. Check out [this Apache Spark guide](https://spark.apache.org/docs/latest/tuning.html#garbage-collection-tuning).

{% endnote %}

## Checking persisted RDDs {#tables}

1. Open the [folder dashboard](https://console.yandex.cloud).
1. Navigate to **Yandex Data Processing**.
1. Click the cluster name.
1. Under **UI Proxy**, select `Spark History Server Web UI`.
1. In the top menu, navigate to **Storage**.

The UI displays the list of persisted resilient distributed datasets ([RDDs](https://spark.apache.org/docs/latest/rdd-programming-guide.html#resilient-distributed-datasets-rdds)). For each RDD, it shows memory consumption, disk usage, and caching progress.

To view details, click the RDD name.

## Checking the list of SQL queries and their plans {#sql}

1. Open the [folder dashboard](https://console.yandex.cloud).
1. Navigate to **Yandex Data Processing**.
1. Click the cluster name.
1. Under **UI Proxy**, select `Spark History Server Web UI`.
1. In the top menu, navigate to **SQL**.

The table lists executed SQL queries, including their start time and duration.

To see the query plan, click the query text in the **Description** column. The query plan is displayed as a flowchart. To view it as text, click **Details** at the bottom of the figure.

The query plan contains stats for each operator along with the number of completed tasks and their duration. If the query is still running, the current stats will be shown.