[Yandex Cloud documentation](../../index.md) > [Yandex Managed Service for Apache Spark™](../index.md) > [Step-by-step guides](index.md) > Logs and monitoring > Cluster state monitoring

# Apache Spark™ cluster state monitoring

Data on the cluster and host state is available in the [management console](https://console.yandex.cloud). You can view them on the **Monitoring** tab of the cluster management page or in [Yandex Monitoring](../../monitoring/concepts/index.md).

Diagnostic information about cluster states is presented as graphs.

Charts are updated every 15 seconds.

{% note info %}

The most appropriate multiple units (MB, GB, and more) are automatically used in charts.

{% endnote %}

You can [configure alerts](#monitoring-integration) in Yandex Monitoring to receive notifications about cluster failures. In Yandex Monitoring, there are two alert thresholds: `Warning` and `Alarm`. If the specified threshold is exceeded, you will receive alerts via the configured [notification channels](../../monitoring/concepts/alerting.md#notification-channel).

## Cluster state monitoring {#monitoring-cluster}

To view detailed information on the health state of a Apache Spark™ cluster:

{% list tabs group=instructions %}

- Management console {#console}

  1. In the [management console](https://console.yandex.cloud), navigate to the relevant folder.
  1. Navigate to **Managed Service for Apache Spark™**.
  1. Click the name of your cluster and select the **Monitoring** tab.

  1. To get started with Yandex Monitoring metrics, dashboards, or alerts, click **Open in Monium** in the top panel.

  The page displays the following charts:

  * Under **Cluster Resource Usage**:

    * **Total Allocated Nodes**: Number of used cluster hosts.
    * **Total Running Containers & Total Running Jobs**: Number of running jobs and containers.

      * **Spark Containers**: Number of running containers.
      * **Spark Jobs**: Number of running jobs.

    * **Pending Containers**: Number of containers waiting to run.
    * **CPU Resources**: Availability of processor cores.
  
      * **Allocated CPU**: Number of CPUs in use.
      * **Allocatable CPU**: Number of CPUs available to containers.
      * **Capacity CPU**: Total CPUs per cluster. Some CPUs may be reserved for system needs.

    * **Available CPU**: Number of available CPUs in the cluster.
    * **CPU Usage/Limits**: CPU utilization by containers.

      * **Additional containers CPU limited**: CPU usage limit for system containers.
      * **Additional containers CPU usage**: Number of CPUs used by the system containers.
      * **Spark containers CPU usage**: CPU usage limit for Spark application containers.
      * **Spark containers CPU limited**: Number of CPUs used by Spark application containers.

    * **Memory Resources**: Available RAM.

      * **Capacity Memory**: Total host RAM. Some RAM may be reserved for system needs.
      * **Allocatable Memory**: Host RAM available to containers.
      * **Allocated Memory**: Host RAM in use.

    * **Available Memory**: Available cluster RAM.
    * **Memory Usage/Limits**: RAM utilization by containers.

      * **Additional containers Memory limited**: RAM limit for system containers.
      * **Additional containers Memory usage**: RAM used by system containers.
      * **Spark containers Memory limited**: RAM limit for Spark application containers.
      * **Spark containers Memory usage**: RAM used by Spark application containers.

  * Under **Driver Pool**:

    * **Driver Pool: Allocated Nodes**: Number of Apache Spark™ driver hosts.
    * **Driver Pool: Running Containers**: Number of running containers in the driver pool.
    * **Spark Drivers: Running Containers By Nodes**: Number of running containers on Apache Spark™ driver hosts.
    * **Spark Drivers: CPU Limits By Nodes**: CPU limit for Apache Spark™ driver hosts.
    * **Spark Drivers: Used CPU By Nodes**: CPUs used by Apache Spark™ driver hosts.
    * **Driver Pool: Available CPU By Nodes**: CPUs available on Apache Spark™ driver hosts.
    * **Spark Drivers: Memory Limits By Nodes**: RAM limit for Apache Spark™ driver hosts.
    * **Spark Drivers: Used Memory By Nodes**: RAM used by Apache Spark™ driver hosts.
    * **Driver Pool: Available Memory By Nodes**: RAM available on Apache Spark™ driver hosts.

  * Under **Executor Pool**:

    * **Executor Pool: Allocated Nodes**: Number of Apache Spark™ executor hosts.
    * **Executor Pool: Running Containers**: Number of running containers in the Apache Spark™ executor pool.
    * **Spark Executors: Running Containers By Node**: Number of running containers on Apache Spark™ executor hosts.
    * **Spark Executors: CPU Limits By Nodes**: CPUs limit for Apache Spark™ executor hosts.
    * **Spark Executors: Used CPU By Nodes**: CPUs used by Apache Spark™ executor hosts.
    * **Executor Pool: Available CPU By Nodes**: CPUs available on Apache Spark™ executor hosts.
    * **Spark Executors: Memory Limits By Nodes**: RAM limit on Apache Spark™ executor hosts.
    * **Spark Executors: Used Memory By Nodes**: RAM used by Apache Spark™ executor hosts.
    * **Executor Pool: Available Memory By Nodes**: RAM available on Apache Spark™ executor hosts.

  * Under **Spark Jobs**:

    * **Running Executors By Jobs**: Number of Apache Spark™ executor hosts by jobs in progress.
    * **Spark Application: Running Stages**: Number of stages in progress by jobs.
    * **Spark Application: Active Tasks**: Number of tasks in progress by jobs.
    * **Spark CPU Limits By Jobs**: CPU limit for jobs.
    * **Spark Used CPU By Jobs**: CPUs used by jobs.
    * **Spark Application: Completed Stages**: Number of completed stages by jobs.
    * **Spark Memory Limits By Jobs**: RAM limit by jobs.
    * **Spark Used Memory By Jobs**: RAM used by jobs.
    * **Spark Application: Completed Tasks**: Number of completed tasks by jobs.
    * **Spark Application: Failed Stages**: Number of failed job stages, by jobs.
    * **Spark Application: Waiting Stages**: Number of pending job stages, by jobs.
    * **Spark Application: Failed Tasks**: Number of failed tasks, by jobs.

{% endlist %}

## Setting up alerts in Yandex Monitoring {#monitoring-integration}

To configure [cluster](#monitoring-cluster) state indicator alerts:

{% list tabs group=instructions %}

- Management console {#console}

  1. In the [management console](https://console.yandex.cloud), select the folder with the cluster for which you want to set up alerts.
  1. Navigate to ![image](../../_assets/console-icons/display-pulse.svg) **Monitoring**.
  1. Under **Service dashboards**, select **Managed Service for Apache Spark™ — Cluster Overview**.
  1. In the chart you need, click ![options](../../_assets/console-icons/ellipsis.svg) and select **Create alert**.
  1. If the chart displays multiple metrics, select the data query for the relevant metric and click **Continue**. You can learn more about the query language in [this Yandex Monitoring article](../../monitoring/concepts/querying.md).
  1. Set the `Alarm` and `Warning` threshold values to trigger the alert.
  1. Click **Create alert**.

{% endlist %}

To have other cluster health indicators monitored automatically:

{% list tabs group=instructions %}

- Management console {#console}

    1. [Create an alert](../../monitoring/operations/alert/create-alert.md).
    1. Add a status metric.
    1. In the alert parameters, set the alert thresholds.

{% endlist %}

For a complete list of supported metrics, see [this Monitoring guide](../../monitoring/metrics-ref/managed-spark-ref.md).

## Cluster health and status {#cluster-health-and-status}

The **State** of a cluster shows the health of its hosts, while the **Status** shows whether the cluster is started, stopped, or is at an intermediate stage.

To view the health state and status of a cluster:

1. Navigate to the folder dashboard and select **Managed Service for Apache Spark™**.
1. In the cluster row, hover over the indicator in the **Availability** column.

### Cluster health states {#cluster-health}

State | Description | Suggested actions
:--- | :--- | :---
**ALIVE** | Cluster is operating normally. | No action is required.
**DEGRADED** | Cluster is not running at its full capacity: the state of at least one of the hosts is other than `ALIVE`. | Run the diagnostics:<ul><li>Go to the **Hosts** tab and see which hosts are not working.</li><li>Go to the **Operations** tab and make sure all operations are completed.</li><li>Make sure the cluster is not under maintenance.</li></ul>If you cannot find the cause yourself, [contact support](https://center.yandex.cloud/support).
**DEAD** | The cluster is down: none of its hosts are running. | [Make a support request](https://center.yandex.cloud/support) stating the following:<ul><li>Cluster ID.</li><li>IDs of the last operations performed on it.</li><li>Time the cluster entered the `DEAD` state according to the [availability charts](#monitoring-cluster).</li></ul>
**UNKNOWN** | Cluster state is unknown. | [Make a support request](https://center.yandex.cloud/support) stating the following:<ul><li>Cluster ID.</li><li>IDs of the last operations performed on it.</li><li>Time the cluster entered the `UNKNOWN` state according to the [availability charts](#monitoring-cluster).</li></ul>

### Cluster statuses {#cluster-status}

Status | Description | Suggested actions
:--- | :--- | :---
**CREATING** | Preparing for the first start | Wait a while and get started. The time it takes to create a cluster depends on the host class.
**RUNNING** | The cluster is operating normally | No action is required.
**STOPPING** | The cluster is stopping | After a while, the cluster status will switch to `STOPPED` and the cluster will be disabled. No action is required.
**STOPPED** | The cluster is stopped | Start the cluster to get it running again.
**STARTING** | Starting the cluster that was stopped earlier | After a while, the cluster status will switch to `RUNNING`. Wait a while and get started.
**UPDATING** | Updating the cluster's configuration | Once the update is complete, the cluster will get the status it had prior to the update: `RUNNING` or `STOPPED`.
**ERROR** | Error when performing an operation with the cluster or during a maintenance window | If the cluster remains in this status for a long time, [contact support](https://center.yandex.cloud/support). You can see whether a cluster is available by its status.
**STATUS_UNKNOWN** | The cluster is unable to determine its status | If the cluster remains in this status for a long time, [contact support](https://center.yandex.cloud/support).