[Yandex Cloud documentation](../../index.md) > [Yandex Managed Service for OpenSearch](../index.md) > [Step-by-step guides](index.md) > Logs and monitoring > Cluster and host state monitoring # OpenSearch cluster and host state monitoring Data on the cluster and host state is available in the [management console](https://console.yandex.cloud). You can view them on the **Monitoring** tab of the cluster management page or in [Yandex Monitoring](../../monitoring/concepts/index.md). Diagnostic information about cluster states is presented as graphs. Charts are updated every 15 seconds. {% note info %} The most appropriate multiple units (MB, GB, and more) are automatically used in charts. {% endnote %} You can [configure alerts](#monitoring-integration) in Yandex Monitoring to receive notifications about cluster failures. In Yandex Monitoring, there are two alert thresholds: `Warning` and `Alarm`. If the specified threshold is exceeded, you will receive alerts via the configured [notification channels](../../monitoring/concepts/alerting.md#notification-channel). ## Cluster state monitoring {#monitoring-cluster} To view detailed information on the health state of a Managed Service for OpenSearch cluster: {% list tabs group=instructions %} - Management console {#console} 1. In the [management console](https://console.yandex.cloud), navigate to the folder page. 1. Navigate to **Managed Service for OpenSearch**. 1. Click the cluster name and open the ![image](../../_assets/console-icons/display-pulse.svg) **Monitoring** tab. The page displays the following charts: * Under **Cluster state**: * **Health status**: Cluster health and technical condition: * `0` (red): Cluster is unhealthy or partially functional. At least one of the [primary shards](../concepts/scalability-and-resilience.md) is not available. If the cluster responds to queries, search results will be incomplete. * `1` (yellow): Cluster is functional. There is no access to at least one of the shard replicas. The search results in the cluster's responses are full, but the cluster's operation will be disrupted if more unavailable shards appear. * `2` (green): Cluster is healthy. All cluster shards are available. * **Current master**: FQDN of one of the hosts with the `MANAGER` [role](../concepts/host-roles.md#data). * **Nodes**: Total number of hosts in the cluster (excluding `Dashboards` hosts) and the number of hosts with the `DATA` role. * **Pending tasks**: Number of enqueued tasks. * Under **Indices and load info**: * **Top indices by size**: Largest indexes in terms of occupied storage space and their size (in bytes). * **Active shards**: Number of active primary shards and the total number of active shards in the cluster. * **Search rate**: Number of search queries per second, per host. * **Top indices by docs count**: Indexes with the largest document count and the number of documents in them. * **Other shards**: Number of inactive shards in each of the following states: * `Delayed unassigned`: Host assignment is delayed. * `Unassigned`: No host is assigned. * `Unassigned Primary`: No host is assigned (primary shards only). * `Relocating`: Shards are being moved to another host. * `Initializing`: Shards are initializing. * **Indexing rate**: Indexing speed for each host (operations per second). * Under **Indices segments info**: * **Total indices segments per host**: Number of index segments for each host. * Under **Latest backup info**: * **Backup size**: Size of the latest backup: * `backup_total_size`: Total size of all indexes in the backup. * `backup_incremental_size`: Size of the indexes included in the backup increment. * `backup_free_space_required`: Storage size required to restore a cluster from a backup. {% note info %} To get started with Monitoring metrics, dashboards, or alerts, click **Open in Monium** in the top panel. {% endnote %} {% endlist %} ## Monitoring the state of hosts {#monitoring-hosts} To view detailed information on the state of individual Managed Service for OpenSearch hosts: 1. In the [management console](https://console.yandex.cloud), navigate to the folder page. 1. Navigate to **Managed Service for OpenSearch**. 1. Click the cluster name and open the ![image](../../_assets/console-icons/cube.svg) **Hosts** tab. 1. Select the **Monitoring** tab. 1. Select the host from the drop-down list. This page displays the charts showing workloads of individual cluster hosts. It depends on the host type: {% list tabs %} * MANAGER * **Process CPU**: Processor core workload generated by the JVM OpenSearch process. * **Memory usage**: Amount of RAM used, in bytes. * **JVM heap**: Use of JVM heap memory, in bytes. * **Disk space usage percent**: Percentage of the disk space used. * **Management thread pool**: Number of cluster management requests. * **Generic thread pool**: Number of requests for running general operations. * **Thread pool queued**: Number of enqueued requests. * **Thread pool rejected**: Number of rejected requests. * Under **Disk Metrics Details**: * **Disk write latency (percentiles)**: Disk write time, in percentiles. * **Disk write bytes**: Average and maximum disk write rate. * **Disk write operations**: Average and maximum number of write operations per second. * **Disk read latency (percentiles)**: Disk read time, in percentiles. * **Disk read bytes**: Average and maximum disk read rate. * **Disk read operations**: Average and maximum number of read operations per second. * **Disk write throttler latency (percentiles)**: Write delay introduced by exceeding disk quota, percentiles. * **Disk read throttler latency (percentiles)**: Read delay introduced by exceeding disk quota, percentiles. * **Disk used quota**: Disk operation quota usage. * DATA * **Process CPU**: Processor core workload generated by the JVM OpenSearch process. * **Memory usage**: Use of RAM, in bytes. * **JVM heap percent**: Percentage of the JVM heap memory used. * **Disk space usage percent**: Percentage of the disk space used. * **Shards count**: Number of index shards. * **Primary shards count**: Number of primary index shards. * **Open file descriptors**: Number of open file descriptors. * **Indexing rate**: Indexing speed (operations per second). * **Search queries**: Number of search queries per second. * **Write thread pool**: Requests for indexing, deleting, or updating documents. * **Average query time**: Average query execution time. * **Average indexing time**: Average time spent on document indexing. * **Thread pool queued**: Number of enqueued requests. * **Thread pool rejected**: Number of rejected requests. * **Merging time**: Time spent to merge the documents. * Under **Disk Metrics Details**: * **Disk write latency (percentiles)**: Disk write time, in percentiles. * **Disk write bytes**: Average and maximum disk write rate. * **Disk write operations**: Average and maximum number of write operations per second. * **Disk read latency (percentiles)**: Disk read time, in percentiles. * **Disk read bytes**: Average and maximum disk read rate. * **Disk read operations**: Average and maximum number of read operations per second. * **Disk write throttler latency (percentiles)**: Write delay introduced by exceeding disk quota, percentiles. * **Disk read throttler latency (percentiles)**: Read delay introduced by exceeding disk quota, percentiles. * **Disk used quota**: Disk operation quota usage. * DASHBOARDS * **Is Alive**: Status that shows the host is available. * **Requests Total**: Total number of host requests. * **Process CPU**: Processor core workload generated by the JVM OpenSearch process. * **Memory usage**: Use of RAM, in bytes. * **Disk read/write bytes**: Speed of disk operations, in bytes per second. * **Disk IOPS**: Number of disk operations per second. * **Network Packets**: Network packet exchange rate, in packets per second. * **Network bytes**: Speed of network data exchange, in bytes per second. {% endlist %} ## Monitoring the state of host groups {#monitoring-host-groups} To view detailed information on the state of a Managed Service for OpenSearch host group: 1. In the [management console](https://console.yandex.cloud), navigate to the folder page. 1. Navigate to **Managed Service for OpenSearch**. 1. Click the cluster name and open the ![image](../../_assets/console-icons/copy-transparent.svg) **Node groups** tab. 1. Select the **Monitoring** tab. 1. Select the host group from the drop-down list. This page displays the charts showing workloads of a cluster host group. The list depends on the type of hosts in the group and matches the [charts shown for individual hosts](#monitoring-hosts). ## Setting up alerts in Yandex Monitoring {#monitoring-integration} {% list tabs group=instructions %} - Management console {#console} 1. In the [management console](https://console.yandex.cloud), select the folder with the cluster for which you want to set up alerts. 1. Navigate to ![image](../../_assets/console-icons/display-pulse.svg) **Monitoring**. 1. Under **Service dashboards**, select: * **Managed Service for OpenSearch** to configure cluster alerts. * **Managed Service for OpenSearch — Dashboards** to configure alerts for hosts with the `DASHBOARDS` role. * **Managed Service for OpenSearch — Data** to configure alerts for hosts with the `DATA` role. * **Managed Service for OpenSearch — Manager** to configure alerts for hosts with the `MANAGER` role. 1. In the chart you need, click ![options](../../_assets/console-icons/ellipsis.svg) and select **Create alert**. 1. If the chart displays multiple metrics, select the data query for the relevant metric and click **Continue**. Learn more about the query language in [this Yandex Monitoring guide](../../monitoring/concepts/querying.md). 1. Set the `Alarm` and `Warning` alert thresholds. 1. Click **Create alert**. {% endlist %} To have other cluster health indicators monitored automatically: {% list tabs group=instructions %} - Management console {#console} 1. [Create an alert](../../monitoring/operations/alert/create-alert.md). 1. Add a status metric. 1. In the alert parameters, set the alert thresholds. {% endlist %} Recommended threshold values for selected metrics: | Metric | Designation | Formula | `Alarm` | `Warning` | |----------------------------------------------|:--------------------------------------:|:----------------------:|:---------------------------------:|:---------------------------------:| | Cluster status | `opensearch_status` | `bottom_last(1)` | `equal to 0` | `equal to 1` | | Number of unassigned shards | `opensearch_unassigned_shards` | `top_last(1)` | `greater than 0` | | | Number of shards being relocated | `opensearch_relocating_shards` | `top_last(1)` | `greater than 0` | | | Number of initializing shards | `opensearch_initializing_shards` | `top_last(1)` | `greater than 0` | | | Number of delayed unassigned shards | `opensearch_delayed_unassigned_shards` | `top_last(1)` | `greater than 0` | | | JVM heap memory used | `opensearch_jvm_mem_heap_used_percent` | `top_last(1)` | Over 90% of host RAM | | | Storage space used | `opensearch_fs_total_used_percent` | `top_last(1)` | Over 90% of the storage size | Over 85% of the storage size | | Using the JVM long-lived object pool | `opensearch_jvm_mem_heap_pressure` | `top_last(1)` | Over 90% of host RAM | Over 75% of host RAM | | Storage space used | `disk.used_bytes` | — | 90% of the storage size | 80% of the storage size | For the `disk.used_bytes` metric, the `Alarm` and `Warning` thresholds are only set in bytes. For example, the recommended values for a 100 GB disk are as follows: * `Alarm`: `96636764160` bytes (90%). * `Warning`: `85899345920` bytes (80%). You can view the current storage size and RAM of the hosts in the [detailed information about the cluster](cluster-list.md#get-cluster). For a complete list of supported metrics, see [this Monitoring guide](../../monitoring/metrics-ref/managed-opensearch-ref.md). ## Cluster health and status {#cluster-health-and-status} The **State** of a cluster shows the health of its hosts, while the **Status** shows whether the cluster is started, stopped, or is at an intermediate stage. To check the cluster state and status: {% list tabs group=instructions %} - Management console {#console} 1. In the [management console](https://console.yandex.cloud), navigate to the folder page. 1. Navigate to **Managed Service for OpenSearch**. 1. In the cluster row, hover over the indicator in the **Availability** column. - CLI {#cli} If you do not have the Yandex Cloud CLI yet, [install and initialize it](../../cli/quickstart.md#install). The folder used by default is the one specified when [creating](../../cli/operations/profile/profile-create.md) the CLI profile. To change the default folder, use the `yc config set folder-id ` command. You can also specify a different folder for any command using `--folder-name` or `--folder-id`. If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions. To find out the state and status of a cluster, get information about it: ```bash yc managed-opensearch cluster get ``` You will see the cluster state in the `health` parameter and the cluster status, in the `status` parameter. You can get the cluster name and ID with the [list of clusters in the folder](cluster-list.md#list-clusters). - REST API {#api} 1. [Get an IAM token for API authentication](../api-ref/authentication.md) and put it into an environment variable: ```bash export IAM_TOKEN="" ``` 1. Call the [Cluster.Get](../api-ref/Cluster/get.md) method, e.g., via the following [cURL](https://curl.se/) request: ```bash curl \ --request GET \ --header "Authorization: Bearer $IAM_TOKEN" \ --url 'https://mdb.api.cloud.yandex.net/managed-opensearch/v1/clusters/' ``` You can get the cluster ID with the [list of clusters in the folder](#list-clusters). 1. Check the [server response](../api-ref/Cluster/get.md#yandex.cloud.mdb.opensearch.v1.Cluster) to make sure your request was successful. You will see the cluster health and status in the `health` and `status` parameters, respectively. - gRPC API {#grpc-api} 1. [Get an IAM token for API authentication](../api-ref/authentication.md) and put it into an environment variable: ```bash export IAM_TOKEN="" ``` 1. Clone the [cloudapi](https://github.com/yandex-cloud/cloudapi) repository: ```bash cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi ``` Below, we assume that the repository contents reside in the `~/cloudapi/` directory. 1. Call the [ClusterService.Get](../api-ref/grpc/Cluster/get.md) method, e.g., via the following [gRPCurl](https://github.com/fullstorydev/grpcurl) request: ```bash grpcurl \ -format json \ -import-path ~/cloudapi/ \ -import-path ~/cloudapi/third_party/googleapis/ \ -proto ~/cloudapi/yandex/cloud/mdb/opensearch/v1/cluster_service.proto \ -rpc-header "Authorization: Bearer $IAM_TOKEN" \ -d '{ "cluster_id": "" }' \ mdb.api.cloud.yandex.net:443 \ yandex.cloud.mdb.opensearch.v1.ClusterService.Get ``` You can get the cluster ID with the [list of clusters in the folder](#list-clusters). 1. Check the [server response](../api-ref/grpc/Cluster/get.md#yandex.cloud.mdb.opensearch.v1.Cluster) to make sure your request was successful. You will see the cluster health and status in the `health` and `status` parameters, respectively. {% endlist %} ### Cluster states {#cluster-health} State | Description | Suggested actions :--- | :--- | :--- **ALIVE** | Cluster is operating normally. | No action is required. **DEGRADED** | Cluster is not running at its full capacity: the state of at least one of the hosts is other than `ALIVE`. | Run the diagnostics:
  • Go to the **Hosts** tab and see which hosts are not working.
  • Go to the **Operations** tab and make sure all operations are completed.
  • Make sure the cluster is not under maintenance.
If you cannot find the cause yourself, [contact support](https://center.yandex.cloud/support). **DEAD** | The cluster is down: none of its hosts are running. | [Make a support request](https://center.yandex.cloud/support) stating the following:
  • Cluster ID.
  • IDs of the last operations performed on it.
  • Time the cluster entered the `DEAD` state according to the [availability charts](#monitoring-cluster).
**UNKNOWN** | Cluster state is unknown. | [Make a support request](https://center.yandex.cloud/support) stating the following:
  • Cluster ID.
  • IDs of the last operations performed on it.
  • Time the cluster entered the `UNKNOWN` state according to the [availability charts](#monitoring-cluster).
### Cluster statuses {#cluster-status} Status | Description | Suggested actions :--- | :--- | :--- **CREATING** | Preparing for the first start | Wait a while and get started. The time it takes to create a cluster depends on the host class. **RUNNING** | The cluster is operating normally | No action is required. **STOPPING** | The cluster is stopping | After a while, the cluster status will switch to `STOPPED` and the cluster will be disabled. No action is required. **STOPPED** | The cluster is stopped | Start the cluster to get it running again. **STARTING** | Starting the cluster that was stopped earlier | After a while, the cluster status will switch to `RUNNING`. Wait a while and get started. **UPDATING** | Updating the cluster's configuration | Once the update is complete, the cluster will get the status it had prior to the update: `RUNNING` or `STOPPED`. **ERROR** | Error when performing an operation with the cluster or during a maintenance window | If the cluster remains in this status for a long time, [contact support](https://center.yandex.cloud/support). You can see whether a cluster is available by its status. **STATUS_UNKNOWN** | The cluster is unable to determine its status | If the cluster remains in this status for a long time, [contact support](https://center.yandex.cloud/support).