[Yandex Cloud documentation](../../index.md) > [Yandex Managed Service for Valkey™](../index.md) > [Step-by-step guides](index.md) > Logs and monitoring > Cluster, host, and shard state monitoring # Cluster, host, and shard state monitoring in Yandex Managed Service for Valkey™ Data on the cluster and host state is available in the [management console](https://console.yandex.cloud). You can view them on the **Monitoring** tab of the cluster management page or in [Yandex Monitoring](../../monitoring/concepts/index.md). Diagnostic information about cluster states is presented as graphs. Chart update rate: * Standard hosts and hosts with an increased RAM to vCPU ratio (`memory-optimized`): 15 seconds. * Hosts with a guaranteed vCPU share under 100% (`burstable`): 150 seconds. {% note info %} The most appropriate multiple units (MB, GB, and more) are automatically used in charts. {% endnote %} You can [configure alerts](#monitoring-integration) in Yandex Monitoring to receive notifications about cluster failures. In Yandex Monitoring, there are two alert thresholds: `Warning` and `Alarm`. If the specified threshold is exceeded, you will receive alerts via the configured [notification channels](../../monitoring/concepts/alerting.md#notification-channel). ## Cluster health state monitoring {#monitoring-cluster} To view detailed information on the health state of a Yandex Managed Service for Valkey™ cluster: {% list tabs group=instructions %} - Management console {#console} 1. In the [management console](https://console.yandex.cloud), select the folder containing your cluster. 1. Navigate to **Yandex Managed Service for Valkey™**. 1. Click the name of your cluster and select the **Monitoring** tab. The page that opens will display performance charts for the cluster. To get started with Yandex Monitoring metrics, dashboards, or alerts, click **Open in Monium** in the top panel. The following charts are displayed for a cluster: * **Hosts available for read**: Cluster host read availability. * **Hosts available for write**: Cluster host write availability. Under **Traffic**: * **Connected Clients**: Number of open connections to services on cluster hosts. If the cluster is [sharded](../concepts/sharding.md) or uses [replication](../concepts/replication.md), some of the connections will be used for inter-host communication within the cluster. If you encounter errors when connecting to the cluster, inactive applications may be keeping connections open for too long. If this is the case, [update the Valkey™ settings](update.md#change-valkey-config) by editing the [Timeout](../concepts/settings-list.md#settings-timeout) value. * **Commands Processed**: Rate of command processing on each cluster host. * **Commandstats**: Rate of command calls on each cluster host. Under **Latency**: * **Latencystats, p50**: Median command execution time. * **Latencystats, p99**: Command execution time in the 99th percentile. * **Latencystats, p99.9**: Command execution time in the 99.9th percentile. Under **DB Metrics**: * **DB keys**: Number of keys on each cluster host. * **Evicted keys**: Number of keys deleted from memory when inserting new data. The information is displayed for each cluster host. * **Cache Hit Rate**: Percentage of requests for which data was retrieved from the cache. The information is displayed for each cluster host. * **Replication Lag**: Replica's lag behind the master, in seconds. * **IO threads active**: Number of active threads on each cluster host. * **Valkey-server OOM kills (for last hour)**: Number of `Out of Memory` errors on each cluster host. Under **Resources** → **Memory**: * **Outer memory limit**: Limit and RAM usage by processes on each cluster host. * **Inner Memory limit**: Limit and RAM usage on each cluster host. * **Memory fragmentation ratio**: Ratio of allocated memory to actually used memory on each cluster host. * **Valkey Used Memory on Masters**: RAM usage on the cluster's master hosts. * **Valkey Used Memory on Replicas**: RAM usage in replicated cluster hosts. * **Replication buffer size**: Size of used and available memory for the [replication](../concepts/replication.md#replication) buffer on each cluster host. * **Client recent max input buffer size**: Maximum buffer size for handling incoming client connections on each cluster host. * **Client recent max output buffer size**: Maximum buffer size for handling outgoing client connections on each cluster host. * **Copy-on-write allocations**: Amount of memory allocated by the COW mechanism when creating child processes. The information is displayed for each cluster host. Under **Resources** → **CPU**: * **CPU usage main thread**: CPU time, system and user, consumed by the main thread on each cluster host. * **CPU usage main thread on Masters**: CPU time, system and user, used by the main thread on the cluster's master hosts. * **CPU usage main thread on Replicas**: CPU time, system and user, consumed by the main thread on the cluster's replica hosts. Under **Resources** → **Network**: * **Network usage**: Amount of incoming and outgoing network traffic on each cluster host. * **Network usage on Masters**: Amount of incoming and outgoing network traffic on the cluster's master hosts. * **Network usage on Replicas**: Amount of incoming and outgoing network traffic on the cluster's replica hosts. * **Replication network usage**: Amount of incoming and outgoing network traffic for replication on each cluster host. * **Replication network usage on Masters**: Amount of incoming and outgoing replication network traffic on the cluster's master hosts. * **Replication network usage on Replicas**: Amount of incoming and outgoing network traffic for replication on the cluster's replica hosts. Under **Resources** → **Data**: * **Disk IO**: Rate of disk operations on each cluster host. * **Disk space usage percentage**: Percentage of disk space used on each cluster host. * **Disk space usage**: Amount of disk space used on each cluster host. Under **Errors**: * **Errorstats**: Rate of failed queries on each cluster host. * **Sentinel error replies**: Number of Valkey™ Sentinel's error responses on each cluster host. * **ACL errors**: Rate of access errors on each cluster host. Under **Slowlog**: * **Slowlog top operations on Master**: Number of records for each command in the Slowlog on the cluster's master hosts. * **Slowlog top operations on Replicas**: Number of records for each command in the Slowlog on the cluster's replica hosts. * **Slowlog new records**: Number of new records in the Slowlog on each cluster host. {% endlist %} ## Host state monitoring {#monitoring-hosts} To view detailed information on the state of individual Yandex Managed Service for Valkey™ hosts: {% list tabs group=instructions %} - Management console {#console} 1. In the [management console](https://console.yandex.cloud), select the folder containing your cluster. 1. Navigate to **Yandex Managed Service for Valkey™**. 1. Click the name of your cluster and select **Hosts** → **Monitoring**. 1. Select the host from the drop-down list. This page displays the charts showing workloads of individual cluster hosts: * **CPU usage**: Processor core workload. As the workload goes up, the `idle` value goes down. * **Memory usage**: Use of RAM, in bytes. At high workloads, the `Free` value goes down, while the other values go up. * **Disk IOPS**: Number of disk operations per second. The `Read` value increases during database read activity, and `Write`, during database write activity. * **Network bytes**: Network data exchange rate, in bytes per second. For **Replica** hosts, the `Received` value is normally greater than `Sent`. * **Network packets**: Network packet exchange rate, in packets per second. For **Replica** hosts, the `Received` value is normally greater than `Sent`. * **Disk space usage**: Amount of used and total disk space. Under **Disk Metrics Details**: * **Disk write latency (percentiles)**: Disk write latency, percentiles. * **Disk write bytes**: Average and maximum disk write rate, bytes per second. * **Disk write operations**: Average and maximum number of write operations per second. * **Disk read latency (percentiles)**: Disk read latency, percentiles. * **Disk read bytes**: Average and maximum disk read rate, bytes per second. * **Disk read operations**: Average and maximum number of read operations per second. * **Disk write throttler latency (percentiles)**: Write delay introduced by exceeding disk quota, percentiles. * **Disk read throttler latency (percentiles)**: Read delay introduced by exceeding disk quota, percentiles. * **Disk used quota**: Average and maximum quota used percentage for disk operations. {% endlist %} ## Shard state monitoring {#monitoring-shards} To view detailed information on the health state of Yandex Managed Service for Valkey™ shards: {% list tabs group=instructions %} - Management console {#console} 1. In the [management console](https://console.yandex.cloud), select the folder containing your cluster. 1. Navigate to **Yandex Managed Service for Valkey™**. 1. Click the name of your cluster and select the **Monitoring** tab. 1. Navigate to the **Shards** tab and select a shard. The page that opens will display health state charts for the selected shard and its hosts. To get started with Yandex Monitoring metrics, dashboards, or alerts, click **Open in Monium** in the top panel. The following charts are displayed for shards: * **Hosts available for read**: Shard host read availability. * **Hosts available for write**: Shard host write availability. Under **Traffic**: * **Connected Clients**: Number of open connections to services on shard hosts. If the cluster is [sharded](../concepts/sharding.md) or uses [replication](../concepts/replication.md), some of the connections will be used for inter-host communication within the cluster. If you encounter errors when connecting to the cluster, inactive applications may be keeping connections open for too long. If this is the case, [update the Valkey™](update.md#change-valkey-config) settings by editing the [Timeout](../concepts/settings-list.md#settings-timeout) value. * **Commands Processed**: Rate of command processing on each shard host. * **Commandstats**: Rate of command calls on each shard host. Under **Latency**: * **Latencystats, p50**: Median command execution time on the shard. * **Latencystats, p99**: Command execution time in the 99th percentile on the shard. * **Latencystats, p99.9**: Command execution time in the 99.9th percentile on the shard. Under **DB Metrics**: * **DB keys**: Number of keys on each shard host. * **Evicted keys**: Number of keys deleted from memory when inserting new data. The information is displayed for each shard host. * **Cache Hit Rate**: Percentage of requests for which data was retrieved from the cache. The information is displayed for each shard host. * **Replication Lag**: Replica's lag behind the master, in seconds. * **IO threads active**: Number of active threads on each shard host. * **Valkey-server OOM kills (for last hour)**: Number of `Out of Memory` errors on each shard host. Under **Resources** → **Memory**: * **Outer memory limit**: Limit and RAM usage by processes on each shard host. * **Inner Memory limit**: Limit and RAM usage on each shard host. * **Memory fragmentation ratio**: Ratio of allocated memory to actually used memory on each shard host. * **Valkey Used Memory on Masters**: RAM usage on the shard's master hosts. * **Valkey Used Memory on Replicas**: RAM usage in replicated shard hosts. * **Replication buffer size**: Size of used and available memory for the [replication](../concepts/replication.md#replication) buffer on each shard host. * **Client recent max input buffer size**: Maximum buffer size for handling incoming client connections on each shard host. * **Client recent max output buffer size**: Maximum buffer size for handling outgoing client connections on each shard host. * **Copy-on-write allocations**: Amount of memory allocated by the COW mechanism when creating child processes. The information is displayed for each shard host. Under **Resources** → **CPU**: * **CPU usage main thread**: CPU time, system and user, consumed by the main thread on each shard host. * **CPU usage main thread on Masters**: CPU time, system and user, consumed by the main thread on the shard's master hosts. * **CPU usage main thread on Replicas**: CPU time, system and user, used by the main thread on the shard's replica hosts. Under **Resources** → **Network**: * **Network usage**: Amount of incoming and outgoing network traffic on each shard host. * **Network usage on Masters**: Amount of incoming and outgoing network traffic on the shard's master hosts. * **Network usage on Replicas**: Amount of incoming and outgoing network traffic on the shard's replica hosts. * **Replication network usage**: Amount of incoming and outgoing network traffic for replication on each shard host. * **Replication network usage on Masters**: Amount of incoming and outgoing network traffic for replication on the shard's master hosts. * **Replication network usage on Replicas**: Amount of incoming and outgoing network traffic for replication on the shard's replica hosts. Under **Resources** → **Data**: * **Disk IO**: Rate of disk operations on each shard host. * **Disk space usage percentage**: Percentage of disk space used on each shard host. * **Disk space usage**: Amount of disk space used on each shard host. Under **Errors**: * **Errorstats**: Rate of failed queries on each shard host. * **Sentinel error replies**: Number of Valkey™ Sentinel's error responses on each shard host. * **ACL errors**: Rate of access errors on each shard host. Under **Slowlog**: * **Slowlog top operations on Master**: Number of records for each command in the Slowlog on the shard's master hosts. * **Slowlog top operations on Replicas**: Number of records for each command in the Slowlog on the shard's replica hosts. * **Slowlog new records**: Number of new records in the Slowlog on each shard host. {% endlist %} ## Setting up alerts in Yandex Monitoring {#monitoring-integration} To configure state indicator alerts for a [cluster](#monitoring-cluster) and [hosts](#monitoring-hosts): {% list tabs group=instructions %} - Management console {#console} 1. In the [management console](https://console.yandex.cloud), select the folder with the cluster for which you want to set up alerts. 1. Navigate to ![image](../../_assets/console-icons/display-pulse.svg) **Monitoring**. 1. Under **Service dashboards**, select: * **Yandex Managed Service for Valkey™ — Cluster Overview** to set up cluster alerts. * **Yandex Managed Service for Valkey™ — Host Overview** to set up host alerts. 1. In the chart you need, click ![options](../../_assets/console-icons/ellipsis.svg) and select **Create alert**. 1. If the chart displays multiple metrics, select the data query for the relevant metric and click **Continue**. Learn more about the query language in [this Yandex Monitoring guide](../../monitoring/concepts/querying.md). 1. Set the `Alarm` and `Warning` alert thresholds. 1. Click **Create alert**. {% endlist %} To have other cluster health indicators monitored automatically: {% list tabs group=instructions %} - Management console {#console} 1. [Create an alert](../../monitoring/operations/alert/create-alert.md). 1. Add a status metric. 1. In the alert parameters, set the alert thresholds. {% endlist %} Recommended threshold values for selected metrics: | Metric | Internal metric name | `Alarm` | `Warning` | |----------------------------------------------------------------------------------------------------------:|:-------------------:|:------------------------:|:------------------------:| | Database write availability | `can_write` | `Equal to 0` | — | | Number of out-of-memory errors, per hour | `redis_oom_count` | `More than 2` | `More than 0` | | RAM utilization (only for the [noeviction policy](../concepts/settings-list.md#settings-maxmemory-policy)) | `redis_used_memory` | 90% of RAM | 75% of RAM | | Storage space used | `disk.used_bytes` | 90% of the storage size | 80% of the storage size | For the `disk.used_bytes` metric, the `Alarm` and `Warning` thresholds are only set in bytes. For example, the recommended values for a 100 GB disk are as follows: * `Alarm`: `96636764160` bytes (90%). * `Warning`: `85899345920` bytes (80%). You can view the current RAM of the hosts in the [detailed information about the cluster](cluster-list.md#get-cluster). For a complete list of supported metrics, see [this Monitoring guide](../../monitoring/metrics-ref/managed-valkey-ref.md). ## Cluster health and status {#cluster-health-and-status} The **State** of a cluster shows the health of its hosts, while the **Status** shows whether the cluster is started, stopped, or is at an intermediate stage. To check the cluster state and status: {% list tabs group=instructions %} - Management console {#console} 1. In the [management console](https://console.yandex.cloud), select the folder containing your cluster. 1. Navigate to **Yandex Managed Service for Valkey™**. 1. In the cluster row, hover over the indicator in the **Availability** column. {% endlist %} ### Cluster health states {#cluster-health} State | Description | Suggested actions :--- | :--- | :--- **ALIVE** | Cluster is operating normally. | No action is required. **DEGRADED** | Cluster is not running at its full capacity: the state of at least one of the hosts is other than `ALIVE`. | Run the diagnostics:If you cannot find the cause yourself, [contact support](https://center.yandex.cloud/support). **DEAD** | The cluster is down: none of its hosts are running. | [Make a support request](https://center.yandex.cloud/support) stating the following: **UNKNOWN** | Cluster state is unknown. | [Make a support request](https://center.yandex.cloud/support) stating the following: ### Cluster statuses {#cluster-status} Status | Description | Suggested actions :--- | :--- | :--- **CREATING** | Preparing for the first start | Wait a while and get started. The time it takes to create a cluster depends on the host class. **RUNNING** | The cluster is operating normally | No action is required. **STOPPING** | The cluster is stopping | After a while, the cluster status will switch to `STOPPED` and the cluster will be disabled. No action is required. **STOPPED** | The cluster is stopped | Start the cluster to get it running again. **STARTING** | Starting the cluster that was stopped earlier | After a while, the cluster status will switch to `RUNNING`. Wait a while and get started. **UPDATING** | Updating the cluster's configuration | Once the update is complete, the cluster will get the status it had prior to the update: `RUNNING` or `STOPPED`. **ERROR** | Error when performing an operation with the cluster or during a maintenance window | If the cluster remains in this status for a long time, [contact support](https://center.yandex.cloud/support). You can see whether a cluster is available by its status. **STATUS_UNKNOWN** | The cluster is unable to determine its status | If the cluster remains in this status for a long time, [contact support](https://center.yandex.cloud/support).