[Yandex Cloud documentation](../../index.md) > [Tutorials](../index.md) > [Container infrastructure](index.md) > Managed Service for Kubernetes > Monitoring > Cluster monitoring with Prometheus and Grafana

# Yandex Managed Service for Kubernetes cluster monitoring with the help of Prometheus Operator with Yandex Monitoring support

# Cluster monitoring with the help of Prometheus Operator with Yandex Monitoring support

Managed Service for Kubernetes enables you to export cluster object metrics to monitoring systems.

In this article, you will learn how to configure [Yandex Managed Service for Kubernetes cluster](../../managed-kubernetes/concepts/index.md#kubernetes-cluster) monitoring with the help of Prometheus Operator with Yandex Monitoring support.

Prometheus Operator with Monitoring support streamlines the installation of Prometheus Operator, a popular solution for Kubernetes cluster monitoring, providing integration with Yandex Monitoring. You can use it to collect, store, and read metrics from containers, applications, and the infrastructure. The system leverages the Prometheus data model and [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/).

To set up monitoring for your Managed Service for Kubernetes cluster:

1. Install Prometheus Operator using a [Helm chart](#helm-install) or [Yandex Cloud Marketplace](#marketplace-install).
1. [Set up and test alerts](#set-alerting).
1. [Visualize metrics](#check-metrics).

If you no longer need the resources you created, [delete them](#clear-out).


## Required paid resources {#paid-resources}

The support cost for this solution includes:

* Fee for the Managed Service for Kubernetes master: (see [Managed Service for Kubernetes pricing](../../managed-kubernetes/pricing.md)).
* Fee for Managed Service for Kubernetes cluster nodes: specifically using computing resources and storage (see [Yandex Compute Cloud pricing](../../compute/pricing.md)).
* Fee for public IP addresses for Managed Service for Kubernetes cluster hosts and Managed Service for Kubernetes cluster nodes with public access enabled (see [Yandex Virtual Private Cloud pricing](../../vpc/pricing.md#prices-public-ip)).


## Getting started {#before-you-begin}

1. [Create security groups](../../managed-kubernetes/operations/connect/security-groups.md) for the Managed Service for Kubernetes cluster and its node groups.

    {% note warning %}
    
    The configuration of security groups determines performance and availability of the cluster and the services and applications running in it.
    
    {% endnote %}

1. [Create a Managed Service for Kubernetes cluster](../../managed-kubernetes/operations/kubernetes-cluster/kubernetes-cluster-create.md) and [node group](../../managed-kubernetes/operations/node-group/node-group-create.md) in any suitable configuration with internet access and the security groups you set up earlier.

1. [Install kubect](https://kubernetes.io/docs/tasks/tools/install-kubectl) and [configure it to work with the new cluster](../../managed-kubernetes/operations/connect/index.md#kubectl-connect).

1. [Create a service account](../../iam/operations/sa/create.md) with the [monitoring.editor](../../monitoring/security/index.md#monitoring-editor) role.

1. [Create an API key](../../iam/operations/authentication/manage-api-keys.md#create-api-key) for the service account:

   * If you want to install Prometheus Operator using a [Helm chart](#helm-install), run the following command and save the `secret` key you get:

      ```bash
      yc iam api-key create --service-account-name=<service_account_name>
      ```

   * If you want to install Prometheus Operator using [Cloud Marketplace](#marketplace-install), create an API key in JSON format and save it to the `sa-key.json` file:

     ```bash
     yc iam api-key create \
        --service-account-name=<service_account_name> \
        --format=json | jq -c > sa-key.json
     ```

      {% note tip %}

      You can also automatically create a new API key during the app installation.

      {% endnote %}

1. Create a Prometheus workspace:

   {% list tabs group=instructions %}

   - Management console {#console}

      1. Open the Monitoring [home page](https://monitoring.yandex.cloud).
      1. In the left-hand panel, select **Prometheus**.
      1. Click **Create workspace**.

   {% endlist %}

## Install Prometheus Operator using a Helm chart {#helm-install}

1. [Install Helm](https://helm.sh/docs/intro/install/) v3.8.0 or higher.

1. Get the Prometheus workspace ID:

   {% list tabs group=instructions %}

   - Management console {#console}

      1. Open the Monitoring [home page](https://monitoring.yandex.cloud).
      1. Select **Prometheus** on the left.
      1. Select a Prometheus workspace.
      1. This will open the **General information** tab displaying the workspace ID. Save the ID.

   {% endlist %}

1. To install a [Helm chart](https://helm.sh/docs/topics/charts/) with Prometheus Operator, run this command:

   ```bash
   helm pull oci://cr.yandex/yc-marketplace/yandex-cloud/prometheus/kube-prometheus-stack \
     --version 72.6.2-1 \
     --untar && \
   helm install \
     --namespace <namespace_for_Prometheus_Operator> \
     --create-namespace \
     --set prometheusWorkspaceId=<Prometheus_workspace_ID> \
     --set iam_api_key_value_generated.secretAccessKey=<secret_part_of_API_key> \
     prometheus ./kube-prometheus-stack/
   ```

   This command will also create a new namespace for Prometheus Operator.

   If you set `namespace` to the default namespace, Prometheus Operator may work incorrectly. Use a value different from all the existing namespaces, e.g., `prometheus-operator-space`.

   {% note info %}
   
   If you are using a Helm version below 3.8.0, add the `export HELM_EXPERIMENTAL_OCI=1 && \` string at the beginning of the command to enable [Open Container Initiative](https://opencontainers.org/) (OCI) support in the Helm client.
   
   {% endnote %}

1. Make sure the Prometheus Operator pods have changed their status to `Running`:

   ```bash
   kubectl get pods --namespace=<namespace_for_Prometheus_Operator> \
      -l "release=prometheus"
   ```

## Install Prometheus Operator using Cloud Marketplace {#marketplace-install}

{% list tabs group=instructions %}

- Management console {#console}

   1. In the [management console](https://console.yandex.cloud), select a folder.
   1. Navigate to **Managed Service for&nbsp;Kubernetes**.
   1. Click the name of the Kubernetes cluster you need and select the ![image](../../_assets/marketplace.svg) **Marketplace** tab.
   1. Under **Application available for installation**, select [Prometheus Operator with Monitoring](https://yandex.cloud/en/marketplace/products/yc/prometheus-operator) support and click **Go to install**.
   1. Configure the application:

      * **Namespace**: Create a new [namespace](../../managed-kubernetes/concepts/index.md#namespace), e.g., `prometheus-operator-space`. If you leave the default namespace, Prometheus Operator may work incorrectly.
      * **Application name**: Specify the application name.
      * **Prometheus Workspace**: Select a Prometheus workspace.
      * **API key**: Specify the contents of the `sa-key.json` file.

   1. Click **Install**.
   1. Wait for the application to change its status to `Deployed`.  

{% endlist %}

## Configure alerts based on metrics {#set-alerting}

1. [Create](../../monitoring/operations/alert/create-channel.md) a notification channel by selecting an available [sending method](../../monitoring/operations/prometheus/alerting-rules.md#rule-processing).
1. Set up the [Alert Manager](https://prometheus.io/docs/alerting/latest/alertmanager/) responsible for sending notifications:

    1. Open the Monitoring [home page](https://monitoring.yandex.cloud).
    1. Select **Prometheus** on the left.
    1. Select a Prometheus workspace.
    1. Navigate to the **Alert manager configuration** tab.
    1. Click **Upload a configuration file** and upload the [configuration](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) file in `YAML` format.

        {% cut "Sample configuration file" %}

        ```yml
        global:
          resolve_timeout: 5m
        route:
          receiver: telegram
          routes:
            - receiver: "null"
              matchers:
                - severity="none"
              continue: false

        receivers:
          - name: telegram
            yandex_monitoring_configs:
              - channel_names:
                - "<notification_channel_name>"
          - name: "null"
            yandex_monitoring_configs:
              - channel_names: []
        ```

        {% endcut %}

1. Add a test alerting rule:

    ```bash
    cat <<EOF | kubectl apply -f -
    apiVersion: monitoring.coreos.com/v1                                                        
    kind: PrometheusRule
    metadata:
      labels:
        release: kube-prometheus-stack
      name: test
    spec:
      groups:
      - name: test
        rules:
          - alert: Test
            annotations:
              description: this is a test alert
              summary: test alert
            expr: vector(1)
            labels:
              severity: warning
    EOF
    ```

1. Wait until you get alerted through the selected notification channel or check the alert status in the management console:

    1. Open the Monitoring [home page](https://monitoring.yandex.cloud).
    1. Select **Prometheus** on the left.
    1. Select a Prometheus workspace.
    1. Navigate to the **Managing rules** tab.
    1. Click the test alert name.
    1. Navigate to the **Alerts** tab.
    1. Make sure the test alert status changed to `FIRING`.

## Visualize metrics {#check-metrics}

Build a chart based on the metrics and add it to the dashboard.

### Build a chart based on metrics {#add-graph}

{% list tabs group=instructions %}

- Management console {#console}

  1. Open the Monitoring [home page](https://monitoring.yandex.cloud).
  1. In the left-hand panel, select **Metrics**.
  1. In the **Data source** list at the top, select **Prometheus**.
  1. Select a workspace.
  1. Enter a query in [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/).
  1. For the chart to display several lines, click **Add query** and enter a new query.
  1. Click **Execute query** to build a chart.

      You can [add](#add-to-dashboard) the chart to the [dashboard](../../monitoring/concepts/visualization/dashboard.md) as a [widget](../../monitoring/concepts/visualization/widget.md).

{% endlist %}

### Add the chart to the dashboard {#add-to-dashboard}

{% list tabs group=instructions %}

- Management console {#console}

   1. [Build a chart](#add-graph).
   1. At the top right, click **Add to dashboard**.
   1. Enter a name for your chart.
   1. Select a cloud and folder.
   1. Select an existing dashboard or create a new one.
   1. Select a chart adding option:
      * **Add**: To add the chart to the dashboard and stay in the **Metrics** section.
      * **Add and edit**: To add the chart to the dashboard and go to the **Dashboards** section. The chart will not be saved in **Metrics**.

{% endlist %}

{% note tip %}

You can [create](../../monitoring/operations/dashboard/create.md) an empty dashboard and then add the charts to it.

{% endnote %}

## Delete the resources you created {#clear-out}

Some resources are not free of charge. Delete the resources you no longer need to avoid paying for them:

1. [Delete the Managed Service for Kubernetes cluster](../../managed-kubernetes/operations/kubernetes-cluster/kubernetes-cluster-delete.md).
1. [Delete](../../vpc/operations/address-delete.md) the [public static IP address](../../vpc/concepts/address.md#public-addresses) for your Managed Service for Kubernetes cluster if you reserved one.