[Yandex Cloud documentation](../../index.md) > [Monium](../index.md) > [Data delivery](index.md) > Otel Collector > Example of Nginx monitoring in Kubernetes # Setting up Nginx telemetry collection in Kubernetes You will set up an Nginx server in a Kubernetes cluster and send its metrics and logs to Monium. This guide uses Managed Service for Kubernetes to deploy a cluster, but you can any of your Kubernetes clusters. To set up web server telemetry collection in a cluster: 1. [Configure the Kubernetes](#cluster-settings) cluster. 1. [Set up authentication](#auth-settings): Create a service account and API key to send data to Monium. 1. [Install and configure Nginx](#nginx-install): Deploy a web server with a metric exporter. 1. [Install OpenTelemetry Collector](#install-otel-collector): Set up metric collection and sending. 1. [View metrics in Monium](#view-metrics). 1. [Configure log collection](#logs-settings). 1. [View logs in Monium](#view-logs). 1. [Configure additional metrics based on logs](#metrics-from-logs). 1. [Create a dashboard and alerts](#dashboard-alerts). ## Getting started {#before-you-begin} Sign up for Yandex Cloud and create a [billing account](../../billing/concepts/billing-account.md): 1. Navigate to the [management console](https://console.yandex.cloud) and log in to Yandex Cloud or create a new account. 1. On the **[Yandex Cloud Billing](https://center.yandex.cloud/billing/accounts)** page, make sure you have a billing account linked and it has the `ACTIVE` or `TRIAL_ACTIVE` [status](../../billing/concepts/billing-account-statuses.md). If you do not have a billing account, [create one](../../billing/quickstart/index.md) and [link](../../billing/operations/pin-cloud.md) a cloud to it. If you have an active billing account, you can create or select a [folder](../../resource-manager/concepts/resources-hierarchy.md#folder) for your infrastructure on the [cloud page](https://console.yandex.cloud/cloud). [Learn more about clouds and folders here](../../resource-manager/concepts/resources-hierarchy.md). ### Required paid resources {#paid-resources} The cost of resources you need for Monium includes: * Fee for using a [Managed Service for Kubernetes](../../managed-kubernetes/concepts/index.md#master) master (see [Managed Service for Kubernetes pricing](../../managed-kubernetes/pricing.md)). * Fee for the [Managed Service for Kubernetes](../../managed-kubernetes/concepts/index.md#node-group) node group's [computing resources](../../compute/concepts/vm-platforms.md) and [disks](../../compute/concepts/disk.md) (see [Yandex Compute Cloud pricing](../../compute/pricing.md)). * Fee for using Monium (see [Monium pricing](../pricing.md)). ## Setting up a cluster {#cluster-settings} {% list tabs group=instructions %} - Management console {#console} 1. Create an [Kubernetes cluster](../../managed-kubernetes/quickstart.md). 1. [Install kubect](https://kubernetes.io/docs/tasks/tools/install-kubectl) and [configure it to work with the new cluster](../../managed-kubernetes/operations/connect/index.md#kubectl-connect). {% endlist %} ## Setting up authentication {#auth} In this step, you need to get and save the API key and folder ID to use them to [configure OpenTelemetry Collector](#install-otel-collector). {% list tabs group=instructions %} - Management console {#console} 1. Create a [service account](../../iam/operations/sa/create.md) with the `monium.telemetry.writer` role. 1. Navigate to **Identity and Access Management**. 1. Click **Create service account**. 1. Name your service account, e.g., `monium-ca`. 1. Click ![image](../../_assets/console-icons/plus.svg) **Add role** and select `monium.telemetry.writer`. 1. Click **Create**. 1. Create an [API key](../../iam/operations/authentication/manage-api-keys.md) with the `yc.monium.telemetry.write` scope: 1. Select the service account you created from the list. 1. Click ![image](../../_assets/console-icons/plus.svg) **Create new key** and select **Create API key**. 1. In the **Scope** field, select `yc.monium.telemetry.write`. 1. Click **Create**. 1. Copy the API key and save it to a secure location as you will need it later. 1. Copy the ID of the folder the cluster and service account were created in. To do this, click the folder name at the top and click ![image](../../_assets/console-icons/ellipsis.svg) → ![image](../../_assets/console-icons/copy.svg) **Copy ID** next to the folder in the list. {% endlist %} ## Installing and configuring Nginx {#nginx-install} 1. Create a file with the web server configuration and Nginx HTML page: {% cut "configmap.yaml" %} ```yaml apiVersion: v1 kind: ConfigMap metadata: name: nginx-config namespace: nginx-demo data: nginx.conf: | server { listen 80; root /usr/share/nginx/html; index index.html; ssi on; ssi_types text/html; ssi_last_modified on; location / { try_files $uri $uri/ =404; } location /nginx_status { stub_status; access_log off; allow 127.0.0.1; deny all; } } index.html: | Nginx Demo

🚀 Nginx is serving this page

This is a simple demo page served from a Kubernetes pod.

Status: OK

Endpoint: /

Static HTML from /usr/share/nginx/html/index.html

Metrics: /metrics (exporter)

Nginx stub status: /nginx_status (localhost-only)

Pod (hostname):

Pod IP:

Client IP:

Request time:

Refresh the page to see load balancing between pods.

If you see this page via LoadBalancer/Ingress — publishing works.

``` {% endcut %} 1. Create a file to manage the running and scaling of Nginx pods. {% cut "deployment.yaml" %} ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-server namespace: nginx-demo labels: app: nginx spec: replicas: 3 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:stable ports: - name: http containerPort: 80 volumeMounts: - name: nginx-config mountPath: /etc/nginx/conf.d/default.conf subPath: nginx.conf - name: nginx-config mountPath: /usr/share/nginx/html/index.html subPath: index.html - name: nginx-exporter image: nginx/nginx-prometheus-exporter:0.10.0 args: - -nginx.scrape-uri=http://localhost/nginx_status ports: - name: metrics containerPort: 9113 resources: limits: memory: 128Mi cpu: 500m volumes: - name: nginx-config configMap: name: nginx-config ``` {% endcut %} To monitor Nginx in Kubernetes, you will need `nginx-exporter` to convert Nginx internal statistics into OpenTelemetry Collector format. In the `deployment.yaml` file, the exporter is added as a sidecar: ```yaml - name: nginx-exporter image: nginx/nginx-prometheus-exporter:0.10.0 args: - -nginx.scrape-uri=http://localhost/nginx_status ports: - name: metrics containerPort: 9113 resources: limits: memory: 128Mi cpu: 500m ``` Each Nginx app pod will have two containers, `nginx` and `nginx-exporter`. 1. Create a file with Nginx access credentials: {% cut "services.yaml" %} ```yaml apiVersion: v1 kind: Service metadata: name: nginx-web namespace: nginx-demo labels: app: nginx spec: externalTrafficPolicy: Local type: LoadBalancer selector: app: nginx ports: - name: http port: 80 targetPort: 80 --- apiVersion: v1 kind: Service metadata: name: nginx-metrics namespace: nginx-demo labels: app: nginx spec: selector: app: nginx ports: - name: metrics port: 9113 targetPort: metrics ``` {% endcut %} 1. Deploy Nginx: ```bash kubectl create namespace nginx-demo kubectl apply -f configmap.yaml kubectl apply -f deployment.yaml kubectl apply -f services.yaml ``` 1. Make sure the application is running: ```bash kubectl get service -n nginx-demo nginx-web ``` Result: ```bash NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE nginx-web LoadBalancer 10.96.238.139 158.260.329.4 80:32761/TCP 104s ``` 1. In your browser, open the address from the `EXTERNAL-IP` column, e.g., `http://158.260.329.4`. You should see the HTML page from `configmap.yaml`. {% cut "Nginx web server page" %} ![image](../../_assets/monium/nginx-page.png) {% endcut %} ## Installing and configuring OpenTelemetry Collector {#install-otel-collector} In this step, you will install [OpenTelemetry Collector Contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib), an extended collector version with additional components for log collection, parsing, and other tasks. The [OpenTelemetry Collector](https://github.com/open-telemetry/opentelemetry-collector) basic version is enough for metric collection, but to work with logs, you need the Contrib version, which is why we will use it from the beginning. 1. Create a file to install OpenTelemetry Collector and send metrics to Monium: {% cut "otel-config.yaml" %} ```yaml --- apiVersion: v1 kind: Secret metadata: name: monium-secrets namespace: nginx-demo stringData: MONIUM_API_KEY: '' --- apiVersion: v1 kind: ConfigMap metadata: name: otel-collector-conf namespace: nginx-demo labels: app: opentelemetry component: otel-collector-conf data: otel-collector-config: | receivers: prometheus: config: scrape_configs: - job_name: "nginx" scrape_interval: 15s kubernetes_sd_configs: - role: pod namespaces: names: - nginx-demo relabel_configs: - source_labels: [__meta_kubernetes_namespace] action: keep regex: nginx-demo - source_labels: [__meta_kubernetes_pod_label_app] action: keep regex: nginx - source_labels: [__meta_kubernetes_pod_container_port_name] action: keep regex: metrics - source_labels: [__meta_kubernetes_pod_name] target_label: pod - source_labels: [__meta_kubernetes_namespace] target_label: namespace processors: batch: timeout: 5s send_batch_size: 1024 memory_limiter: limit_mib: 1500 spike_limit_mib: 512 check_interval: 5s resource: attributes: - key: service.name value: nginx action: upsert - key: service.namespace value: nginx-demo action: upsert exporters: otlp/monium: compression: none endpoint: ingest.monium.yandex.cloud:443 headers: Authorization: Api-Key ${env:MONIUM_API_KEY} x-monium-project: folder__ debug: verbosity: basic extensions: zpages: {} service: extensions: [zpages] pipelines: metrics: receivers: [prometheus] processors: [memory_limiter, resource, batch] exporters: [otlp/monium, debug] --- apiVersion: v1 kind: Service metadata: name: otel-collector namespace: nginx-demo labels: app: opentelemetry component: otel-collector spec: ports: - name: otlp-grpc port: 4317 protocol: TCP targetPort: 4317 - name: otlp-http port: 4318 protocol: TCP targetPort: 4318 - name: metrics port: 8888 selector: component: otel-collector --- apiVersion: apps/v1 kind: Deployment metadata: name: otel-collector namespace: nginx-demo labels: app: opentelemetry component: otel-collector spec: selector: matchLabels: app: opentelemetry component: otel-collector replicas: 1 template: metadata: labels: app: opentelemetry component: otel-collector spec: serviceAccountName: otel-collector containers: - command: - "/otelcol-contrib" - "--config=/conf/otel-collector-config.yaml" image: otel/opentelemetry-collector-contrib:latest name: otel-collector resources: limits: cpu: 1 memory: 2Gi requests: cpu: 200m memory: 400Mi ports: - containerPort: 55679 - containerPort: 4317 - containerPort: 4318 - containerPort: 8888 env: - name: MY_POD_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: GOMEMLIMIT value: 1600MiB - name: MONIUM_API_KEY valueFrom: secretKeyRef: name: monium-secrets key: MONIUM_API_KEY volumeMounts: - name: otel-collector-config-vol mountPath: /conf volumes: - name: otel-collector-config-vol configMap: name: otel-collector-conf items: - key: otel-collector-config path: otel-collector-config.yaml --- apiVersion: v1 kind: ServiceAccount metadata: name: otel-collector namespace: nginx-demo --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: otel-collector-discovery namespace: nginx-demo rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "list", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: otel-collector-discovery namespace: nginx-demo subjects: - kind: ServiceAccount name: otel-collector namespace: nginx-demo roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: otel-collector-discovery ``` {% endcut %} In this file, provide the `` and `` values you saved when [setting up authentication](#auth-settings). 1. Install OTel Collector and start sending metrics: ```bash kubectl apply -f otel-config.yaml ``` 1. Make sure there is a new pod named `otel-collector`: ```bash kubectl get pods --namespace nginx-demo ``` Result: ```bash NAME READY STATUS RESTARTS AGE nginx-server-949d9f98b-kzlrd 2/2 Running 0 2d2h nginx-server-949d9f98b-ngtsp 2/2 Running 0 2d2h nginx-server-949d9f98b-tclvh 2/2 Running 0 2d2h otel-collector-6cd848c59d-k6g2p 1/1 Running 0 12s ``` ## Viewing metrics in Monium {#view-metrics} {% list tabs group=instructions %} - Monium UI {#console} 1. On the [Monium](https://monium.yandex.cloud) home page, select **Metrics** on the left. 1. In the query string, select the following one by one * `project=folder__` * `cluster=default` * `service=nginx` * `name=nginx_up` You can use [text mode](../concepts/visualization/query-string.md#query-text) for the query. To do this, click ![image](../../_assets/console-icons/code.svg) and enter your query as text: ```text `project=folder__; cluster=default; service=nginx; name=nginx_up`. ``` This metric shows the Nginx availability status: `1` for available, `0` for unavailable. The aggregate value of `3` indicates the number of pods. 1. Click **Execute query**. {% endlist %} More on [metrics](../metrics/metric-explorer.md). If there is no data in Monium, see [Data delivery troubleshooting](troubleshooting.md). By combining various metrics in queries, you can check if the web server is handling the current load effectively, detect connection drops caused by insufficient resources, and estimate query processing speeds. ### Number of active connections {#active-connections} To estimate the number of active connections, specify the following value for the `name` metric: * `nginx_connections_accepted`: Number of accepted connections since Nginx startup. It is used to monitor the general load. * `nginx_connections_reading`: Number of connections Nginx reads client requests from. Shows how active incoming requests are. * `nginx_connections_writing`: Number of connections where Nginx sends responses to clients. Shows how active outgoing requests are. ### Number of unprocessed connections {#unprocessed-connections} To estimate the number of connections unprocessed by Nginx, use these metrics: * `nginx_connections_accepted`: Number of accepted connections. * `nginx_connections_handled`: Number of processed connections. The difference between processed and accepted connections will be the number of connections that were accepted but not correctly processed by Nginx. Create three queries: `Query A` for `nginx_connections_accepted`, `Query B` for `nginx_connections_handled`, and `Query C` to calculate the difference between these metrics. To create an extra query, click **Add query**. * Query A: ```text `project=folder__; cluster=default; service=nginx; name=nginx_connections_accepted`. ``` * Query B: ```text `project=folder__; cluster=default; service=nginx; name=nginx_connections_handled`. ``` * Query C: enter `A - B` in the query string. {% cut "Chart displaying unprocessed connections" %} ![image](../../_assets/monium/nginx-unprocessed.png) {% endcut %} To display only the final result, click ![image](../../_assets/console-icons/eye.svg) next to queries `A` and `B` to hide them from the chart. To show the hidden charts, click ![image](../../_assets/console-icons/eye-slash.svg). In the test example, there are no unprocessed connections, so all query `C` values are null. In real systems, you can visualize these values as the `(A - B) / A` ratio, i.e., the share of connections accepted by Nginx but not processed. You can create an [alert](../concepts/alerting/alert.md) for this indicator to notify you, e.g., when the share of dropped connections exceeds `5%`. ## Configuring log collection {#logs-settings} 1. Create a new configuration file for log collection: {% cut "otel-config-logs.yaml" %} ```yaml --- apiVersion: v1 kind: Secret metadata: name: monium-secrets namespace: nginx-demo stringData: MONIUM_API_KEY: '' --- apiVersion: v1 kind: ConfigMap metadata: name: otel-collector-conf namespace: nginx-demo labels: app: opentelemetry component: otel-collector-conf data: otel-collector-config: | receivers: prometheus: config: scrape_configs: - job_name: "nginx" scrape_interval: 15s kubernetes_sd_configs: - role: pod namespaces: names: - nginx-demo relabel_configs: - source_labels: [__meta_kubernetes_namespace] action: keep regex: nginx-demo - source_labels: [__meta_kubernetes_pod_label_app] action: keep regex: nginx - source_labels: [__meta_kubernetes_pod_container_port_name] action: keep regex: metrics - source_labels: [__meta_kubernetes_pod_name] target_label: pod - source_labels: [__meta_kubernetes_namespace] target_label: namespace filelog: include: - /var/log/pods/nginx-demo_nginx-server-*/nginx/*.log operators: - type: container id: container-parser processors: batch: timeout: 5s send_batch_size: 1024 memory_limiter: limit_mib: 1500 spike_limit_mib: 512 check_interval: 5s resource: attributes: - key: service.name value: nginx action: upsert - key: service.namespace value: nginx-demo action: upsert exporters: otlp/monium: compression: none endpoint: ingest.monium.yandex.cloud:443 headers: Authorization: Api-Key ${env:MONIUM_API_KEY} x-monium-project: folder__ debug: verbosity: basic extensions: zpages: {} service: extensions: [zpages] pipelines: metrics: receivers: [prometheus] processors: [memory_limiter, resource, batch] exporters: [otlp/monium, debug] logs: receivers: [filelog] processors: [memory_limiter, resource, batch] exporters: [otlp/monium, debug] --- apiVersion: v1 kind: Service metadata: name: otel-collector namespace: nginx-demo labels: app: opentelemetry component: otel-collector spec: ports: - name: otlp-grpc port: 4317 protocol: TCP targetPort: 4317 - name: otlp-http port: 4318 protocol: TCP targetPort: 4318 - name: metrics port: 8888 selector: component: otel-collector --- apiVersion: apps/v1 kind: DaemonSet metadata: name: otel-collector namespace: nginx-demo labels: app: opentelemetry component: otel-collector spec: selector: matchLabels: app: opentelemetry component: otel-collector template: metadata: labels: app: opentelemetry component: otel-collector spec: serviceAccountName: otel-collector containers: - command: - "/otelcol-contrib" - "--config=/conf/otel-collector-config.yaml" image: otel/opentelemetry-collector-contrib:latest name: otel-collector resources: limits: cpu: 1 memory: 2Gi requests: cpu: 200m memory: 400Mi ports: - containerPort: 55679 - containerPort: 4317 - containerPort: 4318 - containerPort: 8888 env: - name: MY_POD_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: GOMEMLIMIT value: 1600MiB - name: MONIUM_API_KEY valueFrom: secretKeyRef: name: monium-secrets key: MONIUM_API_KEY volumeMounts: - name: otel-collector-config-vol mountPath: /conf - name: varlogpods mountPath: /var/log/pods readOnly: true volumes: - name: otel-collector-config-vol configMap: name: otel-collector-conf items: - key: otel-collector-config path: otel-collector-config.yaml - name: varlogpods hostPath: path: /var/log/pods --- apiVersion: v1 kind: ServiceAccount metadata: name: otel-collector namespace: nginx-demo --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: otel-collector-discovery namespace: nginx-demo rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "list", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: otel-collector-discovery namespace: nginx-demo subjects: - kind: ServiceAccount name: otel-collector namespace: nginx-demo roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: otel-collector-discovery ``` {% endcut %} This file contains a receiver section: ```yaml filelog: include: - /var/log/pods/nginx-demo_nginx-server-*/nginx/*.log operators: - type: container id: container-parser ``` As well as the log collection pipeline (`pipeline`) and mount parameters (`volumeMounts`, `volumes`) for accessing the pod logs. 1. Delete the previous configuration file and apply the new one: ```bash kubectl delete -f otel-config.yaml kubectl apply -f otel-config-logs.yaml ``` ## Viewing logs in Monium {#view-logs} {% list tabs group=instructions %} - Monium UI {#console} 1. On the [Monium](https://monium.yandex.cloud) home page, select **Logs** on the left. 1. In the query string, specify the following: * `project=folder__` * `service=nginx` 1. Click **Execute query**. {% cut "Page example with Nginx logs" %} ![image](../../_assets/monium/nginx-logs.png) {% endcut %} {% endlist %} More on [logs](../logs/logs-explorer.md). ## Setting up metric collection from logs {#metrics-from-logs} Apart from the standard metrics provided by Nginx via the Prometheus exporter, OTel Collector enables you to retrieve additional metrics from application logs. For example, you can track the numbers of HTTP responses with different status codes. To get these metrics, create and apply a new configuration file: {% cut "otel-config-metrics.yaml" %} ```yaml --- apiVersion: v1 kind: Secret metadata: name: monium-secrets namespace: nginx-demo stringData: MONIUM_API_KEY: "" --- apiVersion: v1 kind: ConfigMap metadata: name: otel-collector-conf namespace: nginx-demo labels: app: opentelemetry component: otel-collector-conf data: otel-collector-config: | receivers: prometheus: config: scrape_configs: - job_name: "nginx" scrape_interval: 15s kubernetes_sd_configs: - role: pod namespaces: names: - nginx-demo relabel_configs: - source_labels: [__meta_kubernetes_namespace] action: keep regex: nginx-demo - source_labels: [__meta_kubernetes_pod_label_app] action: keep regex: nginx - source_labels: [__meta_kubernetes_pod_container_port_name] action: keep regex: metrics - source_labels: [__meta_kubernetes_pod_name] target_label: pod - source_labels: [__meta_kubernetes_namespace] target_label: namespace filelog: include: - /var/log/pods/nginx-demo_nginx-server-*/nginx/*.log operators: - type: container id: container-parser - type: regex_parser id: nginx-access parse_from: body regex: '^(?P\S+) - (?P\S+) \[(?P[^\]]+)\] "(?P\S+) (?P\S+) (?P[^"]+)" (?P\d{3}) (?P\d+)' processors: batch: timeout: 5s send_batch_size: 1024 memory_limiter: limit_mib: 1500 spike_limit_mib: 512 check_interval: 5s resource: attributes: - key: service.name value: nginx action: upsert - key: service.namespace value: nginx-demo action: upsert connectors: count/nginx: logs: nginx_http_2xx_total: description: nginx 2xx responses conditions: - 'IsMatch(attributes["status"], "^2..$")' nginx_http_4xx_total: description: nginx 4xx responses conditions: - 'IsMatch(attributes["status"], "^4..$")' nginx_http_5xx_total: description: nginx 5xx responses conditions: - 'IsMatch(attributes["status"], "^5..$")' exporters: otlp/monium: compression: none endpoint: ingest.monium.yandex.cloud:443 headers: Authorization: Api-Key ${env:MONIUM_API_KEY} x-monium-project: folder__ debug: verbosity: basic extensions: zpages: {} service: extensions: [zpages] pipelines: metrics: receivers: [prometheus] processors: [memory_limiter, resource, batch] exporters: [otlp/monium, debug] logs: receivers: [filelog] processors: [memory_limiter, resource, batch] exporters: [count/nginx, otlp/monium, debug] metrics/logs: receivers: [count/nginx] processors: [memory_limiter, resource, batch] exporters: [otlp/monium, debug] --- apiVersion: v1 kind: Service metadata: name: otel-collector namespace: nginx-demo labels: app: opentelemetry component: otel-collector spec: ports: - name: otlp-grpc port: 4317 protocol: TCP targetPort: 4317 - name: otlp-http port: 4318 protocol: TCP targetPort: 4318 - name: metrics port: 8888 selector: component: otel-collector --- apiVersion: apps/v1 kind: DaemonSet metadata: name: otel-collector namespace: nginx-demo labels: app: opentelemetry component: otel-collector spec: selector: matchLabels: app: opentelemetry component: otel-collector template: metadata: labels: app: opentelemetry component: otel-collector spec: serviceAccountName: otel-collector containers: - command: - "/otelcol-contrib" - "--config=/conf/otel-collector-config.yaml" image: otel/opentelemetry-collector-contrib:0.147.0 name: otel-collector resources: limits: cpu: 1 memory: 2Gi requests: cpu: 200m memory: 400Mi ports: - containerPort: 55679 - containerPort: 4317 - containerPort: 4318 - containerPort: 8888 env: - name: MY_POD_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: GOMEMLIMIT value: 1600MiB - name: MONIUM_API_KEY valueFrom: secretKeyRef: name: monium-secrets key: MONIUM_API_KEY volumeMounts: - name: otel-collector-config-vol mountPath: /conf - name: varlogpods mountPath: /var/log/pods readOnly: true volumes: - name: otel-collector-config-vol configMap: name: otel-collector-conf items: - key: otel-collector-config path: otel-collector-config.yaml - name: varlogpods hostPath: path: /var/log/pods --- apiVersion: v1 kind: ServiceAccount metadata: name: otel-collector namespace: nginx-demo --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: otel-collector-discovery namespace: nginx-demo rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "list", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: otel-collector-discovery namespace: nginx-demo subjects: - kind: ServiceAccount name: otel-collector namespace: nginx-demo roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: otel-collector-discovery ``` {% endcut %} This configuration is updated with the following components: 1. Retrieving log data. The `filelog` section includes the `regex_parser` statement to retrieve the relevant fields, e.g., the request method, URL, HTTP response code, etc., from the Nginx access log lines: ```yaml filelog: include: - /var/log/pods/nginx-demo_nginx-server-*/nginx/*.log operators: - type: container id: container-parser - type: regex_parser id: nginx-access parse_from: body regex: '^(?P\S+) - (?P\S+) \[(?P[^\]]+)\] "(?P\S+) (?P\S+) (?P[^"]+)" (?P\d{3}) (?P\d+)' ``` This statement will get the `status` field (HTTP response code) from each log line. We will use this value to generate metrics. 1. Creating metrics from logs. Added the `count` connector to monitor the number of log entries based on specific conditions. The example below counts responses with `2xx`, `4xx`, and `5xx` codes: ```yaml connectors: count/nginx: logs: nginx_http_2xx_total: description: Number of successful responses (2xx) conditions: - 'IsMatch(attributes["status"], "^2..$")' nginx_http_4xx_total: description: Number of client errors (4xx) conditions: - 'IsMatch(attributes["status"], "^4..$")' nginx_http_5xx_total: description: Number of server errors (5xx) conditions: - 'IsMatch(attributes["status"], "^5..$")' ``` 1. Setting up pipelines. Added the `metrics/logs` pipeline to receive metrics from the `count/nginx` connector and send them to Monium together with other metrics: ```yaml service: extensions: [zpages] pipelines: metrics: receivers: [prometheus] processors: [memory_limiter, resource, batch] exporters: [otlp/monium, debug] logs: receivers: [filelog] processors: [memory_limiter, resource, batch] exporters: [count/nginx, otlp/monium, debug] metrics/logs: receivers: [count/nginx] processors: [memory_limiter, resource, batch] exporters: [otlp/monium, debug] ``` Once you apply the configuration, new metrics will be available in Monium: * `nginx_http_2xx_total`: Number of successful responses. * `nginx_http_4xx_total`: Number of client errors. * `nginx_http_5xx_total`: Number of server errors. {% cut "Example of a page with Nginx metrics collected from logs" %} ![image](../../_assets/monium/nginx-metrics-4xx.png) {% endcut %} These metrics can be used for quality-of-service monitoring and troubleshooting. For example, a spike in `5xx` errors may indicate an application failure. ## Creating a dashboard and alerts {#dashboard-alerts} For continuous Nginx health monitoring, [create a dashboard](../operations/dashboard/create.md) with key metrics and [set up alerts](../operations/alert/create-alert.md) to get notifications about issues. For Nginx, we recommend adding charts with the following metrics to the dashboard: * `nginx_connections_accepted`: Number of accepted connections since startup. * `nginx_connections_active`: Number of current active connections. * `nginx_connections_handled`: Number of processed connections. * `nginx_connections_reading`: Number of connections Nginx reads requests from. * `nginx_connections_waiting`: Number of idle connections. * `nginx_connections_writing`: Number of connections Nginx sends responses to. * `nginx_requests_total`: Total number of processed requests. * `nginx_http_2xx_total`: Number of successful responses (code 2xx). * `nginx_http_4xx_total`: Number of client errors (code 4xx). * `nginx_http_5xx_total`: Number of server errors (code 5xx). To receive notifications about critical situations, set the following alerts: * **High server error rate**: Increase in `nginx_http_5xx_total`. Trigger condition: Value greater than `10` over a five-minute window. * **High client error rate**: Increase in `nginx_http_4xx_total`. Trigger condition: Value greater than `50` over a five-minute window. * [Unprocessed connections](#unprocessed-connections): Difference between accepted and processed connections. Trigger condition: Value greater than `0` over a five-minute window. Adjust these the thresholds and time intervals based on your application’s specific requirements.