[Yandex Cloud documentation](../../index.md) > [Security in Yandex Cloud](../index.md) > [Best practices for securing your cloud infrastructure](index.md) > Kubernetes security

# Security Kubernetes

The section includes recommendations for Yandex Cloud users on security settings in [Yandex Managed Service for Kubernetes](../../managed-kubernetes/index.md).

## Responsibility {#division-of-responsibility}

The user is responsible for all actions made inside the Kubernetes node. The user is responsible for the security of the nodes and their proper setup in accordance with PCI DSS requirements and other security standards.

Yandex Cloud is responsible for the Kubernetes API security.

The user is responsible for correctly choosing security settings in Managed Service for Kubernetes, including selecting the [channel](../../managed-kubernetes/concepts/release-channels-and-updates.md) and the update schedule.  

## Sensitive data {#critical-data}

To comply with PCI DSS or other security standards when using Managed Service for Kubernetes, do not:

* Use sensitive data in names and descriptions of clusters, node groups, namespaces, services, and pods.
* Use sensitive data in [Kubernetes node labels](../../managed-kubernetes/concepts/index.md#node-labels) and [Yandex Cloud service resource labels](../../resource-manager/concepts/labels.md).
* Use sensitive data in pod manifests.
* Use sensitive data in etcd in clear text.
* Write sensitive data to Managed Service for Kubernetes logs.

## Resource model {#resource-model}

Wherever possible, ensure maximum isolation between resources:

* Use a separate organization for each _big_ project.
* Use a separate cloud for each development team.
* Use a separate Kubernetes cluster located in a separate folder for each service.
* Use a separate namespace for each microservice.

Your clouds must have no shared resources. Cloud members must have access only to their clouds.

Less strong isolation models are also possible, for example:

* Projects are split between different clouds.
* Development teams are assigned independent folders.
* Services have separate Kubernetes clusters.
* Microservices have independent namespaces.

## Managed Service for Kubernetes network security {#network-security}

We do not recommend granting access to the Kubernetes API and node groups from non-trusted networks, e.g., from the internet.
Use firewall protection where needed (for example, [security groups](../../vpc/concepts/security-groups.md)). In the section below, you can find links to instructions on how to set up firewall protection in security groups.

### Segmentation {#segmentation}

#### Cloud level {#cloud-level}

Restrict network access to the Kubernetes API (master) and node groups using [instructions for security groups](../../managed-kubernetes/operations/connect/security-groups.md).

When using an ALB as an [Ingress Gateway](../../application-load-balancer/tools/k8s-ingress-controller/index.md), also complete the following steps:

1. Apply the security group to the ALB.
2. Additionally, apply the security group to the node group:

    * Source type: `<security group applied for the ALB>`.
    * Destination type: `node group`.
    * Port range: 30000-32767.

#### Kubernetes level {#kubernetes-level}

Restrict network access inside Kubernetes using the [Network Policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/).

You can use two network plugins in Yandex Cloud:

* [Calico](../../managed-kubernetes/concepts/network-policy.md#calico): a basic plugin.
* [Cilium CNI](../../managed-kubernetes/concepts/network-policy.md#cilium): an advanced plugin that uses advanced network policies applied [at the L7 layer (REST/HTTP, gRPC and Kafka)](https://docs.cilium.io/en/v1.10/gettingstarted/http/).

We recommend using the `default deny` rules for incoming and outgoing traffic by default with only the relevant traffic allowed.

To generate policies, you can use the Cilium CNI built-in Hubble platform to analyze the traffic manually. Various solutions for automatic generation of network policies are also available on the market.

For useful examples of network policies, see the [repo](https://github.com/ahmetb/kubernetes-network-policy-recipes).

A helpful tool to create both basic and advanced network policies is available [here](https://editor.cilium.io/).

#### Setting up incoming network access {#ingress}

For online endpoints, we recommend allocating an independent Kubernetes cluster or independent node groups (using such mechanisms as: [Taints and Tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/#:~:text=Node%20affinity%20is%20a%20property,onto%20nodes%20with%20matching%20taints) + [Node affinity](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/)). By doing this, you establish a DMZ so that if your nodes are compromised online, your attack surface is small.

To enable incoming network access to your workloads via HTTP/HTTPS, use the [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) resource.

There are at least two controller options that you can use in Yandex Cloud:
- [Yandex Cloud Gwin Controller](../../managed-kubernetes/alb-ref/gwin-index.md).
- [Application Load Balancer Ingress controller](../../application-load-balancer/tools/k8s-ingress-controller/index.md).


#### Restricting access to the metadata of VMs in the node group {#metadata-access-restriction}

For all pods, create a network policy to block network traffic to port 169.254.169.254 or use the default-deny policy from the [example](../../managed-kubernetes/operations/calico.md#enable-isolation). The policy must block workload node group metadata access because these node groups contain sensitive data, such as the token of the service account assigned to the node.

## Managed Service for Kubernetes authentication and access management {#authentication-and-access-control}

The access of IAM accounts to Managed Service for Kubernetes resources is managed at the following levels:

* [Managed Service for Kubernetes service roles](../../managed-kubernetes/security/index.md#yc-api) (access to the Yandex Cloud API): These enable you to control clusters and node groups (for example, create a cluster, create/edit/delete a node group, and so on).
* Service roles to access the Kubernetes API: These let you control cluster resources via the Kubernetes API (for example, perform standard actions with Kubernetes: create, delete, view namespaces, work with pods, deployments, create roles, and so on). Only the basic global roles are available at cluster level: `k8s.cluster-api.cluster-admin`, `k8s.cluster-api.editor`, and `k8s.cluster-api.viewer`.
* Primitive roles: These are global primitive IAM roles that include service roles (for example, the primitive role admin includes both the service administration role and the administrative role to access the Kubernetes API).
* Standard Kubernetes roles: Inside the Kubernetes cluster itself, you can use Kubernetes tools to create both regular roles and cluster roles. Thus you can control IAM accounts in terms of access at the namespace level. To assign IAM roles at the namespace level, you can manually create RoleBinding objects in a relevant namespace stating the cloud user's IAM ID in the **subjects name** field. Here is an example:

    ```
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
    name: iam-user-aje0jndkhkvu04ek #object name RoleBinding
    namespace: micro1-ns 
    roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: ClusterRole
    name: admin
    subjects:
    - kind: User
    name: aje0jndkq855llvu04ek #cloud user ID
    ```

For the Managed Service for Kubernetes cluster to run, you need two service accounts: [the service account of the cluster and the service account of the node group](../../managed-kubernetes/security/index.md#sa-annotation).

![](../../_assets/overview/solution-library-icon.svg)[Example of setting up role models and policies in Managed Service for Kubernetes](https://github.com/yandex-cloud-examples/yc-mk8s-roles-and-policies).

## Managed Service for Kubernetes secure configuration {#secure-config-1}

### Secure configuration {#secure-config-2}

In Managed Service for Kubernetes, the user is fully in control of all node group settings, but only partially in control of the [master](../../managed-kubernetes/concepts/index.md#master) settings. These settings are part of the user's overall cluster security responsibility.

The [CIS Kubernetes Benchmark](https://www.cisecurity.org/benchmark/kubernetes) standard is designed to build a secure Kubernetes configuration, including node configurations.

In Yandex Cloud, the Kubernetes node groups are deployed by default with the configuration that complies with CIS Kubernetes Benchmark.

The [kube-bench](https://github.com/aquasecurity/kube-bench) tool enables you to check whether the node group configuration is compliant with CIS Kubernetes Benchmark. The tool officially supports the Yandex Cloud node groups.

[Here](https://github.com/aquasecurity/kube-bench/blob/main/docs/running.md) you can see the examples of launching kube-bench on the nodes.

In addition, kube-bench supports integration with [Starboard Operator](https://blog.aquasec.com/automate-kubernetes-compliance) — another product for kube-bench automatic launching.

Starboard Operator is a free tool that helps you automate scanning of images for vulnerabilities and checking that the configuration complies with CIS Kubernetes Benchmark.

![](../../_assets/overview/solution-library-icon.svg) [Integration between Starboard and Yandex Container Registry to scan running images](https://github.com/yandex-cloud-examples/yc-cr-trivy-integration)

### Integrity control (FIM — File integrity monitoring) {#fim}

You must control two levels of file integrity in node groups:

* Node OS files, e.g., configuration files.
* Container files, e.g., critical files the user application writes to the [volume](../../managed-kubernetes/concepts/volume.md).

#### Node OS files {#fim-OS-files}

You can use, for example, [Osquery](https://osquery.io/) as an agent installed on the nodes using [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) and uses specific node folders mounted as a volume into the DaemonSet container (redirected file system).

![](../../_assets/overview/solution-library-icon.svg)A comprehensive solution in [Osquery and kubequery in K8s.](https://github.com/yandex-cloud-examples/yc-mk8s-osquery-kubequery)

#### Container files {#fim-container-files}

One of the methods to solve this task:

1. Use [readOnlyRootFilesystem](https://www.thorsten-hans.com/read-only-filesystems-in-docker-and-kubernetes/) in pods.
2. Make sure to mount folders to write the data to as separate volumes: as emptydir or individual disks.

If you mount folders as emptydir, files are stored on the node in the `/var/lib/kubelet/pods/PODUID/volumes/kubernetes.ioempty-dir/VOLUMENAME` folder. To ensure data integrity, you can monitor this folder by Osquery as [node OS files](#fim-OS-files).

In the case of separate disks (not emptydir), you can mount volumes in read mode to the above-mentioned DaemonSet running Osquery.

To control file integrity on the Kubernetes nodes, you can also use the tools listed in [Integrity control](../standard/virtualenv-safe-config.md#integrity-control).

There exist dedicated free solutions for Kubernetes nodes from Google or Argus, including [file-integrity-operator](https://github.com/openshift/file-integrity-operator).

## Data encryption and Managed Service for Kubernetes secret management {#encryption-and-secret-management}

At the Kubernetes etcd level, encrypt secrets using an in-built [mechanism from Yandex Cloud](../../managed-kubernetes/concepts/encryption.md).

We recommend that you use SecretManager solutions to work with Kubernetes secrets. [Yandex Lockbox](../../lockbox/index.md) is such a solution in Yandex Cloud.

 Yandex Lockbox was integrated with Kubernetes using the [External Secrets](https://external-secrets.io/latest/) open-source project. In Cloud Marketplace, the solution is available in the basic simplified scenario: [External Secrets Operator with Yandex Lockbox support](https://yandex.cloud/en/marketplace/products/yc/external-secrets).


Useful instructions on working with External Secrets:

* [Guide](https://external-secrets.io/latest/provider/yandex-lockbox/) on External Secrets and Yandex Lockbox from the project description.
* [Guide](../../lockbox/tutorials/kubernetes-lockbox-secrets.md) on External Secrets and Yandex Lockbox from the Yandex Cloud documentation.


Many methods to differentiate access to secrets using this tool have been [described](https://external-secrets.io/latest/guides/multi-tenancy/#eso-as-a-service).

The most secure recommended option for encrypting secrets is ESO as a Service (External Secrets Operator as a service). In this case, the global administrator has access to the namespace where ESO is installed, and administrators of individual namespaces create their own [`SecretStore`](https://external-secrets.io/latest/api/secretstore/) objects (where they specify IAM authorized access keys for their Lockbox secrets). If this `SecretStore` object is compromised, the authorized key of only one namespace will be compromised (not all of them, as in the case of Shared ClusterSecretStore).

### Encryption in transit {#encryption-in-transist}

For in-transit encryption, use TLS interaction between pods. If TLS cannot be used, use the service mesh solutions:

* [Istio](https://istio.io/)
* [Linkerd](https://linkerd.io/)

### At-rest encryption {#encryption-at-rest}

If you need to encrypt your stored data, you can use:

* Key Management Service, for encrypting data at the application level (including when you use persistent volumes).
* A custom method of data encryption. However, in this case, protection of the keys and of the key management procedure is the sole responsibility of the user.

## Protection against malicious code in Managed Service for Kubernetes {#malware-protection}

There are two levels where you can enable malicious code protection in Kubernetes:

* Container Registry level protection.
* OS-level protection of Kubernetes nodes.

Security scanner in [Container Registry](../../container-registry/concepts/vulnerability-scanner.md).

To protect the containerization host levels, you can use a variety of paid and free solutions from the "Runtime security" and "Antivirus engine" classes. Examples of free solutions:

* [Kubernetes ClamAV](https://cloud.google.com/community/tutorials/gcp-cos-clamav)
* [Sysdig Falco](https://falco.org/) (it can also function as an Intrusion Detection System)

Be sure to also use the Kubernetes built-in support for [AppArmor](https://kubernetes.io/docs/tutorials/security/apparmor/) and [Seccomp](https://kubernetes.io/docs/tutorials/security/seccomp/).

![](../../_assets/overview/solution-library-icon.svg)[Analyzing Kubernetes security logs in ELK: audit logs, Policy Engine, Falco.](https://github.com/yandex-cloud-examples/yc-export-mk8s-auditlogs-to-elk)

## Managing Managed Service for Kubernetes vulnerabilities {#vulnerability-management}

Yandex Cloud within Managed Service for Kubernetes is in charge of vulnerability management and security updates on the [master](../../managed-kubernetes/concepts/index.md#master). The user must independently control vulnerabilities on the Kubernetes worker nodes.

### Scanning for vulnerabilities {#vulnerability-scanning}

You can break vulnerability scanning into the following levels:

* Image-level vulnerability scanning.
* Vulnerability scanning of Kubernetes node OSes.

Vulnerability scanning at the image level is detailed in [Protection against malicious code in Managed Service for Kubernetes](#malware-protection).

Examples of free universal solutions for node OS vulnerability scanning in Kubernetes are given in [Kubernetes security requirements](../standard/kubernetes-security.md).

There are also paid and free specialized solutions for scanning Kubernetes node OSes and Kubernetes hosts for vulnerabilities, e.g., the free kube-hunter and trivi (scan filesystem).

## Security updates {#security-updates}

Managed Service for Kubernetes issues updates in a regular manner. To meet the Information Security standards:

* Select a relevant update channel and enable either automatic installation of updates, or manual installation immediately after publication in the selected channel.
* Check that the update settings meet the Information Security standards.
* Use one of the three latest Kubernetes versions, because updates (including security updates) are only released for these versions.

## Backup and recovery {#backup-and-restore}

Configure a Managed Service for Kubernetes backup job according to the [guide](../../managed-kubernetes/tutorials/kubernetes-backup.md). When storing your backups in Object Storage, follow recommendations from the [Secure configuration for Object Storage](../standard/virtualenv-safe-config.md#objstorage).

## Security policies in Kubernetes {#kubernetes-security-policies}

Requirements listed in [Pod Security Standards](https://kubernetes.io/docs/concepts/security/pod-security-standards/) from Kubernetes let you prevent threats related to Kubernetes objects.

To implement the requirements, you can either use the built-in Kubernetes [Pod Security Admission Controller](https://kubernetes.io/docs/setup/best-practices/enforcing-pod-security-standards/) tool or open-source software, e.g., other admission controllers: OPA Gatekeeper, [Kyverno](https://yandex.cloud/en/marketplace/products/yc/kyverno).

Examples using Kyverno:

* ![](../../_assets/overview/solution-library-icon.svg)[Analyzing Kubernetes security logs in ELK: audit logs, Policy Engine, Falco.](https://github.com/yandex-cloud-examples/yc-export-mk8s-auditlogs-to-elk)
* ![](../../_assets/overview/solution-library-icon.svg)[Example of setting up role models and policies in Managed Service for Kubernetes.](https://github.com/yandex-cloud-examples/yc-mk8s-roles-and-policies)

To control compliance with Pod Security Standards, you can also use the following tools within CI/CD:

* [Kyverno CLI](https://kyverno.io/docs/kyverno-cli/)
* [The gator CLI](https://open-policy-agent.github.io/gatekeeper/website/docs/gator)

Or a separate [Kubesec](https://kubesec.io/) tool.

## Secure practices of creating and using a Docker image {#best-practices-docker-image}

Use these check lists to meet requirements for secure creation of images:

* [Dockerfile best practices](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/)
* [Kubernetes Security Checklist and Requirements](https://github.com/Vinum-Security/kubernetes-security-checklist/blob/main/README.md)

You can control Dockerfile in your CI/CD pipeline using the [Conftest](https://www.conftest.dev/) utility.

## Runtime protection {#runtime-protection}

When using minimal images or distroless images without a shell, use [ephemeral containers](https://kubernetes.io/docs/concepts/workloads/pods/ephemeral-containers/).

![](../../_assets/overview/solution-library-icon.svg)[Solution running Osquery.](https://github.com/yandex-cloud-examples/yc-mk8s-osquery-kubequery)

## Load sharing between nodes {#load-sharing}

Data loads with different security contexts (such as different severities of data processed) must be processed on different Kubernetes nodes. To enable load sharing within a cluster, use different node groups with different settings for [`node labels`](https://kubernetes.io/docs/tasks/configure-pod-container/assign-pods-nodes/) and [`node taints`](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/). Be sure to combine both settings.

## Collecting, monitoring, and analyzing Managed Service for Kubernetes audit logs {#collection-monitoring-analysis-audit-logs}

Events available to the user in the Managed Service for Kubernetes service can be classified as levels:

* Kubernetes API events (Kubernetes audit logging)
* Kubernetes node events
* Kubernetes pod events
* Kubernetes metrics
* Kubernetes flow logs

### Kubernetes API level (Kubernetes audit logging) {#kubernetes-api-level}

Audit events are collected from the Kubernetes API level by Cloud Logging.

![](../../_assets/overview/solution-library-icon.svg)[Analyzing Kubernetes security logs in ELK: audit logs, Policy Engine, Falco.](https://github.com/yandex-cloud-examples/yc-export-mk8s-auditlogs-to-elk)

### Kubernetes node level {#kubernetes-nodes-level}

Kubernetes node level events are collected and exported similarly to [collecting OS audit logs](../standard/audit-logs.md#os-level).

### Kubernetes pod level {#kubernetes-pods-level}

To learn more about options for collecting and exporting pod-level events in Kubernetes, refer to the [Kubernetes official documentation](https://kubernetes.io/docs/concepts/cluster-administration/logging/).

Examples of collecting and exporting pod logs:

* Exporting logs to Cloud Logging using Fluent Bit is described in the [Managed Service for Kubernetes documentation](../../managed-kubernetes/tutorials/fluent-bit-logging.md).
* Exporting pod logs into Elastic or Splunk is described in the [Yandex Cloud Security Solution Library](https://github.com/yandex-cloud-examples/yc-mk8s-osquery-kubequery/blob/main/README-en.md).

Cloud Marketplace offers the Filebeat plugin for transferring logs to Elastic and [Fluent Bit with the Cloud Logging plugin](https://yandex.cloud/en/marketplace/products/yc/fluent-bit).

### Kubernetes metrics {#kubernetes-metrics}

Monitoring includes a set of metrics to analyze availability of Kubernetes objects and their behavioral anomalies.

For a guide on exporting Monitoring metrics, see the [Exporting events to SIEM](../standard/audit-logs.md#events) section.

### Kubernetes flow logs {#flow-logs-kubernetes}

![](../../_assets/overview/solution-library-icon.svg)[Exporting flow logs to Yandex Object Storage.](https://github.com/yandex-cloud-examples/yc-export-mk8s-cilium-logs-to-s3)

### Managed Service for Kubernetes role model audit {#role-model-audit}

In the Managed Service for Kubernetes console, you can audit the current role model used in the service. For this, go to the **Access management** tab in the service.

You can also use:
* [KubiScan](https://github.com/cyberark/KubiScan)
* [Krane](https://github.com/appvia/krane)