[Yandex Cloud documentation](../../index.md) > [Yandex Managed Service for Kubernetes](../index.md) > FAQ > All questions on one page # FAQ about Managed Service for Kubernetes ### General questions {#toc-general} * [What services are available in Managed Service for Kubernetes clusters by default?](#defaults) * [Which Kubernetes CLI (kubectl) version do I need to install for comprehensive cluster management?](#kubectl-version) * [Can Yandex Cloud restore the health of my cluster if I configure it incorrectly?](#tech-support-cases) * [Who will be monitoring the cluster health?](#health-check) * [How quickly does Yandex Cloud address vulnerabilities discovered in the security system? What should I do if an attacker has taken advantage of a vulnerability and my data is compromised?](#security-updates) * [Can I connect to a cluster node via OS Login, similar to a Yandex Cloud VM?](#connect-via-oslogin) * [What operating system is used on cluster nodes?](#cluster-node-os) * [Does Yandex Virtual Private Cloud support IPv6?](#ipv6-support) ### Data storage {#toc-volumes} * [What should I consider when using disk storage for a database, such as MySQL® or PostgreSQL, deployed in a Kubernetes cluster?](#bd) * [How do I connect a pod to managed Yandex Cloud databases?](#mdb) * [How do I correctly attach a persistent volume to a container?](#persistent-volume) * [What types of volumes does Managed Service for Kubernetes support?](#supported-volumes) ### Autoscaling {#toc-autosscaling} * [Why does my cluster have N nodes and is not scaling down?](#not-scaling-down) * [In an autoscaling group, the number of nodes never scales down to one, even when there is no load](#autoscaler-one-node) * [Why does the node group fail to scale down after the pod deletion?](#not-scaling-pod) * [Why does autoscaling fail to trigger even though the number of nodes is below the minimum or above the maximum?](#beyond-limits) * [Why do _Terminated_ pods remain in my cluster?](#terminated-pod) * [Is Horizontal Pod Autoscaler supported?](#horizontal-pod-autoscaler) * [How do I choose the minimum master preset to reduce costs?](#master-preset-cost) * [Can I change the master’s autoscaling thresholds on the user side?](#master-autoscaler-thresholds) * [Can I set an upper limit on the master node's auto-scaling?](#master-autoscaler-max) * [How can I prevent the master resources from scaling down during auto-scaling?](#master-autoscaler-no-scaledown) ### Setup and updates {#toc-settings} * [What should I do if I lose some of my data during a Kubernetes version upgrade?](#backups-update) * [Can I configure a backup for a Kubernetes cluster?](#cluster-backups) * [Will my resources be unavailable while Kubernetes is going through a version upgrade?](#downtime-update) * [Can I upgrade my Managed Service for Kubernetes cluster in one step?](#upgrade-in-one-step) * [Is the Container Network Interface plugin upgraded together with the Managed Service for Kubernetes cluster?](#upgrade-cni) * [Can I send you a YAML configuration file so that you apply it to my cluster?](#configs) * [Can you install Web UI Dashboard, Rook, and other tools?](#install-tools) * [What should I do if volumes fail to attach after a Kubernetes upgrade?](#pvc) ### Resources {#toc-resources} * [What resources do I need to maintain a Kubernetes cluster with a group of, say, three nodes?](#required-resources) * [Can I change resources for each node in a Kubernetes cluster?](#change-resources) * [Who handles Kubernetes cluster scaling?](#scaling) * [Do the Kubernetes cluster nodes need internet access?](#internet-access) ### Logs {#toc-logs} * [How can I monitor the Managed Service for Kubernetes cluster state?](#monitoring) * [Can I access the logs of my activity in Yandex Cloud services?](#logs) * [Can I save logs myself?](#auto-logging) * [Can I use Yandex Cloud Logging for viewing logs?](#master-logging) ### Troubleshooting {#toc-troubleshooting} * [Error creating a cluster in a different folder's cloud network](#neighbour-catalog-permission-denied) * [Namespace fails to delete and remains _Terminating_](#namespace-terminating) * [I am using Yandex Network Load Balancer together with an ingress controller. Why are some of my cluster's nodes _UNHEALTHY_?](#nlb-ingress) * [Why does the newly created PersistentVolumeClaim remain _Pending_?](#pvc-pending) * [Why does my Managed Service for Kubernetes cluster fail to start after I update its node configuration?](#not-starting) * [Error updating ingress controller certificate](#ingress-certificate) * [Why is DNS resolution not working in my cluster?](#not-resolve-dns) * [Creating a node group with the CLI results in a parameter conflict. How do I fix it?](#conflicting-flags) * [Error connecting to a cluster using kubectl](#connect-to-cluster) * [Errors connecting to a node over SSH](#node-connect) * [How do I provide internet access to my Managed Service for Kubernetes cluster nodes?](#internet) * [Why cannot I choose Docker as the container runtime?](#docker-runtime) * [Error connecting a GitLab repository to Argo CD](#argo-cd) * [Traffic loss when deploying app updates in a cluster with Yandex Application Load Balancer](#alb-traffic-lost) * [System time displayed incorrectly in the Linux console, as well as in container and Managed Service for Kubernetes cluster pod logs](#time) * [What should I do if I deleted my Yandex Network Load Balancer or its target groups that were automatically created for a LoadBalancer service?](#deleted-loadbalancer-service) * [Error connecting a Yandex Compute Cloud VM instance as an external Managed Service for Kubernetes node](#vm-as-external-node) ## General questions {#general} #### What services are available in Managed Service for Kubernetes clusters by default? {#defaults} The following services are available by default: * [Metrics Server](https://github.com/kubernetes-sigs/metrics-server) for aggregating data on resource usage in a [Kubernetes cluster](../concepts/index.md#kubernetes-cluster). * [Kubernetes plugin for CoreDNS](https://coredns.io/plugins/kubernetes/) for name resolution in a cluster. * [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) supporting [CSI plugins](https://github.com/container-storage-interface/spec) to work with [persistent volumes](../concepts/volume.md) (`PersistentVolume`). #### Which Kubernetes CLI (kubectl) version do I need to install for comprehensive cluster management? {#kubectl-version} We recommend using the latest official version of [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) to avoid compatibility issues. #### Can Yandex Cloud restore the health of my cluster if I configure it incorrectly? {#tech-support-cases} The [master](../concepts/index.md#master) is fully managed by Yandex Cloud, so you cannot damage it. If you have issues with Kubernetes cluster components, contact [support](https://center.yandex.cloud/support). #### Who will be monitoring the cluster health? {#health-check} Yandex Cloud. We will monitor your cluster for corrupted file system, kernel deadlock, lost internet connection, and Kubernetes component issues. We are also developing a solution to automatically restore faulty components. #### How quickly does Yandex Cloud address vulnerabilities discovered in the security system? What should I do if an attacker has taken advantage of a vulnerability and my data is compromised? {#security-updates} Yandex Cloud services, images and master configuration undergo [various security and compliance checks](../../security/index.md) from the start. Users can set the [update frequency](../concepts/release-channels-and-updates.md#updates) depending on their workloads and cluster configuration. It is important to consider attack vectors and the vulnerabilities of applications deployed in the Kubernetes cluster. Such factors as [network security policies](../concepts/network-policy.md) between applications, Docker container vulnerabilities, and improper container runtime configuration in the cluster may affect application security. #### Can I connect to a cluster node via OS Login, similar to a Yandex Cloud VM? {#connect-via-oslogin} Yes, you can. To do this, [follow this guide](../operations/node-connect-oslogin.md). #### What operating system is used on cluster nodes? {#cluster-node-os} Depending on the [update channel](../concepts/release-channels-and-updates.md) and Kubernetes version, cluster nodes come with pre-installed Ubuntu 20.04 or Ubuntu 22.04. #### Does Yandex Virtual Private Cloud support IPv6? {#ipv6-support} Yandex Virtual Private Cloud [does not support IPv6](../../vpc/concepts/network-overview.md#limits); however, IPv6 is on by default on cluster nodes at the OS level. ## Data storage {#volumes} #### What should I consider when using disk storage for a database, such as MySQL® or PostgreSQL, deployed in a Kubernetes cluster? {#bd} For a database deployed in a Kubernetes cluster, use [StatefulSet](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/) controllers. We do not recommend running stateful services with persistent volumes in Kubernetes. To work with databases of stateful applications, use [Yandex Cloud managed databases](https://yandex.cloud/en/services#data-platform), e.g., Managed Service for MySQL® or Managed Service for PostgreSQL. #### How do I connect a pod to managed Yandex Cloud databases? {#mdb} To connect to a [Yandex Cloud managed database](https://yandex.cloud/en/services#data-platform) in the same [network](../../vpc/concepts/network.md#network), specify its [host name and FQDN](../../compute/concepts/network.md#hostname). To connect a database certificate to a [pod](../concepts/index.md#pod), use the `secret` or `configmap` objects. #### How do I correctly attach a persistent volume to a container? {#persistent-volume} You can choose how to attach Compute Cloud [disks](../../compute/concepts/disk.md) depending on your use case: * If you need Kubernetes to automatically provision a `PersistentVolume` and configure a new disk, create a pod with a [dynamically provisioned](../operations/volumes/dynamic-create-pv.md) volume. * To use existing Compute Cloud volumes, create a pod with a [statically provisioned](../operations/volumes/static-create-pv.md) pod. For more information, see [Working with persistent volumes](../concepts/volume.md#persistent-volume). #### What types of volumes does Managed Service for Kubernetes support? {#supported-volumes} Managed Service for Kubernetes supports temporary (`Volume`) and persistent (`PersistentVolume`) volumes. For more information, see [Volume](../concepts/volume.md). ## Autoscaling {#autosscaling} #### Why does my cluster have N nodes and is not scaling down? {#not-scaling-down} [Autoscaling](../concepts/autoscale.md) does not stop nodes with [pods](../concepts/index.md#pod) that cannot be evicted. The following prevents scaling: * Pods with a [PodDisruptionBudget](../concepts/node-group/node-drain.md) that restricts their eviction. * Pods in the `kube-system` [namespace](../concepts/index.md#namespace): * Those not managed by a [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) controller. * Those without a `PodDisruptionBudget` or those with a `PodDisruptionBudget` restricting their eviction. * Pods not managed by a replication controller, such as [ReplicaSet](https://kubernetes.io/docs/concepts/workloads/controllers/replicaset/), [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/), or [StatefulSet](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/). * Pods with `local-storage`. * Pods that cannot be scheduled anywhere due to restrictions, e.g., due to insufficient resources or lack of nodes matching the [affinity or anti-affinity](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity) selectors. * Pods annotated with `"cluster-autoscaler.kubernetes.io/safe-to-evict": "false"`. {% note info %} You can evict `kube-system` pods, pods with `local-storage`, and pods without a replication controller. To do this, set `"safe-to-evict": "true"`: ```bash kubectl annotate pod cluster-autoscaler.kubernetes.io/safe-to-evict=true ``` {% endnote %} Other possible causes include: * The [node group](../concepts/index.md#node-group) has already reached its minimum size. * The node has been idle for less than 10 minutes. * The node group was scaled up in the last 10 minutes. * There was a failed attempt to scale down the node group in the last three minutes. * There was an unsuccessful attempt to stop a certain node. In this case, the next attempt occurs in 5 minutes. * The node is annotated to prevent it from being stopped during downscaling: `"cluster-autoscaler.kubernetes.io/scale-down-disabled": "true"`. You can add or remove the annotation using `kubectl`. Check the node for annotations: ```bash kubectl describe node | grep scale-down-disabled ``` Result: ```bash Annotations: cluster-autoscaler.kubernetes.io/scale-down-disabled: true ``` Set the annotation: ```bash kubectl annotate node cluster-autoscaler.kubernetes.io/scale-down-disabled=true ``` You can remove the annotation by running the `kubectl` command with `-`: ```bash kubectl annotate node cluster-autoscaler.kubernetes.io/scale-down-disabled- ``` #### In an autoscaling group, the number of nodes never scales down to one, even when there is no load {#autoscaler-one-node} In a Managed Service for Kubernetes cluster, the `kube-dns-autoscaler` app decides on the number of CoreDNS replicas. If the `preventSinglePointFailure` parameter in the `kube-dns-autoscaler` configuration is set to `true` and there is more than one node in the group, the minimum number of CoreDNS replicas is two. In this case, the Cluster Autoscaler cannot scale down the number of nodes in the cluster below that of CoreDNS pods. Learn more about DNS scaling based on the cluster size [here](../../tutorials/container-infrastructure/dns-autoscaler.md). **Solution**: 1. Disable the protection setting that limits the minimum number of CoreDNS replicas to two. To do this, set the `preventSinglePointFailure` parameter to `false` in the `kube-dns-autoscaler` [ConfigMap](https://kubernetes.io/docs/concepts/configuration/configmap/). 1. Enable the `kube-dns-autoscaler` pod eviction by adding the `save-to-evict` annotation to [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/): ```bash kubectl patch deployment kube-dns-autoscaler -n kube-system \ --type merge \ -p '{"spec":{"template":{"metadata":{"annotations":{"cluster-autoscaler.kubernetes.io/safe-to-evict":"true"}}}}}' ``` #### Why does the node group fail to scale down after the pod deletion? {#not-scaling-pod} If a node is underutilized, it will be deleted after 10 minutes. #### Why does autoscaling fail to trigger even though the number of nodes is below the minimum or above the maximum? {#beyond-limits} Autoscaling will not violate the preset limits, but Managed Service for Kubernetes does not explicitly enforce the limits. Upscaling will only happen if there are `unschedulable` pods. #### Why do _Terminated_ pods remain in my cluster? {#terminated-pod} This happens because the [Pod garbage collector (PodGC)](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-garbage-collection) fails to timely clean up these pods during autoscaling. For more information, see [Deleting terminated pods](../operations/autoscale.md#delete-terminated). To get answers to other questions about autoscaling, see [Kubernetes FAQ](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#table-of-contents). #### Is Horizontal Pod Autoscaler supported? {#horizontal-pod-autoscaler} Yes, Managed Service for Kubernetes supports [Horizontal Pod Autoscaler](../concepts/autoscale.md#hpa). #### How do I choose the minimum master preset to reduce costs? {#master-preset-cost} Choose the master configuration that matches the real cluster load. You may want to check out our [recommended configuration](../concepts/master-configuration.md) options that depend on the number of nodes, maximum number of pods, and CNI in use. #### Can I change the master’s autoscaling thresholds on the user side? {#master-autoscaler-thresholds} No, you cannot. The thresholds con only be managed on the Managed Service for Kubernetes side. If you have any feedback on how autoscaler works, please contact our [support](https://center.yandex.cloud/support). You can indirectly impact the autoscaler’s behavior by selecting the master configuration, since Master Autoscaler will not scale the resources down below the specified configuration. #### Can I set an upper limit on the master node's auto-scaling? {#master-autoscaler-max} No, you cannot specify maximum values for auto-scaling. #### How can I prevent the master node from scaling down during auto-scaling? {#master-autoscaler-no-scaledown} Choose the master configuration that [best suits your load](../concepts/master-configuration.md). Master Autoscaler does not scale the resources down below the specified configuration. ## Setup and updates {#settings} #### What should I do if I lose some of my data during a Kubernetes version upgrade? {#backups-update} Your data will not get lost as Managed Service for Kubernetes creates a data backup prior to [upgrading the Kubernetes version](../concepts/release-channels-and-updates.md). You can manually configure [cluster backup in Yandex Object Storage](../tutorials/kubernetes-backup.md). We also recommend using the application's native features to back up your databases. #### Can I configure a backup for a Kubernetes cluster? {#cluster-backups} Yandex Cloud provides secure storage and replication for data in [Managed Service for Kubernetes clusters](../concepts/index.md#kubernetes-cluster). However, you can back up data from your [Managed Service for Kubernetes cluster node groups](../concepts/index.md#node-group) at any time and store it in [Object Storage](../../storage/index.md) or other types of storage. For more information, see [Backing up a Managed Service for Kubernetes cluster to Object Storage](../tutorials/kubernetes-backup.md). #### Will my resources be unavailable while Kubernetes is going through a version upgrade? {#downtime-update} When a [master](../concepts/index.md#master) is going through an upgrade, control plane resources will experience downtime. For this reason, operations like [creating](../operations/node-group/node-group-create.md) or [deleting](../operations/node-group/node-group-delete.md) a [Managed Service for Kubernetes node group](../concepts/index.md#node-group) will be unavailable. Still, the application will continue serving user requests. If `max_expansion` is greater than zero, Managed Service for Kubernetes creates new nodes when upgrading node groups. The system moves all workloads to the new nodes and deletes the old node groups. The downtime in this case equals the time it takes for a [pod](../concepts/index.md#pod) to restart when moved to the new Managed Service for Kubernetes node group. #### Can I upgrade my Managed Service for Kubernetes cluster in one step? {#upgrade-in-one-step} It depends on the source and target versions of the Managed Service for Kubernetes cluster upgrade. In one step, you can only upgrade your Managed Service for Kubernetes cluster to the next minor version from the current one. Upgrading to newer versions is done in multiple steps, e.g., 1.19 → 1.20 → 1.21. For more information, see [Updating a cluster](../operations/update-kubernetes.md#cluster-upgrade). If you want to skip intermediate versions, [create a Managed Service for Kubernetes cluster](../operations/kubernetes-cluster/kubernetes-cluster-create.md) of the version you need and migrate the workloads from the old cluster to the new one. #### Is the Container Network Interface plugin upgraded together with the Managed Service for Kubernetes cluster? {#upgrade-cni} Yes. If you are using the [Calico](../concepts/network-policy.md#calico) and [Cilium](../concepts/network-policy.md#cilium) controllers, they are upgraded along with your Managed Service for Kubernetes cluster. To upgrade your Managed Service for Kubernetes cluster, do one of the following: * [Create a Managed Service for Kubernetes cluster](../operations/kubernetes-cluster/kubernetes-cluster-create.md) of the version you need and migrate the workloads from the old cluster to the new one. * [Upgrade your Managed Service for Kubernetes cluster manually](../operations/update-kubernetes.md#cluster-manual-upgrade). To get timely Managed Service for Kubernetes cluster version upgrades, [set up auto upgrading](../operations/update-kubernetes.md#cluster-auto-upgrade). #### Can I send you a YAML configuration file so that you apply it to my cluster? {#configs} No. You can use a `kubeconfig` file to apply the YAML file with cluster configuration on your own. #### Can you install Web UI Dashboard, Rook, and other tools? {#install-tools} No. You can install all the required tools on your own. #### What should I do if volumes fail to attach after a Kubernetes upgrade? {#pvc} If you get the following error after a Kubernetes upgrade: ```text AttachVolume.Attach failed for volume "pvc": Attach timeout for volume yadp-k8s-volumes/pvc ``` Update the [s3-CSI driver](https://github.com/ctrox/csi-s3) to the latest version. ## Resources {#resources} #### What resources do I need to maintain a Kubernetes cluster with a group of, say, three nodes? {#required-resources} Each [node](../concepts/index.md#node-group) needs resources to run the Kubernetes components that allow the node to work as part of the [Kubernetes cluster](../concepts/index.md#kubernetes-cluster). For more information, see [Dynamic resource allocation](../concepts/node-group/allocatable-resources.md). #### Can I change resources for each node in a Kubernetes cluster? {#change-resources} You can only change resources for a node group. You can create groups with different configurations in a single Kubernetes cluster and spread them across multiple [availability zones](../../overview/concepts/geo-scope.md). For more information, see [Updating a Managed Service for Kubernetes node group](../operations/node-group/node-group-update.md). #### Who handles Kubernetes cluster scaling? {#scaling} In Managed Service for Kubernetes, you can enable [cluster autoscaling](../concepts/autoscale.md#ca). #### Do the Kubernetes cluster nodes need internet access? {#internet-access} To connect to external resources, e.g., Docker image registries such as [Container Registry](../../container-registry/concepts/index.md), [Cloud Registry](../../cloud-registry/concepts/index.md), or [Docker Hub](https://hub.docker.com/) as well as [Object Storage buckets](../../storage/concepts/bucket.md), the group nodes must have internet access. To enable internet access, [assign](../operations/node-group/node-group-update.md#node-internet-access) a public IP address to the nodes and [configure](../operations/connect/security-groups.md#rules-internal-nodegroup) a security group. As an alternative to public IP addresses, you can also create and configure a [NAT gateway](../../vpc/operations/create-nat-gateway.md) or a [NAT instance](../../vpc/tutorials/nat-instance/index.md) in the subnet hosting the nodes. For more information, see [Internet access for cluster worker nodes](../concepts/network.md#nodes-internet). ## Logs {#logs} #### How can I monitor the Managed Service for Kubernetes cluster state? {#monitoring} [Get cluster statistics](../operations/kubernetes-cluster/kubernetes-cluster-get-stats.md). You can find the description of the available cluster metrics in [this reference](../metrics.md). #### Can I get logs of my operations in Yandex Cloud? {#logs} Yes, you can request information about operations with your resources from Yandex Cloud logs. Do it by contacting [support](https://center.yandex.cloud/support). #### Can I save logs myself? {#auto-logging} For log collection and storage, use [Fluent Bit](../tutorials/fluent-bit-logging.md). #### Can I use Yandex Cloud Logging for viewing logs? {#master-logging} Yes, you can. To do this, set up sending logs to [Cloud Logging](../../logging/index.md) when [creating](../operations/kubernetes-cluster/kubernetes-cluster-create.md) or [updating](../operations/kubernetes-cluster/kubernetes-cluster-update.md) a [Managed Service for Kubernetes cluster](../concepts/index.md#kubernetes-cluster). This setting is only available in the CLI, Terraform, and API. ## Troubleshooting {#troubleshooting} This section describes typical issues you may encounter while using Managed Service for Kubernetes and gives troubleshooting recommendations. #### Error creating a cluster in a different folder's cloud network {#neighbour-catalog-permission-denied} Error message: ```text Permission denied ``` This error occurs when the [resource service account](../security/index.md#sa-annotation) has no required [roles](../../iam/concepts/access-control/roles.md) in the [folder](../../resource-manager/concepts/resources-hierarchy.md#folder) that contains the [cloud network](../../vpc/concepts/network.md#network) selected when creating the cluster. To create a [Managed Service for Kubernetes cluster](../concepts/index.md#kubernetes-cluster) in a cloud network of another folder, [assign](../../iam/operations/sa/assign-role-for-sa.md) the resource [service account](../../iam/concepts/users/service-accounts.md) the following roles in that folder: * [vpc.privateAdmin](../../vpc/security/index.md#vpc-private-admin) * [vpc.user](../../vpc/security/index.md#vpc-user) To use a [public IP address](../../vpc/concepts/address.md#public-addresses), also [assign](../../iam/operations/sa/assign-role-for-sa.md) the [vpc.publicAdmin](../../vpc/security/index.md#vpc-public-admin) role. #### Namespace fails to delete and remains _Terminating_ {#namespace-terminating} This issue occurs when your [namespace](../concepts/index.md#namespace) contains stuck resources that the namespace controller cannot delete. To fix it, delete the stuck resources manually. {% list tabs %} - CLI If you do not have the Yandex Cloud CLI yet, [install and initialize it](../../cli/quickstart.md#install). The folder used by default is the one specified when [creating](../../cli/operations/profile/profile-create.md) the CLI profile. To change the default folder, use the `yc config set folder-id ` command. You can also specify a different folder for any command using `--folder-name` or `--folder-id`. If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions. 1. [Connect to the Managed Service for Kubernetes cluster](../operations/connect/index.md). 1. Get the list of resources remaining in the namespace: ```bash kubectl api-resources --verbs=list --namespaced --output=name \ | xargs --max-args=1 kubectl get --show-kind \ --ignore-not-found --namespace= ``` 1. Delete the listed resources: ```bash kubectl delete --namespace= ``` If the namespace is still in the `Terminating` status and cannot be deleted, delete it forcibly using `finalizer`: 1. Run a local proxy to the Kubernetes API: ```bash kubectl proxy ``` 1. Delete the namespace: ```bash kubectl get namespace --output=json \ | jq '.spec = {"finalizers":[]}' > temp.json && \ curl --insecure --header "Content-Type: application/json" \ --request PUT --data-binary @temp.json \ 127.0.0.1:8001/api/v1/namespaces//finalize ``` We do not recommend deleting the namespace with the `Terminating` status using `finalizer` right away, as this may cause the stuck resources to remain in your Managed Service for Kubernetes cluster. {% endlist %} #### I am using Yandex Network Load Balancer together with an ingress controller. Why are some of my cluster's nodes _UNHEALTHY_? {#nlb-ingress} This is normal behavior for a [load balancer](../../network-load-balancer/concepts/index.md) with `External Traffic Policy: Local` enabled. Only the [Managed Service for Kubernetes nodes](../concepts/index.md#node-group) whose [pods](../concepts/index.md#pod) are ready to handle user traffic get the `HEALTHY` status. All other nodes are labeled as `UNHEALTHY`. To check the policy type of a load balancer created using a `LoadBalancer` service, run this command: ```bash kubectl describe svc \ | grep 'External Traffic Policy' ``` For more information, see [Parameters of a LoadBalancer service](../operations/create-load-balancer.md#advanced). #### Why does the newly created PersistentVolumeClaim remain _Pending_? {#pvc-pending} This is normal for a [PersistentVolumeClaim](../concepts/volume.md#persistent-volume) (PVC). The newly created PVC remains **Pending** until you create a pod that will use it. To change the PVC status to **Running**: 1. View the PVC details: ```bash kubectl describe pvc \ --namespace= ``` Where `--namespace` is the namespace containing the PVC. The `waiting for first consumer to be created before binding` message means that the PVC is awaiting pod creation. 1. [Create a pod](../operations/volumes/dynamic-create-pv.md#create-pod) for this PVC. #### Why does my Managed Service for Kubernetes cluster fail to start after I update its node configuration? {#not-starting} Make sure the new configuration of Managed Service for Kubernetes nodes is within the [quota](../concepts/limits.md): {% list tabs %} - CLI If you do not have the Yandex Cloud CLI yet, [install and initialize it](../../cli/quickstart.md#install). The folder used by default is the one specified when [creating](../../cli/operations/profile/profile-create.md) the CLI profile. To change the default folder, use the `yc config set folder-id ` command. You can also specify a different folder for any command using `--folder-name` or `--folder-id`. If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions. To run diagnostics for your Managed Service for Kubernetes cluster nodes: 1. [Connect to the Managed Service for Kubernetes cluster](../operations/connect/index.md). 1. Check the state of Managed Service for Kubernetes nodes: ```bash yc managed-kubernetes cluster list-nodes ``` A message saying that the limit of Managed Service for Kubernetes cluster resources has been exceeded appears in the first column of the command output. Here is an example: ```text +--------------------------------+-----------------+------------------+-------------+--------------+ | CLOUD INSTANCE | KUBERNETES NODE | RESOURCES | DISK | STATUS | +--------------------------------+-----------------+------------------+-------------+--------------+ | fhmil14sdienhr5uh89no | | 2 100% core(s), | 64.0 GB hdd | PROVISIONING | | CREATING_INSTANCE | | 4.0 GB of memory | | | | [RESOURCE_EXHAUSTED] The limit | | | | | | on total size of network-hdd | | | | | | disks has exceeded., | | | | | | [RESOURCE_EXHAUSTED] The limit | | | | | | on total size of network-hdd | | | | | | disks has exceeded. | | | | | +--------------------------------+-----------------+------------------+-------------+--------------+ ``` {% endlist %} To start your Managed Service for Kubernetes cluster, [increase the quotas](../concepts/limits.md). #### After changing the node subnet mask in the cluster settings, the number of pods per node is not as expected {#count-pods} **Solution**: Recreate the node group. #### Error updating ingress controller certificate {#ingress-certificate} Error message: ```text ERROR controller-runtime.manager.controller.ingressgroup Reconciler error {"name": "some-prod", "namespace": , "error": "rpc error: code = InvalidArgument desc = Validation error:\nlistener_specs[1].tls.sni_handlers[2].handler.certificate_ids: Number of elements must be less than or equal to 1"} ``` The error occurs if different certificates are specified for the same ingress controller handler. **Solution**: Edit and apply the ingress controller specifications so that each handler has only one certificate. #### Why is DNS resolution not working in my cluster? {#not-resolve-dns} A Managed Service for Kubernetes cluster may fail to resolve internal and external DNS requests for several reasons. To fix the issue: 1. [Check the version of your Managed Service for Kubernetes cluster and node groups](#check-version). 1. [Make sure CoreDNS is up and running](#check-coredns). 1. [Make sure your Managed Service for Kubernetes cluster has enough CPU resources available](#check-cpu). 1. [Set up autoscaling](#dns-autoscaler). 1. [Set up local DNS caching](#node-local-dns). ##### Check the version of your cluster and node groups {#check-version} 1. Get the list of current Kubernetes versions: ```bash yc managed-kubernetes list-versions ``` 1. Get the Managed Service for Kubernetes cluster version: ```bash yc managed-kubernetes cluster get | grep version: ``` You can get the Managed Service for Kubernetes cluster ID and name with the [list of clusters in the folder](../operations/kubernetes-cluster/kubernetes-cluster-list.md#list). 1. Get the Managed Service for Kubernetes node group version: ```bash yc managed-kubernetes node-group get | grep version: ``` You can get the Managed Service for Kubernetes node group ID and name with the [list of node groups in the cluster](../operations/node-group/node-group-list.md#list). 1. If the versions of your Managed Service for Kubernetes cluster and node groups are not on the list of current Kubernetes versions, [upgrade them](../operations/update-kubernetes.md). ##### Make sure CoreDNS is up and running {#check-coredns} Get the list of CoreDNS pods and their statuses: ```bash kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide ``` Make sure all pods have the `Running` status. ##### Make sure your cluster has enough CPU resources available {#check-cpu} 1. Navigate to the [folder dashboard](https://console.yandex.cloud) and select **Managed Service for Kubernetes**. 1. Click the name of the Managed Service for Kubernetes cluster you need and select the **Node management** tab. 1. Go to the **Nodes** tab and click the name of any Managed Service for Kubernetes node. 1. Navigate to the **Monitoring** tab. 1. Make sure that, in the **CPU, [cores]** chart, the `used` CPU values have not reached the `total` available CPU values. Check this for each Managed Service for Kubernetes cluster node. ##### Set up autoscaling {#dns-autoscaler} Set up [DNS autoscaling based on the Managed Service for Kubernetes cluster size](../tutorials/dns-autoscaler.md). ##### Set up local DNS caching {#node-local-dns} [Set up NodeLocal DNS Cache](../tutorials/node-local-dns.md). For optimal settings, [install NodeLocal DNS Cache from Yandex Cloud Marketplace](../operations/applications/node-local-dns.md#marketplace-install). #### Creating a node group with the CLI results in a parameter conflict. How do I fix it? {#conflicting-flags} Check whether you are specifying the `--location`, `--network-interface`, and `--public-ip` parameters in the same command. Providing them together causes the following errors: * For the `--location` and `--public-ip` or `--location` and `--network-interface` pairs: ```text ERROR: rpc error: code = InvalidArgument desc = Validation error: allocation_policy.locations[0].subnet_id: can't use "allocation_policy.locations[0].subnet_id" together with "node_template.network_interface_specs" ``` * For the `--network-interface` and `--public-ip` pair: ```text ERROR: flag --public-ip cannot be used together with --network-interface. Use '--network-interface' option 'nat' to get public address ``` Make sure you only provide one of the three parameters in a command. It is enough to specify the location of a Managed Service for Kubernetes node group either in `--location` or in `--network-interface`. To grant internet access to Managed Service for Kubernetes cluster nodes, do one of the following: * Assign a [public IP address](../../vpc/concepts/address.md#public-addresses) to the cluster nodes, specifying `--network-interface ipv4-address=nat` or `--network-interface ipv6-address=nat`. * [Enable access to Managed Service for Kubernetes nodes from the internet](../operations/node-group/node-group-update.md#node-internet-access) after creating a node group. #### Error connecting to a cluster using kubectl {#connect-to-cluster} Error message: ```text ERROR: cluster has empty endpoint ``` This error occurs if you try to [connect to a cluster](../operations/connect/index.md#kubectl-connect) with no public IP address and get `kubectl` credentials for a public IP address using this command: ```bash yc managed-kubernetes cluster \ get-credentials \ --external ``` To connect to the cluster's private IP address from a VM in the same network, get `kubectl` credentials using this command: ```bash yc managed-kubernetes cluster \ get-credentials \ --internal ``` If you need to connect to a cluster from the internet, [recreate the cluster and assign](../operations/kubernetes-cluster/kubernetes-cluster-create.md) it a public IP address. #### Errors connecting to a node over SSH {#node-connect} Error messages: ```text Permission denied (publickey,password) ``` ```text Too many authentication failures ``` The following situations cause errors [when connecting to a Managed Service for Kubernetes node](../operations/node-connect-ssh.md): * No public SSH key is added to the Managed Service for Kubernetes node group metadata. **Solution**: [Update the Managed Service for Kubernetes node group keys](../operations/node-connect-ssh.md#node-add-metadata). * An invalid public SSH key is added to the Managed Service for Kubernetes node group metadata. **Solution**: [Change the format of the public key file to the appropriate one](../operations/node-connect-ssh.md#key-format) and [update the Managed Service for Kubernetes node group keys](../operations/node-connect-ssh.md#node-add-metadata). * No private SSH key is added to an authentication agent (`ssh-agent`). **Solution**: Add a private key by running the `ssh-add ` command. #### How do I provide internet access to my Managed Service for Kubernetes cluster nodes? {#internet} If Managed Service for Kubernetes cluster nodes have no internet access, the following error occurs when trying to connect to the internet: ```text Failed to pull image "cr.yandex/***": rpc error: code = Unknown desc = Error response from daemon: Gethttps://cr.yandex/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) ``` You can provide internet access to your Managed Service for Kubernetes cluster nodes in several ways: * Set up a [NAT gateway](../../vpc/operations/create-nat-gateway.md) or [NAT instance](../../vpc/tutorials/nat-instance/index.md). With [static routing](../../vpc/concepts/routing.md) in place, traffic will go through a gateway or a separate NAT instance. * [Assign a public IP addresses to your Managed Service for Kubernetes node group](../operations/node-group/node-group-update.md#node-internet-access). {% note info %} If you assign public IP addresses to your cluster nodes and then configure a NAT gateway or NAT instance, you will lose internet access via the public addresses. For more information, see [this Yandex Virtual Private Cloud guide](../../vpc/concepts/routing.md#internet-routes). {% endnote %} #### Why cannot I choose Docker as the container runtime? {#docker-runtime} Clusters running Kubernetes 1.24 or higher do not support the Docker container runtime. [containerd](https://containerd.io/) is the only available runtime. #### Error connecting a GitLab repository to Argo CD {#argo-cd} Error message: ```text FATA[0000] rpc error: code = Unknown desc = error testing repository connectivity: authorization failed ``` This error occurs if access to GitLab over HTTP(S) is disabled. **Solution**: Enable HTTP(S) access. Proceed as follows: 1. In GitLab, in the left-hand panel, select **Admin → Settings → General**. 1. Under **Visibility and access controls**, find the **Enabled Git access protocols** setting. 1. In the list, select the item which allows access over HTTP(S). For more information, see [this GitLab guide](https://docs.gitlab.com/administration/settings/visibility_and_access_controls/#configure-enabled-git-access-protocols). #### Traffic loss when deploying app updates in a cluster with Yandex Application Load Balancer {#alb-traffic-lost} When your app traffic is managed by an Application Load Balancer and the load balancer's ingress controller [traffic policy](../nlb-ref/service.md#servicespec) is set to `externalTrafficPolicy: Local`, the app processes requests on the same node they were delivered to by the load balancer. There is no traffic flow between nodes. The [default health check](../../network-load-balancer/concepts/health-check.md) monitors the status of the node, not application. Therefore, Application Load Balancer traffic may go to a node where there is no application running. When you deploy a new app version in a cluster, the [Application Load Balancer ingress controller](../../application-load-balancer/tools/k8s-ingress-controller/index.md) requests the load balancer to update the backend group configuration. It takes at least 30 seconds to process the request, and the app may not receive any user traffic during that time. To prevent this, we recommend setting up backend health checks on your Application Load Balancer. With health checks, the load balancer timely spots unavailable backends and reroutes traffic to healthy backends. Once the application is updated, traffic will again be distributed across all backends. For more information, see [Tips for configuring Yandex Application Load Balancer health checks](../../application-load-balancer/concepts/best-practices.md) and [Metadata.annotations](../../application-load-balancer/k8s-ref/service-for-ingress.md#annotations). #### System time displayed incorrectly on nodes, as well as in container and Managed Service for Kubernetes cluster pod logs {#time} Managed Service for Kubernetes cluster time may not match the time of other resources, such as VMs, if they use different time synchronization sources. For example, a Managed Service for Kubernetes cluster synchronizes with a time server (by default), whereas a VM synchronizes with a private or public NTP server. **Solution**: Set up Managed Service for Kubernetes cluster time synchronization with your private NTP server. Proceed as follows: 1. Specify the NTP server addresses in the [DHCP settings](../../vpc/concepts/dhcp-options.md) of the master subnets. {% list tabs group=instructions %} - Management console {#console} 1. Navigate to the folder dashboard and select **Managed Service for Kubernetes**. 1. Click the name of the Kubernetes cluster. 1. Under **Master configuration**, click the subnet name. 1. Click ![subnets](../../_assets/console-icons/pencil.svg) **Edit** in the top-right corner. 1. In the window that opens, expand the **DHCP settings** section. 1. Click **Add** and specify the IP address of your NTP server. 1. Click **Save changes**. - CLI {#cli} If you do not have the Yandex Cloud CLI yet, [install and initialize it](../../cli/quickstart.md#install). The folder used by default is the one specified when [creating](../../cli/operations/profile/profile-create.md) the CLI profile. To change the default folder, use the `yc config set folder-id ` command. You can also specify a different folder for any command using `--folder-name` or `--folder-id`. If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions. 1. See the description of the CLI command for updating subnet settings: ```bash yc vpc subnet update --help ``` 1. Run the `subnet` command, specifying the NTP server IP address in the `--ntp-server` parameter: ```bash yc vpc subnet update --ntp-server ``` {% note tip %} To find out the IDs of the subnets containing the cluster, [get detailed information about the cluster](../operations/kubernetes-cluster/kubernetes-cluster-list.md#get). {% endnote %} - Terraform {#tf} 1. In the Terraform configuration file, change the cluster subnet description. Add the `dhcp_options` section (if missing) with the `ntp_servers` parameter specifying the IP address of your NTP server: ```hcl ... resource "yandex_vpc_subnet" "lab-subnet-a" { ... v4_cidr_blocks = [""] network_id = "" ... dhcp_options { ntp_servers = [""] ... } } ... ``` For more information about `yandex_vpc_subnet` properties in Terraform, see [this provider guide](../../terraform/resources/vpc_subnet.md). 1. Apply the changes: 1. In the terminal, navigate to the configuration file directory. 1. Make sure the configuration is correct using this command: ```bash terraform validate ``` If the configuration is valid, you will get this message: ```bash Success! The configuration is valid. ``` 1. Run this command: ```bash terraform plan ``` You will see a list of resources and their properties. No changes will be made at this step. Terraform will show any errors in the configuration. 1. Apply the configuration changes: ```bash terraform apply ``` 1. Type `yes` and press **Enter** to confirm the changes. Terraform will update all required resources. You can check the subnet update using the [management console](https://console.yandex.cloud) or this [CLI](../../cli/quickstart.md) command: ```bash yc vpc subnet get ``` - API {#api} Use the [update](../../vpc/api-ref/Subnet/update.md) method for the [Subnet](../../vpc/api-ref/Subnet/index.md) resource and provide the following in the request: * NTP server IP address in the `dhcpOptions.ntpServers` parameter. * `dhcpOptions.ntpServers` parameter to update in the `updateMask` parameter. {% note tip %} To find out the IDs of the subnets containing the cluster, [get detailed information about the cluster](../operations/kubernetes-cluster/kubernetes-cluster-list.md#get). {% endnote %} {% endlist %} {% note warning %} For a highly available master hosted across three availability zones, you need to update each of the three subnets. {% endnote %} 1. Enable connections from the cluster to NTP servers. [Create a rule](../../vpc/operations/security-group-add-rule.md) for outbound traffic in the [cluster and node group security group](../operations/connect/security-groups.md#rules-internal-cluster): * **Port range**: `123`. If your NTP server uses a port other than `123`, specify that port. * **Protocol**: `UDP`. * **Destination name**: `CIDR`. * **CIDR blocks**: `/32`. For a master hosted across three availability zones, specify three sections: `/32`, `/32`, and `/32`. 1. Update the network settings in the cluster node group using one of the following methods: * Connect to each node in the group [over SSH](../operations/node-connect-ssh.md) or [via OS Login](../operations/node-connect-oslogin.md) and run the `sudo dhclient -v -r && sudo dhclient` command. * Reboot the group nodes at any convenient time. {% note warning %} Updating network settings may cause the services within the cluster to become unavailable for a few minutes. {% endnote %} #### What should I do if I deleted my Yandex Network Load Balancer or its target groups that were automatically created for a LoadBalancer service? {#deleted-loadbalancer-service} You cannot manually restore a Network Load Balancer or its target groups. [Recreate](../operations/create-load-balancer.md#lb-create) your `LoadBalancer` service. This will automatically create a load balancer and target groups. #### Error connecting a Yandex Compute Cloud VM instance as an external Managed Service for Kubernetes node {#vm-as-external-node} Error message: ```text Unable to create remote dir /home/kubernetes/bin/: ssh run `mkdir -p -m 0644 /home/kubernetes/bin/': Process exited with status 142 Please login as the user "NONE" rather than the user "root". ``` To fix the issue, [recreate](../../compute/operations/index.md#vm-create) the VM instance, specifying `disable_root: false` in the metadata for the `user-data` key. {% cut "Metadata example" %} ```yaml #cloud-config datasource: Ec2: strict_id: false disable_root: false users: - name: sudo: ALL=(ALL) NOPASSWD:ALL shell: /bin/bash ssh_authorized_keys: - ssh-rsa ``` {% endcut %}