[Yandex Cloud documentation](../../../index.md) > [Yandex Managed Service for Kubernetes](../../index.md) > [Step-by-step guides](../index.md) > Managing a node group > Creating a node group # Creating a node group A [node group](../../concepts/index.md#node-group) is a group of VMs in a [Managed Service for Kubernetes cluster](../../concepts/index.md#kubernetes-cluster) that have the same configuration and run user containers. Before creating a node group, [create](../kubernetes-cluster/kubernetes-cluster-create.md) a Managed Service for Kubernetes cluster and make sure your [cloud](../../../resource-manager/concepts/resources-hierarchy.md#cloud) has enough [resources](../../concepts/limits.md). {% note warning %} Starting with Kubernetes 1.30, the Managed Service for Kubernetes cluster node base image changed from Ubuntu 20.04 to Ubuntu 22.04 in all [release channels](../../concepts/release-channels-and-updates.md). In the existing clusters and node groups, the OS version will be upgraded using the method you select. For OS upgrade details and recommendations, see [Updating node group OS](../../concepts/node-os-update.md). {% endnote %} {% list tabs group=instructions %} - Management console {#console} To create a [Managed Service for Kubernetes node group](../../concepts/index.md#node-group): 1. In the [management console](https://console.yandex.cloud), select the [folder](../../../resource-manager/concepts/resources-hierarchy.md#folder) where you want to create a [Managed Service for Kubernetes cluster](../../concepts/index.md#kubernetes-cluster). 1. Navigate to **Managed Service for Kubernetes**. 1. Select the Managed Service for Kubernetes cluster to create a node group for. 1. On the Managed Service for Kubernetes cluster page, go to the **Node management** tab. 1. Click **Create a node group**. 1. Enter a name and description for the Managed Service for Kubernetes node group. 1. In the **Kubernetes version** field, select the Kubernetes version for the Managed Service for Kubernetes nodes. 1. In the **Container runtime** field, select `containerd`. 1. In the **Labels** field, add the [node cloud labels](../../concepts/index.md#node-labels). 1. Under **Scaling**, select its type: * `Fixed`: To keep a fixed number of Managed Service for Kubernetes nodes in the group. Specify the number of Managed Service for Kubernetes nodes in the group. The **Number of nodes** setting will become available. * `Automatic`: To manage the number of group nodes using the [Managed Service for Kubernetes cluster autoscaling](../../concepts/autoscale.md#ca). The following settings will become available: * **Minimum number of nodes**. * **Maximum number of nodes**. * **Initial number of nodes** of the new Managed Service for Kubernetes group. {% note warning %} You cannot change the scaling type after creating a Managed Service for Kubernetes node group. {% endnote %} 1. Under **Changes during creation and updates**, specify the maximum number of nodes by which you can exceed the size of the group when updating it, as well as the maximum number of unavailable nodes during the update. {% note warning %} If the node group settings allow the group to scale up automatically during updates, make sure your cloud has enough [resources](../../concepts/limits.md). [Increase](../../../quota-manager/operations/set-quotas.md) quotas, if needed. {% endnote %} 1. Under **Computing resources**: * Select a [platform](../../../compute/concepts/vm-platforms.md). * Enter the required number of [GPUs](../../../compute/concepts/gpus.md) and vCPUs, [guaranteed vCPU performance](../../../compute/concepts/performance-levels.md), and the amount of RAM. * Optionally, make the VM instance [preemptible](../../../compute/concepts/preemptible-vm.md) by checking the relevant box. {% note tip %} To evict pods automatically from a preemptible node when shutting it down, use the Yandex Cloud Marketplace product called [node-sitter](../applications/node-sitter.md). {% endnote %} * Optionally, enable a [software accelerated network](../../../compute/concepts/software-accelerated-network.md). {% note warning %} Before enabling a [software-accelerated network](../../../compute/concepts/software-accelerated-network.md), make sure you have [enough cloud resources](../../concepts/limits.md) to create an additional [Managed Service for Kubernetes node](../../concepts/index.md#node-group). {% endnote %} {% note info %} The set of parameters depends on the platform you select. {% endnote %} 1. Optionally, under **GPU settings**, specify whether the Managed Service for Kubernetes node group should have no pre-installed NVIDIA® drivers and CUDA® libraries for [GPU acceleration](../../../compute/concepts/gpus.md). 1. Optionally, under **Placement**, enter a name for the Managed Service for Kubernetes node [placement group](../../../compute/concepts/placement-groups.md). You will not be able to edit this setting after creating the Managed Service for Kubernetes node group. {% note info %} The placement group determines the maximum available node group size: * In an instance group with the [spread placement](../../../compute/concepts/placement-groups.md#spread) strategy, the maximum number of instances depends on the [limits](../../../compute/concepts/limits.md#compute-limits-vm). * In an instance group with the [partition placement](../../../compute/concepts/placement-groups.md#partition) strategy, the maximum number of instances in a partition depends on the [quotas](../../../compute/concepts/limits.md#vm-quotas). {% endnote %} 1. Under **Storage**: * Specify the Managed Service for Kubernetes node **Disk type**: * `HDD`: [Standard network drive](../../../compute/concepts/disk.md#disks-types); HDD network block storage. * `SSD`: Fast network drive; SSD network block storage. * `Non-replicated SSD`: Network drive with enhanced performance achieved by eliminating redundancy. You can only change the size of this disk type in 93 GB increments. {% note alert %} Non-replicated disks have no redundancy. If a disk fails, its data will be permanently lost. Learn more in [Non-replicated disks and ultra high-speed network storages with three replicas (SSD)](../../../compute/concepts/disk.md#nr-disks). {% endnote %} * `SSD IO`: Network drive with the same performance specifications as `Non-replicated SSD`, plus redundancy. You can only change the size of this disk type in 93 GB increments. * Specify the Managed Service for Kubernetes node disk size. 1. Under **Network settings**: * In the **Public address** field, select the IP address assignment method: * `Auto` to assign random [IP addresses](../../../vpc/concepts/address.md) from the Yandex Cloud address pool to the nodes. * `No address` not to assign public IP addresses. Public IP addresses are required for [connecting](../connect/index.md) to nodes from the internet and to connect nodes to external resources, such as Docker image registries ([Container Registry](../../../container-registry/concepts/index.md), [Cloud Registry](../../../cloud-registry/concepts/index.md), or [Docker Hub](https://hub.docker.com/)) and [Object Storage](../../../storage/concepts/bucket.md) buckets. To provide nodes with internet access, assign them a public IP address and [configure](../connect/security-groups.md#rules-internal-nodegroup) a security group. As an alternative to public IP addresses, you can also create and configure a [NAT gateway](../../../vpc/operations/create-nat-gateway.md) or a [NAT instance](../../../vpc/tutorials/nat-instance/index.md) in the subnet hosting the nodes. If you assign public IP addresses to your cluster nodes and then configure a NAT gateway or NAT instance, you will lose internet access via the public addresses. For more information, see [this Yandex Virtual Private Cloud guide](../../../vpc/concepts/routing.md#internet-routes). * Select [security groups](../../../vpc/concepts/security-groups.md). {% note warning %} The [configuration of security groups](../connect/security-groups.md) determines performance and availability of the cluster and services running in it. {% endnote %} 1. Under **Location**: * Select the [availability zone](../../../overview/concepts/geo-scope.md) and [subnet](../../../vpc/concepts/network.md#subnet) to place the group nodes in. {% note warning %} Do not change or delete Virtual Private Cloud resources used by the Managed Service for Kubernetes cluster. This may cause cluster errors or make it impossible to delete later. {% endnote %} * Optionally, you can place nodes of a group with the fixed scaling type across multiple availability zones. To do this, click **Add location** and specify an additional availability zone and subnet. {% note warning %} You can only place an autoscaling node group in one [availability zone](../../../overview/concepts/geo-scope.md). {% endnote %} 1. Under **Access**, configure one of the methods of connecting to nodes in a Managed Service for Kubernetes node group: * To connect to nodes via OS Login, select **Access by OS Login**. In this case, you will not be able to specify SSH keys because these connection methods are mutually exclusive. For more information on how to configure and use OS Login, see [Connecting to a node via OS Login](../node-connect-oslogin.md). * To connect to nodes using SSH keys, specify the required credentials: * In the **Login** field, enter a username. * In the **SSH key** field, paste the contents of the [public key](../node-connect-ssh.md#creating-ssh-keys) file. For more information about preparing, configuring, and using SSH keys, see [Connecting to a node over SSH](../node-connect-ssh.md). You can [change the metadata list after you create a cluster](node-group-update.md#switch-node-connect-mode). 1. Under **Maintenance window settings**: * In the **Maintenance frequency / Disable** field, select your preferred maintenance window: * `Disable`: Automatic updates disabled. * `Anytime`: Updates allowed at any time. * `Daily`: Updates will take place within the time interval specified in the **Maintenance window start time in UTC** field. * `Custom`: Updates will take place within the time interval specified in the **Weekly schedule** field. 1. Under **Additional**: * To be able to edit [unsafe kernel parameters](../../concepts/index.md#node-group) on the Managed Service for Kubernetes group nodes, click **Add variable**. To enter the name of each kernel parameter, create a separate field. * To set up [taints for Managed Service for Kubernetes nodes](../../concepts/index.md#taints-tolerations), click **Add policy**. Enter the key, value, and effect for each taint in a separate set of fields. * To set up [Kubernetes labels](../../concepts/index.md#node-labels) for group nodes, click **Add label**. Enter the key and value for each Kubernetes label in a separate set of fields. 1. Optionally, expand the **Metadata** section and add metadata for the nodes. {% note warning %} Metadata settings can affect the behavior and health of nodes in a group. Change these settings only if you know exactly what you want to do. Providing user data in the metadata with the `user-data` key is not supported. {% endnote %} To add metadata, click **Add field**. Specify the key and value for each metadata element in a separate set of fields. You can [change the metadata list after you create a cluster](node-group-update.md#switch-node-connect-mode). 1. Click **Create**. - CLI {#cli} If you do not have the Yandex Cloud CLI yet, [install and initialize it](../../../cli/quickstart.md#install). The folder used by default is the one specified when [creating](../../../cli/operations/profile/profile-create.md) the CLI profile. To change the default folder, use the `yc config set folder-id ` command. You can also specify a different folder for any command using `--folder-name` or `--folder-id`. If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions. 1. View the description of the CLI command to create a Managed Service for Kubernetes node group: ```bash yc managed-kubernetes node-group create --help ``` 1. Specify Managed Service for Kubernetes node group parameters in the create command (our example does not include all available parameters): ```bash yc managed-kubernetes node-group create \ --allowed-unsafe-sysctls \ --cluster-name \ --cores \ --core-fraction \ --daily-maintenance-window \ --disk-size \ --disk-type \ --fixed-size \ --max-expansion \ --max-unavailable \ --location zone=[],subnet-id=[] \ --memory \ --name \ --network-acceleration-type \ --network-interface security-group-ids=[],ipv4-address= \ --platform-id \ --container-runtime containerd \ --preemptible \ --public-ip \ --template-labels \ --node-labels --version \ --node-name \ --node-taints \ --container-network-settings pod-mtu= \ --max-expansion \ --max-unavailable \ --reserved-instance-pool-id ``` Where: * `--allowed-unsafe-sysctls`: Permission for Managed Service for Kubernetes group nodes to use [unsafe kernel parameters](../../concepts/index.md#node-group), comma-separated. * `--cluster-name`: Name of the [Managed Service for Kubernetes cluster](../../concepts/index.md#kubernetes-cluster) to create the node group in. * `--cores`: Number of vCPUs for Managed Service for Kubernetes nodes. * `--core-fraction`: [Guaranteed vCPU share](../../../compute/concepts/performance-levels.md) for Managed Service for Kubernetes nodes. * `--daily-maintenance-window`: [Maintenance](../../concepts/release-channels-and-updates.md#updates) window settings. * `--disk-size`: [Disk size](../../../compute/concepts/disk.md#maximum-disk-size) of the Managed Service for Kubernetes node. * `--disk-type`: [Disk type](../../../compute/concepts/disk.md#disks_types) of the Managed Service for Kubernetes node, `network-nvme` or `network-hdd`. * Type of scaling: * `--fixed-size`: Fixed number of nodes in a Managed Service for Kubernetes node group. * `--auto-scale`: Settings for [Managed Service for Kubernetes cluster autoscaling](../../concepts/node-group/cluster-autoscaler.md): * `min`: Minimum number of nodes in the group. * `max`: Maximum number of nodes in the group. * `initial`: Initial number of nodes in the group. You cannot change the scaling type after creating a node group. * `--max-expansion`: Maximum number of nodes by which you can expand the group when updating it. {% note warning %} If the node group settings allow the group to scale up automatically during updates, make sure your cloud has enough [resources](../../concepts/limits.md). [Increase](../../../quota-manager/operations/set-quotas.md) quotas, if needed. {% endnote %} * `--max-unavailable`: Maximum number of unavailable nodes in the group when updating it. * `--location`: [Availability zone](../../../overview/concepts/geo-scope.md) and [subnet](../../../vpc/concepts/network.md#subnet) to host Managed Service for Kubernetes nodes. You can specify more than one option but only a single subnet per zone. Use a separate `--location` parameter for each availability zone. {% note warning %} You can only place an autoscaling node group in one [availability zone](../../../overview/concepts/geo-scope.md). {% endnote %} If you provide `--location`, `--network-interface`, and `--public-ip` in the same command, you will [get an error](../../qa/troubleshooting.md#conflicting-flags). It is enough to specify the location of a Managed Service for Kubernetes node group either in `--location` or `--network-interface`. To grant internet access to Managed Service for Kubernetes cluster nodes, do one of the following: * Assign a [public IP address](../../../vpc/concepts/address.md#public-addresses) to the cluster nodes, specifying `--network-interface ipv4-address=nat` or `--network-interface ipv6-address=nat`. * [Enable access to Managed Service for Kubernetes nodes from the internet](node-group-update.md#node-internet-access) after creating a node group. {% note warning %} Do not change or delete Virtual Private Cloud resources used by the Managed Service for Kubernetes cluster. This may cause cluster errors or make it impossible to delete later. {% endnote %} * `--memory`: Amount of memory allocated for Managed Service for Kubernetes nodes. * `--name`: Managed Service for Kubernetes node group name. * `--network-acceleration-type`: Select the [network acceleration](../../../compute/concepts/software-accelerated-network.md) type: * `standard`: No acceleration. * `software-accelerated`: Software-accelerated network. {% note warning %} Before enabling a [software-accelerated network](../../../compute/concepts/software-accelerated-network.md), make sure you have [enough cloud resources](../../concepts/limits.md) to create an additional [Managed Service for Kubernetes node](../../concepts/index.md#node-group). {% endnote %} * `--network-interface`: Network settings: * `security-group-ids`: [Security group](../connect/security-groups.md) IDs. * `subnets`: Names of subnets that will host the nodes. * `ipv4-address`: IPv4 address assignment method. * `ipv6-address`: IPv6 address assignment method. The `ipv4-address` and `ipv6-address` parameters set the method of assigning IP addresses: * `auto`: Assign only internal IP addresses to nodes. * `nat`: Assign public and internal IP addresses to nodes. Public IP addresses are required for [connecting](../connect/index.md) to nodes from the internet and to connect nodes to external resources, such as Docker image registries ([Container Registry](../../../container-registry/concepts/index.md), [Cloud Registry](../../../cloud-registry/concepts/index.md), or [Docker Hub](https://hub.docker.com/)) and [Object Storage](../../../storage/concepts/bucket.md) buckets. To provide nodes with internet access, assign them a public IP address and [configure](../connect/security-groups.md#rules-internal-nodegroup) a security group. As an alternative to public IP addresses, you can also create and configure a [NAT gateway](../../../vpc/operations/create-nat-gateway.md) or a [NAT instance](../../../vpc/tutorials/nat-instance/index.md) in the subnet hosting the nodes. If you assign public IP addresses to your cluster nodes and then configure a NAT gateway or NAT instance, you will lose internet access via the public addresses. For more information, see [this Yandex Virtual Private Cloud guide](../../../vpc/concepts/routing.md#internet-routes). * `--platform-id`: Managed Service for Kubernetes node [platform](../../../compute/concepts/vm-platforms.md). * `--container-runtime`: [containerd](https://containerd.io/) runtime environment. * `--preemptible`: Flag you set for [preemptible](../../../compute/concepts/preemptible-vm.md) VMs. {% note tip %} To evict pods automatically from a preemptible node when shutting it down, use the Yandex Cloud Marketplace product called [node-sitter](../applications/node-sitter.md). {% endnote %} * `--public-ip`: Flag you set if the Managed Service for Kubernetes node group needs a [public IP address](../../../vpc/concepts/address.md#public-addresses). * `--template-labels`: Node group [cloud labels](../../concepts/index.md#node-labels). You can specify multiple labels separated by commas. * `--node-labels`: Node group [Kubernetes labels](../../concepts/index.md#node-labels). * `--version`: Kubernetes version on the Managed Service for Kubernetes group nodes. * `--node-name`: Managed Service for Kubernetes node name template. The name is unique if the template contains at least one of the following variables: * `{instance_group.id}`: Instance group ID. * `{instance.index}`: Unique instance number in the group that can be any value from 1 to N, where N is the number of instances in the group. * `{instance.index_in_zone}`: Instance number in the zone. It is unique for a specific instance group within a zone. * `{instance.short_id}`: Instance ID that is unique within the group. It consists of four alphabetic characters. * `{instance.zone_id}`: Zone ID. For example, `prod-{instance.short_id}-{instance_group.id}`. If you do not specify the name, the default value is used: `{instance_group.id}-{instance.short_id}`. * `--node-taints`: Kubernetes [taints](../../concepts/index.md#taints-tolerations). You can specify multiple values. * `--container-network-settings`: [MTU](https://en.wikipedia.org/wiki/Maximum_transmission_unit) value for network connections to group pods. This setting is not applicable for clusters with the Calico or Cilium network policy controllers. * `--reserved-instance-pool-id`: Reserved instance pool [ID](../../../compute/cli-ref/reserved-instance-pool/list.md). For more information, see [Creating a group with nodes from a Yandex Compute Cloud reserved instance pool](node-group-create-in-instance-pool.md). * [Deployment policy](../../concepts/node-group/deploy-policy.md) parameters: * `--max-expansion`: Maximum number of nodes by which you can expand the group when modifying or updating it. The default value is `3`. * `--max-unavailable`: Maximum number of nodes that may be unavailable while the group is being modified or updated. The default value is `0`. For more information, see [Node group deployment policy in Managed Service for Kubernetes](../../concepts/node-group/deploy-policy.md). Result: ```text done (1m17s) id: catpl8c44kii******** cluster_id: catcsqidoos7******** ... start_time: hours: 22 duration: 36000s ``` 1. To add metadata for nodes, use the `--metadata` or `--metadata-from-file` parameter. Use metadata to configure the method of connecting to nodes in a node group. You can configure one method only because they are mutually exclusive. To connect to nodes in a node group, specify metadata for the selected connection method: * To connect to nodes via OS Login, add metadata with the `enable-oslogin` key set to `true`. For more information on how to configure and use OS Login, see [Connecting to a node via OS Login](../node-connect-oslogin.md). * To connect to nodes using SSH keys, add metadata with the `ssh-keys` key and its value listing the connection credentials. For more information about preparing, configuring, and using SSH keys, see [Connecting to a node over SSH](../node-connect-ssh.md). {% note warning %} Metadata settings can affect the behavior and health of nodes in a group. Change these settings only if you know exactly what you want to do. Providing user data in the metadata with the `user-data` key is not supported. {% endnote %} Add metadata using one of the following methods: * Using `--metadata`, specify one or multiple `key=value` pairs separated by commas. The key value is provided explicitly. * Using `--metadata-from-file`, specify one or multiple `key=path_to_file_with_value` pairs separated by commas. The key value will be read from a file. This may be of use if the value is too long to provide it explicitly or if it contains line breaks or other special characters. You can [change the metadata list after you create a cluster](node-group-update.md#switch-node-connect-mode). 1. To specify a [placement group](../../../compute/concepts/placement-groups.md) for Managed Service for Kubernetes nodes: 1. Get a list of placement groups using the `yc compute placement-group list` command. 1. Provide a placement group name or ID in the `--placement-group` parameter when creating a Managed Service for Kubernetes node group: ```bash yc managed-kubernetes node-group create \ ... --placement-group ``` {% note info %} The placement group determines the maximum available node group size: * In an instance group with the [spread placement](../../../compute/concepts/placement-groups.md#spread) strategy, the maximum number of instances depends on the [limits](../../../compute/concepts/limits.md#compute-limits-vm). * In an instance group with the [partition placement](../../../compute/concepts/placement-groups.md#partition) strategy, the maximum number of instances in a partition depends on the [quotas](../../../compute/concepts/limits.md#vm-quotas). {% endnote %} - Terraform {#tf} To create a [Managed Service for Kubernetes node group](../../concepts/index.md#node-group): 1. In the folder containing the [cluster description file](../kubernetes-cluster/kubernetes-cluster-create.md#kubernetes-cluster-create), create a configuration file with the new Managed Service for Kubernetes node group's parameters. Here is an example of the configuration file structure: ```hcl resource "yandex_kubernetes_node_group" "" { cluster_id = yandex_kubernetes_cluster..id name = "" ... instance_template { name = "" platform_id = "" placement_policy { placement_group_id = "" } network_acceleration_type = "" container_runtime { type = "containerd" } network_interface { nat = } labels { ""="" } node_labels { ""="" } ... } ... scale_policy { } deploy_policy { max_expansion = max_unavailable = } ... allocation_policy { location { zone = "" } } } ``` Where: * `cluster_id`: [Managed Service for Kubernetes cluster](../../concepts/index.md#kubernetes-cluster) ID. * `name`: Managed Service for Kubernetes node group name. * `instance_template`: Managed Service for Kubernetes node parameters: * `name`: Managed Service for Kubernetes node name template. The name is unique if the template contains at least one of the following variables: * `{instance_group.id}`: Instance group ID. * `{instance.index}`: Unique instance number in the group that can be any value from 1 to N, where N is the number of instances in the group. * `{instance.index_in_zone}`: Instance number in the zone. It is unique for a specific instance group within a zone. * `{instance.short_id}`: Instance ID that is unique within the group. It consists of four alphabetic characters. * `{instance.zone_id}`: Zone ID. For example, `prod-{instance.short_id}-{instance_group.id}`. If you do not specify the name, the default value is used: `{instance_group.id}-{instance.short_id}`. * `platform_id`: Managed Service for Kubernetes node [platform](../../../compute/concepts/vm-platforms.md). * `placement_group_id`: [Placement group](../../../compute/concepts/placement-groups.md) for Managed Service for Kubernetes nodes. {% note info %} The placement group determines the maximum available node group size: * In an instance group with the [spread placement](../../../compute/concepts/placement-groups.md#spread) strategy, the maximum number of instances depends on the [limits](../../../compute/concepts/limits.md#compute-limits-vm). * In an instance group with the [partition placement](../../../compute/concepts/placement-groups.md#partition) strategy, the maximum number of instances in a partition depends on the [quotas](../../../compute/concepts/limits.md#vm-quotas). {% endnote %} * `network_acceleration_type`: [Network acceleration](../../../compute/concepts/software-accelerated-network.md) type: * `standard`: No acceleration. * `software-accelerated`: Software-accelerated network. {% note warning %} Before enabling a [software-accelerated network](../../../compute/concepts/software-accelerated-network.md), make sure you have [enough cloud resources](../../concepts/limits.md) to create an additional [Managed Service for Kubernetes node](../../concepts/index.md#node-group). {% endnote %} * `container_runtime`, `type`: [containerd](https://containerd.io/) runtime environment. * `network_interface.nat`: Assign random public [IP addresses](../../../vpc/concepts/address.md) from the Yandex Cloud pool to the nodes, `true` or `false`. Public IP addresses are required for [connecting](../connect/index.md) to nodes from the internet and to connect nodes to external resources, such as Docker image registries ([Container Registry](../../../container-registry/concepts/index.md), [Cloud Registry](../../../cloud-registry/concepts/index.md), or [Docker Hub](https://hub.docker.com/)) and [Object Storage](../../../storage/concepts/bucket.md) buckets. To provide nodes with internet access, assign them a public IP address and [configure](../connect/security-groups.md#rules-internal-nodegroup) a security group. As an alternative to public IP addresses, you can also create and configure a [NAT gateway](../../../vpc/operations/create-nat-gateway.md) or a [NAT instance](../../../vpc/tutorials/nat-instance/index.md) in the subnet hosting the nodes. If you assign public IP addresses to your cluster nodes and then configure a NAT gateway or NAT instance, you will lose internet access via the public addresses. For more information, see [this Yandex Virtual Private Cloud guide](../../../vpc/concepts/routing.md#internet-routes). * `labels`: Node group [cloud labels](../../concepts/index.md#node-labels). You can specify multiple labels separated by commas. * `node_labels`: Node group [Kubernetes labels](../../concepts/index.md#node-labels). * `reserved_instance_pool_id`: Reserved instance pool [ID](../../../compute/cli-ref/reserved-instance-pool/list.md). For more information, see [Creating a group with nodes from a Yandex Compute Cloud reserved instance pool](node-group-create-in-instance-pool.md). * `scale_policy`: Scaling settings. You cannot change the scaling type after creating a node group. * `deploy_policy`: Group deployment settings: * `max_expansion`: Maximum number of nodes by which you can expand the group when updating it. {% note warning %} If the node group settings allow the group to scale up automatically during updates, make sure your cloud has enough [resources](../../concepts/limits.md). [Increase](../../../quota-manager/operations/set-quotas.md) quotas, if needed. {% endnote %} * `max_unavailable`: Maximum number of unavailable nodes in the group when updating it. * `allocation_policy`: Placement settings. These contain the `location` section with the `zone` parameter, i.e., the [availability zone](../../../overview/concepts/geo-scope.md) where you want to place the group nodes. You can place nodes of a group with the fixed scaling type across multiple availability zones. To do this, specify each availability zone you need in a separate `location` section. {% note warning %} You can only place an autoscaling node group in one [availability zone](../../../overview/concepts/geo-scope.md). {% endnote %} {% note warning %} Do not change or delete Virtual Private Cloud resources used by the Managed Service for Kubernetes cluster. This may cause cluster errors or make it impossible to delete later. {% endnote %} * To create a node group with a fixed number of nodes, add the `fixed_scale` section: ```hcl resource "yandex_kubernetes_node_group" "" { ... scale_policy { fixed_scale { size = } } } ``` * To create an [autoscaling](../../concepts/node-group/cluster-autoscaler.md) Managed Service for Kubernetes node group, add the `auto_scale` section: ```hcl resource "yandex_kubernetes_node_group" "" { ... scale_policy { auto_scale { min = max = initial = } } } ``` * To create a Managed Service for Kubernetes node group with [preemptible VMs](../../../compute/concepts/preemptible-vm.md), add the `scheduling_policy` section: ```hcl resource "yandex_kubernetes_node_group" "" { ... instance_template { scheduling_policy { preemptible = true } } } ``` {% note tip %} To evict pods automatically from a preemptible node when shutting it down, use the Yandex Cloud Marketplace product called [node-sitter](../applications/node-sitter.md). {% endnote %} * To add metadata for nodes, provide it in the `instance_template.metadata` parameter. Use metadata to configure the method of connecting to nodes in a node group. You can configure one method only because they are mutually exclusive. To connect to nodes in a node group, specify metadata for the selected connection method: * To connect to nodes via OS Login, add metadata with the `enable-oslogin` key set to `true`. For more information on how to configure and use OS Login, see [Connecting to a node via OS Login](../node-connect-oslogin.md). * To connect to nodes using SSH keys, add metadata with the `ssh-keys` key and its value listing the connection credentials. For more information about preparing, configuring, and using SSH keys, see [Connecting to a node over SSH](../node-connect-ssh.md). {% note warning %} Metadata settings can affect the behavior and health of nodes in a group. Change these settings only if you know exactly what you want to do. Providing user data in the metadata with the `user-data` key is not supported. {% endnote %} Add metadata using one of the following methods: * Specify one or multiple `key=value` pairs. The key value is provided explicitly. * Specify one or multiple `key=file(path_to_file_with_value)` pairs. The key value will be read from a file. This may be of use if the value is too long to provide it explicitly or if it contains line breaks or other special characters. ```hcl resource "yandex_kubernetes_node_group" "" { ... instance_template { metadata = { "key_1" = "value" "key_2" = file("") ... } ... } ... } ``` You can [change the metadata list after you create a cluster](node-group-update.md#switch-node-connect-mode). * To add [DNS records](../../../dns/concepts/resource-record.md): 1. Create a [private](../../../dns/operations/zone-create-private.md) or [public](../../../dns/operations/zone-create-public.md) DNS zone. 1. Add the `instance_template.network_interface.ipv4_dns_records` section: ```hcl resource "yandex_kubernetes_node_group" "" { ... instance_template { network_interface { ipv4_dns_records { fqdn = "" dns_zone_id = "" ttl = "" ptr = "" } } } } ``` Where `ptr` indicates if you need to create a PTR record, `true` or `false`. In a DNS record FQDN, you can use a template with variables: * `{instance_group.id}`: Instance group ID. * `{instance.index}`: Unique instance number in the group that can be any value from 1 to N, where N is the number of instances in the group. * `{instance.index_in_zone}`: Instance number in the zone. It is unique for a specific instance group within a zone. * `{instance.short_id}`: Instance ID that is unique within the group. It consists of four alphabetic characters. * `{instance.zone_id}`: Zone ID. For more information, see [this Terraform provider guide](../../../terraform/resources/kubernetes_node_group.md). 1. Validate your configuration files. 1. In the command line, go to the folder where you created the configuration file. 1. Run a check using this command: ```bash terraform plan ``` If the configuration is described correctly, the terminal will display a list of created resources and their parameters. If the configuration contains any errors, Terraform will point them out. This is a test step; no resources will be created. 1. Create a Managed Service for Kubernetes node group. 1. If the configuration does not contain any errors, run this command: ```bash terraform apply ``` 1. Confirm that you want to create the resources. After this, all required resources will be created in the specified folder and the IP addresses of the VMs will be displayed in the terminal. You can check the new resources and their configuration using the [management console](https://console.yandex.cloud). {% note warning "Timeouts" %} The Terraform provider sets time limits for operations with Managed Service for Kubernetes cluster node groups: * Creating and editing: 60 minutes. * Deleting: 20 minutes. Operations in excess of this time will be interrupted. {% cut "How do I modify these limits?" %} Add the `timeouts` section to the cluster node group description, e.g.: ```hcl resource "yandex_kubernetes_node_group" "" { ... timeouts { create = "1h30m" update = "1h30m" delete = "60m" } } ``` {% endcut %} {% endnote %} - API {#api} {% note info %} Below, see parameters for the REST API request body. To specify a parameter in a gRPC API call, convert its name from [lower CamelCase](https://en.wikipedia.org/wiki/Camel_case) to [snake_case](https://en.wikipedia.org/wiki/Snake_case), e.g., `minResourcePresetId` becomes `min_resource_preset_id`. {% endnote %} Use the [create](../../managed-kubernetes/api-ref/NodeGroup/create.md) REST API method for the [NodeGroup](../../managed-kubernetes/api-ref/NodeGroup/index.md) resource or the [NodeGroupService/Create](../../managed-kubernetes/api-ref/grpc/NodeGroup/create.md) gRPC API call, and provide the following in the request: * [Managed Service for Kubernetes cluster](../../concepts/index.md#kubernetes-cluster) ID in the `clusterId` parameter. You can get it with the [list of Managed Service for Kubernetes clusters in the folder](../kubernetes-cluster/kubernetes-cluster-list.md#list). * [Managed Service for Kubernetes node group configuration](../../concepts/index.md#config) in the `nodeTemplate` parameter. * [Network acceleration type](../../../compute/concepts/software-accelerated-network.md) in the `nodeTemplate.networkSettings.type` parameter. {% note warning %} Before enabling a [software-accelerated network](../../../compute/concepts/software-accelerated-network.md), make sure you have [enough cloud resources](../../concepts/limits.md) to create an additional [Managed Service for Kubernetes node](../../concepts/index.md#node-group). {% endnote %} * [containerd](https://containerd.io/) runtime environment in the `nodeTemplate.containerRuntimeSettings.type` parameter. * Node group [cloud labels](../../concepts/index.md#node-labels) in the `nodeTemplate.labels` parameter. * Node group [Kubernetes labels](../../concepts/index.md#node-labels) in the `nodeLabels` parameter. * [Reserved instance pool](../../../compute/concepts/reserved-pools.md) ID in the `nodeTemplate.reservedInstancePoolId` parameter. For more information, see [Creating a group with nodes from a Yandex Compute Cloud reserved instance pool](node-group-create-in-instance-pool.md). * [Scaling settings](../../concepts/autoscale.md#ca) in the `scalePolicy` parameter. You cannot change the scaling type after creating a node group. * Node group deployment settings in the `deployPolicy` parameter: * `maxExpansion`: Maximum number of nodes by which you can expand the group when updating it. {% note warning %} If the node group settings allow the group to scale up automatically during updates, make sure your cloud has enough [resources](../../concepts/limits.md). [Increase](../../../quota-manager/operations/set-quotas.md) quotas, if needed. {% endnote %} * `maxUnavailable`: Maximum number of unavailable nodes in the group when updating it. * Managed Service for Kubernetes node group [placement settings](../../../overview/concepts/geo-scope.md) in the `allocationPolicy` parameters. {% note warning %} You can only place an autoscaling node group in one [availability zone](../../../overview/concepts/geo-scope.md). {% endnote %} {% note warning %} Do not change or delete Virtual Private Cloud resources used by the Managed Service for Kubernetes cluster. This may cause cluster errors or make it impossible to delete later. {% endnote %} * [Maintenance](../../concepts/release-channels-and-updates.md#updates) window settings in the `maintenancePolicy` parameters. * [Deployment policy](../../concepts/node-group/deploy-policy.md) parameters: * `deployPolicy.maxExpansion`: Maximum number of nodes by which you can expand the group when modifying or updating it. The default value is `3`. * `deployPolicy.maxUnavailable`: Maximum number of nodes that may be unavailable while the group is being modified or updated. The default value is `0`. For more information, see [Node group deployment policy in Managed Service for Kubernetes](../../concepts/node-group/deploy-policy.md). * List of settings to update in the `updateMask` parameter. {% note warning %} The API method will assign default values to all the parameters of the object you are modifying unless you explicitly provide them in your request. To avoid this, list the settings you want to change in the `updateMask` parameter as a single comma-separated string. {% endnote %} * To assign random public [IP addresses](../../../vpc/concepts/address.md) from the Yandex Cloud pool to the nodes, provide the `IPV4` value in the `nodeTemplate.networkInterfaceSpecs.primaryV4AddressSpec.oneToOneNatSpec.ipVersion` parameter. Public IP addresses are required for [connecting](../connect/index.md) to nodes from the internet and to connect nodes to external resources, such as Docker image registries ([Container Registry](../../../container-registry/concepts/index.md), [Cloud Registry](../../../cloud-registry/concepts/index.md), or [Docker Hub](https://hub.docker.com/)) and [Object Storage](../../../storage/concepts/bucket.md) buckets. To provide nodes with internet access, assign them a public IP address and [configure](../connect/security-groups.md#rules-internal-nodegroup) a security group. As an alternative to public IP addresses, you can also create and configure a [NAT gateway](../../../vpc/operations/create-nat-gateway.md) or a [NAT instance](../../../vpc/tutorials/nat-instance/index.md) in the subnet hosting the nodes. If you assign public IP addresses to your cluster nodes and then configure a NAT gateway or NAT instance, you will lose internet access via the public addresses. For more information, see [this Yandex Virtual Private Cloud guide](../../../vpc/concepts/routing.md#internet-routes). * For nodes to use [non-replicated disks](../../../compute/concepts/disk.md#disks_types), provide `network-ssd-nonreplicated` for the `nodeTemplate.bootDiskSpec.diskTypeId` parameter. You can only change the size of non-replicated disks in 93 GB increments. The maximum size of this type of disk is 4 TB. {% note alert %} Non-replicated disks have no redundancy. If a disk fails, its data will be permanently lost. Learn more in [Non-replicated disks and ultra high-speed network storages with three replicas (SSD)](../../../compute/concepts/disk.md#nr-disks). {% endnote %} * To enable Managed Service for Kubernetes group nodes to use [unsafe kernel parameters](../../concepts/index.md#node-group), provide their names in the `allowedUnsafeSysctls` parameter. * To set [taints](../../concepts/index.md#taints-tolerations), provide their values in the `nodeTaints` parameter. * To set a template for Managed Service for Kubernetes node names, provide it in the `nodeTemplate.name` parameter. The name is unique if the template contains at least one of the following variables: * `{instance_group.id}`: Instance group ID. * `{instance.index}`: Unique instance number in the group that can be any value from 1 to N, where N is the number of instances in the group. * `{instance.index_in_zone}`: Instance number in the zone. It is unique for a specific instance group within a zone. * `{instance.short_id}`: Instance ID that is unique within the group. It consists of four alphabetic characters. * `{instance.zone_id}`: Zone ID. For example, `prod-{instance.short_id}-{instance_group.id}`. If you do not specify the name, the default value is used: `{instance_group.id}-{instance.short_id}`. * To create a node group with [preemptible VMs](../../../compute/concepts/preemptible-vm.md), provide the `nodeTemplate.schedulingPolicy.preemptible` parameter: {% note tip %} To evict pods automatically from a preemptible node when shutting it down, use the Yandex Cloud Marketplace product called [node-sitter](../applications/node-sitter.md). {% endnote %} * To specify a [placement group](../../../compute/concepts/placement-groups.md) for Managed Service for Kubernetes nodes, provide the placement group ID in the `nodeTemplate.placementPolicy.placementGroupId` parameter. {% note info %} The placement group determines the maximum available node group size: * In an instance group with the [spread placement](../../../compute/concepts/placement-groups.md#spread) strategy, the maximum number of instances depends on the [limits](../../../compute/concepts/limits.md#compute-limits-vm). * In an instance group with the [partition placement](../../../compute/concepts/placement-groups.md#partition) strategy, the maximum number of instances in a partition depends on the [quotas](../../../compute/concepts/limits.md#vm-quotas). {% endnote %} * To add metadata for nodes, provide it in the `nodeTemplate.metadata` parameter. Use metadata to configure the method of connecting to nodes in a node group. You can configure one method only because they are mutually exclusive. To connect to nodes in a node group, specify metadata for the selected connection method: * To connect to nodes via OS Login, add metadata with the `enable-oslogin` key set to `true`. For more information on how to configure and use OS Login, see [Connecting to a node via OS Login](../node-connect-oslogin.md). * To connect to nodes using SSH keys, add metadata with the `ssh-keys` key and its value listing the connection credentials. For more information about preparing, configuring, and using SSH keys, see [Connecting to a node over SSH](../node-connect-ssh.md). {% note warning %} Metadata settings can affect the behavior and health of nodes in a group. Change these settings only if you know exactly what you want to do. Providing user data in the metadata with the `user-data` key is not supported. {% endnote %} Add metadata by specifying one or multiple `key=value` pairs separated by commas. The key value is provided explicitly. You can [change the metadata list after you create a cluster](node-group-update.md#switch-node-connect-mode). * To add [DNS records](../../../dns/concepts/resource-record.md), provide their settings in the `nodeTemplate.networkInterfaceSpecs.primaryV4AddressSpec.dnsRecordSpecs` parameter. In a DNS record's FQDN, you can use the `nodeTemplate.name` node name template with variables. {% endlist %} Creating a group of Managed Service for Kubernetes nodes may take a few minutes depending on the number of nodes. Individual nodes in node groups are Yandex Compute Cloud virtual machines with automatically generated names. To configure nodes, [follow the node group management guides](../index.md#node-group). {% note alert %} Do not change node VM settings, including names, network interfaces, and SSH keys, using the Compute Cloud interfaces or SSH connections to the VM. This can disrupt the operation of individual nodes, node groups, and the whole Managed Service for Kubernetes cluster. {% endnote %} ## Examples {#examples} Create a node group for the Managed Service for Kubernetes cluster with the following test specifications: * Name: `k8s-demo-ng`. * Description: `Test node group`. * Node name template: `test-{instance.short_id}-{instance_group.id}`. * [Kubernetes cluster](../../concepts/index.md#kubernetes-cluster): Specify the [ID](../kubernetes-cluster/kubernetes-cluster-list.md) of an existing cluster, e.g., `cat0adul1fj0********`. * [Kubernetes version](../../concepts/release-channels-and-updates.md) on group nodes: `1.29`. * Node [platform](../../../compute/concepts/vm-platforms.md): `standard-v3`. * Number of vCPUs for nodes: Two. * [Guaranteed vCPU share](../../../compute/concepts/performance-levels.md): 50%. * [Disk size](../../../compute/concepts/disk.md#maximum-disk-size): 64 GB. * [Disk type](../../../compute/concepts/disk.md#disks_types): `network-ssd`. * Number of nodes: One. * [Deployment policy](../../concepts/node-group/deploy-policy.md): * Maximum number of nodes by which you can expand the group when modifying or updating it: `3`. * Maximum number of nodes that may be unavailable while the group is being modified or updated: `1`. * RAM: 2 GB. * [Update](../../concepts/release-channels-and-updates.md#updates) time: From 22:00 to 08:00 UTC. * [Network acceleration](../../../compute/concepts/software-accelerated-network.md) type: `standard` (no acceleration). * Network settings: * [Security group ID](../../../vpc/operations/security-group-get-info.md), e.g., `enp6saqnq4ie244g67sb`. * [Subnet ID](../../../vpc/operations/subnet-get-info.md), e.g., `e9bj3s90g9hm********`. * Assigning public and internal IP addresses to nodes: Enabled. * [Kubernetes label](../../concepts/index.md#node-labels): `node-label1=node-value1`. * Kubernetes [taint](../../concepts/index.md#taints-tolerations): `taint1=taint-value1:NoSchedule`. * [Cloud label](../../concepts/index.md#node-labels): `template-label1=template-value1`. * Permission to use [unsafe kernel parameters](../../concepts/index.md#config): Enabled. We added the `kernel.msg*` and `net.core.somaxconn` parameters. * VM being the only node of the group: [Preemptible](../../../compute/concepts/preemptible-vm.md). {% list tabs group=instructions %} - CLI {#cli} Run this command: ```bash yc managed-kubernetes node-group create \ --name k8s-demo-ng \ --description 'Test node group' \ --node-name test-{instance.short_id}-{instance_group.id} \ --cluster-id cat0adul1fj0******** \ --version 1.29 \ --platform-id standard-v3 \ --cores 2 \ --core-fraction 50 \ --disk-size 64 \ --disk-type network-ssd \ --fixed-size 1 \ --max-expansion 3 \ --max-unavailable 1 \ --memory 2 \ --daily-maintenance-window 'start=22:00,duration=10h' \ --network-acceleration-type standard \ --network-interface security-group-ids=enp6saqnq4ie244g67sb,subnets=e9bj3s90g9hm********,ipv4-address=nat \ --node-labels node-label1=node-value1 \ --node-taints taint1=taint-value1:NoSchedule \ --template-labels template-label1=template-value1 \ --allowed-unsafe-sysctls='kernel.msg*,net.core.somaxconn' \ --preemptible ``` - Terraform 1. Place the node group configuration file in the same folder as the [cluster description file](../kubernetes-cluster/kubernetes-cluster-create.md#kubernetes-cluster-create). ```hcl resource "yandex_kubernetes_node_group" "k8s-demo-ng" { name = "k8s-demo-ng" description = "Test node group" cluster_id = "cat0adul1fj0********" version = "1.29" instance_template { name = "test-{instance.short_id}-{instance_group.id}" platform_id = "standard-v3" resources { cores = 2 core_fraction = 50 memory = 2 } boot_disk { size = 64 type = "network-ssd" } network_acceleration_type = "standard" network_interface { security_group_ids = ["enp6saqnq4ie244g67sb"] subnet_ids = ["e9bj3s90g9hm********"] nat = true } scheduling_policy { preemptible = true } } scale_policy { fixed_scale { size = 1 } } deploy_policy { max_expansion = 3 max_unavailable = 1 } maintenance_policy { auto_upgrade = true auto_repair = true maintenance_window { start_time = "22:00" duration = "10h" } } node_labels = { node-label1 = "node-value1" } node_taints = ["taint1=taint-value1:NoSchedule"] labels = { "template-label1" = "template-value1" } allowed_unsafe_sysctls = ["kernel.msg*", "net.core.somaxconn"] } ``` 1. Make sure the configuration file is correct. 1. In the command line, go to the folder where you created the configuration file. 1. Run a check using this command: ```bash terraform plan ``` If the configuration is described correctly, the terminal will display a list of created resources and their parameters. If the configuration contains any errors, Terraform will point them out. This is a test step; no resources will be created. 1. Create a Managed Service for Kubernetes node group. 1. If the configuration does not contain any errors, run this command: ```bash terraform apply ``` 1. Confirm that you want to create the resources. After this, all required resources will be created in the specified folder and the IP addresses of the VMs will be displayed in the terminal. You can check the new resources and their configuration using the [management console](https://console.yandex.cloud). {% endlist %} ### See also {#see-also} * [Creating a group with nodes from a Yandex Compute Cloud reserved instance pool](node-group-create-in-instance-pool.md) * [Node group](../../concepts/index.md#node-group)