[Yandex Cloud documentation](../../index.md) > [Yandex Managed Service for Kubernetes](../index.md) > [Tutorials](index.md) > Migrating resources to a different availability zone

# Migrating Kubernetes resources to a different availability zone


In a Managed Service for Kubernetes cluster, you can [migrate a node group and pod workload](#transfer-a-node-group) from one availability zone to another.

## Getting started {#before-you-begin}

If you do not have the Yandex Cloud CLI yet, [install and initialize it](../../cli/quickstart.md#install).

The folder used by default is the one specified when [creating](../../cli/operations/profile/profile-create.md) the CLI profile. To change the default folder, use the `yc config set folder-id <folder_ID>` command. You can also specify a different folder for any command using `--folder-name` or `--folder-id`. If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions.

If you have already installed the CLI, update to its latest version:

```bash
yc components update
```


## Migrate the node group and pod workload to a different availability zone {#transfer-a-node-group}

[Set up the node group](#prepare) and proceed with migration using one of the following methods:

* Migrating a node group directly to the new availability zone. It depends on the type of workload in the pods:

   * [Stateless workload](#stateless): Application functioning in pods during migration depends on how workload is distributed between cluster nodes. If the pods reside both in the node group you are migrating and the groups for which the availability zone remains unchanged, the applications keep running. If the pods only reside in the group you are migrating, both the pods and the applications in them must be briefly stopped.

      Stateless workload examples include the web server, Yandex Application Load Balancer [ingress controller](../../application-load-balancer/tools/k8s-ingress-controller/index.md), and REST API apps.

   * [Stateful workloads](#stateful): Pods and applications must be briefly stopped, regardless of how workload is distributed between cluster nodes.

      Stateful workload examples include databases and storages.

* [Gradually migrating stateless and stateful workloads](#gradual-migration) to a new node group: It involves creating a new node group in the new availability zone and gradually discontinuing the old nodes. This enables you to manage workload transfer.

### Getting started {#prepare}

1. Check if the `nodeSelector`, `affinity`, or `topology spread constraints` strategies are used to assign the pods to the group's nodes. For more information on strategies, see [this Kubernetes article](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/) and [High availability and fault tolerance](../concepts/usage-recommendations.md#high-availability). To check a pod's assignment to nodes and remove it:

   {% list tabs group=instructions %}

   - Management console {#console}

      1. In the [management console](https://console.yandex.cloud), select the folder with your Managed Service for Kubernetes cluster.
      1. Navigate to **Managed Service for&nbsp;Kubernetes**.
      1. Open the cluster page and navigate to **Workload**.
      1. On the **Pods** tab, open the pod page.
      1. Navigate to the **YAML** tab.
      1. Check if the pod manifest contains the following parameters and Kubernetes labels in them:

         * Parameters:

            * `spec.nodeSelector`
            * `spec.affinity`
            * `spec.topologySpreadConstraints`

         * Kubernetes labels set in the parameters:

            * `failure-domain.beta.kubernetes.io/zone`: `<availability_zone>`
            * `topology.kubernetes.io/zone`: `<availability_zone>`

         When the configuration includes at least one of these parameters containing at least one of the listed Kubernetes labels, node group and workload migration will not proceed.

      1. Check the pod manifest for dependencies on the following entities:

         * Availability zone you are migrating your resources from.
         * Specific nodes within the group.

      1. If you find any of the parameters, assignments, or dependencies listed above, remove them from the pod configuration:

         1. Copy the YAML configuration from the management console.
         1. Create a local YAML file and paste the copied configuration into it.
         1. Remove any availability zone assignments from the configuration. For example, if the `spec.affinity` parameter includes the `failure-domain.beta.kubernetes.io/zone` Kubernetes label, remove it.
         1. Apply the new configuration:

            ```bash
            kubectl apply -f <YAML_file_path>
            ```

         1. Make sure the pod status changed to `Running`:

            ```bash
            kubectl get pods
            ```

      1. Repeat these steps for each pod to check and update its configuration.

   {% endlist %}

1. Transfer persistent data, such as databases, message queues, monitoring servers, and log servers, to the new availability zone.

### Migrating stateless workloads {#stateless}

1. Create a subnet in the new availability zone and migrate the node group:

   {% list tabs group=instructions %}
   
   - CLI {#cli}
   
      1. Create a subnet:
   
         ```bash
         yc vpc subnet create \
            --name <subnet_name> \
            --zone <availability_zone> \
            --network-id <network_ID> \
            --range <subnet_CIDR>
         ```
   
         Where:
   
         * `--name`: Subnet name.
         * `--zone`: Availability zone, `ru-central1-a`, `ru-central1-b`, or `ru-central1-d`.
         * `--network-id`: ID of the network that contains the new subnet.
         * `--range`: List of IPv4 addresses for incoming or outgoing traffic, e.g., `10.0.0.0/22` or `192.168.0.0/16`. Make sure the addresses are unique within the network. The minimum subnet size is `/28`, while the maximum subnet size is `/16`. Only IPv4 is supported.
   
      1. Migrate the node group to the new availability zone. The example below shows a command for migrating a group placed in a single zone:
   
         ```bash
         yc managed-kubernetes node-group update \
            --id <node_group_ID> \
            --location zone=<availability_zone>,subnet-id=<subnet_ID> \
            --network-interface subnets=<subnet_ID>,`
                 `ipv4-address=nat,`
                 `security-group-ids=[<security_group_IDs>]
         ```
   
         Where:
   
         * `id`: ID of the node group you want to migrate to a different availability zone.
         * `zone`: Availability zone you want to migrate your node group to, `ru-central1-a`, `ru-central1-b`, or `ru-central1-d`.
         * `subnet-id` and `subnets`: ID of the new subnet you created earlier.
         * `ipv4-address`: IPv4 address assignment method. The `nat` value allows assigning public and internal IP addresses to nodes.
         * `security-group-ids`: List of [security group](../operations/connect/security-groups.md) IDs.
   
         {% note warning %}
   
         If you want to keep the values of other network parameters for the node group, specify them in the `network-interface` parameter as well. Otherwise, the group may be recreated with default values. For more information, see [Updating a Managed Service for Kubernetes node group](../operations/node-group/node-group-update.md).
   
         {% endnote %}
   
         Make sure to provide the `ipv4-address` and `security-group-ids` parameters in the command: this will allow assigning public IP addresses to the node group and keeping security groups within it.
   
         This command recreates the nodes within their group in the specified availability zone and subnet. When recreating, the deployment settings are considered: the maximum number of nodes by which you can increase or decrease the group size against the initial number of nodes.
   
         {% cut "How to migrate a node group placed in different availability zones" %}
   
         In this case, use the following command:
   
         ```bash
         yc managed-kubernetes node-group update \
            --id <node_group_ID> \
            --location zone=<availability_zone>,subnet-id=<subnet_ID> \
            ...
            --location zone=<availability_zone>,subnet-id=<subnet_ID> \
            --network-interface subnets=[<subnet_IDs>],`
                 `ipv4-address=nat,`
                 `security-group-ids=[<security_group_IDs>]
         ```
   
         Where:
   
         * `id`: ID of the node group you want to migrate to a different availability zone.
         * `zone`: Availability zone, `ru-central1-a`, `ru-central1-b`, or `ru-central1-d`. Specify the `location` parameters for each availability zone that will host the node group.
         * `subnet-id` and `subnets`: IDs of the subnets for the specified availability zones.
         * `ipv4-address`: IPv4 address assignment method. The `nat` value allows assigning public and internal IP addresses to nodes.
         * `security-group-ids`: List of [security group](../operations/connect/security-groups.md) IDs.
   
         {% endcut %}
   
   - Terraform {#tf}
   
      {% note alert %}
   
      To make sure your node group is not recreated (unless it is your intention to do so), analyze the output of the `terraform plan` and `terraform apply` commands before applying the configuration.
   
      You can migrate a node group to a different availability zone without recreating it only if the configuration file contains the `allocation_policy` section.
   
      {% endnote %}
   
      1. Edit the configuration file as follows:
         * Add a new subnet manifest (the `yandex_vpc_subnet` resource) in the availability zone to which you want to migrate the node group.
         * Change the node group location parameters (the `yandex_kubernetes_node_group` resource):
           * `allocation_policy.location.subnet_id`: Remove this parameter from the manifest.
           * `allocation_policy.location.zone`: Specify the availability zone you want to migrate the node group to.
           * `instance_template.network_interface.subnet_ids`: Specify the new subnet ID. Add this parameter to the manifest if missing.
   
         ```hcl
         resource "yandex_vpc_subnet" "my-new-subnet" {
            name           = "<subnet_name>"
            zone           = "<availability_zone>"
            network_id     = "<network_ID>"
            v4_cidr_blocks = ["<subnet_CIDR>"]
         }
         ...
         resource "yandex_kubernetes_node_group" "k8s-node-group" {
            allocation_policy {
               location {
                  zone = "<availability_zone>"
               }
            }
            ...
            instance_template {
               network_interface {
                  subnet_ids = [yandex_vpc_subnet.my-new-subnet.id]
                  ...
               }
               ...
            }
            ...
         }
         ```
   
         Where:
   
         * `name`: Subnet name.
         * `zone`: Availability zone you want to migrate your node group to, `ru-central1-a`, `ru-central1-b`, or `ru-central1-d`.
         * `network_id`: ID of the network that contains the new subnet.
         * `v4_cidr_blocks`: List of IPv4 addresses for incoming or outgoing traffic, e.g., `10.0.0.0/22` or `192.168.0.0/16`. Make sure the addresses are unique within the network. The minimum subnet size is `/28`, while the maximum subnet size is `/16`. Only IPv4 is supported.
         * `subnet_ids`: New subnet ID.
   
      1. Make sure the configuration file is correct.
   
         1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.
         1. Run this command:
         
            ```bash
            terraform validate
            ```
         
            Terraform will show any errors found in your configuration files.
   
      1. Confirm updating the resources.
   
         1. Run this command to view the planned changes:
         
            ```bash
            terraform plan
            ```
         
            If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.
         
         1. If everything looks correct, apply the changes:
            1. Run this command:
         
               ```bash
               terraform apply
               ```
         
            1. Confirm updating the resources.
            1. Wait for the operation to complete.
   
      The group nodes will be recreated in the specified availability zone and subnet. When recreating, the deployment settings are considered: the maximum number of nodes by which you can increase or decrease the group size against the initial number of nodes.
   
   {% endlist %}

1. Make sure the pods are running in the migrated node group:

   ```bash
   kubectl get po --output wide
   ```

   The output of this command shows the nodes on which your pods are currently running.

### Migrating stateful workloads {#stateful}

The migration is based on scaling the `StatefulSet` controller. To migrate stateful workloads:

1. Get a list of `StatefulSet` controllers to find the name of the one you need:

   ```bash
   kubectl get statefulsets
   ```

1. Get the number of pods managed by the controller:

   ```bash
   kubectl get statefulsets <controller_name> \
      -n default -o=jsonpath='{.status.replicas}'
   ```

   Save this value. You will need it to scale the `StatefulSet` controller once the migration of your stateful workloads is complete.

1. Reduce the number of pods to zero:

   ```bash
   kubectl scale statefulset <controller_name> --replicas=0
   ```

   This will stop the pods that are using disks, while retaining the [PersistentVolumeClaim](../concepts/volume.md#persistent-volume) Kubernetes API object (PVC).

1. For the [PersistentVolume](../concepts/volume.md#persistent-volume) object (PV) associated with `PersistentVolumeClaim`, change the `persistentVolumeReclaimPolicy` value from `Delete` to `Retain` to prevent accidental data loss.

   1. Get the name of the `PersistentVolume` object:

      ```bash
      kubectl get pv
      ```

   1. Edit the `PersistentVolume` object:

      ```bash
      kubectl edit pv <PV_name>
      ```

1. Check if the `PersistentVolume` object manifest contains the `spec.nodeAffinity` parameter:

    ```bash
    kubectl get pv <PV_name> --output='yaml'
    ```

    If the manifest contains the `spec.nodeAffinity` parameter with an availability zone specified in it, save this parameter. You will need to specify it in a new `PersistentVolume` object.

1. Create a `PersistentVolume` snapshot. For more information about snapshots, see [this Kubernetes article](https://kubernetes.io/docs/concepts/storage/volume-snapshots/).

   1. Get the name of the `PersistentVolumeClaim` object:

      ```bash
      kubectl get pvc
      ```

   1. Create the `snapshot.yaml` with the snapshot manifest and specify the `PersistentVolumeClaim` name in it:

      ```yaml
      apiVersion: snapshot.storage.k8s.io/v1
      kind: VolumeSnapshot
      metadata:
         name: new-snapshot-test-<number>
      spec:
         volumeSnapshotClassName: yc-csi-snapclass
         source:
            persistentVolumeClaimName: <PVC_name>
      ```

      If you are creating several snapshots for different `PersistentVolumeClaim` objects, specify the `<number>` (consecutive) to make sure each snapshot gets a unique `metadata.name` value.

   1. Create a snapshot:

      ```bash
      kubectl apply -f snapshot.yaml
      ```

   1. Make sure the snapshot was created:

      ```bash
      kubectl get volumesnapshots.snapshot.storage.k8s.io
      ```

   1. Make sure the [VolumeSnapshotContent](https://kubernetes.io/docs/concepts/storage/volume-snapshots/#introduction) Kubernetes API object was created:

      ```bash
      kubectl get volumesnapshotcontents.snapshot.storage.k8s.io
      ```

1. Get the snapshot ID:

   ```bash
   yc compute snapshot list
   ```

1. Create a [VM disk](../../compute/concepts/disk.md) from the snapshot:

   ```bash
   yc compute disk create \
      --source-snapshot-id <snapshot_ID> \
      --zone <availability_zone>
   ```

   In the command, specify the availability zone to which you are migrating your Managed Service for Kubernetes node group.

   Save the following parameters from the command output:
   * Disk ID from the `id` field.
   * Disk type from the `type_id` field.
   * Disk size from the `size` field.

1. Create a `PersistentVolume` Kubernetes API object from the new disk:

   1. Create the `persistent-volume.yaml` file with the `PersistentVolume` manifest:

      ```yaml
      apiVersion: v1
      kind: PersistentVolume
      metadata:
         name: new-pv-test-<number>
      spec:
         capacity:
            storage: <PersistentVolume_size>
         accessModes:
            - ReadWriteOnce
         csi:
            driver: disk-csi-driver.mks.ycloud.io
            fsType: ext4
            volumeHandle: <disk_ID>
         storageClassName: <disk_type>
      ```

      In the file, specify the parameters of the disk created from the snapshot:

      * `spec.capacity.storage`: Disk size.
      * `spec.csi.volumeHandle`: Disk ID.
      * `spec.storageClassName`: Disk type. Specify it in accordance with the following table:

         | Snapshot-based disk type | Disk type for the YAML file |
         | ----------- | ----------- |
         | `network-ssd` | `yc-network-ssd` |
         | `network-ssd-nonreplicated` | `yc-network-ssd-nonreplicated` |
         | `network-nvme` | `yc-network-nvme` |
         | `network-hdd` | `yc-network-hdd` |

      If you are creating several `PersistentVolume` objects, specify the `<number>` (consecutive) to make sure each gets a unique `metadata.name` value.

      Add the previously saved `spec.nodeAffinity` parameter to the manifest and specify the availability zone to which you are migrating your Managed Service for Kubernetes node group. If you skip it, the workload may run in a different availability zone where `PersistentVolume` is not available, causing an error.

      Here is an example of the `spec.nodeAffinity` parameter:

      ```yaml
      spec:
         ...
         nodeAffinity:
            required:
               nodeSelectorTerms:
               - matchExpressions:
                  - key: failure-domain.beta.kubernetes.io/zone
                    operator: In
                    values:
                       - ru-central1-d
      ```

   1. Create a `PersistentVolume` object:

      ```bash
      kubectl apply -f persistent-volume.yaml
      ```

   1. Make sure `PersistentVolume` was created:

      ```bash
      kubectl get pv
      ```

      The command output will display the `new-pv-test<number>` object.

   1. If you specified the `spec.nodeAffinity` parameter in the manifest, make sure to apply it for the `PersistentVolume` object:

       {% list tabs group=instructions %}

       - Management console {#console}

          1. In the [management console](https://console.yandex.cloud), select the folder with your Managed Service for Kubernetes cluster.
          1. Navigate to **Managed Service for&nbsp;Kubernetes**.
          1. Open the cluster page and navigate to **Persistent volumes**.
          1. On the **PersistentVolumes** tab, find the `new-pv-test-<number>` object and check the **Availability zone** field value. It should specify an availability zone. A dash means there is no assignment to an availability zone.

       {% endlist %}

   1. If you skipped the `spec.nodeAffinity` parameter in the manifest, you can add it by editing the `PersistentVolume` object:

      ```bash
      kubectl edit pv new-pv-test-<number>
      ```

1. Create a `PersistentVolumeClaim` object from the new `PersistentVolume`:

   1. Create the `persistent-volume-claim.yaml` file with the `PersistentVolumeClaim` manifest:

      ```yaml
      apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
         name: <PVC_name>
      spec:
         accessModes:
            - ReadWriteOnce
         resources:
            requests:
               storage: <PV_size>
         storageClassName: <disk_type>
         volumeName: new-pv-test-<number>
      ```

      In the file, set the following parameters:

      * `metadata.name`: Name of the `PersistentVolumeClaim` object you used to create the snapshot. You can get this name by running the `kubectl get pvc` command.
      * `spec.resources.requests.storage`: `PersistentVolume` size, which matches the size of the created disk.
      * `spec.storageClassName`: `PersistentVolume` disk type, which matches the disk type of the new `PersistentVolume`.
      * `spec.volumeName`: Name of the `PersistentVolume` object to base `PersistentVolumeClaim` on. You can get this name by running the `kubectl get pv` command.

   1. Delete the original `PersistentVolumeClaim` so you can replace it:

      ```bash
      kubectl delete pvc <PVC_name>
      ```

   1. Create a `PersistentVolumeClaim` object:

      ```bash
      kubectl apply -f persistent-volume-claim.yaml
      ```

   1. Make sure `PersistentVolumeClaim` was created:

      ```bash
      kubectl get pvc
      ```

      The command output will return the `PersistentVolumeClaim` size you specified in the YAML file.

1. Create a subnet in the new availability zone and migrate the node group:

   {% list tabs group=instructions %}
   
   - CLI {#cli}
   
      1. Create a subnet:
   
         ```bash
         yc vpc subnet create \
            --name <subnet_name> \
            --zone <availability_zone> \
            --network-id <network_ID> \
            --range <subnet_CIDR>
         ```
   
         Where:
   
         * `--name`: Subnet name.
         * `--zone`: Availability zone, `ru-central1-a`, `ru-central1-b`, or `ru-central1-d`.
         * `--network-id`: ID of the network that contains the new subnet.
         * `--range`: List of IPv4 addresses for incoming or outgoing traffic, e.g., `10.0.0.0/22` or `192.168.0.0/16`. Make sure the addresses are unique within the network. The minimum subnet size is `/28`, while the maximum subnet size is `/16`. Only IPv4 is supported.
   
      1. Migrate the node group to the new availability zone. The example below shows a command for migrating a group placed in a single zone:
   
         ```bash
         yc managed-kubernetes node-group update \
            --id <node_group_ID> \
            --location zone=<availability_zone>,subnet-id=<subnet_ID> \
            --network-interface subnets=<subnet_ID>,`
                 `ipv4-address=nat,`
                 `security-group-ids=[<security_group_IDs>]
         ```
   
         Where:
   
         * `id`: ID of the node group you want to migrate to a different availability zone.
         * `zone`: Availability zone you want to migrate your node group to, `ru-central1-a`, `ru-central1-b`, or `ru-central1-d`.
         * `subnet-id` and `subnets`: ID of the new subnet you created earlier.
         * `ipv4-address`: IPv4 address assignment method. The `nat` value allows assigning public and internal IP addresses to nodes.
         * `security-group-ids`: List of [security group](../operations/connect/security-groups.md) IDs.
   
         {% note warning %}
   
         If you want to keep the values of other network parameters for the node group, specify them in the `network-interface` parameter as well. Otherwise, the group may be recreated with default values. For more information, see [Updating a Managed Service for Kubernetes node group](../operations/node-group/node-group-update.md).
   
         {% endnote %}
   
         Make sure to provide the `ipv4-address` and `security-group-ids` parameters in the command: this will allow assigning public IP addresses to the node group and keeping security groups within it.
   
         This command recreates the nodes within their group in the specified availability zone and subnet. When recreating, the deployment settings are considered: the maximum number of nodes by which you can increase or decrease the group size against the initial number of nodes.
   
         {% cut "How to migrate a node group placed in different availability zones" %}
   
         In this case, use the following command:
   
         ```bash
         yc managed-kubernetes node-group update \
            --id <node_group_ID> \
            --location zone=<availability_zone>,subnet-id=<subnet_ID> \
            ...
            --location zone=<availability_zone>,subnet-id=<subnet_ID> \
            --network-interface subnets=[<subnet_IDs>],`
                 `ipv4-address=nat,`
                 `security-group-ids=[<security_group_IDs>]
         ```
   
         Where:
   
         * `id`: ID of the node group you want to migrate to a different availability zone.
         * `zone`: Availability zone, `ru-central1-a`, `ru-central1-b`, or `ru-central1-d`. Specify the `location` parameters for each availability zone that will host the node group.
         * `subnet-id` and `subnets`: IDs of the subnets for the specified availability zones.
         * `ipv4-address`: IPv4 address assignment method. The `nat` value allows assigning public and internal IP addresses to nodes.
         * `security-group-ids`: List of [security group](../operations/connect/security-groups.md) IDs.
   
         {% endcut %}
   
   - Terraform {#tf}
   
      {% note alert %}
   
      To make sure your node group is not recreated (unless it is your intention to do so), analyze the output of the `terraform plan` and `terraform apply` commands before applying the configuration.
   
      You can migrate a node group to a different availability zone without recreating it only if the configuration file contains the `allocation_policy` section.
   
      {% endnote %}
   
      1. Edit the configuration file as follows:
         * Add a new subnet manifest (the `yandex_vpc_subnet` resource) in the availability zone to which you want to migrate the node group.
         * Change the node group location parameters (the `yandex_kubernetes_node_group` resource):
           * `allocation_policy.location.subnet_id`: Remove this parameter from the manifest.
           * `allocation_policy.location.zone`: Specify the availability zone you want to migrate the node group to.
           * `instance_template.network_interface.subnet_ids`: Specify the new subnet ID. Add this parameter to the manifest if missing.
   
         ```hcl
         resource "yandex_vpc_subnet" "my-new-subnet" {
            name           = "<subnet_name>"
            zone           = "<availability_zone>"
            network_id     = "<network_ID>"
            v4_cidr_blocks = ["<subnet_CIDR>"]
         }
         ...
         resource "yandex_kubernetes_node_group" "k8s-node-group" {
            allocation_policy {
               location {
                  zone = "<availability_zone>"
               }
            }
            ...
            instance_template {
               network_interface {
                  subnet_ids = [yandex_vpc_subnet.my-new-subnet.id]
                  ...
               }
               ...
            }
            ...
         }
         ```
   
         Where:
   
         * `name`: Subnet name.
         * `zone`: Availability zone you want to migrate your node group to, `ru-central1-a`, `ru-central1-b`, or `ru-central1-d`.
         * `network_id`: ID of the network that contains the new subnet.
         * `v4_cidr_blocks`: List of IPv4 addresses for incoming or outgoing traffic, e.g., `10.0.0.0/22` or `192.168.0.0/16`. Make sure the addresses are unique within the network. The minimum subnet size is `/28`, while the maximum subnet size is `/16`. Only IPv4 is supported.
         * `subnet_ids`: New subnet ID.
   
      1. Make sure the configuration file is correct.
   
         1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.
         1. Run this command:
         
            ```bash
            terraform validate
            ```
         
            Terraform will show any errors found in your configuration files.
   
      1. Confirm updating the resources.
   
         1. Run this command to view the planned changes:
         
            ```bash
            terraform plan
            ```
         
            If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.
         
         1. If everything looks correct, apply the changes:
            1. Run this command:
         
               ```bash
               terraform apply
               ```
         
            1. Confirm updating the resources.
            1. Wait for the operation to complete.
   
      The group nodes will be recreated in the specified availability zone and subnet. When recreating, the deployment settings are considered: the maximum number of nodes by which you can increase or decrease the group size against the initial number of nodes.
   
   {% endlist %}

1. Restore the original number of pods managed by the `StatefulSet` controller:

   ```bash
   kubectl scale statefulset <controller_name> --replicas=<number_of_pods>
   ```

   This will start the pods in the migrated node group.

   In the command, specify the following parameters:

   * Name of the `StatefulSet` controller. You can get it by running the `kubectl get statefulsets` command.
   * Number of pods prior to scaling down the controller.

1. Make sure the pods are running in the migrated node group:

   ```bash
   kubectl get po --output wide
   ```

   The output of this command shows the nodes on which your pods are currently running.

1. Delete the unused `PersistentVolume` object, i.e., the one with the `Released` status.

   1. Get the name of the `PersistentVolume` object:

      ```bash
      kubectl get pv
      ```

   1. Delete the `PersistentVolume` object:

      ```bash
      kubectl delete pv <PV_name>
      ```

### Gradually migrating stateless and stateful workloads {#gradual-migration}

See below how to gradually migrate workloads from the old node group to the new one. For `PersistentVolume` and `PersistentVolumeClaim` migration steps, see [Migrating stateful workloads](#stateful).

1. [Create a new Managed Service for Kubernetes node group](../operations/node-group/node-group-create.md) in the new availability zone.

1. Disable running new pods in the old node group:

   ```bash
   kubectl cordon -l yandex.cloud/node-group-id=<old_node_group_ID>
   ```

1. For each node in the old node group, run the following command:

   ```bash
   kubectl drain <node_name> --ignore-daemonsets --delete-emptydir-data
   ```

   The pods will gradually move to the new node group.

1. Make sure the pods are running in the new node group:

   ```bash
   kubectl get po --output wide
   ```

1. [Delete the old node group](../operations/node-group/node-group-delete.md).