[Yandex Cloud documentation](../../../index.md) > [Yandex Cloud Stackland](../../index.md) > [Step-by-step guides](../index.md) > Users and groups > Restoring a YDB from a backup

# Restoring a YDB from a backup

The `ydb-restore.sh` script restores an [Identity and Access Management](../../concepts/components/iam.md) database in YDB from an archive created using the [backup script](ydb-backup.md).

The script is supported in Stackland version **26.3 and higher**.

{% note alert %}

Restoring a database irreversibly deletes the existing YDB and all its PVCs. Identity and Access Management data changes introduced after restoring a database from the selected backup will be lost.

The cluster will become unavailable during recovery. Schedule a maintenance window. If the script terminates with an error, the cluster may remain unavailable until it is successfully restarted.

{% endnote %}

## Getting started {#prerequisites}

1. Make sure the workstation (Linux or macOS) has `bash`, `awk`, `du`, `kubectl`, `jq`, `tar`, and `mktemp` installed. Run the script from the interactive terminal, as you will need to confirm the action before changing the cluster.
1. Prepare a kubeconfig file with administrative access to the cluster to restore. This access should remain when Identity and Access Management is not accessible. For example, you can use a kubeconfig with the `cluster-admin` role created [when installing the cluster](../../quickstart.md#installation-common).
1. Prepare the `tar.gz` archive created by the `ydb-backup.sh` script. If the current YDS is available and you need to save its data, [create a separate backup](ydb-backup.md) before restoring.
1. Make sure the workstation has space for the archive unpacking. In the cluster, the unpacked data copy is stored in a temporary PVC. The script reads its size automatically as the size of unpacked data + 10% rounded up to a GiB but not less than 10 GiB. After completion, the script deletes the temporary pod and PVC.
1. Specify the absolute paths to the kubeconfig and archive. Check the selected cluster, available storage classes, and archive readability:

   ```bash
   KUBECONFIG_PATH="<absolute_path_to_kubeconfig>"
   BACKUP_PATH="<absolute_path_to_tar.gz>"

   kubectl --kubeconfig "${KUBECONFIG_PATH}" config current-context
   kubectl --kubeconfig "${KUBECONFIG_PATH}" cluster-info
   kubectl --kubeconfig "${KUBECONFIG_PATH}" get storageclass
   tar -tzf "${BACKUP_PATH}" >/dev/null
   ```

1. Select a `StorageClass` supporting block (`volumeMode: Block`) and file (`volumeMode: Filesystem`) volumes. The script uses this storage class for new YDB disks and a temporary PVC with the backup.

   For a YDB, the script will create nine PVCs of `100Gi` each: three volumes for each of the three YDB storage nodes. Make sure the storage you selected has enough space for these PVCs and a temporary PVC with a backup, and the cluster has at least three nodes that disk preparation pods can run on.

   After restoring, the system uses one storage group with the selected `StorageClass` and three disks per node, even if the configuration was different prior to restoring.

## Recovering data {#restore}

1. Save the script below as `ydb-restore.sh` on your workstation.

   {% cut "ydb-restore.sh script" %}

   ```bash
   #!/usr/bin/env bash
   
   set -Eeuo pipefail
   
   usage() {
     printf 'Usage: %s --kubeconfig PATH --storage-class NAME [--timeout SECONDS] BACKUP.tar.gz\n' "$(basename -- "$0")"
   }
   
   log() {
     printf '[ydb-restore] %s\n' "$*"
   }
   
   fail() {
     printf '[ydb-restore] ERROR: %s\n' "$*" >&2
     exit 1
   }
   
   kubectl_target() {
     kubectl --kubeconfig "${KUBECONFIG_PATH}" "$@"
   }
   
   kubectl_iam() {
     kubectl_target --namespace "${IAM_NAMESPACE}" "$@"
   }
   
   wait_for_deployment_scale_down() {
     local namespace="$1"
     local deployment="$2"
     local selector="$3"
     local pod_name=""
     local -a pod_names=()
   
     kubectl_target --namespace "${namespace}" scale deployment "${deployment}" --replicas=0
     kubectl_target --namespace "${namespace}" rollout status deployment "${deployment}" \
       --timeout="${TIMEOUT}s"
     while IFS= read -r pod_name; do
       [[ -n "${pod_name}" ]] && pod_names+=("${pod_name}")
     done < <(kubectl_target --namespace "${namespace}" get pod \
       --selector "${selector}" --output name)
     if (( ${#pod_names[@]} > 0 )); then
       kubectl_target --namespace "${namespace}" wait --for=delete "${pod_names[@]}" \
         --timeout="${TIMEOUT}s"
     fi
   }
   
   wait_for_any_ready_pod() {
     local namespace="$1"
     local selector="$2"
     local description="$3"
     local deadline=$((SECONDS + TIMEOUT))
     local pod_name=""
   
     while (( SECONDS < deadline )); do
       while IFS= read -r pod_name; do
         [[ -n "${pod_name}" ]] || continue
         if kubectl_target --namespace "${namespace}" wait \
           --for=condition=Ready "${pod_name}" --timeout=1s >/dev/null 2>&1; then
           log "${description} is ready: ${namespace}/${pod_name#pod/}"
           return
         fi
       done < <(kubectl_target --namespace "${namespace}" get pod \
         --selector "${selector}" --output name)
       sleep 5
     done
   
     fail "No Ready ${description} Pod found in namespace ${namespace} within ${TIMEOUT} seconds"
   }
   
   secret_value() {
     local secret_name="$1"
     local key="$2"
   
     kubectl_iam get secret "${secret_name}" \
       --output "go-template={{ index .data \"${key}\" | base64decode }}"
   }
   
   confirm_restore() {
     local confirmation=""
   
     printf '\n'
     printf 'Restore YDB from backup\n'
     printf '\n'
     printf '  Cluster:     %s\n' "${target_api_server}"
     printf '  Backup:      %s\n' "${BACKUP_ARCHIVE}"
     printf '  Volume size: %s\n' "${VOLUME_SIZE}"
     printf '  Restore PVC: %s\n' "${RESTORE_PVC_SIZE}"
     printf '\n'
     printf 'WARNING: DESTRUCTIVE OPERATION\n'
     printf '\n'
     printf '  Existing YDB data and all PVCs will be permanently deleted.\n'
     printf '  The cluster will be unavailable during the restore.\n'
     printf '  A failed restore may leave it unavailable until the next attempt.\n'
     printf '\n'
     printf 'Proceed? Type "yes": '
   
     if ! IFS= read -r confirmation </dev/tty; then
       fail "Cannot read confirmation from the terminal"
     fi
     [[ "${confirmation}" == "yes" ]] \
       || fail "Restore was not confirmed"
   }
   
   decimal_at_least() {
     local value="$1"
     local minimum="$2"
   
     if (( ${#value} > ${#minimum} )); then
       return 0
     fi
     if (( ${#value} < ${#minimum} )); then
       return 1
     fi
     (( 10#${value} >= 10#${minimum} ))
   }
   
   volume_size_at_least_100gi() {
     local amount=""
     local suffix=""
   
     [[ "${VOLUME_SIZE}" =~ ^([1-9][0-9]*)(Ei|Pi|Ti|Gi|Mi|Ki|E|P|T|G|M|K)?$ ]] \
       || return 1
     amount="${BASH_REMATCH[1]}"
     suffix="${BASH_REMATCH[2]:-}"
   
     case "${suffix}" in
       Ei|Pi|Ti|E|P|T)
         return 0
         ;;
       Gi)
         decimal_at_least "${amount}" 100
         ;;
       G)
         decimal_at_least "${amount}" 108
         ;;
       Mi)
         decimal_at_least "${amount}" 102400
         ;;
       M)
         decimal_at_least "${amount}" 107375
         ;;
       Ki)
         decimal_at_least "${amount}" 104857600
         ;;
       K)
         decimal_at_least "${amount}" 107374183
         ;;
       '')
         decimal_at_least "${amount}" 107374182400
         ;;
     esac
   }
   
   confirm_small_volume_size() {
     local confirmation=""
   
     volume_size_at_least_100gi && return
   
     printf '\n'
     printf 'WARNING: SMALL YDB VOLUME\n'
     printf '\n'
     printf '  Requested:   %s\n' "${VOLUME_SIZE}"
     printf '  Recommended: at least 100Gi\n'
     printf '\n'
     printf '  Smaller volumes may make YDB unstable.\n'
     printf '\n'
     printf 'Proceed anyway? Type "yes": '
   
     if ! IFS= read -r confirmation </dev/tty; then
       fail "Cannot read volume size confirmation from the terminal"
     fi
     [[ "${confirmation}" == "yes" ]] \
       || fail "Volume size was not confirmed"
   }
   
   target_kubeconfig=""
   storage_class=""
   volume_size="100Gi"
   timeout="3600"
   backup_archive=""
   while (( $# > 0 )); do
     case "$1" in
       --kubeconfig)
         (( $# >= 2 )) || fail "Missing value for --kubeconfig"
         target_kubeconfig="$2"
         shift 2
         ;;
       --kubeconfig=*)
         target_kubeconfig="${1#*=}"
         shift
         ;;
       --volume-size)
         (( $# >= 2 )) || fail "Missing value for --volume-size"
         volume_size="$2"
         shift 2
         ;;
       --volume-size=*)
         volume_size="${1#*=}"
         shift
         ;;
       --storage-class)
         (( $# >= 2 )) || fail "Missing value for --storage-class"
         storage_class="$2"
         shift 2
         ;;
       --storage-class=*)
         storage_class="${1#*=}"
         shift
         ;;
       --timeout)
         (( $# >= 2 )) || fail "Missing value for --timeout"
         timeout="$2"
         shift 2
         ;;
       --timeout=*)
         timeout="${1#*=}"
         shift
         ;;
       -h|--help)
         usage
         exit 0
         ;;
       -*)
         printf 'Unknown option: %s\n' "$1" >&2
         usage >&2
         exit 2
         ;;
       *)
         [[ -z "${backup_archive}" ]] || {
           printf 'Only one backup archive may be specified\n' >&2
           usage >&2
           exit 2
         }
         backup_archive="$1"
         shift
         ;;
     esac
   done
   
   [[ -n "${target_kubeconfig}" ]] || {
     printf -- '--kubeconfig must explicitly identify the target cluster\n' >&2
     usage >&2
     exit 2
   }
   [[ -n "${storage_class}" ]] || {
     printf -- '--storage-class is required\n' >&2
     usage >&2
     exit 2
   }
   [[ -n "${backup_archive}" ]] || {
     printf 'Backup archive is required\n' >&2
     usage >&2
     exit 2
   }
   
   readonly KUBECONFIG_PATH="${target_kubeconfig}"
   readonly YDB_STORAGE_CLASS="${storage_class}"
   readonly VOLUME_SIZE="${volume_size}"
   readonly BACKUP_ARCHIVE="${backup_archive}"
   readonly TIMEOUT="${timeout}"
   readonly IAM_NAMESPACE="stackland-iam"
   readonly IAM_CONFIG_NAME="main"
   readonly IAM_CONFIG_RESOURCE="iamconfigs.stackland.yandex.cloud"
   readonly IAM_CONFIG_CRD="iamconfigs.stackland.yandex.cloud"
   readonly CILIUM_NETWORK_POLICY_CRD="ciliumnetworkpolicies.cilium.io"
   readonly RESTORE_STATE_SECRET="ydb-restore-state"
   readonly COMPONENT_MANAGER_NAMESPACE="stackland-component-manager"
   readonly COMPONENT_MANAGER_DEPLOYMENT="component-manager"
   readonly COMPONENT_MANAGER_SELECTOR="control-plane=component-manager,app.kubernetes.io/name=component-manager"
   readonly IAM_OPERATOR_DEPLOYMENT="iam-operator"
   readonly IAM_OPERATOR_SELECTOR="app.kubernetes.io/name=iam-operator,app.kubernetes.io/instance=iam-operator"
   readonly IAM_UBER_DEPLOYMENT="iam-uber"
   readonly IAM_UBER_SELECTOR="app=iam-uber"
   readonly YDB_RELEASE="ydb"
   readonly YDB_STORAGE_NAME="ydb-storage"
   readonly YDB_DATABASE_NAME="ydb-database"
   readonly YDB_ENDPOINT="grpcs://ydb-storage-grpc.${IAM_NAMESPACE}.svc.cluster.local:2135"
   readonly YDB_DATABASE="/ydb-storage/iam"
   readonly YDB_USER="operator"
   readonly CILIUM_NETWORK_POLICY_NAME="ydb-restore-block-iam-uber"
   readonly RESTORE_POD_NAME="ydb-restore"
   readonly RESTORE_PVC_NAME="ydb-restore"
   
   WORK_DIR=""
   RESTORE_POD_CREATED=false
   RESTORE_PVC_CREATED=false
   
   cleanup() {
     local exit_code=$?
     trap - EXIT INT TERM
   
     if [[ "${RESTORE_POD_CREATED}" == true ]]; then
       log "Deleting restore Pod ${IAM_NAMESPACE}/${RESTORE_POD_NAME}"
       kubectl_iam delete pod "${RESTORE_POD_NAME}" --ignore-not-found --wait=false >/dev/null || true
     fi
   
     if [[ "${RESTORE_PVC_CREATED}" == true ]]; then
       log "Deleting restore PVC ${IAM_NAMESPACE}/${RESTORE_PVC_NAME}"
       kubectl_iam delete pvc "${RESTORE_PVC_NAME}" --ignore-not-found --wait=false >/dev/null || true
     fi
   
     if [[ -n "${WORK_DIR}" && -d "${WORK_DIR}" ]]; then
       rm -rf -- "${WORK_DIR}"
     fi
   
     if (( exit_code != 0 )); then
       printf '[ydb-restore] Recovery stopped after an error. Re-run the same command to continue safely.\n' >&2
     fi
     exit "${exit_code}"
   }
   
   trap cleanup EXIT
   trap 'exit 130' INT
   trap 'exit 143' TERM
   
   for command_name in awk du jq kubectl tar mktemp; do
     command -v "${command_name}" >/dev/null 2>&1 \
       || fail "Required command is not installed: ${command_name}"
   done
   
   [[ -f "${KUBECONFIG_PATH}" ]] || fail "Target kubeconfig does not exist: ${KUBECONFIG_PATH}"
   [[ -f "${BACKUP_ARCHIVE}" ]] || fail "Backup archive does not exist: ${BACKUP_ARCHIVE}"
   [[ "${TIMEOUT}" =~ ^[1-9][0-9]*$ ]] || fail "Timeout must be a positive number of seconds"
   [[ "${VOLUME_SIZE}" =~ ^[1-9][0-9]*(Ei|Pi|Ti|Gi|Mi|Ki|E|P|T|G|M|K)?$ ]] \
     || fail "Volume size must be a positive Kubernetes quantity, for example 100Gi"
   
   target_api_server="$(kubectl_target config view --minify --output jsonpath='{.clusters[0].cluster.server}')"
   [[ -n "${target_api_server}" ]] || fail "Cannot resolve target Kubernetes API server"
   
   kubectl_target get namespace "${IAM_NAMESPACE}" >/dev/null \
     || fail "Required namespace does not exist: ${IAM_NAMESPACE}"
   kubectl_target --namespace "${COMPONENT_MANAGER_NAMESPACE}" \
     get deployment "${COMPONENT_MANAGER_DEPLOYMENT}" >/dev/null \
     || fail "Required Deployment does not exist: ${COMPONENT_MANAGER_NAMESPACE}/${COMPONENT_MANAGER_DEPLOYMENT}"
   kubectl_iam get deployment "${IAM_OPERATOR_DEPLOYMENT}" >/dev/null \
     || fail "Required Deployment does not exist: ${IAM_NAMESPACE}/${IAM_OPERATOR_DEPLOYMENT}"
   kubectl_iam get deployment "${IAM_UBER_DEPLOYMENT}" >/dev/null \
     || fail "Required Deployment does not exist: ${IAM_NAMESPACE}/${IAM_UBER_DEPLOYMENT}"
   kubectl_target get storageclass "${YDB_STORAGE_CLASS}" >/dev/null \
     || fail "Required StorageClass does not exist: ${YDB_STORAGE_CLASS}"
   kubectl_target get customresourcedefinition "${CILIUM_NETWORK_POLICY_CRD}" >/dev/null \
     || fail "Required CiliumNetworkPolicy CRD does not exist: ${CILIUM_NETWORK_POLICY_CRD}"
   
   log "Validating backup archive"
   tar -tzf "${BACKUP_ARCHIVE}" >/dev/null || fail "Backup archive is not a readable tar.gz file"
   backup_entry_count=0
   while IFS= read -r archive_entry; do
     normalized_entry="${archive_entry#./}"
     if [[ "${normalized_entry}" == /* ||
           "${normalized_entry}" == ".." ||
           "${normalized_entry}" == ../* ||
           "${normalized_entry}" == */.. ||
           "${normalized_entry}" == */../* ]]; then
       fail "Backup archive contains an unsafe path: ${archive_entry}"
     fi
     [[ -z "${normalized_entry}" ]] || (( backup_entry_count += 1 ))
   done < <(tar -tzf "${BACKUP_ARCHIVE}")
   (( backup_entry_count > 0 )) || fail "Backup archive is empty"
   while IFS= read -r archive_listing; do
     case "${archive_listing:0:1}" in
       l|h)
         fail "Backup archive must not contain symbolic or hard links"
         ;;
     esac
   done < <(tar -tvzf "${BACKUP_ARCHIVE}")
   
   WORK_DIR="$(mktemp -d "${TMPDIR:-/tmp}/ydb-restore.XXXXXX")"
   readonly ORIGINAL_IAM_CONFIG="${WORK_DIR}/iamconfig.json"
   readonly IAM_CONFIG_CEL_PATCH="${WORK_DIR}/iamconfig-cel-patch.json"
   readonly BACKUP_DIRECTORY="${WORK_DIR}/database"
   
   mkdir -p -- "${BACKUP_DIRECTORY}"
   tar -xzf "${BACKUP_ARCHIVE}" -C "${BACKUP_DIRECTORY}"
   [[ -f "${BACKUP_DIRECTORY}/database.pb" && -s "${BACKUP_DIRECTORY}/database.pb" ]] \
     || fail "Backup archive is not a YDB database dump: root database.pb is missing or empty"
   
   backup_size_kib="$(du -sk "${BACKUP_DIRECTORY}" | awk '{print $1}')"
   [[ "${backup_size_kib}" =~ ^[0-9]+$ ]] \
     || fail "Cannot calculate the unpacked backup size"
   padded_backup_size_kib=$(( (backup_size_kib * 110 + 99) / 100 ))
   restore_pvc_size_gib=$(( (padded_backup_size_kib + 1048575) / 1048576 ))
   if (( restore_pvc_size_gib < 10 )); then
     restore_pvc_size_gib=10
   fi
   readonly RESTORE_PVC_SIZE="${restore_pvc_size_gib}Gi"
   log "Unpacked backup occupies ${backup_size_kib}Ki; restore PVC size is ${RESTORE_PVC_SIZE}"
   
   confirm_restore
   confirm_small_volume_size
   
   # 0. Resolve the YDB image and update retry state.
   YDB_IMAGE="$(kubectl_iam get storages.ydb.tech "${YDB_STORAGE_NAME}" \
     --output jsonpath='{.spec.image.name}' 2>/dev/null || true)"
   if [[ -n "${YDB_IMAGE}" ]]; then
     log "Updating the retry state with YDB image ${YDB_IMAGE}"
     kubectl_iam create secret generic "${RESTORE_STATE_SECRET}" \
       --from-literal="ydb-image=${YDB_IMAGE}" \
       --dry-run=client --output yaml \
       | kubectl_iam apply --filename -
   elif kubectl_iam get secret "${RESTORE_STATE_SECRET}" >/dev/null 2>&1; then
     log "YDB Storage is absent; using the image from Secret ${IAM_NAMESPACE}/${RESTORE_STATE_SECRET}"
     YDB_IMAGE="$(secret_value "${RESTORE_STATE_SECRET}" "ydb-image")"
   else
     fail "Cannot resolve the YDB image and retry Secret ${IAM_NAMESPACE}/${RESTORE_STATE_SECRET} does not exist"
   fi
   
   [[ -n "${YDB_IMAGE}" ]] || fail "Resolved YDB image is empty"
   readonly YDB_IMAGE
   log "Using YDB image ${YDB_IMAGE} and StorageClass ${YDB_STORAGE_CLASS}"
   
   # 1. Stop component-manager.
   log "Scaling ${COMPONENT_MANAGER_NAMESPACE}/${COMPONENT_MANAGER_DEPLOYMENT} to 0"
   wait_for_deployment_scale_down \
     "${COMPONENT_MANAGER_NAMESPACE}" "${COMPONENT_MANAGER_DEPLOYMENT}" "${COMPONENT_MANAGER_SELECTOR}"
   
   # 2. Stop IAM workloads.
   log "Scaling ${IAM_NAMESPACE}/${IAM_OPERATOR_DEPLOYMENT} to 0"
   wait_for_deployment_scale_down \
     "${IAM_NAMESPACE}" "${IAM_OPERATOR_DEPLOYMENT}" "${IAM_OPERATOR_SELECTOR}"
   log "Scaling ${IAM_NAMESPACE}/${IAM_UBER_DEPLOYMENT} to 0"
   wait_for_deployment_scale_down \
     "${IAM_NAMESPACE}" "${IAM_UBER_DEPLOYMENT}" "${IAM_UBER_SELECTOR}"
   
   # 3. Delete the existing YDB Helm release without Helm.
   log "Removing finalizers from YDB ServiceAccounts"
   while IFS= read -r service_account_name; do
     [[ -n "${service_account_name}" ]] || continue
     kubectl_iam patch serviceaccount "${service_account_name}" \
       --type=merge --patch '{"metadata":{"finalizers":null}}'
   done < <(kubectl_iam get serviceaccounts \
     --selector "app.kubernetes.io/instance=${YDB_RELEASE},app.kubernetes.io/managed-by=Helm" \
     --output jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}')
   
   log "Deleting resources of Helm release ${IAM_NAMESPACE}/${YDB_RELEASE} without Helm"
   kubectl_iam delete databases.ydb.tech \
     --selector "app.kubernetes.io/instance=${YDB_RELEASE},app.kubernetes.io/managed-by=Helm" \
     --ignore-not-found --wait=true --timeout="${TIMEOUT}s"
   kubectl_iam delete storages.ydb.tech \
     --selector "app.kubernetes.io/instance=${YDB_RELEASE},app.kubernetes.io/managed-by=Helm" \
     --ignore-not-found --wait=true --timeout="${TIMEOUT}s"
   kubectl_iam delete persistentvolumeclaims \
     --selector "ydb.tech/statefulset-name=${YDB_STORAGE_NAME}" \
     --ignore-not-found --wait=true --timeout="${TIMEOUT}s"
   
   while IFS= read -r resource_type; do
     [[ -n "${resource_type}" ]] || continue
     RELEASE_RESOURCES=()
     while IFS= read -r release_resource; do
       [[ -n "${release_resource}" ]] && RELEASE_RESOURCES+=("${release_resource}")
     done < <(kubectl_iam get "${resource_type}" \
       --selector "app.kubernetes.io/instance=${YDB_RELEASE},app.kubernetes.io/managed-by=Helm" \
       --output name --ignore-not-found 2>/dev/null || true)
     if (( ${#RELEASE_RESOURCES[@]} > 0 )); then
       kubectl_iam delete "${RELEASE_RESOURCES[@]}" \
         --ignore-not-found --wait=true --timeout="${TIMEOUT}s"
     fi
   done < <(kubectl_target api-resources --verbs=list,delete --namespaced --output name)
   
   kubectl_iam delete secret --selector "owner=helm,name=${YDB_RELEASE}" \
     --ignore-not-found --wait=true --timeout="${TIMEOUT}s"
   log "Helm release ${IAM_NAMESPACE}/${YDB_RELEASE} resources deleted"
   
   # 4. Update IAMConfig in place.
   kubectl_target get "${IAM_CONFIG_RESOURCE}" "${IAM_CONFIG_NAME}" \
     --output json >"${ORIGINAL_IAM_CONFIG}" \
     || fail "Required IAMConfig does not exist: ${IAM_CONFIG_NAME}"
   
   log "Temporarily removing all IAMConfig CEL rules"
   kubectl_target get customresourcedefinition "${IAM_CONFIG_CRD}" --output json \
     | jq '[
         paths(type == "object" and has("x-kubernetes-validations")) as $path
         | {
             op: "remove",
             path: (
               "/" + (
                 ($path + ["x-kubernetes-validations"])
                 | map(tostring | gsub("~"; "~0") | gsub("/"; "~1"))
                 | join("/")
               )
             )
           }
       ]' >"${IAM_CONFIG_CEL_PATCH}"
   kubectl_target patch customresourcedefinition "${IAM_CONFIG_CRD}" \
     --type=json --patch-file "${IAM_CONFIG_CEL_PATCH}"
   
   restore_iam_config_patch="$(jq --null-input --compact-output \
     --arg storage_class "${YDB_STORAGE_CLASS}" --arg volume_size "${VOLUME_SIZE}" '
       {
         spec: {
           settings: {
             ydb: {
               storage: {
                 disksPerNode: 3,
                 groupsCount: 1,
                 storageClass: $storage_class,
                 volumeSize: $volume_size
               }
             }
           }
         }
       }
     ')"
   
   log "Updating IAMConfig ${IAM_CONFIG_NAME} YDB storage settings in place"
   kubectl_target patch "${IAM_CONFIG_RESOURCE}" "${IAM_CONFIG_NAME}" \
     --type=merge --patch "${restore_iam_config_patch}"
   
   restore_iam_status_patch="$(jq --compact-output '
       (.status // {}) as $status
       | {
           status: (
             reduce [
               "ydbStorageReady",
               "ydbDiskExpand",
               "ydbGroupExpand",
               "ydbCurrentGroupsCount",
               "ydbDatabaseReady",
               "ydbStorageInitialized"
             ][] as $field (
               {};
               if ($status | type) == "object" and ($status | has($field)) then
                 . + {($field): null}
               else
                 .
               end
             )
             + if ($status | type) == "object" and
                 ($status.conditions | type) == "array" and
                 any($status.conditions[]; .type == "YDBStorageReady" or .type == "YDBDatabaseReady") then
               {
                 conditions: [
                   $status.conditions[]
                   | select(
                       .type != "YDBStorageReady" and
                       .type != "YDBDatabaseReady"
                     )
                 ]
               }
             else
               {}
               end
           )
         }
     ' "${ORIGINAL_IAM_CONFIG}")"
   
   log "Removing YDB fields from IAMConfig ${IAM_CONFIG_NAME} status"
   kubectl_target patch "${IAM_CONFIG_RESOURCE}" "${IAM_CONFIG_NAME}" \
     --subresource=status --type=merge --patch "${restore_iam_status_patch}"
   
   # 5. Isolate iam-uber from YDB.
   log "Blocking egress from iam-uber while YDB is restored"
   kubectl_target apply --filename - <<EOF
   apiVersion: cilium.io/v2
   kind: CiliumNetworkPolicy
   metadata:
     name: ${CILIUM_NETWORK_POLICY_NAME}
     namespace: ${IAM_NAMESPACE}
   spec:
     endpointSelector:
       matchLabels:
         app: iam-uber
     egressDeny:
       - toEntities:
           - all
   EOF
   kubectl_iam get "ciliumnetworkpolicy/${CILIUM_NETWORK_POLICY_NAME}" >/dev/null
   log "Waiting for CiliumNetworkPolicy propagation"
   # CiliumNetworkPolicy has no portable readiness condition confirming propagation to every Cilium agent.
   sleep 30
   
   # 6. Create and obliterate the new YDB storage PVCs.
   log "Deleting stale restore and disk preparation workloads"
   kubectl_iam delete pod "${RESTORE_POD_NAME}" --ignore-not-found --wait=true --timeout="${TIMEOUT}s"
   kubectl_iam delete jobs \
     ydb-restore-obliterate-0 \
     ydb-restore-obliterate-1 \
     ydb-restore-obliterate-2 \
     --ignore-not-found --wait=true --timeout="${TIMEOUT}s"
   
   YDB_PVCS=()
   for node_index in 0 1 2; do
     for disk_number in 0 1 2; do
       printf -v disk_index '%02d' "${disk_number}"
       YDB_PVCS+=("${YDB_STORAGE_NAME}-${disk_index}-${YDB_STORAGE_NAME}-${node_index}")
     done
   done
   
   log "Deleting old YDB storage PVCs"
   kubectl_iam delete pvc --selector "ydb.tech/statefulset-name=${YDB_STORAGE_NAME}" \
     --ignore-not-found --wait=true --timeout="${TIMEOUT}s"
   
   log "Creating nine ${VOLUME_SIZE} YDB storage PVCs"
   for pvc_name in "${YDB_PVCS[@]}"; do
     kubectl_target apply --filename - <<EOF
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: ${pvc_name}
     namespace: ${IAM_NAMESPACE}
     labels:
       ydb.tech/statefulset-name: ${YDB_STORAGE_NAME}
   spec:
     accessModes:
       - ReadWriteOnce
     volumeMode: Block
     storageClassName: ${YDB_STORAGE_CLASS}
     resources:
       requests:
         storage: ${VOLUME_SIZE}
   EOF
   done
   
   OBLITERATE_JOBS=()
   log "Creating disk obliterate Jobs"
   for node_index in 0 1 2; do
     obliterate_job="ydb-restore-obliterate-${node_index}"
     OBLITERATE_JOBS+=("${obliterate_job}")
     kubectl_target apply --filename - <<EOF
   apiVersion: batch/v1
   kind: Job
   metadata:
     name: ${obliterate_job}
     namespace: ${IAM_NAMESPACE}
     labels:
       app.kubernetes.io/name: ydb
       app.kubernetes.io/component: ydb-restore-disk-obliterate
   spec:
     backoffLimit: 0
     activeDeadlineSeconds: ${TIMEOUT}
     template:
       metadata:
         labels:
           app.kubernetes.io/name: ydb
           app.kubernetes.io/component: ydb-restore-disk-obliterate
       spec:
         automountServiceAccountToken: false
         restartPolicy: Never
         priorityClassName: stackland-system
         affinity:
           podAntiAffinity:
             requiredDuringSchedulingIgnoredDuringExecution:
               - labelSelector:
                   matchLabels:
                     app.kubernetes.io/component: ydb-restore-disk-obliterate
                 topologyKey: kubernetes.io/hostname
         containers:
           - name: obliterate
             image: ${YDB_IMAGE}
             imagePullPolicy: IfNotPresent
             command:
               - /bin/bash
               - -c
               - |
                 set -Eeuo pipefail
                 for device in /dev/kikimr_ssd_00 /dev/kikimr_ssd_01 /dev/kikimr_ssd_02; do
                   /opt/ydb/bin/ydbd admin blobstorage disk obliterate "\${device}"
                 done
             securityContext:
               runAsUser: 0
               privileged: true
               capabilities:
                 add:
                   - SYS_RAWIO
             volumeDevices:
               - name: disk-00
                 devicePath: /dev/kikimr_ssd_00
               - name: disk-01
                 devicePath: /dev/kikimr_ssd_01
               - name: disk-02
                 devicePath: /dev/kikimr_ssd_02
         volumes:
           - name: disk-00
             persistentVolumeClaim:
               claimName: ${YDB_STORAGE_NAME}-00-${YDB_STORAGE_NAME}-${node_index}
           - name: disk-01
             persistentVolumeClaim:
               claimName: ${YDB_STORAGE_NAME}-01-${YDB_STORAGE_NAME}-${node_index}
           - name: disk-02
             persistentVolumeClaim:
               claimName: ${YDB_STORAGE_NAME}-02-${YDB_STORAGE_NAME}-${node_index}
   EOF
   done
   
   for obliterate_job in "${OBLITERATE_JOBS[@]}"; do
     log "Waiting for disk preparation Job ${IAM_NAMESPACE}/${obliterate_job}"
     if ! kubectl_iam wait --for=condition=Complete "job/${obliterate_job}" --timeout="${TIMEOUT}s"; then
       kubectl_iam logs "job/${obliterate_job}" --all-containers=true --prefix=true || true
       fail "Disk preparation Job failed or timed out: ${IAM_NAMESPACE}/${obliterate_job}"
     fi
   done
   
   for pvc_name in "${YDB_PVCS[@]}"; do
     kubectl_iam annotate pvc "${pvc_name}" ydb.tech/disk-obliterated=true --overwrite
   done
   kubectl_iam delete jobs "${OBLITERATE_JOBS[@]}" --wait=true --timeout="${TIMEOUT}s"
   log "YDB storage PVCs prepared"
   
   # 7. Start component-manager.
   log "Scaling ${COMPONENT_MANAGER_NAMESPACE}/${COMPONENT_MANAGER_DEPLOYMENT} to 1"
   kubectl_target --namespace "${COMPONENT_MANAGER_NAMESPACE}" \
     scale deployment "${COMPONENT_MANAGER_DEPLOYMENT}" --replicas=1
   kubectl_target --namespace "${COMPONENT_MANAGER_NAMESPACE}" \
     rollout status deployment "${COMPONENT_MANAGER_DEPLOYMENT}" --timeout="${TIMEOUT}s"
   
   # 8. Wait for YDB Storage and Database to become ready.
   log "Waiting for Storage ${IAM_NAMESPACE}/${YDB_STORAGE_NAME}"
   kubectl_iam wait --for=create "storages.ydb.tech/${YDB_STORAGE_NAME}" --timeout="${TIMEOUT}s"
   kubectl_iam wait "storages.ydb.tech/${YDB_STORAGE_NAME}" \
     --for=jsonpath='{.status.state}'=Ready --timeout="${TIMEOUT}s"
   
   log "Waiting for Database ${IAM_NAMESPACE}/${YDB_DATABASE_NAME}"
   kubectl_iam wait --for=create "databases.ydb.tech/${YDB_DATABASE_NAME}" --timeout="${TIMEOUT}s"
   kubectl_iam wait "databases.ydb.tech/${YDB_DATABASE_NAME}" \
     --for=jsonpath='{.status.state}'=Ready --timeout="${TIMEOUT}s"
   
   # 9. Upload and restore the database backup.
   log "Deleting stale restore PVC ${IAM_NAMESPACE}/${RESTORE_PVC_NAME}"
   kubectl_iam delete pvc "${RESTORE_PVC_NAME}" \
     --ignore-not-found --wait=true --timeout="${TIMEOUT}s"
   
   log "Creating ${RESTORE_PVC_SIZE} restore PVC ${IAM_NAMESPACE}/${RESTORE_PVC_NAME}"
   RESTORE_PVC_CREATED=true
   kubectl_target apply --filename - <<EOF
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: ${RESTORE_PVC_NAME}
     namespace: ${IAM_NAMESPACE}
     labels:
       app.kubernetes.io/name: ydb
       app.kubernetes.io/component: database-restore
   spec:
     accessModes:
       - ReadWriteOnce
     volumeMode: Filesystem
     storageClassName: ${YDB_STORAGE_CLASS}
     resources:
       requests:
         storage: ${RESTORE_PVC_SIZE}
   EOF
   
   log "Creating restore Pod ${IAM_NAMESPACE}/${RESTORE_POD_NAME}"
   RESTORE_POD_CREATED=true
   kubectl_target apply --filename - <<EOF
   apiVersion: v1
   kind: Pod
   metadata:
     name: ${RESTORE_POD_NAME}
     namespace: ${IAM_NAMESPACE}
     labels:
       app.kubernetes.io/name: ydb
       app.kubernetes.io/component: database-restore
   spec:
     automountServiceAccountToken: false
     restartPolicy: Never
     securityContext:
       runAsNonRoot: true
       runAsUser: 65532
       runAsGroup: 65532
       fsGroup: 65532
       seccompProfile:
         type: RuntimeDefault
     containers:
       - name: restore
         image: ${YDB_IMAGE}
         imagePullPolicy: IfNotPresent
         command:
           - /bin/bash
           - -c
           - sleep infinity
         env:
           - name: HOME
             value: /tmp
         securityContext:
           allowPrivilegeEscalation: false
           capabilities:
             drop:
               - ALL
         volumeMounts:
           - name: backup
             mountPath: /backup
           - name: ydb-ca
             mountPath: /credentials/ydb/ca.crt
             subPath: ca.crt
             readOnly: true
           - name: ydb-password
             mountPath: /credentials/ydb/password
             subPath: password
             readOnly: true
     volumes:
       - name: backup
         persistentVolumeClaim:
           claimName: ${RESTORE_PVC_NAME}
       - name: ydb-ca
         secret:
           secretName: ydb-storage-grpc-tls
           items:
             - key: ca.crt
               path: ca.crt
       - name: ydb-password
         secret:
           secretName: ydb-storage-operator-user
           items:
             - key: password
               path: password
   EOF
   kubectl_iam wait --for=condition=Ready "pod/${RESTORE_POD_NAME}" --timeout="${TIMEOUT}s"
   
   log "Checking database ${YDB_DATABASE} availability"
   readiness_attempts=$(( (TIMEOUT + 4) / 5 ))
   for (( attempt = 1; attempt <= readiness_attempts; attempt++ )); do
     if kubectl_iam exec "${RESTORE_POD_NAME}" --container restore -- \
       /opt/ydb/bin/ydb \
       --ca-file /credentials/ydb/ca.crt \
       -e "${YDB_ENDPOINT}" \
       -d "${YDB_DATABASE}" \
       --user "${YDB_USER}" \
       --password-file /credentials/ydb/password \
       scheme ls >/dev/null 2>&1; then
       break
     fi
     if (( attempt == readiness_attempts )); then
       fail "Database ${YDB_DATABASE} did not become available within ${TIMEOUT} seconds"
     fi
     sleep 5
   done
   
   log "Uploading backup into Pod ${IAM_NAMESPACE}/${RESTORE_POD_NAME}"
   kubectl_iam exec "${RESTORE_POD_NAME}" --container restore -- /bin/mkdir -p /backup/database
   kubectl_iam cp "${BACKUP_DIRECTORY}/." "${RESTORE_POD_NAME}:/backup/database" --container restore
   
   log "Restoring ${YDB_DATABASE}"
   if kubectl_iam exec "${RESTORE_POD_NAME}" --container restore -- \
     timeout "${TIMEOUT}s" \
     /opt/ydb/bin/ydb \
     --ca-file /credentials/ydb/ca.crt \
     -e "${YDB_ENDPOINT}" \
     -d "${YDB_DATABASE}" \
     --user "${YDB_USER}" \
     --password-file /credentials/ydb/password \
     --assume-yes \
     admin database restore \
     --input /backup/database; then
     :
   else
     restore_exit_code=$?
     if (( restore_exit_code == 124 )); then
       fail "YDB restore timed out after ${TIMEOUT} seconds"
     fi
     fail "YDB restore failed with exit code ${restore_exit_code}"
   fi
   log "Database ${YDB_DATABASE} restored successfully"
   
   # wipe sensitive data
   kubectl_iam exec "${RESTORE_POD_NAME}" --container restore -- \
     timeout "${TIMEOUT}s" /bin/sh -eu -c '
       command -v find >/dev/null
       command -v shred >/dev/null
       if [ -d /backup/database ]; then
         find /backup/database -type f -exec shred --iterations=1 --zero --exact -- {} +
       fi
       sync
     ' >/dev/null 2>&1 || true
   
   kubectl_iam delete pod "${RESTORE_POD_NAME}" --wait=true --timeout="${TIMEOUT}s"
   RESTORE_POD_CREATED=false
   kubectl_iam delete pvc "${RESTORE_PVC_NAME}" --wait=true --timeout="${TIMEOUT}s"
   RESTORE_PVC_CREATED=false
   
   # 10. Restore iam-uber connectivity.
   log "Restoring network access for iam-uber"
   kubectl_iam delete ciliumnetworkpolicy "${CILIUM_NETWORK_POLICY_NAME}" \
     --ignore-not-found --wait=true --timeout="${TIMEOUT}s"
   
   # 11. Start IAM workloads.
   log "Scaling ${IAM_NAMESPACE}/${IAM_UBER_DEPLOYMENT} to 1"
   kubectl_iam scale deployment "${IAM_UBER_DEPLOYMENT}" --replicas=1
   log "Waiting for any Ready ${IAM_NAMESPACE}/${IAM_UBER_DEPLOYMENT} Pod"
   wait_for_any_ready_pod "${IAM_NAMESPACE}" "${IAM_UBER_SELECTOR}" "${IAM_UBER_DEPLOYMENT}"
   
   log "Scaling ${IAM_NAMESPACE}/${IAM_OPERATOR_DEPLOYMENT} to 1"
   kubectl_iam scale deployment "${IAM_OPERATOR_DEPLOYMENT}" --replicas=1
   log "Waiting for any Ready ${IAM_NAMESPACE}/${IAM_OPERATOR_DEPLOYMENT} Pod"
   wait_for_any_ready_pod "${IAM_NAMESPACE}" "${IAM_OPERATOR_SELECTOR}" "${IAM_OPERATOR_DEPLOYMENT}"
   
   log "Removing restore state Secret"
   kubectl_iam delete secret "${RESTORE_STATE_SECRET}" \
     --ignore-not-found --wait=false >/dev/null
   
   log "Restore completed successfully"
   ```

   {% endcut %}

1. Make the file executable:

   ```bash
   chmod +x ydb-restore.sh
   ```

1. Specify the selected storage class and run the script:

   ```bash
   STORAGE_CLASS="<StorageClass_name>"

   ./ydb-restore.sh \
     --kubeconfig "${KUBECONFIG_PATH}" \
     --storage-class "${STORAGE_CLASS}" \
     --timeout 3600 \
     "${BACKUP_PATH}"
   ```

   Where:

   * `--kubeconfig`: Required path to the kubeconfig file of the cluster to restore.
   * `--storage-class`: Required name of the existing `StorageClass` for new YDB volumes and a temporary PVC with the backup.
   * `--timeout`: Timeout for individual operations and for restoring the YDB dump, in seconds. The default value is `3600`. This is not a limit on the total duration of the script execution.
   * `BACKUP_PATH`: Required path to the backup archive.

1. Check the Kubernetes API server's address and archive path the script will output. To confirm the current data deletion, enter `yes` in the terminal.

1. Wait for this message: `Restore completed successfully`. Do not run multiple script instances at once and do not create backups during restoring.

## Checking the result {#check-result}

1. Check the YDB and Identity and Access Management state:

   ```bash
   kubectl --kubeconfig "${KUBECONFIG_PATH}" \
     --namespace stackland-iam \
     get storages.ydb.tech,databases.ydb.tech

   kubectl --kubeconfig "${KUBECONFIG_PATH}" \
     --namespace stackland-iam \
     get deployment iam-operator iam-uber

   kubectl --kubeconfig "${KUBECONFIG_PATH}" \
     --namespace stackland-iam \
     get pvc -l ydb.tech/statefulset-name=ydb-storage
   ```

   Expected result:

   * The `Storage/ydb-storage` and `Database/ydb-database` are in the `Ready` state.
   * `iam-operator` and `iam-uber` have one ready replica each.
   * The nine PVCs are in the `Bound` state and have the selected `StorageClass`.

1. Log in to the [management console](https://console.yandex.cloud) and make sure that Identity and Access Management users and settings are recovered from the selected backup.

## If recovery completes with an error {#retry}

Fix the issue specified in the script output and retry the same command with the same archive and parameters. If the timeout is exceeded, check the resource state and increase `--timeout` if needed.

If the `CiliumNetworkPolicy/ydb-restore-block-iam-uber` policy is still there in the `stackland-iam` namespace after the error, do not delete it manually. It blocks Identity and Access Management queries against the database being restored. At the next run, the script will reuse the policy and delete it after data recovery.

Re-running the script also requires confirmation by entering `yes` and recreates the YDB. Wait for the `Restore completed successfully` message and [check the result](#check-result).