[Yandex Cloud documentation](../../index.md) > [Yandex Managed Service for Kubernetes](../index.md) > [Tutorials](index.md) > Creating a Kubernetes cluster with no internet access

# Creating and configuring a Kubernetes cluster with no internet access


You can create and configure a Managed Service for Kubernetes cluster with no internet connectivity. For this, you will need the following configuration:

* Managed Service for Kubernetes cluster and node group have no public address. You can only connect to such a cluster with a Yandex Cloud virtual machine.
* The cluster and node group are hosted by subnets with no internet access.
* Service accounts have no permissions to use resources with internet access, such as [Yandex Network Load Balancer](../../network-load-balancer/index.md).
* Cluster security groups restrict incoming and outgoing traffic.

To create a Managed Service for Kubernetes cluster with no internet access:

1. [Set up the infrastructure for Managed Service for Kubernetes](#infra).
1. [Set up a virtual machine](#vm).
1. [Check cluster availability](#check).
1. Optionally, [set up a connection to NTP servers](#ntp).
1. Optionally, [connect a private Docker image registry](#cert).

If you no longer need the resources you created, [delete them](#clear-out).


## Required paid resources {#paid-resources}

The support cost for this solution includes:

* Fee for using the master in a Managed Service for Kubernetes cluster (see [Managed Service for Kubernetes pricing](../pricing.md)).
* Fee for using computing resources, OS, and storage in cluster nodes and VMs (see [Compute Cloud pricing](../../compute/pricing.md)).
* Fee for a public IP address for a VM, which is used to connect to the cluster (see [Virtual Private Cloud pricing](../../vpc/pricing.md#prices-public-ip)).
* Key Management Service fee for the number of active key versions (with `Active` or `Scheduled For Destruction` for status) and completed cryptographic operations (see [Key Management Service pricing](../../kms/pricing.md)).


## Set up the infrastructure for Managed Service for Kubernetes {#infra}

{% list tabs group=instructions %}

- Manually {#manual}

   1. [Create these service accounts](../../iam/operations/sa/create.md):

      * `resource-sa` with the `k8s.clusters.agent`, `logging.writer`, and `kms.keys.encrypterDecrypter` [roles](../../iam/concepts/access-control/roles.md) for the folder where you want to create a Kubernetes cluster. This account will be used to create the resources for the Kubernetes cluster.
      * `node-sa` with the `container-registry.images.puller` role. The nodes will use this account to pull the required Docker images from the registry.

      {% note tip %}

      You can use the same service account to manage your Kubernetes cluster and its node groups.

      {% endnote %}

   1. [Create a Yandex Key Management Service symmetric encryption key](../../kms/operations/key.md#create) with the following settings:

      * **Name**: `my-kms-key`.
      * **Encryption algorithm**: `AES-256`.
      * **Rotation period, days**: 365 days.

   1. [Create a network](../../vpc/operations/network-create.md) named `my-net`.
   1. [Create a subnet](../../vpc/operations/subnet-create.md) named `my-subnet` with the `internal.` domain name.
   1. [Create security groups](../operations/connect/security-groups.md) for the Managed Service for Kubernetes cluster and its node groups.

        {% note warning %}
        
        The configuration of security groups determines performance and availability of the cluster and the services and applications running in it.
        
        {% endnote %}

   1. [Create a Managed Service for Kubernetes cluster](../operations/kubernetes-cluster/kubernetes-cluster-create.md#kubernetes-cluster-create) with the following settings:

      * **Service account for resources**: `resource-sa`.
      * **Service account for nodes**: `node-sa`.
      * **Encryption key**: `my-kms-key`.
      * **Public address**: No address.
      * **Cloud network**: `my-net`.
      * **Subnet**: `my-subnet`.
      * **Security groups**: Select the previously created security groups containing the rules for service traffic and Kubernetes API access.
      * **CIDR cluster**: `172.19.0.0/16`.
      * **CIDR services**: `172.20.0.0/16`.
      * **Write logs**: Enabled.
      * **Cluster Autoscaler logs**: Enabled.
      * **Event logs**: Enabled.
      * **Kubernetes API server logs**: Enabled.

   1. In the Managed Service for Kubernetes cluster, [create a node group](../operations/node-group/node-group-create.md) with the following settings:

      * **Public address**: No address.
      * **Security groups**: Select the previously created security groups containing the rules for service traffic, internet access to services, and SSH access to nodes.
      * **Location**: `my-subnet`.

- Terraform {#tf}

   1. If you do not have Terraform yet, [install it](../../tutorials/infrastructure-management/terraform-quickstart.md#install-terraform).
   1. [Get the authentication credentials](../../tutorials/infrastructure-management/terraform-quickstart.md#get-credentials). You can add them to environment variables or specify them later in the provider configuration file.
   1. [Configure and initialize a provider](../../tutorials/infrastructure-management/terraform-quickstart.md#configure-provider). There is no need to create a provider configuration file manually, you can [download it](https://github.com/yandex-cloud-examples/yc-terraform-provider-settings/blob/main/provider.tf).
   1. Place the configuration file in a separate working directory and [specify the parameter values](../../tutorials/infrastructure-management/terraform-quickstart.md#configure-provider). If you did not add the authentication credentials to environment variables, specify them in the configuration file.
   1. Download the [k8s-cluster-with-no-internet.tf](https://github.com/yandex-cloud-examples/yc-mk8s-cluster-without-internet/blob/main/k8s-cluster-with-no-internet.tf) configuration file to the same working directory. You need this file to create the following resources:

      * Network.
      * Route table.
      * Subnets.
      * Managed Service for Kubernetes cluster.
      * Managed Service for Kubernetes node group.
      * [Security groups](../../vpc/concepts/security-groups.md) which contain [rules](../operations/connect/security-groups.md) required for the Managed Service for Kubernetes cluster and its node groups.

        {% note warning %}
        
        The configuration of security groups determines performance and availability of the cluster and the services and applications running in it.
        
        {% endnote %}

      * [Service accounts](../../iam/concepts/users/service-accounts.md) for Kubernetes resources and nodes.
      * Yandex Key Management Service [symmetric encryption key](../../kms/concepts/key.md).

      The file is generated using the libraries of the [terraform-yc-vpc](https://github.com/terraform-yc-modules/terraform-yc-vpc) and [terraform-yc-kubernetes](https://github.com/terraform-yc-modules/terraform-yc-kubernetes) modules. For more information on the configuration of the resources you create using these modules, see the relevant library pages.

   1. Validate your Terraform configuration files using this command:

      ```bash
      terraform validate
      ```

      Terraform will display any configuration errors detected in your files.

   1. Create the required infrastructure:

      1. Run this command to view the planned changes:
      
         ```bash
         terraform plan
         ```
      
         If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.
      
      1. If everything looks correct, apply the changes:
         1. Run this command:
      
            ```bash
            terraform apply
            ```
      
         1. Confirm updating the resources.
         1. Wait for the operation to complete.

      All the required resources will be created in the specified folder. You can check resource availability and their settings in the [management console](https://console.yandex.cloud).

{% endlist %}

{% note warning %}

Do not change or delete Virtual Private Cloud resources used by the Managed Service for Kubernetes cluster. This may cause cluster errors or make it impossible to delete later.

{% endnote %}

## Set up a virtual machine {#vm}

As the Managed Service for Kubernetes cluster has no internet access, you can only connect to it from a VM in the same network. Therefore, to check cluster availability, set up the infrastructure:

1. Create the required resources:

   {% list tabs group=instructions %}

   - Manually {#manual}

      1. Create a service account named `vm-sa` with the `k8s.cluster-api.cluster-admin` and `k8s.admin` roles. You will use it to connect to the Managed Service for Kubernetes cluster.
      1. Create a security group named `vm-security-group` and specify a rule for incoming traffic in it:

         * **Port range**: `22`.
         * **Protocol**: `TCP`.
         * **Source**: `CIDR`.
         * **CIDR blocks**: `0.0.0.0/0`.

      1. [Create a Linux VM](../../compute/operations/vm-create/create-linux-vm.md) with the following settings:

         * **Subnet**: `my-subnet`.
         * **Public IP address**: `Auto`, or you can [reserve a static public IP address](../../vpc/operations/get-static-ip.md) and assign it to the new VM.
         * **Security groups**: `vm-security-group`.
         * **Service account**: `vm-sa`.

   - Terraform {#tf}

      1. Download the [virtual-machine-for-k8s.tf](https://github.com/yandex-cloud-examples/yc-mk8s-cluster-without-internet/blob/main/virtual-machine-for-k8s.tf) configuration file to the directory containing the `k8s-cluster-with-no-internet.tf` file.

         This file describes:

         * Service account for the VM.
         * Security group for the VM.
         * VM.

      1. In `virtual-machine-for-k8s.tf`, specify the following:

         * Folder ID.
         * ID of the network created along with the Managed Service for Kubernetes cluster.
         * ID of the subnet created along with the Managed Service for Kubernetes cluster and residing in the `ru-central1-a` availability zone. You can find this zone in the VM settings.
         * Username to use for connecting to the VM over SSH.
         * Absolute path to the public part of the SSH key for connecting to the VM.

      1. Validate your Terraform configuration files using this command:

         ```bash
         terraform validate
         ```

         Terraform will display any configuration errors detected in your files.

      1. Create the required infrastructure:

         1. Run this command to view the planned changes:
         
            ```bash
            terraform plan
            ```
         
            If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.
         
         1. If everything looks correct, apply the changes:
            1. Run this command:
         
               ```bash
               terraform apply
               ```
         
            1. Confirm updating the resources.
            1. Wait for the operation to complete.

         All the required resources will be created in the specified folder. You can check resource availability and their settings in the [management console](https://console.yandex.cloud).

   {% endlist %}

1. [Connect to the VM](../../compute/operations/vm-connect/ssh.md#vm-connect) over SSH:

   ```bash
   ssh <username>@<VM_public_IP_address>
   ```

   Where `<username>` is the VM account username.

1. [Install the Yandex Cloud command line interface](../../cli/operations/install-cli.md#interactive) (YC CLI).
1. [Create a YC CLI profile](../../cli/operations/profile/profile-create.md#create).
1. [Install kubect](https://kubernetes.io/docs/tasks/tools/#kubectl) and [configure it to work with the new cluster](../operations/connect/index.md#kubectl-connect).

## Check cluster availability {#check}

Run this command on the VM:

```bash
kubectl cluster-info
```

The command will return the following Managed Service for Kubernetes cluster information:

```text
Kubernetes control plane is running at https://<cluster_address>
CoreDNS is running at https://<cluster_address>/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
```

## Optionally, set up Managed Service for Kubernetes cluster time synchronization with your private NTP server.

To ensure the Managed Service for Kubernetes cluster time remains synchronized with another resource (in this case, a VM), deploy a private NTP server in `my-subnet` and set up synchronization of the cluster and VM with this server.

1. Specify the NTP server address in the [DHCP settings](../../vpc/concepts/dhcp-options.md) of `my-subnet`.

   {% list tabs group=instructions %}

   - Manually {#manual}

     [Update `my-subnet`](../../vpc/operations/subnet-update.md) by adding the NTP server IP address.

   - Terraform {#tf}

     1. In the Terraform configuration file, change `my-subnet` description. Add the `dhcp_options` section (if missing) with the `ntp_servers` parameter specifying the IP address of your NTP server:

        ```hcl
        ...
        resource "yandex_vpc_subnet" "lab-subnet-a" {
          name           = "subnet-1"
          ...
          v4_cidr_blocks = ["<IPv4_address>"]
          network_id     = "<network_ID>"
          ...
          dhcp_options {
            ntp_servers = ["<IPv4_address>"]
            ...
          }
        }
        ...
        ```

        For more information about the `yandex_vpc_subnet` settings, see [this Terraform provider guide](../../terraform/resources/vpc_subnet.md).

     1. Apply the changes:

        1. In the terminal, navigate to the configuration file directory.
        1. Make sure the configuration is correct using this command:
        
           ```bash
           terraform validate
           ```
        
           If the configuration is valid, you will get this message:
        
           ```bash
           Success! The configuration is valid.
           ```
        
        1. Run this command:
        
           ```bash
           terraform plan
           ```
        
           You will see a list of resources and their properties. No changes will be made at this step. Terraform will show any errors in the configuration.
        1. Apply the configuration changes:
        
           ```bash
           terraform apply
           ```
        
        1. Type `yes` and press **Enter** to confirm the changes.
        
        Terraform will update all required resources. You can check the subnet update using the [management console](https://console.yandex.cloud) or this [CLI](../../cli/quickstart.md) command:

        ```
        yc vpc subnet get <subnet_name>
        ```
     
   {% endlist %}

1. Allow the cluster and VM to connect to the NTP server.
   
   [Create](../../vpc/operations/security-group-add-rule.md) rules in the [security group of the cluster and node groups](../operations/connect/security-groups.md#rules-internal-cluster) and `vm-security-group`:

   * **Port range**: `123`. If your NTP server uses a port other than `123`, specify that port.
   * **Protocol**: `UDP`.
   * **Destination name**: `CIDR`.
   * **CIDR blocks**: `<NTP_server_IP_address>/32`.

1. Update the network settings in the cluster node group and on the VM using one of the following methods:

   * Connect to each node in the group and to the VM [over SSH](../operations/node-connect-ssh.md) or [via OS Login](../operations/node-connect-oslogin.md) and run the `sudo dhclient -v -r && sudo dhclient` command.
   * Reboot the group nodes and VM at any convenient time.

   {% note warning %}

   Updating network settings may cause the services within the cluster to become unavailable for a few minutes.

   {% endnote %}

## Optionally, connect a private Docker image registry {#cert}

You can connect a [private Docker image registry](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/) to your Managed Service for Kubernetes cluster. To get authenticated in the registry and connect to it over HTTPS, the cluster will need trusted CA certificates. Use a [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) controller to add and later automatically update the certificates on cluster nodes. The controller runs the following process in pods:

1. A Bash script runs on the cluster nodes to continuously check for the required certificates.
1. If any certificates are missing, they are copied from the Kubernetes [secret](https://kubernetes.io/docs/concepts/configuration/secret/) and updated.
1. `containerd` is restarted.

To configure certificate updates using DaemonSet, do the following on your VM:

1. Place the `.crt` certificate files.
1. Create a file named `certificate-updater-namespace.yaml` with the namespace configuration. This namespace will be used to run and isolate the DaemonSet.

   ```yaml
   apiVersion: v1
   kind: Namespace
   metadata:
     name: certificate-updater
     labels:
       name: certificate-updater
   ```

1. Create a file named `certificate-updater-daemonset.yaml` with the DaemonSet configuration:

   {% cut "File contents" %}

   ```yaml
   ---
   kind: NetworkPolicy
   apiVersion: networking.k8s.io/v1
   metadata:
     name: certificate-updater-deny-all
     namespace: certificate-updater
   spec:
     podSelector:
       matchLabels:
         k8s-app: certificate-updater
     policyTypes:
       - Ingress
       - Egress
     ingress: []
     egress:  []
   ---
   apiVersion: "apps/v1"
   kind: DaemonSet
   metadata:
     name: certificate-updater
     namespace: certificate-updater
     labels:
       k8s-app: certificate-updater
       version: 1v
   spec:
     selector:
       matchLabels:
         k8s-app: certificate-updater
     template:
       metadata:
         labels:
           k8s-app: certificate-updater
       spec:
         hostPID: true
         hostIPC: true
         containers:
         - name: certificate-updater
           image: cr.yandex/yc/mk8s-openssl:stable
           command: 
             - sh
             - -c
             - |
               while true; do
                 diff -x '.*' -r /mnt/user-cert-path/ /usr/local/share/ca-certificates
                 if [ $? -ne 0 ];
                   then
                       echo "Removing all old certificates"
                       rm -r /usr/local/share/ca-certificates/*
                       echo "Copying certificates from configmap"
                       cp /mnt/sbin/update-ca-certificates /usr/sbin/
                       cp /mnt/user-cert-path/* /usr/local/share/ca-certificates
         
                       echo "Updating cerfificates authorities"
                       update-ca-certificates 

                       echo "Restarting containerd"
                       ps -x -o pid= -o comm= | awk '$2 ~ "^(containerd|dockerd)$" { print $1 }' | xargs kill
                       #systemd will get them back less than a minute
                   else
                     echo "Doing Nothing as no certs has not been changed"
                   fi
                 sleep 60
               done
           imagePullPolicy: Never
           securityContext:
             privileged: true
           resources:
             limits:
               memory: 200Mi
             requests:
               cpu: 100m
               memory: 200Mi
           volumeMounts:
           - mountPath: /etc/
             name: etc
           - mountPath: /usr/local/share/ca-certificates
             name: docker-cert
           - name: secret
             mountPath: /mnt/user-cert-path
           - name: sbin
             mountPath: /mnt/sbin
             readOnly: true
           - name: ca-cert
             mountPath: /usr/share/ca-certificates
         volumes:
         - name: secret
           secret:
             secretName: crt
         - name: sbin
           hostPath:
             path: /usr/sbin/
             type: Directory
         - name: ca-cert
           hostPath:
             path: /usr/share/ca-certificates
             type: Directory
         - name: docker-cert
           hostPath:
             path: /usr/local/share/ca-certificates
             type: DirectoryOrCreate
         - name: etc
           hostPath:
             path: /etc/
             type: Directory
   ```

   {% endcut %}

1. Create a namespace:

   ```bash
   kubectl apply -f certificate-updater-namespace.yaml
   ```

1. Create a secret with the contents of the CA certificates:

   ```bash
   kubectl create secret generic crt \
      --from-file=<certificate_file_path>.crt \
      --namespace="certificate-updater"
   ```

   Specify a certificate with the `.crt` extension in the command. If you need to add multiple certificates, provide each one in the command using the `--from-file` flag.

   You can check the secret configuration using the command below and see if it contains the information about the certificates:

   ```bash
   kubectl get secret crt -o yaml
   ```

1. Create a DaemonSet:

   ```bash
   kubectl apply -f certificate-updater-daemonset.yaml
   ```

Now you can monitor the state of the DaemonSet controller. As soon as the certificates are updated, the `containerd` processes will be restarted.

## Delete the resources you created {#clear-out}

Some resources are not free of charge. Delete the resources you no longer need to avoid paying for them.

{% list tabs group=instructions %}

- Manually {#manual}

   Delete:

   1. [Service accounts](../../iam/operations/sa/delete.md)
   1. Key Management Service [encryption key](../../kms/operations/key.md#delete)
   1. [Security groups](../../vpc/operations/security-group-delete.md)
   1. Managed Service for Kubernetes [node group](../operations/node-group/node-group-delete.md)
   1. Managed Service for Kubernetes [cluster](../operations/kubernetes-cluster/kubernetes-cluster-delete.md)
   1. [Virtual machine](../../compute/operations/vm-control/vm-delete.md)
   1. [Subnet](../../vpc/operations/subnet-delete.md)
   1. [Network](../../vpc/operations/network-delete.md)

- Terraform {#tf}

   1. In the terminal window, go to the directory containing the infrastructure plan.
   
       {% note warning %}
   
       Make sure the directory has no Terraform manifests with the resources you want to keep. Terraform deletes all resources that were created using the manifests in the current directory.
   
       {% endnote %}
   
   1. Delete resources:
   
       1. Run this command:
   
           ```bash
           terraform destroy
           ```
   
       1. Confirm deleting the resources and wait for the operation to complete.
   
       All the resources described in the Terraform manifests will be deleted.

{% endlist %}

If you assigned a static public IP address to your VM, [release and delete it](../../vpc/operations/address-delete.md).