[Yandex Cloud documentation](../../index.md) > [Tutorials](../index.md) > [Basic infrastructure](index.md) > Tools > Running the DeepSeek-R1 language model in a Compute Cloud GPU cluster # Running the DeepSeek-R1 language model in a Yandex Compute Cloud GPU cluster {% note info %} Currently, GPU clusters are only available in the `ru-central1-a` and `ru-central1-d` [availability zones](../../overview/concepts/geo-scope.md). You can only add a virtual machine (VM) to a GPU cluster from the same availability zone. {% endnote %} In this tutorial, you will create a [GPU cluster](../../compute/concepts/gpus.md#gpu-clusters) with two VMs to run the [DeepSeek-R1](https://huggingface.co/deepseek-ai/DeepSeek-R1) language model. To run a language model in a cluster: 1. [Get your cloud ready](#prepare-cloud). 1. [Create a GPU cluster with two VMs](#create-gpu-cluster). 1. [Test cluster state](#test-cluster-state). 1. [Run the language model](#launch-llm). 1. [Test the model](#test-llm). If you no longer need the resources you created, [delete them](#clear-out). ## Get your cloud ready {#prepare-cloud} Sign up for Yandex Cloud and create a [billing account](../../billing/concepts/billing-account.md): 1. Navigate to the [management console](https://console.yandex.cloud) and log in to Yandex Cloud or create a new account. 1. On the **[Yandex Cloud Billing](https://center.yandex.cloud/billing/accounts)** page, make sure you have a billing account linked and it has the `ACTIVE` or `TRIAL_ACTIVE` [status](../../billing/concepts/billing-account-statuses.md). If you do not have a billing account, [create one](../../billing/quickstart/index.md) and [link](../../billing/operations/pin-cloud.md) a cloud to it. If you have an active billing account, you can create or select a [folder](../../resource-manager/concepts/resources-hierarchy.md#folder) for your infrastructure on the [cloud page](https://console.yandex.cloud/cloud). [Learn more about clouds and folders here](../../resource-manager/concepts/resources-hierarchy.md). Make sure the cloud has enough [quotas](../../compute/concepts/limits.md#compute-quotas) for the total number of GPU clusters, total number of `Gen2` GPUs, amount of RAM, number of vCPUs, and the SSD size to create the VMs. To check your quotas, use [Yandex Cloud Quota Manager](../../quota-manager/index.md). ### Required paid resources {#paid-resources} The infrastructure support cost includes a fee for VM computing resources and disks, as well as for the GPU cluster (see [Yandex Compute Cloud pricing](../../compute/pricing.md)). ## Create a GPU cluster with two VMs {#create-gpu-cluster}. ### Create a GPU cluster {#create-cluster} {% list tabs group=instructions %} - Management console {#console} 1. In the [management console](https://console.yandex.cloud), select a [folder](../../resource-manager/concepts/resources-hierarchy.md#folder) to create your cluster in. 1. Navigate to **Compute Cloud**. 1. In the left-hand panel, select ![image](../../_assets/console-icons/cpus.svg) **GPU clusters**. 1. Click **Create a cluster**. 1. In the **Name** field, specify `test-gpu-cluster`. 1. In the **Availability zone** field, select `ru-central1-d`. 1. Click **Save**. {% endlist %} ### Add your VM to a cluster {#add-vm} 1. Create your first VM: {% list tabs group=instructions %} - Management console {#console} 1. In the left-hand panel, select ![image](../../_assets/console-icons/server.svg) **Virtual machines** and click **Create virtual machine**. 1. Under **Boot disk image**, select the [Ubuntu 20.04 LTS Secure Boot CUDA 12.2](https://yandex.cloud/en/marketplace/products/yc/ubuntu-2004-lts-secureboot-cuda-12-2) public image. 1. In the **Availability zone** field, select `ru-central1-d`. 1. Under **Disks and file storages**, select the `SSD` [disk type](../../compute/concepts/disk.md#disks_types) and specify its size: `800 GB`. 1. Under **Computing resources**, navigate to the `Custom` tab and specify: * **Platform**: `Gen2`. * **GPU**: `8`. * **GPU cluster**: Select the `test-gpu-cluster` cluster you created earlier. 1. Under **Access**, select **SSH key** and specify the access credentials: * **Login**: `ubuntu`. * In the **SSH key** field, select the SSH key saved in your [organization user](../../organization/concepts/membership.md) profile. If there are no SSH keys in your profile or you want to add a new key: 1. Click **Add key**. 1. Enter a name for the SSH key. 1. Select one of the following: * `Enter manually`: Paste the contents of the public SSH key. You need to [create](../../compute/operations/vm-connect/ssh.md#creating-ssh-keys) an SSH key pair on your own. * `Load from file`: Upload the public part of the SSH key. You need to create an SSH key pair on your own. * `Generate key`: Automatically create an SSH key pair. When adding a new SSH key, an archive containing the key pair will be created and downloaded. In Linux or macOS-based operating systems, unpack the archive to the `/home//.ssh` directory. In Windows, unpack the archive to the `C:\Users\/.ssh` directory. You do not need additionally enter the public key in the management console. 1. Click **Add**. The system will add the SSH key to your organization user profile. If the organization has [disabled](../../organization/operations/os-login-access.md) the ability for users to add SSH keys to their profiles, the added public SSH key will only be saved in the user profile inside the newly created resource. 1. Click **Create VM**. {% endlist %} 1. Similarly, create a second VM with the same settings. ## Optionally, check the cluster state {#test-cluster-state} Also, you can: * [Test the cluster physical state](../../compute/operations/gpu-cluster/gpu-cluster-test-physical-state.md). * [Run parallel jobs](../../compute/operations/gpu-cluster/gpu-cluster-mpirun-parallel.md). * [Test InfiniBand throughput](../../compute/operations/gpu-cluster/test-infiniband-bandwidth.md). ## Run the language model {#launch-llm} 1. [Connect to both VMs over SSH](../../compute/operations/vm-connect/ssh.md#vm-connect). 1. Add the `ubuntu` user to the `docker` group by running these commands on both VMs: ```bash sudo groupadd docker sudo usermod -aG docker $USER newgrp docker ``` 1. Pull the `SGLang` image to both VMs: ```bash docker pull lmsysorg/sglang:latest ``` 1. On **the first VM**, run the server start command (replace `` with the first VM's internal IP): ```bash docker run --gpus all \ --device=/dev/infiniband \ --ulimit memlock=-1 \ --ulimit stack=67108864 \ --shm-size 32g \ --network=host \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --name sglang_multinode1 \ -e GLOO_SOCKET_IFNAME=eth0 \ -it --rm --ipc=host lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-R1 \ --tp 16 \ --nccl-init-addr :30000 \ --nnodes 2 \ --node-rank 0 \ --trust-remote-code \ --host 0.0.0.0 \ --port 30001 \ --disable-radix \ --max-prefill-tokens 126000 ``` 1. On **the second VM**, run the same command with `--node-rank` set to `1`: ```bash docker run --gpus all \ --device=/dev/infiniband \ --ulimit memlock=-1 \ --ulimit stack=67108864 \ --shm-size 32g \ --network=host \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --name sglang_multinode2 \ -e GLOO_SOCKET_IFNAME=eth0 \ -it --rm --ipc=host lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-R1 \ --tp 16 \ --nccl-init-addr :30000 \ --nnodes 2 \ --node-rank 1 \ --trust-remote-code \ --host 0.0.0.0 \ --port 30001 \ --disable-radix \ --max-prefill-tokens 126000 ``` 1. Wait for the message saying your start was successful: `The server is fired up and ready to roll!`. ## Test the language model performance {#test-llm} 1. Open a new SSH session to the first VM. 1. Install the OpenAI library: ```bash sudo apt update sudo apt install python3-pip -y pip install openai ``` 1. Create a script named `test_model.py` with this code: ```python import openai client = openai.Client( base_url="http://127.0.0.1:30001/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="default", messages=[ {"role": "system", "content": "You are a helpful AI assistant"}, {"role": "user", "content": "List 3 countries and their capitals."}, ], temperature=0.3, max_tokens=1024, ) print(response.choices[0].message.content) ``` 1. Run the script: ```bash python3 test_model.py ``` **Sample response:** ```text Here are three countries and their capitals: 1. **France** - Paris 2. **Japan** - Tokyo 3. **Brazil** - Brasília Let me know if you'd like more examples! 😊 ``` ## How to delete the resources you created {#clear-out} To stop paying for the resources you created, in Compute Cloud: 1. [Delete](../../compute/operations/vm-control/vm-delete.md) the VMs you created. 1. [Delete](../../compute/operations/gpu-cluster/gpu-cluster-delete.md) the GPU cluster you created. #### See also {#see-also} [Questions about GPUs](../../compute/qa/gpu.md)