[Yandex Cloud documentation](../../index.md) > [Yandex Compute Cloud](../index.md) > [Tutorials](index.md) > Running a vLLM library with the Gemma 3 language model on a VM with GPU # Running a vLLM library with the Gemma 3 language model on a Yandex Compute Cloud VM instance with a GPU Using this tutorial, you will create a VM instance with a single GPU to run the [Gemma 3](https://huggingface.co/google/gemma-3-27b-it) lightweight multimodal language model. To run a language model: 1. [Get your cloud ready](#prepare-cloud). 1. [Get access to the Gemma 3 model](#get-gemma3-access). 1. [Create a VM with a GPU](#create-gpu-vm). 1. [Run the language model](#launch-llm). 1. [Test the language model performance](#test-llm). If you no longer need the resources you created, [delete them](#clear-out). ## Get your cloud ready {#prepare-cloud} Sign up for Yandex Cloud and create a [billing account](../../billing/concepts/billing-account.md): 1. Navigate to the [management console](https://console.yandex.cloud) and log in to Yandex Cloud or create a new account. 1. On the **[Yandex Cloud Billing](https://center.yandex.cloud/billing/accounts)** page, make sure you have a billing account linked and it has the `ACTIVE` or `TRIAL_ACTIVE` [status](../../billing/concepts/billing-account-statuses.md). If you do not have a billing account, [create one](../../billing/quickstart/index.md) and [link](../../billing/operations/pin-cloud.md) a cloud to it. If you have an active billing account, you can create or select a [folder](../../resource-manager/concepts/resources-hierarchy.md#folder) for your infrastructure on the [cloud page](https://console.yandex.cloud/cloud). [Learn more about clouds and folders here](../../resource-manager/concepts/resources-hierarchy.md). Make sure the cloud has enough [quotas](../concepts/limits.md#compute-quotas) for the total number of `Gen2` GPUs, amount of RAM, number of vCPUs, and SSD size to create the VM. To do this, use [Yandex Cloud Quota Manager](../../quota-manager/index.md). ### Required paid resources {#paid-resources} The infrastructure support cost includes a fee for continuously running VMs and disks (see [Yandex Compute Cloud pricing](../pricing.md)). ## Get access to the Gemma 3 model {#get-gemma3-access} 1. Sign up for [Hugging Face](https://huggingface.co/). 1. Create an access token: 1. After logging into your account, click your avatar → **Settings** → **Access Tokens**. 1. Click **+ Create new token**. 1. Select the **Read** token type. 1. Enter a token name. 1. Click **Create token**. 1. Copy the token value. 1. Request access to the `Gemma-3-27b-it` model: 1. Go to the [model page](https://huggingface.co/google/gemma-3-27b-it). 1. Click **Request access**. 1. Accept the license terms. 1. Wait for access confirmation. ## Create a VM with a GPU {#create-gpu-vm} {% list tabs group=instructions %} - Management console {#console} 1. In the [management console](https://console.yandex.cloud), select the [folder](../../resource-manager/concepts/resources-hierarchy.md#folder) where you want to create your VM. 1. Navigate to **Compute Cloud**. 1. In the left-hand panel, select ![image](../../_assets/console-icons/server.svg) **Virtual machines**. 1. Click **Create virtual machine**. 1. Under **Boot disk image**, select the [Ubuntu 20.04 LTS Secure Boot CUDA 12.2](https://yandex.cloud/en/marketplace/products/yc/ubuntu-2004-lts-secureboot-cuda-12-2) public image. 1. In the **Availability zone** field, select the `ru-central1-d` [availability zone](../../overview/concepts/geo-scope.md). 1. Under **Disks and file storages**, select the `SSD` [disk type](../concepts/disk.md#disks_types) and set the size to at least `500 GB`. 1. Under **Computing resources**, navigate to the `Custom` tab and specify the [platform](../concepts/vm-platforms.md) and number of GPUs: * **Platform**: `Gen2`. * **GPU**: `1`. 1. Under **Access**, select **SSH key** and specify the VM access credentials: * In the **Login** field, enter a username, e.g., `ubuntu`. Do not use `root` or other names reserved for the OS purposes. For operations requiring root privileges, use the `sudo` command. * In the **SSH key** field, select the SSH key saved in your [organization user](../../organization/concepts/membership.md) profile. If there are no SSH keys in your profile or you want to add a new key: 1. Click **Add key**. 1. Enter a name for the SSH key. 1. Select one of the following: * `Enter manually`: Paste the contents of the public SSH key. You need to [create](../operations/vm-connect/ssh.md#creating-ssh-keys) an SSH key pair on your own. * `Load from file`: Upload the public part of the SSH key. You need to create an SSH key pair on your own. * `Generate key`: Automatically create an SSH key pair. When adding a new SSH key, an archive containing the key pair will be created and downloaded. In Linux or macOS-based operating systems, unpack the archive to the `/home//.ssh` directory. In Windows, unpack the archive to the `C:\Users\/.ssh` directory. You do not need additionally enter the public key in the management console. 1. Click **Add**. The system will add the SSH key to your organization user profile. If the organization has [disabled](../../organization/operations/os-login-access.md) the ability for users to add SSH keys to their profiles, the added public SSH key will only be saved in the user profile inside the newly created resource. 1. Click **Create VM**. {% endlist %} ## Run the language model {#launch-llm} 1. [Connect](../operations/vm-connect/ssh.md#vm-connect) to the VM over SSH. 1. Add the current user to the `docker` group: ```bash sudo groupadd docker sudo usermod -aG docker $USER newgrp docker ``` 1. Fill in the variables: ```bash TOKEN= MODEL=google/gemma-3-27b-it MODEL_OPTS="--max-num-seqs 256 --max-model-len 16384 --gpu-memory-utilization 0.98 --max_num_batched_tokens 2048" ``` Where `HF_token` is the Hugging Face access token. 1. Run this command: ```bash docker run --runtime nvidia --gpus '"device=0"' \ --name vllm-gema3-0 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HUGGING_FACE_HUB_TOKEN=$TOKEN" \ --env "HF_HUB_ENABLE_HF_TRANSFER=0" \ --env "PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True" \ -p 8000:8000 \ --ipc=host \ --shm-size=32g \ vllm/vllm-openai:latest \ --model $MODEL $MODEL_OPTS ``` 1. Wait for the server to start: ```text INFO: Started server process [1] INFO: Waiting for application startup. INFO: Application startup complete. ``` ## Test the language model performance {#test-llm} 1. Connect to the VM over SSH in a new session. 1. Install the `openai` package: ```bash sudo apt update sudo apt install python3-pip pip install openai ``` 1. Create a `test_model.py` script with the following contents: ```python import openai client = openai.Client(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY") response = client.chat.completions.create( model="google/gemma-3-27b-it", messages=[ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" }, }, {"type": "text", "text": "Describe this image in one sentence."}, ], } ], temperature=0.3, max_tokens=128, ) print(response.choices[0].message.content) ``` 1. Run the script: ```bash python3 test_model.py ``` Model response example: ```text Here's a one-sentence description of the image: The Statue of Liberty stands prominently on Liberty Island with the Manhattan skyline, including the Empire State Building, visible in the background across the water on a clear, sunny day. ``` ## How to delete the resources you created {#clear-out} To stop paying for the resources you created, [delete](../operations/vm-control/vm-delete.md) the VM instance you created in Compute Cloud. #### See also {#see-also} * [Questions about GPUs](../qa/gpu.md)