[Yandex Cloud documentation](../../index.md) > [Yandex Compute Cloud](../index.md) > [Step-by-step guides](index.md) > GPU > Enabling MIG on a VM with a GPU # Enabling MIG mode on a VM with a GPU in Compute Cloud {% note info %} This guide was tested on Ubuntu 20.04, Ubuntu 22.04, and Ubuntu 24.04 images with NVIDIA driver version 535. {% endnote %} _MIG (Multi-Instance GPU)_ is an NVIDIA technology that allows splitting a single graphics accelerator into several independent GPU instances. Each GPU instance gets its dedicated computing blocks, L2 cache, and HBM2 memory. Instances are isolated from each other at the hardware level, so a load on one instance does not affect the performance of the others. For a detailed description of this technology and its splitting schema, see the [NVIDIA MIG User Guide](https://docs.nvidia.com/datacenter/tesla/mig-user-guide/) page. With MIG, you can: * Run multiple independent ML or inference jobs on a single VM without them affecting each other in terms of performance and memory. * Use your GPU in a more efficient way because no individual job can load the entire GPU. * Guarantee the allocation of GPU resources among several users or processes. MIG is supported for the following [platforms](../concepts/vm-platforms.md): * AMD EPYC™ with NVIDIA® Ampere® A100 (`gpu-standard-v3`). * Gen2 (`gpu-standard-v3i`). * GPU PLATFORM V4 (`gpu-standard-v4`). {% note warning %} In MIG mode, you cannot transfer data between GPUs over [NVLink](https://en.wikipedia.org/wiki/NVLink). {% endnote %} To enable MIG mode on a VM with a GPU: 1. Create a VM: 1. [Create](vm-create/create-vm-with-gpu.md) a VM with a GPU based on one of the following platforms: * AMD EPYC™ with NVIDIA® Ampere® A100 (`gpu-standard-v3`). * Gen2 (`gpu-standard-v3i`). * GPU PLATFORM V4 (`gpu-standard-v4`). {% note tip %} Make sure the cloud has enough [quotas](../concepts/limits.md#compute-quotas) for the total number of GPUs of the correct platform, RAM, number of vCPUs, and SSD size to create a VM. Use [Yandex Cloud Quota Manager](../../quota-manager/index.md) to run this check. {% endnote %} 1. [Connect](vm-connect/ssh.md) to the VM over SSH. 1. [Install](vm-operate/install-nvidia-drivers.md) NVIDIA drivers version 535. 1. Disable NVLink on the GPUs. {#disable-nvlink} 1. Create a file named `/etc/modprobe.d/nonvlink.conf` with the following contents: ```text options nvidia NVreg_NvLinkDisable=1 ``` 1. Restart the NVIDIA driver: {% note warning %} From now on, run all commands as the `root` user or use `sudo`. {% endnote %} ```bash sudo systemctl stop nvidia-persistenced sudo modprobe -r nvidia_uvm nvidia_drm nvidia_modeset nvidia sudo modprobe nvidia sudo systemctl start nvidia-persistenced nvidia-smi ``` 1. Make sure NVLink is off: ```bash grep NvLink /proc/driver/nvidia/params ``` Expected result: ```text NvLinkDisable: 1 ``` 1. Reset the status of the GPUs: ```bash sudo nvidia-smi -r ``` Result: ```text GPU 00000000:8B:00.0 was successfully reset. GPU 00000000:8C:00.0 was successfully reset. All done. ``` For more on possible errors, see [Troubleshooting](#troubleshooting). 1. Enable MIG mode for all the VM GPUs: ```bash sudo nvidia-smi -mig 1 ``` Result: ```text Enabled MIG Mode for GPU 00000000:8B:00.0 Enabled MIG Mode for GPU 00000000:8C:00.0 All done. ``` {% note info %} If the output alerts you with `Warning: persistence mode is disabled on device ...`, start `nvidia-persistenced`: ```bash sudo systemctl start nvidia-persistenced ``` {% endnote %} 1. Split the GPUs into GPU instances. The example below creates two GPU instances with the `1g.10gb` profile on each GPU: ```bash sudo nvidia-smi mig -cgi 19,1g.10gb -C ``` Result: ```text Successfully created GPU instance ID 13 on GPU 0 using profile MIG 1g.10gb (ID 19) Successfully created compute instance ID 0 on GPU 0 GPU instance ID 13 using profile MIG 1g.10gb (ID 0) Successfully created GPU instance ID 11 on GPU 0 using profile MIG 1g.10gb (ID 19) Successfully created compute instance ID 0 on GPU 0 GPU instance ID 11 using profile MIG 1g.10gb (ID 0) ... ``` For a list of available splitting profiles for NVIDIA® A100, see [NVIDIA MIG User Guide](https://docs.nvidia.com/datacenter/tesla/mig-user-guide/#supported-mig-profiles). Alternatively, you can use the `nvidia-smi mig -lgip` command. 1. Check your new GPU instances: ```bash nvidia-smi -L ``` Result: ```text GPU 0: NVIDIA A100-SXM4-80GB (UUID: GPU-9c73eb7b-8d6d-6c0e-dc13-97873e2ede40) MIG 1g.10gb Device 0: (UUID: MIG-a686709d-6dc2-542a-8851-0d2508448c10) MIG 1g.10gb Device 1: (UUID: MIG-53864714-0a66-5df2-9007-9268e30d9a72) GPU 1: NVIDIA A100-SXM4-80GB (UUID: GPU-c92e2d5d-4a69-f1cd-4df0-305f8606705a) MIG 1g.10gb Device 0: (UUID: MIG-d5a46a2e-1d77-575d-99e3-1f96c8555a2e) MIG 1g.10gb Device 1: (UUID: MIG-b8479d6e-d08f-5a92-9a24-0946db72b2a0) ``` Your GPU instances are ready to use. To run computations on a specific instance, provide its UUID into the `CUDA_VISIBLE_DEVICES` environment variable. ## Troubleshooting {#troubleshooting} #### "Not Supported" GPU reset error {#not-supported} You may get the following error when running `sudo nvidia-smi -r`: ```text The following GPUs could not be reset: GPU 00000000:8B:00.0: Not Supported GPU 00000000:8C:00.0: Not Supported ``` This error indicates an active NVLink on your GPUs. [Disable](enable-mig.md#disable-nvlink) it and try resetting them again. #### "In use by another client" GPU reset error {#in-use} You may get the following error when running `sudo nvidia-smi -r`: ```text The following GPUs could not be reset: GPU 00000000:8B:00.0: In use by another client GPU 00000000:8C:00.0: In use by another client 2 devices are currently being used by one or more other processes (e.g., Fabric Manager, CUDA application, graphics application such as an X server, or a monitoring application such as another instance of nvidia-smi). Please first kill all processes using these devices and all compute applications running in the system. ``` This means your GPUs are currently in use by another process, e.g., the `nvidia_drm` module with enabled `modeset=1`, or a user process holding a lock on the device. To fix the issue: 1. Check if `nvidia_drm` is running with `modeset=1`: ```bash cat /sys/module/nvidia_drm/parameters/modeset ``` If the output returns `Y`, disable `modeset`. Do to this, create a file named `/etc/modprobe.d/nomodeset.conf` with the following contents: ```text options nvidia-drm modeset=0 ``` 1. Update `initramfs` (Initial RAM File System): ```bash sudo update-initramfs -u ``` 1. Identify the processes holding a lock on the device: ```bash sudo ls -l /proc/*/fd/* | grep /dev/nvidia ``` {% cut "Example of the command output" %} ```bash lrwx------ 1 root root 64 Jun 17 12:01 /proc/1574/fd/3 -> /dev/nvidiactl lrwx------ 1 root root 64 Jun 17 12:01 /proc/1574/fd/4 -> /dev/nvidia0 lrwx------ 1 root root 64 Jun 17 12:01 /proc/1574/fd/5 -> /dev/nvidia0 lrwx------ 1 root root 64 Jun 17 12:01 /proc/1574/fd/6 -> /dev/nvidia0 lrwx------ 1 root root 64 Jun 17 12:01 /proc/1574/fd/7 -> /dev/nvidia0 lrwx------ 1 root root 64 Jun 17 11:59 /proc/904/fd/13 -> /dev/nvidiactl lrwx------ 1 root root 64 Jun 17 11:59 /proc/904/fd/17 -> /dev/nvidia0 lrwx------ 1 root root 64 Jun 17 11:59 /proc/904/fd/18 -> /dev/nvidia0 lrwx------ 1 root root 64 Jun 17 11:59 /proc/904/fd/19 -> /dev/nvidia0 lrwx------ 1 root root 64 Jun 17 11:59 /proc/904/fd/20 -> /dev/nvidia0 lrwx------ 1 root root 64 Jun 17 11:59 /proc/904/fd/24 -> /dev/nvidia0 lrwx------ 1 root root 64 Jun 17 11:59 /proc/904/fd/25 -> /dev/nvidia0 lrwx------ 1 root root 64 Jun 17 11:59 /proc/904/fd/27 -> /dev/nvidia0 lrwx------ 1 root root 64 Jun 17 11:59 /proc/904/fd/28 -> /dev/nvidia0 lrwx------ 1 root root 64 Jun 17 11:59 /proc/904/fd/29 -> /dev/nvidia0 lrwx------ 1 root root 64 Jun 17 11:59 /proc/904/fd/3 -> /dev/nvidiactl lrwx------ 1 root root 64 Jun 17 11:59 /proc/904/fd/39 -> /dev/nvidia0 lrwx------ 1 root root 64 Jun 17 11:59 /proc/904/fd/43 -> /dev/nvidia0 lrwx------ 1 root root 64 Jun 17 11:59 /proc/904/fd/8 -> /dev/nvidia-modeset ``` {% endcut %} 1. Terminate these processes by running `sudo kill -9 `. 1. Try resetting your GPUs again: ```bash sudo nvidia-smi -r ``` {% note tip %} If the `modeset` value reverts to `1` after a VM reboot, check that your file system has no other configurations that override this setting: ```bash sudo grep -R "modeset=1" /etc ``` {% endnote %} #### "Unknown Error" when running nvidia-smi commands {#unknown-error} If `nvidia-smi` commands, such as `nvidia-smi --gpu-reset`, return an `Unknown Error`, restart `nvidia-persistenced`: ```bash sudo systemctl restart nvidia-persistenced ``` #### "Warning: persistence mode is disabled on device" error when enabling MIG mode {#persistence-mode} If the `nvidia-smi -mig 1` command output alerts you with `Warning: persistence mode is disabled on device ...`, start `nvidia-persistenced`: ```bash sudo systemctl start nvidia-persistenced ``` #### Useful links {#see-also} * [Graphics processing units (GPUs)](../concepts/gpus.md) * [Creating a VM with a GPU](vm-create/create-vm-with-gpu.md) * [Installing NVIDIA drivers](vm-operate/install-nvidia-drivers.md) * [NVIDIA driver update guide](gpu-driver-update.md)