# Creating test VMs via GitLab CI # Creating test VMs with GitLab CI You can use Yandex Cloud to automate routine tasks, such as running a specific script after each Git `master` branch commit. In the following example, we will create and test a [VM](../concepts/vm.md) after each commit. To configure continuous integration (CI) for VM [disk snapshots](../concepts/snapshot.md): 1. [Create a test application VM](#create-vm): Create a virtual machine. You will use its disk snapshot for creating new CI VMs. 1. [Configure the test application VM](#configure-vm): Install a web server and test application components on the VM. Write a test application reversing words in a text sent to the server. 1. [Check whether the test application is working](#test-app): Send a test request to check whether the server settings are correct. 1. [Create a VM disk snapshot](#create-snapshot): Create a disk snapshot that CI will use to create new VMs. 1. [Create a GitLab VM](#create-gitlab-vm): Create a [GitLab VM](https://about.gitlab.com/) with a repository to store CI settings and a functional testing script. 1. [Configure GitLab](#configure-gitlab): Create a file repository and get configuration parameters. 1. [Configure a runner](#configure-runner): Set up a tool for performing tasks. 1. [Configure CI](#configure-ci): Specify the required command and testing parameters. 1. [Test the application on the CI VM](#test-new-vm): Make sure the system created CI snapshot VMs and they run your test application. If you no longer need the VMs you created, [delete them](#clear-out). ## Get your cloud ready {#before-you-begin} Sign up for Yandex Cloud and create a [billing account](../../billing/concepts/billing-account.md): 1. Navigate to the [management console](https://console.yandex.cloud) and log in to Yandex Cloud or create a new account. 1. On the **[Yandex Cloud Billing](https://center.yandex.cloud/billing/accounts)** page, make sure you have a billing account linked and it has the `ACTIVE` or `TRIAL_ACTIVE` [status](../../billing/concepts/billing-account-statuses.md). If you do not have a billing account, [create one](../../billing/quickstart/index.md) and [link](../../billing/operations/pin-cloud.md) a cloud to it. If you have an active billing account, you can create or select a [folder](../../resource-manager/concepts/resources-hierarchy.md#folder) for your infrastructure on the [cloud page](https://console.yandex.cloud/cloud). [Learn more about clouds and folders here](../../resource-manager/concepts/resources-hierarchy.md). Before creating a VM: 1. Navigate to the Yandex Cloud [management console](https://console.yandex.cloud) and select the [folder](../../resource-manager/concepts/resources-hierarchy.md#folder) to work with. 1. Make sure the selected folder has a [network](../../vpc/concepts/network.md#network) with a [subnet](../../vpc/concepts/network.md#subnet) for your VM. Do it by navigating to **Virtual Private Cloud** on the folder page. If the list contains a network, click its name to see the list of subnets. If there are no networks in the list or the selected network has no subnets, [create them](../../vpc/quickstart.md) as required. ### Required paid resources {#paid-resources} The infrastructure support cost includes: * Fee for continuously running VMs (see [Yandex Compute Cloud pricing](../pricing.md)). * Fee for storing created images (see [Compute Cloud pricing](../pricing.md#prices-storage)). * Fee for dynamic public IP addresses (see [Yandex Virtual Private Cloud pricing](../../vpc/pricing.md)). ## Create a test application VM {#create-vm} Create a VM where you will install a web server, your test application, and its components: 1. In the [management console](https://console.yandex.cloud), select the [folder](../../resource-manager/concepts/resources-hierarchy.md#folder) where you want to create your VM. 1. Navigate to **Compute Cloud**. 1. In the left-hand panel, select ![image](../../_assets/console-icons/server.svg) **Virtual machines**. 1. Click **Create virtual machine**. 1. Under **Boot disk image**, select the [Ubuntu 18.04](https://yandex.cloud/en/marketplace/products/yc/ubuntu-18-04-lts) public image. 1. Under **Location**, select an [availability zone](../../overview/concepts/geo-scope.md) where your VM will reside. 1. Under **Computing resources**, navigate to the **Custom** tab and specify the following: * **Platform**: `Intel Ice Lake` * **vCPU**: `2` * **Guaranteed vCPU performance**: `20%` * **RAM**: `1 GB` 1. In **Network settings**, select the subnet where you will connect your VM. 1. Under **Access**, select **SSH key** and specify the VM access credentials: * In the **Login** field, enter a name for the user you want to create on the VM, e.g., `yc-user`. {% note alert %} Do not use `root` or other OS-reserved usernames. For operations requiring root privileges, use the `sudo` command. {% endnote %} * In the **SSH key** field, select the SSH key saved in your [organization user](../../organization/concepts/membership.md) profile. If there are no SSH keys in your profile or you want to add a new key: 1. Click **Add key**. 1. Enter a name for the SSH key. 1. Select one of the following: * `Enter manually`: Paste the contents of the public SSH key. You need to [create](../operations/vm-connect/ssh.md#creating-ssh-keys) an SSH key pair on your own. * `Load from file`: Upload the public part of the SSH key. You need to create an SSH key pair on your own. * `Generate key`: Automatically create an SSH key pair. When adding a new SSH key, an archive containing the key pair will be created and downloaded. In Linux or macOS-based operating systems, unpack the archive to the `/home//.ssh` directory. In Windows, unpack the archive to the `C:\Users\/.ssh` directory. You do not need additionally enter the public key in the management console. 1. Click **Add**. The system will add the SSH key to your organization user profile. If the organization has [disabled](../../organization/operations/os-login-access.md) the ability for users to add SSH keys to their profiles, the added public SSH key will only be saved in the user profile inside the newly created resource. 1. Under **General information**, specify the VM name: `ci-tutorial-test-app`. 1. Click **Create VM**. It may take a few minutes to create your VM. When its status changes to `RUNNING`, you can start configuring it. Once your VM is created, the system will assign it an [IP address and a host name (FQDN)](../../vpc/concepts/address.md). You can use them for SSH access. ## Configure the test application VM {#configure-vm} On the new VM, install a web server and test application components. You will use Python 2 to create your test application. 1. In the [management console](https://console.yandex.cloud), under **Network** on the VM page, find the VM public IP address. 1. [Connect](../operations/vm-connect/ssh.md#vm-connect) to the VM over SSH. To do this, use `ssh` in Linux or macOS, or [PuTTY](https://www.chiark.greenend.org.uk/~sgtatham/putty/) in Windows. ```bash ssh @ ``` 1. Run the `apt update` command to update the lists of packages available for installation. 1. Install the required packages: `jq`, Git client, `pip`, `virtualenv`, Python C API header files, and Nginx: ```bash sudo apt-get --yes install jq git python-pip virtualenv python-dev nginx-full ``` 1. Create your application directory and make your current user its owner: ```bash sudo mkdir /srv/test-app sudo chown -R $USER /srv/test-app ``` 1. Navigate to this directory and create the `virtualenv` virtual environment in it: ```bash cd /srv/test-app virtualenv test-venv ``` 1. Activate the virtual environment: ```bash . test-venv/bin/activate ``` 1. Install the Flask framework and the uWSGI web server in the virtual environment: ```bash pip install flask uwsgi ``` 1. Deactivate the virtual environment: ```bash deactivate ``` 1. Create the `api.py` file in the `/srv/test-app` directory: ```bash touch api.py ``` 1. Open the `api.py` file in any text editor and paste the Python code below into it; the code does the following: * Accepts a text string from the `text` input argument. * Reverses each word in the input string. * Returns a response: * In JSON format if the client accepts it. * In plain text if the client cannot accept JSON. ```python # api.py import json from flask import Flask, request, make_response as response app = Flask(__name__) @app.route("/") def index(): text = request.args.get('text') json_type = 'application/json' json_accepted = json_type in request.headers.get('Accept', '') if text: words = text.split() reversed_words = [word[::-1] for word in words] if json_accepted: res = response(json.dumps({'text': reversed_words}), 200) else: res = response(' '.join(reversed_words), 200) else: res = response('text not found', 501) res.headers['Content-Type'] = json_type if json_accepted else 'text/plain' return res ``` 1. Create the `wsgi.py` file in the `/srv/test-app` directory: ```bash touch wsgi.py ``` 1. Open the `wsgi.py` file in any text editor and paste the application starting code in it: ```python # wsgi.py from api import app if __name__ == "__main__": app.run() ``` 1. Create the `test-app.ini` file in the `/srv/test-app` directory: ```bash touch test-app.ini ``` 1. Open the `test-app.ini` file in any text editor and paste the uWSGI server configuration in it: ```ini #test-app.ini [uwsgi] module = wsgi:app master = true processes = 1 socket = test-app.sock chmod-socket = 660 vacuum = true die-on-term = true ``` 1. Make the `www-data` user owner of the `/srv/test-app` directory and its contents: ```bash sudo chown -R www-data:www-data /srv/test-app ``` 1. Set up a service that will start your uWSGI server. To do this, edit the `/etc/systemd/system/test-app.service` file as follows: ```ini #/etc/systemd/system/test-app.service [Unit] Description=uWSGI instance to serve test API After=network.target [Service] User=www-data Group=www-data WorkingDirectory=/srv/test-app Environment="PATH=/srv/test-app/test-venv/bin" ExecStart=/srv/test-app/test-venv/bin/uwsgi --ini test-app.ini [Install] WantedBy=multi-user.target ``` 1. Specify the new virtual server settings in the Nginx configuration by changing the `/etc/nginx/sites-available/test-app.conf` file as follows: ```nginx #/etc/nginx/sites-available/test-app.conf server { #server_name test-app.yandex www.test-app.yandex; listen 80; location /test/ { include uwsgi_params; uwsgi_param SCRIPT_NAME /test; uwsgi_modifier1 30; uwsgi_pass unix:/srv/test-app/test-app.sock; } } ``` 1. Create a symbolic link pointing to the `test-app.conf` Nginx configuration file: ```bash sudo ln -s /etc/nginx/sites-available/test-app.conf /etc/nginx/sites-enabled/ ``` 1. Delete the symbolic link pointing to the default Nginx configuration: ```bash sudo unlink /etc/nginx/sites-enabled/default ``` 1. Configure the service to run at boot: ```bash sudo systemctl enable test-app.service ``` ## Check whether the test application is working {#test-app} To make sure the web server is properly configured and the test application is working as expected, make a simple request. 1. In the browser address bar, enter the following URL: ```http http:///test/?text=hello_world ``` 1. If everything works correctly, you will see reversed words from the `text` argument string. ## Create a VM disk snapshot {#create-snapshot} Create a VM disk snapshot to transfer your web server configuration and test application to CI VMs. 1. In the Yandex Cloud [management console](https://console.yandex.cloud), select the folder where you created your VM. 1. Navigate to **Compute Cloud**. 1. Find the `ci-tutorial-test-app` VM and select it. 1. Click **Stop**. 1. In the window that opens, click **Stop**. 1. Once the VM gets stopped, select the **Disks** tab. 1. Next to the disk you need, click ![vertical-ellipsis](../../_assets/console-icons/ellipsis-vertical.svg) and select **Create snapshot**. 1. In the window that opens, specify the snapshot name: `test-app-snap`. 1. Click **Create**. ## Create a GitLab VM {#create-gitlab-vm} You can set up CI in Yandex Cloud by using a public image with GitLab pre-installed. GitLab includes tools for managing Git repositories and configuring CI. 1. In the [management console](https://console.yandex.cloud), select the [folder](../../resource-manager/concepts/resources-hierarchy.md#folder) where you want to create your VM. 1. Navigate to **Compute Cloud**. 1. In the left-hand panel, select ![image](../../_assets/console-icons/server.svg) **Virtual machines**. 1. Click **Create virtual machine**. 1. Under **Boot disk image**, navigate to the **Marketplace** tab, click **Show all Marketplace products**, and select the [GitLab](https://yandex.cloud/en/marketplace/products/yc/gitlab) image. 1. Under **Location**, select an [availability zone](../../overview/concepts/geo-scope.md) where your VM will reside. 1. Under **Computing resources**, navigate to the **Custom** tab and specify the following: * **Platform**: `Intel Ice Lake` * **vCPU**: `2` * **Guaranteed vCPU performance**: `100%` * **RAM**: `2 GB` 1. In **Network settings**, select the subnet where you will connect your VM. 1. Under **Access**, select **SSH key** and specify the VM access credentials: * In the **Login** field, enter a name for the user you want to create on the VM, e.g., `yc-user`. {% note alert %} Do not use `root` or other OS-reserved usernames. For operations requiring root privileges, use the `sudo` command. {% endnote %} * In the **SSH key** field, select the SSH key saved in your [organization user](../../organization/concepts/membership.md) profile. If there are no SSH keys in your profile or you want to add a new key: 1. Click **Add key**. 1. Enter a name for the SSH key. 1. Select one of the following: * `Enter manually`: Paste the contents of the public SSH key. You need to [create](../operations/vm-connect/ssh.md#creating-ssh-keys) an SSH key pair on your own. * `Load from file`: Upload the public part of the SSH key. You need to create an SSH key pair on your own. * `Generate key`: Automatically create an SSH key pair. When adding a new SSH key, an archive containing the key pair will be created and downloaded. In Linux or macOS-based operating systems, unpack the archive to the `/home//.ssh` directory. In Windows, unpack the archive to the `C:\Users\/.ssh` directory. You do not need additionally enter the public key in the management console. 1. Click **Add**. The system will add the SSH key to your organization user profile. If the organization has [disabled](../../organization/operations/os-login-access.md) the ability for users to add SSH keys to their profiles, the added public SSH key will only be saved in the user profile inside the newly created resource. 1. Under **General information**, specify the VM name: `ci-tutorial-gitlab`. 1. Click **Create VM**. It may take a few minutes to create your VM. When its status changes to `RUNNING`, you can start configuring it. Once your VM is created, the system will assign it an [IP address and a host name (FQDN)](../../vpc/concepts/address.md). You can use them for SSH access. ## Configure GitLab {#configure-gitlab} To set GitLab up and configure the CI process, create a new project and enter your CI login credentials: 1. Navigate to **Compute Cloud**. 1. Select the VM instance you created and copy its public IP address. 1. [Connect](../operations/vm-connect/ssh.md#vm-connect) to the VM over SSH. 1. Get the GitLab administrator password with the following VM command: ```bash sudo cat /etc/gitlab/initial_root_password ``` 1. Copy the password without spaces from the `Password` line to the clipboard or a separate file. 1. In your browser, open `http://`. This will take you to the GitLab web UI. 1. Log in as the administrator: * **Username or email**: `root`. * **Password**: Password you copied in the previous step. If you are unable to log in, [reset the administrator password](https://docs.gitlab.com/ee/security/reset_user_password.html#reset-your-root-password). 1. [Change the administrator password](https://docs.gitlab.com/ee/user/profile/#change-your-password). 1. Log in as the administrator with the new password. 1. Select **Create a project**. 1. Specify the project name: `gitlab-test`. 1. Click **Create project**. 1. Get a Yandex OAuth token. To do this, follow [this link](https://oauth.yandex.com/authorize?response_type=token&client_id=1a6990aa636648e9b2ef855fa7bec2fb) and click **Allow**. 1. In your browser, open `http:///root`. 1. Select the `gitlab-test` project. 1. On the screen that opens, click **Settings** on the left and, in the popup menu, select **CI/CD**. 1. Under **Variables**, click **Expand**. 1. Create a new variable: * Specify the variable name: `YC_OAUTH`. * As the value of the variable, specify the OAuth token you obtained earlier. * Click **Save variables**. 1. Under **Runners**, click **Expand**. 1. Under **Set up a specific Runner automatically**, you will see the GitLab **server address** and the Runner **token**. You will need them to register your runner. ## Configure your runner {#configure-runner} Runner is a tool which runs user tasks. You need to install a runner on your VM and then register it in GitLab. To make your runner work, install the Yandex Cloud CLI and create a test to check the new VM. 1. [Connect](../operations/vm-connect/ssh.md#vm-connect) to the GitLab VM over SSH: ```bash ssh @ ``` 1. Add a new repository to the package manager: ```bash curl --location https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | sudo bash ``` 1. Install a runner to run your CI scripts: ```bash sudo apt-get -y install gitlab-runner ``` 1. Register your runner: * In response to `Please enter the gitlab-ci coordinator URL`, specify your GitLab IP address. * In response to `Please enter the gitlab-ci token for this runner`, specify your runner token. * In response to `Please enter the gitlab-ci description for this runner`, specify the `gitlab test runner` description. * In response to `Please enter the gitlab-ci tags for this runner`, press **Enter**. * In response to `Please enter the executor`, specify `shell`. ```bash sudo gitlab-runner register Runtime platform arch=amd64 os=linux pid=8197 revision=3afdaba6 version=11.5.0 Running in system-mode. Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/): http:/// Please enter the gitlab-ci token for this runner: Please enter the gitlab-ci description for this runner: [ci-tutorial-gitlab]: gitlab test runner Please enter the gitlab-ci tags for this runner (comma separated): Registering runner... succeeded runner=wZqzyy9s Please enter the executor: virtualbox, docker+machine, docker-ssh+machine, kubernetes, docker, docker-ssh, shell, ssh, parallels: shell Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded! ``` 1. Install the Yandex Cloud CLI to create VMs with the CI script: ```bash curl https://storage.yandexcloud.net/yandexcloud-yc/install.sh --output install.sh sudo bash install.sh -n -i /opt/yc ``` 1. Install the `pytest` package for functional testing: ```bash sudo apt-get install python-pytest ``` 1. Create the `test.py` file with the functional testing script: 1. Open the `gitlab-test` repository home page. 1. Click **+** and select **New file**. 1. In the window that opens, specify the file name: `test.py`. 1. Add the following code to the file: ```python # test.py import pytest import json import socket as s @pytest.fixture def hostname(request): with open("instance-creation.out", "r") as fd: fqdn = json.loads(fd.read()).get("fqdn") return fqdn @pytest.fixture def socket(request): _socket = s.socket(s.AF_INET, s.SOCK_STREAM) def socket_teardown(): _socket.close() request.addfinalizer(socket_teardown) return _socket def test_server_connect(socket, hostname): socket.connect((hostname, 80)) assert socket ``` 1. Specify your commit message and click **Commit changes**. ## Configure CI {#configure-ci} Now you need to configure CI. 1. Open the `gitlab-test` repository home page: ```http http:///root/gitlab-test ``` 1. Click **Set up CI/CD**. This will open a window for adding a new file. 1. GitLab will automatically name the file as `.gitlab-ci.yml`. Do not rename it. Add the following configuration to the file: ```yaml #.gitlab-ci.yml stages: - build - test build: stage: build variables: snapshot_name: test-app-snap folder_id: subnet_name: script: - export instance_name="ci-tutorial-test-app-$(date +%s)" - export PATH="/opt/yc/bin:${PATH}" - yc config set token $YC_OAUTH - yc compute instance create --format json --name $instance_name --folder-id $folder_id --zone ru-central1-d --network-interface subnet-name=$subnet_name,nat-ip-version=ipv4 --create-boot-disk name=$instance_name-boot,type=network-ssd,size=15,snapshot-name=$snapshot_name,auto-delete=true --memory 1 --cores 1 --hostname $instance_name > instance-creation.out - sleep 30 artifacts: when: on_success paths: - instance-creation.out expire_in: 10 mins test_external: stage: test script: - py.test test.py > pytest-external.out artifacts: paths: - pytest-external.out expire_in: 1 hour ``` 1. In the `snapshot_name` field, specify the first VM snapshot name. In the `folder_id` field, specify the VM folder ID. In the `subnet_name` field, specify your VM subnet name; you can find this name on the Virtual Private Cloud page in the relevant folder of the management console. 1. Click **Commit changes**. ## Test the application on the CI VM {#test-new-vm} After making a commit, make sure that CI ran successfully. You should see a new VM with a web server and your test application in the relevant folder. To test the new VM: 1. Open the Yandex Cloud management console. 1. In the folder the VMs were created in, navigate to **Compute Cloud**. 1. If you did everything correctly, you will see a new VM in the list, e.g., `ci-tutorial-test-app-1543910277`. 1. Select the new VM and copy its public IP address. 1. In your browser, open this link: ```http http:///test/?text=hello_world ``` 1. Your test application should also work on the new VM, returning reversed words from the `text` argument. ### Delete the resources you created {#clear-out} If you no longer need the VMs and images you created: * [Delete the VMs](../operations/vm-control/vm-delete.md). * [Delete the images](../operations/image-control/delete.md).