[Yandex Cloud documentation](../../index.md) > [Tutorials](../index.md) > [Machine learning and artificial intelligence](index.md) > Development with DataSphere > Running computations in DataSphere using the API

# Running computations in Yandex DataSphere using the API.

In [Yandex DataSphere](https://datasphere.yandex.cloud), you can run code using the API without opening your project. It comes in handy when you need to automate routine operations, fine tune a neural network, or deploy a service that does not require quick API responses.

Using a simple convolutional neural network ([CNN](https://en.wikipedia.org/wiki/Convolutional_neural_network)) as an example, this tutorial describes how to deploy a model trained in DataSphere with [Yandex Cloud Functions](../../functions/index.md). The model's output will be saved to the DataSphere project storage.

For information on how to deploy a service returning results via the API, see [Deploying a service based on a Docker image with FastAPI](../../datasphere/tutorials/node-from-docker-fast-api.md).

1. [Set up your infrastructure](#infra).
1. [Prepare notebooks](#set-notebooks).
1. [Train a neural network](#ai-training).
1. [Upload the model architecture and weights](#load-data).
1. [Create a function in Cloud Functions](#create-function).

If you no longer need the resources you created, [delete them](#clear-out).

## Getting started {#before-you-begin}

Before getting started, register in Yandex Cloud, set up a [community](../../datasphere/concepts/community.md), and link your [billing account](../../billing/concepts/billing-account.md) to it.
1. [On the DataSphere home page](https://datasphere.yandex.cloud), click **Try for free** and select an account to log in with: Yandex ID or your working account with the identity federation (SSO).
1. Select the [Yandex Identity Hub organization](../../organization/index.md) you are going to use in Yandex Cloud.
1. [Create a community](../../datasphere/operations/community/create.md).
1. [Link your billing account](../../datasphere/operations/community/link-ba.md) to the DataSphere community you are going to work in. Make sure you have a linked billing account and its [status](../../billing/concepts/billing-account-statuses.md) is `ACTIVE` or `TRIAL_ACTIVE`. If you do not have a billing account yet, create one in the DataSphere interface.

### Required paid resources {#paid-resources}

The cost of implementing regular runs includes:

* Fee for using [DataSphere computing resources](../../datasphere/pricing.md).
* Fee for the number of [Cloud Functions function calls](../../functions/pricing.md).

## Set up your infrastructure {#infra}

Log in to the Yandex Cloud [management console](https://console.yandex.cloud) and select the organization you use to access DataSphere. On the [**Yandex Cloud Billing**](https://center.yandex.cloud/billing/accounts) page, make sure you have a billing account linked.

If you have an active billing account, you can go to the [cloud page](https://console.yandex.cloud/cloud) to create or select a folder to run your infrastructure.

{% note info %}

If you are using an [identity federation](../../organization/concepts/add-federation.md) to work with Yandex Cloud, you might not have access to billing details. In this case, contact your Yandex Cloud organization administrator.

{% endnote %}

### Create a folder {#create-folder}

{% list tabs group=instructions %}

- Management console {#console}

   1. In the [management console](https://console.yandex.cloud), select a cloud and click ![create](../../_assets/console-icons/plus.svg) **Create folder**.
   1. Name your folder, e.g., `data-folder`.
   1. Click **Create**.

{% endlist %}

### Create a service account for the DataSphere project {#create-sa}

To access a DataSphere project from a Cloud Functions function, you need a [service account](../../iam/concepts/users/service-accounts.md) with the `datasphere.community-projects.editor` role.

{% list tabs group=instructions %}

- Management console {#console}

   1. In the [management console](https://console.yandex.cloud), navigate to `data-folder`.
   1. [Go](../../console/operations/select-service.md#select-service) to **Identity and Access Management**.
   1. Click **Create service account**.
   1. Name your service account, e.g., `datasphere-sa`.
   1. Click **Add role** and assign the `datasphere.community-projects.editor` role to this service account.
   1. Click **Create**.

{% endlist %}

### Add the service account to the project {#sa-to-project}

To enable the service account to run a DataSphere project, add it to the list of project members:

1. Select the project in your community or on the DataSphere [home page](https://datasphere.yandex.cloud) in the **Recent projects** tab.
1. In the **Members** tab, click **Add member**.
1. Select the `datasphere-sa` account and click **Add**.

## Prepare notebooks and your neural network architecture {#set-notebooks}

Clone the Git repository containing the notebooks with the examples of the ML model training and testing:
   1. In the top menu, click **Git** and select **Clone**.
   1. In the window that opens, enter `https://github.com/yandex-cloud-examples/yc-datasphere-batch-execution.git` and click **Clone**.
   
   Wait until cloning is complete. It may take some time. You will see the cloned repository folder in the ![folder](../../_assets/datasphere/jupyterlab/folder.svg) **File Browser** section.

The repository contains two notebooks and the neural network architecture:

* `train_classifier.ipynb`: Notebook for downloading a training sample of the `CIFAR10` dataset and training a simple neural network.
* `test_classifier.ipynb`: Notebook for testing the model.
* `my_nn_model.py`: Neural network architecture. For classification, three-dimensional images are input to the neural network. It contains two convolutional layers with the `maxpool` layer between them, and three linear layers:

    ```python
    import torch.nn as nn
    import torch.nn.functional as F
    import torch

    class Net(nn.Module):
        def __init__(self):
            super().__init__()
            self.conv1 = nn.Conv2d(3, 6, 5)
            self.pool = nn.MaxPool2d(2, 2)
            self.conv2 = nn.Conv2d(6, 16, 5)
            self.fc1 = nn.Linear(16 * 5 * 5, 120)
            self.fc2 = nn.Linear(120, 84)
            self.fc3 = nn.Linear(84, 10)

        def forward(self, x):
            x = self.pool(F.relu(self.conv1(x)))
            x = self.pool(F.relu(self.conv2(x)))
            x = torch.flatten(x, 1) # flatten all dimensions except batch
            x = F.relu(self.fc1(x))
            x = F.relu(self.fc2(x))
            x = self.fc3(x)
            return x
    ```

## Train a neural network {#ai-training}

In the `train_classifier.ipynb` notebook, you will download a training sample of the `CIFAR10` dataset to train a simple neural network. The trained model's weights will be saved to the project storage named `cifar_net.pth`.

1. Open the DataSphere project:
   
   1. Select the project in your community or on the DataSphere [home page](https://datasphere.yandex.cloud) in the **Recent projects** tab.
   1. Click **Open project in JupyterLab** and wait for the loading to complete.
   1. Open the notebook tab.

1. Import the libraries required to train the model:

    ```python
    import torch
    import torchvision
    import torchvision.transforms as transforms
    import torch.optim as optim
    from my_nn_model import Net
    ```

1. Upload the `CIFAR10` dataset to train the model. Images in the dataset are of 10 categories:

    ```python
    transform = transforms.Compose(
        [transforms.ToTensor(),
        transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

    batch_size = 4

    trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                            download=True, transform=transform)
    trainloader = torch.utils.data.DataLoader(trainset, batch_size=batch_size,
                                            shuffle=True, num_workers=2)

    classes = ('plane', 'car', 'bird', 'cat',
            'deer', 'dog', 'frog', 'horse', 'ship', 'truck')
    ```

1. Output sample images from the dataset:

    ```python
    import matplotlib.pyplot as plt
    import numpy as np

    def imshow(img):
        img = img / 2 + 0.5     # unnormalize
        npimg = img.numpy()
        plt.imshow(np.transpose(npimg, (1, 2, 0)))
        plt.show()

    dataiter = iter(trainloader)
    images, labels = next(dataiter)
    imshow(torchvision.utils.make_grid(images))
    print(' '.join(f'{classes[labels[j]]:5s}' for j in range(batch_size)))
    ```

1. Create a loss function and an optimizer required to train the neural network:

    ```python
    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
    net = Net()
    net.to(device)

    criterion = nn.CrossEntropyLoss()
    optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)
    ```

1. Run training on five epochs:

    ```python
    for epoch in range(5):
        running_loss = 0.0
        for i, data in enumerate(trainloader, 0):
            inputs, labels = data[0].to(device), data[1].to(device)

            optimizer.zero_grad()

            outputs = net(inputs)
            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()

            running_loss += loss.item()
            if i % 2000 == 1999:
                print(f'[{epoch + 1}, {i + 1:5d}] loss: {running_loss / 2000:.3f}')
                running_loss = 0.0

    print('Finished Training')
    ```

1. Save the resulting model to the project disk:

    ```python
    torch.save(net.state_dict(), './cifar_net.pth')
    ```

## Upload the model architecture and weights {#load-data}

In the `test_classifier.ipynb` notebook, you will upload the model architecture and weights created while running the [`train_classifier.ipynb`](#ai-training) file. The uploaded model is used for predictions based on the test sample. Prediction results are saved to a file named `test_predictions.csv`.

1. Open the DataSphere project:
   
   1. Select the project in your community or on the DataSphere [home page](https://datasphere.yandex.cloud) in the **Recent projects** tab.
   1. Click **Open project in JupyterLab** and wait for the loading to complete.
   1. Open the notebook tab.

1. Import the libraries required to run the model and make predictions:

    ```python
    import torch
    import torchvision
    import torchvision.transforms as transforms
    from my_nn_model import Net
    import pandas as pd
    ```

1. Prepare the objects that will enable you to access the test sample:

    ```python
    transform = transforms.Compose(
        [transforms.ToTensor(),
        transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

    batch_size = 4

    testset = torchvision.datasets.CIFAR10(root='./data', train=False,
                                        download=True, transform=transform)
    testloader = torch.utils.data.DataLoader(testset, batch_size=batch_size,
                                            shuffle=False, num_workers=2)

    classes = ('plane', 'car', 'bird', 'cat',
            'deer', 'dog', 'frog', 'horse', 'ship', 'truck')
    ```

1. Set the resource configuration to run the model on, CPU or GPU:

    ```python
    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
    ```

1. Upload the trained model's weights and make predictions based on the test sample:

    ```python
    net = Net()
    net.to(device)
    net.load_state_dict(torch.load('./cifar_net.pth'))

    predictions = []
    predicted_labels = []
    with torch.no_grad():
        for data in testloader:
            images, labels = data[0].to(device), data[1].to(device)
            outputs = net(images)
            _, predicted = torch.max(outputs.data, 1)
            predictions.append(predicted.tolist())
            predicted_labels.append([classes[predicted[j]] for j in range(batch_size)])
    ```

1. Save the predictions in `pandas.DataFrame` format:

    ```python
    final_pred = pd.DataFrame({'class_idx': [item for sublist in predictions for item in sublist],
                               'class': [item for sublist in predicted_labels for item in sublist]})
    ```

1. Save the model predictions to a file:

    ```python
    final_pred.to_csv('/home/jupyter/datasphere/project/test_predictions.csv')
    ```

## Create a function in Cloud Functions {#create-function}

To run cells without opening JupyterLab, you need a Cloud Functions that will initiate computations in a notebook via the API.

{% list tabs group=instructions %}

- Management console {#console}

    1. In the [management console](https://console.yandex.cloud), select the folder where you want to create a function.
    1. [Go](../../console/operations/select-service.md#select-service) to **Cloud Functions**.
    1. Click **Create function**.
    1. Name the function, e.g., `ai-function`.
    1. Click **Create function**.

{% endlist %}

### Create a Cloud Functions version {#create-function-ver}

[Versions](../../functions/concepts/function.md#version) contain the function code, run parameters, and all required dependencies.

{% list tabs group=instructions %}

- Management console {#console}

    1. In the [management console](https://console.yandex.cloud), select the folder containing the function.
    1. [Go](../../console/operations/select-service.md#select-service) to **Cloud Functions**.
    1. Select the function whose version you want to create.
    1. Under **Last version**, click **Сreate in editor**.
    1. Select the **Python** runtime environment. Do not select **Add files with code examples**.
    1. Choose the **Code editor** method.
    1. Click **Create file** and specify a file name, e.g., `index`.
    1. Enter the function code by inserting your project ID and the absolute path to the project notebook:
    
        ```python
        import requests
    
        def handler(event, context):

            url = 'https://datasphere.api.cloud.yandex.net/datasphere/v2/projects/<project_ID>:execute'
            body = {"notebookId": "/home/jupyter/datasphere/project/test_classifier.ipynb"}
            headers = {"Content-Type" : "application/json",
                       "Authorization": "Bearer {}".format(context.token['access_token'])}
            resp = requests.post(url, json = body, headers=headers)

            return {
            'body': resp.json(),
            }
        ```

       Where:
       * `<project_ID>`: ID of the DataSphere project displayed on the project page under its name.
       * `notebookId`: Absolute path to the project notebook.

    1. Under **Parameters**, set the version parameters:
       * **Entry point**: `index.handler`.
       * **Service account**: `datasphere-sa`.
    1. In the top-right corner, click **Save changes**.

{% endlist %}

## How to delete the resources you created {#clear-out}

To stop paying for the resources you created:
* [Delete](../../functions/operations/function/function-delete.md) the function.
* [Delete](../../datasphere/operations/projects/delete.md) the project.