[Yandex Cloud documentation](../../index.md) > [Yandex DataSphere](../index.md) > [Tutorials](index.md) > Development > Running scheduled computations

# Running computations on a schedule in DataSphere

You can set up regular run scenarios in [Yandex DataSphere](https://datasphere.yandex.cloud) using the API by triggering notebook cell execution in [Yandex Cloud Functions](../../functions/index.md).

In this tutorial, you will get information about the most discussed stocks on [Reddit](https://tradestie.com/api/v1/apps/reddit), analyze the sentiment of the discussion, and set up regular data updates.

DataSphere collects and analyzes information, while a timer created in Cloud Functions triggers regular cell execution.

To set up regular runs of Jupyter notebook:

1. [Set up your infrastructure](#infra).
1. [Create a notebook](#create-notebook).
1. [Upload and process data](#load-data).
1. [Create a function in Cloud Functions](#create-function).
1. [Create a timer](#create-timer).

If you no longer need the resources you created, [delete them](#clear-out).

## Getting started {#before-you-begin}

Before getting started, register in Yandex Cloud, set up a [community](../concepts/community.md), and link your [billing account](../../billing/concepts/billing-account.md) to it.
1. [On the DataSphere home page](https://datasphere.yandex.cloud), click **Try for free** and select an account to log in with: Yandex ID or your working account with the identity federation (SSO).
1. Select the [Yandex Identity Hub organization](../../organization/index.md) you are going to use in Yandex Cloud.
1. [Create a community](../operations/community/create.md).
1. [Link your billing account](../operations/community/link-ba.md) to the DataSphere community you are going to work in. Make sure you have a linked billing account and its [status](../../billing/concepts/billing-account-statuses.md) is `ACTIVE` or `TRIAL_ACTIVE`. If you do not have a billing account yet, create one in the DataSphere interface.

### Required paid resources {#paid-resources}

The cost of implementing regular runs includes:

* Fee for using [DataSphere computing resources](../pricing.md).
* Fee for the number of [Cloud Functions](../../functions/pricing.md) function calls.

## Set up your infrastructure {#infra}

Log in to the Yandex Cloud [management console](https://console.yandex.cloud) and select the organization you use to access DataSphere. On the [**Yandex Cloud Billing**](https://center.yandex.cloud/billing/accounts) page, make sure you have a billing account linked.

If you have an active billing account, you can go to the [cloud page](https://console.yandex.cloud/cloud) to create or select a folder to run your infrastructure.

{% note info %}

If you are using an [identity federation](../../organization/concepts/add-federation.md) to work with Yandex Cloud, you might not have access to billing details. In this case, contact your Yandex Cloud organization administrator.

{% endnote %}

### Create a folder {#create-folder}

{% list tabs group=instructions %}

- Management console {#console}

   1. In the [management console](https://console.yandex.cloud), select a cloud and click ![create](../../_assets/console-icons/plus.svg) **Create folder**.
   1. Name your folder, e.g., `data-folder`.
   1. Click **Create**.

{% endlist %}

### Create a service account for the DataSphere project {#create-sa}

To access a DataSphere project from a function in Cloud Functions, you need a service account with the `datasphere.community-projects.editor` and `functions.functionInvoker` roles.

{% list tabs group=instructions %}

- Management console {#console}

   1. Navigate to `data-folder`.
   1. [Go](../../console/operations/select-service.md#select-service) to **Identity and Access Management**.
   1. Click **Create service account**.
   1. Name the [service account](../../iam/concepts/users/service-accounts.md), e.g., `reddit-user`.
   1. Click **Add role** and assign the `datasphere.community-projects.editor` and `functions.functionInvoker` roles to the service account.
   1. Click **Create**.

{% endlist %}

### Add the service account to the project {#sa-to-project}

To enable the service account to run a DataSphere project, add it to the list of project members.

{% list tabs group=instructions %}

- Management console {#console}

    1. Select the project in your community or on the DataSphere [home page](https://datasphere.yandex.cloud) in the **Recent projects** tab.
    1. In the **Members** tab, click **Add member**.
    1. Select the `reddit-user` account and click **Add**.

{% endlist %}

### Set up the project {#setup-project}

To reduce DataSphere usage costs, configure the time to release the VM attached to the project.

1. Select the project in your community or on the DataSphere [home page](https://datasphere.yandex.cloud) in the **Recent projects** tab.
1. Navigate to the **Settings** tab.
1. Under **General settings**, click ![pencil](../../_assets/console-icons/pencil-to-line.svg) **Edit**.
1. To configure **Stop inactive VM after**, select `Custom` and specify 5 minutes.
1. Click **Save**.

## Create a notebook {#create-notebook}

1. Select the project in your community or on the DataSphere [home page](https://datasphere.yandex.cloud) in the **Recent projects** tab.
1. Click **Open project in JupyterLab** and wait for the loading to complete.
1. In the top panel, click **File** and select **New** → **Notebook**.
1. Select a kernel and click **Select**.
1. Right-click the notebook and select **Rename**. Enter the name: `test_classifier.ipynb`.

## Upload and process data {#load-data}

To upload information on the most discussed stocks on Reddit and the sentiment of the discussion, paste the code to the `test_classifier.ipynb` notebook cells. You will use it to select the top three most discussed stocks and save them to a CSV file in the project storage.

1. Open the DataSphere project:
   
   1. Select the project in your community or on the DataSphere [home page](https://datasphere.yandex.cloud) in the **Recent projects** tab.
   1. Click **Open project in JupyterLab** and wait for the loading to complete.
   1. Open the notebook tab.

1. Import the libraries:

    ```python
    import pandas as pd
    import requests
    import os.path
    ```

1. Initialize the variables:

    ```python
    REQUEST_URL = "https://tradestie.com/api/v1/apps/reddit"
    FILE_NAME = "/home/jupyter/datasphere/project/stock_sentiments_data.csv"
    TICKERS = ['NVDA', 'TSLA', 'AAPL']
    ```

1. Create a function that sends a request to the Tradestie API and returns a response as `pandas.DataFrame`:

    ```python
    def load_data():
        response = requests.get(REQUEST_URL)
        stocks = pd.DataFrame(response.json())
        stocks = stocks[stocks['ticker'].isin(TICKERS)]
        stocks.drop('sentiment', inplace=True, axis=1)
        return stocks
    ```

1. Set the condition that defines a file to write stock information to:

    ```python
    if os.path.isfile(FILE_NAME):
        stocks = pd.read_csv(FILE_NAME)
    else:
        stocks = load_data()
        stocks['count'] = 1
        stocks.to_csv(FILE_NAME, index=False)
    ```

1. Upload the updated stock data:

    ```python
    stocks_update = load_data()
    ```

1. Compare the updated and existing data:

    ```python
    stocks = stocks.merge(stocks_update, how='left', on = 'ticker')
    stocks['no_of_comments_y'] = stocks['no_of_comments_y'].fillna(stocks['no_of_comments_x'])
    stocks['sentiment_score_y'] = stocks['sentiment_score_y'].fillna(stocks['sentiment_score_y'])
    ```

1. Update the arithmetic average count of comments and sentiment scores:

    ```python
    stocks['count'] += 1
    stocks['no_of_comments_x'] += (stocks['no_of_comments_y'] - stocks['no_of_comments_x'])/stocks['count']
    stocks['sentiment_score_x'] += (stocks['sentiment_score_y'] - stocks['sentiment_score_x'])/stocks['count']
    stocks = stocks[['no_of_comments_x', 'sentiment_score_x', 'ticker', 'count']]
    stocks.columns = ['no_of_comments', 'sentiment_score', 'ticker', 'count']
    print(stocks)
    ```

1. Save the results to a file:

    ```python
    stocks.to_csv(FILE_NAME, index=False)
    ```

## Create a function in Cloud Functions {#create-function}

To run computations without opening JupyterLab, you will need a function in Cloud Functions that will trigger computations in a notebook via the API.

{% list tabs group=instructions %}

- Management console {#console}

    1. In the [management console](https://console.yandex.cloud), select the folder where you want to create your function.
    1. [Go](../../console/operations/select-service.md#select-service) to **Cloud Functions**.
    1. Click **Create function**.
    1. Name the function, e.g., `my-function`.
    1. Click **Create function**.

{% endlist %}

### Create a Cloud Functions version {#create-function-ver}

[Versions](../../functions/concepts/function.md#version) contain the function code, run parameters, and all required dependencies.

{% list tabs group=instructions %}

- Management console {#console}

    1. In the [management console](https://console.yandex.cloud), navigate to the folder containing the function.
    1. [Go](../../console/operations/select-service.md#select-service) to **Cloud Functions**.
    1. Select the function whose version you want to create.
    1. Under **Last version**, click **Сreate in editor**.
    1. Select the `Python` runtime environment. Do not select **Add files with code examples**.
    1. Choose the **Code editor** method.
    1. Click **Create file** and specify a file name, e.g., `index`.
    1. Enter the function code by inserting your project ID and the absolute path to the project notebook:

        ```python
        import requests
    
        def handler(event, context):

            url = 'https://datasphere.api.cloud.yandex.net/datasphere/v2/projects/<project_ID>:execute'
            body = {"notebookId": "/home/jupyter/datasphere/project/test_classifier.ipynb"}
            headers = {"Content-Type" : "application/json",
                       "Authorization": "Bearer {}".format(context.token['access_token'])}
            resp = requests.post(url, json = body, headers=headers)

            return {
            'body': resp.json(),
            }
        ```

       Where:
       * `<project_ID>`: ID of the DataSphere project displayed on the project page under its name.
       * `notebookId`: Absolute path to the project notebook.

    1. Under **Parameters**, set the version parameters:
       * **Entry point**: `index.handler`.
       * **Service account**: `reddit-user`.
    1. Click **Save changes**.

{% endlist %}

## Create a timer {#create-timer}

To run a function every 15 minutes, you will need a [timer](../../functions/concepts/trigger/timer.md).

{% list tabs group=instructions %}

- Management console {#console}

    1. In the [management console](https://console.yandex.cloud), select the folder where you want to create a timer.

    1. [Go](../../console/operations/select-service.md#select-service) to **Cloud Functions**.

    1. In the left-hand panel, select ![image](../../_assets/console-icons/gear-play.svg) **Triggers**.

    1. Click **Create trigger**.

    1. Under **Basic settings**:

        * Enter a name and description for the trigger.
        * In the **Type** field, select `Timer`.
        * In the **Launched resource** field, select `Function`.

    1. Under **Timer settings**, set the function call schedule to `Every 15 minutes`.

    1. Under **Function settings**, select a function and specify:

       * [Function version tag](../../functions/concepts/function.md#tag).
       * `reddit-user` Service account to call the function.

    1. Click **Create trigger**.

{% endlist %}

From now on, the `stock_sentiments_data.csv` file will be updated every 15 minutes. You can find it next to the notebook.

## How to delete the resources you created {#clear-out}

To stop paying for the resources you created:

* [Delete](../../functions/operations/function/function-delete.md) the function.
* [Delete](../../functions/operations/trigger/trigger-delete.md) the trigger.
* [Delete](../operations/projects/delete.md) the project.