[Yandex Cloud documentation](../../index.md) > [Yandex DataSphere](../index.md) > [Tutorials](index.md) > Development > Classification of images in video frames

# Classification of images in video frames

Yandex DataSphere allows you to build ML models using the Jupyter Notebook interface.

This tutorial solves the problem of binary image classification. You may have to deal with it when recognizing vehicle types based on CCTV camera images. The CCTV system is assumed to capture images from cameras when detecting motion and save them to a bucket in Yandex Object Storage.

To get an idea of how to solve the problem:
1. [Set up your infrastructure](#deploy-infrastructure).
1. [Configure DataSphere](#project).
1. [Create secrets](#create-secrets).
1. [Prepare notebooks](#set-notebooks).
1. [Install the dependencies](#satisfy-dependencies).
1. [Upload and label the data](#load-dataset).
1. [Prepare the ML model and calculate the features](#get-cnn-model).
1. [Train the classifier using the features you calculated](#classifier-fit).
1. [Predict the feature for the test image](#model-test).
1. [View possible applications of the model](#model-apply).

If you no longer need the resources you created, [delete them](#clear-out).

## Getting started {#before-you-begin}

Before getting started, register in Yandex Cloud, set up a [community](../concepts/community.md), and link your [billing account](../../billing/concepts/billing-account.md) to it.
1. [On the DataSphere home page](https://datasphere.yandex.cloud), click **Try for free** and select an account to log in with: Yandex ID or your working account with the identity federation (SSO).
1. Select the [Yandex Identity Hub organization](../../organization/index.md) you are going to use in Yandex Cloud.
1. [Create a community](../operations/community/create.md).
1. [Link your billing account](../operations/community/link-ba.md) to the DataSphere community you are going to work in. Make sure you have a linked billing account and its [status](../../billing/concepts/billing-account-statuses.md) is `ACTIVE` or `TRIAL_ACTIVE`. If you do not have a billing account yet, create one in the DataSphere interface.

### Required paid resources {#paid-resources}

The model operation cost includes:

* Fee for bucket usage (see [Yandex Object Storage pricing](../../storage/pricing.md)).
* Fee for computing resource usage (see [Yandex DataSphere pricing](../pricing.md)).

## Set up your infrastructure {#deploy-infrastructure}

Log in to the Yandex Cloud [management console](https://console.yandex.cloud) and select the organization you use to access DataSphere. On the [**Yandex Cloud Billing**](https://center.yandex.cloud/billing/accounts) page, make sure you have a billing account linked.

If you have an active billing account, you can go to the [cloud page](https://console.yandex.cloud/cloud) to create or select a folder to run your infrastructure.

{% note info %}

If you are using an [identity federation](../../organization/concepts/add-federation.md) to work with Yandex Cloud, you might not have access to billing details. In this case, contact your Yandex Cloud organization administrator.

{% endnote %}

### Create a folder {#create-folder}

Create a [folder](../../resource-manager/concepts/resources-hierarchy.md) and [network](../../vpc/concepts/network.md#network) with subnets in each [availability zone](../../overview/concepts/geo-scope.md).

{% list tabs group=instructions %}

- Management console {#console}

   1. In the [management console](https://console.yandex.cloud), select a cloud and click ![create](../../_assets/console-icons/plus.svg) **Create folder**.
   1. Name your folder, e.g., `data-folder`.
   1. Click **Create**.

{% endlist %}

### Create a service account {#create-sa}

{% list tabs group=instructions %}

- Management console {#console}

   1. Navigate to `data-folder`.
   1. [Go](../../console/operations/select-service.md#select-service) to **Identity and Access Management**.
   1. Click **Create service account**.
   1. Name the [service account](../../iam/concepts/users/service-accounts.md), e.g., `sa-for-project`.
   1. Click **Add role** and assign the following [roles](../../iam/concepts/access-control/roles.md) to the service account:
      * `storage.viewer` to read data from the Object Storage bucket.
      * `vpc.gateways.user` to use the subnet.

   1. Click **Create**.

{% endlist %}

### Create a static access key for the service account {#create-key}

To allow your service account to get authenticated in Object Storage, create a [static access key](../../iam/concepts/authorization/access-key.md).

{% list tabs group=instructions %}

- Management console {#console}

  1. Navigate to `data-folder`.
  1. [Go](../../console/operations/select-service.md#select-service) to **Identity and Access Management**.
  1. In the left-hand panel, select ![FaceRobot](../../_assets/console-icons/face-robot.svg) **Service accounts**.
  1. In the list that opens, select the `sa-for-project` service account.
  1. Click **Create new key** in the top panel.
  1. Select **Create static access key**.
  1. Enter a description of the key so that you can easily find it in the management console.
  1. Save the ID and secret key. The secret key is not saved in Yandex Cloud, so you will not be able to view it in the management console.

{% endlist %}

### Create an egress NAT gateway {#create-nat}

{% list tabs group=instructions %}

- Management console {#console}

   1. Navigate to `data-folder`.
   1. [Go](../../console/operations/select-service.md#select-service) to **Virtual Private Cloud**.
   1. In the left-hand panel, select **Gateways**.
   1. Click **Create** and set the gateway parameters:
      * Name the gateway, e.g., `nat-for-cluster`.
      * Gateway **Type**: **Egress NAT**.
      * Click **Save**.

   Create a route table:

   1. In the left-hand panel, select **Routing tables**.
   1. Click **Create** and specify the route table parameters:
      1. Enter a name, e.g., `route-table`.
      1. Select the `data-folder` network.
      1. Click **Add**.
         * In the window that opens, select **Gateway** in the **Next hop** field.
         * In the **Gateway** field, select the NAT gateway you created. The destination prefix will apply automatically.
         * Click **Add**.
   1. Click **Create routing table**.

   Associate the route table with a subnet to route traffic from it through the NAT gateway:

   1. In the left-hand panel, select ![image](../../_assets/console-icons/nodes-right.svg) **Subnets**.
   1. In the row with the subnet, click ![image](../../_assets/console-icons/ellipsis.svg).
   1. In the menu that opens, select **Link routing table**.
   1. In the window that opens, select your route table from the list.
   1. Click **Link**.

{% endlist %}

### Create an S3 bucket {#create-bucket}

{% list tabs group=instructions %}

- Management console {#console}

   1. Navigate to `data-folder`.
   1. [Go](../../console/operations/select-service.md#select-service) to **Object Storage**.
   1. Click **Create bucket**.
   1. On the bucket creation page:
      1. Enter a name for the bucket by following the [naming conventions](../../storage/concepts/bucket.md#naming).
      
         {% note warning %}

         Do not use buckets with a dot in their name for connection.

         {% endnote %}

      1. In the **Read objects**, **Read object list**, and **Read settings** fields, select **With authorization**.
      1. Limit the maximum bucket size, if required.
   1. Click **Create bucket** to complete the operation.

{% endlist %}

## Configure DataSphere {#project}

### Create a project {#create-project}

1. Open the DataSphere [home page](https://datasphere.yandex.cloud).
1. In the left-hand panel, select ![image](../../_assets/console-icons/circles-concentric.svg) **Communities**.
1. Select the community where you want to create a project.
1. On the community page, click ![image](../../_assets/console-icons/folder-plus.svg) **Create project**.
1. In the window that opens, enter a name for the project. You can also add a description as needed.
1. Click **Create**.

### Edit the project settings {#change-settings}

1. Select the project in your community or on the DataSphere [home page](https://datasphere.yandex.cloud) in the **Recent projects** tab.
1. Navigate to the **Settings** tab.
1. Under **Advanced settings**, click **![pencil](../../_assets/console-icons/pencil-to-line.svg) Edit**.
1. Specify the parameters:
   * **Default folder**: `data-folder`.
   * **Service account**: `sa-for-project`.
   * **Subnet**: Subnet of the `ru-central1-a` availability zone in `data-folder`.

   {% note info %}
   
   If you specify a subnet in the project settings, the VM preparation during the first computation run may take longer.
   
   {% endnote %}

   * **Security group**: Specify a [security group](../../vpc/concepts/security-groups.md), if used in your organization.
1. Click **Save**.

## Create secrets {#create-secrets}

Create [secrets](../concepts/secrets.md) to store the ID and secret part of the static access key:

1. Under **Project resources** on the project page, click ![secret](../../_assets/console-icons/shield-check.svg)**Secret**.
1. Click **Create**.
1. In the **Name** field, enter a name for the secret. Name the secret with the static key ID as `token`, and the secret with the secret part as `key_value`.
1. In the **Value** field, enter a value to store in encrypted form.
1. Click **Create**. You will see a page with detailed info on the secret you created.

## Prepare notebooks {#set-notebooks}

Clone the Git repository containing the notebooks with the examples of the ML model training and testing:
   1. In the top menu, click **Git** and select **Clone**.
   1. In the window that opens, enter `https://github.com/yandex-cloud-examples/yc-datasphere-video-recognition.git` and click **Clone**.
   
   Wait until cloning is complete. It may take some time. You will see the cloned repository folder in the ![folder](../../_assets/datasphere/jupyterlab/folder.svg) **File Browser** section.

There are two notebooks in the repository:

* `model-building.ipynb`: To set up the environment and train your model using the ResNet50 [convolutional neural network](https://en.wikipedia.org/wiki/Convolutional_neural_network) (CNN).
* `model-testing.ipynb`: To test your model.

## Install dependencies {#satisfy-dependencies}

{% note info %}

In this example, the model is trained and tested using the g1.1 computing resource [configuration](../concepts/configurations.md). You can use a different configuration with a GPU. To do this, change the configuration in the code in all notebook cells as needed.

{% endnote %}

1. Open the **ML** folder and then the **model-building.ipynb** notebook.
1. Click the first cell to select it:

   ```
   #!g1.1
   %matplotlib inline
   import matplotlib
   import matplotlib.pyplot as plt
   import os
   import io
   from os import path
   ...
   ```

1. To run the cell, click ![run](../../_assets/datasphere/jupyterlab/run.svg) or press **Shift** + **Enter**.
1. Wait for the operation to complete.

The solution uses the [Keras interface](https://keras.io/about/) of the TensorFlow library with a [CNTK backend](https://docs.microsoft.com/en-us/cognitive-toolkit/). To connect to the bucket with images, you need the `boto3` package. The cell also sets the environment variables to access the CNTK backend and connect to the bucket.

The packages listed in the cell are already installed in DataSphere and you can import them using the `import` command. For the full list of packages pre-installed in DataSphere, see [List of pre-installed software](../concepts/preinstalled-packages.md).

## Upload and label the data {#load-dataset}

Go to the **Connect S3** section to:

1. Set up a connection to the S3 bucket.
1. Load the list of objects, i.e., images of cars and buses to train the model.
1. Define the function to extract an image using a key (name).

The next section, **Labeling**, is used for labeling data:
* Images are labeled according to the key value (folder name).
* Bus images are labeled with `0`, and car images are labeled with `1`.

To upload and label the data:

1. In the first cell, replace the `bucket_name` variable value with the name of your bucket. The default value is `bucketwithvideo`.
1. Select all the cells containing code in the **Connect S3** and **Labeling** sections by holding **Shift** and clicking to the left of the cells in question:

   ```
   #!g1.1

   session = boto3.session.Session()
   ...
   ```

1. Run the selected cells.
1. Wait for the operation to complete. When the operation is complete, one of the images is displayed to check if the data was uploaded and labeled correctly.

## Prepare the ML model and calculate the features {#get-cnn-model}

Go to the **Calculating the characteristics** section to:

1. Load the ResNet50 model from the Keras package, with weights pretrained on the ImageNet dataset. This dataset contains 1.2 million images in 1,000 categories.
1. Define the `featurize_images` function. It splits the list of images into chunks of 32 images each, scales the images down to 224×224 pixels, and converts them to a four-dimensional [tensor](https://en.wikipedia.org/wiki/Tensor) to feed to the Keras model. Next, the function calculates their features and returns them as a NumPy array.
1. Use the function to calculate binary features (`1` means a car, `0` means other) and save them to a file. This step may take 10-15 minutes. [Learn more about the ResNet50 model](https://www.kaggle.com/keras/resnet50).

To prepare the model and calculate the features:

1. Select all the cells containing code in the **Calculating the characteristics** section:

   ```
   #!g1.1
   model = ResNet50(weights='imagenet',  input_shape=(224, 224, 3))
   ...
   ```

1. Run the selected cells.
1. Wait for the operation to complete.

## Train the classifier using the features you calculated {#classifier-fit}

In this section, you will train the LightGBM classifier on the features calculated in the previous section. To evaluate the model, you will use cross-validation.

The [K-fold](http://scikit-learn.org/0.16/modules/generated/sklearn.cross_validation.StratifiedKFold.html) method of the _scikit-learn_ package is used. Parts of the training sample (folds) are created based on the specified percentage of data from each class.

This is important when your data contains much fewer images of one category than another. This example uses five-fold cross-validation. You can set a different number of folds in the `n_splits` parameter.

Go to the **Training and cross-validation** and **Saving the model** sections to:

1. Define an object for cross-validation of training results using the **K-fold** method.
1. Prepare a table to store the classification quality metrics.
1. Define the `classification_metrics` function for calculating the selected metrics.
1. Start the LightGBM classifier training. This example uses five-fold cross-validation:
   1. The training sample is divided into five equal non-overlapping folds.
   1. Five iterations take place. At each one, the following steps are performed:
      1. The model is trained on four folds from the sample.
      1. The model is tested on the sample fold not used for training.
      1. The selected quality metrics are output.
1. Train the classifier on the full dataset and output the resulting error matrix.

To train the classifier, run all the cells in the **Training and cross-validation** and **Saving the model** sections one by one.

The model thus trained will be saved to a separate file.

## Get the results of feature prediction based on test data {#model-test}

To use the resulting model:

1. Open the **ML** folder and then the **model-testing.ipynb** notebook.

   This notebook uses the previously trained LightGBM classifier to prepare the entities required to illustrate how you can use your model.

   {% note info %}

   You need much less resources to use your model than to train it; therefore, the minimum c1.4 configuration is used by default.

   {% endnote %}

1. In the second cell, replace the `bucket_name` variable value with the name of your bucket. The default value is `bucketwithvideo`.
1. Run the first three cells. Use these cells to:
   1. Import the packages you need for the test.
   1. Identify the name of the bucket to store the images.
   1. Set up a connection to the bucket with CCTV images.
1. Specify a test image featuring a car:

   ```
   test_image = 'car/electric-cars-17.jpeg'
   ```

   {% note info %}

   If you run the cell again, specify a new image.

   {% endnote %}

1. In the cell from the **Prediction** section, load the ResNet50 model and the prepared LightGBM classifier and calculate the probability of the predicted binary feature value (`1` means a car).

   It takes longer to process the cell with prediction calculation for the first time, because the models are loaded into the memory. The subsequent runs will be faster:

   ```
   %%time
   clf = lgb.Booster(model_file='ImageClassificationML/lightgbm_classifier.model')
   model = ResNet50(weights='imagenet',  input_shape=(224, 224, 3))
   ...
   ```

1. Make sure that the probability value is close to 1 (you should get `≈0.98`).
1. Edit the cell code before loading the model:

   ```
   test_image = 'bus/electric_bus-183.jpeg'
   ```

   This is a test image with no car in it.

1. Run the cell.
1. Repeat the probability calculation and make sure that the value is much less than `0.5`.

This means the classifier can successfully predict the feature for both images.

{% note info %}

You can [share](../operations/projects/publication.md) your finished notebook of computations or [export the whole project](../operations/projects/export.md).

{% endnote %}

## Possible model application {#model-apply}

There are several possible applications for your model:

* The code of the solution allows running a web service using [Yandex Cloud Functions](../../functions/index.md) and analyze on-event CCTV content.
* For parallel processing of images collected from multiple CCTV cameras into an S3 bucket, you can upload the code to an Apache Spark™ cluster in [Yandex Data Processing](../../data-proc/index.md) using the PySpark package.


## How to delete the resources you created {#clear-out}

Some resources are not free of charge. To shut down the model and stop paying for the resources you created, delete those you no longer need:

1. [Delete all objects from the bucket](../../storage/operations/objects/delete.md).
1. [Delete the bucket](../../storage/operations/buckets/delete.md).
1. [Delete the project](../operations/projects/delete.md).
1. [Delete the route table](../../vpc/operations/delete-route-table.md).
1. [Delete the NAT gateway](../../vpc/operations/delete-nat-gateway.md).