# Delivering data from Yandex Forms to Yandex DataLens using Yandex Cloud Functions and Yandex Query via Terraform

# Delivering data from Yandex Forms to Yandex DataLens using Yandex Cloud Functions and Yandex Query via Terraform


To set up Forms and DataLens integration via Terraform:

1. [Set up your infrastructure](#prepare-infrastructure).
1. [Create and configure a function in Yandex Cloud Functions](#create-function).
1. [Create and set up a form in Forms](#create-form).
1. [Configure a connection and binding to data in Yandex Query](#yq-integration).
1. [Set up fetching data in DataLens](#set-up-datalens).
1. [Test the new resource integration](#test-forms-integration).

If you no longer need the resources you created, [delete them](#clear-out).

## Get your cloud ready {#prepare-cloud}

Sign up for Yandex Cloud and create a [billing account](../../../billing/concepts/billing-account.md):
1. Navigate to the [management console](https://console.yandex.cloud) and log in to Yandex Cloud or create a new account.
1. On the **[Yandex Cloud Billing](https://center.yandex.cloud/billing/accounts)** page, make sure you have a billing account linked and it has the `ACTIVE` or `TRIAL_ACTIVE` [status](../../../billing/concepts/billing-account-statuses.md). If you do not have a billing account, [create one](../../../billing/quickstart/index.md) and [link](../../../billing/operations/pin-cloud.md) a cloud to it.

If you have an active billing account, you can create or select a [folder](../../../resource-manager/concepts/resources-hierarchy.md#folder) for your infrastructure on the [cloud page](https://console.yandex.cloud/cloud).

[Learn more about clouds and folders here](../../../resource-manager/concepts/resources-hierarchy.md).

### Required paid resources {#paid-resources}

The infrastructure support cost includes:

* Fee for using the Yandex Object Storage bucket (see [Object Storage pricing](../../../storage/pricing.md)).
* Fee for storing and requesting Yandex Lockbox secrets (see [Yandex Lockbox pricing](../../../lockbox/pricing.md)).
* Fee for function calls and computing resources allocated for executing the functions (see [Cloud Functions pricing](../../../functions/pricing.md)).
* Fee for the amount of data read from sources when executing Query queries (see [Query pricing](../../../query/pricing.md)).
* Fee for using DataLens under the service plan (see [DataLens pricing](../../pricing.md)).

## Set up your infrastructure {#prepare-infrastructure}

With [Terraform](https://www.terraform.io/), you can quickly create a cloud infrastructure in Yandex Cloud and manage it using configuration files. These files store the infrastructure description written in HashiCorp Configuration Language (HCL). If you change the configuration files, Terraform automatically detects which part of your configuration is already deployed, and what should be added or removed.

Terraform is distributed under the [Business Source License](https://github.com/hashicorp/terraform/blob/main/LICENSE). The [Yandex Cloud provider for Terraform](https://github.com/yandex-cloud/terraform-provider-yandex) is distributed under the [MPL-2.0](https://www.mozilla.org/en-US/MPL/2.0/) license.

For more information about the provider resources, see the relevant documentation on the [Terraform](https://www.terraform.io/docs/providers/yandex/index.html) website or [its mirror](../../../terraform/index.md).

To set up your infrastructure via Terraform:

1. If you do not have Terraform yet, [install it](../../../tutorials/infrastructure-management/terraform-quickstart.md#install-terraform).
1. [Get the authentication credentials](../../../tutorials/infrastructure-management/terraform-quickstart.md#get-credentials). You can add them to environment variables or specify them later in the provider configuration file.
1. [Configure and initialize a provider](../../../tutorials/infrastructure-management/terraform-quickstart.md#configure-provider). There is no need to create a provider configuration file manually, you can [download it](https://github.com/yandex-cloud-examples/yc-terraform-provider-settings/blob/main/provider.tf).
1. Place the configuration file in a separate working directory and [specify the parameter values](../../../tutorials/infrastructure-management/terraform-quickstart.md#configure-provider). If you did not add the authentication credentials to environment variables, specify them in the configuration file.

1. Download the [forms-and-datalens-integration.tf](https://github.com/yandex-cloud-examples/yc-serverless-forms-datalens-integration/blob/main/forms-and-datalens-integration.tf) configuration file to the same working directory.

    This file describes:

    * Network and subnet for deploying the infrastructure for your function in Cloud Functions.
    * Service account with the following roles:
        * `lockbox.payloadViewer`: For reading Yandex Lockbox secrets.
        * `functions.functionInvoker`: For calling the function in Cloud Functions.
        * `storage.admin`: For reading data from and writing data to an Object Storage bucket, as well as managing the bucket’s ACL.
        * `yq.viewer` and `yq.invoker`: For integrating DataLens and Query.      
    * Static key for the service account used for creating the Object Storage bucket.
    * Static key for the service account used for creating the function in Cloud Functions.
    * Yandex Lockbox secret for storing data on the service account static key for the function from Cloud Functions.
    * `data` resource for the Yandex Lockbox secret the function in Cloud Functions uses to retrieve the secret version ID.
    * Object Storage bucket to store function results.
    * Publicly accessible function in Cloud Functions.

1. In the local variables section of the `forms-and-datalens-integration.tf` file, specify the following:
    * Your folder ID in the `sa_folder_id` variable.
    * Name of the Object Storage bucket in the `bucket-name` variable.

1. Make sure the Terraform configuration files are correct using this command:

    ```bash
    terraform validate
    ```

    If there are any errors in the configuration files, Terraform will point them out.

1. Create the required infrastructure:

    1. Run this command to view the planned changes:
    
       ```bash
       terraform plan
       ```
    
       If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.
    
    1. If everything looks correct, apply the changes:
       1. Run this command:
    
          ```bash
          terraform apply
          ```
    
       1. Confirm updating the resources.
       1. Wait for the operation to complete.

    All the required resources will be created in the specified folder. You can check resource availability and their settings in the [management console](https://console.yandex.cloud).


## Create and configure a function in Cloud Functions {#create-function}

1. Download the [function code archive](https://github.com/yandex-cloud-examples/yc-serverless-forms-datalens-integration/blob/main/function-zip) to the directory with the `forms-and-datalens-integration.tf` file.    

1. Under the local variables section of the `forms-and-datalens-integration.tf` file, specify the following values:

    * `content_path` = `function.zip`.
    * `create-function` = `1`.

1. Make sure the Terraform configuration files are correct using this command:

    ```bash
    terraform validate
    ```

    If there are any errors in the configuration files, Terraform will point them out.

1. Create the required infrastructure:

    1. Run this command to view the planned changes:
    
       ```bash
       terraform plan
       ```
    
       If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.
    
    1. If everything looks correct, apply the changes:
       1. Run this command:
    
          ```bash
          terraform apply
          ```
    
       1. Confirm updating the resources.
       1. Wait for the operation to complete.

    All the required resources will be created in the specified folder. You can check resource availability and their settings in the [management console](https://console.yandex.cloud).

## Create and set up a Forms form {#set-up-form}

### Create an API key {#create-api-key}

You need an [API key](../../../iam/concepts/authorization/api-key.md) to set up integration with Forms.

{% list tabs group=instructions %}

- Management console {#console}

  1. In the [management console](https://console.yandex.cloud), select the relevant folder.
  1. Navigate to **Identity and Access Management**.
  1. Select the `forms-integration` service account.
  1. Navigate to **Overview**.
  1. Click **Create new key** and select **Create API key**.
  1. Click **Create**.
  1. In the **New key** window that opens, you will see **Key ID** and **Your secret key**. Save them for later.

{% endlist %}

### Create a form {#create-form}

{% list tabs group=instructions %}

- Forms UI {#forms}

  1. Go to [Forms](https://forms.yandex.com/cloud/admin).

  1. Click **Create form**.

  1. Add at least two fields to the form and set up any questions in them.

  1. Click **Publish**.

  1. Save the link to the published form: you will need it later.

{% endlist %}

### Set up integration with a Cloud Functions function {#set-up-integration}

{% list tabs group=instructions %}

- Forms UI {#forms}

  1. Go to [Forms](https://forms.yandex.com/cloud/admin).

  1. Open the form you created earlier.

  1. Go to the **Settings** tab.

  1. Open the **Additional** section.

  1. Under **Cloud function key**, enter the data of the [previously created API key](#create-api-key) and click **Save**.

  1. Open the **Integrations** tab.

  1. Click **Cloud Functions** to add a condition for the integration.

  1. In the **Function code** field, enter the Cloud Functions function ID. You can copy the ID in the **Overview** section of the Cloud Functions function in the management console.

  1. In the **Show outcome notification** field, select **Show**.

  1. Click **Save**.

{% endlist %}

### Test the integration you created {#test-integration}

1. Follow the public link to the previously created form.

1. Complete the form and click **Submit**.

1. Go to [Forms](https://forms.yandex.com/cloud/admin).

1. Open the form you created earlier.

1. Open the **Integrations** tab.

1. Go to the **Completed integrations** page.

1. Open the completed action record and make sure that the **Response** section contains the `200 — OK` response.

1. Navigate to the [management console](https://console.yandex.cloud) and open the [bucket created earlier](#create-s3-bucket). Make sure the bucket now contains a JSON file with the data from the form you completed.

   The name of the directory the file will reside in matches the form's internal ID. Save this ID for the next steps.

   All the future form completion outcomes will be saved in this directory.

1. Complete the form several times more, and make sure you fill one of the fields in the same way in at least two forms. This will improve data visualization when testing the integration.

## Configure a connection and binding to data in Query {#yq-integration}

{% list tabs group=instruction %}

- Query UI {#yquery}

  1. Open the [management console](https://console.yandex.cloud).
  1. Navigate to **Yandex Query**.
  1. In the **Connections** tab, click ![info](../../../_assets/console-icons/plus.svg) **Create new**.
  1. Create a connection with the following parameters:
      * **Name**: `forms-connection`.
      * **Type**: `Object Storage`.
      * **Bucket**: Name of the bucket you [created earlier](#create-s3-bucket).
      * **Service account**: `forms-integration-sa`.
  1. In the window that opens, set the data binding parameters:
      * **Type**: `Object Storage`.
      * **Connection**: `forms-connection`.
      * **Path**: `/<form_ID>/`.
        You can copy the ID:
          * Under **Objects** in the Object Storage bucket. The name of the folder containing the file with results of the completed form matches the form ID.
          * In the address bar of the [Forms interface](https://forms.yandex.com/cloud/admin), on the form viewing or editing page.        
      * **Format**: `json_each-row`.
      * **Columns**: Create columns for the fields you specified in the form.
        If you want Query to define columns automatically, click **Detect columns**.

{% endlist %}

## Set up fetching data in DataLens {#set-up-datalens}

{% list tabs group=instructions %}

- DataLens UI {#datalens}

  1. Navigate to [DataLens](https://datalens.ru/promo).
  1. Click **Start in cloud**.
  1. In the left-hand panel, select ![image](../../../_assets/console-icons/thunderbolt.svg) **Connections** and click **Create connection**.
  1. Navigate to **Yandex Query** and set the following in the window that opens:
      * **Cloud and folder**: Select the cloud and folder where you created the rest of the resources.
      * **Service account**: `forms-integration-sa`.
  1. Click **Save changes**. In the window that opens, specify `forms-datalens-connection` as the connection name, then click **Create**.
  1. Go back to the [DataLens](https://datalens.ru/?skipPromo=true) main page and click **Create dataset**.
  1. In the connection panel, click ![icon](../../../_assets/console-icons/plus.svg) **Add** and select the `forms-datalens-connection` connection.
  1. Under **Tables**, select the table and drag it to your DataLens workspace.
      After loading, the data from the table will appear in the **Preview** panel.
  1. Click **Save** and enter `forms-integration-dataset` as the dataset name, then click **Create**.

{% endlist %}

## Test the new resource integration {#test-forms-integration}

To check how the created resources work together, open the `forms-integration-dataset` dataset and click **Create chart**. In the window that opens, create a chart using the data from the form. For better visualization, create a chart based on a field with repeating values.

## Delete the resources you created {#clear-out}

Some resources are not free of charge. Delete the resources you no longer need to avoid paying for them.

1. In the terminal window, go to the directory containing the infrastructure plan.

    {% note warning %}

    Make sure the directory has no Terraform manifests with the resources you want to keep. Terraform deletes all resources that were created using the manifests in the current directory.

    {% endnote %}

1. Delete resources:

    1. Run this command:

        ```bash
        terraform destroy
        ```

    1. Confirm deleting the resources and wait for the operation to complete.

    All the resources described in the Terraform manifests will be deleted.