[Yandex Cloud documentation](../../index.md) > [Tutorials](../index.md) > [Building a data platform](index.md) > Automating operations with Yandex Data Processing using Managed Service for Apache Airflow™

# Automating operations with Yandex Data Processing using Yandex Managed Service for Apache Airflow™

# Automating operations with Yandex Data Processing using Yandex Managed Service for Apache Airflow™


{% note warning %}

This tutorial was tested on clusters with the Apache Airflow™ version below 3.0.

{% endnote %}

In Yandex Managed Service for Apache Airflow™, you can create a [directed acyclic graph (DAG)](../../managed-airflow/concepts/index.md) to automate your operations in [Yandex Data Processing](../../data-proc/index.md). Below is an example of a DAG that includes a number of tasks:

1. Create a Yandex Data Processing cluster.
1. Create and run a [PySpark job](../../data-proc/concepts/jobs.md).
1. Delete the Yandex Data Processing cluster.

With a DAG like this, a cluster is short-lived. Since the cost of Yandex Data Processing resources [depends on their usage time](../../data-proc/pricing.md), you can use resources with higher capacity in the cluster and quickly process a larger amount of data at no additional cost.

In this DAG, a Yandex Data Processing cluster is created without using Hive. In the example below, a [Apache Hive™ Metastore cluster](../../metadata-hub/concepts/metastore.md) is used for storing table metadata. The saved metadata can then be used by another Yandex Data Processing cluster.

To automate operations with Yandex Data Processing using Managed Service for Apache Airflow™:

1. [Set up your infrastructure](#infra).
1. [Prepare a PySpark job](#prepare-a-job).
1. [Prepare and run a DAG file](#dag).
1. [Check the result](#check-out).

If you no longer need the resources you created, [delete them](#clear-out).


## Required paid resources {#paid-resources}

* Managed Service for Apache Airflow™ cluster: computing resources of cluster components (see [Managed Service for Apache Airflow™ pricing](../../managed-airflow/pricing.md)).
* Apache Hive™ Metastore cluster: computing resources of cluster components (see [Yandex MetaData Hub pricing](../../metadata-hub/pricing.md#metastore)).
* NAT gateway: hourly use of the gateway and its outgoing traffic (see [Yandex Virtual Private Cloud pricing](../../vpc/pricing.md)).
* Yandex Object Storage buckets: use of storage, data operations (see [Object Storage pricing](../../storage/pricing.md)).
* Yandex Data Processing cluster: use of computing resources with a Yandex Data Processing markup, use of network drives, retrieval and storage of logs, amount of outgoing traffic (see [Yandex Data Processing pricing](../../data-proc/pricing.md)).


## Set up your infrastructure {#infra}

The example below illustrates two scenarios. Select the one you find most relevant:

* **High security level**. This is a recommended scenario, as it respects the [principle of least privilege](../../iam/best-practices/using-iam-securely.md#restrict-access). This scenario entails the following:

   * Splitting access permissions across different service accounts. You have to create a separate service account for each cluster and assign to it only the roles required for this account's cluster to operate.
   * Using multiple buckets for different tasks and storing different data in separate buckets. For example, the results of running a PySpark job get written to one bucket, and logs, to another.
   * Setting up security groups. This way, you can restrict traffic and grant access only to authorized resources.

* **Simplified setup**. This scenario implies a lower security level:

   * Using a single service account with more privileges than required.
   * Storing all data in a single bucket but in different folders.
   * No security groups are set up.

{% list tabs group=instructions %}

* High security level

   Set up the infrastructure:

   1. [Create service accounts](../../iam/operations/sa/create.md) with the following roles:

      #|
      || **Service account** | **Roles** ||
      || `airflow-agent` for an Apache Airflow™ cluster |
      * [dataproc.editor](../../iam/roles-reference.md#dataproc-editor) to manage a Yandex Data Processing cluster from a DAG.
      * [vpc.user](../../iam/roles-reference.md#vpc-user) to use the [Virtual Private Cloud subnet](../../vpc/concepts/network.md#subnet) in the Apache Airflow™ cluster.
      * [managed-airflow.integrationProvider](../../iam/roles-reference.md#managed-airflow-integrationProvider) to enable the Apache Airflow™ cluster to [interact with other resources](../../managed-airflow/concepts/impersonation.md).
      * [iam.serviceAccounts.user](../../iam/roles-reference.md#iam-serviceAccounts-user) to specify the `data-processing-agent` service account when creating a Yandex Data Processing cluster. ||
      || `metastore-agent` for a Apache Hive™ Metastore cluster |
      * [managed-metastore.integrationProvider](../../iam/roles-reference.md#managed-metastore-integrationProvider) to enable the Apache Hive™ Metastore cluster to [interact with other resources](../../metadata-hub/concepts/metastore-impersonation.md). ||
      || `data-processing-agent` for a Yandex Data Processing cluster |
      * [dataproc.agent](../../data-proc/security/index.md#dataproc-agent): To enable the service account to get info on cluster host states, [jobs](../../data-proc/concepts/jobs.md), and [log groups](../../logging/concepts/log-group.md).
      * [dataproc.provisioner](../../data-proc/security/index.md#dataproc-provisioner): To enable the service account to work with an autoscaling instance group. This will enable [subcluster autoscaling](../../data-proc/concepts/autoscaling.md).
      * [resource-manager.auditor](../../resource-manager/security/index.md#resource-manager-auditor) or higher for the folder where you want to create a Yandex Data Processing cluster: For connection to the cluster using [OS Login](../../organization/concepts/os-login.md).
      
      {% note tip %}
      
      To restrict the permissions of a cluster's service account (its IAM token is available when running jobs):
      
      1. Specify a separate service account for cluster autoscaling when [creating](../../data-proc/operations/cluster-create.md) or [updating](../../data-proc/operations/cluster-update.md) the cluster via the Yandex Cloud CLI, Terraform, or API.
      1. Assign the `dataproc.provisioner` role to this account only.
      
      {% endnote %}
      ||
      |#

   1. [Create buckets](../../storage/operations/buckets/create.md):

      * `<bucket_for_Managed_Airflow>`
      * `<bucket_for_PySpark_job_source_code>`
      * `<bucket_for_PySpark_job_output_data>`
      * `<bucket_for_collecting_Spark_logs>`

      You need multiple buckets with different access permissions.

   1. [Grant permissions](../../storage/operations/buckets/edit-acl.md) for the buckets as follows:

      * `<bucket_for_Managed_Airflow>`: `READ` permission to the `airflow-agent` service account.
      * `<bucket_for_PySpark_job_source_code>`: `READ` permission to the `data-processing-agent` service account.
      * `<bucket_for_PySpark_job_output_data>`: `READ and WRITE` permissions to the `data-processing-agent` and `metastore-agent` service accounts.
      * `<bucket_for_collecting_Spark_logs>`: `READ and WRITE` permissions to the `data-processing-agent` service account.

   1. [Create a cloud network](../../vpc/operations/network-create.md) named `data-processing-network`.

      This will automatically create three subnets in different availability zones.

   1. [Set up a NAT gateway](../../vpc/operations/create-nat-gateway.md) for the `data-processing-network-ru-central1-a` subnet.
   1. For the Apache Hive™ Metastore cluster, [create a security group](../../vpc/operations/security-group-create.md) named `metastore-sg` in `data-processing-network`. Add the following rules to the group:

      * For incoming client traffic:

         * Port range: `30000-32767`
         * Protocol: `Any`
         * Source: `CIDR`
         * CIDR blocks: `0.0.0.0/0`

      * For incoming load balancer traffic:

         * Port range: `10256`
         * Protocol: `Any`
         * Source: `Load balancer health checks`

   1. For the Managed Service for Apache Airflow™ and Yandex Data Processing clusters, create a security group named `airflow-sg` in `data-processing-network`. Add the following rules to it:

      * For incoming service traffic:

         * Port range: `0-65535`
         * Protocol: `Any`
         * Source: `Security group`
         * Security group: `Self`

      * For outgoing service traffic:

         * Port range: `0-65535`
         * Protocol: `Any`
         * Destination: `Security group`
         * Security group: `Self`

      * For outgoing HTTPS traffic:

         * Port range: `443`
         * Protocol: `TCP`
         * Destination: `CIDR`
         * CIDR blocks: `0.0.0.0/0`

      * For outgoing traffic, to allow Yandex Data Processing cluster connections to Apache Hive™ Metastore:

         * Port range: `9083`
         * Protocol: `Any`
         * Destination: `Security group`
         * Security group: `metastore-sg` (`From list`)

   1. [Create a Apache Hive™ Metastore cluster](../../metadata-hub/operations/metastore/cluster-create.md) with the following parameters:

      * **Service account**: `metastore-agent`
      * **Version**: `3.1`
      * **Network**: `data-processing-network`
      * **Subnet**: `data-processing-network-ru-central1-a`
      * **Security group**: `metastore-sg`

   1. [Create a Managed Service for Apache Airflow™ cluster](../../managed-airflow/operations/cluster-create.md) with the following parameters:

      * **Service account**: `airflow-agent`
      * **Availability zone**: `ru-central1-a`
      * **Network**: `data-processing-network`
      * **Subnet**: `data-processing-network-ru-central1-a`
      * **Security group**: `airflow-sg`
      * **Bucket name**: `<bucket_for_Managed_Airflow>`

* Simplified setup

   Set up your infrastructure:

   1. [Create a service account](../../iam/operations/sa/create.md) named `my-editor` with the following roles:

      * [dataproc.editor](../../iam/roles-reference.md#dataproc-editor) to manage a Yandex Data Processing cluster from a DAG.
      * [editor](../../iam/roles-reference.md#editor) to perform other operations.

   1. [Create a bucket](../../storage/operations/buckets/create.md) named `<bucket_for_jobs_and_data>`.

      You do not need to [grant a permission](../../storage/operations/buckets/edit-acl.md) for this bucket to the service account as the `editor` role is enough.

   1. [Create a cloud network](../../vpc/operations/network-create.md) named `data-processing-network`.

      This will automatically create three subnets in different availability zones and a security group.

   1. [Set up a NAT gateway](../../vpc/operations/create-nat-gateway.md) for the `data-processing-network-ru-central1-a` subnet.
   1. [Create a Apache Hive™ Metastore cluster](../../metadata-hub/operations/metastore/cluster-create.md) with the following parameters:

      * **Service account**: `my-editor`
      * **Version**: `3.1`
      * **Network**: `data-processing-network`
      * **Subnet**: `data-processing-network-ru-central1-a`
      * **Security group**: Default group in `data-processing-network`

   1. [Create a Managed Service for Apache Airflow™ cluster](../../managed-airflow/operations/cluster-create.md) with the following parameters:

      * **Service account**: `my-editor`
      * **Availability zone**: `ru-central1-a`
      * **Network**: `data-processing-network`
      * **Subnet**: `data-processing-network-ru-central1-a`
      * **Security group**: Default group in `data-processing-network`
      * **Bucket name**: `<bucket_for_jobs_and_data>`

{% endlist %}

## Prepare a PySpark job {#prepare-a-job}

For a PySpark job, we will use a Python script that creates a table and is stored in the Object Storage bucket. Prepare a script file:

{% list tabs group=instructions %}

* High security level

   1. Create a local file named `create-table.py` and paste the following script to it:

      {% cut "create-table.py" %}

      ```python
      from pyspark.sql.types import *
      from pyspark.sql import SparkSession

      # Creating a Spark session
      spark = SparkSession.builder \
          .appName("create-table") \
          .enableHiveSupport() \
          .getOrCreate()

      # Creating a data schema
      schema = StructType([StructField('Name', StringType(), True),
      StructField('Capital', StringType(), True),
      StructField('Area', IntegerType(), True),
      StructField('Population', IntegerType(), True)])

      # Creating a dataframe
      df = spark.createDataFrame([('Australia', 'Canberra', 7686850, 19731984), ('Austria', 'Vienna', 83855, 7700000)], schema)

      # Writing the dataframe to a bucket as a countries table
      df.write.mode("overwrite").option("path","s3a://<bucket_for_PySpark_job_output_data>/countries").saveAsTable("countries")
      ```

      {% endcut %}

   1. In `<bucket_for_PySpark_job_source_code>`, create a folder named `scripts` and [upload](../../storage/operations/objects/upload.md#simple) the `create-table.py` file to it.

* Simplified setup

   1. Create a local file named `create-table.py` and paste the following script to it:

      {% cut "create-table.py" %}

      ```python
      from pyspark.sql.types import *
      from pyspark.sql import SparkSession

      # Creating a Spark session
      spark = SparkSession.builder \
          .appName("create-table") \
          .enableHiveSupport() \
          .getOrCreate()

      # Creating a data schema
      schema = StructType([StructField('Name', StringType(), True),
      StructField('Capital', StringType(), True),
      StructField('Area', IntegerType(), True),
      StructField('Population', IntegerType(), True)])

      # Creating a dataframe
      df = spark.createDataFrame([('Australia', 'Canberra', 7686850, 19731984), ('Austria', 'Vienna', 83855, 7700000)], schema)

      # Writing the dataframe to a bucket as a countries table
      df.write.mode("overwrite").option("path","s3a://<bucket_for_jobs_and_data>/countries").saveAsTable("countries")
      ```

      {% endcut %}

   1. In `<bucket_for_jobs_and_data>`, create a folder named `scripts` and [upload](../../storage/operations/objects/upload.md#simple) the `create-table.py` file to it.

{% endlist %}

## Prepare and run a DAG file {#dag}

A DAG will have multiple vertices that form a sequence of actions:

1. Managed Service for Apache Airflow™ creates a temporary lightweight Yandex Data Processing cluster whose settings are set in the DAG. This cluster automatically connects to the previously created Apache Hive™ Metastore cluster.
1. When the Yandex Data Processing cluster is ready, a PySpark job is run.
1. Once the job is complete, the temporary Yandex Data Processing cluster is deleted.

To prepare a DAG:

{% list tabs group=instructions %}

* High security level

   1. [Create an SSH key](../../compute/operations/vm-connect/ssh.md#creating-ssh-keys). Save the public part of the key: you will need it to create a Yandex Data Processing cluster.
   1. Create a local file named `Data-Processing-DAG.py`, paste the following script to it and substitute the variables with your infrastructure data:

      {% cut "Data-Processing-DAG.py" %}

      ```python
      import uuid
      import datetime
      from airflow import DAG
      from airflow.utils.trigger_rule import TriggerRule
      from airflow.providers.yandex.operators.yandexcloud_dataproc import (
          DataprocCreateClusterOperator,
          DataprocCreatePysparkJobOperator,
          DataprocDeleteClusterOperator,
      )

      # Your infrastructure data
      YC_DP_AZ = 'ru-central1-a'
      YC_DP_SSH_PUBLIC_KEY = '<public_part_of_SSH_key>'
      YC_DP_SUBNET_ID = '<subnet_ID>'
      YC_DP_SA_ID = '<ID_of_data_processing_agent_service_account>'
      YC_DP_METASTORE_URI = '<IP_address>'
      YC_SOURCE_BUCKET = '<bucket_for_PySpark_job_source_code>'
      YC_DP_LOGS_BUCKET = '<bucket_for_collecting_Spark_logs>'

      # DAG settings
      with DAG(
              'DATA_INGEST',
              schedule='@hourly',
              tags=['data-processing-and-airflow'],
              start_date=datetime.datetime.now(),
              max_active_runs=1,
              catchup=False
      ) as ingest_dag:
          # Step 1: Creating a Yandex Data Proc cluster
          create_spark_cluster = DataprocCreateClusterOperator(
              task_id='dp-cluster-create-task',
              cluster_name=f'tmp-dp-{uuid.uuid4()}',
              cluster_description='Temporary cluster for running a PySpark job orchestrated by Managed Service for Apache Airflow™',
              ssh_public_keys=YC_DP_SSH_PUBLIC_KEY,
              service_account_id=YC_DP_SA_ID,
              subnet_id=YC_DP_SUBNET_ID,
              s3_bucket=YC_DP_LOGS_BUCKET,
              zone=YC_DP_AZ,
              cluster_image_version='2.1',
              masternode_resource_preset='s2.small',
              masternode_disk_type='network-ssd',
              masternode_disk_size=200,
              computenode_resource_preset='m2.large',
              computenode_disk_type='network-ssd',
              computenode_disk_size=200,
              computenode_count=2,
              computenode_max_hosts_count=5,  # The number of data-handling subclusters will automatically scale up if the load is high.
              services=['YARN', 'SPARK'],     # A lightweight cluster is being created.
              datanode_count=0,               # Without data storage subclusters.
              properties={                    # With a reference to a remote Apache Hive™ Metastore cluster.
                  'spark:spark.hive.metastore.uris': f'thrift://{YC_DP_METASTORE_URI}:9083',
              },
          )

          # Step 2: Running a PySpark job
          poke_spark_processing = DataprocCreatePysparkJobOperator(
              task_id='dp-cluster-pyspark-task',
              main_python_file_uri=f's3a://{YC_SOURCE_BUCKET}/scripts/create-table.py',
          )

          # Step 3: Deleting the Yandex Data Processing cluster
          delete_spark_cluster = DataprocDeleteClusterOperator(
              task_id='dp-cluster-delete-task',
              trigger_rule=TriggerRule.ALL_DONE,
          )

          # Building a DAG based on the above steps
          create_spark_cluster >> poke_spark_processing >> delete_spark_cluster
      ```

      Where:

      * `YC_DP_AZ`: Availability zone for the Yandex Data Processing cluster.
      * `YC_DP_SSH_PUBLIC_KEY`: Public part of the SSH key for the Yandex Data Processing cluster.
      * `YC_DP_SUBNET_ID`: Subnet ID.
      * `YC_DP_SA_ID`: ID of the service account for Yandex Data Processing.
      * `YC_DP_METASTORE_URI`: IP address of the Apache Hive™ Metastore cluster.
      * `YC_SOURCE_BUCKET`: Bucket with the Python script for the PySpark job.
      * `YC_DP_LOGS_BUCKET`: Bucket for logs.

      {% endcut %}

   1. Upload the DAG to the Managed Service for Apache Airflow™ cluster: in `<bucket_for_Managed_Airflow>`, create a folder named `dags` and upload the `Data-Processing-DAG.py` file to it.
   1. Open the Apache Airflow™ web interface.
   1. Make sure the **DAGs** section now contains the new DAG named `DATA_INGEST` and tagged as `data-processing-and-airflow`.

      It may take a few minutes to upload a DAG file from the bucket.

   1. To run the DAG, click ![image](../../_assets/managed-airflow/trigger-dag.png =18x) in the line with its name.

* Simplified setup

   1. [Create an SSH key](../../compute/operations/vm-connect/ssh.md#creating-ssh-keys). Save the public part of the key: you will need it to create a Yandex Data Processing cluster.
   1. Create a local file named `Data-Processing-DAG.py`, paste the following script to it and substitute the variables with your infrastructure data:

      {% cut "Data-Processing-DAG.py" %}

      ```python
      import uuid
      import datetime
      from airflow import DAG
      from airflow.utils.trigger_rule import TriggerRule
      from airflow.providers.yandex.operators.yandexcloud_dataproc import (
          DataprocCreateClusterOperator,
          DataprocCreatePysparkJobOperator,
          DataprocDeleteClusterOperator,
      )

      # Your infrastructure data
      YC_DP_AZ = 'ru-central1-a'
      YC_DP_SSH_PUBLIC_KEY = '<public_part_of_SSH_key>'
      YC_DP_SUBNET_ID = '<subnet_ID>'
      YC_DP_SA_ID = '<ID_of_my-editor_service_account>'
      YC_DP_METASTORE_URI = '<IP_address>'
      YC_BUCKET = '<bucket_for_jobs_and_data>'

      # DAG settings
      with DAG(
              'DATA_INGEST',
              schedule='@hourly',
              tags=['data-processing-and-airflow'],
              start_date=datetime.datetime.now(),
              max_active_runs=1,
              catchup=False
      ) as ingest_dag:
          # Step 1: Creating a Yandex Data Proc cluster
          create_spark_cluster = DataprocCreateClusterOperator(
              task_id='dp-cluster-create-task',
              cluster_name=f'tmp-dp-{uuid.uuid4()}',
              cluster_description='Temporary cluster for running a PySpark job orchestrated by Managed Service for Apache Airflow™',
              ssh_public_keys=YC_DP_SSH_PUBLIC_KEY,
              service_account_id=YC_DP_SA_ID,
              subnet_id=YC_DP_SUBNET_ID,
              s3_bucket=YC_BUCKET,
              zone=YC_DP_AZ,
              cluster_image_version='2.1',
              masternode_resource_preset='s2.small',
              masternode_disk_type='network-ssd',
              masternode_disk_size=200,
              computenode_resource_preset='m2.large',
              computenode_disk_type='network-ssd',
              computenode_disk_size=200,
              computenode_count=2,
              computenode_max_hosts_count=5,  # The number of data-handling subclusters will automatically scale up if the load is high.
              services=['YARN', 'SPARK'],     # A lightweight cluster is being created.
              datanode_count=0,               # Without data storage subclusters.
              properties={                    # With a reference to a remote Apache Hive™ Metastore cluster.
                  'spark:spark.hive.metastore.uris': f'thrift://{YC_DP_METASTORE_URI}:9083',
              },
          )

          # Step 2: Running a PySpark job
          poke_spark_processing = DataprocCreatePysparkJobOperator(
              task_id='dp-cluster-pyspark-task',
              main_python_file_uri=f's3a://{YC_BUCKET}/scripts/create-table.py',
          )

          # Step 3: Deleting the Yandex Data Processing cluster
          delete_spark_cluster = DataprocDeleteClusterOperator(
              task_id='dp-cluster-delete-task',
              trigger_rule=TriggerRule.ALL_DONE,
          )

          # Building a DAG based on the above steps
          create_spark_cluster >> poke_spark_processing >> delete_spark_cluster
      ```

      Where:

      * `YC_DP_AZ`: Availability zone for the Yandex Data Processing cluster.
      * `YC_DP_SSH_PUBLIC_KEY`: Public part of the SSH key for the Yandex Data Processing cluster.
      * `YC_DP_SUBNET_ID`: Subnet ID.
      * `YC_DP_SA_ID`: `my-editor` service account ID.
      * `YC_DP_METASTORE_URI`: IP address of the Apache Hive™ Metastore cluster.
      * `YC_BUCKET` — `<bucket_for_jobs_and_data>`.

      {% endcut %}

   1. Upload the DAG to the Managed Service for Apache Airflow™ cluster: in `<bucket_for_jobs_and_data>`, create a folder named `dags` and upload the `Data-Processing-DAG.py` file to it.
   1. Open the Apache Airflow™ web interface.
   1. Make sure the **DAGs** section now contains the new DAG named `DATA_INGEST` and tagged as `data-processing-and-airflow`.

      It may take a few minutes to upload a DAG file from the bucket.

   1. To run a DAG, click ![image](../../_assets/managed-airflow/trigger-dag.png =18x) in the line with its name.

{% endlist %}

## Check the result {#check-out}

{% list tabs group=instructions %}

* High security level

   1. To monitor task execution results, click the DAG name. You can find the results in the **Grid** tab.
   1. Wait until the status of all the three tasks in the DAG changes to **Success**. Simultaneously, you can check that a Yandex Data Processing cluster is being created, the PySpark job is running, and the same cluster is being deleted in the [management console](https://console.yandex.cloud).
   1. Make sure your `<bucket_for_PySpark_job_output_data>` now contains the `countries` folder with the `part-00000-...` file. The data from the created table is now stored in the Object Storage bucket and the table metadata is stored in the Apache Hive™ Metastore cluster.
   1. Make sure there are PySpark job logs in `<bucket_for_collecting_Spark_logs>`.

* Simplified setup

   1. To monitor task execution results, click the DAG name. You can find the results in the **Grid** tab.
   1. Wait until the status of all the three tasks in the DAG changes to **Success**. Simultaneously, you can check that an Yandex Data Processing cluster is being created, the PySpark job is running, and the same cluster is being deleted in the [management console](https://console.yandex.cloud).
   1. Make sure `<bucket_for_jobs_and_data>` now contains the `countries` folder with the `part-00000-...` file. The data from the created table is now stored in the Object Storage bucket and the table metadata is stored in the Apache Hive™ Metastore cluster.
   1. Make sure there are PySpark job logs in `<bucket_for_jobs_and_data>`. These logs are written to the `dataproc`, `user`, and `var` folders.

{% endlist %}

## Delete the resources you created {#clear-out}

Some resources are not free of charge. Delete the resources you no longer need to avoid paying for them:

{% list tabs group=instructions %}

* High security level

   1. [Service accounts](../../iam/operations/sa/delete.md).
   1. [Object Storage buckets](../../storage/operations/buckets/delete.md).
   1. [Apache Hive™ Metastore cluster](../../metadata-hub/operations/metastore/cluster-delete.md).
   1. [Managed Service for Apache Airflow™ cluster](../../managed-airflow/operations/cluster-delete.md).
   1. [Route table](../../vpc/operations/delete-route-table.md).
   1. [NAT gateway](../../vpc/operations/delete-nat-gateway.md).
   1. [Security groups](../../vpc/operations/security-group-delete.md).
   1. [Cloud subnets](../../vpc/operations/subnet-delete.md) created in `data-processing-network` by default.
   1. [Cloud network](../../vpc/operations/network-delete.md).

* Simplified setup

   1. [Service account](../../iam/operations/sa/delete.md).
   1. [Object Storage bucket](../../storage/operations/buckets/delete.md).
   1. [Apache Hive™ Metastore cluster](../../metadata-hub/operations/metastore/cluster-delete.md).
   1. [Managed Service for Apache Airflow™ cluster](../../managed-airflow/operations/cluster-delete.md).
   1. [Route table](../../vpc/operations/delete-route-table.md).
   1. [NAT gateway](../../vpc/operations/delete-nat-gateway.md).
   1. [Security group](../../vpc/operations/security-group-delete.md) created in `data-processing-network` by default.
   1. [Cloud subnets](../../vpc/operations/subnet-delete.md) created in `data-processing-network` by default.
   1. [Cloud network](../../vpc/operations/network-delete.md).

{% endlist %}