[Yandex Cloud documentation](../../index.md) > [Yandex Managed Service for Apache Airflow™](../index.md) > Concepts > Resource relationships

# Resource relationships in Managed Service for Apache Airflow™

Managed Service for Apache Airflow™ helps you deploy and maintain clusters of [Apache Airflow™](https://airflow.apache.org/) servers in the Yandex Cloud infrastructure.

## About Apache Airflow™ {#about-the-service}

Apache Airflow™ is an open-source platform that enables you to create, schedule, and monitor batch-oriented _workflows_. A workflow defines job relationships and their execution sequence. It is presented as a directed acyclic graph (DAG). DAGs in Apache Airflow™ can be used for automation and scheduled runs of any processes, e.g., [data processing in Apache Spark™](../tutorials/data-processing-automation.md).

Apache Airflow™ follows the _Workflows as code_ approach. It implies that each workflow is implemented using a Python 3 script. A file with this script is called a _DAG file_. It describes jobs, their run schedule, and dependencies between them. This approach allows storing workflows in a version control system, running tests, and enabling technology required for workflows.

Apache Airflow™ is not used for streaming and continuous data processing. If such processing is required, you can develop a solution based on [Yandex Managed Service for Apache Kafka®](../../managed-kafka/index.md).

For more information, see the [Apache Airflow™ documentation](https://airflow.apache.org/docs/apache-airflow/stable/#).

## Managed Service for Apache Airflow™ architecture {#architecture}

The Managed Service for Apache Airflow™ architecture is presented on the diagram:

![architecture](../../_assets/managed-airflow/architecture.svg)

Each [Apache Airflow™ cluster](#cluster) runs in a separate Kubernetes node group with the required network infrastructure. This infrastructure includes a virtual network, a security group, and a service account. Node groups are isolated from each other, both through virtual networks and through Kubernetes itself. Node groups are managed by a common Kubernetes master, and Apache Airflow™ clusters use a common PostgreSQL cluster for data storage.

To ensure isolated data storage, the service limits the use of the PostgreSQL cluster:

* A separate database is created for each Apache Airflow™ cluster in the PostgreSQL cluster. Clusters can connect only to their own database.
* Apache Airflow™ clusters can work only with tables created by Apache Airflow™. You cannot create and modify schemas, tables, functions, procedures, and triggers yourself.
* Read and write speed, as well as the available database storage space, are limited.

    {% note warning %}

    Any malicious attempt to bypass these restrictions will result in your cluster being locked under Clause 7 of the [Acceptable Use Policy](https://yandex.ru/legal/cloud_aup/?lang=ru).

    {% endnote %}

## Apache Airflow™ cluster {#cluster}

The main entity Managed Service for Apache Airflow™ operates is a _cluster_. Inside a cluster, [Apache Airflow™ components](#components) are deployed. Cluster resources may reside in different availability zones. You can learn more about the Yandex Cloud availability zones in the [Platform overview](../../overview/concepts/geo-scope.md) section.

A workflow running in a cluster may access any Yandex Cloud resource within the cloud network where the cluster is located. For example, a workflow can send requests to Yandex Cloud VMs or managed DB clusters. You can build a workflow using multiple resources, e.g., a workflow that collects data from one DB and sends it to another DB or [Yandex Data Processing](../../data-proc/index.md).


## Apache Airflow™ main components {#components}

The main Apache Airflow™ components are shown below:

![components](../../_assets/managed-airflow/components.svg)

Apache Airflow™ components:

* _Web server_: Server in Yandex Cloud hosting an Apache Airflow™ instance. The web server receives user commands sent through the Apache Airflow™ web interface and checks, runs, and debugs Python scripts in DAG files.

   For more information on working with the web interface, see [this Apache Airflow™ guide](https://airflow.apache.org/docs/apache-airflow/stable/ui.html).

* _Scheduler_: Server in Yandex Cloud that controls the job run schedule. The scheduler gets schedule information from DAG files. It uses this schedule to notify workers that it is time to run a DAG file.

* _DAG processor_: Yandex Cloud server to process DAG files.

   {% note warning %}
          
   The _DAG processor_ component is available in Apache Airflow™ 3.0 or higher.
          
   {% endnote %}

* _Workers_: Executors of jobs specified in DAG files. The workers run jobs on the schedule received from the scheduler.

* [_Triggerer_](#triggerer): Service that releases a worker if it goes idle while executing a job with a long event timeout (optional component).

* _DAG file storage_: [Yandex Object Storage bucket](../../storage/concepts/bucket.md) that stores DAG files. This storage can be accessed by web servers, schedulers, workers, and Triggerer.

To ensure fault tolerance and enhance performance, web servers, schedulers, and Triggerer may exist in multiple instances. Their number is set when creating a cluster.

For workers, you can also set the minimum and maximum number of instances while creating a cluster. Their number will be scaled dynamically. This feature is provided by the [KEDA](https://airflow.apache.org/docs/helm-chart/stable/keda.html) controller.

### Apache Airflow™ component configurations {#presets}

A configuration decides the computing power allocated for the web server, scheduler, workers, and the Triggerer service. The available [configuration types](instance-types.md) are as follows:

 * **standard**: With 4:1 RAM GB to vCPU ratio.
 * **cpu-optimized**: With reduced RAM to vCPU ratio (2:1). These configurations may be useful for clusters with higher processor performance requirements.

You can select configurations when [creating a cluster](../operations/cluster-create.md) or change them while [editing it](../operations/cluster-update.md).

## Triggerer {#triggerer}

The Triggerer service reduces worker idle time.

DAGs may contain jobs that send requests to an external system (such as a Apache Spark™ cluster) and wait for it to respond for a certain period of time. If [standard operators](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/operators.html) are used, such a job will occupy a worker while awaiting the response. This keeps the worker idle. If this happens to a large number of workers, job queues will form, reducing the job run speed and slowing down execution.

_Deferrable operators_ help avoid a situation like this. They allow pausing a job, releasing a worker, and isolating the external system request into a separate process called a _trigger_. All triggers are independent from each other and processed by Triggerer asynchronously, with separate resources allocated for it in the cluster. Once a response is received from the external system, a trigger fires, and the scheduler returns the job to the worker.

See how to work with Triggerer in the figure below:

![triggerer](../../_assets/managed-airflow/triggerer.svg)

For more information about deferred operatrors, triggers, and Triggerer, see [this Apache Airflow™ guide](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/deferring.html#deferrable-operators-triggers).


## Use cases {#examples}

* [Automating Yandex Data Processing operations](../tutorials/data-processing-automation.md)
* [Automating Yandex Query tasks with Yandex Managed Service for Apache Airflow™](../tutorials/airflow-auto-tasks.md)

#### Useful links {#see-also}

* [Apache Airflow™ in Yandex Cloud: Managed service or independent deployment](https://yandex.cloud/ru/blog/posts/2025/05/apache-airflow-in-yc)