 .. Licensed to the Apache Software Foundation (ASF) under one
    or more contributor license agreements.  See the NOTICE file
    distributed with this work for additional information
    regarding copyright ownership.  The ASF licenses this file
    to you under the Apache License, Version 2.0 (the
    "License"); you may not use this file except in compliance
    with the License.  You may obtain a copy of the License at

 ..   http://www.apache.org/licenses/LICENSE-2.0

 .. Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    KIND, either express or implied.  See the License for the
    specific language governing permissions and limitations
    under the License.

Edge Executor
=============

``EdgeExecutor`` is an option if you want to distribute tasks to workers distributed in different locations.
You can use it also in parallel with other executors if needed. Change your ``airflow.cfg`` to point
the executor parameter to ``EdgeExecutor`` and provide the related settings. The ``EdgeExecutor`` is the component
to schedule tasks to the edge workers. The edge workers need to be set-up separately as described in :doc:`deployment`.

The configuration parameters of the Edge Executor can be found in the Edge provider's :doc:`configurations-ref`.

To understand the setup of the Edge Executor, please also take a look to :doc:`architecture`.

 See more details Airflow documentation
  :ref:`apache-airflow:using-multiple-executors-concurrently`.

.. _edge_executor:queue:

Queues
------

When using the EdgeExecutor, the workers that tasks are sent to
can be specified. ``queue`` is an attribute of BaseOperator, so any
task can be assigned to any queue. The default queue for the environment
is defined in the ``airflow.cfg``'s ``operators -> default_queue``. This defines
the queue that tasks get assigned to when not specified, as well as which
queue Airflow workers listen to when started.

Workers can listen to one or multiple queues of tasks. When a worker is
started (using command ``airflow edge worker``), a set of comma-delimited queue
names (with no whitespace) can be given (e.g. ``airflow edge worker -q remote,wisconsin_site``).
This worker will then only pick up tasks wired to the specified queue(s).
If the ``queue`` attribute is not given then a worker will pick tasks from all queues.

This can be useful if you need specialized workers, either from a
resource perspective (for say very lightweight tasks where one worker
could take thousands of tasks without a problem), or from an environment
perspective (you want a worker running from a specific location where required
infrastructure is available).

When using EdgeExecutor in addition to other executors and EdgeExecutor not being the default executor
(that is to say the first one in the list of executors), be reminded to also define EdgeExecutor
as the executor at task or Dag level in addition to the queues you are targeting.
For more details on multiple executors please see :ref:`apache-airflow:using-multiple-executors-concurrently`.

.. _edge_executor:multi_team:

Multi-Team Support
------------------

When multiple teams share a single Airflow deployment, each team may need its own
set of edge workers — for example, separate on-premise sites, different geographic
regions, or isolated execution environments. The EdgeExecutor integrates with
Airflow's Multi-Team mode so that each team's edge jobs and workers are kept separate.

To use multi-team with the EdgeExecutor, first enable Multi-Team mode in your
Airflow deployment and create the teams you need. Then configure the
EdgeExecutor for each team in your ``airflow.cfg``:

.. code-block:: ini

    [core]
    multi_team = True
    executor = EdgeExecutor;team_a=EdgeExecutor;team_b=EdgeExecutor

With this configuration, the scheduler runs a dedicated EdgeExecutor instance per
team. Each instance only schedules and monitors jobs belonging to its own team, and
each worker only picks up jobs assigned to its team.

**Starting a worker for a specific team:**

.. code-block:: bash

    airflow edge worker --team-name team_a -q queue1,queue2

When ``--team-name`` is omitted, the worker operates as a default-team worker — the
same behavior as a single-team deployment. Existing workers continue to work without
any changes.

.. note::

    ``--team-name`` is an experimental UI/REST API-level hint. The Execution API
    does not currently enforce team-based access boundaries — workloads from
    different teams share the same Execution API, JWT signing keys, and access to
    connections, variables, and XComs. See
    :doc:`apache-airflow:security/workload` (section "No team-level isolation in
    Execution API") for the full security boundary discussion.

**Per-team configuration overrides:**

Each team's EdgeExecutor can have its own settings. Use environment variables with
the ``AIRFLOW__<TEAM_NAME>___<SECTION>__<KEY>`` pattern (triple underscore between
team name and section):

.. code-block:: bash

    # Set a longer heartbeat interval for team_a's edge workers
    export AIRFLOW__TEAM_A___EDGE__HEARTBEAT_INTERVAL=30

    # Point team_b's workers to a different API endpoint
    export AIRFLOW__TEAM_B___EDGE__API_URL=https://team-b-api.example.com/edge_worker/v1/rpcapi

.. warning::

    **Security limitation:** Multi-team in the EdgeExecutor provides **logical
    isolation only**. Worker management CLI commands (maintenance, shutdown, remove,
    etc.) operate without team distinction — any administrator can manage any
    worker regardless of its team. Treat multi-team as an organizational separation
    for trusted administrators, not as a security boundary. Per-team authentication
    tokens are planned for a future release.

.. _edge_executor:concurrency_slots:

Concurrency slot handling
-------------------------

Some tasks may need more resources than other tasks, to handle these use case the Edge worker supports
concurrency slot handling. The logic behind this is the same as the pool slot feature
see :doc:`apache-airflow:administration-and-deployment/pools`.
Edge worker reuses ``pool_slots`` of task_instance to keep number if task instance parameter as low as possible.
The ``pool_slots`` value works together with the ``worker_concurrency`` value which is defined during start of worker.
If a task needs more resources, the ``pool_slots`` value can be increased to reduce number of tasks running in parallel.
The value can be used to block other tasks from being executed in parallel on the same worker.
A ``pool_slots`` of 2 and a ``worker_concurrency`` of 3 means
that a worker which executes this task can only execute a job with a ``pool_slots`` of 1 in parallel.
If no ``pool_slots`` is defined for a task the default value is 1. The ``pool_slots`` value only supports
integer values.

Here is an example setting pool_slots for a task:

.. code-block:: python

    import os

    import pendulum

    from airflow import DAG
    from airflow.decorators import task
    from airflow.example_dags.libs.helper import print_stuff
    from airflow.settings import AIRFLOW_HOME

    with DAG(
        dag_id="example_edge_pool_slots",
        schedule=None,
        start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
        catchup=False,
        tags=["example"],
    ) as dag:

        @task(executor="EdgeExecutor", pool_slots=2)
        def task_with_template():
            print_stuff()

        task_with_template()


.. _edge_executor:execute_callback:

Support ExecuteCallback in Worker
---------------------------------

In addition to executing tasks, the EdgeExecutor can also dispatch executor-level
callbacks (``ExecuteCallback`` workloads, e.g. deadline callbacks) to edge workers.
When the scheduler hands an ``ExecuteCallback`` to ``EdgeExecutor.queue_workload``,
it is enqueued into the same job queue (``EdgeJobModel``) that is used for task
workloads, so an edge worker picks it up alongside regular tasks without any
additional configuration.

Callback jobs share the ``EdgeJobModel`` table with task jobs. They are
distinguished by reserved values in the identifier columns:

- ``dag_id`` is set to the constant tag ``ExecuteCallback``.
- ``task_id`` is set to the callback key (the callback ID).
- ``run_id`` is set to ``ExecuteCallback-<callback_key>``.
- ``map_index`` is fixed to ``-1`` and ``try_number`` to ``0``.

When the worker fetches such a job through the worker API, the command payload is
deserialized back into an ``ExecuteCallback`` workload (instead of an
``ExecuteTask``) based on these identifiers. On Airflow 3.3+, the worker executes
both task and callback workloads via ``BaseExecutor.run_workload`` (or the
``airflow.sdk.execution_time.execute_workload`` entrypoint when using the subprocess path).

.. note::

    This feature is only active on Airflow 3.3 or newer. On earlier Airflow versions
    the EdgeExecutor only handles ``ExecuteTask`` workloads and any
    ``ExecuteCallback`` will be rejected with a ``TypeError``.

Current Limitations Edge Executor
---------------------------------

- Some known limitations

  - Log upload will only work if you use a single api-server / webserver instance or they need to share one log file
    volume. Logs are uploaded in chunks and are transferred via API. If you use multiple api-servers / webservers w/o
    a shared log volume the logs will be scattered across the api-server / webserver instances and if you view the
    logs on UI you will only see fractions of the logs.
  - Performance: No extensive performance assessment and scaling tests have been made. The edge executor package is
    optimized for stability. This will be incrementally improved in future releases. Setups have reported stable
    operation with ~80 workers until now. Note that executed tasks require more api-server / webserver API capacity.
  - Multi-team isolation is logical only — all teams share a single authentication secret. A worker
    administrator could change the team name and access another team's jobs. See
    :ref:`edge_executor:multi_team` for details and planned improvements.
