Airflow agent

Apache Airflow is a widely used open source platform for managing workflows with a robust ecosystem. Flyte provides an Airflow plugin that allows you to run Airflow tasks as Flyte tasks. This allows you to use the Airflow plugin ecosystem in conjunction with Flyte’s powerful task execution and orchestration capabilities.

Note

The Airflow agent does not support all Airflow operators. We have tested many, but if you run into issues, please file a bug report.

Installation

To install the plugin, run the following command:

pip install flytekitplugins-airflow

This plugin has two components:

• Airflow compiler: This component compiles Airflow tasks to Flyte tasks, so Airflow tasks can be directly used inside the Flyte workflow.

• Airflow agent: This component allows you to execute Airflow tasks either locally or on a Flyte cluster.

Example usage

Note

You don’t need an Airflow cluster to run Airflow tasks, since flytekit will automatically compile Airflow tasks to Flyte tasks and execute them on the Airflow agent.

For a usage example, see Airflow agent example usage.

Local testing

Airflow doesn’t support local execution natively. However, Flyte compiles Airflow tasks to Flyte tasks, which enables you to test Airflow tasks locally in flytekit’s local execution mode.

Note

In some cases, you will need to store credentials in your local environment when testing locally.

Flyte deployment configuration

Note

If you are using a managed deployment of Flyte, you will need to contact your deployment administrator to configure agents in your deployment.

To enable the Airflow agent in your Flyte deployment, see the Airflow agent deployment guide.