Apache Airflow agent

Apache Airflow is a widely used open source platform for managing workflows with a robust ecosystem. Union.ai provides an Airflow plugin that allows you to run Airflow tasks as Union.ai tasks. This allows you to use the Airflow plugin ecosystem in conjunction with Union.ai’s powerful task execution and orchestration capabilities.

The Airflow agent does not support all Airflow operators. We have tested many, but if you run into issues, please file a bug report or reach out to the Union.ai team.

Installation

To install the Airflow agent, run the following command:

pip install flytekitplugins-airflow

This integration has two components:

  • Airflow compiler: This component compiles Airflow tasks to Union.ai tasks, so Airflow tasks can be directly used inside the Union.ai workflow.
  • Airflow agent: This component allows you to execute Airflow tasks either locally or on a Union.ai cluster.

Example usage

You don’t need an Airflow cluster to run Airflow tasks, since Union will automatically compile Airflow tasks to Union.ai tasks and execute them on the Airflow agent.

For a usage example, see Airflow agent example.

Local testing

Airflow doesn’t support local execution natively. However, Union.ai compiles Airflow tasks to Union.ai tasks, which enables you to test Airflow tasks locally in Union’s local execution mode.

In some cases, you will need to store credentials in your local environment when testing locally.

Union.ai cluster deployment

After you have finished testing the agent locally, contact the Union.ai team to enable it in your cluster.