Integrations

This section describes how to enable integrations that connect to other services, such as Apache Airflow, ChatGPT, dbt, Databricks, NVidia DGX, Snowflake, and more.

Additional integrations

A variety of agents and plugins enable you to integrate Flyte with additional external services. If you don’t see the integration you need, you can create your own.

AI integrations

• OpenAI agents Send prompts to ChatGPT and receive responses.

Database integrations

• DuckDB plugin Run analytical queries using DuckDB.
• Google BigQuery agent Query Google BigQuery tables from your workflows.
• SQL plugin Execute SQL queries as tasks.

Data pipeline integrations

• Apache Airflow agent Run Apache Airflow jobs in your workflows.
• Databricks agent Run Databricks jobs in your workflows.
• dbt plugin Run and test your dbt pipelines in Flyte.
• MemVerge Memory Machine Cloud agent Execute tasks using the MemVerge Memory Machine Cloud agent
• NVIDIA DGX agent Run jobs on the NVIDIA DGX platform.
• Snowflake agent Run Snowflake jobs in your workflow.

Data science tooling integrations

• Modin plugin Scale pandas workflows with modin.
• Papermill plugin Execute Jupyter Notebooks with papermill.

Data validation integrations

• Dolt plugin Version your SQL database with dolt.
• Great Expectations plugin Validate data with great_expectations.
• Pandera plugin Validate pandas dataframes with pandera.
• whylogs plugin The open standard for data logging.

MLOps integrations

• mlflow plugin The open standard for model tracking.
• ONNX plugin Convert ML models to ONNX models seamlessly.

Misc

• File sensor agent Detect files appearing in your local or remote filesystem.