Integrations

This section describes how to enable integrations with AWS and GCP cloud resources as well as how to use integrations that connect to other services, such as Apache Airflow, ChatGPT, dbt, Databricks, NVidia DGX, Snowflake, and more.

AWS cloud resources integrations

• Enabling AWS S3
• Enabling AWS ECR
• Enabling AWS Secrets Manager

GCP cloud resources integrations

• Enabling BigQuery
• Enabling Google Artifact Registry
• Enabling Google Secret Manager

Additional integrations

A variety of agents and plugins enable you to integrate Flyte with additional external services. If you don’t see the integration you need, you can create your own.

AI integrations

• OpenAI agents Send prompts to ChatGPT and receive responses.

Database integrations

• DuckDB plugin Run analytical queries using DuckDB.
• Google BigQuery agent Query Google BigQuery tables from your workflows.
• SQL plugin Execute SQL queries as tasks.

Data pipeline integrations

• Apache Airflow agent Run Apache Airflow jobs in your workflows.
• Databricks agent Run Databricks jobs in your workflows.
• dbt plugin Run and test your dbt pipelines in Flyte.
• MemVerge Memory Machine Cloud agent Execute tasks using the MemVerge Memory Machine Cloud agent
• NVIDIA DGX agent Run jobs on the NVIDIA DGX platform.
• Snowflake agent Run Snowflake jobs in your workflow.

Data science tooling integrations

• Modin plugin Scale pandas workflows with modin.
• Papermill plugin Execute Jupyter Notebooks with papermill.

Data validation integrations

• Dolt plugin Version your SQL database with dolt.
• Great Expectations plugin Validate data with great_expectations.
• Pandera plugin Validate pandas dataframes with pandera.
• whylogs plugin The open standard for data logging.

MLOps integrations

• mlflow plugin The open standard for model tracking.
• ONNX plugin Convert ML models to ONNX models seamlessly.

Misc

• File sensor agent Detect files appearing in your local or remote filesystem.