Build large-scale Xarray datasets for geospatial computing with Union

We demonstrate the use of Flyte and the Union.ai platform, as a workflow orchestration system for constructing large-scale Xarray mosaics using GDAL’s new GTI driver. We highlight the benefits of this orchestration system, including its integration with Kubernetes Dask, which seamlessly connects with Xarray and Zarr. Additionally, we explore the advantages of GDAL’s GTI driver and key configuration considerations. We present various parallelization strategies, offering insights into their effectiveness across different scenarios. Using GLAD’s ARD dataset—pre-tiled to EPSG:4326 globally and temporally stackable—we showcase how ingest and mosaic workflows can be combined to create an end-to-end Xarray dataset, ready for scientific computation.

About the Speaker

Len has extensive experience building remote sensing, machine learning applications with tools such as Flyte, Xarray, Dask, and Tensorflow. He holds a MsC degree in Statistics from the University of Colorado, Denver.

About Union.ai

Union is an AI platform that simplifies ML infrastructure so you can develop, deploy, and innovate faster.

Write your code in Python, collaborate across departments, and enjoy full reproducibility and auditability. Union lets you focus on what matters.

💬 Join our AI and MLOps Slack Community: slack.flyte.org

⭐ Check out Flyte on GitHub: github.com/flyteorg/flyte

🤝 Get access to $30 in GPU credits and a hosted Flyte platform signup.union.ai