Make a NotebookLM Clone with Open Weights & Models - AI Workshop
Generative AI models have matured to the point where we can use LLMs as a natural language interface to build compound AI systems, where multiple models of different modalities work together to form a single, coherent application. An example of this type of application is Google’s NotebookLM, which can generate audio podcasts based on input text documents. Compound AI systems unlock a whole suite of automation tools that were not possible before, acting like an interoperability layer between multi-modal generative models and traditional software.
In this workshop, attendees will create a minimal NotebookLM clone using open weights models like Llama3 and Parler-TTS (text-to-speech). They will build and deploy their NotebookLM clones using Union Serverless, an end-to-end platform for building AI products, and will learn the building blocks, infrastructure, and abstractions that are helpful for building compound AI systems.
Session Outline
- Part 1: Introduction to compound AI systems (10 minutes)
- What are compound AI systems?
- NotebookLM high-level architecture
- Part 2: Cloning NotebookLM with Union, Llama3 and Kokoro (30 minutes)
- Reusable container environments to create heterogeneous compute applications
- Writing and refining a podcast script with Llama3
- Generating podcast audio with a TTS model
- Part 3: Deploying the serving app (20 minutes)
- Serving a minimal UI application with Gradio on Union
- Productionize deployment with Github Actions and Union
Learning Objectives and Tools
- Use Union to orchestrate and deploy a compound AI application that uses an LLM and Text-to-speech model to create a minimal clone of NotebookLM.
- Learn how to deploy serving infrastructure for self-hosting open weights models with inference frameworks like vLLM.
- Understand how containers, strongly-typed functions, and infrastructure-as-code provide benefits like reproducibility and cost efficiency.
Prerequisites
Working knowledge of Python, orchestration frameworks, and web frameworks. Familiarity with LLM frameworks like LangChain and LlamaIndex, familiarity with machine learning architectures and concepts is recommended.
About the Speaker
Niels is a machine learning engineer and core maintainer of Flyte, an open source ML orchestration tool and author and maintainer of Pandera, a data testing tool for dataframes. He has a Masters in Public Health with a specialization in sociomedical science and public health informatics, and prior to that a background in developmental biology and immunology. His research interests include reinforcement learning, AutoML, creative machine learning, and fairness, accountability, and transparency in automated systems. He enjoys developing open source tools to make data science and machine learning practitioners more productive.
About Union.ai
Union is an AI platform that simplifies ML infrastructure so you can develop, deploy, and innovate faster.
Write your code in Python, collaborate across departments, and enjoy full reproducibility and auditability. Union lets you focus on what matters.
💬 Join our AI and MLOps Slack Community: slack.flyte.org
⭐ Check out Flyte on GitHub: https://github.com/flyteorg/flyte
🤝 Learn about everything else we’re doing at union.ai