Setting up a production project

In Flyte, your work is organized in a hierarchy with the following structure:

  • Organization: Your Flyte instance, accessible at a specific URL like flyte.my-company.com.
  • Domains Within an organization there are (typically) three domains, development, staging, and production, used to organize your code during the development process. You can configure a custom set of domains to suit your needs during onboarding.
  • Projects: Orthogonal to domains, projects are used to organize your code into logical groups. You can create as many projects as you need.

A given workflow will reside in a specific project. For example, let’s say my_workflow is a workflow in my_project.

When you start work on my_workflow you would typically register it in the project-domain my_project/development.

As you work on successive iterations of the workflow you might promote my_workflow to my_project/staging and eventually my_project/production.

Promotion is done simply by re-registering the workflow to the new project-domain.

Terminology

In everyday use, the term “project” is often used to refer to not just the Flyte entity that holds a set of workflows, but also to the local directory in which you are developing those workflows, and to the GitHub (or other SCM) repository that you are using to store the same workflow code.

To avoid confusion, in this guide we will stick to the following naming conventions:

  • Flyte project: The entity in your Flyte instance that holds a set of workflows, as described above. Often referred to simply as a project.
  • Local project: The local directory (usually the working directory of a GitHub repository) in which you are developing workflows.

Create a Flyte project

Ensure that you have flytectl CLI installed and the connection to your Flyte cluster properly configured. Now, create a new project on your Flyte cluster:

$ flytectl create project \
      --id "my-project" \
      --labels "my-label=my-project" \
      --description "My Flyte project" \
      --name "My project"

Creating a local production project directory using pyflyte init

Earlier, in the Getting started section we used pyflyte init to create a new local project based on the flyte-simple.

Here, we will do the same, but use the flyte-production template. Perform the following command:

$ pyflyte init --template union-production my-project

Directory structure

In the basic-example directory you’ll see the following file structure:

├── LICENSE
├── README.md
├── docs
│   └── docs.md
├── pyproject.toml
├── src
│   ├── core
│   │   ├── __init__.py
│   │   └── core.py
│   ├── orchestration
│   │   ├── __init__.py
│   │   └── orchestration.py
│   ├── tasks
│   │   ├── __init__.py
│   │   └── say_hello.py
│   └── workflows
│       ├── __init__.py
│       └── hello_world.py
└── uv.lock

You can create your own conventions and file structure for your production projects, but this tempkate provides a good starting point.

However, the separate workflows subdirectory and the contained __init__.py file are significant. We will discuss them when we cover the registration process.