Setting up a production project
In Flyte, your work is organized in a hierarchy with the following structure:
- Organization: Your Flyte instance, accessible at a specific URL like
flyte.my-company.com
. - Domains Within an organization there are (typically) three domains,
development
,staging
, andproduction
, used to organize your code during the development process. You can configure a custom set of domains to suit your needs during onboarding. - Projects: Orthogonal to domains, projects are used to organize your code into logical groups. You can create as many projects as you need.
A given workflow will reside in a specific project. For example, let’s say my_workflow
is a workflow in my_project
.
When you start work on my_workflow
you would typically register it in the project-domain my_project/development
.
As you work on successive iterations of the workflow you might promote my_workflow
to my_project/staging
and eventually my_project/production
.
Promotion is done simply by re-registering the workflow to the new project-domain.
Terminology
In everyday use, the term “project” is often used to refer to not just the Flyte entity that holds a set of workflows, but also to the local directory in which you are developing those workflows, and to the GitHub (or other SCM) repository that you are using to store the same workflow code.
To avoid confusion, in this guide we will stick to the following naming conventions:
- Flyte project: The entity in your Flyte instance that holds a set of workflows, as described above. Often referred to simply as a project.
- Local project: The local directory (usually the working directory of a GitHub repository) in which you are developing workflows.
Create a Flyte project
Ensure that you have
flytectl
CLI installed and the connection to your Flyte cluster
properly configured.
Now, create a new project on your Flyte cluster:
$ flytectl create project \
--id "my-project" \
--labels "my-label=my-project" \
--description "My Flyte project" \
--name "My project"
Creating a local production project directory using pyflyte init
Earlier, in the
Getting started section we used pyflyte init
to create a new local project based on the flyte-simple
.
Here, we will do the same, but use the flyte-production
template. Perform the following command:
$ pyflyte init --template union-production my-project
Directory structure
In the basic-example
directory you’ll see the following file structure:
├── LICENSE
├── README.md
├── docs
│ └── docs.md
├── pyproject.toml
├── src
│ ├── core
│ │ ├── __init__.py
│ │ └── core.py
│ ├── orchestration
│ │ ├── __init__.py
│ │ └── orchestration.py
│ ├── tasks
│ │ ├── __init__.py
│ │ └── say_hello.py
│ └── workflows
│ ├── __init__.py
│ └── hello_world.py
└── uv.lock
You can create your own conventions and file structure for your production projects, but this tempkate provides a good starting point.
However, the separate workflows
subdirectory and the contained __init__.py
file are significant.
We will discuss them when we cover the
registration process.