Running in a local cluster

Running in a local Kubernetes cluster

Ultimately you will be running your workflows in a Kubernetes cluster in Flyte. But it can be handy to try out a workflow in a cluster on your local machine.

First, ensure that you have Docker (or a similar OCI-compliant container engine) installed locally and that the daemon is running.

Then start the demo cluster using flytectl:

$ flytectl demo start

Configuration

When flytectl starts the cluster in your local container engine it also writes configuration information to the directory ~/.flyte/.

Most importantly, it creates the file ~/.flyte/config-sandbox.yaml. This file holds (among other things) the location of the Kubernetes cluster to which we will be deploying the workflow:

admin:
  endpoint: localhost:30080
  authType: Pkce
  insecure: true
console:
  endpoint: http://localhost:30080
logger:
  show-source: true
  level: 0

Right now this file indicates that the target cluster is your local Docker instance (localhost:30080), but later we will change it to point to your Flyte cluster.

Later invocations of flytectl or pyflyte will need to know the location of the target cluster. This can be provided in two ways:

  1. Explicitly passing the location of the config file on the command line
    • flytectl --config ~/.flyte/config-sandbox.yaml <command>
    • pyflyte --config ~/.flyte/config-sandbox.yaml <command>
  2. Setting the environment variable FLYTECTL_CONFIGto the location of the config file:
    • export FLYTECTL_CONFIG=~/.flyte/config-sandbox.yaml

In this guide, we assume that you have set the FLYTECTL_CONFIG environment variable in your shell to the location of the configuration file.

Start the workflow

Now you can run your workflow in the local cluster simply by adding the --remote flag to your pyflyte command:

$ pyflyte run --remote \
          workflows/example.py \
          training_workflow \
          --hyperparameters '{"C": 0.1}'

The output supplies a URL to your workflow execution in the UI.

Inspect the results

Navigate to the URL produced by pyflyte run to see your workflow in the Flyte UI.

Local cluster with default image

$ pyflyte run --remote my_file.py my_workflow

Where pyflyte is configured to point to the local cluster started with flytectl demo start.

  • Task code runs in the environment of the default image in your local cluster.
  • Python code is dynamically overlaid into the container at runtime.
  • Only supports Python code whose dependencies are installed in the default image (see here).
  • Includes a local S3.
  • Supports some plugins but not all.
  • Single workflow runs immediately.
  • Workflow is registered to a default project.
  • Useful for demos.

Local cluster with custom image

$ pyflyte run --remote \
              --image my_cr.io/my_org/my_image:latest \
              my_file.py \
              my_workflow

Where pyflyte is configured to point to the local cluster started with flytectl demo start.

  • Task code runs in the environment of your custom image (my_cr.io/my_org/my_image:latest) in your local cluster.
  • Python code is dynamically overlaid into the container at runtime
  • Supports any Python dependencies you wish, since you have full control of the image.
  • Includes a local S3.
  • Supports some plugins but not all.
  • Single workflow runs immediately.
  • Workflow is registered to a default project.
  • Useful for advanced testing during the development cycle.