Running a workflow locally

You can run a workflow locally in two ways:

• In a local Python environment: To develop and test your code quickly without the overhead of setting up a local Flyte cluster, you can run your workflow in your local Python environment.

• In a local Flyte cluster: To test your code in a more production-like setting, you can run your workflow in a local cluster, such as the demo Flyte cluster.

Running a workflow in a local Python environment

Prerequisites

• Install development tools

• Create a Flyte project

Steps

1. On the command line, navigate to the workflows directory of your Flyte project:

cd my_project/workflows

2. Run the workflow with pyflyte run:

pyflyte run example.py wf

Note

While you can run the example file like a Python script with python example.py, we recommend using pyflyte run instead. To run the file like a Python script, you would have to add a main module conditional at the end of the script:

if __name__ == "__main__":
    print(wf())

Your code would become even more verbose if you wanted to pass arguments to the workflow:

if __name__ == "__main__":
    from argparse import ArgumentParser

    parser = ArgumentParser()
    parser.add_argument("--name", type=str)

    args = parser.parse_args()
    print(wf(name=args.name))

Running a workflow in a local Flyte cluster

Prerequisites

1. Install development tools

If you have not already done so, follow the steps in “Installing development tools” to install Python, Flytekit, and optionally, conda.

2. Create a Flyte project

If you have not already done so, follow the steps in “Creating a Flyte project” to create a Flyte project.

3. Install Docker

Follow the steps in the Docker installation guide to install Docker.

Flyte supports any OCI-compatible container technology (like Podman, LXD, and Containerd), but for the purpose of this documentation, flytectl uses Docker to start a local Kubernetes cluster that you can interact with on your machine.

4. Install flytectl

You must install flytectl to start and configure a local Flyte cluster, as well as register workflows to a local or remote Flyte cluster.

macOS

To use Homebrew, on the command line, run the following:

brew install flyteorg/homebrew-tap/flytectl

To use curl, on the command line, run the following:

curl -sL https://ctl.flyte.org/install | sudo bash -s -- -b /usr/local/bin

To download manually, see the flytectl releases.

Linux

To use curl, on the command line, run the following:

Warning

jq is a dependency of this script.

curl -sL https://ctl.flyte.org/install | sudo bash -s -- -b /usr/local/bin

To download manually, see the flytectl releases.

Windows

To use curl, in a Linux shell (such as WSL), on the command line, run the following:

curl -sL https://ctl.flyte.org/install | sudo bash -s -- -b /usr/local/bin

To download manually, see the flytectl releases.

Steps

1. Export the FLYTECTL_CONFIG environment variable in your shell:

export FLYTECTL_CONFIG=~/.flyte/config-sandbox.yaml

2. Start the Docker daemon.

3. Start the demo cluster:

flytectl demo start

4. Create a project on the demo cluster to correspond to your local Flyte project:

flytectl create project \
      --id "my-project" \
      --labels "my-label=my-project" \
      --description "My Flyte project" \
      --name "My project"

3. On the command line, navigate to the workflows directory of your Flyte project:

cd my_project/workflows

4. Run the workflow on the Flyte cluster with pyflyte run using the --remote flag and additional parameters for the project name and domain. In this example, you can also optionally pass a name parameter to the workflow:

pyflyte run --remote -p my-project -d development example.py wf --name Ada

You should see a URL to the workflow execution on your demo Flyte cluster, where <execution_name> is a unique identifier for the workflow execution:

Go to http://localhost:30080/console/projects/my-project/domains/development/executions/<execution_name> to see execution in the console.

Inspecting a workflow run in the FlyteConsole web interface

You can inspect the results of a workflow run by navigating to the URL produced by pyflyte run for the workflow execution. You should see FlyteConsole, the web interface used to manage Flyte entities such as tasks, workflows, and executions. The default execution view shows the list of tasks executing in sequential order.

Task panel

Clicking on a single task will open a panel that shows task logs, inputs, outputs, and metadata.

Graph view

The Graph view shows the execution graph of the workflow, providing visual information about the topology of the graph and the state of each node as the workflow progresses.