Getting Started: Scale

A step-by-step guide to building, deploying, and iterating on Flyte tasks and workflows

Scale by Deploying Your Workflows into the Cloud

Prerequisites

Make sure you have Docker and Docker Daemon is running. Also some of our users noted, that if connected to VPN may cause problems downloading the image.

Install flytectl

  1. Install flytectl. Flytectl is a commandline interface for Flyte.

    brew install flyteorg/homebrew-tap/flytectl
    

    Upgrade existing installation using the following command:

    flytectl upgrade
    
    curl -sL https://ctl.flyte.org/install | sudo bash -s -- -b /usr/local/bin # You can change path from /usr/local/bin to any file system path
    export PATH=$(pwd)/bin:$PATH # Only required if user used different path then /usr/local/bin
    

    Upgrade existing installation using the following command:

    flytectl upgrade
    

    Test if Flytectl is installed correctly (your Flytectl version should be >= 0.1.34.) using the following command:

    flytectl version
    
  2. Flyte can be deployed locally using a single Docker container — we refer to this as the flyte-sandbox environment. You can also run this getting started against a hosted or pre-provisioned environment. Refer to Deployment section to learn how to deploy a flyte cluster.

    Tip

    Want to dive under the hood into flyte-sandbox, refer to Sandbox Overview.

    Here ‘.’ represents current directory and assuming you have changed into myflyteapp — the git-cloned directory you created.

    flytectl sandbox start --source .
    

    Setup flytectl sandbox config

    flytectl config init
    

    NOTE if having trouble with starting the sandbox refer to Troubleshooting Guide.

    flytectl config init --host={FLYTEADMIN_URL} --storage
    

Build & Deploy Your Application to the Cluster

  1. Flyte uses Docker containers to package the workflows and tasks and sends them to the remote Flyte cluster. Thus, there is a Dockerfile already included in the cloned repo. You can build the Docker container and push the built image to a registry.

    Since flyte-sandbox runs locally in a Docker container, you need not push the Docker image. You can combine the build and push step by simply building the image inside the Flyte-sandbox container. This can be done using the following command:

    flytectl sandbox exec -- docker build . --tag "myapp:v1"
    

    Tip

    1. Why are we not pushing the Docker image? Want to understand the details — refer to Sandbox Overview

    2. Recommended: Use the bundled ./docker_build_and_tag.sh. It will automatically build the local Dockerfile, name it and tag it with the current git-SHA. This helps in achieving GitOps style workflows.

    If you are using a remote Flyte cluster, then you need to build your container and push it to a registry that is accessible by the Flyte Kubernetes cluster.

    docker build . --tag <registry/repo:version>
    docker push <registry/repo:version>
    

    OR flytekit-python-template ships with a helper docker build script which make it possible to build and image, tag it correctly and optionally use the git-SHA as the version. We recommend using such a script to track versions more effectively and use a CI/CD pipeline to deploy your code.

    ./docker_build_and_tag.sh -r <registry> -a <repo> [-v <version>]
    
  2. Next, package the workflow using the pyflyte cli bundled with Flytekit and upload it to the Flyte backend. Note that the image is the same as the one built in the previous step.

    pyflyte --pkgs myapp.workflows package --image <registry/repo:version>
    
  3. Upload this package to the Flyte backend. We refer to this as registration. The version here v1 does not have to match the version used in the commands above. It’s generally recommended to match the versions to make it easier to track.

    Note

    Note that we are simply using an existing project flytesnacks and an existing domain development to register the workflows and tasks. It is possible to create your own project and configure domains. Refer to Control Plane to understand projects and domains.

    flytectl register files --project flytesnacks --domain development --archive flyte-package.tgz --version v1
    
  4. Finally, visualize the registered workflow.

    flytectl get workflows --project flytesnacks --domain development myapp.workflows.example.my_wf --version v1 -o doturl
    

Execute on Flyte Cluster

Use FlyteConsole to launch an execution and keep tabs on the window!

A quick visual tour for launching a workflow and checking the outputs when they're done.

Alternatively, you can execute using the command line.

Launch and monitor from CLI using Flytectl. More details can be found here.

  1. Generate an execution spec file.

    flytectl get launchplan --project flytesnacks --domain development myapp.workflows.example.my_wf --latest --execFile exec_spec.yaml
    
  2. Create an execution using the exec spec file.

    flytectl create execution --project flytesnacks --domain development --execFile exec_spec.yaml
    
  3. Monitor the execution by providing the execution name from the create execution command.

    flytectl get execution --project flytesnacks --domain development <execname>
    

Recap

🎉 You have successfully packaged your workflow and tasks and pushed them to a Flyte cluster. Let’s learn how to iterate.