Getting Started: Scale¶
A step-by-step guide to building, deploying, and iterating on Flyte tasks and workflows
Scale by Deploying Your Workflows into the Cloud¶
Make sure you have Docker and Docker Daemon is running. Also some of our users noted, that if connected to VPN may cause problems downloading the image.
Flytectlis a commandline interface for Flyte.
brew install flyteorg/homebrew-tap/flytectl
Upgrade existing installation using the following command:
brew upgrade flytectl
curl -s https://raw.githubusercontent.com/lyft/flytectl/master/install.sh | bash export PATH=$(pwd)/bin:$PATH
Test if Flytectl is installed correctly (your Flytectl version should be >= 0.1.34.) using the following command:
Flyte can be deployed locally using a single Docker container — we refer to this as the
flyte-sandboxenvironment. You can also run this getting started against a hosted or pre-provisioned environment. Refer to Deployment section to learn how to deploy a flyte cluster.
Want to dive under the hood into flyte-sandbox, refer to Sandbox Overview.
Here ‘.’ represents current directory and assuming you have changed into
myflyteapp— the git-cloned directory you created.
flytectl sandbox start --source .
NOTE: Output of the command will contain a recommendation to export an environment variable called FLYTECTL_CONFIG. please export as follows or copy paste
NOTE if having trouble with starting the sandbox refer to Troubleshooting Guide.
# COMING SOON! flytectl init
Build & Deploy Your Application to the Cluster¶
Flyte uses Docker containers to package the workflows and tasks and sends them to the remote Flyte cluster. Thus, there is a
Dockerfilealready included in the cloned repo. You can build the Docker container and push the built image to a registry.
flyte-sandboxruns locally in a Docker container, you need not push the Docker image. You can combine the build and push step by simply building the image inside the Flyte-sandbox container. This can be done using the following command:
flytectl sandbox exec -- docker build . --tag "myapp:v1"
Why are we not pushing the Docker image? Want to understand the details — refer to Sandbox Overview
Recommended: Use the bundled ./docker_build_and_tag.sh. It will automatically build the local Dockerfile, name it and tag it with the current git-SHA. This helps in achieving GitOps style workflows.
If you are using a remote Flyte cluster, then you need to build your container and push it to a registry that is accessible by the Flyte Kubernetes cluster.
docker build . --tag <registry/repo:version> docker push <registry/repo:version>
flytekit-python-templateships with a helper docker build script which make it possible to build and image, tag it correctly and optionally use the git-SHA as the version. We recommend using such a script to track versions more effectively and use a CI/CD pipeline to deploy your code.
./docker_build_and_tag.sh -r <registry> -a <repo> [-v <version>]
Next, package the workflow using the
pyflytecli bundled with Flytekit and upload it to the Flyte backend. Note that the image is the same as the one built in the previous step.
pyflyte --pkgs myapp.workflows package --image <registry/repo:version>
Upload this package to the Flyte backend. We refer to this as
registration. The version here
v1does not have to match the version used in the commands above. It’s generally recommended to match the versions to make it easier to track.
Note that we are simply using an existing project
flytesnacksand an existing domain
developmentto register the workflows and tasks. It is possible to create your own project and configure domains. Refer to Control Plane to understand projects and domains.
flytectl register files --project flytesnacks --domain development --archive flyte-package.tgz --version v1
Finally, visualize the registered workflow.
flytectl get workflows --project flytesnacks --domain development myapp.workflows.example.my_wf --version v1 -o doturl
Execute on Flyte Cluster¶
Use FlyteConsole to launch an execution and keep tabs on the window!
Alternatively, you can execute using the command line.
Launch and monitor from CLI using Flytectl. More details can be found here.
Generate an execution spec file.
flytectl get launchplan --project flytesnacks --domain development myapp.workflows.example.my_wf --latest --execFile exec_spec.yaml
Create an execution using the exec spec file.
flytectl create execution --project flytesnacks --domain development --execFile exec_spec.yaml
Monitor the execution by providing the execution name from the
flytectl get execution --project flytesnacks --domain development <execname>
🎉 You have successfully packaged your workflow and tasks and pushed them to a Flyte cluster. Let’s learn how to iterate.