Iterate and Re-deploy

In this guide, you’ll learn how to iterate on and re-deploy your tasks and workflows.

Modify Code and Test Locally

Open example.py in your favorite editor.

flyte/workflows/example.py

"""
Hello World
------------
This simple workflow calls a task that returns "Hello World" and then just sets that as the final output of the workflow.
"""
import typing

# %%
# All imports at the root flytekit level are stable and we maintain backwards
# compatibility for them.
from flytekit import task, workflow


# %%
# Here we define a task called ``say_hello``. Note the @task decorator, Flyte
# uses this to understand that you intend to port this function to flyte.
# If you have normal functions in this file, they are not accessible to
# file, unless they have the @task decorator.
# You can change the signature of the task to take in an argument like this:
# def say_hello(name: str) -> str:
@task
def say_hello() -> str:
    return "hello world"


# %%
# Here we declare a workflow called ``my_wf``. Note the @workflow decorator,
# Flyte finds all workflows that you have declared by finding this decorator.
# A @workflow function, looks like a regular python function, except for some
# important differences, it is never executed by flyte-engine. It is like
# psuedo code, that is analyzed by flytekit to convert to Flyte's native
# Workflow representation. Thus the variables like return values from `tasks`
# are not real values, and trying to interact with them like regular variables
# will result in an error. For example, if a task returns a boolean, and if you
# try to test the truth value for this boolean, an error will be raised. The
# reason, is the tasks are not really executed by the function, but run remote
# and the return variables are supplied to subsequent tasks.
#
# You can treat the outputs of a task as you normally would a Python function. Assign the output to two variables
# and use them in subsequent tasks as normal. See :py:func:`flytekit.workflow`
# You can change the signature of the workflow to take in an argument like this:
# def my_wf(name: str) -> str:
@workflow
def my_wf() -> str:
    res = say_hello()
    return res


# %%
# Execute the Workflow, simply by invoking it like a function and passing in
# the necessary parameters
#
# .. note::
#
#   One thing to remember, currently we only support ``Keyword arguments``. So
#   every argument should be passed in the form ``arg=value``. Failure to do so
#   will result in an error
if __name__ == "__main__":
    print(f"Running my_wf() {my_wf()}")

Add message: str as an argument to both my_wf and say_hello functions. Then update the body of say_hello to consume that argument.

@task
def say_hello(message: str) -> str:
   return f"hello world, {message}"

@workflow
def my_wf(message: str) -> str:
   res = say_hello(message=message)
   return res

Update the simple test at the bottom of the file to pass in a message, e.g.

print(f"Running my_wf(message='what a nice day it is!') {my_wf(message='what a nice day it is!')}")

When you run this file locally, it should output hello world, what a nice day it is!.

python flyte/workflows/example.py

Expected output

Running my_wf(message='what a nice day it is!') hello world, what a nice day it is!

Quickly Re-deploy Your Application

To re-deploy this workflow to the sandbox Flyte cluster, you can repeat the steps previously covered in getting-started-build-deploy. Flyte provides a faster way to iterate on your workflows. Since you have not updated any of the dependencies in your requirements file, it is possible to push just the code to Flyte backend without re-building the entire Docker container. To do so, run the following commands.

pyflyte --pkgs flyte.workflows package --image my_flyte_project:v1 --fast --force

Note

--fast flag will take the code from your local machine and provide it for execution without having to build the container and push it. The --force flag allows overriding your previously created package.

Caution

The fast registration method can only be used if you do not modify any requirements (that is, you re-use an existing environment). But, if you add a dependency to your requirements file or env you have to follow the getting-started-build-deploy method.

The code can now be deployed using Flytectl, similar to what we’ve done previously. flytectl automatically understands that the package is for fast registration.

For this to work, a new storage block has to be added to the Flytectl configuration with appropriate permissions at runtime. The storage block configures Flytectl to write to a specific S3 / GCS bucket.

Tip

If you’re using the sandbox, this is automatically configured by Flytectl.

The dropdown below provides more information on the required configuration depending on your cloud infrastructure.

Flytectl configuration with storage block for Fast registration

Local Flyte Sandbox

Automatically configured for you by flytectl sandbox command.

admin:
   # For GRPC endpoints you might want to use dns:///flyte.myexample.com
   endpoint: dns:///localhost:30081
   insecure: true
storage:
   connection:
      access-key: minio
      auth-type: accesskey
      disable-ssl: true
      endpoint: http://localhost:30084
      region: my-region-here
      secret-key: miniostorage
   container: my-s3-bucket
   type: minio

S3 Configuration

admin:
   # For GRPC endpoints you might want to use dns:///flyte.myexample.com
   endpoint: dns:///<replace-me>
   authType: Pkce # authType: Pkce # if using authentication or just drop this.
   insecure: true # insecure: True # Set to true if the endpoint isn't accessible through TLS/SSL connection (not recommended except on local sandbox deployment)
storage:
   type: stow
   stow:
      kind: s3
      config:
         auth_type: iam
         region: <REGION> # Example: us-east-2
   container: <replace> # Example my-bucket. Flyte k8s cluster / service account for execution should have read access to this bucket

GCS Configuration

admin:
   # For GRPC endpoints you might want to use dns:///flyte.myexample.com
   endpoint: dns:///<replace-me>
   authType: Pkce # authType: Pkce # if using authentication or just drop this.
   insecure: false # insecure: True # Set to true if the endpoint isn't accessible through TLS/SSL connection (not recommended except on local sandbox deployment)
storage:
   type: stow
   stow:
      kind: google
      config:
         json: ""
         project_id: <replace-me> # replace <project-id> with the GCP project ID
         scopes: https://www.googleapis.com/auth/devstorage.read_write
   container: <replace> # Example my-bucket. Flyte k8s cluster / service account for execution should have access to this bucket

Others

For other supported storage backends like Oracle, Azure, etc., refer to the configuration structure here.

Next, you can simply register your packaged workflows with:

flytectl register files --project flytesnacks --domain development --archive flyte-package.tgz  --version v1-fast1

Execute Your Re-deployed Workflow

Finally, you can execute the updated workflow programmatically with flytectl.

To pass arguments to the workflow, update the execution spec file that we previously generated in the Deploying to the Coud step.

Generate an execution spec file. This will prompt you to overwrite and answer ‘y’ on it.

flytectl get launchplan --project flytesnacks --domain development flyte.workflows.example.my_wf --latest --execFile exec_spec.yaml

Modify the execution spec file and update the input params and save the file. Notice that the version would be changed to your latest one.

....
inputs:
   message: "what's up doc? 🐰🥕"
....
version: v1-fast1

Create an execution using the exec spec file.

flytectl create execution --project flytesnacks --domain development --execFile exec_spec.yaml

You should see an output that looks like:

execution identifier project:"flytesnacks" domain:"development" name:"<execution_name>"

Monitor the execution by providing the execution name from the create execution command.

flytectl get execution --project flytesnacks --domain development <execution_name>

Tip

Alternatively, visit the Flyte sandbox console, click launch, and provide a message as input.

Recap

In this guide, we:

1. Setup a Flyte project with pyflyte init my_flyte_project and ran your workflows locally.

2. Started a Flyte sandbox cluster and ran a Flyte workflow on a cluster.

3. Iterated on a Flyte workflow and updated the workflows on the cluster.

What’s Next?

To experience the full power of Flyte on distributed compute, take a look at the Deployment Guides.