Pyflyte CLI#
pyflyte#
Entrypoint for all the user commands.
pyflyte [OPTIONS] COMMAND [ARGS]...
Options
- --verbose#
Show verbose messages and exception traces
- -k, --pkgs <pkgs>#
Dot-delineated python packages to operate on. Multiple may be specified (can use commas, or specify the switch multiple times. Please note that this option will override the option specified in the configuration file, or environment variable
- -c, --config <config>#
Path to config file for use within container
backfill#
The backfill command generates and registers a new workflow based on the input launchplan to run an automated backfill. The workflow can be managed using the Flyte UI and can be canceled, relaunched, and recovered.
launchplan
refers to the name of the Launchplanlaunchplan_version
is optional and should be a valid version for a Launchplan version.
pyflyte backfill [OPTIONS] LAUNCHPLAN [LAUNCHPLAN_VERSION]
Options
- -p, --project <project>#
Project to register and run this workflow in
- -d, --domain <domain>#
Domain to register and run this workflow in
- -v, --version <version>#
Version for the registered workflow. If not specified it is auto-derived using the start and end date
- -n, --execution-name <execution_name>#
Create a named execution for the backfill. This can prevent launching multiple executions.
- --dry-run#
Just generate the workflow - do not register or execute
- Default
False
- --parallel, --serial#
All backfill steps can be run in parallel (limited by max-parallelism), if using
--parallel.
Else all steps will be run sequentially [--serial
].- Default
False
- --execute, --do-not-execute#
Generate the workflow and register, do not execute
- Default
True
- --from-date <from_date>#
Date from which the backfill should begin. Start date is inclusive.
- --to-date <to_date>#
Date to which the backfill should run_until. End date is inclusive
- --backfill-window <backfill_window>#
Timedelta for number of days, minutes hours after the from-date or before the to-date to compute the backfills between. This is needed with from-date / to-date. Optional if both from-date and to-date are provided
Arguments
- LAUNCHPLAN#
Required argument
- LAUNCHPLAN_VERSION#
Optional argument
init#
Create flyte-ready projects.
pyflyte init [OPTIONS] PROJECT_NAME
Options
- --template <template>#
cookiecutter template folder name to be used in the repo - https://github.com/flyteorg/flytekit-python-template.git
Arguments
- PROJECT_NAME#
Required argument
local-cache#
Interact with the local cache.
pyflyte local-cache [OPTIONS] COMMAND [ARGS]...
clear#
This command will remove all stored objects from local cache.
pyflyte local-cache clear [OPTIONS]
package#
This command produces a Flyte backend registrable package of all entities in Flyte. For tasks, one pb file is produced for each task, representing one TaskTemplate object. For workflows, one pb file is produced for each workflow, representing a WorkflowClosure object. The closure object contains the WorkflowTemplate, along with the relevant tasks for that workflow. This serialization step will set the name of the tasks to the fully qualified name of the task function.
pyflyte package [OPTIONS]
Options
- -i, --image <image_config>#
A fully qualified tag for an docker image, for example
somedocker.com/myimage:someversion123
. This is a multi-option and can be of the form--image xyz.io/docker:latest --image my_image=xyz.io/docker2:latest
. Note, thename=image_uri
. The name is optional, if not provided the image will be used as the default image. All the names have to be unique, and thus there can only be one--image
option with no name.
- -s, --source <source>#
Local filesystem path to the root of the package.
- -o, --output <output>#
Filesystem path to the source of the Python package (from where the pkgs will start).
- --fast#
This flag enables fast packaging, that allows no container build deploys of flyte workflows and tasks. Note this needs additional configuration, refer to the docs.
- -f, --force#
This flag enables overriding existing output files. If not specified, package will exit with an error, when an output file already exists.
- -p, --python-interpreter <python_interpreter>#
Use this to override the default location of the in-container python interpreter that will be used by Flyte to load your program. This is usually where you install flytekit within the container.
- -d, --in-container-source-path <in_container_source_path>#
Filesystem path to where the code is copied into within the Dockerfile. look for
COPY . /root
like command.
- --deref-symlinks#
Enables symlink dereferencing when packaging files in fast registration
register#
This command is similar to package
but instead of producing a zip file, all your Flyte entities are compiled,
and then sent to the backend specified by your config file. Think of this as combining the pyflyte package
and the flytectl register
steps in one command. This is why you see switches you’d normally use with flytectl
like service account here.
Note: This command runs “fast” register by default.
This means that a zip is created from the detected root of the packages given and uploaded. Just like with
pyflyte run
, tasks registered from this command will download and unzip that code package before running.
Note: This command only works on regular Python packages, not namespace packages. When determining
the root of your project, it finds the first folder that does not have a __init__.py
file.
pyflyte register [OPTIONS] [PACKAGE_OR_MODULE]...
Options
- -p, --project <project>#
Project to register and run this workflow in
- -d, --domain <domain>#
Domain to register and run this workflow in
- -i, --image <image_config>#
A fully qualified tag for an docker image, for example
somedocker.com/myimage:someversion123
. This is a multi-option and can be of the form--image xyz.io/docker:latest --image my_image=xyz.io/docker2:latest
. Note, thename=image_uri
. The name is optional, if not provided the image will be used as the default image. All the names have to be unique, and thus there can only be one--image
option with no name.
- -o, --output <output>#
Directory to write the output zip file containing the protobuf definitions
- -D, --destination-dir <destination_dir>#
Directory inside the image where the tar file containing the code will be copied to
- --service-account <service_account>#
Service account used when creating launch plans
- --raw-data-prefix <raw_data_prefix>#
Raw output data prefix when creating launch plans, where offloaded data will be stored
- -v, --version <version>#
Version the package or module is registered with
- --deref-symlinks#
Enables symlink dereferencing when packaging files in fast registration
- --non-fast#
Skip zipping and uploading the package
- --dry-run#
Execute registration in dry-run mode. Skips actual registration to remote
Arguments
- PACKAGE_OR_MODULE#
Optional argument(s)
run#
This command can execute either a workflow or a task from the command line, for fully self-contained scripts.
Tasks and workflows cannot be imported from other files currently. Please use pyflyte package
or
pyflyte register
to handle those and then launch from the Flyte UI or flytectl
.
Note: This command only works on regular Python packages, not namespace packages. When determining
the root of your project, it finds the first folder that does not have an __init__.py
file.
pyflyte run [OPTIONS] COMMAND [ARGS]...
Options
- --remote#
Whether to register and run the workflow on a Flyte deployment
- -p, --project <project>#
Project to register and run this workflow in
- -d, --domain <domain>#
Domain to register and run this workflow in
- --name <name>#
Name to assign to this execution
- --destination-dir <destination_dir>#
Directory inside the image where the tar file containing the code will be copied to
- -i, --image <image_config>#
Image used to register and run.
- --service-account <service_account>#
Service account used when executing this workflow
- --wait-execution#
Whether to wait for the execution to finish
- --dump-snippet#
Whether to dump a code snippet instructing how to load the workflow execution using flyteremote
conf.py#
Run a [workflow|task] in a file using script mode
pyflyte run conf.py [OPTIONS] COMMAND [ARGS]...
serialize#
This command produces protobufs for tasks and templates. For tasks, one pb file is produced for each task, representing one TaskTemplate object. For workflows, one pb file is produced for each workflow, representing a WorkflowClosure object. The closure object contains the WorkflowTemplate, along with the relevant tasks for that workflow. In lieu of Admin, this serialization step will set the URN of the tasks to the fully qualified name of the task function.
pyflyte serialize [OPTIONS] COMMAND [ARGS]...
Options
- --image <image>#
Text tag, for example
somedocker.com/myimage:someversion123
- --local-source-root <local_source_root>#
Root dir for Python code containing workflow definitions to operate on when not the current working directory. Optional when running
pyflyte serialize
in out-of-container-mode and your code lies outside of your working directory.
- --in-container-config-path <in_container_config_path>#
This is where the configuration for your task lives inside the container. The reason it needs to be a separate option is because this pyflyte utility cannot know where the Dockerfile writes the config file to. Required for running
pyflyte serialize
in out-of-container-mode
- --in-container-virtualenv-root <in_container_virtualenv_root>#
DEPRECATED: This flag is ignored! This is the root of the flytekit virtual env in your container. The reason it needs to be a separate option is because this pyflyte utility cannot know where flytekit is installed inside your container. Required for running pyflyte serialize in out of container mode when your container installs the flytekit virtualenv outside of the default /opt/venv
fast#
pyflyte serialize fast [OPTIONS] COMMAND [ARGS]...
workflows#
pyflyte serialize fast workflows [OPTIONS]
Options
- --deref-symlinks#
Enables symlink dereferencing when packaging files in fast registration
- -f, --folder <folder>#
workflows#
pyflyte serialize workflows [OPTIONS]
Options
- -f, --folder <folder>#