Configuration#
Flytekit Configuration Sources#
There are multiple ways to configure flytekit settings:
Command Line Arguments: This is the recommended way of setting configuration values for many cases. For example, see pyflyte package command.
Python Config Object: A Config
object can by used directly, e.g. when
initializing a FlyteRemote
object. See here for examples on
how to specify a Config
object.
Environment Variables: Users can specify these at compile time, but when your task is run, Flyte Propeller will
also set configuration to ensure correct interaction with the platform. The environment variables must be specified
with the format FLYTE_{SECTION}_{OPTION}
, all in upper case. For example, to specify the
PlatformConfig.endpoint
setting, the environment variable would
be FLYTE_PLATFORM_URL
.
Note
Environment variables won’t work for image configuration, which need to be specified with the pyflyte package –image … option or in a configuration file.
YAML Format Configuration File: A configuration file that contains settings for both
flytectl and flytekit
. This is the recommended configuration
file format. Invoke the flytectl config init command to create a boilerplate
~/.flyte/config.yaml
file, and flytectl --help
to learn about all of the configuration yaml options.
INI Format Configuration File: A configuration file for flytekit
. By default, flytekit
will look for a
file in two places:
First, a file named
flytekit.config
in the Python interpreter’s working directory.A file in
~/.flyte/config
in the home directory as detected by Python.
Warning
The INI format configuration is considered a legacy configuration format. We recommend using the yaml format instead if you’re using a configuration file.
How is configuration used?#
Configuration usage can roughly be bucketed into the following areas,
Compile-time settings: these are settings like the default image and named images, where to look for Flyte code, etc.
Platform settings: Where to find the Flyte backend (Admin DNS, whether to use SSL)
Registration Run-time settings: these are things like the K8s service account to use, a specific S3/GCS bucket to write off-loaded data (dataframes and files) to, notifications, labels & annotations, etc.
Data access settings: Is there a custom S3 endpoint in use? Backoff/retry behavior for accessing S3/GCS, key and password, etc.
Other settings - Statsd configuration, which is a run-time applicable setting but is not necessarily relevant to the Flyte platform.
Configuration Objects#
The following objects are encapsulated in a parent object called Config
.
This is the parent configuration object and holds all the underlying configuration object types. |
Serialization Time Settings#
These are serialization/compile-time settings that are used when using commands like pyflyte package or pyflyte register. These configuration settings are typically passed in as flags to the above CLI commands.
The image configurations are typically either passed in via an –image flag,
or can be specified in the yaml
or ini
configuration files (see examples above).
Image is a structured wrapper for task container images used in object serialization. |
|
We recommend you to use ImageConfig.auto(img_name=None) to create an ImageConfig. |
|
These settings are provided while serializing a workflow and task, before registration. |
|
This object hold information about settings necessary to serialize an object so that it can be fast-registered. |
Execution Time Settings#
Users typically shouldn’t be concerned with these configurations, as they are typically set by FlytePropeller or FlyteAdmin. The configurations below are useful for authenticating to a Flyte backend, configuring data access credentials, secrets, and statsd metrics.
This object contains the settings to talk to a Flyte backend (the DNS location of your Admin server basically). |
|
Configuration for sending statsd. |
|
Configuration for secrets. |
|
S3 specific configuration |
|
Any GCS specific configuration. |
|
Any data storage specific configuration. |