Configuration#
Flytekit Configuration Ecosystem#
Where can configuration come from?#
Command line arguments. This is the ideal location for settings to go. (See
pyflyte package --help
for example.)Environment variables. Users can specify these at compile time, but when your task is run, Flyte Propeller will also set configuration to ensure correct interaction with the platform.
A config file - an INI style configuration file. By default, flytekit will look for a file in two places 1. First, a file named
flytekit.config
in the Python interpreter’s starting directory 2. A file in~/.flyte/config
in the home directory as detected by Python.
How is configuration used?#
Configuration usage can roughly be bucketed into the following areas,
Compile-time settings - things like the default image, where to look for Flyte code, etc.
Platform settings - Where to find the Flyte backend (Admin DNS, whether to use SSL)
Run time (registration) settings - these are things like the K8s service account to use, a specific S3/GCS bucket to write off-loaded data (dataframes and files) to, notifications, labels & annotations, etc.
Data access settings - Is there a custom S3 endpoint in use? Backoff/retry behavior for accessing S3/GCS, key and password, etc.
Other settings - Statsd configuration, which is a run-time applicable setting but is not necessarily relevant to the Flyte platform.
Configuration Objects#
The following objects are encapsulated in a parent object called Config
.
This is the parent configuration object and holds all the underlying configuration object types. |
Compilation (Serialization) Time Settings#
Image is a structured wrapper for task container images used in object serialization. |
|
ImageConfig holds available images which can be used at registration time. |
|
These settings are provided while serializing a workflow and task, before registration. |
Execution Time Settings#
This object contains the settings to talk to a Flyte backend (the DNS location of your Admin server basically). |
|
Configuration for sending statsd. |
|
Configuration for secrets. |
|
S3 specific configuration |
|
Any GCS specific configuration. |
|
Any data storage specific configuration. |
|
This is the parent configuration object and holds all the underlying configuration object types. |