Configuration#

Flytekit Configuration Ecosystem#

Where can configuration come from?#

  • Command line arguments. This is the ideal location for settings to go. (See pyflyte package --help for example.)

  • Environment variables. Users can specify these at compile time, but when your task is run, Flyte Propeller will also set configuration to ensure correct interaction with the platform.

  • A config file - an INI style configuration file. By default, flytekit will look for a file in two places 1. First, a file named flytekit.config in the Python interpreter’s starting directory 2. A file in ~/.flyte/config in the home directory as detected by Python.

How is configuration used?#

Configuration usage can roughly be bucketed into the following areas,

  • Compile-time settings - things like the default image, where to look for Flyte code, etc.

  • Platform settings - Where to find the Flyte backend (Admin DNS, whether to use SSL)

  • Run time (registration) settings - these are things like the K8s service account to use, a specific S3/GCS bucket to write off-loaded data (dataframes and files) to, notifications, labels & annotations, etc.

  • Data access settings - Is there a custom S3 endpoint in use? Backoff/retry behavior for accessing S3/GCS, key and password, etc.

  • Other settings - Statsd configuration, which is a run-time applicable setting but is not necessarily relevant to the Flyte platform.

Configuration Objects#

The following objects are encapsulated in a parent object called Config.

Config

This is the parent configuration object and holds all the underlying configuration object types.

Compilation (Serialization) Time Settings#

Image

Image is a structured wrapper for task container images used in object serialization.

ImageConfig

ImageConfig holds available images which can be used at registration time.

SerializationSettings

These settings are provided while serializing a workflow and task, before registration.

Execution Time Settings#

PlatformConfig

This object contains the settings to talk to a Flyte backend (the DNS location of your Admin server basically).

StatsConfig

Configuration for sending statsd.

SecretsConfig

Configuration for secrets.

S3Config

S3 specific configuration

GCSConfig

Any GCS specific configuration.

DataConfig

Any data storage specific configuration.

Config

This is the parent configuration object and holds all the underlying configuration object types.