Flytekit Configuration Ecosystem#

Where can configuration come from?#

  • Command line arguments. This is the ideal location for settings to go. (See pyflyte package --help for example.)

  • Environment variables. Users can specify these at compile time, but when your task is run, Flyte Propeller will also set configuration to ensure correct interaction with the platform.

  • A config file - an INI style configuration file. By default, flytekit will look for a file in two places 1. First, a file named flytekit.config in the Python interpreter’s starting directory 2. A file in ~/.flyte/config in the home directory as detected by Python.

How is configuration used?#

Configuration usage can roughly be bucketed into the following areas,

  • Compile-time settings - things like the default image, where to look for Flyte code, etc.

  • Platform settings - Where to find the Flyte backend (Admin DNS, whether to use SSL)

  • Run time (registration) settings - these are things like the K8s service account to use, a specific S3/GCS bucket to write off-loaded data (dataframes and files) to, notifications, labels & annotations, etc.

  • Data access settings - Is there a custom S3 endpoint in use? Backoff/retry behavior for accessing S3/GCS, key and password, etc.

  • Other settings - Statsd configuration, which is a run-time applicable setting but is not necessarily relevant to the Flyte platform.

Configuration Objects#

The following objects are encapsulated in a parent object called Config.


This is the parent configuration object and holds all the underlying configuration object types.

Compilation (Serialization) Time Settings#


Image is a structured wrapper for task container images used in object serialization.


ImageConfig holds available images which can be used at registration time.


These settings are provided while serializing a workflow and task, before registration.

Execution Time Settings#


This object contains the settings to talk to a Flyte backend (the DNS location of your Admin server basically).


Configuration for sending statsd.


Configuration for secrets.


S3 specific configuration


Any GCS specific configuration.


Any data storage specific configuration.


This is the parent configuration object and holds all the underlying configuration object types.