Flyte Scheduler Configuration

Section: admin

endpoint (config.URL)

For admin types, specify where the uri of the service is located.

Default Value:

""

insecure (bool)

Use insecure connection.

Default Value:

"false"

insecureSkipVerify (bool)

InsecureSkipVerify controls whether a client verifies the server’s certificate chain and host name. Caution : shouldn’t be use for production usecases’

Default Value:

"false"

caCertFilePath (string)

Use specified certificate file to verify the admin server peer.

Default Value:

""

maxBackoffDelay (config.Duration)

Max delay for grpc backoff

Default Value:

8s

perRetryTimeout (config.Duration)

gRPC per retry timeout

Default Value:

15s

maxRetries (int)

Max number of gRPC retries

Default Value:

"4"

authType (uint8)

Type of OAuth2 flow used for communicating with admin.ClientSecret,Pkce,ExternalCommand are valid values

Default Value:

ClientSecret

tokenRefreshWindow (config.Duration)

Max duration between token refresh attempt and token expiry.

Default Value:

0s

useAuth (bool)

Deprecated: Auth will be enabled/disabled based on admin’s dynamically discovered information.

Default Value:

"false"

clientId (string)

Client ID

Default Value:

flytepropeller

clientSecretLocation (string)

File containing the client secret

Default Value:

/etc/secrets/client_secret

scopes ([]string)

List of scopes to request

Default Value:

[]

authorizationServerUrl (string)

This is the URL to your IdP’s authorization server. It’ll default to Endpoint

Default Value:

""

tokenUrl (string)

OPTIONAL: Your IdP’s token endpoint. It’ll be discovered from flyte admin’s OAuth Metadata endpoint if not provided.

Default Value:

""

authorizationHeader (string)

Custom metadata header to pass JWT

Default Value:

""

pkceConfig (pkce.Config)

Config for Pkce authentication flow.

Default Value:

refreshTime: 5m0s
timeout: 15s

command ([]string)

Command for external authentication token generation

Default Value:

[]

config.Duration

Duration (int64)

Default Value:

8s

config.URL

URL (url.URL)

Default Value:

ForceQuery: false
Fragment: ""
Host: ""
Opaque: ""
Path: ""
RawFragment: ""
RawPath: ""
RawQuery: ""
Scheme: ""
User: null

url.URL

Scheme (string)

Default Value:

""
Opaque (string)

Default Value:

""
User (url.Userinfo)

Default Value:

null
Host (string)

Default Value:

""
Path (string)

Default Value:

""
RawPath (string)

Default Value:

""
ForceQuery (bool)

Default Value:

"false"
RawQuery (string)

Default Value:

""
Fragment (string)

Default Value:

""
RawFragment (string)

Default Value:

""

pkce.Config

timeout (config.Duration)

Default Value:

15s
refreshTime (config.Duration)

Default Value:

5m0s

Section: cloudevents

enable (bool)

Default Value:

"false"

type (string)

Default Value:

local

aws (interfaces.AWSConfig)

Default Value:

region: ""

gcp (interfaces.GCPConfig)

Default Value:

projectId: ""

kafka (interfaces.KafkaConfig)

Default Value:

Version: {}
brokers: null

eventsPublisher (interfaces.EventsPublisherConfig)

Default Value:

eventTypes: null
topicName: ""

reconnectAttempts (int)

Default Value:

"0"

reconnectDelaySeconds (int)

Default Value:

"0"

interfaces.AWSConfig

region (string)

Default Value:

""

interfaces.EventsPublisherConfig

topicName (string)

Default Value:

""
eventTypes ([]string)

Default Value:

null

interfaces.GCPConfig

projectId (string)

Default Value:

""

interfaces.KafkaConfig

Version (sarama.KafkaVersion)

Default Value:

{}
brokers ([]string)

Default Value:

null

sarama.KafkaVersion

version (array)

Default Value:

'[0 0 0 0]'

Section: cluster_resources

templatePath (string)

Default Value:

""

templateData (map[string]interfaces.DataSource)

Default Value:

{}

refreshInterval (config.Duration)

Default Value:

1m0s

customData (map[string]map[string]interfaces.DataSource)

Default Value:

{}

standaloneDeployment (bool)

Whether the cluster resource sync is running in a standalone deployment and should call flyteadmin service endpoints

Default Value:

"false"

Section: clusters

clusterConfigs ([]interfaces.ClusterConfig)

Default Value:

null

labelClusterMap (map[string][]interfaces.ClusterEntity)

Default Value:

null

Section: database

host (string)

Default Value:

""

port (int)

Default Value:

"0"

dbname (string)

Default Value:

""

username (string)

Default Value:

""

password (string)

Default Value:

""

passwordPath (string)

Default Value:

""

options (string)

Default Value:

""

debug (bool)

Default Value:

"false"

enableForeignKeyConstraintWhenMigrating (bool)

Whether to enable gorm foreign keys when migrating the db

Default Value:

"false"

maxIdleConnections (int)

maxIdleConnections sets the maximum number of connections in the idle connection pool.

Default Value:

"10"

maxOpenConnections (int)

maxOpenConnections sets the maximum number of open connections to the database.

Default Value:

"1000"

connMaxLifeTime (config.Duration)

sets the maximum amount of time a connection may be reused

Default Value:

1h0m0s

postgres (database.PostgresConfig)

Default Value:

dbname: postgres
debug: false
host: postgres
options: sslmode=disable
password: ""
passwordPath: ""
port: 5432
username: postgres

sqlite (database.SQLiteConfig)

Default Value:

file: ""

database.PostgresConfig

host (string)

The host name of the database server

Default Value:

postgres
port (int)

The port name of the database server

Default Value:

"5432"
dbname (string)

The database name

Default Value:

postgres
username (string)

The database user who is connecting to the server.

Default Value:

postgres
password (string)

The database password.

Default Value:

""
passwordPath (string)

Points to the file containing the database password.

Default Value:

""
options (string)

See http://gorm.io/docs/connecting_to_the_database.html for available options passed, in addition to the above.

Default Value:

sslmode=disable
debug (bool)

Whether or not to start the database connection with debug mode enabled.

Default Value:

"false"

database.SQLiteConfig

file (string)

The path to the file (existing or new) where the DB should be created / stored. If existing, then this will be re-used, else a new will be created

Default Value:

""

Section: domains

id (string)

Default Value:

development

name (string)

Default Value:

development

Section: externalevents

enable (bool)

Default Value:

"false"

type (string)

Default Value:

local

aws (interfaces.AWSConfig)

Default Value:

region: ""

gcp (interfaces.GCPConfig)

Default Value:

projectId: ""

eventsPublisher (interfaces.EventsPublisherConfig)

Default Value:

eventTypes: null
topicName: ""

reconnectAttempts (int)

Default Value:

"0"

reconnectDelaySeconds (int)

Default Value:

"0"

Section: flyteadmin

roleNameKey (string)

Default Value:

""

metricsScope (string)

Default Value:

'flyte:'

profilerPort (int)

Default Value:

"10254"

metadataStoragePrefix ([]string)

Default Value:

- metadata
- admin

eventVersion (int)

Default Value:

"2"

asyncEventsBufferSize (int)

Default Value:

"100"

maxParallelism (int32)

Default Value:

"25"

labels (map[string]string)

Default Value:

null

annotations (map[string]string)

Default Value:

null

interruptible (bool)

Default Value:

"false"

assumableIamRole (string)

Default Value:

""

k8sServiceAccount (string)

Default Value:

default

outputLocationPrefix (string)

Default Value:

""

Section: logger

show-source (bool)

Includes source code location in logs.

Default Value:

"false"

mute (bool)

Mutes all logs regardless of severity. Intended for benchmarks/tests only.

Default Value:

"false"

level (int)

Sets the minimum logging level.

Default Value:

"3"

formatter (logger.FormatterConfig)

Sets logging format.

Default Value:

type: json

logger.FormatterConfig

type (string)

Sets logging format type.

Default Value:

json

Section: namespace_mapping

mapping (string)

Default Value:

""

template (string)

Default Value:

'{{ project }}-{{ domain }}'

templateData (map[string]interfaces.DataSource)

Default Value:

null

Section: notifications

type (string)

Default Value:

local

region (string)

Default Value:

""

aws (interfaces.AWSConfig)

Default Value:

region: ""

gcp (interfaces.GCPConfig)

Default Value:

projectId: ""

publisher (interfaces.NotificationsPublisherConfig)

Default Value:

topicName: ""

processor (interfaces.NotificationsProcessorConfig)

Default Value:

accountId: ""
queueName: ""

emailer (interfaces.NotificationsEmailerConfig)

Default Value:

body: ""
emailServerConfig:
  apiKeyEnvVar: ""
  apiKeyFilePath: ""
  serviceName: ""
sender: ""
subject: ""

reconnectAttempts (int)

Default Value:

"0"

reconnectDelaySeconds (int)

Default Value:

"0"

interfaces.NotificationsEmailerConfig

emailServerConfig (interfaces.EmailServerConfig)

Default Value:

apiKeyEnvVar: ""
apiKeyFilePath: ""
serviceName: ""
subject (string)

Default Value:

""
sender (string)

Default Value:

""
body (string)

Default Value:

""

interfaces.EmailServerConfig

serviceName (string)

Default Value:

""
apiKeyEnvVar (string)

Default Value:

""
apiKeyFilePath (string)

Default Value:

""

interfaces.NotificationsProcessorConfig

queueName (string)

Default Value:

""
accountId (string)

Default Value:

""

interfaces.NotificationsPublisherConfig

topicName (string)

Default Value:

""

Section: qualityofservice

tierExecutionValues (map[string]interfaces.QualityOfServiceSpec)

Default Value:

{}

defaultTiers (map[string]string)

Default Value:

{}

Section: queues

executionQueues (interfaces.ExecutionQueues)

Default Value:

[]

workflowConfigs (interfaces.WorkflowConfigs)

Default Value:

[]

Section: registration

maxWorkflowNodes (int)

Default Value:

"100"

maxLabelEntries (int)

Default Value:

"0"

maxAnnotationEntries (int)

Default Value:

"0"

workflowSizeLimit (string)

Default Value:

""

Section: remotedata

scheme (string)

Default Value:

none

region (string)

Default Value:

""

signedUrls (interfaces.SignedURL)

Default Value:

durationMinutes: 0
enabled: false
signingPrincipal: ""

maxSizeInBytes (int64)

Default Value:

"2097152"

inlineEventDataPolicy (int)

Specifies how inline execution event data should be saved in the backend

Default Value:

Offload

interfaces.SignedURL

enabled (bool)

Whether signed urls should even be returned with GetExecutionData, GetNodeExecutionData and GetTaskExecutionData response objects.

Default Value:

"false"
durationMinutes (int)

Default Value:

"0"
signingPrincipal (string)

Default Value:

""

Section: scheduler

profilerPort (config.Port)

Default Value:

10254

eventScheduler (interfaces.EventSchedulerConfig)

Default Value:

aws: null
local: {}
region: ""
scheduleNamePrefix: ""
scheduleRole: ""
scheme: local
targetName: ""

workflowExecutor (interfaces.WorkflowExecutorConfig)

Default Value:

accountId: ""
aws: null
local:
  adminRateLimit:
    burst: 10
    tps: 100
region: ""
scheduleQueueName: ""
scheme: local

reconnectAttempts (int)

Default Value:

"0"

reconnectDelaySeconds (int)

Default Value:

"0"

config.Port

port (int)

Default Value:

"10254"

interfaces.EventSchedulerConfig

scheme (string)

Default Value:

local
region (string)

Default Value:

""
scheduleRole (string)

Default Value:

""
targetName (string)

Default Value:

""
scheduleNamePrefix (string)

Default Value:

""
aws (interfaces.AWSSchedulerConfig)

Default Value:

null
local (interfaces.FlyteSchedulerConfig)

Default Value:

{}

interfaces.FlyteSchedulerConfig

interfaces.WorkflowExecutorConfig

scheme (string)

Default Value:

local
region (string)

Default Value:

""
scheduleQueueName (string)

Default Value:

""
accountId (string)

Default Value:

""
aws (interfaces.AWSWorkflowExecutorConfig)

Default Value:

null
local (interfaces.FlyteWorkflowExecutorConfig)

Default Value:

adminRateLimit:
  burst: 10
  tps: 100

interfaces.FlyteWorkflowExecutorConfig

adminRateLimit (interfaces.AdminRateLimit)

Default Value:

burst: 10
tps: 100

interfaces.AdminRateLimit

tps (float64)

Default Value:

"100"
burst (int)

Default Value:

"10"

Section: storage

type (string)

Sets the type of storage to configure [s3/minio/local/mem/stow].

Default Value:

s3

connection (storage.ConnectionConfig)

Default Value:

access-key: ""
auth-type: iam
disable-ssl: false
endpoint: ""
region: us-east-1
secret-key: ""

stow (storage.StowConfig)

Storage config for stow backend.

Default Value:

{}

container (string)

Initial container (in s3 a bucket) to create -if it doesn’t exist-.’

Default Value:

""

enable-multicontainer (bool)

If this is true, then the container argument is overlooked and redundant. This config will automatically open new connections to new containers/buckets as they are encountered

Default Value:

"false"

cache (storage.CachingConfig)

Default Value:

max_size_mbs: 0
target_gc_percent: 0

limits (storage.LimitsConfig)

Sets limits for stores.

Default Value:

maxDownloadMBs: 2

defaultHttpClient (storage.HTTPClientConfig)

Sets the default http client config.

Default Value:

headers: null
timeout: 0s

signedUrl (storage.SignedURLConfig)

Sets config for SignedURL.

Default Value:

{}

storage.CachingConfig

max_size_mbs (int)

Maximum size of the cache where the Blob store data is cached in-memory. If not specified or set to 0, cache is not used

Default Value:

"0"
target_gc_percent (int)

Sets the garbage collection target percentage.

Default Value:

"0"

storage.ConnectionConfig

endpoint (config.URL)

URL for storage client to connect to.

Default Value:

""
auth-type (string)

Auth Type to use [iam,accesskey].

Default Value:

iam
access-key (string)

Access key to use. Only required when authtype is set to accesskey.

Default Value:

""
secret-key (string)

Secret to use when accesskey is set.

Default Value:

""
region (string)

Region to connect to.

Default Value:

us-east-1
disable-ssl (bool)

Disables SSL connection. Should only be used for development.

Default Value:

"false"

storage.HTTPClientConfig

headers (map[string][]string)

Default Value:

null
timeout (config.Duration)

Sets time out on the http client.

Default Value:

0s

storage.LimitsConfig

maxDownloadMBs (int64)

Maximum allowed download size (in MBs) per call.

Default Value:

"2"

storage.SignedURLConfig

stowConfigOverride (map[string]string)

Default Value:

null

storage.StowConfig

kind (string)

Kind of Stow backend to use. Refer to github/flyteorg/stow

Default Value:

""
config (map[string]string)

Configuration for stow backend. Refer to github/flyteorg/stow

Default Value:

{}

Section: task_resources

defaults (interfaces.TaskResourceSet)

Default Value:

cpu: "2"
ephemeralStorage: "0"
gpu: "0"
memory: 200Mi
storage: "0"

limits (interfaces.TaskResourceSet)

Default Value:

cpu: "2"
ephemeralStorage: "0"
gpu: "1"
memory: 1Gi
storage: "0"

interfaces.TaskResourceSet

cpu (resource.Quantity)

Default Value:

"2"
gpu (resource.Quantity)

Default Value:

"0"
memory (resource.Quantity)

Default Value:

200Mi
storage (resource.Quantity)

Default Value:

"0"
ephemeralStorage (resource.Quantity)

Default Value:

"0"

resource.Quantity

i (resource.int64Amount)

Default Value:

{}
d (resource.infDecAmount)

Default Value:

<nil>
s (string)

Default Value:

"2"
Format (string)

Default Value:

DecimalSI

resource.infDecAmount

Dec (inf.Dec)

Default Value:

null

resource.int64Amount

value (int64)

Default Value:

"2"
scale (int32)

Default Value:

"0"