Flyte Scheduler Configuration¶

admin
cloudevents
cluster_resources
clusters
database
domains
externalevents
flyteadmin
logger
namespace_mapping
notifications
qualityofservice
queues
registration
remotedata
scheduler
storage
task_resources
task_type_whitelist

Section: admin¶

endpoint (config.URL)¶

For admin types, specify where the uri of the service is located.

Default Value:

""

insecure (bool)¶

Use insecure connection.

Default Value:

"false"

insecureSkipVerify (bool)¶

InsecureSkipVerify controls whether a client verifies the server’s certificate chain and host name. Caution : shouldn’t be use for production usecases’

Default Value:

"false"

caCertFilePath (string)¶

Use specified certificate file to verify the admin server peer.

Default Value:

""

maxBackoffDelay (config.Duration)¶

Max delay for grpc backoff

Default Value:

8s

perRetryTimeout (config.Duration)¶

gRPC per retry timeout

Default Value:

15s

maxRetries (int)¶

Max number of gRPC retries

Default Value:

"4"

authType (uint8)¶

Type of OAuth2 flow used for communicating with admin.ClientSecret,Pkce,ExternalCommand are valid values

Default Value:

ClientSecret

tokenRefreshWindow (config.Duration)¶

Max duration between token refresh attempt and token expiry.

Default Value:

0s

useAuth (bool)¶

Deprecated: Auth will be enabled/disabled based on admin’s dynamically discovered information.

Default Value:

"false"

clientId (string)¶

Client ID

Default Value:

flytepropeller

clientSecretLocation (string)¶

File containing the client secret

Default Value:

/etc/secrets/client_secret

scopes ([]string)¶

List of scopes to request

Default Value:

[]

authorizationServerUrl (string)¶

This is the URL to your IdP’s authorization server. It’ll default to Endpoint

Default Value:

""

tokenUrl (string)¶

OPTIONAL: Your IdP’s token endpoint. It’ll be discovered from flyte admin’s OAuth Metadata endpoint if not provided.

Default Value:

""

authorizationHeader (string)¶

Custom metadata header to pass JWT

Default Value:

""

pkceConfig (pkce.Config)¶

Config for Pkce authentication flow.

Default Value:

refreshTime: 5m0s
timeout: 15s

command ([]string)¶

Command for external authentication token generation

Default Value:

[]

config.Duration¶

Duration (int64)¶

Default Value:

8s

config.URL¶

URL (url.URL)¶

Default Value:

ForceQuery: false
Fragment: ""
Host: ""
Opaque: ""
Path: ""
RawFragment: ""
RawPath: ""
RawQuery: ""
Scheme: ""
User: null

url.URL¶

Scheme (string)¶

Default Value:

""

Opaque (string)¶

Default Value:

""

User (url.Userinfo)¶

Default Value:

null

Host (string)¶

Default Value:

""

Path (string)¶

Default Value:

""

RawPath (string)¶

Default Value:

""

ForceQuery (bool)¶

Default Value:

"false"

RawQuery (string)¶

Default Value:

""

Fragment (string)¶

Default Value:

""

RawFragment (string)¶

Default Value:

""

pkce.Config¶

timeout (config.Duration)¶

Default Value:

15s

refreshTime (config.Duration)¶

Default Value:

5m0s

Section: cloudevents¶

enable (bool)¶

Default Value:

"false"

type (string)¶

Default Value:

local

aws (interfaces.AWSConfig)¶

Default Value:

region: ""

gcp (interfaces.GCPConfig)¶

Default Value:

projectId: ""

kafka (interfaces.KafkaConfig)¶

Default Value:

Version: {}
brokers: null

eventsPublisher (interfaces.EventsPublisherConfig)¶

Default Value:

eventTypes: null
topicName: ""

reconnectAttempts (int)¶

Default Value:

"0"

reconnectDelaySeconds (int)¶

Default Value:

"0"

interfaces.AWSConfig¶

region (string)¶

Default Value:

""

interfaces.EventsPublisherConfig¶

topicName (string)¶

Default Value:

""

eventTypes ([]string)¶

Default Value:

null

interfaces.GCPConfig¶

projectId (string)¶

Default Value:

""

interfaces.KafkaConfig¶

Version (sarama.KafkaVersion)¶

Default Value:

{}

brokers ([]string)¶

Default Value:

null

sarama.KafkaVersion¶

version (array)¶

Default Value:

'[0 0 0 0]'

Section: cluster_resources¶

templatePath (string)¶

Default Value:

""

templateData (map[string]interfaces.DataSource)¶

Default Value:

{}

refreshInterval (config.Duration)¶

Default Value:

1m0s

customData (map[string]map[string]interfaces.DataSource)¶

Default Value:

{}

standaloneDeployment (bool)¶

Whether the cluster resource sync is running in a standalone deployment and should call flyteadmin service endpoints

Default Value:

"false"

Section: clusters¶

clusterConfigs ([]interfaces.ClusterConfig)¶

Default Value:

null

labelClusterMap (map[string][]interfaces.ClusterEntity)¶

Default Value:

null

Section: database¶

host (string)¶

Default Value:

""

port (int)¶

Default Value:

"0"

dbname (string)¶

Default Value:

""

username (string)¶

Default Value:

""

password (string)¶

Default Value:

""

passwordPath (string)¶

Default Value:

""

options (string)¶

Default Value:

""

debug (bool)¶

Default Value:

"false"

enableForeignKeyConstraintWhenMigrating (bool)¶

Whether to enable gorm foreign keys when migrating the db

Default Value:

"false"

maxIdleConnections (int)¶

maxIdleConnections sets the maximum number of connections in the idle connection pool.

Default Value:

"10"

maxOpenConnections (int)¶

maxOpenConnections sets the maximum number of open connections to the database.

Default Value:

"1000"

connMaxLifeTime (config.Duration)¶

sets the maximum amount of time a connection may be reused

Default Value:

1h0m0s

postgres (database.PostgresConfig)¶

Default Value:

dbname: postgres
debug: false
host: postgres
options: sslmode=disable
password: ""
passwordPath: ""
port: 5432
username: postgres

sqlite (database.SQLiteConfig)¶

Default Value:

file: ""

database.PostgresConfig¶

host (string)¶

The host name of the database server

Default Value:

postgres

port (int)¶

The port name of the database server

Default Value:

"5432"

dbname (string)¶

The database name

Default Value:

postgres

username (string)¶

The database user who is connecting to the server.

Default Value:

postgres

password (string)¶

The database password.

Default Value:

""

passwordPath (string)¶

Points to the file containing the database password.

Default Value:

""

options (string)¶

See http://gorm.io/docs/connecting_to_the_database.html for available options passed, in addition to the above.

Default Value:

sslmode=disable

debug (bool)¶

Whether or not to start the database connection with debug mode enabled.

Default Value:

"false"

database.SQLiteConfig¶

file (string)¶

The path to the file (existing or new) where the DB should be created / stored. If existing, then this will be re-used, else a new will be created

Default Value:

""

Section: domains¶

id (string)¶

Default Value:

development

name (string)¶

Default Value:

development

Section: externalevents¶

enable (bool)¶

Default Value:

"false"

type (string)¶

Default Value:

local

aws (interfaces.AWSConfig)¶

Default Value:

region: ""

gcp (interfaces.GCPConfig)¶

Default Value:

projectId: ""

eventsPublisher (interfaces.EventsPublisherConfig)¶

Default Value:

eventTypes: null
topicName: ""

reconnectAttempts (int)¶

Default Value:

"0"

reconnectDelaySeconds (int)¶

Default Value:

"0"

Section: flyteadmin¶

roleNameKey (string)¶

Default Value:

""

metricsScope (string)¶

Default Value:

'flyte:'

profilerPort (int)¶

Default Value:

"10254"

metadataStoragePrefix ([]string)¶

Default Value:

- metadata
- admin

eventVersion (int)¶

Default Value:

"2"

asyncEventsBufferSize (int)¶

Default Value:

"100"

maxParallelism (int32)¶

Default Value:

"25"

labels (map[string]string)¶

Default Value:

null

annotations (map[string]string)¶

Default Value:

null

interruptible (bool)¶

Default Value:

"false"

assumableIamRole (string)¶

Default Value:

""

k8sServiceAccount (string)¶

Default Value:

default

outputLocationPrefix (string)¶

Default Value:

""

Section: logger¶

show-source (bool)¶

Includes source code location in logs.

Default Value:

"false"

mute (bool)¶

Mutes all logs regardless of severity. Intended for benchmarks/tests only.

Default Value:

"false"

level (int)¶

Sets the minimum logging level.

Default Value:

"3"

formatter (logger.FormatterConfig)¶

Sets logging format.

Default Value:

type: json

logger.FormatterConfig¶

type (string)¶

Sets logging format type.

Default Value:

json

Section: namespace_mapping¶

mapping (string)¶

Default Value:

""

template (string)¶

Default Value:

'{{ project }}-{{ domain }}'

templateData (map[string]interfaces.DataSource)¶

Default Value:

null

Section: notifications¶

type (string)¶

Default Value:

local

region (string)¶

Default Value:

""

aws (interfaces.AWSConfig)¶

Default Value:

region: ""

gcp (interfaces.GCPConfig)¶

Default Value:

projectId: ""

publisher (interfaces.NotificationsPublisherConfig)¶

Default Value:

topicName: ""

processor (interfaces.NotificationsProcessorConfig)¶

Default Value:

accountId: ""
queueName: ""

emailer (interfaces.NotificationsEmailerConfig)¶

Default Value:

body: ""
emailServerConfig:
  apiKeyEnvVar: ""
  apiKeyFilePath: ""
  serviceName: ""
sender: ""
subject: ""

reconnectAttempts (int)¶

Default Value:

"0"

reconnectDelaySeconds (int)¶

Default Value:

"0"

interfaces.NotificationsEmailerConfig¶

emailServerConfig (interfaces.EmailServerConfig)¶

Default Value:

apiKeyEnvVar: ""
apiKeyFilePath: ""
serviceName: ""

subject (string)¶

Default Value:

""

sender (string)¶

Default Value:

""

body (string)¶

Default Value:

""

interfaces.EmailServerConfig¶

serviceName (string)¶

Default Value:

""

apiKeyEnvVar (string)¶

Default Value:

""

apiKeyFilePath (string)¶

Default Value:

""

interfaces.NotificationsProcessorConfig¶

queueName (string)¶

Default Value:

""

accountId (string)¶

Default Value:

""

interfaces.NotificationsPublisherConfig¶

topicName (string)¶

Default Value:

""

Section: qualityofservice¶

tierExecutionValues (map[string]interfaces.QualityOfServiceSpec)¶

Default Value:

{}

defaultTiers (map[string]string)¶

Default Value:

{}

Section: queues¶

executionQueues (interfaces.ExecutionQueues)¶

Default Value:

[]

workflowConfigs (interfaces.WorkflowConfigs)¶

Default Value:

[]

Section: registration¶

maxWorkflowNodes (int)¶

Default Value:

"100"

maxLabelEntries (int)¶

Default Value:

"0"

maxAnnotationEntries (int)¶

Default Value:

"0"

workflowSizeLimit (string)¶

Default Value:

""

Section: remotedata¶

scheme (string)¶

Default Value:

none

region (string)¶

Default Value:

""

signedUrls (interfaces.SignedURL)¶

Default Value:

durationMinutes: 0
enabled: false
signingPrincipal: ""

maxSizeInBytes (int64)¶

Default Value:

"2097152"

inlineEventDataPolicy (int)¶

Specifies how inline execution event data should be saved in the backend

Default Value:

Offload

interfaces.SignedURL¶

enabled (bool)¶

Whether signed urls should even be returned with GetExecutionData, GetNodeExecutionData and GetTaskExecutionData response objects.

Default Value:

"false"

durationMinutes (int)¶

Default Value:

"0"

signingPrincipal (string)¶

Default Value:

""

Section: scheduler¶

profilerPort (config.Port)¶

Default Value:

eventScheduler (interfaces.EventSchedulerConfig)¶

Default Value:

aws: null
local: {}
region: ""
scheduleNamePrefix: ""
scheduleRole: ""
scheme: local
targetName: ""

workflowExecutor (interfaces.WorkflowExecutorConfig)¶

Default Value:

accountId: ""
aws: null
local:
  adminRateLimit:
    burst: 10
    tps: 100
region: ""
scheduleQueueName: ""
scheme: local

reconnectAttempts (int)¶

Default Value:

"0"

reconnectDelaySeconds (int)¶

Default Value:

"0"

config.Port¶

port (int)¶

Default Value:

"10254"

interfaces.EventSchedulerConfig¶

scheme (string)¶

Default Value:

local

region (string)¶

Default Value:

""

scheduleRole (string)¶

Default Value:

""

targetName (string)¶

Default Value:

""

scheduleNamePrefix (string)¶

Default Value:

""

aws (interfaces.AWSSchedulerConfig)¶

Default Value:

null

local (interfaces.FlyteSchedulerConfig)¶

Default Value:

{}

interfaces.FlyteSchedulerConfig¶

interfaces.WorkflowExecutorConfig¶

scheme (string)¶

Default Value:

local

region (string)¶

Default Value:

""

scheduleQueueName (string)¶

Default Value:

""

accountId (string)¶

Default Value:

""

aws (interfaces.AWSWorkflowExecutorConfig)¶

Default Value:

null

local (interfaces.FlyteWorkflowExecutorConfig)¶

Default Value:

adminRateLimit:
  burst: 10
  tps: 100

interfaces.FlyteWorkflowExecutorConfig¶

adminRateLimit (interfaces.AdminRateLimit)¶

Default Value:

burst: 10
tps: 100

interfaces.AdminRateLimit¶

tps (float64)¶

Default Value:

"100"

burst (int)¶

Default Value:

"10"

Section: storage¶

type (string)¶

Sets the type of storage to configure [s3/minio/local/mem/stow].

Default Value:

s3

connection (storage.ConnectionConfig)¶

Default Value:

access-key: ""
auth-type: iam
disable-ssl: false
endpoint: ""
region: us-east-1
secret-key: ""

stow (storage.StowConfig)¶

Storage config for stow backend.

Default Value:

{}

container (string)¶

Initial container (in s3 a bucket) to create -if it doesn’t exist-.’

Default Value:

""

enable-multicontainer (bool)¶

If this is true, then the container argument is overlooked and redundant. This config will automatically open new connections to new containers/buckets as they are encountered

Default Value:

"false"

cache (storage.CachingConfig)¶

Default Value:

max_size_mbs: 0
target_gc_percent: 0

limits (storage.LimitsConfig)¶

Sets limits for stores.

Default Value:

maxDownloadMBs: 2

defaultHttpClient (storage.HTTPClientConfig)¶

Sets the default http client config.

Default Value:

headers: null
timeout: 0s

signedUrl (storage.SignedURLConfig)¶

Sets config for SignedURL.

Default Value:

{}

storage.CachingConfig¶

max_size_mbs (int)¶

Maximum size of the cache where the Blob store data is cached in-memory. If not specified or set to 0, cache is not used

Default Value:

"0"

target_gc_percent (int)¶

Sets the garbage collection target percentage.

Default Value:

"0"

storage.ConnectionConfig¶

endpoint (config.URL)¶

URL for storage client to connect to.

Default Value:

""

auth-type (string)¶

Auth Type to use [iam,accesskey].

Default Value:

iam

access-key (string)¶

Access key to use. Only required when authtype is set to accesskey.

Default Value:

""

secret-key (string)¶

Secret to use when accesskey is set.

Default Value:

""

region (string)¶

Region to connect to.

Default Value:

us-east-1

disable-ssl (bool)¶

Disables SSL connection. Should only be used for development.

Default Value:

"false"

storage.HTTPClientConfig¶

headers (map[string][]string)¶

Default Value:

null

timeout (config.Duration)¶

Sets time out on the http client.

Default Value:

0s

storage.LimitsConfig¶

maxDownloadMBs (int64)¶

Maximum allowed download size (in MBs) per call.

Default Value:

"2"

storage.SignedURLConfig¶

stowConfigOverride (map[string]string)¶

Default Value:

null

storage.StowConfig¶

kind (string)¶

Kind of Stow backend to use. Refer to github/flyteorg/stow

Default Value:

""

config (map[string]string)¶

Configuration for stow backend. Refer to github/flyteorg/stow

Default Value:

{}

Section: task_resources¶

defaults (interfaces.TaskResourceSet)¶

Default Value:

cpu: "2"
ephemeralStorage: "0"
gpu: "0"
memory: 200Mi
storage: "0"

limits (interfaces.TaskResourceSet)¶

Default Value:

cpu: "2"
ephemeralStorage: "0"
gpu: "1"
memory: 1Gi
storage: "0"

interfaces.TaskResourceSet¶

cpu (resource.Quantity)¶

Default Value:

"2"

gpu (resource.Quantity)¶

Default Value:

"0"

memory (resource.Quantity)¶

Default Value:

200Mi

storage (resource.Quantity)¶

Default Value:

"0"

ephemeralStorage (resource.Quantity)¶

Default Value:

"0"

resource.Quantity¶

i (resource.int64Amount)¶

Default Value:

{}

d (resource.infDecAmount)¶

Default Value:

<nil>

s (string)¶

Default Value:

"2"

Format (string)¶

Default Value:

DecimalSI

resource.infDecAmount¶

Dec (inf.Dec)¶

Default Value:

null

resource.int64Amount¶

value (int64)¶

Default Value:

"2"

scale (int32)¶

Default Value:

"0"