Flyte Propeller Configuration

Section: admin

endpoint (config.URL)

For admin types, specify where the uri of the service is located.

Default Value:

""

insecure (bool)

Use insecure connection.

Default Value:

"false"

insecureSkipVerify (bool)

InsecureSkipVerify controls whether a client verifies the server’s certificate chain and host name. Caution : shouldn’t be use for production usecases’

Default Value:

"false"

maxBackoffDelay (config.Duration)

Max delay for grpc backoff

Default Value:

8s

perRetryTimeout (config.Duration)

gRPC per retry timeout

Default Value:

15s

maxRetries (int)

Max number of gRPC retries

Default Value:

"4"

authType (uint8)

Default Value:

ClientSecret

useAuth (bool)

Deprecated: Auth will be enabled/disabled based on admin’s dynamically discovered information.

Default Value:

"false"

clientId (string)

Client ID

Default Value:

flytepropeller

clientSecretLocation (string)

File containing the client secret

Default Value:

/etc/secrets/client_secret

scopes ([]string)

List of scopes to request

Default Value:

[]

authorizationServerUrl (string)

This is the URL to your IdP’s authorization server. It’ll default to Endpoint

Default Value:

""

tokenUrl (string)

OPTIONAL: Your IdP’s token endpoint. It’ll be discovered from flyte admin’s OAuth Metadata endpoint if not provided.

Default Value:

""

authorizationHeader (string)

Custom metadata header to pass JWT

Default Value:

""

pkceConfig (pkce.Config)

Config for Pkce authentication flow.

Default Value:

refreshTime: 5m0s
timeout: 15s

command ([]string)

Command for external authentication token generation

Default Value:

null

config.Duration

Duration (int64)

Default Value:

8s

config.URL

URL (url.URL)

Default Value:

ForceQuery: false
Fragment: ""
Host: ""
Opaque: ""
Path: ""
RawFragment: ""
RawPath: ""
RawQuery: ""
Scheme: ""
User: null

url.URL

Scheme (string)

Default Value:

""
Opaque (string)

Default Value:

""
User (url.Userinfo)

Default Value:

null
Host (string)

Default Value:

""
Path (string)

Default Value:

""
RawPath (string)

Default Value:

""
ForceQuery (bool)

Default Value:

"false"
RawQuery (string)

Default Value:

""
Fragment (string)

Default Value:

""
RawFragment (string)

Default Value:

""

pkce.Config

timeout (config.Duration)

Default Value:

15s
refreshTime (config.Duration)

Default Value:

5m0s

Section: catalog-cache

type (string)

Catalog Implementation to use

Default Value:

noop

endpoint (string)

Endpoint for catalog service

Default Value:

""

insecure (bool)

Use insecure grpc connection

Default Value:

"false"

max-cache-age (config.Duration)

Cache entries past this age will incur cache miss. 0 means cache never expires

Default Value:

0s

Section: event

type (string)

Sets the type of EventSink to configure [log/admin/file].

Default Value:

""

file-path (string)

For file types, specify where the file should be located.

Default Value:

""

rate (int64)

Max rate at which events can be recorded per second.

Default Value:

"500"

capacity (int)

The max bucket size for event recording tokens.

Default Value:

"1000"

Section: logger

show-source (bool)

Includes source code location in logs.

Default Value:

"false"

mute (bool)

Mutes all logs regardless of severity. Intended for benchmarks/tests only.

Default Value:

"false"

level (int)

Sets the minimum logging level.

Default Value:

"4"

formatter (logger.FormatterConfig)

Sets logging format.

Default Value:

type: json

logger.FormatterConfig

type (string)

Sets logging format type.

Default Value:

json

Section: plugins

enabled-plugins ([]string)

List of enabled plugins, default value is to enable all plugins.

Default Value:

- '*'

athena (athena.Config)

Default Value:

defaultCatalog: AwsDataCatalog
defaultWorkGroup: primary
resourceConstraints:
  NamespaceScopeResourceConstraint:
    Value: 50
  ProjectScopeResourceConstraint:
    Value: 100
webApi:
  caching:
    maxSystemFailures: 5
    resyncInterval: 30s
    size: 500000
    workers: 10
  readRateLimiter:
    burst: 100
    qps: 10
  resourceMeta: null
  resourceQuotas:
    default: 1000
  writeRateLimiter:
    burst: 100
    qps: 10

aws (aws.Config)

Default Value:

accountId: ""
logLevel: 0
region: us-east-2
retries: 3

catalogcache (catalog.Config)

Default Value:

reader:
  maxItems: 1000
  maxRetries: 3
  workers: 10
writer:
  maxItems: 1000
  maxRetries: 3
  workers: 10

k8s (config.K8sPluginConfig)

Default Value:

co-pilot:
  cpu: 500m
  default-input-path: /var/flyte/inputs
  default-output-path: /var/flyte/outputs
  image: cr.flyte.org/flyteorg/flytecopilot:v0.0.9
  input-vol-name: flyte-inputs
  memory: 128Mi
  name: flyte-copilot-
  output-vol-name: flyte-outputs
  start-timeout: 1m0s
  storage: ""
create-container-error-grace-period: 3m0s
default-annotations:
  cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
default-cpus: "1"
default-env-vars: null
default-env-vars-from-env: null
default-labels: null
default-memory: 1Gi
default-node-selector: null
default-tolerations: null
delete-resource-on-finalize: false
gpu-resource-name: nvidia.com/gpu
inject-finalizer: false
interruptible-node-selector: null
interruptible-node-selector-requirement: null
interruptible-tolerations: null
non-interruptible-node-selector-requirement: null
resource-tolerations: null
scheduler-name: ""

k8s-array (k8s.Config)

Default Value:

ErrorAssembler:
  maxItems: 100000
  maxRetries: 5
  workers: 10
OutputAssembler:
  maxItems: 100000
  maxRetries: 5
  workers: 10
logs:
  config:
    cloudwatch-enabled: false
    cloudwatch-log-group: ""
    cloudwatch-region: ""
    cloudwatch-template-uri: ""
    gcp-project: ""
    kubernetes-enabled: false
    kubernetes-template-uri: ""
    kubernetes-url: ""
    stackdriver-enabled: false
    stackdriver-logresourcename: ""
    stackdriver-template-uri: ""
    templates: null
maxArrayJobSize: 5000
maxErrorLength: 1000
namespaceTemplate: ""
node-selector: null
remoteClusterConfig:
  auth:
    certPath: ""
    tokenPath: ""
    type: ""
  enabled: false
  endpoint: ""
  name: ""
resourceConfig:
  limit: 0
  primaryLabel: ""
scheduler: ""
tolerations: null

logs (logs.LogConfig)

Default Value:

cloudwatch-enabled: false
cloudwatch-log-group: ""
cloudwatch-region: ""
cloudwatch-template-uri: ""
gcp-project: ""
kubernetes-enabled: false
kubernetes-template-uri: ""
kubernetes-url: ""
stackdriver-enabled: false
stackdriver-logresourcename: ""
stackdriver-template-uri: ""
templates: null

qubole (config.Config)

Default Value:

analyzeLinkPath: /v2/analyze
clusterConfigs:
- labels:
  - default
  limit: 100
  namespaceScopeQuotaProportionCap: 0.7
  primaryLabel: default
  projectScopeQuotaProportionCap: 0.7
commandApiPath: /api/v1.2/commands/
defaultClusterLabel: default
destinationClusterConfigs: []
endpoint: https://wellness.qubole.com
lruCacheSize: 2000
quboleTokenKey: FLYTE_QUBOLE_CLIENT_TOKEN
workers: 15

sagemaker (config.Config (sagemaker))

Default Value:

prebuiltAlgorithms:
- name: xgboost
  regionalConfigs:
  - region: us-east-1
    versionConfigs:
    - image: 683313688378.dkr.ecr.us-east-1.amazonaws.com/sagemaker-xgboost:0.90-2-cpu-py3
      version: "0.90"
region: us-east-1
roleAnnotationKey: ""
roleArn: default_role

snowflake (snowflake.Config)

Default Value:

defaultWarehouse: COMPUTE_WH
resourceConstraints:
  NamespaceScopeResourceConstraint:
    Value: 50
  ProjectScopeResourceConstraint:
    Value: 100
snowflakeTokenKey: FLYTE_SNOWFLAKE_CLIENT_TOKEN
webApi:
  caching:
    maxSystemFailures: 5
    resyncInterval: 30s
    size: 500000
    workers: 10
  readRateLimiter:
    burst: 100
    qps: 10
  resourceMeta: null
  resourceQuotas:
    default: 1000
  writeRateLimiter:
    burst: 100
    qps: 10

spark (spark.Config)

Default Value:

features: null
logs:
  all-user:
    cloudwatch-enabled: false
    cloudwatch-log-group: ""
    cloudwatch-region: ""
    cloudwatch-template-uri: ""
    gcp-project: ""
    kubernetes-enabled: false
    kubernetes-template-uri: ""
    kubernetes-url: ""
    stackdriver-enabled: false
    stackdriver-logresourcename: ""
    stackdriver-template-uri: ""
    templates: null
  mixed:
    cloudwatch-enabled: false
    cloudwatch-log-group: ""
    cloudwatch-region: ""
    cloudwatch-template-uri: ""
    gcp-project: ""
    kubernetes-enabled: true
    kubernetes-template-uri: http://localhost:30082/#!/log/{{ .namespace }}/{{ .podName
      }}/pod?namespace={{ .namespace }}
    kubernetes-url: ""
    stackdriver-enabled: false
    stackdriver-logresourcename: ""
    stackdriver-template-uri: ""
    templates: null
  system:
    cloudwatch-enabled: false
    cloudwatch-log-group: ""
    cloudwatch-region: ""
    cloudwatch-template-uri: ""
    gcp-project: ""
    kubernetes-enabled: false
    kubernetes-template-uri: ""
    kubernetes-url: ""
    stackdriver-enabled: false
    stackdriver-logresourcename: ""
    stackdriver-template-uri: ""
    templates: null
  user:
    cloudwatch-enabled: false
    cloudwatch-log-group: ""
    cloudwatch-region: ""
    cloudwatch-template-uri: ""
    gcp-project: ""
    kubernetes-enabled: false
    kubernetes-template-uri: ""
    kubernetes-url: ""
    stackdriver-enabled: false
    stackdriver-logresourcename: ""
    stackdriver-template-uri: ""
    templates: null
spark-config-default: null
spark-history-server-url: ""

athena.Config

webApi (webapi.PluginConfig)

Defines config for the base WebAPI plugin.

Default Value:

caching:
  maxSystemFailures: 5
  resyncInterval: 30s
  size: 500000
  workers: 10
readRateLimiter:
  burst: 100
  qps: 10
resourceMeta: null
resourceQuotas:
  default: 1000
writeRateLimiter:
  burst: 100
  qps: 10
resourceConstraints (core.ResourceConstraintsSpec)

Default Value:

NamespaceScopeResourceConstraint:
  Value: 50
ProjectScopeResourceConstraint:
  Value: 100
defaultWorkGroup (string)

Defines the default workgroup to use when running on Athena unless overwritten by the task.

Default Value:

primary
defaultCatalog (string)

Defines the default catalog to use when running on Athena unless overwritten by the task.

Default Value:

AwsDataCatalog

core.ResourceConstraintsSpec

ProjectScopeResourceConstraint (core.ResourceConstraint)

Default Value:

Value: 100
NamespaceScopeResourceConstraint (core.ResourceConstraint)

Default Value:

Value: 50

core.ResourceConstraint

Value (int64)

Default Value:

"100"

webapi.PluginConfig

resourceQuotas (webapi.ResourceQuotas)

Default Value:

default: 1000
readRateLimiter (webapi.RateLimiterConfig)

Defines rate limiter properties for read actions (e.g. retrieve status).

Default Value:

burst: 100
qps: 10
writeRateLimiter (webapi.RateLimiterConfig)

Defines rate limiter properties for write actions.

Default Value:

burst: 100
qps: 10
caching (webapi.CachingConfig)

Defines caching characteristics.

Default Value:

maxSystemFailures: 5
resyncInterval: 30s
size: 500000
workers: 10
resourceMeta (interface)

Default Value:

<nil>

webapi.CachingConfig

size (int)

Defines the maximum number of items to cache.

Default Value:

"500000"
resyncInterval (config.Duration)

Defines the sync interval.

Default Value:

30s
workers (int)

Defines the number of workers to start up to process items.

Default Value:

"10"
maxSystemFailures (int)

Defines the number of failures to fetch a task before failing the task.

Default Value:

"5"

webapi.RateLimiterConfig

qps (int)

Defines the max rate of calls per second.

Default Value:

"10"
burst (int)

Defines the maximum burst size.

Default Value:

"100"

aws.Config

region (string)

AWS Region to connect to.

Default Value:

us-east-2
accountId (string)

AWS Account Identifier.

Default Value:

""
retries (int)

Number of retries.

Default Value:

"3"
logLevel (uint64)

Default Value:

"0"

catalog.Config

reader (workqueue.Config)

Catalog reader workqueue config. Make sure the index cache must be big enough to accommodate the biggest array task allowed to run on the system.

Default Value:

maxItems: 1000
maxRetries: 3
workers: 10
writer (workqueue.Config)

Catalog writer workqueue config. Make sure the index cache must be big enough to accommodate the biggest array task allowed to run on the system.

Default Value:

maxItems: 1000
maxRetries: 3
workers: 10

workqueue.Config

workers (int)

Number of concurrent workers to start processing the queue.

Default Value:

"10"
maxRetries (int)

Maximum number of retries per item.

Default Value:

"3"
maxItems (int)

Maximum number of entries to keep in the index.

Default Value:

"1000"

config.Config

endpoint (config.URL)

Endpoint for qubole to use

Default Value:

https://wellness.qubole.com
commandApiPath (config.URL)

API Path where commands can be launched on Qubole. Should be a valid url.

Default Value:

/api/v1.2/commands/
analyzeLinkPath (config.URL)

URL path where queries can be visualized on qubole website. Should be a valid url.

Default Value:

/v2/analyze
quboleTokenKey (string)

Name of the key where to find Qubole token in the secret manager.

Default Value:

FLYTE_QUBOLE_CLIENT_TOKEN
lruCacheSize (int)

Size of the AutoRefreshCache

Default Value:

"2000"
workers (int)

Number of parallel workers to refresh the cache

Default Value:

"15"
defaultClusterLabel (string)

The default cluster label. This will be used if label is not specified on the hive job.

Default Value:

default
clusterConfigs ([]config.ClusterConfig)

Default Value:

- labels:
  - default
  limit: 100
  namespaceScopeQuotaProportionCap: 0.7
  primaryLabel: default
  projectScopeQuotaProportionCap: 0.7
destinationClusterConfigs ([]config.DestinationClusterConfig)

Default Value:

[]

config.Config (sagemaker)

roleArn (string)

The role the SageMaker plugin uses to communicate with the SageMaker service

Default Value:

default_role
region (string)

The AWS region the SageMaker plugin communicates to

Default Value:

us-east-1
roleAnnotationKey (string)

Map key to use to lookup role from task annotations.

Default Value:

""
prebuiltAlgorithms ([]config.PrebuiltAlgorithmConfig)

Default Value:

- name: xgboost
  regionalConfigs:
  - region: us-east-1
    versionConfigs:
    - image: 683313688378.dkr.ecr.us-east-1.amazonaws.com/sagemaker-xgboost:0.90-2-cpu-py3
      version: "0.90"

config.K8sPluginConfig

inject-finalizer (bool)

Instructs the plugin to inject a finalizer on startTask and remove it on task termination.

Default Value:

"false"
default-annotations (map[string]string)

Default Value:

cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
default-labels (map[string]string)

Default Value:

null
default-env-vars (map[string]string)

Default Value:

null
default-env-vars-from-env (map[string]string)

Default Value:

null
default-cpus (resource.Quantity)

Defines a default value for cpu for containers if not specified.

Default Value:

"1"
default-memory (resource.Quantity)

Defines a default value for memory for containers if not specified.

Default Value:

1Gi
default-tolerations ([]v1.Toleration)

Default Value:

null
default-node-selector (map[string]string)

Default Value:

null
default-affinity (v1.Affinity)

Default Value:

null
scheduler-name (string)

Defines scheduler name.

Default Value:

""
interruptible-tolerations ([]v1.Toleration)

Default Value:

null
interruptible-node-selector (map[string]string)

Default Value:

null
interruptible-node-selector-requirement (v1.NodeSelectorRequirement)

Default Value:

null
non-interruptible-node-selector-requirement (v1.NodeSelectorRequirement)

Default Value:

null
resource-tolerations (map[v1.ResourceName][]v1.Toleration)

Default Value:

null
co-pilot (config.FlyteCoPilotConfig)

Co-Pilot Configuration

Default Value:

cpu: 500m
default-input-path: /var/flyte/inputs
default-output-path: /var/flyte/outputs
image: cr.flyte.org/flyteorg/flytecopilot:v0.0.9
input-vol-name: flyte-inputs
memory: 128Mi
name: flyte-copilot-
output-vol-name: flyte-outputs
start-timeout: 1m0s
storage: ""
delete-resource-on-finalize (bool)

Instructs the system to delete the resource on finalize. This ensures that no resources are kept around (potentially consuming cluster resources). This, however, will cause k8s log links to expire as soon as the resource is finalized.

Default Value:

"false"
create-container-error-grace-period (config.Duration)

Default Value:

3m0s
gpu-resource-name (string)

The name of the GPU resource to use when the task resource requests GPUs.

Default Value:

nvidia.com/gpu

config.FlyteCoPilotConfig

name (string)

Flyte co-pilot sidecar container name prefix. (additional bits will be added after this)

Default Value:

flyte-copilot-
image (string)

Flyte co-pilot Docker Image FQN

Default Value:

cr.flyte.org/flyteorg/flytecopilot:v0.0.9
default-input-path (string)

Default path where the volume should be mounted

Default Value:

/var/flyte/inputs
default-output-path (string)

Default path where the volume should be mounted

Default Value:

/var/flyte/outputs
input-vol-name (string)

Name of the data volume that is created for storing inputs

Default Value:

flyte-inputs
output-vol-name (string)

Name of the data volume that is created for storing outputs

Default Value:

flyte-outputs
start-timeout (config.Duration)

Default Value:

1m0s
cpu (string)

Used to set cpu for co-pilot containers

Default Value:

500m
memory (string)

Used to set memory for co-pilot containers

Default Value:

128Mi
storage (string)

Default storage limit for individual inputs / outputs

Default Value:

""

resource.Quantity

i (resource.int64Amount)

Default Value:

{}
d (resource.infDecAmount)

Default Value:

<nil>
s (string)

Default Value:

"1"
Format (string)

Default Value:

DecimalSI

resource.infDecAmount

Dec (inf.Dec)

Default Value:

null

resource.int64Amount

value (int64)

Default Value:

"1"
scale (int32)

Default Value:

"0"

k8s.Config

scheduler (string)

Decides the scheduler to use when launching array-pods.

Default Value:

""
maxErrorLength (int)

Determines the maximum length of the error string returned for the array.

Default Value:

"1000"
maxArrayJobSize (int64)

Maximum size of array job.

Default Value:

"5000"
resourceConfig (k8s.ResourceConfig)

Default Value:

limit: 0
primaryLabel: ""
remoteClusterConfig (k8s.ClusterConfig)

Default Value:

auth:
  certPath: ""
  tokenPath: ""
  type: ""
enabled: false
endpoint: ""
name: ""
node-selector (map[string]string)

Default Value:

null
tolerations ([]v1.Toleration)

Default Value:

null
namespaceTemplate (string)

Default Value:

""
OutputAssembler (workqueue.Config)

Default Value:

maxItems: 100000
maxRetries: 5
workers: 10
ErrorAssembler (workqueue.Config)

Default Value:

maxItems: 100000
maxRetries: 5
workers: 10
logs (k8s.LogConfig)

Config for log links for k8s array jobs.

Default Value:

config:
  cloudwatch-enabled: false
  cloudwatch-log-group: ""
  cloudwatch-region: ""
  cloudwatch-template-uri: ""
  gcp-project: ""
  kubernetes-enabled: false
  kubernetes-template-uri: ""
  kubernetes-url: ""
  stackdriver-enabled: false
  stackdriver-logresourcename: ""
  stackdriver-template-uri: ""
  templates: null

k8s.ClusterConfig

name (string)

Friendly name of the remote cluster

Default Value:

""
endpoint (string)

Remote K8s cluster endpoint

Default Value:

""
auth (k8s.Auth)

Default Value:

certPath: ""
tokenPath: ""
type: ""
enabled (bool)

Boolean flag to enable or disable

Default Value:

"false"

k8s.Auth

type (string)

Authentication type

Default Value:

""
tokenPath (string)

Token path

Default Value:

""
certPath (string)

Certificate path

Default Value:

""

k8s.LogConfig

config (logs.LogConfig (config))

Defines the log config for k8s logs.

Default Value:

cloudwatch-enabled: false
cloudwatch-log-group: ""
cloudwatch-region: ""
cloudwatch-template-uri: ""
gcp-project: ""
kubernetes-enabled: false
kubernetes-template-uri: ""
kubernetes-url: ""
stackdriver-enabled: false
stackdriver-logresourcename: ""
stackdriver-template-uri: ""
templates: null

logs.LogConfig (config)

cloudwatch-enabled (bool)

Enable Cloudwatch Logging

Default Value:

"false"
cloudwatch-region (string)

AWS region in which Cloudwatch logs are stored.

Default Value:

""
cloudwatch-log-group (string)

Log group to which streams are associated.

Default Value:

""
cloudwatch-template-uri (string)

Template Uri to use when building cloudwatch log links

Default Value:

""
kubernetes-enabled (bool)

Enable Kubernetes Logging

Default Value:

"false"
kubernetes-url (string)

Console URL for Kubernetes logs

Default Value:

""
kubernetes-template-uri (string)

Template Uri to use when building kubernetes log links

Default Value:

""
stackdriver-enabled (bool)

Enable Log-links to stackdriver

Default Value:

"false"
gcp-project (string)

Name of the project in GCP

Default Value:

""
stackdriver-logresourcename (string)

Name of the logresource in stackdriver

Default Value:

""
stackdriver-template-uri (string)

Template Uri to use when building stackdriver log links

Default Value:

""
templates ([]logs.TemplateLogPluginConfig)

Default Value:

null

k8s.ResourceConfig

primaryLabel (string)

PrimaryLabel of a given service cluster

Default Value:

""
limit (int)

Resource quota (in the number of outstanding requests) for the cluster

Default Value:

"0"

logs.LogConfig

cloudwatch-enabled (bool)

Enable Cloudwatch Logging

Default Value:

"false"
cloudwatch-region (string)

AWS region in which Cloudwatch logs are stored.

Default Value:

""
cloudwatch-log-group (string)

Log group to which streams are associated.

Default Value:

""
cloudwatch-template-uri (string)

Template Uri to use when building cloudwatch log links

Default Value:

""
kubernetes-enabled (bool)

Enable Kubernetes Logging

Default Value:

"false"
kubernetes-url (string)

Console URL for Kubernetes logs

Default Value:

""
kubernetes-template-uri (string)

Template Uri to use when building kubernetes log links

Default Value:

""
stackdriver-enabled (bool)

Enable Log-links to stackdriver

Default Value:

"false"
gcp-project (string)

Name of the project in GCP

Default Value:

""
stackdriver-logresourcename (string)

Name of the logresource in stackdriver

Default Value:

""
stackdriver-template-uri (string)

Template Uri to use when building stackdriver log links

Default Value:

""
templates ([]logs.TemplateLogPluginConfig)

Default Value:

null

snowflake.Config

webApi (webapi.PluginConfig)

Defines config for the base WebAPI plugin.

Default Value:

caching:
  maxSystemFailures: 5
  resyncInterval: 30s
  size: 500000
  workers: 10
readRateLimiter:
  burst: 100
  qps: 10
resourceMeta: null
resourceQuotas:
  default: 1000
writeRateLimiter:
  burst: 100
  qps: 10
resourceConstraints (core.ResourceConstraintsSpec)

Default Value:

NamespaceScopeResourceConstraint:
  Value: 50
ProjectScopeResourceConstraint:
  Value: 100
defaultWarehouse (string)

Defines the default warehouse to use when running on Snowflake unless overwritten by the task.

Default Value:

COMPUTE_WH
snowflakeTokenKey (string)

Name of the key where to find Snowflake token in the secret manager.

Default Value:

FLYTE_SNOWFLAKE_CLIENT_TOKEN
snowflakeEndpoint (string)

Default Value:

""

spark.Config

spark-config-default (map[string]string)

Default Value:

null
spark-history-server-url (string)

URL for SparkHistory Server that each job will publish the execution history to.

Default Value:

""
features ([]spark.Feature)

Default Value:

null
logs (spark.LogConfig)

Config for log links for spark applications.

Default Value:

all-user:
  cloudwatch-enabled: false
  cloudwatch-log-group: ""
  cloudwatch-region: ""
  cloudwatch-template-uri: ""
  gcp-project: ""
  kubernetes-enabled: false
  kubernetes-template-uri: ""
  kubernetes-url: ""
  stackdriver-enabled: false
  stackdriver-logresourcename: ""
  stackdriver-template-uri: ""
  templates: null
mixed:
  cloudwatch-enabled: false
  cloudwatch-log-group: ""
  cloudwatch-region: ""
  cloudwatch-template-uri: ""
  gcp-project: ""
  kubernetes-enabled: true
  kubernetes-template-uri: http://localhost:30082/#!/log/{{ .namespace }}/{{ .podName
    }}/pod?namespace={{ .namespace }}
  kubernetes-url: ""
  stackdriver-enabled: false
  stackdriver-logresourcename: ""
  stackdriver-template-uri: ""
  templates: null
system:
  cloudwatch-enabled: false
  cloudwatch-log-group: ""
  cloudwatch-region: ""
  cloudwatch-template-uri: ""
  gcp-project: ""
  kubernetes-enabled: false
  kubernetes-template-uri: ""
  kubernetes-url: ""
  stackdriver-enabled: false
  stackdriver-logresourcename: ""
  stackdriver-template-uri: ""
  templates: null
user:
  cloudwatch-enabled: false
  cloudwatch-log-group: ""
  cloudwatch-region: ""
  cloudwatch-template-uri: ""
  gcp-project: ""
  kubernetes-enabled: false
  kubernetes-template-uri: ""
  kubernetes-url: ""
  stackdriver-enabled: false
  stackdriver-logresourcename: ""
  stackdriver-template-uri: ""
  templates: null

spark.LogConfig

mixed (logs.LogConfig)

Defines the log config that’s not split into user/system.

Default Value:

cloudwatch-enabled: false
cloudwatch-log-group: ""
cloudwatch-region: ""
cloudwatch-template-uri: ""
gcp-project: ""
kubernetes-enabled: true
kubernetes-template-uri: http://localhost:30082/#!/log/{{ .namespace }}/{{ .podName
  }}/pod?namespace={{ .namespace }}
kubernetes-url: ""
stackdriver-enabled: false
stackdriver-logresourcename: ""
stackdriver-template-uri: ""
templates: null
user (logs.LogConfig)

Defines the log config for user logs.

Default Value:

cloudwatch-enabled: false
cloudwatch-log-group: ""
cloudwatch-region: ""
cloudwatch-template-uri: ""
gcp-project: ""
kubernetes-enabled: false
kubernetes-template-uri: ""
kubernetes-url: ""
stackdriver-enabled: false
stackdriver-logresourcename: ""
stackdriver-template-uri: ""
templates: null
system (logs.LogConfig)

Defines the log config for system logs.

Default Value:

cloudwatch-enabled: false
cloudwatch-log-group: ""
cloudwatch-region: ""
cloudwatch-template-uri: ""
gcp-project: ""
kubernetes-enabled: false
kubernetes-template-uri: ""
kubernetes-url: ""
stackdriver-enabled: false
stackdriver-logresourcename: ""
stackdriver-template-uri: ""
templates: null
all-user (logs.LogConfig)

All user logs across driver and executors.

Default Value:

cloudwatch-enabled: false
cloudwatch-log-group: ""
cloudwatch-region: ""
cloudwatch-template-uri: ""
gcp-project: ""
kubernetes-enabled: false
kubernetes-template-uri: ""
kubernetes-url: ""
stackdriver-enabled: false
stackdriver-logresourcename: ""
stackdriver-template-uri: ""
templates: null

Section: propeller

kube-config (string)

Path to kubernetes client config file.

Default Value:

""

master (string)

Default Value:

""

workers (int)

Number of threads to process workflows

Default Value:

"20"

workflow-reeval-duration (config.Duration)

Frequency of re-evaluating workflows

Default Value:

10s

downstream-eval-duration (config.Duration)

Frequency of re-evaluating downstream tasks

Default Value:

30s

limit-namespace (string)

Namespaces to watch for this propeller

Default Value:

all

prof-port (config.Port)

Profiler port

Default Value:

10254

metadata-prefix (string)

MetadataPrefix should be used if all the metadata for Flyte executions should be stored under a specific prefix in CloudStorage. If not specified, the data will be stored in the base container directly.

Default Value:

metadata/propeller

rawoutput-prefix (string)

a fully qualified storage path of the form s3://flyte/abc/…, where all data sandboxes should be stored.

Default Value:

""

queue (config.CompositeQueueConfig)

Workflow workqueue configuration, affects the way the work is consumed from the queue.

Default Value:

batch-size: -1
batching-interval: 1s
queue:
  base-delay: 5s
  capacity: 1000
  max-delay: 1m0s
  rate: 100
  type: maxof
sub-queue:
  base-delay: 0s
  capacity: 1000
  max-delay: 0s
  rate: 100
  type: bucket
type: batch

metrics-prefix (string)

An optional prefix for all published metrics.

Default Value:

flyte

enable-admin-launcher (bool)

Enable remote Workflow launcher to Admin

Default Value:

"true"

max-workflow-retries (int)

Maximum number of retries per workflow

Default Value:

"10"

max-ttl-hours (int)

Maximum number of hours a completed workflow should be retained. Number between 1-23 hours

Default Value:

"23"

gc-interval (config.Duration)

Run periodic GC every 30 minutes

Default Value:

30m0s

leader-election (config.LeaderElectionConfig)

Config for leader election.

Default Value:

enabled: false
lease-duration: 15s
lock-config-map:
  Name: ""
  Namespace: ""
renew-deadline: 10s
retry-period: 2s

publish-k8s-events (bool)

Enable events publishing to K8s events API.

Default Value:

"false"

max-output-size-bytes (int64)

Maximum size of outputs per task

Default Value:

"10485760"

kube-client-config (config.KubeClientConfig)

Configuration to control the Kubernetes client

Default Value:

burst: 25
qps: 100
timeout: 30s

node-config (config.NodeConfig)

config for a workflow node

Default Value:

default-deadlines:
  node-active-deadline: 48h0m0s
  node-execution-deadline: 48h0m0s
  workflow-active-deadline: 72h0m0s
interruptible-failure-threshold: 1
max-node-retries-system-failures: 3

max-streak-length (int)

Maximum number of consecutive rounds that one propeller worker can use for one workflow - >1 => turbo-mode is enabled.

Default Value:

"8"

event-config (config.EventConfig)

Configures execution event behavior.

Default Value:

fallback-to-output-reference: false
raw-output-policy: reference

admin-launcher (launchplan.AdminConfig)

Default Value:

burst: 10
cacheSize: 10000
tps: 100
workers: 10

resourcemanager (config.Config (resourcemanager))

Default Value:

redis:
  hostKey: ""
  hostPath: ""
  hostPaths: []
  maxRetries: 0
  primaryName: ""
resourceMaxQuota: 1000
type: noop

workflowstore (workflowstore.Config)

Default Value:

policy: ResourceVersionCache

config.CompositeQueueConfig

type (string)

Type of composite queue to use for the WorkQueue

Default Value:

batch
queue (config.WorkqueueConfig)

Workflow workqueue configuration, affects the way the work is consumed from the queue.

Default Value:

base-delay: 5s
capacity: 1000
max-delay: 1m0s
rate: 100
type: maxof
sub-queue (config.WorkqueueConfig)

SubQueue configuration, affects the way the nodes cause the top-level Work to be re-evaluated.

Default Value:

base-delay: 0s
capacity: 1000
max-delay: 0s
rate: 100
type: bucket
batching-interval (config.Duration)

Duration for which downstream updates are buffered

Default Value:

1s
batch-size (int)

Default Value:

"-1"

config.WorkqueueConfig

type (string)

Type of RateLimiter to use for the WorkQueue

Default Value:

maxof
base-delay (config.Duration)

base backoff delay for failure

Default Value:

5s
max-delay (config.Duration)

Max backoff delay for failure

Default Value:

1m0s
rate (int64)

Bucket Refill rate per second

Default Value:

"100"
capacity (int)

Bucket capacity as number of items

Default Value:

"1000"

config.Config (resourcemanager)

type (string)

Which resource manager to use

Default Value:

noop
resourceMaxQuota (int)

Global limit for concurrent Qubole queries

Default Value:

"1000"
redis (config.RedisConfig)

Config for Redis resourcemanager.

Default Value:

hostKey: ""
hostPath: ""
hostPaths: []
maxRetries: 0
primaryName: ""

config.RedisConfig

hostPaths ([]string)

Redis hosts locations.

Default Value:

[]
primaryName (string)

Redis primary name, fill in only if you are connecting to a redis sentinel cluster.

Default Value:

""
hostPath (string)

Redis host location

Default Value:

""
hostKey (string)

Key for local Redis access

Default Value:

""
maxRetries (int)

See Redis client options for more info

Default Value:

"0"

config.EventConfig

raw-output-policy (string)

How output data should be passed along in execution events.

Default Value:

reference
fallback-to-output-reference (bool)

Whether output data should be sent by reference when it is too large to be sent inline in execution events.

Default Value:

"false"

config.KubeClientConfig

qps (float32)

Default Value:

"100"
burst (int)

Max burst rate for throttle. 0 defaults to 10

Default Value:

"25"
timeout (config.Duration)

Max duration allowed for every request to KubeAPI before giving up. 0 implies no timeout.

Default Value:

30s

config.LeaderElectionConfig

enabled (bool)

Enables/Disables leader election.

Default Value:

"false"
lock-config-map (types.NamespacedName)

ConfigMap namespace/name to use for resource lock.

Default Value:

Name: ""
Namespace: ""
lease-duration (config.Duration)

Duration that non-leader candidates will wait to force acquire leadership. This is measured against time of last observed ack.

Default Value:

15s
renew-deadline (config.Duration)

Duration that the acting master will retry refreshing leadership before giving up.

Default Value:

10s
retry-period (config.Duration)

Duration the LeaderElector clients should wait between tries of actions.

Default Value:

2s

types.NamespacedName

Namespace (string)

Default Value:

""
Name (string)

Default Value:

""

config.NodeConfig

default-deadlines (config.DefaultDeadlines)

Default value for timeouts

Default Value:

node-active-deadline: 48h0m0s
node-execution-deadline: 48h0m0s
workflow-active-deadline: 72h0m0s
max-node-retries-system-failures (int64)

Maximum number of retries per node for node failure due to infra issues

Default Value:

"3"
interruptible-failure-threshold (int64)

number of failures for a node to be still considered interruptible’

Default Value:

"1"

config.DefaultDeadlines

node-execution-deadline (config.Duration)

Default value of node execution timeout

Default Value:

48h0m0s
node-active-deadline (config.Duration)

Default value of node timeout

Default Value:

48h0m0s
workflow-active-deadline (config.Duration)

Default value of workflow timeout

Default Value:

72h0m0s

config.Port

port (int)

Default Value:

"10254"

launchplan.AdminConfig

tps (int64)

The maximum number of transactions per second to flyte admin from this client.

Default Value:

"100"
burst (int)

Maximum burst for throttle

Default Value:

"10"
cacheSize (int)

Maximum cache in terms of number of items stored.

Default Value:

"10000"
workers (int)

Number of parallel workers to work on the queue.

Default Value:

"10"

workflowstore.Config

policy (string)

Workflow Store Policy to initialize

Default Value:

ResourceVersionCache

Section: secrets

secrets-prefix (string)

Prefix where to look for secrets file

Default Value:

/etc/secrets

env-prefix (string)

Prefix for environment variables

Default Value:

FLYTE_SECRET_

Section: storage

type (string)

Sets the type of storage to configure [s3/minio/local/mem/stow].

Default Value:

s3

connection (storage.ConnectionConfig)

Default Value:

access-key: ""
auth-type: iam
disable-ssl: false
endpoint: ""
region: us-east-1
secret-key: ""

stow (storage.StowConfig)

Storage config for stow backend.

Default Value:

{}

container (string)

Initial container (in s3 a bucket) to create -if it doesn’t exist-.’

Default Value:

""

enable-multicontainer (bool)

If this is true, then the container argument is overlooked and redundant. This config will automatically open new connections to new containers/buckets as they are encountered

Default Value:

"false"

cache (storage.CachingConfig)

Default Value:

max_size_mbs: 0
target_gc_percent: 0

limits (storage.LimitsConfig)

Sets limits for stores.

Default Value:

maxDownloadMBs: 2

defaultHttpClient (storage.HTTPClientConfig)

Sets the default http client config.

Default Value:

headers: null
timeout: 0s

storage.CachingConfig

max_size_mbs (int)

Maximum size of the cache where the Blob store data is cached in-memory. If not specified or set to 0, cache is not used

Default Value:

"0"
target_gc_percent (int)

Sets the garbage collection target percentage.

Default Value:

"0"

storage.ConnectionConfig

endpoint (config.URL)

URL for storage client to connect to.

Default Value:

""
auth-type (string)

Auth Type to use [iam,accesskey].

Default Value:

iam
access-key (string)

Access key to use. Only required when authtype is set to accesskey.

Default Value:

""
secret-key (string)

Secret to use when accesskey is set.

Default Value:

""
region (string)

Region to connect to.

Default Value:

us-east-1
disable-ssl (bool)

Disables SSL connection. Should only be used for development.

Default Value:

"false"

storage.HTTPClientConfig

headers (map[string][]string)

Default Value:

null
timeout (config.Duration)

Sets time out on the http client.

Default Value:

0s

storage.LimitsConfig

maxDownloadMBs (int64)

Maximum allowed download size (in MBs) per call.

Default Value:

"2"

storage.StowConfig

kind (string)

Kind of Stow backend to use. Refer to github/graymeta/stow

Default Value:

""
config (map[string]string)

Configuration for stow backend. Refer to github/graymeta/stow

Default Value:

{}

Section: tasks

task-plugins (config.TaskPluginConfig)

Task plugin configuration

Default Value:

default-for-task-types: {}
enabled-plugins: []

max-plugin-phase-versions (int32)

Maximum number of plugin phase versions allowed for one phase.

Default Value:

"100000"

barrier (config.BarrierConfig)

Config for Barrier implementation

Default Value:

cache-size: 10000
cache-ttl: 30m0s
enabled: true

backoff (config.BackOffConfig)

Config for Exponential BackOff implementation

Default Value:

base-second: 2
max-duration: 20s

maxLogMessageLength (int)

Max length of error message.

Default Value:

"2048"

config.BackOffConfig

base-second (int)

The number of seconds representing the base duration of the exponential backoff

Default Value:

"2"
max-duration (config.Duration)

The cap of the backoff duration

Default Value:

20s

config.BarrierConfig

enabled (bool)

Enable Barrier transitions using inmemory context

Default Value:

"true"
cache-size (int)

Max number of barrier to preserve in memory

Default Value:

"10000"
cache-ttl (config.Duration)

Max duration that a barrier would be respected if the process is not restarted. This should account for time required to store the record into persistent storage (across multiple rounds.

Default Value:

30m0s

config.TaskPluginConfig

enabled-plugins ([]string)

deprecated

Default Value:

[]
default-for-task-types (map[string]string)

Default Value:

{}

Section: webhook

metrics-prefix (string)

An optional prefix for all published metrics.

Default Value:

'flyte:'

certDir (string)

Certificate directory to use to write generated certs. Defaults to /etc/webhook/certs/

Default Value:

/etc/webhook/certs

listenPort (int)

The port to use to listen to webhook calls. Defaults to 9443

Default Value:

"9443"

serviceName (string)

The name of the webhook service.

Default Value:

flyte-pod-webhook

secretName (string)

Secret name to write generated certs to.

Default Value:

flyte-pod-webhook

secretManagerType (int)

Default Value:

K8s

awsSecretManager (config.AWSSecretManagerConfig)

AWS Secret Manager config.

Default Value:

resources:
  limits:
    cpu: 200m
    memory: 500Mi
  requests:
    cpu: 200m
    memory: 500Mi
sidecarImage: docker.io/amazon/aws-secrets-manager-secret-sidecar:v0.1.4

vaultSecretManager (config.VaultSecretManagerConfig)

Vault Secret Manager config.

Default Value:

kvVersion: "2"
role: flyte

config.AWSSecretManagerConfig

sidecarImage (string)

Specifies the sidecar docker image to use

Default Value:

docker.io/amazon/aws-secrets-manager-secret-sidecar:v0.1.4
resources (v1.ResourceRequirements)

Default Value:

limits:
  cpu: 200m
  memory: 500Mi
requests:
  cpu: 200m
  memory: 500Mi

v1.ResourceRequirements

limits (v1.ResourceList)

Default Value:

cpu: 200m
memory: 500Mi
requests (v1.ResourceList)

Default Value:

cpu: 200m
memory: 500Mi

config.VaultSecretManagerConfig

role (string)

Specifies the vault role to use

Default Value:

flyte
kvVersion (int)

Default Value:

"2"