Configuring Your Flyte Deployment

Configurable Resource Types

Flyte allows these custom settings along with the following combination of dimensions:

  • domain

  • project and domain

  • project, domain, and name (must be either the name of a workflow name or a launch plan)

Please see the Control Plane document to get to know about projects and domains. Along these dimensions, the following settings are configurable.

Alert

Not all three of the combinations mentioned above are valid for each of these settings.

  • Defaults are used for task resource requests and limits (when not specified by the author of the task).

  • Settings for project-namespaced cluster resource configuration that feeds into admin’s cluster resource manager.

  • Execution queues that are used for dynamic workflows. Effectively, they’re meant to be used with constructs like AWS Batch.

Note

Execution queues are used to determine where tasks yielded by a flytekit.dynamic() workflow or map task run.

  • Determining how workflow executions are assigned to clusters in a multi-cluster Flyte deployment.

Tip

The proto definition is the definitive source encapsulating which Matchable Resource attributes can be customized.

Each of the four above settings is discussed below.

Task Resources

Configuring task resources includes setting default values for the requests and limits for the following resources:

  • cpu

  • gpu

  • memory

  • storage

In the absence of an override, the global default values in the FlyteAdmin config are used.

The override values from the database are assigned at execution time.

To update individual project-domain attributes, use the following command for your reference.

curl --request PUT 'https://flyte.company.net/api/v1/project_domain_attributes/projectname/staging' \
    --header 'Content-Type: application/json' --data-raw \
    '{"attributes":{"matchingAttributes":{"taskResourceAttributes":{"defaults":{"cpu": "1000", "memory": "5000Gi"}, "limits": {"cpu": "4000"}}}}'

Tip

The equivalent flytectl command is:

flytectl update task-resource-attribute

Refer to the :ref:`docs <flytectl:flytectl_update_task-resource-attribute>` to learn more about the command and its supported flag(s).

Cluster Resources

These are free-form key-value pairs that are used when filling in the templates that the admin feeds into its cluster manager. The keys represent templatized variables in cluster resource template YAML and the values are what you want to see filled in.

In the absence of custom override values, templateData from the FlyteAdmin config is used as a default.

Note

The above-referenced settings can only take on domain, project, and domain specificity. Since Flyte has not tied in the notion of a workflow or a launch plan to any Kubernetes constructs, specifying a workflow or launch plan name doesn’t make any sense.

Running the following will ensure that when the admin fills in cluster resource templates, the Kubernetes namespace flyteexamples-development will have a resource quota of 1000 CPU cores and 5TB of memory.

flyte-cli -h localhost:30081 -p flyteexamples -d development update-cluster-resource-attributes  \
--attributes projectQuotaCpu 1000 --attributes projectQuotaMemory 5000Gi

Tip

The equivalent flytectl command is:

flytectl update cluster-resource-attribute

Refer to the docs to learn more about the command and its supported flag(s).

The above-updated values will, in turn, be used to fill in the template fields.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: project-quota
  namespace: {{ namespace }} 
spec:
  hard:
    limits.cpu: {{ projectQuotaCpu }} 
    limits.memory: {{ projectQuotaMemory }}

The values can now be copied from the base of this repository to the flyteexamples-development namespace only.

For other namespaces, the platform defaults apply.

Note

The template values, e.g. projectQuotaCpu or projectQuotaMemory are freeform strings. You must ensure that they match the template placeholders in your template file for your changes to take effect.

Execution Queues

Execution queues themselves are currently defined in the flyteadmin config.

The attributes associated with an execution queue must match the tags for workflow executions. The tags are associated with configurable resources stored in the admin database.

flyte-cli -h localhost:30081 -p flyteexamples -d development update-execution-queue-attributes  \
--tags critical --tags gpu_intensive

Tip

The equivalent command in flytectl is:

flytectl update execution-queue-attribute

Refer to the docs to learn more about the command and its supported flag(s).

You can view existing attributes for which tags can be assigned by visiting protocol://<host>/api/v1/matchable_attributes?resource_type=3.

Execution Cluster Label

This allows forcing a matching execution to consistently execute on a specific Kubernetes cluster.

You can set this using flyte-cli:

flyte-cli -h localhost:30081 -p flyteexamples -d development update-execution-cluster-label --value mycluster

Tip

The equivalent command in flytectl is:

flytectl update execution-cluster-label

Refer to the docs to learn more about the command and its supported flag(s).

Hierarchy

Increasing specificity defines how matchable resource attributes get applied. The available configurations, in order of decreasing specifity, are:

  1. Domain, project, workflow name, and launch plan.

  2. Domain, project, and workflow name

  3. Domain and project

  4. Domain

Default values for all and per-domain attributes may be specified in the FlyteAdmin config as documented in the Adding customizable resources.

Example

If the database includes the following:

Domain

Project

Workflow

Launch Plan

Tags

production

widgetmodels

critical

production

widgetmodels

Demand

supply

Any inbound CreateExecution requests with [Domain: Production, Project: widgetmodels, Workflow: Demand] for any launch plan will have a tag value of “supply”.

Any inbound CreateExecution requests with [Domain: Production, Project: widgetmodels] for any workflow other than Demand and any launch plan will have a tag value “critical”.

All other inbound CreateExecution requests will use the default values specified in the FlyteAdmin config (if any).

Debugging

To get the matchable resources of execution queue attributes, run the command:

flytectl get execution-queue-attribute

Note

Alternatively, you can also hit the URL: protocol://<host/api/v1/project_domain_attributes/widgetmodels/production?resource_type=2>.

To get the global state of the world, list all endpoints. For example, visit protocol://<host>/api/v1/matchable_attributes?resource_type=2.

The resource type enum (int) is defined in the Matchable Resource.