Configuring Custom K8s Resources#

Configurable Resource Types#

Many platform specifications such as task resource defaults, project namespace Kubernetes quota, and more can be assigned using default values or custom overrides. Defaults are specified in the FlyteAdmin config and overrides for specific projects are registered with the FlyteAdmin service.

You can customize these settings along increasing levels of specificity with Flyte:

  • domain

  • project and domain

  • project, domain, and workflow name

  • project, domain, workflow name and launch plan name

See Control Plane to understand projects and domains. Along these dimensions, the following settings are configurable:

Task Resources#

Configuring task resources includes setting default values for unspecified task requests and limits. Task resources also include limits which specify the maximum value that a task request or a limit can have.

In the absence of an override, the global default values in the FlyteAdmin config are used.

The override values from the database are assigned at execution, rather than registration time.

To customize resources for project-domain attributes, define a tra.yaml file with your overrides:

defaults:
    cpu: "1"
    memory: 150Mi
limits:
    cpu: "2"
    memory: 450Mi
project: flyteexamples
domain: development

Update the task resource attributes for a project-domain combination:

flytectl update task-resource-attribute --attrFile tra.yaml

Refer to the docs to learn more about the command and its supported flag(s).

To fetch and verify the individual project-domain attributes:

flytectl get task-resource-attribute -p flyteexamples -d development

Refer to the docs to learn more about the command and its supported flag(s).

You can view all custom task-resource-attributes by visiting protocol://<host/api/v1/matchable_attributes?resource_type=0> and substitute the protocol and host appropriately.

Cluster Resources#

These are free-form key-value pairs used when filling the templates that the admin feeds into the cluster manager; the process that syncs Kubernetes resources.

The keys represent templatized variables in cluster resource template YAML and the values are what you want to see filled in.

In the absence of custom override values, you can use templateData from the FlyteAdmin config as a default. Flyte specifies these defaults by domain and applies them to every project-domain namespace combination.

Note

The settings above can be specified on domain, and project-and-domain. Since Flyte hasn’t tied the notion of a workflow or a launch plan to any Kubernetes construct, specifying a workflow or launch plan name doesn’t make sense. This is a departure from the usual hierarchy for customizable resources.

Define an attributes file, cra.yaml:

attributes:
    projectQuotaCpu: "1000"
    projectQuotaMemory: 5TB
domain: development
project: flyteexamples

To ensure that the overrides reflect in the Kubernetes namespace flyteexamples-development (that is, the namespace has a resource quota of 1000 CPU cores and 5TB of memory) when the admin fills in cluster resource templates:

flytectl update cluster-resource-attribute --attrFile cra.yaml

Refer to the docs to learn more about the command and its supported flag(s).

To fetch and verify the individual project-domain attributes:

flytectl get cluster-resource-attribute -p flyteexamples -d development

Refer to the docs to learn more about the command and its supported flag(s).

Flyte uses these updated values to fill the template fields for the flyteexamples-development namespace.

For other namespaces, the platform defaults apply.

Note

The template values, for example, projectQuotaCpu or projectQuotaMemory are free-form strings. Ensure that they match the template placeholders in your template file for your changes to take effect and custom values to be substituted.

You can view all custom cluster-resource-attributes by visiting protocol://<host/api/v1/matchable_attributes?resource_type=1> and substitute the protocol and host appropriately.

Execution Cluster Label#

This allows forcing a matching execution to consistently execute on a specific Kubernetes cluster for multi-cluster Flyte deployment set-up.

Define an attributes file in ec.yaml:

value: mycluster
domain: development
project: flyteexamples

Ensure that admin places executions in the flyteexamples project and development domain onto mycluster:

flytectl update execution-cluster-label --attrFile ec.yaml

Refer to the docs to learn more about the command and its supported flag(s).

To fetch and verify the individual project-domain attributes:

flytectl get execution-cluster-label -p flyteexamples -d development

Refer to the docs to learn more about the command and its supported flag(s).

You can view all custom execution cluster attributes by visiting protocol://<host/api/v1/matchable_attributes?resource_type=3> and substitute the protocol and host appropriately.

Execution Queues#

Execution queues are defined in flyteadmin config. These are used for execution placement for constructs like AWS Batch.

The attributes associated with an execution queue must match the tags for workflow executions. The tags associated with configurable resources are stored in the admin database.

flytectl update execution-queue-attribute

Refer to the :ref:`docs <flytectl:flytectl_update_execution-queue-attribute>` to learn more about the command and its supported flag(s).

You can view existing attributes for which tags can be assigned by visiting protocol://<host>/api/v1/matchable_attributes?resource_type=2 and substitute the protocol and host appropriately.

Hierarchy#

Increasing specificity defines how matchable resource attributes get applied. The available configurations, in order of decreasing specifity are:

  1. Domain, project, workflow name, and launch plan.

  2. Domain, project, and workflow name

  3. Domain and project

  4. Domain

Default values for all and per-domain attributes may be specified in the FlyteAdmin config as documented in the Adding New Customizable Resources.

Example#

If the database includes the following:

Domain

Project

Workflow

Launch Plan

Tags

production

widgetmodels

critical

production

widgetmodels

Demand

supply

Any inbound CreateExecution requests with [Domain: Production, Project: widgetmodels, Workflow: Demand] for any launch plan will have a tag value of “supply”.

Any inbound CreateExecution requests with [Domain: Production, Project: widgetmodels] for any workflow other than Demand and any launch plan will have a tag value “critical”.

All other inbound CreateExecution requests will use the default values specified in the FlyteAdmin config (if any).