Understanding the State Transition in a Workflow#

Tags: Basic, Design

High Level Overview of How a Workflow Progresses to Success#

        flowchart TD
  id1(( ))
  id1 --> Ready
  Ready --> Running
  subgraph Running
  id2(( ))
  id2 --> NodeQueued
  NodeQueued --> NodeRunning
  subgraph NodeRunning
  id3(( ))
  id3 --> TaskQueued
  TaskQueued --> TaskRunning
  TaskRunning --> TaskSuccess
  end
  TaskSuccess --> NodeSuccess
  end
  NodeSuccess --> Success

This state diagram illustrates a high-level, simplistic view of the state transitions that a workflow with a single task and node would go through as the user observes success.

The following sections explain the various observable (and some hidden) states for workflow, node, and task state transitions.

Workflow States#

        flowchart TD
  Queued -->|On system errors more than threshold| Aborted
  Queued --> Ready
  Ready--> |Write inputs to workflow| Running
  Running--> |On system error| Running
  Running--> |On all Nodes Success| Succeeding
  Succeeding--> |On successful event send to Admin| Succeeded
  Succeeding--> |On system error| Succeeding
  Ready--> |On precondition failure| Failing
  Running--> |On any Node Failure| Failing
  Ready--> |On user initiated abort| Aborting
  Running--> |On user initiated abort| Aborting
  Succeeding--> |On user initiated abort| Aborting
  Failing--> |If Failure node exists| HandleFailureNode
  Failing--> |On user initiated abort| Aborting
  HandleFailureNode--> |On completing failure node| Failed
  HandleFailureNode--> |On user initiated abort| Aborting
  Failing--> |On successful send of Failure node| Failed
  Aborting--> |On successful event send to Admin| Aborted

A workflow always starts in the Ready state and ends either in Failed, Succeeded, or Aborted state. Any system error within a state causes a retry on that state. These retries are capped by system retries which eventually lead to an Aborted state if the failure persists.

Every transition between states is recorded in FlyteAdmin using workflowexecutionevent.

The phases in the above state diagram are captured in the admin database as specified here workflowexecution.phase and are sent as a part of the Execution event.

The state machine specification for the illustration can be found here.

Node States#

        flowchart TD
  id1(( ))
  id1-->NotYetStarted
  id1-->|Will stop the node execution |Aborted
  NotYetStarted-->|If all upstream nodes are ready, i.e, inputs are ready | Queued
  NotYetStarted--> |If the branch was not taken |Skipped
  Queued-->|Start task execution- attempt 0 | Running
  Running-->|If task timeout has elapsed and retry_attempts >= max_retries|TimingOut
  Running-->|Internal state|Succeeding
  Running-->|For dynamic nodes generating workflows| DynamicRunning
  DynamicRunning-->TimingOut
  DynamicRunning-->RetryableFailure
  TimingOut-->|If total node timeout has elapsed|TimedOut
  DynamicRunning-->Succeeding
  Succeeding-->|User observes the task as succeeded| Succeeded
  Running-->|on retryable failure| RetryableFailure
  RetryableFailure-->|if retry_attempts < max_retries|Running
  RetryableFailure-->|retry_attempts >= max_retries|Failing
  Failing-->Failed
  Succeeded-->id2(( ))
  Failed-->id2(( ))

This state diagram illustrates the node transition through various states. This is the core finite state machine for a node. From the user’s perspective, a workflow simply consists of a sequence of tasks. But to Flyte, a workflow internally creates a meta entity known as node.

Once a Workflow enters the Running state, it triggers the phantom start node of the workflow. The start node is considered to be the entry node of any workflow. The start node begins by executing all its child-nodes using a modified Depth First Search algorithm recursively.

Nodes can be of different types as listed below, but all the nodes traverse through the same transitions:

  1. Start Node - Only exists during the execution and is not modeled in the core spec.

  2. Task Node

  3. Branch Node

  4. Workflow Node

  5. Dynamic Node - Just a task node that does not return output but constitutes a dynamic workflow. When the task runs, it remains in the RUNNING state. Once the task completes and Flyte starts executing the dynamic workflow, the overarching node that contains both the original task and the dynamic workflow enters DYNAMIC_RUNNING state.

  6. End Node - Only exists during the execution and is not modeled in the core spec

Every transition between states is recorded in FlyteAdmin using nodeexecutionevent.

Every NodeExecutionEvent can have any nodeexecution.phase.

Note

TODO: Add explanation for each phase.

The state machine specification for the illustration can be found here.

Task States#

        flowchart TD
  id1(( ))
  id1-->|Aborted by NodeHandler- timeouts, external abort, etc,.| NotReady
  id1-->Aborted
  NotReady-->|Optional-Blocked on resource quota or resource pool | WaitingForResources
  WaitingForResources--> |Optional- Has been submitted, but hasn't started |Queued
  Queued-->|Optional- Prestart initialization | Initializing
  Initializing-->|Actual execution of user code has started|Running
  Running-->|Successful execution|Success
  Running-->|Failed with a retryable error|RetryableFailure
  Running-->|Unrecoverable failure, will stop all execution|PermanentFailure
  Success-->id2(( ))
  RetryableFailure-->id2(( ))
  PermanentFailure-->id2(( ))

The state diagram above illustrates the various states through which a task transitions. This is the core finite state machine for a task.

Every transition between states is recorded in FlyteAdmin using taskexecutionevent.

Every TaskExecutionEvent can have any taskexecution.phase.

Note

TODO: Add explanation for each phase.

The state machine specification for the illustration can be found here.