Build your custom task types

exception flytekit.core.base_task.IgnoreOutputs[source]

This exception should be used to indicate that the outputs generated by this can be safely ignored. This is useful in case of distributed training or peer-to-peer parallel algorithms.

For example look at Sagemaker training.

class flytekit.core.base_task.PythonTask(*args, **kwargs)[source]

Base Class for all Tasks with a Python native Interface. This should be directly used for task types, that do not have a python function to be executed. Otherwise refer to flytekit.PythonFunctionTask.

dispatch_execute(ctx: flytekit.core.context_manager.FlyteContext, input_literal_map: flytekit.models.literals.LiteralMap)Union[flytekit.models.literals.LiteralMap, flytekit.models.dynamic_job.DynamicJobSpec][source]

This method translates Flyte’s Type system based input values and invokes the actual call to the executor This method is also invoked during runtime.

  • VoidPromise is returned in the case when the task itself declares no outputs.

  • Literal Map is returned when the task returns either one more outputs in the declaration. Individual outputs may be none

  • DynamicJobSpec is returned when a dynamic workflow is executed

get_input_types()Optional[Dict[str, type]][source]

Returns python native types for inputs. In case this is not a python native task (base class) and hence returns a None. we could deduce the type from literal types, but that is not a required excercise # TODO we could use literal type to determine this

get_type_for_input_var(k: str, v: Any)Optional[Type[Any]][source]

Returns the python native type for the given input variable # TODO we could use literal type to determine this

get_type_for_output_var(k: str, v: Any)Optional[Type[Any]][source]

Returns the python native type for the given output variable # TODO we could use literal type to determine this

post_execute(user_params: flytekit.common.tasks.sdk_runnable.ExecutionParameters, rval: Any)Any[source]

Post execute is called after the execution has completed, with the user_params and can be used to clean-up, or alter the outputs to match the intended tasks outputs. If not overriden, then this function is a No-op

Parameters
  • is returned value from call to execute (rval) –

  • user_params – are the modified user params as created during the pre_execute step

pre_execute(user_params: flytekit.common.tasks.sdk_runnable.ExecutionParameters)flytekit.common.tasks.sdk_runnable.ExecutionParameters[source]

This is the method that will be invoked directly before executing the task method and before all the inputs are converted. One particular case where this is useful is if the context is to be modified for the user process to get some user space parameters. This also ensures that things like SparkSession are already correctly setup before the type transformers are called

This should return either the same context of the mutated context

class flytekit.core.base_task.Task(task_type: str, name: str, interface: Optional[flytekit.models.interface.TypedInterface] = None, metadata: Optional[flytekit.core.base_task.TaskMetadata] = None, task_type_version=0, security_ctx: Optional[flytekit.models.security.SecurityContext] = None, **kwargs)[source]

The base of all Tasks in flytekit. This task is closest to the FlyteIDL TaskTemplate and captures information in FlyteIDL specification and does not have python native interfaces associated. For any real extension please refer to the derived classes.

abstract dispatch_execute(ctx: flytekit.core.context_manager.FlyteContext, input_literal_map: flytekit.models.literals.LiteralMap)flytekit.models.literals.LiteralMap[source]

This method translates Flyte’s Type system based input values and invokes the actual call to the executor This method is also invoked during runtime.

get_input_types()Dict[str, type][source]

Returns python native types for inputs. In case this is not a python native task (base class) and hence returns a None. we could deduce the type from literal types, but that is not a required excercise # TODO we could use literal type to determine this

get_type_for_input_var(k: str, v: Any)type[source]

Returns the python native type for the given input variable # TODO we could use literal type to determine this

get_type_for_output_var(k: str, v: Any)type[source]

Returns the python native type for the given output variable # TODO we could use literal type to determine this

abstract pre_execute(user_params: flytekit.common.tasks.sdk_runnable.ExecutionParameters)flytekit.common.tasks.sdk_runnable.ExecutionParameters[source]

This is the method that will be invoked directly before executing the task method and before all the inputs are converted. One particular case where this is useful is if the context is to be modified for the user process to get some user space parameters. This also ensures that things like SparkSession are already correctly setup before the type transformers are called

This should return either the same context of the mutated context

class flytekit.core.base_task.TaskMetadata(cache: bool = False, cache_version: str = '', interruptable: bool = False, deprecated: str = '', retries: int = 0, timeout: Optional[Union[datetime.timedelta, int]] = None)[source]

Create Metadata to be associated with this Task

Parameters
  • cache – Boolean that indicates if caching should be enabled

  • cache_version – Version string to be used for the cached value

  • interruptable – Boolean that indicates that this task can be interrupted and/or scheduled on nodes with lower QoS guarantees. This will directly reduce the $/execution cost associated, at the cost of performance penalties due to potential interruptions

  • deprecated – A string that can be used to provide a warning message for deprecated task. Absence / empty str indicates that the task is active and not deprecated

  • retries – for retries=n; n > 0, on failures of this task, the task will be retried at-least n number of times.

  • timeout – the max amount of time for which one execution of this task should be executed for. If the execution will be terminated if the runtime exceeds the given timeout (approximately)

to_taskmetadata_model()flytekit.models.task.TaskMetadata[source]

Converts to _task_model.TaskMetadata

flytekit.core.base_task.kwtypes(**kwargs)Dict[str, Type][source]

Converts the keyword arguments to typed dictionary

class flytekit.core.python_function_task.PythonFunctionTask(*args, **kwargs)[source]

A Python Function task should be used as the base for all extensions that have a python function. It will automatically detect interface of the python function and also, create the write execution command to execute the function

It is advised this task is used using the @task decorator as follows

In the above code, the name of the function, the module, and the interface (inputs = int and outputs = str) will be auto detected.

class ExecutionBehavior(value)[source]

An enumeration.

dynamic_execute(task_function: Callable, **kwargs)Any[source]

By the time this function is invoked, the _local_execute function should have unwrapped the Promises and Flyte literal wrappers so that the kwargs we are working with here are now Python native literal values. This function is also expected to return Python native literal values.

Since the user code within a dynamic task constitute a workflow, we have to first compile the workflow, and then execute that workflow.

When running for real in production, the task would stop after the compilation step, and then create a file representing that newly generated workflow, instead of executing it.

execute(**kwargs)Any[source]

This method will be invoked to execute the task. If you do decide to override this method you must also handle dynamic tasks or you will no longer be able to use the task as a dynamic task generator.

class flytekit.core.python_function_task.PythonInstanceTask(*args, **kwargs)[source]

This class should be used as the base class for all Tasks that do not have a user defined function body, but have a platform defined execute method. (Execute needs to be overriden). This base class ensures that the module loader will invoke the right class automatically, by capturing the module name and variable in the module name.