flytekit.remote.remote.FlyteRemote#

class flytekit.remote.remote.FlyteRemote(config, default_project=None, default_domain=None, data_upload_location='s3://my-s3-bucket/data', **kwargs)[source]#

Main entrypoint for programmatically accessing a Flyte remote backend.

The term ‘remote’ is synonymous with ‘backend’ or ‘deployment’ and refers to a hosted instance of the Flyte platform, which comes with a Flyte Admin server on some known URI.

Initialize a FlyteRemote object.

Parameters

default_project (Optional[str]) – default project to use when fetching or executing flyte entities.
default_domain (Optional[str]) – default domain to use when fetching or executing flyte entities.
data_upload_location (str) – this is where all the default data will be uploaded when providing inputs. The default location - s3://my-s3-bucket/data works for sandbox/demo environment. Please override this for non-sandbox cases.
config (Config) –

Methods

execute(entity, inputs, project=None, domain=None, name=None, version=None, execution_name=None, image_config=None, options=None, wait=False, type_hints=None)[source]#

Execute a task, workflow, or launchplan, either something that’s been declared locally, or a fetched entity.

This method supports: - Flyte{Task, Workflow, LaunchPlan} remote module objects. - @task-decorated functions and TaskTemplate tasks. - @workflow-decorated functions. - LaunchPlan objects.

For local entities, this code will attempt to find the entity first, and if missing, will compile and register the object.

Not all arguments are relevant in all circumstances. For example, there’s no reason to use the serialization settings for entities that have already been registered on Admin.

Parameters

options (Optional[flytekit.tools.translator.Options]) –
entity (Union[flytekit.remote.task.FlyteTask, flytekit.remote.launch_plan.FlyteLaunchPlan, flytekit.remote.workflow.FlyteWorkflow, flytekit.core.base_task.PythonTask, flytekit.core.workflow.WorkflowBase, flytekit.core.launch_plan.LaunchPlan]) – entity to execute
inputs (Dict[str, Any]) – dictionary mapping argument names to values
project (Optional[str]) – execute entity in this project. If entity doesn’t exist in the project, register the entity first before executing.
domain (Optional[str]) – execute entity in this domain. If entity doesn’t exist in the domain, register the entity first before executing.
name (Optional[str]) – execute entity using this name. If not None, use this value instead of entity.name
version (Optional[str]) – execute entity using this version. If None, uses auto-generated value.
execution_name (Optional[str]) – name of the execution. If None, uses auto-generated value.
image_config (Optional[flytekit.configuration.ImageConfig]) –
wait (bool) – if True, waits for execution to complete
type_hints (Optional[Dict[str, Type]]) – Python types to be passed to the TypeEngine so that it knows how to properly convert the input values for the execution into Flyte literals. If missing, will default to first guessing the type using the type engine, and then to type(v). Providing the correct Python types is particularly important if the inputs are containers like lists or maps, or if the Python type is one of the more complex Flyte provided classes (like a StructuredDataset that’s annotated with columns).

Return type

flytekit.remote.executions.FlyteWorkflowExecution

execute_local_launch_plan(entity, inputs, version, project=None, domain=None, execution_name=None, options=None, wait=False)[source]#

Parameters

entity (flytekit.core.launch_plan.LaunchPlan) – The locally defined launch plan object
inputs (Dict[str, Any]) – Inputs to be passed into the execution as a dict with Python native values.
version (str) – The version to look up/register the launch plan (if not already exists)
project (Optional[str]) – The same as version, but will default to the Remote object’s project
domain (Optional[str]) – The same as version, but will default to the Remote object’s domain
execution_name (Optional[str]) – If specified, will be used as the execution name instead of randomly generating.
options (Optional[flytekit.tools.translator.Options]) –
wait (bool) –

Returns

Return type

flytekit.remote.executions.FlyteWorkflowExecution

execute_local_task(entity, inputs, project=None, domain=None, name=None, version=None, execution_name=None, image_config=None, wait=False)[source]#

Execute an @task-decorated function or TaskTemplate task.

Parameters

entity (flytekit.core.base_task.PythonTask) –
inputs (Dict[str, Any]) – register the task, which requires compiling the task, before running it.
project (Optional[str]) –
domain (Optional[str]) –
name (Optional[str]) –
version (Optional[str]) –
execution_name (Optional[str]) –
image_config (Optional[flytekit.configuration.ImageConfig]) –
wait (bool) –

Returns

Return type

flytekit.remote.executions.FlyteWorkflowExecution

execute_local_workflow(entity, inputs, project=None, domain=None, name=None, version=None, execution_name=None, image_config=None, options=None, wait=False)[source]#

Execute an @workflow decorated function. :param entity: :param inputs: :param project: :param domain: :param name: :param version: :param execution_name: :param image_config: :param options: :param wait: :return:

Parameters

entity (flytekit.core.workflow.WorkflowBase) –
inputs (Dict[str, Any]) –
project (Optional[str]) –
domain (Optional[str]) –
name (Optional[str]) –
version (Optional[str]) –
execution_name (Optional[str]) –
image_config (Optional[flytekit.configuration.ImageConfig]) –
options (Optional[flytekit.tools.translator.Options]) –
wait (bool) –

Return type

flytekit.remote.executions.FlyteWorkflowExecution

execute_remote_task_lp(entity, inputs, project=None, domain=None, execution_name=None, options=None, wait=False, type_hints=None)[source]#

Execute a FlyteTask, or FlyteLaunchplan.

NOTE: the name and version arguments are currently not used and only there consistency in the function signature

Parameters

entity (Union[flytekit.remote.task.FlyteTask, flytekit.remote.launch_plan.FlyteLaunchPlan]) –
inputs (Dict[str, Any]) –
project (Optional[str]) –
domain (Optional[str]) –
execution_name (Optional[str]) –
options (Optional[flytekit.tools.translator.Options]) –
wait (bool) –
type_hints (Optional[Dict[str, Type]]) –

Return type

flytekit.remote.executions.FlyteWorkflowExecution

execute_remote_wf(entity, inputs, project=None, domain=None, execution_name=None, options=None, wait=False, type_hints=None)[source]#

Execute a FlyteWorkflow.

NOTE: the name and version arguments are currently not used and only there consistency in the function signature

Parameters

entity (flytekit.remote.workflow.FlyteWorkflow) –
inputs (Dict[str, Any]) –
project (Optional[str]) –
domain (Optional[str]) –
execution_name (Optional[str]) –
options (Optional[flytekit.tools.translator.Options]) –
wait (bool) –
type_hints (Optional[Dict[str, Type]]) –

Return type

flytekit.remote.executions.FlyteWorkflowExecution

fetch_execution(project=None, domain=None, name=None)[source]#

Fetch a workflow execution entity from flyte admin.

Parameters

project (Optional[str]) – fetch entity from this project. If None, uses the default_project attribute.
domain (Optional[str]) – fetch entity from this domain. If None, uses the default_domain attribute.
name (Optional[str]) – fetch entity with matching name.

Returns

FlyteWorkflowExecution

Raises

FlyteAssertion if name is None

Return type

flytekit.remote.executions.FlyteWorkflowExecution

fetch_launch_plan(project=None, domain=None, name=None, version=None)[source]#

Fetch a launchplan entity from flyte admin.

Parameters

project (Optional[str]) – fetch entity from this project. If None, uses the default_project attribute.
domain (Optional[str]) – fetch entity from this domain. If None, uses the default_domain attribute.
name (Optional[str]) – fetch entity with matching name.
version (Optional[str]) – fetch entity with matching version. If None, gets the latest version of the entity.

Returns

FlyteLaunchPlan

Raises

FlyteAssertion if name is None

Return type

flytekit.remote.launch_plan.FlyteLaunchPlan

fetch_task(project=None, domain=None, name=None, version=None)[source]#

Fetch a task entity from flyte admin.

Parameters

project (Optional[str]) – fetch entity from this project. If None, uses the default_project attribute.
domain (Optional[str]) – fetch entity from this domain. If None, uses the default_domain attribute.
name (Optional[str]) – fetch entity with matching name.
version (Optional[str]) – fetch entity with matching version. If None, gets the latest version of the entity.

Returns

FlyteTask

Raises

FlyteAssertion if name is None

Return type

flytekit.remote.task.FlyteTask

fetch_workflow(project=None, domain=None, name=None, version=None)[source]#

Fetch a workflow entity from flyte admin.

Parameters

project (Optional[str]) – fetch entity from this project. If None, uses the default_project attribute.
domain (Optional[str]) – fetch entity from this domain. If None, uses the default_domain attribute.
name (Optional[str]) – fetch entity with matching name.
version (Optional[str]) – fetch entity with matching version. If None, gets the latest version of the entity.

Raises

FlyteAssertion if name is None

Return type

flytekit.remote.workflow.FlyteWorkflow

generate_console_url(execution)[source]#

Parameters: execution (Union[flytekit.remote.executions.FlyteWorkflowExecution, flytekit.remote.executions.FlyteNodeExecution, flytekit.remote.executions.FlyteTaskExecution]) –

generate_http_domain()[source]#

This should generate the domain where the HTTP endpoints for the Flyte backend are hosted. This should be the domain that console is hosted on.

Returns
Return type: str

list_tasks_by_version(version, project=None, domain=None, limit=100)[source]#

Parameters

version (str) –
project (Optional[str]) –
domain (Optional[str]) –
limit (Optional[int]) –

Return type

List[flytekit.remote.task.FlyteTask]

recent_executions(project=None, domain=None, limit=100)[source]#

Parameters

project (Optional[str]) –
domain (Optional[str]) –
limit (Optional[int]) –

Return type

List[flytekit.remote.executions.FlyteWorkflowExecution]

register_launch_plan(entity, version, project=None, domain=None, options=None)[source]#

Register a given launchplan, possibly applying overrides from the provided options. :param entity: Launchplan to be registered :param version: :param project: Optionally provide a project, if not already provided in flyteremote constructor or a separate one :param domain: Optionally provide a domain, if not already provided in FlyteRemote constructor or a separate one :param options: :return:

Parameters

entity (flytekit.core.launch_plan.LaunchPlan) –
version (str) –
project (Optional[str]) –
domain (Optional[str]) –
options (Optional[flytekit.tools.translator.Options]) –

Return type

flytekit.remote.launch_plan.FlyteLaunchPlan

register_script(entity, image_config=None, version=None, project=None, domain=None, destination_dir='.', default_launch_plan=True, options=None, source_path=None, module_name=None)[source]#

Use this method to register a workflow via script mode. :param destination_dir: :param domain: :param project: :param image_config: :param version: version for the entity to be registered as :param entity: The workflow to be registered or the task to be registered :param default_launch_plan: This should be true if a default launch plan should be created for the workflow :param options: Additional execution options that can be configured for the default launchplan :param source_path: The root of the project path :param module_name: the name of the module :return:

Parameters

entity (Union[flytekit.core.workflow.WorkflowBase, flytekit.core.base_task.PythonTask]) –
image_config (Optional[flytekit.configuration.ImageConfig]) –
version (Optional[str]) –
project (Optional[str]) –
domain (Optional[str]) –
destination_dir (str) –
default_launch_plan (Optional[bool]) –
options (Optional[flytekit.tools.translator.Options]) –
source_path (Optional[str]) –
module_name (Optional[str]) –

Return type

Union[flytekit.remote.workflow.FlyteWorkflow, flytekit.remote.task.FlyteTask]

register_task(entity, serialization_settings, version=None)[source]#

Register a qualified task (PythonTask) with Remote For any conflicting parameters method arguments are regarded as overrides

Parameters

entity (flytekit.core.base_task.PythonTask) – PythonTask can be either @task or a instance of a Task class
serialization_settings (flytekit.configuration.SerializationSettings) – Settings that will be used to override various serialization parameters.
version (Optional[str]) – version that will be used to register. If not specified will default to using the serialization settings default

Returns

Return type

flytekit.remote.task.FlyteTask

register_workflow(entity, serialization_settings=None, version=None, default_launch_plan=True, options=None)[source]#

Use this method to register a workflow. :param version: version for the entity to be registered as :param entity: The workflow to be registered :param serialization_settings: The serialization settings to be used :param default_launch_plan: This should be true if a default launch plan should be created for the workflow :param options: Additional execution options that can be configured for the default launchplan :return:

Parameters

entity (flytekit.core.workflow.WorkflowBase) –
serialization_settings (Optional[flytekit.configuration.SerializationSettings]) –
version (Optional[str]) –
default_launch_plan (Optional[bool]) –
options (Optional[flytekit.tools.translator.Options]) –

Return type

flytekit.remote.workflow.FlyteWorkflow

remote_context()[source]#: Context manager with remote-specific configuration.

sync(execution, entity_definition=None, sync_nodes=False)[source]#

This function was previously a singledispatchmethod. We’ve removed that but this function remains so that we don’t break people.

Parameters

execution (flytekit.remote.executions.FlyteWorkflowExecution) –
entity_definition (Optional[Union[flytekit.remote.workflow.FlyteWorkflow, flytekit.remote.task.FlyteTask]]) –
sync_nodes (bool) – By default sync will fetch data on all underlying node executions (recursively, so subworkflows will also get picked up). Set this to False in order to prevent that (which will make this call faster).

Returns

Returns the same execution object, but with additional information pulled in.

Return type

flytekit.remote.executions.FlyteWorkflowExecution

sync_execution(execution, entity_definition=None, sync_nodes=False)[source]#

Sync a FlyteWorkflowExecution object with its corresponding remote state.

Parameters

execution (flytekit.remote.executions.FlyteWorkflowExecution) –
entity_definition (Optional[Union[flytekit.remote.workflow.FlyteWorkflow, flytekit.remote.task.FlyteTask]]) –
sync_nodes (bool) –

Return type

flytekit.remote.executions.FlyteWorkflowExecution

sync_node_execution(execution, node_mapping)[source]#

Get data backing a node execution. These FlyteNodeExecution objects should’ve come from Admin with the model fields already populated correctly. For purposes of the remote experience, we’d like to supplement the object with some additional fields:

inputs/outputs

task/workflow executions, and/or underlying node executions in the case of parent nodes

TypedInterface (remote wrapper type)

A node can have several different types of executions behind it. That is, the node could’ve run (perhaps multiple times because of retries):

A task

A static subworkflow

A dynamic subworkflow (which in turn may have run additional tasks, subwfs, and/or launch plans)

A launch plan

The data model is complicated, so ascertaining which of these happened is a bit tricky. That logic is encapsulated in this function.

Parameters

execution (flytekit.remote.executions.FlyteNodeExecution) –
node_mapping (Dict[str, flytekit.remote.nodes.FlyteNode]) –

Return type

flytekit.remote.executions.FlyteNodeExecution

sync_task_execution(execution, entity_definition=None)[source]#

Sync a FlyteTaskExecution object with its corresponding remote state.

Parameters

execution (flytekit.remote.executions.FlyteTaskExecution) –
entity_definition (Optional[Union[flytekit.remote.workflow.FlyteWorkflow, flytekit.remote.task.FlyteTask]]) –

Return type

flytekit.remote.executions.FlyteTaskExecution

terminate(execution, cause)[source]#

Terminate a workflow execution.

Parameters

execution (flytekit.remote.executions.FlyteWorkflowExecution) – workflow execution to terminate
cause (str) – reason for termination

wait(execution, timeout=None, poll_interval=None, sync_nodes=True)[source]#

Wait for an execution to finish.

Parameters

execution (flytekit.remote.executions.FlyteWorkflowExecution) – execution object to wait on
timeout (Optional[datetime.timedelta]) – maximum amount of time to wait
poll_interval (Optional[datetime.timedelta]) – sync workflow execution at this interval
sync_nodes (bool) – passed along to the sync call for the workflow execution

Return type

flytekit.remote.executions.FlyteWorkflowExecution

Attributes

client#: Return a SynchronousFlyteClient for additional operations.

config#: Image config.

context#

default_domain#: Default project to use when fetching or executing flyte entities.

default_project#: Default project to use when fetching or executing flyte entities.

file_access#: File access provider to use for offloading non-literal inputs/outputs.