- flytekit.map_task(task_function, concurrency=0, min_success_ratio=1.0, **kwargs)#
Use a map task for parallelizable tasks that run across a list of an input type. A map task can be composed of any individual
Invoke a map task with arguments using the
listversion of the expected input.
@task def my_mappable_task(a: int) -> str: return str(a) @workflow def my_wf(x: typing.List[int]) -> typing.List[str]: return map_task(my_mappable_task, metadata=TaskMetadata(retries=1), concurrency=10, min_success_ratio=0.75,)( a=x ).with_overrides(cpu="10M")
At run time, the underlying map task will be run for every value in the input collection. Attributes such as
with_overridesare applied to individual instances of the mapped task.
Map Task Plugins
There are two plugins to run maptasks that ship as part of flyteplugins:
Enabling a plugin is controlled in the plugin configuration at values-sandbox.yaml.
By default, the map task uses the
K8s Arrayplugin. It executes array tasks by launching a pod for every instance in the array. It’s simple to use, has a straightforward implementation, and works out of the box.
Learn more about
AWS batchsetup configuration here.
A custom plugin can also be implemented to handle the task type.
task_function (flytekit.core.python_function_task.PythonFunctionTask) – This argument is implicitly passed and represents the repeatable function
concurrency (int) – If specified, this limits the number of mapped tasks than can run in parallel to the given batch size. If the size of the input exceeds the concurrency value, then multiple batches will be run serially until all inputs are processed. If left unspecified, this means unbounded concurrency.
min_success_ratio (float) – If specified, this determines the minimum fraction of total jobs which can complete successfully before terminating this task and marking it successful.