AWS Batch#

Tags: Data, Integration, AWS, Advanced

Executing Batch Job#

Flyte backend can be connected with batch. Once enabled, it allows you to run regular task on AWS batch.

This section provides a guide on how to use the AWS Batch Plugin using flytekit python.

Installation#

To use the flytekit batch plugin simply run the following:

pip install flytekitplugins-awsbatch

Configuring the backend to get AWS Batch working#

Follow this guide to setting up the AWS Batch Plugin

Quick Start#

This plugin allows you to run batch tasks on AWS and only requires you to change a few lines of code. We can then move workflow execution from Kubernetes to AWS.

config = AWSBatch(
    parameters={"codec": "mp4"},
    platformCapabilities="EC2",
    propagateTags=True,
    retryStrategy={"attempts": 10},
    tags={"hello": "world"},
    timeout={"attemptDurationSeconds": 60},
)

@task(task_config=config)
def t1(a: int) -> str:
    return str(a)

AWS Batch