AWS Batch

Once you have a Union account, install union:

pip install union

Export the following environment variable to build and push images to your own container registry:

        
# replace with your registry name
export IMAGE_SPEC_REGISTRY="<your-container-registry>"

Then run the following commands to run the workflow:

        
    
git clone https://github.com/unionai/unionai-examples
cd unionai-examples
union run --remote tutorials/sentiment_classifier/sentiment_classifier.py main --model distilbert-base-uncased

The source code for this tutorial can be found here {octicon}mark-github.

This example shows how to use a Flyte AWS batch plugin to execute a tasks on batch service. With AWS Batch, there is no need to install and manage batch computing software or server clusters that you use to run your jobs, allowing you to focus on analyzing results and solving problems.

        
from flytekit import task, workflow
from flytekitplugins.awsbatch import AWSBatchConfig

Use this to configure SubmitJobInput for a AWS batch job. Task’s marked with this will automatically execute natively onto AWS batch service. Refer to the official AWS SubmitJobInput documentation for more detail.

        
    
config = AWSBatchConfig(
    parameters={"codec": "mp4"},
    platformCapabilities="EC2",
    tags={"name": "flyte-example"},
)


@task(task_config=config)
def t1(a: int) -> int:
    return a + 2


@task(task_config=config)
def t2(b: int) -> int:
    return b * 10


@workflow
def my_wf(a: int) -> int:
    b = t1(a=a)
    return t2(b=b)


if __name__ == "__main__":
    print(f"Running my_wf(a=3') {my_wf(a=3)}")