AWS Batch Strangeness
Published on
In a recent project at work I stumbled across an annoying gotcha with AWS Batch. In case you don’t know what AWS Batch is there are more details here. In short, it’s a fairly simple way to scale out workloads that happen on an irregular basis. In our case, create a Docker image with everything needed and then run Batch when you have data to analyse. At the end of the day you end up with something similar in functionality to a SGE/PBS/SLURM (insert HPC queueing system here…) HPC cluster, which in previous roles I found were often used for this type of batch job.