A synchronous task runner for AWS Fargate on Amazon ECS

Supported tags and respective Dockerfile links:
・latest (versions/3.0/Dockerfile)
・3.0 (versions/3.0/Dockerfile)
・2.3 (versions/2.3/Dockerfile)
・1 (versions/1.2/Dockerfile)

Description

This is a synchronous task runner for AWS Fargate. It runs a docker container on Fargate and waits for its done. Then it returns its standard output logs from CloudWatch Logs. All resources we need are created temporarily and remove them after the task finished.

Installation

curl (macOS):

$ curl -Lo ecs-task-runner https://github.com/pottava/ecs-task-runner/releases/download/3.0/ecs-task-runner_darwin_amd64 \
    && chmod +x ecs-task-runner

curl (Linux):

$ curl -Lo ecs-task-runner https://github.com/pottava/ecs-task-runner/releases/download/3.0/ecs-task-runner_linux_amd64 \
    && chmod +x ecs-task-runner

go:

go get github.com/pottava/ecs-task-runner/...

docker:

docker pull pottava/ecs-task-runner

Parameters

Common parameters:

Environment Variables	Argument	Description	Required	Default
AWS_ACCESS_KEY_ID	access-key, a	AWS access key for API access
AWS_SECRET_ACCESS_KEY	secret-key, s	AWS secret key for API access
AWS_DEFAULT_REGION	region, r	AWS region for API access		us-east-1
AWS_PROFILE	profile	AWS profile for API access		default
AWS_ASSUME_ROLE	assume-role	IAM Role ARN to be assumed
AWS_MFA_SERIAL_NUMBER	mfa-serial-num	A serial number of MFA device
AWS_MFA_TOKEN	mfa-token	A token for MFA
ECS_CLUSTER	cluster, c	Amazon ECS cluster name
EXEC_ROLE_NAME	exec-role-name	Name of an execution role		ecs-task-runner
TASK_TIMEOUT	timeout, t	Timeout minutes for the task		30
EXTENDED_OUTPUT	extended-output	True: meta data also returns		false

for the run command:

Environment Variables	Argument	Description	Required	Default
DOCKER_IMAGE		Docker image to be run on ECS	*
FARGATE_SPOT	spot	True: fargate spot will be used		false
FORCE_ECR	force-ecr, f	True: you can use shortened name		false
ENTRYPOINT	entrypoint	Override ENTRYPOINT of the image
COMMAND	command	Override CMD of the image
PORT	port, p	Publish ports
ENVIRONMENT	environment, e	Add ENV to the container
USER	docker-user	The user inside the container
LABEL	label, l	Add LABEL to the container
SUBNETS	subnets	Fargate's Subnets
SECURITY_GROUPS	security-groups	Fargate's SecurityGroups
TASKDEF_FAMILY	taskdef-family	ECS Task Definition family name		ecs-task-runner
TASK_ROLE	task-role-arn	ARN of an IAM Role for the task
CPU	cpu	Requested vCPU to run Fargate		256
MEMORY	memory	Requested memory to run Fargate		512
NUMBER	number, n	Number of tasks		1
PRIVATE_REGISTRY_USER	user	PrivateRegistry Username
PRIVATE_REGISTRY_PASSWORD	password	PrivateRegistry Password
KMS_CUSTOMKEY_ID	kms-key-id	KMS custom key ID for SecretsManager
ASSIGN_PUBLIC_IP	assign-pub-ip	True: Assigns public IP		true
READONLY_ROOOTFS	readonly-rootfs	Make the root file system read-only		false
ASYNC	async	True: Does not wait for the job done		false

for the stop command:

Environment Variables	Argument	Description	Required	Default
REQUEST_ID		Resources ID to be stopped	*
TASK_ARN	task-arn	Task ARNs to be stopped

Samples

$ export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
$ export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
$ ecs-task-runner run alpine --entrypoint env
{
  "container-1": [
    "2018-09-23T11:42:01+09:00: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
    "2018-09-23T11:42:01+09:00: HOSTNAME=ip-172-31-40-206.us-east-1.compute.internal",
    "2018-09-23T11:42:01+09:00: AWS_DEFAULT_REGION=ap-northeast-1",
    "2018-09-23T11:42:01+09:00: AWS_REGION=ap-northeast-1",
    "2018-09-23T11:42:01+09:00: HOME=/root"
  ]
}

Using an assume role with MFA token

$ unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY
$
$ export AWS_PROFILE=project-alpha-dev
$ export AWS_ASSUME_ROLE=arn:aws:iam::123456789012:role/rolename
$ export AWS_MFA_SERIAL_NUMBER=arn:aws:iam::123456789012:mfa/mfaname
$ export AWS_MFA_TOKEN=123456
$
$ ecs-task-runner run alpine --entrypoint env
{
  "container-1": [
    "2018-09-23T11:42:01+09:00: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
    "2018-09-23T11:42:01+09:00: HOSTNAME=ip-172-31-40-206.us-east-1.compute.internal",
    "2018-09-23T11:42:01+09:00: AWS_DEFAULT_REGION=ap-northeast-1",
    "2018-09-23T11:42:01+09:00: AWS_REGION=ap-northeast-1",
    "2018-09-23T11:42:01+09:00: HOME=/root"
  ]
}
$ echo $?
0

This app will return with an exit code of the containers:

$ ecs-task-runner run alpine --entrypoint sh,-c --command "exit 255"
{
  "container-1": []
}
$ echo $?
255

Run a container asynchronously with --async flag:

$ ecs-task-runner run nginx --async -p 80 --security-groups sg-public-80-abcdefg
{
  "RequestID": "ecs-task-runner-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "Tasks": [
    {
      "PublicIP": "xx.xxx.xxx.xx",
      "TaskARN": "arn:aws:ecs:us-east-1:xxxxxxxxxxxx:task/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
    }
  ]
}

To stop the asynchronous tasks:

$ ecs-task-runner stop ecs-task-runner-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
{
  "container-1": [
    "2018-09-23T22:34:37+09:00: zzz.zz.z.zzz - - [23/Sep/2018:13:34:37 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0 Safari/537.36\" \"-\""
  ]
}

Usage

With arguments:

ecs-task-runner -a AKIAIOSFODNN7EXAMPLE -s wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY run sample/image

With environment variables:

export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
ecs-task-runner run sample/image

With ECR shortened image name:

export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
ecs-task-runner run --force-ecr my-ecr/image

With a private registory:

export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
export PRIVATE_REGISTRY_USER=user
export PRIVATE_REGISTRY_PASSWORD=password
ecs-task-runner run sample/secret

With the docker container:

export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
docker run --rm -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY pottava/ecs-task-runner sample/image

Troubleshooting

If the command returns non-zero exit code, you can try --extended-output to analyze the cause of failure.

export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
ecs-task-runner run sample/image --extended-output