Hello reddit,

We are the Amazon EC2 Container Service (ECS) team.

My name is Deepak Singh and I am the General Manager of Amazon ECS.

Today I'm joined by:

  • Chris Barclay (Product Manager)
  • Anthony Suarez (Dev Manager)
  • Kiran Meduri (Software Engineer)
  • Dan Gerdesmeier (Software Engineer)
  • William Thurston (Software Engineer)
  • Jessica Chen (Software Engineer)
  • Uttara Sridhar (Software Engineer)
  • and Samuel Karp (Software Engineer).

We are here to answer any technical questions you have about Amazon ECS. We’ll start answering questions at 11:00 AM PST for as long as possible.

PROOF: https://twitter.com/awscloud/status/591297651620114432

UPDATE: Love the questions. Will keep answering for another 30 minutes.

UPDATE: That's all for now. Thank you for all the questions. We had a blast answering them.

Comments: 171 • Responses: 41  • Date: 

dorfsmay21 karma

There is a bug currently in the console, if you try to "Add Container definition" with a small screen (13 inch laptop), the window to add the container is too low, and we cannot access the buttons to complete the form. I have send feedback but haven't heard back.

What's the best way to feed that back and get it fixed?

AmazonWebServices23 karma

Thank you. We are aware of this bug and are in the process of fixing it and will be rolling it out soon.

nivertech16 karma

Q1. Any plans to integrate ECS with Lambda? I.e. ability to map Lambda event sources to ECS task/service instead of Lambda function?

Q2. Any plans to implement cron-like scheduler for ECS tasks, either native or via external schedulers like AirBnB Chronos?

AmazonWebServices14 karma

Q1. That does sound super cool. We'll take that back to the Lambda team.

AmazonWebServices9 karma

Q2. We have a Mesos Scheduler Driver than allows you to use Chronos with ECS (https://github.com/awslabs/ecs-mesos-scheduler-driver). Note that this is just a proof of concept to show how you can do this. We've heard this request from others customers as well and are looking at how we can make this even easier.

cahaseler8 karma

Welcome to IAMA. We're happily using EC2 for a couple of our behind the scenes bots here - so thanks!

Other than the IAMA bots, what's the coolest thing going on in your datacenters? What kinds of crazy technology are you using to run what feels like half of the modern internet?

AmazonWebServices9 karma

Hi, Thank you for your question. There are a number of cool things going on in our datacenters. Instead of listing all of them, let me point you to a couple of talks from the last re:Invent. The first one is James Hamilton's talk on Innovation at Scale: https://www.youtube.com/watch?v=JIQETrFC_SQ and the other one is a talk on some of the core distributed systems that we use at AWS: https://www.youtube.com/watch?v=QVvFVwyElLY

dulax7 karma

What kind of monitoring is available today for ECS and can you tell us what will be coming soon?

Also interested in seeing a dashboard which tells me where my resources are and what they're up to. At scale, any of the docker management solutions will need to make it simple to track patterns.

AmazonWebServices7 karma

Thanks for your feedback. You can send custom metrics to cloudwatch, e.g. from cAdvisor today. We've also heard from customers that they are including a log pulling and metrics generation container in their Task Definitions

We are also interested in the kinds of metrics you would be interested in and how you would use them. If you could post these on the ECS forums that would be super awesome (https://forums.aws.amazon.com/forum.jspa?forumID=187)

ASTU107 karma

Amazon EC2 container service team, what's your favourite cheese?

AmazonWebServices10 karma

Lots of folks here seem to like Harvati. Brie and Gouda are right on the heels. Jeff Barr seems to like smelly ones and Sam likes Beecher's Flagship.

dorfsmay7 karma

Is there a way to get the cluster to scale on demand, à la beanstalk?

If not, any plan for the the near future?

AmazonWebServices7 karma

Multi-container Docker environments for Beanstalk leverage ECS. So you can continue to use that. But if you want to use ECS, you can use AutoScaling to scale your fleet of EC2 instances. Keep an eye out on our blog for examples of how you can do that.

dorfsmay7 karma

Multi-container Docker environments for Beanstalk

Wow! How did I miss this:


If you're going to use ECS anyway, are there any advantage to Docker in Beanstalk rather than ECS?

AmazonWebServices6 karma

Elastic Beanstalk is what you want to use of you want the simplicity of deploying applications within the Beanstalk application model. If you are running multiple applications across a fleet of instances or want more control over the application architectre, ECS will give you more flexibility.

kore_sar3 karma

What's your blog URI?

kore_sar6 karma

Hi. Thank you for the great service, guys,

First of all, I'd like to tell that I had to wake up at 4 am in order to ask a bunch of questions. We like ECS very and are thinking of moving our architecture to Microsevices.

Question: Auto Scaling Group

It can spin up and shut down EC2 instances. Will ASG be able to spin up and shut down containers (TaskDefinitions) dynamicaly? When?

Related StackOverflow question I asked: http://stackoverflow.com/questions/29737034/does-aws-ecs-support-per-container-dynamic-scalability

AmazonWebServices4 karma

Thank you for getting up in the wee hours. Certainly appreciate that.

ECS has a CreateService and UpdateService API for long running tasks. For tasks started with CreateService you can trigger UpdateService based on the metrics you care about. Keep an eye out on our blog for examples of how you can scale tasks and integrate with Auto Scaling.

kore_sar3 karma

I quickly browsed the docs for CreateService and UpdateService API. I am probably wrong, but these API calls

Runs and maintains a desired number of tasks

The key word is desired. I do not know what is my desired number because I haven't met the high load yet. When my containers gets hot on CPU/RAM/Bandwidth the desired number increases. Otherwise, the number of running TaskDefinitions should decrease.

Am I missing something?

AmazonWebServices3 karma

You can start with a number that you think is reasonable, but using CloudWatch (for example) you can trigger the UpdateService API to increment the number of tasks based on the metric(s) you care about.

set_uk5 karma

I have 2 concerns about ECS. 1) How do you ensure optimal instance sizing to ensure you arent paying for compute resources that containers arent yet using? 2) Say you stand up your services on a customer specific basis and you are running many customers. How can you ensure that ECS scales so that one set of services doesnt compromise the function of the rest of the services.

AmazonWebServices7 karma

  1. Two strategies here today. You can start with smaller instances (as long as they meet the minimum requirements of your applications) and autoscale. Alternatively you can be conservative and start with larger instances and monitor overall cluster utilization to keep the right number. We are working on providing you task and cluster-level metrics that can help you track cluster utilization: https://github.com/aaithal/amazon-ecs-agent/tree/telemetryacs

  2. I think the metrics mention above will help you achieve that (Assuming each customer is their own task). We would also be interested in more discussion on this topic, so it would be awesome if you add this question to our forums: https://forums.aws.amazon.com/forum.jspa?forumID=187

monstl2 karma

Intentional that the GitHub link goes to a fork of ecs-agent rather than the source?

AmazonWebServices4 karma

It's our fork under development

monstl4 karma

Howdy again! Are there any near-term plans for a single ELB to support mapping from a single ingress port (e.g. 443) to multiple service ports on multiple endpoints/containers (e.g. 6743, 4632, and 7322)? Thanks!

AmazonWebServices10 karma

We've heard this request from other customers. We are talking to the ELB team about this. Their GM and I have coffee together on most mornings. I suspect he is amenable to me buying him coffee everyday.

jeffreyparker4 karma

Awesome work team! I'm having a blast experimenting with ECS on elastic beanstalk.

Are there any plans to ease hosting private images, like the Google Container Registry? I've hacked together a solution using the S3-backed docker registry container, but I'd love to simply point to an S3 bucket as a "registry" and use IAM for auth.

AmazonWebServices2 karma


Looks like this is a popular request. I'd definitely recommend dropping by the ECS forums (http://amzn.to/1PYrErc) so that we can have a deeper discussion and include other customers.

kore_sar4 karma

Currently, we are using ELB+ASG+EC2. It works great. We have a dozen of instances which automatically gets more than a dozen on high load. But we would like to migrate our VPC from spinning AMI's to spinning containers.

Please, recommend a strategy, step-by-step guide, or document to migrate from AMI's to ECS Docker Containers as smooth as possible. Is there one already?

AmazonWebServices4 karma

Looks like this is an area we could improve our documentation based on the questions today. Use CreateService to launch services and then the UpdateService API to scale your tasks. Watch the AWS Compute Blog for best practices on scaling tasks.

dorfsmay2 karma

use CreateService to launch services and then the UpdateService API to scale your tasks

Can UpdateService scale up/down the number of instances?

AmazonWebServices4 karma

UpdateService only scales tasks. You can use Auto Scaling to scale your instances.

monstl3 karma

Howdy! Are there any near-term plans to support running > 1 copies of a given container on a single instance w/o specifying them as additional task definitions? Thanks!

AmazonWebServices2 karma


You can run multiple tasks from a single task definition on a single container instance as long as the resources are available.

monstl1 karma

Hm, not to devolve into support - we started a test service task with 5 requested copies against 3 container instances, and received only 3 service tasks. Could this limitation have been a result of something port mapping-related?

AmazonWebServices2 karma

If the resources are not available and you are using host port binding that is the likely problem. Check your event stream on the service to see if that is indeed the case.

rocklob03 karma

I was trying out ECS the other day, but it looks like it doesn't work with our private registry running Docker Registry 2.0. Am I doing something wrong or does ECS just not support it?

AmazonWebServices2 karma

Currently we support Docker 1.5. Docker Registry 2.0 is officially supported with Docker 1.6. We are qualifying Docker 1.6 as we speak.

klocekPL2 karma

We would like to use Amazon ECS, but we have Service-Oriented Architecture + Docker, so we have lots of internal APIs behind the scene, we have 3 environments (prod, test, dev), ~40 Services per env (foo-api, bar-api, foobar-api ...) so in current shape (ECS), we need to spend at least 18 USD per service with ELB * 40 services * 3 envs = 2160 USD per month only for ELB (not including traffic). Do you have any other plan for service discovery, not related to ELB? We really like AWS and we don't want to setup our own Mesos + Kubernetes at EC2 or move to GCloud but ECS IMHO is not SOA friendly. Can you give us some advices?

AmazonWebServices2 karma

There are many options to do this today. You can use Eureka from Netflix (https://github.com/Netflix/eureka) or take a look at AirBnB Synapse and this pull request: https://github.com/airbnb/synapse/pull/110. The pull request allows linking a Synapse container with your application and the Synapse container with Haproxy will forward requests to the location in your cluster where the targeted services are running and dynamically reconfigure as services stop and start by using our Describe APIs.

dorfsmay2 karma

json is great for software to exchange data between themselves, but it's terrible for human being: No tooling, no comments, commas hell (if you miss one, it invalidates the whole file), difficult to debug, no schema to test against etc...

Have you guys thought of adding another config language like yaml or even xml (hey, at least there's tooling for it)?

AmazonWebServices3 karma


YAML is definitely easier for humans to consume. I would recommend that you take a look at this Github repository: https://github.com/ambitioninc/container-transform that allows you to transform YAML-based Compose templates to ECS Task Definitions (and back)

rocklob02 karma

When I created an ECS instance, I noticed that my full list of security groups was not available for selection. Instead, I had to manually add the desired rules into the assigned security group. Why is this?

AmazonWebServices2 karma

We're interested in getting additional detail on this one. If you drop by the ECS forums (http://amzn.to/1PYrErc) we can definitely dig into this.

dorfsmay2 karma

With bigger images, deploy feels like they take forever.

When will AWS have its own docker registry?

AmazonWebServices2 karma

Thank you for the feature request. Could you please post this on the ECS forums (http://amzn.to/1PYrErc)? Your feedback definitely helps us prioritize features and allows other customers to weigh in.

dulax2 karma

Is there a best practice or planned support for local environments which use ECS for production/test workloads?

There was mention of eb local (elasticbeanstalk engine for local use) but that has yet to be released.

AmazonWebServices2 karma

We'd love to ask you more questions about this. Could you post this question on the ECS forums (http://amzn.to/1PYrErc)?

jcf42 karma

What is the recommended approach for IAM permission use within Docker containers, any plans to have the ecs agent be able to forward into the containers ?

AmazonWebServices2 karma

You can use IAM roles just like you do today (they are accessible from within your container). You should be aware that the IAM role is available to other containers running on the instance.

cloudninja01 karma

Is some cases this might be security issue. For instance I need have an access for one container to S3, but I have to give to all on this instance. Is there any way to prevent it?

AmazonWebServices3 karma

Remember that you own the instances that your containers are running on and can set up the appropriate privileges. But it looks like you would be interested in per-container or per-task roles. If yes, we'd love to discuss this with you on the forums (https://forums.aws.amazon.com/forum.jspa?forumID=187)

ragge_2 karma


My two main issues with ECS as currently delivered are: - I think the EC2 security group per instance model is a poor fit for a container world where many containers run on the same instance. - I believe any networking model that provides less than a unique, routable IP per container is flawed.

Are any of these issues something you're looking to address?

AmazonWebServices2 karma

Yes! :-)

ragge_1 karma

Care to expand? I'm especially interested in what you believe is an ideal networking model for containers. Overlay network (VXLAN etc)?

AmazonWebServices4 karma

The one that works!! But seriously, any answer to this question requires a lot more time than we have in this AMA. Even within our team there is more than one opinion on this question.

xrothgarx2 karma

How much of what you get to work on is open source? Are you allowed to contribute back to upstream projects?

AmazonWebServices2 karma

I'll answer for the ECS team. The ECS Agent https://github.com/aws/amazon-ecs-agent) and the ECS Init RPM (https://github.com/aws/amazon-ecs-init) are both on Github under OSS licenses. We've also contributed code to Docker (https://github.com/docker/docker/pull/5868)

Matteo12 karma

Thanks for the AMA.

Can you ELI5 what ECS is and how someone with >200 ec2 instances with multiple environments can take advantage of it?

Say I have puppetmaster, ELB, frontends, backends, NFS, RDS, Cloudfront. Would ECS make my life easier?

AmazonWebServices2 karma

On the first part of your question: ECS is ideal for people you run a number of instances, are running a mix of service-oriented and batch applications, and want to orchestrate them efficiently across a cluster.

On the second part, I am in the "yes" camp.

monstl2 karma

Howdy again! Any ETA on facilities to delete task definitions? Thanks!

AmazonWebServices1 karma

The team is looking into this.

jkmcf2 karma

Are there plans for ECS to work with dedicated servers? Due to security concerns, we can only run our software on dedicated instances, so it would be awesome to have a cluster of dedicated instances with containers managed by ECS.

AmazonWebServices3 karma

ECS will run on whatever EC2 instances you provide (we recommend instances running within a VPC). This includes dedicated instances. So you should be able to launch your own dedicated instances and use them with ECS. The key is to make sure you are running an AMI that supports Docker and is running the Amazon ECS agent.

oba112 karma

Thanks so much for providing the platform. I've been using ECS for our dev stuffs in Ireland region but we can push to pre-prod because we currently use singapore region (ap-southeast-1) as our main dc, when will ECS be available in singapore region?

AmazonWebServices1 karma

Thank you for your feedback. As with all our services we plan to expand to AWS regions and your feedback definitely helps us prioritize rollout.

xrothgarx2 karma

Are there plans to provide hosted orchestration layers for containers as a service (kubernetes, fleet) on top of ECS? Does Amazon intend to build their own?

AmazonWebServices2 karma

Does our Service Scheduler (http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_services.html) work for you? It is designed for long running services.

SafeSituation2 karma

Very excited for ECS! Any chance we can get support for zipped Dockerfile + context like on Elastic Beanstalk?

AmazonWebServices2 karma

If I understand your question correctly, you will find this blog post on using Jenkins with ECS useful: https://blogs.aws.amazon.com/application-management/post/Tx32RHFZHXY6ME1/Set-up-a-build-pipeline-with-Jenkins-and-Amazon-ECS

Neilsome1 karma

For the uninitiated here, could you please briefly introduce the EC2 Container Service?

For my question: Can it benefit users of EC2 Spot Instances?


AmazonWebServices3 karma

TL;DR> https://www.youtube.com/watch?v=zBqjh61QcB4

Absolutely. We think of ECS as a way for you to efficiently run Docker-enabled applications across a cluster of EC2 instances. We focus on the complexity of state and cluster management, and let you focus on how you want to split up your applications into containers.

And yes, you can use Spot Instances with ECS.

fenster_blick1 karma

How often do you all roll out changes to your Production service? What does your deployment process look like - do you roll out changes to a small # of customers, monitor for bad effects, and then continue with the rest of the Production environment?

Do you use any open source tools for your deployments, or is everything built inside of AWS?

I'm curious what patterns AWS use for their own services.

AmazonWebServices2 karma

We are like many other Amazon teams. This talk will give you a sense of how we do deployments (https://www.youtube.com/watch?v=KzziRHOa5X4).

Long story short: We do a lot of deployments :).

FrankPapageorgio1 karma

I realize Amazon EC2 has nothing to do with Amazon Mechanical Turk... but do any of you know a guy who knows a guy that can get me the mythical Masters Certification?

AmazonWebServices2 karma

I have no idea what that is, but I want it too :-)

WhatsUrRingtone1 karma

What are your ringtones?

AmazonWebServices2 karma

  • Deepak: Led Zepellin's "Kashmir"
  • Anthony: Whatever the default iOS ringtone is
  • Sam: UCLA fight song
  • Dan: AT&T Firefly
  • Everyone else: Silent

Pickman1 karma

I'm a layman and just stumbled in here.

What is Amazon ec2 container service?