Reduce traffic between deploy-agent and deploy-service

[mingzhaodotname]

For each deployment, there are about 8 deployment stages/steps (from PRE_DOWNLOAD to SERVING_BUILD), and for each step the deploy-agent talks to deploy-service to find out what's the next step to do. So, there are at least 8 rounds of traffic for each deployment, which also creates more database access.

Another way to handle each deployment is to let the deploy-agent handle all the deployment stages/steps together and then report back to the deploy-service. In this way, there will be only one or two rounds of communications for each deployment. This would give the deploy-service and also the database less load.

This might be helpful for adding new features, e.g.
#491, where there are concerns about hitting scale limit because of too many database access.

How do you think?

[lilida]

I don't think the round trips in deployment is the bottleneck from our observation. Deployments are not happening all the time. The load here is much much smaller comparing to the regular ping to check if there is a new deployment. Even the latter, from the metric we observed, it can support very large scale

[sulixu]

@mingzhaodotname there are no increased db query in that pull request. lida's concern is about how to retrieve ec2 tags from host, we don't want to make an ec2 AWS request for every ping request on each host.