Using Spot Instances as build agents, or “How to save money without really trying.”

Using Spot Instances as build agents

At Irdeto we have been working with AWS for some time. Our standard deployments are on AWS and this has led to improved visibility on costs. Of course, once you have that visibility there is always the question – “How do we reduce our costs?” We have gone down the usual routes, reserving instances, stopping unused PIOPS volumes, using a 3rd party cost analyzing tool (Cloudyn – in case you’re interested) and having frank discussions as to the cost of each solution.

As we develop our products we need certain tools, one of which is a build server which has associated build agents. The build agents are spun up for each pipeline and after the pipeline has completed the agents are spun down. Now this follows good AWS principles in that you only use the resources you need for as long as you need and then you shut them down. We costed out the shared platform services VPC that the build server and its agents run in and projected that the cost would be somewhere around $2000 per month for all our supporting services, before instance reservation. Not too bad and within our projected budget.

Fast forward a few months and things had changed somewhat – that’s ok, we expect our solutions to evolve. We had one request from our development teams and increased the number of build agent licenses (previously 6 and now 14). The type of instances in use as build agents had also changed as we needed to prioritize the speed of the pipelines over the cost. The build agents were now costing an average of around $1300 per month. This had to change without impacting the delivery capabilities of our development teams.

A review was done by the Devops team to see where we could cut costs. At this point we could see that all pipelines were using the same instance type, something which wasn’t really needed. Only one pipeline required the largest instance type to implement the steps in parallel. The other pipelines would be moved to using smaller and cheaper t2.medium instances. At this point we had saved costs but we still want to save more. Our build server was upgraded to a version allowing the use of Spot instances as build agents and we have reviewed the cost difference so far and the results look very promising.

As the cost differential between an “on-demand” c4.2xl and the current spot instance is approximately $0.32/hr, the savings can quickly increase. During 1 very busy month we had costs of $5200 just for the build agents (which were that month split between c4.2xl and c4.xl instance types), which, had we been using spot instances at that stage (and nothing else had changed), would have been reduced to around $1365 instead – quite some saving.

We are now down to a projected amount for our build agents of around $150-$200 per month (depending on workload). The reduction in costs has been around 50% from the changes we made to the pipelines after reviewing requirements, while the change to spot instances has added a further 70-80% cost saving.

 

The running cost of our management services vpc is now within the projected budget which means we are back on track. 

 

As you can see from our experience there are 2 key factors to take away from this.

  1. Don’t assume that all pipelines are equal and use the same instance type on each pipeline, review the requirements and size the build agents accordingly.
  2. If you can use spot instances do so. Build agents are a prime example of a good use for spot agents. They are stateless and if the instance is terminated by AWS before the pipeline has finished then you aren’t charged for the time used. Another instance will be created by the build server to re-run the pipeline.