Custom Build Environments - AWS CodeBuild

Part 1 - The drivers behind custom build images

AWS CodeBuild offers a number of curated images that offer a wide range of build tools. This includes all the usual suspects such as python, go, node, java and of course my personal language of choice dotnet along with powershell core. Until very recently, my approach was to simply choose the appropriate image (usually the latest ubuntu (standard 7.0 at the time of writing)), then add any additional dependencies I required to build in the buildspec.yml file in the install phase.

install:
    runtime-versions:
      dotnet: 6.0
    commands:
      
      #Install Amazon Lambda Tools  
      - dotnet tool install Amazon.Lambda.Tools -g
      - export PATH="$PATH:/root/.dotnet/tools"
      # install any other tools/dependencies
      - ...

I would often usually use apt-get to perform an upgrade and update ensuring I had the latest bug fixes and security patches before I started my build. However, recently this approach cost my team valuable time when a component I was relying on completely changed the way it was distributed. This caused our pipeline to break and prevented the team from releasing any other updates while myself and another team member attempted to find a fix. What was even worse was that this same issue caused another one of our dependencies to break, and so we were in a position where we had to deliberately hard code the solution to retrieve an older version of the component, while we waited for the other open source library to also respond to the change. This essentially meant fixing it twice.

This kind of disruption was extremely undesirable, and it got me thinking about my approach in general. I had a feeling for a while that something like this could happen. Afterall, when you are always getting the latest version of any components and dependencies required to build and/or test your application, then you are not fully in control of the end result. Who’s to say that one of your components won’t make a breaking change. Dependency pinning will definitely help here, but even doing the apt-get upgrade/update could introduce a version of a tool that causes an issue, or even worse, has a zero day security vulnerability. Even if you think this is being overly paranoid, the fact that as time goes by, your builds get longer and longer as there are more binaries on your image that require patching, is also a concern. This exact same cycle is repeated over and over for every build you perform, and when you are paying for the build agent by the second, you really want to be doing only what’s absolutely necessary for each build.

This got me thinking about the benefits of using a custom build image. This is an image I maintain in my own ECR repository. This would have a few advantages.

1. I could be very specific about exactly what was installed. The AWS CodeBuild images contain a lot of tools I don’t need, and this makes them quite large.
2. I have full control over the version of everything installed making each build a lot more repeatable and reliable.
3. I wouldn’t have to waste time in the install phase adding additional tools.
4. I could maintain a history of images in ECR so that if I found a problem with a new version of my build environemt I could easily switch back to an older version.

Of course it does come with some overhead in that I would be responsible for ensuring it is maintained and up to date, but that’s a cost I’m willing to bare.

In part 1 of this post, I just wanted to discuss the motivation for using custom build images. This blog post from AWS Describes the basics of how to do this to build a php application. In Part 2, I will outline a fully working example for dotnet.