It’s been 2 months now since I embarked on this journey, and I promised I would be transparent about it all on my blog… mainly to keep myself honest. I posted about the reasons here, and put a number of baselines in place. In this post, I want to give an update, and talk a little about how I got to where I am today.

For just over a year now, I have been noticing some very worrying heath issues. It started in December 2024 when I had serious asthma for about 5 weeks straight. Coughing almost constantly, out of breath regularly and resulting in me withdrawing from the Lorne Peer to Pub swim that I was actively training for. Eventually the coughing subsided but I was left with an issue that persisted. Even the mildest amount of physical exhertion like walking would quickly result in my heart rate spiking (often to over 130bpm), becoming out of breath and sweating profusely.

Generative AI coding assistants have been gaining in popularity recently, so I have been trialling a few different AI coding agents myself. Roo Code, Amazon Q Developer, Github CoPilot. So far the experience has been mixed, with certain scenarios costing me more time and energy in fixing the issues they have created than I would have if I’d just done it myself, and at other times, seeing the AI do things a better way than I would have thought to do it. One thing though is clear, you really want to isolate your AI coding agents from the rest of your system. Once you give an agent power to change code and execute commands, you can get into some trouble if you’re not careful, and there have been stories of AI agents wiping entire hard drives. Then there’s always the questions of leaking data. If the AI has the ability to read your code files, what’s stopping it from potentially reading other things on your system. Because of this, it is essential that any serious AI coding should be performed in a completely isolated environment. For Visual Studio Code the best way to achieve this is to use Dev Containers. In this post I’ll go over a few tips and tricks I’ve learnt to set up a Dev Container that is ready for developing serverless AWS applications.

It’s been 2 years since I first took a look at the Melbourne region (ap-southeast-4) (my home region) from a developer’s perspective, and given the length of time, and the huge volume of features that have been rolled out to the Melbourne region over that time, I decided that I should give it a try for my latest personal project. I’m using the same SAM template I discussed in my previous post, and checking the availability of the services I needed, things were looking pretty good. I thought it would be a simple process of using the same scripts I crafted for deployment to the Sydney region (ap-southeast-2), but replacing the region with ap-southeast-4. This was working fine for the most part, ap-southeast-4 now supports the services I use: Lambda API Gateway DynamoDB Cognito CodePipeline CodeBuild CodeDeploy I was going fine until I attempted to connect my codepipeline to my github repository, at which point I discovered that ap-southeast-4 does NOT currently support CodeConnections. As described in this article: AWS CodeConnections is available in all regions where AWS CodePipeline is supported except in the Asia Pacific (Hong Kong), Africa (Cape Town), Middle East (Bahrain), Europe (Zurich), Asia Pacific (Jakarta), Asia Pacific (Hyderabad), Asia Pacific (Osaka), Asia Pacific (Melbourne), Israel (Tel Aviv), Europe (Spain), and Middle East (UAE), and the AWS GovCloud (US-West) Regions.

Sometimes it pays to think twice about the recommended approach to achieve your ends. Especially when it comes to performing major upgrades to your applications most crucial system… the database.

Day 3 Keynote with Dr. Swami Sivasubramanian After the heavy focus on Generative AI in Adam Selipsky’s keynote the morning before, I was kind of feeling a bit sorry for the Vice President of Data and AI. Dr Swami began by talking about the human/AI relationship being a mutually beneficial symbiosis. Still not quite sure how I feel about that analogy. He then went on to talk about Ada Lovelace, and credited her with predicting the emergence of generative AI.

Day 2 - Adam Selipsky Kenote Early start to walk to the Venetian for a coffee and some breakfast. I usually don’t function without a coffee so the 20 minute walk between Caeser’s and the Venetian was a bit of a challenge.

Day 1 - First Day of Conference Content RDS Telstra dedicated session My first session was a special dedicated session targetted at Telstra and our use of RDS. It consisted of myself and 2 internal Telstra executives, and 8 people from the RDS team. We did a round of introductions, and then we they began to dive deep into some of the issues Telstra has. I had never been exposed to any of our internal applications, and so it was a real revelation to me to learn the extent of our data footprint. The sheer number of databases, including many different database engines and the enormous volume of the data we store. In retrospect, it shouldn’t have been that surprising. A Telco the size and age of Telstra would inevitably have many different applications and teams that maintain them. The real challenge as we move towards our goal of 90% of workloads running in the public cloud is inevitably the cost. The AWS team offered some suggestions, and promised to do a deep dive with the relevant engineering groups to understand the various challenges. They did have one really big announcement for us that they had literally announced publicly minutes before the meeting. RDS will add DB2 to its list of supported database engines. This is great news for big enterprise clients that have data in DB2. As per usual, there are some caveats, for instance, while Java based stored procedures are supported, RDS won’t initially support Cobol stored procedures.

Orientation and Welcome Drinks As I have every other year I’ve attended AWS Re:Invent, I’ll be blogging my journey and my thoughts as I go. The first challenge with re:Invent is getting there. From Melbourne it’s a 14.5 hour flight to Los Angeles, and then a 1 hour flight to Las Vegas. When all of the additional time (travel to the airport, check-in, waiting for the flight to board, getting lost at LAX, and then waiting for the connecting flight) is taken into account, it is in excess of 24 hours worth of travel. I was lucky enough to be let into the Qantas lounge by my collegue Angie, and we whipped together a quick video of our initial thoughts and expectations.

I really love the high level abstraction you get from using the object persistence model for DynamoDB in .Net. It allows for a very natural feel for a .Net developer to use a set of attributes to mark up simple .Net objects, and to have all of the underlying implementation details taken care of for you. I use the object persistence model regularly in many of my projects.

AWS CodeBuild offers a number of curated images that offer a wide range of build tools. This includes all the usual suspects such as python, go, node, java and of course my personal language of choice dotnet along with powershell core. Until very recently, my approach was to simply choose the appropriate image (usually the latest ubuntu (standard 7.0 at the time of writing)), then add any additional dependencies I required to build in the buildspec.yml file in the install phase. install: runtime-versions: dotnet: 6.0 commands: #Install Amazon Lambda Tools - dotnet tool install Amazon.Lambda.Tools -g - export PATH="$PATH:/root/.dotnet/tools" # install any other tools/dependencies - ... I would often usually use apt-get to perform an upgrade and update ensuring I had the latest bug fixes and security patches before I started my build. However, recently this approach cost my team valuable time when a component I was relying on completely changed the way it was distributed. This caused our pipeline to break and prevented the team from releasing any other updates while myself and another team member attempted to find a fix. What was even worse was that this same issue caused another one of our dependencies to break, and so we were in a position where we had to deliberately hard code the solution to retrieve an older version of the component, while we waited for the other open source library to also respond to the change. This essentially meant fixing it twice. This kind of disruption was extremely undesirable, and it got me thinking about my approach in general. I had a feeling for a while that something like this could happen. Afterall, when you are always getting the latest version of any components and dependencies required to build and/or test your application, then you are not fully in control of the end result. Who’s to say that one of your components won’t make a breaking change. Dependency pinning will definitely help here, but even doing the apt-get upgrade/update could introduce a version of a tool that causes an issue, or even worse, has a zero day security vulnerability. Even if you think this is being overly paranoid, the fact that as time goes by, your builds get longer and longer as there are more binaries on your image that require patching, is also a concern. This exact same cycle is repeated over and over for every build you perform, and when you are paying for the build agent by the second, you really want to be doing only what’s absolutely necessary for each build.

This week I had the awesome opportunity to represent Telstra Purple at the AWS Public Sector Symposium. This is an annual event run by AWS specifically targeted at the needs of public sector agencies. While a big part of my time there was spent manning the Telstra stand attempting to explain what Telstra Purple in fact was in relation to the monolith of Telstra, I did also manage to get to some sessions.

When I wrote Part 1, and Part 2 of “How to Productionize an AWS .Net Serverless Application”, I thought that would be it. I felt I had covered most of the generic activities that can be done, and that anything beyond that point would need to be application specific. Over the past few months, I’ve realised there are still few very specific yet improtant things I’d left out. As a result, I give you “Part 3”… who know’s, there may be a “Part 4” down the track.

If you are writing a serverless application on AWS that requires a relational database backend, then you’ll no doubt be using Aurora Serverless (v1 or v2) in either the MySQL or PostgreSql flavours. One challenge with using a relational database is ensuring that the database schema is appropriately maintained as you develop your application. The necessity to specify a very strict schema can often work at odds with an incremental/agile approach to application development. As such you need to be able to reliably take your database from an old version of the schema to a new version of the schema before you start using new code against the database. Ideally this should be integrated into your CI/CD pipeline to ensure that it is repeatable and reliable. Using AWS Lambda, and a .Net library called DbUp, you can achieve a schema upgrade process controlled by commiting scripts to your source cotrol repository. In this post, I will set up the basics of a working approach. I will continue to explore and improve this approach over upcoming posts.

There was much rejoicing when AWS finally announced the long awaited melbourne region ap-southeast-4 a few months ago. For many years Azure has boasted a Melbourne “region”, but without the support of “Availbility Zones”. AWS hold each region to a higher standard, in that a region MUST have a minimum number of availability zones. This minimum number used to be 2, but has been increased to 3. The Melbourne region has 3 availability zones. At the time of writing Azure still doesn’t have support for availability zones in the Melbourne region, but plans to bring this support online soon. These subtleties are often lost on decision makers, or at best they have chosen latency over availability.

In Part 1, we took the serverless.AspNetCoreWebAPI blueprint, and began to implement some best practice changes in order to make it a production ready workload. In Part 2, I’ll show you some more things I like to do to ensure your serverless application is ready for prime time.

The AWS .Net serverless blueprints are a great way to get started coding your serverless application. My personal favourite is the serverless.AspNetCoreWebAPI which sets up an AWS Web API complete with controllers to show you how to start coding. For a quick proof of concept, or some internal experimentation, this is a perfect starting point. However, before you deploy your fancy new serverless application into a production environment, and expose it to your end users, there are a few things you should consider doing to harden or “productionize” your application.

On Wednesday the 31st of August, I was given the privilege of speaking at the Melbourne AWS User Group on one of my favourite topics, Performance tuning .Net Lambdas. I thought I would take the opportunity to post a link to the talk, as well as the powerpoint deck, and dive into some of the details in my analysis.

I mentioned briefly in a past blog post about the AWS dotnet lambda serverless template serverless.ASPNetCoreWebAPI. This is a super easy way for a .Net developer to either migrate an existing Web API into AWS Lambda, or even to start a new API without having to dive too deeply into understanding the intricacies of AWS Lambda and API Gateway. Recently AWS have even added the new .Net annotations framework and a new template serverless.AspNetCoreMinimalAPI which makes it even easier to just start coding up your .Net Web API with reckless abandon. At first glance, these approaches are very tempting to use, and seems like the best way forward for producing APIs in lambda if you’re a .Net developer, but a detailed analysis raises a few awkward questions that deserve to be answered. There are also some really good reasons to use Web APIs in lambda.

Setting up Isolated Stacks One thing I love about writing serverless software is the idea of scale-to-zero. Basically, when you are not using a serverless application, you are not paying for any compute resources whatsoever. While this is a boon for apps that have spikey and unpredictable loads, it also has a very real benefit during development. It means that developers can realistically spin up their own isolated environment in order to validate their work.

AWS CI/CD By Example A while back I wrote an article about AWS CI/CD tooling, and the realisation that forced me to re-evaluate my opinions. Initially I thought it was an inferior or limited set of tooling, however, I came to realise that it was in fact just an “Aggressively opinionated framework for safely deploying highly available microservices across the globe at scale to AWS”. I have since presented on this at the Melbourne AWS User Group. In that talk, I demonstrated a powerful feature, “Automated Rollback”, using an example pipeline that touches many other things I discuss in my blog post. This example pipeline was created as a way to demonstrate many of the concepts I talked about in the original post. It is my honour to announce that I have open sourced this project and in this blog post I’d like to take you through a few of the projects features.

ReadyToRun’s Impact on Cold and Lukewarm Starts In a previous post I discussed an issue with .Net Lambda functions that I labelled “Lukewarm starts”. This effect is due to the JIT compiled nature of the .Net framework. I also discussed a number of possible ways of alleviating this issue. One of which was using the “ReadyToRun” compilation option that was introduced as part of .Net core 3.1. In this article, I’d like to dive into some details about the impact this feature has on both cold and lukewarm starts.

.NET Lambda Lukewarm Starts The Problem One of the biggest barriers to the adoption of serverless is the cold start performance hit. There are many blog posts about AWS Lambda cold start performance and how to improve it. In this blog post I want to raise a related issue that is specific to .NET Lambdas, which I have labelled “lukewarm starts”.

AWS CI/CD Tooling – I’ve Seen the Light Although I am a big fan of almost all things AWS, I have always been a little critical of their CI/CD tooling, CodeBuild, CodePipeline and CodeDeploy. Having used many CI/CD tools such as Azure DevOps (formerly VSTS), Jenkins, Hudson, BuildKite, and a few more, I felt that I was in a position to judge AWS’s offerings. Until this week, I would have made statements like “their offering lacks the maturity of their competitors”, or “there are some significant gaps”. This week, however, while discussing a fellow colleague’s approach to AWS CodePipeline, I had what can only be described as a “Come to Jesus” moment… or perhaps a “Come to AWS” moment.

Day #3 The fact that AWS Re:Invent is spread out over 5 separate casinos, makes it really challenging to get to the content you want to see. The alternatives are either book sessions only in 1 hotel, which is fine if all the sessions you want to see are there, and you can get reserve seating, or wear yourself out treking between casinos using either the monorail, the conference shuttles, or walking… I choose the latter. First session at the Mirage, then off to the Aria for second and third session (with a quick bite to eat in between), then the MGM Grand for the final session of the day. Unfortunately, my laptop ran out of battery in the middle of the third session, so my summary of the last 2 sessions will be from memory rather than well structured notes.

Day #2 Keynote #1 Andy Jassy [watch](https://www.youtube.com/watch?v=7-31KgImGgU] I’ve often found keynote sessions at these big conferences (whether AWS or Microsoft) to walk a fine line between a necessary marketing exercise, and a full on revival crusade. Having Andy’s keynote points framed by covers of popular songs by the in house re:Invent band, certainly helped to blur that line further. The only thing really missing was someone up the back yelling “Amen brother” everytime Andy drilled home a salient point. My biases aside, Andy did his best to weave the necessary product announcements into a structured 6 point story on public cloud transformation. Don’t ask me what the 6 points exactly were, I got the first 2; Leadership and Broadest and deepest platform (no guesses who has that), then the rest seemed to get lost in a sea of product announcements.

Back for my Second Re:Invent It’s an honour to be back representing Telstra Purple at Re:Invent 2019, and just like last year, I’ll be posting each day, going through all the highlights from AWS’s premier event from my own unique perspective. This year the Purple contingent are all staying at the New York New York hotel.

Repository Not Found If, for whatever reason, you find yourself trying to connect to AWS CodeCommit repositories hosted in more than 1 AWS account from a Windows machine, you may run into this frustrating issue.

Day 4 Session Summaries Keynote Straight out of the gate Werner made the statement that “Serverless is the future of application development”. I couldn’t agree more, and judging by the interest at Re:Invent, it is certainly the new “black”. The key note was a story weaving exercise as Werner described his worst day ever at AWS when they had a pre-Christmas 12 hour outage of their oracle database. He described how this event drove their approach and the way they think about systems at scale now. This then fed into a discussion of Aurora, and the decisions they have made to make it a cloud scale database. Eventually Werner arrived at one of his pet topics. Relational databases are not a good choice for the majority of workloads for a modern web scale, microservice based application.

Day 3 Session Summaries Keynote At the keynote on Day 3, there were so many machine learning product announcements that I am convinced that in a post-apocalyptic terminator-esque future if there are any historians left, they will look back on this day and realise that this was the inflection point where skynet went from being a mere possibility to an inevitability. Of course everyone was far too busy training their DeepRacers to notice or do anything about it. I keep looking over my shoulder expecting to see Arnold Shwarzenegger trying to get to the AWS Machine Learning team to destroy skynet before it happens.

Day 2 Session Summaries Managing State for Serverless Applications This session would probably have been better named “Step Functions, and how customers are using them”. While I was kind of hoping for a deeper dive (it was advertised as level 300) into the various types of state that is required in a serverless app, and a comparison of different approaches, I was happy to learn a little bit about Step Functions. So basically, they are similar to Logic Apps on Azure, but without the nice editor experience, and without the connector ecosystem. Still it’s obvious the potential in using Step Functions to orchestrate flows in a serverless application. The talk focussed on a number of real customer scenarios; Basic ETL workflow, intricate deployment scenarios, integrating legacy applications, Large scale image processing and analysis, and (importantly) detecting if your boss has entered the team office, and (more importantly) whether or not you should be concerned based on his emotional state as perceived by ML.

Day 1 Session Summaries While at AWS Re:Invent 2018, I am concentrating mainly on serverless, and I’ll be keeping a daily update of the sessions I attend, and highlighting the key takeaways.

Part 1 Part 2 Cleaning up the Mess Software development can be a very messy process. You follow a path you think might work, only to realize that the class you are relying on doesn’t expose a method with the right parameters, or you suddnely see an opportunity to refactor a piece of code to make it more generic and get some code re-use out of it. You have to dive into a piece of the code that hasn’t been touched in years, and realise that the coding style has changed, and it looks so weird you just have to correct it to make sense of it. A complex feature could require a number of changes to core libraries along with corresponding unit/integration/UI tests, not to mention database scripts. If you’re doing TDD, you may have a number of Red/Green/Refactor cycles. At the end of a feature, your code (hopefully) will be elegant and readable, but the process you got there by is most likely a winding road that is anything but elegant. Don’t get me wrong, regular committing is a great thing that I strongly endorse… commit early, commit often is my motto. The great thing about git is, if you’re committed, it’s never lost.

Part 1 Embracing Rebase Rebase (in my opinion) is one of the most mis-understood and under used features of git. There are many attempts at explaining rebase, some highlighting the difference between merge and rebase, and some dire warnings against using rebase. Git, at its core is surprisingly simple and elegant, and it is from this simplicity and elegance that it derives its power. It is only when this simplicity is abstracted away with tools that hide what is really going on that complexity seemingly emerges and begins to befuddle those who haven’t attempted to understand it. As such I will try to explain rebase alluding to some of the simple and powerful concepts at its core, but without delving too much into the inner workings of git.

Introduction The Problem Git is by far the world’s most popular source control system, but most people only use a tiny proportion of its true power. (Source: https://www.xkcd.com/)

First Post on my New Blog After a few years hiatus from blogging, I have decided it is time to dust off the cob-webs and give it another go. I started blogging back in 2004, and maintained 2 seperate blogger blogs Musings of a Morbid Mind, and Daily Dribblings of a Demented Developer which I’ll be the first to admit the titles of these blogs, are essentially a ‘how-to’ guide for over using aliteration.