It’s been 2 years since I first took a look at the Melbourne region (ap-southeast-4) (my home region) from a developer’s perspective, and given the length of time, and the huge volume of features that have been rolled out to the Melbourne region over that time, I decided that I should give it a try for my latest personal project. I’m using the same SAM template I discussed in my previous post, and checking the availability of the services I needed, things were looking pretty good. I thought it would be a simple process of using the same scripts I crafted for deployment to the Sydney region (ap-southeast-2), but replacing the region with ap-southeast-4. This was working fine for the most part, ap-southeast-4 now supports the services I use:
- Lambda
- API Gateway
- DynamoDB
- Cognito
- CodePipeline
- CodeBuild
- CodeDeploy
I was going fine until I attempted to connect my codepipeline to my github repository, at which point I discovered that ap-southeast-4 does NOT currently support CodeConnections. As described in this article:
AWS CodeConnections is available in all regions where AWS CodePipeline is supported except in the Asia Pacific (Hong Kong), Africa (Cape Town), Middle East (Bahrain), Europe (Zurich), Asia Pacific (Jakarta), Asia Pacific (Hyderabad), Asia Pacific (Osaka), Asia Pacific (Melbourne), Israel (Tel Aviv), Europe (Spain), and Middle East (UAE), and the AWS GovCloud (US-West) Regions.
Hrmmmmmm…. That’s quite a few regions that don’t have AWS CodeConnections. So what’s the alternative? I guess there are 2 alternatives: the simplest would be to simply use an S3 bucket for your source, but this just seems wrong to me, so the other alternative is to create your pipeline in a region that can use CodeConnections, and then use it to deploy to the other region. So I decided to take this challenge on.
Cross Region CodePipelines
Because of the way CodePipeline was designed, it is absolutely capable of deploying code across different regions. When using the pipeline action, you simply need to specify the region.
...
ActionTypeId:
Category: Deploy
Owner: AWS
Provider: CloudFormation
Version: '1'
Configuration:
...
InputArtifacts: ...
Region: ap-southeast-4
...
Of course that also means having a bucket and corresponding KMS Key provisioned in each region you target to support the
InputArtifacts
, and then granting the deployment role the permissions required to use those keys and buckets. This
turned out to be a major refactoring of the way my pipeline was set up to work.
The first pain point was that you can no longer use the Fn::ImportValue
intrinsic function to reference things
like IAM roles that have been provisioned by a cloudformation script and deployed to another region, even
though IAM is global. Fortunately I’m a big believer in creating helper scripts, and already had a powershell
script that deployed my cloudformation stacks. I adapted the script to call aws cloudformation describe-stack
and extract the Outputs of the stack that created the IAM role I needed, and passed it as a parameter to the
stacks that required it. This is fine, but my powershell scripts are now diverging from very simple
shortcuts to the aws cli, and moving into an orchestration of complex moving parts. While there may be an
advantage to a more loosely coupled approach, there are also now no guarantees if someone accidentally
deletes the IAM stack.
The next annoyance came when I realised that now I had to give the codepipeline and codebuild service accounts permissions to S3 buckets (and the corresponding KMS keys) in each region I wanted to use. Sure this time I only
have 2 regions, Sydney and Melbourne, but what if I wanted to add another down the track? It means re-writing my
cloudformation. I tried using a naming convention for my S3 buckets, but this didn’t solve the issue, I also tired using
Fn::ForEach
and again no dice. This is honestly the closest I’ve ever come to seeing the value in CDK, and
will most likely re-write it in CDK if I have to add more regions, but given I only need 2 regions (for now) I
will stick with cloudformation. So now it looks like this:
Parameters:
...
ArtifactKMSKeySydArn:
Type: String
ArtifactKMSKeyMelArn:
Type: String
...
CodeBuildServicePolicy:
Type: AWS::IAM::Role
Properties:
...
Policies:
- PolicyDocument:
Version: '2012-10-17'
Statement:
Effect: Allow
Action:
- kms:Encrypt
- kms:Decrypt
- kms:ReEncrypt*
- kms:DescribeKey*
- kms:GenerateDataKey
Resource:
- !Ref ArtifactKMSKeySydArn
- !Ref ArtifactKMSKeyMelArn
...
And similarly for S3 buckets, and also for the codepipeline service role.
You also need to list all of the artifact stores in the pipeline resource like so
Pipeline:
Type: AWS::CodePipeline::Pipeline
Properties:
...
ArtifactStores:
- ArtifactStore:
Location: !Ref ArtifactBucketSydName
Type: S3
EncryptionKey:
Id: !Ref ArtifactKMSKeySydArn
Type: KMS
- ArtifactStore:
Location: !Ref ArtifactBucketSydName
Type: S3
EncryptionKey:
Id: !Ref ArtifactKMSKeySydArn
Type: KMS
All in all, this is a bit of pain to go through especially considering I never really wanted to do a Cross Region pipeline but was forced into it by the lack of CodeConnections in my chosen region. That said, I am glad to have gained an understanding of what is required to implement Cross Region pipelines.
Additional Bucket Issue
Once I had my cross region pipelines up and running, I was still getting an issue when I deployed my cloudfront website. This should be straight forward, I’ve done it many times before, and it’s fundamental to hosting a SPA based application. Unfortunately I was getting an error stating:
IllegalLocationConstraintException
The ap-southeast-4 location constraint is incompatible for the region specific endpoint this request was sent to
it took me a while to figure this out, but ultimately the issue is that for any region created since 2019 (ap-southeast-4 was launched Jan 2022), you need to specify the full bucket name when referring to the bucket, otherwise the DNS won’t resolve it. This was an issue when setting up the Origin for the static website. My cloudformation script needed to be changed from this:
StaticWebsiteDistribution:
Type: AWS::CloudFront::Distribution
Properties:
DistributionConfig:
DefaultRootObject: index.html
Origins:
- DomainName: !GetAtt StaticWebsiteBucket.DomainName
Id: StaticWebsiteOrigin
...
which rendered the domain name as: <bucket_name>.s3.amazonaws.com
to:
StaticWebsiteDistribution:
Type: AWS::CloudFront::Distribution
Properties:
DistributionConfig:
DefaultRootObject: index.html
Origins:
- DomainName: !GetAtt StaticWebsiteBucket.RegionalDomainName
Id: StaticWebsiteOrigin
...
ensuring the domain name becomes <bucket_name>.s3.ap-southeast-4.amazonaws.com
. Of course this still works for
regions created before 2019, so there is no downside to making this change across all cloudformation scripts regardless of which region you are hoping to deploy to.
Conclusion
This was way more effort than I originally thought it would be, and am left wondering why certain regions are excluded from CodeConnections, and if they will ever be added to these regions. I just hope this helps someone else out there.