Unit testing DynamoDB Code

Specifically using the .Net Object Persistence Model

I really love the high level abstraction you get from using the object persistence model for DynamoDB in .Net. It allows for a very natural feel for a .Net developer to use a set of attributes to mark up simple .Net objects, and to have all of the underlying implementation details taken care of for you. I use the object persistence model regularly in many of my projects.

Recently, I was looking at the unit test code coverage for one of my projects which is almost 3 years old now, and noticed that certain parts of my code had very little code coverage. I dug a little deeper only to find that it was a class that made use of the object persistence model, and I immediately started to think of how I would go about mocking out the DynamoDBContext. “Easy” I though, “it implements the IDynamoDBContext, I can simply pass in a mock version of that to whatever uses it, and start testing my class”.

I was struggling to see why this hadn’t been done sooner in the project, and was busy preparing a lecture for my team on the importance of unit testing their code. I began refactoring accordingly, but was suddenly stopped in my tracks…

It wasn’t possible to create a type of AsyncSearch to return from the QueryAsync method. Suddenly, I realised why this class had not been previously unit tested. It wasn’t possible to mock out the interaction with DynamoDB. To me this represented significant risk for the project. Convinced I would’t be the only poor sod to have encountered this limitation I started googling. Sure enough eventually I found a GitHub issue with some suitably frustrated people experiencing the exact same issue dating back to 2017. It seems as though this may have been an oversight in the original implementation of the object persistence model, as many suggested fixes would have caused unacceptable breaking changes. Eventually an acceptable fix was found and merged into the library about a year ago now. It’s probably not how one would design this if you had a clean slate, but it fulfills the purpose. By having a protected default constructor for the AsyncSearch class it is now possible to subclass AsyncSearch and use that subclass and simulate the kinds of returns that you might expect from DynamoDB.

This allows you to test your client class under all conditions such as expected results, unexpected results (e.g. no results), or even testing the call throwing an exception. I have demonstrated how to use this here in my AWSCodePipelineExample project by unit testing a small part of my LocationsController class, but the cruclial code is

    public class MockAsyncSearch<T> : AsyncSearch<T>
    {
        private List<T> _data;
        public MockAsyncSearch(IEnumerable<T> data) : base()
        {
            _data = new List<T>(data);
        }

        public override Task<List<T>> GetRemainingAsync(CancellationToken cancellationToken = default(CancellationToken))
        {
            return Task.FromResult(_data);
        }
    }

which is then used in the unit tests to set up the mock return for the QueryAsync<Location>(...) method like this

    // Arrange
    var contextCreator = new Mock<IDynamoDbContextCreator>();
    var dynamoDbContext = new Mock<IDynamoDBContext>();

    var locationId = Guid.NewGuid();

    dynamoDbContext.Setup(d => d.QueryAsync<Location>(
        It.IsAny<object>(), 
        It.IsAny<QueryOperator>(), 
        It.IsAny<IEnumerable<object>>(), 
        It.IsAny<DynamoDBOperationConfig>())).Returns(
            new MockAsyncSearch<Location>(
                new List<Location>{
                    new Location
                    {
                        LocationId = locationId,
                        LocationName = "Test Location"
                    }
                }
            )
        );

Of course, unit testing will only take you so far. Given you mock the return results, unit tests won’t help you validate if your queries are working correctly on real data stored in DynamoDB. To do this you need to run integration tests against real data. The easiest way to do this is obviously against DynamoDB itself, but this of course requires deploying your application to AWS, and setting up access keys etc… but this is how I’ve been testing the majority of my DynamoDB code to date. If you’d like the ability to execute integration tests locally, you can use DynamoDB Local which will do a very good job of emulating DynamoDB either on your local computer or in a Docker container. There are some caveats to be aware of, but should be suitable for most integration testing purposes.

I am very relieved that this issue has been addressed by the team at AWS, and will be making extensive use of this in future projects.