November 13, 2020

Getting started with DynamoDB and .NET Core

In this post we are going to look at some of the components and terms used in DynamoDB. We will follow this with some .NET Core code examples of commonly used functions.

Getting started with DynamoDB and .NET Core

DynamoDB is a fully managed NoSQL database service provided by Amazon Web Services (AWS). DynamoDB provides fast performance with seamless scalability.

In this post, we are going to look at some of the components and terms used in DynamoDB. We will follow this with some .NET Core code examples of commonly used functions.

If you are interested in diving deeper inside DynamoDB, then I have a Pluralsight course that goes through working with DynamoDB with .NET Core in a lot more detail.

https://app.pluralsight.com/library/courses/aws-dotnet-core-developing-applications-dynamodb/table-of-contents

DynamoDB in a nutshell

AWS has done a great job at removing a lot of the setup and maintenance that is commonly needed when dealing with databases such as SQL. Tasks such as hardware provisioning, setup and configuration, replication and software updates are managed and done behind the scenes.

DynamoDB allows you to create tables that can store and retrieve massive amounts of data. A benefit of NoSQL databases such as DynamoDB is the amount of data the database can hold due to horizontal scalability.

Horizontal scaling is the process of adding more machines into your pool of resources to increase the number of resources you have available to you and your database. If we look at the opposite, relational databases are scaled vertically, meaning you need to add more CPU, RAM to the existing machine. As you can probably guess, vertical scaling has a limit, where horizontal scaling is almost limitless. Due to this power, DynamoDB can serve any level of requested traffic.

DynamoDB is both a key-value and document-based database. You store a collection of keys with associated values to those keys. Key-value is great for horizontal scaling at scale. DynamoDB also supports native JSON. You can write JSON documents directly into DynamoDB tables that are associated with your key/value pair.

Partition, Sort and Primary Keys

The partition key which is sometimes referred to as a HashKey is mandatory.

partition key

The Sort Key can be used in conjunction with the partition key. When setting the sort key, this allows us to query data that is related to the partition key. We can use filters such as begins with, between and greater than.

partition key

Each table that is created must have a primary key. The primary key is set by either having a partition key by itself

partition key

or we can create the primary key using a combination of the partition key and sort key. When using the partition and sort key this is called a composite primary key.

partition key

If you are only using the partition key as the primary key, then this must be unique. If you are using a combination of the partition and sort key as your primary key, then both of these values must make up a unique value, i.e we could have the same value set for the partition key, but we must have a different sort key value and vice versa.

Secondary Indexes

To efficiently find your data in DynamoDB, sometimes you need to query data using an attribute that is not your primary key or composite primary key. DynamoDB offers a way to achieve this by offering Secondary indexes. You can create a secondary index with either a different sort key based on your partition key that you set when creating the table or a completely different partition and optionally a sort key. There are two types of secondary indexes that we can create.

Local Secondary Index

A local secondary index is where you use the partition key that you have specified when creating the table. You can then choose any of your other attributes in your table to be the sort key

Global Secondary Index

A global secondary index is where you specify a different partition key and optionally a sort key from the attributes in your table.

You are able to create up to 5 global secondary indexes and up to 5 local secondary indexes per table. You need to be careful when creating secondary indexes, they come at a cost. Don’t create secondary indexes on attributes that you don’t query often. They will contribute to increased storage and I/O input/output costs that will not improve your application performance.

Scan and Query

To retrieve data from our DynamoDB table, we can use two options. The scan and Query function.

It’s important to understand how these both work and what the performance costs are for both of these.

The Scan function

When using the Scan function, DynamoDB reads all items in your table. You can optionally add a filter to match items based on what you are searching for, but all items need to be read before the filter is applied.

When you have a lot of items in your table, this operation is going to have poor performance. The costs associated to run the scan could be significant, depending on the number of items your table has to scan and how often you need to query your data using the scan function.

The Query Function

The query function allows you to query your items more efficiently. To be able to query more efficiently you need to supply and partition key and operational a sort key.

To be able to use the query function to get the items you want, you need to make sure you have set the partition and sort key on the correct attributes that will allow you to query your data.

Remember Secondary indexes are available to allow you to create second indexes that allow you to query with different attributes then what was set when creating your table.

DynamoDB .NET SDK

DynamoDB SDK for .NET contains three models to interact with DynamoDB from within your application. A low-level model, Document Model and an Object Persistence Model, we will mainly focus on the Low-Level and Object Persistence model.

If we first look at the Low-Level model, this model gives us the ability to do almost everything that DynamoDB offers we can create, delete and update tables we can query, scan, add and update items in our table. However, this model is considered the hardest of the 3 models to use and requires us to write the most amount of code between the models.

The Object-Persistence and Document Model contain higher-level interface abstraction that we can use. We can do everything that the Low-level model can apart from creating, deleting and updating tables. We can still, query, scan, add and update items. These models are a lot easier to use when interacting with DynamoDb and easier to read.
In the examples below, we will see what the code looks like from the low-level and object persistence model.

AWSSDK.Extensions

The final SDK I want to talk about is the AWSSDK.Extensions.NETCore.Setup. It’s worth highlighting and speaking about what this package is and what it gives us.


The AWSSDK.Extensions.NETCore.Setup library package gives us two main things, first, it adds extension methods to the IConfiguration interface.
This allows us to easily read configuration from our appsettings.json file.

Let’s have a look at what we might store in our appsettings.

{
  "AWS": {
  "Profile": "test-profile",
  "Region": "us-east-1"
  }
}
appsettings.json

We have added a section named AWS and set the profile and region.
Inside our startup file, we want to read these config values.

This brings us to the second main point of the AWSSDK.Extensions.NETCore.Setup library. We spoke about having extension methods on the IConfiguration, but we also have extension methods on the IServiceColleciton interface.

We can use the AddDefaultAWSOptions extension method to read the AWS config values.

public void ConfigureServices(IServiceCollection services)
{
    services.AddControllers();
    services.AddDefaultAWSOptions(Configuration.GetAWSOptions());
}
startup.cs

We are also able to register the IAmazonDynamoDB client through .NET Core dependency injection framework.

public void ConfigureServices(IServiceCollection services)
{
    services.AddControllers();
    services.AddDefaultAWSOptions(Configuration.GetAWSOptions());
    services.AddAWSService<IAmazonDynamoDB>();
}
startup.cs

We are then able to use our client in our application, in the class you want to use the IAmazonDynamoDB client, we can inject this like so.

public class ItemsController : ControllerBase
{
    private readonly IAmazonDynamoDB DynamoDbClient = _dynamoDbClient
    
    public ItemsController(IAmazonDynamoDB  dynamoDbClient)
    {
        _dynamoDbClient = dynamoDbClient;
    }
...
}
ItemsController.cs

Now that we have an understanding of DynamoDB and some of its concepts. Let’s have a look at some code examples of comment messages for DynamoDb. I spoke earlier about the .NET DynamoDB SDK having three models that we can use to interact with DynamoDB from within our .NET Core application. In the below examples I’m going to show examples using the Object Persistence Model and the Low-Level Model, apart from creating and deleting tables as the only way to do this currently is by using the Low-Level model.

Remember the Object Persistence model wraps the low-level model, the reason for showing the low-level examples is to give you an idea on how much easier it is to use one of the models that wrap the low-level model.


Creating a Table

The following example shows how we go about creating a table from within our .NET Core application. We need to ensure we

public async Task CreateDynamoTable(string tableName)
{
    var request = new CreateTableRequest
    {
        TableName = tableName,
        AttributeDefinitions = new List<AttributeDefinition>()
        {
            new AttributeDefinition
            {
                AttributeName = "Id",
                AttributeType = "N"
            }
        },
        KeySchema = new List<KeySchemaElement>()
        {
            new KeySchemaElement
            {
                AttributeName = "Id",
                KeyType = "HASH"
            }
        },
        ProvisionedThroughput = new ProvisionedThroughput
        {
            ReadCapacityUnits = 1,
            WriteCapacityUnits = 1
        }
    };

    await _dynamoDbClient.CreateTableAsync(request);
}

Add Items

Low Level Model

public async Task AddItem(int userId, ItemRequest itemRequest)
{
    var request = new PutItemRequest
    {
        TableName = TableName,
        Item = new Dictionary<string, AttributeValue>
        {
            {"Id", new AttributeValue {N = userId.ToString()}},
            {"Name", new AttributeValue {S = itemRequest.MovieName}},
            {"Description", new AttributeValue {S = itemRequest.Description}},
        }
    };

    await _dynamoDbClient.PutItemAsync(request);
}

Object Persistence Model

When using the Object persistence model, we want to instanuate the DynamoDBContext this is part of the DynamoDB SDK that we spoke about earlier. By instanuating the DynamoDBContext we gain access to the methods contained in the Object Persistence Model, once we have instiated the DynamoDBContext we can pass in our injected dynamoDBClient

public class DynamoDbRepository : IDynamoDbRepository
{
    private readonly DynamoDBContext _context;
    
    public DynamoDbRepository(IAmazonDynamoDB dynamoDbClient)
    {
        _context = new DynamoDBContext(dynamoDbClient);
    }
...

}

Once we can use the object persistence model, we can add our item using the following

public async Task AddItem(ItemRequest itemRequest)
{
    await _context.SaveAsync(itemRequest);
}

Below shows what our item request model looks like. You will know two attributes the first DynamoDBTable is set to the name of our DynamoDB table. The second is the DynamoDBHashKey this is what we have set as our Partition key when creating our DynamoDb table.

[DynamoDBTable("TableName")]
public class ItemRequest
{
    [DynamoDBHashKey]
    public int Id { get; set; }
    public string Name { get; set; }
    public string Description { get; set; }
}

Remove Items

Low Level Model

public async Task DeleteItem(string tableName)
{
    var request = new DeleteItemRequest
    {
        TableName = tableName,
        Key = new Dictionary<string, AttributeValue>
        {
            { "Id", new AttributeValue { N = "999" } }
        },
    };

    await _dynamoDbClient.DeleteItemAsync(request);
}

Object Persistence Model

Using the same model above that includes the table name and partition key we can do the following.

public async Task DeleteItem(ItemRequest itemRequest)
{
    await _context.DeleteAsync(itemRequest);
}

Get Items using Scan function

Object Persistence Model

public async Task<IEnumerable<ItemRequest>> GetAllItems()
{
    return await _context.ScanAsync<ItemRequest>(
       new List<ScanCondition>()).GetRemainingAsync();
}

Low Level Model

public async Task<ScanResponse> GetAllItems()
{
    var scanRequest = new ScanRequest(TableName);
    return await _dynamoDbClient.ScanAsync(scanRequest);
}

Get Items using Query Filter

Object Persistence Model

public async Task<IEnumerable<ItemRequest>> GetUsersItemsByName(
int id, string name)
{
    var config = new DynamoDBOperationConfig
    {
        QueryFilter = new List<ScanCondition>
        {
            new ScanCondition("Name", ScanOperator.BeginsWith, name)
        }
    };

    return await _context.QueryAsync<ItemRequest>(
       id, config).GetRemainingAsync();


Low Level Model

public async Task<QueryResponse> GetItemsByName(int id, string name)
{
    var request = new QueryRequest
    {
        TableName = TableName,
        KeyConditionExpression = "Id = :userId and begins_with (Name, :name)",
        ExpressionAttributeValues = new Dictionary<string, AttributeValue> {
            {":id", new AttributeValue { N =  id.ToString() }},
            {":name", new AttributeValue { S = name }}
    }
 };

    return await _dynamoDbClient.QueryAsync(request);
}

Developing .NET Core Applications with DynamoDB on AWS

Configuring AWS SDK with .NET Core

.NET: Object Persistence Model

DynamoDB Low-Level API