Testing with Azure CosmosDB Emulator in Azure DevOps CI/CD Pipeline with ASP.NET Core

During local development, I often use Azure CosmosDB emulator instead of having a running instance in the cloud. Naturally, my unit tests also use the emulator.

Since our gated check-in requires all unit tests to complete we need to find a way to complete all tests in Azure DevOps.

Luckily there is this handy pipeline task from the Visual Studio Marketplace at hand!

In case you never heard of Azure DevOps (formerly known as VSTS) or Azure pipelines you can get a good overview here. You can sign up for free for Azure DevOps here.

Setting up an ASP.NET Core sample

For our purpose I have created a new MVC Asp.NET Core 2.1 web application in Visual Studio including an additional .NET Core MSTest project ‚EmulatorDemoTests‘ which will contain my unit tests.

In the sample code I use the slightly adapted class DocumentDBRepository from the ToDo-App CosmosDB sample. We add the following nuget packages:
– „Microsoft.Azure.DocumentDB.Core“
– „System.Configuration.ConfigurationManager“.

For the test project i created a file „.runsettings“, which is required to configure the access to our local cosmosdb instance.

<RunSettings>
    <TestRunParameters>
    <!-- Path to the local CosmosDB instance -->
    <Parameter name="endpoint" value="https://localhost:8081" />
    <!-- Wellknown Secret to acccess the emulator instance -->
    <Parameter name="authKey" value="C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==" />
    <!-- Database and collection name -->
    <Parameter name="database" value="demodb" />
    <Parameter name="collection" value="democol" />
  </TestRunParameters>
</RunSettings>

In order to have Visual Studio pick up that file, you need to set the Test Settings file with following commands:
– Test – Test Settings – Select Test Settings File (Select your .runsettings file)

In the „CosmosDBTest“ class I ensure that our DocumentDBRepository is initialized properly with the settings from .runsettings file:

[TestInitialize()]
public void CosmosDBInitialize()
{
    this._EndpointUrl = TestContext.Properties["endpoint"].ToString();
    this._AuthKey = TestContext.Properties["authKey"].ToString();
    this._DatabaseName = TestContext.Properties["database"].ToString();
    this._CollectionName = TestContext.Properties["collection"].ToString();
    DocumentDBRepository<Person>.Initialize(
            this._EndpointUrl, 
            this._AuthKey, 
            this._DatabaseName, 
            this._CollectionName);
}

I have written a simple test case which will suffice for our little sample.

[TestMethod]
public async Task TestInsertDocuments()
{
    var document = await DocumentDBRepository<Person>.CreateItemAsync(new Person
    {
        Age = 38,
        FirstName = "Andreas",
        LastName = "Pollak"
    });
    Assert.IsNotNull(document);
    Assert.IsFalse(string.IsNullOrEmpty(document.Id));

    var person = (await DocumentDBRepository<Person>.GetItemsAsync(
        p => p.LastName == "Pollak")).FirstOrDefault();

    Assert.IsNotNull(person);
    Assert.IsTrue(person.FirstName == "Andreas");
    Assert.IsTrue(person.LastName == "Pollak");
    Assert.IsTrue(person.Age == 38);
    await DocumentDBRepository<Person>.DeleteItemAsync(document.Id);
}

You can find the complete code on GitHub.

Setting up CI/CD pipeline in Azure DevOps

First of all you need an azure devops account. You can sign up for free for Azure DevOps here. After you have created a new DevOps Project (in this case with a GIT repository) you can add your source to the git repository like i did below:
Source Code

To enable Azure CosmosDB Emulator in you CI/CD pipelines you need to install the Azure DevOps pipeline task.
Navigate to the Azure CosmosDB Emulator Task in your browser and click „Get it free“ – Button. After authentication with either your organizational or Microsoft account you can choose the DevOps Account you want to install this task to. In case your devops account is located at https://youraccount.visualstudio.com your account will be listed as: „youraccount“.

Install to location

Click „Install“ and after the successful install „Proceed to organization“. Select your DevOps project.

Click the Pipelines menu and create a new pipeline by clicking on the „New pipeline“ button in the center of the screen.

Select Pipelines

First select your pipeline source. You have a varity of options to host your code including DevOps Repos, TFVC, GitHub, GitHub Enterprise, Subversion, BitBucket or any external git repository. In this case just use „Azure Repos Git“ as the source is stored in the DevOps project.

Select Source

Next choose from many available templates (which allow us also to build Python, Java,… code). Select „ASP.NET Core“ as our template:

Select AspNet Core Template

The initial pipeline looks like this. Since no emulator is running in the DevOps envionment the tests will fail.
Initial pipeline

And sure enough they fail:
Failed test run

To enable the Azure CosmosDB Emulator add an additional task to our devops pipeline. This task will run a docker container for windows containing an emulator instance. Since per default the agent host is Linux you need to switch the Build Agent from Linux to Windows:
Change Pipeline to Windows Agent

Now you can select the + sign in the agent job section and add a new task „Azure CosmosDB emulator“ from the „test“ category. Use drag n drop to move it between the tasks „Build“ and „Test“.

AddPipelineTask

It is important to know that the CosmosDB Emulator Task will export an pipeline variable „$(CosmosDbEmulator.Endpoint)“ which will contain the endpoint where the CosmosDB instance will be available.

You can configure your Emulator instance as you like. For example configure the consistency model or the amount of partitions to allocate,…

Now you need to configure the „Test .NET Core“ task to have the unit tests use the endpoint of the emulator you just created. While you can configure a runsettings file, there is currently no way to override parameters (see this open github issue).

Therefore you need to work around this limitation. First of all configure the test task to use a runsettings file that does not yet exist. Right now there is only a „.runnsettings“ file in that folder.

--configuration $(BuildConfiguration) --settings "$(Build.SourcesDirectory)\EmulatorDemo\EmulatorDemoTests\test.runsettings"

Configure NetCore Test Task

Next use a small powershell script task to create this file dynamically. Click on the +-Icon in the „Agent job“ section and find unter the category „Utility“ the task „PowerShell“. Place that script between „Run Azure CosmosDB Emulator“ and „Test .NET Core“.

Now you need to configure two pipeline parameters as environment variables within the script. Open the „Environment variables“ section and configure those values. Attention: Environment variables MUST NOT contain a „.“.

CosmosDbEmulatorEndpoint  = $(CosmosDbEmulator.Endpoint)
BuildSourcesDirectory     = $(Build.SourcesDirectory)

Set EnvVariables For PowerShell Script

Add your little script. Make sure to select Type = Inline. and copy the following script into the script text field.

# Create test.runsettings file
Write-Host CosmosDB Endpoint: $env:CosmosDbEmulatorEndpoint
Write-Host Source Path: $env:BuildSourcesDirectory

$sourceConfig = $env:BuildSourcesDirectory+"\EmulatorDemo\EmulatorDemoTests\.runsettings"

$parameter = Select-Xml -Path $sourceConfig -XPath '//Parameter[@name="endpoint"]'
$parameter.Node.value = $env:CosmosDbEmulatorEndpoint
$newFileName = $parameter.Path.Replace(".runsettings","test.runsettings")
$parameter.Node.OwnerDocument.Save($newFileName)

Add Script To Power Shell Task

And you are ready to roll! And the test run succeeds:

Test run succeeds

Have fun with Azure DevOps AndiP

Adding Changefeed support to CosmosDB Datamigration Tool

In a production environment I usually would use Azure Functions which use the Change Feed Processor Library internally to continuously push changes in my CosmosDB to other destination(s).

However, for some small testing scenarios, demos and also some coding fun, I decided to add ChangeFeed support to the Azure CosmosDB data migration tool (link to original version).

So with this special version of the Azure CosmosDB Data Migration Tool you have additional options for the DocumentDB-Source available unter „Advanced Options“:

Once you check „Use change feed of the collection“ you get following options:

You can select where you want start reading from the change feed. Either start at the creation time of the collection or select a specific date.  I admit I could have added a DateTime-Picker :-P.

At „File to read/store continuation tokens“ you can provide a file name to store the continuation tokens. If you re-run the wizard and provide the file again only the new updated documents will be processed.

Last but not least you need to set, if you want to update the provided continuation token file with the new continuation tokens which in most situations is desired.

Thnx

AndiP

Cosmos DB access control with users and permissions

Overview

I have written a small lab on using user permissions with CosmosDB. This article dwells a bit on the scenarios when to use permissions instead of masterkey and how you can use a set of permissions to grant access to multiple documents, collections, partitions at once.

Scenarios for user/permissions resource keys

Before we dive into the details of „How we can work with users/permissions to access data in Cosmos DB“ let’s first discuss some scenarios where that can be applied.

In an ideal scenario we ensure that our authenticated user can access our services only through a limited set of API’s as shown in this simplified architecture:

In this scenario we have two options.
Option A: Use the master key and ensure access to the right data in the data services depending on the users permissions.
Option B: Depending on the authenticated user create a set of user permissions (correlating it either directly 1:1 to a cosmos db user, or using a user per tenant) and use this to connect to Cosmos DB in your data services.

In the following scenarios we want to enable direct access to Cosmos DB from applications and services that are not under our full control. We need to ensure that those clients can only access data they are supposed to.

Let’s say the Vendor B needs very fast regional access to the data stored in Cosmos DB. She wants to avoid additional latency by running constantly through business services provided by Vendor A.

In this case Vendor A can grant access to Vendor B’s specific data in Cosmos DB by limiting the access to a single document, single tenant collection or a specific partition within a collection.

Vendor B can use that token/permissions to access the database directly without having access to data of other tenants.

Another scenario might be a mobile application that should be able to fetch the users profile/data quickly directly globally from Cosmos DB.

In this case the application service could provide a resourcetoken/permission to access the database directly. I personally do not like this scenario and would rather use a global distributed API governed by API-Management because that way I have more control in how the user accesses the data (metrics, throttling, additional protection mechanisms). Such a scenario is described with a Xamarin Forms application in the documentation including sample code.

Implementing Users/Permissions

Users are stored within the context of the database in Cosmos DB. Each user has a set of unique named permissions. To learn more about the structure within CosmosDB read this.

Each permission object consists out of

  • Token (string)… Access token to access cosmos db
  • Id (string)… unique permission id (for user)
  • PermissionMode… either Read or All
  • ResourceId (string)… ID of the resource the permission applies to
  • ResourceLink (string)… Self-Link to resource where perm apply.
  • ResourcePartitionKey (string)… PartitionKey of resource perm applies.

Once you have acquired a permission you need only to transfer the Token to the client that should access Cosmos DB. A token is represented as a string.

// token: "type=resource&ver=1&sig=uobSDos7JRdEUfj ... w==;"
string tokenToTransfer = permission.Token;

The client can create a connection to Cosmos DB using that token.

DocumentClient userClient = new DocumentClient(
    new Uri(docDBEndPoint), 
    transferedToken);

!Important! The token stays valid until it expires even if you delete the permission in Cosmos DB. The expiration time can be configured!

You can access multiple resources by providing a set of Permissions. Use the constructor allowing to pass a list of Permissions.
Serializing and Deserializing those permissions as JSON strings is a bit painfully:

Serialization:

// Serializing to JSON
MemoryStream memStream = new MemoryStream();
somePermission.SaveTo(memStream);
memStream.Position = 0L;
StreamReader sr = new StreamReader(memStream);
string jsonPermission = sr.ReadToEnd();

The serialized token looks like this:

{
    "permissionMode":"Read",
    "resource":"dbs/q0dwAA==/colls/q0dwAIDfWAU=/",
    "resourcePartitionKey":["apollak"],
    "id":"mydata",
    "_rid":"q0dwAHOyMQBzifcvvscGAA==",
    "_self":"dbs/q0dwAA==/users/q0dwAHOyMQA=/permissions/q0dw ... AA==/",
    "_etag":"\"00001500-0000-0000-0000-5aaa4a6b0000\"",
    "_token":"type=resource&ver=1&sig=uobSD ... 2kWYxA==;2L9WD ... Yxw==;",
    "_ts":1521109611
}

De-serialization:

memStream = new MemoryStream();
StreamWriter sw = new StreamWriter(memStream);
sw.Write(jsonPermission);
sw.Flush();
memStream.Position = 0L;
Permission somePermission = Permission.LoadFrom<Permission>(memStream);
sw.Close();

By adding multiple permissions to a list you can create a document client with access to multiple resources (f.e. 1..n documents, 1..n collections, 1..n partitions).

List<Permission> permList = new List<Permission>();
// adding two permissions
permList.Add(Permission.LoadFrom<Permission>(memStream));
permList.Add(lpollakDataPermission);
DocumentClient userClient = new DocumentClient(
    new Uri(docDBEndPoint,UriKind.Absolute), 
    permList);

Important things to know:
– If you restrict the partition with a Permission you MUST always set the partition key accessing CosmosDB!
– Permission IDs must be unique for each user and must not be longer than 255 characters
– Tokens expire after an hour per default. You can set an expiration starting by 10 minutes up to 24 hours. This is passed within the RequestOptions in seconds.
– Each time you read the permission from the permission feed of a user a new token gets generated

Example to customize the expiration time

lpollakDataPermission = await client.UpsertPermissionAsync(
    UriFactory.CreateUserUri(dbName, apollakUserid), 
    lpollakDataPermission, 
    new RequestOptions() { ResourceTokenExpirySeconds = 600});

Simple sample implementation as LAB

I have written a small LAB that you can use with CosmosDB Emulator to play with the permissions. There is a student version to start from and also a finished version. In the project folder you will find a „ReadMe.md“ file describing the steps. Download Cosmos DB Security Lab

Learn more about securing CosmosDB

Global Azure Bootcamp 2017 Nachlese

 

Wissensturm in Linz 2017 (c) by Andreas PollakWie jedes Jahr fand auch heuer das Global Azure Bootcamp  für Österreich im Linzer Wissensturm statt. Wie immer , dank Rainer Stropek und Karin Huber, ausgezeichnet organisiert und mit spannenden Vorträgen versehen.

Diesmal durfte ich gleich zwei Vorträge zu den Themen “Azure API Management” und “Azure DocumentDB” beisteuern.

Speaker Foto (c) by Rainer StropekUnterhalb findet ihr die Verweise zu den Vortragsslides. Dort könnt ihr auch die Codebeispiele von GitHub herunterladen.

 

Slidedecks & Source

Viel Spaß und bis demnächst

AndiP