Metrics in CosmosDB (DocumentDB-Focus)

Update 19.11.2018: Checkout my experimental implementation of a Metrics-Client Library for CosmosDB on github.

With Azure Monitor you have a single way to access metrics for a lot of supported azure resources. My personal guess on how the implemented Azure Monitor is that each Azure Resource provider (like Microsoft.DocumentDB) built an addon-provider „microsoft.insights“ that is used by Azure Monitor.

Azure Monitor only supports these metrics for CosmosDB. The Azure Monitor REST API documentation to query metrics describes the input and output values.

Of these I found these most relevant for the DocumentDB-API:
– Consumed request units: TotalRequestUnits (Unit:Count, Aggregation: count/total)
– Number of requests made: TotalRequests (Unit:Count, Aggregation: count/total)

which can be filtered by
– DatabaseName
– CollectionName
– Region
– StatusCode

Update: During development of the sample I noticed that the documentation is quite outdated. The new Version „2018-01-01“ supports additional metrics that are not documented on the page above. Here the DocumentDB relevant ones:
– AvailableStorage
– DataUsage
– DocumentCount
– DocumentQuota
– IndexUsage
– ReplicationLatency
– ServiceAvailability

(Un)Fortunately CosmosDB provides other interesting metrics as well that cannot be retrieved by Azure Montior. You might have noticed additional metric data in the metrics blade of CosmosDB like:

  • Available Storage
  • Data Size
  • Index Size
  • Max RUs per Second (filterd on a collections partition in a specific region)
  • Observed Read Latency
  • Observed Write Latency

These and more metrics can be retrieved directly from the CosmosDB Azure Resource Provider. This was/is the „old way“ to retrieve metrics before the arrival of Azure Monitor, if i got that right. The reference chapter describes all the various metrics you can consume from CosmosDB Resource Provider.

While Azure Monitor provides a nice nuget package (enable preview!) that you can use to access Azure Monitor metrics, you need to work with the REST-API to access the other metrics.

In this article I will focus on the DocumentDB metrics retrieved by REST (also the Azure Monitor ones). You can find a Azure Monitor Sample with .NET Core here.

Structure of Uri(s) for the Metric REST interfaces

The {resourceUri} is actually a path to the requested Azure resource. Azure Monitor basically always uses the path to the CosmosDB account. If you work directly with the Azure Resource Provider of CosmosDB you need the other paths to:

Resource Uri – Definition {resourceUri}

Resource Uri -> Database Account

This path is basically used whenever we work with Azure Monitor REST-API
– subscriptions/{subscriptionId}/resourceGroups/{rgName}/providers/Microsoft.DocumentDb/databaseAccounts/{cosmosDBAccountName}

Resource Uri -> DocumentDB Database
– subscriptions/{subscriptionId}/resourceGroups/{rgName}/providers/Microsoft.DocumentDb/databaseAccounts/{cosmosDBAccountName}/databases/{databaseResourceId}

Resource Uri -> DocumentDB Collection
(Mostly used in Azure resource Metric queries)
– subscriptions/{subscriptionId}/resourceGroups/{rgName}/providers/Microsoft.DocumentDb/databaseAccounts/{cosmosDBAccountName}/databases/{databaseResourceId}/collections/{collectionResourceId}

Resource Uri -> DocumentDB Collection Partition in a specific region
– subscriptions/{subscriptionId}/resourceGroups/{rgName}/providers/Microsoft.DocumentDb/databaseAccounts/{cosmosDBAccountName}/region/{regionName}/databases/{databaseResourceId}/collections/{collectionResourceId}/partitions

Region Names {regionName}

You can find out which regions your CosmosDB is available by querying the ARM REST API of the CosmosDB Account Azure Resource. Use the „Resources Get REST API“. For CosmosDB you find the documentation on how to retrieve the details of the CosmosDB Account Resource here.

The documentations misses out additional values in „properties“. While „enableAutomaticFailover“ and „enableMultipleWriteLocations“ (multimaster) is quite easy to guess I have no idea what „capabilities“ and „configurationOverrides“ will contain (maybe other API’s?):
– capabilites: []
– configurationOverrides: {}
– enableAutomaticFailover: false
– enableMultipleWriteLocations: false

Here a non exhaustive list of potential regions:

  • North Europe
  • West Europe

Request Examples

CosmosDB Resource Provider GET metrics sample

splitted in multiple rows for better reading
This request will fetch the „Available Storage“, „Data Size“, „Index Size“ in the time frame:
2018-10-10T06:55:00.000Z to 2018-10-10T07:55:00.000Z with a 5 minute interval (PT5M). Since the resource uri path points to a specific collection, only the data of this collection is retrieved!
        name.value eq 'Available Storage' or 
        name.value eq 'Data Size' or 
        name.value eq 'Index Size'
        ) and 
        startTime eq 2018-10-10T06%3A55%3A00.000Z and 
        endTime eq 2018-10-10T07%3A55%3A00.000Z and 
        timeGrain eq duration'PT5M'

Azure Monitor GET metrics sample

splitted in multiple rows for better reading

This request will fetch the „TotalRequests“-metric within the timespan from: 10.Oct. 2018 07:57 to 10.Oct. 2018 08:57 (one hour). The result will be delivered in 1 Minute invervals (PT1M). In this case we want all Databases, Collections, Regions and StatusCodes.
    &$filter=DatabaseName eq '*' and CollectionName eq '*' and Region eq '*' and StatusCode eq '*'

The azure portal on the CosmosDB metrics blade currently uses this API-Version: 2017-05-01-preview. There is a more recent one „2018-01-01“. To get the supported API-Versions send in a wrong one :-).

Note that the new version requires „metricNames“ instead of „metric“!
    &$filter=DatabaseName eq '*' and CollectionName eq '*' and Region eq '*' and StatusCode eq '*'

Other intervals:
– PT1M (1 minute)
– PT5M (5 minutes)
– PT1H (1 hour)
– PT1D (1 day)
– P7D (7 days)

Azure ComosDB requests the Azure Portal uses for the metrics blade

Overview TAB

Number of requests (aggregated over 1 minute interval)

  • TYPE: Azure Monitor (microsoft.insights provider)
  • ResourceUri -> Database Account
  • API-Version -> 2017-05-01-preview
  • timespan -> 2018-10-10T07%3A57%3A00.000Z/2018-10-10T07%3A58%3A00.000Z
  • metric -> TotalRequests
  • aggregation -> total
  • interval -> PT1M
  • $filter ->
    DatabaseName eq 'databaseName' and 
    CollectionName eq '*' and 
    Region eq '*' and 
    ddStatusCode eq '*'

Number of requests (counted over 1 minute interval)

  • TYPE: Azure Monitor (microsoft.insights provider)
  • ResourceUri -> Database Account
  • API-Version -> 2017-05-01-preview
  • timespan-> 2018-10-10T06%3A58%3A00.000Z/2018-10-10T07%3A58%3A00.000Z
  • metric-> TotalRequests
  • aggregation-> count
  • interval-> PT1M
  • $filter ->
    DatabaseName eq 'databaseName' and 
    CollectionName eq 'colName' and 
    StatusCode eq '*'

Data and Index storage consumed

  • TYPE: Azure Resource Provider Metric
  • ResourceUri -> DocumentDB Collection
  • API-Version -> 2014-04-01
  • $filter->
    name.value eq 'Available Storage' or 
    name.value eq 'Data Size' or 
    name.value eq 'Index Size'
    ) and 
    endTime eq 2018-10-10T07%3A55%3A00.000Z and 
    startTime eq 2018-10-10T06%3A55%3A00.000Z and 
    timeGrain eq duration'PT5M'

Documentation for fetching metrics from the Collection:

Max consumed RU/s per partition key range

  • TYPE: Azure Resource Provider Metric
  • ResourceUri -> DocumentDB Collection
  • API-Version -> 2014-04-01
  • $filter->
        name.value eq 'Max RUs Per Second'
    ) and 
    endTime eq 2018-10-10T07%3A58%3A00.000Z and 
    startTime eq 2018-10-10T06%3A58%3A00.000Z and 
    timeGrain eq duration'PT1M'

Depending on the given resourceUri path the result will vary. The portal uses these three combinations of ResourceUri(s):

  • DocumentDB Database
  • DocumentDB Collection
  • DocumentDB Collection Partition in a specific region

You can find the respective documentation here:

For the „DocumentDB Collection Partition in a specific region“ I missed out the documented „partitionId“-value in my results. I got only „partitionKeyRangeId“. I also got a „region“-value for each entry in my value array. The portal uses the MAX value of all retrieved metric values to display the MAX-RUs for a partition.

Throughput TAB

Number of requests (aggregated over 1 minute interval)

see next section, uses the results from the same query

Number of requests exceeded capacity (aggregated over 1 minute interval) Status:429

This request had been used in the „Overview-Tab“ as well. The result is basically grouped by Database, Collection and Statuscode. So we can filter the 429 requests to get result we need.

  • TYPE: Azure Monitor (microsoft.insights provider)
  • ResourceUri -> Database Account
  • API-Version -> 2017-05-01-preview
  • timespan-> 2018-10-10T08%3A30%3A00.000Z/2018-10-10T09%3A30%3A00.000Z
  • metric-> TotalRequests
  • aggregation-> count
  • interval-> PT1M
  • $filter ->
    DatabaseName eq '*' and 
    CollectionName eq '*' and 
    StatusCode eq '*'

The generic result structure is documented here.

Within the the value the Metric ID will be „/subscriptions/12a34456-bc78-9d0e-fa1b-c2d3e4567890/resourceGroups/myRG/providers/Microsoft.DocumentDb/databaseAccounts/cosmosDBAccount/providers/Microsoft.Insights/metrics/TotalRequests“.

The timeseries array contains entries that basically represent the groups (by DBName, CollectionName and Status Code). Each group contains (in this case 60 items, one per minute PT1M) all the values for that group. The will be one of the following:

  • collectionname
  • databasename
  • statuscode

End to end observed read latency at the 99th percentile – metrics – Azure Resource

Latency for a 1KB document lookup operation observed in North Europe in the 99th percentile

  • TYPE: Azure Resource Provider Metric
  • ResourceUri -> Database Account
  • API-Version -> 2014-04-01
  • $filter->
        name.value eq 'Observed Read Latency' or 
        name.value eq 'Observed Write Latency'
    ) and 
    endTime eq 2018-10-10T15%3A00%3A00.000Z and 
    startTime eq 2018-10-10T14%3A00%3A00.000Z and 
    timeGrain eq duration'PT5M'

I was really missing out a single page that describes all the metric possibilities with CosmosDB. I hope that this fills the gap.

Enjoy and have a great day!


Testing with Azure CosmosDB Emulator in Azure DevOps CI/CD Pipeline with ASP.NET Core

During local development, I often use Azure CosmosDB emulator instead of having a running instance in the cloud. Naturally, my unit tests also use the emulator.

Since our gated check-in requires all unit tests to complete we need to find a way to complete all tests in Azure DevOps.

Luckily there is this handy pipeline task from the Visual Studio Marketplace at hand!

In case you never heard of Azure DevOps (formerly known as VSTS) or Azure pipelines you can get a good overview here. You can sign up for free for Azure DevOps here.

Setting up an ASP.NET Core sample

For our purpose I have created a new MVC Asp.NET Core 2.1 web application in Visual Studio including an additional .NET Core MSTest project ‚EmulatorDemoTests‘ which will contain my unit tests.

In the sample code I use the slightly adapted class DocumentDBRepository from the ToDo-App CosmosDB sample. We add the following nuget packages:
– „Microsoft.Azure.DocumentDB.Core“
– „System.Configuration.ConfigurationManager“.

For the test project i created a file „.runsettings“, which is required to configure the access to our local cosmosdb instance.

    <!-- Path to the local CosmosDB instance -->
    <Parameter name="endpoint" value="https://localhost:8081" />
    <!-- Wellknown Secret to acccess the emulator instance -->
    <Parameter name="authKey" value="C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==" />
    <!-- Database and collection name -->
    <Parameter name="database" value="demodb" />
    <Parameter name="collection" value="democol" />

In order to have Visual Studio pick up that file, you need to set the Test Settings file with following commands:
– Test – Test Settings – Select Test Settings File (Select your .runsettings file)

In the „CosmosDBTest“ class I ensure that our DocumentDBRepository is initialized properly with the settings from .runsettings file:

public void CosmosDBInitialize()
    this._EndpointUrl = TestContext.Properties["endpoint"].ToString();
    this._AuthKey = TestContext.Properties["authKey"].ToString();
    this._DatabaseName = TestContext.Properties["database"].ToString();
    this._CollectionName = TestContext.Properties["collection"].ToString();

I have written a simple test case which will suffice for our little sample.

public async Task TestInsertDocuments()
    var document = await DocumentDBRepository<Person>.CreateItemAsync(new Person
        Age = 38,
        FirstName = "Andreas",
        LastName = "Pollak"

    var person = (await DocumentDBRepository<Person>.GetItemsAsync(
        p => p.LastName == "Pollak")).FirstOrDefault();

    Assert.IsTrue(person.FirstName == "Andreas");
    Assert.IsTrue(person.LastName == "Pollak");
    Assert.IsTrue(person.Age == 38);
    await DocumentDBRepository<Person>.DeleteItemAsync(document.Id);

You can find the complete code on GitHub.

Setting up CI/CD pipeline in Azure DevOps

First of all you need an azure devops account. You can sign up for free for Azure DevOps here. After you have created a new DevOps Project (in this case with a GIT repository) you can add your source to the git repository like i did below:
Source Code

To enable Azure CosmosDB Emulator in you CI/CD pipelines you need to install the Azure DevOps pipeline task.
Navigate to the Azure CosmosDB Emulator Task in your browser and click „Get it free“ – Button. After authentication with either your organizational or Microsoft account you can choose the DevOps Account you want to install this task to. In case your devops account is located at your account will be listed as: „youraccount“.

Install to location

Click „Install“ and after the successful install „Proceed to organization“. Select your DevOps project.

Click the Pipelines menu and create a new pipeline by clicking on the „New pipeline“ button in the center of the screen.

Select Pipelines

First select your pipeline source. You have a varity of options to host your code including DevOps Repos, TFVC, GitHub, GitHub Enterprise, Subversion, BitBucket or any external git repository. In this case just use „Azure Repos Git“ as the source is stored in the DevOps project.

Select Source

Next choose from many available templates (which allow us also to build Python, Java,… code). Select „ASP.NET Core“ as our template:

Select AspNet Core Template

The initial pipeline looks like this. Since no emulator is running in the DevOps envionment the tests will fail.
Initial pipeline

And sure enough they fail:
Failed test run

To enable the Azure CosmosDB Emulator add an additional task to our devops pipeline. This task will run a docker container for windows containing an emulator instance. Since per default the agent host is Linux you need to switch the Build Agent from Linux to Windows:
Change Pipeline to Windows Agent

Now you can select the + sign in the agent job section and add a new task „Azure CosmosDB emulator“ from the „test“ category. Use drag n drop to move it between the tasks „Build“ and „Test“.


It is important to know that the CosmosDB Emulator Task will export an pipeline variable „$(CosmosDbEmulator.Endpoint)“ which will contain the endpoint where the CosmosDB instance will be available.

You can configure your Emulator instance as you like. For example configure the consistency model or the amount of partitions to allocate,…

Now you need to configure the „Test .NET Core“ task to have the unit tests use the endpoint of the emulator you just created. While you can configure a runsettings file, there is currently no way to override parameters (see this open github issue).

Therefore you need to work around this limitation. First of all configure the test task to use a runsettings file that does not yet exist. Right now there is only a „.runnsettings“ file in that folder.

--configuration $(BuildConfiguration) --settings "$(Build.SourcesDirectory)\EmulatorDemo\EmulatorDemoTests\test.runsettings"

Configure NetCore Test Task

Next use a small powershell script task to create this file dynamically. Click on the +-Icon in the „Agent job“ section and find unter the category „Utility“ the task „PowerShell“. Place that script between „Run Azure CosmosDB Emulator“ and „Test .NET Core“.

Now you need to configure two pipeline parameters as environment variables within the script. Open the „Environment variables“ section and configure those values. Attention: Environment variables MUST NOT contain a „.“.

CosmosDbEmulatorEndpoint  = $(CosmosDbEmulator.Endpoint)
BuildSourcesDirectory     = $(Build.SourcesDirectory)

Set EnvVariables For PowerShell Script

Add your little script. Make sure to select Type = Inline. and copy the following script into the script text field.

# Create test.runsettings file
Write-Host CosmosDB Endpoint: $env:CosmosDbEmulatorEndpoint
Write-Host Source Path: $env:BuildSourcesDirectory

$sourceConfig = $env:BuildSourcesDirectory+"\EmulatorDemo\EmulatorDemoTests\.runsettings"

$parameter = Select-Xml -Path $sourceConfig -XPath '//Parameter[@name="endpoint"]'
$parameter.Node.value = $env:CosmosDbEmulatorEndpoint
$newFileName = $parameter.Path.Replace(".runsettings","test.runsettings")

Add Script To Power Shell Task

And you are ready to roll! And the test run succeeds:

Test run succeeds

Have fun with Azure DevOps AndiP

Adding Changefeed support to CosmosDB Datamigration Tool

In a production environment I usually would use Azure Functions which use the Change Feed Processor Library internally to continuously push changes in my CosmosDB to other destination(s).

However, for some small testing scenarios, demos and also some coding fun, I decided to add ChangeFeed support to the Azure CosmosDB data migration tool (link to original version).

So with this special version of the Azure CosmosDB Data Migration Tool you have additional options for the DocumentDB-Source available unter „Advanced Options“:

Once you check „Use change feed of the collection“ you get following options:

You can select where you want start reading from the change feed. Either start at the creation time of the collection or select a specific date.  I admit I could have added a DateTime-Picker :-P.

At „File to read/store continuation tokens“ you can provide a file name to store the continuation tokens. If you re-run the wizard and provide the file again only the new updated documents will be processed.

Last but not least you need to set, if you want to update the provided continuation token file with the new continuation tokens which in most situations is desired.



Cosmos DB access control with users and permissions


I have written a small lab on using user permissions with CosmosDB. This article dwells a bit on the scenarios when to use permissions instead of masterkey and how you can use a set of permissions to grant access to multiple documents, collections, partitions at once.

Scenarios for user/permissions resource keys

Before we dive into the details of „How we can work with users/permissions to access data in Cosmos DB“ let’s first discuss some scenarios where that can be applied.

In an ideal scenario we ensure that our authenticated user can access our services only through a limited set of API’s as shown in this simplified architecture:

In this scenario we have two options.
Option A: Use the master key and ensure access to the right data in the data services depending on the users permissions.
Option B: Depending on the authenticated user create a set of user permissions (correlating it either directly 1:1 to a cosmos db user, or using a user per tenant) and use this to connect to Cosmos DB in your data services.

In the following scenarios we want to enable direct access to Cosmos DB from applications and services that are not under our full control. We need to ensure that those clients can only access data they are supposed to.

Let’s say the Vendor B needs very fast regional access to the data stored in Cosmos DB. She wants to avoid additional latency by running constantly through business services provided by Vendor A.

In this case Vendor A can grant access to Vendor B’s specific data in Cosmos DB by limiting the access to a single document, single tenant collection or a specific partition within a collection.

Vendor B can use that token/permissions to access the database directly without having access to data of other tenants.

Another scenario might be a mobile application that should be able to fetch the users profile/data quickly directly globally from Cosmos DB.

In this case the application service could provide a resourcetoken/permission to access the database directly. I personally do not like this scenario and would rather use a global distributed API governed by API-Management because that way I have more control in how the user accesses the data (metrics, throttling, additional protection mechanisms). Such a scenario is described with a Xamarin Forms application in the documentation including sample code.

Implementing Users/Permissions

Users are stored within the context of the database in Cosmos DB. Each user has a set of unique named permissions. To learn more about the structure within CosmosDB read this.

Each permission object consists out of

  • Token (string)… Access token to access cosmos db
  • Id (string)… unique permission id (for user)
  • PermissionMode… either Read or All
  • ResourceId (string)… ID of the resource the permission applies to
  • ResourceLink (string)… Self-Link to resource where perm apply.
  • ResourcePartitionKey (string)… PartitionKey of resource perm applies.

Once you have acquired a permission you need only to transfer the Token to the client that should access Cosmos DB. A token is represented as a string.

// token: "type=resource&ver=1&sig=uobSDos7JRdEUfj ... w==;"
string tokenToTransfer = permission.Token;

The client can create a connection to Cosmos DB using that token.

DocumentClient userClient = new DocumentClient(
    new Uri(docDBEndPoint), 

!Important! The token stays valid until it expires even if you delete the permission in Cosmos DB. The expiration time can be configured!

You can access multiple resources by providing a set of Permissions. Use the constructor allowing to pass a list of Permissions.
Serializing and Deserializing those permissions as JSON strings is a bit painfully:


// Serializing to JSON
MemoryStream memStream = new MemoryStream();
memStream.Position = 0L;
StreamReader sr = new StreamReader(memStream);
string jsonPermission = sr.ReadToEnd();

The serialized token looks like this:

    "_self":"dbs/q0dwAA==/users/q0dwAHOyMQA=/permissions/q0dw ... AA==/",
    "_token":"type=resource&ver=1&sig=uobSD ... 2kWYxA==;2L9WD ... Yxw==;",


memStream = new MemoryStream();
StreamWriter sw = new StreamWriter(memStream);
memStream.Position = 0L;
Permission somePermission = Permission.LoadFrom<Permission>(memStream);

By adding multiple permissions to a list you can create a document client with access to multiple resources (f.e. 1..n documents, 1..n collections, 1..n partitions).

List<Permission> permList = new List<Permission>();
// adding two permissions
DocumentClient userClient = new DocumentClient(
    new Uri(docDBEndPoint,UriKind.Absolute), 

Important things to know:
– If you restrict the partition with a Permission you MUST always set the partition key accessing CosmosDB!
– Permission IDs must be unique for each user and must not be longer than 255 characters
– Tokens expire after an hour per default. You can set an expiration starting by 10 minutes up to 24 hours. This is passed within the RequestOptions in seconds.
– Each time you read the permission from the permission feed of a user a new token gets generated

Example to customize the expiration time

lpollakDataPermission = await client.UpsertPermissionAsync(
    UriFactory.CreateUserUri(dbName, apollakUserid), 
    new RequestOptions() { ResourceTokenExpirySeconds = 600});

Simple sample implementation as LAB

I have written a small LAB that you can use with CosmosDB Emulator to play with the permissions. There is a student version to start from and also a finished version. In the project folder you will find a „“ file describing the steps. Download Cosmos DB Security Lab

Learn more about securing CosmosDB

Global Azure Bootcamp 2017 Nachlese


Wissensturm in Linz 2017 (c) by Andreas PollakWie jedes Jahr fand auch heuer das Global Azure Bootcamp  für Österreich im Linzer Wissensturm statt. Wie immer , dank Rainer Stropek und Karin Huber, ausgezeichnet organisiert und mit spannenden Vorträgen versehen.

Diesmal durfte ich gleich zwei Vorträge zu den Themen “Azure API Management” und “Azure DocumentDB” beisteuern.

Speaker Foto (c) by Rainer StropekUnterhalb findet ihr die Verweise zu den Vortragsslides. Dort könnt ihr auch die Codebeispiele von GitHub herunterladen.


Slidedecks & Source

Viel Spaß und bis demnächst