Reliability

Azure Cosmos DB–Multi Master

October 8, 2018 .NET, .NET Core, .NET Framework, ASP.NET, Azure, Azure CLI, Azure Cosmos DB, CosmosDB, Data Consistancy, Data Integrity, Microsoft, Multi-master, Performance, Reliability, Resilliancy, Scalability, Scale Up No comments

During the Ignite 2018, Microsoft has announced the general availability of Multi-Master feature being introduced to Azure Cosmos DB to provide more control into data redundancy and elastic scalability for your data from different regions with multiple writes and read instances.

What is Multi-Master essentially?

Multi-master is a capability that provided as part of Cosmos DB, that would provide you multiple write regions and provides an option to handle conflict resolution automatically through different options provided by the platform. Most of the major scenarios you would encounter the conflict can be resolved with these simple configurations.

A sample diagram depicting a use case of load balanced web app writing to respective regional master:-

image

With multi-master, Azure Cosmos DB delivers a single digit millisecond write latency at the 99th percentile anywhere in the world, and now offers 99.999 percent write availability (in addition to 99.999 percent read availability) backed by the industry-leading SLAs.

image

Wow! That’s an amazing performance Cosmos DB guarantees to provide so that your mission-critical systems will have zero downtime, if they start using Cosmos DB.

 

How to Enabled Multi-Master support in your Cosmos DB solutions?

Currently multi-master can only be enabled for new Cosmos DB instances using “Enable Multi-Master” option in Azure Portal or through PowerShell or ARM templates or through SDK.

These options are detailed below with necessary examples:

1.) Azure Portal – Enable Multi-region writes and Enable geo-redundancy

image

2.) Azure CLI 
Set the “enable-multiple-write-locations” parameter to “true”

az cosmosdb create \
   –-name "thingx-cosmosdb-dev" \
   --resource-group "consmosify-dev" \
   --default-consistency-level "Session" \
   --enable-automatic-failover "true" \
   --locations "EastUS=0" "WestUS=1" \
   --enable-multiple-write-locations true \

3.) AzureRM PowerShell
In AzureRM PowerShell cmdlet – Set enableMultipleWriteLocations parameter to “true”

$locations = @(@{"locationName"="East US"; "failoverPriority"=0},
             @{"locationName"="West US"; "failoverPriority"=1})

$iprangefilter = ""

$consistencyPolicy = @{"defaultConsistencyLevel"="Session";
                       "maxIntervalInSeconds"= "10";
                       "maxStalenessPrefix"="200"}

$CosmosDBProperties = @{"databaseAccountOfferType"="Standard";
                        "locations"=$locations;
                        "consistencyPolicy"=$consistencyPolicy;
                        "ipRangeFilter"=$iprangefilter;
                        "enableMultipleWriteLocations"="true"}

New-AzureRmResource -ResourceType "Microsoft.DocumentDb/databaseAccounts" `
  -ApiVersion "2015-04-08" `
  -ResourceGroupName "consmosify-dev" `
  -Location "East US" `
  -Name "thingx-cosmosdb-dev" `
  -Properties $CosmosDBProperties

4.) Through CosmosDB SDK
Setting connection policy in DocumentDBClient and set UseMultipleWriteLocations to true.

ConnectionPolicy policy = new ConnectionPolicy
{
   ConnectionMode = ConnectionMode.Direct,
   ConnectionProtocol = Protocol.Tcp,
   UseMultipleWriteLocations = true,
};
policy.PreferredLocations.Add("East US");
policy.PreferredLocations.Add("West US");
policy.PreferredLocations.Add("West Europe");
policy.PreferredLocations.Add("North Europe");
policy.PreferredLocations.Add("Southeast Asia");
policy.PreferredLocations.Add("Japan East");
policy.PreferredLocations.Add("Japan West");

Azure Cosmos DB multi-master configuration is the game changes that really makes it a true global scale database with automatic conflict resolution capabilities for data synchronization and consistancy.

In my later sessions I will write examples to cover how conflict resolutions can be configured and used in realtime scenarios.

Useful Refs:

Azure Cosmos DB – 429 Too Many Requests

October 6, 2018 .NET, Azure, CosmosDB, Document DB, Microsoft, Performance, Reliability, Resilliancy, Scalability, Visual Studio 2017, VisualStudio, VS2017 No comments

Recently while I was doing Performance Testing in one of the APIs interacting with Cosmos DB, I encountered a problem as Azure Cosmos DB API’s started returning Http Code 429.  Http Status Code 429 indicates that too many request been received or request rate is very large. This error would happen when we have concurrent users trying to write or read from same cosmos db collection.

Following diagram covers the architecture of the performance test I am performing:

image

Based on analysis it found out to be the Throttling happening from Azure Cosmos DB, as we make requests that may use more than provisioned Request Units(RU) per second. We were using default Cosmos DB configuration for a fixed collection of 1000 RU’s per second which is sufficient enough for a 500 reads and 100 writes for a 1 kb file. You can refer more about Request Units from Azure Docs.

image

 

 

 

Solution(s):

1. Now first logical step we can do is to get rid off this error by increasing the Throughput for the collection.  I am going to increase to 10000 RU/s maximum allocatable for a Storage Capacity: Fixed.   This should ideally improve the Throughput for 250 or more virtual users hitting.

image

2. Second logical step is to improve the code: Improve the connection parameters in the Document DB SDK –> DocumentDbClient. For this I referred to the Microsoft Docs: Performance tips for Azure Cosmos DB and .NET

Providing optimum values to the following Properties in RetryOption class   to be passed as parameter to Connection Policy.

image

 

In my case I provided a value of 30 to give ultimate results:

new RetryOptions() { MaxRetryAttemptsOnThrottledRequests = 30, MaxRetryWaitTimeInSeconds = 30  }

That should resolve most of the 429 issues when dealing with Cosmos DB SDK

Azure Cosmos DB – Consistency Levels

June 2, 2018 Azure, CosmosDB, Data Consistancy, Data Integrity, Higher Availability, Microsoft, Reliability, Resilliancy, Scalability No comments

CosmosDB is a planet scale multi model, multi-region NoSQL database service provided as part of Azure Platform. Azure Cosmos DB is designed to provide global distribution for every data model you choose while creating Cosmos DB.  It is promised to provide low latency and various well-defined consistency models to ensure data redundancy and high availability.

In this short diagram I will be covering the different consistency models available with Cosmos DB and their benefits:

image

There said depending on your data criticality and needs of faster accessibility, you can choose between any of the above consistency models. I strongly trusts in session consistency, as it ensures a balance b/w both.  But again it is totally depending on your business case and how critical is your system depends on the accuracy of this data.

Hope you enjoyed this short article!.

Further reads: https://docs.microsoft.com/en-us/azure/cosmos-db/consistency-levels

Microsoft Azure IoT Suite–Provisioned solutions for Faster Time to Market IoT enabled solutions

January 7, 2017 Analytics, Azure, Azure SDK, Cloud Computing, Communication Protocols, Contrained Networks/Devices, Data Collection, Data Integration, Emerging Technologies, Identity of Things (IDoT), Internet of Things, Interoperability, IoT, PaaS, Performance, Predictive Analytics, Predictive Maintenance, Realtime Analytics, Reliability, Scalability, Self Driven Cars, Solutions, Stream Analytics, Tech-Trends, Windowz Azure No comments

Microsoft Azure IoT Suite Provisioned solutions will help you create your own fully integrated solutions tailored for your specific needs in the following 3 sections. Using these ready to consume solutions will accelerate your time to market IoT(Internet of Things) requirements.

image

  1. Remote Monitoring  – Connect and monitor your devices to analyze untapped data and improve business outcomes by automating processes.  For ex: As a car  manufacturing company, provide an option to customer to remotely monitor their car condition, and suggest if they need a re-fuel or oil change.
  2. Connected Factory – Anticipate maintenance needs and avoid unscheduled downtime by connecting and monitoring your devices. For ex: As a car manufacturing  factory and spare parts are essential for car manufacturing. Automated solutions can ensure timely availability of necessary spare parts inventory to meet daily, weekly or monthly manufacturing needs.
  3. Predictive Maintenance – Connect and monitor your factory industrial devices for insights using OPC UA to drive operational productivity.  For ex: As a car service support, you can get near real-time performance data from the cars manufactured by your company, predict the health of each components in a car and offer timely maintenance to their cars.  Send real-time reminders and notifications to customers. Their by ensuring higher satisfaction levels for customers and more business value to the organization as it attracts more sales and good customer feedback.

image

These solutions will help you to:

  1. Connect and scale quickly – Use preconfigured solutions, and accelerate the development of your Internet of Things (IoT) solution.

  2. Analyze and process dataCollect previously untapped data from devices and sensors, and use built-in capabilities to visualize—and act on—that data.

  3. Integration and Digital TransformationEasily integrate with your systems and applications, including Salesforce, SAP, Oracle Database, and Microsoft Dynamics, making it simple to access your data and keep your disparate systems up to date.

  4. Enhanced security Set up individual identities and credentials for each of your connected devices—and help retain the confidentiality of both cloud-to-device and device-to-cloud messages.

Useful Links:

IoT Protocols–Quick Comparison

January 5, 2017 Communication Protocols, Connectivity, Contrained Networks/Devices, Internet of Things, Interoperability, IoT, Performance, Reliability, Scalability, Tech-Trends No comments

The table below contains a quick summary of the IoT protocols:

Protocol CoAP XMPP RESTful HTTP MQTT AMQP
Transport UDP TCP TCP TCP TCP
Messaging Request/Response Publish/Subscribe Request/Response Request/Response Publish/Subscribe Request/Response Topic based Publish/Subscribe
2G, 3G, 4G Suitability (1000s nodes) Excellent Excellent Excellent Excellent Excellent
LLN Suitability (1000s nodes) Excellent Fair Fair Fair Excellent
Compute Resources 10Ks RAM/Flash 10Ks RAM/Flash 10Ks RAM/Flash 10Ks RAM/Flash 10Ks RAM/Flash
Success Stories Utility Field Area Networks Remote management of consumer white goods Smart Energy Profile 2 (premise energy management, home services) Extending enterprise messaging into IoT applications Extending enterprise messaging into IoT applications
designed for resource-constrained devices and low bandwidth, high latency networks interoperability
reliable queuing, flexible routing, transactions, and security

Source: Beyond MQTT: A Cisco View on IoT Protocols, Paul Duffy, April 30 2013

IoT Hub vs Event Hub–A quick comparison

December 11, 2016 Azure, Cloud Computing, Cloud to Device, Communication Protocols, Connectivity, Contrained Networks/Devices, Data Hubs, Device Shadow, Device to Cloud, Device Twin, Emerging Technologies, Event Hubs, HTTP2, Identity of Things (IDoT), Intelligent Cloud, Internet of Things, Interoperability, IoT, IoT Hub, IoT Privacy, IoT Security, Messaging, Microsoft, Performance, Protocols, Reliability, Scalability, Tech-Trends No comments

With this article I am trying to provide you a birds eye view comparison of IoT Hub and Azure Event Hub, so that some of you may stop feeling that there is nothing new in IoT Hub.

For the interest of this article, I put together a table with side-by-side comparison of some important features/desired features from an IoT Hub like platform.

Feature IoT Hub Event Hub
Communication Supports both device-to-cloud and cloud-to-device bidirectional communication Supports only device-to-cloud communication
State Management Can maintain device state using Device Twins and query them whenever needed. Not Supported
Protocol Support AMQP 1.1, AMQP over Web Sockets, MQTT 3.2, MQTT over Web Sockets, HTTP 1.1, Web Sockets. AMQP 1.1, AMQP over Web Sockets, HTTP 11 , Web Sockets only
Protocol Extensions Provides IoT protocol gateway a customizable implementation for industrial protocol channelling. Not Supported
Security Provides identity to each device and easily revocable through IoT Hub Device Management portal. Shared access policies with limited revocation capabilities are provided.
Monitoring/ Operations Provides a rich set of features through Device Management capability. Includes individually enable/disable or provision new device. Change security keys as needed. View/identify individual device problems easily. Does not provide individual performance metrics. Can provide only a high level aggregated metrics only.
Scalability Scalable to thousands/millions of simultaneous devices Limited number of simultaneous connections up to 5000 connections per Azure Service Bus Quotas. Event Hub provides a capability to partition your message to channel it in to associated Service Bus quotas.
SDK Support/ Developer Support Provides very good Integration SDK and developer support. Both Azure IoT  Device SDK and IoT Gateway SDK are the most essential kits provided for almost all devices/OS platforms. It also support all the latest programming languages such as C#, Node.js, Java and Python.
Also provides  direct MQTT, AMQP and REST based HTTP APIs.
Very detail oriented documentation provided.
.NET, Java and C apart from protocols such as AMQP, HTTP API interfaces.
Files/Images Upload Capability Supports IoT devices/solutions to upload files/images/snapshots to cloud and define a workflow for processing them. Not Available
Message Routing Very decent message routing capability is available out of the box. Up to 10 end points can be defined and Advanced Rules can be defined on how routing should occur. Requires additional programming and hosting to support as per the need.

From this comparison table, you can analyse that IoTHub is the right candidate for your IoT solution needs, as Event Hub lacking certain capabilities that are essential for an IoT Ingestion point. If you are only requiring to send messages to cloud and doesn’t require any fancy stuff as IoTHub provides, you can choose Event Hub.

Remember with more power comes more responsibility, that’s what IotHub intend to provide to you.

Hope this overview was helpful. Please feel free to comment or initiate a discussion any time. Please share your feedbacks on this article as well.