Azure queue storage: overview, benefits and usage in .NET/ .NET core application.

I have seen applications which use a database to simulate a queue as part of processing some workflow tasks. Using a database to simulate a messaging queue mean you have to write more boilerplate codes, sql queries for the CRUD operations as well as handle concurrency issues if your applications run on multiple nodes.

Microsoft has made it easy to utilize messaging and queueing in your application with azure queue storage, which is what I talk about in this post.

Overview of Azure Queue Storage

In a nutshell, an azure queue storage is a temporary storage of messages. I say temporary because if you don’t act on a message within a certain time, the system automatically deletes the message. Currently, the default message live time is 7 days. You know have the option to specify a message should stay indefinitely in the queue until it is dequeued. Before, the maximum live time is 7 days, and you could not go beyond that.

Prior to version 2017-07-29, the maximum time-to-live allowed is 7 days. For version 2017-07-29 or later, the maximum time-to-live can be any positive number, as well as -1 indicating that the message does not expire. If this parameter is omitted, the default time-to-live is 7 days.

Azure Queue Storage Put Messages API

The content of a message is a string. A string is arguably a universal way to represent also anything: an object, a state of the program, a simple value etc … For instance, in my program, I encapsulate the data I need to process a transaction as an object which I serialize as a JSON string and use as a message’s content.

Each message has a maximum size of 64 KB. In addition, if you encode the data in base64, then you can also store up to 48 KB. Basically, don’t try to store blob data unless you know the content size is never going to be above 48 KB, as you probably want to encode the data.

When a consumer consumes and processes a message, the message becomes invisible to other consumers for a certain period which you can define programmatically. As a result, you don’t need to write codes to handle concurrent consumers. For instance, if for whatever reasons your program has not finished processing a message for a predefined period of time, the message becomes visible again for another consumer to process. Another advantage of using azure queue storage is that f your program fails midway during processing because of a network program or whatever reasons, you don’t have to worry about losing data as the message would still be there after the timeout. However, you do need to take care to delete the message or move it somewhere else after you have finished processing.

To interact with an azure queue storage, Microsoft provides libraries for various languages and frameworks: .NET, Java, Python, Ruby, PowerShell, Node.js, PHP, C++. Azure queue storage also has a RESTful API which you can use if for some reasons you cannot use a library.

What is azure queue storage good for?

Azure queue storage is good for processing a long running task asynchronously. For instance, suppose as part of an online transaction, your system need to generate a document and store into a database. The call to the database and/or the document generation can be slow. However, the process runs in the background and is transparent to the user. As such, you don’t want the user to have to wait for the process to complete. Using an azure queue would be a good way to handle this situation. For example, you could encapsulate all the data you need to process the job into an object. Then you can serialize the object into a JSON string which you set as the content of a message and push to the queue. Another service can consume from the queue and process the job. By using an azure queue storage, your application does not need to get bogged down with slow operations.

Using azure queue storage can help with throttling. A consumer can consume messages from a queue when it is ready. You can easily scale up or scale down by adding or removing consumers to suite your need. Suppose you did not use a queue and processed the work right away, if your system was busy or low on resources, the task could fail or require a long time to finish which would not result in a good user’s experience.

Another benefit of azure queue storage is making a system more resilient to failures. Producers can push messages to an azure queue storage, and if no consumer is available, the messages can stay on the queue until a consumer comes online and process it. As such, even if the services of your system that consume from the queue are down, the services can start processing the messages again when they are backed up.

Using a messaging system such as an azure queue storage is good for when you have multiple components that need to communicate with each other, and you don’t want them to depend on each other. For instance, suppose you have a separate web service that handles generating a document, instead of calling the service directly, you can encapsulate the work into a message and pushes it out the the queue. Then the web service can consume from the queue and process the message. The queue sits between multiple components and facilitates the communication such that the components don’t directly depend on each other.

An azure queue storage is also a great choice if you want a simple messaging system. Another messaging system azure has is azure service bus, which is a full, traditional messaging broker but is more complex when compared to azure queue storage. If you use an azure queue storage, you mostly need to know the basic operations to get, put, delete, and peak messages. That said, azure queue storage does not offer publish/subscribe semantics , complex routing rules and other integration features often found in a traditional messaging broker. For instance, azure queue storage does not support delivering a message to multiple consumers. For more details, check out the Storage queues and Service Bus queues – compared and contrasted article.

Using azure queue storage in a .NET/ .NET core application

Follow the documentation to get started using an azure queue storage in your .NET application. Below is a recap:

• Create an azure storage account in azure.
• Add the WindowsAzure.Storage.Queue Nuget package.
• Use the library to connect to the queue, create the queue, put or get messages.

Follow this tutorial to create a storage account. Once you have an account, you can authenticate and connect to the queue using a connection string. Another way which Microsoft recommends is using Azure Active Directory to authenticate to your storage. The feature is still in preview. For more details, checkout the documentation.

Although the documentation on azure queue storage is specific for .NET, The WindowsAzure.Storage.Queue library works for both .NET and .NET core. I have been able to use the Windows.Azure.Storage in my .NET core application without any issue.

References:

Microsoft documentation – how to use queues

Microsoft documentation – comparing azure service bus and azure qeuue storage

Microsoft documentation – managing storage account

Microsoft documentation – getting started with azure queue storage in .NET