With Amazon Easy Queue Service (Amazon SQS), you possibly can ship, retailer, and obtain messages between software program elements at any quantity. At present, Amazon SQS has launched two new capabilities for first-in, first-out (FIFO) queues:

• Most throughput has been elevated as much as 70,000 transactions per second (TPS) per API motion in chosen AWS Areas, supporting sending or receiving as much as 700,000 messages per second with batching.
• Useless letter queue (DLQ) redrive assist to deal with messages that aren’t consumed after a particular variety of retries in a method much like what was already obtainable for normal queues.

Let’s take a extra in-depth take a look at how these work in apply.

FIFO queues throughput enhance as much as 70K TPS
FIFO queues are designed for purposes that require messages to be processed precisely as soon as and within the order through which they’re despatched. Whereas commonplace queues have an infinite throughput, FIFO queues have an higher quota within the variety of TPS per API motion.

Normal and FIFO queues assist batch actions that may ship and obtain as much as 10 messages with a single API name (as much as a most whole payload of 256 KB). Because of this a FIFO queue can course of as much as 10 occasions extra messages per second than its most throughput.

At launch in 2016, FIFO queues supported as much as 300 TPS per API motion (3,000 messages per second with batching). This was sufficient for a lot of use circumstances, however some prospects requested for extra throughput.

With excessive throughput mode launched in 2021, FIFO queues launched a tenfold enhance of the utmost throughput and will course of as much as 3,000 TPS per API motion, relying on the Area. One 12 months later, that quota was doubled to as much as 6,000 TPS per API motion.

This 12 months, Amazon SQS has already elevated FIFO queue throughput quota two occasions, to as much as 9,000 TPS per API motion in August and as much as 18,000 TPS per API motion in October (relying on the Area).

At present, the Amazon SQS group has been in a position to enhance the FIFO queue throughput quota once more, permitting you to course of as much as 70,000 TPS per API motion (as much as 700,000 messages per second with batching) within the US East (N. Virginia), US West (Oregon), and Europe (Eire) Areas. That is greater than 2 hundred occasions the utmost throughput at launch.

DLQ redrive assist for FIFO queues
With Amazon SQS, messages that aren’t consumed after a particular variety of retries can robotically be moved to a DLQ. There, messages will be analyzed to know the explanation why they haven’t been processed accurately. Typically there’s a bug or a misconfiguration within the shopper utility. Different occasions the messages include invalid information from the supply purposes that must be fastened to permit the messages to be processed once more.

Both method, you possibly can outline a plan to reprocess these messages. For instance, you possibly can repair the patron utility and redrive all messages to the supply queue. Or you possibly can create a devoted queue the place a customized utility receives the messages, fixes their content material, after which sends them to the supply queue.

To simplify shifting the messages again to the supply queue or to a special queue, Amazon SQS means that you can create a redrive job. Redrive duties are already obtainable for normal queues. Beginning as we speak, you can even begin a redrive job for FIFO queues.

Utilizing the Amazon SQS console, I create a primary queue (my-dlq.fifo) for use as a DLQ. To redrive messages again to the supply FIFO queue, the queue kind should match, so that is additionally a FIFO queue.

Then, I create a supply FIFO queue (my-source-queue.fifo) to deal with messages as ordinary. Once I create the supply queue, I configure the primary queue (my-dlq.fifo) because the DLQ and specify 3 because the Most receives situation underneath which messages are moved from the supply queue to the DLQ.

When a message has been obtained by a shopper for greater than the variety of occasions specified by this situation, Amazon SQS strikes the message to the DLQ. The unique message ID is retained and can be utilized to uniquely observe the message.

To check this setup, I exploit the console to ship a message to the supply queue. Then, I exploit the AWS Command Line Interface (AWS CLI) to obtain the message a number of occasions with out deleting it.

aws sqs receive-message --queue-url https://sqs.eu-west-1.amazonaws.com/123412341234/my-source-queue.fifo

{
    "Messages": [
        {
            "MessageId": "ef2f1c72-4bfe-4093-a451-03fe2dbd4d0f",
            "ReceiptHandle": "...",
            "MD5OfBody": "0f445a578fbcb0c06ca8aeb90a36fcfb",
            "Body": "My important message."
        }
    ]
}

To obtain the identical message greater than as soon as, I anticipate the time specified within the queue visibility timeout to go (30 seconds by default).

After the third time, the message is just not within the supply queue as a result of it has been moved to the DLQ. When I attempt to obtain messages from the supply queue, the record is empty.

aws sqs receive-message --queue-url https://sqs.eu-west-1.amazonaws.com/123412341234/my-source-queue.fifo

{
    "Messages": []
}

To substantiate that the message has been moved, I ballot the DLQ to see if the message is there.

aws sqs receive-message --queue-url https://sqs.eu-west-1.amazonaws.com/123412341234/my-dlq.fifo  

{
    "Messages": [
        {
            "MessageId": "ef2f1c72-4bfe-4093-a451-03fe2dbd4d0f",
            "ReceiptHandle": "...",
            "MD5OfBody": "0f445a578fbcb0c06ca8aeb90a36fcfb",
            "Body": "My important message."
        }
    ]
}

Now that the message is within the DLQ, I can examine why the message has not been processed (effectively, I do know the explanation this time) and resolve whether or not to redrive messages from the DLQ utilizing the Amazon SQS console or the brand new redrive API that was launched a couple of months in the past. For this instance, I exploit the console. Again on the Amazon SQS console, I choose the DLQ queue and select Begin DLQ redrive.

In Redrive configuration, I select to redrive the messages to the supply queue. Optionally, I can specify one other FIFO queue as a customized vacation spot. I exploit System optimized in Velocity management settings to redrive messages with the utmost variety of messages per second optimized by Amazon SQS. Optionally, if there may be a lot of messages within the DLQ, I can configure a customized most charge of messages per second to keep away from overloading shoppers.

Earlier than beginning the redrive job, I can use the Examine messages part to ballot and test messages. I already determined what to do, so I select DLQ redrive to start out the duty. I’ve just one message to course of, so the redrive job completes in a short time.

As anticipated, the message is again within the supply queue and is able to be processed once more.

Issues to know
Useless letter queue (DLQ) assist for FIFO queues is accessible as we speak in all AWS Areas the place Amazon SQS is obtainable excluding GovCloud Areas and people based mostly in China.

Within the DLQ configuration, the utmost variety of receives ought to be between 1 and 1,000.

There isn’t a further price for utilizing excessive throughput mode or a DLQ. Each Amazon SQS motion counts as a request. A single request can ship or obtain from 1 to 10 messages, as much as a most whole payload of 256 KB. You pay based mostly on the variety of requests, and requests are priced in a different way between commonplace and FIFO queues.

As a part of the AWS Free Tier, there is no such thing as a price for the primary million requests per 30 days for normal queues and for the primary million requests per 30 days for FIFO queues. For extra data, see Amazon SQS pricing.

With these updates and the elevated throughput, you possibly can cowl the overwhelming majority of use circumstances with FIFO queues.

Use Amazon SQS FIFO queues to have excessive throughput, exactly-once processing, and first-in-first-out supply.

— Danilo