Simple Queue Service (SQS)
Last updated
Last updated
One of the oldest offerings by AWS.
Fully managed service, used to decouple applications.
Producer/s will send (produce) a message using SendMessage API offered by AWS SDK and put it in AWS SQS service queue.
Message will be persisted in SQS untill a consumer deletes it.
Messages will be retained as per retention policy of SQS.
Everytime a message is received for reprocessing the receive count is increased.
Consumer/s will poll (consume) the message/s from SQS. Typically consumer can receive upto 10 messages at a time.
Once a message is read/consumed by Consumer/s from the queue, they will be deleted using DeleteMessage API.
Consumers typically read more than one message per time, but by default it is 1 message per read.
Multiple consumers will receive different messages and process in parallel.
Consumers may be in an ASG and will be polling in an SQS group.
CloudWatch Metric can be set to trigger an CloudWatch Alarm, which in response may scale out the consumers.
After a message is polled/consumed by a consumer, the message becomes invisible to other consumers. The time duration starts as soon as the ReceiveMessageRequest is returned by the SQS to consumer.
Once the timeout duration ends, the message is put back to the queue if the message doesnt get processed in that duration. Due to this the message may get duplicated and processed more than once by different consumers.
One can change the visibility timeout duration by calling the ChangeMessageVisibility API to get more time.
By default, the message visibility timeout is 30 seconds and can be increased up to 12 hours.
When message is not processed successfully within the visibility timeout period, as mentioned earlier they gets added back to the queue.
When these keeps on occurring for undeliverable messsage, could lead to failure loop. One can limit the number of times this reprocessing is being done, by adding a dead letter queue(DLQ).
Once MaximumReceives threshold is exceeded, the message goes to dead letter queue.
Retention period should be set so that the messages dont get expired in DLQ.
DLQ of standard queue must be a standard queue and DLQ of FIFO queue must be a FIFO queue.
Maximum retention period is 14 days, as mentioned earlier.
This feature helps messages in DLQ to be redriven back to source queue or any other queue or custom destination in batches without writing custom code.
Messages redriven from DLQ to Source will be as if new message has been received with receive count set to 1.
Delay Queue offers a mechanism to delay the delivery of messages to consumer by some time period, by default it is 0 seconds (max 15 minutes).
DeliveryDelay parameter helps to configure the delay at queue level.
One can set the delay seconds value at send time by setting DelaySeconds parameter.
Long polling allows consumers to wait for messages when no messages are available in the queue.
This reduces the number of API calls made to SQS while increasing efficiency and decreasing the latency of your application.
One can set wait time from 1 second to 20 seconds (preferrable) when long polling.
Long polling is preferrable to short polling.
Can be confugured at queue level or at the API level using ReceiveMessageWaitTimeSeconds.
To send messages with size above 256 KB, one can use the SQS Extended Client (Java library).
The client sends the large message to S3 bucket and producer will send the metadata about S3 bucket and the message send to SQS queue.
This message will be consumed by consumer and retrieve the larger message (S3 Object) from S3 using the metadata and continue the process.
CreateQueue
To create queue. MessageRetentionPeriod parameter can be used to set the duration after which message is discarded.
DeleteQueue
To delete queue and messages in queue
PurgeQueue
To delete all messages in queue
SendMessage
To send messages, supports batch API. (DelaySeconds) parameter can be used to set the delay seconds.
ReceiveMessage
To receive messages
DeleteMessage
To delete messages once processed by consumer, supports batch API
MaxNumberOfMessages
Default 1, Max 10 (for ReceiveMessage)
ReceiveMessageWaitTimeSeconds
Long polling
ChangeMessageVisibility
To change message visibility timeout period, supports batch API
Unlimited throughput, unlimited number of messages in queue.
Default retention period of messages is 4 days, maximum of 14 days.
Low latency (< 10 ms on publish and receive).
Message size cannot exceed 256 KB.
Duplicate messages are possible i.e., a message may be delivered twice. But it aims for atleast once delivery.
Out of order messages may occur, as it offers best-effort ordering.
DLQ of a Standard queue must be Standard Queue.
Queue name should be suffixed with .fifo,
Message ordering is preserved, first message coming into queue is the first to leave the queue. Hence, consumer process messages in order.
Limited throughput with 300 messages/seconds with out batching and 3000 messages/seconds with batching.
Messages are send exactly once, as it uses deduplication ID.
Optionally, it can do content-based deduplication as well.
Deduplication
Deduplication interval is 5 minutes, which enables a duplicate message received within this interval is refused.
Types of deduplication
Content based : Done by calculating SHA-256 SHA of message body.
Message deduplication ID based : Done by passing unique deduplication ID, is optional if Content based deduplication is enabled.
Message Grouping
If you specify same message group ID in SQS FIFO queue, you can only have one consumer, all the messages are in order.
Messages with common Message Group ID will be ordered with in the group, each group ID can have a different consumer (helpful for parallel processing).
Ordering across group is not guaranteed.
Offers in-flight encryption using HTTPS API.
Can also have at rest encryption using KMS keys.
Client-side encryption can also be done if the client wants to perform encryption/decryption itself.
AWS IAM policies
Can be used to control access to SQS API.
SQS Access policy
Can also be used to control the access, which is useful when needed to make a cross account access or control access to write to SQS Queue by other AWS services.
A typical SQS based architecture flow will look in below diagram,
