Amazon SQS: Simple Queue Service
A few days later than expected, it was the….erm….writers strike!
Amazon’s Simple Queue Service provides reliable storage and delivery
of messages between any number of computers that have at least periodic
access to the Internet.
SQS
has everything you expect from a messaging system, including loose
coupling and fault tolerance, with some things you might not, like a
distributed infrastructure that stores messages redundantly over
multiple data centres. And of course, being web service based, clients can be built on any platform that can use HTTP requests.
SQS Architecture
Like the other Amazon Web Services, SQS is elegant in its simplicity. It contains just 2 constructs:
- Messages. A message is a piece of textual data up to 8KB in size.
- Queues. Queues store related messages together with a common configuration.
SQS is implemented as a distributed system. It stores messages in
multiple servers and potentially across multiple data centres. Although
this has a positive impact on redundancy, reliability and scalability,
it also has its drawbacks. The distributed nature brings in a few
quirks that you need to bear in mind when developing an application
with SQS.
Message retrievals might return incomplete results. When
a request comes in from a client, the SQS system samples only a subset
of physical servers for messages to return. Although the subset
changes, you cannot be sure that any particular retrieval
contains all the messages in a given queue.
Messages may not be delivered speedily. You cannot
rely on SQS as a massaging system if you require instant message
delivery. In the normal run of things, messages can take from 2 to 10
seconds to be delivered.
Messages may not be delivered in sequence. Although
SQS will try to deliver your messages in the order in which they were
sent, in a distributed system it’s not always possible. If you need to
have ordered delivery, you will need to add some sort of sequencing
layer on top of SQS.
Messages may be redelivered. SQS uses 2 criteria to
determine whether or not a message should be delivered; whether it
still exists in a queue, and its visibility state (see below). In a
distributed system however, one can never be certain that these
properties are synchronised across multiple physical servers, so your
application must gracefully handle the delivery of a message that
should be invisible or have already been deleted.
Managing Messages
SQS
will deliver a message as many times as it needs to in order to ensure
that it is properly received, processed and acknowledged. This approach
means that no message can be lost. Even if a message-receiving
component crashes, or a network goes down before a message can be
processed.
Messages are managed by changing a message’s visibility state. A messages state can be set to invisible for a certain amount of time. This amount of time varies from 0 seconds to 2 hours, and is set by the queue.
Because
a message only remains invisible for a set period of time, the only way
to prevent eventual redelivery is to delete the message from the queue.
Setting the visibility timeout appropriately is essential for efficient
running of your application. If you set it too short, the message might
be delivered again whilst it is still being processed. Ideally, a
message’s visibility timeout should be slightly higher than the time it
would take to process the message.
SQS Roles
Message Sender. A message sender contacts SQS and asks it to create a message on a given queue with some data.
Message Receiver.
The message receiver periodically polls SQS to see if there are any new
messages available on a specific queue. If there are messages on a
specific queue, and on the subset of sampled physical servers, SQS will
return up to 10 of these messages for processing. If a message is
processed successfully, the receiver will delete the message from the
queue. If it is not processed successfully, the receiver can opt to do
nothing, and let the visibility timeout expire.
Administrator. The administrator monitors the
queues and keeps the infrastructure running smoothly. This role can
bounce quite nicely off of EC2, in the sense that if a queue starts to
back up, it can simply spawn a new server to process the messages!
Pricing
$0.01 for 10,000 requests.
Data transfer is on a sliding scale – $0.18/GB from 0 to 10TB, $0.16/GB from 10 to 50TB and $0.13/GB for any amount over 50TB.
All in all, an enterprise level messaging system, but for a very low cost. Just the right formula for the micro ISV!
