or: "What's the difference between SQS, Kinesis, and SNS?"
One of the core principles of "cloud-native" applications is "loose coupling". Let's explore what AWS has to offer and how they various options differ from each other.
Why decoupling? (Why "Microservices"?)
I'll keep this section short since it's probably obvious why you would want to decouple the components of your architecture. Also, I know people like looking at pictures much more than reading long texts... :) (If you do like to read a long text have a look at this blog post where I'm outlining some ideas on how to apply these concepts to making Magento scale.)
- Scalability: This allows you to scale the different parts of your architecture independently. If you need more time processing an order than capturing it, you can scale up your auto-scaling group of worker servers instead of also having to add more frontend servers at the same time.
- Performance/Offloading: Some tasks don't have to happen in the request/response workflow. Sending an email later or generating a report in the background and have an email sent to you are some examples.
- Reliability (graceful degradation): Splitting your architecture into smaller independent parts allows you to react better if some parts have problems. Your fulfillment system or inventory management is down? That's not a good reason to stop accepting orders, right?
- Flexibility: "Rewire" your components differently or replace one component without having to rebuild (and re-deploy) everything.
- Separation of concerns: Why should everything live in the same code base and be deployed at the same time? Splitting your business logic into multiple services and connecting them via queues and/or stream will make it easy to develop them separately (by different teams?) and interact via a well-defined message-format/API.
The request-reply (or request-response) pattern is the simplest and most common one. If you need an immediate response there's no way around making a synchronous call (even if that's handled in the background via AJAX). Of course most of my earlier points don't apply to this pattern, but in many cases that's what you need.
Publish-subscribe: Amazon SNS
Amazon SNS implements the pub/sub (publish-subscribe) pattern. The producer posts a message to a "topic" without knowing who will be consuming it. The consumers subscribe to this topic and will be notified in real-time. Like the "request-reply" pattern this one is also synchronous, but since the producer isn't waiting for any response it's decoupled (as in "if one of the consumers fail the producer is not affected").
You have a couple of options for SNS subscribers. One of those is pushing your notifications into a queue (see below). Since any number of consumers can subscribe to the same topic this can be used to "fan-out" messages in case you need to process the same messages by multiple consumers.
Also, SNS notifications are one of the supported source events for Lambda function, which makes Lambda a great option if you need a lightweight and cheap consumer.
Push-Pull: Amazon SQS
Amazon SQS provides queues. One ore more consumers can push messages into the queue and one ore more consumers can poll the queue and process the messages. SQS is a "at-least-once" queue (and in most cases even "exactly-once"). This means that SQS won't deliver the same message to another consumer after it has been read for a configurable period (VisibilityTimeout). If the consumer doesn't delete the message within the VisibilityTimout period SQS assumes that something went wrong and will re-deliver the message to another consumer. As a consumer you actively have to delete the message after it has been processed successfully. In rare situations SQS might still deliver the message more than once. So unless it's not critical that the same message is being processed multiple times you have to makes sure your consumer is "idempotent" by tracking the execution of a specific message outside SQS.
Another big difference compared to SNS (and pub/sub in general) is that with queues the consumers don't get notified. It's the consumer's responsibility to poll the queue. In order to makes this more efficient and to allow processing incoming messages faster, SQS offers "long polling" where SQS allows you to keep the connection open for up to 20 seconds if there's currently no messages in the queue. As soon as a new message appears within this time frame SNS will deliver it right away to you.
One more thing that's important to mention is that the message order is not guaranteed (btw, it IS in Kinesis), so if the message order matters to you you'll have to take care of this yourself (e.g. by adding a sequence number in the message body)
Here's some sample code showing how to interact with SQS in PHP.
Stream: Amazon Kinesis
On the surface a Kinesis stream appears very similar to SQS queues, but they are fundamentally different: Besides the fact that Kinesis optimized for high-throughput writing to a stream is "write-only": The consumers will never delete anything from a stream. It's the individual consumer's responsibility to track the current position of the stream and they're free to go back and read the stream multiple times. It's comparable to a big log file with one or more processes reading the same file. The stream will be "trimmed" automatically (by default after 24h).
Actually, a stream is a little bit more complex than this. In order to increase the throughput, incoming messages will be distributed automatically (using the partition key you provide when writing to the stream) to a configurable number of "shards". The more shards you configure the faster you can write to and read from the queue:
Unless you use the Kinesis Client Library (which sadly isn't available for PHP) you'll have to manually iterate over all the shards and track the state. Again, here's some example code for PHP. A much more convenient option is to use Lambda to process the stream without you having to deal with any details. Behind the scenes Lambda is still polling the stream, but to your function it looks truly event-driven where your Lambda function is executed whenever new unprocessed records are available on the stream. Lambda allows you to process them in configurable batches and keeps track of sequence numbers and shards in the background.