Synchronous vs Asynchronous Communication 🔄
There are two types of communication:
- Synchronous: Immediate response but problematic with large traffic.
- Asynchronous: Decouples components to improve scalability and reliability.
- SQS: Queue-based model.
- SNS: Publish/Subscribe model.
- Kinesis: Real-time streaming.
For more detailed information visit AWS SNS and SQS
AWS SQS (Simple Queue Service) 📩
AWS SQS is a fully managed queue service used to decouple microservices and handle large-scale message processing.
Key Features ⚡
- Message Queue: Producers send messages; consumers poll and process them.
- Scalability: Unlimited throughput and messages.
- Retention: Default 4 days, max 14 days.
- Message Size: Up to 256 KB per message.
- Ordering: Standard SQS can have duplicate and out-of-order messages.
How SQS Works 🛠️
- Producing Messages: Messages are persisted in the queue until the consumer deletes them.
- Consuming Messages: Poll messages (up to 10 at a time), process them (e.g., insert them into RDS), and then delete them from the queue.
- Auto Scaling with EC2:
- EC2 instances running consumers in an Auto Scaling Group (ASG).
- Queue length acts as a scaling metric.
- When messages accumulate, an alarm triggers ASG to scale out/in.
SQS Message Visibility Timeout ⏳
- When a consumer polls a message, it becomes invisible to other consumers for 30 seconds.
- It becomes visible again if not processed within this time, risking duplicate processing.
- Consumers can extend visibility timeout using
changeMessageVisibility
.
FIFO Queues 📌
- Maintains message order.
- Deduplication ID prevents duplicate processing.
- Ordering is maintained using Message Group ID.
🔹 SQS as a Buffer for Database Writes
- Clients send requests to SQS.
- Messages are dequeued and inserted into the database.
- Best when the client does not require immediate database confirmation.
💡 Important Notes:
- FIFO queue names must end with
.fifo
. - Maximum 3,000 messages per second for FIFO queues.
Temporary Queues 🕒
Temporary queues provide lightweight communication channels and reduce deployment costs.
Benefits:
- Act as lightweight threads/process communication channels.
- Created/deleted without extra cost.
- API-compatible with normal SQS queues.
AWS SNS (Simple Notification Service) 📢
AWS SNS is a fully managed publish/subscribe (pub/sub) service for sending messages to multiple receivers simultaneously.
Key Features ⚡
- Event Producer sends messages to one SNS topic.
- Subscribers (e.g., SQS, Lambda, Email, HTTP) receive messages from the topic.
- Scalable: Supports up to 12.5M subscribers per topic and 100K topics.
- Access Policies: Similar to SQS for managing permissions.
- Message Filtering: Example:
- If
state = placed
, send to SQS Placed Queue. - If
state = completed
, send to SQS Completed Queue.
SNS + SQS Fanout Architecture 🚀
Fanout allows one message to be sent to multiple destinations.
Benefits of Fanout 🎯
- Push once, and receive in all subscribed SQS queues.
- Fully decoupled for no data loss.
- Retries and delayed processing via SQS.
- Easily add more subscribers over time.
- Cross-region delivery supported.
Application Example: S3 Event Notifications 📁
- S3 allows only one event rule per bucket.
- To send the same S3 event to multiple SQS queues, use SNS fanout.
SNS FIFO + SQS FIFO Fanout 🔄
- Enables ordering + deduplication while using fanout architecture.
TL;DR 📝
- SQS: Queue-based model for decoupling, scaling, and message persistence.
- SNS: Pub/Sub model for broadcasting messages to multiple subscribers.
- Use SQS FIFO for ordered message processing.
- SNS + SQS Fanout: Best for sending one message to multiple queues.
- Visibility Timeout prevents duplicate processing but should be tuned based on the use case.
- Temporary Queues reduce costs and simplify request-response messaging.
- Database writes via SQS ensure reliability but require clients to handle asynchronous responses.
AWS messaging services provide scalable, decoupled architectures for microservices and event-driven applications. Choose SQS for queueing, and SNS for broadcasting, and combine them for fanout architectures! 🚀
Tags:
Cloud_Computing