How to make your service resilient to failures | Decoupling using AWS

 Synchronous vs Asynchronous Communication 🔄

There are two types of communication:

• Synchronous: Immediate response but problematic with large traffic.
• Asynchronous: Decouples components to improve scalability and reliability.
• SQS: Queue-based model.
• SNS: Publish/Subscribe model.
• Kinesis: Real-time streaming.

For more detailed information visit AWS SNS and SQS 

AWS SQS (Simple Queue Service) 📩

AWS SQS is a fully managed queue service used to decouple microservices and handle large-scale message processing.

Key Features ⚡

• Message Queue: Producers send messages; consumers poll and process them.
• Scalability: Unlimited throughput and messages.
• Retention: Default 4 days, max 14 days.
• Message Size: Up to 256 KB per message.
• Ordering: Standard SQS can have duplicate and out-of-order messages.

How SQS Works 🛠️

1. Producing Messages: Messages are persisted in the queue until the consumer deletes them.
2. Consuming Messages: Poll messages (up to 10 at a time), process them (e.g., insert them into RDS), and then delete them from the queue.
3. Auto Scaling with EC2:

• EC2 instances running consumers in an Auto Scaling Group (ASG).
• Queue length acts as a scaling metric.
• When messages accumulate, an alarm triggers ASG to scale out/in.

SQS Message Visibility Timeout ⏳

• When a consumer polls a message, it becomes invisible to other consumers for 30 seconds.
• It becomes visible again if not processed within this time, risking duplicate processing.
• Consumers can extend visibility timeout using changeMessageVisibility.

FIFO Queues 📌

• Maintains message order.
• Deduplication ID prevents duplicate processing.
• Ordering is maintained using Message Group ID.

🔹 SQS as a Buffer for Database Writes

• Clients send requests to SQS.
• Messages are dequeued and inserted into the database.
• Best when the client does not require immediate database confirmation.

💡 Important Notes:

• FIFO queue names must end with .fifo.
• Maximum 3,000 messages per second for FIFO queues.

Temporary Queues 🕒

Temporary queues provide lightweight communication channels and reduce deployment costs.

Benefits:

1. Act as lightweight threads/process communication channels.
2. Created/deleted without extra cost.
3. API-compatible with normal SQS queues.

---

AWS SNS (Simple Notification Service) 📢

AWS SNS is a fully managed publish/subscribe (pub/sub) service for sending messages to multiple receivers simultaneously.

Key Features ⚡

• Event Producer sends messages to one SNS topic.
• Subscribers (e.g., SQS, Lambda, Email, HTTP) receive messages from the topic.
• Scalable: Supports up to 12.5M subscribers per topic and 100K topics.
• Access Policies: Similar to SQS for managing permissions.
• Message Filtering: Example:
• If state = placed, send to SQS Placed Queue.
• If state = completed, send to SQS Completed Queue.

---

SNS + SQS Fanout Architecture 🚀

Fanout allows one message to be sent to multiple destinations.

Benefits of Fanout 🎯

• Push once, and receive in all subscribed SQS queues.
• Fully decoupled for no data loss.
• Retries and delayed processing via SQS.
• Easily add more subscribers over time.
• Cross-region delivery supported.

Application Example: S3 Event Notifications 📁

• S3 allows only one event rule per bucket.
• To send the same S3 event to multiple SQS queues, use SNS fanout.

SNS FIFO + SQS FIFO Fanout 🔄

• Enables ordering + deduplication while using fanout architecture.

---

TL;DR 📝

• SQS: Queue-based model for decoupling, scaling, and message persistence.
• SNS: Pub/Sub model for broadcasting messages to multiple subscribers.
• Use SQS FIFO for ordered message processing.
• SNS + SQS Fanout: Best for sending one message to multiple queues.
• Visibility Timeout prevents duplicate processing but should be tuned based on the use case.
• Temporary Queues reduce costs and simplify request-response messaging.
• Database writes via SQS ensure reliability but require clients to handle asynchronous responses.

AWS messaging services provide scalable, decoupled architectures for microservices and event-driven applications. Choose SQS for queueing, and SNS for broadcasting, and combine them for fanout architectures! 🚀