How to make your service resilient to failures | Decoupling using AWS

How to make your service resilient to failures | Decoupling using AWS

 Synchronous vs Asynchronous Communication 🔄

There are two types of communication:

  • Synchronous: Immediate response but problematic with large traffic.
  • Asynchronous: Decouples components to improve scalability and reliability.
  • SQS: Queue-based model.
  • SNS: Publish/Subscribe model.
  • Kinesis: Real-time streaming.

For more detailed information visit AWS SNS and SQS 

AWS SQS (Simple Queue Service) 📩

AWS SQS is a fully managed queue service used to decouple microservices and handle large-scale message processing.

Key Features ⚡

  • Message Queue: Producers send messages; consumers poll and process them.
  • Scalability: Unlimited throughput and messages.
  • Retention: Default 4 days, max 14 days.
  • Message Size: Up to 256 KB per message.
  • Ordering: Standard SQS can have duplicate and out-of-order messages.

How SQS Works 🛠️

  1. Producing Messages: Messages are persisted in the queue until the consumer deletes them.
  2. Consuming Messages: Poll messages (up to 10 at a time), process them (e.g., insert them into RDS), and then delete them from the queue.
  3. Auto Scaling with EC2:
  • EC2 instances running consumers in an Auto Scaling Group (ASG).
  • Queue length acts as a scaling metric.
  • When messages accumulate, an alarm triggers ASG to scale out/in.

SQS Message Visibility Timeout ⏳

  • When a consumer polls a message, it becomes invisible to other consumers for 30 seconds.
  • It becomes visible again if not processed within this time, risking duplicate processing.
  • Consumers can extend visibility timeout using changeMessageVisibility.

FIFO Queues 📌

  • Maintains message order.
  • Deduplication ID prevents duplicate processing.
  • Ordering is maintained using Message Group ID.

🔹 SQS as a Buffer for Database Writes

  • Clients send requests to SQS.
  • Messages are dequeued and inserted into the database.
  • Best when the client does not require immediate database confirmation.

💡 Important Notes:

  • FIFO queue names must end with .fifo.
  • Maximum 3,000 messages per second for FIFO queues.

Temporary Queues 🕒

Temporary queues provide lightweight communication channels and reduce deployment costs.

Benefits:

  1. Act as lightweight threads/process communication channels.
  2. Created/deleted without extra cost.
  3. API-compatible with normal SQS queues.

AWS SNS (Simple Notification Service) 📢

AWS SNS is a fully managed publish/subscribe (pub/sub) service for sending messages to multiple receivers simultaneously.

Key Features ⚡

  • Event Producer sends messages to one SNS topic.
  • Subscribers (e.g., SQS, Lambda, Email, HTTP) receive messages from the topic.
  • Scalable: Supports up to 12.5M subscribers per topic and 100K topics.
  • Access Policies: Similar to SQS for managing permissions.
  • Message Filtering: Example:
  • If state = placed, send to SQS Placed Queue.
  • If state = completed, send to SQS Completed Queue.

SNS + SQS Fanout Architecture 🚀

Fanout allows one message to be sent to multiple destinations.

Benefits of Fanout 🎯

  • Push once, and receive in all subscribed SQS queues.
  • Fully decoupled for no data loss.
  • Retries and delayed processing via SQS.
  • Easily add more subscribers over time.
  • Cross-region delivery supported.

Application Example: S3 Event Notifications 📁

  • S3 allows only one event rule per bucket.
  • To send the same S3 event to multiple SQS queues, use SNS fanout.

SNS FIFO + SQS FIFO Fanout 🔄

  • Enables ordering + deduplication while using fanout architecture.

TL;DR 📝

  • SQS: Queue-based model for decoupling, scaling, and message persistence.
  • SNS: Pub/Sub model for broadcasting messages to multiple subscribers.
  • Use SQS FIFO for ordered message processing.
  • SNS + SQS Fanout: Best for sending one message to multiple queues.
  • Visibility Timeout prevents duplicate processing but should be tuned based on the use case.
  • Temporary Queues reduce costs and simplify request-response messaging.
  • Database writes via SQS ensure reliability but require clients to handle asynchronous responses.

AWS messaging services provide scalable, decoupled architectures for microservices and event-driven applications. Choose SQS for queueing, and SNS for broadcasting, and combine them for fanout architectures! 🚀

Post a Comment

Previous Post Next Post