Message Queues

Message queues enable asynchronous, decoupled communication between services. A producer sends messages to a queue, a consumer processes them later. Key benefits: decoupling, load leveling, fault tolerance. Popular options: RabbitMQ (AMQP, routing), Apache Kafka (distributed log, high throughput), Amazon SQS (managed). Know Pub/Sub vs Point-to-Point, and at-most-once vs at-least-once vs exactly-once delivery.

Key Concepts

Deep Dive: Point-to-Point vs Pub/Sub

Point-to-Point (Queue):

Producer → Queue → Consumer 1 (processes message)
                   Consumer 2 (gets next message)

Each message consumed by ONE consumer. Good for task distribution.

Pub/Sub (Topic):

Publisher → Topic → Subscriber 1 (gets ALL messages)
                  → Subscriber 2 (gets ALL messages)

Each message delivered to ALL subscribers. Good for event broadcasting.

Deep Dive: Kafka vs RabbitMQ

Feature	Kafka	RabbitMQ
Model	Distributed log	Message broker
Ordering	Per partition	Per queue
Retention	Configurable (days/weeks)	Until consumed
Throughput	Very high (millions/sec)	Moderate (thousands/sec)
Replay	Yes (re-read messages)	No (once consumed, gone)
Routing	Basic (topics + partitions)	Advanced (exchanges, routing keys)
Best for	Event streaming, logs	Task queues, RPC

Kafka architecture:

Producer → Topic (partitioned) → Consumer Group

Topic: [Partition 0] [Partition 1] [Partition 2]
            ↓             ↓             ↓
       Consumer 1    Consumer 2    Consumer 3

Deep Dive: Delivery Guarantees

Guarantee	Description	Use Case
At-most-once	May lose messages. No retries.	Logging, metrics
At-least-once	May duplicate. Retry on failure.	Most applications
Exactly-once	No loss, no duplicates. Hard to achieve.	Financial transactions

At-least-once + idempotency is the most practical approach:

// Consumer with idempotency key
public void processOrder(OrderEvent event) {
    if (processedEvents.contains(event.getId())) {
        return;  // Already processed — skip
    }
    processOrder(event);
    processedEvents.add(event.getId());
}

Deep Dive: Common Use Cases

1. Order processing: User places order → Queue → Payment service processes
2. Email/notifications: User action → Queue → Email service sends async
3. Event sourcing: All state changes stored as events in log
4. Data pipelines: Kafka ingests → Stream processing → Data warehouse
5. Rate leveling: Absorb traffic spikes, process at steady rate

Common Interview Questions

• What is a message queue? Why use one?
• What is the difference between Kafka and RabbitMQ?
• What is Pub/Sub vs Point-to-Point?
• What are delivery guarantees?
• How do you handle duplicate messages?
• What is event sourcing?
• When would you use Kafka vs SQS vs RabbitMQ?
• How does Kafka achieve high throughput?