|$ curl https://forge-ai.dev/api/markdown?path=docs/system-design/messaging

$cat docs/messaging.md

updated Recently·30 min read·published

Messaging

◆System Design◆Advanced🎯Free Tools

Introduction

Messaging systems are the nervous system of distributed architectures. They decouple services in time and space, buffer traffic spikes, and enable event-driven workflows. A service can publish a message without knowing who will consume it or when they will be available.

The two dominant messaging models are queues and publish/subscribe. Queues deliver each message to exactly one consumer, making them ideal for work distribution. Pub/sub broadcasts each message to all subscribers, making it ideal for event propagation.

This guide covers queues, pub/sub, Kafka, event-driven architecture, delivery guarantees, ordering, and the idempotent consumers that make messaging reliable.

Message Queues

A message queue holds messages until a consumer processes them. Multiple producers can write to the queue, and multiple consumers can read from it, but each message is typically delivered to only one consumer. This pattern balances load and isolates producers from consumer speed.

queue.txt

TEXT

1	Queue architecture:
2
3	Producer A ──┐
4	│
5	Producer B ──┼──▶ Queue ──┐
6	│ │
7	Producer C ──┘ │
8	┌────────┼────────┐
9	▼ ▼ ▼
10	Consumer1 Consumer2 Consumer3
11
12	Each message is picked up by one consumer.
13	If a consumer fails, the message is redelivered
14	or moved to a dead-letter queue.

Feature	Use Case
Load leveling	Absorb traffic spikes without overwhelming consumers
Work distribution	Spread tasks across a pool of workers
Decoupling	Producer does not wait for consumer
Retry	Failed messages are retried or dead-lettered

✓

best practice

Keep messages small. Large messages increase broker memory pressure and slow replication. If payloads are large, store them in object storage and pass a reference in the message.

Publish/Subscribe

In pub/sub, publishers send messages to topics without knowledge of subscribers. Subscribers register interest in topics and receive all messages published to them. This pattern enables one-to-many communication and event-driven architectures.

pubsub.txt

TEXT

1	Pub/sub architecture:
2
3	Publisher ──▶ Topic: order-placed
4	│
5	┌───────────┼───────────┐
6	▼ ▼ ▼
7	Subscriber1 Subscriber2 Subscriber3
8	Inventory Notification Analytics
9
10	All subscribers receive a copy of the event.
11	Subscribers process independently and at their own pace.

ℹ

info

Use topics to model business events, not commands. A topic named order-placed describes something that happened. A topic named process-payment is a command and couples publishers to subscriber behavior.

Delivery Guarantees

Message brokers offer different delivery guarantees. The choice affects complexity, performance, and correctness. In practice, most distributed systems choose at-least-once delivery and build idempotent consumers.

Guarantee	Meaning	Tradeoff
At-most-once	Message may be lost, never duplicated	Fast, but unsafe for critical data
At-least-once	Message delivered one or more times	Requires idempotent consumers
Exactly-once	Message delivered exactly once	Higher latency, more coordination

⚠

warning

Exactly-once semantics are expensive and often unnecessary. Idempotent at-least-once delivery is the pragmatic default for most systems. True exactly-once requires distributed transactions or idempotent deduplication at the consumer.

Message Ordering

Ordering is one of the trickiest aspects of messaging. Global ordering across all messages is expensive and rarely needed. Most systems need ordering only within a partition or for a specific entity, such as all events for a given user or order.

ordering.txt

TEXT

1	Partitioned ordering:
2
3	Topic: user-events
4	Partitions: 8
5	Key: user_id
6
7	Events for user A ──▶ partition 3
8	Events for user B ──▶ partition 7
9	Events for user C ──▶ partition 1
10
11	Ordering is preserved per user because all
12	messages for that user go to the same partition.
13	Different users can be processed in parallel.

📝

note

To preserve order for an entity, route all related events to the same partition using a consistent key. If a single partition becomes a hotspot, consider further sharding or relaxing ordering requirements.

Apache Kafka

Apache Kafka is a distributed event streaming platform. It combines the durability of a log with the scalability of a partitioned, replicated system. Kafka is used for event sourcing, real-time analytics, log aggregation, and as a backbone for event-driven microservices.

kafka.txt

TEXT

1	Kafka core concepts:
2
3	Topic: A category of events (e.g., order-events)
4	Partition: A shard of a topic; ordered, append-only log
5	Offset: Position of a record within a partition
6	Producer: Writes events to topics
7	Consumer: Reads events from topics
8	Consumer group: A set of consumers that share work
9	Broker: A Kafka server that stores partitions
10
11	Replication factor 3 means each partition has 3 copies
12	across different brokers for fault tolerance.

Kafka Feature	Use Case
High throughput	Millions of events per second
Durability	Persistent log, configurable retention
Replay	Reprocess historical events
Partitions	Parallelism and ordered streams
Consumer groups	Scalable consumption

ℹ

info

Tune retention carefully. Too short and you lose the ability to replay events; too long and storage costs grow. Use compacted topics for event sourcing where only the latest state per key matters.

Idempotent Consumers

Because messages can be redelivered, consumers must handle duplicates without side effects. An idempotent consumer produces the same outcome whether it processes a message once or many times. This is usually achieved by tracking processed message IDs or making operations naturally idempotent.

idempotent-consumer.txt

TEXT

1	Idempotent consumer pattern:
2
3	function processMessage(message) {
4	const idempotencyKey = message.idempotencyKey;
5
6	if (processedStore.has(idempotencyKey)) {
7	return; // already handled
8	}
9
10	db.transaction(() => {
11	applyBusinessLogic(message);
12	processedStore.save(idempotencyKey);
13	});
14	}
15
16	The idempotency store is usually a database table
17	with a unique constraint on the key.

✓

best practice

Store the idempotency record in the same transaction as the business update. If either fails, both roll back, preventing partial state and lost idempotency records.

Event-Driven Architecture

Event-driven architecture uses events as the primary mechanism for communication between services. Services react to events rather than invoking each other directly. This decouples producers and consumers, enables independent evolution, and supports real-time processing.

event-driven.txt

TEXT

1	Event-driven order flow:
2
3	Order Service ──OrderPlaced──▶ Event Bus
4	│
5	┌─────────────────────┼─────────────────────┐
6	▼ ▼ ▼
7	Payment Service Inventory Service Notification Service
8	│ │ │
9	▼ ▼ ▼
10	PaymentProcessed InventoryReserved EmailSent
11	│ │
12	└──────────┬──────────┘
13	▼
14	Fulfillment Service
15
16	Each service owns its reaction to the event.
17	No central orchestrator is required.

⚠

warning

Event-driven systems are harder to debug than synchronous systems. Distributed traces and event schemas are essential. Without them, understanding the flow of a single business transaction becomes painful.

Sagas and Compensation

In event-driven systems, business transactions often span multiple services. A saga coordinates these steps using a sequence of events and compensating actions. If one step fails, compensating actions undo earlier steps.

Saga Type	Coordination	Best For
Choreography	Services react to each other's events	Simple flows, loose coupling
Orchestration	Central coordinator emits commands	Complex flows, visibility

🔥

pro tip

Use choreography for simple, linear sagas with few participants. Use orchestration when the flow has branches, loops, or timeouts and you need a single place to monitor progress.

$Blueprint — Engineering Documentation·Section ID: SD-MSG-01·Revision: 1.0

X GitHub YouTube Instagram Facebook Discord Slack Telegram Buy me a coffee Ko-fi Sponsor