Publish Subscribe - Learning Module

Loading content...

0/273

Topics and Subscriptions

The Architecture of Event Flow

If one-to-many messaging is the engine of pub-sub, then topics and subscriptions are the road network—the infrastructure that determines where events flow, how they're organized, and who receives them.

Poorly designed topic structures lead to chaos: events end up in wrong places, consumers process irrelevant messages, and the system becomes impossible to reason about. Well-designed topic hierarchies create clarity: events have obvious homes, consumers receive precisely what they need, and the entire system is self-documenting.

This page explores both sides of this fundamental design challenge. We'll examine how to think about topic design, the different subscription models available, and the semantic guarantees that govern message delivery.

What You Will Learn

By the end of this page, you will understand topic naming conventions and hierarchies, the difference between exclusive and shared subscriptions, how to configure acknowledgment and dead-letter handling, and the lifecycle management practices that keep pub-sub systems maintainable.

Understanding Topics

A topic is a named channel or category to which publishers send messages. Topics are the primary organization mechanism in pub-sub systems, providing logical grouping for related events.

What is a Topic?

Depending on the implementation, a topic might be:

A logical routing label (RabbitMQ exchanges): Messages tagged with the topic name are routed to matching subscriptions
A persistent log (Kafka): An append-only data structure that stores messages durably
A messaging destination (Google Pub/Sub, AWS SNS): A managed resource with its own configuration, permissions, and metadata

Regardless of implementation, topics serve the same conceptual purpose: grouping events by domain or type to enable organized fan-out to interested consumers.

Topic Properties

•Name/Identifier: Unique identifier within the namespace (e.g., orders, user-events.signups, analytics.pageviews)
•Schema (optional): Message format specification (Protobuf, Avro, JSON Schema) that validators enforce
•Partitions (in partitioned systems): Subdivisions that enable parallel processing and ordering guarantees
•Retention Policy: How long messages are kept (time-based, size-based, or log-compacted)
•Replication Factor: How many copies of data are maintained for durability
•Access Controls: Which services can publish or subscribe
•Message Configuration: Max message size, compression settings, encryption requirements

Topic vs Queue: A Clarification

Topics and queues serve different purposes, though some systems conflate them:

Concept	Purpose	Delivery Model
Topic	Organize events by category	One-to-many (fan-out)
Queue	Buffer work for processing	One-to-one (competing consumers)

In Kafka, this distinction is subtle because topics ARE logs, and consumer groups create queue-like behavior within topics. In Google Pub/Sub, topics fan out, and each subscription acts as a queue. Understanding your platform's model prevents architectural confusion.

Topic Naming Conventions

Topic names are the primary interface developers interact with. Good naming conventions make systems self-documenting; poor names create confusion and coupling. Establishing naming standards early prevents costly migrations later.

Naming Best Practices

•Use domain prefixes: Group related topics under domain namespaces (e.g., orders.created, orders.shipped, inventory.updated)
•Be specific about events: Include the event type in the name (e.g., user.signup.completed vs just user)
•Use consistent delimiters: Pick one (dots, hyphens, underscores) and stick with it across the organization
•Include environment (if shared cluster): prod.orders.created, staging.orders.created
•Avoid version numbers in names: Use schema versioning instead of orders.v2.created
•Make names searchable: Use full words (payment-processed) not abbreviations (pmt-proc)
•Consider wildcards: If your system supports wildcard subscriptions, design hierarchies that enable them (analytics.* captures all analytics events)

Topic Naming Examples
Pattern	Example	Use Case
{domain}.{entity}.{action}	orders.order.created	Standard event sourcing pattern
{env}.{domain}.{event}	prod.payments.payment-completed	Multi-environment clusters
{team}.{service}.{event}	platform.auth.session-started	Team ownership model
{aggregate}.{version}.{event}	customer.v2.address-changed	Versioned event streams (use sparingly)
{region}.{domain}.{event}	us-east.inventory.stock-updated	Geo-distributed systems

Avoid These Anti-Patterns

Generic names like 'events', 'messages', or 'data'—impossible to find or understand. Technology in names like 'kafka-orders' or 'pubsub-events'—couples to implementation. Consumer in names like 'orders-for-analytics'—creates implicit coupling to specific consumers.

Topic Hierarchies and Wildcards

Many pub-sub systems support hierarchical topic structures with wildcard subscriptions, enabling flexible routing patterns without enumerating every topic explicitly.

Hierarchical Structure

Topics organized in a tree-like hierarchy:

ecommerce/
├── orders/
│   ├── created
│   ├── updated
│   ├── shipped
│   └── cancelled
├── inventory/
│   ├── stock-updated
│   └── reorder-triggered
└── payments/
    ├── initiated
    ├── completed
    └── failed

Wildcard Subscriptions (where supported)

Wildcards let subscribers match multiple topics:

Wildcard	Meaning	Example	Matches
`*` (single-level)	Any single segment	`orders.*`	`orders.created`, `orders.shipped`
`#` or `>` (multi-level)	Any remaining path	`ecommerce.#`	`ecommerce.orders.created`, `ecommerce.payments.failed`

This enables powerful patterns like:

Audit service subscribes to # (all events)
Analytics subscribes to *.created (all creation events)
Order processing subscribes to orders.* (all order events)

Platform Differences

RabbitMQ supports wildcards natively with topic exchanges. Kafka does not support wildcard subscriptions—consumers specify exact topic names (though Kafka Streams supports pattern matching). Google Pub/Sub requires explicit subscriptions per topic. Design your hierarchy based on your platform's capabilities.

Converting Mermaid diagram...

Understanding Subscriptions

A subscription is the binding between a topic and a consumer (or consumer group). It defines who receives messages from a topic and how they receive them.

Subscriptions are where pub-sub systems provide control over delivery semantics, acknowledgment behavior, and failure handling.

Subscription Properties

•Subscription Name/ID: Unique identifier for this subscription (e.g., portfolio-service-trades-subscription)
•Topic Binding: Which topic(s) this subscription receives from
•Delivery Type: Push (broker initiates) or pull (consumer initiates)
•Acknowledgment Mode: Manual ack, auto-ack, or batch ack
•Acknowledgment Deadline: How long before unacknowledged messages are redelivered
•Retry Policy: How many times and with what backoff to retry failed deliveries
•Dead Letter Configuration: Where to send messages that exhaust retries
•Filter Expression (if supported): Rules to receive only matching messages
•Message Ordering (if applicable): Whether to preserve order within partitions/keys

subscription-configuration.ts
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
// Example: Google Pub/Sub subscription configuration
import { PubSub } from '@google-cloud/pubsub';
 
const pubsub = new PubSub();
 
async function createSubscription(): Promise<void> {
  const [subscription] = await pubsub.topic('trades').createSubscription(
    'portfolio-service-trades-sub',
    {
      // Acknowledgment deadline: 30 seconds to process
      ackDeadlineSeconds: 30,
      
      // Retry policy for failed deliveries
      retryPolicy: {
        minimumBackoff: { seconds: 10 },
        maximumBackoff: { seconds: 600 }, // 10 minutes max
      },
      
      // Dead letter queue for messages that fail permanently
      deadLetterPolicy: {
        deadLetterTopic: 'projects/my-project/topics/trades-dlq',
        maxDeliveryAttempts: 5,
      },
      
      // Message retention (24 hours if not acked, independent of topic)
      messageRetentionDuration: { seconds: 86400 },
      
      // Filter to only receive BUY orders
      filter: 'attributes.side = "BUY"',
      
      // Enable message ordering (requires ordering key on publish)
      enableMessageOrdering: true,
    }
  );
  
  console.log(`Subscription ${subscription.name} created`);
}

Subscription Types: Exclusive vs Shared

How messages are distributed to consumers within a subscription determines whether you get pub-sub (fan-out) or queue (competing consumer) behavior. Most systems support both, controlled by the subscription type.

Exclusive Subscription: Each subscription receives all messages, and only one consumer can be active per subscription at a time.

Behavior:

One active consumer per subscription (second consumer rejected or put on standby)
Consumer receives ALL messages from topic (true fan-out to subscriptions)
If consumer fails, another can take over (failover)

Use Cases:

Stateful consumers that must see complete event stream
Event sourcing where order and completeness matter
Each service: own subscription, complete view

Example Architecture:

Topic: trades
Subscription A: portfolio-service-trades → Portfolio Service (exclusive)
Subscription B: risk-service-trades → Risk Engine (exclusive)
Each service sees ALL trades, processed exclusively by that service's single instance

Apache Pulsar
1
2
3
4
5
6
7
8
9
10
// Apache Pulsar: Exclusive subscription
// Only ONE consumer can attach at a time
Consumer<TradeEvent> consumer = pulsarClient.newConsumer(Schema.JSON(TradeEvent.class))
    .topic("trades")
    .subscriptionName("portfolio-service-trades")
    .subscriptionType(SubscriptionType.Exclusive) // Key setting
    .subscribe();
 
// Second consumer with same subscription name will fail
// Use for stateful processing that needs complete stream

Kafka's Approach: Consumer Groups

Kafka combines subscription and consumer group concepts. A consumer group name IS the subscription. Within the group:

Partitions are assigned to consumers (not individual messages)
Each partition → exactly one consumer (exclusive at partition level)
Multiple partitions → can go to same or different consumers
Adding consumers → partitions rebalance

This hybrid model provides both ordered processing (per partition) and horizontal scaling (across partitions).

Acknowledgment Semantics

Acknowledgment is the mechanism by which consumers communicate to the broker that messages have been successfully processed. Proper acknowledgment handling is critical for reliability—misconfiguration leads to message loss or infinite redelivery loops.

Acknowledgment Modes
Mode	Description	Risk	Use Case
Auto-Ack on Receive	Broker considers delivered = acknowledged	Message loss if consumer crashes during processing	Logs, metrics where loss acceptable
Manual Ack After Process	Consumer explicitly acks after successful processing	Redelivery if slow; must handle duplicates	Standard reliable processing
Batch Ack	Consumer acks multiple messages at once	Larger window of potential redelivery	High-throughput when individual ack too expensive
Negative Ack (Nack)	Consumer signals failure; broker redelivers	Must avoid infinite retry loops	Transient failures that may succeed on retry

The Ack Lifecycle

Message Delivered: Broker sends message to consumer
Ack Deadline Starts: Timer begins (e.g., 30 seconds)
Consumer Processes: Business logic executes
Consumer Acks: Success → message complete; Nack → redelivery
Deadline Expires: If no ack/nack → automatic redelivery

Critical Considerations:

Set appropriate deadlines: Too short → unnecessary redelivery; too long → stuck messages
Extend deadlines for long processing: Some APIs let you extend the deadline mid-processing
Ack AFTER side effects complete: Don't ack until database writes, API calls, etc. are confirmed
Handle duplicates: Redelivery means you WILL see messages more than once; ensure idempotency

acknowledgment-patterns.ts
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
// Pattern: Reliable acknowledgment after processing
async function processWithReliableAck(message: Message): Promise<void> {
  const event = deserialize(message.data);
  
  try {
    // 1. Validate message (nak immediately if invalid)
    if (!isValid(event)) {
      await message.nack(); // Send to dead-letter after max retries
      return;
    }
    
    // 2. Extend deadline if processing will be long
    if (estimateProcessingTime(event) > 25_000) {
      await message.modifyAckDeadline(60); // Extend to 60 seconds
    }
    
    // 3. Process with idempotency check
    const alreadyProcessed = await checkIdempotencyKey(event.eventId);
    if (!alreadyProcessed) {
      await processEvent(event);
      await recordIdempotencyKey(event.eventId);
    }
    
    // 4. ACK only after ALL side effects complete
    await message.ack();
    
  } catch (error) {
    if (isRetryable(error)) {
      // Transient failure - nack for redelivery
      await message.nack();
    } else {
      // Permanent failure - ack to prevent infinite retry
      // (in production, send to dead-letter queue first)
      logger.error(`Permanent failure for ${event.eventId}`, error);
      await message.ack(); // Or let DLQ policy handle it
    }
  }
}

The Ack-Before-Process Anti-Pattern

Never acknowledge a message before processing completes. If your consumer crashes after ack but before the database write, the message is lost forever. This is the most common cause of data loss in messaging systems.

Dead Letter Queues (DLQ)

A Dead Letter Queue (DLQ) is a specialized destination for messages that cannot be processed successfully after all retry attempts. DLQs prevent poison messages from blocking the pipeline indefinitely while preserving them for investigation.

Why DLQs Are Essential:

Without a DLQ, a malformed or unprocessable message creates an infinite retry loop:

Message arrives
Consumer crashes or fails
Message redelivered
Consumer fails again
Repeat forever... blocking all subsequent messages (head-of-line blocking)

With a DLQ:

Message arrives
Consumer fails (attempt 1)
Redelivered, fails again (attempts 2, 3, 4, 5)
Exceeds max attempts → moved to DLQ
Main queue continues processing; DLQ message awaits human review

DLQ Best Practices

•Configure DLQ for every subscription: Don't assume failures won't happen
•Preserve context: Include original message, error details, attempt count, timestamps
•Alert on DLQ arrivals: Dead letters indicate problems; don't let them accumulate silently
•Build reprocessing tooling: Ability to fix and replay DLQ messages back to main topic
•Set DLQ retention: DLQ messages need retention too; don't expire before you can investigate
•Separate DLQ per subscription: Enables targeted debugging per consumer

Converting Mermaid diagram...

dead-letter-handling.ts
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
// Example: DLQ message structure with full context
interface DeadLetterMessage<T> {
  originalMessage: T;
  originalTopic: string;
  subscription: string;
  
  // Failure context
  errorMessage: string;
  errorStack: string;
  failureTimestamp: string;
  attemptCount: number;
  
  // Original metadata
  originalEventId: string;
  originalTimestamp: string;
  originalHeaders: Record<string, string>;
}
 
// Publishing to DLQ with context
async function sendToDeadLetter<T>(
  message: Message,
  error: Error,
  attemptCount: number
): Promise<void> {
  const dlqMessage: DeadLetterMessage<T> = {
    originalMessage: JSON.parse(message.data.toString()),
    originalTopic: message.attributes.topic,
    subscription: 'order-processor-sub',
    
    errorMessage: error.message,
    errorStack: error.stack || '',
    failureTimestamp: new Date().toISOString(),
    attemptCount,
    
    originalEventId: message.id,
    originalTimestamp: message.publishTime.toISOString(),
    originalHeaders: message.attributes,
  };
  
  await deadLetterTopic.publish(
    Buffer.from(JSON.stringify(dlqMessage)),
    { originalTopic: message.attributes.topic }
  );
  
  // Alert operations team
  await alerting.notify('dlq-arrival', {
    topic: message.attributes.topic,
    error: error.message,
    messageId: message.id,
  });
}

Subscription Lifecycle Management

Subscriptions have lifecycles that must be managed carefully. Creating, updating, and deleting subscriptions requires coordination to avoid message loss or processing gaps.

Lifecycle Stages and Considerations

•Creation: Define before deploying consumers. Messages to a topic with no subscriptions may be lost (depends on system). Consider subscription backlog retention.
•Initial Position: Start from earliest message, latest message, or specific timestamp? This determines whether you process historical events.
•Consumer Attachment: Consumers connect to existing subscription. Handle connection failures, reconnection logic, and rebalancing gracefully.
•Configuration Updates: Some properties can be changed live (ack deadline); others require recreation (filter expression varies by platform).
•Scaling: Adding/removing consumers triggers rebalancing. Design for partition reassignment without losing progress.
•Pause/Resume: Some systems support pausing delivery without deleting the subscription. Messages accumulate during pause.
•Deletion: Deleting a subscription is permanent. Ensure all consumers are stopped first. Consider the subscription's backlog—unprocessed messages are lost.

Safe Subscription Deletion

•Stop all consumers gracefully
•Drain backlog (process remaining messages)
•Verify no messages in flight
•Delete subscription
•Remove configuration from deployment manifests

Dangerous Deletion Patterns

•Deleting while consumers active (processing interruption)
•Deleting with unprocessed backlog (data loss)
•Deleting without removing consumer code (restart creates new subscription with different position)
•Deleting shared subscription without coordinating all consumer owners

Infrastructure as Code

Manage subscriptions through infrastructure-as-code (Terraform, Pulumi). This ensures subscriptions exist before deployment, tracks configuration in version control, and prevents drift between environments. Never create subscriptions manually in production.

Summary: Topics and Subscriptions

We've explored the organizational primitives that structure every pub-sub system. Let's consolidate the key concepts:

Key Takeaways

•Topics organize events by domain: Use consistent naming conventions with hierarchical prefixes to create self-documenting structures.
•Subscriptions define how consumers receive: Exclusive subscriptions provide complete ordered streams; shared subscriptions enable horizontal scaling.
•Acknowledgment semantics control reliability: Manual ack after processing prevents message loss; set appropriate deadlines and handle retries.
•Dead Letter Queues catch poison messages: Configure DLQs on every subscription to prevent infinite retry loops and enable investigation.
•Lifecycle management prevents gaps: Create subscriptions before deploying consumers; delete carefully after draining.
•Platform differences matter: Kafka consumer groups, Pulsar subscription types, and cloud pub-sub services have different models—understand your platform.

What's Next:

With a solid understanding of topics and subscriptions, we'll explore fan-out patterns—the architectural approaches that leverage pub-sub's one-to-many capabilities for building reactive, event-driven systems at scale.

Page Complete

You now understand the building blocks of pub-sub organization. Topics provide logical grouping for events; subscriptions define delivery to consumers; acknowledgments ensure reliability; and dead-letter queues handle failures gracefully. These primitives combine to create the flexible, scalable messaging infrastructure that powers event-driven architectures.

Topics and Subscriptions

The Architecture of Event Flow

What You Will Learn

Understanding Topics

A topic is a named channel or category to which publishers send messages. Topics are the primary organization mechanism in pub-sub systems, providing logical grouping for related events.

What is a Topic?

Depending on the implementation, a topic might be:

A logical routing label (RabbitMQ exchanges): Messages tagged with the topic name are routed to matching subscriptions
A persistent log (Kafka): An append-only data structure that stores messages durably
A messaging destination (Google Pub/Sub, AWS SNS): A managed resource with its own configuration, permissions, and metadata

Regardless of implementation, topics serve the same conceptual purpose: grouping events by domain or type to enable organized fan-out to interested consumers.

Topic Properties

•Name/Identifier: Unique identifier within the namespace (e.g., orders, user-events.signups, analytics.pageviews)
•Schema (optional): Message format specification (Protobuf, Avro, JSON Schema) that validators enforce
•Partitions (in partitioned systems): Subdivisions that enable parallel processing and ordering guarantees
•Retention Policy: How long messages are kept (time-based, size-based, or log-compacted)
•Replication Factor: How many copies of data are maintained for durability
•Access Controls: Which services can publish or subscribe
•Message Configuration: Max message size, compression settings, encryption requirements

Topic vs Queue: A Clarification

Topics and queues serve different purposes, though some systems conflate them:

Concept	Purpose	Delivery Model
Topic	Organize events by category	One-to-many (fan-out)
Queue	Buffer work for processing	One-to-one (competing consumers)

Topic Naming Conventions

Naming Best Practices

•Use domain prefixes: Group related topics under domain namespaces (e.g., orders.created, orders.shipped, inventory.updated)
•Be specific about events: Include the event type in the name (e.g., user.signup.completed vs just user)
•Use consistent delimiters: Pick one (dots, hyphens, underscores) and stick with it across the organization
•Include environment (if shared cluster): prod.orders.created, staging.orders.created
•Avoid version numbers in names: Use schema versioning instead of orders.v2.created
•Make names searchable: Use full words (payment-processed) not abbreviations (pmt-proc)
•Consider wildcards: If your system supports wildcard subscriptions, design hierarchies that enable them (analytics.* captures all analytics events)

Topic Naming Examples
Pattern	Example	Use Case
{domain}.{entity}.{action}	orders.order.created	Standard event sourcing pattern
{env}.{domain}.{event}	prod.payments.payment-completed	Multi-environment clusters
{team}.{service}.{event}	platform.auth.session-started	Team ownership model
{aggregate}.{version}.{event}	customer.v2.address-changed	Versioned event streams (use sparingly)
{region}.{domain}.{event}	us-east.inventory.stock-updated	Geo-distributed systems

Avoid These Anti-Patterns

Topic Hierarchies and Wildcards

Many pub-sub systems support hierarchical topic structures with wildcard subscriptions, enabling flexible routing patterns without enumerating every topic explicitly.

Hierarchical Structure

Topics organized in a tree-like hierarchy:

ecommerce/
├── orders/
│   ├── created
│   ├── updated
│   ├── shipped
│   └── cancelled
├── inventory/
│   ├── stock-updated
│   └── reorder-triggered
└── payments/
    ├── initiated
    ├── completed
    └── failed

Wildcard Subscriptions (where supported)

Wildcards let subscribers match multiple topics:

Wildcard	Meaning	Example	Matches
`*` (single-level)	Any single segment	`orders.*`	`orders.created`, `orders.shipped`
`#` or `>` (multi-level)	Any remaining path	`ecommerce.#`	`ecommerce.orders.created`, `ecommerce.payments.failed`

This enables powerful patterns like:

Audit service subscribes to # (all events)
Analytics subscribes to *.created (all creation events)
Order processing subscribes to orders.* (all order events)

Platform Differences

Converting Mermaid diagram...

Understanding Subscriptions

A subscription is the binding between a topic and a consumer (or consumer group). It defines who receives messages from a topic and how they receive them.

Subscriptions are where pub-sub systems provide control over delivery semantics, acknowledgment behavior, and failure handling.

Subscription Properties

•Subscription Name/ID: Unique identifier for this subscription (e.g., portfolio-service-trades-subscription)
•Topic Binding: Which topic(s) this subscription receives from
•Delivery Type: Push (broker initiates) or pull (consumer initiates)
•Acknowledgment Mode: Manual ack, auto-ack, or batch ack
•Acknowledgment Deadline: How long before unacknowledged messages are redelivered
•Retry Policy: How many times and with what backoff to retry failed deliveries
•Dead Letter Configuration: Where to send messages that exhaust retries
•Filter Expression (if supported): Rules to receive only matching messages
•Message Ordering (if applicable): Whether to preserve order within partitions/keys

subscription-configuration.ts
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
// Example: Google Pub/Sub subscription configuration
import { PubSub } from '@google-cloud/pubsub';
 
const pubsub = new PubSub();
 
async function createSubscription(): Promise<void> {
  const [subscription] = await pubsub.topic('trades').createSubscription(
    'portfolio-service-trades-sub',
    {
      // Acknowledgment deadline: 30 seconds to process
      ackDeadlineSeconds: 30,
      
      // Retry policy for failed deliveries
      retryPolicy: {
        minimumBackoff: { seconds: 10 },
        maximumBackoff: { seconds: 600 }, // 10 minutes max
      },
      
      // Dead letter queue for messages that fail permanently
      deadLetterPolicy: {
        deadLetterTopic: 'projects/my-project/topics/trades-dlq',
        maxDeliveryAttempts: 5,
      },
      
      // Message retention (24 hours if not acked, independent of topic)
      messageRetentionDuration: { seconds: 86400 },
      
      // Filter to only receive BUY orders
      filter: 'attributes.side = "BUY"',
      
      // Enable message ordering (requires ordering key on publish)
      enableMessageOrdering: true,
    }
  );
  
  console.log(`Subscription ${subscription.name} created`);
}

Subscription Types: Exclusive vs Shared

Exclusive Subscription: Each subscription receives all messages, and only one consumer can be active per subscription at a time.

Behavior:

One active consumer per subscription (second consumer rejected or put on standby)
Consumer receives ALL messages from topic (true fan-out to subscriptions)
If consumer fails, another can take over (failover)

Use Cases:

Stateful consumers that must see complete event stream
Event sourcing where order and completeness matter
Each service: own subscription, complete view

Example Architecture:

Topic: trades
Subscription A: portfolio-service-trades → Portfolio Service (exclusive)
Subscription B: risk-service-trades → Risk Engine (exclusive)
Each service sees ALL trades, processed exclusively by that service's single instance

Apache Pulsar
1
2
3
4
5
6
7
8
9
10
// Apache Pulsar: Exclusive subscription
// Only ONE consumer can attach at a time
Consumer<TradeEvent> consumer = pulsarClient.newConsumer(Schema.JSON(TradeEvent.class))
    .topic("trades")
    .subscriptionName("portfolio-service-trades")
    .subscriptionType(SubscriptionType.Exclusive) // Key setting
    .subscribe();
 
// Second consumer with same subscription name will fail
// Use for stateful processing that needs complete stream

Kafka's Approach: Consumer Groups

Kafka combines subscription and consumer group concepts. A consumer group name IS the subscription. Within the group:

Partitions are assigned to consumers (not individual messages)
Each partition → exactly one consumer (exclusive at partition level)
Multiple partitions → can go to same or different consumers
Adding consumers → partitions rebalance

This hybrid model provides both ordered processing (per partition) and horizontal scaling (across partitions).

Acknowledgment Semantics

Acknowledgment Modes
Mode	Description	Risk	Use Case
Auto-Ack on Receive	Broker considers delivered = acknowledged	Message loss if consumer crashes during processing	Logs, metrics where loss acceptable
Manual Ack After Process	Consumer explicitly acks after successful processing	Redelivery if slow; must handle duplicates	Standard reliable processing
Batch Ack	Consumer acks multiple messages at once	Larger window of potential redelivery	High-throughput when individual ack too expensive
Negative Ack (Nack)	Consumer signals failure; broker redelivers	Must avoid infinite retry loops	Transient failures that may succeed on retry

The Ack Lifecycle

Message Delivered: Broker sends message to consumer
Ack Deadline Starts: Timer begins (e.g., 30 seconds)
Consumer Processes: Business logic executes
Consumer Acks: Success → message complete; Nack → redelivery
Deadline Expires: If no ack/nack → automatic redelivery

Critical Considerations:

Set appropriate deadlines: Too short → unnecessary redelivery; too long → stuck messages
Extend deadlines for long processing: Some APIs let you extend the deadline mid-processing
Ack AFTER side effects complete: Don't ack until database writes, API calls, etc. are confirmed
Handle duplicates: Redelivery means you WILL see messages more than once; ensure idempotency

acknowledgment-patterns.ts
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
// Pattern: Reliable acknowledgment after processing
async function processWithReliableAck(message: Message): Promise<void> {
  const event = deserialize(message.data);
  
  try {
    // 1. Validate message (nak immediately if invalid)
    if (!isValid(event)) {
      await message.nack(); // Send to dead-letter after max retries
      return;
    }
    
    // 2. Extend deadline if processing will be long
    if (estimateProcessingTime(event) > 25_000) {
      await message.modifyAckDeadline(60); // Extend to 60 seconds
    }
    
    // 3. Process with idempotency check
    const alreadyProcessed = await checkIdempotencyKey(event.eventId);
    if (!alreadyProcessed) {
      await processEvent(event);
      await recordIdempotencyKey(event.eventId);
    }
    
    // 4. ACK only after ALL side effects complete
    await message.ack();
    
  } catch (error) {
    if (isRetryable(error)) {
      // Transient failure - nack for redelivery
      await message.nack();
    } else {
      // Permanent failure - ack to prevent infinite retry
      // (in production, send to dead-letter queue first)
      logger.error(`Permanent failure for ${event.eventId}`, error);
      await message.ack(); // Or let DLQ policy handle it
    }
  }
}

The Ack-Before-Process Anti-Pattern

Dead Letter Queues (DLQ)

Why DLQs Are Essential:

Without a DLQ, a malformed or unprocessable message creates an infinite retry loop:

Message arrives
Consumer crashes or fails
Message redelivered
Consumer fails again
Repeat forever... blocking all subsequent messages (head-of-line blocking)

With a DLQ:

Message arrives
Consumer fails (attempt 1)
Redelivered, fails again (attempts 2, 3, 4, 5)
Exceeds max attempts → moved to DLQ
Main queue continues processing; DLQ message awaits human review

DLQ Best Practices

•Configure DLQ for every subscription: Don't assume failures won't happen
•Preserve context: Include original message, error details, attempt count, timestamps
•Alert on DLQ arrivals: Dead letters indicate problems; don't let them accumulate silently
•Build reprocessing tooling: Ability to fix and replay DLQ messages back to main topic
•Set DLQ retention: DLQ messages need retention too; don't expire before you can investigate
•Separate DLQ per subscription: Enables targeted debugging per consumer

Converting Mermaid diagram...

dead-letter-handling.ts
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
// Example: DLQ message structure with full context
interface DeadLetterMessage<T> {
  originalMessage: T;
  originalTopic: string;
  subscription: string;
  
  // Failure context
  errorMessage: string;
  errorStack: string;
  failureTimestamp: string;
  attemptCount: number;
  
  // Original metadata
  originalEventId: string;
  originalTimestamp: string;
  originalHeaders: Record<string, string>;
}
 
// Publishing to DLQ with context
async function sendToDeadLetter<T>(
  message: Message,
  error: Error,
  attemptCount: number
): Promise<void> {
  const dlqMessage: DeadLetterMessage<T> = {
    originalMessage: JSON.parse(message.data.toString()),
    originalTopic: message.attributes.topic,
    subscription: 'order-processor-sub',
    
    errorMessage: error.message,
    errorStack: error.stack || '',
    failureTimestamp: new Date().toISOString(),
    attemptCount,
    
    originalEventId: message.id,
    originalTimestamp: message.publishTime.toISOString(),
    originalHeaders: message.attributes,
  };
  
  await deadLetterTopic.publish(
    Buffer.from(JSON.stringify(dlqMessage)),
    { originalTopic: message.attributes.topic }
  );
  
  // Alert operations team
  await alerting.notify('dlq-arrival', {
    topic: message.attributes.topic,
    error: error.message,
    messageId: message.id,
  });
}

Subscription Lifecycle Management

Subscriptions have lifecycles that must be managed carefully. Creating, updating, and deleting subscriptions requires coordination to avoid message loss or processing gaps.

Lifecycle Stages and Considerations

•Creation: Define before deploying consumers. Messages to a topic with no subscriptions may be lost (depends on system). Consider subscription backlog retention.
•Initial Position: Start from earliest message, latest message, or specific timestamp? This determines whether you process historical events.
•Consumer Attachment: Consumers connect to existing subscription. Handle connection failures, reconnection logic, and rebalancing gracefully.
•Configuration Updates: Some properties can be changed live (ack deadline); others require recreation (filter expression varies by platform).
•Scaling: Adding/removing consumers triggers rebalancing. Design for partition reassignment without losing progress.
•Pause/Resume: Some systems support pausing delivery without deleting the subscription. Messages accumulate during pause.
•Deletion: Deleting a subscription is permanent. Ensure all consumers are stopped first. Consider the subscription's backlog—unprocessed messages are lost.

Safe Subscription Deletion

•Stop all consumers gracefully
•Drain backlog (process remaining messages)
•Verify no messages in flight
•Delete subscription
•Remove configuration from deployment manifests

Dangerous Deletion Patterns

•Deleting while consumers active (processing interruption)
•Deleting with unprocessed backlog (data loss)
•Deleting without removing consumer code (restart creates new subscription with different position)
•Deleting shared subscription without coordinating all consumer owners

Infrastructure as Code

Summary: Topics and Subscriptions

We've explored the organizational primitives that structure every pub-sub system. Let's consolidate the key concepts:

Key Takeaways

•Topics organize events by domain: Use consistent naming conventions with hierarchical prefixes to create self-documenting structures.
•Subscriptions define how consumers receive: Exclusive subscriptions provide complete ordered streams; shared subscriptions enable horizontal scaling.
•Acknowledgment semantics control reliability: Manual ack after processing prevents message loss; set appropriate deadlines and handle retries.
•Dead Letter Queues catch poison messages: Configure DLQs on every subscription to prevent infinite retry loops and enable investigation.
•Lifecycle management prevents gaps: Create subscriptions before deploying consumers; delete carefully after draining.
•Platform differences matter: Kafka consumer groups, Pulsar subscription types, and cloud pub-sub services have different models—understand your platform.

What's Next:

Page Complete