Imagine receiving 50 separate notifications in 10 minutes because 50 people liked your viral post. Or getting the same "Your order has shipped" notification three times because of a retry race condition. These experiences erode user trust and lead to notification opt-outs—the death knell for engagement.

Batching and deduplication are two sides of the same coin: respecting user attention. Batching groups related notifications ("50 people liked your post") instead of bombarding users. Deduplication ensures users never receive the same notification twice, even when distributed systems make this surprisingly difficult.

Notification batching serves multiple stakeholders:

For Users:

• Reduces notification fatigue and interruption frequency
• Provides contextual summaries ("5 new messages" vs. 5 separate alerts)
• Respects attention and improves signal-to-noise ratio
• Prevents the "cry wolf" effect where users ignore all notifications

For the System:

• Reduces load on delivery infrastructure
• Lowers costs (especially for SMS and email)
• Decreases provider rate limit pressure
• Enables more efficient rendering (single grouped notification)

Different scenarios call for different batching approaches. Understanding the trade-offs helps you choose the right strategy for each notification type.

Implementing batching in a distributed system requires careful architecture to handle state, timers, and coordination across multiple workers.

Key Components:

1. Partitioned Processing: Partition notifications by user_id to ensure all notifications for a user are processed by the same worker. This prevents race conditions and enables local batching state.

2. Distributed State (Redis):

# Redis data structures for batching
# Batch contents: List of notification IDs
RPUSH batch:{user_id}:{type} notification_id

# Batch metadata: First notification time, count
HSET batch_meta:{user_id}:{type} first_ts 1703001234 count 5

# TTL ensures orphaned batches expire
EXPIRE batch:{user_id}:{type} 3600

3. Timer Service: Manages delayed flush events. Options include:

• Redis ZADD with scored timestamps, polled periodically
• SQS delay queues or similar
• In-memory timer wheels (for single-process systems)
• Kafka scheduled topics with delayed delivery

4. Exactly-Once Flush: Prevent double-flush when timer fires and count threshold hits simultaneously:

def flush_with_lock(batch_key: str):
    lock_key = f"flush_lock:{batch_key}"
    
    # Try to acquire lock with short TTL
    if redis.set(lock_key, "1", nx=True, ex=5):
        try:
            notifications = redis.lrange(f"batch:{batch_key}", 0, -1)
            redis.delete(f"batch:{batch_key}")
            
            if notifications:
                self.deliver_batch(notifications)
        finally:
            redis.delete(lock_key)

Batched notifications need intelligent summarization. "10 people liked your post" is more useful than just a count—it should highlight who liked it and provide quick actions.

Summary Generation Strategies:

class NotificationSummarizer:
    def summarize(self, notifications: List[Notification]) -> BatchedNotification:
        notification_type = notifications[0].type
        
        if notification_type == 'like':
            return self.summarize_likes(notifications)
        elif notification_type == 'comment':
            return self.summarize_comments(notifications)
        elif notification_type == 'follow':
            return self.summarize_follows(notifications)
        else:
            return self.generic_summary(notifications)
    
    def summarize_likes(self, notifications: List[Notification]) -> BatchedNotification:
        actors = [n.actor for n in notifications]
        content = notifications[0].target_content
        
        # Prioritize actors: friends first, then verified, then by recency
        prioritized = self.prioritize_actors(actors)
        
        count = len(notifications)
        if count == 1:
            title = f"{prioritized[0].name} liked your post"
        elif count == 2:
            title = f"{prioritized[0].name} and {prioritized[1].name} liked your post"
        elif count <= 5:
            title = f"{prioritized[0].name}, {prioritized[1].name}, and {count-2} others liked your post"
        else:
            title = f"{prioritized[0].name} and {count-1} others liked your post"
        
        return BatchedNotification(
            title=title,
            body=f'"{content.preview}"',
            data={
                'type': 'batched_likes',
                'content_id': content.id,
                'actor_ids': [a.id for a in prioritized[:5]],
                'total_count': count,
            },
            actions=[
                Action('view', 'View Post', f'/post/{content.id}'),
            ]
        )

Deduplication ensures users never receive the same notification twice—even when distributed systems, retries, and race conditions conspire to create duplicates. This is harder than it sounds.

Root Cause Analysis:

Duplicates arise from the inherent tension between:

• At-least-once delivery: Messages are never lost (may duplicate)
• Exactly-once delivery: Each message processed exactly once (impossible in distributed systems without coordination)

Since true exactly-once is impractical at scale, we implement at-least-once with application-level deduplication.

Idempotency Keys:

Every notification must carry a unique, deterministic identifier:

def generate_idempotency_key(event: Event) -> str:
    """
    Generate a stable key that identifies this unique notification.
    Same input always produces same key.
    """
    components = [
        event.type,
        event.user_id,
        event.source_id,    # e.g., the post that was liked
        event.actor_id,     # e.g., who liked it
        event.timestamp.date().isoformat(),  # Daily uniqueness
    ]
    
    # Hash to fixed-length key
    content = "|".join(str(c) for c in components)
    return hashlib.sha256(content.encode()).hexdigest()[:32]

Key Design Considerations:

• Include enough context for semantic uniqueness (same like shouldn't notify twice)
• Don't include timestamps at full precision (retries have different timestamps)
• Consider whether re-occurrence should notify (daily login bonus: yes, same like: no)

Implementing deduplication requires a fast lookup mechanism to check if a notification has already been processed or delivered.

Multi-Layer Deduplication:

class DeduplicationService:
    def __init__(self):
        self.bloom = BloomFilter(expected_items=100_000_000, fp_rate=0.01)
        self.redis = RedisClient()
        self.db = DatabaseClient()
    
    def check_and_mark(self, key: str) -> bool:
        """
        Returns True if this is a new notification (should process).
        Returns False if duplicate (should skip).
        """
        
        # Layer 1: Bloom filter (fast negative check)
        if self.bloom.check(key):
            # Might be duplicate, need definitive check
            
            # Layer 2: Redis (recent dedup window)
            if not self.redis.set(f"dedup:{key}", "1", nx=True, ex=86400):
                # Already exists in Redis - definite duplicate
                return False
            
            # Layer 3: Database (persist for compliance)
            try:
                self.db.insert_dedup_key(key)
            except UniqueViolation:
                # Race condition: another process inserted first
                return False
        else:
            # Not in bloom filter - definitely new
            self.bloom.add(key)
            self.redis.set(f"dedup:{key}", "1", ex=86400)
            self.db.insert_dedup_key(key)
        
        return True

TTL Strategy:

When notifications go through multiple channels (push + email for important updates), deduplication becomes more nuanced. Users shouldn't receive the same information twice, but the rules depend on intent.

Implementation Pattern:

class CrossChannelDeduplicator:
    def __init__(self):
        self.delivery_tracker = DeliveryTracker()
    
    def should_deliver(self, notification: Notification, channel: str) -> bool:
        dedup_strategy = notification.cross_channel_strategy
        tracking_key = notification.idempotency_key
        
        if dedup_strategy == 'independent':
            # Each channel operates independently
            return self.single_channel_dedup(tracking_key, channel)
        
        elif dedup_strategy == 'full_dedup':
            # Check if ANY channel already delivered
            return not self.delivery_tracker.any_channel_delivered(tracking_key)
        
        elif dedup_strategy == 'acknowledgment_based':
            # Check if user acknowledged via any channel
            return not self.delivery_tracker.user_acknowledged(tracking_key)
        
        elif dedup_strategy == 'time_delayed':
            # Primary has N minutes before secondary fires
            primary_delivered = self.delivery_tracker.get_delivery_time(
                tracking_key, 
                notification.primary_channel
            )
            
            if primary_delivered:
                time_since = now() - primary_delivered
                if time_since < notification.fallback_delay:
                    return False  # Give primary more time
                    
                if self.delivery_tracker.user_acknowledged(tracking_key):
                    return False  # User already responded
            
            return True
    
    def record_delivery(self, notification: Notification, channel: str):
        self.delivery_tracker.mark_delivered(
            notification.idempotency_key,
            channel,
            timestamp=now()
        )

Real-world batching and deduplication systems encounter numerous edge cases that can cause surprising behavior if not handled carefully.

The "Unlike" Problem in Detail:

def handle_like_event(event: LikeEvent) -> Optional[Notification]:
    """
    Handle the complexity of like/unlike sequences.
    """
    # Get current like state from source of truth
    current_state = self.content_service.get_like_state(
        event.user_id, 
        event.content_id
    )
    
    if not current_state.is_liked:
        # User has since unliked - don't notify
        return None
    
    # Check if we already notified for a like on this content
    dedup_key = f"like:{event.content_owner_id}:{event.content_id}:{event.user_id}"
    notification_window = 86400 * 7  # 7 days
    
    last_notification = self.dedup.get_last_notification_time(dedup_key)
    
    if last_notification and (now() - last_notification) < notification_window:
        # Already notified within window - skip
        return None
    
    # All checks passed - create notification
    return create_like_notification(event)

This handles the sequence: Like → Notify → Unlike → Like (don't notify again within window).

Batching and deduplication are essential for user experience and system efficiency. They transform a potential flood of notifications into a manageable, valuable stream of information.

What's Next:

With batching and deduplication covered, we'll explore User Preferences—the system that gives users control over their notification experience. You'll learn how to design preference schemas, handle inheritance and defaults, and implement preference-aware routing.