The Instagram home feed is where users spend most of their time—scrolling through a curated stream of photos, videos, and stories from accounts they follow. What appears to be a simple chronological list is actually the output of one of the most sophisticated content ranking systems in existence.

Consider the challenge: A typical user follows 200-500 accounts. Those accounts collectively produce dozens of new posts per day. Instagram must determine which posts to show, in what order, and when—all while rendering a feed that feels personal, timely, and engaging.

Now multiply this by 500 million daily active users, each requesting their feed multiple times per day. The feed generation system handles 10+ billion feed renders per day, each requiring aggregation from hundreds of sources, ranking through ML models, and delivery in under 200 milliseconds.

The fanout problem is the central architectural challenge in social feed systems. When a user posts a photo, how do we ensure it appears in all their followers' feeds?

There are two fundamental approaches, each with profound trade-offs:

Fanout-on-Write (Push Model):

When a user posts, immediately write to every follower's pre-computed feed cache.

User A posts photo
→ For each follower of A:
    → Append photo to follower's feed cache
→ All followers have updated feeds immediately

Fanout-on-Read (Pull Model):

When a user requests their feed, query all followed accounts and aggregate their recent posts.

User B requests feed
→ For each account B follows:
    → Fetch recent posts from that account
→ Merge and rank all posts
→ Return top N posts as feed

The Celebrity Problem:

Neither approach works universally. Consider these extremes:

If Cristiano Ronaldo (600M+ followers) used fanout-on-write, a single post would generate 600 million write operations. At 10 posts/day, that's 6 billion writes daily from one user—more than Instagram's total daily post volume.

At the heart of feed generation is the social graph—the data structure representing who follows whom. This graph is massive and accessed on nearly every operation.

Graph Scale:

Graph Operations Needed:

Graph Storage Strategies:

Storing 100+ billion edges efficiently requires specialized infrastructure:

Option 1: Adjacency List in Key-Value Store

# Store outgoing edges (following)
Key: user_id:following
Value: [followed_user_id_1, followed_user_id_2, ...]

# Store incoming edges (followers) 
Key: user_id:followers
Value: [follower_id_1, follower_id_2, ...]

• Simple and fast for common queries
• Problematic for users with millions of followers (large values)
• Requires pagination for large lists

Option 2: Edge Table in Relational/Wide-Column DB

CREATE TABLE follows (
    follower_id BIGINT,
    followed_id BIGINT,
    created_at TIMESTAMP,
    PRIMARY KEY (follower_id, followed_id)
);

-- Index for reverse lookup
CREATE INDEX idx_followed ON follows(followed_id, follower_id);

• Clean relational model
• Handles any list size gracefully
• Requires careful sharding for scale

Option 3: Graph Database

// Neo4j style
(:User {id: 'A'})-[:FOLLOWS]->(:User {id: 'B'})

• Native graph operations (traversal, pattern matching)
• Often harder to scale horizontally
• May be used for recommendation graph, not primary storage

Instagram's Approach: TAO (The Associations and Objects)

Meta (Facebook/Instagram) uses TAO, a purpose-built graph storage system optimized for social graph access patterns:

Graph Sharding:

With billions of users, the graph must be sharded across many machines:

# Shard by user_id hash
Shard(user_id) = hash(user_id) % num_shards

# User A's following list → Shard for A
# User B's followers list → Shard for B

# Problem: Following query (A follows B) might span two shards
# Solution: Store edges in both shards (write amplification for read efficiency)

When a user requests their home feed, the system must determine which posts are candidates for inclusion. This happens in stages:

Stage 1: Identify Content Sources

User requests feed
→ Get list of followed accounts
→ Separate into 'push' accounts (regular users) and 'pull' accounts (celebrities)
→ Identify other content sources (ads, suggested posts, etc.)

Stage 2: Fetch Pre-pushed Content

For regular accounts (fanout-on-write), the user has a pre-computed feed cache:

Feed cache structure:
{
  user_id: "viewer_123",
  posts: [
    {post_id: "abc", author_id: "456", timestamp: 1704300000, score_features: {...}},
    {post_id: "def", author_id: "789", timestamp: 1704290000, score_features: {...}},
    // ... up to ~1000 recent posts
  ],
  last_updated: 1704310000
}

Stage 3: Pull Celebrity Content

For high-follower accounts, fetch their recent posts at request time:

async def fetch_celebrity_posts(user_id: str, celebrity_ids: List[str]) -> List[Post]:
    """
    Fetch recent posts from celebrity accounts the user follows.
    Runs at feed-request time for accounts above the push threshold.
    """
    # Parallel fetch from all celebrity accounts
    tasks = [fetch_recent_posts(celeb_id, limit=20) for celeb_id in celebrity_ids]
    results = await asyncio.gather(*tasks)
    
    # Flatten and deduplicate
    all_posts = []
    seen_post_ids = set()
    for posts in results:
        for post in posts:
            if post.id not in seen_post_ids:
                all_posts.append(post)
                seen_post_ids.add(post.id)
    
    return all_posts

Stage 4: Merge Candidate Pools

Candidate pool sources:
├── Pre-pushed feed items (from fanout-on-write)
├── Pulled celebrity posts (from fanout-on-read)
├── Suggested posts (from recommendation system)
├── Ads (from advertising system)
├── Reshared content (friends reshared to Stories)
└── Cross-posted content (e.g., Reels that appear in feed)

Merge all into single candidate pool → 500-2000 candidates

Once candidates are assembled, the ranking system determines the order in which posts appear. Modern Instagram feeds are heavily optimized by machine learning models that predict which posts each user will find most engaging.

Ranking Objectives:

The ranking system optimizes for multiple outcomes simultaneously:

The Ranking Pipeline:

Multi-Stage Ranking:

Ranking uses a cascade of increasingly sophisticated (and expensive) models:

Feature Categories for Ranking:

With billions of feed requests daily, aggressive caching is essential. However, feed caching is complex because feeds are personalized, time-sensitive, and constantly changing.

What Can Be Cached?

Feed Cache Architecture:

Cache Invalidation Strategies:

The classic challenge: "When do we invalidate the cache?"

1. Time-based Expiration (TTL):

# Simple but can serve stale content
feed_cache.set(user_id, feed_data, ttl=300)  # 5-minute TTL

2. Event-driven Invalidation:

# When followed account posts
def on_new_post(author_id: str, post_id: str):
    # Get all followers who have cached feeds
    followers = get_followers(author_id)
    for follower_id in followers:
        # Invalidate or update their feed cache
        feed_cache.invalidate(follower_id)
        # Or: Append new post to cached feed (lazy update)
        feed_cache.append_post(follower_id, post_id)

# When user follows/unfollows
def on_follow_change(user_id: str, target_id: str):
    feed_cache.invalidate(user_id)  # Feed sources changed

3. Hybrid: TTL + Incremental Updates:

# Cache stores feed with version number
cached_feed = {
    'version': 123,
    'posts': [...],
    'last_update': timestamp
}

# On request, check for updates since last version
def get_feed_with_refresh(user_id: str):
    cached = feed_cache.get(user_id)
    if cached:
        new_posts = get_posts_since(user_id, cached['last_update'])
        if new_posts:
            # Merge new posts into cached feed and re-rank
            merged = merge_and_rank(cached['posts'], new_posts)
            cached['posts'] = merged
            cached['version'] += 1
            feed_cache.set(user_id, cached)
        return cached['posts']
    else:
        # Full feed generation
        return generate_full_feed(user_id)

Modern users expect feeds to update in real-time. When a friend posts while you're scrolling, you expect to see "New posts available" appear. This requires push infrastructure that can reach hundreds of millions of concurrent users.

Real-Time Update Channels:

Push Notification Architecture:

WebSocket Connection Management:

Maintaining persistent connections to 100+ million concurrent users is a significant infrastructure challenge:

Long-Polling Fallback:

For clients that can't maintain WebSocket connections (firewalls, proxies, old devices):

# Long-polling endpoint
GET /api/v1/feed/updates?since=<timestamp>&timeout=30

# Server holds connection open for up to 30 seconds
# Returns immediately if updates available
# Returns empty after timeout if no updates

# Client immediately reconnects after each response

Instagram's feed is an infinite scroll experience—users continuously scroll and new content loads seamlessly. Implementing this correctly is more complex than it appears.

The Pagination Challenge:

Traditional offset-based pagination fails for dynamic feeds:

-- Offset pagination: Simple but broken for feeds
SELECT * FROM posts ORDER BY rank DESC LIMIT 20 OFFSET 40;

-- Problem: If new posts are inserted or rankings change,
-- the user may see duplicate posts or miss posts entirely

Cursor-based Pagination:

Instagram uses cursor-based (keyset) pagination for stable scrolling:

# Cursor encodes the position in the feed
# Using last-seen post ID and its rank score

@dataclass
class FeedCursor:
    user_id: str
    last_post_id: str
    last_rank_score: float
    page_number: int
    session_id: str  # For analytics

def get_feed_page(cursor: FeedCursor) -> Tuple[List[Post], FeedCursor]:
    """
    Get next page of feed starting after the cursor position.
    """
    # Fetch posts ranked after the cursor
    posts = db.query(
        """SELECT * FROM ranked_feed 
           WHERE user_id = :user_id 
           AND (rank_score, post_id) < (:last_score, :last_post_id)
           ORDER BY rank_score DESC, post_id DESC
           LIMIT 20""",
        user_id=cursor.user_id,
        last_score=cursor.last_rank_score,
        last_post_id=cursor.last_post_id
    )
    
    # Create cursor for next page
    if posts:
        next_cursor = FeedCursor(
            user_id=cursor.user_id,
            last_post_id=posts[-1].id,
            last_rank_score=posts[-1].rank_score,
            page_number=cursor.page_number + 1,
            session_id=cursor.session_id
        )
    else:
        next_cursor = None
    
    return posts, next_cursor

Feed Session Stability:

A user's feed should remain stable during a scrolling session, even if rankings change:

Implementation: Session-Scoped Feed Snapshot:

class FeedSession:
    """
    Represents a single scrolling session with stable feed content.
    """
    
    def __init__(self, user_id: str):
        self.session_id = generate_uuid()
        self.user_id = user_id
        self.created_at = time.now()
        self.seen_post_ids: Set[str] = set()
        self.ranked_posts: List[RankedPost] = []
        
    def initialize(self):
        """Generate and cache the ranked feed for this session."""
        candidates = fetch_candidates(self.user_id)
        self.ranked_posts = rank_candidates(self.user_id, candidates)
        # Store in Redis with TTL
        cache.set(
            f"feed_session:{self.session_id}",
            self.serialize(),
            ttl=3600  # 1 hour max session
        )
    
    def get_page(self, page_num: int, page_size: int = 20) -> List[Post]:
        """Get a page from the frozen session feed."""
        start = page_num * page_size
        end = start + page_size
        
        posts = self.ranked_posts[start:end]
        
        # Track seen posts
        for post in posts:
            self.seen_post_ids.add(post.id)
        
        return posts

Pure engagement-based ranking can create degenerate feeds—too many posts from one author, too much of one content type, or filter bubbles that limit exposure. Instagram applies diversity rules after ML ranking to ensure feed quality.

Post-Ranking Rules:

Diversity Re-Ranking Algorithm:

def apply_diversity_rules(ranked_posts: List[RankedPost]) -> List[RankedPost]:
    """
    Apply diversity rules to re-order posts after ML ranking.
    Maintains a sliding window to enforce spacing constraints.
    """
    final_feed = []
    remaining = list(ranked_posts)
    
    # Track recent authors/topics for spacing
    recent_authors = []  # Last 5 authors
    recent_topics = []   # Last 3 topics
    
    while remaining and len(final_feed) < MAX_FEED_LENGTH:
        # Find highest-ranked post that satisfies constraints
        for i, post in enumerate(remaining):
            # Check author spacing
            if post.author_id in recent_authors:
                continue
            
            # Check topic diversity
            if post.primary_topic in recent_topics[-3:]:
                # Allow if high enough rank (don't sacrifice too much quality)
                if i < 3:  # Top 3 ranked remaining
                    pass  # Allow despite topic repeat
                else:
                    continue
            
            # Post passes constraints - add to feed
            final_feed.append(post)
            remaining.pop(i)
            
            # Update tracking
            recent_authors.append(post.author_id)
            if len(recent_authors) > 5:
                recent_authors.pop(0)
            
            recent_topics.append(post.primary_topic)
            if len(recent_topics) > 6:
                recent_topics.pop(0)
            
            break
        else:
            # No post satisfied constraints - take top remaining
            final_feed.append(remaining.pop(0))
    
    return final_feed

Ad Slot Injection:

Ads are inserted after ranking, not ranked alongside organic content:

def inject_ads(feed: List[Post], user_context: UserContext) -> List[FeedItem]:
    """Insert ads at appropriate intervals."""
    feed_with_ads = []
    ad_interval = calculate_ad_interval(user_context)  # ~6-8 for most users
    
    for i, post in enumerate(feed):
        feed_with_ads.append(post)
        
        if (i + 1) % ad_interval == 0:
            ad = fetch_ad_for_position(user_context, position=i)
            if ad:
                feed_with_ads.append(ad)
    
    return feed_with_ads

The feed generation system is Instagram's core value creator—it turns raw uploads into personalized, engaging experiences. Let's consolidate the key learnings:

What's Next: Stories Architecture

Stories present unique architectural challenges distinct from the main feed: 24-hour ephemeral content, sequential viewing within stories, and the prominent 'story ring' UI. We'll explore:

• Story creation and expiration lifecycle
• The story tray (ring of stories at top)
• Sequential story viewing and progress tracking
• Story interactions (polls, questions, reactions)
• Close Friends and audience controls
• Story analytics and view tracking