Indexing vs Querying: The Two Paths of Search

Every search system lives in a constant tension between two competing concerns: how fast can we add new content versus how fast can we find existing content. These are not just different operations—they are fundamentally different paths through the system, with different architectures, different bottlenecks, and often different teams responsible for them.

Understanding this dichotomy is not academic trivia. It directly impacts:

• System design: Where do you invest engineering effort?
• Capacity planning: How do you size machines and clusters?
• Latency optimization: Where do you look when performance degrades?
• Feature prioritization: What trade-offs do you accept?

This page will make you fluent in thinking about search systems as two interrelated but distinct paths, each with its own optimization strategies.

The separation of indexing and querying is not an arbitrary design choice—it emerges from fundamental physics and mathematics. Let's understand why.

The Brute Force Problem:

The naive approach to search—scan all documents for each query—has linear time complexity: O(n) per query. For small collections, this works fine:

Clearly, brute force doesn't scale. But if we can't scan documents at query time, when can we scan them?

The Write-Read Asymmetry:

This trade-off is justified by a crucial asymmetry in real systems:

1. Writes are rare, reads are common: Most search systems see 100-10,000x more read traffic than write traffic.

2. Writes can be batched, reads cannot: Users expect instant search results, but indexing can happen in the background.

3. Writes are produced by systems, reads by humans: Humans are impatient; systems can wait.

Because of this asymmetry, it makes economic sense to invest heavily in indexing complexity to achieve query simplicity.

Exception: Write-Heavy Systems

Log search and observability platforms are interesting exceptions—they are write-heavy. Systems like Elasticsearch for logs, Splunk, and Datadog ingest terabytes of logs per day while serving relatively few queries. These systems still use indexes, but their optimization focus shifts toward write throughput and storage efficiency rather than query latency.

The indexing path is responsible for transforming raw documents into optimized data structures. Let's trace a document's journey through this path and understand each step's purpose and challenges.

The Indexing Pipeline:

Document → Ingestion → Analysis → Index Building → Commit → Refresh → Searchable

Each stage has distinct characteristics:

Near-Real-Time (NRT) Indexing:

Modern search systems offer near-real-time indexing, where documents become searchable within seconds of ingestion. This is achieved through:

1. Memory buffers: Documents accumulate in memory before disk writes
2. Frequent refreshes: New segments are created every 1-5 seconds
3. Segment replication: New segments replicate to replicas quickly

However, NRT involves trade-offs:

Indexing Throughput Optimization:

When indexing throughput is critical (bulk imports, migration, log ingestion), these strategies help:

1. Disable refresh during bulk indexing: Set refresh_interval: -1, refresh manually when done
2. Increase indexing buffer size: More memory for in-memory segments
3. Reduce replica count during bulk: Index to primary only, replicate after
4. Use bulk APIs: Batch documents into single requests (100-1000 docs per bulk)
5. Parallelize ingestion: Multiple indexing threads/processes
6. Optimize analysis: Simpler analyzers for non-search fields
7. Consider index sorting: Sorted indexes compress better, but slower to build

Typical indexing throughput:

• Single node: 10,000-50,000 docs/second (depends on doc size, analysis)
• Cluster: Linear scaling with nodes (until coordination overhead dominates)

The querying path is where all the indexing investment pays off. Users expect sub-second responses regardless of corpus size. Let's trace a query through the system and understand where time is spent.

The Query Pipeline:

Query → Parse → Analyze → Plan → Route → Execute → Score → Merge → Return

Each stage has a latency budget—typically summing to under 200ms for competitive search:

Query Latency Optimization:

Query latency optimization is a deep art. Key strategies include:

Index-Level Optimizations:

• Fewer, larger segments (reduce segment overhead)
• Hot indexes in memory (filed system caches or explicit warming)
• Appropriate shard sizing (too many = coordination overhead, too few = per-query work)
• Index sorting (improves compression and skip efficiency)

Query-Level Optimizations:

• Filter before score (eliminate documents cheaply)
• Terminate early (stop after top-K found with sufficient confidence)
• Avoid deep pagination (offset 10000 requires computing 10000 results)
• Use routing (send query only to relevant shards when possible)

The separation of indexing and querying creates an inherent consistency challenge: there is a window between when a document is indexed and when it becomes searchable. Understanding and managing this window is crucial for system correctness.

The Visibility Window:

Time:       T0          T1          T2          T3          T4
            │           │           │           │           │
Document:   Created → Indexed → Committed → Refreshed → Searchable
                        │           │           │
                     Memory      Disk       Query Path

Between T0 and T4, the document exists but isn't visible to search. This window can range from milliseconds to minutes depending on configuration.

Managing Consistency Expectations:

1. Read-Your-Writes Consistency

Users expect to immediately see content they just created. Solutions:

• Force refresh after write (expensive, use sparingly)
• Primary routing (route user's queries to shard where they write)
• Optimistic UI (show document before indexing confirms, hide if fails)

2. Delete Consistency

Deleted documents appearing in search is jarring. Solutions:

• Soft delete (filter by deletion flag, garbage collect later)
• Wait for refresh before confirming delete
• Combine with above read-your-writes strategy

3. Update Consistency

Partially updated documents are confusing. Solutions:

• Index entire document atomically (no field-level updates)
• Use versioning (optimistic concurrency control)
• Accept eventual consistency for most fields

Distributed Consistency Complications:

In a distributed search cluster, consistency becomes more complex:

Search systems typically choose availability over consistency (AP in CAP theorem). Missing a document for a few seconds is acceptable; being completely unavailable is not.

Although indexing and querying are logically separate, they share physical resources. Understanding this contention is essential for capacity planning and performance tuning.

Mitigation Strategies:

1. Physical Separation

Dedicate nodes to specific roles:

• Data nodes: Hold shards, serve queries and indexing
• Ingest nodes: Pre-process documents (parsing, enrichment)
• Coordinator nodes: Route queries, merge results

For critical workloads, separate query-serving data nodes from indexing-receiving data nodes.

2. Temporal Separation

Schedule heavy operations during off-peak hours:

• Run merges at night
• Bulk imports during low-traffic periods
• Backup operations scheduled around query patterns

3. Resource Quotas

Limit resource consumption:

• Thread pool sizing (separate indexing and search pools)
• Memory allocation (indexing buffer vs. query cache)
• I/O throttling (merge bandwidth limits)

4. Priority Queuing

Prioritize query path:

• Query requests preempt indexing
• Search thread pool priority > bulk indexing priority
• Merges pause during query spikes

Different systems organize indexing and querying differently based on their requirements. Let's examine common patterns and when to use each.

Pattern 1: Unified Cluster (Simple)

                  ┌─────────────────────┐
   Documents  ──► │   Search Cluster    │ ◄── Queries
                  │ (Index + Query)     │
                  └─────────────────────┘

All nodes handle both indexing and querying. Simple, but indexing can impact query latency.

When to use: Small to medium deployments, balanced workloads, limited operational complexity.

Pattern 2: Role-Based Separation (Medium)

   Documents  ──► [Ingest Nodes] ──► [Data Nodes] ◄── [Coordinator] ◄── Queries
                                           │
                                      (Shards)

Specialized node roles reduce contention. Ingest nodes handle document processing; coordinators handle query routing.

When to use: Larger clusters, predictable workloads, some operational maturity.

Pattern 3: Full Separation (Complex)

   Documents  ──► [Indexing Cluster] ──► [Object Storage] ──► [Query Cluster]
                  (Build segments)        (Segment files)      (Serve queries)

Completely separate indexing and serving clusters. Segments are built offline and pushed to query servers.

When to use: Very high scale, strict query latency SLAs, different scaling requirements for read vs. write.

Pattern 4: Lambda Architecture for Search

Combine batch and streaming indexing:

                        ┌─────────────────┐
   Documents ──────────►│  Stream Layer   │──► Real-time Index
        │               │  (Kafka, etc.)  │         │
        │               └─────────────────┘         │
        │                                           ▼
        │               ┌─────────────────┐    ┌─────────┐
        └──────────────►│  Batch Layer   │───►│  Merge  │──► Query Cluster
                        │  (Spark, etc.) │    │         │
                        └─────────────────┘    └─────────┘

• Stream layer: Indexes recent data with minimal latency
• Batch layer: Reprocesses all data periodically for accuracy
• Serving layer: Merges real-time and batch indexes

When to use: When freshness matters but complete processing is too slow, when data requires complex enrichment.

Trade-off: Complexity of maintaining two data paths, potential for inconsistency between layers.

Let's examine how major search systems handle the indexing-querying balance: