Every distributed data system faces the same fundamental challenge: how do you spread data across multiple machines while maintaining coherent query behavior and surviving failures? Elasticsearch's answer is sharding with replication—a design that enables both horizontal scaling and fault tolerance.

Sharding isn't unique to Elasticsearch. PostgreSQL has table partitioning. MongoDB has sharded clusters. Kafka has partitions. Redis has cluster slots. But each system implements partitioning differently, with distinct trade-offs.

What makes Elasticsearch's sharding model distinctive is its tight coupling with Lucene indexes. Each shard is a complete, self-contained Lucene index. This design choice has profound implications for sizing, allocation, and operational behavior that we'll explore in depth.

When you create an Elasticsearch index, you specify the number of primary shards. This number determines how your data will be partitioned and has permanent implications—you cannot change the primary shard count without reindexing.

What is a primary shard?

A primary shard is the authoritative copy of a portion of an index's data. When a document is indexed, Elasticsearch uses a routing formula to determine which primary shard should hold it:

shard_id = hash(_routing) % number_of_primary_shards

By default, _routing equals the document's _id. This ensures deterministic placement—given a document ID, any node can calculate which shard contains that document without consulting a lookup table.

This hash-based routing is elegant but inflexible. Once shard count is set, changing it would alter the hash results, making existing documents unfindable. Hence the immutability.

Each shard is a Lucene index:

Here's the crucial point: every primary shard is a complete Lucene index with its own inverted indexes, stored fields, and segment files. This means:

• Each shard can be searched independently
• Each shard maintains its own term statistics (affecting TF-IDF/BM25 scores)
• Each shard has fixed overhead regardless of document count

The 'shard as Lucene index' design enables distribution but creates a floor on resource usage. Even an empty shard consumes memory for buffers, file handles for open segments, and CPU for background operations. This overhead accumulates with shard count.

While primary shards handle write operations, replica shards provide two critical capabilities: data redundancy for fault tolerance and read distribution for query scaling.

What is a replica shard?

A replica shard is a copy of a primary shard. When documents are indexed to a primary, they're replicated to all associated replicas. If the node holding a primary shard fails, one of its replicas is automatically promoted to primary.

Unlike primary shard count, the replica count can be changed dynamically at any time:

PUT /products/_settings
{
  "number_of_replicas": 2
}

Increasing replicas triggers data copying to new shards. Decreasing removes shards immediately (data still exists on primaries). This flexibility lets you tune redundancy based on changing requirements.

How replication works:

When you index a document, the following sequence occurs:

1. Coordinating node routes request to the appropriate primary shard
2. Primary shard performs the indexing operation
3. Primary shard writes to its translog (for durability)
4. Primary shard forwards the operation to all replica shards in parallel
5. Replicas perform the same operation and acknowledge
6. Primary acknowledges to coordinator once enough replicas respond

The 'enough replicas' threshold is controlled by wait_for_active_shards. By default, it's 1 (just the primary). Setting it higher (all, or a specific number) provides stronger durability guarantees at the cost of latency.

Read distribution:

During search operations, queries can be served by either primary or replica shards. The coordinating node selects which copy to query using a round-robin or adaptive algorithm. This means:

• More replicas = more copies available for parallel search
• Read throughput scales roughly linearly with replica count
• Geographic distribution of replicas can reduce latency for distant users

However, replicas incur costs:

• Each replica consumes storage equal to its primary
• Each replica requires memory for caching and operations
• Network bandwidth for replication traffic
• Indexing latency increases with more replicas

Shard sizing is one of the most consequential decisions in Elasticsearch deployments. Get it wrong, and you'll face performance problems, scaling constraints, or operational headaches. Unfortunately, there's no universal answer—optimal shard size depends on hardware, workload patterns, and query characteristics.

The competing pressures:

The sizing guidelines:

Elastic's official guidance has evolved over time, but the current recommendations are:

Target shard size: 10GB to 50GB

This range balances Lucene efficiency with operational flexibility. Below 10GB, the overhead-to-data ratio is poor. Above 50GB, recovery and rebalancing take too long.

Shards per node: 20 per GB of heap (rough guide)

For a node with 30GB heap, aim for under 600 shards. This is a ceiling, not a target—fewer is usually better.

Calculate before creating:

If you expect 200GB of data, don't create an index with 100 shards (2GB each). Instead, use 4-6 shards (33-50GB each) and add replicas for redundancy and read scaling.

Time-based indexing for scaling:

For logging, metrics, and other time-series data, a common pattern is time-based indexing:

• Create a new index daily/weekly/monthly (e.g., logs-2024.01.15)
• Each index is sized based on daily data volume
• Old indexes can be optimized (force merged), moved to cold storage, or deleted

This pattern avoids the static sizing problem: you're not guessing future data volume, just sizing for predictable time periods. Index lifecycle management (ILM) automates this workflow.

Shard allocation is the process by which the master node decides where each shard should live. This seemingly simple task involves complex decisions balancing disk space, node capacity, failure domains, and administrative policies.

Default allocation behavior:

Without manual intervention, Elasticsearch allocates shards to:

• Maximize balance across nodes (similar shard counts per node)
• Ensure primaries and their replicas are on different nodes
• Consider disk usage (won't allocate to nearly-full disks)
• Spread shards across failure domains when configured (rack/zone awareness)

The allocation process runs continuously—whenever a new index is created, a node joins/leaves, or a recovery is needed.

Disk-based allocation:

Elasticsearch applies watermarks to prevent disk exhaustion:

Low watermark (default 85%): Elasticsearch avoids allocating new shards to nodes above this threshold.

High watermark (default 90%): Elasticsearch attempts to relocate shards away from nodes above this threshold.

Flood stage (default 95%): Elasticsearch blocks writes to indexes with shards on nodes above this threshold. This is a safety mechanism—better to reject writes than corrupt data.

These thresholds are configurable but should be changed carefully. At 95% disk usage, a sudden spike in segment creation (during heavy indexing) could fill the remaining space instantly.

Default shard allocation ensures primaries and replicas aren't on the same node. But what if that node's rack loses power? Or that cloud availability zone goes down? Both copies are lost simultaneously.

Allocation awareness extends failure domain thinking beyond individual nodes. You define attributes that categorize nodes (zone, rack, data center), and Elasticsearch ensures replicas are spread across these failure domains.

Configuring zone awareness:

How awareness affects allocation:

With zone awareness enabled, Elasticsearch's allocation logic changes:

• When allocating a replica, it prefers nodes in different zones from the primary
• If no nodes are available in other zones, the replica remains unassigned (yellow cluster)
• Rebalancing considers zone distribution, not just node-level balance

This means a 2-zone cluster with 1 replica achieves true redundancy: losing an entire zone doesn't lose any data.

Forced awareness (strict zone distribution):

By default, if a zone is unavailable, Elasticsearch can still allocate replicas to remaining zones. Forced awareness prevents this—it requires all configured zones to be present before allocating replicas.

This is useful when you want to guarantee data doesn't concentrate in a single zone during temporary outages.

Beyond automated allocation, Elasticsearch provides allocation filtering—rules that explicitly include, exclude, or require nodes for certain indexes. This enables sophisticated multi-tier architectures and operational control.

Use cases for allocation filtering:

• Hot-warm-cold architecture: Route new indexes to fast (hot) nodes, older indexes to slower (warm/cold) nodes
• Hardware specialization: Keep aggregation-heavy indexes on high-memory nodes
• Maintenance operations: Drain a node before hardware maintenance
• Data residency: Keep certain data in specific geographic locations

Filter types:

require — All specified conditions must be met. The shard can only allocate to nodes matching ALL criteria.

include — At least one specified condition must be met. The shard can allocate to nodes matching ANY of the criteria.

exclude — Specified conditions must NOT be met. The shard avoids nodes matching ANY of the criteria.

Node attributes you can filter on:

• _name — Node name
• _ip — Node IP address
• _host — Node hostname
• _id — Node ID
• Custom attributes (any node.attr.* value)

The most powerful patterns combine custom attributes with lifecycle policies. As data ages, ILM automatically moves indexes through tiers by updating allocation filters.

A healthy Elasticsearch cluster is in constant motion. Shards relocate to balance load, replicas recover after failures, and new shards are allocated as indexes grow. Understanding these processes helps diagnose performance issues and plan capacity.

Rebalancing:

Rebalancing moves shards between nodes to maintain even distribution. This happens when:

• New nodes join the cluster (shards spread to new capacity)
• Nodes leave the cluster (shards redistribute to remaining nodes)
• Index operations create uneven distribution

Rebalancing is automatic but configurable. You can control:

• Threshold: How unbalanced before rebalancing triggers
• Concurrency: How many shards move simultaneously
• Timing: Temporary disabling during maintenance

Recovery:

Recovery is the process of restoring shard data after failures or when initializing replicas. There are several recovery types:

Peer recovery — Copying data from a primary to a replica, or from one node to another during relocation. This involves streaming segment files over the network.

Translog replay — After a node restart, replaying the transaction log to recover uncommitted operations.

Snapshot restore — Restoring from a snapshot repository for disaster recovery or data migration.

Recovery can be I/O and network intensive. Too aggressive recovery can impact production traffic; too conservative extends time at risk.

Sharding and replication are the heart of Elasticsearch's distributed nature. Every performance, availability, and scaling decision traces back to how you've configured shards. Let's consolidate the key principles:

What's next:

With distribution mechanics understood, we'll explore mapping and analysis—how Elasticsearch processes, stores, and makes documents searchable. Proper mapping design is as critical as shard sizing for query performance and storage efficiency.