Distributed File Systems
Hadoop Distributed File System (HDFS)
Read aloud is unavailable in this browser.
The Challenge of Petabyte-Scale Storage
In 2003, a fundamental question emerged in computer science: How do you store and process datasets so large that no single machine could possibly contain them? Traditional file systems, designed for single-machine architectures with disks measured in gigabytes, simply could not scale to handle the petabytes of data being generated by web-scale companies.
The Hadoop Distributed File System (HDFS) emerged as the definitive answer to this challenge. Inspired by Google's groundbreaking GFS paper, HDFS fundamentally reimagined what a file system could be—not a component of a single machine, but a unified storage layer spanning thousands of commodity servers, capable of storing files that dwarf the capacity of any individual disk.
Today, HDFS serves as the storage foundation for some of the world's largest data processing systems, handling exabytes of data across organizations ranging from Facebook and Yahoo to financial institutions and scientific research facilities. Understanding HDFS is essential for any engineer working with large-scale data systems.
By the end of this page, you will understand HDFS's master-slave architecture, its block-based storage model, the roles of NameNode and DataNodes, how data is read and written across the cluster, and the design decisions that enable reliability at massive scale. You'll gain the foundational knowledge needed to work with or design distributed storage systems.
Genesis and Design Philosophy
HDFS was born from the Apache Nutch project's need to crawl and index the entire World Wide Web—a task requiring storage and processing capabilities far beyond any single machine. The project's creators, Doug Cutting and Mike Cafarella, drew inspiration from Google's 2003 GFS paper, which described an internally-developed distributed file system designed around a radical set of assumptions.
The foundational assumptions that shaped HDFS:
- •Hardware Failure is the Norm — When you operate thousands of commodity servers, component failures happen constantly. A system with 10,000 disks, each with a 2% annual failure rate, experiences a disk failure nearly every day. HDFS treats failure as the expected case, not the exception.
- •Files are Massive — HDFS is optimized for files measured in gigabytes or terabytes, not kilobytes. A typical HDFS deployment might store billions of web pages, years of log files, or petabytes of scientific data. Small files are an anti-pattern.
- •Access Patterns are Streaming — Applications read data sequentially, often processing entire files. Random access within files is rare. This assumption enables aggressive prefetching and simplifies caching strategies.
- •Write-Once, Read-Many — Files are written once and then read repeatedly. Appends are supported, but random writes within files are not. This simplifies consistency guarantees and enables aggressive replication.
- •Data Locality Matters — Moving computation to data is vastly cheaper than moving data to computation. HDFS exposes data location information to enable computation frameworks to schedule tasks near the data they process.
- •Commodity Hardware Over Specialized Hardware — Rather than relying on expensive, highly-reliable hardware, HDFS uses redundancy across cheap, failure-prone machines to achieve reliability. This dramatically reduces cost per byte stored.
Traditional storage systems like RAID arrays and SANs achieve reliability through expensive, specialized hardware with redundant power supplies, ECC memory, and hot-swappable components. HDFS inverts this approach: use cheap, unreliable components and handle failures in software. This philosophical shift enabled storage at previously impossible scales and costs.
What HDFS explicitly does not optimize for:
- Low-latency data access (use HBase or other key-value stores instead)
- Lots of small files (metadata overhead becomes prohibitive)
- Multiple writers or arbitrary file modifications (consistency complexity explodes)
- POSIX-compliant semantics (relaxed consistency enables better scalability)
These non-goals are as important as the goals—they enable HDFS to excel at its intended purpose rather than attempting to be a general-purpose file system.
Master-Slave Architecture
HDFS implements a master-slave architecture with two distinct node types: a single NameNode (the master) that manages file system metadata, and many DataNodes (slaves) that store actual data blocks. This separation of concerns enables independent scaling of metadata management and data storage.
The architecture at a glance:
Critical insight: Clients never transfer data through the NameNode. For read operations, the NameNode provides block locations, and clients connect directly to DataNodes. For write operations, the NameNode allocates blocks, and clients stream data directly to DataNodes. This design prevents the NameNode from becoming a data transfer bottleneck, enabling aggregate throughput that scales with the number of DataNodes.
- •Manages file system namespace (directories, files, permissions)
- •Maintains file-to-block mapping
- •Tracks which DataNodes hold which blocks
- •Handles block replication decisions
- •Processes client metadata operations
- •Coordinates block creation and deletion
- •Manages lease recovery for failed writers
- •Enforces access control and quotas
- •Stores data blocks on local disks
- •Serves read requests from clients
- •Receives write data and creates blocks
- •Sends periodic heartbeats to NameNode
- •Reports list of hosted blocks
- •Executes block replication commands
- •Performs block deletion as instructed
- •Validates block checksums on reads
The NameNode: Metadata Management
The NameNode is the brain of HDFS—a single point of coordination that maintains the entire file system namespace in memory for fast access. Understanding the NameNode's data structures and persistence mechanisms is crucial for both operating and designing distributed file systems.
What the NameNode stores in memory:
- •Namespace tree — The complete directory hierarchy with file and directory metadata (names, permissions, timestamps, replication factors)
- •File-to-block mapping — For each file, an ordered list of block IDs that compose the file
- •Block-to-DataNode mapping — For each block, the list of DataNodes that hold replicas (this is NOT persisted—rebuilt from DataNode reports)
- •DataNode registration info — Network addresses, capacity, and status of each DataNode
- •Pending replication queue — Blocks that need additional replicas created
- •Invalidation queue — Blocks that should be deleted from specific DataNodes
Every file and block consumes approximately 150 bytes of NameNode memory. A cluster with 100 million files and 200 million blocks requires roughly 45 GB of heap memory. This memory constraint is why HDFS discourages small files—a 10 MB file consumes the same metadata as a 10 GB file, but provides 1000x less storage efficiency.
Namespace persistence:
The NameNode persists its namespace to durable storage using two mechanisms:
1. FSImage (Checkpoint) A complete serialized snapshot of the file system namespace at a point in time. Contains all directories, files, and their block lists. Loaded into memory at NameNode startup. Creating a new FSImage is expensive and done periodically.
2. Edit Log (Journal) A sequential log of all namespace-modifying operations since the last checkpoint. Every mkdir, create, delete, and rename is recorded. Replayed on top of FSImage during startup to reach current state.
The startup process:
123456789101112131415161718192021222324252627
function startNameNode(): // Phase 1: Load namespace from disk namespace = loadFSImage(latestCheckpoint) // Phase 2: Replay edit log to reach current state for each transaction in editLog: applyTransaction(namespace, transaction) // Phase 3: Enter "safe mode" - read-only operations only enterSafeMode() // Phase 4: Wait for DataNodes to report their blocks while not sufficientBlockReports(): waitForHeartbeats() processBlockReports() // Phase 5: Identify under-replicated blocks for each block in namespace: replicas = getReportedReplicas(block) if replicas < block.replicationFactor: scheduleReplication(block) // Phase 6: Exit safe mode - full operations enabled leaveSafeMode() // Phase 7: Begin periodic checkpointing startCheckpointThread()Safe mode: During startup, the NameNode enters a special read-only state called safe mode. It knows what files exist and what blocks compose them (from FSImage + edit log), but it doesn't yet know which DataNodes hold those blocks. It waits for DataNodes to send block reports, building the block-to-location mapping dynamically. Only after receiving reports for a sufficient percentage of blocks (typically 99.9%) does it exit safe mode and allow writes.
The NameNode deliberately does not persist block-to-DataNode mappings. DataNodes may have failed, had disks replaced, or been decommissioned while the NameNode was offline. By rebuilding this mapping from live DataNode reports at startup, the NameNode always has an accurate view of actual data location.
Block-Based Storage Model
HDFS stores files as sequences of blocks—fixed-size chunks that are distributed across DataNodes independently. This block-based approach is fundamental to HDFS's ability to store files larger than any single disk and to parallelize both storage and processing.
Block characteristics:
| Property | Default Value | Rationale |
|---|---|---|
| Block size | 128 MB (was 64 MB) | Amortizes seek time; reduces NameNode memory; enables efficient MapReduce splits |
| Replication factor | 3 | Survives two concurrent failures; enables rack-aware placement |
| Minimum replication | 1 | Emergency mode for low-importance data |
| Maximum replication | 512 | For extremely critical data; rarely used |
| Block ID | 64-bit unique ID | Globally unique across the cluster |
Why 128 MB blocks?
The block size represents a careful tradeoff:
-
Too small (e.g., 4 KB like local file systems): Millions of blocks for a terabyte file. NameNode memory explodes. Clients must contact NameNode millions of times per file. Seeks dominate throughput.
-
Too large (e.g., 1 GB): Parallelism is reduced. MapReduce jobs have fewer tasks than cluster capacity. Tail latency increases due to stragglers. Block replication after failure takes longer.
-
128 MB sweet spot: A terabyte file has ~8,000 blocks—manageable metadata. Typical disk sequential throughput (~100-200 MB/s) means block read completes in 1-2 seconds. MapReduce gets sufficient parallelism while tasks are meaningful units of work.
Do not confuse HDFS blocks with disk blocks (typically 4 KB) or local file system blocks. An HDFS block is a much higher-level abstraction—an independently stored, replicated unit that happens to be stored as a regular file on each DataNode's local file system.
How DataNodes store blocks:
On each DataNode, blocks are stored as regular files in the local file system (typically ext4 or XFS). A DataNode with 12 disks might have 12 storage directories, one per disk. Within each directory:
123456789101112131415161718
/data/disk1/hdfs/└── current/ ├── VERSION # DataNode storage version info ├── BP-random-nn-ts/ # Block Pool (one per namespace) │ └── current/ │ └── finalized/ │ ├── subdir0/ │ │ ├── blk_1073741825 # Block data file │ │ ├── blk_1073741825.meta # Checksum metadata │ │ ├── blk_1073741826 │ │ ├── blk_1073741826.meta │ │ └── ... │ ├── subdir1/ │ │ └── ... │ └── subdir63/ │ └── ... │ └── rbw/ # Replica Being Written (incomplete blocks) └── in_use.lock # Prevents concurrent accessThe .meta file contains:
- Version information
- Checksum type (CRC32C by default)
- Bytes per checksum (typically 512 bytes)
- Actual checksums for each chunk of the block
Every read operation verifies checksums, detecting silent data corruption.
Replication and Rack Awareness
HDFS achieves reliability through replication—storing multiple copies of each block across different machines. The default replication factor of 3 means every block exists in three locations, enabling the cluster to survive up to two simultaneous failures of nodes holding replicas of any given block.
But not all replica placements are equal. HDFS implements rack-aware replica placement to balance reliability against network efficiency:
- •First replica: Placed on the same node as the writer (for local writes) or a random node in the writer's rack
- •Second replica: Placed on a different node in a DIFFERENT rack (protects against entire rack failure)
- •Third replica: Placed on a different node in the SAME rack as the second replica (reduces cross-rack traffic)
Why this specific placement?
Rack failure tolerance: If Rack 1 loses power or its switch fails, replicas 2 and 3 on Rack 2 ensure data remains available. This is crucial—rack failures (power, switch, cooling) affect all nodes simultaneously.
Network efficiency: Two replicas share a rack, so reads from compute nodes in that rack avoid cross-rack traffic. Writes also benefit: the pipeline from replica 2 to replica 3 stays within Rack 2.
Failure scenarios survived:
- Any single node failure
- Any two node failures
- Any single rack failure
- One node failure plus different rack failure
HDFS cannot discover rack topology automatically. Administrators must configure a rack mapping script that maps IP addresses or hostnames to rack identifiers. Without this configuration, HDFS assumes all nodes are on a single rack, losing rack-failure protection.
Re-replication triggers:
The NameNode continuously monitors replica counts and initiates re-replication when:
- A DataNode fails to send heartbeats (node failure detected)
- A DataNode reports a corrupted block
- The replication factor for a file is increased
- A DataNode is decommissioned
Re-replication is prioritized: blocks with only one remaining replica are replicated before those with two.
Read and Write Data Paths
Understanding how data flows during read and write operations reveals HDFS's architectural elegance. The separation of metadata (NameNode) from data (DataNodes) enables high aggregate throughput.
The Write Path:
123456789101112131415161718192021222324252627282930313233343536
Client NameNode DataNodes | | | |---- create(path) ------->| | | |-- check permissions | | |-- check file exists | | |-- create file entry | |<--- file handle ---------| | | | | |---- addBlock() --------->| | | |-- select DataNodes | | |-- create block entry | |<--- [DN1, DN2, DN3] -----| | | | | |============= Data Pipeline Setup ===================| | | | |---- connect ----------->[DN1] | | |---- connect ----------->[DN2] | | |---- connect ----------->[DN3] | | | |============= Streaming Data ========================| | | | |---- packet 1 ---------->[DN1] | | |---- packet 1 ---------->[DN2] | | |---- packet 1 ---------->[DN3] | | |<--- ack 1 ---------------| | |<--- ack 1 ---------------| |<--- ack 1 ---------------| | | | | |---- packet N ---------->[DN1] ... (continues) | | | | |============= Block Complete ========================| | | | |---- close() ------------>| | | |<--- blockReceived -------| | | (from all DNs) | |<--- success -------------| |Key write path characteristics:
-
Pipeline writes: Data flows from client to DN1 to DN2 to DN3, not from client to each DN in parallel. This reduces client bandwidth requirements by 3x.
-
Packet-based streaming: Data is broken into 64 KB packets. Each packet contains checksums. Acks flow backward through the pipeline.
-
Lease mechanism: The NameNode grants the client a lease on the file. If the client fails, the lease expires, and recovery procedures reclaim the file.
-
Minimum replica acknowledgment: Writes succeed when a minimum number of replicas (default: 1) acknowledge. The remaining replicas complete asynchronously.
The Read Path:
123456789101112131415161718192021222324
Client NameNode DataNodes | | | |---- open(path) --------->| | | |-- check permissions | | |-- look up file | |<--- block locations -----| | | [B1: DN2, DN3, DN5] | | | [B2: DN1, DN4, DN6] | | | [B3: DN2, DN4, DN5] | | | | | |============= Read Block 1 ==========================| | | | |---- connect to DN2 ----------------------->[DN2] | |---- read B1 request ---------------------->[DN2] | |<--- B1 data stream -----------------------| | | (verify checksums) | | | | | |============= Read Block 2 ==========================| | | | |---- connect to DN1 ------->[DN1] | |---- read B2 request ------>[DN1] | |<--- B2 data stream -------| | | | | |============= (continues for remaining blocks) ======|- •Sorted replica lists: NameNode returns replicas sorted by distance to client (same node > same rack > different rack)
- •Failover on error: If a DataNode fails or returns corrupt data, client automatically tries the next replica
- •Client-side caching: Block locations are cached to avoid repeated NameNode queries for sequential reads
- •Short-circuit reads: If client runs on a DataNode, it can read directly from the local disk, bypassing the network entirely
- •Checksum verification: Every chunk is verified; corruption is detected and reported to NameNode
Fault Tolerance Mechanisms
Operating at scale with commodity hardware requires robust fault tolerance at every level. HDFS implements multiple complementary mechanisms to detect, handle, and recover from failures.
DataNode failure detection:
| Mechanism | Interval | Purpose |
|---|---|---|
| Heartbeat | Every 3 seconds | Prove DataNode is alive; carry NameNode commands |
| Block Report | Every 6 hours | Full inventory of all blocks on DataNode |
| Incremental Block Report | Immediate | Report newly received or deleted blocks |
| Failure detection | 10 minutes (10 missed heartbeats) | Declare DataNode dead; schedule re-replication |
Data integrity protection:
- •Write-time checksums: Client computes CRC32C for every 512-byte chunk before sending
- •Storage checksums: DataNode stores checksums in .meta files alongside block data
- •Read-time verification: Every chunk is verified on read; corruption triggers automatic failover to another replica
- •Background scanning: DataNodes periodically scan all blocks, verifying checksums and reporting corruption
- •Corruption reporting: NameNode marks corrupt replicas, schedules re-replication from healthy copies
Studies show that disks silently corrupt data more often than most engineers expect—approximately 1 in every 10^14 to 10^15 bits. For petabyte-scale clusters, this means corruption events happen regularly. Checksums are not optional at scale; they are essential.
NameNode high availability:
The original HDFS design had a single NameNode—a glaring single point of failure. Modern HDFS deployments use NameNode High Availability (HA) with an Active-Standby configuration:
- •Active NameNode: Handles all client operations; writes to shared edit log
- •Standby NameNode: Continuously reads shared edit log; maintains in-memory namespace state; ready for immediate failover
- •Shared storage (JournalNodes or NFS): Both NameNodes access the same edit log; ensures standby sees all mutations
- •ZooKeeper-based failover controller: Monitors NameNode health; triggers automatic failover on active failure
- •Fencing mechanism: Prevents split-brain by ensuring only one NameNode is active (uses STONITH—Shoot The Other Node In The Head)
KEEP LEARNING
Keep learning for free.
Read the complete lesson, save your notes and pick up where you left off with a free account. No card needed.