Loading learning content...
Read aloud is unavailable in this browser.
In 2003, a fundamental question emerged in computer science: How do you store and process datasets so large that no single machine could possibly contain them? Traditional file systems, designed for single-machine architectures with disks measured in gigabytes, simply could not scale to handle the petabytes of data being generated by web-scale companies.
The Hadoop Distributed File System (HDFS) emerged as the definitive answer to this challenge. Inspired by Google's groundbreaking GFS paper, HDFS fundamentally reimagined what a file system could be—not a component of a single machine, but a unified storage layer spanning thousands of commodity servers, capable of storing files that dwarf the capacity of any individual disk.
Today, HDFS serves as the storage foundation for some of the world's largest data processing systems, handling exabytes of data across organizations ranging from Facebook and Yahoo to financial institutions and scientific research facilities. Understanding HDFS is essential for any engineer working with large-scale data systems.
By the end of this page, you will understand HDFS's master-slave architecture, its block-based storage model, the roles of NameNode and DataNodes, how data is read and written across the cluster, and the design decisions that enable reliability at massive scale. You'll gain the foundational knowledge needed to work with or design distributed storage systems.
HDFS was born from the Apache Nutch project's need to crawl and index the entire World Wide Web—a task requiring storage and processing capabilities far beyond any single machine. The project's creators, Doug Cutting and Mike Cafarella, drew inspiration from Google's 2003 GFS paper, which described an internally-developed distributed file system designed around a radical set of assumptions.
The foundational assumptions that shaped HDFS:
Traditional storage systems like RAID arrays and SANs achieve reliability through expensive, specialized hardware with redundant power supplies, ECC memory, and hot-swappable components. HDFS inverts this approach: use cheap, unreliable components and handle failures in software. This philosophical shift enabled storage at previously impossible scales and costs.
What HDFS explicitly does not optimize for:
These non-goals are as important as the goals—they enable HDFS to excel at its intended purpose rather than attempting to be a general-purpose file system.
HDFS implements a master-slave architecture with two distinct node types: a single NameNode (the master) that manages file system metadata, and many DataNodes (slaves) that store actual data blocks. This separation of concerns enables independent scaling of metadata management and data storage.
The architecture at a glance:
Critical insight: Clients never transfer data through the NameNode. For read operations, the NameNode provides block locations, and clients connect directly to DataNodes. For write operations, the NameNode allocates blocks, and clients stream data directly to DataNodes. This design prevents the NameNode from becoming a data transfer bottleneck, enabling aggregate throughput that scales with the number of DataNodes.
The NameNode is the brain of HDFS—a single point of coordination that maintains the entire file system namespace in memory for fast access. Understanding the NameNode's data structures and persistence mechanisms is crucial for both operating and designing distributed file systems.
What the NameNode stores in memory:
Every file and block consumes approximately 150 bytes of NameNode memory. A cluster with 100 million files and 200 million blocks requires roughly 45 GB of heap memory. This memory constraint is why HDFS discourages small files—a 10 MB file consumes the same metadata as a 10 GB file, but provides 1000x less storage efficiency.
Namespace persistence:
The NameNode persists its namespace to durable storage using two mechanisms:
1. FSImage (Checkpoint) A complete serialized snapshot of the file system namespace at a point in time. Contains all directories, files, and their block lists. Loaded into memory at NameNode startup. Creating a new FSImage is expensive and done periodically.
2. Edit Log (Journal) A sequential log of all namespace-modifying operations since the last checkpoint. Every mkdir, create, delete, and rename is recorded. Replayed on top of FSImage during startup to reach current state.
The startup process:
123456789101112131415161718192021222324252627
function startNameNode(): // Phase 1: Load namespace from disk namespace = loadFSImage(latestCheckpoint) // Phase 2: Replay edit log to reach current state for each transaction in editLog: applyTransaction(namespace, transaction) // Phase 3: Enter "safe mode" - read-only operations only enterSafeMode() // Phase 4: Wait for DataNodes to report their blocks while not sufficientBlockReports(): waitForHeartbeats() processBlockReports() // Phase 5: Identify under-replicated blocks for each block in namespace: replicas = getReportedReplicas(block) if replicas < block.replicationFactor: scheduleReplication(block) // Phase 6: Exit safe mode - full operations enabled leaveSafeMode() // Phase 7: Begin periodic checkpointing startCheckpointThread()Safe mode: During startup, the NameNode enters a special read-only state called safe mode. It knows what files exist and what blocks compose them (from FSImage + edit log), but it doesn't yet know which DataNodes hold those blocks. It waits for DataNodes to send block reports, building the block-to-location mapping dynamically. Only after receiving reports for a sufficient percentage of blocks (typically 99.9%) does it exit safe mode and allow writes.
The NameNode deliberately does not persist block-to-DataNode mappings. DataNodes may have failed, had disks replaced, or been decommissioned while the NameNode was offline. By rebuilding this mapping from live DataNode reports at startup, the NameNode always has an accurate view of actual data location.
HDFS stores files as sequences of blocks—fixed-size chunks that are distributed across DataNodes independently. This block-based approach is fundamental to HDFS's ability to store files larger than any single disk and to parallelize both storage and processing.
Block characteristics:
| Property | Default Value | Rationale |
|---|---|---|
| Block size | 128 MB (was 64 MB) | Amortizes seek time; reduces NameNode memory; enables efficient MapReduce splits |
| Replication factor | 3 | Survives two concurrent failures; enables rack-aware placement |
| Minimum replication | 1 | Emergency mode for low-importance data |
| Maximum replication | 512 | For extremely critical data; rarely used |
| Block ID | 64-bit unique ID | Globally unique across the cluster |
Why 128 MB blocks?
The block size represents a careful tradeoff:
Too small (e.g., 4 KB like local file systems): Millions of blocks for a terabyte file. NameNode memory explodes. Clients must contact NameNode millions of times per file. Seeks dominate throughput.
Too large (e.g., 1 GB): Parallelism is reduced. MapReduce jobs have fewer tasks than cluster capacity. Tail latency increases due to stragglers. Block replication after failure takes longer.
128 MB sweet spot: A terabyte file has ~8,000 blocks—manageable metadata. Typical disk sequential throughput (~100-200 MB/s) means block read completes in 1-2 seconds. MapReduce gets sufficient parallelism while tasks are meaningful units of work.
Do not confuse HDFS blocks with disk blocks (typically 4 KB) or local file system blocks. An HDFS block is a much higher-level abstraction—an independently stored, replicated unit that happens to be stored as a regular file on each DataNode's local file system.
How DataNodes store blocks:
On each DataNode, blocks are stored as regular files in the local file system (typically ext4 or XFS). A DataNode with 12 disks might have 12 storage directories, one per disk. Within each directory:
123456789101112131415161718
/data/disk1/hdfs/└── current/ ├── VERSION # DataNode storage version info ├── BP-random-nn-ts/ # Block Pool (one per namespace) │ └── current/ │ └── finalized/ │ ├── subdir0/ │ │ ├── blk_1073741825 # Block data file │ │ ├── blk_1073741825.meta # Checksum metadata │ │ ├── blk_1073741826 │ │ ├── blk_1073741826.meta │ │ └── ... │ ├── subdir1/ │ │ └── ... │ └── subdir63/ │ └── ... │ └── rbw/ # Replica Being Written (incomplete blocks) └── in_use.lock # Prevents concurrent accessThe .meta file contains:
Every read operation verifies checksums, detecting silent data corruption.
HDFS achieves reliability through replication—storing multiple copies of each block across different machines. The default replication factor of 3 means every block exists in three locations, enabling the cluster to survive up to two simultaneous failures of nodes holding replicas of any given block.
But not all replica placements are equal. HDFS implements rack-aware replica placement to balance reliability against network efficiency:
Why this specific placement?
Rack failure tolerance: If Rack 1 loses power or its switch fails, replicas 2 and 3 on Rack 2 ensure data remains available. This is crucial—rack failures (power, switch, cooling) affect all nodes simultaneously.
Network efficiency: Two replicas share a rack, so reads from compute nodes in that rack avoid cross-rack traffic. Writes also benefit: the pipeline from replica 2 to replica 3 stays within Rack 2.
Failure scenarios survived:
HDFS cannot discover rack topology automatically. Administrators must configure a rack mapping script that maps IP addresses or hostnames to rack identifiers. Without this configuration, HDFS assumes all nodes are on a single rack, losing rack-failure protection.
Re-replication triggers:
The NameNode continuously monitors replica counts and initiates re-replication when:
Re-replication is prioritized: blocks with only one remaining replica are replicated before those with two.
Understanding how data flows during read and write operations reveals HDFS's architectural elegance. The separation of metadata (NameNode) from data (DataNodes) enables high aggregate throughput.
The Write Path:
123456789101112131415161718192021222324252627282930313233343536
Client NameNode DataNodes | | | |---- create(path) ------->| | | |-- check permissions | | |-- check file exists | | |-- create file entry | |<--- file handle ---------| | | | | |---- addBlock() --------->| | | |-- select DataNodes | | |-- create block entry | |<--- [DN1, DN2, DN3] -----| | | | | |============= Data Pipeline Setup ===================| | | | |---- connect ----------->[DN1] | | |---- connect ----------->[DN2] | | |---- connect ----------->[DN3] | | | |============= Streaming Data ========================| | | | |---- packet 1 ---------->[DN1] | | |---- packet 1 ---------->[DN2] | | |---- packet 1 ---------->[DN3] | | |<--- ack 1 ---------------| | |<--- ack 1 ---------------| |<--- ack 1 ---------------| | | | | |---- packet N ---------->[DN1] ... (continues) | | | | |============= Block Complete ========================| | | | |---- close() ------------>| | | |<--- blockReceived -------| | | (from all DNs) | |<--- success -------------| |Key write path characteristics:
Pipeline writes: Data flows from client to DN1 to DN2 to DN3, not from client to each DN in parallel. This reduces client bandwidth requirements by 3x.
Packet-based streaming: Data is broken into 64 KB packets. Each packet contains checksums. Acks flow backward through the pipeline.
Lease mechanism: The NameNode grants the client a lease on the file. If the client fails, the lease expires, and recovery procedures reclaim the file.
Minimum replica acknowledgment: Writes succeed when a minimum number of replicas (default: 1) acknowledge. The remaining replicas complete asynchronously.
The Read Path:
123456789101112131415161718192021222324
Client NameNode DataNodes | | | |---- open(path) --------->| | | |-- check permissions | | |-- look up file | |<--- block locations -----| | | [B1: DN2, DN3, DN5] | | | [B2: DN1, DN4, DN6] | | | [B3: DN2, DN4, DN5] | | | | | |============= Read Block 1 ==========================| | | | |---- connect to DN2 ----------------------->[DN2] | |---- read B1 request ---------------------->[DN2] | |<--- B1 data stream -----------------------| | | (verify checksums) | | | | | |============= Read Block 2 ==========================| | | | |---- connect to DN1 ------->[DN1] | |---- read B2 request ------>[DN1] | |<--- B2 data stream -------| | | | | |============= (continues for remaining blocks) ======|Operating at scale with commodity hardware requires robust fault tolerance at every level. HDFS implements multiple complementary mechanisms to detect, handle, and recover from failures.
DataNode failure detection:
| Mechanism | Interval | Purpose |
|---|---|---|
| Heartbeat | Every 3 seconds | Prove DataNode is alive; carry NameNode commands |
| Block Report | Every 6 hours | Full inventory of all blocks on DataNode |
| Incremental Block Report | Immediate | Report newly received or deleted blocks |
| Failure detection | 10 minutes (10 missed heartbeats) | Declare DataNode dead; schedule re-replication |
Data integrity protection:
Studies show that disks silently corrupt data more often than most engineers expect—approximately 1 in every 10^14 to 10^15 bits. For petabyte-scale clusters, this means corruption events happen regularly. Checksums are not optional at scale; they are essential.
NameNode high availability:
The original HDFS design had a single NameNode—a glaring single point of failure. Modern HDFS deployments use NameNode High Availability (HA) with an Active-Standby configuration:
We have explored the Hadoop Distributed File System from its design philosophy through its operational mechanisms. Here are the essential takeaways:
What's next:
HDFS was inspired by Google's GFS paper. In the next page, we'll examine the original GFS design concepts—the ideas that spawned HDFS and shaped how the industry thinks about distributed storage. Understanding GFS provides deeper insight into the tradeoffs and innovations that made large-scale distributed file systems possible.
You now understand HDFS's architecture, from its design philosophy through its operational mechanisms. You can reason about block placement, replication strategies, and fault tolerance in distributed storage systems. Next, we'll explore the GFS concepts that inspired HDFS and continue to influence distributed system design.