Distributed File Systems

In 2004, Google published a paper describing the Google File System (GFS), a distributed file system designed to store and process petabytes of data across thousands of commodity servers. This paper inspired Doug Cutting and Mike Cafarella to create an open-source implementation called the Hadoop Distributed File System (HDFS), which would go on to become the foundation of the big data revolution.

HDFS fundamentally changed how organizations think about data storage. Before HDFS, storing petabytes of data required expensive, specialized hardware—SAN arrays, NAS appliances, and custom high-availability systems costing millions of dollars. HDFS proved that you could achieve similar reliability and far greater scalability using thousands of cheap commodity servers, accepting that failures are inevitable and designing the system to handle them gracefully.

Today, HDFS remains the backbone of countless data platforms, powering analytics at companies like Facebook (storing over 300 petabytes), Yahoo (40+ petabytes), and thousands of enterprises worldwide. Understanding HDFS architecture isn't just academically interesting—it's essential knowledge for anyone designing systems that must handle massive scale.

Before diving into architecture, we must understand the assumptions and design goals that shaped HDFS. These aren't arbitrary choices—they reflect deep insights about hardware economics, failure patterns, and workload characteristics.

HDFS was designed with specific assumptions:

The core insight behind HDFS:

Traditional storage systems tried to prevent failures through expensive, specialized hardware—redundant disk controllers, battery-backed caches, and enterprise-grade components. This approach doesn't scale economically to petabyte levels.

HDFS takes a fundamentally different approach: accept that failures are inevitable and build recovery into the software layer. Using commodity hardware and aggressive replication, HDFS achieves reliability through redundancy rather than prevention. A three-way replica on a $500 commodity server is more cost-effective than a single copy on a $50,000 enterprise array—and often more reliable because failures are truly independent.

HDFS employs a master-slave architecture with two primary component types: a single NameNode (the master) and multiple DataNodes (the slaves). This separation of concerns is fundamental to understanding how HDFS operates.

Key architectural insight: The NameNode never handles actual data—it only manages metadata. All data transfers happen directly between clients and DataNodes. This design prevents the NameNode from becoming a throughput bottleneck, allowing the cluster to scale data bandwidth by simply adding more DataNodes.

Let's examine each component in detail:

The NameNode is the central coordinator of an HDFS cluster. It maintains the entire file system namespace—the directory tree, file attributes, and the mapping from files to blocks. Understanding the NameNode's internal structure is crucial for capacity planning and troubleshooting.

NameNode's In-Memory Data Structures:

Metadata Persistence Mechanism:

The NameNode must persist metadata durably to survive restarts. It uses a combination of periodic checkpointing and continuous journaling:

1. On each metadata operation: The change is first written to the edit log (WAL pattern) before being applied to in-memory structures.

2. Periodically: The Secondary NameNode (or Checkpoint Node) fetches the FsImage and recent edit logs, applies the edits to create a new FsImage, and returns it to the NameNode.

3. On NameNode restart: It loads the latest FsImage, replays all edit logs since that checkpoint, and waits for DataNodes to report their blocks. Only after receiving a sufficient number of block reports does the NameNode leave "safe mode" and begin normal operations.

DataNodes are the workhorses of HDFS—they store the actual data blocks and handle all data transfer operations. Understanding how DataNodes manage storage is essential for capacity planning and performance tuning.

Block Storage Model:

When a file is written to HDFS, it's split into fixed-size blocks (default 128MB, configurable per file). Each block is stored as a separate file on the DataNode's local file system, along with a corresponding metadata file containing checksums.

Why large blocks?

HDFS uses large blocks compared to traditional file systems (4KB-64KB) for several important reasons:

1. Reduced NameNode memory: Each block requires ~150 bytes in NameNode memory. A 1TB file at 128MB blocks = 8,192 blocks; at 4KB blocks = 268 million blocks.

2. Optimized for sequential reads: Large blocks minimize seek overhead. MapReduce jobs typically scan entire blocks sequentially.

3. Lower network overhead: Fewer block transfers mean fewer TCP connections and metadata lookups.

4. Efficient replication: Replicating larger chunks is more efficient than many small transfers.

DataNode Operations:

Replication is HDFS's primary mechanism for ensuring data durability and availability. By default, every block is replicated to three DataNodes. The placement of these replicas follows a carefully designed policy that balances reliability, write bandwidth, and read performance.

Rack-Aware Replica Placement:

HDFS assumes that nodes are organized into racks, and that network bandwidth within a rack is higher than across racks. The default placement policy for three replicas is:

Why this specific placement?

The placement policy optimizes multiple objectives:

Replication Pipeline:

When writing a block, HDFS uses a pipeline replication strategy rather than parallel writes:

1. Client writes to first DataNode
2. First DataNode simultaneously writes to disk AND forwards to second DataNode
3. Second DataNode writes to disk AND forwards to third DataNode
4. Acknowledgments flow back up the pipeline
5. Client receives ACK only after all replicas confirm

This pipeline approach efficiently uses network bandwidth—the client only needs bandwidth to reach one DataNode, and the replication work is distributed.

Understanding the complete data flow for reads and writes reveals why HDFS achieves high throughput and how it maintains consistency. Let's trace both paths step by step.

The original HDFS design had a critical weakness: a single NameNode was a single point of failure. If the NameNode crashed, the entire cluster was unavailable until manual recovery. HDFS 2.0 introduced High Availability (HA) to address this limitation.

NameNode HA Architecture:

HDFS HA provides two NameNodes in an active-standby configuration:

Failover Process:

1. ZKFC detects Active NameNode failure (health check failures or ZooKeeper session timeout)
2. ZKFC initiates fencing to prevent split-brain (ensures old Active cannot respond to requests)
3. ZKFC triggers Standby to transition to Active
4. New Active NameNode begins accepting client requests
5. DataNodes now report to new Active

Typical failover time: 30-60 seconds (configurable), compared to hours for manual recovery in non-HA configurations.

Even with HA, the NameNode remains a scalability bottleneck—all namespace operations funnel through a single NameNode's memory. For clusters with hundreds of millions of files, this hits hard limits. HDFS Federation addresses this through horizontal namespace scaling.

Federation Architecture:

Federation allows multiple independent NameNodes to share datanodes, each managing a portion of the namespace:

Benefits of Federation:

• Horizontal namespace scaling: Add more NameNodes to manage more files. No single NameNode memory limit.
• Isolation: Different organizations/applications use different NameNodes. Failure and performance isolation.
• Improved throughput: Multiple NameNodes handle metadata operations in parallel.

Limitations:

• No cross-NameNode operations: Can't move files across namespaces atomically; appears as copy+delete.
• Operational complexity: Multiple NameNodes to monitor, backup, and upgrade.
• ViewFS configuration: Clients need consistent mount table configuration.

HDFS's design choices optimize for specific workloads at the expense of others. Understanding these trade-offs prevents misapplication and guides technology selection.

When NOT to use HDFS:

HDFS excels when your workload matches its assumptions: large files, write-once/read-many, sequential access, and batch processing. For other patterns, consider purpose-built systems.

We've explored the architecture that enables HDFS to store petabytes of data reliably on commodity hardware. Let's consolidate the key concepts:

What's Next:

HDFS pioneered distributed storage for big data, but it's not the only approach. In the next page, we'll explore Ceph, a unified storage platform that provides object, block, and file interfaces with a fundamentally different architecture—eliminating the single-point-of-failure metadata server entirely through distributed hashing.