79:T4d0,In 2004, Google published a paper describing the Google File System (GFS), a distributed file system designed to store and process petabytes of data across thousands of commodity servers. This paper inspired Doug Cutting and Mike Cafarella to create an open-source implementation called the **Hadoop Distributed File System (HDFS)**, which would go on to become the foundation of the big data revolution. **HDFS fundamentally changed how organizations think about data storage.** Before HDFS, storing petabytes of data required expensive, specialized hardware—SAN arrays, NAS appliances, and custom high-availability systems costing millions of dollars. HDFS proved that you could achieve similar reliability and far greater scalability using thousands of cheap commodity servers, accepting that failures are inevitable and designing the system to handle them gracefully. Today, HDFS remains the backbone of countless data platforms, powering analytics at companies like Facebook (storing over 300 petabytes), Yahoo (40+ petabytes), and thousands of enterprises worldwide. Understanding HDFS architecture isn't just academically interesting—it's essential knowledge for anyone designing systems that must handle massive scale.71:{"json":{"id":"seed_9dd6eb84c391305448b3eb1fc55e8fb1","slug":"hld-distributed-file-systems","category":"System Design HLD","topic":"Distributed File Systems","title":"Distributed File Systems","description":"Master distributed file systems that power large-scale data storage and processing. Explore HDFS, Ceph, GlusterFS, and MinIO architectures, understanding how they achieve massive scalability, fault tolerance, and high throughput for modern data-intensive applications.","difficulty":"ADVANCED","estimatedMinutes":90,"pages":[{"id":"seed_41f4309f88c4c4441916dfd99a767d53","title":"HDFS Architecture","order":0,"content":[{"id":"opening-hook","title":"The File System That Changed Big Data","content":[{"data":{"content":"$79"},"type":"text"},{"data":{"icon":"Target","title":"What You Will Learn","content":"By the end of this page, you will understand HDFS's master-slave architecture, the role of NameNodes and DataNodes, how data is distributed and replicated for fault tolerance, the write and read paths in detail, federation and high availability mechanisms, and the trade-offs that make HDFS optimal for batch processing workloads.","variant":"info"},"type":"callout"}]},{"id":"core-design-philosophy","title":"Core Design Philosophy","content":[{"data":{"content":"Before diving into architecture, we must understand **the assumptions and design goals** that shaped HDFS. These aren't arbitrary choices—they reflect deep insights about hardware economics, failure patterns, and workload characteristics.\n\n**HDFS was designed with specific assumptions:**"},"type":"text"},{"data":{"items":["**Hardware failures are inevitable** — In a cluster of thousands of nodes, multiple failures happen daily. The system must detect and recover from failures automatically, without human intervention.","**Files are written once and read many times** — Typical big data workloads involve batch processing where data is ingested once and analyzed repeatedly. Optimizing for this pattern simplifies design significantly.","**Large files are the norm** — HDFS targets multi-gigabyte to terabyte-scale files (log files, sensor data, scientific datasets). Optimizing for small files would introduce unnecessary overhead.","**High throughput matters more than low latency** — Batch analytics needs to scan terabytes efficiently. Interactive, low-latency access is not the primary use case.","**Moving computation is cheaper than moving data** — With petabytes of data, network bandwidth becomes the bottleneck. HDFS provides data locality information so computation can be scheduled where data resides."],"title":{"icon":"Target","text":"HDFS Design Assumptions","type":"icon","color":"primary","variant":"outline-circle"},"variant":"accent"},"type":"list"},{"data":{"icon":"AlertCircle","title":"Understanding Trade-offs","content":"These assumptions have profound implications. HDFS is *not* a general-purpose file system. It excels at batch processing workloads but is poorly suited for interactive queries, random writes, or applications requiring millions of small files. Choosing HDFS for the wrong workload leads to frustration; understanding its design philosophy prevents this mistake.","variant":"warning"},"type":"callout"},{"data":{"content":"**The core insight behind HDFS:**\n\nTraditional storage systems tried to prevent failures through expensive, specialized hardware—redundant disk controllers, battery-backed caches, and enterprise-grade components. This approach doesn't scale economically to petabyte levels.\n\nHDFS takes a fundamentally different approach: **accept that failures are inevitable and build recovery into the software layer.** Using commodity hardware and aggressive replication, HDFS achieves reliability through redundancy rather than prevention. A three-way replica on a $500 commodity server is more cost-effective than a single copy on a $50,000 enterprise array—and often more reliable because failures are truly independent."},"type":"text"}],"titleType":"number","titleNumber":1},{"id":"master-slave-architecture","title":"Master-Slave Architecture Overview","content":[{"data":{"content":"HDFS employs a **master-slave architecture** with two primary component types: a single **NameNode** (the master) and multiple **DataNodes** (the slaves). This separation of concerns is fundamental to understanding how HDFS operates."},"type":"text"},{"data":{"height":"450px","mermaidSyntax":"graph TB\n subgraph \"HDFS Cluster\"\n NN[NameNode\nMaster]\n subgraph \"DataNodes - Slaves\"\n DN1[DataNode 1]\n DN2[DataNode 2]\n DN3[DataNode 3]\n DN4[DataNode N...]\n end\n end\n \n Client[Client Application]\n \n Client -->|\"1. Metadata Ops\"| NN\n NN -->|\"2. Block Locations\"| Client\n Client -->|\"3. Direct Data Transfer\"| DN1\n Client -->|\"3. Direct Data Transfer\"| DN2\n Client -->|\"3. Direct Data Transfer\"| DN3\n \n DN1 -.->|Heartbeat + Block Reports| NN\n DN2 -.->|Heartbeat + Block Reports| NN\n DN3 -.->|Heartbeat + Block Reports| NN\n DN4 -.->|Heartbeat + Block Reports| NN"},"type":"mermaid-excalidraw"},{"data":{"content":"**Key architectural insight:** The NameNode never handles actual data—it only manages metadata. All data transfers happen directly between clients and DataNodes. This design prevents the NameNode from becoming a throughput bottleneck, allowing the cluster to scale data bandwidth by simply adding more DataNodes.\n\nLet's examine each component in detail:"},"type":"text"},{"data":{"rows":[["NameNode","Manages namespace, file-to-block mapping, block locations","All state in memory; persists metadata to edit log and fsimage"],["Secondary NameNode","Periodically merges edit log with fsimage (checkpointing)","Not a failover node; reduces NameNode recovery time"],["DataNode","Stores and retrieves data blocks on local disk","Stateless from NameNode perspective; reports blocks on startup"],["Client","Initiates reads/writes, handles data pipeline logic","Uses HDFS client library (Java, C, or WebHDFS)"]],"caption":"HDFS Component Responsibilities","headers":["Component","Responsibility","State Characteristics"]},"type":"table"}],"titleType":"number","titleNumber":2},{"id":"namenode-in-depth","title":"The NameNode: Brain of the File System","content":[{"data":{"content":"The **NameNode** is the central coordinator of an HDFS cluster. It maintains the entire file system namespace—the directory tree, file attributes, and the mapping from files to blocks. Understanding the NameNode's internal structure is crucial for capacity planning and troubleshooting.\n\n**NameNode's In-Memory Data Structures:**"},"type":"text"},{"data":{"items":["**FsImage** — A persistent checkpoint of the file system metadata, including namespace tree, file-to-block mappings, replication factors, permissions, and quotas.","**Edit Log** — An append-only transaction log recording every metadata operation since the last checkpoint. Enables durability without constantly rewriting FsImage.","**Block Map** — In-memory structure mapping blocks to their physical locations on DataNodes. *This is never persisted*—it's reconstructed from DataNode block reports on every startup.","**DataNode Registry** — Tracks live DataNodes, their capacity, and health status based on heartbeats."],"title":{"icon":"Info","text":"NameNode Memory Model","type":"icon","color":"primary","variant":"outline-circle"},"variant":"accent"},"type":"list"},{"data":{"icon":"Info","title":"Critical Design Decision: Block Locations in Memory","content":"The NameNode keeps the *entire* block map in memory. For a cluster with 100 million blocks, this requires significant RAM (roughly 150-200 bytes per block). This design enables O(1) lookup for any block location but limits cluster size by NameNode memory. A typical NameNode with 128GB RAM can manage ~400 million blocks, which at 128MB block size equals ~50PB of data.","variant":"note"},"type":"callout"},{"data":{"content":"**Metadata Persistence Mechanism:**\n\nThe NameNode must persist metadata durably to survive restarts. It uses a combination of periodic checkpointing and continuous journaling:\n\n1. **On each metadata operation:** The change is first written to the edit log (WAL pattern) before being applied to in-memory structures.\n\n2. **Periodically:** The Secondary NameNode (or Checkpoint Node) fetches the FsImage and recent edit logs, applies the edits to create a new FsImage, and returns it to the NameNode.\n\n3. **On NameNode restart:** It loads the latest FsImage, replays all edit logs since that checkpoint, and waits for DataNodes to report their blocks. Only after receiving a sufficient number of block reports does the NameNode leave \"safe mode\" and begin normal operations."},"type":"text"},{"data":{"code":"# NameNode Startup Sequence\n \n1. Load FsImage into memory\n └─ Contains: namespace tree, file metadata, block IDs\n └─ Does NOT contain: block locations (DataNode mapping)\n\n2. Replay Edit Log\n └─ Apply all transactions since checkpoint\n └─ Rebuild current in-memory state\n\n3. Enter Safe Mode\n └─ Read-only operation allowed\n └─ Block reports received from DataNodes\n └─ Block map reconstructed progressively\n\n4. Check Block Health\n └─ Count blocks with minimum replicas\n └─ Threshold: 99.9% of blocks must have ≥1 replica\n\n5. Leave Safe Mode\n └─ Full read/write operations enabled\n └─ Background replication begins for under-replicated blocks","fileName":"namenode-recovery-sequence.md","showLineNumbers":false},"type":"code"}],"titleType":"number","titleNumber":3},{"id":"datanodes-block-storage","title":"DataNodes and Block Storage","content":[{"data":{"content":"**DataNodes** are the workhorses of HDFS—they store the actual data blocks and handle all data transfer operations. Understanding how DataNodes manage storage is essential for capacity planning and performance tuning.\n\n**Block Storage Model:**\n\nWhen a file is written to HDFS, it's split into fixed-size **blocks** (default 128MB, configurable per file). Each block is stored as a separate file on the DataNode's local file system, along with a corresponding metadata file containing checksums."},"type":"text"},{"data":{"rows":[["HDFS 1.x","64 MB","Original design, balanced memory overhead vs. locality"],["HDFS 2.x+","128 MB","Increased to reduce NameNode memory pressure at scale"],["Large Clusters","256 MB or 512 MB","Custom configuration for massive files to minimize metadata"]],"caption":"HDFS Block Size Evolution","headers":["Version","Default Block Size","Rationale"]},"type":"table"},{"data":{"content":"**Why large blocks?**\n\nHDFS uses large blocks compared to traditional file systems (4KB-64KB) for several important reasons:\n\n1. **Reduced NameNode memory:** Each block requires ~150 bytes in NameNode memory. A 1TB file at 128MB blocks = 8,192 blocks; at 4KB blocks = 268 million blocks.\n\n2. **Optimized for sequential reads:** Large blocks minimize seek overhead. MapReduce jobs typically scan entire blocks sequentially.\n\n3. **Lower network overhead:** Fewer block transfers mean fewer TCP connections and metadata lookups.\n\n4. **Efficient replication:** Replicating larger chunks is more efficient than many small transfers.\n\n**DataNode Operations:**"},"type":"text"},{"data":{"items":["**Block Storage** — Store blocks as files in local file system, maintaining data and checksum files.","**Heartbeat Signals** — Send periodic heartbeats to NameNode (default: every 3 seconds) indicating liveness.","**Block Reports** — Periodically send complete list of all blocks stored (default: every 6 hours).","**Data Integrity** — Verify checksums on read and during background scanning; report corrupted blocks to NameNode.","**Pipeline Operations** — Participate in write pipelines, receiving data and forwarding to next DataNode in chain.","**Block Operations** — Execute NameNode commands: replicate blocks, delete blocks, recover blocks."],"title":{"icon":"Zap","text":"DataNode Responsibilities","type":"icon","color":"primary","variant":"outline-circle"},"variant":"glass"},"type":"list"},{"data":{"icon":"Lightbulb","title":"Block Scanner: Silent Guardian","content":"DataNodes continuously verify stored blocks through a 'block scanner' that reads all blocks and validates checksums over a configurable period (default: 3 weeks). This proactive scanning catches disk corruption before it's discovered during reads, giving the NameNode time to create additional replicas.","variant":"tip"},"type":"callout"}],"titleType":"number","titleNumber":4},{"id":"data-replication-strategy","title":"Data Replication Strategy","content":[{"data":{"content":"Replication is HDFS's primary mechanism for ensuring **data durability and availability**. By default, every block is replicated to three DataNodes. The placement of these replicas follows a carefully designed policy that balances reliability, write bandwidth, and read performance.\n\n**Rack-Aware Replica Placement:**\n\nHDFS assumes that nodes are organized into racks, and that network bandwidth within a rack is higher than across racks. The default placement policy for three replicas is:"},"type":"text"},{"data":{"type":"ordered","items":["**First replica:** Placed on the node where the client is running (or a random node if client is external).","**Second replica:** Placed on a *different rack* than the first replica. This ensures data survives a complete rack failure.","**Third replica:** Placed on the *same rack as the second replica* but on a different node. This optimizes write bandwidth (only one cross-rack transfer) while maintaining rack-level redundancy."],"variant":"accent"},"type":"list"},{"data":{"height":"400px","mermaidSyntax":"graph TB\n subgraph \"Rack 1\"\n DN1[\"DataNode 1\nReplica 1\"]\n end\n subgraph \"Rack 2\"\n DN2[\"DataNode 2\nReplica 2\"]\n DN3[\"DataNode 3\nReplica 3\"]\n end\n \n Client[Client] -->|\"1. Write\"| DN1\n DN1 -->|\"2. Cross-rack transfer\"| DN2\n DN2 -->|\"3. Intra-rack transfer\"| DN3\n \n style DN1 fill:#4CAF50\n style DN2 fill:#2196F3\n style DN3 fill:#2196F3"},"type":"mermaid-excalidraw"},{"data":{"content":"**Why this specific placement?**\n\nThe placement policy optimizes multiple objectives:"},"type":"text"},{"data":{"rows":[["Rack failure tolerance","Two racks contain replicas; one full rack can fail"],["Write performance","Only one cross-rack hop; second and third replicas share rack"],["Read performance","High probability of finding replica on same rack as reader"],["Data locality","First replica placed on writer's node when possible"]],"headers":["Objective","How Policy Addresses It"]},"type":"table"},{"data":{"content":"**Replication Pipeline:**\n\nWhen writing a block, HDFS uses a **pipeline replication** strategy rather than parallel writes:\n\n1. Client writes to first DataNode\n2. First DataNode simultaneously writes to disk AND forwards to second DataNode\n3. Second DataNode writes to disk AND forwards to third DataNode\n4. Acknowledgments flow back up the pipeline\n5. Client receives ACK only after all replicas confirm\n\nThis pipeline approach efficiently uses network bandwidth—the client only needs bandwidth to reach one DataNode, and the replication work is distributed."},"type":"text"},{"data":{"icon":"AlertCircle","title":"Under-Replication Detection and Recovery","content":"When the NameNode detects an under-replicated block (via missing heartbeats or block reports), it schedules re-replication to another DataNode. The priority is based on how under-replicated the block is: blocks with only one replica are replicated before those with two. This automatic recovery is how HDFS tolerates DataNode failures without data loss.","variant":"warning"},"type":"callout"}],"titleType":"number","titleNumber":5},{"id":"read-write-paths","title":"Read and Write Paths in Detail","content":[{"data":{"content":"Understanding the complete data flow for reads and writes reveals why HDFS achieves high throughput and how it maintains consistency. Let's trace both paths step by step."},"type":"text"},{"data":{"tabs":[{"id":"write-path","label":"Write Path","content":[{"data":{"content":"**HDFS Write Path (Creating a New File):**"},"type":"text"},{"data":{"type":"ordered","items":["**Client contacts NameNode** — Requests to create file `/user/data/file.txt`. NameNode checks permissions, validates path doesn't exist, creates entry in namespace.","**NameNode logs operation** — File creation logged to edit log BEFORE response. Client receives success but file has no blocks yet.","**Client requests block allocation** — When ready to write data, client asks NameNode for DataNodes to store first block.","**NameNode selects DataNodes** — Using rack-aware policy, selects three DataNodes (e.g., DN1, DN5, DN6) and returns ordered list to client.","**Client initiates pipeline** — Opens TCP connection to DN1, passing the list of remaining nodes (DN5, DN6).","**Pipeline assembly** — DN1 connects to DN5, DN5 connects to DN6. Success confirmed back to client.","**Data streaming begins** — Client writes packets (64KB default) to DN1. DN1 simultaneously writes to disk AND forwards to DN5.","**Packet acknowledgment** — ACKs flow back through pipeline. Client tracks outstanding packets in 'ack queue'.","**Block completion** — When block is full, client notifies NameNode of successful block storage, then requests allocation for next block.","**File close** — After all blocks written, client issues close. NameNode verifies minimum replicas, finalizes file."],"variant":"accent"},"type":"list"},{"data":{"icon":"Info","title":"Write Semantics","content":"HDFS provides write-once semantics. Once a file is closed, it cannot be modified—only appended to (added in Hadoop 2.x). This simplifies consistency and enables aggressive caching.","variant":"note"},"type":"callout"}],"titleIcon":"Zap"},{"id":"read-path","label":"Read Path","content":[{"data":{"content":"**HDFS Read Path (Reading a File):**"},"type":"text"},{"data":{"type":"ordered","items":["**Client contacts NameNode** — Requests to open `/user/data/file.txt`. NameNode checks permissions and file existence.","**NameNode returns block list** — Returns list of blocks comprising the file, along with DataNode locations for each block, sorted by distance from client.","**Client reads first block** — Opens stream to closest DataNode holding first block. 'Closest' = same node > same rack > different rack.","**Checksum verification** — Client verifies packet checksums during read. If corruption detected, reports to NameNode and tries next replica.","**Sequential block reads** — After finishing first block, client moves to next block, opening new connection to appropriate DataNode.","**Transparent failover** — If DataNode fails mid-read, client automatically retries with another replica. Reads are idempotent.","**Connection pooling** — Client may reuse connections to same DataNode for subsequent blocks (implementation-dependent)."],"variant":"accent"},"type":"list"},{"data":{"icon":"Lightbulb","title":"Short-Circuit Local Reads","content":"When the client runs on the same machine as a DataNode holding the desired block, HDFS can bypass the network entirely. The client reads directly from the local file system using shared memory, dramatically improving read performance for data-local computations.","variant":"tip"},"type":"callout"}],"titleIcon":"BookOpen"}]},"type":"tabs"}],"titleType":"number","titleNumber":6},{"id":"high-availability","title":"High Availability Architecture","content":[{"data":{"content":"The original HDFS design had a critical weakness: **a single NameNode was a single point of failure**. If the NameNode crashed, the entire cluster was unavailable until manual recovery. HDFS 2.0 introduced High Availability (HA) to address this limitation.\n\n**NameNode HA Architecture:**\n\nHDFS HA provides **two NameNodes** in an active-standby configuration:"},"type":"text"},{"data":{"height":"450px","mermaidSyntax":"graph TB\n subgraph \"NameNode HA\"\n Active[Active NameNode]\n Standby[Standby NameNode]\n end\n \n subgraph \"Shared Edit Log\"\n JN1[JournalNode 1]\n JN2[JournalNode 2]\n JN3[JournalNode 3]\n end\n \n ZK[ZooKeeper Cluster]\n \n Active -->|\"Write Edits\"| JN1\n Active -->|\"Write Edits\"| JN2\n Active -->|\"Write Edits\"| JN3\n \n Standby -->|\"Read Edits\"| JN1\n Standby -->|\"Read Edits\"| JN2\n Standby -->|\"Read Edits\"| JN3\n \n Active <-->|\"ZKFC Health Check\"| ZK\n Standby <-->|\"ZKFC Health Check\"| ZK\n \n DN1[DataNode] -.->|\"Block Reports\"| Active\n DN1 -.->|\"Block Reports\"| Standby"},"type":"mermaid-excalidraw"},{"data":{"items":["**Active NameNode** — Handles all client operations and writes edits to JournalNodes.","**Standby NameNode** — Continuously reads edits from JournalNodes, maintaining synchronized state. Ready to take over instantly.","**JournalNodes (QJM)** — Quorum-based shared edit log. Active writes to majority; Standby reads any. Typically 3 or 5 nodes for odd quorum.","**ZooKeeper** — Provides failure detection and leader election. Ensures only one Active at a time (prevents split-brain).","**ZKFC (ZooKeeper Failover Controller)** — Daemon running on each NameNode machine, monitoring health and coordinating failover."],"title":{"icon":"CheckCircle","text":"HA Components","type":"icon","color":"primary","variant":"outline-circle"},"variant":"accent"},"type":"list"},{"data":{"content":"**Failover Process:**\n\n1. ZKFC detects Active NameNode failure (health check failures or ZooKeeper session timeout)\n2. ZKFC initiates fencing to prevent split-brain (ensures old Active cannot respond to requests)\n3. ZKFC triggers Standby to transition to Active\n4. New Active NameNode begins accepting client requests\n5. DataNodes now report to new Active\n\n**Typical failover time:** 30-60 seconds (configurable), compared to hours for manual recovery in non-HA configurations."},"type":"text"},{"data":{"icon":"AlertCircle","title":"Fencing: Preventing Split-Brain","content":"Split-brain—where both NameNodes believe they're Active—would cause catastrophic data corruption. HDFS uses 'fencing' mechanisms to guarantee the old Active is truly stopped before promoting the Standby. Common fencing methods include SSH-based process killing, STONITH (Shoot The Other Node In The Head) via power control, or preventing the old Active from writing to JournalNodes.","variant":"warning"},"type":"callout"}],"titleType":"number","titleNumber":7},{"id":"federation-scalability","title":"Federation and Horizontal Scalability","content":[{"data":{"content":"Even with HA, the NameNode remains a scalability bottleneck—all namespace operations funnel through a single NameNode's memory. For clusters with hundreds of millions of files, this hits hard limits. **HDFS Federation** addresses this through horizontal namespace scaling.\n\n**Federation Architecture:**\n\nFederation allows multiple independent NameNodes to share datanodes, each managing a portion of the namespace:"},"type":"text"},{"data":{"rows":[["Namespace Volume","Each NameNode manages an independent namespace (directory tree). No coordination between NameNodes."],["Block Pool","Blocks belonging to a single namespace. Each NameNode has its own block pool."],["DataNode","Stores blocks from ALL block pools, serving all NameNodes. Single DataNode, multiple pools."],["Client ViewFS","Client-side mount table mapping paths to NameNodes. Creates unified namespace view."]],"caption":"Federation Concepts","headers":["Concept","Description"]},"type":"table"},{"data":{"code":"\n\n \n fs.viewfs.mounttable.default.link./user\n hdfs://namenode1:8020/user\n \n \n fs.viewfs.mounttable.default.link./data\n hdfs://namenode2:8020/data\n \n \n fs.viewfs.mounttable.default.link./logs\n hdfs://namenode3:8020/logs\n \n","fileName":"viewfs-mount-table.xml","showLineNumbers":true},"type":"code"},{"data":{"content":"**Benefits of Federation:**\n\n- **Horizontal namespace scaling:** Add more NameNodes to manage more files. No single NameNode memory limit.\n- **Isolation:** Different organizations/applications use different NameNodes. Failure and performance isolation.\n- **Improved throughput:** Multiple NameNodes handle metadata operations in parallel.\n\n**Limitations:**\n\n- **No cross-NameNode operations:** Can't move files across namespaces atomically; appears as copy+delete.\n- **Operational complexity:** Multiple NameNodes to monitor, backup, and upgrade.\n- **ViewFS configuration:** Clients need consistent mount table configuration."},"type":"text"}],"titleType":"number","titleNumber":8},{"id":"tradeoffs-limitations","title":"Trade-offs and Limitations","content":[{"data":{"content":"HDFS's design choices optimize for specific workloads at the expense of others. Understanding these trade-offs prevents misapplication and guides technology selection."},"type":"text"},{"data":{"left":[{"data":{"items":["Petabyte-scale storage on commodity hardware","Extremely high sequential read/write throughput","Automatic failure detection and recovery","Data locality awareness for compute frameworks","Cost-effective compared to enterprise storage","Proven at scale (Facebook: 300+ PB)"],"title":{"icon":"CheckCircle","text":"HDFS Strengths","type":"icon","color":"primary","variant":"outline-circle"},"variant":"accent"},"type":"list"}],"right":[{"data":{"items":["High latency for small files (NameNode overhead)","No random write support (append only)","Single-writer semantics (no concurrent writes)","Small files problem (millions → NameNode OOM)","No native encryption at rest (added in 3.x)","NameNode memory limits cluster file count"],"title":{"icon":"XCircle","text":"HDFS Limitations","type":"icon","color":"destructive","variant":"outline-circle"},"variant":"accent"},"type":"list"}]},"type":"two-column"},{"data":{"icon":"AlertCircle","title":"The Small Files Problem","content":"Each file, directory, and block consumes ~150 bytes in NameNode memory. With millions of small files (e.g., storing 1KB files), you hit NameNode memory limits long before exhausting DataNode storage. Solutions include HAR files (Hadoop Archives), SequenceFiles (container format), or using HBase for small object storage.","variant":"warning"},"type":"callout"},{"data":{"content":"**When NOT to use HDFS:**"},"type":"text"},{"data":{"rows":[["Interactive queries requiring <100ms latency","NameNode lookup overhead, no caching","Apache Ignite, ClickHouse"],["Millions of small files (

Ceph: Unified Storage

Ceph: Unified Storage