Distributed File Systems

We've examined HDFS with its centralized NameNode and Ceph with its sophisticated CRUSH algorithm. Both are powerful, but both come with significant operational complexity. What if you need distributed storage but want to prioritize simplicity?

GlusterFS takes a fundamentally different approach. Instead of inventing new data formats or complex placement algorithms, GlusterFS leverages existing file systems (XFS, ext4) and aggregates them into a unified namespace. There's no metadata server, no specialized storage format—just directories and files organized in a way that enables any node to locate any file using simple path hashing.

Originally developed by Gluster Inc. and now part of Red Hat, GlusterFS has found its niche in scenarios where simplicity, POSIX compliance, and linear scalability matter more than absolute performance optimization. Media streaming, NAS replacement, container persistent volumes, and backup targets are common use cases.

GlusterFS was designed with a set of guiding principles that prioritize simplicity and practicality over theoretical optimization. Understanding these principles explains many architectural decisions.

The key insight: GlusterFS treats each server's local file system as a building block called a brick. A volume is created by combining bricks from multiple servers using various strategies (distribution, replication, dispersion). The client-side translator stack determines how files map to bricks without consulting any central server.

This user-space approach means GlusterFS can run without kernel modifications, though kernel modules (FUSE or native) are used for mounting the file system.

GlusterFS architecture centers around three key concepts: bricks (storage units), volumes (logical aggregations), and translators (processing modules).

Brick Storage Format:

Unlike HDFS (which uses proprietary block format) or Ceph (custom OSD format), GlusterFS stores files as regular files on a standard Linux file system:

/data/brick1/
├── file1.txt           # Actual file
├── directory1/
│   └── file2.txt
└── .glusterfs/         # Metadata (extended attributes cache)
    └── indices/

This means you can browse brick contents with standard Unix tools, and in emergencies, access data directly from the underlying file system. This simplicity is a key operational advantage.

GlusterFS's power comes from its translator architecture. Every operation (open, read, write, stat) passes through a stack of translators, each adding specific functionality. This design enables features to be composed without monolithic code.

How Translators Work:

Each translator implements a set of file operation interfaces (called FOPs—file operations). A translator receives a request from the layer above, processes it, and either handles it locally or passes it to the layer below.

Key Translators:

GlusterFS supports multiple volume types, each with different characteristics for capacity, redundancy, and performance. The choice fundamentally affects how data is stored.

Volume Type Comparison:

The DHT (Distribute Hash Table) translator is the heart of GlusterFS's metadata-free architecture. It enables any client to determine a file's location without consulting a central server.

How DHT Works:

1. Each brick is assigned a hash range (e.g., 0-25%, 25-50%, etc.)
2. When accessing a file, DHT hashes the file's path
3. The hash determines which brick 'owns' the file
4. The client contacts that brick directly

Hash Layout:

DHT Directory Layout:

Each directory stores its hash layout in extended attributes. When you ls a directory, the client checks all bricks because files within a directory can be on any brick based on their individual path hashes.

Adding Bricks - Rebalancing:

When bricks are added, DHT layouts change. Existing files are in 'wrong' locations until rebalanced:

1. New brick gets a hash range (split from existing bricks)
2. Files in migrated ranges are copied to new brick
3. Hard links (via symlink-like mechanism) redirect until migration complete
4. Old copies deleted after successful migration

Lookup Optimization (Linkto Files):

During and after rebalancing, a file might be looked up on the 'wrong' brick. DHT handles this with linkto files—small marker files that point to the actual location. This adds one extra lookup but maintains consistency.

GlusterFS replicated and dispersed volumes include self-healing mechanisms to detect and repair inconsistencies. Understanding healing is crucial for operational reliability.

When Healing Is Needed:

Healing Mechanisms:

Automatic File Replication (AFR) Translator:

AFR tracks 'changelog' extended attributes on each replica. When replicas diverge (different changelog values), AFR determines which copy is authoritative based on which has more recent acknowledged writes.

Split-Brain Detection:

Split-brain occurs when both replicas believe they're the source of truth (both were written during a partition). AFR detects this condition and marks the file as requiring manual intervention rather than risking data loss by arbitrarily choosing one version.

GlusterFS provides multiple access methods to suit different deployment scenarios.

Native FUSE Mount:

The most common access method mounts GlusterFS as a POSIX file system:

# Mount using FUSE
mount -t glusterfs server1:/volume-name /mnt/gluster

# Or in /etc/fstab
server1:/volume-name /mnt/gluster glusterfs defaults,_netdev 0 0

libgfapi - High Performance:

For applications that can be modified (like QEMU/KVM), libgfapi provides direct integration without FUSE overhead:

// Application using libgfapi
glfs_t *fs = glfs_new("volume-name");
glfs_set_volfile_server(fs, "tcp", "server1", 24007);
glfs_init(fs);

glfs_fd_t *fd = glfs_creat(fs, "/file.txt", O_RDWR, 0644);
glfs_write(fd, data, size, 0);
glfs_close(fd);

QEMU uses libgfapi for VM disk images, avoiding double-caching and context switching, achieving near-native storage performance.

Operating GlusterFS in production requires attention to several key areas.

Performance Tuning:

Common Troubleshooting:

Choosing between distributed file systems requires understanding their strengths and ideal use cases.

We've explored GlusterFS's approach to distributed storage—using simple building blocks (bricks) and composable modules (translators) to create scale-out file systems. Let's consolidate the key concepts:

What's Next:

While GlusterFS provides general-purpose distributed file system capabilities, sometimes you need a lightweight, S3-compatible object store without the complexity of Ceph. In the next page, we'll explore MinIO, a high-performance, S3-compatible object storage system that's become the de facto standard for self-hosted object storage.