Distributed File Systems

Amazon S3 has become the de facto standard API for object storage. Countless applications, frameworks, and tools are built assuming S3 compatibility. But what if you need to run object storage on-premises? What if regulatory requirements prevent cloud usage? What if you simply want to avoid cloud vendor lock-in?

MinIO answers these questions with radical simplicity. Instead of building a complex distributed system with multiple component types (like Ceph), MinIO focuses on doing one thing exceptionally well: providing high-performance S3-compatible object storage that's easy to deploy and operate.

Launched in 2014 and written in Go, MinIO has become the most popular self-hosted S3-compatible object storage, with over 44,000 GitHub stars and adoption by companies like Adobe, PayPal, and Siemens. Its appeal lies in its simplicity: a single binary that can run standalone on a laptop for development or distributed across hundreds of nodes for enterprise scale.

MinIO was built with a contrarian philosophy: distributed storage doesn't need to be complicated. While systems like Ceph and HDFS introduce multiple specialized components, MinIO takes a minimalist approach.

The key insight: MinIO recognized that most distributed storage complexity exists to handle edge cases and provide features that many workloads don't need. By focusing on object storage (not file or block) and accepting certain constraints (immutable objects, erasure coding only), MinIO achieves remarkable simplicity.

Comparison of Architecture Complexity:

MinIO's architecture is elegantly simple compared to other distributed storage systems. A MinIO deployment consists of MinIO server processes, each managing local drives, with all nodes being equal peers (no master/slave distinction).

Key Architectural Concepts:

Deployment Modes:

1. Standalone (single-node single-drive) — For development and testing. No redundancy.

2. Standalone erasure code (single-node multi-drive) — One node with 4+ drives. Erasure coding within the node.

3. Distributed (multi-node) — Multiple nodes, each with multiple drives. Erasure coding spans nodes for node-level fault tolerance.

# Single node, 4 drives
minio server /data1 /data2 /data3 /data4

# Distributed, 4 nodes × 4 drives each
minio server http://node{1...4}/data{1...4}

The distributed syntax ({1...4}) is MinIO-specific shorthand for specifying multiple nodes and drives.

MinIO uses erasure coding (not replication) as its primary durability mechanism. This provides space efficiency comparable to RAID while tolerating multiple drive failures.

How Erasure Coding Works in MinIO:

When an object is written, MinIO divides it into data shards and computes parity shards using Reed-Solomon encoding:

• Data shards — Original data split into pieces
• Parity shards — Computed redundancy that allows reconstruction
• Erasure set — The group of drives across which shards are distributed

Erasure Set Sizing:

The number of drives per erasure set determines fault tolerance and storage efficiency:

MinIO automatically calculates erasure set size based on total drives. For example, 16 drives across 4 nodes creates one erasure set of 16 drives.

Inline vs. Multi-Part:

Small objects (default <128KB) are written inline—all shards in a single write. Larger objects use multi-part upload where each part is independently striped.

MinIO stores objects in a straightforward directory structure on each drive. Understanding this format helps with troubleshooting and recovery.

On-Disk Layout:

xl.meta File:

The xl.meta file contains object metadata in JSON format:

{
  "version": "1.0.0",
  "checksum": {
    "algorithm": "highwayhash256S",
    "sum": "..."
  },
  "erasure": {
    "algorithm": "ReedSolomon",
    "data": 4,
    "parity": 4
  },
  "meta": {
    "content-type": "image/jpeg",
    "x-amz-meta-custom": "value"
  },
  "parts": [...]
}

Key Points:

• No separate metadata store — Metadata lives with data on each drive
• Self-describing — Each shard knows its position and the overall encoding scheme
• Human-readable — You can inspect metadata with standard tools
• Checksums everywhere — Both metadata and data are checksummed for integrity

MinIO's primary interface is the S3 API. Compatibility is extensive—most S3 SDKs and tools work without modification.

Using Standard S3 Tools:

# AWS CLI with MinIO
aws --endpoint-url http://minio:9000 s3 ls
aws --endpoint-url http://minio:9000 s3 cp file.txt s3://bucket/

# Python boto3
import boto3
s3 = boto3.client('s3',
    endpoint_url='http://minio:9000',
    aws_access_key_id='minioadmin',
    aws_secret_access_key='minioadmin'
)
s3.upload_file('file.txt', 'bucket', 'file.txt')

# mc (MinIO Client - specialized tool)
mc alias set myminio http://minio:9000 minioadmin minioadmin
mc cp file.txt myminio/bucket/

MinIO Client (mc):

While standard S3 tools work, MinIO provides mc—a feature-rich CLI for advanced operations:

MinIO provides full S3-compatible versioning and object lock capabilities, essential for regulatory compliance and data protection.

Bucket Versioning:

When enabled, every write creates a new version instead of overwriting:

Object Lock (WORM):

Object Lock prevents objects from being deleted or overwritten, even by administrators:

Use Cases:

• Compliance mode: Regulated industries (SEC 17a-4, HIPAA) requiring immutable records
• Governance mode: Protection against accidental deletion with administrative override
• Legal hold: Indefinite hold regardless of retention period

MinIO supports bucket-level replication between clusters for disaster recovery, geo-distribution, and data synchronization.

Replication Modes:

Configuring Bucket Replication:

# Set up remote target
mc admin bucket remote add myminio/source-bucket 
  http://accesskey:secretkey@remote-minio:9000/target-bucket 
  --service replication

# Enable replication
mc replicate add myminio/source-bucket 
  --remote-bucket target-bucket 
  --replicate "delete,delete-marker,existing-objects"

# Check replication status
mc replicate status myminio/source-bucket

What Gets Replicated:

• Object data and metadata
• Object versions (with versioning enabled)
• Object tags
• Object lock / retention settings
• Delete markers (optional)
• Existing objects (optional, requires batch replication)

MinIO is designed for high performance on modern hardware. Understanding its performance characteristics helps optimize deployments.

Benchmark Results (MinIO published):

Results from 32-node cluster with NVMe drives and 100GbE networking

Performance Factors:

Small Object Optimization:

Small objects (<128KB by default) are handled specially:

• Written inline to reduce I/O operations
• Metadata and data in single file
• Less overhead from erasure coding per object

Performance Tuning:

# Increase GOMAXPROCS for more CPU utilization
export GOMAXPROCS=
minio server ...

# Use multiple drives with high queue depth
minio server /mnt/disk{1...12}

# Enable drive caching for metadata-heavy workloads
minio server --config-dir=/config --certs-dir=/certs /data{1...4}

MinIO is built for cloud-native environments, with first-class Kubernetes support through the MinIO Operator.

MinIO Operator:

The operator manages MinIO tenants (clusters) on Kubernetes:

apiVersion: minio.min.io/v2
kind: Tenant
metadata:
  name: minio-tenant
  namespace: minio
spec:
  image: minio/minio:latest
  pools:
    - servers: 4
      volumesPerServer: 4
      volumeClaimTemplate:
        spec:
          storageClassName: local-nvme
          resources:
            requests:
              storage: 1Ti
  requestAutoCert: true
  s3:
    bucketDNS: true

Key Kubernetes Features:

We've explored MinIO's approach to self-hosted object storage—achieving S3 compatibility and high performance through radical simplicity. Let's consolidate the key concepts:

What's Next:

We've now explored four major distributed storage systems: HDFS, Ceph, GlusterFS, and MinIO. Each has distinct strengths and ideal use cases. In the final page of this module, we'll synthesize this knowledge with a framework for choosing distributed storage—helping you select the right system for your specific requirements.