Key-Value Stores

If you could only add one technology to your stack beyond your primary database, a strong argument could be made for Redis. Created by Salvatore Sanfilippo in 2009, Redis (Remote Dictionary Server) has become the de facto standard for caching, session management, real-time analytics, messaging, and dozens of other use cases that require sub-millisecond latency and high throughput.

What sets Redis apart from other key-value stores is its rich set of native data structures. While Memcached treats values as opaque blobs, Redis understands strings, lists, sets, sorted sets, hashes, bitmaps, hyperloglogs, and streams. This structural awareness enables sophisticated operations—atomic list pushes, set intersections, sorted set range queries—that would require complex application logic with a simpler store.

Redis is fast. Extraordinarily fast. Single-threaded by design (for command processing), it can handle hundreds of thousands of operations per second on commodity hardware. And with Redis Cluster, it scales horizontally to handle millions of operations per second across terabytes of data.

Redis's performance stems from deliberate architectural choices that prioritize simplicity and speed. Understanding these choices is essential for using Redis effectively.

In-memory data store:

Redis keeps all data in RAM. This is its most fundamental characteristic:

• Read performance: No disk I/O for reads. Data is already in memory.
• Write performance: Writes update memory first, optionally persisted to disk asynchronously.
• Capacity constraint: Dataset must fit in memory (plus overhead).
• Cost implications: RAM is expensive; Redis for multi-TB datasets is costly.

Single-threaded command processing:

Redis processes commands using a single thread. This sounds limiting but is actually a key performance optimization:

But isn't single-threaded slow?

Not when operations are fast. A single Redis thread can execute 500,000+ simple operations per second because:

1. All data is in memory (no I/O wait)
2. Operations are O(1) or O(log N) (efficient algorithms)
3. Network I/O is multiplexed (one thread handles many connections)
4. No locking overhead (no contention delays)

Redis 6.0+ threading:

Modern Redis uses multiple threads for I/O (network reads/writes) while maintaining single-threaded command processing. This improves throughput for I/O-bound workloads without sacrificing the simplicity of single-threaded execution.

┌─────────────────────────────────────────────────────────────┐
│                    Redis Server                             │
├─────────────────────────────────────────────────────────────┤
│  I/O Threads (multiple)     │   Main Thread (single)       │
│  ─────────────────────────  │   ────────────────────────   │
│  • Read client requests     │   • Parse commands           │
│  • Write responses          │   • Execute commands         │
│  • Network multiplexing     │   • Update data structures   │
│                             │   • Trigger persistence      │
└─────────────────────────────────────────────────────────────┘

Redis strings are the most basic and most used data type. Despite the name, they're not just text—they're binary-safe sequences of bytes that can hold any data up to 512MB.

What strings can store:

• Text strings (user names, serialized JSON)
• Numbers (counters, timestamps)
• Binary data (images, serialized objects)
• Bit arrays (feature flags, bloom filters)

Core string operations:

Pattern: Atomic counters

# Page view counter - atomically increment
INCR page:views:home
INCR page:views:home
GET page:views:home  → "2"

# Rate limiting - increment and check in one operation
MULTI
INCR ratelimit:api:user:123
EXPIRE ratelimit:api:user:123 60
EXEC

Pattern: Distributed locking

# Acquire lock with protection against deadlocks
SET lock:order-processing "server-1" NX EX 30
# NX = only if not exists
# EX 30 = expires in 30 seconds

# If SET returns OK, lock acquired
# If SET returns nil, lock held by another

# Release lock (only if we own it - use Lua for atomicity)
EVAL "if redis.call('get',KEYS[1]) == ARGV[1] then return redis.call('del',KEYS[1]) else return 0 end" 1 lock:order-processing server-1

Pattern: Caching with TTL

# Cache API response for 5 minutes
SETEX api:response:products:category:electronics 300 '{"products": [...]}'

# Check cache, fall back to database if miss
GET api:response:products:category:electronics

Redis's collection types—Lists, Sets, and Sorted Sets—transform it from a simple cache into a powerful data processing platform. Each serves distinct use cases with specialized operations.

Lists: Ordered sequences

Redis lists are linked lists of strings. They support efficient insertion and removal at both ends, making them ideal for queues, stacks, and recent-item lists.

List patterns:

# Recent notifications (capped list)
LPUSH notifications:user:123 '{"type":"like","from":456}'
LTRIM notifications:user:123 0 99  # Keep only last 100
LRANGE notifications:user:123 0 9   # Get 10 most recent

# Job queue (producer-consumer)
RPUSH queue:emails '{"to":"a@b.com","subject":"Hello"}'
BLPOP queue:emails 30  # Worker blocks until job available

Sets: Unique unordered collections

Redis sets are unordered collections of unique strings. They enable membership tests, deduplication, and powerful set operations (union, intersection, difference).

Set patterns:

# Track unique visitors
SADD visitors:2024-01-15 "user:123"
SADD visitors:2024-01-15 "user:456"
SCARD visitors:2024-01-15  → 2 (unique count)

# Common friends
SINTER friends:user:123 friends:user:456  → common friends

# Tagging system
SADD product:123:tags "electronics" "sale" "featured"
SMEMBERS product:123:tags  → all tags
SINTER tag:electronics tag:sale  → products in both categories

Sorted Sets: Ordered unique collections with scores

Sorted sets combine set uniqueness with ordering by score. Each member has an associated floating-point score that determines its position. This enables leaderboards, time-series data, and priority queues.

Sorted set patterns:

# Real-time leaderboard
ZADD leaderboard:game:chess 2500 "player:magnus"
ZADD leaderboard:game:chess 2400 "player:hikaru"
ZINCRBY leaderboard:game:chess 10 "player:hikaru"  # Win increases rating
ZREVRANK leaderboard:game:chess "player:hikaru"    # Get ranking (0 = top)
ZREVRANGE leaderboard:game:chess 0 9 WITHSCORES   # Top 10 with scores

# Rate limiting with sliding window
local_time = 1705320000
ZADD ratelimit:user:123 <timestamp> <request_id>
ZREMRANGEBYSCORE ratelimit:user:123 0 <timestamp-60>  # Remove old
ZCARD ratelimit:user:123  # Count in window

Beyond lists and sets, Redis offers hashes for structured objects and specialized types for specific use cases.

Hashes: Field-value maps within a key

Redis hashes are perfect for representing objects with multiple fields. Instead of serializing an entire object as a JSON string, you store individual fields that can be read and updated independently.

Hash patterns:

# User profile with independent fields
HSET user:123 name "Alice" email "alice@example.com" balance 1000
HGET user:123 name         → "Alice"
HINCRBY user:123 balance 50  # Add $50 without reading whole object
HGETALL user:123           → {name: "Alice", email: "...", balance: "1050"}

# Shopping cart
HSET cart:user:123 prod:456 2 prod:789 1  # productId -> quantity
HINCRBY cart:user:123 prod:456 1          # Add one more of product 456
HGETALL cart:user:123                      # Get entire cart

Hash vs String (JSON):

Advanced data types:

Bitmaps (Bit arrays):

Redis strings can be manipulated as bit arrays. Extremely memory-efficient for tracking boolean states across millions of entities.

# User login tracking (1 bit per user)
SETBIT logins:2024-01-15 123 1    # User 123 logged in
SETBIT logins:2024-01-15 456 1    # User 456 logged in
GETBIT logins:2024-01-15 123      → 1 (did log in)
BITCOUNT logins:2024-01-15        → 2 (total logins)

# Users active on both days
BITOP AND active:both logins:2024-01-15 logins:2024-01-16

HyperLogLog (Cardinality estimation):

Estimate unique counts with ~0.81% error using only 12KB regardless of cardinality.

# Count unique visitors (millions) with constant memory
PFADD visitors:2024-01 "user:123" "user:456" "user:123"  # Duplicates ignored
PFCOUNT visitors:2024-01  → 2 (approximate unique count)

# Merge months for quarterly uniques
PFMERGE visitors:Q1-2024 visitors:2024-01 visitors:2024-02 visitors:2024-03

Streams (Append-only log):

Redis Streams provide a persistent, consumer-group-aware message log—Redis's answer to Kafka.

# Append events to stream
XADD events:orders * action create orderId 12345 amount 99.99
XADD events:orders * action payment orderId 12345 status success

# Read events (consumer group)
XREADGROUP GROUP processors worker1 STREAMS events:orders >

Redis is an in-memory database, but it offers two persistence mechanisms to survive restarts: RDB (Redis Database Backup) snapshots and AOF (Append-Only File) logs. Understanding when to use each—or both—is crucial for production deployments.

RDB: Point-in-Time Snapshots

RDB persistence saves the entire dataset as a compact binary file at specified intervals. Think of it as taking a photograph of your data at a moment in time.

How RDB works:

1. Redis forks the main process
2. Child process has copy-on-write access to memory
3. Child writes all data to temporary RDB file
4. Child atomically replaces old RDB file
5. Child exits; parent continues serving requests

Fork behavior (critical for production):

The fork() system call creates a child process with a copy of parent's memory. On Linux, this uses copy-on-write: memory pages are shared until modified. This means:

• Fork is fast (no actual copy initially)
• Memory used increases as parent modifies data during snapshot
• Peak memory can be 2x dataset size during heavy writes
• Fork on large datasets (>10GB) can cause brief latency spike

AOF: Write-Ahead Logging

AOF persistence logs every write operation to a file. On restart, Redis replays the log to reconstruct the dataset. This provides much better durability than RDB at the cost of larger files and slower restarts.

How AOF works:

1. Every write command is appended to AOF file
2. File is synced to disk according to policy
3. On restart, Redis replays commands from AOF
4. Periodic rewriting compacts the file

Sync policies (the durability/performance trade-off):

AOF Rewriting:

The AOF file grows continuously as operations are appended. Rewriting creates a minimal AOF that represents the current dataset:

# Original AOF (verbose)
SET key 1
INCR key
INCR key
INCR key
DEL key
SET key 100

# After rewrite (minimal)
SET key 100

Rewriting happens in a background process (similar to RDB snapshots) and atomically replaces the old AOF.

Configuration:

appendonly yes
appendfsync everysec
auto-aof-rewrite-percentage 100    # Rewrite when AOF doubles since last rewrite
auto-aof-rewrite-min-size 64mb     # Don't rewrite if AOF is tiny

Hybrid persistence (Redis 4.0+):

Redis can combine RDB and AOF: the AOF file starts with an RDB snapshot, followed by incremental AOF entries. This gives fast restarts (RDB) with excellent durability (AOF).

aof-use-rdb-preamble yes

Choosing a persistence strategy:

Since Redis stores everything in memory, understanding memory behavior is essential for production operations. Running out of memory is catastrophic; Redis will stop accepting writes or crash.

Memory usage components:

• Dataset: Your actual data (keys, values, data structures)
• Overhead: Internal bookkeeping, per-key metadata (~40 bytes/key)
• Client buffers: Memory for client connections and pending responses
• Replication buffers: Data queued for replicas
• AOF/RDB: Memory used during persistence operations

Monitoring memory:

INFO memory
# used_memory: 1234567           # Total allocated by Redis
# used_memory_human: 1.18M
# used_memory_rss: 2345678       # Memory from OS perspective
# used_memory_peak: 3456789      # Historical maximum
# maxmemory: 1073741824          # Configured limit
# maxmemory_policy: allkeys-lru   # Eviction policy

Eviction policies:

When maxmemory is reached, Redis must decide what to do. The eviction policy controls this:

LRU vs LFU:

• LRU (Least Recently Used): Evicts keys not accessed recently. Good when recent access predicts future access.
• LFU (Least Frequently Used): Evicts keys accessed least often overall. Better when some keys are consistently hot.

Memory optimization techniques:

We've explored Redis's architecture, data structures, and persistence—the foundation for using Redis effectively in production. Let's consolidate the key concepts:

What's next:

With Redis's architecture and features understood, we'll explore Memcached—the simpler, specialized caching system that influenced Redis's design. Understanding Memcached helps you choose between them and appreciate Redis's additional capabilities.