NewSQL Databases

For decades, database engineers faced a seemingly impossible choice: ACID transactions and SQL with traditional relational databases, or horizontal scalability with NoSQL systems. This dichotomy—often called the 'SQL vs NoSQL' divide—forced architects to sacrifice either consistency guarantees or the ability to scale across multiple machines.

Then came NewSQL databases.

NewSQL represents a paradigm shift in database engineering. These systems promise something that was previously considered architecturally impossible: the full transactional guarantees of traditional relational databases combined with the elastic scalability of NoSQL distributed systems. This isn't mere marketing—it's a genuine technological breakthrough built on decades of distributed systems research, novel consensus algorithms, and innovative storage architectures.

To truly understand NewSQL, we must first understand the technological forces and limitations that created the demand for a new database category. The emergence of NewSQL wasn't accidental—it was an inevitable response to the shortcomings of existing solutions.

The Traditional SQL Era (1970s-2000s)

For decades, relational database management systems (RDBMS) like Oracle, PostgreSQL, MySQL, and SQL Server dominated enterprise data management. These systems offered:

• ACID transactions: Atomicity, Consistency, Isolation, Durability guarantees that ensured data integrity
• SQL: A powerful, declarative query language with rich semantics
• Relational model: A mathematically rigorous foundation for data organization
• Mature tooling: Decades of optimization, monitoring, and administration tools

However, traditional RDBMS were designed for single-node operation. While they could scale vertically (adding more CPU, RAM, and faster disks to a single machine), they struggled with horizontal scaling (distributing data across multiple machines).

The NoSQL Revolution (2005-2015)

Faced with scaling limitations, internet-scale companies developed new database architectures that prioritized horizontal scalability:

• Google Bigtable (2006): Column-family storage for web indexing
• Amazon Dynamo (2007): Key-value store for shopping cart availability
• MongoDB (2009): Document database with flexible schemas
• Apache Cassandra (2008): Wide-column distributed database

These NoSQL systems achieved massive scale by relaxing consistency guarantees. They embraced the CAP theorem's trade-offs, often choosing availability and partition tolerance over strong consistency.

NoSQL databases typically offered:

• Horizontal scalability across commodity hardware
• Schema flexibility (no rigid table structures)
• Eventual consistency (data would eventually converge, but reads might return stale values)
• High availability through replication

The Gap That Created NewSQL

NoSQL solved the scalability problem but created new challenges:

• Loss of ACID semantics: Applications had to handle consistency at the application layer—error-prone and complex
• No standard query language: Each NoSQL system had its own API, increasing learning curves and lock-in
• Limited transaction support: Operations spanning multiple records required custom, fragile workarounds
• Join complexity: Denormalization or application-level joins increased data redundancy and complexity

Meanwhile, many applications—particularly in finance, healthcare, and enterprise systems—required strong consistency and couldn't tolerate the trade-offs NoSQL demanded.

The question became: Is it possible to have both ACID guarantees AND horizontal scalability?

The term NewSQL was coined by 451 Research analyst Matthew Aslett in 2011 to describe a new class of database systems that provide the scalability of NoSQL while maintaining the ACID guarantees and SQL interface of traditional relational databases.

Formal Definition:

NewSQL is a class of relational database management systems that seek to provide the same scalable performance of NoSQL systems for online transaction processing (OLTP) workloads while maintaining the ACID guarantees of a traditional database system.

This definition encompasses three essential characteristics:

What NewSQL Is NOT

To understand NewSQL precisely, it's equally important to clarify what it's not:

• Not just 'SQL on NoSQL': Some systems layer SQL interfaces over NoSQL storage. True NewSQL systems are designed from the ground up for distributed ACID transactions, not retrofitted.

• Not extended traditional RDBMS: Products like MySQL Cluster or PostgreSQL with Citus add distribution capabilities but often with limitations on cross-shard transactions or consistency guarantees.

• Not sacrificing consistency for scale: Unlike NoSQL, NewSQL maintains serializable isolation levels and strong consistency across all operations, even distributed ones.

• Not limited to specific workloads: While optimized for OLTP, NewSQL systems support a broad range of SQL operations, not just simple key-value lookups.

NewSQL wasn't possible until several key technological and theoretical breakthroughs matured. Understanding these foundations reveals why NewSQL emerged when it did—not earlier, not later.

1. Consensus Algorithms: Paxos and Raft

Distributed consensus—getting multiple machines to agree on a value even when some machines fail—is the foundation of distributed transactions. The Paxos algorithm (Leslie Lamport, 1989/1998) proved consensus was possible but was notoriously difficult to implement correctly.

The Raft consensus algorithm (Diego Ongaro, 2013) provided a more understandable alternative, accelerating the development of correct distributed systems. NewSQL databases use these algorithms to:

• Elect leaders for data partitions
• Replicate transaction logs across nodes
• Ensure all replicas agree on the order of transactions

2. Multi-Version Concurrency Control (MVCC)

MVCC allows readers to access consistent snapshots of data without blocking writers, and vice versa. This technique, refined over decades in traditional databases, became essential for NewSQL:

• Each transaction sees a consistent snapshot at its start time
• Writes create new versions rather than overwriting data
• Garbage collection removes old versions when no longer needed
• Enables serializable isolation without excessive locking

NewSQL systems extend MVCC to work across distributed nodes, maintaining consistent snapshots even when data spans multiple machines.

3. Hybrid Logical Clocks and TrueTime

To order events correctly across distributed nodes, NewSQL systems need accurate, synchronized timestamps. Two approaches emerged:

• Hybrid Logical Clocks (HLC): Combine physical timestamps with logical counters to ensure causal ordering without requiring perfect clock synchronization. Used by CockroachDB.

• TrueTime: Google's solution using GPS receivers and atomic clocks to provide globally synchronized time with bounded uncertainty (typically ±7ms). Used by Google Spanner.

These timing mechanisms enable consistent transaction ordering across geographically distributed datacenters.

4. Range-Based Sharding with Automatic Rebalancing

NewSQL databases partition data into ranges (contiguous key spans) rather than fixed hash-based sharding. Range-based partitioning enables:

• Efficient range scans for analytics and reporting
• Locality for related data (e.g., all orders for a customer)
• Automatic splitting when ranges grow too large
• Automatic rebalancing when nodes are added or removed

This dynamic partitioning, combined with Raft-based replication, allows NewSQL databases to automatically adjust to changing data volumes and cluster sizes.

While specific implementations vary, NewSQL databases share common architectural patterns that enable their distinctive capabilities. Understanding these patterns provides insight into how NewSQL achieves the 'impossible' combination of scale and consistency.

Layered Architecture

Most NewSQL systems use a layered architecture separating concerns:

Data Distribution Model

NewSQL databases partition the keyspace into ranges (also called regions, tablets, or shards). Each range:

• Contains a contiguous span of the keyspace (e.g., keys 'a' through 'm')
• Is replicated across multiple nodes using Raft consensus (typically 3 or 5 replicas)
• Has one leader that handles all reads and writes, with followers for redundancy
• Can be split automatically when it grows too large
• Can be moved between nodes for load balancing

This design means:

• Writes go to the range leader, are replicated via Raft, and committed when a majority acknowledges
• Reads can go to the leader (for linearizable reads) or followers (for slightly stale but load-balanced reads)
• Distributed transactions touching multiple ranges use two-phase commit coordinated across range leaders

A common misconception is that NewSQL 'solves' the CAP theorem. This is incorrect—the CAP theorem is a mathematical proof, not a limitation to be overcome. However, NewSQL makes different trade-offs than traditional NoSQL systems.

CAP Theorem Recap

The CAP theorem states that a distributed data store can provide at most two of three guarantees:

• Consistency: Every read receives the most recent write
• Availability: Every request receives a response (success or failure)
• Partition tolerance: The system continues operating despite network partitions

Since network partitions are inevitable in distributed systems, the real choice is between CP (consistency + partition tolerance) and AP (availability + partition tolerance) during a partition event.

NewSQL's Pragmatic Approach

NewSQL systems acknowledge that network partitions are rare and brief in modern data centers. They optimize for the common case (no partition) while maintaining correctness in the failure case (partition occurs):

1. During normal operation: Full ACID transactions, serializable isolation, high throughput
2. During partition: Operations involving the minority partition become unavailable; majority partition continues normally
3. After partition heals: Raft automatically reconciles any divergence; no application intervention needed

This design means NewSQL systems are almost always fully available (because partitions are rare) while always consistent (because they never sacrifice correctness).

To solidify understanding, let's directly compare NewSQL with the two paradigms it bridges:

Compared to Traditional SQL (PostgreSQL, MySQL, Oracle)

Compared to NoSQL (Cassandra, MongoDB, DynamoDB)

The NewSQL landscape includes several mature, production-ready systems. Here's a brief overview of the most significant ones (we'll explore Google Spanner and CockroachDB in detail in subsequent pages):

We've covered substantial ground in understanding what NewSQL databases are and why they represent a significant advancement in database technology. Let's consolidate the key concepts:

What's Next

With the foundational understanding of NewSQL established, the next pages will dive deeper into:

• SQL with Scalability: How NewSQL systems achieve scale-out architecture while maintaining SQL semantics
• Google Spanner: The pioneering system that proved global-scale ACID transactions were possible
• CockroachDB: An open-source Spanner-inspired system accessible to all organizations
• Use Cases: When NewSQL is the right choice—and when it isn't