Understanding Real-Time Systems

In real-time systems, latency is the currency by which all architectural decisions are evaluated. While conventional systems might accept latency variability in exchange for throughput or cost efficiency, real-time systems treat latency as a hard constraint that shapes every design choice.

But what does "low latency" actually mean? The answer depends entirely on context:

• A 1-second response is unacceptably slow for a video game, but impressively fast for a batch data pipeline
• 100 milliseconds feels instant for a web page load, but introduces noticeable lag in a phone conversation
• 10 milliseconds is imperceptible for human interaction, but far too slow for a motor control loop
• 1 microsecond is achievable in specialized hardware, but impossible for general-purpose software

This page establishes a rigorous framework for understanding latency expectations: how they vary across domains, how they're measured, and how system architects budget latency across components to meet end-to-end requirements.

Different application domains operate at vastly different latency scales. Understanding where your system falls on this spectrum is the first step in setting appropriate expectations.

The latency hierarchy:

Physical constraints at the extremes:

At the lowest latencies, physics becomes the limiting factor:

• Speed of light: Light travels approximately 30cm per nanosecond. A round-trip across a 10-meter data center network cable takes at least 67 nanoseconds—just for photons to travel, ignoring all processing.

• Memory access hierarchy: L1 cache hit: ~1ns. L2 cache: ~4ns. L3 cache: ~15ns. Main memory: ~100ns. SSD: ~100μs. HDD: ~10ms. Each level represents roughly an order of magnitude increase.

• CPU clock cycles: At 4GHz, each cycle is 0.25 nanoseconds. Even a "simple" operation requiring 100 cycles takes 25 nanoseconds.

These physical realities create hard floors below which no software optimization can push latency. High-frequency trading firms spend billions locating their servers meters closer to exchanges because at their scale, nanoseconds matter.

For systems that interact with humans, human perception sets the latency targets. Decades of human-computer interaction research have established clear thresholds that should guide your requirements.

The foundational research:

Jakob Nielsen's "Response Time Limits" (derived from Miller's and Card's earlier research) established three fundamental thresholds that remain relevant today:

Domain-specific perception thresholds:

Different human senses have different latency sensitivity:

The conversation threshold:

For real-time communication systems (video calls, VoIP, gaming voice chat), the ITU-T G.114 standard establishes critical thresholds:

• < 150ms one-way delay: Excellent quality, natural conversation flow
• 150-300ms: Acceptable for most users, slight awkwardness in turn-taking
• 300-450ms: Poor quality, noticeable delay, talkers interrupt each other
• > 450ms: Unacceptable for interactive conversation

These thresholds explain why satellite phone calls feel unnatural (600ms+ round trip due to geostationary orbit distance) despite high audio quality.

Accurate latency measurement is surprisingly difficult. Many teams measure latency incorrectly, leading to false confidence in their systems' performance characteristics.

What to measure:

End-to-end latency is the gold standard—the time from when a user initiates an action to when they perceive the result. This includes:

• Client-side processing time
• Network transmission (both directions)
• Server queuing and processing
• Response generation and serialization
• Client-side rendering

Measuring only server-side processing time can dramatically underestimate actual user-perceived latency.

Coordinated Omission:

One of the most insidious measurement errors is "coordinated omission," identified by Gil Tene. This occurs when a load generator waits for a response before sending the next request. If responses are delayed, the load generator sends fewer requests, hiding the true queuing delays.

Example:

• System normally responds in 1ms
• Under load, responses take 100ms
• With coordinated omission, you measure ~100ms latency
• Actual user experience: 100ms + queue wait (potentially seconds)

Tools that "fire and forget" requests on a fixed schedule (regardless of responses) expose the true latency including queue wait times.

Latency is not a single number—it's a distribution. Understanding this distribution is essential for real-time system design because the tail of the distribution often determines whether you meet your requirements.

Why averages lie:

Consider two systems:

• System A: Every request takes exactly 50ms. Average: 50ms.
• System B: 99% of requests take 10ms, 1% take 4010ms. Average: 50ms.

Both have the same average, but System B delivers unusable experience for 1% of users. At 1 million requests per day, that's 10,000 terrible experiences daily.

Percentile thinking:

Real-time requirements should be specified using percentiles:

The amplification problem:

In distributed systems, latency compounds across service calls. If your request touches 10 backend services in parallel, the end-to-end latency is the maximum of the 10 individual latencies.

Mathematical example:

• Each service has p99 latency of 10ms
• Single call: 99% of requests complete in ≤10ms
• With 10 parallel calls: 0.99^10 = 0.904
• Only 90.4% of requests will have all 10 complete in ≤10ms
• Your end-to-end p99 requires looking at p99.9 or higher of individual services

This is why systems with many dependencies struggle with tail latency—the "long tail" of each dependency contributes to a very long tail at the system level.

For many real-time applications, consistency matters as much as speed. A system with steady 50ms latency often provides better user experience than one oscillating between 10ms and 100ms, even though the latter has better average latency.

Defining jitter:

Jitter is the variation in latency over time. Formally, it's often measured as:

• Peak-to-peak jitter: Maximum latency minus minimum latency
• Standard deviation: Statistical measure of variation around the mean
• Inter-packet jitter (for streaming): Variation in delay between consecutive packets

Why jitter matters:

Jitter buffering:

The classic solution to jitter is buffering—accumulate some data before processing/playback to smooth out variations. But buffering adds latency:

Effective latency = Transmission latency + Buffer size

There's a fundamental tradeoff:

• Large buffers: Smooth playback but high latency (problematic for interactive applications)
• Small buffers: Low latency but vulnerable to jitter (gaps, stuttering)

Adaptive jitter buffers dynamically adjust based on observed jitter, but they can't eliminate the underlying constraint.

Sources of jitter:

A latency budget divides an end-to-end latency requirement into allocations for each component of the system. This is the foundational artifact of real-time system design.

The budgeting process:

Example: E-commerce search latency budget

Target: p99 search results < 200ms from keypress to results displayed

Latency requirements are formalized through Service Level Indicators (SLIs) and Service Level Objectives (SLOs), which provide measurable, enforceable targets.

Terminology:

• SLI (Service Level Indicator): A quantitative measure of some aspect of service quality. For latency: "The proportion of requests that complete within X milliseconds."

• SLO (Service Level Objective): A target value or range for an SLI. For latency: "99% of requests will complete within 100ms."

• SLA (Service Level Agreement): A contract specifying what happens when SLOs are not met (usually external, with financial consequences).

Defining latency SLOs:

Best practices for latency SLOs:

We've established a comprehensive framework for understanding, measuring, and managing latency in real-time systems. Let's consolidate the key concepts:

What's next:

With latency expectations established, the next page explores the critical distinction between soft real-time and hard real-time systems. This classification determines the appropriate architecture, implementation technologies, and failure handling strategies for your real-time application.