Distributed Tracing: Following Requests Across Services

Picture this scenario: Your e-commerce platform is experiencing slowdowns. Users are complaining that checkout takes 15 seconds instead of the usual 2 seconds. You have 47 microservices, 12 databases, 5 message queues, and 3 external API dependencies. The million-dollar question: Where is the time going?

You check your metrics dashboard—CPU and memory look normal across all services. You scan your logs—nothing obviously wrong in any individual service. Your load balancer metrics show requests are balanced. Yet users continue to suffer.

This is the debugging nightmare that distributed tracing was designed to solve.

In a monolithic application, debugging a slow request is straightforward. You add some timing logs, reproduce the issue, and examine a single log file to find the bottleneck. All the code runs in one process, on one machine, writing to one log.

Microservices shatter this simplicity.

A single user request now becomes a cascade of inter-service communications. That checkout request might touch:

Each of these services writes its own logs, emits its own metrics, and runs on potentially different machines. When the checkout is slow, how do you correlate what happened?

Without tracing, you're left with fragments:

• Cart Service logged that it completed in 50ms
• Pricing Service logged that it completed in 30ms
• Payment Service logged that it completed in 100ms
• But the user experienced 15 seconds of latency

The individual pieces look fine. The problem is somewhere in the gaps—the time between services, the retries you didn't know about, the database connection that timed out and failed over, the cache miss that forced a cold read.

Distributed tracing gives requests an identity. It stitches together the fragmented story across all these services into a single, unified timeline that you can visualize, analyze, and debug.

Metrics, logs, and traces are the three pillars of observability—each serving a distinct purpose. To understand why tracing is essential, we must understand what the other pillars cannot do.

The specific gaps that tracing fills:

1. Understanding Latency Distribution

Metrics can tell you that your P99 latency is 2 seconds. But they cannot tell you where in the request that 2 seconds is spent. Is it the database? A downstream service? Network latency? The client? A trace shows you a waterfall view: 50ms in Service A, 100ms waiting for Service B, 1800ms in the database query, 50ms in serialization.

2. Detecting Intermittent Issues

Imagine 1% of requests are slow due to a specific code path triggered by certain user attributes. Aggregate metrics might not flag this—the P50 and P95 look fine. Logs from each service individually look normal. Only by tracing slow requests can you discover that they all share a common path: they hit the legacy discount calculation service that queries an unindexed table.

3. Understanding System Topology

Which services depend on which? As systems evolve, documentation becomes stale. Traces are generated from actual traffic, providing a living, accurate dependency graph. You discover that the Report Service unexpectedly calls the User Service 47 times per request—something no one documented.

4. Debugging Distributed Transactions

When a saga fails halfway through, which services completed their steps? What compensating transactions ran? Traces capture the full choreography, showing exactly what happened and where it broke.

Let's examine concrete scenarios where tracing transforms debugging from impossible to straightforward.

Case Study: The Phantom Retry Storm

A major fintech company experienced periodic latency spikes. Their metrics showed brief CPU spikes, but nothing actionable. Engineers spent weeks investigating before implementing tracing.

The traces immediately revealed the issue: a flaky downstream service was causing the payment service to retry. Each retry was itself triggering retries in its downstream dependencies. A single slow response cascaded into 27 downstream calls instead of the expected 3.

Without tracing, this retry amplification was invisible—each service's metrics showed 'normal' retry rates. Only the end-to-end trace view exposed the multiplicative effect.

Resolution time without tracing: 6 weeks of investigation Resolution time with tracing: 2 hours after deployment

One reason engineers underestimate the need for tracing is that they underestimate the complexity of actual request flows. Let's make this concrete.

What you might expect:

User → API Gateway → Application Service → Database → Response

What actually happens in production:

In this realistic scenario, the request touches:

• 3+ network hops before reaching application code
• 5+ external data stores (Redis, Memcached, primary DB, replica DB, Consul)
• 2+ downstream services, each with their own complexity
• 10+ internal subsystems (rate limiter, auth, circuit breaker, connection pools)

Any of these can be the bottleneck. Many of them can be slow intermittently—a database connection pool that's normally instant but occasionally waits 500ms under load, a circuit breaker that opened briefly, a DNS resolution that had to retry.

Tracing captures all of this. Each step becomes a span with precise timing. The spans are linked together as a trace, showing the complete journey of the request. What took hours of manual log correlation now takes seconds of visual inspection.

Distributed tracing isn't just about debugging production issues—it's infrastructure that enables a suite of advanced engineering capabilities.

Tracing does have costs—instrumentation effort, storage, and operational overhead. But the cost of not tracing in a distributed system is far greater.

The Inflection Point

Tracing becomes essential at the following inflection points:

• More than 5-10 services: The cognitive load of holding multiple service interactions in your head becomes unsustainable
• Multiple teams owning different services: Cross-team debugging without tracing devolves into endless Slack threads and blame games
• SLA/SLO requirements: When you need to prove why an SLO was missed and who is accountable, traces provide evidence
• Asynchronous processing: Message queues, event-driven architectures, and background jobs make request flows non-linear and traces essential

If your system hasn't hit these inflection points yet, it will. Organizations that implement tracing before they desperately need it have a significant advantage over those that scramble to implement it during an incident.

Distributed tracing represents a fundamental paradigm shift in how we debug software. Understanding this shift is crucial for adopting tracing effectively.

The Mental Model Shift

In monolithic systems, you think: "I'll add a breakpoint here and step through the code."

In distributed systems, you think: "I'll look at the trace to understand which service was executing what, when, and for how long."

This shift requires new skills:

• Reading trace waterfalls to understand timing and parallelism
• Understanding how trace context flows through synchronous calls, queues, and async processing
• Recognizing patterns in traces that indicate common problems (retry storms, fan-out explosions, cold start latency)
• Using trace data to drive hypotheses that you can then verify with logs and metrics

Tracing is not a replacement for debugging skills—it's an amplifier. The engineer who understands distributed systems and knows how to read traces can solve problems that would stump an engineer limited to traditional debugging tools.

We've established the foundational case for distributed tracing. Let's consolidate what we've learned:

What's Next:

Now that we understand why tracing matters, we need to understand how it works. The next page dives into the fundamental building blocks of distributed tracing: traces and spans. We'll explore how these abstractions model distributed request flow, the anatomy of a span, parent-child relationships, and the data model that makes tracing possible.