Sampling Strategies

HEAD-BASED SAMPLING
═══════════════════
 
Request arrives at edge:
 │
 ├─ Sample decision: random(0.0-1.0) < 0.10 (10% rate)
 │   │
 │   ├─ TRUE (sampled = 1):
 │   │   • All downstream services create and export spans
 │   │   • Complete trace stored
 │   │   
 │   └─ FALSE (sampled = 0):
 │       • All downstream services skip span creation entirely
 │       • No trace data generated
 │       • Zero overhead beyond decision propagation
 
Risk: If this request later errors, we have no trace!
 
 
TAIL-BASED SAMPLING
═══════════════════
 
Request arrives and progresses:
 │
 ├─ Service A creates span → buffered
 ├─ Service B creates span → buffered
 ├─ Service C creates span → buffered
 ├─ Service D creates span → buffered
 │
 └─ Request completes, all spans arrive at collector
     │
     └─ Collector evaluates complete trace:
         │
         ├─ Has error spans? → KEEP (100%)
         ├─ Latency > P99? → KEEP (100%)
         ├─ Important user tier? → KEEP (100%)
         ├─ Otherwise → KEEP (1%)
         │
         └─ Only kept traces written to storage
 
Benefit: We ALWAYS have traces for errors and slow requests!
Cost: All spans buffered until trace completes (memory, processing)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

// How probabilistic sampling works internally
 
function shouldSample(traceId: string, samplingRate: number): boolean {
  // Use trace ID to make deterministic decision
  // This ensures all services agree on the decision for the same trace
  const hash = hashTraceId(traceId);  // Returns 0.0 - 1.0
  return hash < samplingRate;
}
 
// Example: 10% sampling rate
const SAMPLING_RATE = 0.10;
 
function onTraceStart(traceId: string): SamplingDecision {
  if (shouldSample(traceId, SAMPLING_RATE)) {
    return {
      decision: 'RECORD_AND_SAMPLE',  // Record and export
      attributes: {
        'sampling.probability': SAMPLING_RATE,
      },
    };
  }
  return {
    decision: 'DROP',  // Don't record
  };
}
 
// Why hash the trace ID?
// - Trace ID is generated before sampling decision
// - Hash function produces deterministic output for same input
// - All services with same trace ID get same decision
// - No coordination needed between services

Traffic Level:  |  100 req/s  |  1,000 req/s  |  10,000 req/s  |
════════════════════════════════════════════════════════════════
                                                              
Probabilistic 10% sampling:                                   
  Sampled:      |  10/s       |  100/s        |  1,000/s       |
  Problem: Sampling rate constant, but VOLUME scales with traffic
           During traffic spikes, you still get 10x more data
                                                              
Rate-Limiting at 50 traces/s:                                 
  Sampled:      |  50/s       |  50/s         |  50/s          |
  Sampled %:    |  50%        |  5%           |  0.5%          |
  Benefit: Predictable output regardless of input traffic
           Storage costs are bounded and predictable
                                                              
                   Normal        Spike          Mega-spike     
                   Traffic       Traffic        Traffic        
                     │             │               │           
                     ▼             ▼               ▼           
Probabilistic:  ▓▓▓▓▓▓▓▓▓▓   ▓▓▓▓▓▓▓▓▓▓▓▓▓   ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓
                (scales up)                  (keeps scaling!)  
                                                              
Rate-Limited:   ▓▓▓▓▓▓▓▓▓▓   ▓▓▓▓▓▓▓▓▓▓     ▓▓▓▓▓▓▓▓▓▓        
                (capped)      (still capped) (still capped!)   

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

// Token bucket rate limiter for sampling
class RateLimitingSampler {
  private tokens: number;
  private readonly maxTokens: number;
  private readonly refillRate: number;  // tokens per second
  private lastRefill: number;
  
  constructor(tracesPerSecond: number) {
    this.maxTokens = tracesPerSecond;
    this.tokens = tracesPerSecond;
    this.refillRate = tracesPerSecond;
    this.lastRefill = Date.now();
  }
  
  shouldSample(traceId: string): boolean {
    this.refillTokens();
    
    if (this.tokens >= 1) {
      this.tokens -= 1;
      return true;  // Sample this trace
    }
    return false;  // Drop this trace
  }
  
  private refillTokens(): void {
    const now = Date.now();
    const elapsed = (now - this.lastRefill) / 1000;  // seconds
    this.tokens = Math.min(
      this.maxTokens,
      this.tokens + (elapsed * this.refillRate)
    );
    this.lastRefill = now;
  }
}
 
// Usage: Sample at most 100 traces per second
const sampler = new RateLimitingSampler(100);
 
function onTraceStart(traceId: string): boolean {
  return sampler.shouldSample(traceId);
}

Consider a system with two endpoints:
 
/api/search:       10,000 requests/second (high volume, well-tested)
/api/admin/export: 1 request/minute (rare, complex, often breaks)
 
With 1% probabilistic sampling:
  /api/search:       100 samples/second ✓ (plenty of visibility)
  /api/admin/export: ~0.6 samples/hour ✗ (might miss issues entirely!)
 
Problem: Low-volume operations are under-represented in samples.
 
With adaptive sampling targeting 10 samples/minute per operation:
  /api/search:       10/min = 0.001% sampling rate
  /api/admin/export: 10/min = 16.7% sampling rate (or 100% since volume < target)
 
Result: Every operation has representation regardless of volume.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

# Jaeger adaptive sampling configuration
# This requires jaeger-collector to compute and distribute rates
 
# Remote sampling endpoint (collector provides rates to SDKs)
sampling:
  strategies_file: /etc/jaeger/sampling_strategies.json
  sampling_refresh_interval: 60s
 
# Example adaptive sampling strategy
{
  "service_strategies": [
    {
      "service": "order-service",
      "type": "adaptive",
      "options": {
        "sampling_refresh_interval": "60s",
        "sampling_store_type": "cassandra",  # or in-memory
        "initial_sampling_rate": 0.1,
        "target_samples_per_second": 10,
        "min_sampling_rate": 0.001,
        "max_sampling_rate": 1.0
      },
      "operation_strategies": [
        {
          "operation": "GET /api/checkout",
          "target_samples_per_second": 50  # Important operation
        },
        {
          "operation": "GET /api/health",
          "target_samples_per_second": 1   # Noisy, less important
        }
      ]
    }
  ]
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52

import { context, trace, SpanKind, TraceFlags } from '@opentelemetry/api';
 
// Priority sampler that always samples certain traces
class PrioritySampler {
  private readonly baseSampler: Sampler;
  
  constructor(baseSamplingRate: number) {
    this.baseSampler = new ProbabilisticSampler(baseSamplingRate);
  }
  
  shouldSample(
    context: Context,
    traceId: string,
    spanName: string,
    spanKind: SpanKind,
    attributes: Attributes
  ): SamplingResult {
    
    // Priority 1: Always sample if debug flag is set
    const debugHeader = context.getValue('x-trace-debug');
    if (debugHeader === 'true') {
      return { decision: SamplingDecision.RECORD_AND_SAMPLED };
    }
    
    // Priority 2: Always sample high-value users
    const userTier = attributes['user.tier'];
    if (userTier === 'enterprise' || userTier === 'premium') {
      return { decision: SamplingDecision.RECORD_AND_SAMPLED };
    }
    
    // Priority 3: Always sample critical operations
    const criticalOperations = [
      '/api/checkout',
      '/api/payment/process',
      '/api/refund',
    ];
    if (criticalOperations.some(op => spanName.includes(op))) {
      return { decision: SamplingDecision.RECORD_AND_SAMPLED };
    }
    
    // Priority 4: Always sample synthetic/canary traffic
    const syntheticHeader = attributes['http.header.x-synthetic-request'];
    if (syntheticHeader === 'true') {
      return { decision: SamplingDecision.RECORD_AND_SAMPLED };
    }
    
    // Default: Fall back to base probabilistic sampling
    return this.baseSampler.shouldSample(
      context, traceId, spanName, spanKind, attributes
    );
  }
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66

# OpenTelemetry Collector tail-based sampling configuration
 
processors:
  tail_sampling:
    # How long to wait for all spans of a trace
    decision_wait: 10s
    
    # Number of traces to hold in memory
    num_traces: 100000
    
    # Expected max spans per trace
    expected_new_traces_per_sec: 10000
    
    policies:
      # Policy 1: Always keep error traces
      - name: errors-policy
        type: status_code
        status_code:
          status_codes: [ERROR]
      
      # Policy 2: Always keep slow traces (> 2 seconds)
      - name: latency-policy
        type: latency
        latency:
          threshold_ms: 2000
      
      # Policy 3: Always keep traces with specific attributes
      - name: important-users-policy
        type: string_attribute
        string_attribute:
          key: user.tier
          values: [enterprise, premium, vip]
      
      # Policy 4: Sample 100% of specific operations
      - name: checkout-policy
        type: string_attribute
        string_attribute:
          key: http.route
          values: [/api/checkout, /api/payment]
      
      # Policy 5: Rate limit everything else
      - name: default-policy
        type: rate_limiting
        rate_limiting:
          spans_per_second: 500
      
      # Policy 6: Composite policy example
      - name: composite-policy
        type: composite
        composite:
          max_total_spans_per_second: 1000
          policy_order: [errors-policy, latency-policy, default-policy]
          rate_allocation:
            - policy: errors-policy
              percent: 50
            - policy: latency-policy
              percent: 30
            - policy: default-policy
              percent: 20
 
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling]  # Tail sampling here
      exporters: [jaeger]