Clock Skew — Failure Injection | OneNoughtOne

0/105

System DesignFailure Injection

Failure Injection

LevelAdvanced

Duration90 mins

TopicFailure Injection

5 / 5

Clock Skew

The Hidden Enemy: Time in Distributed Systems

Of all the failure modes in distributed systems, clock-related failures are perhaps the most insidious. Unlike network failures (which cause visible errors) or resource exhaustion (which causes observable slowdowns), clock skew can cause silent correctness violations —your system continues running, returns no errors, yet produces wrong results. Distributed systems make countless implicit assumptions about time: If event A has a timestamp earlier than event B, then A happened before B If a cache entry has a TTL of 5 minutes, it will expire approximately 5 minutes from now If a lease is valid for 30 seconds, the holder has exclusive access for 30 seconds If a certificate expires at midnight, it's valid until midnight Every one of these assumptions can be violated when clocks disagree. And in distributed systems, clocks always disagree . The question is only by how much, and whether your system handles that disagreement correctly.

What You Will Learn

By the end of this page, you will understand how to inject and analyze clock-related failures: clock skew between nodes, sudden time jumps, NTP synchronization failures, and timezone anomalies. You'll learn to identify code that makes unsafe time assumptions, test distributed coordination mechanisms, and implement clock-resilient patterns.

Understanding Time in Distributed Systems

Before injecting clock failures, we must understand how computers track time and why perfect synchronization is impossible. Types of Clocks:

Clock Type	What It Measures	Monotonicity	Use Case
Wall clock (time-of-day)	Current date/time	Can jump backward	Timestamps, scheduling
Monotonic clock	Duration since arbitrary point	Always increases	Measuring elapsed time
Logical clock (Lamport)	Causal ordering	Always increases	Distributed ordering
Vector clock	Per-node logical time	Always increases	Causality tracking
Hybrid logical clock (HLC)	Wall time + logical	Always increases	Spanner-style ordering

Why Perfect Synchronization Is Impossible: Network latency varies — NTP synchronizes over the network, but network delay is unpredictable Clock drift — Hardware clocks drift at different rates (typically 10-100 ppm) Temperature effects — CPU temperature affects crystal oscillator frequency Virtualization overhead — VMs may have less accurate time than physical machines Leap seconds — UTC system inserts leap seconds that not all systems handle correctly NTP availability — If NTP servers are unreachable, clocks drift further

Typical Clock Accuracy:

Environment	Expected Accuracy	Skew in 1 Hour
NTP over internet	10-100ms	Minutes
NTP in datacenter	1-10ms	Seconds
PTP (Precision Time Protocol)	< 1μs	Microseconds
GPS synchronization	< 100ns	Nanoseconds
Same machine	Perfectly synchronized	Zero
Without NTP	Hours per day	Hours

For most distributed systems using NTP within a datacenter, you can expect clocks to be within a few milliseconds of each other—but this bound is not guaranteed, and during NTP failures or network issues, skew can grow significantly.

The Millisecond Assumption

Many developers assume clocks are 'close enough' and write code that breaks with even millisecond skew. Operations like 'find the newest record' or 'check if the token expired' can produce wrong results when clocks disagree. Test what actually happens with realistic skew levels.

Clock Skew Injection

Clock skew injection makes different nodes in your distributed system have different notions of the current time. This tests whether your system relies on clock agreement for correctness. Skew Scenarios to Test:

Scenario	How To Inject	What It Tests
Small constant skew	One node +/- 100ms	Subtle ordering issues
Large constant skew	One node +/- 10s	Timeout and expiration logic
Clock ahead	One node +5 minutes	Token expiration, cache TTL
Clock behind	One node -5 minutes	Timestamp ordering, 'newest' logic
Variable skew	Skew changes over time	Sensitivity to drift rate
Asymmetric communication	A→B normal, B→A skewed	Request/response timing

What Skew Breaks: Distributed locks — Node 2 might think Node 1's lease expired when it hasn't Event ordering — Events appear out of order when sorted by timestamp Cache invalidation — Entry might be cached with 'future' timestamp Token expiration — Token might appear valid/invalid depending on which node checks Conflict resolution — 'Last-write-wins' becomes 'last-clock-wins' Consensus protocols — Leader election timeouts might fire incorrectly

clock-skew-injection.sh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
# WARNING: These commands modify system time
# Only run in isolated test environments, never in production
 
# Check current time and NTP status
timedatectl status
ntpq -p  # or chronyc sources
 
# Set time manually (requires root, disables NTP)
sudo timedatectl set-ntp false
sudo date -s "$(date -d '+5 minutes')"  # Set clock 5 minutes ahead
sudo date -s "$(date -d '-5 minutes')"  # Set clock 5 minutes behind
 
# Temporary time shift (returns to normal after test)
# Uses libfaketime to intercept time calls
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/faketime/libfaketime.so.1 \
  FAKETIME="+5m" \
  ./my-service
 
# Container-level time manipulation using Chaos Mesh
cat <<EOF | kubectl apply -f -
apiVersion: chaos-mesh.org/v1alpha1
kind: TimeChaos
metadata:
  name: clock-skew-ahead
spec:
  mode: one
  selector:
    labelSelectors:
      app: service-a
  timeOffset: "5m"        # 5 minutes ahead
  duration: "10m"
EOF
 
# Skew clock backward
cat <<EOF | kubectl apply -f -
apiVersion: chaos-mesh.org/v1alpha1
kind: TimeChaos
metadata:
  name: clock-skew-behind
spec:
  mode: one
  selector:
    labelSelectors:
      app: service-b
  timeOffset: "-5m"       # 5 minutes behind
  duration: "10m"
EOF
 
# Using Docker with faketime
docker run --rm \
  -e LD_PRELOAD=/usr/lib/x86_64-linux-gnu/faketime/libfaketime.so.1 \
  -e FAKETIME="+1h" \
  my-service:latest

Clock Skew Test Scenarios

•Database Leader/Replica — Skew the replica clock ahead. Do reads see 'future' data?
•Distributed Lock — Skew lock holder's clock behind. Does it hold lock past expiration?
•Cache Cluster — Skew one cache node ahead. Do TTLs expire prematurely?
•Message Queue — Skew consumer's clock. Are messages processed in wrong order?
•API Gateway — Skew gateway's clock. Are JWT tokens validated correctly?
•Metrics/Logging — Skew application's clock. Are timestamps usable for debugging?

Test Both Directions

Clock ahead and clock behind cause different failures. A clock running ahead causes premature expiration; a clock behind causes stale data to appear valid. Test both directions for each component.

Time Jumps

Time jumps are sudden, large changes in the system clock—as opposed to gradual drift. They occur when: NTP corrects a significantly skewed clock A VM resumes from suspension with stale time, then corrects An administrator manually sets the time The system clock is stepped (rather than slewed) to correct drift Hardware clock battery fails and system boots with wrong time Why Time Jumps Are Dangerous: Unlike gradual skew (which might add or remove a few milliseconds per second), time jumps change the clock by seconds, minutes, or even hours instantaneously. Code that assumes time moves forward smoothly can: Miss scheduled events if time jumps past their scheduled time Run scheduled events multiple times if time jumps backward then forward Calculate negative durations if end time is before start time after backward jump Expire all cache entries at once if time jumps forward Invalidate all tokens if time jumps past expiration Violate causality in event logs if timestamps go backward

Time Jump Scenarios to Test

•Forward Jump (1 hour) — Simulate VM resume or large NTP correction. Test scheduled job handling, cache expiration, session validity.
•Backward Jump (1 hour) — Simulate backward NTP correction. Test duration calculations, monotonic vs wall clock usage.
•Small Forward Jump (1 minute) — Simulate typical NTP correction. Test sensitivity to common corrections.
•Small Backward Jump (30 seconds) — Test whether code handles time going backward gracefully.
•Rapid Oscillation — Jump forward then backward repeatedly. Test for race conditions in time handling.
•Jump During Long Operation — Start operation, jump time mid-operation. Test elapsed time calculations.

time-jump-injection.sh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
# Forward time jump
sudo date -s "$(date -d '+1 hour')"
 
# Backward time jump (DANGEROUS - can cause issues)
sudo date -s "$(date -d '-30 seconds')"
 
# Kubernetes TimeChaos for jumps
cat <<EOF | kubectl apply -f -
apiVersion: chaos-mesh.org/v1alpha1
kind: TimeChaos
metadata:
  name: time-jump-forward
spec:
  mode: all
  selector:
    labelSelectors:
      app: scheduler-service
  timeOffset: "1h"        # Jump 1 hour ahead
  clockIds:
    - CLOCK_REALTIME      # Affects wall clock, not monotonic
  duration: "30m"
EOF
 
# Test time jump during operation (script example)
#!/bin/bash
echo "Starting long operation at $(date)"
./start-long-operation &
OP_PID=$!
sleep 5
echo "Jumping time forward 10 minutes"
sudo date -s "$(date -d '+10 minutes')"
wait $OP_PID
echo "Operation completed at $(date)"
 
# Using faketime for time jumps
# Start at normal time
FAKETIME="@2024-01-01 00:00:00" ./my-service &
sleep 10
# Kill and restart with jumped time
kill $!
FAKETIME="@2024-01-01 01:00:00" ./my-service &

Observations During Time Jump Testing:

Component	Forward Jump Effect	Backward Jump Effect
Scheduled jobs	Jobs may be skipped	Jobs may run twice
Cache entries	Mass expiration	Entries appear fresh
Session tokens	Mass invalidation	Expired tokens appear valid
Rate limiters	Quotas reset	Negative available tokens
Distributed locks	Leases expire	Holder thinks lease is valid
Metrics	Gaps in time series	Duplicate data points
Logs	Timestamps jump	Non-monotonic timestamps

Backward Time Jumps Are Particularly Dangerous

Many systems don't handle backward time jumps at all. NTP typically 'slews' the clock (adjusts rate) rather than stepping backward, but steps can occur after VM resume or manual correction. If your logs show 'negative elapsed time' errors, your code isn't using monotonic clocks where it should.

NTP Failures

NTP (Network Time Protocol) keeps distributed system clocks synchronized. When NTP fails or becomes unavailable, clocks drift apart at the rate of their hardware clock inaccuracy—typically 10-100 parts per million (ppm). This means: At 100 ppm drift: ~8.6 seconds per day, ~6 minutes per month Two nodes drifting in opposite directions could diverge twice as fast Extended NTP outages can cause significant clock divergence NTP Failure Modes:

Failure Mode	Effect	Time to Significant Skew
NTP servers unreachable	Clocks drift at hardware rate	Hours to days
NTP returning wrong time	Clocks steered toward wrong value	Minutes
NTP latency variations	Reduced accuracy	Gradual
NTP server compromise	Malicious time values	Immediate
Leap second insertion	61-second minute	Once per insertion
Clock stepping vs slewing	Sudden vs gradual correction	Immediate

NTP Failure Scenarios to Test

•NTP Server Unavailable — Block NTP traffic (UDP 123). Monitor how long until clock skew becomes problematic.
•Delayed NTP Responses — Add latency to NTP packets. Test whether synchronization accuracy degrades.
•Intermittent NTP — Block NTP periodically (5 min on, 5 min off). Test recovery behavior.
•Stale NTP Data — If using local NTP cache, test behavior when cache is stale.
•NTP Stratum Degradation — If using hierarchical NTP, test behavior when primary servers are unavailable.
•Extended NTP Outage — Block NTP for 24 hours. Measure actual clock drift in your environment.

ntp-failure-injection.sh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
# Check current NTP status
timedatectl status
systemctl status chronyd  # or systemctl status ntpd
 
# Check NTP server connectivity
chronyc sources    # for chronyd
ntpq -p           # for ntpd
 
# Block NTP traffic (UDP port 123)
sudo iptables -A OUTPUT -p udp --dport 123 -j DROP
sudo iptables -A INPUT -p udp --sport 123 -j DROP
 
# Monitor clock drift during NTP outage
while true; do
  echo "$(date +%s.%N) - $(curl -s http://time.google.com | grep -oP '\d+\.\d+')"
  sleep 60
done
 
# Add latency to NTP packets
sudo tc qdisc add dev eth0 root handle 1: prio
sudo tc qdisc add dev eth0 parent 1:3 handle 30: netem delay 500ms
sudo tc filter add dev eth0 protocol ip parent 1:0 prio 3 u32 \
  match ip dport 123 0xffff flowid 1:3
 
# Disable NTP synchronization temporarily
sudo timedatectl set-ntp false
# ... run experiment ...
sudo timedatectl set-ntp true
 
# Force NTP resync after experiment
sudo chronyc makestep  # for chronyd
sudo ntpdate -u pool.ntp.org  # for ntpd
 
# Kubernetes: Block NTP at pod level
cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: block-ntp
spec:
  podSelector:
    matchLabels:
      app: test-service
  policyTypes:
    - Egress
  egress:
    - ports:
        - protocol: UDP
          port: 53        # Allow DNS
    - to:
        - podSelector: {}  # Allow pod-to-pod
    # NTP (UDP 123) is not listed, so it's blocked
EOF

Hardware Clock Drift Rate

The rate of clock drift without NTP depends on hardware quality and temperature. Virtual machines often have higher drift rates than physical hardware. Measure your actual drift rate during NTP outage testing—it affects how quickly skew becomes problematic.

Leap Seconds and Calendar Edge Cases

Leap seconds are occasional one-second adjustments to UTC to account for irregularities in Earth's rotation. While seemingly minor, leap seconds have caused numerous production incidents because they create situations that 'normal' time handling doesn't anticipate. Famous Leap Second Incidents:

Year	Affected System	What Happened
2012	Linux kernel, Java	High CPU from spinning processes, crashes
2012	Reddit	Multiple hour outage
2012	Mozilla	Bug in code handling 61-second minute
2015	Cloudflare	Negative duration calculation, DNS issues
2016	Multiple	Similar issues, better preparation

How Operating Systems Handle Leap Seconds: Smear : Google and AWS spread the leap second across the day, slightly slowing/speeding clocks Step : Linux can insert a 61st second at 23:59:60 Repeat : Some systems repeat second 59 twice Ignore : Some systems don't handle at all (until NTP corrects)

Time Edge Cases to Test

•Leap Second Simulation — Insert timestamp at 23:59:60. Test whether parsing and handling code breaks.
•61-Second Minute — Test scheduling code with 61 seconds in a minute.
•Repeated Second — Test with same timestamp appearing twice (second 59 repeated).
•Daylight Saving Transitions — Test during DST changes. Does scheduled work run twice or skip?
•Timezone Changes — Test behavior when system timezone changes at runtime.
•Year Boundaries — Test at 2024-12-31 23:59:59 → 2025-01-01 00:00:00 transition.
•Month-End Processing — Test batch jobs that run on last day of month (what about Feb 28/29?).

leap-second-testing.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
# Testing leap second handling in code
import datetime
from dateutil import parser
 
# Test parsing a leap second timestamp
try:
    # This is technically a valid UTC timestamp during leap second
    leap_second = "2016-12-31T23:59:60Z"
    parsed = parser.parse(leap_second)
    print(f"Parsed leap second: {parsed}")
except Exception as e:
    print(f"Failed to parse leap second: {e}")
 
# Test duration calculation across leap second
t1 = datetime.datetime(2016, 12, 31, 23, 59, 59)
t2 = datetime.datetime(2017, 1, 1, 0, 0, 0)
duration = (t2 - t1).total_seconds()
print(f"Duration: {duration}s (should be 1s, or 2s with leap)")
 
# Test for negative duration after time jump
def measure_operation():
    start = datetime.datetime.now()
    # ... operation ...
    end = datetime.datetime.now()
    duration = (end - start).total_seconds()
    if duration < 0:
        raise RuntimeError(f"Negative duration: {duration}")
    return duration
 
# Better: use monotonic clock for durations
import time
 
def measure_operation_safe():
    start = time.monotonic()
    # ... operation ...
    end = time.monotonic()
    duration = end - start  # Always non-negative
    return duration
 
# Test DST transition
import pytz
 
tz = pytz.timezone('US/Eastern')
# This time happens twice during fall-back
ambiguous = datetime.datetime(2024, 11, 3, 1, 30, 0)
try:
    localized = tz.localize(ambiguous)
except pytz.exceptions.AmbiguousTimeError:
    print("Ambiguous time during DST transition")
 
# This time doesn't exist during spring-forward
missing = datetime.datetime(2024, 3, 10, 2, 30, 0)
try:
    localized = tz.localize(missing)
except pytz.exceptions.NonExistentTimeError:
    print("Non-existent time during DST transition")

Use Leap Second Smearing

If your cloud provider offers leap second smearing (Google, AWS), use it. Smearing makes the leap second invisible to applications by gradually adjusting clocks, eliminating the need for applications to handle 61-second minutes. Verify your provider's approach and configure accordingly.

Patterns Vulnerable to Clock Issues

Certain distributed system patterns are inherently sensitive to clock accuracy. Understanding which patterns are at risk helps focus testing efforts.

Clock-Sensitive Distributed Patterns
Pattern	Clock Dependency	Risk of Failure	Mitigation
Distributed locks with TTL	Lease expiration timing	High	Use fencing tokens, conservative TTLs
Token-based authentication	Token expiry validation	High	Include server time in response, use generous buffers
Last-write-wins conflict resolution	Timestamp comparison	High	Use vector clocks or HLC instead
Event sourcing timestamps	Event ordering	Medium	Use logical clocks for ordering
Time-windowed rate limiting	Window boundary detection	Medium	Use sliding windows, be conservative
Cache TTL	Expiration timing	Medium	Use monotonic time for durations
Scheduled jobs	Trigger timing	Medium	Use dedicated scheduler, handle missed runs
Audit logging	Timestamp accuracy	Low	Include source and sync status

Clock-Resilient Patterns

•Use monotonic clocks for durations — Never compute elapsed time using wall clock differences. Use clock_gettime(CLOCK_MONOTONIC) or language equivalents.
•Use logical clocks for ordering — Lamport clocks, vector clocks, or HLC provide ordering guarantees without wall clock dependency.
•Include clock uncertainty — Like Google Spanner's TrueTime, maintain bounds on clock error and wait out uncertainty before committing.
•Use fencing tokens — For distributed locks, use monotonically increasing tokens to detect stale lock holders.
•Validate external timestamps — When receiving timestamps from external sources, validate against expected ranges.
•Log clock sync status — Include NTP sync status and offset in logs to aid debugging time-related issues.

clock-resilient-patterns.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
// Examples of clock-resilient coding patterns
 
// BAD: Using wall clock for elapsed time
function measureOperationBad(): number {
  const start = Date.now();
  doOperation();
  const end = Date.now();
  return end - start;  // Can be negative if clock jumps backward!
}
 
// GOOD: Using monotonic time (performance.now() in JS)
function measureOperationGood(): number {
  const start = performance.now();
  doOperation();
  const end = performance.now();
  return end - start;  // Always non-negative
}
 
// BAD: Last-write-wins using wall clock
interface Record {
  value: string;
  timestamp: number;
}
 
function mergeRecordsBad(a: Record, b: Record): Record {
  return a.timestamp > b.timestamp ? a : b;  // Sensitive to clock skew!
}
 
// GOOD: Using vector clock for conflict detection
interface VectorClock {
  [nodeId: string]: number;
}
 
interface RecordWithVector {
  value: string;
  vectorClock: VectorClock;
}
 
function mergeRecordsGood(a: RecordWithVector, b: RecordWithVector): RecordWithVector[] {
  if (happensBefore(a.vectorClock, b.vectorClock)) return [b];
  if (happensBefore(b.vectorClock, a.vectorClock)) return [a];
  return [a, b];  // Concurrent - return both for conflict resolution
}
 
// BAD: Distributed lock with only TTL
async function acquireLockBad(key: string, ttlMs: number): Promise<boolean> {
  const acquired = await redis.set(key, 'locked', 'NX', 'PX', ttlMs);
  return acquired === 'OK';
}
 
// GOOD: Distributed lock with fencing token
async function acquireLockGood(key: string, ttlMs: number): Promise<number | null> {
  const token = await redis.incr('lock:token:' + key);
  const acquired = await redis.set(
    key, 
    JSON.stringify({ token, holder: nodeId }), 
    'NX', 
    'PX', 
    ttlMs
  );
  return acquired === 'OK' ? token : null;
}
 
// When using the lock, pass fencing token to downstream:
async function writeWithFencingToken(value: string, fencingToken: number) {
  // Storage system rejects writes with lower token
  await database.write(value, { fencingToken });
}

Review All Wall Clock Usage

Search your codebase for Date.now(), System.currentTimeMillis(), time.time(), and similar calls. Each one is a potential clock skew vulnerability. Most should be replaced with monotonic time or logical clocks.

Observing Clock-Related Failures

Clock-related failures are notoriously difficult to observe because they often don't cause visible errors—just incorrect behavior. Special instrumentation is needed to detect and diagnose these issues.

What to Monitor During Clock Experiments

•NTP offset and sync status — Export chronyd/ntpd offset metrics. Alert when offset exceeds threshold.
•Clock comparison across nodes — Periodically log wall clock time from all nodes. Compare for drift.
•Negative duration errors — Log when elapsed time calculations produce negative values.
•Timestamp ordering violations — Log when events arrive with timestamps older than previously seen.
•TTL-related anomalies — Monitor for premature expiration (clock ahead) or stale data (clock behind).
•Scheduled job execution times — Track if jobs run early, late, or are skipped entirely.
•Authentication failures — Monitor for spikes in token validation failures that might indicate clock issues.

clock-monitoring.sh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
# Export NTP metrics to Prometheus
 
# For chronyd, use chrony_exporter
# https://github.com/superq/chrony_exporter
 
# Key metrics to monitor:
# - chrony_tracking_stratum: NTP stratum level
# - chrony_tracking_root_delay_seconds: delay to root time source
# - chrony_tracking_root_dispersion_seconds: dispersion from root
# - chrony_tracking_system_time: current system time adjustment
 
# Alert on clock skew
cat <<EOF | kubectl apply -f -
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: clock-skew-alerts
spec:
  groups:
    - name: clock
      rules:
        - alert: ClockSkewHigh
          expr: abs(chrony_tracking_system_time) > 0.1
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "Clock skew > 100ms on {{ $labels.instance }}"
        
        - alert: NTPNotSynced
          expr: chrony_tracking_stratum == 0
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "NTP not synced on {{ $labels.instance }}"
EOF
 
# Log cluster clock comparison (run on each node)
#!/bin/bash
while true; do
  TIMESTAMP=$(date +%s.%N)
  NTP_OFFSET=$(chronyc tracking | grep "System time" | awk '{print $4}')
  echo "{"node": "$(hostname)", "time": $TIMESTAMP, "offset": $NTP_OFFSET}"
  sleep 10
done

Symptoms and Diagnosis of Clock Issues
Symptom	Possible Clock Issue	Diagnostic Steps
Tokens expire immediately	Local clock is ahead	Compare local time to token issue time
Expired tokens accepted	Local clock is behind	Check NTP sync status
Cache entries evict too soon	Clock skew in cache cluster	Compare time across cache nodes
Events appear out of order in logs	Clock skew between producers	Add source timestamp to log entries
Last-write-wins gives wrong result	Writer clock is skewed	Use vector clocks instead
Distributed lock contention	Lease timing incorrect	Add fencing tokens to locks
Scheduled jobs run early/late	Scheduler clock drifted	Check NTP offset on scheduler nodes

System DesignFailure Injection

Failure Injection

LevelAdvanced

Duration90 mins

TopicFailure Injection

5 / 5

Clock Skew

The Hidden Enemy: Time in Distributed Systems

Of all the failure modes in distributed systems, clock-related failures are perhaps the most insidious. Unlike network failures (which cause visible errors) or resource exhaustion (which causes observable slowdowns), clock skew can cause silent correctness violations —your system continues running, returns no errors, yet produces wrong results. Distributed systems make countless implicit assumptions about time: If event A has a timestamp earlier than event B, then A happened before B If a cache entry has a TTL of 5 minutes, it will expire approximately 5 minutes from now If a lease is valid for 30 seconds, the holder has exclusive access for 30 seconds If a certificate expires at midnight, it's valid until midnight Every one of these assumptions can be violated when clocks disagree. And in distributed systems, clocks always disagree . The question is only by how much, and whether your system handles that disagreement correctly.

What You Will Learn

By the end of this page, you will understand how to inject and analyze clock-related failures: clock skew between nodes, sudden time jumps, NTP synchronization failures, and timezone anomalies. You'll learn to identify code that makes unsafe time assumptions, test distributed coordination mechanisms, and implement clock-resilient patterns.

Understanding Time in Distributed Systems

Before injecting clock failures, we must understand how computers track time and why perfect synchronization is impossible. Types of Clocks:

Clock Type	What It Measures	Monotonicity	Use Case
Wall clock (time-of-day)	Current date/time	Can jump backward	Timestamps, scheduling
Monotonic clock	Duration since arbitrary point	Always increases	Measuring elapsed time
Logical clock (Lamport)	Causal ordering	Always increases	Distributed ordering
Vector clock	Per-node logical time	Always increases	Causality tracking
Hybrid logical clock (HLC)	Wall time + logical	Always increases	Spanner-style ordering

Why Perfect Synchronization Is Impossible: Network latency varies — NTP synchronizes over the network, but network delay is unpredictable Clock drift — Hardware clocks drift at different rates (typically 10-100 ppm) Temperature effects — CPU temperature affects crystal oscillator frequency Virtualization overhead — VMs may have less accurate time than physical machines Leap seconds — UTC system inserts leap seconds that not all systems handle correctly NTP availability — If NTP servers are unreachable, clocks drift further

Typical Clock Accuracy:

Environment	Expected Accuracy	Skew in 1 Hour
NTP over internet	10-100ms	Minutes
NTP in datacenter	1-10ms	Seconds
PTP (Precision Time Protocol)	< 1μs	Microseconds
GPS synchronization	< 100ns	Nanoseconds
Same machine	Perfectly synchronized	Zero
Without NTP	Hours per day	Hours

For most distributed systems using NTP within a datacenter, you can expect clocks to be within a few milliseconds of each other—but this bound is not guaranteed, and during NTP failures or network issues, skew can grow significantly.

The Millisecond Assumption

Many developers assume clocks are 'close enough' and write code that breaks with even millisecond skew. Operations like 'find the newest record' or 'check if the token expired' can produce wrong results when clocks disagree. Test what actually happens with realistic skew levels.

Clock Skew Injection

Clock skew injection makes different nodes in your distributed system have different notions of the current time. This tests whether your system relies on clock agreement for correctness. Skew Scenarios to Test:

Scenario	How To Inject	What It Tests
Small constant skew	One node +/- 100ms	Subtle ordering issues
Large constant skew	One node +/- 10s	Timeout and expiration logic
Clock ahead	One node +5 minutes	Token expiration, cache TTL
Clock behind	One node -5 minutes	Timestamp ordering, 'newest' logic
Variable skew	Skew changes over time	Sensitivity to drift rate
Asymmetric communication	A→B normal, B→A skewed	Request/response timing

What Skew Breaks: Distributed locks — Node 2 might think Node 1's lease expired when it hasn't Event ordering — Events appear out of order when sorted by timestamp Cache invalidation — Entry might be cached with 'future' timestamp Token expiration — Token might appear valid/invalid depending on which node checks Conflict resolution — 'Last-write-wins' becomes 'last-clock-wins' Consensus protocols — Leader election timeouts might fire incorrectly

clock-skew-injection.sh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
# WARNING: These commands modify system time
# Only run in isolated test environments, never in production
 
# Check current time and NTP status
timedatectl status
ntpq -p  # or chronyc sources
 
# Set time manually (requires root, disables NTP)
sudo timedatectl set-ntp false
sudo date -s "$(date -d '+5 minutes')"  # Set clock 5 minutes ahead
sudo date -s "$(date -d '-5 minutes')"  # Set clock 5 minutes behind
 
# Temporary time shift (returns to normal after test)
# Uses libfaketime to intercept time calls
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/faketime/libfaketime.so.1 \
  FAKETIME="+5m" \
  ./my-service
 
# Container-level time manipulation using Chaos Mesh
cat <<EOF | kubectl apply -f -
apiVersion: chaos-mesh.org/v1alpha1
kind: TimeChaos
metadata:
  name: clock-skew-ahead
spec:
  mode: one
  selector:
    labelSelectors:
      app: service-a
  timeOffset: "5m"        # 5 minutes ahead
  duration: "10m"
EOF
 
# Skew clock backward
cat <<EOF | kubectl apply -f -
apiVersion: chaos-mesh.org/v1alpha1
kind: TimeChaos
metadata:
  name: clock-skew-behind
spec:
  mode: one
  selector:
    labelSelectors:
      app: service-b
  timeOffset: "-5m"       # 5 minutes behind
  duration: "10m"
EOF
 
# Using Docker with faketime
docker run --rm \
  -e LD_PRELOAD=/usr/lib/x86_64-linux-gnu/faketime/libfaketime.so.1 \
  -e FAKETIME="+1h" \
  my-service:latest

Clock Skew Test Scenarios

•Database Leader/Replica — Skew the replica clock ahead. Do reads see 'future' data?
•Distributed Lock — Skew lock holder's clock behind. Does it hold lock past expiration?
•Cache Cluster — Skew one cache node ahead. Do TTLs expire prematurely?
•Message Queue — Skew consumer's clock. Are messages processed in wrong order?
•API Gateway — Skew gateway's clock. Are JWT tokens validated correctly?
•Metrics/Logging — Skew application's clock. Are timestamps usable for debugging?

Test Both Directions

Clock ahead and clock behind cause different failures. A clock running ahead causes premature expiration; a clock behind causes stale data to appear valid. Test both directions for each component.

Time Jumps

Time jumps are sudden, large changes in the system clock—as opposed to gradual drift. They occur when: NTP corrects a significantly skewed clock A VM resumes from suspension with stale time, then corrects An administrator manually sets the time The system clock is stepped (rather than slewed) to correct drift Hardware clock battery fails and system boots with wrong time Why Time Jumps Are Dangerous: Unlike gradual skew (which might add or remove a few milliseconds per second), time jumps change the clock by seconds, minutes, or even hours instantaneously. Code that assumes time moves forward smoothly can: Miss scheduled events if time jumps past their scheduled time Run scheduled events multiple times if time jumps backward then forward Calculate negative durations if end time is before start time after backward jump Expire all cache entries at once if time jumps forward Invalidate all tokens if time jumps past expiration Violate causality in event logs if timestamps go backward

Time Jump Scenarios to Test

•Forward Jump (1 hour) — Simulate VM resume or large NTP correction. Test scheduled job handling, cache expiration, session validity.
•Backward Jump (1 hour) — Simulate backward NTP correction. Test duration calculations, monotonic vs wall clock usage.
•Small Forward Jump (1 minute) — Simulate typical NTP correction. Test sensitivity to common corrections.
•Small Backward Jump (30 seconds) — Test whether code handles time going backward gracefully.
•Rapid Oscillation — Jump forward then backward repeatedly. Test for race conditions in time handling.
•Jump During Long Operation — Start operation, jump time mid-operation. Test elapsed time calculations.

time-jump-injection.sh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
# Forward time jump
sudo date -s "$(date -d '+1 hour')"
 
# Backward time jump (DANGEROUS - can cause issues)
sudo date -s "$(date -d '-30 seconds')"
 
# Kubernetes TimeChaos for jumps
cat <<EOF | kubectl apply -f -
apiVersion: chaos-mesh.org/v1alpha1
kind: TimeChaos
metadata:
  name: time-jump-forward
spec:
  mode: all
  selector:
    labelSelectors:
      app: scheduler-service
  timeOffset: "1h"        # Jump 1 hour ahead
  clockIds:
    - CLOCK_REALTIME      # Affects wall clock, not monotonic
  duration: "30m"
EOF
 
# Test time jump during operation (script example)
#!/bin/bash
echo "Starting long operation at $(date)"
./start-long-operation &
OP_PID=$!
sleep 5
echo "Jumping time forward 10 minutes"
sudo date -s "$(date -d '+10 minutes')"
wait $OP_PID
echo "Operation completed at $(date)"
 
# Using faketime for time jumps
# Start at normal time
FAKETIME="@2024-01-01 00:00:00" ./my-service &
sleep 10
# Kill and restart with jumped time
kill $!
FAKETIME="@2024-01-01 01:00:00" ./my-service &

Observations During Time Jump Testing:

Component	Forward Jump Effect	Backward Jump Effect
Scheduled jobs	Jobs may be skipped	Jobs may run twice
Cache entries	Mass expiration	Entries appear fresh
Session tokens	Mass invalidation	Expired tokens appear valid
Rate limiters	Quotas reset	Negative available tokens
Distributed locks	Leases expire	Holder thinks lease is valid
Metrics	Gaps in time series	Duplicate data points
Logs	Timestamps jump	Non-monotonic timestamps

Backward Time Jumps Are Particularly Dangerous

Many systems don't handle backward time jumps at all. NTP typically 'slews' the clock (adjusts rate) rather than stepping backward, but steps can occur after VM resume or manual correction. If your logs show 'negative elapsed time' errors, your code isn't using monotonic clocks where it should.

NTP Failures

NTP (Network Time Protocol) keeps distributed system clocks synchronized. When NTP fails or becomes unavailable, clocks drift apart at the rate of their hardware clock inaccuracy—typically 10-100 parts per million (ppm). This means: At 100 ppm drift: ~8.6 seconds per day, ~6 minutes per month Two nodes drifting in opposite directions could diverge twice as fast Extended NTP outages can cause significant clock divergence NTP Failure Modes:

Failure Mode	Effect	Time to Significant Skew
NTP servers unreachable	Clocks drift at hardware rate	Hours to days
NTP returning wrong time	Clocks steered toward wrong value	Minutes
NTP latency variations	Reduced accuracy	Gradual
NTP server compromise	Malicious time values	Immediate
Leap second insertion	61-second minute	Once per insertion
Clock stepping vs slewing	Sudden vs gradual correction	Immediate

NTP Failure Scenarios to Test

•NTP Server Unavailable — Block NTP traffic (UDP 123). Monitor how long until clock skew becomes problematic.
•Delayed NTP Responses — Add latency to NTP packets. Test whether synchronization accuracy degrades.
•Intermittent NTP — Block NTP periodically (5 min on, 5 min off). Test recovery behavior.
•Stale NTP Data — If using local NTP cache, test behavior when cache is stale.
•NTP Stratum Degradation — If using hierarchical NTP, test behavior when primary servers are unavailable.
•Extended NTP Outage — Block NTP for 24 hours. Measure actual clock drift in your environment.

ntp-failure-injection.sh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
# Check current NTP status
timedatectl status
systemctl status chronyd  # or systemctl status ntpd
 
# Check NTP server connectivity
chronyc sources    # for chronyd
ntpq -p           # for ntpd
 
# Block NTP traffic (UDP port 123)
sudo iptables -A OUTPUT -p udp --dport 123 -j DROP
sudo iptables -A INPUT -p udp --sport 123 -j DROP
 
# Monitor clock drift during NTP outage
while true; do
  echo "$(date +%s.%N) - $(curl -s http://time.google.com | grep -oP '\d+\.\d+')"
  sleep 60
done
 
# Add latency to NTP packets
sudo tc qdisc add dev eth0 root handle 1: prio
sudo tc qdisc add dev eth0 parent 1:3 handle 30: netem delay 500ms
sudo tc filter add dev eth0 protocol ip parent 1:0 prio 3 u32 \
  match ip dport 123 0xffff flowid 1:3
 
# Disable NTP synchronization temporarily
sudo timedatectl set-ntp false
# ... run experiment ...
sudo timedatectl set-ntp true
 
# Force NTP resync after experiment
sudo chronyc makestep  # for chronyd
sudo ntpdate -u pool.ntp.org  # for ntpd
 
# Kubernetes: Block NTP at pod level
cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: block-ntp
spec:
  podSelector:
    matchLabels:
      app: test-service
  policyTypes:
    - Egress
  egress:
    - ports:
        - protocol: UDP
          port: 53        # Allow DNS
    - to:
        - podSelector: {}  # Allow pod-to-pod
    # NTP (UDP 123) is not listed, so it's blocked
EOF

Hardware Clock Drift Rate

The rate of clock drift without NTP depends on hardware quality and temperature. Virtual machines often have higher drift rates than physical hardware. Measure your actual drift rate during NTP outage testing—it affects how quickly skew becomes problematic.

Leap Seconds and Calendar Edge Cases

Leap seconds are occasional one-second adjustments to UTC to account for irregularities in Earth's rotation. While seemingly minor, leap seconds have caused numerous production incidents because they create situations that 'normal' time handling doesn't anticipate. Famous Leap Second Incidents:

Year	Affected System	What Happened
2012	Linux kernel, Java	High CPU from spinning processes, crashes
2012	Reddit	Multiple hour outage
2012	Mozilla	Bug in code handling 61-second minute
2015	Cloudflare	Negative duration calculation, DNS issues
2016	Multiple	Similar issues, better preparation

How Operating Systems Handle Leap Seconds: Smear : Google and AWS spread the leap second across the day, slightly slowing/speeding clocks Step : Linux can insert a 61st second at 23:59:60 Repeat : Some systems repeat second 59 twice Ignore : Some systems don't handle at all (until NTP corrects)

Time Edge Cases to Test

•Leap Second Simulation — Insert timestamp at 23:59:60. Test whether parsing and handling code breaks.
•61-Second Minute — Test scheduling code with 61 seconds in a minute.
•Repeated Second — Test with same timestamp appearing twice (second 59 repeated).
•Daylight Saving Transitions — Test during DST changes. Does scheduled work run twice or skip?
•Timezone Changes — Test behavior when system timezone changes at runtime.
•Year Boundaries — Test at 2024-12-31 23:59:59 → 2025-01-01 00:00:00 transition.
•Month-End Processing — Test batch jobs that run on last day of month (what about Feb 28/29?).

leap-second-testing.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
# Testing leap second handling in code
import datetime
from dateutil import parser
 
# Test parsing a leap second timestamp
try:
    # This is technically a valid UTC timestamp during leap second
    leap_second = "2016-12-31T23:59:60Z"
    parsed = parser.parse(leap_second)
    print(f"Parsed leap second: {parsed}")
except Exception as e:
    print(f"Failed to parse leap second: {e}")
 
# Test duration calculation across leap second
t1 = datetime.datetime(2016, 12, 31, 23, 59, 59)
t2 = datetime.datetime(2017, 1, 1, 0, 0, 0)
duration = (t2 - t1).total_seconds()
print(f"Duration: {duration}s (should be 1s, or 2s with leap)")
 
# Test for negative duration after time jump
def measure_operation():
    start = datetime.datetime.now()
    # ... operation ...
    end = datetime.datetime.now()
    duration = (end - start).total_seconds()
    if duration < 0:
        raise RuntimeError(f"Negative duration: {duration}")
    return duration
 
# Better: use monotonic clock for durations
import time
 
def measure_operation_safe():
    start = time.monotonic()
    # ... operation ...
    end = time.monotonic()
    duration = end - start  # Always non-negative
    return duration
 
# Test DST transition
import pytz
 
tz = pytz.timezone('US/Eastern')
# This time happens twice during fall-back
ambiguous = datetime.datetime(2024, 11, 3, 1, 30, 0)
try:
    localized = tz.localize(ambiguous)
except pytz.exceptions.AmbiguousTimeError:
    print("Ambiguous time during DST transition")
 
# This time doesn't exist during spring-forward
missing = datetime.datetime(2024, 3, 10, 2, 30, 0)
try:
    localized = tz.localize(missing)
except pytz.exceptions.NonExistentTimeError:
    print("Non-existent time during DST transition")

Use Leap Second Smearing

If your cloud provider offers leap second smearing (Google, AWS), use it. Smearing makes the leap second invisible to applications by gradually adjusting clocks, eliminating the need for applications to handle 61-second minutes. Verify your provider's approach and configure accordingly.

Patterns Vulnerable to Clock Issues

Certain distributed system patterns are inherently sensitive to clock accuracy. Understanding which patterns are at risk helps focus testing efforts.

Clock-Sensitive Distributed Patterns
Pattern	Clock Dependency	Risk of Failure	Mitigation
Distributed locks with TTL	Lease expiration timing	High	Use fencing tokens, conservative TTLs
Token-based authentication	Token expiry validation	High	Include server time in response, use generous buffers
Last-write-wins conflict resolution	Timestamp comparison	High	Use vector clocks or HLC instead
Event sourcing timestamps	Event ordering	Medium	Use logical clocks for ordering
Time-windowed rate limiting	Window boundary detection	Medium	Use sliding windows, be conservative
Cache TTL	Expiration timing	Medium	Use monotonic time for durations
Scheduled jobs	Trigger timing	Medium	Use dedicated scheduler, handle missed runs
Audit logging	Timestamp accuracy	Low	Include source and sync status

Clock-Resilient Patterns

•Use monotonic clocks for durations — Never compute elapsed time using wall clock differences. Use clock_gettime(CLOCK_MONOTONIC) or language equivalents.
•Use logical clocks for ordering — Lamport clocks, vector clocks, or HLC provide ordering guarantees without wall clock dependency.
•Include clock uncertainty — Like Google Spanner's TrueTime, maintain bounds on clock error and wait out uncertainty before committing.
•Use fencing tokens — For distributed locks, use monotonically increasing tokens to detect stale lock holders.
•Validate external timestamps — When receiving timestamps from external sources, validate against expected ranges.
•Log clock sync status — Include NTP sync status and offset in logs to aid debugging time-related issues.

clock-resilient-patterns.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
// Examples of clock-resilient coding patterns
 
// BAD: Using wall clock for elapsed time
function measureOperationBad(): number {
  const start = Date.now();
  doOperation();
  const end = Date.now();
  return end - start;  // Can be negative if clock jumps backward!
}
 
// GOOD: Using monotonic time (performance.now() in JS)
function measureOperationGood(): number {
  const start = performance.now();
  doOperation();
  const end = performance.now();
  return end - start;  // Always non-negative
}
 
// BAD: Last-write-wins using wall clock
interface Record {
  value: string;
  timestamp: number;
}
 
function mergeRecordsBad(a: Record, b: Record): Record {
  return a.timestamp > b.timestamp ? a : b;  // Sensitive to clock skew!
}
 
// GOOD: Using vector clock for conflict detection
interface VectorClock {
  [nodeId: string]: number;
}
 
interface RecordWithVector {
  value: string;
  vectorClock: VectorClock;
}
 
function mergeRecordsGood(a: RecordWithVector, b: RecordWithVector): RecordWithVector[] {
  if (happensBefore(a.vectorClock, b.vectorClock)) return [b];
  if (happensBefore(b.vectorClock, a.vectorClock)) return [a];
  return [a, b];  // Concurrent - return both for conflict resolution
}
 
// BAD: Distributed lock with only TTL
async function acquireLockBad(key: string, ttlMs: number): Promise<boolean> {
  const acquired = await redis.set(key, 'locked', 'NX', 'PX', ttlMs);
  return acquired === 'OK';
}
 
// GOOD: Distributed lock with fencing token
async function acquireLockGood(key: string, ttlMs: number): Promise<number | null> {
  const token = await redis.incr('lock:token:' + key);
  const acquired = await redis.set(
    key, 
    JSON.stringify({ token, holder: nodeId }), 
    'NX', 
    'PX', 
    ttlMs
  );
  return acquired === 'OK' ? token : null;
}
 
// When using the lock, pass fencing token to downstream:
async function writeWithFencingToken(value: string, fencingToken: number) {
  // Storage system rejects writes with lower token
  await database.write(value, { fencingToken });
}

Review All Wall Clock Usage

Search your codebase for Date.now(), System.currentTimeMillis(), time.time(), and similar calls. Each one is a potential clock skew vulnerability. Most should be replaced with monotonic time or logical clocks.

Observing Clock-Related Failures

Clock-related failures are notoriously difficult to observe because they often don't cause visible errors—just incorrect behavior. Special instrumentation is needed to detect and diagnose these issues.

What to Monitor During Clock Experiments

•NTP offset and sync status — Export chronyd/ntpd offset metrics. Alert when offset exceeds threshold.
•Clock comparison across nodes — Periodically log wall clock time from all nodes. Compare for drift.
•Negative duration errors — Log when elapsed time calculations produce negative values.
•Timestamp ordering violations — Log when events arrive with timestamps older than previously seen.
•TTL-related anomalies — Monitor for premature expiration (clock ahead) or stale data (clock behind).
•Scheduled job execution times — Track if jobs run early, late, or are skipped entirely.
•Authentication failures — Monitor for spikes in token validation failures that might indicate clock issues.

clock-monitoring.sh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
# Export NTP metrics to Prometheus
 
# For chronyd, use chrony_exporter
# https://github.com/superq/chrony_exporter
 
# Key metrics to monitor:
# - chrony_tracking_stratum: NTP stratum level
# - chrony_tracking_root_delay_seconds: delay to root time source
# - chrony_tracking_root_dispersion_seconds: dispersion from root
# - chrony_tracking_system_time: current system time adjustment
 
# Alert on clock skew
cat <<EOF | kubectl apply -f -
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: clock-skew-alerts
spec:
  groups:
    - name: clock
      rules:
        - alert: ClockSkewHigh
          expr: abs(chrony_tracking_system_time) > 0.1
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "Clock skew > 100ms on {{ $labels.instance }}"
        
        - alert: NTPNotSynced
          expr: chrony_tracking_stratum == 0
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "NTP not synced on {{ $labels.instance }}"
EOF
 
# Log cluster clock comparison (run on each node)
#!/bin/bash
while true; do
  TIMESTAMP=$(date +%s.%N)
  NTP_OFFSET=$(chronyc tracking | grep "System time" | awk '{print $4}')
  echo "{"node": "$(hostname)", "time": $TIMESTAMP, "offset": $NTP_OFFSET}"
  sleep 10
done

Symptoms and Diagnosis of Clock Issues
Symptom	Possible Clock Issue	Diagnostic Steps
Tokens expire immediately	Local clock is ahead	Compare local time to token issue time
Expired tokens accepted	Local clock is behind	Check NTP sync status
Cache entries evict too soon	Clock skew in cache cluster	Compare time across cache nodes
Events appear out of order in logs	Clock skew between producers	Add source timestamp to log entries
Last-write-wins gives wrong result	Writer clock is skewed	Use vector clocks instead
Distributed lock contention	Lease timing incorrect	Add fencing tokens to locks
Scheduled jobs run early/late	Scheduler clock drifted	Check NTP offset on scheduler nodes

Clock Skew

The Hidden Enemy: Time in Distributed Systems

Understanding Time in Distributed Systems

Clock Skew Injection

Time Jumps

NTP Failures

Leap Seconds and Calendar Edge Cases

Patterns Vulnerable to Clock Issues

Observing Clock-Related Failures

There’s more to understand.

Clock Skew

The Hidden Enemy: Time in Distributed Systems

Understanding Time in Distributed Systems

Clock Skew Injection

Time Jumps

NTP Failures

Leap Seconds and Calendar Edge Cases

Patterns Vulnerable to Clock Issues

Observing Clock-Related Failures

There’s more to understand.