Network Troubleshooting

If you had to troubleshoot networks with only two tools, you'd choose ping and traceroute. These deceptively simple utilities answer the two most fundamental questions in network troubleshooting:

• Ping: Can I reach the destination, and how long does it take?
• Traceroute: What path do packets take to get there, and where might they fail?

Most engineers use these tools daily but never develop true mastery. They know how to run the commands but not how to extract deep diagnostic insights from the results. This page takes you from basic usage to expert-level interpretation.

Both ping and traceroute rely on the Internet Control Message Protocol (ICMP). Understanding ICMP is essential for interpreting results correctly and understanding tool limitations.

ICMP Overview:

ICMP is a Layer 3 protocol encapsulated directly in IP (IP protocol number 1). Unlike TCP and UDP, ICMP doesn't carry application data—it carries control and error messages about network conditions.

ICMP Message Types Used in Troubleshooting:

ICMP Echo Request/Reply Structure:

+-------------------+-------------------+
|   Type (1 byte)   |   Code (1 byte)   |
+-------------------+-------------------+
|          Checksum (2 bytes)           |
+------------------+--------------------+
|  Identifier (2)  |   Sequence (2)    |
+------------------+--------------------+
|              Data (variable)          |
+---------------------------------------+

• Type/Code: 8/0 for request, 0/0 for reply
• Identifier: Matches requests with replies (usually process ID)
• Sequence Number: Increments with each ping, correlates specific packets
• Data: Optional payload (used for timing and size testing)

ICMP and Firewalls:

ICMP is frequently filtered by firewalls. This creates complications:

Always consider that ICMP tools may be affected by filtering, not just network issues.

Beyond basic reachability testing, ping provides detailed metrics for network analysis.

Ping Metrics Explained:

1. Round-Trip Time (RTT):

RTT measures the time from sending the Echo Request to receiving the Echo Reply. It includes:

• Propagation delay (speed of light through medium)
• Transmission delay (time to push bits onto wire)
• Processing delay (router/host processing)
• Queuing delay (waiting in buffers)

Typical baseline RTT values:

2. Packet Loss:

Percentage of pings that don't receive replies. Causes include:

• Congestion (buffers overflow)
• Physical issues (bad cables, interference)
• Firewall filtering (sometimes rate-limited)
• Host overload (can't respond to all pings)

3. Jitter (RTT Variation):

The difference between minimum and maximum RTT, or the variance across pings. High jitter indicates inconsistent network performance.

# Example ping statistics analysis
64 bytes from 8.8.8.8: icmp_seq=1 ttl=117 time=12.3 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=117 time=14.1 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=117 time=11.8 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=117 time=45.2 ms  <- Spike!
64 bytes from 8.8.8.8: icmp_seq=5 ttl=117 time=12.0 ms

Statistics:
- Min: 11.8ms, Max: 45.2ms, Avg: 19.1ms
- Jitter: 45.2 - 11.8 = 33.4ms (concerning)

4. TTL (Time to Live):

The TTL in the reply indicates how many hops are left after traversing the path. Different OS default TTL values:

If you ping a host and get TTL=52, and Linux uses TTL=64, the host is approximately 12 hops away (64 - 52 = 12).

Expert interpretation goes beyond 'it works' or 'it doesn't.' Each result pattern tells a story.

Pattern 1: Complete Failure

C:\> ping 192.168.1.100

Pinging 192.168.1.100 with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.

Analysis:

• No ICMP errors returned, just silence
• Could be: host down, firewall dropping silently, routing black hole
• Next step: traceroute to see where packets stop, check ARP for local hosts

Pattern 2: Destination Unreachable

C:\> ping 10.99.99.99

Pinging 10.99.99.99 with 32 bytes of data:
Reply from 192.168.1.1: Destination host unreachable.

Analysis:

• A router (192.168.1.1) is responding with ICMP error
• Router can't find valid route or ARP for destination
• The router IP tells you where the problem is
• Next step: Check routing on that router

Pattern 3: Intermittent Loss

64 bytes from 8.8.8.8: icmp_seq=1 ttl=117 time=12.3 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=117 time=11.8 ms
Request timed out.
64 bytes from 8.8.8.8: icmp_seq=4 ttl=117 time=12.1 ms
Request timed out.
64 bytes from 8.8.8.8: icmp_seq=6 ttl=117 time=11.9 ms

Analysis:

• Some packets succeed, some fail
• Could be: congestion, flapping link, ICMP rate limiting
• If loss is regular (every Nth packet), likely rate limiting
• If random, likely congestion or physical issue
• Next step: mtr for continuous path analysis

Pattern 4: High Latency Spikes

64 bytes from target: icmp_seq=1 ttl=117 time=11.2 ms
64 bytes from target: icmp_seq=2 ttl=117 time=11.5 ms
64 bytes from target: icmp_seq=3 ttl=117 time=312.6 ms  <- Spike!
64 bytes from target: icmp_seq=4 ttl=117 time=11.3 ms

Analysis:

• Occasional massive latency spikes
• Classic symptom of buffer bloat or momentary congestion
• One packet queued behind large transfers
• Common in residential connections during uploads
• Next step: Test with background load (simulates real usage)

Pattern 5: TTL Changes

64 bytes from target: icmp_seq=1 ttl=117 time=12.1 ms
64 bytes from target: icmp_seq=2 ttl=116 time=13.2 ms
64 bytes from target: icmp_seq=3 ttl=117 time=11.9 ms
64 bytes from target: icmp_seq=4 ttl=116 time=14.1 ms

Analysis:

• TTL alternates between 117 and 116
• Different paths with different hop counts (load balancing)
• ECMP (Equal-Cost Multi-Path) routing in effect
• If RTT also alternates, different path characteristics

Traceroute reveals the path packets take through the network. It exploits the TTL field in IP headers—each router decrements TTL and, when it hits zero, returns an ICMP TTL Exceeded message.

How Traceroute Works:

1. Send packet with TTL=1
2. First router decrements to 0, returns ICMP TTL Exceeded
3. Note router IP and round-trip time
4. Send packet with TTL=2
5. Second router returns TTL Exceeded
6. Continue until destination responds (or max hops reached)

Traceroute Variants:

The probe type matters because different protocols are filtered differently by firewalls.

Traceroute output requires careful interpretation. Many patterns that look problematic are actually normal.

Sample Traceroute Analysis:

traceroute to example.com (93.184.216.34), 30 hops max
  1  gateway (192.168.1.1)  1.234 ms  0.987 ms  0.923 ms
  2  10.0.0.1             12.432 ms  11.876 ms  12.001 ms
  3  isp-edge.example.net  15.234 ms  14.987 ms  15.123 ms
  4  core-router.peer.net  25.432 ms  24.876 ms  25.001 ms
  5  * * *
  6  cdn-edge.example.net  23.234 ms  22.987 ms  23.123 ms
  7  93.184.216.34         24.432 ms  23.876 ms  24.001 ms

Understanding Each Line:

Key Insight: Hop 5's asterisks don't indicate a problem!

That router simply doesn't respond to traceroute probes (ICMP filtered). Since hops 6 and 7 respond, connectivity is fine.

Common Traceroute Patterns:

Pattern 1: * * * at Intermediate Hops (Usually Normal)

  4  core-router.isp.net    25.1 ms  24.8 ms  25.0 ms
  5  * * *
  6  * * *
  7  edge-server.cdn.net    28.4 ms  27.9 ms  28.2 ms

Interpretation: Hops 5-6 are routers that don't respond to traceroute. Since the trace continues and completes, this is normal filtering, not a problem.

Pattern 2: * * * to End (Potential Problem)

  4  core-router.isp.net    25.1 ms  24.8 ms  25.0 ms
  5  peering.exchange.net   35.2 ms  34.9 ms  35.1 ms
  6  * * *
  7  * * *
  8  * * *
  ...

Interpretation: After hop 5, nothing responds. Could be:

• Destination is down
• Firewall blocking all probes
• Routing issue beyond hop 5
• Try ICMP vs TCP probes to confirm

Pattern 3: Sudden Latency Jump

  3  isp-core.example.net    15.2 ms  14.9 ms  15.1 ms
  4  trans-atlantic.net      95.3 ms  94.8 ms  95.0 ms  <- 80ms jump!
  5  eu-router.example.net   98.4 ms  97.9 ms  98.1 ms

Interpretation: 80ms jump at hop 4 indicates a long-distance link (continental hop). This is normal physics (speed of light across Atlantic). RTT stays in that range afterward.

mtr (My Traceroute) combines the functionality of ping and traceroute, providing continuous statistics for each hop. It's the gold standard for diagnosing intermittent path issues.

Sample mtr Output:

                              My traceroute  [v0.95]
host                    Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. gateway              0.0%    50    0.5   0.4   0.3   0.8   0.1
 2. isp-access           0.0%    50   11.2  10.8   9.5  15.2   1.2
 3. isp-core             0.0%    50   14.5  14.2  12.8  18.9   1.5
 4. peering              0.0%    50   25.3  24.8  22.5  32.1   2.1
 5. ???                   ???     0    0.0   0.0   0.0   0.0   0.0
 6. cdn-edge             0.0%    50   28.2  27.5  25.8  35.4   2.3
 7. destination          0.0%    50   29.1  28.4  26.5  36.2   2.4

Reading mtr Output:

Interpreting mtr Statistics:

The Loss Pattern Analysis:

Scenario 1: Loss at intermediate hop, not at destination

 4. problematic-router    15.0%    50   25.1  24.8  22.5  35.1   2.5
 5. next-router            0.0%    50   28.2  27.5  25.8  35.4   2.3
 6. destination            0.0%    50   29.1  28.4  26.5  36.2   2.4

Interpretation: Hop 4 shows 15% loss, but destination shows 0%. This router likely deprioritizes or rate-limits ICMP responses. It's NOT actually dropping traffic—if it were, the destination would also show loss.

Scenario 2: Loss starts at hop and continues to end

 4. problematic-router    12.0%    50   25.1  24.8  22.5  35.1   2.5
 5. next-router           12.0%    50   28.2  27.5  25.8  35.4   2.3
 6. destination           12.0%    50   29.1  28.4  26.5  36.2   2.4

Interpretation: Loss begins at hop 4 and persists (same percentage) through all subsequent hops. Hop 4 is genuinely dropping packets. This is a real problem.

Scenario 3: High jitter (StDev) at specific hop

 3. stable-router          0.0%    50   14.5  14.2  12.8  16.9   0.8
 4. unstable-router        0.0%    50   75.1  44.8  22.5 195.1  45.2  <- High StDev!
 5. next-router            0.0%    50   78.2  47.5  25.8 198.4  46.3

Interpretation: Hop 4 has 0% loss but massive jitter (StDev=45.2). This indicates buffer bloat or intermittent congestion at that router. The jitter propagates to subsequent hops.

Maximum diagnostic power comes from using both tools strategically together.

Diagnostic Workflow:

1. Symptom: "Can't reach server X"
        |
        v
2. [ping server-x]
        |
   +----+----+
   |         |
   Success   Failure
   |         |
   v         v
3. ping works, but    [traceroute server-x]
   app doesn't:             |
   - Check ports      Where does it stop?
   - Check app              |
   - Check DNS        +-----+-----+
                      |           |
               Mid-path      Never reaches
               death         destination
               (hop N * * *       |
               to end)      Routing/firewall
                   |         issue
                   v
           [ping hop-N directly]
                   |
4.            Does it respond?
              Yes: routing issue beyond hop N
              No: hop N device issue

Workflow Example - Diagnosing 'Slow Website':

Step 1: Baseline ping

$ ping website.com
ROund-trip min/avg/max = 11.2/145.8/312.4 ms
Packet loss = 5%

High average, high variance, some loss. Problem confirmed.

Step 2: Traceroute to find where

$ traceroute website.com
  1  gateway           0.5 ms
  2  isp-access       11.2 ms
  3  isp-core         14.5 ms
  4  peering          25.3 ms
  5  congested-hop   312.8 ms  <- HUGE jump!
  6  cdn-edge        315.2 ms
  7  website.com     318.4 ms

Hop 5 shows 290ms jump. Problem isolated.

Step 3: Ping problematic hop directly

$ ping 10.0.5.1 -c 100
5% packet loss, avg=285ms, max=450ms

Confirmed: hop 5 is the problem.

Step 4: Identify ownership

$ whois 10.0.5.1
[Shows ISP/upstream provider info]

Now you know who to contact.

Step 5: Use mtr for evidence

$ mtr -rwc 100 website.com > mtr-report.txt

Statistical evidence to share with provider.

Ping and traceroute are foundational tools that, when truly mastered, provide extraordinary diagnostic power. Let's consolidate the key insights:

What's Next:

With connectivity and path analysis covered, we'll dive into packet capture tools. When ping and traceroute don't explain the problem, capturing actual packets reveals exactly what's happening at the protocol level.