Computer NetworksPing and Traceroute

Ping and Traceroute: Network Diagnostic Tools

LevelIntermediate

Duration75 mins

TopicPing and Traceroute

5 / 5

Diagnostic Usage: Professional Network Troubleshooting

From Tools to Mastery

You now possess deep knowledge of ICMP Echo, ping, traceroute, and TTL manipulation. But knowing individual tools is different from knowing how to solve problems. Real network issues don't announce which tool will fix them—you face symptoms like "the website is slow," "users can't connect," or "something changed and now it's broken."

This final page transforms your tool knowledge into diagnostic methodology—the systematic approach expert network engineers use to:

Quickly isolate connectivity vs. performance vs. configuration issues
Determine the responsible party (local network, ISP, destination)
Gather evidence suitable for escalation to vendors or ISPs
Document findings professionally
Monitor for and prevent future occurrences

Effective diagnosis isn't about running random commands—it's about forming hypotheses, testing them efficiently, and converging on root causes. The techniques here represent years of professional experience distilled into actionable methodologies.

What You Will Master

By the end of this page, you will command systematic troubleshooting frameworks, know how to diagnose common network problem categories efficiently, integrate multiple diagnostic tools effectively, prepare professional-quality troubleshooting documentation, and understand proactive monitoring strategies using these tools.

The Systematic Troubleshooting Framework

Professional network engineers follow structured methodologies rather than ad-hoc testing. The most effective framework combines the OSI layer approach with binary search isolation:

The Bottom-Up OSI Approach:

Start at the physical layer and work upward. If lower layers fail, higher layers cannot work—so confirming lower layers first saves time:

OSI Layer Diagnostic Progression
Layer	What to Check	Diagnostic Method	Common Failures
Physical	Cable, port, link light	Visual inspection, cable tester	Damage, loose connection, bad port
Data Link	MAC address, VLAN, switch port	`arp -a`, switch logs, spanning tree	Wrong VLAN, MAC conflict, STP blocking
Network	IP config, routing, connectivity	`ping`, `traceroute`, `ip route`	Wrong IP, missing route, firewall block
Transport	Ports, connections, sockets	`netstat`, `ss`, `tcpdump`	Port closed, firewall, connection refused
5-7. Application	Service status, config, logs	Service-specific tools, logs	Wrong config, crash, resource exhaustion

The Binary Search Isolation Technique:

When you have a path with multiple segments, cut it in half repeatedly to find the failure point:

Binary Search Path Isolation
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# Problem: Cannot reach external website from workstation
# Path: Workstation → Switch → Router → ISP → Internet → Website
 
# Step 1: Test middle point (Router/Gateway)
ping 192.168.1.1
# ✓ Works → Problem is beyond router
# ✗ Fails → Problem is between workstation and router
 
# Assuming Step 1 works, test next middle point:
# Step 2: Test ISP/First external hop  
ping 10.0.0.1  # ISP gateway from traceroute
# ✓ Works → Problem is beyond ISP
# ✗ Fails → Problem is router ↔ ISP link
 
# Continue halving until problem is isolated to single link/device
 
# This is more efficient than sequential testing:
# Binary search: O(log n) tests to find failure in n segments
# Sequential: O(n) tests in worst case
 
# Example: 8-hop path
# Sequential worst case: 8 tests
# Binary search worst case: 3 tests (log₂ 8 = 3)

The STOP-THINK-TEST Framework

•STOP: Don't immediately start running commands. Gather information about the symptoms first. What exactly isn't working? When did it stop? What changed?
•THINK: Form a hypothesis. Based on symptoms, what are the most likely causes? Where in the path would those causes manifest?
•TEST: Run the minimum diagnostic to confirm or eliminate your hypothesis. One clear test is worth ten ambiguous ones.
•Repeat: Based on test results, refine hypothesis and test again. Converge systematically on root cause.

The 'What Changed?' Question

Before running any diagnostics, always ask: 'What changed?' Networks that worked yesterday don't spontaneously break—something changed. Recent deployments, config changes, patches, hardware replacements, ISP maintenance, certificate expirations—identifying the change often points directly to the cause.

Common Diagnostic Scenarios and Solutions

Let's work through the most common network problems you'll encounter, with step-by-step diagnostic procedures:

Scenario 1: "I can't reach the website"

Scenario 1: Website Unreachable
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
# Step 1: Determine if it's DNS or connectivity
nslookup www.example.com
# Output: 93.184.216.34
 
# Step 2: Ping the IP directly (bypass DNS caching issues)
ping 93.184.216.34
# ✗ 100% packet loss
 
# Step 3: Test basic connectivity infrastructure
ping 8.8.8.8  # Well-known Google DNS
# ✓ Works → Internet connectivity exists
# ✗ Fails → General connectivity issue (go to local troubleshooting)
 
# Step 4: Traceroute to problematic destination
traceroute 93.184.216.34
 1  192.168.1.1 (gateway)    1.2 ms
 2  10.0.0.1                  8.5 ms
 3  ae-5.r21.lsanca07...     12.3 ms
 4  * * *
 5  * * *
 ...
 
# Diagnosis: Path fails at hop 4 (after hop 3 which is ISP backbone)
# Likely cause: Routing issue at ISP level, destination network issue,
#               or destination firewall blocking ICMP
 
# Step 5: Test if web service is reachable even if ping blocked
curl -I --connect-timeout 5 https://www.example.com
# HTTP/1.1 200 OK → Website is fine! They just block ping.
# timeout → Website truly unreachable
 
# Conclusion paths:
# - If curl works but ping fails: Destination blocks ICMP (normal)
# - If both fail: Real connectivity issue at identified hop
# - If DNS failed: DNS server issue, not connectivity

Scenario 2: "The connection is slow"

Scenario 2: Slow Connection
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
# Step 1: Quantify "slow" - measure actual latency
ping -c 20 target.example.com
# Output: rtt min/avg/max/mdev = 45.2/156.8/892.3/234.7 ms
 
# Analysis: Average 156ms with huge variance (mdev 234ms)
# This indicates severe jitter/congestion, not just distance
 
# Step 2: Identify where latency is added
mtr -r -c 50 target.example.com
 
#                              Loss%  Snt  Last  Avg  Best Wrst StDev
# 1. router.local               0.0%   50   0.5  0.6  0.3  1.2  0.2
# 2. 10.0.0.1                   0.0%   50   8.2  8.5  7.9 12.3  0.8
# 3. ae-5.r21.lsanca07...       0.0%   50  12.1 12.4 11.8 15.6  0.6
# 4. congested-link.isp.net     5.0%   50  85.3 98.7 25.4 456.2 89.3 # ← PROBLEM
# 5. 72.14.196.226              4.8%   50  92.4 105.3 28.1 478.9 92.1
# 6. target.example.com         4.9%   50  95.2 108.7 30.2 485.3 95.8
 
# Analysis:
# - Hop 4 shows huge jump (12ms → 98ms average)
# - High StDev at hop 4 (89.3ms) indicates severe variance
# - Loss starts at hop 4 (5%)
# - Problem persists through remaining hops
 
# Diagnosis: Congestion at hop 4 (ISP link)
# Action: Report to ISP with MTR data as evidence
 
# Step 3: Correlate with time of day
# Run periodic tests to identify patterns
while true; do
    echo "$(date): $(ping -c 1 -W 2 target.example.com | grep time=)"
    sleep 60
done > latency_log.txt
 
# If latency is high only during business hours: bandwidth contention
# If latency is high only at night: different user patterns (streaming?)
# If latency is always high: capacity issue or misconfiguration

Scenario 3: "It was working, now it's broken"

Scenario 3: Sudden Failure
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
# The most common scenario: something changed!
 
# Step 1: Test basic connectivity
ping google.com
# ✗ ping: unknown host google.com
 
# Step 2: Test by IP (bypass DNS)
ping 8.8.8.8
# ✓ 64 bytes from 8.8.8.8: icmp_seq=1 ttl=116 time=15.2 ms
 
# Diagnosis: DNS is broken, not connectivity
# Check DNS configuration:
cat /etc/resolv.conf
# nameserver 192.168.1.53  # Internal DNS server
 
ping 192.168.1.53
# ✗ Request timeout
 
# Root cause: Internal DNS server is down
# Solution: Fix DNS server or use alternative (temporarily 8.8.8.8)
 
# Alternative scenario: IP works but different path
# Compare current traceroute to known-good baseline
 
# Current:
traceroute 10.20.30.40
 1  192.168.1.1    0.5 ms
 2  10.0.0.1       8.2 ms
 3  172.16.0.1    45.3 ms   # NEW: unexpected router!
 4  10.20.30.40   78.9 ms   # Much slower total
 
# Baseline (from documentation):
# 1  192.168.1.1    0.5 ms
# 2  10.10.10.1     5.2 ms  # Different router - old path
# 3  10.20.30.40   12.3 ms
 
# Diagnosis: Routing changed. Traffic now goes through 172.16.0.1
# instead of 10.10.10.1, adding latency.
# Action: Check routing tables, BGP sessions, link status on old path

Scenario 4: Intermittent Issues

Intermittent problems are the hardest to diagnose. Use continuous monitoring (MTR, long-running ping) to capture the issue when it occurs. Log with timestamps to correlate with other events. Packet captures (tcpdump/Wireshark) may be needed for transient issues. Set up alerts to notify you when the problem recurs.

Integrating Multiple Diagnostic Tools

Real troubleshooting rarely uses a single tool. Effective diagnosis combines ping, traceroute, and TTL analysis with complementary tools:

The Core Diagnostic Toolkit:

Complementary Diagnostic Tools
Tool	Purpose	When to Use
`ping`	Basic reachability, RTT measurement	First test for any connectivity issue
`traceroute/mtr`	Path mapping, per-hop analysis	When you need to know WHERE in the path
`dig/nslookup`	DNS resolution testing	When names don't resolve or resolve wrong
`curl/wget`	Application-layer testing	When ICMP blocked but services may work
`telnet/nc`	Port connectivity testing	Testing specific TCP/UDP port access
`ss/netstat`	Local socket/connection state	Connection issues on local host
`tcpdump/wireshark`	Packet capture and analysis	Deep inspection, timing issues, protocol errors
`arp -a`	ARP cache inspection	Layer 2 issues, wrong MAC associations
`ip route`	Routing table inspection	Checking local routing decisions

Combined Diagnostic Script
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
#!/bin/bash
# comprehensive_diag.sh - Full connectivity diagnostic
 
TARGET=$1
if [ -z "$TARGET" ]; then
    echo "Usage: $0 <target_hostname_or_ip>"
    exit 1
fi
 
echo "=========================================="
echo "  Comprehensive Network Diagnostic"
echo "  Target: $TARGET"
echo "  Time: $(date)"
echo "=========================================="
 
# Step 1: DNS Resolution
echo -e "
[1/6] DNS Resolution"
echo "-------------------"
if host "$TARGET" > /dev/null 2>&1; then
    IP=$(host "$TARGET" | grep "has address" | head -1 | awk '{print $4}')
    echo "✓ Resolved: $TARGET → $IP"
else
    echo "✗ DNS resolution failed for $TARGET"
    echo "  Testing DNS server connectivity..."
    ping -c 1 -W 2 8.8.8.8 > /dev/null 2>&1 && echo "  (8.8.8.8 reachable - DNS server may be down)" || echo "  (No internet connectivity?)"
    exit 1
fi
 
# Step 2: ICMP Reachability
echo -e "
[2/6] ICMP Reachability (ping)"
echo "-----------------------------"
PING_RESULT=$(ping -c 5 -W 2 "$IP" 2>&1)
if echo "$PING_RESULT" | grep -q "5 received"; then
    echo "✓ Ping successful:"
    echo "$PING_RESULT" | grep -E "^(PING|rtt|5 packets)"
else
    echo "✗ Ping failed or packet loss detected:"
    echo "$PING_RESULT" | grep -E "^(PING|--- |5 packets|rtt)"
fi
 
# Step 3: Path Analysis
echo -e "
[3/6] Path Analysis (traceroute)"
echo "--------------------------------"
traceroute -n -w 2 -q 1 "$IP" 2>&1 | head -20
 
# Step 4: TCP Port Check (common ports)
echo -e "
[4/6] TCP Port Connectivity"
echo "--------------------------"
for PORT in 80 443 22; do
    if timeout 2 bash -c "echo > /dev/tcp/$IP/$PORT" 2>/dev/null; then
        echo "✓ Port $PORT: Open"
    else
        echo "✗ Port $PORT: Closed/Filtered"
    fi
done
 
# Step 5: HTTP(S) Check
echo -e "
[5/6] HTTP/HTTPS Connectivity"
echo "----------------------------"
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" --connect-timeout 5 "http://$TARGET" 2>/dev/null)
HTTPS_CODE=$(curl -s -o /dev/null -w "%{http_code}" --connect-timeout 5 "https://$TARGET" 2>/dev/null)
echo "HTTP response code: $HTTP_CODE"
echo "HTTPS response code: $HTTPS_CODE"
 
# Step 6: Summary
echo -e "
[6/6] Diagnostic Summary"
echo "------------------------"
echo "Target: $TARGET ($IP)"
echo "Timestamp: $(date +%Y-%m-%d_%H:%M:%S)"
echo "Ping: $(echo "$PING_RESULT" | grep -oE '[0-9]+% packet loss')"
echo "HTTP: $HTTP_CODE | HTTPS: $HTTPS_CODE"
echo "=========================================="

Tool Selection Decision Tree:

Problem reported
      │
      ▼
┌─────────────────────────────────────────┐
│  Can you ping the default gateway?      │
└─────────────────────────────────────────┘
      │
      ├── NO → Local network issue
      │         • Check cable/WiFi
      │         • Check IP config (ip addr)
      │         • Check ARP (arp -a)
      │
      └── YES → Continue
              │
              ▼
      ┌─────────────────────────────────────────┐
      │  Can you ping 8.8.8.8 (Internet)?       │
      └─────────────────────────────────────────┘
              │
              ├── NO → Gateway/ISP issue
              │         • traceroute 8.8.8.8
              │         • Check gateway config
              │
              └── YES → Continue
                      │
                      ▼
              ┌─────────────────────────────────────────┐
              │  Can you resolve DNS names?              │
              └─────────────────────────────────────────┘
                      │
                      ├── NO → DNS issue
                      │         • Check /etc/resolv.conf
                      │         • Test dig @8.8.8.8 name
                      │
                      └── YES → Destination-specific issue
                                • traceroute to target
                                • curl/telnet to target
                                • tcpdump if needed

The Power of Comparison

Always compare problematic hosts to working ones. If Server A is unreachable but Server B in the same subnet works, the problem is server-specific. If neither works, it's network-wide. Comparison testing isolates variables rapidly.

Gathering Evidence for Escalation

When the problem exists outside your control (ISP issue, vendor service, partner network), you need compelling evidence to get action. Poor escalations get ignored; well-documented ones get resolved.

Essential Elements of an Escalation Package:

Professional Escalation Checklist

•Clear problem statement: One sentence summarizing the issue (e.g., 'Complete packet loss to your server 203.0.113.50 from our network 198.51.100.0/24')
•Impact description: What's affected and how severely (e.g., 'Production e-commerce site unavailable, affecting 10,000 customers/hour')
•Timeline: When it started, any intermittency pattern (e.g., 'Started 2024-01-15 09:45 UTC, still ongoing')
•Source information: Your public IP(s), ASN if relevant, geographic location
•Diagnostic output: Ping results, traceroute output, MTR data—with timestamps
•Comparisons: Evidence from different sources if available ('traceroute from backup site succeeds')
•What you've already tried: Show competence and eliminate obvious suggestions

Escalation Report Template

Text

Subject: Packet Loss to [Destination] from AS[XXXXX] - Ticket #[REF]
 
Hi [ISP/Vendor] NOC,
 
We are experiencing complete connectivity loss from our network to your infrastructure.
 
ISSUE SUMMARY:
- Problem: 100% packet loss to 203.0.113.50 from 198.51.100.0/24
- Impact: Production payment gateway unavailable
- Start Time: 2024-01-15 09:45:23 UTC
- Current Status: Ongoing as of 2024-01-15 10:15:00 UTC
 
SOURCE INFORMATION:
- Our Public IP: 198.51.100.10
- Our ASN: AS12345
- Location: San Francisco, CA, USA
- ISP: Example Communications
 
DIAGNOSTIC EVIDENCE:
 
1. PING OUTPUT (current):
$ ping -c 10 203.0.113.50
PING 203.0.113.50 (203.0.113.50) 56(84) bytes of data.
--- 203.0.113.50 ping statistics ---
10 packets transmitted, 0 received, 100% packet loss, time 9215ms
 
2. MTR OUTPUT (50 samples):
HOST                      Loss%  Snt  Last  Avg  Best Wrst StDev
1. 198.51.100.1             0.0%  50   0.5  0.6  0.3  1.2  0.1
2. 10.0.0.1                 0.0%  50   5.2  5.4  5.0  8.1  0.4
3. edge-router.isp.net      0.0%  50  12.3 12.5 12.0 15.2  0.5
4. your-peering.isp.net   100.0%  50   0.0  0.0  0.0  0.0  0.0 <-- Loss starts
5. ???                    100.0%  50   0.0  0.0  0.0  0.0  0.0
 
3. TRACEROUTE OUTPUT:
$ traceroute -n 203.0.113.50
 1  198.51.100.1     0.5 ms
 2  10.0.0.1         5.3 ms
 3  68.87.64.1      12.4 ms
 4  * * *
 5  * * *
 
4. COMPARISON (working destination on same path):
$ ping -c 5 203.0.113.100  (different host, same /24)
64 bytes from 203.0.113.100: icmp_seq=1 ttl=56 time=45.2 ms
[Works fine - issue is specific to .50]
 
ACTIONS WE HAVE TAKEN:
- Verified our local routing and firewall (no changes, other traffic works)
- Tested from secondary ISP connection (same result)
- Confirmed DNS resolution works correctly
- Rebooted our edge router (no effect)
 
REQUEST:
Please investigate connectivity to 203.0.113.50 from the direction of 
68.87.64.1 (our ISP's peering point with your network).
 
Contact: [Your Name]
Phone: [+1-XXX-XXX-XXXX]
Ticket Ref: [Internal ticket number]

Getting Faster Response

ISP NOC teams receive hundreds of tickets daily. Tickets with clear evidence, specific details, and professional formatting get prioritized. Vague reports like 'your network is slow' get queued behind others. Include all evidence in the first message—back-and-forth requests for details add hours or days.

Proactive Monitoring and Baseline Establishment

The best troubleshooting happens before problems occur. Proactive monitoring using ping-based tools establishes baselines, detects degradation early, and provides historical data for comparison during incidents.

Key Metrics to Monitor:

Essential Connectivity Metrics
Metric	What It Shows	Alert Threshold (Example)	Collection Method
RTT (latency)	Network delay	2× baseline average	`ping` with timestamp logging
Jitter (RTT variance)	Latency consistency	StdDev > 20% of average	MTR or custom ping parser
Packet loss %	Reliability	0.1% over 5 minutes	Continuous ping with loss count
Path changes	Routing stability	Any change from baseline	Periodic traceroute comparison
DNS resolution time	Lookup performance	500ms average	`dig` with +stats option

Simple Monitoring Script
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
#!/bin/bash
# monitor_connectivity.sh - Continuous monitoring with alerting
 
TARGETS="8.8.8.8 google.com production-server.example.com"
LOG_DIR="/var/log/network_monitor"
ALERT_EMAIL="oncall@example.com"
CHECK_INTERVAL=60  # seconds
 
mkdir -p "$LOG_DIR"
 
monitor_target() {
    local target=$1
    local timestamp=$(date +%Y-%m-%d_%H:%M:%S)
    local log_file="$LOG_DIR/${target//[.:]/_}.log"
    
    # Run ping
    result=$(ping - c 5 - W 2 "$target" 2 >& 1)
    loss=$(echo "$result" | grep - oE '[0-9]+% packet loss' | grep - oE '[0-9]+')
    avg_rtt=$(echo "$result" | grep - oE 'avg[^=]*= [0-9.]+' | grep - oE '[0-9.]+')
    
    # Log result
    echo "$timestamp | loss=$loss% | avg_rtt=${avg_rtt}ms" >> "$log_file"
    
    # Alert on issues
    if ["$loss" - gt 10]; then
        echo "ALERT: $target - $loss% packet loss at $timestamp" |                                 mail - s "Network Alert: High packet loss to $target" "$ALERT_EMAIL"
    fi
    
    if(($(echo "$avg_rtt > 200" | bc - l 2 > /dev/null || echo 0))); then
        echo "ALERT: $target - High latency ${avg_rtt}ms at $timestamp" |     mail - s "Network Alert: High latency to $target" "$ALERT_EMAIL"
    fi
}
 
echo "Starting network monitoring..."
echo "Targets: $TARGETS"
echo "Interval: ${CHECK_INTERVAL}s"
echo "Logs: $LOG_DIR"
 
while true; do
    for target in $TARGETS; do
        monitor_target "$target" &
    done
    wait
    sleep "$CHECK_INTERVAL"
done

Baseline Establishment:

Before you can detect anomalies, you need to know what's normal:

# Collect baseline data over 24 - 48 hours of normal operation
mtr - r - c 100 - i 1 target.example.com > baseline_mtr.txt

# Capture hourly averages to understand daily patterns
for hour in $(seq 0 23); do 
    echo "Hour $hour:" >> hourly_baseline.txt
    ping -c 60 target.example.com | tail -1 >> hourly_baseline.txt
done

# Store baseline traceroute for path comparison
traceroute -n target.example.com > baseline_traceroute.txt

What goes into your baseline documentation:

Target	Normal RTT	Normal Loss	Expected Path	Notes
8.8.8.8	15-25ms	<0.01%	4 hops	Google DNS, always available
production-db	2-5ms	0%	2 hops	Same datacenter
partner-api.com	45-60ms	<0.1%	8 hops	Cross-country, normal variance

Monitoring Tool Options

For production environments, consider dedicated monitoring tools: Smokeping (RRD-based latency graphs), Nagios/Icinga (alerts and dashboards), Prometheus + Blackbox Exporter (modern metrics), PRTG (commercial, comprehensive), ThousandEyes (SaaS, global perspective). These provide visualization, historical data, and sophisticated alerting beyond simple scripts.

Documentation Best Practices

Effective network troubleshooting produces documentation that helps resolve future issues faster. Well-documented diagnostics create organizational knowledge that persists beyond individual engineers.

What to Document:

Network Documentation Essentials

•Network topology diagrams: Visual maps showing subnets, routers, firewalls—updated when changes occur
•Baseline performance data: Normal latency, loss rates, path information for critical targets
•Known issues database: Previous problems, symptoms, and solutions for quick reference
•Escalation contacts: Who to call at each ISP/vendor with account numbers and SLA details
•Runbooks: Step-by-step procedures for common issues (written when you have time, used when you don't)
•Post-incident reviews: What happened, why, how it was fixed, how to prevent recurrence

Incident Documentation Template
Markdown
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
# Incident Report: [Brief Title]
 
## Timeline
- **Detected**: 2024-01-15 09:45 UTC
- **Acknowledged**: 2024-01-15 09:48 UTC
- **Mitigated**: 2024-01-15 10:23 UTC
- **Resolved**: 2024-01-15 14:30 UTC
- **Duration**: 4 hours 45 minutes (35 minutes to mitigation)
 
## Impact
- Services affected: Customer portal, API endpoints
- Users impacted: ~5,000 during peak window
- Revenue impact: Estimated $12,000 in lost transactions
 
## Root Cause
Upstream ISP experienced fiber cut on their primary path. Failover to 
backup path succeeded but backup had insufficient capacity for our 
traffic volume, causing packet loss and high latency.
 
## Detection
- Monitoring alert: "High packet loss to customer-portal.example.com"
- Customer complaints started arriving 3 minutes after alert
- MTR showed 40% loss at hop 4 (ISP edge router)
 
## Diagnostic Steps Taken
1. Verified internal network (all local segments healthy)
2. Traceroute showed path unchanged but high loss at ISP hop
3. Tested alternate paths (backup ISP connection) - worked
4. Contacted primary ISP NOC with evidence package
 
## Resolution
- Short-term: Shifted traffic to backup ISP connection
- Long-term: ISP repaired fiber, validated capacity, restored primary
 
## Lessons Learned
1. Backup ISP should be primary failover, not just "last resort"
2. Need automated traffic shifting based on loss metrics
3. ISP SLA review scheduled - current terms don't cover this scenario
 
## Action Items
- [ ] Implement automated ISP failover (Owner: NetOps, Due: 2024-02-01)
- [ ] Review/renegotiate ISP SLA (Owner: Vendor Mgmt, Due: 2024-01-30)
- [ ] Update runbook with ISP-specific escalation steps (Owner: Doc team, Due: 2024-01-20)

The Runbook Investment

Write runbooks when you're NOT in an incident. During an outage, you don't have time to write comprehensive guides. After an incident, document the steps you took so the next person (or future you at 3 AM) can follow a checklist instead of figuring it out from scratch. Each incident should produce or update a runbook.

Advanced Scenarios and Edge Cases

Some network issues require thinking beyond standard ping/traceroute. Here are advanced scenarios and the diagnostic approaches they require:

Scenario A: Asymmetric Routing Causing One-Way Traffic

Asymmetric Routing Diagnosis
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
# Symptom: Outbound connections work, inbound fail
# SSH to remote server works, but reverse SSH from server fails
 
# Diagnosis approach:
# 1. You can reach them
ping server.remote.com     # Works
 
# 2. They can't reach you (run from remote server)
ssh server.remote.com
$ ping your-ip-address   # Fails
 
# 3. Compare traceroutes from both ends
# Your side:
traceroute server.remote.com
 1  your-gw       ...
 2  your-isp      ...
 3  peering       ...
 4  their-isp     ...
 5  server        ...
 
# Their side (run on remote server):
traceroute your-ip-address
 1  their-gw      ...
 2  their-isp     ...
 3  DIFFERENT-peering  ...  # <-- Different path!
 4  * * *               # <-- Fails here
 5  * * *
 
# Root cause: Return traffic takes different path that has a block/failure
# Solution: Fix routing/firewall on the asymmetric return path

Scenario B: PMTU Black Hole

Path MTU Discovery fails when ICMP is blocked, causing large packets to silently disappear:

PMTU Black Hole Diagnosis
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
# Symptom: Small requests work, large responses fail
# Web page loads partially then hangs
# SSH works but SCP/large transfers stall
 
# Test with different packet sizes:
ping -s 100 target.com   # Works (small packet)
ping -s 1400 target.com  # Works (near MTU)
ping -s 1472 target.com  # Fails! (exceeds path MTU)
 
# Test with Don't Fragment bit:
ping -M do -s 1472 target.com  # "Frag needed but DF set" = PMTU issue
 
# Find the Path MTU:
for size in $(seq 1400 1500); do
    if ping -M do -c 1 -s $size target.com > /dev/null 2>&1; then
        echo "Size $size: OK"
    else
        echo "Size $size: FAIL (Path MTU is $((size + 28)))"
        break
    fi
done
 
# Common culprits: VPN tunnels, PPPoE (1492), certain carrier links
# Solution: Reduce local MTU, enable PMTU clamping on firewall,
#           or fix ICMP blocking in path

Scenario C: DNS-Specific vs Connectivity Issues

Sometimes what looks like a connectivity problem is actually DNS:

DNS vs Connectivity Differentiation
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# Symptom: "Website is down" but sometimes works
 
# Quick differentiation:
ping google.com          # Fails
ping 8.8.8.8            # Works!
 
# DNS issue confirmed. Further diagnosis:
dig google.com @192.168.1.1    # Test local DNS server
dig google.com @8.8.8.8        # Test external DNS server
 
# Common DNS issues:
# - Local DNS server overloaded/down
# - DNS cache corruption
# - Incorrect DNS server configured
# - DNS queries blocked by firewall (port 53)
# - DNSSEC validation failures
 
# Fix options:
# Temporary: echo "nameserver 8.8.8.8" > /etc/resolv.conf
# Permanent: Fix local DNS infrastructure

When Standard Tools Aren't Enough

Some issues require packet captures (tcpdump/Wireshark) for deep analysis. Timing issues, connection state problems, and protocol-level bugs are invisible to ping/traceroute. If you've narrowed down to 'packets reach destination but something's wrong,' it's time for packet capture analysis.

Summary: Becoming a Diagnostic Expert

We've covered the complete spectrum of diagnostic application—from systematic frameworks to advanced edge cases. Let's consolidate the essential knowledge:

Key Takeaways

•Systematic troubleshooting beats random testing: Use OSI layer progression and binary search isolation to converge on root causes efficiently.
•'What changed?' is often the fastest path to resolution: Networks don't spontaneously break—identify the change and you often identify the cause.
•Combine multiple tools for complete diagnosis: Ping for reachability, traceroute for path, dig for DNS, curl for application layer—each reveals different information.
•Document as you go: Evidence gathered during troubleshooting becomes the escalation package, the incident report, and the runbook for next time.
•Proactive monitoring prevents incidents: Baselines let you detect degradation before failure, and historical data accelerates troubleshooting when problems occur.
•Professional escalation gets results: Clear problem statements, specific evidence, and demonstrated competence make ISPs and vendors prioritize your tickets.
•Edge cases require deeper analysis: Asymmetric routing, PMTU issues, and DNS problems may hide behind simple symptoms—expand your toolkit as needed.

Module Complete: Ping and Traceroute

You've now mastered the complete spectrum of ICMP-based network diagnostics:

Page 1: ICMP Echo Request/Reply fundamentals
Page 2: Ping operation and output interpretation
Page 3: Traceroute mechanism and path analysis
Page 4: TTL manipulation for advanced diagnostics
Page 5: Professional diagnostic methodology

These skills form the foundation of network troubleshooting. Whether you're debugging a development environment, managing enterprise infrastructure, or responding to production incidents, the techniques in this module will serve you throughout your career.

Module Complete

Congratulations! You've completed the Ping and Traceroute module. You now possess professional-level diagnostic capabilities—from understanding the ICMP protocol at the byte level to applying systematic troubleshooting methodologies in production environments. The next modules will build on these fundamentals as we explore ARP, RARP, and additional network layer protocols.

5 / 5

Loading learning content...

Computer NetworksPing and Traceroute

Ping and Traceroute: Network Diagnostic Tools

LevelIntermediate

Duration75 mins

TopicPing and Traceroute

5 / 5

Diagnostic Usage: Professional Network Troubleshooting

From Tools to Mastery

This final page transforms your tool knowledge into diagnostic methodology—the systematic approach expert network engineers use to:

Quickly isolate connectivity vs. performance vs. configuration issues
Determine the responsible party (local network, ISP, destination)
Gather evidence suitable for escalation to vendors or ISPs
Document findings professionally
Monitor for and prevent future occurrences

What You Will Master

The Systematic Troubleshooting Framework

Professional network engineers follow structured methodologies rather than ad-hoc testing. The most effective framework combines the OSI layer approach with binary search isolation:

The Bottom-Up OSI Approach:

Start at the physical layer and work upward. If lower layers fail, higher layers cannot work—so confirming lower layers first saves time:

OSI Layer Diagnostic Progression
Layer	What to Check	Diagnostic Method	Common Failures
Physical	Cable, port, link light	Visual inspection, cable tester	Damage, loose connection, bad port
Data Link	MAC address, VLAN, switch port	`arp -a`, switch logs, spanning tree	Wrong VLAN, MAC conflict, STP blocking
Network	IP config, routing, connectivity	`ping`, `traceroute`, `ip route`	Wrong IP, missing route, firewall block
Transport	Ports, connections, sockets	`netstat`, `ss`, `tcpdump`	Port closed, firewall, connection refused
5-7. Application	Service status, config, logs	Service-specific tools, logs	Wrong config, crash, resource exhaustion

The Binary Search Isolation Technique:

When you have a path with multiple segments, cut it in half repeatedly to find the failure point:

Binary Search Path Isolation
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# Problem: Cannot reach external website from workstation
# Path: Workstation → Switch → Router → ISP → Internet → Website
 
# Step 1: Test middle point (Router/Gateway)
ping 192.168.1.1
# ✓ Works → Problem is beyond router
# ✗ Fails → Problem is between workstation and router
 
# Assuming Step 1 works, test next middle point:
# Step 2: Test ISP/First external hop  
ping 10.0.0.1  # ISP gateway from traceroute
# ✓ Works → Problem is beyond ISP
# ✗ Fails → Problem is router ↔ ISP link
 
# Continue halving until problem is isolated to single link/device
 
# This is more efficient than sequential testing:
# Binary search: O(log n) tests to find failure in n segments
# Sequential: O(n) tests in worst case
 
# Example: 8-hop path
# Sequential worst case: 8 tests
# Binary search worst case: 3 tests (log₂ 8 = 3)

The STOP-THINK-TEST Framework

•STOP: Don't immediately start running commands. Gather information about the symptoms first. What exactly isn't working? When did it stop? What changed?
•THINK: Form a hypothesis. Based on symptoms, what are the most likely causes? Where in the path would those causes manifest?
•TEST: Run the minimum diagnostic to confirm or eliminate your hypothesis. One clear test is worth ten ambiguous ones.
•Repeat: Based on test results, refine hypothesis and test again. Converge systematically on root cause.

The 'What Changed?' Question

Common Diagnostic Scenarios and Solutions

Let's work through the most common network problems you'll encounter, with step-by-step diagnostic procedures:

Scenario 1: "I can't reach the website"

Scenario 1: Website Unreachable
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
# Step 1: Determine if it's DNS or connectivity
nslookup www.example.com
# Output: 93.184.216.34
 
# Step 2: Ping the IP directly (bypass DNS caching issues)
ping 93.184.216.34
# ✗ 100% packet loss
 
# Step 3: Test basic connectivity infrastructure
ping 8.8.8.8  # Well-known Google DNS
# ✓ Works → Internet connectivity exists
# ✗ Fails → General connectivity issue (go to local troubleshooting)
 
# Step 4: Traceroute to problematic destination
traceroute 93.184.216.34
 1  192.168.1.1 (gateway)    1.2 ms
 2  10.0.0.1                  8.5 ms
 3  ae-5.r21.lsanca07...     12.3 ms
 4  * * *
 5  * * *
 ...
 
# Diagnosis: Path fails at hop 4 (after hop 3 which is ISP backbone)
# Likely cause: Routing issue at ISP level, destination network issue,
#               or destination firewall blocking ICMP
 
# Step 5: Test if web service is reachable even if ping blocked
curl -I --connect-timeout 5 https://www.example.com
# HTTP/1.1 200 OK → Website is fine! They just block ping.
# timeout → Website truly unreachable
 
# Conclusion paths:
# - If curl works but ping fails: Destination blocks ICMP (normal)
# - If both fail: Real connectivity issue at identified hop
# - If DNS failed: DNS server issue, not connectivity

Scenario 2: "The connection is slow"

Scenario 2: Slow Connection
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
# Step 1: Quantify "slow" - measure actual latency
ping -c 20 target.example.com
# Output: rtt min/avg/max/mdev = 45.2/156.8/892.3/234.7 ms
 
# Analysis: Average 156ms with huge variance (mdev 234ms)
# This indicates severe jitter/congestion, not just distance
 
# Step 2: Identify where latency is added
mtr -r -c 50 target.example.com
 
#                              Loss%  Snt  Last  Avg  Best Wrst StDev
# 1. router.local               0.0%   50   0.5  0.6  0.3  1.2  0.2
# 2. 10.0.0.1                   0.0%   50   8.2  8.5  7.9 12.3  0.8
# 3. ae-5.r21.lsanca07...       0.0%   50  12.1 12.4 11.8 15.6  0.6
# 4. congested-link.isp.net     5.0%   50  85.3 98.7 25.4 456.2 89.3 # ← PROBLEM
# 5. 72.14.196.226              4.8%   50  92.4 105.3 28.1 478.9 92.1
# 6. target.example.com         4.9%   50  95.2 108.7 30.2 485.3 95.8
 
# Analysis:
# - Hop 4 shows huge jump (12ms → 98ms average)
# - High StDev at hop 4 (89.3ms) indicates severe variance
# - Loss starts at hop 4 (5%)
# - Problem persists through remaining hops
 
# Diagnosis: Congestion at hop 4 (ISP link)
# Action: Report to ISP with MTR data as evidence
 
# Step 3: Correlate with time of day
# Run periodic tests to identify patterns
while true; do
    echo "$(date): $(ping -c 1 -W 2 target.example.com | grep time=)"
    sleep 60
done > latency_log.txt
 
# If latency is high only during business hours: bandwidth contention
# If latency is high only at night: different user patterns (streaming?)
# If latency is always high: capacity issue or misconfiguration

Scenario 3: "It was working, now it's broken"

Scenario 3: Sudden Failure
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
# The most common scenario: something changed!
 
# Step 1: Test basic connectivity
ping google.com
# ✗ ping: unknown host google.com
 
# Step 2: Test by IP (bypass DNS)
ping 8.8.8.8
# ✓ 64 bytes from 8.8.8.8: icmp_seq=1 ttl=116 time=15.2 ms
 
# Diagnosis: DNS is broken, not connectivity
# Check DNS configuration:
cat /etc/resolv.conf
# nameserver 192.168.1.53  # Internal DNS server
 
ping 192.168.1.53
# ✗ Request timeout
 
# Root cause: Internal DNS server is down
# Solution: Fix DNS server or use alternative (temporarily 8.8.8.8)
 
# Alternative scenario: IP works but different path
# Compare current traceroute to known-good baseline
 
# Current:
traceroute 10.20.30.40
 1  192.168.1.1    0.5 ms
 2  10.0.0.1       8.2 ms
 3  172.16.0.1    45.3 ms   # NEW: unexpected router!
 4  10.20.30.40   78.9 ms   # Much slower total
 
# Baseline (from documentation):
# 1  192.168.1.1    0.5 ms
# 2  10.10.10.1     5.2 ms  # Different router - old path
# 3  10.20.30.40   12.3 ms
 
# Diagnosis: Routing changed. Traffic now goes through 172.16.0.1
# instead of 10.10.10.1, adding latency.
# Action: Check routing tables, BGP sessions, link status on old path

Scenario 4: Intermittent Issues

Integrating Multiple Diagnostic Tools

Real troubleshooting rarely uses a single tool. Effective diagnosis combines ping, traceroute, and TTL analysis with complementary tools:

The Core Diagnostic Toolkit:

Complementary Diagnostic Tools
Tool	Purpose	When to Use
`ping`	Basic reachability, RTT measurement	First test for any connectivity issue
`traceroute/mtr`	Path mapping, per-hop analysis	When you need to know WHERE in the path
`dig/nslookup`	DNS resolution testing	When names don't resolve or resolve wrong
`curl/wget`	Application-layer testing	When ICMP blocked but services may work
`telnet/nc`	Port connectivity testing	Testing specific TCP/UDP port access
`ss/netstat`	Local socket/connection state	Connection issues on local host
`tcpdump/wireshark`	Packet capture and analysis	Deep inspection, timing issues, protocol errors
`arp -a`	ARP cache inspection	Layer 2 issues, wrong MAC associations
`ip route`	Routing table inspection	Checking local routing decisions

Combined Diagnostic Script
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
#!/bin/bash
# comprehensive_diag.sh - Full connectivity diagnostic
 
TARGET=$1
if [ -z "$TARGET" ]; then
    echo "Usage: $0 <target_hostname_or_ip>"
    exit 1
fi
 
echo "=========================================="
echo "  Comprehensive Network Diagnostic"
echo "  Target: $TARGET"
echo "  Time: $(date)"
echo "=========================================="
 
# Step 1: DNS Resolution
echo -e "
[1/6] DNS Resolution"
echo "-------------------"
if host "$TARGET" > /dev/null 2>&1; then
    IP=$(host "$TARGET" | grep "has address" | head -1 | awk '{print $4}')
    echo "✓ Resolved: $TARGET → $IP"
else
    echo "✗ DNS resolution failed for $TARGET"
    echo "  Testing DNS server connectivity..."
    ping -c 1 -W 2 8.8.8.8 > /dev/null 2>&1 && echo "  (8.8.8.8 reachable - DNS server may be down)" || echo "  (No internet connectivity?)"
    exit 1
fi
 
# Step 2: ICMP Reachability
echo -e "
[2/6] ICMP Reachability (ping)"
echo "-----------------------------"
PING_RESULT=$(ping -c 5 -W 2 "$IP" 2>&1)
if echo "$PING_RESULT" | grep -q "5 received"; then
    echo "✓ Ping successful:"
    echo "$PING_RESULT" | grep -E "^(PING|rtt|5 packets)"
else
    echo "✗ Ping failed or packet loss detected:"
    echo "$PING_RESULT" | grep -E "^(PING|--- |5 packets|rtt)"
fi
 
# Step 3: Path Analysis
echo -e "
[3/6] Path Analysis (traceroute)"
echo "--------------------------------"
traceroute -n -w 2 -q 1 "$IP" 2>&1 | head -20
 
# Step 4: TCP Port Check (common ports)
echo -e "
[4/6] TCP Port Connectivity"
echo "--------------------------"
for PORT in 80 443 22; do
    if timeout 2 bash -c "echo > /dev/tcp/$IP/$PORT" 2>/dev/null; then
        echo "✓ Port $PORT: Open"
    else
        echo "✗ Port $PORT: Closed/Filtered"
    fi
done
 
# Step 5: HTTP(S) Check
echo -e "
[5/6] HTTP/HTTPS Connectivity"
echo "----------------------------"
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" --connect-timeout 5 "http://$TARGET" 2>/dev/null)
HTTPS_CODE=$(curl -s -o /dev/null -w "%{http_code}" --connect-timeout 5 "https://$TARGET" 2>/dev/null)
echo "HTTP response code: $HTTP_CODE"
echo "HTTPS response code: $HTTPS_CODE"
 
# Step 6: Summary
echo -e "
[6/6] Diagnostic Summary"
echo "------------------------"
echo "Target: $TARGET ($IP)"
echo "Timestamp: $(date +%Y-%m-%d_%H:%M:%S)"
echo "Ping: $(echo "$PING_RESULT" | grep -oE '[0-9]+% packet loss')"
echo "HTTP: $HTTP_CODE | HTTPS: $HTTPS_CODE"
echo "=========================================="

Tool Selection Decision Tree:

Problem reported
      │
      ▼
┌─────────────────────────────────────────┐
│  Can you ping the default gateway?      │
└─────────────────────────────────────────┘
      │
      ├── NO → Local network issue
      │         • Check cable/WiFi
      │         • Check IP config (ip addr)
      │         • Check ARP (arp -a)
      │
      └── YES → Continue
              │
              ▼
      ┌─────────────────────────────────────────┐
      │  Can you ping 8.8.8.8 (Internet)?       │
      └─────────────────────────────────────────┘
              │
              ├── NO → Gateway/ISP issue
              │         • traceroute 8.8.8.8
              │         • Check gateway config
              │
              └── YES → Continue
                      │
                      ▼
              ┌─────────────────────────────────────────┐
              │  Can you resolve DNS names?              │
              └─────────────────────────────────────────┘
                      │
                      ├── NO → DNS issue
                      │         • Check /etc/resolv.conf
                      │         • Test dig @8.8.8.8 name
                      │
                      └── YES → Destination-specific issue
                                • traceroute to target
                                • curl/telnet to target
                                • tcpdump if needed

The Power of Comparison

Gathering Evidence for Escalation

When the problem exists outside your control (ISP issue, vendor service, partner network), you need compelling evidence to get action. Poor escalations get ignored; well-documented ones get resolved.

Essential Elements of an Escalation Package:

Professional Escalation Checklist

•Clear problem statement: One sentence summarizing the issue (e.g., 'Complete packet loss to your server 203.0.113.50 from our network 198.51.100.0/24')
•Impact description: What's affected and how severely (e.g., 'Production e-commerce site unavailable, affecting 10,000 customers/hour')
•Timeline: When it started, any intermittency pattern (e.g., 'Started 2024-01-15 09:45 UTC, still ongoing')
•Source information: Your public IP(s), ASN if relevant, geographic location
•Diagnostic output: Ping results, traceroute output, MTR data—with timestamps
•Comparisons: Evidence from different sources if available ('traceroute from backup site succeeds')
•What you've already tried: Show competence and eliminate obvious suggestions

Escalation Report Template

Text

Subject: Packet Loss to [Destination] from AS[XXXXX] - Ticket #[REF]
 
Hi [ISP/Vendor] NOC,
 
We are experiencing complete connectivity loss from our network to your infrastructure.
 
ISSUE SUMMARY:
- Problem: 100% packet loss to 203.0.113.50 from 198.51.100.0/24
- Impact: Production payment gateway unavailable
- Start Time: 2024-01-15 09:45:23 UTC
- Current Status: Ongoing as of 2024-01-15 10:15:00 UTC
 
SOURCE INFORMATION:
- Our Public IP: 198.51.100.10
- Our ASN: AS12345
- Location: San Francisco, CA, USA
- ISP: Example Communications
 
DIAGNOSTIC EVIDENCE:
 
1. PING OUTPUT (current):
$ ping -c 10 203.0.113.50
PING 203.0.113.50 (203.0.113.50) 56(84) bytes of data.
--- 203.0.113.50 ping statistics ---
10 packets transmitted, 0 received, 100% packet loss, time 9215ms
 
2. MTR OUTPUT (50 samples):
HOST                      Loss%  Snt  Last  Avg  Best Wrst StDev
1. 198.51.100.1             0.0%  50   0.5  0.6  0.3  1.2  0.1
2. 10.0.0.1                 0.0%  50   5.2  5.4  5.0  8.1  0.4
3. edge-router.isp.net      0.0%  50  12.3 12.5 12.0 15.2  0.5
4. your-peering.isp.net   100.0%  50   0.0  0.0  0.0  0.0  0.0 <-- Loss starts
5. ???                    100.0%  50   0.0  0.0  0.0  0.0  0.0
 
3. TRACEROUTE OUTPUT:
$ traceroute -n 203.0.113.50
 1  198.51.100.1     0.5 ms
 2  10.0.0.1         5.3 ms
 3  68.87.64.1      12.4 ms
 4  * * *
 5  * * *
 
4. COMPARISON (working destination on same path):
$ ping -c 5 203.0.113.100  (different host, same /24)
64 bytes from 203.0.113.100: icmp_seq=1 ttl=56 time=45.2 ms
[Works fine - issue is specific to .50]
 
ACTIONS WE HAVE TAKEN:
- Verified our local routing and firewall (no changes, other traffic works)
- Tested from secondary ISP connection (same result)
- Confirmed DNS resolution works correctly
- Rebooted our edge router (no effect)
 
REQUEST:
Please investigate connectivity to 203.0.113.50 from the direction of 
68.87.64.1 (our ISP's peering point with your network).
 
Contact: [Your Name]
Phone: [+1-XXX-XXX-XXXX]
Ticket Ref: [Internal ticket number]

Getting Faster Response

Proactive Monitoring and Baseline Establishment

Key Metrics to Monitor:

Essential Connectivity Metrics
Metric	What It Shows	Alert Threshold (Example)	Collection Method
RTT (latency)	Network delay	2× baseline average	`ping` with timestamp logging
Jitter (RTT variance)	Latency consistency	StdDev > 20% of average	MTR or custom ping parser
Packet loss %	Reliability	0.1% over 5 minutes	Continuous ping with loss count
Path changes	Routing stability	Any change from baseline	Periodic traceroute comparison
DNS resolution time	Lookup performance	500ms average	`dig` with +stats option

Simple Monitoring Script
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
#!/bin/bash
# monitor_connectivity.sh - Continuous monitoring with alerting
 
TARGETS="8.8.8.8 google.com production-server.example.com"
LOG_DIR="/var/log/network_monitor"
ALERT_EMAIL="oncall@example.com"
CHECK_INTERVAL=60  # seconds
 
mkdir -p "$LOG_DIR"
 
monitor_target() {
    local target=$1
    local timestamp=$(date +%Y-%m-%d_%H:%M:%S)
    local log_file="$LOG_DIR/${target//[.:]/_}.log"
    
    # Run ping
    result=$(ping - c 5 - W 2 "$target" 2 >& 1)
    loss=$(echo "$result" | grep - oE '[0-9]+% packet loss' | grep - oE '[0-9]+')
    avg_rtt=$(echo "$result" | grep - oE 'avg[^=]*= [0-9.]+' | grep - oE '[0-9.]+')
    
    # Log result
    echo "$timestamp | loss=$loss% | avg_rtt=${avg_rtt}ms" >> "$log_file"
    
    # Alert on issues
    if ["$loss" - gt 10]; then
        echo "ALERT: $target - $loss% packet loss at $timestamp" |                                 mail - s "Network Alert: High packet loss to $target" "$ALERT_EMAIL"
    fi
    
    if(($(echo "$avg_rtt > 200" | bc - l 2 > /dev/null || echo 0))); then
        echo "ALERT: $target - High latency ${avg_rtt}ms at $timestamp" |     mail - s "Network Alert: High latency to $target" "$ALERT_EMAIL"
    fi
}
 
echo "Starting network monitoring..."
echo "Targets: $TARGETS"
echo "Interval: ${CHECK_INTERVAL}s"
echo "Logs: $LOG_DIR"
 
while true; do
    for target in $TARGETS; do
        monitor_target "$target" &
    done
    wait
    sleep "$CHECK_INTERVAL"
done

Baseline Establishment:

Before you can detect anomalies, you need to know what's normal:

# Collect baseline data over 24 - 48 hours of normal operation
mtr - r - c 100 - i 1 target.example.com > baseline_mtr.txt

# Capture hourly averages to understand daily patterns
for hour in $(seq 0 23); do 
    echo "Hour $hour:" >> hourly_baseline.txt
    ping -c 60 target.example.com | tail -1 >> hourly_baseline.txt
done

# Store baseline traceroute for path comparison
traceroute -n target.example.com > baseline_traceroute.txt

What goes into your baseline documentation:

Target	Normal RTT	Normal Loss	Expected Path	Notes
8.8.8.8	15-25ms	<0.01%	4 hops	Google DNS, always available
production-db	2-5ms	0%	2 hops	Same datacenter
partner-api.com	45-60ms	<0.1%	8 hops	Cross-country, normal variance

Monitoring Tool Options

Documentation Best Practices

What to Document:

Network Documentation Essentials

•Network topology diagrams: Visual maps showing subnets, routers, firewalls—updated when changes occur
•Baseline performance data: Normal latency, loss rates, path information for critical targets
•Known issues database: Previous problems, symptoms, and solutions for quick reference
•Escalation contacts: Who to call at each ISP/vendor with account numbers and SLA details
•Runbooks: Step-by-step procedures for common issues (written when you have time, used when you don't)
•Post-incident reviews: What happened, why, how it was fixed, how to prevent recurrence

Incident Documentation Template
Markdown
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
# Incident Report: [Brief Title]
 
## Timeline
- **Detected**: 2024-01-15 09:45 UTC
- **Acknowledged**: 2024-01-15 09:48 UTC
- **Mitigated**: 2024-01-15 10:23 UTC
- **Resolved**: 2024-01-15 14:30 UTC
- **Duration**: 4 hours 45 minutes (35 minutes to mitigation)
 
## Impact
- Services affected: Customer portal, API endpoints
- Users impacted: ~5,000 during peak window
- Revenue impact: Estimated $12,000 in lost transactions
 
## Root Cause
Upstream ISP experienced fiber cut on their primary path. Failover to 
backup path succeeded but backup had insufficient capacity for our 
traffic volume, causing packet loss and high latency.
 
## Detection
- Monitoring alert: "High packet loss to customer-portal.example.com"
- Customer complaints started arriving 3 minutes after alert
- MTR showed 40% loss at hop 4 (ISP edge router)
 
## Diagnostic Steps Taken
1. Verified internal network (all local segments healthy)
2. Traceroute showed path unchanged but high loss at ISP hop
3. Tested alternate paths (backup ISP connection) - worked
4. Contacted primary ISP NOC with evidence package
 
## Resolution
- Short-term: Shifted traffic to backup ISP connection
- Long-term: ISP repaired fiber, validated capacity, restored primary
 
## Lessons Learned
1. Backup ISP should be primary failover, not just "last resort"
2. Need automated traffic shifting based on loss metrics
3. ISP SLA review scheduled - current terms don't cover this scenario
 
## Action Items
- [ ] Implement automated ISP failover (Owner: NetOps, Due: 2024-02-01)
- [ ] Review/renegotiate ISP SLA (Owner: Vendor Mgmt, Due: 2024-01-30)
- [ ] Update runbook with ISP-specific escalation steps (Owner: Doc team, Due: 2024-01-20)

The Runbook Investment

Advanced Scenarios and Edge Cases

Some network issues require thinking beyond standard ping/traceroute. Here are advanced scenarios and the diagnostic approaches they require:

Scenario A: Asymmetric Routing Causing One-Way Traffic

Asymmetric Routing Diagnosis
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
# Symptom: Outbound connections work, inbound fail
# SSH to remote server works, but reverse SSH from server fails
 
# Diagnosis approach:
# 1. You can reach them
ping server.remote.com     # Works
 
# 2. They can't reach you (run from remote server)
ssh server.remote.com
$ ping your-ip-address   # Fails
 
# 3. Compare traceroutes from both ends
# Your side:
traceroute server.remote.com
 1  your-gw       ...
 2  your-isp      ...
 3  peering       ...
 4  their-isp     ...
 5  server        ...
 
# Their side (run on remote server):
traceroute your-ip-address
 1  their-gw      ...
 2  their-isp     ...
 3  DIFFERENT-peering  ...  # <-- Different path!
 4  * * *               # <-- Fails here
 5  * * *
 
# Root cause: Return traffic takes different path that has a block/failure
# Solution: Fix routing/firewall on the asymmetric return path

Scenario B: PMTU Black Hole

Path MTU Discovery fails when ICMP is blocked, causing large packets to silently disappear:

PMTU Black Hole Diagnosis
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
# Symptom: Small requests work, large responses fail
# Web page loads partially then hangs
# SSH works but SCP/large transfers stall
 
# Test with different packet sizes:
ping -s 100 target.com   # Works (small packet)
ping -s 1400 target.com  # Works (near MTU)
ping -s 1472 target.com  # Fails! (exceeds path MTU)
 
# Test with Don't Fragment bit:
ping -M do -s 1472 target.com  # "Frag needed but DF set" = PMTU issue
 
# Find the Path MTU:
for size in $(seq 1400 1500); do
    if ping -M do -c 1 -s $size target.com > /dev/null 2>&1; then
        echo "Size $size: OK"
    else
        echo "Size $size: FAIL (Path MTU is $((size + 28)))"
        break
    fi
done
 
# Common culprits: VPN tunnels, PPPoE (1492), certain carrier links
# Solution: Reduce local MTU, enable PMTU clamping on firewall,
#           or fix ICMP blocking in path

Scenario C: DNS-Specific vs Connectivity Issues

Sometimes what looks like a connectivity problem is actually DNS:

DNS vs Connectivity Differentiation
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# Symptom: "Website is down" but sometimes works
 
# Quick differentiation:
ping google.com          # Fails
ping 8.8.8.8            # Works!
 
# DNS issue confirmed. Further diagnosis:
dig google.com @192.168.1.1    # Test local DNS server
dig google.com @8.8.8.8        # Test external DNS server
 
# Common DNS issues:
# - Local DNS server overloaded/down
# - DNS cache corruption
# - Incorrect DNS server configured
# - DNS queries blocked by firewall (port 53)
# - DNSSEC validation failures
 
# Fix options:
# Temporary: echo "nameserver 8.8.8.8" > /etc/resolv.conf
# Permanent: Fix local DNS infrastructure

When Standard Tools Aren't Enough

Summary: Becoming a Diagnostic Expert

We've covered the complete spectrum of diagnostic application—from systematic frameworks to advanced edge cases. Let's consolidate the essential knowledge:

Key Takeaways

•Systematic troubleshooting beats random testing: Use OSI layer progression and binary search isolation to converge on root causes efficiently.
•'What changed?' is often the fastest path to resolution: Networks don't spontaneously break—identify the change and you often identify the cause.
•Combine multiple tools for complete diagnosis: Ping for reachability, traceroute for path, dig for DNS, curl for application layer—each reveals different information.
•Document as you go: Evidence gathered during troubleshooting becomes the escalation package, the incident report, and the runbook for next time.
•Proactive monitoring prevents incidents: Baselines let you detect degradation before failure, and historical data accelerates troubleshooting when problems occur.
•Professional escalation gets results: Clear problem statements, specific evidence, and demonstrated competence make ISPs and vendors prioritize your tickets.
•Edge cases require deeper analysis: Asymmetric routing, PMTU issues, and DNS problems may hide behind simple symptoms—expand your toolkit as needed.

Module Complete: Ping and Traceroute

You've now mastered the complete spectrum of ICMP-based network diagnostics:

Page 1: ICMP Echo Request/Reply fundamentals
Page 2: Ping operation and output interpretation
Page 3: Traceroute mechanism and path analysis
Page 4: TTL manipulation for advanced diagnostics
Page 5: Professional diagnostic methodology

Module Complete

5 / 5