When you access a website halfway around the world, your request traverses a remarkable infrastructure—submarine cables spanning oceans, fiber optic networks crossing continents, and massive data centers interconnecting thousands of networks. This is the Internet backbone, and it runs entirely on BGP.

The backbone isn't a single network or company—it's an emergent structure arising from thousands of independent networks choosing to interconnect. Understanding how these interconnections work, how traffic flows between networks, and how BGP enables global reachability is essential for anyone working with Internet infrastructure at scale.

The Internet's structure is often described as a hierarchy of network "tiers," though the reality is more nuanced than a simple pyramid.

Tier 1 Networks:

Tier 1 networks sit at the top of the Internet hierarchy. They have the defining characteristic of being able to reach every destination on the Internet without paying for transit—either through direct peering or through peering with other Tier 1 networks.

Major Tier 1 Networks (as of recent classifications):

• Lumen (formerly CenturyLink/Level 3)
• NTT Communications
• Cogent Communications
• Tata Communications
• GTT Communications
• Zayo/Allstream
• Telecom Italia Sparkle
• Telia Carrier
• Arelion (formerly Telia Carrier)

The exact list is debated, as the criteria for "Tier 1" aren't formally defined, and the landscape evolves through mergers and peering changes.

Tier 2 Networks:

Tier 2 networks have significant infrastructure but rely on at least one transit provider to reach parts of the Internet. They typically:

• Peer settlement-free with similar-sized networks
• Purchase transit from Tier 1 or larger Tier 2 networks
• Sell transit to smaller networks and enterprises

Tier 3 Networks:

Tier 3 networks are primarily consumers of transit. They include:

• Small ISPs serving end users
• Enterprise networks with Internet connectivity
• Content providers without significant peering

The Tier System Reality:

The neat tier hierarchy is a simplification. In practice:

1. Content Networks: Companies like Google, Meta, and Netflix have massive networks that peer directly with most ISPs, bypassing the traditional hierarchy entirely.

2. Hybrid Relationships: A network might be Tier 2 in one region and Tier 3 in another.

3. Peering vs. Transit: The same two networks might have peering in one location and transit in another.

4. Constant Evolution: Mergers, new submarine cables, and shifting traffic patterns continuously reshape the landscape.

Understanding the economics of Internet interconnection is essential for network operators. These relationships directly influence routing policy and network design.

Transit Agreements:

In a transit relationship, one network pays another for full Internet connectivity:

Typical Transit Pricing (varies significantly by region/volume):

• Large wholesale: $0.20-$1.00 per Mbps/month
• Enterprise: $5-$50 per Mbps/month
• Pricing depends on location, volume, commitment term

Peering Agreements:

In a peering relationship, networks exchange traffic without payment:

Types of Peering:

1. Public Peering (IXP):

• Meet at Internet Exchange Point
• Multiple peers via single connection
• Lower cost per peer
• Often requires IXP membership fee

2. Private Network Interconnect (PNI):

• Direct fiber connection between two networks
• Dedicated capacity
• Higher performance
• More expensive (cross-connect, port costs)

3. Paid Peering:

• Settlement-free peering with a twist
• One party pays (usually smaller or more traffic-heavy network)
• Less than transit, more than free peering
• Common when networks have unequal value

Why Peer?

1. Cost Reduction: Traffic to peer is "free" vs. transit costs
2. Performance: Direct path vs. multiple hops through transit
3. Redundancy: Multiple paths improve resilience
4. Strategic: Dependency on fewer transit providers

Peering Policies:

Networks define peering policies that specify requirements:

• Open: Will peer with anyone who asks
• Selective: Evaluate requests against criteria (traffic volumes, geographic presence)
• Restrictive: Only peer with similar-sized networks

Internet Exchange Points (IXPs) are physical locations where networks interconnect to exchange traffic. They're the meeting grounds of the Internet, reducing costs and improving performance for all participants.

How IXPs Work:

An IXP typically consists of:

1. Switch Fabric: High-capacity Ethernet switches connecting member ports
2. Colocation Facility: Data center housing the infrastructure
3. Route Servers: Optional service simplifying peering (BGP sessions with route server instead of each peer)
4. Operations Team: Manages the exchange, enforces policies

IXP Architecture:

[Member A] ---+
              |
[Member B] ---+---- IXP Switch Fabric ----+--- Route Server
              |                           |
[Member C] ---+                     [Member D]
              |
[Member E] ---+

Members connect to the switch fabric and can:

• Establish bilateral BGP sessions with specific peers
• Use the route server to peer with all route server participants

Route Servers:

Route servers simplify IXP peering by providing a central BGP "hub":

Without Route Server:

• 10 members = 45 bilateral BGP sessions (10×9÷2)
• Each new member needs sessions with all existing members
• O(n²) scaling

With Route Server:

• Each member peers only with route server(s)
• Route server reflects routes between members
• One session provides access to all participating members
• O(n) scaling

Route Server Policies:

Members can configure which routes to receive/send via the route server:

! Accept routes from all RS participants
route-map RS-IN permit 10

! Advertise to all RS participants except AS 65003
route-map RS-OUT permit 10
  set community 0:65003 additive
  ! 0:ASN = do not announce to ASN (route server convention)

IXP Membership Considerations:

1. Port Cost: Monthly fee for switch port (1G, 10G, 100G, 400G)
2. Cross-Connect: Physical connection to the exchange
3. Traffic Volume: Some IXPs have minimum traffic requirements
4. Peering Opportunities: Who's at this IXP?
5. Geographic Relevance: Is traffic served by this location?

The largest sources of Internet traffic—video streaming, web content, software updates—come from content networks that have radically reshaped Internet topology.

Content Network Strategies:

1. Traditional CDN (Akamai, Cloudflare, Fastly):

• Operate edge servers in many locations
• Cache content close to users
• Anycast routing for automatic failover
• Customer pays CDN for delivery

2. Private Content Networks (Netflix Open Connect, Google GGC):

• Content provider's own infrastructure
• Cache appliances placed directly in ISP networks
• BGP (or private routing) for traffic delivery
• Free to ISPs (content provider saves transit costs)

3. Cloud Provider Edge (AWS CloudFront, Azure CDN):

• Integrated with cloud infrastructure
• Leverages cloud provider's extensive peering
• Dynamic and static content delivery

BGP and Content Delivery:

Content networks use BGP to:

1. Establish peering at IXPs and via PNI
2. Announce prefixes for their edge infrastructure
3. Implement anycast for geographic load balancing
4. Traffic engineer to preferred paths

Anycast in Practice:

Anycast allows the same IP address to be announced from multiple locations:

Location: NYC      Location: LON      Location: TOK
  |                   |                   |
  | Announce          | Announce          | Announce
  | 192.0.2.0/24      | 192.0.2.0/24      | 192.0.2.0/24
  v                   v                   v
Internet              Internet              Internet

Users automatically route to the "closest" (by BGP metrics) instance. This provides:

• Geographic load balancing
• Automatic failover (if one site fails, traffic goes to next closest)
• DDoS resilience (attack traffic distributed across locations)

Embedded Cache Model:

Netflix's Open Connect and Google's GGC place cache servers inside ISP networks:

[End User] → [ISP Access] → [Embedded Cache] → Content
                  ↓
             [ISP Core] → [Internet] → [Content Origin]
                            (only for uncached content)

Benefits:

• ISP: Reduces transit costs, improves performance for customers
• Content Provider: Reduces transit costs, better user experience
• User: Lower latency, higher bandwidth

BGP may or may not be involved—some deployments use private routing or simple static routes.

International Internet connectivity relies primarily on submarine cables—fiber optic cables laid across ocean floors connecting continents. These are the true backbone of global connectivity.

Submarine Cable Infrastructure:

Cable Ownership Models:

1. Consortium Cables: Multiple carriers jointly fund and operate (e.g., SEA-ME-WE cables)
2. Private Cables: Single company builds for their own use (e.g., Google's Dunant, Meta's 2Africa)
3. Commercial Cables: Built to sell capacity wholesale

BGP and Submarine Cables:

Submarine cables appear in BGP as segments of AS paths:

User in USA → US ISP (AS 65001) → Cable Landing (US)
                     ↓
              Submarine Cable (no AS, Layer 1)
                     ↓
           Cable Landing (EU) → EU ISP (AS 65002) → Destination

At cable landing stations, networks peer or exchange traffic. The cable itself is transparent to BGP—it's just the physical medium.

Cable Landing Points and Diversity:

Cable landing points are critical infrastructure:

• Limited locations due to geographic, regulatory requirements
• Natural concentration points for traffic
• Single points of failure for entire regions

Resilience Considerations:

1. Geographic Diversity: Multiple cables via different routes
2. Landing Point Diversity: Different landing locations in same region
3. Ownership Diversity: Cables from different operators

Lessons from Outages:

Submarine cable damage (anchors, earthquakes, sabotage) has caused significant outages:

• 2006 Taiwan earthquake: 7 cables damaged, Asia-US traffic severely impacted
• 2008 Mediterranean cuts: Multiple cables damaged within days, Middle East connectivity disrupted
• 2012 Hurricane Sandy: Cable landing points flooded, but cables survived

BGP enables automatic rerouting when cables fail—but only if diverse paths exist and BGP sessions are configured for redundancy.

The Internet backbone's reliance on BGP creates significant security challenges. Understanding threats and mitigations is essential for anyone operating production networks.

Major BGP Security Threats:

Notable BGP Security Incidents:

1. Pakistan/YouTube (2008): Pakistan Telecom hijacked YouTube's prefixes to block access domestically; leak propagated globally.

2. China Telecom (2010, 2019+): Suspicious route announcements affecting US government and military prefixes.

3. Cloudflare Leak (2019): Customer's route leak through Verizon caused widespread Cloudflare service degradation.

4. Russian Hijacks (2022): Multiple incidents during Ukraine conflict, including Twitter route hijacking.

Defense Mechanisms:

1. RPKI (Resource Public Key Infrastructure):

Cryptographically sign route origins:

• Create ROAs (Route Origin Authorizations) for your prefixes
• Validate received routes against ROAs
• Reject or deprioritize invalid routes

RPKI Adoption (as of 2024):

• ~50% of IPv4 routes have valid ROAs
• ~60% of IPv6 routes have valid ROAs
• Major networks increasingly dropping RPKI-invalid routes

2. BGP Prefix Filtering:

• IRR-based filters: Build filters from Internet Routing Registry data
• Customer prefix limits: Maximum-prefix to prevent leaks
• Bogon filtering: Reject private/unallocated space

3. BGPSec (Emerging):

Cryptographically sign AS paths:

• Each AS signs the path, proving it handled the route
• Prevents AS path manipulation
• Very limited deployment due to complexity and performance

4. Mutually Agreed Norms for Routing Security (MANRS):

Industry initiative with commitments:

• Filtering: Prevent propagation of incorrect routing
• Anti-spoofing: Prevent traffic with spoofed source addresses
• Coordination: Maintain contact information, respond to incidents
• Global validation: RPKI ROAs for own prefixes

Not every router on the Internet carries the complete global routing table. The routers that do form the Default-Free Zone (DFZ)—they have no default route because they know explicit paths to every destination.

Default-Free Zone Characteristics:

Routing Table Growth:

The DFZ routing table has grown dramatically:

Year     IPv4 Prefixes    IPv6 Prefixes
2000     ~80,000          ~200
2010     ~350,000         ~4,000
2020     ~850,000         ~100,000
2024     ~950,000+        ~200,000+

Factors Driving Growth:

1. More networks: Continued Internet expansion
2. Deaggregation: Traffic engineering via more-specifics
3. IPv4 fragmentation: Smaller allocations as space exhausted
4. Multihoming: More enterprises getting their own AS/prefixes

Hardware Requirements:

DFZ routers require significant resources:

Reducing DFZ Participation:

Not every router needs the full table:

1. Default Route: Small networks send all non-local traffic to upstream
2. Partial Tables: Accept local/regional routes, default for rest
3. BGP Communities: Request only specific routes from transit

Example Partial Table Request:

! Request only North American routes
route-map TRANSIT-IN permit 10
  match community NA-ROUTES
route-map TRANSIT-IN permit 20
  match ip address prefix-list DEFAULT-ONLY
  ! Accept only 0.0.0.0/0

neighbor 10.0.0.1 route-map TRANSIT-IN in

Convergence Implications:

A full BGP table means:

• Millions of path calculations during convergence
• Significant CPU/memory during peer reset
• Longer reconvergence after major failures

Smaller networks should carefully consider whether DFZ participation is necessary.

BGP has evolved significantly since its creation in 1989, but the Internet continues to change. Several developments are shaping the future of inter-domain routing.

Emerging Technologies:

1. Segment Routing:

SR enables source routing across network domains:

• Traffic engineering without signaling protocols
• Integration with BGP via BGP-SR extensions
• Inter-domain SR emerging for multi-domain paths

2. Secure Path Validation:

Beyond RPKI origin validation:

• BGPSec for full path validation (limited adoption)
• ASPA (Autonomous System Provider Authorization) for path validation
• Open-source validation tools improving accessibility

3. Software-Defined Networking:

SDN principles applied to inter-domain:

• Centralized controllers for better visibility
• Faster, more intelligent route selection
• API-driven configuration and policy

4. AI/ML for Routing:

Machine learning applications:

• Anomaly detection for hijacks/leaks
• Predictive routing for failure avoidance
• Traffic engineering optimization

Challenges Ahead:

1. Security at Scale: Full path validation remains elusive; new attack vectors emerge

2. Table Growth: More networks, more prefixes, more paths to process

3. Convergence Speed: Users expect instant failover; BGP takes seconds to minutes

4. Automation Complexity: More automation enables faster changes but also faster mistakes

5. Geopolitical Factors: Internet fragmentation, regulatory restrictions, sovereignty concerns

The Irreplaceable Protocol:

Despite decades of proposals, no BGP replacement has emerged. The protocol is:

• Incrementally improved rather than replaced
• Deeply embedded in operations and training
• Adequate for current needs (mostly)
• Too critical to undergo disruptive change

BGP will likely remain the Internet's inter-domain routing protocol for decades to come, evolving gradually as new needs emerge.

The Internet backbone is a remarkable emergent structure—thousands of independent networks choosing to interconnect, running BGP to exchange routing information. Understanding this infrastructure is essential for anyone operating networks at scale.

Module Complete:

You have now completed Module 6: BGP Operation. You understand the complete operational mechanics of the Border Gateway Protocol—from session establishment through route advertisement, best path selection, policy routing, and the protocol's critical role in the Internet backbone.

This knowledge positions you to design, operate, and troubleshoot BGP in production networks, from single-homed enterprises to global backbone providers.