Hybrid Cloud - Learning Module

Loading content...

0/273

VPN and Direct Connect

The Two Pillars of Hybrid Connectivity

When engineers discuss hybrid cloud connectivity, two technologies dominate the conversation: VPN (Virtual Private Network) and dedicated private connections (AWS Direct Connect, Azure ExpressRoute, GCP Cloud Interconnect). These aren't competing technologies—they're complementary tools that solve different problems and often coexist in production architectures.

Understanding when and how to use each is fundamental to hybrid cloud design. A misconfigured VPN can introduce crippling latency; an oversized Direct Connect circuit can waste thousands monthly. The difference between a hybrid architecture that enables business agility and one that becomes a maintenance burden often comes down to these connectivity decisions.

This page provides the technical depth needed to design, implement, and operate both connectivity types at production scale.

What You Will Learn

By the end of this page, you will understand the protocols, architectures, and operational characteristics of VPN and private connection technologies. You'll learn how to design for performance, redundancy, and cost optimization—and critically, how to choose between them based on workload requirements.

VPN Fundamentals

A Virtual Private Network creates an encrypted tunnel through the public internet, making geographically separated networks appear as a single, unified private network. For hybrid cloud, VPN connects on-premises data centers to cloud Virtual Private Clouds (VPCs).

VPN Protocol Deep Dive: IPSec

•IPSec (Internet Protocol Security) — The dominant protocol for site-to-site VPN. Operates at the network layer (Layer 3), encrypting and authenticating all IP packets.
•IKE (Internet Key Exchange) — Establishes the secure tunnel before data transfer. IKEv2 is preferred for its improved reliability, NAT traversal, and faster reconnection.
•ESP (Encapsulating Security Payload) — Provides confidentiality (encryption), authentication, and integrity for the actual data packets. Supported ciphers include AES-256-GCM.
•Tunnel Mode vs Transport Mode — Site-to-site VPNs use tunnel mode, encapsulating entire IP packets. The original source/destination IPs are hidden inside the encrypted payload.
•Perfect Forward Secrecy (PFS) — Ensures that session keys are not compromised even if long-term keys are later exposed. Critical for security-conscious deployments.

IPSec Phase 1 vs Phase 2 Negotiation
Phase	Purpose	Key Parameters	Result
Phase 1 (IKE)	Establish secure channel for negotiation	Encryption algo, hash algo, DH group, lifetime, authentication method	IKE Security Association (SA)
Phase 2 (IPSec)	Negotiate parameters for data encryption	ESP cipher, hash, PFS group, proxy identities (subnets)	IPSec Security Association (SA)

Cloud Provider VPN Implementations

AWS VPN uses IKEv2 with AES-256-GCM. Azure VPN Gateway also supports IKEv2. GCP Cloud VPN offers both Classic VPN (static routing) and HA VPN (IKEv2 with BGP). Always check provider documentation for supported parameters—mismatches cause tunnel establishment failures.

VPN Architecture Patterns

Cloud providers offer several VPN deployment patterns, each with different availability, performance, and complexity characteristics. Selecting the right pattern depends on your SLA requirements and operational maturity.

Single VPN Gateway

•Single cloud VPN endpoint
•Two tunnels for redundancy (same endpoint)
•SLA: 99.95% (AWS), 99.9% (Azure Basic)
•Simplest configuration
•Suitable for dev/test, non-critical workloads
•Risk: Gateway maintenance causes downtime

High Availability VPN

•Two VPN endpoints in different AZs
•Four tunnels total (2 per endpoint)
•SLA: 99.99% (AWS HA, GCP HA VPN)
•BGP dynamic routing for failover
•Production-grade availability
•Slightly higher cost and complexity

Active-Active vs Active-Passive:

VPN tunnels can operate in two modes:

Active-Active — Both tunnels carry traffic simultaneously using ECMP (Equal-Cost Multi-Path) routing. Doubles effective bandwidth but requires BGP and customer gateway support.
Active-Passive — Primary tunnel carries all traffic; secondary activates only on primary failure. Simpler but wastes standby capacity and introduces failover delay.

Production deployments should prefer Active-Active for both bandwidth and seamless failover.

aws-vpn-terraform.tf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
# AWS Site-to-Site VPN with High Availability
# Creates two VPN connections to two Customer Gateways (on-prem side)
 
resource "aws_vpn_gateway" "main" {
  vpc_id = aws_vpc.main.id
  
  tags = {
    Name = "hybrid-vpn-gateway"
  }
}
 
# First Customer Gateway (on-prem router 1)
resource "aws_customer_gateway" "onprem_1" {
  bgp_asn    = 65000
  ip_address = var.onprem_router_1_public_ip
  type       = "ipsec.1"
  
  tags = {
    Name = "onprem-cgw-1"
  }
}
 
# Second Customer Gateway (on-prem router 2)
resource "aws_customer_gateway" "onprem_2" {
  bgp_asn    = 65000
  ip_address = var.onprem_router_2_public_ip
  type       = "ipsec.1"
  
  tags = {
    Name = "onprem-cgw-2"
  }
}
 
# VPN Connection 1 - Uses BGP for dynamic routing
resource "aws_vpn_connection" "vpn_1" {
  vpn_gateway_id      = aws_vpn_gateway.main.id
  customer_gateway_id = aws_customer_gateway.onprem_1.id
  type                = "ipsec.1"
  static_routes_only  = false  # Use BGP
  
  tunnel1_inside_cidr = "169.254.10.0/30"
  tunnel2_inside_cidr = "169.254.10.4/30"
  
  tags = {
    Name = "vpn-connection-1"
  }
}
 
# VPN Connection 2 - Redundant path through second router
resource "aws_vpn_connection" "vpn_2" {
  vpn_gateway_id      = aws_vpn_gateway.main.id
  customer_gateway_id = aws_customer_gateway.onprem_2.id
  type                = "ipsec.1"
  static_routes_only  = false
  
  tunnel1_inside_cidr = "169.254.11.0/30"
  tunnel2_inside_cidr = "169.254.11.4/30"
  
  tags = {
    Name = "vpn-connection-2"
  }
}

VPN Bandwidth Limitations

Cloud VPN throughput per tunnel is typically limited: AWS ~1.25 Gbps, Azure ~1.25 Gbps (VpnGw1), GCP ~3 Gbps. For higher bandwidth, use multiple tunnels with ECMP or consider Direct Connect. Never assume VPN can handle bulk data migrations without testing.

Direct Connect Deep Dive

Private connections (Direct Connect, ExpressRoute, Cloud Interconnect) establish dedicated network links between on-premises infrastructure and cloud providers, bypassing the public internet entirely. This provides consistent latency, higher bandwidth, and enhanced security—at significantly higher cost and complexity.

AWS Direct Connect Architecture

•Physical Connection — A dedicated 1 Gbps or 10 Gbps Ethernet port at an AWS Direct Connect location. You or your partner provides cross-connect to your network.
•Virtual Interfaces (VIFs) — Logical connections over the physical link. Private VIFs connect to VPCs; Public VIFs access AWS public services; Transit VIFs connect to Transit Gateway.
•BGP Peering — All Direct Connect routing uses BGP. You must configure BGP sessions for each VIF. AWS advertises VPC CIDRs; you advertise on-prem routes.
•VLANs — Each VIF gets a unique 802.1Q VLAN tag. Multiple VIFs can share a single physical connection for different VPCs or accounts.
•Direct Connect Gateway — Enables a single Direct Connect to reach VPCs in multiple regions. Central control point for global hybrid architectures.

Cloud Provider Private Connection Comparison
Feature	AWS Direct Connect	Azure ExpressRoute	GCP Cloud Interconnect
Port Speeds	1 Gbps, 10 Gbps, 100 Gbps	50 Mbps - 100 Gbps	10 Gbps, 100 Gbps (Dedicated)
Locations	100+ worldwide	180+ globally	150+ locations
BGP Required	Yes	Yes	Yes (Dedicated), Optional (Partner)
Layer 2 Extension	Via partner overlay	ExpressRoute Global Reach	Partner Interconnect
Encryption Option	MACsec at edge routers	ExpressRoute Direct with MACsec	MACsec on 100G
Multi-Cloud	Via Transit Gateway + peering	ExpressRoute Global Reach	Partner Interconnect
Pricing Model	Port-hour + data egress	Bandwidth tier + data egress	Port + egress (no per-GB on some)

Connection Types:

Dedicated Connection — You lease a physical port directly from the cloud provider's cage in a colocation facility. Requires your equipment to be in or connected to that facility. Highest performance, most complex.
Hosted Connection / Partner Connection — An AWS Partner / Azure Partner provides a sub-interface on their existing aggregation port. Simpler provisioning, potentially lower bandwidth options, shared infrastructure.
Colocation Model — Your equipment is in the same facility as the cloud provider's edge routers. Direct cross-connect establishment. Lowest latency.
Last-Mile Provider — A telecom provides a circuit from your data center to the colocation facility. Adds latency and cost but enables connectivity from anywhere.

High Availability for Private Connections

A single Direct Connect port provides no redundancy. Physical circuit failures, equipment issues, or colocation facility problems will cause complete connectivity loss. Production deployments must design for resilience.

Redundancy Architectures

•Single Location, Dual Ports — Two connections in the same Direct Connect location. Protects against port or device failure. SLA: ~99.9%. Risk: Facility-level events affect both.
•Dual Location — Connections in two geographically separate Direct Connect locations. Protects against facility outages. SLA: ~99.99%. Recommended for production.
•Dual Location + VPN Backup — Private connections as primary, VPN tunnels as tertiary failover. Provides fallback even if both Direct Connect locations fail.
•Maximum Resiliency — AWS recommends four connections: two ports in two locations, each in different device racks. Achieves highest SLA with complete redundancy.

dx-ha-architecture.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
# AWS Direct Connect High Availability Reference Architecture
# Using AWS CloudFormation to configure resilient Direct Connect
 
AWSTemplateFormatVersion: '2010-09-09'
Description: Direct Connect Resilient Architecture
 
Resources:
  # Direct Connect Gateway - shared across connections
  DirectConnectGateway:
    Type: AWS::DirectConnect::DirectConnectGateway
    Properties:
      Name: production-dx-gateway
      AmazonSideAsn: 64512
 
  # Virtual Private Gateway association to VPC
  GatewayAssociation:
    Type: AWS::DirectConnect::VirtualPrivateGatewayAssociation
    Properties:
      DirectConnectGatewayId: !Ref DirectConnectGateway
      VirtualPrivateGatewayId: !Ref ProductionVGW
 
  # Primary Private VIF - Location 1
  PrimaryVIF:
    Type: AWS::DirectConnect::VirtualInterface
    Properties:
      ConnectionId: !Ref PrimaryConnectionId  # Physical connection 1
      VirtualInterfaceName: primary-private-vif
      Vlan: 100
      Asn: 65000  # Customer ASN
      DirectConnectGatewayId: !Ref DirectConnectGateway
      AddressFamily: ipv4
      AmazonAddress: 169.254.100.1/30
      CustomerAddress: 169.254.100.2/30
 
  # Secondary Private VIF - Location 2  
  SecondaryVIF:
    Type: AWS::DirectConnect::VirtualInterface
    Properties:
      ConnectionId: !Ref SecondaryConnectionId  # Physical connection 2
      VirtualInterfaceName: secondary-private-vif
      Vlan: 200
      Asn: 65000
      DirectConnectGatewayId: !Ref DirectConnectGateway
      AddressFamily: ipv4
      AmazonAddress: 169.254.200.1/30
      CustomerAddress: 169.254.200.2/30
 
Outputs:
  DXGatewayId:
    Value: !Ref DirectConnectGateway
    Description: Direct Connect Gateway for VIF attachments

Single Direct Connect is Not Production-Ready

A single Direct Connect circuit has no SLA. AWS explicitly states: 'For production workloads or workloads where availability is critical, we recommend configuring two connections at different Direct Connect locations.' Never deploy mission-critical workloads on single-path connectivity.

BGP Routing Strategies

Border Gateway Protocol (BGP) is the routing protocol that makes hybrid connectivity work. Understanding BGP is essential for designing traffic paths, implementing failover, and troubleshooting connectivity issues.

BGP Concepts for Hybrid Cloud

•Autonomous System Number (ASN) — Unique identifier for your network. Use private ASNs (64512-65534) for on-prem; cloud providers have their own ASNs.
•Route Advertisement — BGP speakers announce which networks they can reach. On-prem advertises data center CIDRs; AWS advertises VPC CIDRs.
•AS_PATH — List of ASNs a route has traversed. Used for loop prevention and path selection. Prepending your ASN artificially lengthens the path to de-preference a route.
•LOCAL_PREF — Outbound preference for multiple paths. Higher LOCAL_PREF is preferred. Used to prefer Direct Connect over VPN.
•MED (Multi-Exit Discriminator) — Suggestion to external neighbors about which entry point to use. Lower MED is preferred.
•BGP Communities — Tags attached to routes for policy application. AWS's communities allow route scope control (e.g., local region, all regions).

BGP Traffic Engineering Scenarios
Scenario	Technique	Configuration
Prefer Direct Connect over VPN	LOCAL_PREF	Set higher LOCAL_PREF (e.g., 200) on Direct Connect learned routes
Prefer primary DC location	AS_PATH Prepending	Prepend ASN on secondary location advertisements
Load balance across paths	ECMP	Ensure identical AS_PATH length and metrics on all paths
Control AWS-side preference	MED	Set lower MED on preferred customer gateway
Limit route propagation	BGP Communities	Use AWS-specific communities to control regional advertisement

Failover Behavior:

When a BGP session fails (tunnel down, interface failure), routes are withdrawn. The BGP Hold Timer (default 90s in many implementations, configurable lower) determines how long before routes are considered stale.

For faster failover:

Reduce BGP Hold Timer to 30s
Enable BGP Keepalive (default 30s, reduce to 10s)
Use BFD (Bidirectional Forwarding Detection) for sub-second detection

AWS VPN supports BGP with configurable timers. Direct Connect supports BFD for rapid failure detection.

BFD for Sub-Second Failover

Bidirectional Forwarding Detection (BFD) can detect link failures in milliseconds, triggering BGP route withdrawal immediately. For production hybrid architectures requiring minimal failover time, enabling BFD on Direct Connect Virtual Interfaces is highly recommended.

Performance Optimization

Connectivity is only useful if it performs well. Hybrid architectures introduce latency, potential bandwidth constraints, and TCP optimization challenges that require careful attention.

Performance Optimization Techniques

•MTU Sizing — Tunnel encapsulation adds headers, reducing effective MTU. VPN typically supports 1400 bytes; Direct Connect up to 9001 (jumbo frames). Set TCP MSS clamping to avoid fragmentation.
•TCP Window Scaling — High-latency links benefit from larger TCP windows. Ensure TCP Window Scale option is enabled (RFC 1323). Aim for bandwidth-delay product coverage.
•Bandwidth Provisioning — Private connections require upfront sizing. Monitor utilization; 80% average utilization indicates need for upgrade. Burstable traffic may exceed provisioned rates briefly.
•Path Selection for Latency — Traffic may traverse unexpected paths. Use traceroute to verify end-to-end path. Cloud provider inter-region latencies vary; test before architecting.
•Application-Level Optimization — Design applications for WAN latency: batch API calls, use connection pooling, implement caching. Chatty protocols suffer over hybrid links.

Latency Expectations by Connectivity Type
Connection Type	Typical Latency	Latency Variability	Suitable Workloads
VPN over Internet	20-100ms+	High (jitter, congestion)	Async operations, batch jobs, fault-tolerant apps
Direct Connect (same metro)	1-5ms	Very Low	Transactional databases, real-time sync
Direct Connect (cross-region)	10-50ms	Low	Cross-region replication, DR workloads
Direct Connect + VPN overlay	5-15ms overhead	Low	Encrypted Direct Connect traffic

tcp-optimization.sh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
#!/bin/bash
# Linux TCP optimization for high-latency hybrid links
# Apply on application servers connecting across hybrid boundary
 
# Enable TCP SACK (Selective Acknowledgment)
sysctl -w net.ipv4.tcp_sack=1
 
# Enable TCP Window Scaling
sysctl -w net.ipv4.tcp_window_scaling=1
 
# Increase TCP buffer sizes for high BDP links
# BDP = Bandwidth * Delay (e.g., 100 Mbps * 50ms = 625KB)
sysctl -w net.core.rmem_max=16777216
sysctl -w net.core.wmem_max=16777216
sysctl -w net.ipv4.tcp_rmem="4096 87380 16777216"
sysctl -w net.ipv4.tcp_wmem="4096 65536 16777216"
 
# Enable TCP timestamps for PAWS (Protection Against Wrapped Sequences)
sysctl -w net.ipv4.tcp_timestamps=1
 
# Congestion control: consider bbr for WAN links
modprobe tcp_bbr
sysctl -w net.ipv4.tcp_congestion_control=bbr
 
# MTU path discovery
sysctl -w net.ipv4.ip_no_pmtu_disc=0
 
echo "TCP optimization applied for hybrid connectivity"

The Bandwidth-Delay Product

On high-latency links, small TCP windows cause underutilization regardless of available bandwidth. Calculate BDP: if you have 100 Mbps and 50ms RTT, you need 625KB in-flight data to saturate the link. Default TCP window sizes may be insufficient.

Cost Analysis and Optimization

Connectivity costs are often underestimated in hybrid cloud budgets. Understanding the pricing models and optimization strategies can result in significant savings.

VPN Costs:

AWS: ~$0.05/hour per VPN connection ($36/month) + data transfer
Azure: $0.038-$1.25/hour depending on gateway SKU
GCP: ~$0.05/hour per tunnel hour

Direct Connect Cost Components (AWS Example)
Cost Component	Pricing (Example)	Notes
Port Hour (1 Gbps)	~$0.30/hour ($219/month)	Fixed cost regardless of utilization
Port Hour (10 Gbps)	~$2.25/hour ($1,642/month)	Significant commitment
Data Transfer Out (same region)	~$0.02/GB	Reduced vs internet egress (~$0.09/GB)
Data Transfer Out (cross-region)	~$0.02/GB	Same as standard inter-region rates
Partner Port (Hosted)	Partner-dependent	Often lower for sub-1Gbps needs
Cross-Connect	$100-500/month	Colocation facility charges

Cost Optimization Strategies

•Right-Size Port Speed — Don't provision 10 Gbps if 1 Gbps meets 80th percentile utilization with headroom. Port costs scale significantly with speed.
•Use Hosted/Partner Connections for Lower Bandwidth — If you need 200 Mbps, partner connections offer fractional pricing without dedicated port costs.
•Aggregate Traffic via Transit Gateway — Rather than multiple Direct Connect VIFs to each VPC, use Transit Gateway to share single Direct Connect across VPCs.
•Monitor Utilization — Overprovisioned connections waste money; underprovisioned connections throttle applications. Track metrics to right-size.
•Compare Data Transfer Costs — Sometimes internet egress is cheaper than maintaining private connectivity for low-volume workloads. Do the math.
•Commitment Discounts — Some providers offer discounts for 1-year or 3-year commitments on private connections. Evaluate long-term stability before committing.

Break-Even Analysis

Direct Connect's reduced data transfer rates ($0.02/GB vs $0.09/GB) mean it becomes cost-effective at scale. For AWS, if you transfer >4TB/month, Direct Connect 1 Gbps often pays for itself through egress savings alone. Build a spreadsheet with your expected traffic patterns.

Summary: VPN vs Direct Connect Decision Framework

We've covered the technical depth of both connectivity options. Let's consolidate into a decision framework:

When to Use Each Connectivity Type
Factor	Choose VPN	Choose Direct Connect
Budget	Tight; under $500/month	Can invest $1000+/month
Deployment Speed	Need connectivity in hours/days	Can wait weeks/months
Latency Requirement	Tolerant of 50-100ms+ variability	Requires consistent <10ms
Bandwidth	Under 500 Mbps sustained	Gbps-level throughput needed
Data Volume	Low-medium (< 5 TB/month)	High (many TBs/month)
Compliance	Standard security OK	Data must not traverse internet
Redundancy	Acceptable periodic downtime	99.99% availability required

Key Takeaways

•VPN is quick, cheap, and sufficient for many workloads — Don't over-engineer connectivity for dev/test or low-volume production.
•Direct Connect provides the gold standard for production — When latency, bandwidth, and compliance matter, private connectivity is worth the investment.
•Use both for resilience — Direct Connect as primary, VPN as backup ensures business continuity even during private link failures.
•BGP is your friend — Understanding BGP enables traffic engineering, failover, and multi-path optimization that static routing cannot achieve.
•Optimize TCP for WAN — Hybrid links expose TCP inefficiencies. Tune buffer sizes and congestion control for your latency-bandwidth profile.
•Mind the costs — Connectivity has ongoing costs. Monitor utilization and optimize port sizes, data paths, and commitment levels.

What's next:

With connectivity established, the next challenge is data. How do you synchronize databases across hybrid boundaries? How do you handle data sovereignty requirements? The next page explores Hybrid Data Strategies—replication patterns, caching, and consistency models for data that spans on-prem and cloud.

Page Complete

You now have comprehensive knowledge of VPN and Direct Connect technologies. This technical foundation enables you to design, implement, and operate hybrid connectivity that meets your organization's performance, security, and budget requirements.