Cloud Networking

The Virtual Private Cloud (VPC) is the fundamental building block of cloud network architecture. While the previous page covered virtual networking concepts abstractly, this page focuses on the practical design and implementation of VPCs in production environments.

A well-designed VPC provides:

• Isolation between workloads and from other tenants
• Segmentation to limit blast radius and enforce least privilege
• Connectivity to the internet, on-premises networks, and other cloud resources
• Security through layered firewalls, flow logging, and encryption
• Scalability to accommodate growth without re-architecture

This page examines production VPC patterns, multi-account/multi-VPC strategies, VPC peering and transit architectures, and provider-specific advanced features that distinguish robust cloud networks from fragile ones.

Production VPCs follow established patterns that balance security, operational simplicity, and cost. While every organization's requirements differ, these patterns serve as proven starting points.

The Three-Tier Architecture

The classic web application architecture translates directly to VPC design:

Tier 1: Presentation (Public)

• Contains load balancers, CDN endpoints, bastion hosts
• Public subnets with Internet Gateway routes
• Security groups allowing inbound HTTP/HTTPS (443/80)

Tier 2: Application (Private)

• Contains compute instances, containers, application servers
• Private subnets with NAT Gateway routes for outbound internet
• Security groups allowing inbound only from Tier 1

Tier 3: Data (Private, Restricted)

• Contains databases, caches, message queues
• Private subnets, typically no internet route at all
• Security groups allowing inbound only from Tier 2
• May reside in dedicated database subnets with additional controls

CIDR Block Allocation Strategy

Production VPCs require careful CIDR planning. A typical /16 VPC allocation:

10.0.0.0/16  (65,536 addresses)
│
├── 10.0.0.0/22   Public Subnets (1,024 addresses)
│   ├── 10.0.0.0/24   Public-AZ-A (256)
│   ├── 10.0.1.0/24   Public-AZ-B (256)
│   └── 10.0.2.0/24   Public-AZ-C (256)
│   └── 10.0.3.0/24   Reserved
│
├── 10.0.16.0/20  Private Subnets (4,096 addresses)
│   ├── 10.0.16.0/22  Private-AZ-A (1,024)
│   ├── 10.0.20.0/22  Private-AZ-B (1,024)
│   └── 10.0.24.0/22  Private-AZ-C (1,024)
│   └── 10.0.28.0/22  Reserved
│
├── 10.0.64.0/20  Data Subnets (4,096 addresses)
│   ├── 10.0.64.0/24  Data-AZ-A (256)
│   ├── 10.0.65.0/24  Data-AZ-B (256)
│   └── 10.0.66.0/24  Data-AZ-C (256)
│   └── Reserved for growth
│
├── 10.0.128.0/20 Reserved for future tiers
│
└── 10.0.192.0/18 Reserved for infrastructure
    ├── 10.0.192.0/24 VPN Tunnel interfaces
    ├── 10.0.193.0/24 Transit Gateway attachments
    └── Reserved

Key Principles:

1. Leave large gaps for future growth
2. Private tiers get more address space than public (most resources are private)
3. Align CIDR boundaries for clean route aggregation
4. Reserve infrastructure ranges explicitly

Essential VPC Components

Beyond subnets, production VPCs require several supporting components:

Internet Gateway (IGW)

• One per VPC (free)
• Enables bidirectional internet connectivity for public subnets
• High availability; managed by provider

NAT Gateway

• One per AZ for high availability (per-hour + per-GB cost)
• Enables outbound internet from private subnets
• Managed, scales automatically to 45 Gbps

VPC Endpoints

• Interface Endpoints: ENI in your VPC for AWS services (per-hour + per-GB)
• Gateway Endpoints: Route table entries for S3/DynamoDB (free)
• Reduce egress costs, improve security, lower latency

DNS Resolution

• Enable enableDnsHostnames and enableDnsSupport
• Private hosted zones for internal service discovery
• Resolver endpoints for hybrid DNS scenarios

Flow Logs

• VPC, subnet, or ENI-level traffic logging
• Export to CloudWatch Logs, S3, or Kinesis
• Essential for security monitoring and troubleshooting

Security Layers

• Security Groups: Required for all resources
• Network ACLs: Recommended for defense in depth
• AWS Network Firewall or third-party appliances for advanced inspection

Enterprise cloud deployments rarely use a single account/subscription. Multi-account strategies provide isolation between workloads, teams, and environments—often with separate VPCs in each account.

Why Multiple Accounts?

Blast Radius Containment: A misconfiguration or compromise in one account doesn't affect others. IAM boundaries prevent cross-account access by default.

Clear Billing Boundaries: Per-account cost visibility enables chargeback to teams or projects.

Service Limits: Account-level limits (VPCs, subnets, EIPs) multiply across accounts.

Compliance Isolation: Regulated workloads (PCI, HIPAA) can reside in hardened accounts with strict controls, separate from general workloads.

Organizational Structure: Accounts can map to teams, business units, or environments (dev/staging/prod).

AWS Landing Zone / Control Tower Pattern

AWS Landing Zone (now AWS Control Tower) establishes a multi-account architecture with centralized governance:

Account Structure:

Organization Root
├── Core OU
│   ├── Management Account (billing, organizations)
│   ├── Log Archive Account (centralized logging)
│   ├── Audit Account (security tooling)
│   └── Network Account (shared VPCs, Transit Gateway)
│
├── Production OU
│   ├── Prod-App-1 Account
│   └── Prod-App-2 Account
│
├── Non-Production OU
│   ├── Dev Account
│   └── Staging Account
│
└── Sandbox OU
    └── Experiments Account

Networking Implications:

1. Network Account hosts the Transit Gateway and core connectivity
2. Workload Accounts contain application VPCs attached to Transit Gateway
3. Shared Services VPC in Network Account provides central DNS, directory services, egress filtering
4. RAM (Resource Access Manager) shares Transit Gateway with workload accounts

Centralized vs Distributed Egress

Centralized Egress:

• All internet-bound traffic routes through a single Egress VPC
• Enables centralized NAT, firewall inspection, logging
• Increases latency for inter-AZ traffic
• Single point of control (and potential failure)

Distributed Egress:

• Each VPC has its own NAT Gateways and direct internet path
• Lower latency, simpler routing
• Harder to enforce consistent egress policies
• More NAT Gateway instances to pay for

Hybrid Approach:

• Production traffic routes through centralized firewall for inspection
• Non-sensitive traffic (OS updates, telemetry) uses local NAT
• Implement via route table differentiation (specific routes to firewall, default to local NAT)

Connecting multiple VPCs is a fundamental requirement for multi-account and multi-region architectures. Two primary mechanisms exist: VPC Peering and Transit Gateway.

VPC Peering

VPC Peering creates a private, high-bandwidth connection between two VPCs. Traffic flows over AWS/Azure/GCP backbone, never traversing public internet.

Characteristics:

• Point-to-point connection between exactly two VPCs
• Non-transitive: If A peers with B, and B peers with C, A cannot reach C through B
• Supports cross-account and cross-region peering
• No bandwidth limits (uses provider backbone)
• Minimal latency (direct path)
• Free for same-region; cross-region incurs data transfer charges

When to Use Peering:

• Small number of VPCs (< 10)
• Simple point-to-point connectivity needs
• Cost-sensitive scenarios (no hourly charges)
• When Transit Gateway limits are constraining

Limitations:

• Full mesh requires N×(N-1)/2 peering connections
• 125 peering connections per VPC limit (AWS)
• CIDR blocks cannot overlap
• No transitive routing; complex topologies are unwieldy

Transit Gateway (AWS) / Virtual WAN (Azure) / Network Connectivity Center (GCP)

Transit Gateway solves the full mesh problem by providing a regional hub that all VPCs connect to. Instead of N×(N-1)/2 peering connections, you create N attachments to the Transit Gateway.

Characteristics:

• Hub-and-spoke model: VPCs attach to Transit Gateway
• Transitive routing: A → TGW → B works out of the box
• Supports VPN and Direct Connect attachments
• Route tables within TGW for segmentation
• Cross-region peering between Transit Gateways
• Per-hour charge per attachment + data processing charges

Transit Gateway Route Tables:

TGW maintains its own route tables, separate from VPC route tables. You can create multiple TGW route tables for segmentation:

Transit Gateway Route Tables:

Production RT:
  10.0.0.0/16  → attachment-prod-vpc     (Production)
  10.1.0.0/16  → attachment-shared-vpc   (Shared Services)
  172.16.0.0/12 → attachment-vpn         (On-premises)
  0.0.0.0/0    → attachment-egress-vpc   (Internet via firewall)

Development RT:
  10.32.0.0/16 → attachment-dev-vpc      (Development)
  10.1.0.0/16  → attachment-shared-vpc   (Shared Services)
  0.0.0.0/0    → attachment-egress-vpc   (Internet via firewall)
  # Note: No route to 10.0.0.0/16 - dev cannot reach prod

This segmentation ensures development VPCs cannot route to production VPCs, even though both connect to the same Transit Gateway.

Hybrid Topology: Peering + Transit Gateway

Many organizations combine both approaches:

1. High-bandwidth, low-latency paths use direct VPC peering (e.g., application VPC to database VPC)
2. General connectivity flows through Transit Gateway (shared services, egress, on-premises)
3. Most-specific-route wins: VPC route tables can have both peering routes and TGW routes

Example Route Table:

Application VPC Route Table:
  10.0.0.0/16   local                 (VPC CIDR)
  10.1.64.0/20  pcx-dbpeer            (Direct peering to DB VPC - most specific)
  10.1.0.0/16   tgw-main              (Other traffic to Shared VPC via TGW)
  172.16.0.0/12 tgw-main              (On-premises via TGW)
  0.0.0.0/0     tgw-main              (Internet via centralized egress)

The direct peering route for 10.1.64.0/20 (database subnet) is more specific than the TGW route for 10.1.0.0/16, so database traffic takes the direct path while other traffic to Shared VPC routes through Transit Gateway.

Visibility into network traffic is essential for security monitoring, troubleshooting, and compliance. VPC Flow Logs capture metadata about traffic flowing through your network interfaces.

What Flow Logs Capture

Flow logs record metadata, not packet contents:

• Source and destination IP addresses
• Source and destination ports
• Protocol number (6=TCP, 17=UDP, 1=ICMP)
• Number of packets and bytes
• Start and end timestamps
• Action taken (ACCEPT or REJECT)
• Log status

Sample Flow Log Record:

2 123456789012 eni-abc123 10.0.1.5 10.0.2.100 443 54321 6 25 5000 1620000000 1620000060 ACCEPT OK

Interpretation:

• Version 2 format
• Account 123456789012, ENI eni-abc123
• Traffic: 10.0.1.5:443 ← 10.0.2.100:54321
• Protocol 6 (TCP)
• 25 packets, 5000 bytes
• Accepted by security group/NACL

Flow Log Scope Levels

VPC-Level: Captures all traffic for all ENIs in the VPC Subnet-Level: Captures traffic for all ENIs in the subnet ENI-Level: Captures traffic for a specific network interface

Best Practice: Enable VPC-level flow logs for comprehensive visibility. Use subnet or ENI-level logs for focused debugging or when VPC-level logs are too voluminous.

Custom Flow Log Fields (Version 5)

Recent flow log versions support additional fields:

${version} ${account-id} ${interface-id} ${srcaddr} ${dstaddr} ${srcport} 
${dstport} ${protocol} ${packets} ${bytes} ${start} ${end} ${action} ${log-status}
${vpc-id} ${subnet-id} ${instance-id} ${tcp-flags} ${type} ${pkt-srcaddr} 
${pkt-dstaddr} ${region} ${az-id} ${sublocation-type} ${sublocation-id}
${pkt-src-aws-service} ${pkt-dst-aws-service} ${flow-direction} ${traffic-path}

Useful Fields:

• tcp-flags: SYN, ACK, FIN flags for connection state analysis
• pkt-src-aws-service: Identifies traffic from AWS services (S3, Lambda)
• flow-direction: ingress or egress relative to the ENI
• traffic-path: How traffic reached the ENI (through IGW, TGW, peering, etc.)

Security Use Cases for Flow Logs

1. Detect Reconnaissance: Many REJECT entries from a single source to sequential ports indicates port scanning.

2. Identify Data Exfiltration: Unusual volume of egress traffic from unexpected instances, especially to unfamiliar IP ranges.

3. Validate Security Groups: Flow logs show what traffic actually flows. Compare against intended security group rules.

4. Troubleshoot Connectivity: "Connection refused" could be security group, NACL, or routing. Flow logs reveal where traffic is REJECTed.

5. Compliance Evidence: Demonstrate that prohibited traffic is blocked; provide audit trail for incident response.

Traffic Mirroring

Flow Logs capture metadata. For deep packet inspection, Traffic Mirroring copies actual packets to analysis tools.

Architecture:

┌─────────────┐                    ┌──────────────────────┐
│ Source ENI  │ ──── Mirrored ────→│ Network Load Balancer│
│ (App Server)│      Traffic       │  (Mirror Target)     │
└─────────────┘                    └──────────┬───────────┘
                                              │
                                   ┌──────────▼───────────┐
                                   │ IDS/IPS Appliances   │
                                   │ (Suricata, Zeek)     │
                                   └──────────────────────┘

Use Cases:

• Deep packet inspection for threat hunting
• Content inspection that flow logs can't provide
• Network forensics during incident response
• Compliance monitoring for regulated industries

Limitations:

• Significant bandwidth overhead (mirrored traffic counts against limits)
• Cost of processing infrastructure
• Privacy concerns if mirroring contains PII
• Not all instance types support mirroring

DNS resolution in cloud VPCs is more complex than traditional environments. VPCs have built-in DNS servers, but enterprise deployments often require custom DNS architectures for hybrid connectivity and service discovery.

VPC DNS Defaults

AWS Route 53 Resolver:

• Every VPC has an implicit DNS server at VPC_CIDR + 2 (e.g., 10.0.0.2)
• Resolves public DNS queries via recursive resolution
• Resolves Route 53 Private Hosted Zones associated with the VPC
• Provides automatic DNS hostnames for instances (if enabled)

Azure DNS:

• Azure-provided DNS for VNets
• Or configure custom DNS servers per VNet/NIC
• Private DNS Zones for internal name resolution

GCP Cloud DNS:

• Automatic DNS resolution for internal VPC addresses
• Private zones for custom internal domains
• DNS peering for cross-VPC resolution

Private Hosted Zones

Private Hosted Zones (AWS Route 53) / Private DNS Zones (Azure) enable custom internal domain names:

Example:

• Zone: internal.example.com
• Records:
  • api.internal.example.com → 10.0.5.25
  • db.internal.example.com → 10.0.10.100
• Associated with production VPC

Instances in the VPC resolve api.internal.example.com to the internal IP. External queries for this zone return NXDOMAIN (domain not found).

Hybrid DNS Architecture

When VPCs connect to on-premises networks, DNS becomes complex. Two scenarios:

Scenario 1: Cloud Resources Need On-Premises DNS

Cloud instances need to resolve legacy-app.corp.internal pointing to an on-premises server.

Solution: Outbound Resolver Endpoints (AWS) / DNS Forwarding (Azure/GCP)

Create endpoints in VPC subnets that forward queries for specific domains to on-premises DNS servers:

Route 53 Resolver Outbound Rule:
  Domain: corp.internal
  Forward to: 172.16.1.10, 172.16.1.11 (on-prem DNS servers)
  Via: VPN/Direct Connect

Scenario 2: On-Premises Needs Cloud DNS

On-premises servers need to resolve api.cloud.example.com pointing to VPC resources.

Solution: Inbound Resolver Endpoints

Expose DNS endpoints with private IPs in your VPC. On-premises DNS conditionally forwards to these IPs:

Route 53 Resolver Inbound Endpoint:
  ENI: 10.0.4.50, 10.0.5.50 (in different AZs)
  
On-Premises DNS Server:
  Forward zone cloud.example.com → 10.0.4.50, 10.0.5.50 (via VPN/DX)

VPCs are the cornerstone of cloud network architecture. A well-designed VPC enables security, scalability, and operational excellence; a poorly designed VPC becomes a limiting constraint that's expensive to fix.

What's Next:

With VPC architecture mastered, we'll explore Cloud Connectivity—the mechanisms for connecting cloud resources to on-premises networks, other cloud providers, and the internet. VPNs, Direct Connect, and SD-WAN will be examined in detail.