Understanding what load balancing does is only half the battle. Knowing where to place load balancers in your architecture is equally critical. The same load balancing concept manifests very differently when positioned at the network edge versus in the middle tier versus in front of databases.

Think of load balancer placement as deciding where to station traffic controllers in a city. Controllers at highway entrances manage very different traffic than controllers at downtown intersections or parking garage entrances. Each position has unique responsibilities, constraints, and optimization opportunities.

In modern distributed systems, you'll often need load balancing at multiple layers simultaneously—and understanding the distinct role of each layer is essential for designing robust, high-performance architectures.

Modern distributed architectures typically have three distinct zones where load balancing occurs. Each zone has fundamentally different characteristics:

Why Multiple Tiers?

You might wonder: if edge load balancers distribute traffic, why do we need more load balancers internally?

The answer lies in traffic multiplication and specialization:

1. Traffic Multiplication: One external request might generate 10 internal service calls. Those 10 calls each need load balancing to reach their destination services.

2. Specialization: Edge load balancers optimize for external threats (DDoS, TLS) while internal load balancers optimize for microservice patterns (service discovery, circuit breaking).

3. Failure Isolation: If an internal service is overwhelmed, internal load balancers' health checks catch it before it cascades to the edge.

4. Independent Scaling: Internal services scale independently; they need their own load balancing that adapts to their specific patterns.

Edge load balancers sit at the boundary between the public internet and your infrastructure. They are the first line of defense and the primary entry point for all external traffic.

Characteristics of Edge Load Balancing:

Edge Load Balancer Architecture Patterns:

Pattern 1: Single Region Edge

Simplest architecture: one region, one or more edge load balancers behind a single public IP (using DNS or BGP anycast).

Users → DNS → Edge LB (with HA pair) → Backend Servers

Pattern 2: Multi-Region with GSLB

Global Server Load Balancing (GSLB) uses intelligent DNS or anycast to route users to the nearest region. Each region has its own edge load balancers.

Users → GSLB/GeoDNS → Region-specific Edge LB → Regional Backends

Pattern 3: CDN as Edge Load Balancer

CDNs (Cloudflare, Akamai, CloudFront) act as a global edge load balancing tier, with hundreds of PoPs worldwide. They handle static content and can proxy dynamic requests to origin servers.

Users → CDN PoP (caches + load balances) → Origin Servers

Middle-tier load balancers handle traffic between internal services—what's often called 'east-west' traffic as opposed to 'north-south' traffic from external clients. In microservice architectures, this internal traffic often vastly exceeds external traffic.

The Scale of Internal Traffic:

Consider a typical e-commerce request:

1. Edge LB receives request (1 external request)
2. API Gateway calls User Service (1 internal request)
3. API Gateway calls Product Service (1 internal request)
4. User Service calls Auth Service (1 internal request)
5. Product Service calls Inventory Service (1 internal request)
6. Product Service calls Pricing Service (1 internal request)
7. All services call Logging Service (5 internal requests)

Result: 1 external request → 10+ internal requests

This multiplication means internal load balancing often handles 10-100x the traffic that edge load balancing handles.

Middle-Tier Load Balancing Architectures:

Architecture 1: Centralized Internal Load Balancer

A dedicated load balancer cluster handles all internal traffic. Services call the load balancer, which routes to the appropriate backend.

Service A → Internal LB → Service B instances

• Pros: Centralized management, simpler service code
• Cons: Single point of failure risk, potential bottleneck, added network hop

Architecture 2: Client-Side Load Balancing

Each service maintains a list of available instances for target services and makes load balancing decisions locally.

Service A (with built-in LB logic) → directly to Service B instance

• Pros: No central bottleneck, lower latency, no single point of failure
• Cons: More complex service code, harder to manage consistently

Architecture 3: Sidecar/Service Mesh

A proxy runs alongside each service instance (as a 'sidecar'), handling all network communication including load balancing.

Service A → Sidecar Proxy → Sidecar Proxy → Service B

• Pros: Decouples networking from application, consistent policies, rich observability
• Cons: Added resource overhead, operational complexity

Data layer load balancing sits between application services and data stores (databases, caches, message queues). This tier has unique requirements because data access patterns are fundamentally different from HTTP traffic.

Database Load Balancing Architecture:

Connection Pooling Deep Dive:

Connection pooling is perhaps the most critical function of data layer load balancing. Consider:

• Each database connection consumes memory on the database server (~5-10MB for PostgreSQL)
• A PostgreSQL instance with 16GB RAM might support only 500 connections
• But you might have 100 application instances, each wanting 20 connections = 2000 connections

Without a connection pooler:

• Applications fight for limited connections
• Connection storms cause database overload
• Scaling applications breaks the database

With a connection pooler:

• The pooler maintains 500 connections to the database
• 2000 application 'connections' are multiplexed over 500 real connections
• Connection overhead is absorbed by the pooler, not the database
• Database can focus on query processing, not connection management

When designing a system, how do you decide which tiers need load balancing? Use this decision framework:

Tier-Specific Decision Factors:

Common Mistakes in Placement:

1. Skipping middle-tier load balancing — "Our services are simple, they can call each other directly." This works until services scale differently or fail.

2. Over-centralizing at edge — Routing all internal calls through edge load balancers adds latency and creates unnecessary coupling.

3. Ignoring data tier — Assuming databases 'just work.' Connection pooling and read distribution are almost always beneficial.

4. One-size-fits-all configuration — Using identical health check settings across tiers. Each tier has different latency tolerances.

Let's trace a request through a realistic multi-tier load-balanced architecture—an e-commerce checkout flow:

Step-by-Step Request Flow:

Step 1-2: Edge Load Balancing

• User's browser makes HTTPS request
• Cloudflare (CDN/Edge LB) receives it at nearest PoP
• Performs DDoS protection, WAF inspection, TLS termination
• Routes to AWS ALB in the nearest region

Step 3-4: Cluster Edge

• ALB distributes to Kubernetes Ingress Controllers
• Ingress Controller routes based on path/headers to appropriate service

Step 5-8: Service Mesh (Middle Tier)

• Every service-to-service call goes through Envoy sidecars
• Sidecars perform load balancing across service instances
• Circuit breakers prevent cascade failures
• Distributed tracing captures the call graph

Step 9-10: Database Tier

• PgBouncer pools connections to PostgreSQL
• Write queries go to primary
• Read queries distributed across replicas
• Connection pooling reduces database load

Step 11: Cache Tier

• Redis Cluster routes to appropriate node based on key hash
• Client library handles slot mapping

Total Load Balancing Decisions for One Request: 6-10

This illustrates why understanding placement matters—load balancing touches every tier, and each tier has distinct requirements and failure modes.

Each placement decision involves trade-offs. Here's a summary to guide your architectural choices:

When to Simplify (Fewer Tiers):

• Early-stage startups with minimal traffic
• Monolithic applications with single database
• Very simple microservice architectures (2-3 services)
• Cost-sensitive environments where managed services aren't an option

When to Invest in All Tiers:

• High-traffic applications (>10K requests/second)
• Complex microservice architectures (10+ services)
• Strict availability requirements (99.9%+)
• Global user base requiring geographic distribution
• Regulated industries requiring security perimeters

Let's consolidate the key concepts from this page:

What's Next:

With a solid understanding of load balancing fundamentals, benefits, and placement, the final page of this module addresses a critical concern: the load balancer as a single point of failure. We'll explore how to make load balancing infrastructure itself highly available.