In 2007, Netflix started streaming video using third-party CDNs like Akamai and Limelight. By 2012, as streaming exploded in popularity, Netflix was paying hundreds of millions annually for CDN services—and still facing quality limitations. The solution was radical: build a proprietary CDN from scratch.

Open Connect launched in 2012 and now delivers over 95% of Netflix traffic globally. It's not just a cost-saving measure—it's a strategic asset that enables quality and features impossible with generic CDNs.

This page examines Open Connect in depth: its architecture, deployment models, the control plane that orchestrates it, ISP partnership dynamics, and the technical innovations that make it the world's most efficient video delivery network.

Building infrastructure from scratch is usually a mistake—buy don't build. Why did Netflix choose differently? The answer lies in video streaming's unique characteristics and Netflix's scale.

The Build vs Buy Calculation:

At 2012 scale (30 million subscribers, ~3% of today's traffic), the math was already compelling:

Third-Party CDN Cost:

• ~$0.02 per GB delivered
• Billions of GB per month
• Hundreds of millions annually

Open Connect Cost:

• Hardware: $10-20K per server
• Operational cost: $2-5K per server per year
• 15,000 servers = $150-300M upfront + $30-75M/year operations

With Netflix serving exabytes of content, Open Connect pays for itself multiple times over each year. The economics only improve with scale.

The Open Connect Appliance (OCA) is a purpose-built server optimized for a single task: serving video efficiently. Every component—hardware and software—is chosen for this specific workload.

Software Stack Deep Dive:

Operating System: FreeBSD

Netflix chose FreeBSD over Linux for OCAs due to:

• Superior network stack for high-throughput streaming
• ZFS for reliable, self-healing storage
• BSD license allows proprietary modifications
• Netflix engineers actively contribute to FreeBSD kernel

HTTP Server: Custom nginx Fork

Netflix forked nginx and heavily modified it:

• Sendfile (zero-copy I/O) optimizations
• TCP tuning specifically for streaming workloads
• Custom logging for Netflix telemetry
• Integration with FreeBSD's network stack optimizations

TLS Termination:

Every video byte is encrypted. OCAs must terminate TLS at 100+ Gbps:

• Hardware acceleration via AES-NI instructions
• Session ticket reuse to minimize handshakes
• OCSP stapling for certificate validation
• TLS 1.3 for reduced round trips

Content Storage:

Content is stored as simple files on ZFS:

• Content-addressable naming (hash in filename)
• No database—filesystem is the database
• ZFS compression for rarely-accessed cold content
• Automatic scrubbing for corruption detection

Open Connect servers are deployed in three primary models, each serving different purposes and requirements. The choice of model depends on traffic volume, ISP relationships, and geographic needs.

Location Selection Criteria:

Netflix evaluates potential locations using multiple factors:

Traffic Analysis:

• Current Netflix traffic volume from that network/region
• Growth trajectory and projections
• Peak vs. average traffic ratios

Network Quality:

• Existing latency to nearest OCAs
• Packet loss rates on current paths
• Bandwidth constraints on current paths

Economic Factors:

• Cost of placement (colocation fees, power)
• Transit/peering cost savings from local serving
• ISP willingness to participate (for embedded)

Operational Considerations:

• Physical security and access
• Power reliability and redundancy
• Network connectivity redundancy
• Remote management capability

A location is added when the ROI is positive—when traffic volume justifies the capital expense and operational overhead.

While OCAs are simple content servers, the control plane that manages them is sophisticated. It handles steering decisions, cache management, health monitoring, and configuration across 15,000+ servers.

Steering Deep Dive:

When a Netflix player initiates playback, it doesn't know which OCA to connect to. The steering process:

1. Player Request — Client requests playback of title X
2. Control Plane Lookup — Service identifies client's network (via IP), desired content
3. Candidate Selection — Control plane computes ranked list of OCAs that:
   • Have the content cached (or can fetch quickly)
   • Are topologically close to client's network
   • Have available capacity
   • Have acceptable historical quality to this network
4. URL Generation — Returns URLs pointing to selected OCA(s) with fallbacks
5. Player Connection — Player connects to primary OCA; uses fallback if primary fails

Real-Time Adaptation:

Steering decisions aren't static. If an OCA becomes overloaded mid-session:

• Telemetry detects rising latency/queue depth
• Control plane removes OCA from new steering decisions
• Active sessions continue (graceful handling)
• Player can switch to different OCA at segment boundaries

This all happens in seconds—users don't notice the orchestration.

Optimal traffic steering is a complex optimization problem. Netflix has developed sophisticated algorithms that continuously learn and adapt to network conditions.

Multi-Armed Bandit Approach:

Netflix uses a variant of the multi-armed bandit algorithm for steering decisions:

The Problem:

• Many candidate OCAs could serve a request
• Historical data provides quality estimates for each
• But the best choice might be an OCA we haven't tried recently
• Need to balance exploitation (use known-good) with exploration (try others)

The Solution:

• Model each (client-network, OCA) pair as a 'bandit arm'
• Track expected quality (mean) and uncertainty (variance)
• Use Upper Confidence Bound (UCB) to select:
  • If confident about quality, pick highest expected quality
  • If uncertain, occasionally explore to update estimates

Continuous Learning:

• Every stream provides feedback: actual quality achieved
• Update models in real-time
• Network conditions change (ISP upgrades, congestion patterns)
• Model automatically adapts without manual intervention

Failover Logic:

Steering always provides multiple fallback OCAs:

Primary: oca1.netflix.com (best predicted quality)
Secondary: oca2.netflix.com (second best, different cluster)
Tertiary: oca3.netflix.com (different region, worst-case fallback)

Players try primary first, fall back on errors. This provides resilience without control plane involvement during playback.

Getting content onto OCAs before users request it is critical for quality streaming. The fill system is a complex choreography of prediction, prioritization, and network utilization.

Fill Traffic Engineering:

Fill operations must not interfere with viewer traffic. Several mechanisms ensure this:

Bandwidth Limiting:

• Fill traffic rate-limited per OCA and per path
• Automatically throttled if viewer traffic increases
• Night-hour fills can use 100% of available capacity
• Day-hour fills limited to spare capacity only

Hierarchical Filling:

• Don't fetch from origin for every OCA
• Regional fill servers aggregate demand
• Content flows: Origin → Fill servers → Edge OCAs
• Reduces origin egress by 90%+

Deduplication:

• If 10 OCAs in a cluster need the same content, fetch once and replicate locally
• Intra-cluster copying is cheap (local network)
• One upstream fetch serves multiple OCAs

Graceful Degradation:

• If fill infrastructure is overloaded, delay non-critical fills
• New releases always get priority
• Long-tail content fills can wait

Open Connect's success depends on ISP partnerships. These relationships are complex—technical, commercial, and sometimes political. Understanding them is essential for any large-scale content delivery system.

Partnership Requirements:

To qualify for Open Connect, ISPs must meet technical and business requirements:

Technical Requirements:

• Minimum subscriber base (typically 500K+ Netflix subscribers)
• Adequate space in a BGP-speaking data center
• 10 Gbps minimum connectivity (100 Gbps preferred)
• Ability to provide secure physical access for maintenance
• Support for remote out-of-band management

Operational Requirements:

• Named engineering contact for coordination
• Timely response for hardware issues (RMA)
• Notification of network changes affecting OCAs
• Cooperation on traffic steering optimization

Relationship Tiers:

Operating 15,000+ servers across 1,000+ locations requires sophisticated monitoring and streamlined operations. Netflix has developed tools and processes that enable a relatively small team to manage this massive fleet.

Operational Philosophy:

Automate Everything:

With 15,000+ servers, human-driven operations don't scale. Netflix automates:

• Health check and automatic removal of failing servers
• Software updates rolled out progressively
• Certificate rotation across entire fleet
• Capacity planning and server provisioning recommendations

No On-Site Requirements:

OCAs are designed for fully remote operation:

• Remote console access (IPMI/BMC)
• Remote power cycling
• Remote firmware updates
• Only hardware replacement requires hands-on

Hardware Replacement Process:

1. Automated detection of hardware failure
2. Server automatically drained of traffic
3. Ticket created with ISP partner
4. ISP technician (or Netflix contractor) performs replacement
5. Server rejoins fleet after automated validation

Typical time from failure to replacement: 24-72 hours depending on location.

Incident Response:

When issues occur, the incident process:

1. Automatic Detection — Anomaly detection on quality metrics
2. Automatic Mitigation — Traffic steered away from affected servers/regions
3. Alerting — Page on-call if automatic mitigation insufficient
4. Investigation — Root cause analysis using telemetry
5. Resolution — Fix applied, traffic restored
6. Post-mortem — Document learnings, prevent recurrence