Three Way Handshake - Learning Module

Loading content...

0/228

Connection Establishment: The Complete Process

The Art of Establishing Reliable Connections

We've examined each segment of the three-way handshake in isolation: SYN, SYN-ACK, and ACK. Now it's time to step back and see connection establishment as a complete process—a carefully choreographed dance between two endpoints that creates a reliable, bidirectional communication channel from nothing.

This page integrates our knowledge into a unified understanding of how connections are born. We'll trace the complete state machine, see how the socket API maps to protocol events, explore edge cases like simultaneous open and connection timeouts, and understand what happens when things go wrong.

What You Will Master

By the end of this page, you will understand the complete connection establishment state machine, how socket API calls map to TCP events, the active vs passive open distinction, simultaneous open mechanics, connection timeout handling, and debugging connection establishment failures.

The Complete Connection Establishment State Machine

TCP is a finite state machine—at any moment, each endpoint is in exactly one of 11 possible states. Connection establishment involves transitions through a subset of these states, from CLOSED to ESTABLISHED.

The Connection Establishment States:

TCP States Involved in Connection Establishment
State	Who	Description	Entry Condition
CLOSED	Both	No connection exists	Initial state / after close
LISTEN	Server	Waiting for connection requests	Application calls listen()
SYN_SENT	Client	SYN sent, waiting for SYN-ACK	Application calls connect()
SYN_RECEIVED	Server	SYN received, SYN-ACK sent	SYN arrives on listening socket
ESTABLISHED	Both	Connection open, data can flow	Handshake complete

State Transitions:

The canonical three-way handshake follows these transitions:

Client (Active Open):

CLOSED → [send SYN] → SYN_SENT → [receive SYN-ACK, send ACK] → ESTABLISHED

Server (Passive Open):

CLOSED → [listen()] → LISTEN → [receive SYN, send SYN-ACK] → SYN_RECEIVED → [receive ACK] → ESTABLISHED

Visual Representation:

TCP Connection Establishment State Machine

State Diagram

CLIENT                                    SERVER
═══════                                   ═══════
 
CLOSED                                    CLOSED
   │                                         │
   │ connect()                               │ listen()
   ↓                                         ↓
SYN_SENT ─────────── SYN ─────────────→   LISTEN
   │                                         │
   │                                         │ SYN received
   │                                         ↓
   │              SYN-ACK               SYN_RECEIVED
   │←─────────────────────────────────────   │
   │                                         │
   │ send ACK                                │
   ↓                                         │
ESTABLISHED                                  │
   │                                         │
   │ ────────────── ACK ────────────────→    │
                                             │ ACK received
                                             ↓
                                        ESTABLISHED

The 11 TCP States

The five connection establishment states are only part of TCP's full state machine. The remaining six states (FIN_WAIT_1, FIN_WAIT_2, CLOSING, TIME_WAIT, CLOSE_WAIT, LAST_ACK) handle connection termination. A connection must pass through ESTABLISHED to reach any termination state.

Active Open vs Passive Open

TCP distinguishes between two types of connection initiation based on which side acts first:

Active Open (Client): The host initiates the connection by sending a SYN. This is called 'active' because the application takes action to establish the connection.

Passive Open (Server): The host waits for incoming connections. This is 'passive' because the application simply prepares to accept connections without initiating them.

Active Open Characteristics

•Application calls connect()
•Specifies target IP and port
•Uses ephemeral source port
•Sends the first SYN
•Transitions: CLOSED → SYN_SENT
•Waits for SYN-ACK response
•Typical role: client

Passive Open Characteristics

•Application calls bind() then listen()
•Binds to specific or wildcard IP/port
•Uses well-known or registered port
•Waits for incoming SYN
•Transitions: CLOSED → LISTEN
•Responds with SYN-ACK
•Typical role: server

Simultaneous Active Open:

An unusual but valid scenario: both sides perform active open at the same time. Neither waits for an incoming connection—both send SYN to each other simultaneously.

Host A: CLOSED → [send SYN to B] → SYN_SENT
Host B: CLOSED → [send SYN to A] → SYN_SENT

Host A: SYN_SENT → [receive SYN from B, send SYN-ACK] → SYN_RECEIVED
Host B: SYN_SENT → [receive SYN from A, send SYN-ACK] → SYN_RECEIVED

Host A: SYN_RECEIVED → [receive SYN-ACK] → ESTABLISHED
Host B: SYN_RECEIVED → [receive SYN-ACK] → ESTABLISHED

The result is a valid ESTABLISHED connection. This is rare in practice because it requires both sides to know each other's address and initiate connection simultaneously.

Peer-to-Peer Connections

Simultaneous open is useful in peer-to-peer scenarios where neither host is a traditional 'server.' NAT traversal techniques like TCP hole punching rely on both peers sending SYN to each other, hoping the simultaneous open establishes a connection through NAT devices.

Socket API and Connection Establishment

Applications interact with TCP through the socket API—a set of system calls that abstract the complexity of the protocol. Understanding how API calls map to handshake events is essential for application debugging.

Server-Side Socket API Sequence:

Server Connection Setup
C/Python Pseudocode
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
// 1. Create socket
server_socket = socket(AF_INET, SOCK_STREAM, 0)
// Creates kernel data structure for managing connections
 
// 2. Bind to address
bind(server_socket, {addr: "0.0.0.0", port: 443})
// Associates socket with specific IP/port; socket still in CLOSED state
 
// 3. Start listening
listen(server_socket, backlog=128)
// Transitions socket to LISTEN state; can now receive SYNs
// backlog specifies maximum accept queue size
 
// 4. Wait for connection
client_socket = accept(server_socket)
// BLOCKS until a connection reaches ESTABLISHED
// Returns new socket for this specific connection
// Original server_socket remains in LISTEN for more connections
 
// 5. Communicate
data = recv(client_socket, buffer_size)
send(client_socket, response)

Client-Side Socket API Sequence:

Client Connection Setup
C/Python Pseudocode
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
// 1. Create socket
client_socket = socket(AF_INET, SOCK_STREAM, 0)
// Creates kernel data structure; socket in CLOSED state
 
// 2. Optional: bind to specific local address
// bind(client_socket, {addr: "192.168.1.10", port: 0})
// Usually skipped; kernel assigns ephemeral port automatically
 
// 3. Connect to server
connect(client_socket, {addr: "server.example.com", port: 443})
// Triggers SYN transmission
// BLOCKS until ESTABLISHED or error (timeout, refused, etc.)
// Socket transitions: CLOSED → SYN_SENT → ESTABLISHED
 
// 4. Communicate
send(client_socket, request)
data = recv(client_socket, buffer_size)

Socket API Calls and TCP State Transitions
API Call	Side	TCP Action	State Transition
socket()	Both	Create TCB structure	None (CLOSED)
bind()	Both	Assign local address	None (CLOSED)
listen()	Server	Enable passive open	CLOSED → LISTEN
accept()	Server	Wait for ESTABLISHED connection	Returns after SYN_RECEIVED → ESTABLISHED
connect()	Client	Initiate active open, send SYN	CLOSED → SYN_SENT → ESTABLISHED

Blocking vs Non-Blocking

By default, connect() and accept() are blocking calls—the application waits until the operation completes. Non-blocking sockets and event loops (epoll, kqueue, IOCP) allow applications to handle many connections without dedicating a thread to each. The underlying handshake is identical; only the application's waiting strategy differs.

Connection Establishment Timeline

Let's trace a complete connection establishment with precise timing, showing the interplay between TCP protocol events and application behavior.

Scenario: Client connects to a web server (RTT = 50ms)

Complete Connection Establishment Timeline
Time (ms)	Client Event	Network	Server Event
0	Application calls connect()		accept() blocking, waiting
0	Kernel: CLOSED → SYN_SENT, send SYN	→ SYN →
25		SYN in transit
50			Receive SYN
50			Kernel: LISTEN → SYN_RECEIVED
50			Send SYN-ACK
50		← SYN-ACK ←
75		SYN-ACK in transit
100	Receive SYN-ACK
100	Kernel: SYN_SENT → ESTABLISHED
100	Send ACK	→ ACK →
100	connect() returns SUCCESS
125		ACK in transit
150			Receive ACK
150			Kernel: SYN_RECEIVED → ESTABLISHED
150			accept() returns new socket

Key Timing Observations:

Client connect() blocks for ~100ms (2× one-way latency = 1 RTT)
Server accept() returns at 150ms — half an RTT after client considers connection established
First data can flow at 100ms — client can piggyback data on ACK immediately
Total handshake time: 1.5 RTT — SYN to server (0.5 RTT) + SYN-ACK to client (0.5 RTT) + ACK to server (0.5 RTT)

With Data Piggybacking:

If client sends HTTP request with the ACK:

Time 100ms: Client sends ACK + "GET /index.html HTTP/1.1..."
Time 150ms: Server receives ACK+Request, begins processing
Time 150+X: Server sends response
Time 200+X: Client receives first response byte

Total time to first response byte: ~200ms + processing time

Why Latency Matters

For a user in New York connecting to a server in Tokyo (RTT ~200ms), the handshake alone takes 300ms before any useful data exchange. Add TLS (another 1-2 RTT) and you're at 500-700ms before HTTP can begin. This is why CDNs, edge computing, and QUIC's 0-RTT are critical for global services.

Connection Timeouts and Failure Handling

Not every connection attempt succeeds. TCP must handle scenarios where packets are lost, the server is unreachable, or the connection is refused. Different failure modes have distinct behaviors.

Timeout Scenarios:

Connection Establishment Failure Modes
Scenario	Client Sees	Cause	Detection Time
Server not listening	RST received, 'Connection refused'	No application on port	~1 RTT
Server unreachable (no route)	ICMP error, 'Network unreachable'	No path to network	Immediate to seconds
Server unreachable (filtered)	Timeout, 'Connection timed out'	Firewall silently drops SYN	Kernel's connect timeout (30-120s)
SYN lost	SYN retransmits, eventual timeout	Network packet loss	After retransmit exhaustion (~30s)
SYN-ACK lost	SYN retransmits (looks like SYN lost)	Network packet loss	After retransmit exhaustion
ACK lost	Client established, server retransmits SYN-ACK	Network packet loss	Usually recovers via SYN-ACK retry

SYN Retransmission:

When a SYN receives no response, TCP retransmits with exponential backoff:

Attempt 1: SYN sent at T=0
Attempt 2: SYN retransmit at T=1s (if no SYN-ACK)
Attempt 3: SYN retransmit at T=3s (1 + 2)
Attempt 4: SYN retransmit at T=7s (1 + 2 + 4)
Attempt 5: SYN retransmit at T=15s (1 + 2 + 4 + 8)
Attempt 6: SYN retransmit at T=31s (1 + 2 + 4 + 8 + 16)
Timeout: connect() fails at T=63s (varies by OS)

Linux default: tcp_syn_retries = 6 (approximately 127 seconds total timeout) This can be tuned per-socket or system-wide.

The Silent Firewall Problem

When a firewall silently drops SYNs (no RST, no ICMP), the client has no way to know the connection will fail—it must wait through all retransmissions until timeout. This is why connection timeouts can exceed 2 minutes. Applications often implement shorter application-level timeouts to fail faster.

Application-Level Timeout Strategies:

# Python example: setting socket timeout
import socket

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.settimeout(5.0)  # Fail if connect takes > 5 seconds

try:
    sock.connect(("server.example.com", 443))
except socket.timeout:
    print("Connection timed out after 5s")
except ConnectionRefusedError:
    print("Connection refused (server sent RST)")
except OSError as e:
    print(f"Connection failed: {e}")

Application timeouts are typically 5-30 seconds—much shorter than TCP's default ~127 seconds.

Simultaneous Open: The Four-Way Handshake

When both hosts simultaneously attempt to establish a connection to each other, TCP handles this gracefully through a four-message handshake (though it's really two SYNs crossing each other, then two SYN-ACKs).

Simultaneous Open Sequence:

Simultaneous Open Message Exchange

Sequence Diagram

Host A                                    Host B
═══════                                   ═══════
 
CLOSED                                    CLOSED
   │                                         │
   │ connect()                               │ connect()
   ↓                                         ↓
SYN_SENT ─────────── SYN (A→B) ─────────→
                                          SYN_SENT
         ←───────── SYN (B→A) ────────────
   │                                         │
   │ Receive SYN while in SYN_SENT           │ Receive SYN while in SYN_SENT
   │ This is NOT the SYN-ACK we expected     │ This is NOT the SYN-ACK we expected
   │ But RFC 793 says: send SYN-ACK          │ But RFC 793 says: send SYN-ACK
   ↓                                         ↓
SYN_RECEIVED ────── SYN-ACK (A→B) ───────→
                                          SYN_RECEIVED
         ←───────── SYN-ACK (B→A) ────────
   │                                         │
   │ Receive SYN-ACK (for our SYN)           │ Receive SYN-ACK (for our SYN)
   ↓                                         ↓
ESTABLISHED                               ESTABLISHED

The Key Insight:

When a host in SYN_SENT receives a SYN (not SYN-ACK), it recognizes this as simultaneous open. Instead of being confused, it:

Notes the peer's ISN from the incoming SYN
Sends a SYN-ACK (combining its own SYN with acknowledgment of peer's SYN)
Transitions to SYN_RECEIVED
Waits for peer's SYN-ACK to complete the handshake

Why This Works:

The SYN-ACK from each side serves as both:

Acknowledgment of the peer's SYN (the ACK part)
Retransmission of our own SYN (the SYN part, redundant but harmless)

Once each side receives the peer's SYN-ACK, both ISNs have been acknowledged, and both can transition to ESTABLISHED.

Simultaneous Open Is Rare

In typical client-server applications, simultaneous open almost never happens—the server is in LISTEN, not SYN_SENT. Simultaneous open requires both hosts to know each other's address and call connect() at nearly the same time. It's mainly significant for peer-to-peer applications and NAT traversal.

Security Considerations During Establishment

Connection establishment is a particularly vulnerable phase for TCP. The protocol must handle untrusted input while allocating resources, creating opportunities for attacks.

Attack Surface During Handshake:

Handshake Attack Vectors

•SYN Flood — Overwhelm server's half-open connection capacity with SYNs from spoofed sources
•Connection Exhaustion — Complete handshakes rapidly, exhausting server's connection slots
•Slowloris (partial handshake variant) — Complete handshake but send data very slowly, tying up resources
•RST Injection — Guess sequence numbers to reset connections during establishment
•TCP Session Hijacking — Predict ISN to inject data into establishing connections

Defense Mechanisms:

Connection Establishment Security Measures
Defense	Attack Mitigated	Mechanism	Trade-off
SYN Cookies	SYN Flood	Encode state in ISN, defer TCB allocation	Option negotiation limited under attack
SYN Proxy	SYN Flood	Front-end device validates before passing to server	Additional latency, device scaling
Rate Limiting	Connection Exhaustion	Limit connections per source IP	May block legitimate proxies/NATs
Cryptographic ISN	RST Injection, Hijacking	Unpredictable sequence numbers (RFC 6528)	Minimal (standard practice)
TCP MD5 Authentication	RST Injection	Segment-level authentication via shared secret	Complex key management, limited deployment
Connection Limits	Exhaustion	Per-source-IP connection limits	May affect legitimate users behind NAT

Defense in Depth

Modern servers deploy multiple layers: SYN cookies for volumetric attacks, rate limiting for per-source abuse, cryptographic ISN for injection prevention, and application-level limits for resource protection. No single mechanism is sufficient; the combination provides robust defense.

Debugging Connection Establishment Issues

Connection establishment failures are among the most common networking issues. Systematic debugging helps identify the root cause quickly.

Debugging Toolkit:

Connection Debugging Tools and Commands
Tool	Platform	Use Case	Example
`netstat -an`	Linux/Windows	View listening ports and connection states	`netstat -an \| grep LISTEN`
`ss -tan`	Linux	Fast socket statistics, includes state	`ss -tan state syn-recv`
`tcpdump`	Linux/Mac	Capture handshake packets	`tcpdump -i eth0 'tcp[tcpflags] & tcp-syn != 0'`
`Wireshark`	All	GUI packet analysis, TCP conversation tracking	Filter: `tcp.flags.syn==1`
`telnet`	All	Test if port is reachable and accepting	`telnet server.com 443`
`nc -vz`	Linux/Mac	Connection test without protocol	`nc -vz server.com 443`
`curl -v`	All	Verbose HTTP connection details	`curl -v https://server.com/`

Systematic Debugging Approach:

Connection Failure Debugging Steps

•Verify server is listening — ss -tln or netstat -tln on server; confirm process bound to port
•Test local connectivity — Can you connect from the server itself? curl localhost:port
•Check firewall rules — iptables -L (Linux), Windows Firewall rules, cloud security groups
•Verify network path — traceroute/tracert server.com, check for network partitions
•Capture packets at client — Is SYN being sent? tcpdump should show outgoing SYN
•Capture packets at server — Is SYN arriving? If not, blocked by network device
•Check for SYN-ACK — If server receives SYN, is it responding? Check firewall, tcp configs
•Verify return path — Asymmetric routing can cause SYN-ACK to take different path and be filtered
•Examine RST — If RST received, was it from server (port closed) or intermediate device?
•Review TCP system settings — Check SYN cookie settings, backlog sizes, connection limits

The Most Common Issues

In practice, most connection establishment failures are: (1) Application not listening (typo in port, crash), (2) Firewall blocking SYN or SYN-ACK, (3) DNS resolution failure (can't get server IP), (4) Server overloaded (SYN queue full). Checking these four covers 90% of issues.

Optimizing Connection Establishment

For high-performance applications, connection establishment overhead can be significant. Several techniques reduce this impact:

Connection Reuse:

The most effective optimization is avoiding new connections entirely. HTTP Keep-Alive and HTTP/2 multiplexing allow multiple requests over a single connection, amortizing the handshake cost.

Connection Pooling:

Applications maintain a pool of pre-established connections to frequently-accessed servers. When a request arrives, an existing connection is borrowed rather than created.

Pool initialization:
  - Create 10 connections to database server
  - Store in pool

Request handling:
  - Borrow connection from pool
  - Execute query
  - Return connection to pool

Benefit: No handshake latency on each request

Connection Establishment Optimization Techniques
Technique	Description	Latency Reduction	Applicability
Connection Reuse	Keep connections open for multiple requests	1.5 RTT × (N-1) for N requests	HTTP, databases, any stateless protocol
Connection Pooling	Pre-establish connections before needed	Complete elimination of connect latency	Application servers, databases
TCP Fast Open	Data in SYN for repeat connections	1 RTT per connection	Web servers, repeat visitors
Happy Eyeballs	Race IPv4/IPv6, use winner	Avoids timeout on broken path	Dual-stack clients
Preconnect	Browser hints to establish early	Overlap with page processing	Web browsers, resource hints
QUIC/HTTP/3	0-RTT resumption, no TCP handshake	Up to 2 RTT (TCP+TLS)	Modern web traffic

TCP Fast Open (TFO) Deep Dive:

TCP Fast Open (RFC 7413) allows data transmission before the handshake completes:

FirstConnection (TFO Cookie Request):
  Client: SYN + TFO Cookie Request
  Server: SYN-ACK + TFO Cookie
  Client: ACK (+ data)
  [Normal 1.5 RTT, but client receives cookie]

Subsequent Connections (Using Cookie):
  Client: SYN + TFO Cookie + DATA (GET /)
  Server: SYN-ACK + RESPONSE DATA
  Client: ACK
  [Server can respond before ACK arrives = 0 RTT handshake]

The TFO cookie proves the client previously completed a handshake legitimately.

TFO Limitations

TCP Fast Open requires support on both client and server, may be blocked by middleboxes, and only helps repeat connections (first connection is normal). Still, for services with repeat visitors (most websites), TFO provides significant latency reduction.

Summary: Connection Establishment

We've examined TCP connection establishment as a complete process—integrating the three handshake segments with state machines, socket APIs, timing, failures, and optimization. Let's consolidate the essential knowledge:

Key Takeaways

•The state machine governs all behavior — CLOSED → LISTEN/SYN_SENT → SYN_RECEIVED → ESTABLISHED, with clear transitions for each event
•Active open vs passive open — Clients initiate (connect()), servers wait (listen()/accept())
•Socket API maps directly to TCP events — connect() triggers SYN, accept() returns after handshake completes
•1.5 RTT is the fundamental cost — Optimizations like Fast Open and connection reuse target this overhead
•Timeouts use exponential backoff — SYN retries at 1s, 2s, 4s, 8s... totaling ~127 seconds by default
•Simultaneous open is valid — Both hosts in SYN_SENT receive peer's SYN, both send SYN-ACK, both establish
•Security during establishment is critical — SYN cookies, rate limiting, cryptographic ISN all mitigate attacks
•Systematic debugging finds issues — Check listening, firewall, packet capture, return path

What's Next:

With connection establishment complete, we have an ESTABLISHED connection ready for data transfer. But the handshake does more than open a channel—it negotiates parameters that govern the connection's entire lifetime. In the next page, we'll explore parameter negotiation: MSS determination, window scaling, SACK enablement, and how options agreed during the handshake shape connection behavior.

Page Complete

You now understand TCP connection establishment as a complete process—from socket API calls through state transitions to established connections. This comprehensive view enables you to debug issues, optimize performance, and understand why TCP behaves as it does in production environments.