The three-way handshake does more than establish sequence numbers—it's a capability discovery mechanism that shapes how the connection behaves for its entire lifetime. Every TCP option, every parameter, every performance characteristic is locked in during these three segments.

Modern TCP connections use sophisticated features like window scaling for high-bandwidth paths, selective acknowledgments for efficient loss recovery, and timestamps for accurate RTT measurement. But none of these work unless both endpoints agree during the handshake. Miss the negotiation window, and you're stuck with 1985-era TCP capabilities.

TCP was designed in 1981 when networks were slow (kilobits), lossy, and simple. Over decades, TCP has evolved through backward-compatible extensions—optional features that enhance performance when both endpoints support them.

The Negotiation Challenge:

How do two endpoints discover each other's capabilities without a pre-arranged agreement? The solution: option fields in the TCP header. Each SYN and SYN-ACK carries optional capability advertisements. If both sides include an option, the feature is enabled. If either omits it, the feature is unavailable.

Why This Design?

1. Backward Compatibility — A new client with advanced options can still connect to an old server. The old server ignores unknown options and responds without them.

2. Path Transparency — Middleboxes (firewalls, NATs) may alter or strip options. Negotiation happens end-to-end with whatever survives.

3. Flexibility — Different connections can negotiate different capabilities based on endpoint support and path characteristics.

The Maximum Segment Size (MSS) option is arguably the most important parameter negotiated during the handshake. It specifies the largest TCP segment (data portion) that an endpoint is willing to receive.

Why MSS Matters:

TCP segments are encapsulated in IP datagrams. If a segment is too large, the IP datagram will be fragmented—broken into smaller pieces that are reassembled at the destination. Fragmentation is extremely undesirable:

• Overhead — Each fragment has its own IP header (20-40 bytes of waste)
• Reliability — If any fragment is lost, the entire segment is lost
• Performance — Reassembly consumes CPU cycles and memory
• Middlebox Issues — Many firewalls/NATs handle fragments poorly

MSS Calculation:

MSS = MTU - IP_Header - TCP_Header

For standard Ethernet with IPv4:

MSS = 1500 - 20 - 20 = 1460 bytes

For Ethernet with IPv6:

MSS = 1500 - 40 - 20 = 1440 bytes

MSS Negotiation Mechanics:

1. Client includes MSS option in SYN — Declares max segment size it can receive
2. Server includes MSS option in SYN-ACK — Declares its max segment size
3. Each side uses peer's MSS — When sending data, use the MSS the receiver advertised

Key Insight: MSS is per-direction. The client might send MSS=1460 while server sends MSS=1400. The client then sends segments ≤1400, and server sends segments ≤1460.

Path MTU Discovery (PMTUD):

MSS from the handshake assumes local interface MTU, but the path between endpoints may have lower MTU (e.g., VPN tunnels). Path MTU Discovery:

1. Send large segments with Don't Fragment (DF) bit set
2. If path has smaller MTU, router returns ICMP "Fragmentation Needed"
3. TCP reduces segment size based on ICMP feedback

PMTUD adjusts MSS after the handshake, but initial MSS from negotiation is the starting point.

The original TCP header provides only 16 bits for window size, limiting the maximum receive window to 65,535 bytes. For modern high-bandwidth, high-latency networks, this is catastrophically insufficient.

The Bandwidth-Delay Product Problem:

Maximum Throughput = Window Size / RTT

With 64KB window and 100ms RTT:
Max Throughput = 65,535 bytes / 0.1s = 655,350 bytes/s ≈ 5.2 Mbps

A 100ms RTT path with 64KB window is capped at 5.2 Mbps—unusable for modern networks. A 1 Gbps link with 100ms RTT needs:

Required Window = 1 Gbps × 0.1s = 100 Mbit = 12.5 MB

Window scaling (RFC 7323) extends the window to up to 1 GB.

How Window Scaling Works:

The Window Scale option specifies a shift count (0-14). The actual window is:

Actual_Window = Window_Field × 2^(Scale_Factor)

Examples:

Window Scale Negotiation Rules:

1. Both sides must participate — If client includes Window Scale but server doesn't respond with Window Scale, the feature is disabled for the entire connection

2. Each side chooses its own scale — Client might use scale=7, server might use scale=5. They don't need to match.

3. Scale applies to advertised windows — When client advertises window with scale=7, server multiplies the window field by 128 to get actual receive window

4. Scale is fixed for connection lifetime — Cannot change after handshake

Negotiation Example:

Client SYN:     Window=65535, Window_Scale=7
Server SYN-ACK: Window=32768, Window_Scale=5

Client's actual receive window: 65535 × 128 = 8,388,480 bytes
Server's actual receive window: 32768 × 32 = 1,048,576 bytes

Standard TCP acknowledgments are cumulative—they acknowledge all bytes up to a sequence number but can't specify gaps. When packet 3 of 5 is lost, the receiver can only say "I have bytes up to packet 2." The sender doesn't know if packets 4 and 5 arrived.

The Problem with Cumulative ACKs:

Sent:     [1][2][3][4][5]
Received: [1][2][ ][4][5]   (Packet 3 lost)
ACK:      "I have up to sequence 2001"

Sender knows packet 3 is missing, but:
- Did 4 and 5 arrive?
- Should I retransmit only 3, or 3, 4, and 5?

Without SACK, TCP often retransmits more than necessary (Go-Back-N behavior) or uses complex algorithms (NewReno, etc.) to probe what was received.

Selective Acknowledgments (SACK):

SACK (RFC 2018) lets receivers explicitly report which non-contiguous blocks were received:

ACK: "Cumulative ACK for 2001; SACK blocks: [4001-5000], [6001-7000]"

Now the sender knows:

• Bytes 0-2000 received (cumulative)
• Bytes 2001-3999 missing (gap before first SACK block)
• Bytes 4001-5000 received (first SACK block)
• Bytes 5001-5999 missing (gap between SACK blocks)
• Bytes 6001-7000 received (second SACK block)

Retransmit only the missing ranges: major efficiency gain.

SACK Negotiation:

SACK Negotiation Success:

If both SYN and SYN-ACK contain SACK-Permitted:

• SACK is enabled for the connection
• Receiver can include SACK blocks in ACKs when appropriate
• Sender can use SACK information for precise retransmission

If SACK Not Negotiated:

• Receiver can still receive out-of-order data and buffer it
• But receiver cannot inform sender of what was received
• Sender falls back to cumulative-ACK-based recovery (NewReno, etc.)

The TCP Timestamps option (RFC 7323) provides two critical capabilities: accurate RTT measurement and Protection Against Wrapped Sequences (PAWS).

Timestamps Option Structure:

The option contains two 32-bit values:

+-------+-------+-------------------+-------------------+
| Kind  | Len   |   TSval (4 bytes) |   TSecr (4 bytes) |
|  8    | 10    |   Sender's time   |   Echo of peer's  |
+-------+-------+-------------------+-------------------+

• TSval (Timestamp Value): Sender's current timestamp
• TSecr (Timestamp Echo Reply): Most recent TSval received from peer

RTT Measurement:

Timestamps enable per-acknowledgment RTT calculation:

1. Sender includes TSval = T1 in outgoing segment
2. Receiver echoes TSval in TSecr of ACK
3. Sender receives ACK with TSecr = T1
4. Sender calculates RTT = CurrentTime - TSecr = CurrentTime - T1

Without timestamps, RTT can only be measured when retransmission timer is used—coarse and infrequent. Timestamps enable fine-grained, continuous RTT tracking.

Protection Against Wrapped Sequences (PAWS):

On very fast connections (10+ Gbps), the 32-bit sequence space can wrap in seconds:

10 Gbps = 1.25 GB/s → 4 GB (sequence space) wraps in ~3.4 seconds

An old duplicate packet with sequence X might arrive after the sequence has wrapped past X again. Without PAWS, TCP might accept this old data as new.

PAWS uses timestamps to reject such duplicates:

• Segments with timestamps older than recent ones are discarded
• Prevents sequence space wrap-around issues

Timestamps Negotiation:

1. Client includes Timestamps in SYN: TSval = client_time, TSecr = 0
2. Server includes Timestamps in SYN-ACK: TSval = server_time, TSecr = client_time
3. Client includes in ACK: TSval = client_time2, TSecr = server_time

If both sides include Timestamps in their handshake segments, timestamps are enabled. If either omits them, timestamps are disabled for the connection.

Timestamps in SYN Example:

Traditional TCP detects congestion only through packet loss—by the time packets are dropped, the network is already severely congested. Explicit Congestion Notification (ECN) allows routers to signal congestion before dropping packets.

How ECN Works:

1. Router marking: When queues build up, ECN-capable routers set the CE (Congestion Experienced) bit in IP header instead of dropping
2. Receiver notification: Receiver sees CE bit, sets ECE (ECN-Echo) flag in TCP header
3. Sender response: Sender sees ECE, reduces congestion window, sets CWR (Congestion Window Reduced)

Result: Congestion control without packet loss—lower latency, less retransmission.

ECN Negotiation During Handshake:

ECN is negotiated using two TCP flags: ECE (ECN-Echo) and CWR (Congestion Window Reduced).

SYN (Client):

Flags: SYN, ECE, CWR

Setting both ECE and CWR in SYN indicates "I support ECN."

SYN-ACK (Server):

Flags: SYN, ACK, ECE  (if ECN supported)
     or
Flags: SYN, ACK       (if ECN not supported)

Server responds with ECE only (not CWR) to indicate ECN support.

ECN Benefits:

1. Lower latency — Congestion signaled before queues overflow; no retransmission wait
2. Better utilization — Networks can run closer to capacity without packet loss
3. Fairness with loss-based flows — ECN flows respond to congestion earlier

ECN Challenges:

• Middlebox issues — Some old firewalls drop packets with ECN flags set
• Limited deployment — Many internet paths still don't support ECN marking
• Game-ability — Malicious endpoints could ignore ECN and get more bandwidth

TCP options must fit within the header's size constraints and follow encoding rules. Understanding these mechanics helps diagnose option-related issues.

TCP Header Size Limit:

The Data Offset field is 4 bits, specifying header length in 32-bit words:

• Minimum header: 5 × 4 = 20 bytes (no options)
• Maximum header: 15 × 4 = 60 bytes
• Maximum options space: 60 - 20 = 40 bytes

Common Option Sizes:

Option Encoding Details:

Options follow a Type-Length-Value (TLV) format:

+------+--------+-----------------+
| Kind | Length |     Data        |
+------+--------+-----------------+
   1B     1B        (Length-2) B

Exceptions:

• Kind 0 (End of Options): Single byte, no length or data
• Kind 1 (NOP): Single byte, no length or data, used for padding

Alignment Requirements:

Options don't require specific alignment, but implementations often pad to 32-bit boundaries for efficiency. NOP (0x01) is inserted as needed:

Actual option bytes: [MSS:4][WSc:3][SACK:2][TS:10] = 19 bytes
With padding:        [MSS:4][NOP:1][WSc:3][SACK:2][NOP:1][NOP:1][TS:10] = 22 bytes

Option Processing Rules:

1. Unknown options should be ignored — Old hosts encountering new options skip them
2. Malformed options should cause RST — Invalid length or truncated options indicate errors
3. Order doesn't matter — Options can appear in any sequence
4. Duplicates are implementation-defined — Usually last occurrence wins

SYN Cookie Option Limitations:

When SYN cookies are active, the server doesn't store client options. The cookie can only encode:

• Limited MSS (3-4 bits, mapping to common values)
• No Window Scale, SACK, or Timestamps state

If the connection completes with cookies, these options may not be properly enabled. Modern implementations activate cookies only under attack, minimizing impact.

Let's examine real-world scenarios where parameter negotiation has dramatic effects on performance.

Scenario 1: Transcontinental High-Bandwidth Connection

Path: New York to Tokyo
RTT: 200ms
Bandwidth: 1 Gbps
Bandwidth-Delay Product: 1 Gbps × 0.2s = 25 MB

Scenario 2: Lossy Wireless Network

Path: Mobile device to cloud server
Loss rate: 2%
RTT: 80ms

Scenario 3: High-Frequency Data Center Traffic

Path: In-rack server communication
RTT: 0.1ms
Bandwidth: 25 Gbps

When TCP performance is poor, parameter negotiation issues are a common culprit. Here's how to diagnose them.

Capture and Analyze the Handshake:

Using tcpdump or Wireshark, capture the SYN and SYN-ACK:

tcpdump -i any 'tcp[tcpflags] & (tcp-syn) != 0' -vvv

Look for:

• MSS values on each side
• Window Scale presence and values
• SACK Permitted presence
• Timestamp option presence

Common Issues and Solutions:

We've thoroughly examined how TCP endpoints negotiate capabilities during the three-way handshake, and how these negotiated parameters shape connection behavior. Let's consolidate the essential knowledge:

Module Complete:

With this page, we've completed our deep dive into the TCP three-way handshake. From the initial SYN that proposes a connection, through the SYN-ACK that responds and synchronizes, to the final ACK that establishes the channel—and the parameter negotiation that shapes its behavior—you now understand this fundamental protocol mechanism at a deep, operational level.

This knowledge enables you to debug connection issues, optimize performance, understand security implications, and appreciate why TCP behaves as it does in production environments.