SSH Protocol: Deep Dive into Protocol Mechanics

Every SSH connection you've ever made—whether typing ssh server.example.com or pushing code to GitHub—executes a sophisticated cryptographic protocol in milliseconds. Within 100-200 milliseconds, two machines that have never communicated establish provable identities, negotiate encryption algorithms, derive shared secrets without transmitting them, and create an encrypted channel impervious to eavesdropping.

This seamless experience conceals remarkable complexity. The SSH protocol orchestrates public key cryptography, symmetric encryption, message authentication, and careful sequencing to achieve security that was once thought impossible over untrusted networks.

SSH-2 defines three core protocols that operate as a layered stack. This separation enables independent evolution and allows different implementations to share common components.

Transport Layer Protocol (RFC 4253)

The foundation providing:

• Server authentication via host keys
• Encryption and integrity protection
• Key exchange for establishing shared secrets
• Data compression (optional)

User Authentication Protocol (RFC 4252)

Builds on the transport layer to:

• Authenticate the client to the server
• Support multiple authentication methods
• Enable multi-factor authentication requirements

Connection Protocol (RFC 4254)

Provides:

• Multiplexed logical channels
• Interactive sessions and command execution
• Port forwarding and tunneling
• X11 forwarding for graphical applications

Every SSH communication after version string exchange uses a consistent packet format. Understanding this format is essential for protocol analysis and debugging.

Binary Packet Protocol:

SSH packets have this structure:

Field Details:

packet_length (4 bytes): The total length of the packet excluding this field and the MAC. This is the first value encrypted (except during key exchange).

padding_length (1 byte): The amount of random padding appended to align the encrypted block to cipher block size boundaries. The total packet (excluding MAC) must be a multiple of the cipher block size or 8 bytes, whichever is larger.

payload (variable): The actual SSH message. The first byte of the payload is always the message type identifier.

random_padding (4-255 bytes): Random bytes that:

• Ensure block size alignment for block ciphers
• Make traffic analysis more difficult by varying packet sizes
• Prevent certain cryptographic attacks

MAC (variable): Message Authentication Code computed over:

• Implicit sequence number (not transmitted)
• Unencrypted packet_length, padding_length, payload, and padding

For AEAD ciphers (AES-GCM, ChaCha20-Poly1305), the MAC is integrated into the cipher operation.

Key exchange is the cryptographic heart of SSH, establishing shared secrets over an untrusted network. SSH-2 supports multiple key exchange algorithms, with the protocol selecting based on negotiated preferences.

Key Exchange Goals:

1. Shared Secret (K): Both parties derive the same secret value without transmitting it
2. Exchange Hash (H): A hash binding the entire negotiation, signed by the server
3. Session Identifier: H from the initial key exchange, used throughout the session
4. Key Derivation: Session keys for encryption and MAC derived from K and H

SSH_MSG_KEXINIT Structure:

This message initiates algorithm negotiation:

Algorithm Selection:

For each category, the first algorithm in the client's list that also appears in the server's list is selected. This gives clients preference control while ensuring compatibility.

Exchange Hash Computation:

The exchange hash H binds the entire negotiation:

The server signs H with its host private key. This signature proves:

1. The server possesses the private key matching its claimed public host key
2. The server generated this specific response to this specific negotiation
3. No attacker modified any message during the exchange

Session Key Derivation:

From K (shared secret) and H (exchange hash), SSH derives multiple keys:

SSH-2 supports multiple key exchange algorithms. Modern deployments prefer elliptic curve methods for their combination of security and efficiency.

Curve25519 (curve25519-sha256@libssh.org, curve25519-sha256)

The current gold standard for SSH key exchange:

• Uses Daniel Bernstein's Curve25519 elliptic curve
• 128-bit security level with 256-bit keys
• Constant-time implementation resists timing attacks
• Fast: designed for efficient software implementation
• Default in modern OpenSSH versions

Diffie-Hellman Primer:

Classic DH operates in a multiplicative group modulo a large prime p:

1. Both parties agree on p and generator g
2. Client chooses random secret x, computes e = g^x mod p
3. Server chooses random secret y, computes f = g^y mod p
4. They exchange e and f
5. Client computes K = f^x mod p = g^(xy) mod p
6. Server computes K = e^y mod p = g^(xy) mod p

Both derive the same K without transmitting x or y.

Elliptic Curve Diffie-Hellman:

ECDH achieves equivalent security with smaller values:

1. Operations occur on an elliptic curve over a finite field
2. Multiplication replaces exponentiation as the "hard" operation
3. 256-bit ECDH provides ~128-bit security, matching 3072-bit DH
4. Smaller keys = faster computation and less bandwidth

Curve25519 specifically is designed for:

• Resistance to timing side-channels
• Efficient constant-time software implementation
• Security against known and theoretical attacks

After key exchange completes and SSH_MSG_NEWKEYS is exchanged, all traffic uses the negotiated encryption and integrity algorithms with derived session keys.

Authenticated Encryption with Associated Data (AEAD):

Modern SSH prefers AEAD ciphers that combine encryption and integrity in a single operation:

ChaCha20-Poly1305:

Daniel Bernstein's cipher design, adopted by OpenSSH:

• ChaCha20: 256-bit stream cipher, 20 rounds
• Poly1305: Message authentication with 128-bit tags
• Excellent software performance without hardware acceleration
• Constant-time implementation natural to the algorithm
• No padding required (stream cipher)

AES-GCM:

NIST standardized AEAD mode:

• Galois/Counter Mode provides encryption and authentication
• Extremely fast with AES-NI hardware instructions
• 128-bit authentication tags
• Used by TLS, IPsec, and SSH

Non-AEAD Ciphers (Legacy):

Older configurations use separate encryption and MAC:

• CTR mode ciphers (aes256-ctr) are secure but require separate HMAC
• CBC mode ciphers are vulnerable to padding oracle attacks and should be disabled
• The MAC is computed over the unencrypted packet (encrypt-and-MAC), which can leak information

Server authentication relies on host keys—cryptographic key pairs that identify servers. The server's public key is verified against known keys, and the private key signs the key exchange to prove identity.

Host Key Verification:

When connecting to a new server, SSH clients display the host key fingerprint:

The authenticity of host 'server.example.com (192.0.2.1)' can't be established.
ED25519 key fingerprint is SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.
Are you sure you want to continue connecting (yes/no/[fingerprint])?

This "Trust on First Use" (TOFU) model establishes initial trust. Subsequent connections verify the server presents the same key—a changed key triggers a warning indicating potential man-in-the-middle attack.

Ed25519: The Modern Choice

Ed25519 offers compelling advantages:

• Fast: Signature generation and verification are very quick
• Small: 32-byte public keys, 64-byte signatures
• Secure: Edwards curve design resists known attacks
• Side-channel resistant: Constant-time implementation

RSA Considerations:

RSA remains widely deployed but requires attention:

• Key size: 3072+ bits recommended, 4096 for higher security
• Signature algorithm: rsa-sha2-256 or rsa-sha2-512 required; ssh-rsa uses SHA-1
• Performance: Larger keys = slower operations

Host Key Storage:

Servers store host keys in /etc/ssh/:

• ssh_host_ed25519_key / ssh_host_ed25519_key.pub
• ssh_host_rsa_key / ssh_host_rsa_key.pub
• etc.

Clients store known host keys in ~/.ssh/known_hosts:

server.example.com ssh-ed25519 AAAA...base64-encoded-public-key...

Long-running SSH sessions must manage cryptographic resources to maintain security. SSH supports rekeying—performing a new key exchange during an active session to refresh session keys.

Why Rekey?

1. Limit key exposure: The more data encrypted with a key, the more material for potential attacks
2. 32-bit sequence numbers: SSH sequence numbers are 32-bit; after 2^32 packets, they wrap, potentially enabling replay attacks
3. Cipher-specific limits: Some ciphers have limits on data encrypted with a single key (e.g., AES-GCM)
4. Policy compliance: Security policies may mandate periodic rekeying

OpenSSH Rekeying Defaults:

Rekeying Process:

Rekeying follows the same protocol as initial key exchange, but within the encrypted session:

1. Either party sends SSH_MSG_KEXINIT
2. Algorithm negotiation proceeds (may select different algorithms)
3. DH/ECDH key exchange establishes new shared secret K'
4. New session keys derive from K' and original session_id
5. Both parties send SSH_MSG_NEWKEYS
6. All subsequent packets use new keys

The original session identifier remains constant—it binds the entire session to the initial key exchange.

Configuration Options:

# In ssh_config or sshd_config:
RekeyLimit 1G 1h    # Rekey after 1GB or 1 hour, whichever first
RekeyLimit 500M 30m # More aggressive: 500MB or 30 minutes
RekeyLimit default  # Use cipher-specific defaults