TLS Cipher Negotiation

When a TLS handshake begins, the client and server face an immediate challenge: they must agree on a common set of cryptographic algorithms before any secure communication can occur. This agreement process is called cipher negotiation, and it's far more complex and security-critical than it might initially appear.

Cipher negotiation determines everything about the connection's security:

• How keys will be exchanged (RSA, Diffie-Hellman, Elliptic Curve)
• How the server will prove its identity (RSA signature, ECDSA, EdDSA)
• How data will be encrypted (AES, ChaCha20)
• How message integrity will be verified (HMAC-SHA256, GCM, Poly1305)

A poor choice in cipher negotiation can undermine the entire security model. Using deprecated algorithms exposes connections to known attacks. Misconfigured servers might allow attackers to force weak ciphers through downgrade attacks. Understanding cipher negotiation is essential for anyone configuring secure systems.

A cipher suite is a combination of cryptographic algorithms that together provide all the security functions needed for a TLS connection. Understanding cipher suite naming is essential for security configuration.

TLS 1.2 Cipher Suite Format:

TLS 1.2 cipher suites follow a specific naming pattern that describes all components:

TLS_[Key Exchange]_[Authentication]_WITH_[Encryption]_[MAC]

Let's dissect a concrete example:

TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

• TLS: Protocol identifier
• ECDHE: Elliptic Curve Diffie-Hellman Ephemeral (key exchange)
• RSA: RSA signature (server authentication)
• WITH: Separator
• AES_256_GCM: AES-256 in Galois/Counter Mode (encryption + authentication)
• SHA384: Hash function for key derivation (not integrity—GCM handles that)

TLS 1.3 Cipher Suite Simplification:

TLS 1.3 dramatically simplified cipher suite naming by separating key exchange from the cipher specification. Key exchange is now negotiated via the supported_groups extension, and signature algorithms via the signature_algorithms extension.

TLS 1.3 cipher suites only specify:

TLS_[AEAD Algorithm]_[Hash Algorithm]

TLS 1.3 Cipher Suites:

• TLS_AES_128_GCM_SHA256 — AES-128-GCM with SHA-256 HKDF
• TLS_AES_256_GCM_SHA384 — AES-256-GCM with SHA-384 HKDF
• TLS_CHACHA20_POLY1305_SHA256 — ChaCha20-Poly1305 with SHA-256 HKDF
• TLS_AES_128_CCM_SHA256 — AES-128-CCM with SHA-256 (constrained devices)
• TLS_AES_128_CCM_8_SHA256 — AES-128-CCM with 8-byte tag (IoT)

Cipher negotiation occurs during the TLS handshake's Hello message exchange. The process is asymmetric: the client proposes, and the server chooses.

ClientHello Cipher Proposal:

The client includes an ordered list of cipher suites in its ClientHello message. This list typically contains:

• Most preferred cipher suites first
• All cipher suites the client supports
• Order based on client's security preferences

Example ClientHello cipher_suites (TLS 1.2 client):

cipher_suites (12 suites)
  TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 (0xc02c)
  TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (0xc030)
  TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b)
  TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f)
  TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 (0xcca9)
  TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (0xcca8)
  TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 (0x009f)
  TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (0x009e)
  TLS_RSA_WITH_AES_256_GCM_SHA384 (0x009d)
  TLS_RSA_WITH_AES_128_GCM_SHA256 (0x009c)
  TLS_RSA_WITH_AES_256_CBC_SHA (0x0035)
  TLS_RSA_WITH_AES_128_CBC_SHA (0x002f)

ServerHello Cipher Selection:

The server examines the client's list and selects ONE cipher suite that:

1. Appears in the client's list
2. Is supported by the server
3. Is compatible with the server's certificate
4. Meets the server's security requirements

Selection Strategies:

Servers can use different selection strategies:

• Client preference order: Select the first suite from the client's list that the server supports. This respects client preferences but can result in weaker security if clients are poorly configured.

• Server preference order: Ignore client order; select the first suite from the SERVER's preference list that the client supports. This enforces server security policy.

Modern best practice: Server preference order with a carefully curated server list.

The key exchange algorithm determines how the client and server establish a shared secret over an insecure channel. This is perhaps the most security-critical component of the cipher suite.

RSA Key Exchange (Deprecated in TLS 1.3):

In RSA key exchange, the client generates a random pre-master secret, encrypts it with the server's public key (from the certificate), and sends it to the server. Only the server, possessing the private key, can decrypt it.

Critical Weakness: No forward secrecy. If the server's private key is ever compromised, all recorded sessions can be decrypted. This motivated TLS 1.3's removal of RSA key exchange.

ECDHE (Elliptic Curve Diffie-Hellman Ephemeral):

The modern standard for key exchange. ECDHE uses elliptic curve mathematics to achieve the same security as DHE with much smaller key sizes:

X25519 is the most widely deployed curve, designed by Daniel J. Bernstein. It offers excellent performance, constant-time implementations (resistance to timing attacks), and avoids potential weaknesses in NIST curves.

DHE (Finite Field Diffie-Hellman Ephemeral):

The classical Diffie-Hellman algorithm uses modular exponentiation. Still supported but generally slower than ECDHE and requires larger key sizes:

• 2048-bit DHE ≈ 112 bits of security
• 3072-bit DHE ≈ 128 bits of security
• 4096-bit DHE ≈ 140 bits of security

Prefer ECDHE when available. DHE remains a fallback for legacy systems.

Authentication in TLS serves a distinct purpose from key exchange: it proves the identity of the server (and optionally the client). The authentication algorithm must match the type of certificate deployed.

RSA Authentication:

The server's certificate contains an RSA public key. Authentication occurs through:

• TLS 1.2 (DHE/ECDHE): Server signs the ServerKeyExchange message with its RSA private key
• TLS 1.3: Server signs the handshake transcript in the CertificateVerify message

RSA Signature Schemes (TLS 1.3):

• rsa_pkcs1_sha256 — PKCS#1 v1.5 with SHA-256 (legacy compatibility)
• rsa_pss_rsae_sha256 — RSA-PSS with SHA-256 (recommended)
• rsa_pss_pss_sha256 — RSA-PSS with PSS OID in certificate

RSA-PSS (Probabilistic Signature Scheme) is preferred over PKCS#1 v1.5 due to stronger security proofs.

ECDSA (Elliptic Curve Digital Signature Algorithm):

The server's certificate contains an EC public key. ECDSA offers:

• Smaller signatures than RSA at equivalent security levels
• Faster signature verification
• Native curve compatibility with ECDHE key exchange

ECDSA Curves:

• P-256 (secp256r1) — Most widely deployed
• P-384 (secp384r1) — Higher security, slower
• P-521 (secp521r1) — Maximum security, rarely used

EdDSA (Edwards-Curve Digital Signature Algorithm):

The newest authentication algorithm, based on Edwards curves:

• ed25519 — 128-bit security, constant-time by design
• ed448 — 224-bit security

EdDSA offers deterministic signatures (no randomness needed), resistance to timing attacks, and simple, fast implementations. Ed25519 is gaining adoption in TLS 1.3.

Signature Scheme Selection (TLS 1.3):

Clients and servers communicate supported signature algorithms via the signature_algorithms extension. The intersection determines which can be used.

Once keys are established, the encryption algorithm protects all application data. Modern TLS exclusively uses AEAD (Authenticated Encryption with Associated Data) ciphers, which provide both confidentiality and integrity in a single operation.

Why AEAD?

Historically, encryption (confidentiality) and MAC (integrity) were separate operations:

1. Encrypt data with AES-CBC
2. MAC the ciphertext with HMAC-SHA

This "MAC-then-Encrypt" or "Encrypt-then-MAC" approach created vulnerabilities:

• Padding oracle attacks (POODLE, Lucky13): CBC mode padding errors leaked information
• Timing attacks: MAC verification time revealed plaintext length
• Implementation complexity: Two operations = more room for errors

AEAD ciphers integrate encryption and authentication into a single, atomic operation, eliminating these attack surfaces.

AES-GCM (Galois/Counter Mode):

The dominant AEAD cipher in TLS. Hardware acceleration (AES-NI) makes it extremely fast on modern processors.

Properties:

• Block cipher: AES-128 or AES-256
• Mode: Galois/Counter Mode
• Tag size: 128 bits (16 bytes)
• Nonce: 96 bits (12 bytes) — must never repeat with same key

AES-GCM Operation:

1. Counter mode encryption (CTR) provides confidentiality
2. Galois field multiplication (GHASH) provides authentication
3. Single pass over the data
4. Authentication tag covers ciphertext + associated data (e.g., sequence number)

Vulnerability: Nonce Reuse

If the same nonce is ever used twice with the same key, AES-GCM's security completely collapses:

• XOR of plaintexts is revealed
• Authentication key is recoverable
• All data encrypted with that key is compromised

TLS mitigates this by deriving unique nonces from the record sequence number, which is guaranteed to be unique and sequential.

ChaCha20-Poly1305:

Alternative AEAD cipher designed by Daniel J. Bernstein. Optimized for software implementations without hardware AES support.

Properties:

• Stream cipher: ChaCha20 (20 rounds)
• MAC: Poly1305 one-time authenticator
• Key: 256 bits
• Nonce: 96 bits (12 bytes)
• Tag: 128 bits (16 bytes)

Why ChaCha20-Poly1305?

• Mobile devices: Many ARM processors lack AES-NI; ChaCha20 is 3x faster in software
• Constant-time: Naturally resistant to timing side-channels
• Simplicity: Easier to implement correctly than AES-GCM
• Independent design: If AES has undiscovered weaknesses, ChaCha20 provides backup

Google deployed ChaCha20-Poly1305 in Chrome/Android for connections to their servers, specifically for mobile battery and performance improvements.

Not all cipher suites provide equal security. They can be categorized into tiers based on their cryptographic properties and known vulnerabilities.

Known Vulnerabilities by Algorithm:

RC4 — Stream cipher with measurable biases. BEAST attack mitigation drove RC4 adoption, then RC4 attacks (Bar-Mitzvah, NOMORE) led to deprecation.

3DES — Sweet32 attack: after 2^32 blocks (~32GB), birthday collision reveals plaintext patterns. Too slow and weak.

CBC mode — Padding oracles (POODLE, Lucky13) enabled plaintext recovery. Timing side-channels in MAC verification. Requires complex countermeasures.

SHA-1 — Collision attacks demonstrated (SHAttered). No longer considered secure for signatures.

RSA key exchange — No forward secrecy. Bleichenbacher attacks on PKCS#1 padding (F1 Racing, ROBOT).

Export ciphers — Deliberately weakened (40-bit, 56-bit keys) for 1990s US export regulations. FREAK, Logjam attacks exploited servers still supporting these.

Configuring cipher suites correctly is critical for security. Here are current best practices for server configuration.