TLS Handshake

When you connect to https://bank.example.com, how do you know you're actually communicating with your bank and not an attacker? The answer lies in certificate exchange—the process by which servers prove their identity using digital certificates.

Certificate exchange is the authentication cornerstone of TLS. Without it, encrypted communication would be meaningless; you'd have a perfectly secure channel to an adversary. The certificate system provides:

• Server identity verification: Proof that the server controls the domain you're connecting to
• Public key distribution: The server's public key for key exchange and signature verification
• Trust chain establishment: A verifiable path from the server to a trusted authority

This page explores the mechanics of certificate exchange: what certificates contain, how they're transmitted, how clients validate them, and the trust model that makes global secure communication possible.

Digital certificates in TLS use the X.509 standard, defined by ITU-T and documented in RFC 5280. An X.509 certificate is essentially a signed document binding a public key to an identity.

Certificate Fields:

Every X.509 v3 certificate contains these standard fields:

Critical Extensions:

X.509 v3 introduced extensions that provide essential functionality:

Subject Alternative Name (SAN): Lists all domain names and IP addresses the certificate is valid for. Modern browsers require SAN; the Subject CN field is deprecated for domain validation.

X509v3 Subject Alternative Name:
    DNS:www.example.com
    DNS:example.com
    DNS:api.example.com
    IP Address:192.0.2.1

Key Usage: Defines allowed uses of the public key—prevents a signing key from being used for encryption or vice versa.

X509v3 Key Usage: critical
    Digital Signature, Key Encipherment

Extended Key Usage: More specific usage constraints—server authentication, client authentication, code signing, etc.

X509v3 Extended Key Usage:
    TLS Web Server Authentication (1.3.6.1.5.5.7.3.1)
    TLS Web Client Authentication (1.3.6.1.5.5.7.3.2)

Basic Constraints: Indicates if the certificate can sign other certificates (CA:TRUE) or is an end-entity.

X509v3 Basic Constraints: critical
    CA:FALSE

During the TLS handshake, certificate exchange occurs after the Hello messages. The timing and encryption status differ between TLS versions.

TLS 1.2 Certificate Message:

In TLS 1.2, the Certificate message is sent in clear text, immediately after ServerHello:

ServerHello
Certificate <-- Unencrypted, visible to observers
ServerKeyExchange (if DHE/ECDHE)
ServerHelloDone

The Certificate message contains:

1. The server's end-entity certificate
2. Intermediate CA certificates (chain building)
3. (Optionally) The root CA certificate (usually omitted)

TLS 1.3 Encrypted Certificate:

In TLS 1.3, the Certificate message is encrypted with handshake traffic keys:

ServerHello
EncryptedExtensions
{Certificate} <-- Encrypted, not visible to passive observers
{CertificateVerify}
{Finished}

This is a privacy improvement—attackers cannot see which certificate (and thus which website) the server presents.

Server Name Indication (SNI):

SNI, sent in the ClientHello, tells the server which hostname the client is connecting to. This allows:

• Virtual hosting: One IP address can serve multiple HTTPS sites
• Certificate selection: Server chooses the appropriate certificate
• Privacy issue: In TLS 1.2, SNI is unencrypted—observers see the hostname

TLS 1.3 and Encrypted Client Hello (ECH):

To address SNI privacy, Encrypted Client Hello (formerly ESNI) encrypts the SNI using a public key obtained via DNS. This prevents network observers from seeing which site you're connecting to.

Certificate Chain Ordering:

The Certificate message should contain certificates in order:

1. End-entity certificate (server's certificate) — first
2. Intermediate CA certificates — in order from signer of end-entity to root
3. Root CA certificate — usually omitted (clients have their own root store)

Some servers misconfigure this order, causing validation failures on strict clients.

Certificate validation is the process of verifying that a certificate can be trusted. It's a multi-step procedure that must complete successfully for the handshake to proceed.

The Chain of Trust:

Certificate trust is established through a chain:

Root CA (self-signed, trusted by client)
    |
    └── Intermediate CA (signed by Root)
            |
            └── End-Entity Certificate (signed by Intermediate)

The client holds a list of trusted root CAs. If it can construct a chain from the server's certificate to a trusted root, and all signatures verify, the certificate is trusted.

Validation Steps:

Signature Verification Details:

For each certificate pair (subject, issuer) in the chain:

1. Extract the tbsCertificate (to-be-signed portion) from the subject certificate
2. Extract the issuer's public key from the issuer certificate
3. Verify the signature field using the public key and specified algorithm

isValid = Verify(
    issuerPublicKey,
    signature,
    Hash(tbsCertificate)
)

If verification fails at any step, the chain is invalid and the handshake must abort.

Path Length Constraints:

The pathLenConstraint in Basic Constraints limits how many intermediate CAs can exist below a given CA:

Basic Constraints: critical
    CA:TRUE, pathLenConstraint:0

This means the CA can only sign end-entity certificates, not other CAs. It's a defense against CA compromise—a stolen intermediate with pathLenConstraint:0 cannot create further intermediates.

Even with a valid certificate chain, the certificate must be for the correct host. Hostname verification ensures the certificate matches the server you intended to reach.

Where to Check:

1. Primary: Subject Alternative Name (SAN) extension — DNS: entries
2. Fallback (deprecated): Subject Common Name (CN) — only if no SAN exists

Matching Rules (RFC 6125):

Wildcard Certificate Rules:

Wildcard certificates (*.example.com) are common for covering multiple subdomains but have strict rules:

1. Single label only: *.example.com matches www.example.com but not sub.www.example.com
2. Leftmost position only: Valid: *.example.com. Invalid: www.*.com, *.*.example.com
3. No partial wildcards: w*.example.com is not valid
4. No TLD wildcards: *.com is explicitly prohibited
5. Apex domain not covered: *.example.com does NOT match example.com

Public Suffix Considerations:

Certificate Authorities cannot issue wildcards for public suffixes (like *.com, *.co.uk, *.github.io). The Public Suffix List (maintained by Mozilla) defines these boundaries, preventing overly-broad wildcards.

Verification in Practice:

Most TLS libraries perform hostname verification automatically, but some require explicit configuration:

• Python ssl: Set check_hostname=True (default since 3.4)
• Java: SSLParameters.setEndpointIdentificationAlgorithm("HTTPS")
• Go: tls.Config.ServerName must be set
• Node.js: Enabled by default in https module

Certificates may need to be invalidated before their natural expiration—typically due to private key compromise, CA errors, or organization changes. Revocation checking attempts to verify certificates haven't been revoked.

Certificate Revocation List (CRL):

Traditional revocation mechanism. CAs periodically publish signed lists of revoked certificates.

How CRL works:

1. Certificate contains CRL Distribution Points extension with URL(s)
2. Client downloads CRL from the URL
3. Client checks if certificate's serial number appears in the list
4. CRLs are cached until their next update time

CRL Limitations:

• CRLs can grow very large (DigiCert's can exceed 30MB)
• Privacy issue: CA learns which sites you visit
• Availability: If CRL server is down, what to do?
• Timeliness: CRLs might be updated hourly; revocations aren't instant

OCSP (Online Certificate Status Protocol):

A more targeted approach—clients query the CA about a specific certificate's status.

How OCSP works:

1. Certificate contains Authority Information Access extension with OCSP URL
2. Client sends request containing certificate serial number
3. CA responds with signed status: Good, Revoked, or Unknown
4. Response is cached for nextUpdate period

OCSP Stapling:

To address OCSP privacy and availability concerns, TLS supports OCSP stapling:

1. Server periodically fetches its own OCSP response from the CA
2. Server "staples" the fresh, signed response to its Certificate message
3. Client validates the stapled OCSP response (it's signed by the CA)
4. No client-to-CA communication needed

Enabling OCSP stapling (Nginx):

ssl_stapling on;
ssl_stapling_verify on;
resolver 8.8.8.8 8.8.4.4 valid=300s;

Must-Staple:

A certificate extension requiring servers to staple OCSP responses. If the server fails to staple, clients must reject the connection. This prevents attackers from blocking OCSP checks.

Certificates come in different validation levels, reflecting how thoroughly the CA verified the applicant's identity. All provide the same cryptographic security, but differ in the identity assurance they provide.

Domain Validated (DV) Certificates:

The most basic validation—CA only verifies you control the domain.

Validation methods:

• HTTP challenge: Place a specific file at http://domain/.well-known/acme-challenge/token
• DNS challenge: Create a specific TXT record
• Email challenge: Respond to email sent to admin@domain, webmaster@domain, etc.

Characteristics:

• Issued in minutes (automated)
• Lowest cost (often free via Let's Encrypt)
• No organization information in certificate
• Sufficient for most websites

Organization Validated (OV) Certificates:

CA verifies you control the domain AND verifies your organization exists.

Additional validation:

• Business registration documents
• Phone verification
• Physical address verification

Characteristics:

• Issued in days
• Organization name appears in certificate Subject
• Provides assurance of organization identity
• Prior to EV deprecation, displayed in browser

Extended Validation (EV) Certificates:

The highest validation level—extensive vetting of the organization's legal identity and operational existence.

Additional validation:

• Legal entity verification through government databases
• Physical address verification
• Operational existence (3+ years in business or verified references)
• Due diligence on the organization and domain authorization
• Verification of authority of the certificate requester

The EV UI Debate:

Historically, browsers displayed EV certificates prominently (green bar, company name). Research showed users didn't understand or notice these indicators. As of 2019-2020, major browsers removed EV visual distinctions, displaying all certificates identically. This deprecation sparked debate about whether EV provides meaningful user security.

Modern Recommendations:

• Most websites: DV certificates (via Let's Encrypt ACME) are sufficient
• Financial/healthcare: OV or EV for regulatory compliance and partner requirements
• Enterprise/B2B: OV certificates provide organization identity
• High-value targets: EV plus Certificate Transparency monitoring

In standard TLS, only the server presents a certificate. Mutual TLS (mTLS) adds client certificate authentication—both parties prove their identity.

When is mTLS used?

• Enterprise networks: Device certificates for corporate laptops/phones
• API security: Service-to-service authentication (microservices, cloud)
• IoT: Device identity in manufacturing and deployment
• Financial services: Regulatory requirements for strong authentication
• VPN and remote access: Certificate-based user authentication
• Zero-trust architecture: Continuous verification of all connections

mTLS Message Flow (TLS 1.2):

CertificateRequest Message:

The server indicates it wants client authentication by sending CertificateRequest, which contains:

• Certificate types: RSA, DSS, ECDSA—what key types are acceptable
• Signature algorithms: Which signature algorithms the server supports
• Certificate authorities: Distinguished names of accepted CA issuers

The client must present a certificate signed by one of the specified CAs.

CertificateVerify Message:

The client proves it possesses the private key for its certificate by signing a hash of all handshake messages seen so far:

CertificateVerify = Sign(ClientPrivateKey, Hash(HandshakeMessages))

The server verifies this signature using the client certificate's public key.

mTLS in TLS 1.3:

TLS 1.3 made client authentication changes:

• CertificateRequest is encrypted (sent after ServerHello)
• Post-handshake authentication is possible (request client cert after data exchange begins)
• More flexibility in when authentication occurs

Certificate exchange is the trust anchor of TLS. Without it, encryption would be pointless—you'd have a secure channel to an unknown (possibly malicious) party. The certificate system provides the identity assurance that makes secure communication meaningful.

What's Next:

With certificate exchange understood, we move to the cryptographic core of the handshake: key derivation. The next page explores how the handshake's inputs are transformed into session keys through key derivation functions, ensuring each connection has unique, secure encryption keys.