Simple Mail Transfer Protocol (SMTP)

After your email leaves your organization's mail server, it embarks on a journey across the Internet—hopping between servers, navigating DNS lookups, and traversing organizational boundaries until it reaches its destination. This mail relay process is the backbone of global email communication, enabling billions of messages to cross domains, networks, and continents every day.

Understanding mail relay is essential for email administrators who configure routing policies, security professionals who protect against unauthorized relay abuse, and developers who need to diagnose why mail isn't reaching its destination. The relay process involves intricate DNS lookups, trust decisions, and protocol exchanges that determine whether an email successfully arrives or disappears into a spam filter.

Mail relay is the process by which a Mail Transfer Agent (MTA) accepts a message destined for another domain and forwards it toward the recipient's mail server. Unlike mail submission (client-to-server), relay is a server-to-server operation that happens automatically, without direct user involvement.

The Relay Decision:

When an MTA receives a message via SMTP, it must make a critical decision: Should I accept and relay this message? This decision depends on:

1. Who is sending? — Is this a trusted source (internal user, authenticated client, authorized partner)?
2. Where is it going? — Is the recipient local (I should deliver) or remote (I should relay)?
3. Am I authorized? — Do policies permit me to relay for this sender/recipient combination?

Types of Relay:

1. Outbound Relay: Relaying mail from internal users to external domains. This is normal operation—your mail server sends your organization's email to the world.

2. Inbound Delivery: Receiving mail for your domain from external senders. Technically not "relay" but often handled by the same infrastructure.

3. Open Relay (Dangerous!): Relaying mail from anyone to anyone—with no authorization. This is a security disaster and the primary source of spam. An open relay forwards mail from arbitrary senders to arbitrary recipients, often exploited within hours of going online.

4. Authorized Third-Party Relay: Relaying for specific authorized parties (backup MX, smarthost relay, email service providers). This is legitimate but requires explicit configuration.

Before relaying email to a recipient domain, the sending MTA must discover which server(s) accept mail for that domain. This is accomplished through DNS MX (Mail Exchanger) records.

MX Record Structure:

An MX record specifies the hostname of a mail server and a priority value (lower = higher priority):

example.com.    IN  MX  10 mail.example.com.
example.com.    IN  MX  20 backup.example.com.
example.com.    IN  MX  30 emergency.example.net.

Resolution Algorithm:

Edge Cases in MX Resolution:

No MX Record: If no MX record exists, RFC 5321 specifies falling back to the domain's A/AAAA record. The sender should attempt delivery directly to the domain's IP address.

Null MX (MX 0 .): RFC 7505 introduced the "null MX" record indicating a domain explicitly does not accept email:

no-email.example.com.  IN  MX  0 .

Senders should immediately generate a bounce for recipients at such domains.

MX Pointing to CNAME: RFC 5321 discourages MX records pointing to CNAMEs. The MX should resolve directly to A/AAAA records. Some servers reject or mishandle CNAME chains.

Same Priority (Round-Robin): When multiple MX records share the same priority, senders should distribute load randomly among them, not always choosing the same one.

Controlling who can relay mail through your server is the most critical security configuration for any MTA. Misconfiguration creates open relays; over-restriction blocks legitimate mail.

Policy Components:

Relay authorization involves checking multiple factors:

1. Client Identity — IP address, authenticated username, TLS certificate
2. Sender Address — MAIL FROM envelope address
3. Recipient Address — RCPT TO envelope address
4. Message Content — Sometimes scanned for spam indicators

Testing Relay Configuration:

ALWAYS test your relay configuration after changes. A misconfiguration can turn your server into a spam source within hours.

# From an EXTERNAL IP (not in mynetworks), without auth:
telnet mail.example.com 25
EHLO test.external.com
MAIL FROM:<spammer@evil.com>
RCPT TO:<victim@gmail.com>

# Expected response:
# 554 5.7.1 <victim@gmail.com>: Relay access denied

# If you see "250 OK" — YOU HAVE AN OPEN RELAY!

Use External Testing Services:

• MX Toolbox relay test
• mail-tester.com
• abuse.net open relay test
• Your organization's security scanning

Email often traverses multiple servers between sender and recipient. Each hop adds a Received header, creating an audit trail of the message's journey. Understanding multi-hop routing is essential for troubleshooting delivery issues and investigating message origins.

Common Multi-Hop Scenarios:

Tracing the Route via Received Headers:

Every MTA prepends a Received header when handling a message. Reading these headers from bottom to top reveals the message path:

Key Information in Received Headers:

• from: Hostname the sender claimed (HELO/EHLO argument)
• (hostname [IP]): Actual hostname and IP from connection
• by: Server that received the message
• with ESMTPS/ESMTP: Protocol used (TLS if 'S' suffix)
• id: Server's internal queue ID for tracking
• for: Recipient (may be redacted for privacy)
• timestamp: When this hop occurred

Reading Received Headers for Troubleshooting:

1. Find the first (bottom) Received header to identify true origin
2. Check timestamps for unusual delays between hops
3. Verify ESMTPS (with TLS) was used where expected
4. Look for discrepancies between claimed and actual hostnames

Unlike submission (where TLS is mandatory for credential protection), server-to-server relay traditionally used opportunistic encryption—encrypt if possible, fall back to plaintext if not. Modern standards increasingly push toward mandatory encryption to protect message confidentiality.

Opportunistic TLS (Traditional):

C: EHLO sender.example.org
S: 250-mx.receiver.com
S: 250-STARTTLS
S: 250 OK
C: STARTTLS
S: 220 Ready
[TLS negotiation]
# ... encrypted session continues

If the server doesn't advertise STARTTLS, or TLS negotiation fails, the sender continues in plaintext. This prevents encryption from blocking mail delivery but exposes content to eavesdroppers.

The STARTTLS Stripping Attack:

Because STARTTLS is negotiated over plaintext, an active attacker can:

1. Intercept the connection between sender and receiver MTA
2. Remove STARTTLS from the server's EHLO response
3. Sender thinks TLS isn't available and continues in plaintext
4. Attacker reads/modifies all email content

This is why opportunistic TLS provides "best effort" security but not guaranteed protection.

MTA-STS: Mandatory TLS with Policy:

RFC 8461 introduced MTA-STS (Mail Transfer Agent Strict Transport Security) to solve STARTTLS stripping:

1. Receiving domain publishes TLS policy via HTTPS (https://mta-sts.example.com/.well-known/mta-sts.txt)
2. Policy specifies MX hostnames that MUST use TLS
3. Sending MTA fetches and caches policy
4. Future connections to that domain REQUIRE TLS; stripping attacks fail

# Example: https://mta-sts.example.com/.well-known/mta-sts.txt
version: STSv1
mode: enforce
mx: mail.example.com
mx: backup.example.com
max_age: 604800

DANE (DNS-based Authentication of Named Entities) provides cryptographic authentication of mail server TLS certificates using DNSSEC. It solves a fundamental problem: how can a sending server verify it's talking to the legitimate receiving server and not an imposter?

The Problem with Traditional TLS:

With standard TLS, the sender:

1. Connects to what DNS says is the MX server
2. Server presents a TLS certificate
3. Sender verifies certificate is signed by a trusted CA

But if an attacker compromises DNS, they can redirect traffic to their server with a certificate from any trusted CA (compromise one of 100+ CAs). DNSSEC + DANE closes this gap.

How DANE Works:

1. DNSSEC secures DNS responses cryptographically, preventing DNS spoofing
2. TLSA Records published via DNSSEC specify the expected TLS certificate
3. Sending MTA looks up TLSA record for receiving MX
4. Sender verifies that the server's certificate matches the TLSA record
5. If mismatch → connection rejected (attacker detected)

TLSA Record Format:

_25._tcp.mail.example.com. IN TLSA 3 1 1 <certificate_hash>

Parameters:

• Certificate Usage (3): DANE-EE (end-entity certificate)
• Selector (1): SubjectPublicKeyInfo (verify public key)
• Matching Type (1): SHA-256 hash

Multiple MX records with different priorities provide redundancy. When the primary MX is unavailable, sending servers automatically route mail to backup servers. Understanding this mechanism is essential for reliable email infrastructure.

How Backup MX Works:

example.com.  IN  MX  10 primary.example.com.
example.com.  IN  MX  20 backup1.example.com.
example.com.  IN  MX  30 backup2.example.net.

1. Sender tries primary (priority 10) first
2. If connection fails or returns 4xx error, try backup1 (priority 20)
3. Continue through list until success or all exhausted
4. If all fail, queue message and retry later

Security Concerns with Backup MX:

Backup MX servers are high-value targets for attackers because:

1. Lower Security: Backups often have weaker spam filtering ("just accept it")
2. Spam Staging: Attackers send spam to backup; when primary recovers, backup delivers the spam
3. Unauthorized Backup: Attacker adds their server as low-priority MX in compromised DNS
4. Outdated Software: Backup servers may be less frequently updated

Best Practices for Backup MX:

When a relay attempt fails, the sending MTA must decide: retry later, or give up and notify the sender? This decision affects user experience, server resources, and spam handling.

Temporary Failures (4xx) — Retry:

Temporary failures indicate the receiving server couldn't handle the message right now but might succeed later:

• 421: Service not available (server busy/shutting down)
• 450: Mailbox unavailable (quota issues, locking)
• 451: Local error in processing
• 452: Insufficient storage

Retry Behavior:

Permanent Failures (5xx) — Bounce Immediately:

Permanent failures indicate the message can never be delivered and should not be retried:

• 550: User unknown / mailbox unavailable
• 551: User not local
• 552: Message size exceeded
• 553: Invalid address syntax
• 554: Transaction failed

Delivery Status Notification (DSN / Bounce):

When giving up, the MTA generates a DSN (bounce message) sent to the envelope sender (MAIL FROM address):

From: MAILER-DAEMON@sender.example.org
To: alice@sender.example.org
Subject: Undelivered Mail Returned to Sender

This is the mail delivery agent at sender.example.org.

I was unable to deliver your message to the following address:

<unknown@receiver.com>:
    550 5.1.1 User unknown

--- Original message follows ---
[First ~50KB of original message]

DSN Security Considerations:

Backscatter Problem: Spammers forge sender addresses. When spam bounces, the DSN goes to the forged (innocent) address. This "backscatter" is itself a form of spam/DoS.

Mitigation Strategies:

1. Reject at RCPT TO, not after DATA — If you know a recipient is invalid, reject during SMTP conversation so no bounce is needed
2. Sender verification — Use SPF/DKIM/DMARC to detect forged senders; don't bounce forged mail, just drop it
3. Rate-limit bounces — Limit DSN generation to prevent being used for bounce attacks
4. Null sender policy — Be suspicious of mail with null return path (MAIL FROM:<>) except for legitimate DSNs

We've comprehensively explored mail relay—the mechanism that routes email across the Internet from organization to organization. Let's consolidate the critical knowledge:

What's Next:

With relay mechanisms understood, we turn to SMTP Extensions (ESMTP)—the evolutionary enhancements that have kept SMTP relevant for 40+ years. We'll explore the EHLO negotiation mechanism and key extensions including AUTH, STARTTLS, SIZE, 8BITMIME, SMTPUTF8, and the emerging standards that continue to improve email security and functionality.