When Bahdanau and colleagues introduced attention in 2014, they faced a fundamental design choice: should attention be a discrete selection (pick one position) or a continuous weighting (blend all positions)?

They chose the latter—soft attention—and this choice had profound implications. Soft attention is fully differentiable, enabling end-to-end training with standard backpropagation. It computes a weighted average over all positions rather than selecting a single one, providing smooth gradient flow and stable optimization.

Today, virtually all production attention mechanisms use soft attention. Transformers, BERT, GPT, and their descendants all rely on the soft, differentiable approach we'll study in depth on this page.

Soft attention defines attention as a continuous, differentiable weighted combination of values. Let's formalize this precisely.

Formal Definition:

Given:

• Query vector q ∈ ℝ^{d_q}
• Key vectors K = [k₁, ..., kₘ] ∈ ℝ^{m × d_k}
• Value vectors V = [v₁, ..., vₘ] ∈ ℝ^{m × d_v}
• Scoring function score(q, k) → ℝ

Soft attention computes:

e_i = score(q, k_i)           for i = 1, ..., m
α_i = softmax(e)_i = exp(e_i) / Σⱼ exp(e_j)
c = Σᵢ α_i · v_i             (weighted sum)

The output c is a weighted combination where:

• All α_i ≥ 0 (non-negative weights)
• Σᵢ α_i = 1 (weights sum to 1)
• Each v_i contributes proportionally to its weight α_i

Key Mathematical Properties:

1. Convex Combination: The output c lies within the convex hull of the value vectors:

c = Σᵢ α_i · v_i  where  Σᵢ α_i = 1, α_i ≥ 0

This means c is "between" the values in a geometric sense—it can be any mixture but cannot extrapolate beyond them.

2. Probabilistic Interpretation: The attention weights α form a valid probability distribution over positions:

• α_i can be interpreted as P(attend to position i | query q)
• c can be seen as the expected value under this distribution
• E[v | positions selected according to α]

3. Continuous Relaxation: Soft attention is a continuous relaxation of hard selection:

• Hard: Pick position i* = argmax_i e_i, return v_{i*}
• Soft: Weight all positions by softmax(e), return weighted sum

As softmax temperature τ → 0, soft attention approaches hard (argmax) selection.

4. Full Differentiability: Every operation in soft attention has well-defined gradients:

• Score computation: ∂e/∂q, ∂e/∂k are smooth
• Softmax: ∂α/∂e has closed form
• Weighted sum: ∂c/∂α and ∂c/∂v are trivial

This enables gradient-based optimization of all components.

The choice of soft attention over hard attention is fundamentally about trainability. Let's understand why differentiability is so crucial.

The Gradient Flow Requirement:

Neural network training requires computing ∂Loss/∂θ for all parameters θ. For attention, we need:

∂Loss/∂W_Q = ∂Loss/∂c · ∂c/∂α · ∂α/∂e · ∂e/∂Q · ∂Q/∂W_Q

Every term in this chain must be computable. Soft attention ensures this because:

∂c/∂α: Trivial, since c = Σᵢ αᵢvᵢ, we have ∂c/∂αᵢ = vᵢ

∂α/∂e: The softmax Jacobian:

∂αᵢ/∂eⱼ = αᵢ(δᵢⱼ - αⱼ)

where δᵢⱼ is the Kronecker delta.

∂e/∂Q: Depends on scoring function, but all common ones are smooth.

Why All Positions Receive Gradients:

A critical property of soft attention: every position receives gradient signal, even those with very low attention weight. Here's why this matters:

Exploration During Training:

• Early in training, the network doesn't know which positions matter
• Gradients to all positions allow the network to "explore" different attention patterns
• Positions that are useful will strengthen; useless ones will weaken

Credit Assignment:

• The loss signal propagates to all positions proportionally to their influence
• This enables the network to learn which positions should receive more attention
• Without this, the network couldn't learn to shift attention to new positions

Contrast with Hard Attention:

• Hard attention: Only the selected position receives gradient
• Other positions receive zero gradient—no learning signal
• The network cannot learn to attend elsewhere based on task loss
• Requires special techniques (REINFORCE, Gumbel-softmax) to train

Let's analyze how gradients flow through soft attention in detail. Understanding this is crucial for debugging attention-based models and designing effective architectures.

The Backward Pass:

Given upstream gradient ∂L/∂c from the loss, we need to compute:

∂L/∂v_i = α_i · ∂L/∂c        (gradient to values)
∂L/∂α_i = v_i · ∂L/∂c        (intermediate: gradient to weights)
∂L/∂e_i = Σⱼ ∂L/∂αⱼ · ∂αⱼ/∂e_i   (gradient to scores)
∂L/∂q, ∂L/∂k_i = from ∂L/∂e_i via score function

Gradient Magnitudes:

Let's trace gradient magnitudes through soft attention:

1. Values Gradient:

∂L/∂v_i = α_i · ∂L/∂c

• Scales linearly with attention weight α_i
• Position with α_i = 0.9 gets 9× gradient of position with α_i = 0.1
• Positions with near-zero weight get near-zero gradient

2. Weights Gradient:

∂L/∂α_i = v_i^T · ∂L/∂c

• Depends on alignment between value v_i and upstream gradient
• All positions receive this regardless of current weight

3. Scores Gradient (via softmax):

∂L/∂e_i = α_i · (∂L/∂α_i - Σⱼ αⱼ · ∂L/∂αⱼ)

• Complex interaction: depends on both this position's weight and the weighted average of all gradients
• Relative differences matter, not absolute scores

The Softmax Gradient Bottleneck:

When attention is very peaked (one α_i ≈ 1, others ≈ 0):

• ∂α_i/∂e_i ≈ α_i(1 - α_i) ≈ 0 for the dominant position
• Gradients become very small
• Training slows significantly

This is why the √d_k scaling matters: it prevents premature peaking.

Soft attention can be analyzed through the lens of information theory, providing insights into what attention learns and how it trades off focus versus coverage.

Attention Entropy:

The entropy of attention weights measures how "spread out" attention is:

H(α) = -Σᵢ αᵢ log αᵢ

Minimum entropy (H = 0):

• One α_i = 1, all others = 0
• "Hard" attention to single position
• Maximum focus, minimum coverage

Maximum entropy (H = log m):

• All α_i = 1/m (uniform)
• Attention spread equally over all positions
• Minimum focus, maximum coverage

Mutual Information Perspective:

Attention can be viewed as estimating mutual information between positions:

P(attend to j | decoding position i) ∝ exp(score(q_i, k_j))

The learned attention distribution approximates which source positions are informative for each target position. Training optimizes:

max I(source positions; decoder output | attention)

The network learns to attend to positions that reduce uncertainty about the correct output.

The Bottleneck Trade-off:

Soft attention creates an information bottleneck between source and target:

Source → Attention Weights → Context → Output

• Compression: Attention weights "summarize" relevance
• Sufficiency: Weighted values must contain enough info for the task

The network learns attention patterns that preserve task-relevant information while discarding noise.

Entropy Regularization:

We can explicitly control attention entropy:

Entropy Penalty (encourage focus):

Loss = Task_Loss - λ · H(α)

Encourages peaked attention by penalizing high entropy.

Entropy Bonus (encourage spread):

Loss = Task_Loss + λ · H(α)

Encourages diverse attention by rewarding high entropy.

When to use each:

• Entropy penalty: When task requires precise alignment (translation, copying)
• Entropy bonus: When task requires broad context (summarization, classification)
• Often: Just let the task loss determine natural entropy level

Implementing soft attention efficiently requires careful attention to numerical stability, parallelization, and memory usage.

Numerical Stability Considerations:

1. Softmax Overflow Prevention:

# Naive (overflows for large scores):
weights = exp(scores) / sum(exp(scores))

# Stable (shift by max):
scores_shifted = scores - max(scores)
weights = exp(scores_shifted) / sum(exp(scores_shifted))

The shift doesn't change the result (cancels in ratio) but prevents exp() overflow.

2. Log-Space Computation: When scores are very large/small, compute in log-space:

log_weights = scores - logsumexp(scores)
weights = exp(log_weights)

3. Handling Masked Positions:

# Use -inf for masked positions (becomes 0 after softmax)
scores = scores.masked_fill(mask == 0, float('-inf'))

Memory Efficiency Considerations:

Standard soft attention has memory complexity O(n × m) for the attention weight matrix. For large sequences, this becomes prohibitive:

For long-context models, this motivates:

• Flash Attention: Tile-based computation avoiding materialization
• Linear Attention: O(n) complexity variants
• Sparse Attention: Attend to subsets of positions

We'll explore these in later modules.

While the core soft attention mechanism is standardized, several variants modify its behavior for specific purposes.

1. Additive (Bahdanau) vs Multiplicative (Luong) Attention:

We covered scoring functions previously, but their soft attention behavior differs:

Additive: More expressive non-linear combination, but slower. Multiplicative: Efficient dot-product, less expressive, faster.

2. Local vs Global Attention:

Global: Attend to all positions (standard). Local: Attend only to a window around predicted alignment position.

Local attention bridges soft and hard—it's differentiable like soft attention but focused like hard attention.

3. Attention with Coverage:

For tasks like summarization, we want to ensure all source content is covered without excessive repetition. Coverage mechanisms track what's been attended to:

coverage_t = Σ_{t'<t} α_{t'}  (cumulative attention to each position)
score_t = f(query_t, key, coverage_t)  (coverage-aware scoring)

Coverage penalizes re-attending to already-covered positions, promoting diverse attention.

4. Multi-Head Attention (Preview):

Instead of one attention distribution, compute multiple in parallel:

head_i = Attention(Q · W_Q^i, K · W_K^i, V · W_V^i)
output = Concat(head_1, ..., head_h) · W_O

Each head can learn different attention patterns. We'll study this in detail in Module 3.

5. Attention Dropout:

Apply dropout to attention weights after softmax:

weights = softmax(scores)
weights = dropout(weights, p=0.1)
context = weights @ values

This regularizes attention, preventing over-reliance on specific positions.

While soft attention is the dominant paradigm, it has inherent limitations that motivate research into alternatives.

1. The "Blurriness" Problem:

Soft attention produces weighted combinations—it cannot truly "select" one item:

# Hard selection:
result = items[index]  # Exactly one item

# Soft selection:
result = Σ weights[i] * items[i]  # Blur of all items

For tasks requiring discrete selection (e.g., "which item should I copy?"), soft attention provides an approximation, not exact selection. The output is always a blend.

2. Convex Hull Constraint:

The soft attention output lies within the convex hull of values:

context = α₁v₁ + α₂v₂ + ... + αₘvₘ  where Σαᵢ = 1, αᵢ ≥ 0

The output cannot "extrapolate" beyond the extremes of the values. If all values have norm ≤ 1, output has norm ≤ 1. This limits expressiveness for tasks needing outputs outside the value range.

3. Computational Complexity:

Soft attention requires computing all pairwise interactions:

Scores: O(n × m)   space and time

For self-attention with n = m = 10,000:

• 100 million score computations
• 100 million floats in memory (400 MB for float32)

This quadratic scaling limits application to long sequences.

4. All-or-Nothing Gradient Flow:

While all positions receive gradients, the magnitudes differ dramatically:

• Position with α = 0.9 gets 9× gradient of position with α = 0.1
• Position with α = 0.01 gets 1/90th gradient of the dominant position

Learning to shift attention from a dominant position to a minor one is slow—the minor position's gradient is too small to compete.

When Soft Attention Is Sufficient:

Despite limitations, soft attention works exceptionally well when:

• Tasks benefit from contextual blending (most NLP tasks)
• Discrete selection isn't strictly required
• Sequence lengths are manageable (< 10K typical, < 100K with efficiency techniques)
• The loss provides sufficient gradient signal

When to Consider Alternatives:

• Tasks requiring discrete selection (pointer networks, copying)
• Very long sequences (genome, long documents)
• Tasks with sparse relevance patterns
• Systems where interpretability requires truly sparse attention

We've thoroughly examined soft attention—the differentiable, weighted-sum approach that powers virtually all modern attention mechanisms. Let's consolidate:

What's Next:

The next page explores the alternative: hard attention. Instead of weighted combinations, hard attention samples or argmaxes a single position. This brings different tradeoffs—discrete selection but training challenges. Understanding both soft and hard attention reveals the full spectrum of attention design choices.