The power of LSTM lies not in any single component, but in the coordinated interplay of three gates—forget, input, and output—that together create a sophisticated memory controller. Each gate learns to recognize specific patterns in the input and hidden state, and responses by either opening (allowing information to flow) or closing (blocking information).

In the previous page, we established what each gate does. Now we explore how gates work at a deeper level—their mathematical properties, learning dynamics, and the emergent behaviors that arise from their interaction. Understanding gates thoroughly is essential for diagnosing issues, designing LSTM variants, and knowing when to apply (or not apply) LSTM architectures.

Gates in LSTM use the sigmoid function to produce values between 0 and 1. This choice is not arbitrary—it has profound mathematical consequences that enable LSTM's memory capabilities.

The Sigmoid Function:

$$\sigma(z) = \frac{1}{1 + e^{-z}}$$

Key Properties:

1. Bounded Output: $\sigma(z) \in (0, 1)$ for all $z \in \mathbb{R}$
2. Monotonic: Strictly increasing, so larger pre-activations mean more "open" gates
3. Symmetric: $\sigma(-z) = 1 - \sigma(z)$
4. Smooth Everywhere: Infinitely differentiable, enabling gradient-based learning
5. Efficient Gradient: $\sigma'(z) = \sigma(z)(1 - \sigma(z))$—can be computed from $\sigma(z)$ alone

The Gating Operation:

When a gate $g = \sigma(z)$ modulates a signal $x$, the output is $g \cdot x$. This multiplication has crucial properties:

1. Linear in $x$: For fixed $g$, the operation is linear in $x$, preserving gradient magnitudes
2. Scaling Behavior: $g$ scales the signal—when $g \approx 1$, $x$ passes; when $g \approx 0$, $x$ is blocked
3. Gradient Distribution: $\frac{\partial (gx)}{\partial x} = g$, so the gate directly controls gradient flow

Why Not Hard Gates?

One might ask: why use soft sigmoid gates instead of hard 0/1 decisions? The answer is learnability:

• Hard gates ($g \in {0, 1}$) have zero gradient almost everywhere—no learning signal
• Soft gates ($g \in (0, 1)$) always have non-zero gradient through $\sigma'(z)$
• The network can learn to make gates nearly hard (saturated) when beneficial, while remaining trainable

The forget gate $f_t$ is perhaps the most critical gate in the LSTM. It directly controls the gradient highway and determines whether information persists or is erased.

Mathematical Definition:

$$f_t = \sigma(W_f \cdot [h_{t-1}, x_t] + b_f)$$

What the Forget Gate Learns to Detect:

Through training, the forget gate learns to recognize patterns that signal "erase memory":

• Explicit boundaries: Period at end of sentence, paragraph breaks, topic markers
• Structural shifts: Question → answer, premise → conclusion
• Semantic completions: End of a named entity, completion of a phrase
• Contradiction signals: "but", "however", "contrary to"

Conversely, it stays open (high $f_t$) when context must persist:

• Mid-phrase: Remembering subject for verb agreement
• Long-range references: Tracking pronouns back to antecedents
• Accumulating information: Summing quantities, building descriptions

Forget Gate Bias: The Critical Hyperparameter

The initialization of $b_f$ is one of the most consequential choices in LSTM training:

The Jozefowicz et al. Recommendation:

In "An Empirical Exploration of Recurrent Network Architectures" (2015), the authors found that initializing $b_f = 1.0$ consistently improved performance across tasks. Some practitioners use even higher values (1.5 or 2.0) for tasks requiring very long dependencies.

Intuition: At initialization, before the network has learned anything, it's better to err on the side of remembering (gradients flow) than forgetting (gradient death).

The input gate $i_t$ and candidate values $\tilde{c}_t$ work together to control what new information enters the cell state. Their separation into two components is a key design decision.

Mathematical Definitions:

$$i_t = \sigma(W_i \cdot [h_{t-1}, x_t] + b_i)$$ $$\tilde{c}t = \tanh(W_c \cdot [h{t-1}, x_t] + b_c)$$

Why Two Separate Components?

Consider the alternative: a single component $\Delta c_t = f(W \cdot [h, x] + b)$ added to cell state.

The problem is that this conflates two decisions:

1. What information to add (content)
2. How much to add (magnitude)

By separating these:

• $\tilde{c}_t$ (via $\tanh$) represents the content of potential updates, bounded to [-1, 1]
• $i_t$ (via sigmoid) represents the amount to actually add, bounded to [0, 1]
• Their product $i_t \odot \tilde{c}_t$ gives controlled, bounded updates

The Coordinate Behavior of Input and Forget Gates:

In practice, input and forget gates often develop complementary behaviors:

This coordination emerges naturally through training—the loss function shapes both gates to work together for the task at hand.

The output gate $o_t$ serves a fundamentally different purpose from the input and forget gates. While those control storage, the output gate controls visibility—what information from the cell state is exposed to the rest of the network.

Mathematical Definition:

$$o_t = \sigma(W_o \cdot [h_{t-1}, x_t] + b_o)$$ $$h_t = o_t \odot \tanh(c_t)$$

Why Separate Storage from Output?

This separation is crucial for long-term dependencies:

1. Information can be stored but hidden: Facts relevant 100 steps later don't need to pollute current predictions
2. Selective relevance: The same stored information might be relevant in some contexts but not others
3. Reduced interference: Outputs feeding into predictions don't corrupt long-term storage

Example: Subject-Verb Agreement Across a Long Clause

Consider: "The trophy, which the athletes who won the championship game received from the governor, [was/were] very heavy."

The subject is "trophy" (singular), but many words intervene before the verb. An LSTM might:

1. At "trophy": Store "singular subject" in cell state, with high input gate
2. During the relative clause: Keep forget gate high (retain) but output gate low (don't let it interfere with clause processing)
3. At "was/were": Finally open output gate to expose the singular marker for prediction

Without the output gate, the singular marker would constantly influence hidden states, potentially confusing clause-internal processing.

When all three gates operate together, complex memory behaviors emerge that go beyond what any single gate could achieve. Understanding these emergent patterns helps diagnose LSTM behavior and design better architectures.

The Five Fundamental Memory Operations:

Temporal Patterns in Gate Activations:

Well-trained LSTMs exhibit characteristic temporal patterns:

1. Periodic Gating: For structured data (music, formatted text), gates often show periodic patterns matching the structure

2. Burst Writing: At semantically rich points (entity mentions, key events), input gates spike while forget gates may simultaneously dip

3. Sustained Carry: During parenthetical expressions or subclauses, forget gates stay high and input gates stay low—pure memory preservation

4. Delayed Read: Information stored early may only be read (output gate high) much later when it becomes relevant

5. Cascading Resets: Major transitions can trigger reset cascades where forget gates go low across multiple dimensions simultaneously

How do gates learn their specific roles? The answer lies in the gradient signals they receive during training and how the loss function shapes their behavior.

Gradient Flow to Gates:

Considering the cell state update $c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t$:

$$\frac{\partial L}{\partial f_t} = \frac{\partial L}{\partial c_t} \odot c_{t-1}$$

$$\frac{\partial L}{\partial i_t} = \frac{\partial L}{\partial c_t} \odot \tilde{c}_t$$

Interpretation:

• The forget gate receives gradient proportional to how useful the previous cell state is. If $c_{t-1}$ contains information needed for the current loss, $f_t$ is pushed higher.

• The input gate receives gradient proportional to how useful the candidate write would be. If $\tilde{c}_t$ contains missing information, $i_t$ is pushed higher.

The Credit Assignment Problem:

A key challenge in gate learning is temporal credit assignment. If information written at time $t_1$ enables correct prediction at time $t_2 >> t_1$:

1. The loss at $t_2$ generates gradient $\frac{\partial L}{\partial c_{t_2}}$
2. This propagates back through time: $\frac{\partial L}{\partial c_{t_1}} = \frac{\partial L}{\partial c_{t_2}} \cdot \prod_{k=t_1+1}^{t_2} f_k$
3. The gradient to $i_{t_1}$ is then $\frac{\partial L}{\partial c_{t_1}} \odot \tilde{c}_{t_1}$

The crucial observation: If forget gates maintain high values ($f_k \approx 1$), this gradient survives the journey back through time, and $i_{t_1}$ learns "writing here was useful." If forget gates are too low, the signal is lost, and the network can't learn to write information that will be needed later.

This is why forget gate initialization is so critical—it directly determines whether the network can learn long-range dependencies at all.

While gates learn their behavior from the task, we can guide them through regularization techniques—either encouraging certain behaviors or preventing pathological ones.

Activation Regularization (AR):

Penalizes hidden state magnitude, indirectly affecting gates: $$L_{AR} = \alpha \sum_t ||h_t||^2$$

This encourages sparser, more selective output gate activations.

Temporal Activation Regularization (TAR):

Penalizes changes in hidden state, encouraging smoothness: $$L_{TAR} = \beta \sum_t ||h_t - h_{t-1}||^2$$

This discourages rapid gate switching, promoting more coherent memory behavior.

Gate Activity Regularization:

Directly regularize gate activations:

$$L_{gate} = \gamma \sum_t \sum_{g \in {f,i,o}} ||g_t||^2$$

This pushes gates toward 0, encouraging sparsity in memory operations.

Entropy Regularization:

Gate activations can be viewed as probabilities. High entropy means uncertain, soft gates; low entropy means decisive, hard gates:

$$H(g) = -g \log(g) - (1-g) \log(1-g)$$

We can:

• Maximize entropy to keep gates soft (easier learning, less sharp behavior)
• Minimize entropy to encourage saturation (sharper decisions, better gradient flow when saturated high)

We've deeply explored the three gates that make LSTM a powerful memory-enabled architecture.

Key Insights:

Looking Ahead:

We've now understood the individual gates and their dynamics. The next page focuses on the Cell State Highway—how the linear flow of cell state through time creates the constant error carousel, enabling gradient flow and long-term memory that was impossible with vanilla RNNs.