Recurrent Network with Temporal Gradient Learning

In the realm of sequence modeling, Recurrent Neural Networks (RNNs) represent a foundational architecture designed to process sequential data by maintaining an internal memory of previous inputs. Unlike feedforward networks that treat each input independently, RNNs create a temporal dependency structure that allows information to persist across time steps, making them exceptionally well-suited for tasks involving temporal or sequential patterns.

Core Concept: Temporal State Evolution

At the heart of an RNN lies the hidden state—a dynamically evolving representation that captures the contextual history of the sequence processed thus far. At each time step t, the network computes a new hidden state by combining the current input with the previous hidden state through learned weight transformations:

$$h_t = \tanh(W_{xh} \cdot x_t + W_{hh} \cdot h_{t-1} + b_h)$$

Where:

• x_t is the input vector at time step t
• h_{t-1} is the hidden state from the previous time step
• W_{xh} is the input-to-hidden weight matrix
• W_{hh} is the hidden-to-hidden (recurrent) weight matrix
• b_h is the hidden layer bias
• tanh is the hyperbolic tangent activation function

The output at each time step is then computed from the current hidden state:

$$y_t = W_{hy} \cdot h_t + b_y$$

Temporal Gradient Propagation (Backpropagation Through Time)

Training an RNN requires a specialized gradient computation technique called Backpropagation Through Time (BPTT). Since the network's computations unfold across multiple time steps, gradients must flow backward through the entire sequence to capture temporal dependencies.

The key insight of BPTT is that at each time step, the hidden state depends on all previous hidden states. Therefore, when computing gradients, we must:

1. Unroll the network across all time steps
2. Compute output error at each step using the loss function
3. Propagate gradients backward through both the output layer and the recurrent connections
4. Accumulate gradients across all time steps for each weight matrix

For this problem, use half Mean Squared Error (MSE) as the loss function:

$$L = \frac{1}{2} \sum_{t=1}^{T} (y_t - \hat{y}_t)^2$$

The total loss is the sum of losses at each individual time step.

Your Task

Implement a complete recurrent neural network class with the following specifications:

Class: SequentialRecurrentNetwork

Constructor: __init__(self, input_size, hidden_size, output_size)

• Initialize weight matrices using Xavier/Glorot initialization: sample from a normal distribution with mean 0 and standard deviation sqrt(2 / (fan_in + fan_out))
• Initialize all bias vectors to zeros
• Initialize the hidden state to a zero vector

Forward Pass: forward(self, x)

• Process the input sequence step by step
• At each time step, update the hidden state and compute the output
• Return the outputs for all time steps

Backward Pass: backward(self, x, y, learning_rate)

• Compute gradients using temporal gradient propagation (BPTT)
• Update all weight matrices and biases using gradient descent
• Handle gradient flow through the recurrent connections

Important Implementation Notes:

• Use np.random.seed(42) at the start of your weight initialization for reproducibility
• Use the tanh activation function for the hidden layer
• No activation function is applied to the output layer (linear output)
• The initial hidden state before processing the sequence should be zeros