Gradient Descent with L2 Regularization Penalty

In deep learning and machine learning optimization, L2 regularization (also known as weight decay) is a fundamental technique used to prevent overfitting by penalizing large parameter values. This regularization approach adds a term proportional to the sum of squared weights to the loss function, encouraging the model to keep weights small and thus improve generalization.

When training neural networks, the standard gradient descent update rule is:

$$\theta_{\text{new}} = \theta_{\text{old}} - \eta \cdot abla L$$

where θ represents the model parameters, η is the learning rate, and ∇L is the gradient of the loss function. With L2 regularization, the update rule becomes:

$$\theta_{\text{new}} = \theta_{\text{old}} - \eta \cdot abla L - \eta \cdot \lambda \cdot \theta_{\text{old}}$$

This can be rewritten as:

$$\theta_{\text{new}} = \theta_{\text{old}} \cdot (1 - \eta \cdot \lambda) - \eta \cdot abla L$$

where λ is the weight decay coefficient that controls the strength of regularization.

Selective Application: In practice, weight decay is typically applied only to weight parameters and not to bias terms. This is because biases don't contribute to overfitting in the same way weights do, and penalizing biases can prevent the model from learning necessary offsets. Your implementation should support selectively applying weight decay to specific parameter groups based on a boolean flag array.

Your Task: Implement a function that performs gradient descent parameter updates with optional L2 regularization. The function should:

• Take arrays of parameters, their corresponding gradients, a learning rate, and a weight decay coefficient
• Use a boolean array to determine which parameter groups should have weight decay applied
• Return the updated parameters after applying the appropriate update rule to each group