Gradient Vector Analysis for Optimization

In optimization and machine learning, the gradient vector is a fundamental mathematical object that guides iterative algorithms toward optimal solutions. Understanding the properties of a gradient—its magnitude and direction—is essential for implementing and tuning optimization algorithms like gradient descent, Adam, and RMSprop.

Understanding the Gradient

The gradient of a scalar function ( f(\mathbf{x}) ) with respect to its input vector ( \mathbf{x} = [x_1, x_2, \ldots, x_n] ) is itself a vector:

$$ abla f = \left[ \frac{\partial f}{\partial x_1}, \frac{\partial f}{\partial x_2}, \ldots, \frac{\partial f}{\partial x_n} \right]$$

This gradient vector points in the direction of the steepest increase of the function at any given point. For optimization problems where we want to minimize a loss function, we move in the opposite direction—the direction of steepest descent.

Key Quantities to Compute

Given a gradient vector ( \mathbf{g} = [g_1, g_2, \ldots, g_n] ), your task is to compute:

1. Magnitude (L2 Norm)

The magnitude tells us how steep the function is at the current point:

$$|\mathbf{g}| = \sqrt{g_1^2 + g_2^2 + \cdots + g_n^2} = \sqrt{\sum_{i=1}^{n} g_i^2}$$

A larger magnitude means a steeper slope and typically larger parameter updates during optimization.

2. Ascent Direction (Unit Vector)

The normalized gradient vector pointing toward steepest increase:

$$\hat{\mathbf{g}} = \frac{\mathbf{g}}{|\mathbf{g}|} = \left[ \frac{g_1}{|\mathbf{g}|}, \frac{g_2}{|\mathbf{g}|}, \ldots, \frac{g_n}{|\mathbf{g}|} \right]$$

3. Descent Direction

The negation of the ascent direction, pointing toward steepest decrease:

$$-\hat{\mathbf{g}} = -\frac{\mathbf{g}}{|\mathbf{g}|}$$

This is the direction used in gradient descent to minimize loss functions.

Edge Case: Zero Gradient

When the gradient vector is a zero vector (all components are zero), we are at a critical point—either a local minimum, local maximum, or saddle point. In this case:

• The magnitude is 0
• Both direction vectors should be returned as zero vectors (since there is no defined direction)

Your Task

Write a function that accepts a gradient vector and returns a dictionary containing:

• 'magnitude': The L2 norm of the gradient (a float)
• 'direction': The unit vector of steepest ascent (a list of floats)
• 'descent_direction': The unit vector of steepest descent (a list of floats)

gradient = [3.0, 4.0]

{'magnitude': 5.0, 'direction': [0.6, 0.8], 'descent_direction': [-0.6, -0.8]}

gradient = [1.0, 2.0, 2.0]

{'magnitude': 3.0, 'direction': [0.3333333333, 0.6666666667, 0.6666666667], 'descent_direction': [-0.3333333333, -0.6666666667, -0.6666666667]}

gradient = [0.0, 0.0]

{'magnitude': 0.0, 'direction': [0.0, 0.0], 'descent_direction': [0.0, 0.0]}