Consider a remarkable fact: near any point where a function is smooth, you can approximate it arbitrarily well using just polynomials. The exponential function? A polynomial. Sine and cosine? Polynomials. The complex loss landscape of a neural network? Locally, a polynomial.

This is the power of Taylor series—one of the most useful tools in all of applied mathematics. By expanding a function as a polynomial, we can:

• Approximate complex functions with simple, analytically tractable forms
• Analyze behavior near critical points (minima, maxima, saddle points)
• Design optimization algorithms that exploit local curvature
• Understand why gradient descent converges (or fails to converge)

In machine learning, Taylor expansions underlie:

• Second-order optimization methods (Newton's method, natural gradient)
• Convergence proofs for gradient descent
• Analysis of loss landscape structure
• Approximation theory for neural networks

This page develops Taylor series for multivariable functions, building toward the crucial second-order approximation that captures curvature through the Hessian matrix.

Before tackling multiple dimensions, let's solidify our understanding of the single-variable case.

Taylor Series Expansion:

For a function f: ℝ → ℝ that is infinitely differentiable at point a:

$$f(x) = \sum_{n=0}^{\infty} \frac{f^{(n)}(a)}{n!} (x - a)^n$$

Expanded:

$$f(x) = f(a) + f'(a)(x-a) + \frac{f''(a)}{2!}(x-a)^2 + \frac{f'''(a)}{3!}(x-a)^3 + \cdots$$

Key Terms:

• Zeroth order (n=0): f(a) — the function value
• First order (n=1): f'(a)(x-a) — linear approximation (tangent line)
• Second order (n=2): f''(a)(x-a)²/2 — curvature correction
• Higher orders: Increasingly fine corrections

Truncated Expansions (Taylor Polynomials):

In practice, we truncate after finite terms:

• First-order (linear): f(x) ≈ f(a) + f'(a)(x-a)
• Second-order (quadratic): f(x) ≈ f(a) + f'(a)(x-a) + f''(a)(x-a)²/2

The truncation error is O((x-a)^{n+1}) for an n-th order expansion.

Classic Examples:

The Taylor series extends naturally to functions of multiple variables, though the notation becomes more complex.

First-Order (Linear) Approximation:

For f: ℝⁿ → ℝ near point a:

$$f(\mathbf{x}) \approx f(\mathbf{a}) + \nabla f(\mathbf{a})^\top (\mathbf{x} - \mathbf{a})$$

This is the tangent hyperplane to the surface f(x) at a.

Second-Order (Quadratic) Approximation:

$$f(\mathbf{x}) \approx f(\mathbf{a}) + \nabla f(\mathbf{a})^\top (\mathbf{x} - \mathbf{a}) + \frac{1}{2} (\mathbf{x} - \mathbf{a})^\top \mathbf{H}_f(\mathbf{a}) (\mathbf{x} - \mathbf{a})$$

where H_f(a) is the Hessian matrix of second partial derivatives:

$$\mathbf{H} = \begin{bmatrix} \frac{\partial^2 f}{\partial x_1^2} & \frac{\partial^2 f}{\partial x_1 \partial x_2} & \cdots \ \frac{\partial^2 f}{\partial x_2 \partial x_1} & \frac{\partial^2 f}{\partial x_2^2} & \cdots \ \vdots & \vdots & \ddots \end{bmatrix}$$

General Form (Multi-Index Notation):

For the full infinite series, we use multi-indices α = (α₁, ..., αₙ) with |α| = α₁+...+αₙ:

$$f(\mathbf{x}) = \sum_{|\alpha| = 0}^{\infty} \frac{D^\alpha f(\mathbf{a})}{\alpha!} (\mathbf{x} - \mathbf{a})^\alpha$$

where α! = α₁!·α₂!·...·αₙ! and (x - a)^α = (x₁-a₁)^{α₁}...(xₙ-aₙ)^{αₙ}.

Don't worry about this notation—we'll primarily use first and second-order approximations.

Let's understand why the multivariate Taylor series takes its form by building it up from directional derivatives.

Reduction to One Dimension:

Consider moving from a toward x along the straight line:

$$\mathbf{r}(t) = \mathbf{a} + t(\mathbf{x} - \mathbf{a}), \quad t \in [0, 1]$$

Define g(t) = f(r(t)). This is a single-variable function!

Apply the 1D Taylor series:

$$g(1) = g(0) + g'(0) + \frac{1}{2}g''(0) + \cdots$$

But g(0) = f(a) and g(1) = f(x). So:

$$f(\mathbf{x}) = f(\mathbf{a}) + g'(0) + \frac{1}{2}g''(0) + \cdots$$

Computing g'(0):

By the chain rule:

$$g'(t) = \nabla f(\mathbf{r}(t))^\top \mathbf{r}'(t) = \nabla f(\mathbf{r}(t))^\top (\mathbf{x} - \mathbf{a})$$

At t=0:

$$g'(0) = \nabla f(\mathbf{a})^\top (\mathbf{x} - \mathbf{a})$$

Computing g''(0):

Differentiating again:

$$g''(t) = (\mathbf{x} - \mathbf{a})^\top \mathbf{H}_f(\mathbf{r}(t)) (\mathbf{x} - \mathbf{a})$$

At t=0:

$$g''(0) = (\mathbf{x} - \mathbf{a})^\top \mathbf{H}_f(\mathbf{a}) (\mathbf{x} - \mathbf{a})$$

Assembling the Pieces:

Putting it together:

$$f(\mathbf{x}) = f(\mathbf{a}) + \nabla f(\mathbf{a})^\top (\mathbf{x} - \mathbf{a}) + \frac{1}{2} (\mathbf{x} - \mathbf{a})^\top \mathbf{H}_f(\mathbf{a}) (\mathbf{x} - \mathbf{a}) + O(|\mathbf{x}-\mathbf{a}|^3)$$

This derivation shows the multivariate Taylor series is just the 1D version applied along any straight-line path!

The second-order term involves the Hessian matrix—the matrix of all second partial derivatives. The Hessian captures curvature information about the function.

Definition (Hessian Matrix):

$$\mathbf{H}_f(\mathbf{x}) = \begin{bmatrix} \frac{\partial^2 f}{\partial x_1^2} & \frac{\partial^2 f}{\partial x_1 \partial x_2} & \cdots & \frac{\partial^2 f}{\partial x_1 \partial x_n} \ \frac{\partial^2 f}{\partial x_2 \partial x_1} & \frac{\partial^2 f}{\partial x_2^2} & \cdots & \frac{\partial^2 f}{\partial x_2 \partial x_n} \ \vdots & \vdots & \ddots & \vdots \ \frac{\partial^2 f}{\partial x_n \partial x_1} & \frac{\partial^2 f}{\partial x_n \partial x_2} & \cdots & \frac{\partial^2 f}{\partial x_n^2} \end{bmatrix}$$

Key Properties:

1. Symmetry: If f has continuous second partial derivatives, ∂²f/∂xᵢ∂xⱼ = ∂²f/∂xⱼ∂xᵢ (Schwarz's theorem). Thus H = Hᵀ.

2. Dimension: n × n for f: ℝⁿ → ℝ

3. Eigenvalues: All real (since symmetric). They determine curvature magnitude along principal axes.

4. Relationship to Jacobian: The Hessian is the Jacobian of the gradient: Hf = J{∇f}

Curvature Interpretation:

The eigenvalues λ₁, ..., λₙ of H at a critical point (where ∇f = 0) determine the point's nature:

• All λᵢ > 0: Local minimum (bowl curving up)
• All λᵢ < 0: Local maximum (bowl curving down)
• Mixed signs: Saddle point (curves up in some directions, down in others)
• Some λᵢ = 0: Degenerate critical point (flat in some direction)

The second-order Taylor expansion provides a quadratic model of the loss function. This model is the foundation for second-order optimization methods.

The Quadratic Model:

Near current parameters θ, approximate the loss:

$$L(\mathbf{\theta} + \Delta\mathbf{\theta}) \approx L(\mathbf{\theta}) + \nabla L^\top \Delta\mathbf{\theta} + \frac{1}{2} \Delta\mathbf{\theta}^\top \mathbf{H} \Delta\mathbf{\theta}$$

Newton's Method:

Set derivative of quadratic model to zero:

$$\nabla_{\Delta\theta}\left[ L + \nabla L^\top \Delta\theta + \frac{1}{2} \Delta\theta^\top \mathbf{H} \Delta\theta \right] = \nabla L + \mathbf{H} \Delta\theta = \mathbf{0}$$

Solving: Δθ = -H⁻¹∇L

Newton Update: θ ← θ - H⁻¹∇L

Comparison to Gradient Descent:

Newton converges much faster (quadratically!) near the minimum, but each step costs O(n³) to invert the Hessian—often prohibitive for large n.

Practical Variants:

• Quasi-Newton (BFGS, L-BFGS): Approximate H⁻¹ incrementally. O(n²) or O(n) per step.
• Gauss-Newton: Approximate H using gradient outer products. For least squares.
• Natural Gradient: Use Fisher information matrix instead of Hessian.
• Second-order with diagonal: Approximate H as diagonal (cheap inverse).
• Shampoo, K-FAC: Structured approximations for neural networks.

Taylor series provide the mathematical framework to analyze why and when gradient descent converges.

Setting Up the Analysis:

Let f be a smooth function we're minimizing. Gradient descent updates:

$$\mathbf{x}_{k+1} = \mathbf{x}_k - \alpha \nabla f(\mathbf{x}_k)$$

Taylor expansion of f at the new point:

$$f(\mathbf{x}_{k+1}) = f(\mathbf{x}_k) + \nabla f(\mathbf{x}k)^\top (\mathbf{x}{k+1} - \mathbf{x}k) + O(|\mathbf{x}{k+1} - \mathbf{x}_k|^2)$$

Substituting the GD update:

$$f(\mathbf{x}_{k+1}) = f(\mathbf{x}_k) - \alpha |\nabla f(\mathbf{x}_k)|^2 + O(\alpha^2)$$

Key Insight:

For small α, the first-order term dominates: each step decreases f by approximately α‖∇f‖². This proves gradient descent makes progress (when gradient is nonzero and α is small enough).

Optimal Learning Rate:

Including the second-order term:

$$f(\mathbf{x}_{k+1}) \approx f(\mathbf{x}_k) - \alpha |\nabla f|^2 + \frac{\alpha^2}{2} \nabla f^\top \mathbf{H} \nabla f$$

Minimizing over α, the optimal (for quadratic approximation) is:

$$\alpha^* = \frac{|\nabla f|^2}{\nabla f^\top \mathbf{H} \nabla f}$$

This is the exact line search result for quadratic functions.

Condition Number and Convergence:

Define the condition number κ = λ_max / λ_min (ratio of largest to smallest eigenvalue of H).

For a quadratic function, GD has convergence rate:

$$f(\mathbf{x}_k) - f^* \leq \left(\frac{\kappa - 1}{\kappa + 1}\right)^{2k} (f(\mathbf{x}_0) - f^*)$$

Large κ (ill-conditioned) means slow convergence. This is why poorly conditioned problems are hard for gradient descent.

Taylor series appear throughout machine learning beyond just optimization analysis. Here are key applications:

1. Activation Function Approximations:

Many activations are defined or approximated via Taylor series:

• GELU: Gaussian Error Linear Unit uses ≈ x·Φ(x) ≈ 0.5x(1 + tanh(√(2/π)(x + 0.044715x³)))
• Swish: x·σ(x) ≈ x/2 + x²/4 - x⁴/48 + ... near 0
• Softplus: log(1 + eˣ) ≈ x + log(2) for large x; ≈ eˣ for small x

2. Loss Landscape Visualization:

To visualize high-dimensional loss landscapes, project onto 2 random directions u, v:

$$\text{plot } f(\mathbf{\theta} + \alpha \mathbf{u} + \beta \mathbf{v})$$

Taylor expansion predicts this is approximately quadratic near θ:

$$\approx f(\mathbf{\theta}) + \alpha \nabla f \cdot \mathbf{u} + \beta \nabla f \cdot \mathbf{v} + \frac{1}{2}(\alpha^2 \mathbf{u}^\top\mathbf{H}\mathbf{u} + \cdots)$$

3. Fisher Information and Natural Gradient:

The Fisher information matrix F is the expected Hessian of the log-likelihood:

$$\mathbf{F} = \mathbb{E}[\nabla \log p(x|\theta) \nabla \log p(x|\theta)^\top]$$

Natural gradient uses F instead of H: θ ← θ - αF⁻¹∇L. This accounts for the geometry of probability distributions.

4. Laplace Approximation:

In Bayesian deep learning, the posterior p(θ|data) is intractable. The Laplace approximation fits a Gaussian using the second-order Taylor expansion at the mode:

$$p(\theta|\text{data}) \approx \mathcal{N}(\theta^*, \mathbf{H}^{-1})$$

where θ* is the MAP estimate and H is the Hessian of the negative log-posterior.

Taylor series provide the mathematical lens through which we understand local function behavior—essential for optimization theory and practice.

Key Concepts:

What's Next:

We've now seen how the Hessian appears in the second-order Taylor expansion. The final page dives deeper into second-order approximations and the Hessian matrix specifically: properties, computation, eigenvalue analysis for critical point classification, and practical approaches for leveraging curvature in large-scale optimization.