Information Divergence Between Gaussian Distributions

In probability theory and information theory, the Kullback-Leibler (KL) divergence is a fundamental measure that quantifies how one probability distribution differs from a reference probability distribution. It originated from information theory and answers the question: "How much information is lost when we use distribution Q to approximate distribution P?"

For univariate Gaussian (normal) distributions, the KL divergence has a elegant closed-form solution. Given two Gaussian distributions:

• P ~ N(μₚ, σₚ²) — the "true" distribution
• Q ~ N(μ_q, σ_q²) — the "approximating" or reference distribution

The KL divergence from P to Q is defined as:

$$D_{KL}(P | Q) = \log\left(\frac{\sigma_Q}{\sigma_P}\right) + \frac{\sigma_P^2 + (\mu_P - \mu_Q)^2}{2\sigma_Q^2} - \frac{1}{2}$$

Key Properties of KL Divergence:

1. Non-negative: D_KL(P || Q) ≥ 0 for all distributions P and Q
2. Zero when identical: D_KL(P || Q) = 0 if and only if P = Q almost everywhere
3. Asymmetric: D_KL(P || Q) ≠ D_KL(Q || P) in general — the order of arguments matters!
4. Not a true distance: KL divergence violates the triangle inequality and symmetry

Intuitive Understanding:

Think of KL divergence as measuring the "surprise" or "extra bits" needed when you expect data from distribution Q but it actually comes from distribution P. If Q is a poor approximation of P, you'll be "surprised" more often, resulting in a higher divergence value.

Your Task:

Implement a function that computes the KL divergence between two univariate Gaussian distributions. The function should accept the mean and standard deviation of both distributions and return the KL divergence value, rounded to 4 decimal places.