101
0/318
Loading content...
In information theory and statistical learning, comparing probability distributions is a fundamental task. While several divergence measures exist, many suffer from asymmetry—meaning the order of inputs affects the result. The Symmetric Distribution Divergence addresses this limitation by computing a balanced measure that treats both distributions equally.
Given two probability distributions P and Q over the same discrete sample space, this metric first computes an average mixture distribution M, then evaluates how each original distribution differs from this midpoint:
$$M = \frac{1}{2}(P + Q)$$
The divergence is then calculated as the average of the Kullback-Leibler divergences from each original distribution to the mixture:
$$D_{sym}(P | Q) = \frac{1}{2} D_{KL}(P | M) + \frac{1}{2} D_{KL}(Q | M)$$
where the KL divergence is defined as:
$$D_{KL}(X | Y) = \sum_{i} X_i \cdot \ln\left(\frac{X_i}{Y_i}\right)$$
Key Properties:
Your Task: Write a Python function that computes the symmetric distribution divergence between two probability distributions. The function should return the result rounded to 6 decimal places.
P = [0.5, 0.5]
Q = [0.4, 0.6]0.005059Two similar distributions are compared:
Step 1: Compute the mixture distribution M: • M = 0.5 × (P + Q) = 0.5 × ([0.5, 0.5] + [0.4, 0.6]) = [0.45, 0.55]
Step 2: Compute KL(P || M): • KL(P || M) = 0.5 × ln(0.5/0.45) + 0.5 × ln(0.5/0.55) • = 0.5 × ln(1.111) + 0.5 × ln(0.909) • = 0.5 × 0.1054 + 0.5 × (-0.0953) • ≈ 0.005025
Step 3: Compute KL(Q || M): • KL(Q || M) = 0.4 × ln(0.4/0.45) + 0.6 × ln(0.6/0.55) • = 0.4 × ln(0.889) + 0.6 × ln(1.091) • = 0.4 × (-0.1178) + 0.6 × 0.0870 • ≈ 0.005094
Step 4: Final divergence: • D = 0.5 × 0.005025 + 0.5 × 0.005094 ≈ 0.005059
The small value indicates the distributions are quite similar.
P = [0.25, 0.25, 0.25, 0.25]
Q = [0.25, 0.25, 0.25, 0.25]0.0When comparing identical distributions:
Mixture Distribution: • M = 0.5 × ([0.25, 0.25, 0.25, 0.25] + [0.25, 0.25, 0.25, 0.25]) = [0.25, 0.25, 0.25, 0.25]
Since P = Q = M, both KL divergences become: • KL(P || M) = Σ 0.25 × ln(0.25/0.25) = Σ 0.25 × ln(1) = Σ 0.25 × 0 = 0 • KL(Q || M) = 0
Final divergence: D = 0.5 × 0 + 0.5 × 0 = 0.0
This confirms the fundamental property: identical distributions have zero divergence.
P = [0.9, 0.05, 0.05]
Q = [0.05, 0.05, 0.9]0.462607Comparing highly dissimilar distributions:
Mixture Distribution: • M = 0.5 × ([0.9, 0.05, 0.05] + [0.05, 0.05, 0.9]) = [0.475, 0.05, 0.475]
KL(P || M): • = 0.9 × ln(0.9/0.475) + 0.05 × ln(0.05/0.05) + 0.05 × ln(0.05/0.475) • = 0.9 × ln(1.895) + 0 + 0.05 × ln(0.105) • = 0.9 × 0.639 + 0.05 × (-2.254) • ≈ 0.462
KL(Q || M): By symmetry, KL(Q || M) ≈ 0.462
Final divergence: D ≈ 0.5 × 0.462 + 0.5 × 0.462 ≈ 0.462607
This high value (approaching the maximum of ln(2) ≈ 0.693) indicates the distributions are significantly different, with probability mass concentrated on opposite outcomes.
Constraints