At its core, classification partitions feature space into distinct regions—one for each class. The boundaries separating these regions are decision boundaries, and understanding their geometry is essential for interpreting classifier behavior, diagnosing problems, and building intuition about model assumptions.

In LDA and QDA, decision boundaries have precise mathematical forms: hyperplanes for LDA, quadric surfaces for QDA. These aren't arbitrary shapes but directly reflect the underlying Gaussian assumptions. A classifier's boundary reveals what it 'believes' about the data—where it's confident, where it's uncertain, and where assumptions may be violated.

This page develops a deep understanding of decision boundaries: their mathematical characterization, geometric properties, how to visualize and interpret them, and what they tell us about model behavior at different points in feature space.

A decision boundary is the set of points where two or more classes have equal posterior probability. For a two-class problem, the boundary is:

$$\mathcal{B} = {x : P(Y = 1 | X = x) = P(Y = 2 | X = x)}$$

Equivalently, using discriminant functions:

$$\mathcal{B} = {x : \delta_1(x) = \delta_2(x)}$$

LDA boundary equation:

For LDA with shared covariance $\Sigma$:

$$\delta_1(x) - \delta_2(x) = (\mu_1 - \mu_2)^T\Sigma^{-1}x - \frac{1}{2}(\mu_1^T\Sigma^{-1}\mu_1 - \mu_2^T\Sigma^{-1}\mu_2) + \log\frac{\pi_1}{\pi_2} = 0$$

This can be written as:

$$w^T x + b = 0$$

where:

• $w = \Sigma^{-1}(\mu_1 - \mu_2)$ is the normal vector to the hyperplane
• $b = -\frac{1}{2}(\mu_1 + \mu_2)^T\Sigma^{-1}(\mu_1 - \mu_2) + \log\frac{\pi_1}{\pi_2}$ is the bias

This is a hyperplane in $\mathbb{R}^p$.

QDA boundary equation:

For QDA with class-specific covariances $\Sigma_1, \Sigma_2$:

$$\delta_1(x) - \delta_2(x) = 0$$

Expanding:

$$-\frac{1}{2}x^T(\Sigma_1^{-1} - \Sigma_2^{-1})x + (\mu_1^T\Sigma_1^{-1} - \mu_2^T\Sigma_2^{-1})x + c = 0$$

where $c$ absorbs constant terms (means, determinants, priors).

This is a quadratic equation in $x$:

$$x^T A x + b^T x + c = 0$$

where:

• $A = \frac{1}{2}(\Sigma_2^{-1} - \Sigma_1^{-1})$ is a symmetric matrix determining the quadratic form
• $b = \Sigma_1^{-1}\mu_1 - \Sigma_2^{-1}\mu_2$ is the linear coefficient
• $c$ is a scalar constant

This defines a quadric surface (conic section in 2D, quadric in higher dimensions).

LDA's linear boundaries have elegant geometric interpretations that provide deep insight into the classifier's behavior.

The boundary as a perpendicular bisector:

With equal priors ($\pi_1 = \pi_2$), the LDA boundary is the perpendicular bisector of the line segment connecting $\mu_1$ and $\mu_2$—but in Mahalanobis space, not Euclidean space.

In Mahalanobis space (after whitening by $\Sigma^{-1/2}$):

• The boundary passes through the midpoint $\frac{1}{2}(\mu_1 + \mu_2)$
• The boundary is perpendicular to the line connecting $\mu_1$ to $\mu_2$

In original space, this manifests as a hyperplane that may not appear perpendicular to ($\mu_1 - \mu_2$) if features are correlated.

Distance interpretation:

Classification is based on Mahalanobis distance to class means. For a point $x$:

$$d_M(x, \mu_k) = \sqrt{(x - \mu_k)^T\Sigma^{-1}(x - \mu_k)}$$

With equal priors, classification reduces to: assign to the class with smaller Mahalanobis distance. The boundary is the locus of points equidistant (in Mahalanobis terms) from both means.

Effect of priors:

When priors are unequal, the boundary shifts:

• If $\pi_1 > \pi_2$: boundary shifts toward class 2 (more of the space is assigned to the more common class 1)
• The shift amount is proportional to $\log(\pi_1/\pi_2)$

Geometrically, the boundary no longer passes through the midpoint; it moves closer to the less common class. This implements Bayesian reasoning: without evidence from features, bet on the common class.

Multi-class boundaries:

With $K > 2$ classes:

• There are $\binom{K}{2}$ pairwise boundaries (hyperplanes)
• Each class's region is the intersection of half-spaces where that class beats all others
• All boundaries pass through a common point only in special symmetric cases
• The overall partition is a Voronoi diagram in Mahalanobis space

The decision regions are convex polytopes—a key property ensuring no 'islands' of one class inside another.

The discriminant direction:

For two classes, the most informative direction is $w = \Sigma^{-1}(\mu_1 - \mu_2)$. Projecting data onto this direction compresses the classification problem to 1D.

The projection $z = w^T x$ has the property that:

• Class means are maximally separated relative to within-class variance
• The 1D threshold classification is optimal (under LDA assumptions)

This is exactly Fisher's Linear Discriminant—the supervised projection that best separates classes.

Boundary margin:

Define the margin as the distance from a class mean to the boundary:

$$\text{margin}_k = \frac{|w^T\mu_k + b|}{|w|}$$

With equal priors, both classes have the same margin (the boundary is equidistant). The total 'gap' between class means projected onto the discriminant direction is:

$$\text{separation} = \frac{|w^T(\mu_1 - \mu_2)|}{|w|} = \frac{(\mu_1 - \mu_2)^T\Sigma^{-1}(\mu_1 - \mu_2)}{|\Sigma^{-1}(\mu_1 - \mu_2)|}$$

Larger separation means better class discrimination.

QDA boundaries are more complex than LDA's hyperplanes, taking the form of quadric surfaces. Understanding these geometries helps interpret QDA behavior.

Quadric surfaces in 2D (conic sections):

In two dimensions, the QDA boundary $x^T A x + b^T x + c = 0$ is a conic section:

1. Ellipse (when $\det(A) > 0$, same-sign eigenvalues):

   • One class is 'inside' the ellipse, one outside
   • Occurs when one class has much smaller variance
   • Example: detecting outliers, anomaly detection

2. Hyperbola (when $\det(A) < 0$, opposite-sign eigenvalues):

   • Classes on opposite branches
   • Occurs when classes have different orientations/correlations
   • Example: classes spread in orthogonal directions

3. Parabola (when $\det(A) = 0$ but $A eq 0$):

   • Asymmetric boundary
   • Occurs in degenerate cases

4. Line (when $A \approx 0$):

   • Degenerate quadric = hyperplane
   • Occurs when $\Sigma_1 \approx \Sigma_2$ (QDA ≈ LDA)

Quadric surfaces in higher dimensions:

In $p$ dimensions, the boundary $x^T A x + b^T x + c = 0$ is a quadric hypersurface:

• Ellipsoid: All eigenvalues of $A$ have the same sign
• Hyperboloid: Eigenvalues have mixed signs
• Paraboloid: Some eigenvalues are zero
• Cylinder: Embedded lower-dimensional quadric

Non-convexity and disconnected regions:

Unlike LDA, QDA decision regions can be:

• Non-convex: A region can be 'waisted' or curved inward
• Disconnected: A class can have multiple separate regions

This happens when one class's covariance is small in some directions—the boundary curves around the tight cluster, potentially isolating it or creating disjoint regions.

Effect of priors in QDA:

Priors shift the constant term $c$, which moves the entire quadric surface without changing its shape. For an elliptical boundary:

• Higher prior for the 'inside' class: ellipse shrinks
• Higher prior for the 'outside' class: ellipse expands

For hyperbolic boundaries, prior changes shift the balance between the two branches.

The decision boundary is where classification is most uncertain—where posterior probabilities are exactly equal. Understanding how confidence varies near boundaries is crucial for reliability.

Posterior probabilities near the boundary:

The posterior probability for the two-class case follows a sigmoid curve as you move perpendicular to the boundary:

$$P(Y = 1 | X = x) = \frac{1}{1 + \exp(-(\delta_1(x) - \delta_2(x)))}$$

• At the boundary: $P(Y = 1 | X) = 0.5$
• Far from boundary (on class 1 side): $P(Y = 1 | X) \to 1$
• Far from boundary (on class 2 side): $P(Y = 1 | X) \to 0$

The rate of transition from 0.5 to high confidence depends on class separation. Well-separated classes have sharp transitions; overlapping classes have gradual transitions.

The confidence gradient:

Define the confidence score $s(x) = \max_k P(Y = k | X = x)$. The gradient of this score (in feature space) points away from boundaries:

• Near boundaries: confidence is low (~$1/K$)
• Far from all boundaries: confidence is high (~1)

For LDA, isoclines of constant confidence are parallel to the decision boundaries (since discriminants are linear). For QDA, isoclines are more complex curved surfaces.

Practical implications:

1. Prediction reliability: Points near boundaries should be flagged as uncertain
2. Active learning: Query labels for points near boundaries to most improve the model
3. Reject option: Abstain from prediction when $\max_k P(Y = k | X) < \tau$

Margin-based approaches:

Define the margin as the distance to the nearest decision boundary:

$$\text{margin}(x) = \min_{\text{boundaries } \mathcal{B}} d(x, \mathcal{B})$$

For LDA, this is easy to compute (perpendicular distance to hyperplane). For QDA, finding the closest point on a quadric is more involved.

Larger margins correlate with higher confidence and more reliable predictions.

Visualization brings abstract boundaries to life, revealing how classifiers partition feature space. Different techniques suit different purposes.

2D contour plots:

For two features:

1. Create a grid of points covering the feature space
2. Compute $\hat{y}$ or $P(Y = k | X)$ at each grid point
3. Plot filled contours or class labels

This shows the boundary as the interface between colored regions.

Decision surface plots (3D):

For two features showing a probability surface:

1. Compute $P(Y = 1 | X = (x_1, x_2))$ on a grid
2. Plot as a 3D surface with height = probability
3. The boundary is where the surface crosses $P = 0.5$

Projection methods for high dimensions:

When $p > 2$, directly visualizing boundaries is impossible. Strategies:

1. Project onto Fisher's discriminant directions (the most discriminative 2D subspace)
2. Use PCA to reduce to 2D (may not preserve boundary structure)
3. Pairwise feature plots (each plot shows one pair of features, all others marginalized)
4. UMAP/t-SNE (nonlinear embeddings that preserve some structure)

Decision boundaries reveal much about both the data and the model. Learning to 'read' boundaries provides diagnostic insight.

Signs of good fit (LDA):

• Boundary passes through the region where classes overlap
• Few training points on the 'wrong' side of boundary
• Boundary orientation aligns with apparent class separation direction
• Classes appear roughly elliptical with similar spread on both sides

Signs of LDA misspecification:

• Linear boundary cuts through one class's region (covariances likely differ)
• Boundary seems 'tilted' relative to natural separation
• One class has more misclassifications than another
• Data appears to need a curved boundary

Signs of good QDA fit:

• Curved boundary follows the natural interface between class distributions
• Boundary shape consistent with observed covariance structure
• Reasonable balance of errors across classes

Feature importance from boundaries:

The boundary normal vector reveals feature importance:

For LDA with weights $w = \Sigma^{-1}(\mu_1 - \mu_2)$:

• Large $|w_j|$: Feature $j$ strongly influences classification
• Small $|w_j|$: Feature $j$ has little discriminative power
• Sign of $w_j$: Indicates direction of effect

However, interpretation requires care:

• Correlated features share importance
• Scaling affects magnitude but not boundary shape
• Importance in LDA differs from marginal class differences

Boundary sensitivity analysis:

How robust is the boundary to perturbations?

1. To data changes: Bootstrap the data and refit; stable boundary → robust classifier
2. To hyperparameters: Vary priors slightly; sensitive boundary → priors matter
3. To outliers: Remove extreme points and refit; movement → outlier influence

Methods like influence functions can quantify how each training point affects the boundary position.

With $K > 2$ classes, decision boundaries form a network of surfaces partitioning feature space into $K$ regions. Understanding this network structure adds insight.

Pairwise boundaries:

There are $\binom{K}{2} = \frac{K(K-1)}{2}$ pairwise boundaries between classes. Each boundary $\mathcal{B}_{kl}$ separates classes $k$ and $l$:

• LDA: $\binom{K}{2}$ hyperplanes, with normal vectors $\Sigma^{-1}(\mu_k - \mu_l)$
• QDA: $\binom{K}{2}$ quadric surfaces

The overall class $k$ region is:

$$\mathcal{R}_k = {x : \delta_k(x) > \delta_l(x) \text{ for all } l eq k}$$

This is the intersection of $K-1$ half-spaces (LDA) or $K-1$ quadric constraints (QDA).

Vertices, edges, and faces:

For LDA in $p$ dimensions:

• Faces ($p-1$ dimensional): The pairwise boundary hyperplanes
• Edges ($p-2$ dimensional): Intersections of pairs of boundaries
• Vertices (0 dimensional): Intersections of $p$ boundaries

Vertices are points where multiple classes have equal posterior—typically where 3+ class regions meet.

Adjacency structure:

Not all class pairs share a 'visible' boundary. If class $m$ is 'between' classes $k$ and $l$, the $k$-$l$ boundary might not be reachable—you'd pass through class $m$ first.

The adjacency graph has:

• Nodes = classes
• Edges = pairs of classes that share a boundary region

This graph depends on the class configuration and may not be complete.

Probability transition along paths:

Moving from the center of class $k$'s region toward class $l$:

$$P(Y = k | X) \xrightarrow{\text{decreases}}$$ $$P(Y = l | X) \xrightarrow{\text{increases}}$$

At the boundary: $P(Y = k | X) = P(Y = l | X)$.

If another class $m$ intervenes, you cross the $k$-$m$ boundary first, then the $m$-$l$ boundary.

Implications for classification:

• Multi-class classification isn't just pairwise comparison—regions interact
• A point may be 'between' two classes but assigned to a third
• The argmax rule resolves all pairwise comparisons simultaneously

What's next:

We've now deeply explored LDA with shared covariance, QDA with class-specific covariances, and the geometry of their decision boundaries. The final topic in this module addresses a critical practical issue: Regularized Discriminant Analysis—methods that interpolate between LDA and QDA to handle high-dimensional settings where both pure methods struggle.