Kernel Design for Gaussian Processes

If Gaussian Processes are the engine of non-parametric Bayesian inference, then kernels are the soul that determines what kinds of functions we believe in. Every choice we make about the kernel directly translates into assumptions about function smoothness, periodicity, correlation length, and ultimate behavior.

This page focuses on stationary kernels—also called shift-invariant kernels—which represent the most widely-used class of covariance functions. Stationary kernels encode the idea that the statistical relationship between function values depends only on the distance between input points, not their absolute location. This property makes them extraordinarily versatile for modeling phenomena where we expect translation invariance.

Before diving into specific kernels, we must establish precisely what stationarity means in the context of covariance functions. This concept is foundational to understanding why certain kernels are appropriate for certain problems.

Definition: Stationary Kernel

A kernel (covariance function) $k(\mathbf{x}, \mathbf{x}')$ is stationary if it can be expressed as a function of only the difference $\mathbf{x} - \mathbf{x}'$ rather than the absolute positions:

$$k(\mathbf{x}, \mathbf{x}') = k(\mathbf{x} - \mathbf{x}')$$

This means the covariance between any two function values $f(\mathbf{x})$ and $f(\mathbf{x}')$ depends only on the vector separating them, not where they are in input space. If we translate both points by the same amount, the covariance remains unchanged.

Definition: Isotropic Kernel

A stationary kernel is isotropic (or radial) if it depends only on the Euclidean distance $r = |\mathbf{x} - \mathbf{x}'|$:

$$k(\mathbf{x}, \mathbf{x}') = k(r) \quad \text{where } r = |\mathbf{x} - \mathbf{x}'|$$

Isotropic kernels are rotationally invariant—they treat all directions in input space identically. Most common stationary kernels are isotropic.

Connection to Stochastic Processes

The term 'stationary' in kernel theory directly connects to stationarity in stochastic processes. A Gaussian Process with a stationary kernel is a wide-sense stationary stochastic process, meaning:

1. The mean function is constant (typically zero)
2. The autocovariance function depends only on the lag (difference in inputs)

This connection provides powerful tools for analysis. In particular, Bochner's theorem gives us a complete characterization: a continuous, stationary kernel is positive definite if and only if it is the Fourier transform of a finite, non-negative measure (the spectral measure). This spectral representation is the foundation of spectral analysis of GPs, which we'll explore in a later page.

The Positive Definiteness Requirement

Not every function of $|\mathbf{x} - \mathbf{x}'|$ is a valid kernel. A kernel must be positive semi-definite (PSD): for any set of points ${\mathbf{x}_1, ..., \mathbf{x}n}$, the resulting covariance matrix $\mathbf{K}$ with entries $K{ij} = k(\mathbf{x}_i, \mathbf{x}_j)$ must be positive semi-definite (all eigenvalues $\geq 0$).

This constraint ensures that the GP defines a valid probability distribution. Violating it leads to nonsensical 'negative variances.' Fortunately, the kernels we discuss here are all provably PSD.

The Radial Basis Function (RBF) kernel, also known as the Squared Exponential or Gaussian kernel, is the most widely used kernel in GP practice. Its ubiquity stems from its simplicity, smoothness properties, and well-understood behavior.

Mathematical Definition

$$k_{\text{RBF}}(\mathbf{x}, \mathbf{x}') = \sigma^2 \exp\left(-\frac{|\mathbf{x} - \mathbf{x}'|^2}{2\ell^2}\right)$$

Where:

• $\sigma^2$ is the output variance (signal variance): controls the overall vertical scale of function variations
• $\ell$ is the length-scale: controls how quickly the correlation decays with distance

Alternatively, using the squared distance $r^2 = |\mathbf{x} - \mathbf{x}'|^2$:

$$k_{\text{RBF}}(r) = \sigma^2 \exp\left(-\frac{r^2}{2\ell^2}\right)$$

Understanding the Length-Scale Parameter

The length-scale $\ell$ is perhaps the most important hyperparameter in GP modeling. It answers the question: over what distance do function values remain correlated?

• Small $\ell$: Rapid decorrelation. Function can vary quickly over short distances. More wiggly samples.
• Large $\ell$: Slow decorrelation. Function values remain similar over large distances. Smoother, more slowly-varying samples.

Rule of Thumb: If $|\mathbf{x} - \mathbf{x}'| \gg \ell$, then $k(\mathbf{x}, \mathbf{x}') \approx 0$. Points separated by more than a few length-scales are essentially uncorrelated.

Quantitative Relationship:

This exponential decay is characteristic of the RBF kernel.

When to Use the RBF Kernel

✅ Appropriate for:

• Smooth, slowly-varying functions without sharp transitions
• Physical processes governed by diffusion or wave equations
• Initial exploration when you have no strong prior beliefs about function roughness
• Problems where infinite differentiability is a reasonable assumption

❌ Inappropriate for:

• Functions with discontinuities or sharp transitions
• Rough textures in spatial modeling (terrain, surfaces)
• Time series with irregular patterns or sudden changes
• Any domain where infinite smoothness is too strong an assumption

The Over-Smoothing Problem

A notorious issue with the RBF kernel is its tendency toward over-smoothing or over-confidence. Because it assumes infinite differentiability, the posterior predictions can be overly certain in regions between data points, failing to capture the true uncertainty when the underlying function is rough.

The Matérn kernel family addresses the RBF kernel's over-smoothness problem by providing explicit control over function differentiability. This makes it arguably the most versatile family of stationary kernels for practical applications.

Mathematical Definition

The general Matérn kernel is:

$$k_{\text{Matérn}}(r) = \sigma^2 \frac{2^{1-\nu}}{\Gamma(\nu)} \left(\frac{\sqrt{2\nu}, r}{\ell}\right)^\nu K_\nu\left(\frac{\sqrt{2\nu}, r}{\ell}\right)$$

Where:

• $r = |\mathbf{x} - \mathbf{x}'|$ is the Euclidean distance
• $\sigma^2$ is the output variance
• $\ell$ is the length-scale
• $\nu > 0$ is the smoothness parameter (not a length-scale!)
• $K_\nu$ is the modified Bessel function of the second kind
• $\Gamma$ is the Gamma function

The Smoothness Parameter ν

The parameter $\nu$ directly controls the mean-squared differentiability of the GP samples:

• A GP with the Matérn kernel is $\lceil \nu \rceil - 1$ times differentiable (in the mean-squared sense)
• As $\nu \to \infty$, the Matérn kernel converges to the RBF kernel

Choosing the Smoothness Parameter

Unlike the length-scale, which is typically learned from data, the smoothness parameter $\nu$ often requires domain knowledge or careful consideration:

1. Physical constraints: If derivatives of the modeled function have physical meaning (velocity, acceleration), choose $\nu$ to ensure they exist

2. Visual inspection: Examine your data—does it appear smooth or rough? Sharp transitions suggest lower $\nu$

3. Model comparison: Use marginal likelihood to compare different $\nu$ values

4. Default to Matérn-5/2: When uncertain, this provides a good balance of smoothness and flexibility

Why Not Learn ν?

While $\nu$ can be treated as a hyperparameter to optimize, this is often problematic:

• The marginal likelihood landscape with respect to $\nu$ is typically flat
• Small changes in $\nu$ can dramatically change the posterior
• The interaction between $\nu$ and $\ell$ can make optimization difficult

For these reasons, practitioners usually fix $\nu$ to one of the special cases (0.5, 1.5, 2.5) and learn only $\sigma^2$ and $\ell$.

Many real-world phenomena exhibit periodicity: daily temperature cycles, seasonal patterns in sales data, weekly traffic patterns, or annual biological rhythms. The periodic kernel encodes this structure directly into the GP prior.

Mathematical Definition

$$k_{\text{Periodic}}(\mathbf{x}, \mathbf{x}') = \sigma^2 \exp\left(-\frac{2\sin^2\left(\pi |\mathbf{x} - \mathbf{x}'| / p\right)}{\ell^2}\right)$$

Where:

• $\sigma^2$ is the output variance
• $p$ is the period (the distance after which the pattern repeats)
• $\ell$ is the length-scale (controls smoothness within each period)

Key Properties

1. Exact periodicity: $k(x, x') = k(x+p, x'+p)$ — the covariance structure repeats exactly every $p$ units

2. Cross-period correlation: Points separated by exactly one period ($|x - x'| = p$) have the same correlation as points at the same location ($|x - x'| = 0$)

3. Infinite repetition: The kernel assumes the periodic pattern continues forever in both directions

Understanding the Period and Length-Scale Interaction

The periodic kernel has a subtle interplay between its parameters:

• Period $p$: Determines where the pattern repeats. This is often known from domain knowledge (24 hours for daily, 365.25 days for yearly)

• Length-scale $\ell$: Controls how smoothly the pattern varies within each period. Smaller $\ell$ allows more wiggly variations within the period; larger $\ell$ produces smoother within-period behavior

When $\ell \ll p$: The function can vary rapidly within each period When $\ell \gg p$: The function becomes nearly sinusoidal (only the fundamental frequency matters)

Extensions and Variations

1. Locally Periodic: $k_{\text{LP}} = k_{\text{Periodic}} \times k_{\text{RBF}}$

   • Captures periodic patterns that decay or evolve over time

2. Quasi-Periodic: Period itself varies smoothly

   • Useful for phenomena with drifting frequencies

3. Multiple Periods: Sum of periodic kernels with different periods

   • Models multi-frequency phenomena (e.g., daily + weekly + yearly)

Beyond the RBF, Matérn, and Periodic kernels, several other stationary kernels serve specific purposes. Understanding their characteristics helps you select the right tool for specialized applications.

The Rational Quadratic Kernel

The Rational Quadratic (RQ) kernel deserves special attention due to its unique interpretation:

$$k_{\text{RQ}}(r) = \sigma^2 \left(1 + \frac{r^2}{2\alpha\ell^2}\right)^{-\alpha}$$

The parameter $\alpha > 0$ controls how the different length-scales are mixed:

• As $\alpha \to \infty$: RQ kernel → RBF kernel
• Small $\alpha$: Significant contribution from both short and long length-scales

Mathematical Insight: The RQ kernel can be derived as a scale mixture of RBF kernels where the length-scale follows an inverse-gamma distribution. This makes it ideal for functions that exhibit multi-scale behavior—smooth in some regions, with finer variations in others.

Practical Guidance:

• Start with simpler kernels (RBF or Matérn-5/2) unless you have evidence of multi-scale structure
• RQ can help when a single length-scale seems inadequate but you don't want to commit to a specific composition
• The extra parameter $\alpha$ adds flexibility but also complexity in hyperparameter optimization

Selecting the right kernel is both an art and a science. While hyperparameters can be learned from data, the kernel family encodes structural assumptions that must often come from domain knowledge or exploratory analysis.

A Systematic Selection Process

Common Pitfalls and How to Avoid Them

1. Over-Reliance on RBF: The RBF kernel is the 'Gaussian distribution of kernels'—simple and often good enough, but its infinite smoothness assumption is rarely accurate. Default to Matérn-5/2 instead.

2. Ignoring Periodicity: Failing to model known periodic structure leads to poor predictions. If you know the period, encode it!

3. Too Many Parameters: Complex kernels with many hyperparameters can overfit or be hard to optimize. Start simple and add complexity incrementally.

4. Forgetting Scaling: Input features should typically be normalized. A length-scale of 1.0 means different things for features ranging [0,1] vs [0,1000].

5. Not Visualizing Samples: Before fitting, sample from the prior to check if the kernel produces reasonable-looking functions. If prior samples don't resemble plausible data, reconsider your kernel choice.

Stationary kernels form the foundation of most Gaussian Process applications. Their shift-invariance property—that correlations depend only on distances, not absolute positions—makes them versatile and interpretable.

What's Next:

Stationary kernels assume uniform properties across the input space. In the next page, we'll explore non-stationary kernels that can model functions whose characteristics vary with location—essential for handling trends, boundaries, and heterogeneous domains.