Generalized Additive Models (GAMs)

A machine learning model that predicts well but offers no insight is a black box. GAMs are not black boxes. They are glass boxes—models that combine predictive power with transparent, interpretable structure.

The additive structure $f(\mathbf{x}) = \alpha + \sum_j f_j(x_j)$ is not just a computational convenience. It is an interpretation machine: each component function $f_j$ isolates the effect of feature $j$, making complex nonlinear relationships visible, understandable, and actionable.

The primary tool for GAM interpretation is the partial effect plot (also called component plot or smooth term plot). This is simply a plot of the estimated function $\hat{f}_j(x_j)$ against $x_j$.

What the plot shows:

• Horizontal axis: Values of feature $x_j$ (often shown with rug marks at observed values)
• Vertical axis: Contribution to the linear predictor $\hat{f}_j(x_j)$
• Curve: The estimated smooth function
• Shaded region: Confidence band (typically pointwise 95% CI)

The centering convention:

Recall that $\hat{f}_j$ is centered: $\sum_i \hat{f}j(x{ij}) = 0$. This means:

• The y-axis represents deviation from the mean response (on the link scale for GAMs)
• $\hat{f}_j(x_j) > 0$: This value of $x_j$ is associated with above-average response
• $\hat{f}_j(x_j) < 0$: This value is associated with below-average response
• $\hat{f}_j(x_j) = 0$: This value has 'average' effect

Reading partial effect plots:

Example interpretation:

If $\hat{f}{\text{age}}(30) = -0.5$ and $\hat{f}{\text{age}}(60) = 0.8$:

• Age 30 is associated with a response 0.5 below the mean
• Age 60 is associated with a response 0.8 above the mean
• The difference of 1.3 units is the effect of moving from age 30 to 60

Visualizing uncertainty is essential for responsible interpretation. GAMs provide confidence intervals for the smooth functions, but understanding what these intervals mean requires care.

Pointwise confidence bands:

The standard confidence bands shown in partial effect plots are pointwise: at each value $x_j$, the interval covers the true $f_j(x_j)$ with the stated probability (e.g., 95%), if considered in isolation.

Construction: $$\hat{f}j(x_j) \pm z{\alpha/2} \cdot \text{se}(\hat{f}_j(x_j))$$

where the standard error is derived from the covariance matrix of the estimated coefficients.

Across-the-function coverage:

Pointwise intervals do not provide 95% coverage for the entire curve. If you evaluate the interval at many points, some will fail to cover by chance.

Global confidence bands correct for this by widening the intervals. For a curve to be fully within the band with 95% probability:

$$\hat{f}j(x_j) \pm c{\alpha} \cdot \text{se}(\hat{f}_j(x_j))$$

where $c_\alpha > z_{\alpha/2}$ accounts for multiple comparisons. Simulation-based methods can compute $c_\alpha$ for specific situations.

A key question in GAM interpretation: is the smooth term for $x_j$ significantly different from zero? This tests whether feature $j$ contributes to the model beyond noise.

The null hypothesis:

$$H_0: f_j(x) = 0 \text{ for all } x$$

vs.

$$H_1: f_j(x) \neq 0 \text{ for some } x$$

Approximate F-test:

The standard test compares the null model (without $f_j$) to the full model:

$$F = \frac{(\text{RSS}_0 - \text{RSS}_1) / \Delta \text{df}}{\text{RSS}_1 / \text{df}_1}$$

where $\Delta \text{df}$ is the difference in effective degrees of freedom.

Complication: The effective degrees of freedom are not integers, so the F-distribution is only approximate. Modern software uses refined approximations.

Interpreting p-values for smooth terms:

GAM summary output typically shows for each smooth term:

• edf: Effective degrees of freedom (complexity of the fitted curve)
• Ref.df: Reference degrees of freedom for the F-test
• F or Chi-sq: Test statistic
• p-value: Significance level

P-value interpretation:

• $p < 0.05$: Evidence that $f_j \neq 0$; feature contributes to prediction
• $p \geq 0.05$: No significant evidence; consider removing from model

Which features matter most? In linear models, standardized coefficients answer this. In GAMs, we need different approaches since each effect is a curve, not a single number.

Approach 1: Deviance explained

Compare deviance with and without each term:

$$\text{Importance}j = \frac{D{-j} - D_{\text{full}}}{D_{\text{null}} - D_{\text{full}}}$$

where $D_{-j}$ is deviance without feature $j$. This measures the fraction of explained variation attributable to $f_j$.

Approach 2: Effect range

The range of $\hat{f}_j$ over observed data:

$$\text{Range}_j = \max_i \hat{f}j(x{ij}) - \min_i \hat{f}j(x{ij})$$

Larger range = larger effect on predictions. But this doesn't account for frequency of extreme values.

Approach 3: Sum of squared effects

Weight by the data distribution:

$$\text{SSE}j = \sum{i=1}^n \hat{f}j(x{ij})^2$$

Features that contribute large values for many observations rank higher.

Approach 4: Permutation importance

Shuffle feature $j$'s values and measure degradation in prediction:

$$\text{PI}_j = \text{Loss}(\text{shuffled } x_j) - \text{Loss}(\text{original})$$

This measures how much the model relies on the relationship with $x_j$.

When presenting GAM results, it's often useful to compare the magnitude of different feature effects. This requires some normalization since features have different scales.

Scaling considerations:

The y-axis of partial effect plots is in units of the linear predictor. For Gaussian response, this is the response units. For logistic GAMs, it's log-odds. Comparisons are meaningful within the same model but require interpretation.

Effect comparison strategies:

Example: In a model predicting credit risk:

The IQR effect tells us: moving from the 25th to 75th percentile of credit history has a larger impact than the same move for income.

GAMs produce predictions by summing component contributions:

$$\hat{y}^{(i)} = \hat{\alpha} + \sum_{j=1}^p \hat{f}j(x{ij})$$

For GLM-style GAMs, this is the linear predictor; the response scale prediction applies the inverse link:

$$\hat{\mu}^{(i)} = g^{-1}(\hat{y}^{(i)})$$

Decomposing predictions:

One powerful feature of GAMs: we can decompose any individual prediction into feature contributions:

$$\hat{y}^{(i)} = \underbrace{\hat{\alpha}}{\text{baseline}} + \underbrace{\hat{f}1(x{i1})}{\text{effect of } x_1} + \cdots + \underbrace{\hat{f}p(x{ip})}_{\text{effect of } x_p}$$

This provides local explanation: why did this observation get this prediction?

Example decomposition:

For a loan applicant predicted to have 15% default probability:

This shows that employment years is the main concern for this applicant—despite decent income, the short employment history raises risk.

Model checking is essential before trusting GAM interpretations. Residual diagnostics reveal model misspecification.

Standard residual plots:

Checking for missing interactions:

The key GAM assumption—additivity—can be tested by examining residuals for interaction patterns:

1. Plot residuals against $x_j \cdot x_k$ for pairs of features
2. Fit a tensor product smooth $s(x_j, x_k)$ and test significance
3. If interactions are important, extend to GAMM or use interaction terms

Checking smoothness:

• If fitted curves look 'too wiggly,' consider increasing $\lambda$ or reducing $K$
• If curves look 'too smooth' (failing to capture known patterns), decrease $\lambda$ or increase $K$
• The gam.check() function (in mgcv) automates much of this

GAM interpretations must often be communicated to stakeholders who aren't statisticians. Effective communication requires translating technical plots into actionable insights.

Principles for non-technical audiences:

Visualization enhancements:

Despite their interpretability, GAMs can mislead if carelessly interpreted. Here are the most common pitfalls:

Statistical interpretation must connect to domain knowledge. The shape of $\hat{f}_j$ should be evaluated against what's scientifically plausible.

Domain validation questions:

Example: Age effect on mortality

A GAM on mortality data shows:

• Low mortality for ages 5–20
• Rising mortality from 20–80
• Flattening at 80+

Domain validation:

• ✓ Low infant mortality is expected (infants in data may be selected survivors)
• ✓ Rising mortality with age matches known biology
• ? Flattening at 80+ could be real (mortality deceleration is debated) or could be sparse data
• Check: Are there enough 90+ observations to trust the flat trend?

The data pattern makes sense only in conjunction with domain knowledge.

We have explored the rich interpretation tools that make GAMs uniquely valuable for understanding complex relationships.

What's next:

The additive structure provides remarkable interpretability, but it's also limiting: no interactions. The final page explores extensions to GAMs—tensor products, mixed models, and beyond—that relax these limitations while preserving interpretability.