The histogram is perhaps the most recognizable statistical graphic in existence. From introductory statistics courses to high-dimensional data exploration, histograms serve as our primary window into the distributional properties of data. Yet beneath this familiar visualization lies a rigorous framework for nonparametric density estimation.

Introduced by Karl Pearson in 1895 (the term "histogram" itself was coined in a 1891 lecture), the histogram remains ubiquitous for good reason: it is simple to compute, easy to interpret, and often sufficient for exploratory analysis. But histograms are more than just bar charts—they are consistent density estimators with well-understood mathematical properties.

In this page, we'll move far beyond the superficial understanding of histograms. We'll explore their mathematical foundations, analyze their statistical properties, understand their limitations, and learn principled approaches to one of the most vexing questions in practical statistics: How do we choose the bin width?

Let's begin with a precise definition. Given data $\mathcal{D} = {x_1, x_2, \ldots, x_n}$, a histogram partitions the data range into bins and estimates density based on the proportion of points falling into each bin.

Formal Definition:

Choose bin boundaries $b_0 < b_1 < \cdots < b_m$ such that $b_0 \leq \min_i x_i$ and $b_m \geq \max_i x_i$. Define bins $B_j = [b_{j-1}, b_j)$ for $j = 1, \ldots, m$ (with the last bin typically closed on both ends).

Let $n_j$ denote the count of observations in bin $B_j$: $$n_j = \sum_{i=1}^{n} \mathbf{1}(x_i \in B_j)$$

where $\mathbf{1}(\cdot)$ is the indicator function.

The histogram density estimator at point $x$ is: $$\hat{f}_h(x) = \frac{n_j}{n \cdot h_j} \quad \text{for } x \in B_j$$

where $h_j = b_j - b_{j-1}$ is the width of bin $B_j$.

For equal-width bins with common width $h = h_j$ for all $j$: $$\hat{f}_h(x) = \frac{n_j}{nh} \quad \text{for } x \in B_j$$

Verification That Histogram Is a Valid Density:

$$\int_{-\infty}^{\infty} \hat{f}h(x) , dx = \sum{j=1}^{m} \frac{n_j}{nh_j} \cdot h_j = \sum_{j=1}^{m} \frac{n_j}{n} = \frac{\sum_{j=1}^{m} n_j}{n} = \frac{n}{n} = 1$$

Also, $\hat{f}_h(x) \geq 0$ everywhere, so the histogram is indeed a valid probability density function.

Two Critical Choices:

1. Bin width (h): Controls the resolution/smoothness tradeoff
2. Bin origin ($b_0$): Determines where bins start

The bin width is by far the more important choice—it determines the fundamental properties of the estimate. The bin origin has minor effects but can shift features slightly. We'll focus primarily on bin width selection.

Understanding the statistical properties of the histogram estimator requires analyzing its bias, variance, and mean squared error. These calculations reveal how bin width controls the fundamental tradeoff between smoothness and fidelity to data.

Expected Value of Bin Count:

For a point $x$ in bin $B_j$, the bin count $n_j$ follows a binomial distribution: $$n_j \sim \text{Binomial}(n, p_j)$$

where $p_j = \int_{B_j} f(x) , dx = \mathbb{P}(X \in B_j)$ is the probability of landing in bin $B_j$.

Thus: $$\mathbb{E}[n_j] = n \cdot p_j \approx n \cdot h \cdot f(x)$$

where the approximation uses $p_j \approx h \cdot f(x)$ for small bin width.

Bias Analysis:

The expected value of the histogram estimator at a point $x$ in bin $B_j$ is: $$\mathbb{E}[\hat{f}_h(x)] = \mathbb{E}\left[\frac{n_j}{nh}\right] = \frac{p_j}{h}$$

Using Taylor expansion, if $f$ is twice differentiable, the probability of falling in bin $B_j = [b_{j-1}, b_j)$ is: $$p_j = \int_{b_{j-1}}^{b_j} f(t) , dt = h \cdot f(c_j) + \frac{h^3}{24} f''(c_j) + O(h^5)$$

where $c_j = (b_{j-1} + b_j)/2$ is the bin center.

Therefore: $$\mathbb{E}[\hat{f}_h(x)] = f(c_j) + \frac{h^2}{24} f''(c_j) + O(h^4)$$

The bias at point $x$ (which may not equal the bin center) involves two components:

1. Discretization bias: $f(c_j) - f(x)$ — we estimate density at the bin center, not at $x$
2. Smoothing bias: $\frac{h^2}{24} f''(c_j)$ — averaging within bins smooths the density

The integrated squared bias is: $$\int \text{Bias}^2[\hat{f}_h(x)] , dx = O(h^2)$$

Bias decreases as $h \to 0$ (smaller bins, less averaging).

Variance Analysis:

Since $n_j \sim \text{Binomial}(n, p_j)$, we have $\text{Var}(n_j) = np_j(1-p_j)$.

For small bin width, $p_j \approx hf(x) \ll 1$, so: $$\text{Var}(n_j) \approx np_j \approx nhf(x)$$

The variance of the histogram estimator is: $$\text{Var}[\hat{f}_h(x)] = \text{Var}\left[\frac{n_j}{nh}\right] = \frac{\text{Var}(n_j)}{n^2h^2} \approx \frac{nhf(x)}{n^2h^2} = \frac{f(x)}{nh}$$

The integrated variance is: $$\int \text{Var}[\hat{f}_h(x)] , dx \approx \frac{1}{nh}$$

Variance decreases as $h$ increases (larger bins, more points per bin) or as $n$ increases.

Mean Integrated Squared Error (MISE):

Combining bias and variance: $$\text{MISE}(\hat{f}_h) = \int \mathbb{E}[(\hat{f}_h(x) - f(x))^2] , dx = \int \text{Bias}^2 , dx + \int \text{Var} , dx$$

For the histogram: $$\text{MISE}(\hat{f}_h) \approx \frac{h^2}{12} \int [f'(x)]^2 , dx + \frac{1}{nh}$$

(The bias term comes from discretization; the exact coefficient depends on assumptions.)

Optimal Bin Width:

Minimizing MISE with respect to $h$: $$\frac{d}{dh} \text{MISE} = \frac{2h}{12} R(f') - \frac{1}{nh^2} = 0$$

where $R(f') = \int [f'(x)]^2 , dx$ is the roughness of $f'$.

Solving: $$h^* = \left(\frac{6}{n \cdot R(f')}\right)^{1/3}$$

The optimal MISE scales as: $$\text{MISE}^* = O(n^{-2/3})$$

Compare this to KDE's $O(n^{-4/5})$ rate—histograms converge slower due to their piecewise-constant nature.

The theoretical optimal bin width depends on $R(f')$, which involves the unknown density $f$—a classic chicken-and-egg problem. Practical bin width selection rules make assumptions or use data-driven approaches to estimate reasonable values.

1. Sturges' Rule (1926):

One of the oldest rules, based on the assumption that data follow a normal distribution: $$m = 1 + \log_2(n) = 1 + 3.322 \cdot \log_{10}(n)$$ $$h = \frac{\text{range}}{m} = \frac{\max(x) - \min(x)}{1 + \log_2(n)}$$

Derivation: Sturges assumed the data come from a normal distribution and that the bin counts should follow a binomial distribution approximating the normal. Under this idealized scenario, $1 + \log_2(n)$ bins suffice.

Limitations:

• Severely underestimates bins for large $n$ (10,000 points → only 14 bins)
• Assumes normality—fails for skewed or multimodal data
• Ignores data variability

2. Scott's Rule (1979):

Derived by minimizing MISE assuming a normal reference distribution: $$h = 3.49 \cdot \hat{\sigma} \cdot n^{-1/3}$$

where $\hat{\sigma}$ is the sample standard deviation.

Derivation: For a normal density $N(\mu, \sigma^2)$, we have $R(f') = \frac{1}{2\sqrt{\pi}\sigma^3}$. Substituting into the optimal bin width formula gives: $$h^* = \left(\frac{24\sqrt{\pi}}{n}\right)^{1/3} \sigma \approx 3.49 \cdot \sigma \cdot n^{-1/3}$$

Advantages over Sturges:

• Accounts for data variability through $\hat{\sigma}$
• Correct $n^{-1/3}$ scaling
• Works reasonably for unimodal, symmetric distributions

Limitations:

• Normal reference can oversmooth for multimodal data
• Standard deviation is sensitive to outliers

3. Freedman-Diaconis Rule (1981):

A robust alternative using the interquartile range (IQR): $$h = 2 \cdot \text{IQR} \cdot n^{-1/3}$$

where $\text{IQR} = Q_3 - Q_1$ (75th percentile minus 25th percentile).

Rationale: For a normal distribution, $\text{IQR} \approx 1.35\sigma$, so this gives approximately the same result as Scott's rule. However, IQR is more robust to outliers—even 25% of the data being outliers won't affect it.

Advantages:

• Robust to outliers and heavy tails
• Less sensitive to non-normality than Scott's rule
• Often the default in statistical software

Limitations:

• Can undersmooth for heavy-tailed distributions
• Still assumes roughly unimodal symmetric shape

4. Cross-Validation:

A data-driven approach that doesn't assume a reference distribution:

Leave-One-Out Cross-Validation: $$\text{CV}(h) = \frac{2}{(n-1)h} - \frac{n+1}{(n-1)n^2h} \sum_{j=1}^{m} n_j^2$$

Minimize CV$(h)$ over a grid of $h$ values. This optimizes an unbiased estimate of ISE without knowing $f$.

Advantages:

• Fully data-driven
• Adapts to any distributional shape
• Theoretically optimal under mild conditions

Disadvantages:

• Computationally intensive
• Solution may not be unique
• High variability in selected $h$ for small samples

Despite their utility, histograms have fundamental limitations that motivated the development of more sophisticated density estimators. Understanding these limitations is essential for knowing when to use alternative methods.

Demonstration: Bin Origin Sensitivity

Consider data ${0.9, 1.1, 2.9, 3.1}$ representing two clusters near 1 and 3.

• Origin at 0, width 1: Bins [0,1), [1,2), [2,3), [3,4) give counts {1, 1, 1, 1}—appears uniform!
• Origin at 0.5, width 1: Bins [0.5,1.5), [1.5,2.5), [2.5,3.5) give counts {2, 0, 2}—correctly shows bimodality.

Same data, same bin width, completely different visual impression. The "true" structure (two clusters) is revealed or hidden based on an arbitrary choice.

Discontinuity and Derivative Failure:

For many applications, we need not just the density but also its derivatives. For example:

• Mode finding: Requires $f'(x) = 0$
• Curvature analysis: Requires $f''(x)$
• Score matching: Uses $ abla \log f(x)$

Histograms fail here entirely—their derivative is zero almost everywhere (within bins) and undefined at bin boundaries. This makes them unsuitable for applications requiring the density's shape characteristics.

Kernel density estimation, covered in the next page, addresses all these limitations by producing smooth, infinitely differentiable estimates (with appropriate kernel choice).

The limitations of fixed-width histograms have inspired extensions that allow bin widths to vary across the data range.

Variable-Width Histograms:

Rather than fixing bin width, we can fix the number of points per bin. The resulting histogram has:

• Narrow bins in high-density regions (fine resolution where data is abundant)
• Wide bins in low-density regions (reduced variance where data is sparse)

Equal-Frequency Binning: $$n_j = \frac{n}{m} \quad \text{for all bins } j$$

The density estimate becomes: $$\hat{f}(x) = \frac{1}{nh_j} \cdot \frac{n}{m} = \frac{1}{mh_j}$$

Narrow $h_j$ → high density estimate; wide $h_j$ → low density estimate.

Advantages:

• Automatically adapts to local density
• Reduces variance in sparse regions
• Maintains resolution in dense regions

Disadvantages:

• Bins may have unnatural boundaries
• Harder to interpret visually
• Doesn't achieve optimal convergence rates

Bayesian Blocks:

A more sophisticated approach uses Bayesian model selection to choose bin boundaries optimally.

Idea: Treat the number and location of bins as unknown parameters. Use Bayesian inference to select the configuration that best balances fit and complexity.

The algorithm proceeds by dynamic programming:

1. Compute the best single-block representation
2. Consider all possible places to add a bin boundary
3. Choose the configuration that maximizes a fitness function penalizing complexity

Formally: Maximize: $$\text{Fitness} = \sum_{j=1}^{m} \text{block_fitness}(B_j) - m \cdot \text{penalty}$$

where block fitness measures how well the constant rate within a bin explains the data, and the penalty prevents overfitting.

Advantages:

• Automatically determines both number and location of bins
• No ad-hoc bin width selection
• Particularly useful for detecting change points in time series

Disadvantages:

• Computationally intensive
• May produce many narrow bins near modes
• Less familiar to general audiences

Extending histograms to multiple dimensions reveals the first serious encounter with the curse of dimensionality—a phenomenon that plagues all nonparametric methods.

Construction in $d$ Dimensions:

For data in $\mathbb{R}^d$, we partition each dimension into bins, creating a grid of hypercubes. If each dimension has $m$ bins:

• Total number of bins: $m^d$
• Expected points per bin: $n/m^d$

The Curse Emerges:

Consider $n = 1000$ points and $m = 10$ bins per dimension:

By dimension 4, we expect fewer than one point per bin on average. Most bins are empty, making density estimation essentially impossible.

Theoretical Analysis:

For a $d$-dimensional histogram with bin widths $h$, the MISE becomes: $$\text{MISE} \approx \frac{h^2}{12} \sum_{i=1}^{d} \int \left(\frac{\partial f}{\partial x_i}\right)^2 dx + \frac{1}{nh^d}$$

Optimizing over $h$: $$h^* = O(n^{-1/(d+2)})$$ $$\text{MISE}^* = O(n^{-2/(d+2)})$$

As $d$ increases, the convergence rate deteriorates:

The sample size required for a given accuracy grows exponentially with dimension.

Practical Multivariate Approaches:

1. 2D Histograms: Still viable for joint distributions of two variables. Displayed as heatmaps.

2. Marginal Histograms: Estimate 1D marginal densities separately. Loses dependence structure.

3. Pair Plots: Grid of 2D histograms for all variable pairs. Good for exploration.

4. Hexagonal Binning: For 2D, hexagonal bins are visually superior and reduce boundary artifacts.

5. Dimensionality Reduction + Histogram: Project data to 2-3 dimensions (PCA, t-SNE, UMAP), then histogram.

When implementing histograms in practice, several considerations beyond basic construction affect the quality and utility of results.

Edge Handling:

• Inclusion/Exclusion: Standard convention: bins are $[b_{j-1}, b_j)$ except the last bin $[b_{m-1}, b_m]$. This ensures all points are included exactly once.
• Boundary Extension: Sometimes extend bins slightly beyond data range to avoid boundary artifacts.
• Open-ended Bins: For unbounded data, first and last bins can have infinite extent (but finite count).

Computational Efficiency:

For finding which bin a point falls into:

• Equal-width bins: Direct computation: $j = \lfloor (x - b_0) / h \rfloor + 1$. O(1) per point.
• Unequal bins: Binary search over bin boundaries. O(log m) per point.

Total complexity: O(n) for equal-width, O(n log m) for unequal.

Memory Efficiency:

• Only need to store bin counts, not individual points
• For very large data, histograms provide massive compression
• Streaming construction: update counts as data arrives, no need to store raw data

Common Pitfalls:

1. Forgetting to Normalize: Dividing counts by $n$ gives probability, but dividing by $nh$ gives density. Know which you need.

2. Ignoring Empty Bins: Empty bins have zero density, not undefined. Handle in downstream calculations.

3. Extreme Values: A single outlier can drastically expand the range, wasting bins on empty regions. Consider trimming or log transformation.

4. Integer Data: Discrete data may cluster at integers. Align bin boundaries between integers to reveal true structure.

5. Sparse Regions: Bins with few points have high variance. Don't overinterpret small bumps.

Quality Checks:

• Sum of bin areas should equal 1 (for density histogram)
• Sum of bin counts should equal n
• Visual inspection: do modes align with expected features?
• Sensitivity analysis: how much do conclusions change with different bin widths?

We've now developed a thorough understanding of histogram density estimation—from basic construction through advanced topics. Let's consolidate the key insights:

What's Next:

The histogram's limitations—particularly discontinuity and bin origin sensitivity—motivate the search for smoother estimators. The next page introduces Kernel Density Estimation (KDE), which addresses these issues elegantly by centering a smooth kernel function at each data point. KDE achieves faster convergence rates, produces differentiable estimates, and eliminates dependence on arbitrary bin boundaries—making it the modern workhorse of nonparametric density estimation.