Convolutional Layers

Consider a profound question: When you shift an image of a cat two pixels to the right, what should happen to the features detected by a neural network?

With a fully-connected network, the answer is unpredictable. Shifting pixels rearranges the input vector, potentially activating completely different neurons. A network trained to recognize cats at the image center might fail catastrophically when the same cat appears at the corner.

With a convolutional network, something elegant happens: the detected features shift by exactly the same amount as the input. A cat detector that fires at position (100, 100) for a cat centered there will fire at position (102, 100) when that cat shifts right by two pixels. This property—where output features shift in perfect correspondence with input shifts—is called translation equivariance.

Translation equivariance isn't an accident or an optimization; it's a mathematical guarantee that emerges directly from parameter sharing. Understanding this property is essential for understanding why convolutional networks are so effective for spatial data.

Let's begin with precise mathematical definitions.

Translation Operator:

Define the translation operator T_τ that shifts a function (or image) by displacement vector τ = (τₓ, τᵧ):

$$[T_\tau f](x, y) = f(x - \tau_x, y - \tau_y)$$

This shifts the function in the positive direction by τ. If f has a peak at (10, 10), then T_τf with τ = (3, 5) has a peak at (13, 15).

Equivariance Definition:

A function (or layer) Φ is equivariant to a transformation T if applying T before Φ gives the same result as applying T after Φ:

$$\Phi(T_\tau[f]) = T_\tau[\Phi(f)]$$

In words: transforming the input then processing equals processing then transforming the output. The transformation 'commutes' with the function.

Visualizing Equivariance:

             Φ
   f ──────────────▶ Φ(f)
   │                  │
T_τ│                  │T_τ
   ▼                  ▼
 T_τ[f] ──────────▶ T_τ[Φ(f)] = Φ(T_τ[f])
             Φ

Both paths lead to the same result!

Why Equivariance Matters:

1. Efficient Generalization

If a network is translation equivariant, then learning to detect a feature at one location automatically provides detection at all locations. A single training example of a cat at position A teaches the network about cats at every position.

2. Predictable Feature Behavior

Equivariance guarantees how features transform. We can reason about what a layer does mathematically, not just empirically.

3. Robust Recognition

Objects in real images appear at arbitrary positions. Equivariant representations ensure consistent recognition regardless of position.

4. Meaningful Feature Maps

With equivariance, feature map positions have semantic meaning—activations correspond to spatial locations in the input.

Let's rigorously prove that discrete 2D convolution is translation equivariant.

Setup:

Let f: ℤ² → ℝ be an input image and k: ℤ² → ℝ be a convolution kernel. The convolution is defined as:

$$(f * k)[i, j] = \sum_{m} \sum_{n} f[m, n] \cdot k[i-m, j-n]$$

Alternatively, using the cross-correlation convention common in deep learning:

$$(f \star k)[i, j] = \sum_{m} \sum_{n} f[i+m, j+n] \cdot k[m, n]$$

The Theorem:

For any translation τ = (τₓ, τᵧ):

$$T_\tau[f * k] = (T_\tau f) * k$$

or equivalently:

$$T_\tau[f] * k = T_\tau[f * k]$$

Proof:

Let g = f * k be the convolution output. We need to show that translating f before convolving gives the same result as translating g.

Detailed Proof:

Start with the left side: $(T_\tau f) * k$ at position (i, j):

$$[(T_\tau f) * k][i, j] = \sum_{m} \sum_{n} (T_\tau f)[i+m, j+n] \cdot k[m, n]$$

By definition of translation: $$(T_\tau f)[i+m, j+n] = f[i+m-\tau_x, j+n-\tau_y]$$

Substituting: $$= \sum_{m} \sum_{n} f[i+m-\tau_x, j+n-\tau_y] \cdot k[m, n]$$

Change variables: let m' = m, n' = n (indices sum over all positions, so unchanged): $$= \sum_{m'} \sum_{n'} f[(i-\tau_x)+m', (j-\tau_y)+n'] \cdot k[m', n']$$

This is exactly the convolution evaluated at (i-τₓ, j-τᵧ): $$= (f * k)[i-\tau_x, j-\tau_y]$$

By definition of translation: $$= (T_\tau[f * k])[i, j]$$

QED: $(T_\tau f) * k = T_\tau[f * k]$ ∎

The Key Insight:

Equivariance emerges from the homogeneity of convolution—the same operation is applied everywhere. Parameter sharing guarantees this homogeneity, making equivariance a mathematical certainty, not an empirical observation.

Translation equivariance is best understood through the lens of group theory—the mathematical study of symmetry. This perspective unifies CNNs with a broader family of equivariant architectures.

The Translation Group:

The set of all 2D translations forms a group (ℝ², +) where:

• Elements: Translation vectors τ = (τₓ, τᵧ) ∈ ℝ²
• Operation: Vector addition (composing translations)
• Identity: Zero translation (0, 0)
• Inverse: Negative translation (-τₓ, -τᵧ)

This is an abelian (commutative) group: τ₁ + τ₂ = τ₂ + τ₁

Group Actions:

The translation group acts on the space of images. Each translation τ defines a transformation T_τ of images: $$T_\tau: f \mapsto T_\tau[f]$$

where $T_\tau f = f(x - τ)$

Equivariant Maps:

A function Φ is G-equivariant if it commutes with all group actions: $$\Phi(T_g[f]) = T_g[\Phi(f)] \quad \forall g \in G$$

Convolution is equivariant to the translation group.

Extending Beyond Translations:

The group-theoretic view suggests natural extensions:

1. Rotation Equivariance

Replace the translation group with the rotation group SO(2). Group-equivariant CNNs (G-CNNs) achieve rotation equivariance by convolving with rotated copies of filters:

$$[f * k]\theta = \int f(r{-\theta}(x)) k(x) dx$$

where r_θ is rotation by angle θ.

2. Scale Equivariance

The multiplicative group (ℝ⁺, ×) of positive scalings. Scale-equivariant networks process images at multiple scales with shared weights.

3. Affine and Projective Equivariance

More complex groups handling perspective transformations, useful for 3D vision tasks.

The Group Convolution Theorem:

A linear map Φ: L²(G) → L²(G) is G-equivariant if and only if it can be expressed as convolution on the group:

$$\Phi[f] = f * k$$

for some kernel k. This establishes convolution as the unique linear equivariant operation!

A complete CNN consists of multiple layers. Let's trace how equivariance propagates and where it's intentionally broken.

Layer-by-Layer Analysis:

1. Convolutional Layers: Equivariant ✓

As proven, convolution is translation equivariant. Stacking multiple conv layers preserves equivariance: $$\Phi_2(\Phi_1(T_\tau[x])) = \Phi_2(T_\tau[\Phi_1(x)]) = T_\tau[\Phi_2(\Phi_1(x))]$$

2. Pointwise Nonlinearities (ReLU, etc.): Equivariant ✓

Activation functions applied elementwise preserve equivariance: $$\sigma(T_\tau[f])[i,j] = \sigma(f[i-\tau_x, j-\tau_y]) = T_\tau[\sigma(f)][i,j]$$

3. Batch/Layer Normalization: Approximately Equivariant ~

Statistics computed over spatial positions. Slightly breaks equivariance due to boundary effects and running statistics, but effect is minor.

4. Pooling Layers: Break Equivariance ✗

Max pooling and average pooling over windows break strict equivariance but introduce desired invariance to small translations.

From Equivariance to Invariance:

CNNs build partial invariance through a hierarchy of equivariant layers followed by pooling:

1. Early layers: Fully equivariant, features track spatial position precisely
2. Pooling: Reduces resolution, averages out small translations
3. Deeper layers: Equivariant at coarser scale, invariant to small shifts
4. Global pooling: Completely discards position, full translation invariance
5. Fully-connected classifier: Operates on globally-pooled features

                     Translation Sensitivity
                            ▲
                            │
           High ────────────┼──○ Input pixels
                            │   ╲
                            │    ○ Conv1
                            │     ╲
                            │      ○ Pool1
                            │       ╲
                            │        ○ Conv2
                            │         ╲
                            │          ○ Pool2
                            │           ╲
           Low ─────────────┼────────────○ Global Pool
                            │             ╲
           None ────────────┼──────────────○ Classifier
                            │
                            └──────────────────────────▶ Layer Depth

Design Principle:

Equivariance preserves where features are found. Invariance ultimately discards this for classification (a cat is a cat regardless of position). The key insight is that we want equivariance in intermediate representations (to detect and localize features) but invariance in final predictions (to classify regardless of position).

Understanding equivariance has profound practical implications for training, testing, and deploying CNNs.

1. Data Efficiency

Equivariance provides implicit data augmentation. A network that learns to detect an edge at position (10, 10) automatically detects edges at all positions. Without equivariance, the network would need separate training examples for edges at each location—an exponentially larger dataset.

Formal Analysis:

Consider an image with an object that could appear at H × W different positions. Without equivariance, we might need O(H × W) training examples per object class. With equivariance, a single example teaches the network about all positions, reducing sample complexity by factor O(H × W).

2. Generalization to Unseen Positions

Objects in test images may appear at positions never seen during training. Equivariance guarantees consistent recognition regardless:

$$P(\text{cat} | \text{image with cat at } (x_1, y_1)) = P(\text{cat} | \text{image with cat at } (x_2, y_2))$$

This position-invariant prediction emerges from equivariant representations combined with global pooling.

3. Localization and Detection

Equivariance enables spatial localization. Because features shift with input, we can determine WHERE an object is by finding WHERE features activate:

• Object detection: Bounding box regression at feature map positions
• Semantic segmentation: Per-pixel classification using feature maps
• Pose estimation: Joint location prediction from feature activations

Without equivariance, there would be no spatial correspondence between features and input locations.

4. Transfer Learning

Features learned on one dataset transfer to new datasets with different object positions. An edge detector learned on ImageNet works on medical images where edges appear at different locations. Equivariance ensures this transfer works regardless of spatial statistics.

5. Fully Convolutional Networks

Equivariance enables processing images of arbitrary size:

• Train on 224×224 images
• Test on 1024×1024 images
• Same conv layers, no retraining needed

The network applies the same local operations everywhere, scaling seamlessly.

While equivariance is generally desirable, some tasks require position-dependent processing. Modern architectures selectively break equivariance when beneficial.

When Position Matters:

1. Structured Documents

• Page headers vs body vs footer have different semantics
• Position indicates semantic role, not just content

2. Face Recognition

• Eyes in the upper third, mouth in lower third
• Position provides strong prior on facial features

3. Scene Understanding

• Sky typically above, ground below
• Horizon line provides reference frame

4. Video Prediction

• Object trajectories depend on position (physics)
• Gravity affects objects differently based on height

Methods for Position-Aware Processing:

1. Position Embeddings (Vision Transformers)

Add learnable position vectors to input patches: $$x'_i = x_i + p_i$$

where p_i encodes position i. This allows learning of position-dependent patterns while preserving local equivariance within patches.

2. CoordConv

Concatenate coordinate channels to input:

x_coord = normalized_x_coordinates  # -1 to 1
y_coord = normalized_y_coordinates  # -1 to 1
x_augmented = concat([x, x_coord, y_coord], dim=1)

The network can now learn position-dependent filters using these coordinate channels.

3. Relative Position Bias

Add position-dependent bias to attention scores: $$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d}} + B\right)V$$

where B is a relative position bias matrix.

4. Spatially-Varying Convolutions

Use different kernels at different positions: $$Y[i,j] = \sum_{m,n} K_{i,j}[m,n] \cdot X[i+m, j+n]$$

This fully breaks equivariance but allows position-specific feature extraction.

Translation equivariance in CNNs is a special case of a broader principle: designing neural networks with built-in geometric priors. This field, called Geometric Deep Learning, provides a unified framework for understanding architectures across domains.

The Geometric Deep Learning Blueprint:

1. Identify the domain: Images, graphs, point clouds, meshes, molecules
2. Identify symmetries: What transformations preserve meaning?
3. Design equivariant layers: Ensure network respects those symmetries
4. Build invariant readouts: Aggregate to produce symmetry-invariant predictions

Examples Across Domains:

Images (Grids):

• Symmetry: Translations (optionally rotations)
• Equivariant layer: Convolution
• Invariant readout: Global pooling

Graphs:

• Symmetry: Node permutations (graph isomorphism)
• Equivariant layer: Message passing / Graph convolution
• Invariant readout: Sum/mean over nodes

Point Clouds:

• Symmetry: Permutations + 3D rigid motions
• Equivariant layer: PointNet, SE(3)-transformers
• Invariant readout: Symmetric aggregation

Mathematical Foundation:

The key result connecting group theory to neural networks:

Theorem (Characterization of Equivariant Linear Maps): Let G be a compact group acting on input space X and output space Y. A linear map Φ: X → Y is G-equivariant if and only if:

$$\Phi(x) = \int_G T_g^{(Y)} \cdot k(g) \cdot T_g^{(X)}[x] , dg$$

for some kernel function k: G → L(X, Y). For the translation group on grids, this integral becomes the familiar convolution.

Implications:

1. Convolution is canonical: It's the unique linear equivariant operation for translations
2. Design by symmetry: Identify symmetries first, then architecture follows
3. Guaranteed generalization: Equivariant networks generalize across symmetry transformations
4. Reduced hypothesis space: Symmetry constraints reduce what the network can represent, improving sample efficiency

Translation equivariance is the mathematical property that makes CNNs effective for visual recognition. It emerges directly from parameter sharing and has profound implications for network behavior.

Connection to Next Topic:

Equivariance tells us about global properties—how the entire feature map transforms. The next page explores the receptive field—the local region of input that influences each feature. Understanding receptive fields reveals how CNNs build hierarchical representations, from local edges to global objects.