Vector Spaces and Subspaces

So far, we've worked with vectors as concrete objects—ordered lists of numbers in $\mathbb{R}^n$. But the true power of linear algebra emerges from abstraction: recognizing that many different mathematical objects behave like vectors.

Polynomials. Functions. Matrices. Signals. Probability distributions. All of these can form vector spaces—systems where addition and scalar multiplication make sense and follow the same fundamental rules.

This abstraction is not merely academic elegance. Machine learning algorithms are designed to work in vector spaces, and understanding the abstract structure explains why these algorithms apply equally well to image pixels, word embeddings, graph features, or any other vectorizable data.

This page formalizes the concept of vector spaces and subspaces, explores basis and dimension, and introduces the fundamental subspaces that arise from matrices—structures that directly determine the behavior of linear transformations in ML.

Formal definition:

A vector space (over the real numbers $\mathbb{R}$) is a set $V$ together with two operations:

• Vector addition: $+: V \times V \to V$
• Scalar multiplication: $\cdot: \mathbb{R} \times V \to V$

satisfying the following eight axioms for all vectors $\mathbf{u}, \mathbf{v}, \mathbf{w} \in V$ and all scalars $c, d \in \mathbb{R}$:

Additional axioms (sometimes listed as 9 and 10):

• Scalar multiplication associativity: $(cd)\mathbf{v} = c(d\mathbf{v})$
• Multiplicative identity: $1 \cdot \mathbf{v} = \mathbf{v}$

Immediate consequences of the axioms:

1. Zero vector is unique: There's exactly one zero vector in any vector space
2. Additive inverse is unique: Each vector has exactly one negative
3. $0 \cdot \mathbf{v} = \mathbf{0}$: Scaling by zero gives the zero vector
4. $c \cdot \mathbf{0} = \mathbf{0}$: Scaling zero gives zero
5. $(-1)\mathbf{v} = -\mathbf{v}$: Scaling by -1 gives the additive inverse

These aren't axioms—they're theorems proven from the axioms.

The axioms apply far beyond lists of numbers. Here are important examples of vector spaces:

1. $\mathbb{R}^n$ — n-dimensional real coordinate space

The canonical example. Vectors are n-tuples $(x_1, \ldots, x_n)$, addition and scaling are component-wise. All axioms are satisfied.

2. $\mathbb{P}_n$ — Polynomials of degree at most n

Vectors are polynomials $p(x) = a_0 + a_1 x + a_2 x^2 + \cdots + a_n x^n$. Addition and scaling are the natural operations on polynomials. The zero vector is the zero polynomial.

3. $\mathcal{C}[a,b]$ — Continuous functions on $[a,b]$

Vectors are continuous functions. Addition: $(f+g)(x) = f(x) + g(x)$. Scaling: $(cf)(x) = c \cdot f(x)$. The zero vector is the function $f(x) = 0$.

4. $M_{m \times n}$ — Matrices of size $m \times n$

Vectors are matrices. Addition and scaling are entry-wise (same as viewing the matrix as a vector of $mn$ entries).

5. Solution space of homogeneous linear system

The set of all solutions to $A\mathbf{x} = \mathbf{0}$ forms a vector space (the null space of $A$).

6. Sequences with finite support

Sequences $(a_0, a_1, a_2, \ldots)$ where only finitely many terms are non-zero. This is the foundation of some signal processing applications.

Non-examples (things that are NOT vector spaces):

Definition of subspace:

A subspace $W$ of a vector space $V$ is a non-empty subset of $V$ that is itself a vector space under the same operations.

Subspace test (simpler than checking all axioms):

To verify that $W$ is a subspace of $V$, check:

1. Non-empty: $W$ contains at least one vector (the zero vector suffices)
2. Closed under addition: If $\mathbf{u}, \mathbf{v} \in W$, then $\mathbf{u} + \mathbf{v} \in W$
3. Closed under scalar multiplication: If $\mathbf{v} \in W$ and $c \in \mathbb{R}$, then $c\mathbf{v} \in W$

Or equivalently (combining 2 and 3):

• $W$ is non-empty
• For all $\mathbf{u}, \mathbf{v} \in W$ and all $c, d \in \mathbb{R}$: $c\mathbf{u} + d\mathbf{v} \in W$

Examples of subspaces:

In $\mathbb{R}^3$:

• ${\mathbf{0}}$ — The trivial subspace (just the origin)
• Any line through the origin — A 1D subspace
• Any plane through the origin — A 2D subspace
• $\mathbb{R}^3$ itself — The full space is a subspace of itself

In polynomial space $\mathbb{P}_n$:

• Polynomials with $p(0) = 0$ — A subspace
• Polynomials where the coefficient of $x^k$ is zero — A subspace

The span of any set of vectors is always a subspace. This is fundamental: given any vectors, their span is the smallest subspace containing them.

Definition of basis:

A basis for a vector space $V$ is a set of vectors ${\mathbf{b}_1, \mathbf{b}_2, \ldots, \mathbf{b}_n}$ that is:

1. Linearly independent: No vector is a linear combination of the others
2. Spanning: Every vector in $V$ can be expressed as a linear combination of the basis vectors

In other words, a basis is a minimal spanning set or equivalently a maximal independent set.

Definition of dimension:

The dimension of a vector space $V$, denoted $\dim(V)$, is the number of vectors in any basis of $V$.

Remarkable fact: Every basis of the same vector space has the same number of vectors. This is why dimension is well-defined.

Key theorem (Unique Representation):

If ${\mathbf{b}_1, \ldots, \mathbf{b}_n}$ is a basis for $V$, then every vector $\mathbf{v} \in V$ can be written uniquely as:

$$\mathbf{v} = c_1 \mathbf{b}_1 + c_2 \mathbf{b}_2 + \cdots + c_n \mathbf{b}_n$$

The scalars $(c_1, c_2, \ldots, c_n)$ are called the coordinates of $\mathbf{v}$ with respect to this basis.

Standard bases:

Properties of bases:

• Any n independent vectors in $\mathbb{R}^n$ form a basis
• Any n spanning vectors in $\mathbb{R}^n$ form a basis (if they span, there must be exactly n)
• Fewer than n vectors cannot span $\mathbb{R}^n$
• More than n vectors in $\mathbb{R}^n$ cannot be independent

Infinite-dimensional spaces:

Some vector spaces (like continuous functions) have no finite basis—they're infinite-dimensional. This is important for understanding function spaces in ML (e.g., Gaussian processes, kernel methods).

Every matrix $A \in \mathbb{R}^{m \times n}$ has four fundamental subspaces that completely characterize its behavior. Understanding these is essential for understanding linear transformations, least squares, SVD, and many ML algorithms.

The four subspaces:

1. Column Space (Range): $\mathcal{C}(A)$ or $\text{col}(A)$

• Definition: Span of the columns of $A$
• Lives in: $\mathbb{R}^m$
• Dimension: $r$ (the rank of $A$)
• Meaning: All vectors $\mathbf{b}$ for which $A\mathbf{x} = \mathbf{b}$ has a solution

2. Null Space (Kernel): $\mathcal{N}(A)$ or $\ker(A)$

• Definition: All solutions to $A\mathbf{x} = \mathbf{0}$
• Lives in: $\mathbb{R}^n$
• Dimension: $n - r$ (nullity)
• Meaning: Inputs that the transformation $A$ maps to zero

3. Row Space: $\mathcal{C}(A^\top)$

• Definition: Span of the rows of $A$ (= columns of $A^\top$)
• Lives in: $\mathbb{R}^n$
• Dimension: $r$
• Meaning: The "effective" input directions

4. Left Null Space: $\mathcal{N}(A^\top)$

• Definition: All solutions to $A^\top\mathbf{y} = \mathbf{0}$
• Lives in: $\mathbb{R}^m$
• Dimension: $m - r$
• Meaning: Output directions not reachable by $A$

Dimension relationships:

$$\dim(\mathcal{C}(A)) + \dim(\mathcal{N}(A)) = n \quad \text{(number of columns)}$$ $$\dim(\mathcal{C}(A^\top)) + \dim(\mathcal{N}(A^\top)) = m \quad \text{(number of rows)}$$ $$\dim(\mathcal{C}(A)) = \dim(\mathcal{C}(A^\top)) = r \quad \text{(rank)}$$

ML interpretation:

For a data matrix $X$ or weight matrix $W$:

• Column space: What outputs/predictions are achievable
• Null space: What input variations have no effect (information lost)
• Row space: What input directions matter for the output
• Left null space: What output directions are unreachable

The same vector has different coordinates in different bases. Change of basis describes how coordinates transform when we switch from one basis to another.

Setup:

Let $\mathcal{B} = {\mathbf{b}_1, \ldots, \mathbf{b}_n}$ and $\mathcal{C} = {\mathbf{c}_1, \ldots, \mathbf{c}_n}$ be two bases for a vector space $V$.

A vector $\mathbf{v}$ has:

• Coordinates $[\mathbf{v}]_\mathcal{B} = (\alpha_1, \ldots, \alpha_n)$ in basis $\mathcal{B}$
• Coordinates $[\mathbf{v}]_\mathcal{C} = (\beta_1, \ldots, \beta_n)$ in basis $\mathcal{C}$

The change of basis matrix:

$$[\mathbf{v}]\mathcal{C} = P{\mathcal{B} \to \mathcal{C}} \cdot [\mathbf{v}]_\mathcal{B}$$

where $P_{\mathcal{B} \to \mathcal{C}}$ is the change of basis matrix from $\mathcal{B}$ to $\mathcal{C}$.

To construct $P_{\mathcal{B} \to \mathcal{C}}$:

• Express each basis vector $\mathbf{b}_j$ in terms of basis $\mathcal{C}$
• These coordinates form the columns of $P$

Special case: Standard basis to custom basis

If $\mathcal{B}$ is custom and $\mathcal{C}$ is the standard basis:

• Place basis vectors of $\mathcal{B}$ as columns of matrix $B$
• $[\mathbf{v}]\text{standard} = B \cdot [\mathbf{v}]\mathcal{B}$
• $[\mathbf{v}]\mathcal{B} = B^{-1} \cdot [\mathbf{v}]\text{standard}$

Transformation in different bases:

A linear transformation $T$ has different matrix representations in different bases: $$[T]\mathcal{C} = P^{-1} [T]\mathcal{B} P$$

This is similarity transformation—the transformation is the same, but its matrix representation changes with basis.

The abstract framework of vector spaces illuminates many ML concepts.

1. Feature spaces:

Each data point lives in a feature space $\mathbb{R}^d$. The dimension $d$ is the number of features. Understanding this as a vector space explains:

• Why linear models find hyperplanes (subspaces)
• Why feature engineering changes the space structure
• Why dimensionality reduction projects to subspaces

2. Weight spaces:

Neural network weights form vectors in high-dimensional spaces. The loss surface is defined on this weight space. Optimization navigates this space.

3. Null space and solutions:

For $A\mathbf{x} = \mathbf{b}$:

• If solution exists, all solutions have form $\mathbf{x}_p + \mathbf{n}$ where $\mathbf{n}$ is in null space
• Regularization picks among these solutions (prefer small $|\mathbf{x}|$)

4. Rank and expressivity:

• Low-rank approximations compress data (SVD, matrix factorization)
• Rank of weight matrix limits layer's expressive power
• Rank of feature matrix determines degrees of freedom in fitting