Loading learning content...
When a linear transformation acts on a vector space, something interesting happens: some dimensions might collapse. A 3D space might be flattened into a 2D plane, or even crushed to a line. The concepts of rank and nullity precisely measure this dimensional behavior.
Rank tells us how much of the input space's structure survives transformation—the dimension of the output.
Nullity tells us how much is destroyed—the dimension of the 'invisible' subspace that maps to zero.
Together, they reveal the fundamental structure of any linear map and explain why some linear systems have unique solutions, others have infinitely many, and some have none at all.
By the end of this page, you will understand rank and nullity geometrically, apply the rank-nullity theorem, recognize rank-deficient situations in ML (multicollinearity, singular matrices), and understand why these concepts matter for solving linear systems and training models.
The column space (or range or image) of a matrix $A$ is the set of all possible outputs when we apply $A$ to any input:
$$\text{Col}(A) = \{A\mathbf{x} : \mathbf{x} \in \mathbb{R}^n\}$$
From the column view of matrix-vector multiplication, $A\mathbf{x}$ is a linear combination of the columns of $A$: $$A\mathbf{x} = x_1\mathbf{a}_1 + x_2\mathbf{a}_2 + ... + x_n\mathbf{a}_n$$
Therefore, the column space is exactly the span of the columns of $A$.
The rank of a matrix is the dimension of its column space:
$$\text{rank}(A) = \dim(\text{Col}(A))$$
Equivalently, rank is the number of linearly independent columns of $A$ (or equivalently, the number of linearly independent rows—these two numbers are always equal).
Geometric interpretation:
Rank tells you the dimension of the output space. If $A$ is a $5 \times 3$ matrix with rank 2:
$$A = \\begin{bmatrix} 1 & 2 & 3 \\\\ 2 & 4 & 6 \\\\ 1 & 1 & 1 \\end{bmatrix}$$Column 2 = 2 × Column 1. Column 3 = 3 × Column 1.
Only column 1 and column 3 are independent... wait, let's check: are columns 1 and 3 independent?
Row 3: [1, 1, 1]. Columns 1 and 3 are both contributions to this row.
Actually, we need row reduction to be sure. After reduction: rank = 2.Row 2 is 2× Row 1, so it's redundant. The remaining two rows are independent. Rank = 2. The transformation collapses 3D space to a 2D plane.
The most reliable way to compute rank is to row-reduce the matrix to echelon form. The rank equals the number of non-zero rows (equivalently, the number of pivot positions). This is what numerical libraries do internally.
The null space (or kernel) of a matrix $A$ is the set of all vectors that map to zero:
$$\text{Null}(A) = \{\mathbf{x} : A\mathbf{x} = \mathbf{0}\}$$
The nullity is the dimension of this null space:
$$\text{nullity}(A) = \dim(\text{Null}(A))$$
Geometric interpretation:
The null space is the set of input directions that the transformation "ignores" or "annihilates." If you're standing in the null space and the transformation acts on you, you don't move (you map to the origin).
Examples:
Key insight: The null space is always a subspace—it contains the origin, and it's closed under addition and scalar multiplication. If $A\mathbf{u} = \mathbf{0}$ and $A\mathbf{v} = \mathbf{0}$, then $A(\mathbf{u} + \mathbf{v}) = A\mathbf{u} + A\mathbf{v} = \mathbf{0}$. ✓
If $A\mathbf{x} = \mathbf{b}$ has a solution $\mathbf{x}_0$, then $\mathbf{x}_0 + \mathbf{n}$ is also a solution for any $\mathbf{n}$ in the null space (since $A(\mathbf{x}_0 + \mathbf{n}) = A\mathbf{x}_0 + A\mathbf{n} = \mathbf{b} + \mathbf{0} = \mathbf{b}$). A non-trivial null space means infinitely many solutions!
| Nullity | Null Space | Solutions to $A\mathbf{x} = \mathbf{b}$ |
|---|---|---|
| $0$ | Just $\{\mathbf{0}\}$ | At most one solution (unique if it exists) |
| $> 0$ | A subspace of dimension > 0 | Either no solution or infinitely many |
| $= n$ (full dimension) | Entire $\mathbb{R}^n$ | $A = O$ (zero matrix); $\mathbf{b} = \mathbf{0}$ gives all solutions |
The Rank-Nullity Theorem (also called the Fundamental Theorem of Linear Maps) is one of the most important results in linear algebra:
$$\boxed{\text{rank}(A) + \text{nullity}(A) = n}$$
where $n$ is the number of columns of $A$ (the dimension of the domain).
In words: The dimension of what's preserved (rank) plus the dimension of what's destroyed (nullity) equals the dimension of the input space.
Conservation law interpretation:
Dimension can't be created or destroyed, only transferred. Of the $n$ dimensions going in:
Nothing is lost in the accounting—it's a dimensional conservation law.
$A$ is $5 \\times 4$, so domain is $\\mathbb{R}^4$ ($n = 4$).
Given: $\\text{rank}(A) = 3$.By rank-nullity: $\\text{nullity}(A) = 4 - 3 = 1$.
The null space is 1-dimensional (a line through the origin in $\\mathbb{R}^4$).
The column space is 3-dimensional (a 3D hyperplane in $\\mathbb{R}^5$).Four input dimensions → three output dimensions + one collapsed dimension. The transformation projects 4D space onto a 3D subspace, with one direction getting flattened to zero.
A matrix has 'full rank' when rank equals the smaller of its dimensions. For an $m \times n$ matrix: full rank means rank = min($m$, $n$). For square $n \times n$ matrices, full rank (= $n$) implies nullity = 0, which means the matrix is invertible. Rank and invertibility are intimately connected.
Every matrix $A_{m \times n}$ defines four fundamental subspaces that completely characterize its behavior. Understanding these is essential for grasping linear algebra's deepest structure.
| Subspace | Definition | Dimension | Lives In |
|---|---|---|---|
| Column Space $C(A)$ | All $A\mathbf{x}$ for $\mathbf{x} \in \mathbb{R}^n$ | $r$ (rank) | $\mathbb{R}^m$ |
| Null Space $N(A)$ | All $\mathbf{x}$ with $A\mathbf{x} = \mathbf{0}$ | $n - r$ | $\mathbb{R}^n$ |
| Row Space $C(A^T)$ | All $A^T\mathbf{y}$ for $\mathbf{y} \in \mathbb{R}^m$ | $r$ (rank) | $\mathbb{R}^n$ |
| Left Null Space $N(A^T)$ | All $\mathbf{y}$ with $A^T\mathbf{y} = \mathbf{0}$ | $m - r$ | $\mathbb{R}^m$ |
The orthogonality relationships:
These subspaces come in orthogonal pairs:
Why the orthogonality?
If $\mathbf{x}$ is in the null space, then $A\mathbf{x} = \mathbf{0}$. This means $\mathbf{x}$ is orthogonal to every row of $A$ (since row$_i \cdot \mathbf{x} = 0$). The rows of $A$ span the row space, so $\mathbf{x}$ is orthogonal to the entire row space.
In $\mathbb{R}^n$: dim(Row Space) + dim(Null Space) = $r + (n-r) = n$ ✓ In $\mathbb{R}^m$: dim(Column Space) + dim(Left Null Space) = $r + (m-r) = m$ ✓
Orthogonal complements always sum to the ambient dimension.
ML significance:
Rank is the key to understanding when the system $A\mathbf{x} = \mathbf{b}$ has solutions and how many.
The Existence Condition:
The system $A\mathbf{x} = \mathbf{b}$ has a solution iff $\mathbf{b}$ is in the column space of $A$.
This is equivalent to: $\text{rank}([A | \mathbf{b}]) = \text{rank}(A)$
If augmenting $A$ with $\mathbf{b}$ increases the rank, then $\mathbf{b}$ introduces a new independent direction—one the columns of $A$ can't reach.
The Uniqueness Condition:
If a solution exists, it's unique iff the null space is trivial (nullity = 0).
This happens when $A$ has full column rank: $\text{rank}(A) = n$ (number of columns).
| Condition | Solutions | Interpretation |
|---|---|---|
| $\mathbf{b} otin C(A)$ | None | $\mathbf{b}$ unreachable by any linear combination |
| $\mathbf{b} \in C(A)$, nullity = 0 | Exactly one | Unique representation |
| $\mathbf{b} \in C(A)$, nullity > 0 | Infinitely many | Add any null space vector to a particular solution |
For square $n \times n$ matrices: full rank ($r = n$) implies both existence and uniqueness for ANY right-hand side $\mathbf{b}$. This is equivalent to being invertible. For non-square matrices, we must check both conditions separately.
Rank deficiency—when a matrix doesn't have full rank—causes problems throughout machine learning. Understanding when and why it happens helps you diagnose and fix issues.
Multicollinearity (Feature Correlation)
When features are linearly dependent, the design matrix $X$ becomes rank-deficient (or nearly so).
What happens:
Example: $$\text{Feature 1: Height in cm}$$ $$\text{Feature 2: Height in inches} = 0.394 \times \text{Feature 1}$$
These are perfectly linearly dependent. The design matrix has a column that's a scalar multiple of another. Rank is reduced by 1.
Solutions: Remove redundant features, use regularization (ridge regression), apply PCA to decorrelate.
In theory, rank is a clean integer. In practice with floating-point arithmetic, it becomes murky. Let's understand the challenges and solutions.
The numerical challenge:
Consider a matrix with singular values [3.0, 2.0, 0.0000001]. Is the rank 2 or 3?
There's no perfect answer. We must choose a numerical rank based on a tolerance threshold.
123456789101112131415161718192021222324252627282930313233343536373839404142
import numpy as np # Computing numerical rank using SVDdef numerical_rank(A, tol=None): """ Compute numerical rank using SVD. Parameters: ----------- A : array-like Input matrix tol : float, optional Threshold for singular values. Default: max(m,n) * max(singular_values) * machine_epsilon Returns: -------- rank : int Numerical rank of A """ A = np.asarray(A) s = np.linalg.svd(A, compute_uv=False) if tol is None: tol = max(A.shape) * np.max(s) * np.finfo(A.dtype).eps rank = np.sum(s > tol) return rank # ExampleA = np.array([[1, 2, 3], [2, 4, 6], # Linearly dependent (2 × row 1) [1, 1, 1]]) print(f"Rank of A: {numerical_rank(A)}")print(f"Singular values: {np.linalg.svd(A, compute_uv=False)}") # Nearly rank-deficient matrixB = np.array([[1, 2], [2, 4.0001]]) # Almost singularprint(f"Rank of B: {np.linalg.matrix_rank(B)}")print(f"Singular values of B: {np.linalg.svd(B, compute_uv=False)}")The ratio of largest to smallest singular value (condition number) indicates numerical sensitivity. If condition number is $10^{16}$ or larger, the matrix is effectively singular in double precision. Even if mathematically full rank, computations will be unreliable.
Rank and nullity provide deep insight into what a linear transformation preserves and destroys. Here are the essential takeaways:
What's next:
We've seen that rank deficiency causes matrices to be non-invertible. But what if a matrix IS invertible? The next page explores inverse matrices in depth—what they mean geometrically, when they exist, how to compute them, and their critical role in solving linear systems.
You now understand rank and nullity as measures of dimensional preservation and collapse. These aren't abstract concepts—they determine whether your linear regression is stable, whether your covariance matrix is invertible, and whether your optimization problem has a unique solution. Next: the theory and practice of matrix inverses.