Matrices as Linear Transformations

When you first encounter matrices in mathematics, they often appear as rectangular grids of numbers—notation to be manipulated according to seemingly arbitrary rules. You learn to multiply matrices using the 'row-by-column' procedure without ever understanding why that procedure exists or what it means.

This superficial understanding is a barrier to mastering machine learning. In ML, matrices are everywhere: weight matrices in neural networks, covariance matrices in statistics, transformation matrices in data preprocessing, kernel matrices in support vector machines. Without geometric intuition, these are just symbols to shuffle around. With it, deep patterns emerge that make complex algorithms feel natural.

The central insight: A matrix is not a table of numbers. A matrix is a complete description of a linear transformation—a function that takes vectors as input and produces transformed vectors as output, preserving the essential structure of the space.

Let's begin with a mental reset. Forget everything you know about matrix mechanics. Instead, consider the simplest possible question: What does a matrix do?

A matrix A is a function that takes a vector x as input and produces a new vector y as output:

$$\mathbf{y} = A\mathbf{x}$$

That's it. Everything else—all the rules for matrix multiplication, the definitions of determinants and eigenvalues, the entire apparatus of linear algebra—follows from understanding this one idea deeply.

The function analogy:

Just as the function $f(x) = 2x$ takes a number and doubles it, a matrix takes a vector and transforms it according to a specific rule. The difference is that vectors inhabit multi-dimensional space, so the transformation is richer—it can affect direction as well as magnitude.

Consider a 2×2 matrix acting on 2D vectors:

$$A = \begin{bmatrix} 2 & 0 \\ 0 & 3 \end{bmatrix}, \quad \mathbf{x} = \begin{bmatrix} 1 \\ 1 \end{bmatrix}$$

$$A\mathbf{x} = \begin{bmatrix} 2 \cdot 1 + 0 \cdot 1 \\ 0 \cdot 1 + 3 \cdot 1 \end{bmatrix} = \begin{bmatrix} 2 \\ 3 \end{bmatrix}$$

The input vector $(1, 1)$ becomes $(2, 3)$. The matrix stretched the x-component by factor 2 and the y-component by factor 3. This is scaling—one of the fundamental transformation types.

Not every function on vectors is a linear transformation. The term "linear" imposes two strict constraints that together define the essence of what matrices can represent.

A function $T: \mathbb{R}^n \to \mathbb{R}^m$ is a linear transformation if and only if:

1. Additivity (Preservation of Addition):
   $T(\mathbf{u} + \mathbf{v}) = T(\mathbf{u}) + T(\mathbf{v})$

2. Homogeneity (Preservation of Scalar Multiplication):
   $T(c\mathbf{u}) = cT(\mathbf{u})$

These can be combined into a single condition: $$T(c_1\mathbf{u} + c_2\mathbf{v}) = c_1 T(\mathbf{u}) + c_2 T(\mathbf{v})$$

This says: the transformation of a linear combination equals the linear combination of the transformations. In prose: linear transformations preserve the structure of vector spaces.

Examples of linear transformations:

• Rotation around the origin
• Scaling (uniform or non-uniform)
• Reflection across a line through the origin
• Shearing
• Projection onto a subspace

Examples of NON-linear transformations:

• Translation: $T(\mathbf{x}) = \mathbf{x} + \mathbf{b}$ — violates $T(\mathbf{0}) = \mathbf{0}$
• Affine: $T(\mathbf{x}) = A\mathbf{x} + \mathbf{b}$ — linear part plus translation
• Any function involving powers, products of components, etc.

The fundamental theorem (informal):

Every linear transformation between finite-dimensional vector spaces can be represented by a matrix, and every matrix represents a linear transformation. The matrix IS the transformation, encoded in a specific way.

Here's the key insight that connects algebra to geometry: the columns of a matrix tell you where the standard basis vectors go.

In 2D, the standard basis vectors are: $$\mathbf{e}_1 = \begin{bmatrix} 1 \\ 0 \end{bmatrix}, \quad \mathbf{e}_2 = \begin{bmatrix} 0 \\ 1 \end{bmatrix}$$

These are the unit vectors pointing along the x-axis and y-axis respectively.

When you apply a matrix $A$ to these basis vectors: $$A\mathbf{e}_1 = \text{first column of } A$$ $$A\mathbf{e}_2 = \text{second column of } A$$

This is profound. The entire behavior of the transformation is determined by what it does to the basis vectors. Since any vector can be written as a linear combination of basis vectors, and linear transformations preserve linear combinations, knowing what happens to the basis tells you what happens to everything.

Constructing transformation matrices:

This principle works in reverse. To build a matrix for any linear transformation:

1. Determine what the transformation does to each standard basis vector
2. Place those destination vectors as columns of the matrix
3. Done—you have the matrix representation

Example: Reflection across the x-axis

• $\mathbf{e}_1 = (1, 0)$ stays at $(1, 0)$ (on the axis, no change)
• $\mathbf{e}_2 = (0, 1)$ goes to $(0, -1)$ (flipped below the axis)

$$\text{Reflection matrix} = \begin{bmatrix} 1 & 0 \\ 0 & -1 \end{bmatrix}$$

Example: Scaling by factor 3 in x, factor 2 in y

• $\mathbf{e}_1 = (1, 0)$ goes to $(3, 0)$
• $\mathbf{e}_2 = (0, 1)$ goes to $(0, 2)$

$$\text{Scaling matrix} = \begin{bmatrix} 3 & 0 \\ 0 & 2 \end{bmatrix}$$

All 2D linear transformations fall into categories that combine these fundamental types. Understanding each type geometrically makes complex transformations decomposable and intuitive.

Here's where the matrix perspective becomes powerful: composing transformations corresponds to multiplying matrices.

If transformation $A$ is applied first, then transformation $B$, the combined effect is: $$\mathbf{y} = B(A\mathbf{x}) = (BA)\mathbf{x}$$

The matrix $BA$ encapsulates both transformations in one. This is why matrix multiplication is defined the way it is—it's designed to make function composition work correctly.

Order matters:

Matrix multiplication is NOT commutative: $AB eq BA$ in general. This reflects the geometric reality that rotation then scaling differs from scaling then rotation.

The power of composition:

Complex transformations can be decomposed into sequences of simple ones. The Singular Value Decomposition (SVD), which we'll study later, shows that ANY matrix can be written as: $$A = U \Sigma V^T$$

This means: any linear transformation is equivalent to a rotation ($V^T$), followed by a scaling ($\Sigma$), followed by another rotation ($U$). This decomposition is foundational for understanding everything from PCA to matrix compression to the numerics of neural network training.

So far, we've focused on square matrices that transform n-dimensional space to n-dimensional space. But matrices can also change the dimension of vectors.

An $m \times n$ matrix transforms n-dimensional vectors into m-dimensional vectors: $$A: \mathbb{R}^n \to \mathbb{R}^m$$

$$A_{m \times n} \cdot \mathbf{x}{n \times 1} = \mathbf{y}{m \times 1}$$

Dimension reduction example:

A $2 \times 3$ matrix projects 3D vectors onto a 2D plane:

$$A = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \end{bmatrix}, \quad \mathbf{x} = \begin{bmatrix} 3 \\ 4 \\ 5 \end{bmatrix}$$

$$A\mathbf{x} = \begin{bmatrix} 3 \\ 4 \end{bmatrix}$$

The z-component is discarded—this is projection onto the xy-plane.

Dimension increase example:

A $3 \times 2$ matrix lifts 2D vectors into 3D:

$$A = \begin{bmatrix} 1 & 0 \\ 0 & 1 \\ 0 & 0 \end{bmatrix}, \quad \mathbf{x} = \begin{bmatrix} 3 \\ 4 \end{bmatrix}$$

$$A\mathbf{x} = \begin{bmatrix} 3 \\ 4 \\ 0 \end{bmatrix}$$

The 2D vector is embedded in 3D space, lying in the xy-plane.

Two transformations deserve special attention for their role as 'boundary cases' in the space of all linear transformations.

The spectrum between identity and zero:

Think of all possible matrices as sitting on a spectrum. At one extreme is the identity—preserving everything. At the other extreme is the zero matrix—destroying everything. In between are the interesting transformations: rotations, scalings, projections, shears, and their combinations.

Eigendecomposition preview:

We'll later see that eigenvalues tell us where a matrix sits on this spectrum along different directions. Eigenvalue of 1 means that direction is preserved (identity-like). Eigenvalue of 0 means that direction is destroyed (zero-like). Eigenvalues between 0 and 1 mean contraction; greater than 1 mean expansion.