Vectors as Ordered Lists

Every machine learning algorithm, without exception, operates on vectors. When you feed an image to a neural network, it sees a vector of pixel values. When a recommendation system profiles your preferences, it represents you as a vector in taste-space. When a language model processes your text, it converts words into vectors of numerical embeddings.

Vectors are not merely a mathematical convenience—they are the universal representation through which machines perceive and process information. Understanding vectors deeply is therefore not optional for machine learning practitioners; it is the prerequisite for understanding anything else.

This page begins our journey into linear algebra by establishing what vectors are, how they're represented, and why they provide such a powerful abstraction for computation.

The word "vector" carries different meanings across physics, mathematics, and computer science. Let us establish a precise definition that serves our purposes in machine learning while acknowledging these perspectives.

The algebraic definition:

A vector is an ordered list of numbers. That's it. Deceptively simple, profoundly powerful.

Formally, an n-dimensional vector is an ordered n-tuple of real numbers:

$$\mathbf{v} = (v_1, v_2, \ldots, v_n)$$

where each $v_i \in \mathbb{R}$ is called a component or entry of the vector, and $n$ is the dimension or length of the vector.

The key word is ordered—the sequence matters. The vector $(1, 2, 3)$ is fundamentally different from $(3, 2, 1)$ or $(2, 1, 3)$, even though they contain the same numbers.

Multiple perspectives on vectors:

While we'll primarily work with the algebraic view (vectors as ordered lists), it helps to understand other perspectives that provide complementary intuition:

All three perspectives describe the same mathematical objects—they simply emphasize different aspects. For machine learning, we move fluidly between them:

• Algebraic when implementing algorithms and processing data
• Geometric when developing intuition and visualizing transformations
• Abstract when understanding why certain properties hold or proving algorithm correctness

Developing comfort with all three perspectives is essential for mastering machine learning mathematics.

Mathematical notation for vectors varies across textbooks, papers, and programming contexts. Familiarity with all conventions is essential for reading machine learning literature fluently.

Common notational conventions:

Column vectors vs. row vectors:

A critical distinction in linear algebra is between column vectors and row vectors. By convention in machine learning, vectors are typically column vectors unless stated otherwise:

$$\mathbf{v} = \begin{bmatrix} v_1 \ v_2 \ \vdots \ v_n \end{bmatrix}$$

A row vector is the transpose of a column vector:

$$\mathbf{v}^\top = \begin{bmatrix} v_1 & v_2 & \cdots & v_n \end{bmatrix}$$

This distinction matters crucially when we multiply vectors and matrices. The transpose operation (denoted $\top$ or sometimes $T$) converts between them.

Component indexing:

Vector components are accessed by their index. Different conventions exist:

• 1-indexed (mathematics): Components are $v_1, v_2, \ldots, v_n$
• 0-indexed (most programming): Components are $v_0, v_1, \ldots, v_{n-1}$

Mathematical notation uses 1-indexing, while Python and most ML frameworks use 0-indexing. Being comfortable converting between them is essential.

Notation for common vector spaces:

• $\mathbb{R}^n$ — The space of all n-dimensional real vectors
• $\mathbb{R}^2$ — The 2D plane (pairs of real numbers)
• $\mathbb{R}^3$ — 3D space (triples of real numbers)
• $\mathbb{R}^{784}$ — 784-dimensional space (e.g., flattened 28×28 MNIST images)
• $\mathbb{R}^d$ — d-dimensional space (when dimension is a variable)

While vectors are algebraically just lists of numbers, their geometric interpretation provides crucial intuition. This intuition, developed in 2D and 3D, extends naturally (if invisibly) to higher dimensions.

Vectors as arrows:

Geometrically, a vector $\mathbf{v} = (v_1, v_2)$ in $\mathbb{R}^2$ represents an arrow from the origin $(0, 0)$ to the point $(v_1, v_2)$. The vector has two properties:

1. Magnitude (length): How long the arrow is
2. Direction: Which way the arrow points

Critically, what matters is the displacement—the relative movement—not the absolute starting position. The vector $(3, 2)$ represents "move 3 units right and 2 units up" regardless of where you start.

Visualizing in 2D and 3D:

Consider the vector $\mathbf{v} = (3, 2)$ in $\mathbb{R}^2$:

• It points from the origin to the point $(3, 2)$
• Its horizontal component is 3 (rightward)
• Its vertical component is 2 (upward)
• It lies in the first quadrant at an angle of $\arctan(2/3) \approx 33.7°$ from the positive x-axis

In $\mathbb{R}^3$, the vector $\mathbf{v} = (3, 2, 1)$ has an additional z-component:

• It extends 3 units along x, 2 units along y, and 1 unit along z
• It points to a location in 3D space

Beyond 3D:

In machine learning, we routinely work with vectors in $\mathbb{R}^{1000}$ or $\mathbb{R}^{1,000,000}$. We cannot visualize these spaces, but the algebraic operations and properties carry over identically. This is the power of abstraction: intuition developed in 2D and 3D guides reasoning in any dimension.

The magnitude (length) of a vector:

The magnitude or norm of a vector, denoted $|\mathbf{v}|$, is its length calculated via the Pythagorean theorem:

$$|\mathbf{v}| = \sqrt{v_1^2 + v_2^2 + \cdots + v_n^2} = \sqrt{\sum_{i=1}^{n} v_i^2}$$

This is called the Euclidean norm or L2 norm (we'll explore other norms later).

Unit vectors:

A unit vector has magnitude exactly 1. Any non-zero vector can be normalized to a unit vector by dividing by its magnitude:

$$\hat{\mathbf{v}} = \frac{\mathbf{v}}{|\mathbf{v}|}$$

Unit vectors are fundamental for representing pure directions without magnitude information.

Certain vectors appear so frequently in linear algebra and machine learning that they have standard names and notation.

The zero vector:

The zero vector $\mathbf{0}$ has all components equal to zero:

$$\mathbf{0} = \begin{bmatrix} 0 \ 0 \ \vdots \ 0 \end{bmatrix}$$

The zero vector is the additive identity—adding it to any vector leaves that vector unchanged: $$\mathbf{v} + \mathbf{0} = \mathbf{v}$$

Geometrically, the zero vector has no direction (or, equivalently, points in every direction) and has zero magnitude.

Standard basis vectors (canonical vectors):

The standard basis vectors or canonical vectors are unit vectors pointing along each coordinate axis. In $\mathbb{R}^n$, the $i$-th standard basis vector $\mathbf{e}_i$ has a 1 in position $i$ and 0 everywhere else:

$$\mathbf{e}_1 = \begin{bmatrix} 1 \ 0 \ 0 \ \vdots \ 0 \end{bmatrix}, \quad \mathbf{e}_2 = \begin{bmatrix} 0 \ 1 \ 0 \ \vdots \ 0 \end{bmatrix}, \quad \ldots \quad \mathbf{e}_n = \begin{bmatrix} 0 \ 0 \ 0 \ \vdots \ 1 \end{bmatrix}$$

The ones vector:

The ones vector $\mathbf{1}$ (or $\mathbf{1}_n$) has all components equal to 1:

$$\mathbf{1} = \begin{bmatrix} 1 \ 1 \ \vdots \ 1 \end{bmatrix}$$

This vector appears frequently in statistics (for computing sums and means) and machine learning (as a bias term in linear models).

Notation summary:

The true power of vectors emerges when we recognize that almost any data can be encoded as a vector. This encoding is the bridge between the real world and machine learning algorithms.

The vectorization principle:

Machine learning requires numerical input. Since vectors are ordered collections of numbers, converting data to vectors (called vectorization or feature extraction) is the essential first step in any ML pipeline.

Different data types require different vectorization strategies:

Example: Image as a vector

Consider a tiny 3×3 grayscale image where each pixel has an intensity from 0 (black) to 255 (white):

[100, 150, 200]
[50,  255,  50]
[0,   100, 255]

Flattening row by row creates a 9-dimensional vector: $$\mathbf{x} = (100, 150, 200, 50, 255, 50, 0, 100, 255)$$

This vector lives in $\mathbb{R}^9$. Every possible 3×3 grayscale image corresponds to exactly one point in this 9-dimensional space.

Example: User profile as a vector

A streaming service might represent a user's preferences as: $$\mathbf{u} = (\text{action}, \text{comedy}, \text{drama}, \text{horror}, \text{scifi})$$

For example, $(0.8, 0.3, 0.5, 0.1, 0.9)$ indicates strong preference for action and sci-fi, moderate preference for drama, and low preference for horror.

Why this representation matters:

Once data is vectorized:

1. Distance between vectors measures similarity
2. Directions in vector space encode semantic relationships
3. Linear algebra operations enable efficient computation
4. ML algorithms can process any data uniformly

Understanding when two vectors are equal—and how to compare vectors—is fundamental.

Vector equality:

Two vectors $\mathbf{u}$ and $\mathbf{v}$ are equal if and only if:

1. They have the same dimension
2. Every corresponding component is equal

Formally: $\mathbf{u} = \mathbf{v} \iff u_i = v_i \text{ for all } i$

This means:

• $(1, 2, 3) eq (1, 2)$ (different dimensions)
• $(1, 2, 3) eq (1, 3, 2)$ (different ordering)
• $(1, 2, 3) = (1, 2, 3)$ (exactly equal)

Approximate equality (numerical computing):

In practice, floating-point computations introduce small errors. We rarely check exact equality; instead, we check if vectors are close enough:

$$|\mathbf{u} - \mathbf{v}| < \epsilon$$

where $\epsilon$ is a small tolerance (e.g., $10^{-10}$).

Comparing vectors:

Vectors themselves are not inherently ordered—there's no universal "less than" relationship. The vector $(1, 3)$ is neither less than nor greater than $(2, 2)$.

However, we can compare vectors by:

• Magnitude: $|\mathbf{u}| < |\mathbf{v}|$ (which is longer?)
• Component-wise: $\mathbf{u} \leq \mathbf{v}$ if $u_i \leq v_i$ for all $i$ (partial ordering)
• Dot product: Compare projections (we'll cover this soon)

Before we proceed to vector operations (next page), let's establish the fundamental properties that make vectors so useful mathematically.

Vectors form a mathematical structure:

The set of all n-dimensional real vectors, $\mathbb{R}^n$, together with addition and scalar multiplication (which we'll define next page), forms a vector space. This structure guarantees certain properties that enable all of linear algebra.

Key properties (preview):

We'll explore these axioms rigorously when we cover vector spaces formally. For now, recognize that these properties are not arbitrary—they capture the essential structure that makes vectors useful for representing and manipulating data.