Key Concepts and Terminology

When humans observe the world, we perceive objects through a rich tapestry of sensory input—colors, textures, sounds, context, and meaning accumulated over a lifetime of experience. A doctor glancing at a patient's face sees not just features, but subtle signs of health or illness. An experienced car buyer evaluates not just specifications, but an intuitive sense of value.

Machine learning algorithms have no such intuition. They cannot 'see' in any meaningful sense. They cannot understand. What they can do is process numbers—vast matrices of numerical values, computed with extraordinary speed and precision.

This creates a fundamental challenge: How do we translate the richness of the real world into numbers that machines can process?

The answer lies in two foundational concepts that form the bedrock of every machine learning system: features and labels. These are not mere vocabulary—they represent a profound philosophical stance about how knowledge can be represented, quantified, and learned from data.

A feature (also called an attribute, predictor, input variable, or independent variable) is a measurable property or characteristic of an entity that we believe contains information relevant to the prediction task at hand.

Formally, given an entity $x$ from our domain of interest, a feature is a function $\phi: \mathcal{X} \rightarrow \mathbb{R}$ (or more generally, to some measurable space) that extracts a numerical value from that entity.

The Feature Vector:

When we combine multiple features, we create a feature vector—an ordered collection of feature values that completely describes an entity from the machine learning algorithm's perspective:

$$\mathbf{x} = [x_1, x_2, x_3, \ldots, x_d]^T \in \mathbb{R}^d$$

where $d$ is the dimensionality of the feature space, and each $x_i$ represents the value of the $i$-th feature for this particular entity.

Concrete Example: Predicting House Prices

Consider a house in a real estate dataset. The physical house is a complex object with countless properties. But for our ML model, we might represent it as:

$$\mathbf{x}_{house} = \begin{bmatrix} 2500 & \text{(square footage)} \ 4 & \text{(bedrooms)} \ 2.5 & \text{(bathrooms)} \ 15 & \text{(age in years)} \ 0.25 & \text{(lot size in acres)} \ 3 & \text{(garage capacity)} \ 8.5 & \text{(school district rating)} \end{bmatrix}$$

This 7-dimensional feature vector is all the algorithm 'knows' about the house. Any information not captured in these features is invisible to the model. This is both a limitation and a feature (pun intended)—it forces us to be explicit about what information we believe is predictively relevant.

When we represent entities as feature vectors, we implicitly place them in a feature space—a mathematical space where each dimension corresponds to a feature. This geometric perspective is fundamental to understanding how ML algorithms work.

The Feature Space $\mathcal{X}$:

For a problem with $d$ features, each entity becomes a point in $\mathbb{R}^d$. This transformation is profound:

• A dataset of 1,000 houses becomes 1,000 points in a 7-dimensional space
• A corpus of 10,000 emails becomes 10,000 points in perhaps a 50,000-dimensional space (if using word counts)
• A collection of images becomes points in a space with millions of dimensions (one per pixel per color channel)

Why This Matters:

In this geometric view, similar entities are nearby points, and dissimilar entities are distant points. Most ML algorithms exploit this geometry in some way:

• K-Nearest Neighbors: Literally finds the closest points in feature space
• Support Vector Machines: Finds hyperplanes that separate points by class
• Clustering algorithms: Group nearby points together
• Neural networks: Learn to transform the feature space to make patterns more apparent

Key Insight: Features Define What the Model Can Learn

The feature space is the universe within which your ML algorithm operates. If relevant information isn't encoded in the features, it's as if that information doesn't exist. The algorithm cannot discover patterns that aren't represented in its input.

This is why feature engineering—the art and science of creating informative features—is often more impactful than algorithm selection. A simple model with excellent features will frequently outperform a complex model with poor features.

A label (also called target, outcome, response variable, dependent variable, or ground truth) is the value we want our model to predict. In supervised learning, labels are the 'answers' that the model learns to produce given input features.

Formally, for a supervised learning problem, we have:

• Input: feature vector $\mathbf{x} \in \mathcal{X}$
• Output: label $y \in \mathcal{Y}$

The nature of $\mathcal{Y}$ (the label space) determines the type of learning problem:

Regression ($\mathcal{Y} = \mathbb{R}$ or $\mathbb{R}^k$): Labels are continuous values

• Predicting house prices: $y \in [0, \infty)$
• Forecasting temperature: $y \in \mathbb{R}$
• Estimating age from photos: $y \in [0, 120]$

Classification ($\mathcal{Y}$ is a finite set): Labels are discrete categories

• Binary: $\mathcal{Y} = {0, 1}$ or ${negative, positive}$
• Multi-class: $\mathcal{Y} = {cat, dog, bird, fish, ...}$
• Multi-label: $\mathcal{Y} = \mathcal{P}({label_1, label_2, ..., label_k})$ (power set)

With features and labels defined, we can now formalize the supervised learning setup. This framework underlies everything from simple linear regression to deep neural networks.

The Dataset:

A labeled dataset $\mathcal{D}$ consists of $n$ examples:

$$\mathcal{D} = {(\mathbf{x}^{(1)}, y^{(1)}), (\mathbf{x}^{(2)}, y^{(2)}), \ldots, (\mathbf{x}^{(n)}, y^{(n)})}$$

This is often written in matrix form:

$$\mathbf{X} = \begin{bmatrix} — (\mathbf{x}^{(1)})^T — \ — (\mathbf{x}^{(2)})^T — \ \vdots \ — (\mathbf{x}^{(n)})^T — \end{bmatrix} \in \mathbb{R}^{n \times d}, \quad \mathbf{y} = \begin{bmatrix} y^{(1)} \ y^{(2)} \ \vdots \ y^{(n)} \end{bmatrix} \in \mathbb{R}^n$$

where:

• $n$ = number of examples (samples, observations)
• $d$ = number of features (dimensionality)
• $\mathbf{X}$ = design matrix (each row is a feature vector)
• $\mathbf{y}$ = label vector (corresponding outputs)

The Learning Goal:

Given the dataset $\mathcal{D}$, the goal is to learn a function $f: \mathcal{X} \rightarrow \mathcal{Y}$ that maps feature vectors to labels. We want $f$ to:

1. Fit the training data: $f(\mathbf{x}^{(i)}) \approx y^{(i)}$ for $(\mathbf{x}^{(i)}, y^{(i)}) \in \mathcal{D}$
2. Generalize to new data: $f(\mathbf{x}{new}) \approx y{new}$ for unseen examples

The tension between these goals—fitting training data while generalizing well—is the central challenge of machine learning and leads directly to concepts like overfitting, regularization, and model selection.

Feature engineering is the process of using domain knowledge to create, transform, and select features that make machine learning algorithms work better. It is often the difference between a mediocre model and a highly effective one.

"Coming up with features is difficult, time-consuming, requires expert knowledge. Applied machine learning is basically feature engineering." — Andrew Ng, Stanford

Feature engineering encompasses several activities:

Example: Feature Engineering for Date/Time

Consider a raw timestamp feature: 2024 - 12 - 25 14: 30:00

A naive approach might encode this as a single number (Unix timestamp). But this loses valuable patterns. A skilled feature engineer might extract:

This transformation makes patterns that were implicit in the raw timestamp explicit and learnable.

Understanding the different types of features and their proper handling is essential for effective machine learning. Each feature type requires specific preprocessing and encoding strategies.

The concepts of features and labels apply universally across machine learning applications. Let's examine how they manifest in different domains:

Case Study: Email Spam Detection

Let's trace through a complete example of defining features and labels for spam detection:

The Label:

• Binary classification: y ∈ { spam, not_spam } or { 1, 0 }
• Ground truth comes from human annotation or user actions (reporting, moving to spam)

Feature Engineering Process:

Step 1: Text-Based Features

• Word frequencies (especially words like 'free', 'winner', 'click')
• Character ratios (ALL CAPS percentage, special characters)
• N-gram patterns ('act now', 'limited time offer')

Step 2: Structural Features

• Email length, subject line length
• Number of links, link-to-text ratio
• Presence of attachments, attachment types
• HTML complexity

Step 3: Sender Features

• Sender domain reputation
• SPF/DKIM authentication status
• Past interaction with sender

Step 4: Behavioral Features

• Time of day sent
• Whether sender is in contacts
• Similar emails to other users

The final feature vector might have hundreds of dimensions, each capturing a different signal of spamminess. The model learns which combination of features best predicts the label.

Even experienced practitioners make mistakes when defining features and labels. Here are critical pitfalls to avoid:

We have established the foundational vocabulary for how machines perceive and process data. Let's consolidate the key concepts:

What's Next:

With features and labels understood, we now turn to a critical question: How do we properly evaluate our models? The next page explores training, validation, and test sets—the methodology that ensures our models generalize beyond the data they were trained on.