Foundation Models

In the history of machine learning, no single factor has been more transformative than scale. The models that power today's most impressive AI capabilities—ChatGPT, Claude, GPT-4, Gemini, LLaMA—are not fundamentally different architectures from their predecessors. They are the same architectures, but at a scale that seemed computationally infeasible just a decade ago.

This page explores the profound implications of scale in machine learning. We will examine how scale has transformed from a mere engineering consideration into a fundamental research paradigm, one that has upended decades of conventional wisdom about how to improve AI systems. The story of modern AI is, in many ways, the story of scale—and understanding this story is essential for any practitioner seeking to work with or understand foundation models.

To appreciate the significance of scale in modern ML, we must understand the paradigm it replaced. For most of machine learning's history, progress came primarily through algorithmic innovation—developing better models, better features, and better optimization techniques.

The Traditional ML Paradigm (1950s-2010s):

For decades, the dominant approach to improving ML systems followed a predictable pattern:

1. Careful Feature Engineering: Domain experts would spend months or years crafting features that captured the essential structure of problems. Computer vision practitioners developed SIFT, HOG, and SURF features. NLP researchers built syntactic parsers and hand-crafted lexicons.

2. Clever Architectural Innovations: Researchers focused on developing objectively 'better' models—support vector machines over perceptrons, random forests over single decision trees, deep networks over shallow ones.

3. Sophisticated Optimization: Training algorithms became increasingly sophisticated—from gradient descent to momentum, RMSprop, Adam, and beyond.

In this paradigm, data and compute were secondary concerns. The prevailing wisdom held that more data helped, but with diminishing returns. Extra compute could speed up experiments but wouldn't fundamentally change what was possible.

The Turning Point: Deep Learning at Scale (2012-2017)

The shift began with AlexNet in 2012, which demonstrated that deep neural networks trained on large datasets (ImageNet's 1.2 million images) using GPUs could dramatically outperform hand-engineered approaches. But AlexNet was still modest by modern standards—60 million parameters, trained on a few GPUs.

Over the next five years, a growing body of evidence suggested that scale followed predictable laws. Researchers at OpenAI, Google, and elsewhere noticed that:

• Larger models consistently performed better on the same tasks
• More training data consistently led to improvements
• More compute consistently translated to better results
• These improvements were smooth and predictable, not subject to diminishing returns as rapidly as previously thought

This observation—that you could buy better performance by simply scaling up—upended the traditional paradigm. If more compute reliably means better results, then the bottleneck shifts from algorithm design to compute access and efficient training.

The most remarkable discovery of the scaling era is that improvements from scale are not arbitrary—they follow precise mathematical laws. These scaling laws allow researchers to predict how much performance will improve from additional compute, data, or parameters, enabling rational planning of massive training runs.

The Power Law Relationship:

Scaling laws typically take the form of power laws:

$$L(X) = \left(\frac{X_c}{X}\right)^{\alpha_X}$$

where:

• $L$ is the loss (lower is better)
• $X$ is the scaling variable (compute, data, or parameters)
• $X_c$ is a constant
• $\alpha_X$ is the scaling exponent

This relationship means that each order of magnitude increase in scale yields a consistent, predictable decrease in loss. The exponent $\alpha$ determines how quickly performance improves with scale.

The Chinchilla Scaling Laws:

A pivotal contribution came from DeepMind's 2022 'Chinchilla' paper, which refined our understanding of optimal scaling. The key insight was that previous models were undertrained relative to their size.

The Chinchilla team proposed that compute-optimal training requires balancing model size and training data according to:

$$N_{opt} \propto C^{0.5}, \quad D_{opt} \propto C^{0.5}$$

where $N$ is parameters, $D$ is training tokens, and $C$ is compute. This means:

• Parameters and data should scale equally with compute
• For every 2× increase in compute, increase both model size and data by √2 ≈ 1.41×
• GPT-3 (175B parameters, 300B tokens) was significantly undertrained
• A 'Chinchilla-optimal' 175B model would need ~3.5T tokens

Abstract discussions of scale become concrete when we consider the actual computational requirements of training modern foundation models. The numbers are staggering and have profound implications for who can participate in frontier ML research.

Measuring Compute: FLOPs and GPU-hours

We measure training compute in FLOPs (floating point operations)—specifically, total FLOPs across the entire training run. The relationship between model parameters, training tokens, and compute follows the approximation:

$$C \approx 6ND$$

where:

• $C$ = total FLOPs
• $N$ = model parameters
• $D$ = training tokens
• The factor 6 comes from forward + backward passes through the model

This allows us to estimate compute for any training configuration. For example, training GPT-3 (175B parameters, 300B tokens):

$$C = 6 \times 175 \times 10^9 \times 300 \times 10^9 = 3.15 \times 10^{23} \text{ FLOPs}$$

The Hardware Landscape:

Frontier model training runs on specialized hardware. The dominant platform is NVIDIA's A100 and H100 GPUs, organized into massive clusters:

Example: Training LLaMA 2 70B

Meta reported training LLaMA 2 70B used:

• 2,000 A100 (80GB) GPUs
• ~1.7 million GPU-hours
• ~35 days of continuous training
• Estimated cost: $5-10 million in compute alone

This excludes:

• Engineering salaries (team of 50+ for months)
• Failed experiments and hyperparameter tuning
• Data collection and preprocessing
• Evaluation and red-teaming
• Infrastructure and electricity costs

Scale isn't just about compute and parameters—data is equally fundamental. The shift to web-scale training data represents a transformation as significant as the architectural innovations. Understanding data scaling is crucial for anyone working with foundation models.

The Data Scaling Journey:

The progression of training datasets tells the story of scale:

The Data Quality-Quantity Tradeoff:

Scaling data is not simply 'more is better.' Research has shown that data quality interacts nonlinearly with scale:

1. Deduplication Matters Enormously: Training on duplicated data leads to memorization rather than generalization. Near-duplicate detection at scale requires sophisticated MinHash and locality-sensitive hashing techniques.

2. Quality Filtering Improves Efficiency: Filtering low-quality data (spam, boilerplate, machine-generated text) can improve performance more than adding equivalent amounts of unfiltered data.

3. Domain Mixing Requires Care: The ratio of data from different domains (code, books, web, academic papers) significantly affects downstream capabilities.

4. Repetition Has Costs: Training on the same data multiple epochs provides diminishing returns and can harm generalization.

The empirical success of scaling is undeniable, but why does it work so well? This remains an active area of research. We present several theoretical perspectives that help explain the scaling phenomenon.

Perspective 1: Capacity and Task Coverage

Language modeling requires representing a vast space of knowledge and skills. A more capacious model can:

• Store more factual associations
• Represent more complex patterns and relationships
• Capture more linguistic structures and stylistic variations
• Encode more problem-solving strategies

Under this view, scaling provides the raw capacity to represent the full complexity of language and knowledge. Smaller models must make compromises—forgetting rare information, simplifying patterns, or failing to capture subtle distinctions.

Perspective 2: Feature Learning and Representation Quality

Neural networks learn hierarchical representations. Empirically, larger models learn:

• More disentangled features
• More transferable representations
• Better separation of semantic concepts
• Higher-fidelity internal world models

The quality of learned representations improves smoothly with scale, enabling better generalization to new tasks.

Perspective 3: The Manifold Hypothesis

High-dimensional data often lies on lower-dimensional manifolds. Language—despite its apparent complexity—has significant structure. Larger models can:

• Better approximate these complex manifolds
• Reduce approximation error in high-curvature regions
• Capture long-range dependencies that smaller models miss
• Represent multiple interconnected manifolds (different skills/knowledge areas)

Perspective 4: Compression and Generalization

Kolmogorov complexity theory suggests that learning is compression—finding short programs that generate training data. Larger models with more training compute can:

• Find better compressions of the training data
• Discover more efficient representations
• Identify deeper regularities that enable generalization
• Better approximate the true data-generating process

Perspective 5: Phase Transitions and Emergent Structure

Some capabilities appear to 'emerge' suddenly at specific scales, suggesting phase transitions in the model's internal structure. These may represent:

• Qualitative changes in how information is organized
• The activation of dormant computational circuits
• Threshold effects where multiple sub-capabilities combine
• Transitions in the geometry of the loss landscape

The scaling paradigm does not mean that efficiency is irrelevant—quite the opposite. Given the enormous costs of frontier training, efficiency gains are multiplicative. A 2× efficiency improvement saves millions of dollars. This section surveys key approaches to efficient scaling.

Architectural Efficiency:

Mixture of Experts (MoE):

MoE models activate only a subset of parameters for each input, dramatically reducing compute while maintaining (or exceeding) the effective capacity of dense models:

• GPT-4 is widely believed to use MoE with 8 experts
• Mixtral 8x7B activates only 2 of 8 experts per token
• Each forward pass uses ~12B parameters despite 46B total
• Training efficiency improves by 2-4× for equal performance

While scale has delivered remarkable progress, it is not a panacea. Critical questions remain about the limits and limitations of the scaling paradigm.

Question 1: Do Scaling Laws Eventually Break?

Current scaling laws are empirical observations, not theoretical guarantees. They may:

• Eventually flatten (diminishing returns kick in)
• Hit phase transitions (qualitative changes in behavior)
• Face data bottlenecks (running out of quality training data)
• Encounter irreducible loss floors

Some researchers observe that loss improvements with scale are slowing on certain benchmarks, though interpretations differ.

Question 2: Does Lower Loss Mean Better Capabilities?

Scaling laws predict loss (perplexity), not downstream task performance. The relationship between:

• Cross-entropy loss and task accuracy is imperfect
• Perplexity and 'understanding' is unclear
• Training metrics and safety is unknown

Models with similar losses can have very different capabilities and failure modes.

Question 3: What Can't Scale Solve?

Some challenges may require more than scale:

• Reliable factual accuracy: Models still hallucinate despite scale
• Precise multi-step reasoning: Complex logical chains remain fragile
• Robust agent behavior: Autonomous action requires more than pattern matching
• Genuine understanding: Whether scale leads to 'understanding' is philosophically contested

Question 4: Environmental and Economic Sustainability

Training costs are growing faster than efficiency improvements:

• GPT-4 training reportedly cost $100M+
• Next generation may cost $1B+
• Carbon footprint of frontier training is substantial
• Only a handful of organizations can afford frontier research

This raises questions about sustainable scaling and equitable access.

Question 5: What Comes After Scale?

Scale may be necessary but not sufficient. Emerging research directions include:

• Better architectures (post-transformer designs)
• Improved data quality and curation
• Test-time compute scaling (inference scaling)
• Neurosymbolic integration
• Self-improvement and recursive capability gain

We have explored the role of scale in modern machine learning—from historical context through mathematical foundations to practical implications. Let's consolidate the key insights:

What's Next:

Now that we understand the role of scale, we'll explore its most surprising consequence: emergent capabilities. In the next page, we examine how scaling sometimes produces capabilities that were not present at smaller scales—behaviors that 'emerge' abruptly and unpredictably, challenging our understanding of what these models actually learn.