Large Language Models

How does GPT-4 know that water is wet, that Paris is in France, that print('Hello') outputs text, and that stories should have coherent plots? It was never explicitly taught these facts. Instead, it learned them through a remarkably simple objective: predict the next token.

Pre-training objectives are the loss functions that shape what language models learn during the initial, compute-intensive phase of training on massive text corpora. These objectives seem almost absurdly simple—predict missing words, guess the next token—yet they produce models with stunning breadth and depth of knowledge. Understanding pre-training objectives reveals how neural networks extract and organize the latent structure of human knowledge from raw text.

Modern language models follow a two-stage training paradigm:

1. Pre-training: Self-supervised learning on massive unlabeled text corpora
2. Post-training: Supervised fine-tuning for specific tasks or alignment objectives

Pre-training is where the majority of compute is spent and where models acquire their general capabilities. The objectives used during this phase determine the fundamental nature of the resulting model.

Why Self-Supervised Learning?

The key insight enabling modern LLMs is that text itself provides supervision. Unlike traditional supervised learning requiring expensive human labels, we can construct training signals directly from text:

• Given "The capital of France is ___", predict "Paris"
• Given "She picked up the [MASK] and began to read", predict "book"
• Given corrupted text, reconstruct the original

This enables training on virtually unlimited data—the entire internet becomes a training set. But not all self-supervised objectives are equal.

The Surprising Power of Prediction

Why does predicting the next token produce capable models? The answer lies in what prediction requires:

To predict the next token accurately, a model must implicitly learn:

• Grammar and syntax (to predict grammatical continuations)
• Semantics (to predict contextually appropriate words)
• World knowledge (to predict factually correct statements)
• Reasoning (to predict logical conclusions)
• Style and register (to maintain consistent voice)
• Format and structure (to complete patterns)

The prediction objective is a proxy for understanding. A model that predicts perfectly must understand deeply. This is why simple objectives produce such capable models—the objective is simple, but doing it well is not.

Autoregressive language modeling (AR-LM) is the dominant pre-training objective for modern LLMs, powering GPT, LLaMA, Claude, and most frontier models. The objective is elegantly simple: given a sequence of tokens, predict the next one.

Mathematical Formulation

Given a sequence of tokens $x = (x_1, x_2, ..., x_n)$, the autoregressive model factorizes the joint probability as:

$$P(x) = \prod_{i=1}^{n} P(x_i | x_1, x_2, ..., x_{i-1})$$

The training objective minimizes the negative log-likelihood:

$$\mathcal{L}{AR} = -\sum{i=1}^{n} \log P(x_i | x_{<i}; \theta)$$

In practice, this is implemented as cross-entropy loss between the model's predicted distribution and the one-hot encoded true next token.

The Causal Attention Mask

Autoregressive models use causal attention to ensure each position can only attend to previous positions:

Attention Pattern (Causal Mask):

        t=1  t=2  t=3  t=4  t=5
t=1     ✓    ✗    ✗    ✗    ✗
t=2     ✓    ✓    ✗    ✗    ✗
t=3     ✓    ✓    ✓    ✗    ✗
t=4     ✓    ✓    ✓    ✓    ✗
t=5     ✓    ✓    ✓    ✓    ✓

This triangular mask ensures the model cannot "cheat" by looking at future tokens during training, maintaining consistency between training and generation.

Why Autoregressive Models Dominate

Several properties make AR-LM particularly effective:

Training Dynamics

Autoregressive loss exhibits characteristic training dynamics:

Early training (perplexity > 50):

• Model learns basic tokenization patterns and word frequencies
• Common words and syntax are prioritized
• Loss decreases rapidly

Mid training (perplexity 10-50):

• Model learns semantic relationships and context dependence
• Factual knowledge begins accumulating
• Grammatical fluency emerges

Late training (perplexity < 10):

• Fine-grained distinctions and rare patterns
• Complex reasoning and long-range coherence
• Loss decreases slowly, capability gains continue

Notably, perplexity and capability don't perfectly correlate. A model might stall in perplexity but continue gaining reasoning ability. Evaluation must go beyond the training metric.

Masked Language Modeling (MLM), introduced by BERT (Bidirectional Encoder Representations from Transformers), takes a different approach: randomly mask tokens and train the model to predict them using bidirectional context.

Mathematical Formulation

Given a sequence $x = (x_1, ..., x_n)$, randomly select positions $M$ to mask. The objective is:

$$\mathcal{L}{MLM} = -\sum{i \in M} \log P(x_i | x_{\backslash M}; \theta)$$

Where $x_{\backslash M}$ denotes the sequence with masked positions replaced by a special [MASK] token or corrupted in some way.

The BERT Masking Strategy

BERT's original procedure:

1. Select 15% of tokens uniformly at random
2. For selected tokens:
   • 80%: Replace with [MASK]
   • 10%: Replace with a random token
   • 10%: Keep unchanged
3. Predict original token at masked positions

The 80-10-10 split addresses a key issue: if we only used [MASK], the model would never see this token at inference time (creating train-test mismatch).

MLM Variants and Extensions

RoBERTa (Robustly Optimized BERT):

• Removes next sentence prediction (NSP) task
• Dynamic masking (different masks each epoch)
• Larger batches, more data
• Result: Significant improvements with same architecture

SpanBERT:

• Masks contiguous spans rather than random tokens
• Span lengths sampled from geometric distribution
• Better for tasks requiring span understanding

ALBERT:

• Factorized embedding parameters
• Cross-layer parameter sharing
• Sentence order prediction instead of NSP
• 18× fewer parameters, competitive performance

ELECTRA:

• Train a generator to corrupt tokens, discriminator to detect
• All tokens provide signal (not just 15% masked)
• More sample-efficient than MLM

When to Use MLM vs AR

A third category of pre-training objectives frames language modeling as a denoising or sequence-to-sequence task. Given corrupted input, the model must produce the original (or a corrected/completed) output. This family includes T5, BART, and UL2.

T5: Text-to-Text Transfer Transformer

T5 unifies all NLP tasks as text-to-text problems with a span corruption objective:

Span Corruption Process:

1. Randomly select spans of tokens to corrupt (15% of tokens, mean span length 3)
2. Replace each span with a unique sentinel token (<extra_id_0>, <extra_id_1>, etc.)
3. Target: produce only the corrupted spans prefixed with their sentinel

Example:

Input:  "The [X] sat on the [Y] and meowed loudly."
Target: "[X] cat [Y] mat"

This is more efficient than BERT's approach: the target sequence is much shorter than the input (only corrupted portions), speeding up training.

BART: Denoising Autoencoder

BART uses a more flexible noise approach with an autoregressive decoder:

Noise Types in BART:

BART's decoder is fully autoregressive, generating the complete original sequence. This makes it naturally suited for generation tasks like summarization.

UL2: Unified Language Learner

UL2 proposes mixing multiple objectives during pre-training:

UL2 Mixture:

• R-Denoiser (Regular): Short spans, like T5
• S-Denoiser (Sequential): Long prefix, complete the rest (like AR-LM)
• X-Denoiser (Extreme): Very long spans, high corruption

Each sample is prefixed with a mode token ([R], [S], [X]) indicating the denoising mode. The model learns to handle all modes, gaining benefits of each.

Key insight: Different objectives are complementary. Span infilling improves bidirectional understanding; sequential denoising improves generation. Combining them produces a more capable model than either alone.

The pre-training objective is only half the story. Equally critical is the data on which that objective is optimized. Data quality, diversity, and composition fundamentally shape what models learn.

Data Sources and Their Properties

Data Quality Filtering

Raw web crawl data is not directly usable. Quality filtering is essential:

1. Deduplication

• Exact deduplication: Remove identical documents
• Near-duplicate removal: MinHash/LSH for fuzzy matching
• n-gram deduplication: Remove documents sharing many n-grams

Why it matters: Duplicated data causes memorization, wasted compute, and benchmark contamination. Models can recite training data verbatim if it appears frequently enough.

2. Quality Scoring

• Rule-based filters: Length, punctuation ratio, word list presence
• Classifier-based: Train classifier on Wikipedia+ vs. low-quality web
• Perplexity filtering: Remove text that is too perplexing (noise) or not perplexing enough (repetitive)

3. Content Filtering

• PII removal: Names, addresses, phone numbers
• Toxicity filtering: Hate speech, explicit content
• Benchmark decontamination: Remove test set instances

Data Mixture and Curriculum

Frontier models train on carefully balanced mixtures:

Typical LLM Data Mixture:

├── Web (filtered): 50-60%
│   └── High-quality web text, deduplicated
│
├── Code: 15-20%
│   └── GitHub, code documentation
│
├── Books: 10-15%
│   └── Fiction and non-fiction
│
├── Academic: 5-10%
│   └── arXiv, PubMed, patents
│
├── Wikipedia: 2-5%
│   └── Upsampled for factual grounding
│
├── Curated dialogue: 2-5%
│   └── Reddit, forums, Q&A sites
│
└── Math/Reasoning: 2-5%
    └── Textbooks, proofs, problem sets

Key insights:

• Higher-quality data often upsampled (Wikipedia may be seen 10×+)
• Code data dramatically improves reasoning capabilities
• Mixture affects emergent capabilities (more math data → better at math)
• The optimal mixture is not known a priori—requires ablation studies

Converting the theoretical objective into a working training pipeline requires careful attention to optimization, stability, and efficiency. Here we cover the key implementation considerations.

Optimizer Configuration

Most LLMs use AdamW with carefully tuned hyperparameters:

optimizer_config = {
    "optimizer": "AdamW",
    "learning_rate": 3e-4,      # Peak LR, lower for larger models
    "beta1": 0.9,                # Momentum term
    "beta2": 0.95,               # Second moment (lower than default 0.999)
    "epsilon": 1e-8,
    "weight_decay": 0.1,         # L2 regularization
    "grad_clip": 1.0,            # Gradient clipping norm
}

lr_schedule = {
    "type": "cosine_with_warmup",
    "warmup_steps": 2000,        # Linear warmup
    "min_lr_ratio": 0.1,         # 10× decay from peak
}

Critical hyperparameter scaling:

Rule of thumb: Larger models need lower learning rates and larger batches for stability.

Mixed Precision Training

Modern training uses multiple precision levels:

BFloat16 (BF16) is now standard for LLM training:

• Same exponent range as FP32 (no overflow issues)
• 3-bit mantissa reduction acceptable for deep learning
• Supports tensor cores for accelerated matrix multiply

Loss scaling is required with FP16 to prevent underflow in gradients:

# Automatic mixed precision with loss scaling
scaler = torch.cuda.amp.GradScaler()

with torch.cuda.amp.autocast(dtype=torch.bfloat16):
    loss = model(batch).loss
    
scaler.scale(loss).backward()
scaler.step(optimizer)
scaler.update()

Checkpoint and Evaluation Strategy

checkpoint_config = {
    # Checkpointing frequency
    "save_interval": 500,        # Steps between checkpoints
    "keep_last_n": 10,           # Rolling window
    "keep_milestones": [10000, 50000, 100000],  # Permanent saves
    
    # Evaluation schedule
    "eval_interval": 1000,
    "eval_datasets": [
        "validation_set",         # Held-out distribution
        "mmlu_subset",            # Capability tracking
        "hellaswag_subset",
    ],
    
    # Logging
    "log_interval": 10,          # Steps between loss logging
    "log_grad_norm": True,       # Monitor gradient health
    "log_learning_rate": True,
}

Evaluation during pre-training:

While validation loss is the primary metric, tracking downstream task performance reveals capability emergence. A common approach:

1. Evaluate on standardized benchmarks (MMLU, HellaSwag) every 1K-10K steps
2. Monitor for sudden capability gains (emergence)
3. Track multiple capabilities to detect regression
4. Use evaluation results to inform mixture adjustments

Pre-training objectives and data form the foundation upon which all LLM capabilities rest. The remarkable abilities of modern language models emerge from simple prediction tasks applied at massive scale to diverse data.

What's next:

Pre-training produces models with broad capabilities but no particular alignment to user intent. The next page covers instruction tuning—the process of teaching models to follow instructions, answer questions helpfully, and behave as intended. We'll see how supervised fine-tuning transforms a base language model into an assistant.