A base language model trained purely on next-token prediction is a powerful engine, but it behaves like an autocomplete system: give it "The capital of France is" and it will output "Paris", but ask it "What is the capital of France?" and it may continue with "A question for today's geography lesson..." rather than answering.

This is the instruction-following gap. Base models learn to predict text, but not to follow instructions. They lack the alignment between user intent and model behavior that makes an LLM useful as an assistant, coding tool, or reasoning engine.

Instruction tuning bridges this gap. Through supervised fine-tuning on instruction-response pairs, we teach models to recognize instructions and respond appropriately. This seemingly simple step—showing the model examples of helpful responses—produces the dramatic behavioral shift that transforms a language model into ChatGPT.

Base language models are trained to model the distribution of text on the internet. This creates a fundamental misalignment with how we want to use them:

What Base Models Learn

Given the prompt "Write a poem about autumn", a base model predicts what typically follows such text in its training data. This might be:

• An actual poem (if it was trained on poetry books)
• An essay about poetry (if trained on writing guides)
• Instructions on how to write poetry (if trained on educational content)
• A continuation of a conversation ("Sure, here's a poem...")
• Completely unrelated text that happens to be statistically likely

The base model doesn't "understand" that you want a poem—it predicts tokens. Sometimes prediction and instruction-following coincide; often they don't.

The Surprising Efficiency of Instruction Tuning

Given the dramatic behavioral change, one might expect instruction tuning to require retraining from scratch. Remarkably, the opposite is true:

Key insight: Instruction tuning doesn't teach new knowledge—it teaches the model to access and apply knowledge it already has. The capability was latent in the base model; instruction tuning unlocks it.

This efficiency has profound implications:

1. Accessibility: Anyone with a base model can create their own instruction-tuned variant
2. Customization: Different instruction sets produce different assistant behaviors
3. Iteration: Rapid experimentation with alignment approaches becomes feasible
4. Specialization: Domain-specific instruction tuning creates expert assistants

The quality and diversity of instruction data fundamentally determines the resulting model's capabilities. Creating effective instruction datasets is both art and science.

Anatomy of an Instruction Example

Instruction examples typically follow a structured format:

{
    "instruction": "Write a Python function that calculates factorial",
    "input": "(optional) Additional context or parameters",
    "output": "def factorial(n):
    if n <= 1:
        return 1
    return n * factorial(n - 1)"
}

The model learns to map (instruction, input) pairs to appropriate outputs. During inference, users provide the instruction/input, and the model generates the output.

Data Collection Strategies

1. Human-Written Data

The highest quality but most expensive approach. Human annotators create both instructions and responses.

Pros: High quality, nuanced, authentic human intent Cons: Expensive ($10-50 per high-quality example), slow to scale

2. Template-Based Generation

Define templates and fill with variations:

Templates:
- "Explain {concept} as if I were {age_group}"
- "Compare and contrast {A} and {B}"
- "Write a {length} {genre} story about {topic}"

Pros: Scalable, controllable diversity Cons: May lack naturalness, template artifacts

3. Model-Generated Data (Self-Instruct)

Use an existing LLM to generate instruction-response pairs:

1. Seed with small set of human examples
2. Prompt model to generate new instructions
3. Filter for quality and diversity
4. Generate responses to new instructions
5. Human verification (optional)

Pros: Highly scalable, leverages model capability Cons: May amplify model biases, quality ceiling limited by generator

4. Distillation from Stronger Models

Collect outputs from GPT-4 or Claude to train smaller models:

Pros: High-quality outputs, scalable Cons: Legal concerns (ToS violations), capability limited by source

Supervised Fine-Tuning (SFT) is the process of continuing training on instruction-response pairs. While conceptually simple, effective SFT requires careful attention to formatting, hyperparameters, and training dynamics.

The SFT Training Objective

Like pre-training, SFT uses cross-entropy loss on next-token prediction. The key difference is which tokens are included in the loss:

Conversation:
    [System: You are a helpful assistant]
    [User: What is 2+2?]
    [Assistant: The answer is 4.]

Loss Masking:
    [System: ...] → Loss = 0 (not predicting system prompt)
    [User: ...] → Loss = 0 (not predicting user input)
    [Assistant: The answer is 4.] → Loss computed here only

The model only learns to predict the assistant's tokens, not to reconstruct the prompt. This focuses learning on response generation.

SFT Hyperparameters and Best Practices

Common pitfalls:

1. Catastrophic Forgetting: Too many epochs or high LR erases pre-trained knowledge
2. Style Collapse: Model converges to formulaic responses
3. Data Leakage: Training data appears in benchmarks (invalidating evaluation)
4. Distribution Mismatch: Training on one format, deploying with another
5. Overfitting: Model memorizes training examples rather than generalizing

Evaluating instruction-following models is challenging because "good response" is subjective and multidimensional. Multiple complementary approaches are necessary.

Automatic Evaluation

1. Benchmark Accuracy

Standard ML benchmarks measure capability:

2. LLM-as-Judge

Use a stronger model (GPT-4) to evaluate responses:

judge_prompt = """
Evaluate the following response to the given instruction.
Rate on a scale of 1-10 for:
- Helpfulness
- Accuracy
- Relevance
- Completeness

Instruction: {instruction}
Response: {response}

Provide ratings and brief justification.
"""

Advantages: Scalable, correlates with human judgment Disadvantages: Bias toward verbose responses, judge model limitations

3. Rule-Based Evaluation

For specific tasks, programmatic checks work well:

def evaluate_code_response(response: str, test_cases: list) -> dict:
    """Evaluate code generation by running tests."""
    try:
        exec(response)  # Execute generated code
        passed = sum(1 for tc in test_cases if run_test(tc))
        return {"pass_rate": passed / len(test_cases)}
    except Exception as e:
        return {"pass_rate": 0, "error": str(e)}

def evaluate_format_following(response: str, expected_format: str) -> bool:
    """Check if response follows JSON/XML/etc format."""
    try:
        if expected_format == "json":
            json.loads(response)
            return True
    except:
        return False

Human Evaluation

Human evaluation remains the gold standard but is expensive and slow:

Pairwise Comparison (Arena-style):

• Show evaluators responses from two models (anonymized)
• Ask which response is better
• Aggregate into Elo ratings
• Used by Chatbot Arena, produces reliable rankings

Likert Scale Rating:

• Rate individual responses 1-5 on dimensions
• More granular but less reliable (calibration varies)

Preference Ranking:

• Rank 3+ responses best to worst
• Useful for training preference models (RLHF)

Beyond basic SFT, several techniques improve instruction-following quality and efficiency.

Parameter-Efficient Fine-Tuning (PEFT)

Full fine-tuning modifies all parameters. PEFT methods update only a small fraction:

LoRA (Low-Rank Adaptation)

Instead of updating weight matrix $W$, learn low-rank update: $$W' = W + BA$$

Where $B \in \mathbb{R}^{d \times r}$ and $A \in \mathbb{R}^{r \times k}$, with rank $r \ll \min(d, k)$.

QLoRA

Combines LoRA with 4-bit quantization:

• Base model quantized to 4-bit (1.75 bytes/param)
• LoRA adapters in full precision
• Enables fine-tuning 65B models on single 48GB GPU

NEFTune: Noise Embeddings for Fine-Tuning

A simple technique that improves SFT by adding noise to embeddings:

def neftune_forward(embeddings, noise_alpha=5.0):
    """Add noise to embeddings during training."""
    if model.training:
        # Compute noise scaling
        dims = embeddings.shape[-1]
        mag_norm = noise_alpha / (dims ** 0.5)
        
        # Add uniform noise
        noise = torch.zeros_like(embeddings).uniform_(-1, 1) * mag_norm
        embeddings = embeddings + noise
    
    return embeddings

This regularization reduces overfitting and improves generalization, especially on smaller datasets. Accuracy improvements of 10-15% on benchmarks have been reported.

Curriculum Learning

Order training examples from simpler to more complex:

Phase 1: Simple Q&A, single-turn, short responses
Phase 2: Multi-step tasks, structured outputs
Phase 3: Complex reasoning, long-form generation
Phase 4: Multi-turn conversations, edge cases

Intuition: Establish basic patterns before tackling complexity.

Multi-Task Instruction Tuning

Training on diverse tasks jointly improves generalization:

mixture = {
    "general_qa": 0.25,
    "coding": 0.20,
    "reasoning": 0.15,
    "creative_writing": 0.15,
    "summarization": 0.10,
    "translation": 0.10,
    "safety_examples": 0.05,
}

# Sample each batch according to mixture weights

The mixture significantly affects model behavior. Oversampling coding produces a coding-focused model; oversampling safety produces a cautious one.

Let's trace the complete journey from a base language model to a production assistant.

Stage 1: Base Model Selection

Choose a pre-trained model based on:

Popular choices: LLaMA 2/3, Mistral, Qwen, Gemma, Phi

Stage 2: Data Preparation

1. Collect/generate instructions (10K-1M examples)
2. Format consistently (ChatML, Alpaca, custom)
3. Quality filter (length, coherence, safety)
4. Deduplicate (avoid memorization)
5. Create train/val splits (reserve diverse held-out set)

Stage 3: Supervised Fine-Tuning

# Typical training run
training_args = TrainingArguments(
    output_dir="./sft-model",
    per_device_train_batch_size=4,
    gradient_accumulation_steps=8,
    learning_rate=2e-5,
    num_train_epochs=3,
    warmup_ratio=0.03,
    logging_steps=10,
    save_strategy="steps",
    save_steps=200,
    evaluation_strategy="steps",
    eval_steps=200,
    bf16=True,
)

trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=val_dataset,
    tokenizer=tokenizer,
    dataset_text_field="text",
    max_seq_length=2048,
)

trainer.train()

Stage 4: Evaluation and Iteration

Evaluation checklist:

• Validation loss converged, not overfitting
• MT-Bench score improved vs. base model
• Task-specific benchmarks tested (code, math, etc.)
• Manual spot-checking across instruction types
• Safety evaluation passed
• Format consistency verified

Common iteration patterns:

Stage 5: Deployment Preparation

Quantization for efficiency:

• GPTQ/AWQ for 4-bit deployment
• Reduces memory 4×, maintains most accuracy

Merge LoRA if applicable:

model = model.merge_and_unload()  # Bake adapters into base

Convert for inference framework:

# Convert to GGUF for llama.cpp
python convert.py ./sft-model --outtype f16 --outfile model.gguf

# Quantize to 4-bit
./quantize model.gguf model-q4_k_m.gguf q4_k_m

Instruction tuning is the crucial step that transforms powerful but unfocused base models into useful assistants. Through supervised learning on instruction-response pairs, we unlock the latent capabilities that pre-training established.

What's next:

While SFT teaches models to follow instructions, it optimizes for matching training outputs rather than genuine helpfulness. The next page covers RLHF (Reinforcement Learning from Human Feedback)—a technique that directly optimizes models for human preferences, addressing limitations of pure supervised learning and enabling more nuanced alignment.