Language Model Perplexity Score

Perplexity is one of the most fundamental and widely-used metrics for evaluating the quality of language models. It provides an intuitive measure of how "surprised" or "uncertain" a model is when predicting the next token in a sequence—a lower perplexity indicates that the model is more confident and accurate in its predictions.

Understanding Perplexity

At its core, perplexity quantifies how well a probability distribution predicts a sample. For language models, it answers the question: "On average, how many equally likely words would the model be choosing from at each step?"

For example:

• A perplexity of 1 means the model perfectly predicts every token with 100% confidence
• A perplexity of 10 means the model's uncertainty is equivalent to randomly selecting from 10 equally likely options
• A perplexity of 1000 indicates the model is highly uncertain, as if guessing among 1000 possibilities

Mathematical Definition

Given a sequence of N tokens where the model assigns probabilities P(tᵢ | context) to each token, perplexity is computed as:

$$\text{Perplexity} = \exp\left(-\frac{1}{N} \sum_{i=1}^{N} \ln P(t_i | \text{context})\right)$$

This is equivalent to the geometric mean of the inverse probabilities:

$$\text{Perplexity} = \left(\prod_{i=1}^{N} \frac{1}{P(t_i | \text{context})}\right)^{1/N}$$

Your Task

Write a function that computes the perplexity of a language model given a list of token probabilities. Each probability in the list represents how likely the model predicted the actual token that appeared in the sequence.

Input:

• probabilities: A list of floats where each value represents P(token_i | context) for the i-th token in the sequence. All probabilities are in the range (0, 1].

Output:

• A single float representing the perplexity score, rounded to 6 decimal places.