Multiple Choice Log-Probability Evaluation

In the evaluation of large language models (LLMs), one of the most rigorous and widely adopted methodologies is log-probability-based scoring for multiple choice question answering. This approach assesses how well a model can identify correct answers by analyzing the probability distributions it assigns to each answer option.

The Evaluation Framework

When a language model is presented with a multiple choice question, it internally computes a probability distribution over all possible answer choices. Rather than working with raw probabilities (which can suffer from numerical underflow for low-probability events), we use log-probabilities—the natural logarithm of the probability values. This representation offers superior numerical stability and computational efficiency.

For each question, the model produces a log-probability score for every answer option. The predicted answer is the choice that receives the highest log-probability, indicating the model's most confident selection. This aligns with the principle of maximum likelihood estimation.

Key Metrics

Your evaluation function must compute three essential metrics:

1. Predictions: For each question, determine the predicted answer as the index of the answer choice with the maximum log-probability.

2. Accuracy: The proportion of questions where the model's prediction matches the ground truth correct answer. This is calculated as:

$$\text{Accuracy} = \frac{\text{Number of Correct Predictions}}{\text{Total Number of Questions}}$$

3. Average Correct Probability: To understand model calibration, we convert log-probabilities back to probabilities using the softmax function and compute the average probability mass assigned to the correct answer across all questions. For a question with log-probabilities $[l_0, l_1, ..., l_{n-1}]$, the probability for choice $i$ is:

$$P_i = \frac{e^{l_i}}{\sum_{j=0}^{n-1} e^{l_j}}$$

The average of $P_{\text{correct}}$ across all questions gives insight into how confident the model is about correct answers.

Numerical Stability

When converting log-probabilities to probabilities via softmax, direct computation of $e^{l_i}$ can cause numerical overflow or underflow. A numerically stable approach involves subtracting the maximum log-probability before exponentiation:

$$P_i = \frac{e^{l_i - l_{\max}}}{\sum_{j=0}^{n-1} e^{l_j - l_{\max}}}$$

where $l_{\max} = \max(l_0, l_1, ..., l_{n-1})$.

Your Task

Implement a function that takes log-probabilities for multiple choice questions and their correct answers, then returns a dictionary containing the accuracy, predicted answers, and average probability assigned to correct answers.