Pairwise Ranking Score Adjustment for Competitive Models

In competitive machine learning evaluation, ranking models based on their relative performance is essential for establishing leaderboards and understanding model quality. A powerful approach to this problem is the pairwise comparison rating system, which dynamically adjusts skill ratings based on head-to-head outcomes between competitors.

This rating methodology, originally developed for chess tournaments, has become the gold standard for evaluating large language models (LLMs) through human preference assessments. Platforms like the Chatbot Arena leaderboard use this system to rank AI models based on thousands of blind pairwise comparisons.

Core Algorithm

The system works by computing expected win probabilities based on the current rating difference between two competitors, then adjusting ratings according to whether the actual outcome matched or deviated from expectations.

Expected Score Calculation

For two competitors A and B with ratings ( R_A ) and ( R_B ), the expected score for competitor A is:

$$E_A = \frac{1}{1 + 10^{(R_B - R_A)/400}}$$

Similarly, for competitor B:

$$E_B = \frac{1}{1 + 10^{(R_A - R_B)/400}}$$

Note that ( E_A + E_B = 1 ), reflecting the zero-sum nature of each comparison.

Actual Scores

Based on the comparison outcome:

• Win: Actual score = 1.0
• Loss: Actual score = 0.0
• Tie: Actual score = 0.5

Rating Update Formula

After each comparison, ratings are updated sequentially:

$$R_A^{new} = R_A + K \times (S_A - E_A)$$

$$R_B^{new} = R_B + K \times (S_B - E_B)$$

Where:

• ( K ) is the adjustment factor (sensitivity parameter)
• ( S_A ) and ( S_B ) are the actual scores for each competitor
• ( E_A ) and ( E_B ) are the expected scores

Key Insight

The adjustment magnitude reflects the surprise factor of the outcome:

• An underdog victory yields larger rating gains (and losses for the favorite)
• A favorite winning against a weaker opponent results in smaller adjustments
• Ties between equally-rated competitors cause no rating change

Your Task:

Implement a function that processes a sequence of pairwise comparisons and returns the final adjusted ratings. The comparisons must be processed sequentially, with each subsequent comparison using the updated ratings from the previous one.