Intra-Group Advantage Normalization for Policy Training

In modern reinforcement learning systems designed for fine-tuning large language models, a crucial technique involves relative advantage estimation within generation groups. When optimizing a language model's policy, instead of using raw reward signals directly, we compute how each generated output compares to other outputs for the same input prompt.

This technique is fundamental to Group-Based Relative Policy Optimization, a training paradigm where the model generates multiple candidate outputs (a "group") for each prompt, and each output's advantage is computed relative to the group's performance. This relative scoring mechanism ensures that policy gradients push the model toward outputs that are better than the group average, rather than simply toward outputs with high absolute rewards.

The Normalization Process:

Given a group of G outputs with corresponding reward scores, the advantage for each output is computed using z-score normalization:

$$A_i = \frac{r_i - \mu}{\sigma}$$

Where:

• rᵢ is the reward for the i-th output
• μ (mu) is the mean reward across all outputs in the group
• σ (sigma) is the standard deviation of rewards in the group
• Aᵢ is the resulting advantage score

This normalization transforms rewards into a distribution centered at zero, where:

• Positive advantages indicate outputs that performed better than the group average
• Negative advantages indicate outputs that performed worse than the group average
• The magnitude reflects how many standard deviations an output deviates from the mean

Why This Matters:

Raw reward values can vary significantly across different prompts, making it difficult to combine gradients from different training examples. By normalizing within each group, we ensure that every prompt contributes equally to the training signal, regardless of the absolute reward scale. This prevents prompts with larger reward ranges from dominating the optimization process.

Your Task: Write a function that takes a list of reward values for a group of model outputs and returns the normalized advantage scores. Round each result to 4 decimal places for numerical precision.