Reward Modeling

A reward model takes a prompt and candidate response, or a state and trajectory, and produces a scalar or structured assessment. In pairwise language model setups, its parameters are trained so a preferred response receives a higher score than a rejected response. Other formulations use ratings, rankings or process-level judgments. Scores summarize learned preferences rather than objective correctness unless the data specifically supports that interpretation. A reward model may select candidates, evaluate experiments or supply a reinforcement learning objective. These uses expose it to different distributions, particularly when a policy actively searches for outputs that maximize its score.

Write a rubric, choose label granularity and gather comparisons with varied quality gaps. Check inter-annotator agreement and audit whether response length or formatting predicts the labels. Split related prompts and candidate sources together to avoid leakage. Evaluate pairwise accuracy and disagreement by task and response style; inspect score changes on controlled perturbations. Test generated candidates from policies that differ from the training generator. The result is a versioned scorer with a defined validity range and explicit evidence about its errors, suitable for a particular decision rather than an unrestricted quality oracle.

High held-out comparison accuracy can hide systematic bias on rare but consequential cases. Scalar rewards collapse competing objectives, and their scale is usually meaningful only within the model's training setup. Optimization can amplify weaknesses that ordinary evaluation never encounters. A learned reward is distinct from a deterministic verifier and from direct policy preference losses. Validate under the intended selection or training pressure, preserve independent evaluators and avoid interpreting a high score as a probability that an answer is true.