# Generative Verifiers

> Recasts reward modeling as next-token prediction using Yes/No probability, with optional CoT rationale generation

- Authors: Ben
- Published: 2025-08-08
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


# Generative Verifiers: Reward Modeling as Next-Token Prediction
**Paper**: [Generative Verifiers: Reward Modeling as Next-Token Prediction](https://arxiv.org/abs/2408.15240)  
**Presenter**: Hunter  
**Note Taker**: Ben  
**Date**: August 8, 2025  

---

## Overview
Recasts verification/reward modeling as next-token prediction: given a problem and a candidate solution, the verifier answers “Is the answer correct (Yes/No)?” and uses the probability of Yes as the score. A CoT variant first generates a verification rationale, then emits Yes/No; multiple rationales can be sampled and averaged at inference. Reported gains over discriminative reward models and DPO-style verifiers on GSM8K and algorithmic tasks, with transfer to MATH.

---

## Novelty
- Verification as token probability (direct and CoT modes):  
  \( r_{\text{Direct}}(x,y) = p_\theta(\text{Yes}\mid x,y,I) \)  
  \( r_{\text{CoT}}(x,y) = p_\theta(\text{Yes}\mid x,y,I_{\text{CoT}},v_{\text{CoT}},I) \), with \(v_{\text{CoT}}\sim p_\theta(\cdot\mid x,y,I_{\text{CoT}})\)  
  Majority vote over \(K\) rationales: \( r_{\text{MajV@K}}=\frac{1}{K}\sum_{i=1}^K p_\theta(\text{Yes}\mid\cdot,v^{(i)}_{\text{CoT}}) \)
- Unified objective for verification + solution generation:  
  \( L_{\text{GenRM}} = L_{\text{SFT}}(D_{\text{verify}}) + \lambda\,L_{\text{SFT}}(D_{\text{correct}}) \)
- Synthetic verification rationales for CoT training (reference-guided variants and multiple rationales per solution explored).

---

## Learnings
- Best-of-N re-ranking improves when scoring candidates with direct/CoT verifiers; CoT with multiple verification votes increases accuracy.
- Joint training on verification plus correct-solution SFT improves verification; overly large generation mix can reduce verifier quality.
- Performance scales with verifier size and with the number of verification votes \(K\).
- Quality and quantity of CoT rationales both matter; reference-guided rationales and multiple rationales per solution tend to help.

---

## Notes for Application
- Generative verification appears compatible with loss-run extraction: a verifier can score candidate parses by emitting Yes/No and using \(p(\text{Yes})\) as the selection signal.  
- The Yes-token probability can serve as a confidence score for downstream API calls where logprobs are not returned.