# Reflect, Retry, Reward > Two-stage self-reflection RL: reward only reflection tokens when retry succeeds, improving small models significantly - Authors: Ben - Published: 2025-06-20 - Source: Bevaya Labs, --- # Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learning **Paper**: *Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learning* **Presenter**: Hunter **Note Taker**: Ben **Date**: June 20, 2025 --- ## Overview Two-stage procedure: on failure, the model writes a brief self-reflection; the self-reflection is then added to context for a second attempt. If the second attempt succeeds, only the self-reflection tokens receive reward via GRPO. Requires a binary validator (success/fail). Reported gains include up to +34.7% on equation writing and +18.1% on function calling, with trained 1.5–7B models surpassing much larger baselines in some cases. --- ## Novelty - Reward assigned only to reflection tokens when a retry fixes a failure (masked GRPO over multi-step generations). - Multi-step GRPO implementation: first attempt → reflection → second attempt, with masking to isolate reflection tokens. - “Dataset of failures” constructed by sampling multiple completions and keeping only verified failures for efficiency and clearer learning signals. - Task-agnostic setup: relies on validators (function-call exact match; equation evaluates to target) rather than teacher models or synthetic data. - Empirical comparisons include first/second-try performance pre/post training and an analysis of catastrophic forgetting (minimal changes on MMLU-Pro, GSM8K, HellaSwag, MATH). --- ## Learnings - Self-reflection becomes shorter and clearer after training; models improve even on first-try accuracy (without generating explicit reflections). - On APIGen and Countdown, trained small models can outperform untrained much larger ones under two-attempt evaluation. - Average improvements after training are substantial on both tasks; error profiles shift (e.g., fewer tool-choice errors in function calling; better number usage in equations). - Minimal catastrophic forgetting relative to vanilla checkpoints under standard eval suites. --- ## Notes for Application - Feasible where binary validators exist, like loss runs verifier. - Reflection-only reward offers a way to improve retry behavior without altering task-specific outputs directly. - A “failures-only” training set could reduce compute and target the most informative examples in extraction or function-calling pipelines.