# DAPO > System-scale RL recipe with clip-higher, dynamic sampling, and token-level credit assignment - Authors: Hunter - Published: 2025-04-04 - Source: Bevaya Labs, --- # DAPO: An Open-Source LLM Reinforcement Learning System at Scale **Paper**: [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://arxiv.org/abs/2503.14476) **Presenter**: Yosh **Note Taker**: Hunter **Date**: April 4, 2025 --- ## Overview System-scale RL recipe that modifies GRPO-style training with changes to clipping, sampling, token-level credit assignment, and length handling. Also removes the KL term and uses an answer-equality reward. --- ## Novelty - Clip-Higher: decouples upper and lower clip bounds; relaxes the upper bound to avoid restricting generations while keeping lower-bound clipping. - Dynamic sampling: builds batches that are not all-correct or all-incorrect to keep advantages informative. - Token-level policy-gradient loss: replaces “mean of means” with a single averaging step so token contributions are credited directly. - Overlong reward shaping: masks truncated regions instead of penalizing them; optional soft ramp as outputs approach max length. - Additional choices: KL term removed; reward defined by abstract equality of answers; revised gradient averaging. --- ## Learnings - Decoupled clipping provides control over how much updates constrain outputs. - Ensuring batch outcome diversity avoids zero-advantage collapses. - Token-level credit sharpens feedback to helpful and unhelpful tokens. - Treating truncation separately from verbosity avoids punishing context-limit artifacts. - KL-free training shifts reliance to the reward and clipping scheme. --- ## Notes for Application - Dynamic sampling could prevent degenerate batches in extraction RL (e.g., all fields correct or all incorrect). - Overlong shaping can help when context limits cause truncation in long documents.