# SimpleRL-Zoo > Study of r1-style RL training variables: base model choice, reward design, and training curriculum - Authors: Ben - Published: 2025-03-28 - Source: Bevaya Labs, --- # SimpleRL-Zoo **Paper**: [SimpleRL-Zoo](https://arxiv.org/abs/2503.18892) **Presenter**: Hunter **Note Taker**: Ben **Date**: March 28, 2025 --- ## Overview The paper explores r1-style RL training and how base-model choice, reward design, and training curriculum affect downstream performance. It also examines the impact of supervised fine-tuning (SFT) versus RL-only training and proposes evaluation metrics that better capture reasoning behaviors. --- ## Novelty - Reports that pre-RL SFT limits exploration and caps performance. - Emphasizes progressive difficulty scaling of training data rather than rigid supervision. - Uses metrics beyond accuracy and response length: - reasoning behavior ratio (backtracking, verification, subgoal setting, enumeration) - clip ratio and average stopped length - pass@k accuracy as an indicator of exploration - Notes that strict formatting rewards and complex prompts can suppress exploration and induce “overthinking.” - Shows strong effects from exploration-related hyperparameters (temperature, num_generations). --- ## Learnings - Rigid formatting disincentivizes exploration; flexibility supports richer reasoning. - Response length is not a proxy for reasoning quality. - Base-model choice matters; some models (e.g., Qwen) already exhibit strong math reasoning, reducing RL gains. - SFT before RL can reduce accuracy and reasoning sophistication, leading to early plateaus. - RL-only training adapts more freely and can produce the reported “aha” behaviors. - If SFT is used, it should be framed to encourage exploration, not constrain it. --- ## Notes for Application - Reward design: simple correctness-based rewards are viable; extensions could be explored. - Training strategy: construct datasets where difficulty ramps with model capability. - Hyperparameters: increasing num_generations and using higher temperatures (>1) promotes exploration.