# LADDER

> Recursive self-play generating progressively simpler problem variants to improve RL reasoning

- Authors: Yosh
- Published: 2025-03-21
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


# LADDER

**Paper**: [LADDER](https://arxiv.org/abs/2503.00735)

**Presenter**: Hunter

**Note Taker**: Yosh

**Date**: March 21, 2025

## Discussion Notes

-   Recursive generation and solving of progressively simpler variants of original problem drives improved problem-solving capabilities
-   First takeaway: for RL to be effective, there must be a gradient of difficulty in dataset samples, otherwise the model will catastrophically collapse (especially when gap between simple and hard samples is large)
    -   We can do this manually, e.g. chunking that's already implemented in Form reader project
-   LADDER consists of 3 components:
    -   Variant Generation
    -   Solution Verification
    -   Reinforcement Learning
-   Used base model to generate mathematical transformations of integrals categorized by impact on problem difficulty. Then, for each integral, randomly sample 3-5 transformations and provide them as explicit suggestions to variant generation model.
    -   Also utilized temperature cycling and persona-based prompting (e.g. "think like Euler or Gauss") which improved performance
-   Quality control is essential in variant generation process, as small perturbations could lead to potentially intractable integrals, or variants could actually be more difficult
-   Test-Time Reinforcement Learning - Perform the LADDER at test time, retaining the post-training model but disposing of the test-time tuned weights

-   Train on 10 problems and test on 100, generating 500 variants/problem
-   Using a best of N approach with N=1,10 yielded 1%,2% accuracy on test set respectively. RL w/o variants yielded 3% and LADDER RL w/ variants reached 82% performance
-   LADDER+TTRL attained 90% on MIT Integration Bee