# SFT Memorizes, RL Generalizes

> Comparative study showing SFT memorizes training distribution while RL learns generalizable strategies

- Authors: Yosh
- Published: 2025-02-28
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


# SFT Memorizes, RL Generalizes

**Paper**: [SFT Memorizes, RL Generalizes](https://arxiv.org/abs/2501.17161)

**Presenter**: Hunter

**Note Taker**: Yosh

**Date**: February 28, 2025

## Discussion Notes

-   Comparative study of SFT vs RL performance on a toy arithmetic reasoning
    task and a visual navigation task (textual and visual modalities).
-   SFT quickly achieves good performance in-distribution due to memorization but
    generalizes poorly out-of-distribution. RL, on the other hand, generalizes well
    out-of-distribution while following a smoother learning curve in-distribution,
    indicating true progressive learning.
-   Out of distribution performance of RL is very applicable to our use cases
    -   Demand forms classification and/or extractions, and loss runs extractions
    -   Form extraction model being trained on a subset of form types and then being
        tested on unseen form types (ACORDs).
-   RL is more sample efficient than SFT, which is compatible with some of our lower
    sample count real datasets.
-   We can use RL to develop other in-house models to complement our production models
    -   Verifier model to confirm annotations/catch potential errors