# Dynamic Fine-Tuning (DFT)

> DFT reweights the SFT loss by token probability, eliminating implicit instability and substantially improving generalization

- Authors: Ben
- Published: 2025-08-15
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


# On the Generalization of SFT
**Paper**: [On the Generalization of SFT](https://arxiv.org/abs/2508.05629)  
**Presenter**: Hunter  
**Note Taker**: Ben  
**Date**: August 15, 2025  

---

## Overview
This paper analyzes why supervised fine-tuning (SFT) often underperforms reinforcement learning (RL) in terms of generalization. The authors show mathematically that SFT gradients can be seen as a special case of policy gradients, but with an implicit, ill-posed reward structure that creates instability and overfitting. They propose Dynamic Fine-Tuning (DFT), a simple reweighting of the SFT loss by token probability. DFT is presented as a one-line code change that stabilizes training and substantially improves generalization across tasks and models.  

---

## Novelty
- Provides a formal equivalence between SFT and policy gradient methods, identifying an implicit inverse-probability reward that destabilizes SFT.  
- Introduces Dynamic Fine-Tuning (DFT), which rescales the objective with token probabilities to eliminate this instability.  
- The modified loss function is:  

  \[
  L_{\text{DFT}}(\theta) = \mathbb{E}_{(x,y^\star)\sim D}\; -\sum_{t=1}^{|y^\star|} \text{sg}\big(\pi_\theta(y^\star_t \mid y^\star_{<t}, x)\big)\, \log \pi_\theta(y^\star_t \mid y^\star_{<t}, x)
  \]

  where `sg(·)` is the stop-gradient operator.  
- Shows that DFT achieves stronger gains than both standard SFT and importance-weighted SFT (iw-SFT).  
- Extends evaluation beyond SFT settings into offline RL, where DFT outperforms both offline (RFT, DPO) and online methods (PPO, GRPO) despite its simplicity.  
- Observes that DFT changes token probability distributions in a polarized way, boosting some tokens while downweighting others, unlike SFT’s uniform probability increase.  

---

## Learnings
- The poor generalization of SFT comes from its implicit reward weighting, which overemphasizes low-probability expert tokens and increases gradient variance.  
- DFT corrects this by giving uniform rewards to expert tokens, leading to more stable updates.  
- Empirically, DFT consistently outperforms SFT across multiple benchmarks (Math500, Minerva, Olympiad Bench, AIME, AMC) and models (Qwen2.5, LLaMA, DeepSeek). Gains are often several times larger than those achieved by standard SFT.  
- DFT converges faster, achieves better early-stage performance, and avoids the plateau behavior seen in SFT.  
- In offline RL experiments, DFT not only surpasses SFT-based baselines but also outperforms online methods like PPO and GRPO, highlighting its efficiency.  
- Token distribution analysis suggests that not all tokens should be fit equally; deprioritizing connective or low-value tokens may improve robustness.  
- Ablation studies confirm that DFT’s advantage is not due to hyperparameter choices: improvements hold across learning rates and batch sizes.  

---

## Notes for Application
- DFT offers a practical improvement to SFT with minimal implementation cost: a single modification to the loss.  
- Particularly useful in settings where RL is too resource-intensive or reward signals are unavailable.  
- Highlights the importance of examining implicit reward structures in fine-tuning objectives, which may influence broader post-training strategies.