# Unintentional Unalignment > DPO likelihood displacement causes both preferred and dispreferred probabilities to decrease, risking unintended alignment - Authors: Yosh - Published: 2025-04-18 - Source: Bevaya Labs, --- # Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization **Paper**: [Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization](https://arxiv.org/abs/2410.08847) **Presenter**: Nikhil **Note Taker**: Yosh **Date**: April 18, 2025 ## Discussion Notes - During training it can be observed that models unintentionally align itself toward undesirable responses when using DPO - During DPO, the algorithm attempts to increase the gap between desired and undesired responses, which was ostensibly believed to occur by increasing and decreasing their respective probabilities - It was found that the probabilities actually both decrease; however, one just decreases more sharply than another, still expanding the delta - This can yield 'catastrophic responses' where the model must then augment the probability of other potentially dispreferred tokens - This phenomena emerges in very simple settings - At large scales this can cause the entire model to become unintentionally aligned