# SingLoRA

> Single symmetric low-rank adapter replacing LoRA's two-matrix update for better stability and fewer parameters

- Authors: Ben
- Published: 2025-07-11
- Source: Bevaya Labs, <https://labs.bevaya.ai/>

---


# SingLoRA: Low Rank Adaptation Using a Single Matrix
**Paper**: [SingLoRA](https://arxiv.org/abs/2507.05566)
**Presenter**: Hunter  
**Note Taker**: Ben  
**Date**: July 11, 2025

---

## Overview
Reformulates LoRA by replacing the two-matrix update \(W_0 + BA\) with a single low-rank, symmetric update \(W_0 + \tfrac{\alpha}{r}\,u(t)\,AA^\top\). The goal is to remove inter-matrix scale conflicts, improve training stability in the infinite-width analysis, and cut adapter parameters by about half. The work extends to non-square weight matrices, argues that attention expressiveness is preserved despite symmetric updates, and reports gains on GLUE (RoBERTa, GPT-2) and MNLI with LLaMA-7B, plus diffusion fine-tuning on DreamBooth (Tables 1–3; Figure 2). :contentReference[oaicite:0]{index=0}

---

## Novelty
- Single-matrix low-rank adapter with ramp-up scalar \(u(t)=\min(t/T,1)\); initialization keeps the base model unchanged at step 0.  
- Stability analysis: shows stable feature learning with a learning-rate scale of \(O(n^{-1/2})\) in a toy model; avoids the scale-mismatch seen in LoRA’s \(A,B\) updates.  
- Transformation-invariance for the parameterization is proved under standard first-order optimizers.  
- Extension to rectangular layers via truncated \(A^\*\) so \(W_0 + A^\*A^\top\) applies to common transformer blocks.  
- Attention expressiveness: although updates are symmetric per matrix, \(QK^\top\) interactions are not constrained to be symmetric, so general patterns are learnable (Figure 1).

---

## Learnings
- Reported GLUE results: mean accuracy improves over LoRA and matches/exceeds LoRA+ and DoRA with roughly half the adapter parameters (Table 1).  
- LLaMA-7B on MNLI: 91.3% vs. LoRA 89.1, LoRA+ 90.2, DoRA 90.6, with ~40% fewer trainable params (Table 2).  
- Learning-rate robustness: accuracy varies ~1% across LR sweeps vs. ~4.8% for LoRA (Figure 2).  
- DreamBooth: higher DINO similarity at the same or lower parameter budgets; qualitative examples retain subject details (Table 3; Figure 3).  
- Practical read: fewer hyperparameter sensitivities (no split LR for \(A,B\)), reduced parameter budgets, and stable convergence with standard optimizers.

---

## Notes for Application
- For document-processing finetunes, a single-matrix adapter could reduce trainable parameters and lower sensitivity to learning-rate choice while retaining model capacity in attention blocks.  
- When PEFT stability is a concern (e.g., smaller batches or limited LR sweeps), the symmetric \(AA^\top\) update may help avoid scale-mismatch issues seen in two-matrix LoRA.  
- If adapters are deployed across multiple transformer layers, the rectangular extension suggests the approach is compatible with common projection shapes in extraction models.