# xRAG > Compresses retrieved documents to a single token via staged training for extreme RAG context reduction - Authors: Hunter - Published: 2025-05-16 - Source: Bevaya Labs, --- # xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token **Paper**: [xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token](https://arxiv.org/abs/2405.13792) **Presenter**: Olivia **Note Taker**: Hunter **Date**: May 16, 2025 --- ## Overview Compresses each retrieved document into a single token and conditions the LLM on that token instead of the full text. Training is staged: (1) paraphrase-style pretraining to associate an embedding with its source document, then (2) context-aware instruction tuning using only the embedding, including a self-distillation step against a standard RAG setup. Reported results are near or above regular RAG on several QA datasets; authors argue full-text RAG can be distracted by extra snippet text. --- ## Novelty - One-token document representation injected into the LLM in place of full passages. - Paraphrase pretraining to align vectors with documents. - Context-only instruction tuning plus self-distillation from a full-text RAG teacher. --- ## Learnings - Can roughly match or surpass standard RAG on key QA benchmarks despite using only a single token per document. - Claimed robustness when retrieved text contains distracting or irrelevant content. - Open questions remain for multi-hop reasoning and detail-critical, novel documents. --- ## Notes for Application - Could be relevant for very long documents or table-heavy pages that often exceed context budgets. - Likely unnecessary for small documents and short outputs. - Worth comparing against other RAG-compression approaches if fine-grained details must be preserved.