# InftyThink > Iterative think-summarize-continue reasoning to address overthinking and long-context memory limits - Authors: Ben - Published: 2025-03-14 - Source: Bevaya Labs, --- # InftyThink **Paper**: [InftyThink](https://arxiv.org/abs/2503.06692) **Presenter**: Yosh **Note Taker**: Ben **Date**: March 14, 2025 --- ## Overview Addresses “overthinking” and circular reasoning by constraining early-stage reasoning and restructuring the generation process. Proposes multi-step, chunked reasoning: produce a short reasoning segment, summarize it, append the summary to the prompt, and iterate until an answer or an iteration cap is reached. This creates a sawtooth context pattern (vs. steadily growing contexts), reduces memory/compute pressure from long-context attention, and aims to avoid truncation issues. --- ## Novelty - Early-stage capacity restraint (“Occam’s Razor”) to counter the overthinking optimization problem. - Iterative “think → summarize → continue” procedure with an iteration hyperparameter \(n\) as a ceiling (model may answer earlier). - Incremental summaries are over reasoning steps (not document content); the source document remains present as the query at every step. - SFT data construction: chunk r1 traces on paragraph/sentence boundaries, then summarize segments (e.g., with a larger model) to create supervision for the iterative procedure. --- ## Learnings - Reported AIME24 gains of ~5–10% after SFT; larger base models see smaller gains within that range. - Shorter-context models benefit more, consistent with the generation algorithm’s memory profile. - Even with a single iteration, accuracy at fixed completion length exceeds that of a standard reasoning model of the same length. - Practical note: useful when required “thinking” exceeds available GPU memory. --- ## Notes for Application - Multi-step chunked reasoning may be feasible for long or multi-extraction document tasks: set an iteration cap \(n\), keep the full document in context, and summarize only the reasoning. - Training data can mirror the paper’s approach: split existing CoT traces into segments and attach short summaries for SFT. - Evaluate at equal completion lengths and on shorter-context deployments where the method is expected to help most.