# The Illusion of Thinking > Controllable puzzle environments reveal three reasoning regimes and a complexity collapse threshold in LRMs - Authors: Ben - Published: 2025-06-13 - Source: Bevaya Labs, --- # The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity **Paper**: [The Illusion of Thinking](https://arxiv.org/abs/2506.06941) **Presenter**: Hunter **Note Taker**: Ben **Date**: June 13, 2025 --- ## Overview Uses controllable puzzle environments (Tower of Hanoi, Checker Jumping, River Crossing, Blocks World) to probe Large Reasoning Models (LRMs) versus standard LLMs under matched inference compute. Finds three regimes: (i) at low complexity, non-thinking models are more accurate and token-efficient; (ii) at medium complexity, LRMs gain an advantage; (iii) at high complexity, both collapse to near-zero accuracy. Near the collapse point, LRMs allocate fewer thinking tokens despite ample budget, suggesting an inference-time scaling limit. The setup enables analysis of final answers and internal traces, revealing complexity-dependent failure patterns. --- ## Novelty - Controllable complexity via puzzle families enables clean comparisons and avoids benchmark contamination; results reported across matched thinking/no-thinking model pairs (e.g., Claude 3.7 Sonnet, DeepSeek R1/V3). - Three-regime characterization of performance with complexity (low → non-thinking better; medium → thinking advantage; high → collapse), with visualizations across multiple puzzles. - Reasoning-effort dynamics: thinking tokens initially rise with complexity, then decline near the collapse threshold. - Thought-position analysis: correctness vs. position within the reasoning trace changes with complexity. At low N, models are early correct then overthink; at medium N, correct solutions arrive later; at high N, models fixate on early errors. - Algorithm-execution check: even when given explicit solution algorithms to execute, models still fail at similar complexity points. --- ## Learnings - Low/medium/high regimes: non-thinking > thinking at low complexity; thinking helps at medium; both collapse at high. - Collapse behavior: accuracy falls to near zero beyond a model-specific threshold; simultaneously, thinking tokens decrease instead of scaling with difficulty. - Overthinking vs. fixation: at low complexity, models often find a correct plan early but continue exploring incorrect alternatives; at medium, correct solutions appear later; at high, models fixate on early wrong paths and waste budget. - Exact algorithm execution remains brittle: providing step-by-step algorithms does not prevent failure, indicating limits in following and verifying logical procedures. - Failure location: first failure moves occur far earlier than solution length and vary non-monotonically with complexity; error-free spans differ widely across puzzles. - Controls: context limits and sampling do not explain collapse. --- ## Notes for Application - For document-reasoning pipelines, consider controlled difficulty sweeps (synthetic tasks with adjustable compositional depth) and track both final accuracy and where failures occur within multi-step outputs. - Monitor inference-time effort (e.g., reasoning tokens or steps) versus task difficulty; a decline in effort near hard cases may indicate impending failure modes rather than efficiency. - When adding “thinking” to models, compare against non-thinking baselines under equal compute; improvements may concentrate only in a medium-difficulty band.