LLMs Don't Need More Parameters. They Need Loops.

The foundational insight from Kaplan et al.'s scaling laws (0:08) is that larger models need proportionally more data — an 8x model increase requires ~5x more data. However, human-generated internet data is running out (1:25), creating an upper bound on useful compute and motivating the search for architectures that decouple compute from data.

Current reasoning approaches like chain-of-thought face three key problems: extended context risks forgetting critical information (2:15), the base model sets a ceiling on reasoning performance regardless of reinforcement learning (3:05), and reasoning in vocabulary space is suboptimal because different concepts require different numbers of tokens to express (3:45).

Instead of generating chain-of-thought tokens, looped transformers pass latent vectors through the same model repeatedly with an exit gate deciding when to stop (5:00). This compresses the KV cache, operates in latent space rather than vocabulary space, and is optimized during pre-training rather than as a post-hoc process.

The exit gate uses a sigmoid activation to produce exit probabilities at each loop step (6:20). These are converted to unconditional probabilities using a survival function (hazard-based formulation), ensuring proper probability mass allocation. At the maximum loop count, remaining probability mass is assigned to force an exit (7:30).