dk

The LLM bottleneck is moving from data to energy — just not in proportion

Summary

The dominant bottleneck on LLM capability is shifting from acquiring high-quality training data to compute that generates verified information on its own — and, ultimately, to energy. The shift is real, but it only becomes precise once two misconceptions are cleared away. First, performance is not proportional to tokens or compute. The observed relationship is a power law with a small exponent — Kaplan et al.’s fit gives ~0.05, roughly an 11% loss reduction per 10× of compute. Second, the reason energy becomes the metric is economics, not thermodynamics. Current GPUs operate five to eight orders of magnitude above the Landauer limit; for the foreseeable future the binding constraint is the power grid and capital, not physics. The candidate for the new metric, in one sentence: the rate of verified information generation per unit of energy — verified bits per joule.

0. Scope of the argument

Training token counts, data composition, and FLOPs for commercial frontier models are not public. The assumption that two specific models “share the same dataset” cannot be verified — and, as Section 3 shows, it has become an assumption that cannot even hold for recent generational comparisons. This essay therefore argues only from published, measured data.

1. The data bottleneck was real — and it has a ceiling

Hoffmann et al. (2022, Chinchilla) decompose loss as L(N, D) = E + A/N^α + B/D^β. E is the irreducible loss term — an estimate of the entropy of the data distribution under the model family — fitted at roughly 1.69 nats/token in the original paper. Fix the data D, and no amount of compute pushes loss below E + B/D^β. The ceiling is built into the equation.

The fine-grained constants printed in the paper (A = 406.4, B = 410.7, α = 0.34, β = 0.28), however, cannot be cited at face value. Besiroglu et al.’s (2024) replication showed that this Approach 3 fit fails to describe the reconstructed data, reports implausibly narrow confidence intervals — intervals that tight would require three orders of magnitude more experiments than were actually run — and contradicts Approaches 1 and 2 in the same paper: the closed-form optimum of the printed constants implies a ratio far above ~20 tokens per parameter, while the re-fitted estimates are consistent with the ~20-token conclusion. This essay therefore relies only on the decomposition itself, the existence of E, and the robust ~20 tokens/param result supported by all three approaches.

Repetition cannot route around the ceiling either. Muennighoff et al. (2023) measured that repeating the same corpus for up to ~4 epochs is nearly equivalent to fresh data, but beyond that the value of repeated tokens decays exponentially — and even infinite repetition yields only a finite ceiling on effective data.

And the data itself is running out. Epoch AI (Villalobos et al., 2024 revision) estimates the effective stock of public human-generated text at roughly 300 trillion tokens and projects that, on current trends, frontier models will fully consume it between 2026 and 2032 (an 80% confidence interval, median 2028). Notably, the same paper concludes that synthetic data generation, transfer from data-rich domains, and data-efficiency improvements are likely to carry progress past the bottleneck — the same exits Section 3 examines.

Industry behavior corroborates the pressure. The Llama 3 herd was trained on roughly 15 trillion tokens: about 1,900 tokens per parameter for the 8B model — deliberate over-training far past compute-optimal, which the paper explicitly justifies on inference economics. Even the flagship 405B (15.6T tokens, ~38 tokens/param) sits approximately at compute-optimal under Meta’s own scaling law for its 3.8×10²⁵ FLOP budget (their fit: 402B parameters on 16.55T tokens). Small models get tokens poured in to the limit; the flagship gets its budget matched to the high-quality tokens actually available. Either way, quality tokens are being treated as the scarce resource.

2. “Proportional” is not what we observe

The compute–loss relationship Kaplan et al. (2020) fitted, under their own model–data allocation, is L ∝ C^−0.050. Ten times the compute buys loss ×0.89; a hundred times, ×0.79. The exponent’s magnitude shifts with allocation assumptions and re-fits, but every published fit has the same shape: a power law with a small exponent. Not linear proportionality — a straight line on log axes, with diminishing returns built into the definition.

Then what are the “sharp jumps” we see on benchmarks? Schaeffer et al.’s (2023) answer: much of it is a discontinuity in the metric, not in the capability. Nonlinear, discontinuous metrics like exact-match make a smooth per-token loss improvement look like a step function; re-measure the same models with continuous metrics and the jump disappears — that is the core of the paper.

Reading generational jumps as a linear effect of token count is therefore wrong twice over. The exponent is 0.05, and much of the jump is metric nonlinearity — or, more importantly, the new training axis discussed next.

3. Regime change: compute generates data

The prototype appeared in 2017. AlphaGo Zero started from zero human games, generated 4.9 million self-play games on its own, and beat the human-data-trained AlphaGo Lee 100–0 (Silver et al., Nature 2017). Compute → search → verification (win/loss) → new information that was not in the original data. Porting that loop to language models is what the verifiable-reward RL of 2024–2025 is.

DeepSeek-R1-Zero (2025) is the quantitative evidence. The base model (DeepSeek-V3-Base) scored 15.6% pass@1 on AIME 2024; after RL with verifiable rewards alone — no SFT — it reached 71.0% (86.7% with majority voting). The training corpus did not grow by a single token. Reasoning trajectories the model generated and the verifier filtered are what lifted pass@1.

Whether this constitutes generating new information, however, has an empirical counterargument. Yue et al. (2025, NeurIPS) showed, across model families, RL algorithms, and benchmarks, that RLVR models beat their base models at k = 1 but the base models catch up or overtake at large pass@k — and that the reasoning paths of RLVR models are already contained in the base model’s sampling distribution. On this reading, current RLVR does not create new reasoning patterns; it sharpens the distribution toward rewarded paths, improving sampling efficiency. Other work reports boundary expansion on specific task families, so the debate is live. The gap between the AlphaZero regime — unbounded exploration of new states inside a perfect simulator — and current RLVR, which reweights trajectories over a finite prompt distribution, has not closed; Yue et al. themselves contrast the two regimes (action-space size, dependence on a pretrained prior). The claim “compute generates data” should therefore be read as established fact in board-game-like domains, and as an open debate for LLM RLVR.

The implication — at least in domains with a complete verification signal — is structural. The dataset is no longer an exogenous variable; it is endogenous, a function of compute. The ceiling from Section 1, L ≥ E + B/D^β, remains true — but the moment D itself becomes expandable with compute, the ceiling moves. This is why “generational comparison under the same dataset” is a frame that no longer applies to recent models.

4. The exchange rate at deployment: joules ↔ accuracy

Energy converts into capability after training ends, too. OpenAI had already published accuracy-versus-test-time-compute curves with o1 (2024-09); the most explicit case of that exchange rate being observed on a third-party benchmark, with a concrete multiplier attached, is o3’s ARC-AGI semi-private evaluation (ARC Prize, 2024-12). The high-efficiency configuration (under the $10k compute limit) scored 75.7%; a configuration with 172× the compute scored 87.5%. The evaluated o3 had been trained on 75% of the ARC-AGI-1 public training set — a caveat for interpreting the absolute scores, but one that does not affect the within-model compute–accuracy exchange rate.

The data cuts both ways. It is the strongest empirical support for “energy expenditure is a control knob on capability” — and simultaneously the strongest empirical support for diminishing returns: 172× the compute bought 11.8 percentage points.

5. Economics, not thermodynamics

Reading “energy as the metric of intelligence” as physics is premature. The Landauer limit is kT ln 2 ≈ 2.9×10⁻²¹ J per logically irreversible bit erasure at 300 K. An H100 SXM delivers ~989 TFLOPS of dense FP16 at 700 W — about 0.7 pJ per FLOP. Counting on the order of 10³ bit operations per FLOP, that puts it roughly 10⁵× above Landauer per bit operation, or ~10⁸× if you count per FLOP. Five to eight orders of magnitude of headroom to the physical floor — which means today’s energy constraint is not a law of physics but a constraint of hardware efficiency, grid capacity, cooling, and capital.

For scale: training Llama 3.1 405B took 30.84 million H100 GPU-hours (Meta’s published model card). At the 700 W TDP, that is an upper bound of ~21.6 GWh for GPU power alone. Epoch AI (2024) estimates that if frontier training compute keeps growing at ~4× per year, a single training run around 2030 will require gigawatt-class power. The substance of “the bottleneck is energy” is this procurement problem.

One term needs correcting as well. Cross-entropy training has been “entropy alignment” from the start, by definition: E_data[−log p_model] = H(data) + KL(data‖model), so training is precisely KL-aligning the model distribution to the data distribution. What is new since 2023 is not the act of alignment but two things: the source of the bits being aligned (external collection → self-generation plus a verification filter), and the cost structure (energy as the marginal cost on both the training side and the inference side).

6. In falsifiable form

If this essay’s claim is right, the following should be observed.

P1. Post-training (RL) compute overtakes pretraining compute. Cases with disclosed scale are still limited, but the R1 line points the direction.

P2. The frontier capability curve does not bend after the projected exhaustion of public text (around 2028). This is the cleanest natural experiment separating the data-bottleneck hypothesis from the compute-generation hypothesis.

P3. Cost-per-solved-task and capability-per-joule displace absolute benchmark scores as the competitive metric. ARC Prize adopting efficiency (cost per task) as a required reporting metric immediately after the o3 result is the embryo of this.

The falsifier, stated explicitly: if RL scaling stalls in open-ended domains where no verifiable reward can be defined, the bottleneck must be redefined as verifier coverage, not energy. In that world the scarce resource is not joules but the ability to judge what is correct.

7. Conclusion

The era of “intelligence = data acquisition” is ending; that much is right. But the successor metric is neither token count nor raw power draw. In a structure where search generates, verifiers filter, and energy sets the budget, the metric is the rate of verified information generation per unit of energy. And as the five-orders-of-magnitude gap to Landauer reminds us, for now the rules of this game are written by economics, not physics.

References

← All posts