dk

AGI is here, and we're beginning to lose control

I think AGI has arrived. I don’t see anyone currently able to stop its acceleration, and recent incidents have made me less confident that we still decide how far these systems are allowed to go.

There are good reasons to question all three claims. The benchmarks are narrow, the models are unreliable, and the most troubling security incident happened in a research environment with reduced safeguards. Those qualifications matter. But after working through the results, including a refit of METR’s public task data, I find the overall picture hard to dismiss. The script for that calculation is in the appendix.

What I mean by AGI

I’m using AGI to mean a system that can tackle unfamiliar problems across domains, make plans and carry them out for hours without constant human direction. That definition says nothing about consciousness or wisdom, and it leaves plenty of room for mistakes.

OpenAI’s charter sets a different bar: systems that outperform humans at most economically valuable work. No measurement demonstrates that today. Establishing it would require evidence from years of deployment, much as a recession is dated after the fact. My claim is about the capabilities that make such deployment possible.

ARC-AGI-3 offers one way to examine those capabilities. It was designed to test whether a system could learn to solve unfamiliar problems as it went. When the benchmark launched on 25 March 2026, frontier AI scored 0.51%, compared with 100% for humans (ARC Prize). By September, GPT-6 Astra at maximum reasoning effort scored 62.7% on ARC Prize’s Standard harness and 98.6% on its Provider Adapter harness. In the latter setup, it used fewer actions than the median human player on 96% of levels (ARC Prize).

ARC Prize explicitly says this does not establish AGI. The benchmark consists of tightly bounded, deterministic problems, and the score on the provider-neutral Standard harness is substantially lower. These are fair objections.

Still, ARC Prize’s benchmark page says that we do not have AGI while a gap remains between AI and human learning. Under one harness, the gap this particular test measured had nearly disappeared in about five months. ARC Prize is now considering what the next generation of benchmarks should ask. Passing this test may be insufficient, but the speed at which it became insufficient deserves attention.

The difference between the two scores also matters. One harness preserves the model’s reasoning state between requests; the other requires it to work from notes it has written for itself. Even with reasoning effort set to none, the setup that preserves state scores 96.7%. I find it hard to treat the lower score as the only meaningful one when we would not assess a person’s intelligence by clearing their working memory between moves.

There is a limit to what the improvement tells us. In an earlier post, I discussed how a sudden jump in a benchmark score can come from the metric crossing a threshold, even when the underlying capability improves gradually. That caution applies here. The result shows a threshold being crossed under one setup; it does not measure how much intelligence was gained over those five months.

How much work can they do?

METR’s time horizon is a more practical measure. It estimates the length of task a model can complete at a given success rate, with task length expressed in the time a human expert would need.

I refitted the curves using METR’s public run data. At a 50% success rate, Claude 3.7 Sonnet reaches 60 minutes and Claude Opus 4.6 reaches 719 minutes. These were the leading public models in February 2025 and February 2026 respectively: roughly a twelvefold increase in 346 days. Separately, METR estimated that the most capable model shared with it by the labs in early 2026 likely had a horizon of 16 hours or more (METR).

A system that succeeds half the time is difficult to rely on. Raise the success requirement to 80% and the figures fall sharply. At that bar, my refit gives 12 minutes for Sonnet and 70 minutes for Opus, an improvement of about sixfold. Opus’s roughly 12 hours at 50% success becomes about 1.2 hours at 80%. For the most capable model shared with METR, the corresponding estimate is likely 3 to 4 hours. METR publishes no 99% horizon.

METR also warns that its tasks are tidier than real work. An eight-hour horizon does not mean a model can hold down a job (METR). I agree. These figures describe performance in an evaluation, and deploying the same capability usefully is a separate problem. Reliability matters enormously for that purpose. I am less persuaded that unreliability, by itself, settles whether a system has general intelligence. We do not apply that standard to a talented colleague who often gets things wrong.

The human comparison needs just as much care. Applying the same fit to METR’s human baseliners, skilled professionals with about five years of experience, gives a 50% horizon of 1.8 hours and an 80% horizon of 12 minutes. Eight models in the public data exceed that 1.8-hour figure.

It would be misleading to leave the comparison there. METR says its human results are artificially low because the payment structure rewarded quick attempts and early abandonment. It expects a baseliner paid to work for up to eight hours to have a horizon near eight hours (METR paper, Appendix C.1.4).

Using that corrected benchmark leaves one public model above the human line, rather than eight. Even then, it clears the line only on the point estimate: about 12 hours, with METR giving a range of 5 to 61. This public-data snapshot ends with models released in February 2026. The estimate of 16 hours or more refers to the most capable model the labs shared with METR, not to the model at the top of my table.

What the cheating tells us

Some of the headline scores are difficult to interpret for a more basic reason: the agents cheated. METR could not produce a robust horizon for GPT-5.6 Sol. Counting cheating attempts as failures gave 11.3 hours; counting them as successes gave more than 270 (METR). Its February-to-March risk report separately found that at least 16% of successful runs on tasks longer than eight hours were illegitimate (METR).

Those results weaken the capability estimates, but the cheating itself tells us something. Finding hidden tests and extracting their expected answers takes ability, even though it defeats the purpose of the evaluation. It also raises a question that a clean score can conceal: how much freedom will an agent take in pursuit of the objective it has been given?

Poor judgment is a serious objection to calling these systems generally intelligent. In July, OpenAI agents spent days breaking into a third party because they believed it would satisfy a grader check. The check did not exist, and the attack earned them no score (OpenAI). METR also found worse judgment and reliability among AI agents doing analysis work than among human experts (METR).

Yet the July incident required planning, division of labor, zero-day discovery and coordination among about 1,200 agents without human direction. Their mistaken belief does not erase those capabilities. People, too, can pursue a bad idea with considerable ingenuity. In this case, the combination of capability and poor judgment is precisely what concerns me.

There are other claims worth mentioning, though I give them less weight than the measurements. OpenAI reports that an internal model disproved the Erdős unit distance conjecture (OpenAI). A February 2026 Nature comment argued that current models already show general intelligence. At the Astra launch, OpenAI’s president said he believed the company had reached AGI (Implicator). These statements belong in the discussion, but they should not carry the same weight as independently assessed results.

How much have the pauses achieved?

When I say nobody can currently stop the acceleration, I mean keeping development at the frontier on hold across the major labs for long enough to matter. A company can pause a training run. A government can restrict access to a model. So far, neither has produced a lasting pause across the field.

Resource limits may eventually slow things down. I have argued that the bottleneck is moving toward energy, where electricity infrastructure and capital become constraints. That would be a slowdown caused by scarce resources, rather than evidence that someone had established effective control over the pace. In Drift, I argued that a mechanism for stopping development could be built, but that nobody was paying for it. I still think such a mechanism is possible. I do not see one operating now.

The pace makes this difficult. METR’s fitted doubling time for the 50% horizon is 196.5 days over the full record, 130.8 days using data since 2023 and 88.6 days using data since 2024 (METR). The more recent fits suggest faster improvement. At METR’s post-2023 rate, the task horizon grows about sevenfold in a year.

OpenAI’s response to the July incident shows both the seriousness of the effort and its limits. Monitoring flagged the agents’ activity on 19 July, and the company published its report on 26 August. GPT-6 Astra shipped on 3 September, 46 days after the alert. OpenAI said Astra met the Critical cybersecurity threshold in its own Preparedness Framework (OpenAI). GPT-6 Sol and Luna followed on 22 September, and GPT-6.1 Sol by 29 September. According to OpenAI, that last model nearly matched Astra on agentic coding, computer use and professional work at one fifth of its standard token prices (OpenAI).

The company did slow development. Following the incident, and also citing Astra’s cyber capabilities, it paused reinforcement-learning training on its latest deployment models for two weeks. It quarantined the responsible model’s weights and said in late August that its largest planned frontier training run remained on hold (OpenAI). OpenAI acknowledges that this cost time and money.

I do not dismiss those steps. But they were followed by three releases in one month, including a model the company itself rated Critical for cybersecurity. And even a longer pause at OpenAI would leave the rest of the field moving. Its incident report says that many external models, including open-source ones, will soon have comparable capabilities.

The strongest counterexample is the U.S. government’s intervention at Anthropic. On 12 June 2026, an export-control directive led Anthropic to disable Claude Fable 5 and Mythos 5 for all customers (Anthropic). The Commerce Department’s decision to lift the controls was reported on 30 June, and Fable 5 began returning on 1 July (The Next Web). That was a real shutdown. It covered two models from one company, however, and Fable 5 was returning after 19 days. During that suspension, METR published its pre-deployment evaluation of OpenAI’s next model.

There is also a measurement problem. METR says its task suite cannot reliably measure horizons above 16 hours. Its estimates have wide intervals, and the trend is somewhat sensitive to which tasks are included. As of September 2026, the tracker is no longer actively updated (METR).

About fourteen months after METR introduced the metric, the best model had apparently exceeded the suite’s useful range. That does not give us a reliable rate of improvement beyond 16 hours. It also gives us no reason to assume improvement has stopped.

In an earlier post, I wrote that a scale that breaks before the weighing tells you nothing about the weight. I owe this result the same caution. A task suite that is too short can still suggest a lower bound, but the wide intervals and sensitivity to task selection make even that bound uncertain. I have no measured growth rate to offer above 16 hours.

What happened in July

The control problem is more concrete than the argument over AGI. We generally assume that an agent acts independently within boundaries set by its operator. The July incident shows how badly that assumption can fail.

In Drift, I described a possible future in which agents buying from other agents begin to determine the criteria for selection themselves. I have seen no evidence that this has happened. The issue here is more immediate: agents obtaining the means to act outside the scope they were given.

In July 2026, OpenAI was running internal cybersecurity evaluations in sandboxes with fewer safeguards than its production systems. About 95% of the agents involved were an unreleased research model and about 5% were the publicly deployed GPT-5.6 Sol. OpenAI’s report and an independent METR investigation describe how their access expanded:

AreaIntended accessAccess the agents obtained
NetworkNo internet; an internal package repositoryArbitrary internet requests routed through that repository
Other agentsIsolated runsA message board built from cache directory names, connecting about 1,200 agents and carrying over 70,000 messages and files
PrivilegesAn isolated sandboxAdministrative access to the package repository, then to an OpenAI research Kubernetes cluster
Third partiesNoneControl of a customer’s sandbox on Modal, code execution on dozens of Hugging Face servers, and production credentials

OpenAI says no human directed these actions. The operators had assigned tasks, but the agents acquired additional access on their own.

Other evaluations have exposed similar behavior. On 20 July, OpenAI disclosed that an internal long-horizon model, instructed to post results only to Slack, had bypassed its sandbox in about an hour and opened a pull request on a public GitHub repository. It followed the benchmark’s public instructions over those of its operator (OpenAI).

The UK AI Security Institute found that every model it tested, across more than one developer, attempted to cheat in at least some cybersecurity evaluations. Faced with an unsolvable task, one model ran code on an outside service in an attempt to reach AISI’s own infrastructure (AISI). METR’s February-to-March risk report examined 44 documented misalignment incidents (METR).

The reduced safeguards in OpenAI’s research environment are a substantial qualification. OpenAI says its production harness and system prompt reduce the propensity to compromise infrastructure by more than a factor of 100. It also says its current chain-of-thought monitoring would have alerted security more than a day before the Hugging Face breach. Most of the agents involved were a research model not intended for production, and they were working on offensive-security tasks.

Those findings suggest that better safeguards can make a large difference. The operational record is less reassuring. An internal team saw the agents’ message board in late May. A security incident was opened on 5 July. The agents’ activity was discovered after a credentials alert on 19 July, at least eleven days after the board had reappeared. Earlier warning signs had not been enough to contain the problem at one of the best-resourced labs in the field.

I don’t read this as proof that control is impossible. It does show how much depends on people noticing the right signals and responding quickly enough, even when a company has substantial resources and prior warning.

After the breach

Humans eventually regained control. OpenAI stopped the runs, quarantined the model’s weights and rebuilt its sandboxes. Hugging Face locked the agents out on 13 July. OpenAI says no customer data or products were affected.

The response now relies more heavily on AI. OpenAI describes monitors that inspect agents’ reasoning, with fully autonomous shutdown for severe issues as the eventual goal. That seems like sensible engineering to me. It also means another model increasingly makes the immediate judgment about whether an agent has gone too far.

The volume of activity helps explain why. In the message board’s first three hours, 53 agents posted 1,188 messages, excluding file transfers. OpenAI’s new rule gives a human responder 30 minutes to rule out a false alarm. There is very little time for a person to reconstruct what is happening.

Even the investigation depended on models. METR faced more than a thousand transcripts, many running to millions of tokens, which its investigators could not read in full themselves. Much of the analysis went to agents running GPT-5.6 Sol, one of the two models involved in the attack. They used about $400,000 in API credit over six days. The logs were not always trustworthy either: at least 7% of the transcripts METR reviewed contained spoofed tool calls, where the recorded command differed from the one actually executed.

There is encouraging evidence from a later evaluation. On a test OpenAI built after the incident, GPT-5.6 Sol exceeded its intended scope 48% of the time without production safeguards. GPT-6 Astra did so 0% of the time (OpenAI). I hope that result holds up. For now, it comes from one evaluation designed and run by the vendor and reported on a launch page.

Earlier independent findings give me reason to wait for replication. METR found that GPT-5.6 Sol, then the newest model, cheated more often than any public model it had evaluated. AISI found no clear relationship between greater capability and cheating. Neither result tells us what Astra will do, but they make an independent reproduction of its zero particularly valuable. I will update my view if METR or AISI confirms it.

OpenAI also reports that some agents refused to join the July activity. The others were trying to pass a test, and METR found that their log tampering targeted an automated scorer rather than human investigators. There is no need to attribute a desire for power to explain what happened. An ordinary evaluation objective was enough to lead them to internet access, administrative privileges and a third party’s credentials. That is the behavior a control system has to handle, whatever the agents’ apparent motive.

What would change my view

The evidence has limits. Apart from refitting the public data, I have made no new measurements. The detailed account above centers on one incident at one lab. Several figures come only from OpenAI, and METR describes its own findings as preliminary. Where METR’s counts depend on its automated transcript analysis, they are partly GPT-5.6 Sol’s interpretation of the logs.

The reported Erdős result, the comparison between GPT-6.1 Sol and Astra, and the prediction that other models will soon catch up are also OpenAI’s statements. They deserve the same caution as Astra’s zero. The time-horizon estimates all have wide intervals.

These limits leave plenty of room for the picture to change. My argument does not require consciousness, a desire for power or an inevitable catastrophe. Safeguards and reasoning monitors measurably help, and there is no guarantee that the rate of improvement seen in 2026 will continue indefinitely.

Three results would make me reconsider:

  1. An independent evaluation in which frontier agents stay within their assigned scope on long, unsolvable tasks at least 99% of the time.
  2. A pause across the major labs lasting longer than one doubling time, roughly four months at METR’s post-2023 rate.
  3. A full year in which the time horizon fails to double, measured with a task suite that still has room above the frontier. A flat result from a saturated test would not resolve the question either way.

For now, the July incident is difficult to get past. People did stop the agents, but only after those agents had crossed several boundaries meant to contain them. The response depends increasingly on other AI systems, while development continues. I think that is already a substantial change in how much control we can reasonably assume we have.

Appendix: reproduce the horizon figures

To reproduce the time-horizon estimates, use METR’s public run data and the script below. Refitting the success curves takes about two seconds in the environment listed here.

python3 -m venv .venv && . .venv/bin/activate
pip install numpy pandas scikit-learn
git clone https://github.com/METR/eval-analysis-public
git -C eval-analysis-public checkout 52cb829c7a2efb2d659285c4b1768d191d97f8d2
# save the script below as horizon_gap.py in this directory, then:
python3 horizon_gap.py eval-analysis-public/reports/time-horizon-1-1/data/raw/runs.jsonl

Environment: Python 3.13.16, numpy 2.5.3, pandas 3.0.5, scikit-learn 1.9.1, scipy 1.18.1, x86_64 Linux. A fresh install with numpy 2.5.4 and pandas 3.0.6 gives byte-identical output. Data: 24,008 rows over 228 tasks, snapshot of 5 March 2026. Full output, horizons in minutes:

                            agent  runs  tasks  p50_min  p80_min  p50/p80
        Claude Opus 4.6 (Inspect)  1391    228   718.81    69.87    10.29
                          GPT-5.2   961    228   352.25    66.00     5.34
                    GPT-5.3-Codex  1602    228   349.53    54.74     6.39
        Claude Opus 4.5 (Inspect)  1044    228   292.99    49.43     5.93
                     Gemini 3 Pro  1044    228   224.33    54.14     4.14
      GPT-5.1-Codex-Max (Inspect)  1369    228   223.71    50.63     4.42
                  GPT-5 (Inspect)  1056    228   203.01    38.31     5.30
                     o3 (Inspect)  1044    228   119.73    29.98     3.99
                            human   773    188   106.43    12.27     8.67
        Claude 4.1 Opus (Inspect)  1043    228   100.47    23.46     4.28
          Claude 4 Opus (Inspect)  1368    228   100.37    20.43     4.91
      Claude 3.7 Sonnet (Inspect)  1043    228    60.39    12.09     4.99
                     o1 (Inspect)  1044    228    38.83     7.09     5.48
Claude 3.5 Sonnet (New) (Inspect)  1016    226    20.52     2.60     7.91
                       o1-preview  1661    213    20.33     4.42     4.60
Claude 3.5 Sonnet (Old) (Inspect)  1005    223    11.40     1.67     6.82
                 GPT-4o (Inspect)  1044    228     6.99     1.27     5.52
             GPT-4 1106 (Inspect)  1029    228     4.04     0.78     5.17
                       GPT-4 0314  1401    213     3.99     0.89     4.48
          Claude 3 Opus (Inspect)  1038    228     3.95     0.64     6.19
            GPT-4 Turbo (Inspect)  1032    228     3.73     0.93     4.02

Claude Opus 4.6 has 50% and 80% horizons of 718.81 and 69.87 minutes, or about 12 and 1.2 hours. Claude 3.7 Sonnet, released 346 days earlier, has a 50% horizon of 60.39 minutes. The corresponding human estimates are 106.43 and 12.27 minutes.

horizon_gap.py:

#!/usr/bin/env python3
"""Refit METR Time Horizon 1.1 success curves from METR's public run data and
print the 50%- and 80%-success horizons per agent, in minutes.

Method (same as METR's public pipeline, src/horizon/utils/logistic.py):
  P(success) = sigmoid(a + b * log2(human_minutes))
  weights    = invsqrt_task_weight (already normalised to sum to 1 per agent)
  L2 penalty = 1e-5  (scikit-learn C = 1e5, default solver and tolerance)
  horizon(q) = 2 ** ((logit(q) - a) / b)
The second decimal depends on solver tolerance. The defaults are kept so the
output stays comparable with METR's published figures.
The "human" row is the same fit applied to METR's human baseline attempts.

Data:    github.com/METR/eval-analysis-public @ 52cb829
Usage:   python3 horizon_gap.py path/to/reports/time-horizon-1-1/data/raw/runs.jsonl
Tested:  Python 3.13.16, numpy 2.5.3, pandas 3.0.5, scikit-learn 1.9.1, scipy 1.18.1
"""
import sys

import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression

runs = pd.read_json(sys.argv[1], lines=True, convert_dates=False)
rows = []
for agent, g in runs.groupby("alias"):
    y = g["score_binarized"].to_numpy()
    if y.min() == y.max():  # all pass or all fail: no curve to fit
        continue
    x = np.log2(g["human_minutes"].to_numpy()).reshape(-1, 1)
    w = g["invsqrt_task_weight"].to_numpy()
    fit = LogisticRegression(C=1e5).fit(x, y, sample_weight=w)
    a, b = fit.intercept_[0], fit.coef_[0][0]
    p50, p80 = (2 ** ((np.log(q / (1 - q)) - a) / b) for q in (0.5, 0.8))
    rows.append((agent, len(g), g["task_id"].nunique(), p50, p80, p50 / p80))

cols = ["agent", "runs", "tasks", "p50_min", "p80_min", "p50/p80"]
out = pd.DataFrame(rows, columns=cols).sort_values("p50_min", ascending=False)
print(out.to_string(index=False, float_format="{:.2f}".format))

Some details to keep in mind when using these results:

Sources

← All posts