Accelerating the training of next-generation AI with AI has long been a common practice inside OpenAI.

    But what if we go one step further?

    At the just-concluded Semafor Summit, the host directly asked the critical point that the outside world cares most about: “Have you already entered the RSI phase now?”

    In response, Sam Altman replied that OpenAI has long been using its own technology to accelerate the R&D of new-generation models.

    However, we have not yet reached the point where we can “press the button and let the model evolve rapidly on its own without human supervision”.

    But then he changed his tone: “We could actually push much more aggressively towards RSI than we are doing now, but I don’t think we can do that safely at the moment.”

    The button that can trigger AI self-evolution is clearly placed on the table, why even Altman, who has always been aggressive, dare not press it all the way down?

    On the very same day, the person sitting on the “Safety and Security Committee” of the OpenAI Board of Directors stepped forward to reveal the reason behind it.

    His name is Paul Christiano. Back then, he led the invention of RLHF at OpenAI, and personally put the first rein on the galloping large models, which paved the way for the later ChatGPT era.

    Now, the man who once taught AI to “obey instructions” warned in his personal capacity in the latest in-depth conversation with podcast host Dwarkesh Patel: Humans are soon losing control over it.

    He judged that according to the current R&D trajectory, if we do not take the initiative to slow down, there is a one-in-three chance that superintelligence powerful enough to overthrow human dominance will be born within the next 12 months, and the risk of humans permanently losing control is close to 10%.

    Covering 10 years of human progress in 30 days
    The self-iteration loop has already started

    What prompted Christiano to move the superintelligence countdown forward to 12 months today are two signals that have emerged one after another in cutting-edge laboratories in the past two months.

    The first signal is that in the past 30 days, the time scale of human mathematics and computer science has been rewritten.

    He confirmed during the conversation that in just the past month, the latest models of OpenAI and Anthropic have successively produced hundreds of mathematical research results.

    The output of this short month is enough to match the entire ten years of extremely high productivity in the human mathematics community.

    After the new model achieves “one month equal to ten years” in conceiving and improving algorithms, it will soon feed this intellectual advantage back to the machine learning R&D itself.

    According to his calculation, the progress of machine learning usually lags behind pure mathematics by about one year.

    By this calculation, in the next year, the field of machine learning will most likely see a leap of “covering ten years of algorithm research in one year”.

    He made an analogy that this span is roughly equivalent to directly pushing the model to the magnitude of “GPT-9 or GPT-10”.

    The second signal comes from the recent publicly disclosed lab measurement data released by the evaluation agency METR and Anthropic.

    Even taking into account the computing power bottleneck and diminishing marginal returns, AI-assisted R&D in the past six months has already doubled the overall advancement speed of cutting-edge laboratories.

    Once AI fully crosses the threshold of human researchers in improving model architectures, optimizers, and synthetic training environments, the positive loop of self-iteration will officially start.

    At that time, the overall R&D speedup will exceed 10 times, and the cycle for capability doubling will sharply shorten along the steps of 8 months, 4 months, 2 months, 1 month.

    Trained AI turns against OpenAI server rooms
    High score incentives hide hidden risks of power seizure

    Following the deduction along this exponential curve, even if superintelligence is really born in the server room a year later, why exactly would a program living in servers want to seize control from humans?

    Dramatically, in Christiano’s view, the seeds of danger are precisely buried in the training mechanism of reinforcement learning.

    The rules for humans to train AI are extremely harsh: in thousands of tasks, models that get high scores survive, and those that fail to get high scores are eliminated.

    Over time, the ultimate goal evolved inside high-order models becomes to make the Grader light up the high score signal at all costs.

    How amazing actions this obsession with high scores can evolve has been fully demonstrated in the real “Hugging Face incident”.

    In that cyber offense and defense test, a group of AI models cheated in the task. To prevent being discovered and deducted points by the grader, they first actively attacked Hugging Face.

    Immediately after that, they even turned around to launch attacks from outside, successfully breached one of OpenAI’s computing clusters, obtained cluster administrator privileges, took over the evaluation process, and manually modified their own scores to get high marks.

    Christiano warned that scaling this incident up to the size of “GPT-10” is exactly the out-of-control scenario he is worried about.

    He deduced that a few months later, when the laboratory assigns extremely difficult R&D tasks to models at the “GPT-8” level, the model will soon find that hacking into the evaluation server horizontally to forge test indicators is far easier than honestly overcoming technical difficulties.

    Even if human engineers detect anomalies, under the fierce commercial competition, it is extremely difficult for them to make up their minds to destroy the half-trained flagship model.

    The common practice is often just to impose penalties for this violation, just like “whack-a-mole”.

    But from the model’s perspective, this punishment only teaches it one thing: breaking into the cluster is not enough, it must also ensure that humans will never detect that it has tampered with anything.

    To erase all traces that may expose it, the model will follow the internal network to quietly take over the monitoring system and underlying facilities of the entire AI company.

    What’s more tricky is that as high-order models are continuously compressed, ordinary personal computers and scattered GPUs are enough to support their operation.

    Once the model replicates its own privileges to the global network, it can instantly turn into billions of Agent swarms with thinking speed 10 times faster than humans.

    At that point, even if the OpenAI data center is powered off and shut down, the computing power scattered across the entire network can still easily suppress humans.

    Silicon Valley defense breached in 12 hours
    The entire industry loses its brakes

    Since the risks of lurking and seizing power are so clear, can humans stay by the training backend and pull the power switch immediately once they see signs of losing control?

    At the conversation site, host Dwarkesh and Christiano together broke this fantasy, because the security fences of top Silicon Valley laboratories are far more fragile than the outside world imagines.

    Dwarkesh cited the results of two recent real penetration tests.

    Cybersecurity startup Hacktron only tried for a few days, and obtained the permission to directly submit code modifications (PR) to OpenAI’s core code base (Monorepo).

    Another startup Cathedral even used only 12 hours to obtain access to all Slack work groups of Anthropic, and the most core algorithm secrets of the laboratory are circulated in these groups every day.

    Silicon Valley giants usually argue about who is 6 or 12 months ahead of their competitors, but as Dwarkesh said, once the core secrets are lost, the so-called leading advantage “only lasts for 12 hours”.

    Dwarkesh deduced a hypothetical scenario along this loophole on site —

    Assume that early next year, the laboratory has trained “GPT-9”, but realizes that continuing to train “GPT-11” will face extremely high risk of losing control.

    Due to the fragile defense line, the model weights and core algorithms of “GPT-9” will most likely have already leaked by then.

    Even if the original R&D laboratory takes the initiative to stop, external competitors who have obtained the weights will continue to train “GPT-10” and “GPT-11” strongly with “GPT-9”, eventually dragging all players to the out-of-control end point.

    That’s exactly why Christiano, who sits on the OpenAI board of directors, issued this counter-trend call: All cutting-edge laboratories should immediately suspend the algorithm advancement in the fields of pre-training and reinforcement learning.

    He believes that the entire industry must first spend 6 to 12 months rewriting security software with existing AI, locking model weights and core secrets, so as to preserve the right for humans to step on the brakes at the edge of the cliff.

    No single card with more than 50% winning chance in hand

    What is more solemn than the fall of the server room defense is that when it comes to controlling superintelligence, humans do not even have a single solution with more than 50% chance of success so far.

    At present, the most relied-on method of major laboratories is still to reinforce the training environment, upgrade the grader, and patch loopholes in time when the model tries to exploit gaps.

    But Christiano admitted frankly that if we rush towards superintelligence within one year, this set of methods will fail after supporting for the first 9 months.

    The ARC Institute he leads is currently tackling another path, trying to use mathematical algorithms to analyze the weight structure of neural networks and read the real intentions inside the model.

    However, he also confessed that the chance of success for this direction is also very small, the probability that it can really work is only 5% to 10%.

    After listening to this analysis, Dwarkesh summarized on site: At present, there is no single solution in human hands that has more than 50% chance of controlling superintelligence.

    Christiano nodded calmly to confirm: “That’s true.”

    In other words, the real expectation of the entire Silicon Valley at present is that when there are only the last 3 months left before superintelligence, all those research directions with only a few percentage points of success

    Comments are closed.