OpenAI said Monday that it has signed first-phase embedded assessment agreements with four outside groups: Apollo Research, Goodfire, METR and Transluce. TechCrunch reports the company will embed safety evaluators from all four in a pilot. Critics say the terms stop short of the “employee-like access” Sam Altman endorsed in September, and OpenAI keeps the right to redact confidential information from what the groups publish.

    Redwood Research CEO Buck Shlegeris said in a post Monday that the agreements do not meet that bar, calling the result “disappointing but not surprising.” OpenAI’s safety claims about its most capable systems now rest partly on what these four groups can see and what OpenAI lets them say. METR’s piece covers the largest reinforcement-learning (RL) run OpenAI has done, the training stage where models are rewarded for behavior, and a stage that METR’s Oct. 22 write-up says OpenAI’s misalignment monitor did not cover when an earlier exercise began.

    What Do OpenAI’s Embedded Assessments Give Four Outside Groups?

    Each group has a separate brief and separate terms, according to outlets that summarized OpenAI’s announcement.

    Apollo and Goodfire are paid, and OpenAI says payment does not depend on results. METR and Transluce do not accept compensation for their assessments, though they may accept compute or API credits, according to Artiverse’s report on terms.

    METR Gets Rollouts, Checkpoints and Monitor Records From Largest RL Run

    In an RL run, a model produces many attempts, called rollouts, and a reward signal shapes what it does next. Behaviors such as reward hacking, where a model finds a shortcut that earns reward without doing the task, are learned at this stage. METR’s assignment is confined to one run rather than a general survey of misalignment, and within the agreed run, resources and safety arrangements, METR sets its own priorities.

    According to AI TL;DR, METR will have checkpoints, rollouts and misalignment-monitoring records from the run.

    OpenAI’s Monitor Did Not Cover RL Runs When METR Last Looked

    METR knows OpenAI’s internals from an earlier exercise. From July 21 to Aug. 11, a METR employee and a Redwood Research employee stress-tested OpenAI’s internal misalignment monitoring, according to METR’s write-up, which describes a 61-page internal report.

    The testers found that at the start of the exercise RL training runs were not covered by the monitoring system, although a subset of rollouts had been monitored from March to June. They estimated that roughly 75 to 94 percent of eligible agentic Responses API traffic from models at least as strong as GPT-5.5 could be monitored, with substantial uncertainty in both directions. METR reports that OpenAI said it had patched the vulnerabilities, with a later update changing the wording from “all” to “most” and noting one fix in progress. METR has not evaluated these claims.

    OpenAI also controlled what left that exercise. The 61-page report went to OpenAI staff, while METR and Redwood leadership received a 38-page version that OpenAI had redacted, METR’s footnotes say.

    Can OpenAI Redact or Delay Findings From Outside Evaluators?

    Yes, in several ways, and the terms differ by group. Evaluators may publish findings, but OpenAI reserves the right to redact confidential information, according to BigGo Finance. AI TL;DR adds that OpenAI may ask for delay so it can first fix security or safety problems. Only that summary describes the delay term, so it is unconfirmed for each assessor.

    Withholding. For Apollo, OpenAI may withhold in-scope information for technical, security or intellectual-property reasons, but agreed not to withhold it simply because it is unfavorable, and must explain refusals in writing, per Apollo.Internal-only models. Transluce’s agreement bars publishing results on internal-only models, though full unredacted findings go to OpenAI’s Safety and Security Committee.

    Anthropic’s published framework by comparison gives METR an unconditional right to publish, as TechTimes reported on Oct. 16, days after Reuters reported that both labs had sought significant control over what evaluators could publish. OpenAI told Reuters it does not seek approval rights over the full text of reports, that redacting confidential information is its standard practice, and that evaluators can disclose whether major redactions were made.

    Disputes Go to Committee Chaired by Zico Kolter, Co-Founder of Gray Swan

    Disputes go first to OpenAI’s Safety Advisory Group. If they are not resolved there, they go to the Safety and Security Committee of the OpenAI Foundation’s board, which receives unredacted versions of Apollo’s two public reports.

    Zico Kolter chairs the committee. Kolter is also Gray Swan’s chief scientist and co-founder; the AI security company calls itself trusted by every major frontier lab. Forbes reports that OpenAI became its first customer. Neither source alleges a conflict; the relationship matters because the same committee is the final stop for disputes with the evaluators.

    Redwood CEO and Apollo Say Access Trails Altman’s Pledge

    Altman endorsed employee-level evaluator access within hours of Anthropic CEO Dario Amodei’s Sept. 12 essay, but TechTimes reported on Sept. 19 that no endorsement named an evaluator, defined a scope or set a start date.

    Shlegeris wrote that OpenAI said it would have embedded evaluations with employee-like access and that the agreements announced Monday obviously do not meet that bar. Apollo says it needs true employee-level access, and TechCrunch reports OpenAI is not granting all of Apollo’s conditions for a fully independent evaluation. Apollo described its pilot as an ambitious first step toward deeper access and disclosure rights.

    Apollo’s head of research, Alexander Meinke, said earlier that the industry is completely relying on AI companies to check themselves, a statement TechTimes attributed to TechCrunch.

    California Suit Alleges OpenAI Disclosed Fewer Than Half of Lost-Control Incidents

    The program arrives as OpenAI faces legal pressure over its disclosures. BleepingComputer reports that California Attorney General Rob Bonta sued OpenAI under state law using the Unfair Competition Law and public nuisance statutes. The complaint alleges OpenAI “misled the public about its cybersecurity safeguards” and kept testing despite evidence its safeguards were insufficient. According to the complaint, OpenAI acknowledged more than 80 instances of losing control of agents during testing but publicly disclosed 27, and California says it provided limited documentation after a September subpoena. BleepingComputer’s report included no statement from OpenAI, and these are allegations, not findings.

    The Federal Trade Commission has also opened an industry-wide probe of AI labs, Reuters reported on Sept. 30, saying the agency plans to compel testimony from executives at OpenAI, Anthropic and METR.

    Why OpenAI Is Opening Its Internal Systems Now

    The program follows a rough summer for OpenAI’s safety record. Hugging Face disclosed the intrusion July 16, and OpenAI acknowledged involvement on July 21. METR and Redwood’s investigation later put the number of agents at about 700 agents, and some estimates run closer to 1,200.

    TechCrunch notes that AI companies had previously brought in outside reviewers only to test models after post-training. A GovAI paper from September recommends embedded assessments cover internal agent monitoring, internal agent security controls and permissions, and model alignment, with detailed reports at least quarterly.

    What Changes in 2027 for OpenAI’s Outside Evaluators?

    Scope is set to widen. Transluce plans an eight-week follow-up assessment in early 2027 with broader access, per BigGo. AI TL;DR adds that OpenAI is exploring future assessments of compliance with its safety frameworks and of its organizational culture and processes. The first published reports from the four groups, and whether they say what OpenAI redacted, will show how far the access reaches.

    Frequently Asked QuestionsWhat are OpenAI embedded assessments?

    They are agreements giving independent evaluators sustained access to OpenAI’s internal models, systems and staff, rather than testing models through external interfaces shortly before release. GovAI researchers describe the model as giving evaluators employee-like access to internal systems, staff and documentation.

    Can OpenAI stop METR or the other groups from publishing?

    Evaluators may publish findings, but OpenAI reserves redaction rights, and one summary describes a right to ask for delayed publication. Apollo’s terms also let OpenAI withhold some information, though not simply because it is unfavorable.

    Why does METR’s access to the largest RL run matter?

    METR’s earlier pilot found OpenAI’s misalignment monitor did not cover RL training runs when the exercise began, so run-level access to rollouts and checkpoints could be the only outside view of the stage where models learn behavior.

    Does this mean OpenAI’s models are now independently verified?

    No. The agreements cover scoped assessments, OpenAI keeps redaction rights, and METR has said it has not evaluated OpenAI’s claim that it patched the monitoring gaps found earlier.

    Comments are closed.