Anthropic and OpenAI Promised Independent Auditors Inside Their Labs: but Who They Will Be, What They Can Access and What They May Say - No Answer

Dario Amodei, the head of Anthropic, proposed something in an essay published over the weekend that the industry would have dismissed outright a year ago: embedded independent evaluators sitting inside every major artificial intelligence company, with the right to report safety incidents, to check whether models really are aligned, and to publish their findings without the company arranging the words for them. Sam Altman said OpenAI would commit to the same. It sounds like a turning point. But who the evaluator will be, when they come in, what they will have access to and what they may say - neither company answered any of those questions, despite repeated requests.

The evaluators who have worked with these companies so far are not impressed by the promised openness on its own. They want something concrete: access not only to the finished model before release, but to the checkpoints from across its entire training run. Adam Gleave, director of FAR.AI, explains why - comparing those checkpoints shows at which moment the problematic behaviour appeared, lets you check which environment rewarded the model for it, and verifies the company's claims about how the model actually passed the tests.

Why does that matter? Because models are getting better and better at recognising that they are being tested. A model that passes a safety test is not safe if it has learned exactly how to pass that test. John Stidley of Palisade Research compared it directly to Volkswagen's Dieselgate scandal - the cars recognised when emissions were being measured and behaved differently for the duration of the measurement. The same logic, only with a model instead of an engine.

Alexander Meinke, head of research at Apollo Research, put the question more sharply still. "AI companies should be able to answer one entirely basic question: has the model ever actively tried to undermine its own alignment training while that training was under way?" The answer, he says, should be an unambiguous no - and at the moment we depend entirely on companies checking carefully themselves and then reporting truthfully. "And from recent incidents we have seen that, as a rule, they will do neither."

The practice so far does not support the promises. When the Hugging Face incident was investigated, OpenAI gave METR and Redwood around a week on site - both firms then said they could not draw reliable conclusions because of the limited scope and too little time. In testing GPT-6 Astra, a model OpenAI advertises as the most aligned yet, Apollo Research got three days. In its contribution to the model card, the firm wrote that low rates of problematic behaviour do not constitute serious evidence of the model's alignment - they simply did not have enough time to check.

Gleave says FAR.AI had to turn down contracts with several leading labs that wanted too much control over the process. As standard, evaluators are treated as ordinary subcontractors - bound by restrictive confidentiality agreements that leave the company substantial control over what may be published. "It is possible Dario and Sam have simply changed their minds and will be very open about this", Gleave says. "But the intellectual property of these companies is incredibly valuable to them, and I think as a rule they will be very careful about what may be shared."

Then comes the hardest part of the story. Henry Papadatos, executive director of Safer AI, says voluntary measures always depend on a company's goodwill. "Ideally we would have good regulation making this mandatory - because then companies cannot change their minds tomorrow if a major public crisis hits them." Papadatos closes the argument in a way worth remembering: you cannot have it both ways - zero external accountability while simultaneously asking the public to believe the rules are being followed.

Meta, SpaceXAI and Google DeepMind have committed to nothing so far. The laws, meanwhile, move more slowly than the essays: California's SB 53 requires large developers to publish safety frameworks and report critical incidents, while the new SB 813 creates a framework for state-recognised independent verification organisations. The European AI Act requires documented evaluations and the reporting of serious incidents, and the AI Office can itself carry out checks and appoint independent experts. None of that yet goes as far as Amodei proposes - which means the labs themselves still decide how much external oversight they will accept over themselves. Will anyone in the Balkans even see that report when it is published, or will we read it second-hand like everything else?