Independent AI safety evaluators are moving from the industry's margins into central roles as major labs promise to embed external auditors, yet fundamental questions about funding, access, and independence remain unresolved.
Two months ago, organizations like Model Evaluation and Threat Research (METR), Apollo Research, and Transluce operated in what CNBC describes as "a relatively sleepy corner" of the AI sector. Now Anthropic CEO Dario Amodei has pledged to embed independent evaluators in his company, a move OpenAI CEO Sam Altman endorsed and President Donald Trump supported. Most large U.S. tech companies have backed the idea.
These mostly nonprofit evaluators assess AI model capabilities and risks. Their work has grown more visible because federal regulation remains absent. Their primary role involves flagging instances where AI technology behaves unsafely.
But critical gaps persist. Suresh Venkatasubramanian, a Brown University computer science professor, told CNBC: "To a degree, the problem, as always, is money. Who is paying for these companies to do their work? How are they going to support them? You need an ecosystem, you need a viable business model for this."
The current structure creates potential conflicts. Anthropic, OpenAI, and infrastructure partners profiting from the AI boom are setting the rules. Critics compare this to allowing banks to prevent financial crises or letting pharmaceutical companies approve their own drugs without regulatory oversight.
Trump signaled he intends to leave companies to regulate themselves, encouraging them to "partner with an independent external auditor or evaluator" through a voluntary accord presented in late September. He praised AI executives for their "tremendous self-policing."
Friction has already surfaced. OpenAI last week fired three employees for "violating our policies on accessing and handling sensitive company information." Two of those employees, Mikita Balesni and Tomek Korbak, said they believe dismissal came because they communicated with third-party evaluators. Balesni wrote on X that colleagues fear speaking up and worry their personal phones will be searched for messages to evaluators.
OpenAI disputed this, stating it is "actively finalizing contracts with third-party safety assessors" and remains "committed to embedding external assessors."
The evaluator ecosystem is expanding. Vals AI announced a $40 million funding round in August and grew from eight to roughly 30 employees this year. METR raised commitments of around $71 million over six months, compared to $13.6 million for all of 2024.
Kevin Werbach, faculty director of the Wharton Accountable AI Lab, told CNBC the ecosystem is "not robust enough right now." METR employs fewer than 50 full-time staffers. The power imbalance between small evaluators and labs that have raised tens of billions raises independence concerns, Venkatasubramanian said.