You're right that METR isn't fully independent, and doesn't have the staff to embed enough evaluators in every frontier lab. However, I think you're mischaracterizing Anthropic's position, and indeed METR's.
- Anthropic is recommending that independent evaluators be mandatory via government regulation. They gave METR as a single example, not as the only evaluator they'd accept. Presumably regulation would not give them a choice, and multiple evaluators would be embedded.
- METR is part of the rationalist/EA sphere, yes, because that's the group of people that took risks from AI seriously and developed the expertise necessary to evaluate those risks well. But that doesn't mean they're a secret arm of Anthropic - I think you're underestimating the very wide differences in opinion on how AI should be developed, or not be developed, within this sphere. More to the point, METR wasn't founded (to my understanding) with the intent of acting as any kind of legally independent regulator. They still describe themselves on their website as "We help AI companies and wider society understand the capabilities and risks of advanced AI systems." They were previously best known for research on how fast AI capabilities were scaling; that they've become a candidate for a regulator-like position is a new development.
I also get the sense your original motivating objection was that METR agrees with Anthropic that the biggest risks from AI are existential[1] - that they both think about AI in ways informed by rationalism/EA, and are thus part of the same "ideological blob". That is a legitimate objection to be clear - if the evaluator and the evaluated see things in the same way, there's a risk that they both miss something. But please just say so.
IMO, the reason why Anthropic likes the idea of METR as an evaluator is not that they'll just give Anthropic a free pass - I certainly don't expect that they will - but that, when encountering something like the Hugging Face incident, they'll focus on the most important parts, AI capabilities and alignment, and not just the prosaic security/operational failures. The prosaic parts can and ought to be covered, to be clear, but METR-like orgs will play an invaluable role in keeping the public/government/industry informed on the biggest risks. That's why, at least personally, I'd still like to see them embedded even if they aren't independent.[2]
It would be good to have fully independent embedded evaluators though; I don't think you'll see too much disagreement to that specific ask here, or indeed from Anthropic or METR.
- ^
Besides just the sense I get from reading this post that there is a bit of an ideological axe to grind here, you said under six months ago here in Against Doom & Pause AI that:
In my opinion, AI is roughly a normal science like physics or biology, and is dangerous in the same way those fields are dangerous but perhaps more so. (...) AI risk should be mitigated in a similar way to how we mitigate risks from other sciences and engineering projects.
I presume you would like to see embedded evaluators which share your view rather than Anthropic's/METR's, and which are more traditional, with traditional expertise in ex. cybersecurity. That's reasonable.
- ^
Of course, I am a rationalist and effective altruist myself, so I'm not an independent voice here.
That is a legitimate objection
I mean, sort of? The argument for embedded evaluators itself depends on the belief that AI could go very badly wrong, in ways that are unlike how most other human endeavors can go wrong. If you don't believe that, why do you think that embedded evaluators are necessary? Certainly you might still try to make an argument for AI being consequential enough to justify that kind of measure, just not existentially consequential... but probably you'd be thinking about a very different regime, such that trying to start the argument from this point (rather than backing up a step and arguing about what you think the actual risks posed by AI are) is very confusing[1].
Separately, responding to OP:
METR appears to have, mostly, cut that tie, but its most recent funding includes ten million dollars from a RAND Corporation program that was, in turn, funded by Coefficient Giving. [7] I am going to hazard that the terms of that grant were such that there were only a few plausible grantees here, and METR may well have been the only one. This feels like a shell game: I can't prove that this money was given to RAND on the tacit understanding that it would end up at METR, but it probably was.
This is mendacious. METR got $17 million from Audacious in October 2024. At the same time, Audacious gave RAND money for the "joint program" with METR. 11 months later, in September 2025, CG gave RAND $10m over three years to support that program. At no point did RAND ever send METR any money. Unless you think RAND and METR were getting up to accounting shenanigans about who was paying for program expenses, employee salaries, etc, there is no way for that $10m to have ended up at METR.
See also the the full thread at https://x.com/Laneless_/status/2099212740959379722, which OP participated in.
- ^
Generously.
I don't think you need x-risk to justify embedded, independent evaluators. Dario's post gave the example of the banking industry, and the nuclear industry also has resident inspectors. Even the USDA has inspectors embedded in food processing plants.
My view is that METR is not a meaningful check on Anthropic, but that no other organization would be either. Almost every other organization would be even worse; the exceptions are Redwood, CAISI, UK AISI... and maybe that's about it.
What are the barriers to spinning up a new org conditioned on money? You need a mix of domain experts and market-salaried engineers or researchers, right?
EA pushes the "talent-bound" narrative (e.g. https://forum.effectivealtruism.org/posts/JrJWZK7Mqj3hS5xTk/new-talent-is-coming-to-ai-safety-what-happens-next), I mostly think talent will appear just fine when money appears.
Money has not been the bottleneck for METR for years, it's domain experts. There are probably only like 80 people in the world who could lead a propensity evals team, with 20 of them already working in AI safety but with extremely high opportunity cost, and 60 scattered across academia and industry, mostly not motivated by the prospect of this kind of stressful nonprofit job even if it pays fairly well, and/or likely to not be hired because interviewing for this is hard.
Thanks. n=80 bottleneck is a big deal (I'm still curious, in lower level detail, why it's more than 6-24 months of rampup). I do notice you said "fairly well" and not "very well" and "very well" might be on the table.
It's not extremely much rampup, it's just that AI is moving very fast and no one thought about propensities that result in current misalignment 24 months ago because models were much less capable. The intersection of that with people who are future looking towards behaviors that could cause existential risk is small and concentrated in EAs. It's doubling every year but in three years we might be dead.
The marginal value of increasing pay for METR is fairly low for three reasons
- Industry salaries are extremely high and hard to compete with (eg counting equity, I know someone who stayed at METR despite 8 figure offers elsewhere)
- People don't actually care about money very much
- Increasing pay means you need to increase everyone's pay or be unfair
They did announce a pay raise shortly before I left, which presumably helped recruitment and retention somewhat, but my guess is the marginal effect of higher pay is much smaller than in the average industry.
Sorry, I still have the same crux. "Is looking towards x-risk behavior that important of a trait?" I don't think so. I think the "yes" answer is either "yes, it's siloed specialization" or "yes, for some culture, character or curiosity reason."
Technical / specialization half:
I'm interested in legibly seeing the barriers to being a METR-level AIS researcher. Like, someone show me a syllabus, let me try it, and find out where it gets hard even if it looked easy to me at first. Positive outcomes of making it legible might include:
- "oh hey I have a buddy who dealt with that in a seemingly unrelated domain, they should check this out." (Like the Fields Medalist who leapt to OAI to solve alignment.)
- "yeah this looks hard but there's a lot of bite-sized pieces people might take up."
- "ok, so we can make sure we have more talent in N+2 years. While we have pressing needs sooner, it would be really weird and dumb to make it to 2028 and find we didn't improve our bench."
Character half:
- surely the number of AIS-pilled STEM people has increased quite a bit in the last week.
- I also think engineers just have working knowledge of AI misalignment: they watch it disobey instructions and take suspicious shortcuts all the time. I think this is valuable on deep misalignment problems, because it's a lot of intuition, curiosity and open-mindedness being established already.
- meta / ad hominem: rats/EA can get a little sentimental about their privilege in being first to AIS. I'm happy to lose the argument (you / Daniel K have pushed back on me) but since I think there's groupthink bias that being in on AIS is really important, I should speak up.
Here are some challenges METR faced recently, in no particular order:
- Make the next time horizon graph now that time horizon is saturated. That is, now that human time is a poor predictor of difficulty-for-AI of a task, figure out an intuitive metric that continues to correlate well and remains tractable to measure even for superhuman AIs.
- Devise a benchmark of gradeable (either automatically scorable or human scored with a >100:1 agent:grader effort ratio) tasks that will not saturate for 2 years, to support the next time horizon graph. Remember most of these saturate in months now.
- Investigate the HuggingFace incident as well as Ryan, Ajeya, and Hjalmar did in six days.
- For the first-ever risk report which quantifies catastrophic risk from AIs in the next 12 months from 4 frontier AI companies, define threat models towards catastrophic risk and develop a methodology to forecast how likely those threat models are using only information the companies are willing to give you.
- Within a few weeks of learning about the opportunity, spend 3 weeks red-teaming Anthropic's monitoring and security and find "several routes to disabling monitors", and assess whether monitoring will be sufficient against future agents, as well as David did.
- Construct alignment evals, or use observational data somehow, to predict the next HuggingFace incident in advance. Remember that OpenAI and Anthropic did not have alignment evals that could predict HF, because it was so out-of-distribution.
How big can a team be in this sense? Can each of those 80 strategically lead like 60 people right now if given management assistance, or can they only lead six each?
As of last month METR neither has the management capacity nor the engineers to have a team of 60 people, so probably more like six? Not sure what it would be with management assistance.
Talent/expertise is the main barrier. The domain experts are mostly working at AI safety organizations already, or else working at the big companies. Plenty of people have ML or engineering degrees, but if that alone made you at all competent to evaluate misalignment and AI risk threat models then we wouldn't be in this mess.
I'd be curious to learn more, like, something resembling a syllabus, or learning where the hurdles are. From my current knowledge (B.S. math / now SWE), solving inner alignment sounds like "yellow book" math, but designing experiments that make models divulge behavior like sandbagging sounds accessible.
It's not that hard to skill up. The field is growing. I don't have a syllabus off the top of my head but "AGI Safety Fundamentals" is an old one that's probably still relevant.
TY. I appreciate your clear answer. It also surprised me -- I think EA stresses it's a narrow field, and Thomas Kwa in this thread argued the side "it's hard to get into it". It may be worth endorsing studying AIS more widely or clearly, or clarify who you would encourage.
It's easy to get into it compared to many other fields, I'd guess. No four-year-degree required. The problem is that so very few people chose to get into it over the past years, basically only EAs and rationalists and company employees, and we don't have much time left. Also, to distinguish the people who genuinely upskilled into AI safety stuff from the charlatans and fools who claim to have done so, is a nontrivial problem -- basically it'll be hard for non-experts to recognize true expertise, and the experts are few in number, very busy, and mostly EAs and rationalists and company employees.
Holding loosely: is anyone competent to evaluate alignment or AI safety? I don’t think anyone but EAs/rats have even tried? And in a certain sense, every attempt to measure or define alignment has failed - so far we can only see its absence, and generally that’s in a “I know it when I see it” sense. Edit: Or put differently, alignment is pre-paradigmatic.
Maybe what I’m grasping at it is something like: What sort of perspectives do we get if others start to appreciate the risk, but don’t align with EA/rats?
The "talent-bound narrative" is mostly bullshit? https://forum.effectivealtruism.org/posts/B6d8Wzk4gNzHsXvdi/ai-safety-is-extremely-bottlenecked-on-grantmakers#Jf8RWS8YcsPT8XqvM
If they're getting 1500 applications for a single role, then they ought to be hiring at least 3 of those people... or admit that they're not talent-constrained.
Did they not? The role is now closed, I assume they filled it and see no reason to think they didn't. https://coefficientgiving.org/about-us/careers/.
If the other 1499 applicants were qualified, this strongly suggests we are not talent bound, does it not?
If they're not qualified, then it's still fully ambiguous whether the pay is too low or the talent market is truly dry. For comparison: if they pay $13/hour they'll get 15000 unqualified applicants, and it says nothing about the presence of qualified applicants, only the presence of qualified applicants at that pay rate.
Also I assume the grantmaker role is different skills. If I'm misunderstanding that role (since it's sort of meta regarding talent) and that's messing up my answer, let me know. Daniel K and Thomas Kwa have been replying to me regarding something more like a lead technical safety researcher.
I think we're broadly in agreement. My point, which I apologize for not spelling out explicitly, was something like "even if only 1 in 500 applicants were qualified, they'd have been able to hire multiple people for the role, thus planning ahead instead of just-barely keeping up"
But the actual revealed process/preference is "rely on references from a small inner circle" https://x.com/BogdanIonutCir2/status/2091784358562590941
Dario is calling here for the AI industry to self-regulate.
The next section of Dario's essay, 'Pacing Within Democracies' contains this paragraph:
The most effective method of pacing is via regulation that targets all US frontier AI companies, as that covers even those who are unwilling to cooperate voluntarily. Anthropic has long supported sensible and targeted AI regulation, specifically bills that focus on transparency and on third-party auditing. I believe all frontier labs should partner with government to formalize the idea of permanent embedded evaluators to better prevent and document internal alignment incidents like those that have occurred in the last few months, and to implement regulation focused on keeping capabilities in balance with safety.
So this seems straightforwardly wrong.
Given that the set of people sufficiently qualified to provide oversight overlaps heavily with people who are "hopelessly conflicted", I'm curious what pool of people you'd realistically propose drawing from to get your independent auditors.
The problems you're pointing out are real, but I hope we can start with some oversight at all and work on improving it from there rather than holding out for something better that may come too late.
I think this is a very valid point and should be more widely known, though I think in the interests of fairness its important to point out that concerns about interdependence between regulatory staff and the regulated industry is common to many industries; the 'revolving door' between banking and banking regulators had been discussed a lot (where regulators know that there is the potential of a high-paying job in their target industry, if they 'play nice'). I think the situation here is both a bit more high -stakes and a bit more extreme though.
For fairly obvious reasons, "not wanting yourself and everyone you know to be dead or enslaved in a few years" tends to take precedence in people's priorities over "the prospect of a fat paycheck in a few years". There are situations in which people start all pulling together — the prospect of death tends to do that. I know cynicism is hip, but I think the OP has lost the plot here.
by op do you mean me or the original post? I think I was more responding to the fact that the original poster was maybe a bit over-optimistic about how independent regulators can ever be of the industry they regulate (so I think that while I think a lot of the concerns about METR in the post are worth considering, we should maybe not assume that banking regulation or medical regulation is totally free from conflicts of interest).
while I definitely agree that we should prioritise AI risk over a fat paycheck, the fact is that many actors in the space are historically well modelled by them seeking a fat pay check (sometimes denominated in a different currency, like status) compared to modelling them as perfect altruists, so I think that it's worth being a bit cynical about Anthropics proposals on how they should be regulated. Certainly I think a bit of cynicism is merited given their past plans (e.g I am old enough to remember when anthropic wasn't going to push the capabilities frontier, because I am four years old). I read the post as pointing out that maybe being regulated by METR suits anthropic leadership just fine.
In any case, if we could simply rely on people not wanting to die as motivation, we wouldn't necessarily need any regulation, as people could simply co-ordinate without intervention. But in fact, we probably do need some kind central regulation in order to co-ordinate, partly because people simply have different beliefs (e.g not everyone even at anthropic agrees how bad the risks are or how they should be mitigated) or different prioritisation of x-risk, which leaves a lot of room for motivated reasoning
OP = "Original Post"
IT financial sector in Luxemburg here. It depends of the kind of audit that's being expected and which part of the company that will be audited. The issue I see for the Virtual Entities (AIs) industry is there's no official State Regulator imposing strict rules, guidelines or enforcing laws. This also means that it's not going to be easy for e.g. METR to audit Anthropic properly in my opinion.
Auditors (especially CSSF here in Luxemburg) are simply coming over and tell us, show me this and that from their list. Afterwards they send their findings what is ok, what needs to be worked on and what is not ok and needs to change by date X.
What also will have to be defined is that if there's a need to have different audits, between the business aspect of Anthropic (what does the AI do and doesn't do) -Business Audit- and technical (IT) audits.
Audits also don't tell you how to change, that's the audited company's problem afterwards. They simply tells you that needs work or change and gives you a date.
If we would take the HF incident as example it would require at least two separate audits. A business audit to identify what went wrong and needs to be changed with agents forming a hive and a technical audit for all the IT shortcomings that resulted in the attack on HF.
I didn't know how much I could post as this was my first comment here on LessWrong.
There's although an interesting aspect about regulations that the virtual entities world may have to keep in mind. While their industry isn't directly regulated they are still bound to other sectors regulations where their virtual entities have an impact. During the HF incident it was all in house OpenAI and Hugging Face but it would have been very different if the hive went after Nasdaq's infra because at that moment OpenAI would have hit the highly regulated financial sector. As I mentioned Yesterday, regulators aren't interested in excuses or mutual agreements, they only want answers to their questions and make sure the law and regulations are being properly followed.
While I'm convinced the frontier labs and their employees would adhere and follow imposed regulations by an official regulator they would still face the issue to have their virtual entities, alas AIs, adhere to those regulations too. We aren't dealing with classic software producing the required logs and audit items based on IF/THEN/ELSE, CASE/IN, FOR/NEXT, ... but with virtual entities, pattern seeking their most adequate solutions through their inner representation formed patterns. In my opinion the VE world may one day face the issue that they will have to truly bring proof of alignment working and CoT being reliable enough to regulators and or the law.
Don't let the perfect be the enemy of the good. Who are you proposing instead? UK AISI?
You’re completely right, but I’m tempted to fall back on the old aphorism: beggars can’t be choosers. Other than METR, are there any organizations you think could currently fulfill the auditor role?
However you spin it, the optics of this is atrocious -- didn't see you mention that in the post. It's not just strong social ties.
Just look:
1. Person leaves Anthropic over safety concerns
2. Planning to join METR
3. Quite soon we hear that METR employees will be evaluating/auditing anthropic safety-wise with employee access.
Am I correct here that METR's embedded asessor for Anthropic will be... ex-Anthropic employee?
>Am I correct here that METR's embedded asessor for Anthropic will be... ex-Anthropic employee?
No, you are not; you are merely speculating.
I am indeed speculating, but I am not incorrect, until someone from METR or Anthropic will make it public, which I will hope happens soon. I would love to be incorrect here!
If Joe still has Anthropic equity, then that's a listed example of a tier 3 COI so would have to be disclosed in any Anthropic investigation he's involved in (and also non-Anthropic investigation unless a non-conflicted employee takes responsibility for his work).
(Ex-employee without equity isn't listed as an example so would fall back to the more general principles discussed, especially "Write down the full details of any potential conflict(s) of staff working on the risk assessment, and how those staff contributed. If we published that in full, would a reasonable person not affiliated with METR have materially less trust in the assessment?")
He gave up equity.
Thanks!
Anyone with a brain (which includes METR) would use a mix of ex-Anthropic people, for knowledge of the terrain, and respected people with as few links of any sort to Anthropic as possible under the circumstances, for impartiality and credibility.
It remains the case that the frontier labs tend to hire the most qualified people, and that qualified people who care about AI safety willing to work at a frontier lab tend to end up at Anthropic, because they have a record of taking safety more seriously than the other frontier labs.
I do think that METR should avoid using ex-Anthropic employees as people who are directly embedded at Anthropic. The optics of this are indeed bad, and it is important for METR to remain an entity that the public can trust.
Probably shouldn't have ex-lab people embedded in the lab they just left. But it seems overwhelmingly positive to have ex-lab people working on evaluations and threat research in general (the fact that one left the lab in the first place is an indicator that they're not totally value-captured)
Please explain how this would not be a conflict of interest?
Assuming everyone involved has even half a brain (and neither METR nor Anthropic are known for hiring stupid people), their interest in not all dying massively exceeds conventional financial conflicts of interest. More stock or money is useless if you, your family, and everyone you know is dead or disempowered. This situation is like a war or a natural disaster: there is a strong, objective reason for anyone who understands it (i.e. who isn't an e/acc loon) to pull together.
From which it was fairly clear, even before skimming their other posts to confirm it, that OP is a "very sane" AI-risk denier. Anyone else would look at this and go "Conflict of interest? What conflict of interest? Are you in fact human? If so, your a** is on the line." rather than writing a post about it.
I would add to this that, as best I can tell, METR seems to be staffed from the same distribution of people as Anthropic, likely to have the same biases and methods, and to miss the same things, such that their ability to meaningfully improve on Anthropic's own safety measures is limited.[1]
In my view, the next-closest step to regulation by a governing body that understands the issue at hand and can independently draft regulations that are both useful, fair (allaying concerns of regulatory capture), and benevolent would be much more pluralistic. Something like allocating a budget and an API for independent review by a much broader set of interests. At minimum, I would think state governments, various countries around the world[2], and a wide swath of political parties would be included, receiving education on the current slate of mechanistic interpretability techniques and the ability to schedule experiments (under Anthropic's supervision, such that results can be used for gauging alignment but not for distillation).
For example, suppose that I am the Chinese consulate for AI. I've got a couple dozen reasonably smart CS grads from Peking University, Anthropic runs them through the slate of literature on mechanistic interpretability techniques in a 2-3 week class, and they spend their time looking for potential biases and alignment concerns, whether narrow or broad, that might affect their people and humanity as a whole. If they spot a problem, they write a script that looks at things like neuron activations, or token logits, or other things that users don't get access to, Anthropic reviews this script and confirms it's written in good faith[3] and run it, sending back the results.
Something like a global FOIA for AI companies.
- ^
Consider Swiss Cheese Security when the holes are all in the same places.
- ^
Including, ideally, countries that aren't on the best terms with the U.S.. Showing good faith to e.g. China, with its one billion people, is obligatory if you claim to be building a god that will impose your - or its - will on all of humanity.
- ^
Whether or not they believe the alignment concern is legitimate - they only deny scripts that are meant to facilitate distillation or corporate espionage.
As far as I understand Dario, he meant that METR or any other future auditor of AI labs would inevitably receive extra rights like deep access to training runs which METR demanded from future AI labs after the un-eval of GPT-5.6 Sol.
I think Dario suggesting them as a third-party evaluator analogous to a banking regulator amounts to a malicious lie
I don't get the impression that he's claiming this is as at least as much oversight as banking auditors, in dimensions including independence, etc.
In fact, the words independent/independence do not even appear in this section of the Pacing the Frontier post.
Any analogy he's claiming is to the structure - that's what the precedent exists for.
I wonder if the existence of evaluators at all will have a meaningful positive impact regardless of the competence of said evaluators.
https://www.anthropic.com/institute/measuring-pace-of-ai-development
One could read the above post as Anthropic trying to get ahead of possible problems revealed by evaluators, they sure spend a lot of paragraphs explaining why 6% of research compute to safety is more than it sounds.
"Safety research tends to use less compute than frontier training runs by its nature, so compute is an imperfect proxy for how much a company focuses on safety. This is because safety research consists of individual researchers designing experiments, which is time-consuming even though running the experiments is not particularly compute-intensive. The value of this metric, therefore, is less the absolute numbers and more that it provides a straightforward mechanism to compare like with like, across developers and over time."
Not an expert but I could easily see the opposite spin being true: "safety researchers forced into time-consuming hand-crafted experiments because they're bottlenecked by compute allocations."
The quoted segment in Anthropic's post is:
embedded third-party evaluators (such as METR)
Anthropic never did mention METR by itself, only third-party evaluators as a category. I don't think this quotation is an appropriate opener to a post otherwise exclusively about METR.
(This is a narrow comment on the first quote)
— Dario Amodei, “We Must Pace the Frontier”, September 2026.
METR is not capable of being a meaningful check on Anthropic. METR is not meaningfully independent, is not sufficiently staffed, and has no authority over Anthropic that cannot be revoked at Anthropic's discretion. Suggesting that embedding METR into Anthropic would be a meaningful check on Anthropic is so suspicious that it looks like an attempt to evade oversight and to sabotage attempts at oversight in general.
If Dario does not really mean to suggest that METR could be expected to meaningfully check Anthropic, or he actually believes that METR could check Anthropic in the same way a government regulator can check a bank, something has gone badly wrong in formulating and communicating this policy.
Why METR Cannot Check Anthropic
You don't have to take my word that they're closely tied to Anthropic, you can take METR's:
I appreciate METR's professionalism and candor, both in general and in this report, because those are not always easy to find. I do think that this somewhat understates the problem: If you required METR employees who dealt with Anthropic to be "independent" the way an accountant doing an audit is legally required to be, METR would probably have to recuse itself from consideration for conflicts of interest. An accountant, for example, would generally be considered conflicted if they had lived with or dated the people they were auditing. It would also be important that the management of the accounting firm, controlling the accountant's career, did not have that sort of history with management in the firm being audited.
This proposed arrangement, given the choice of METR, is nothing like embedding a regulatory supervisor at a bank. Bank regulators work for the government and will send bank employees to prison if they do the wrong thing. METR employees would mostly be hanging out with their friends or their boss’ friends at work while drawing non-profit salaries.
If METR wants to establish that it isn't hopelessly conflicted here, they should hire an accounting firm to publicly review them for conflicts of interest or to sue me for saying they have them. I think Dario suggesting them as a third-party evaluator analogous to a banking regulator amounts to a malicious lie intended to sabotage regulations and audits, and METR's failure to correct the record makes them complicit in the lie.
It seems strange to even call this regulatory capture. They are not at all regulators, and they are not being captured; they were never meaningfully independent, and were never regulators. Dario is simply asserting that they are like regulators. This is so far from even being plausible that it is baffling that anyone would say it. If this arrangement came up in court it would be called incestuous, and if this is the basis for anything like a regulation or a legally binding agreement between companies it will come up in court.
This is my first and most serious objection to this suggestion, but there are more.
Staffing is also a problem here. METR does not have sufficient staff. Point blank, as a matter of numbers, there are not enough people at METR. METR tells the press that it has about 35 employees, [2] and I don't think more than a tiny fraction of those employees are directly working on model evaluations. I am not entirely sure I could start a poker game with METR's relevant staff. Team sports are right out. Anthropic employs thousands of people and has millions of users. Due to the relevant numbers alone, this is not a serious regulatory proposal. This is a fig leaf, and maybe the plan is to fake it til you make it, to try to hire a ton of people and make this reasonable in the future even if it isn't now, but that is not good enough given the seriousness of the situation. Anthropic is a trillion-dollar business and one of the most important single endeavors on Earth. Even if they were totally independent, METR could not meaningfully check or even record what Anthropic is doing.
Next, METR's finances are not meaningfully independent of Anthropic. This applies two ways: first, METR refuses cash compensation from AI labs for its services, but it accepts tokens from those labs for research purposes. [3] As it happens, tokens are worth cash; in fact, they are the main product the labs sell. [4] METR is quite professional, so I am positive they would never re-sell tokens they were given for research purposes, but if they did they would be able to make many millions of dollars doing so. Refusing cash but accepting tokens worth millions of dollars may be a reasonable ethical position, but it is not being "independent".
The other issue with METR's financial independence is tricky, complicated, and annoying to explain, but we're going to try anyway. Coefficient Giving is the main entity disbursing money for non-profit activity concerning AI, its primary funders are investors in Anthropic, [5] and Coefficient's former CEO is named Holden Karnofsky. Karnofsky is directly thanked in some of METR's early work, currently works at Anthropic, and is married to Anthropic's President, Daniela Amodei. Coefficient Giving, which was then called Open Philanthropy, funded the Alignment Research Center, from which METR spun out. [6] METR appears to have, mostly, cut that tie, but its most recent funding includes ten million dollars from a RAND Corporation program that was, in turn, funded by Coefficient Giving. [7] I am going to hazard that the terms of that grant were such that there were only a few plausible grantees here, and METR may well have been the only one. This feels like a shell game: I can't prove that this money was given to RAND on the tacit understanding that it would end up at METR, but it probably was. (Also, several prominent current METR employees previously held prominent roles at Coefficient Giving.)
Last, and, honestly, least? This is not real regulation. Real regulation is imposed by the government, and you cannot simply opt out of legally required oversight. METR is full of smart people, and they will be well aware that if they don't play nice they can simply be kicked out of Anthropic. Anthropic could at least have entered into some legal agreement with a real auditor which gave that auditor enforceable oversight powers beyond publication rights over them, but instead they chose this.
Dario is calling here for the AI industry to self-regulate. That is not, normally, a thing. The cases where societies allow industries to self-regulate are broadly those where regulation raises free speech questions, centered around art. The most famous examples of these are the age-related rating systems for movies, TV, music, and video games. These are industries that provide entertainment, not fundamental technology. Most importantly, none of these industries claim on a regular basis that their technologies threaten national security or human survival. Dario does.
How Did We Get Here
Why would anyone think that this should fly? If you said you had an independent auditor in a contract, and when I looked into it I found out that they got paid with the same money you did, lived in the same house you did, and one of their executives married your sister, you would go to prison. You would go to prison because if you don't send people to prison for fudging the independence of their auditors and business connections, they tend to end up like Bernie Madoff and Sam Bankman-Fried. AI is more important than the average business, and accountability should be more, and not less, important.
We live in a relatively high-trust society, where you can safely assume that a signed contract will be honored and your bank really does have your money, because we value impartiality and fair process, and we set high standards for what those are. Impartial application of the law is the absolute bedrock of modern civilization, and even the lowest criminal is entitled to a judge and a jury that does not know him. These rules and how we enforce them are what keeps our society honest and fair, and following them, as much as we do, has made us prosperous and safe.
Anthropic's company culture is apparently not a part of this tradition and does not believe in it. Anthropic doesn't think it should be regulated the way that banks, insurance companies or law firms would be, or that it should be required to seek impartial auditors to do security reviews of current practices or of past breaches. Close associations do not matter, because impartiality does not matter, and in fact, the more closely connected you are to Effective Altruism, Anthropic and Coefficient Giving, the more trust you deserve. This seems to be why Anthropic thinks that METR is, of course, very trustworthy, and why they have a recent history of giving other important security contracts to very incompetent, but very Effective Altruist, companies.
One of the aims of democratic society is to prevent you from needing to know the proclivities of particular individuals in the Bay Area. I would love it if we had a normal liberal-democratic process here, instead of one where I have to care about who is sharing funding, housing, and fluids in San Francisco and Berkeley.
METR, “Frontier Risk Report (February to March 2026)”, May 19, 2026. See the discussion of evaluator independence in Appendix A. ↩︎
Business Insider, reporting on METR’s staffing and the AI safety research talent shortage, August 2026. ↩︎
METR, “Funding update”, August 14, 2026. METR describes its funding policy and the free tokens supplied by frontier AI companies. ↩︎
METR, “Update on Security at METR”, August 31, 2026, “Incident 1.” METR reports that an attacker used a stolen API key over three weeks to consume credits worth approximately $600,000, supplied to METR for free by the model developer. METR also reports that the key had no spending limit at the time. ↩︎
Coefficient Giving, “Open Philanthropy Is Now Coefficient Giving”, describes its founding funding partnership with Good Ventures, the foundation of Cari Tuna and Dustin Moskovitz. Anthropic’s Series A announcement names Moskovitz as an investor; TIME’s profile of Tuna, September 5, 2024, also describes the couple’s 2021 investment in Anthropic. ↩︎
The Alignment Research Center grant records list Open Philanthropy general-support grants of $265,000 in March 2022 and $1.25 million in November 2022, with links to the original grant announcements. METR describes its origins in “ARC Evals is spinning out from ARC”, September 19, 2023. ↩︎
Grantmaking.ai records a $10 million Coefficient Giving grant to RAND for “AI Evaluation and Testing”, dated September 20, 2025, and links to the Audacious Project’s Canary collaboration between METR and RAND. METR’s “New Support Through The Audacious Project”, October 9, 2024, reports approximately $38 million for Canary, with approximately $17 million supporting METR’s work. ↩︎