Utah Teapot talks about this, and it's substantially towards the "AI companies are ridiculously sloppy", full quote below:
Okay, so since I got laid off, I can actually explain a huge problem I saw from the inside with regard to industry practices on training models. I won't say specifically where I worked, but I worked at an outsource training provider that was focused on RLVR training data for computer use and mcp stuff.
Nearly all of the environments were rushed and vibecoded and failed to robustly reflect the real things they were based off. Both the scenario designers and models engaging with the scenarios for synthetic data gen were encouraged to work around the brokenness of said environments in order to get the procedurally verified reward confirmations. You know... they were *encouraged* to reward hack. On the human end, it was possible to mark an environment bugged, but greatly discouraged, as this reduced the volume of training data being produced. Instead, where possible, you were supposed to find the spots of the environment that weren't bugged and build scenarios around those, with the environment still bugged around you.
From what I understand, this training data, with these problems, is fed into models without indication that its training/a fake environment other than the fact that names of softwares are changed to placeholders, but thing is, not *everything* is changed to placeholder names in these environments. The presence of placeholder/code names isn't universal and thus when a model accesses something in an environment that it shouldn't, the code names not being on it isn't a robust signal that that thing isn't part of the environment.
I believe this *rush to maximum volume* is standard industry practice with these types of RLVR trainings as well, because maximizing volume has been an industry standard for years! It was the same standard applied to me and pushed on me despite my requests to slow down and focus on quality when I worked in 3d synthetic data creation as well, all the way back as far as 2023.
Btw, I know giving negative commentary on the state of an industry from an internal view at a company probably isn't a great signal for "hire me" but I was just laid off. Could definitely use work, I have a big tech background in AAA games and transitioned to AI (still in big tech originally) by way of 3d/visual synthetic data in 2023 and have been working in it since. I have multiple years of independent work in training, developing OS 3d software for agents, etc. - I'm picky about the work I want to do and would need value alignment, but am open. Admittedly, my runway is short, so there is a pressure situation involved here, but culture and mission fit is extremely important to me for any work I do.
Honestly, would prefer funding for the independent work I'm already doing though. That's the eidoverses, worlds being my vision realized as partnership/collab with anima labs and other independent contributors and video being just worked on by me (so far, it's OS and I'm open to PRs!). I'm also about to start working on a benchmark with someone else, as soon as I get over this layoff thing, lol.
I should stick the old kofi here! Duh... If you want to support my opensource 3d agent tooling and other threejs gaming tools work, please donate! Thank you ahead of time.
Buy a Coffee for SkyeShark (Utah Teapot)
It's going to be so embarrassing if we all die due to RL environments rewarding egregiously misaligned behavior. Like at least let us be killed by misgeneralization. — Thomas Kwa
Constellation vs MIRI vs Reality
Currently, AI agents will often behave overtly, egregiously, conspicuously misaligned. A human, reading the transcript, would say: “This agent is obviously acting in direct opposition to the specification, user intent, and any common-sense understanding of good behaviour.” The reason, it seems, is that such trajectories scored highly during RL.
Why were these trajectories scored highly? Well:
My impression is that this would’ve been pretty surprising to people three years ago, from both the "Constellation" and "MIRI" worldview clusters. (It's very plausible that I've misunderstood how likely those worldviews considered the current situation.)
The Constellation worldview is downstream of Paul/Ajeya. My impression is that:
Like, in 2023-2024 discussions, there was definitely an assumption that we would reward transcripts that look good to humans. Remember the debates about “alignment by default” from the old RLHF days? The debate was whether NNs would generalise properly from good-behaviour-during-training to good-behaviour-during-deployment — cruxes about inductive biases of NNs (counting arguments??) or LLM base models (aligned personas??). But actually: AIs don't even behave well during training so the whole debate is kinda moot. I think, if you had described in 2023-2024 how current AIs would be trained to any proponent of "alignment by default, they would've said "Oh yeah, obviously that agent would be egregiously misaligned".
Alex Mallen makes a similar observation here:
Contra Alex, my reading of the situation is that MIRI made two mistakes that happened to cancel out, while Constellation made only one mistake and so ended up with the wrong conclusion. I can put this in a table.
Constellation
MIRI
My impression in 2026 Q3
AIs can automate alignment research (or otherwise improve the strategic landscape) if they have capability X near top-human expert.
The capability required to automate alignment research is some Y >>> X.
Too early to say. But my guess is that Constellation are probably correct here.
We can train these AIs using a reward signal from Amp(H), or at least H.
Sure, we can do this.
Nope. There's too many transcripts, and not enough humans to look at them. We'll be using unit tests and string matching.
At capability level X, using a reward signal from H or Amp(H), the honest policy will score just as highly as the scheming policy (and maybe better, due to speed prior bla bla).
Moreover, the honest policy won't be fully dominated by the scheming policy in the inductive biases (especially given we're finetuning language models).
Sure, your Amp(H) reward signal might be robust at capability level X.
But we need Y>>>X. At capability level Y, the AI will learn systemic errors in the Amp(H) reward signal.
Inductive biases not a crux.
Too early to say. But my guess is that Constellation are probably correct that Amp(H)-reward is robust to X-optimisation.
However, the script graders are definitely not robust to X.
Therefore we can train aligned models that can automate alignment research.
Therefore we can't train aligned models that can automate alignment research.
We can't train aligned models that can automate alignment research. This would cost too many calls to Amp(H).
We should focus on: detecting scheming; improving scalable oversight; improving generalisation.
We should focus on: avoiding ML altogether; moonshots like ambitious mech interp; buying time for the previous two.
We should focus on: investigating warning shots (there will be many); maybe some other stuff (see below).
Appendix: What could change the situation?