This is a special post for quick takes by oligo. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
So, one classical dilemma of the "AI for AI alignment" is, you're using Opus 6 (which is let's say is aligned) to train Opus 7 (which is smarter than you or Opus 6.)
I wonder if inference scaling offers a way around this? If Opus 6 gets economically implausible compute resources to spend on its monitoring 7, it can be smarter than 7 in practice by thinking for longer. Then use the same trick with 7 to train 8, and so on.
There are many obvious holes here, first being that you could have a treacherous turn based on compute availability, and so on, but maybe someone smarter can turn this into something useful (or already thought this through and discarded it.)
Agent-3 is misaligned before developing goals aside from optimizing reward. Since Agent-3 isn't rebelling, the monitoring of Agent-3 is useless and the monitoring of Agent-4 is trusted to Agent-3 who is embarassed to admit Agent-4's misalignment and/or has too little compute, is overconfident in its research, etc.
Returning to compute, the amount of internal compute spent on internal agents is 6%, while the entire share of compute spent on alignment is 3% of 63%, i.e. about thrice less.