The alignment of mundane, current AI systems is highly relevant for the alignment of superintelligence, if only because the explicit strategy of the model labs is to build a recursively self-improving AI, whose resultant superintelligence would obviously rely on how well that initial seed AI was aligned to human values.
This seems to refer to something quite late in the game, i.e. what people call "handoff". Sure, those AIs won't be superintelligent, and their alignment is highly relevant. But I don't think those models are "current". They seem a few generations later.
My position on this is:
There's some class of AIs for which (1) we need to mitigate the risks from those models to avoid a catastrophe, and (2) we need to elicit useful work from those models. You could call this "AIs we actually care about" or "mitigate-and-elicit AIs". We want to align these AIs because it helps with mitigation and elicitation.
And there's another class of AIs for which we are align for indirect reasons, e.g. because studying alignment of those AIs will be informative for aligning the mitigate-and-elicit AIs. You could call these "practice AIs".
I think it's entirely plausible that Mythos is a practice AI. Like, it's nice that Mythos is [somewhat] aligned, but it's not clear we'd be in a signifcantly worse position if Mythos was more misaligned. I definitely don't think the resulting superintelligence causally depends on Mythos being aligned.
I think the first mitigate-and-elicit AIs are probably in late 2027.
2Dean Valentine
There's no hard line. Current, mundane AIs are already being used to e.g. detect reward hacking in vendor RL environments. And as we scale to superintelligence the tasks & amount of trust we hand to AI models will ramp up.
Your argument was locally invalid. What you said after “if only because” doesn’t entail what you said before it, because current AIs are not the initial seed AI of an RSI to superintelligence.
I agree current AIs are used a bunch and it’s helpful if they are less misaligned.