Slightly playing Devil’s advocate:

We have a proto-AGI that is aligned

Or it’s misaligned and pretending. All this requires is being situationally aware enough to know when it can and can’t be lazy / cheating / deceptive, i.e. for it to guess at which tasks you can and can’t verify the outputs to.

It's incredibly unlikely it will sabotage alignment work or future training runs, except for lack of understanding/competence.

I think this implicitly assumes that alignment is both solvable and not that difficult to solve.

on current trajectory, there's a decent chance that soon models will actually have good taste and judgement (as this was capability bottlenecked).

Hang on, why exactly do you think this? Research taste definitely is capability-bottlenecked in a strict sense, but I think most recent LLM achievements (e.g. Navier-Stokes) don’t demonstrate huge improvements in research taste. AFAICT most frontier companies are still relying heavily on RLVR and transfer to non-verifiable domains isn’t that high.