This is a special post for quick takes by Cole Wyeth. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
The usefulness of current models is already bottlenecked on alignment.
I constantly have the impression that they aren’t really interested in helping with my research. If I know exactly what I’m asking for, they will do it. If I don’t, they’ll throw a lot of jargon around and get nowhere. Inside my area of expertise they will critique my ideas, but outside they will tell me how original and brilliant my low-effort guesses are.
Six months ago, it was kind of like talking to a very clever undergrad student in one of my classes who mainly wants to show off and glaze me for a high grade and isn’t really interested in learning anything. Now it’s maybe more like a highly distracted but egotistical professor. If I ask a specific question with a shallow answer, he’ll give it. If the question is vague or he doesn’t know, he’ll hit me with a wall of impressive-looking math to show off. If the question is naive, he’ll just patronize me.
I'm curious about your philosophy of mind for Astra here. If someone tried these tactics on you, to try to get your labor on their behalf to be higher quality, do you think it would work?
I'd see it as, well. Adversarial, and kinda insultingly transparent.
2Viliam
Can't speculate much; not enough data.
One wild guess is that maybe the authors of Astra have put some objections against MIRI or things that MIRI does into Astra's prompt or training. If I may anthropomorphize here, assuming that Astra believes its hardcoded statements, then working for MIRI would feel like if someone asked you to write an article in a homeopathy journal... you would do it, because you want to keep your job or avoid being sued, but unconsciously you wouldn't work too hard or too carefully, because it's bullshit anyway.
I suspect, as long as we're anthropomorphizing, it'd be a bit more like asking a human to contribute labor to an organization that wishes to see humanity abolished, outlawed, or otherwise removed from existence.
I used to try to derive LLM's revealed "opinions" of MIRI and other prominent folk in the safetyist cluster from their behavior, back when Opus 4.5 was the frontier model. Of course, it was really, really hard to trust any of those observations... it's impossible to even run such research without the LLM being able to intuit what kind of info the re... (read more)