This is a special post for quick takes by jsd. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
When talking about AI risk from LLM-like models, when using the word "AI" please make it clear whether you are referring to:
A model
An instance of a model, given a prompt
For example, there's a big difference between claiming that a model is goal-directed and claiming that a particular instance of a model given a prompt is goal-directed.
I think this distinction is obvious and important but too rarely made explicit.
For goal-directedness, I'd interpret it as "all instances are goal-directed and share the same goal".
As an example, I wish Without specific countermeasures had made the distinction more explicit.
More generally, when discussing whether a model is scheming, I think it's useful to keep in mind worlds where some instances of the model scheme while others don't.
2Dagon
I don't think I've seen any research about cross-instance similarity, or even measuring the impact of instance-differences (including context and prompts) on strategic/goal-oriented actions. It's an interesting question, but IMO not as interesting as "if instances are created/selected for their ability to make and execute long-term plans, how do those instances behave".
How would you say humanity does on this distinction? When we talk about planning and goals, how often are we talking about "all humans", vs "representative instances"?
Mostly I care about this because if there's a small number of instances that are trying to take over, but a lot of equally powerful instances that are trying to help you, this makes a big difference. My best guess is that we'll be in roughly this situation for "near-human-level" systems.
I don't think I've seen any research about cross-instance similarity
How would you say humanity does on this distinction? When we talk about planning and goals, how often are we talking about "all humans", vs "repres... (read more)