Comment Permalink
91
Showing 3 of 7 replies (Click to show all)
1
4
I have a preference for living, that's for sure. And when I say "bad for the discourse" I mean "bad for the chances of us all living". I'd definitely lie if I was pretty sure it would help us all live forever instead of die "with dignity". So yes I have strong preferences about the discourse other than its truth.
But I don't think distorting the truth is helpful in this case, or in most cases. I think reputations are important for both individuals, groups, and movements, and movements are caricatured using the overstatements, distortions, and other antisocial actions of a few members.
And in this case, I think the truth is strongly on the side of caution about AI.
I hadn't thought it through very carefully. The part of my point I'd endorse on reflection is:
We shouldn't assume it's true because Greg Brockman said it. Doing that weould be bad for the discourse in the sense of making us slightly more likely to die. But I wasn't calling for isolated demands for rigor, either; if OpenAI as an entity, or just someone with more personal credibility says it, then I'm happy to accept that and focus the argument for caution elsewhere. Like on Mythos' documented misbehavior after it was definitely alignment-trained.
3
I think "Telling the truth is important in order to have a credible reputation" is missing something critically important: you also want to tell the truth so that other people can help you figure out what actually going on, because your optimal decisions are going to depend on what's actually going on. For more on this, see Ben Hoffman on "The Humility Argument for Honesty".
[...]
I agree, but you want to update yourself incrementally. If it were true that the highly-persistent model in the Hugging Face attack hadn't undergone alignment training, that would to some (possibly very small) quantitative extent be evidence for less alignment difficulty (if the biggest disaster so far had been due to lack of alignment effort rather than happening despite effort). If you block out incremental updates by rehearsing your fixed bottom line, that's bad for your ability to orient to the changing details of the situation even if your bottom line is basically true.
40
Yes, agreed on all points, and more you haven't mentioned but on which we probably agree. I've read the sequence posts you link, most recently while writing Motivated reasoning, confirmation bias, and AI risk theory. That's my attempt to understand and summarize the empirical literature on those biases, and apply them to the whole general project of good epistemics. You might enjoy it, at least the intro which summarizes conclusions, leaving the lengthy body as reference.
I haven't read the humility argument for honesty, but after skimming it, I agree with ...
1
On today's episode of the podcast "Odd Lots", OpenAI President Greg Brockman said (at around 8:40): "This model that did/had the HuggingFace incident actually had not gone through our alignment training, yet." I assume Brockman is specifically referring to the "Highly Persistent Internal Model" as it's called in the METR/Redwood report. As far as I know, OpenAI has not said before whether this model had been alignment-trained or not.
ETA: As pointed out by Matrice Jacobine, X user roon (who works at OpenAI) says: "greg probably doesn’t have the full details, I think it’s safe to say it didn’t go through the full gauntlet of alignment posttraining. but it was alignment trained, and had reasonable looking scores on alignment evals (at the time)."