O O's Shortform

O O

O O's Shortform — LessWrong

Comment Permalink

O O2y2916

Talk through the grapevine:

Safety is implemented in a highly idiotic way in non frontier but well-funded labs (and possibly in frontier ones too?).

Think raising a firestorm over a 10th leading mini LLM being potentially jailbroken.

The effect is employees get mildly disillusioned with saftey-ism, and it gets seen as unserious. There should have been a hard distinction between existential risks and standard corporate censorship. "Notkilleveryoneism" is simply too ridiculous sounding to spread. But maybe memetic selection pressures make it impossible for the irrelevant version of safety to not dominate.

Showing 3 of 14 replies (Click to show all)

Ivan Vendrov2y51

definitely agree there's some power-seeking equivocation going on, but wanted to offer a less sinister explanation from my experiences in AI research contexts. Seems that a lot of equivocation and blurring of boundaries comes from people trying to work on concrete problems and obtain empirical information. a thought process like

alignment seems maybe important?
ok what experiment can I set up that lets me test some hypotheses
can't really test the long-term harms directly, let me test an analogue in a toy environment or on a small model, publish results
when t... (read more)

1Michael Roe2y

Yeah, many of the issues are the same: *RLHF can be jail broken with prompts, so you can get it to tell you a sexy story or a recipe for methamphetamine. If we ever get to a point where LLMs know truly dangerous things, they'll tell you those, too. *Open source weights are fundamentally insecure, because you can finetune out the guardrails. Sexy stories, meth, or whatever. The good thing about the War on Horny * probably doesnt really matter, so not much harm done when people get LLMx to write porn * Turns out, lots of people want to read porn (surprise! who would have guessed?) so there are lots of attackers trying to bypass the guardrails * This gives us good advance warning that the guardrails are worthless

5Viliam2y

People seeing AI-generated boobs is an x-risk. God might get angry and send another flood. More seriously, is this worse than the usual IT security? The average corporate firewall blocks porn, online games, and hate speech, even if that has nothing to do with security per se (i.e. computers getting hacked, sensitive data stolen). Also, many security rules get adopted not because they make sense, but because "other companies do it, so if we don't, we might get in trouble for not following the industry best practices" and "someone from the security department got drunk and proposed it, but if something bad happens one day, I don't want to have my name on the record as the manager who opposed a security measure recommended by a security expert".

See in context